From 6d9f8eb9de452513c00f193af7f190a3c57ee2b9 Mon Sep 17 00:00:00 2001 From: Sam Yuan Date: Mon, 6 Dec 2021 20:35:02 +0800 Subject: [PATCH 1/3] update open tracing rfc Signed-off-by: Sam Yuan --- text/0000-opentelemetry-tracing.md | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/text/0000-opentelemetry-tracing.md b/text/0000-opentelemetry-tracing.md index 0f2d17e..a288905 100644 --- a/text/0000-opentelemetry-tracing.md +++ b/text/0000-opentelemetry-tracing.md @@ -6,6 +6,7 @@ nav_order: 3 - Feature Name: OpenTelemetry Tracing Integration - Start Date: 2021-01-12 +- Update Date: 2021-12-06 - RFC PR: (leave this empty) - Fabric Component: core, sdks - Fabric Issue: (leave this empty) @@ -70,9 +71,31 @@ The trace and span ID must be sent to the chain as a message header. The SDK should use the standard environment variable environment to let users define how and if they want to report trace data to an endpoint of their choosing. +To find bottleneck for fabric processing. At SDK side, we can add open tracing base on txid. Not only at GRPC side, but also each tx processing, a sample here: https://github.com/Hyperledger-TWGC/tape/tree/alpha, looks like: + +![A sample for full id tracing](https://user-images.githubusercontent.com/7820992/141783841-a4cee4c5-3275-4b58-b2f6-0c34e37d6e6f.png) + ## Changes to peers and orderers -Peers and orderers capture and propagate trace information using an optional gRPC metadata header, if enabled. +Peers and orderers capture and propagate trace information using an optional gRPC metadata header, if enabled. + +To find bottleneck for fabric processing, a sample here: https://github.com/SamYuan1990/fabric/tree/opentracing23 if we enable open tracing for peer and orderer it looks like: +![A sample for peer commit phase](https://user-images.githubusercontent.com/7820992/144846677-6044b69f-d72b-490f-bcf6-0d161a7c1755.png) + +### Peer +Open tracing at peer nodes can be discussed in two different point of views. + +- From the workflow point of view: +1. Endorsement considering, basing on txid and tracing. The endorsement interface process tx via txid as unique identifier, we can easily apply with open tracing. +1. Commit considering, basing on block processing. From the workflow point of considering, it's better to tracing block for each channel. +1. Gossip considering, basing on block processing? + +- From business point of view: tx id for transaction only. + +### Orderer +Envelopes and consensus phase, as orderer's interface with line: +https://github.com/hyperledger/fabric/blob/main/orderer/consensus/consensus.go#L63 +we can tracing envelops in two phases, business envelops process and consensus at fundation level. # Drawbacks [drawbacks]: #drawbacks @@ -82,6 +105,11 @@ OpenTelemetry is still relatively young, yet has reached maturity for traces sup The OpenTelemetry reporting system happens securely over Protobuf, with containers and client applications sending data. This requires that an OpenTelemetry-compatible endpoint is present to receive the data. +For ex, +- At SDK side, to process with open telemetry for each txid. We need analysis blocks and it will make additional cost. + +- Together with opentelemetry which will adding more effort on operation and monitoring/integeration works. + # Rationale and alternatives [alternatives]: #alternatives From 96e81c7a9643ae532ed312e00ec9f7da4f88f2fa Mon Sep 17 00:00:00 2001 From: Sam Yuan Date: Sun, 23 Jan 2022 14:54:03 +0800 Subject: [PATCH 2/3] update with sample usage Signed-off-by: Sam Yuan --- text/0000-opentelemetry-tracing.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/text/0000-opentelemetry-tracing.md b/text/0000-opentelemetry-tracing.md index a288905..b39ae6b 100644 --- a/text/0000-opentelemetry-tracing.md +++ b/text/0000-opentelemetry-tracing.md @@ -95,7 +95,17 @@ Open tracing at peer nodes can be discussed in two different point of views. ### Orderer Envelopes and consensus phase, as orderer's interface with line: https://github.com/hyperledger/fabric/blob/main/orderer/consensus/consensus.go#L63 -we can tracing envelops in two phases, business envelops process and consensus at fundation level. +we can tracing envelops in two phases, business envelops process or consensus at fundation level. +- From business point of view: tx id for transaction only. + +### Usage: +as screen shot as +![txid based tracing](https://user-images.githubusercontent.com/7820992/150667822-9dfc93f7-2757-4008-8db1-0d4d589f3c7d.png) +we are able to use +``` +txid=${txid} +``` +to have a end to end tracing for specific transaction. # Drawbacks [drawbacks]: #drawbacks From 7d28e63f4ac05bc84af263631634344b144b8eec Mon Sep 17 00:00:00 2001 From: Sam Yuan Date: Tue, 8 Feb 2022 10:08:57 +0800 Subject: [PATCH 3/3] fix up, adding comments in PR review Signed-off-by: Sam Yuan --- text/0000-opentelemetry-tracing.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/text/0000-opentelemetry-tracing.md b/text/0000-opentelemetry-tracing.md index b39ae6b..6ff0bba 100644 --- a/text/0000-opentelemetry-tracing.md +++ b/text/0000-opentelemetry-tracing.md @@ -76,12 +76,17 @@ To find bottleneck for fabric processing. At SDK side, we can add open tracing b ![A sample for full id tracing](https://user-images.githubusercontent.com/7820992/141783841-a4cee4c5-3275-4b58-b2f6-0c34e37d6e6f.png) ## Changes to peers and orderers - Peers and orderers capture and propagate trace information using an optional gRPC metadata header, if enabled. To find bottleneck for fabric processing, a sample here: https://github.com/SamYuan1990/fabric/tree/opentracing23 if we enable open tracing for peer and orderer it looks like: ![A sample for peer commit phase](https://user-images.githubusercontent.com/7820992/144846677-6044b69f-d72b-490f-bcf6-0d161a7c1755.png) +Reminder, OpenTelemetry/OpenTracing have created systems to coordinate and correlate traces based on trace ID/parent trace ID/span ID, and are better suited for correlation. +The txid can be added to the trace as an attribute, but is not a valid identifier as the trace can originate outside Fabric. +As we have two hash exists(one generated by OpenTelemetry and another one is txid). It’s best to leave the id of the trace or span to the system generating the trace, especially as traces will not just originate from Fabric. +We are able to put txid as business hash value as an attribute, is there any way for us to search the span id by attribute value. +Then we are able to use txid as attribute to search in OpenTelemetry. For details see [Usage](###Usage). + ### Peer Open tracing at peer nodes can be discussed in two different point of views. @@ -99,13 +104,17 @@ we can tracing envelops in two phases, business envelops process or consensus at - From business point of view: tx id for transaction only. ### Usage: -as screen shot as +As screen shot below ![txid based tracing](https://user-images.githubusercontent.com/7820992/150667822-9dfc93f7-2757-4008-8db1-0d4d589f3c7d.png) we are able to use ``` txid=${txid} ``` -to have a end to end tracing for specific transaction. +as condition for our search among OpenTelemetry system. For ex, we searched value for specific txid with value below: +``` +txid=31a230b23c67f854ec4e9505283270055e16259d0a07e210f27cb8decb64d086 +``` +and from result, we get both endorsement and commit phase time usage/span from OpenTelemetry. # Drawbacks [drawbacks]: #drawbacks