OTEL Monitor

[ddrcode]

Status

• Concept approved
• ready to be implemented, although it doesn't specify precisely what traces and metrics it should be implementing. Start simple, define foundation, discover on the go.
• contributors welcome

Description

Define monitor that produces OpenTelemetry traces and metrics for tracking Maiko performance.

Ideas

Automatic metrics

• actor events counter - number of events sent from an actor and received by it
• topic events counter - number of events per topic
• delivery time - histogram - time between event send and receive
• step execution time - per actor
• errors counter - per actor
• overflow metrics - number of events dropped/blocked (per actor) or actors failed due to overflow

Configurable metrics

• time between two correlated events specified by user

Contributing

If you know OTEL well, then implementing it should be straight-forward. The only thing it requires is custom implementation of Monitor trait. If interested - you are very welcome to contribute.

Implementation

• As it requires extra dependencies (opentelemetry crate), define otel feature for it
• Avoid creating any specific OTEL connections and configuration - assume that happen outside of Maiko
• initialize with opentelemetry::metrics::Meter instance and use it for sending metrics. Analogically for traces.
• more advanced (likely - out of scope) - span_id could be added to Meta for additional correlation