Skip to content

use wrpkafka library to add stream for events of type wrp.SchemeEvent #561

@piccione99

Description

@piccione99

library to use: https://github.com/xmidt-org/wrpkafka/tree/main

above library requires go.1.21+ and talaria is at go.1.24

all wrp.SchemeEvents going to caduceus end up in this method:

https://github.com/xmidt-org/talaria/blob/main/eventDispatcher.go#L224

We want to take those same events bound for caduceus and also stream them to kafka at the same time.

The wrpkafka library expects a wrp.Message so we would have to Decode these messages first before sending.

If there is no kafka configuration set, this additional code will do nothing.

For production, we still need to determine how big to set the kafka buffers , max records before flushing etc... (it depends on the receiving stream) and what auth to use.

Talaria does not use uber.fx, so the singleton will need to be manually configured in main.go. Telemetry needs to be passed into the library. TODO - how to use metrics provider with library.

Metadata

Metadata

Assignees

Labels

kafkaadd kafka streams to xmidt

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions