Backpressure handling - ideation

[ddrcode]

I would appreciate some ideas and concepts how to implement backpressure handling in Maiko. With focus on initial shape of it (planned for version 0.3.0), but some longer term, strategic planning is welcome. Please share any suggestion you have in mind.

Things to discuss (as example)

• per actor vs global handling,
• API
• focus on backpressure or actually backpressure prevention (i.e. degrade system - lower priority actors or topics - on 70% capacity). Or both?
• good use cases for testing

Cheers.

[ddrcode]

To start with something, I thought about simple OverflowPolicy defined per actor:

enum OverflowPolicy {
   Fail, // fail on buffer full
   Drop, // drop messages when full, 
   Block // system blocks, until receiver consumes event 
}

// registering an actor with a specific policy and channel size:
supervisor.add_actor(
    "Zebra", 
    Zebra::new, 
    Subscribe::to([Topic1, Topic2]).overflow(Block).capacity(666) )?;

Currently there is a single config param defining channel size for all actors. With the method above it's possible to define buffer size per actor.

OverflowPolicy::Block is a special case, that should be used sparingly. Scenario - imagine two actors FileReader and FileWriter. Reader's speed is fully determined by the Writer - if we read faster, we overflow. In such case defining Writer as Block, would make event broker (and Reader in consequence) wait until writer releases some capacity.

It would be natural to add something like BlockWithTimeout(Duration), but how the system should behave after the timeout? Drop? Fail? Make it configurable (BlockWithTimeout(Duration, Box<OverflowPolicy>))? Doing that we will quickly step into building some form of rule engine.

Thoughts?

[serle]

If the consumers each had different OverflowPolicies you would need to ensure that the broker is not held up for everyone when it is being asked to block on a given consumer. The solution would then be to have the broker spawn a runner task per consumer so the overflow specification, at least in terms of fail and drop would be respected. For overflow::block there is a second order problem (as you mentioned) in that to implement perfect block the runner would need an unbounded buffer or you could get around that with a block until max timeout or number of messages which is more explicit about the semantics your can physically support. More

Fundamentally if backpressure is defined properly, ultimately the impact propagates backward to the actual producer, not just the broker in the middle, and when the producer makes the corresponding adjustment then the slowest consumer with a blocking policy will have inadvertently propagated its limitations to all the other topic consumers, so the whole decoupled nature of the abstraction then starts to breakdown. In this circumstance you could think about producer guarantees e.g. I block on my slowest blocking consumer, I never block etc and then at subscription time you could have some kind of policy compatibility evaluation.

[WojtekRewaj]

Some loose thoughts:
1. Actor/Topic priorities : the boker could drop events for low-priority topics/actors, preserving capacity for high-priority ones
2. Watermarks/tresholds : For example, once an actor hits a 80% threshold, the broker could start applying backpressure (blocking or dropping) to prevent a hard stall of the system
3. Circuit breaker for "slow" actors : If an actor is consistently lagging and blocking the broker, broker should cut its subscription for some time.

[ddrcode]

Thanks for sharing your comments and ideas @serle and @WojtekRewaj. Below are my first impressions after reading the above.

To clarify something - the supervisor actually spawns runner task per actor (see ActorController), so we can introduce granular control on producer/consumer level.

In both comments, directly or indirectly, you pointed into prevention as well as the actual backpressure handling. Prevention is an interesting topic - actors (or topics, or events) could have priority flags. And the system, under heavy load, could degrade the low-priority entities in order to prevent the performance of the important bits.

Propagating the impact (of backpressure) to relevant producers is an interesting concept. Do you have some idea how to assure that, @serle ? I can guarantee that in my proposed Blocked policy, but at the cost of slowing down the entire system. In multi-actor solution this is a high price to pay.

One of the policies that could be very helpful, but harder to implement, would be DropOldest. It fits into Drop domain, but we prioritize the delivery of the latest events, while allowing to drop the oldest ones. That would be a good solution to all stock tick-like scenarios. If the receiver can't keep up with all events, we prioritize the latest ones. Almost like watch channel, but with n-values. But not sure if such channels even exist, or we would need to build dequeue-based buffer for it.

No solutions yet, just thoughts and exploration. What do you think?

[ddrcode]

Ok, the initial version is under development (see #69), but the discussion is till open, as the first version will just cover simple backpressure handling.

Decisions made so far:

• The OverflowPolicy with Drop, Fail, Block options, as proposed above.
• The policy applies on Topic level rather than actor level. A single policy per actor wouldn't make sense, as a single actor an subscribe to many different topis.
• As the policy is on topic side, and actors subscribe to topics (what they read), in practice the policy controls the consumer side.
• Policies apply only when Broker->Actor channel is full. It means that Block policy doesn't block all the time, but only during the congestion
• There is possible to define channel/buffer size per actor (Broker->Actor channel),
• The size of Actors->Broker channel size should be configurable in global config.
• Context to provide is_sender_full method, to allow producers have control over the situation.

Things that won't be implemented in the first iteration, but still worth to be considered:

• multiple brokers (defined per topic groups) - high priority. The current implementation assumes that single broker solution is a special case for multi-broker one.
• priorities (events, actors, topics?)
• prevention: quotas, dynamic throughput monitoring and speed control (limiting aggressive producers)
• circuit breakers

Open questions

• try_send on actor side