Adding Fanout node #636

nirandaperera · 2025-11-05T22:20:05Z

This PR adds Fanout Node.

/**
 * @brief Fanout policy controlling how messages are propagated.
 */
enum class FanoutPolicy : uint8_t {
    /**
     * @brief Process messages as they arrive and immediately forward them.
     *
     * Messages are forwarded as soon as they are received from the input channel.
     * The next message is not processed until all output channels have completed
     * sending the current one, ensuring backpressure and synchronized flow.
     */
    BOUNDED,

    /**
     * @brief Forward messages without enforcing backpressure.
     *
     * In this mode, messages may be accumulated internally before being
     * broadcast, or they may be forwarded immediately depending on the
     * implementation and downstream consumption rate.
     *
     * This mode disables coordinated backpressure between outputs, allowing
     * consumers to process at independent rates, but can lead to unbounded
     * buffering and increased memory usage.
     *
     * @note Consumers might not receive any messages until *all* upstream
     * messages have been sent, depending on the implementation and buffering
     * strategy.
     */
    UNBOUNDED,
};

/**
 * @brief Broadcast messages from one input channel to multiple output channels.
 *
 * The node continuously receives messages from the input channel and forwards
 * them to all output channels according to the selected fanout policy, see
 * ::FanoutPolicy.
 *
 * Each output channel receives a shallow copy of the same message; no payload
 * data is duplicated. All copies share the same underlying payload, ensuring
 * zero-copy broadcast semantics.
 *
 * @param ctx The node context to use.
 * @param ch_in Input channel from which messages are received.
 * @param chs_out Output channels to which messages are broadcast.
 * @param policy The fanout strategy to use (see ::FanoutPolicy).
 *
 * @return Streaming node representing the fanout operation.
 *
 * @throws std::invalid_argument If an unknown fanout policy is specified.
 *
 * @note Since messages are shallow-copied, releasing a payload (`release<T>()`)
 * is only valid on messages that hold exclusive ownership of the payload.
 */
Node fanout(
    std::shared_ptr<Context> ctx,
    std::shared_ptr<Channel> ch_in,
    std::vector<std::shared_ptr<Channel>> chs_out,
    FanoutPolicy policy
);

Depends on #648

Closes #560

Signed-off-by: niranda perera <niranda.perera@gmail.com>

copy-pr-bot · 2025-11-05T22:20:09Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: niranda perera <niranda.perera@gmail.com>

cpp/include/rapidsmpf/streaming/core/fanout.hpp

cpp/src/streaming/core/channel.cpp

cpp/src/streaming/core/fanout.cpp

Co-authored-by: Mads R. B. Kristensen <madsbk@gmail.com>

wence-

I wonder if there is a much simpler implementation lurking here.

cpp/src/streaming/core/channel.cpp

cpp/src/streaming/core/fanout.cpp

wence- · 2025-11-10T10:53:23Z

cpp/src/streaming/core/fanout.cpp

+            RAPIDSMPF_EXPECTS(
+                co_await ch_out->send(msg.copy(res)), "failed to send message"
+            );


This shouldn't be an error I think. Consider the case where the consumer has "seen enough". It wants to shut down the input channel to signal to the producer "I don't need any more inputs". The producer task should then exit.

Oh okay. That's a good point. I didnt think about it. This could make purging messages a little complicated. But let me try this out

Let's do it in a followup.

Ah! I added this capability now.

cpp/src/streaming/core/fanout.cpp

wence- · 2025-11-10T11:08:29Z

python/rapidsmpf/rapidsmpf/streaming/core/fanout.pyx

+class FanoutPolicy(IntEnum):
+    """
+    Fanout policy controlling how messages are propagated.
+
+    Attributes
+    ----------
+    BOUNDED : int
+        Process messages as they arrive and immediately forward them.
+        Messages are forwarded as soon as they are received from the input channel.
+        The next message is not processed until all output channels have completed
+        sending the current one, ensuring backpressure and synchronized flow.
+    UNBOUNDED : int
+        Forward messages without enforcing backpressure.
+        In this mode, messages may be accumulated internally before being
+        broadcast, or they may be forwarded immediately depending on the
+        implementation and downstream consumption rate.
+
+        This mode disables coordinated backpressure between outputs, allowing
+        consumers to process at independent rates, but can lead to unbounded
+        buffering and increased memory usage.
+
+        Note: Consumers might not receive any messages until *all* upstream
+        messages have been sent, depending on the implementation and buffering
+        strategy.
+    """
+    BOUNDED = <uint8_t>cpp_FanoutPolicy.BOUNDED
+    UNBOUNDED = <uint8_t>cpp_FanoutPolicy.UNBOUNDED


Just reimport the C++ enum.

from rapidsmpf.streaming.core.fanout import FanoutPolicy

I think.

wence- · 2025-11-10T11:09:45Z

python/rapidsmpf/rapidsmpf/streaming/core/fanout.pyx

+    # Validate policy
+    if not isinstance(policy, (FanoutPolicy, int)):
+        raise TypeError(f"policy must be a FanoutPolicy enum value, got {type(policy)}")


Can we only accept FanoutPolicy as the type?

I tried that, and cython was complaining that FanoutPolicy enum is not a type identifier

You need to cimport it as well.

wence- · 2025-11-10T11:10:47Z

python/rapidsmpf/rapidsmpf/streaming/core/fanout.pyx

+        owner.append(ch_out)
+        _chs_out.push_back((<Channel>ch_out)._handle)
+
+    cdef cpp_FanoutPolicy _policy = <cpp_FanoutPolicy>(<int>policy)


If you8 use the re-exported cdef enum this is unnecessary. Check some of the pylibcudf cython code to see how it's done there.

Signed-off-by: niranda perera <niranda.perera@gmail.com>

… fanout-node

Signed-off-by: niranda perera <niranda.perera@gmail.com>

This reverts commit d99d7c4.

This reverts commit ea945ca.

Signed-off-by: niranda perera <niranda.perera@gmail.com>

wence- · 2025-11-12T09:19:53Z

cpp/src/streaming/core/fanout.cpp

+    for (auto& ch_out : chs_out) {
+        // do a reservation for each copy, so that it will fallback to host memory if
+        // needed
+        // TODO: change this
+        auto res = ctx->br()->reserve_or_fail(msg.copy_cost(), try_memory_types(msg)[0]);
+        tasks.push_back(ch_out->send(msg.copy(res)));


We own (or should own) the message, so one of these copies is redundant (the "last" output channel can just take ownership of the input message).

Good point. Let me move the msg to the last channel.

@wence- I've addressed this now

wence- · 2025-11-12T09:23:59Z

cpp/src/streaming/core/fanout.cpp

+        // intentionally not locking the mtx here, because we only need to know a
+        // lower-bound on the last completed idx (ch_next_idx values are monotonically
+        // increasing)
+        size_t last_completed_idx = std::ranges::min(state.ch_next_idx);


This is UB, because another thread (one of the consumers) might be updating an entry in state.ch_next_idx simultaneously.

Yes, but its strictly increasing. And we only need an approximate value here. So, a current running min may not be the exact min in std::ranges::min, but it will be strictly less than or equal. Consequence would be, not cleaning up all the finished messages. But I felt it was a better trade-off than trying to relock the mutex.

Hmm. Thinking about this again, maybe I can do a ranges::min_max during request_data.wait, and purge until the min value. That would eliminate this.

I've removed this now

cpp/src/streaming/core/fanout.cpp

wence- · 2025-11-12T09:40:37Z

cpp/src/streaming/core/fanout.cpp

+    // start send tasks for each output channel
+    coro::task_container<coro::thread_pool> tasks(ctx->executor());
+    for (size_t i = 0; i < chs_out.size(); i++) {
+        RAPIDSMPF_EXPECTS(
+            tasks.start(unbounded_fo_send_task(*ctx, i, chs_out[i], state)),
+            "failed to start send task"
+        );
+    }


Do we actually need a task container here, or can in "process inputs" loop be a task as well, and we instead do:

std:vector<...> tasks = {process_inputs(), unbounded_fo_send_task(...), ...}; coro_results(co_await coro::when_all(std::move(tasks)));

With appropriate schedule of the tasks we're running.

wence- · 2025-11-12T09:53:32Z

cpp/src/streaming/core/fanout.cpp

+                return std::ranges::any_of(state.ch_next_idx, [&](size_t next_idx) {
+                    return state.recv_messages.size() == next_idx;
+                });


Is this more complicated than it needs to be?

I think this is "just" a boolean flag for "at least one of the consumers wants more data".

Could we have such a flag in the state struct that is updated when a consumer is ready and the flipped back to false here?

I think then the ch_next_idx of each consumer doesn't need to be part of the state of the struct, it's a local piece of information rather than a signalling mechanism.

Yes, that's a good point. The only issue would be, cleaning up finished messages. Currently I use the ch_next_idx to find the boundary.

The cost of this test is O(n_out_channels) which would be fairly small IMO.

cpp/src/streaming/core/fanout.cpp

… fanout-node Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera · 2025-11-20T21:36:05Z

@wence- @madsbk I think I addressed all the comments now. Could you please take another look?

Signed-off-by: niranda perera <niranda.perera@gmail.com>

python/rapidsmpf/rapidsmpf/streaming/core/fanout.pyx

cpp/src/streaming/core/fanout.cpp

wence- · 2025-11-24T15:06:35Z

cpp/src/streaming/core/fanout.cpp

+                } else {
+                    // request more data from the input channel
+                    lock.unlock();
+                    co_await request_data.notify_one();


I was reminded what was bugging me here. If there are multiple tasks running notify_one (or notify_all) on the same condvar, we can have lost wakeups unfortunately. I have tried to fix this in libcoro here: jbaldwin/libcoro#416

Oh! 🙁 I think your rationale in the jbaldwin/libcoro#416 makes sense. Should hold this PR off until jbaldwin/libcoro#416 merges? I did a bunch of repeat test runs, but didnt encounter a hang yet though.

I think the reason for that is that these tasks are typically not all firing at once. I think we are probably OK to merge now, but keep that in mind in case we see issues.

Co-authored-by: Lawrence Mitchell <wence@gmx.li>

python/rapidsmpf/rapidsmpf/streaming/core/fanout.pyx

cpp/src/streaming/core/fanout.cpp

Signed-off-by: niranda perera <niranda.perera@gmail.com>

cpp/src/streaming/core/fanout.cpp

python/rapidsmpf/rapidsmpf/streaming/core/fanout.pyx

Co-authored-by: Mads R. B. Kristensen <madsbk@gmail.com>

Signed-off-by: niranda perera <niranda.perera@gmail.com>

madsbk

LGTM, thanks @nirandaperera

nirandaperera · 2025-12-01T18:42:30Z

/merge

nirandaperera · 2025-12-02T17:55:11Z

/merge

nirandaperera added 5 commits November 3, 2025 15:57

porting code from mads' PR

feb5b59

Signed-off-by: niranda perera <niranda.perera@gmail.com>

WIP

6ea737c

Signed-off-by: niranda perera <niranda.perera@gmail.com>

adding fanout

a7254ba

Signed-off-by: niranda perera <niranda.perera@gmail.com>

Merge branch 'main' of github.com:rapidsai/rapidsmpf into fanout-node

8b97e94

Signed-off-by: niranda perera <niranda.perera@gmail.com>

working draft

91bd6c7

Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera added 3 commits November 5, 2025 16:55

adding more tests

a4810a1

Signed-off-by: niranda perera <niranda.perera@gmail.com>

adding more tests

0c5e342

Signed-off-by: niranda perera <niranda.perera@gmail.com>

extending tests

8081332

Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera marked this pull request as ready for review November 7, 2025 20:13

nirandaperera requested review from a team as code owners November 7, 2025 20:13

nirandaperera added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Nov 7, 2025

nirandaperera added 3 commits November 7, 2025 12:23

Merge branch 'main' of github.com:rapidsai/rapidsmpf into fanout-node

9217761

minor changes

37ea50d

Signed-off-by: niranda perera <niranda.perera@gmail.com>

add python bindings

ff962c7

Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera requested a review from a team as a code owner November 7, 2025 22:29

precommit

0c7d6a3

Signed-off-by: niranda perera <niranda.perera@gmail.com>

madsbk reviewed Nov 9, 2025

View reviewed changes

Update cpp/src/streaming/core/fanout.cpp

3b124ca

Co-authored-by: Mads R. B. Kristensen <madsbk@gmail.com>

wence- requested changes Nov 10, 2025

View reviewed changes

nirandaperera added 7 commits November 10, 2025 11:25

adding lower mem types

ea945ca

Signed-off-by: niranda perera <niranda.perera@gmail.com>

Merge branch 'fanout-node' of github.com:nirandaperera/rapidsmpf into…

6e90bf8

… fanout-node

Merge branch 'main' of github.com:rapidsai/rapidsmpf into fanout-node

cc92a82

remove size checkl

d99d7c4

Signed-off-by: niranda perera <niranda.perera@gmail.com>

Revert "remove size checkl"

dccd9dd

This reverts commit d99d7c4.

Revert "adding lower mem types"

33eafbc

This reverts commit ea945ca.

addressing comments

a3752ca

Signed-off-by: niranda perera <niranda.perera@gmail.com>

wence- reviewed Nov 12, 2025

View reviewed changes

nirandaperera requested review from madsbk and wence- November 20, 2025 20:47

Merge branch 'fanout-node' of github.com:nirandaperera/rapidsmpf into…

9a5d907

… fanout-node Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera added 4 commits November 20, 2025 14:14

addressing comments

c7740c8

Signed-off-by: niranda perera <niranda.perera@gmail.com>

stashing messages using ref wrappers

0fef621

Signed-off-by: niranda perera <niranda.perera@gmail.com>

Merge branch 'main' into fanout-node

081662e

Merge branch 'main' into fanout-node

5deca7e

wence- reviewed Nov 24, 2025

View reviewed changes

python/rapidsmpf/rapidsmpf/streaming/core/fanout.pyx Outdated Show resolved Hide resolved

wence- reviewed Nov 24, 2025

View reviewed changes

cpp/src/streaming/core/fanout.cpp Outdated Show resolved Hide resolved

wence- reviewed Nov 24, 2025

View reviewed changes

Update cpp/src/streaming/core/fanout.cpp

6bb0bd8

Co-authored-by: Lawrence Mitchell <wence@gmx.li>

nirandaperera requested a review from wence- November 25, 2025 16:12

madsbk reviewed Nov 25, 2025

View reviewed changes

addressing PR comments

4a66c0f

Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera requested a review from madsbk November 25, 2025 19:40

madsbk requested changes Nov 25, 2025

View reviewed changes

nirandaperera and others added 3 commits November 25, 2025 12:23

Apply suggestions from code review

a315b62

Co-authored-by: Mads R. B. Kristensen <madsbk@gmail.com>

Apply suggestions from code review

79bfc0a

Co-authored-by: Mads R. B. Kristensen <madsbk@gmail.com>

Addressing PR comments

8068f2e

Signed-off-by: niranda perera <niranda.perera@gmail.com>

nirandaperera requested a review from madsbk November 25, 2025 22:48

madsbk approved these changes Nov 26, 2025

View reviewed changes

wence- approved these changes Nov 26, 2025

View reviewed changes

Merge branch 'main' into fanout-node

609917c

madsbk and others added 2 commits December 2, 2025 16:31

Merge branch 'main' into fanout-node

86607bc

Merge branch 'main' into fanout-node

de5ebca

Merge branch 'main' into fanout-node

ede0028

rapids-bot bot merged commit 089a73a into rapidsai:main Dec 2, 2025
89 checks passed

Adding Fanout node #636

Adding Fanout node #636

Uh oh!

Conversation

nirandaperera commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot bot commented Nov 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wence- left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirandaperera Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nirandaperera commented Nov 20, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nirandaperera commented Nov 5, 2025 •

edited

Loading

nirandaperera Nov 12, 2025 •

edited

Loading