Inter-node Communication Buffer warning

[bdaoudtdc]

Describe the bug

We have a 3 node cluster.
We see this message in the logs
[warning] <0.646.0> rabbit_sysmon_handler busy_dist_port <0.1287.0> [{name,delegate_management_1},{initial_call,{delegate,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.874>,unknown}

According to the docs ..

Inter-node traffic between a pair of nodes uses a TCP connection with a buffer known as the inter-node communication buffer. Its size is 128 MB by default. This is a reasonable default for most workloads.

We believe we have a very light workload , however this message came up few times so we increased the buffer to 512 MB , but we still see the message.

To Reproduce

It happens randomly ..

Version and environment information

RabbitMQ: [3.8.12]
RabbitMQ Cluster Operator: [1.7.0 ]
Kubernetes: [e.g. 1.17.13]

[ansd]

Converting this to a discussion.