bug: websocket does not reconnect on disconnection under certain conditions

[GabrielePicco]

Report

Description

Websocket does not reconnect on disconnection. Under certain conditions, the ChainPubsubActor does not reconnect unless the process/validator is restarted.

Proposed solution

1. Implement a heartbeat (e.g. ping messages) to detect whether the connection is healthy.
2. Retry reconnection with exponential backoff and harden the reconnection logic.
3. If the connection is not healthy, exclude it from submux redundancy.

Commit / version

master

Environment

No response

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Possible Duplicates

• Resubscription logic loses accounts when all clients disconnect #622

Related Issues

• bug: Accounts get "stuck" when a delegation event is missed for an account that is already present in the validator as readable. #765

👤 Suggested Assignees

• GabrielePicco
• thlorenz

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[thlorenz]

I investigated this and as explained in Slack this is actually behaving correctly.

When you restart the local test validator the slot restarts, thus when we get a subscription update after that (which I verified via logs that we are getting it) then the remote_slot we have is larger than that slot and thus we correctly ignore that update as old.

[2025-12-24T15:45:21.556781Z TRACE magicblock_chainlink::chainlink::fetch_cloner] FetchCloner received subscription update for FyCH6cWhjVhSAA5ZhMKGT47375WGXNSRH6y8Ysgsgz1x at slot 436
[2025-12-24T15:45:21.556907Z WARN  magicblock_chainlink::chainlink::fetch_cloner] Ignoring out-of-order subscription update for FyCH6cWhjVhSAA5ZhMKGT47375WGXNSRH6y8Ysgsgz1x: bank slot 461, update slot 436

I waited long enough for the slot after restart to be large enough and then subscription updates are cloned again.

It can be a bit unexpected when developing locally, but not sure how we could work around this without having a nuke method that devs can call to either remove all accounts from the bank or forcefully resetting the remote slot to 0.

For now we should close this @GabrielePicco and possible open a feature request for one of the mentioned local dev workaround.

Also related to the above recommended solutions

• ping is already performed by the solana rpc client we are using
• reconnect is already handle with backoff as you described
• exclusion of a connection after exceeding a certain number of max attempts could be added, LMK if you want me to create an issue for that so we can discuss the details there (gotta be careful since we may end up with no redundancy but validator still running, etc).

[thlorenz]

Going to make issue for redundancy improvement.

[thlorenz]

Reconnection improvements were made

• slower resub feat: make resubscription delay configurable for pubsub reconnection #822, throttling to 200ms per sub and making it configurable via resubscription-delay

Only remaining action item would be:

• exclusion of a connection after exceeding a certain number of max attempts could be added

However I'd prefer to see if the first fix makes that unnecessary.

[thlorenz]

@GabrielePicco I think we can close this as we're tracking reconnection issues/improvements in other issues.