Disconnected Nomad client nodes never transition to down, are not garbage collected, and generate infinite heartbeat failures

[raunak2004]

Nomad version

v1.9.7

• Client OS: Windows
• Client driver: raw_exec
• Environment: Production (multiple clusters)
• Workload type: UEM Control Plane workloads
• Node lifecycle: Externally decommissioned by SaaS automation

Issue

Nomad client nodes that are permanently decommissioned outside of Nomad remain indefinitely in a disconnected and eligible state. These nodes never transition to down, are not eligible for garbage collection, and cause Nomad servers to continuously emit Node heartbeat missed events.

As a result:
• Nomad servers repeatedly poll nodes that no longer exist
• Logs contain continuous heartbeat failure noise
• Log ingestion costs increase significantly in production
• Manual operator intervention is required to clean up stale nodes

Reproduction steps
1. Register a Nomad client node to a cluster.
2. Decommission or terminate the node externally without graceful Nomad deregistration.
3. Wait for the client to become unreachable.
4. Observe the node state via:
nomad node status
5. Inspect a disconnected node:
nomad node status <NODE_ID>

Expected Result
• After a reasonable period of missed heartbeats, Nomad should:
• Transition the node from disconnected to down
• Mark the node as ineligible
• Garbage collect the node automatically
• Heartbeat retry attempts and related log entries should stop once the node is clearly unrecoverable.

Actual Result
• Node remains indefinitely in:
• Status = disconnected
• Eligibility = eligible
• Node is never marked as down
• Node is never garbage collected
• Nomad servers continuously emit:
• Node heartbeat missed events
• Manual force drain is required to transition the node to a state where GC can remove it.

Nomad Server logs (if appropriate)

2026-01-23T19:05:43.718Z  {
  "@level": "warn",
  "@message": "node TTL expired",
  "@module": "nomad.heartbeat",
  "@timestamp": "2026-01-23T19:05:43.718135Z",
  "node_id": "<REDACTED_NODE_ID>"
}
2026-01-23T19:05:42.507Z  {
  "@level": "warn",
  "@message": "node TTL expired",
  "@module": "nomad.heartbeat",
  "@timestamp": "2026-01-23T19:05:42.507501Z",
  "node_id": "<REDACTED_NODE_ID>"
}

[tgross]

Hi @raunak2004! A node shouldn't enter the disconnected state at all unless it has an allocation with the disconnect.lost_after flag. That's most likely what's keeping the node in the disconnected state.

When lost_after is specified, the Nomad server will mark clients that fail to heartbeat as "disconnected" rather than "down", and will mark allocations on a disconnected client as "unknown" rather than "lost".

There's no way for Nomad to know in this case whether the client will come back, so if you intend to cycle out instances without leave_on_interrupt / leave_on_terminate you're not going to want to use the disconnect block at all. If you've got automation that's cycling out instances, I'd strongly encourage you to use the infrastructure provider's lifecycle hooks to trigger a graceful shutdown. You can even drain the node on shutdown via drain_on_shutdown, if you're running an environment where you know instances only ever get cycled out and not restarted in place.

Small detail here:

Heartbeat retry attempts and related log entries should stop once the node is clearly unrecoverable.

The servers don't heartbeat the clients; the clients send heartbeats to the servers. So the servers are just logging that the clients have gone missing, not sending spurious RPCs to clients about it.

Also, just a heads up that 1.9.7 is out of support and isn't getting security updates or bug fixes. For the community edition, you'll need to stay on the current version. The Enterprise edition has LTS versions.