ncps does not handle NAR streaming in a multi-instance HA setup

[kalbasit]

Problem statement

In a multi-instance HA setup with Redis locks:

• Instance A starts downloading a large NAR file from upstream
• Instance B receives a client request for the same NAR file
• Instance B acquires the Redis download lock and sees the download is in progress
• Instance B cannot stream from Instance A's in-memory download state
• Instance B must wait for Instance A to complete and store the file
• For large files, this wait exceeds client timeouts (HTTP 200 with curl timeout errors)

Extracted from closed #618:

We're seeing this as well, and not even with heavy loads, sometimes just with a single build. It does not appear to be resolved in v0.7.3.

I've dropped our ncps deployment back to 1 instance to see if that helps. FYI, we're using:

• PostgreSQL
• NVMe-backed S3 storage via Ceph RGW
• Redis locking via DragonflyDB

Originally posted by @dhess in #618

[kalbasit]

From my conversation with Claude, I can either fix this short-term and target 0.7+ or fix it long-term and just get #322 done for 0.9 instead of what I had planned for 0.10. I'm inclined to solve this by implement CDC earlier than planned. LMK @dhess if this is breaking your side of things, it will probably take a week give or take to implement, test and release especially considering that I have yet to release 0.8.

---

Fix Issue #660 - Multi-Instance HA Timeout During Downloads

Problem Statement

In a multi-instance HA setup with Redis locks:

• Instance A starts downloading a large NAR file from upstream
• Instance B receives a client request for the same NAR file
• Instance B sees the download locked in Redis
• Instance B cannot access instance A's in-memory downloadState
• Instance B must wait for instance A to complete the download
• For large files, this exceeds client timeouts → "HTTP 200 (curl error: Timeout was reached)"

Root Cause

The downloadState used for streaming is stored in instance-local memory (c.upstreamJobs map[string]*downloadState). When instance B tries to serve the same file, it cannot access instance A's download progress.

Evaluation of Solutions

Option 1: Content-Defined Chunking (CDC) - Long-term Solution

With CDC (as planned in #322), NARs would be broken into content-addressed chunks stored independently in S3. This would completely solve the multi-instance streaming problem:

How CDC Solves It:

1. Instance A downloads NAR from upstream and chunks it using FastCDC
2. Instance A stores each chunk in S3 as it's produced (e.g., chunks/{sha256})
3. Instance A records chunk sequence in database table nar_file_chunks
4. Instance B can independently query the database for chunk list
5. Instance B streams chunks from S3 directly (no dependency on instance A)

Pros:

• Solves multi-instance streaming elegantly
• Provides deduplication benefits across similar NARs
• Enables true distributed caching

Cons:

• Major architectural change (new storage layer, database schema, GC updates)
• Moderate-High complexity to implement correctly
• Read latency overhead for chunk reassembly
• Timeline: Estimated several weeks of development + testing

Verdict: CDC is the ideal long-term solution but too complex for immediate fix.

---

Option 2: Stream from Shared Temp File (Intermediate Solution) ⭐

Use the existing S3 backend to enable concurrent streaming from temp files:

How It Works:

1. Instance A writes download to S3 using a predictable temp key pattern
   Example: tmp-nars/{hash}-{uuid}.nar.xz
2. Instance A stores the temp S3 key in Redis alongside the download lock
3. Instance B detects ongoing download via Redis lock
4. Instance B reads the temp S3 key from Redis
5. Instance B opens the same S3 object and starts streaming (S3 supports concurrent reads during writes)
6. Both instances coordinate completion via Redis

Pros:

• Works with existing S3 infrastructure (already required for HA)
• Minimal code changes (modify temp file creation + lock metadata)
• No new database schema needed
• Can implement quickly

Cons:

• Only solves the immediate problem, doesn't provide deduplication
• Slightly more complex cleanup logic for temp files

Verdict: Best immediate fix that's practical to implement now.

---

Option 3: Fallback to Direct Upstream Download

Instance B directly downloads from upstream when instance A is already downloading:

Pros:

• Simplest implementation

Cons:

• Wastes bandwidth (double downloads)
• Could overwhelm upstream
• Doesn't truly fix the root cause

Verdict: Not recommended.

[dhess]

Thanks. I think we'll drop back to v0.4.0 for the time being, which worked great.

[kalbasit]

Thanks. I think we'll drop back to v0.4.0 for the time being, which worked great.

Single instance 0.7 will also work. This issue comes up from running multiple instances.