feat: add external repair API (Phase 4)

[youngkidwarrior]

Why:
Implements the External Repair API (Phase 4 of SEN-68) to allow manual
re-verification and re-indexing of block ranges. This provides an
operator "escape hatch" for correcting data inconsistencies that automatic
mechanisms might miss or fail to recover.

Includes:

• RepairService with advisory locking and async execution
• HTTP API endpoints: POST /api/v1/repair, GET /api/v1/repair/{id}, GET /api/v1/repairs
• Prometheus metrics for repair requests and reprocessed blocks
• Integration with Manager for Task deletion and recovery

Test plan:

• Run unit tests: go test -v ./shovel -run TestHandleRepairRequest
• Verify repair job creation and status tracking via API
• Verify metrics increment on repair actions

[youngkidwarrior]

• feat: add observability & operator controls (Phase 5) #7
• feat: add external repair API (Phase 4) #6 👈 (View in Graphite)
• Add confirmation audit loop (Phase 3) #5 : 1 other dependent PR (#8 )
• Add receipt-level validation (Phase 2) #3 : 1 other dependent PR (#4 )
• Add shovel consensus engine phase one #2 : 2 other dependent PRs (#1 , #10 )
• Add integration test reproducing missing logs bug (SEN-68) #9
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[0xBigBoss]

Review: PR #6 - External Repair API (Phase 4)

Verdict: ✅ Approve with minor suggestions

This provides the operator "escape hatch" for manual recovery when automatic mechanisms fail.

Strengths

1. Complete REST API - POST create, GET status, GET list with filtering
2. Good validation - Source/integration existence checks, range limits (10k blocks max), start <= end
3. Async execution - Background goroutine with UUID tracking
4. Audit trail - shovel.repair_audit table logs all requests with requester IP
5. Dry-run mode - Count affected rows without deletion
6. Metrics - shovel_repair_requests_total, shovel_repair_blocks_reprocessed_total

API Design

POST /api/v1/repair
{
  "source": "mainnet",
  "integration": "seaport",
  "start_block": 18000000,
  "end_block": 18000100,
  "force_all_providers": true,
  "dry_run": false
}

Clean, RESTful, well-documented.

Issues Found

1. force_all_providers not used - The flag is parsed but executeRepair() doesn't check it. This should disable consensus and rotate through all URLs.

2. Advisory lock scope - Uses wpg.LockHash() for per-repair locking, but should also prevent overlapping repairs on the same block range.

3. Context propagation - executeRepair uses context.Background() - consider passing a cancellable context for graceful shutdown.

Suggestions

1. Add rate limiting to prevent API abuse
2. Consider adding a webhook callback option for completion notification
3. Add support for repair queue inspection

Main Routes

mux.Handle("/api/v1/repair", wh.Authn(wh.HandleRepairRequest))
mux.Handle("/api/v1/repair/", wh.Authn(wh.HandleRepairStatus))
mux.Handle("/api/v1/repairs", wh.Authn(wh.HandleListRepairs))

Properly behind authentication.

Good Phase 4 implementation. Minor fix needed for force_all_providers.

[0xBigBoss]

Review findings:

• force_all_providers is parsed but never used; repair path does not adjust provider set or consensus behavior. shovel/repair.go.
• executeRepair only deletes; it never reindexes or updates blocks_reprocessed, so status reports are misleading and the “repair” doesn’t actually repair data. shovel/repair.go:337-351.
• Repair deletes run without taking the task’s advisory lock (task.lockid), so it can race with live indexing. Consider reusing the same lock pattern used in Task.Converge().
• Overlapping repair rejection (HTTP 409) is specified in docs but not implemented.

Suggest clarifying intended behavior (delete-only vs delete+reindex) and aligning code + runbook/spec before merge.

[youngkidwarrior]

Merge activity

• Dec 30, 1:19 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Dec 30, 1:26 AM UTC: Graphite rebased this pull request as part of a merge.
• Dec 30, 1:27 AM UTC: @youngkidwarrior merged this pull request with Graphite.