Skip to content

Implement optimized catchup when wallet backend falls behind tip#428

Merged
aditya1702 merged 21 commits intoadd-backfillingfrom
optimize-catchup
Dec 23, 2025
Merged

Implement optimized catchup when wallet backend falls behind tip#428
aditya1702 merged 21 commits intoadd-backfillingfrom
optimize-catchup

Conversation

@aditya1702
Copy link
Contributor

@aditya1702 aditya1702 commented Dec 18, 2025

What

  • Add important env flags to tweak the backfill worker size and the batches.
  • Implement db flushing so that we can insert ledgers frequently while backfilling - this helps prevent increased RAM usage as we don't hold the ledger data in memory.
  • Add a BackfillMode iota variable and do an optimized catchup when wallet backend has fallen behind network tip beyond a threshold.
  • Add unit tests for the new functionality.
  • Add a CatchupTestSuite that tests stopping the wallet backend ingest container, letting the network go ahead and then doing optimized catchup using parallel workers.

Why

This optimized catchup is helpful because for normal live ingestion, we process ledgers sequentially as they are produced. However in cases where wallet backend is behind the tip, we can intelligently fetch multiple ledgers from the ledger backend at once and process them parallelly (sort of like an intelligent backfill) and speed up catchup.

Known limitations

N/A

Issue that this PR addresses

#429

@aditya1702 aditya1702 requested a review from a team December 19, 2025 00:15
@aristidesstaffieri
Copy link
Contributor

Would it make sense for there to be periodic check points during live ingestion that can kick off optimized catch ups?
Afaict catchups are only detected at startup.

result.Duration = time.Since(start)
return result
}
batchBuffer.Clear()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it could be nice to emit a metric for buffer size here so we can see how it relates to theBackfillDBInsertBatchSize

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add it in the metrics PR: #435

@aditya1702 aditya1702 merged commit d10e6c4 into add-backfilling Dec 23, 2025
7 checks passed
@aditya1702 aditya1702 deleted the optimize-catchup branch December 23, 2025 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants