Skip to content

Comments

db: prefer batched deletes#41

Merged
DolceTriade merged 1 commit intomainfrom
batchdel
Dec 23, 2025
Merged

db: prefer batched deletes#41
DolceTriade merged 1 commit intomainfrom
batchdel

Conversation

@DolceTriade
Copy link
Owner

@DolceTriade DolceTriade commented Dec 23, 2025

Our invocation output table is huge and deleting so many rows causes major lock contention. Delete them a few at a time instead to decrease lock contention.

A batch size of 10 was intentionally chosen to deal with large write amplification from cascading deletes.

Our invocation output table is huge and deleting so many rows causes major lock contention. Delete them a few at a time instead to decrease lock contention.
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements batched deletion of old invocations to reduce database lock contention during periodic cleanup operations. Instead of deleting all expired invocations in a single transaction, the cleanup process now deletes them in small batches with delays between batches.

Key changes:

  • Added delete_invocations_batch method to the DB trait that accepts a limit parameter for batch size
  • Modified the periodic cleanup task to loop through batches of 10 invocations with 100ms delays between batches
  • Implemented the batched deletion in both SQLite and Postgres database backends using a SELECT-then-DELETE pattern

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
blade/state/lib.rs Adds new delete_invocations_batch method to the DB trait interface
blade/main.rs Replaces single bulk delete with batched deletion loop in periodic cleanup task, adding batch size and delay constants
blade/db/sqlite/mod.rs Implements delete_invocations_batch for SQLite using SELECT-then-DELETE pattern and adds comprehensive test coverage
blade/db/postgres/mod.rs Implements delete_invocations_batch for Postgres using SELECT-then-DELETE pattern and adds comprehensive test coverage

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@DolceTriade DolceTriade merged commit da21bc5 into main Dec 23, 2025
7 checks passed
@DolceTriade DolceTriade deleted the batchdel branch December 23, 2025 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant