Skip to content

fix(filegen): reset backoff on success to respect configured rate#61

Merged
Dylan-M merged 1 commit intomainfrom
fix/backoff-rate-limiting
Jan 30, 2026
Merged

fix(filegen): reset backoff on success to respect configured rate#61
Dylan-M merged 1 commit intomainfrom
fix/backoff-rate-limiting

Conversation

@Dylan-M
Copy link
Collaborator

@Dylan-M Dylan-M commented Jan 29, 2026

Proposed Change

The filegen generator was using exponential backoff for normal rate limiting between file reads, which caused the wait time to grow exponentially instead of respecting the configured rate parameter. This resulted in significantly lower throughput than expected (e.g., 699 ops/sec per worker instead of 1000+ ops/sec per worker with a 1ms rate).

Changes:

  • Modified the worker loop to use backoff.NewTicker() similar to the JSON generator
  • Exponential backoff is now only applied on errors (file read failures)
  • Backoff is reset to the configured InitialInterval on successful operations
  • The configured rate is now properly respected for normal operation

Example behavior:

  • Before: 1ms rate would grow to 1.5ms, 2.25ms, 3.375ms, etc. due to exponential backoff
  • After: 1ms rate stays at 1ms between successful reads, only increases on errors

This fix enables users to achieve their target EPS by properly honoring the rate configuration.

Checklist
  • Changes are tested (filegen tests pass)
  • CI has passed (lint, format, test, completion all pass)

@Dylan-M Dylan-M requested a review from a team as a code owner January 29, 2026 23:10
@Dylan-M Dylan-M added this pull request to the merge queue Jan 30, 2026
Merged via the queue into main with commit ebbf74b Jan 30, 2026
16 checks passed
@Dylan-M Dylan-M deleted the fix/backoff-rate-limiting branch January 30, 2026 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants