Skip to content

Conversation

@Dylan-M
Copy link
Collaborator

@Dylan-M Dylan-M commented Jan 29, 2026

Proposed Change

Fixed the filegen generator to select a single random line from each file on each work cycle, and added file caching to improve performance.

Changes:

  • Fixed readAndWriteFile() to read the entire file into memory, collect non-empty lines, and select one at random (instead of writing all lines sequentially)
  • Added math/rand import for random line selection
  • Implemented file caching with thread-safe RWMutex to avoid repeated disk reads
  • Files are cached with automatic 1-minute TTL expiration
  • Extracted file reading logic into readFileLines() helper function
  • Updated documentation to clarify the random line selection behavior
  • Added Caching section to documentation explaining cache behavior
Checklist
  • Changes are tested
  • CI has passed

@Dylan-M Dylan-M requested a review from a team as a code owner January 29, 2026 20:41
@Dylan-M Dylan-M mentioned this pull request Jan 29, 2026
2 tasks
@Dylan-M Dylan-M force-pushed the dylanmyers/data-library branch from 6afb2e7 to 7a2609f Compare January 29, 2026 20:45
@Dylan-M Dylan-M requested a review from jsirianni January 29, 2026 21:26
Co-authored-by: Joseph Sirianni <joe.sirianni@observiq.com>
@Dylan-M Dylan-M enabled auto-merge January 29, 2026 21:44
@Dylan-M Dylan-M added this pull request to the merge queue Jan 29, 2026
Merged via the queue into main with commit 79ed724 Jan 29, 2026
16 checks passed
@Dylan-M Dylan-M deleted the dylanmyers/data-library branch January 29, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants