Skip to content

Conversation

@samestrin
Copy link
Owner

Summary

Technical debt cleanup addressing 4 unrelated items: performance gaps, feature gaps, and test coverage issues.

Changes

  • Storage Performance Optimization: Refactored SearchMemory to use batched processing with LIMIT/OFFSET and min-heap priority queue, reducing memory usage from O(n) to O(batch_size + topK)
  • Markdown Chunker Enhancement: Added list hierarchy tracking to preserve parent list item context for code blocks inside nested lists
  • HTML Chunker Test Coverage: Added comprehensive tests for fallbackToText error handling path (0% coverage issue)
  • Memory Stats Query Optimization: Fixed N+1 query pattern by using LEFT JOIN to fetch Question/CreatedAt fields in single query

Files Changed

  • 9 files modified (8 modified, 1 new)
  • +688 insertions, -92 deletions
  • New: storage_sqlite_bench_test.go (benchmark tests for large indexes)

Test Plan

  • Code review passed (8/8 items verified)
  • Sprint complete validation passed
  • All SQLite tests passing
  • Benchmarks show constant memory profile (~8840 B/op)
  • Coverage maintained at 66%

Related

  • Branch: feature/8.17_miscellaneous_cleanup
  • Sprint: 8.17 - MiscellaneousCleanup
  • Code Review: 2026-01-18_code-review.md

- Task 01: Optimize SearchMemory with batched processing
- Task 02: Add list context tracking to markdown chunker
- Task 03: Add test coverage for HTML fallbackToText
- Task 04: Optimize memory stats query (fix N+1)
@samestrin samestrin merged commit 381e2b8 into main Jan 18, 2026
1 check passed
@samestrin samestrin deleted the feature/8.17_miscellaneous_cleanup branch January 18, 2026 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant