Implement lazy file replay reader and integrate with Repo::open_file()

[Copilot]

Adds LazyFileReplay reader for on-demand file reconstruction from fragment sequences without loading entire files into memory. Addresses the need to replay UNCHANGED/ADDED/DELETED fragments lazily while supporting bidirectional seeking. The reader is now integrated into Repo::open_file() to provide lazy file access throughout the system.

Implementation

• LazyFileReplay struct implementing Read + Seek traits
  • Dual position tracking: logical position (output) vs old file position (source)
  • Pre-loaded ADDED fragment data (stored in indexed Vec at construction)
  • Position calculation: find_fragment_at_position() maps logical→(fragment_idx, offset, old_pos)
  • No lifetime parameters - self-contained and returnable as Box<dyn ReadSeek>

Fragment Handling

Type	Output	Logical Pos	Old Pos
UNCHANGED{len}	Read from old file	+len	+len
ADDED{body,len}	Pre-loaded data	+len	unchanged
DELETED{len}	(none)	unchanged	+len

Seeking Behavior

DELETED fragments don't exist in output—they only shift the old file position mapping. Seeking to logical position N calculates which fragment contains N and determines the corresponding old file offset accounting for all preceding DELETED fragments.

Example:

Fragments: [UNCHANGED:100] [DELETED:50] [UNCHANGED:100]
Output:     0─────────99   (invisible)  100───────199
Old file:   0─────────99   100─────149  150───────249

seek(150) → fragment 2, offset 50, old_pos=200

Integration with Repo::open_file()

The Repo::open_file() function now returns a LazyFileReplay reader instead of eagerly reconstructing files into memory:

// Old behavior: eagerly loads entire file into memory
let mut reader = repo.open_file(file_path, hash)?;

// New behavior: returns lazy reader with on-demand fragment loading
let mut reader = repo.open_file(file_path, hash)?; // Same API, lazy internally

Key changes:

• Removed eager reconstruction loop from open_file()
• Recursively opens parent files as lazy readers
• Pre-loads ADDED fragment data at construction to avoid lifetime issues
• Returns Box<dyn ReadSeek> wrapping LazyFileReplay

API

// Direct usage
let mut reader = LazyFileReplay::new(&repo, old_file, fragments)?;
reader.read(&mut buf)?;           // Sequential reading
reader.seek(SeekFrom::Start(n))?; // Random access

// Via Repo::open_file() - now uses LazyFileReplay internally
let mut reader = repo.open_file(file_path, commit_hash)?;
reader.read(&mut buf)?;           // Lazy reading
reader.seek(SeekFrom::Start(n))?; // Seekable

Testing

• Unit tests for position calculation logic across all fragment types
• Comprehensive seeking test (test_seeking_forward_and_backward) validates:
  • Forward seeking with SeekFrom::Start
  • Backward seeking to earlier positions
  • Relative seeking with SeekFrom::Current (both directions)
  • Seeking from end with SeekFrom::End
  • Multiple back-and-forth seeks
  • Data correctness after each seek operation
  • Proper handling of fragment boundaries
• Integration: All existing tests pass with lazy implementation

Files Changed

• lib/src/replay.rs - Core LazyFileReplay implementation with comprehensive tests
• lib/src/repo.rs - Integrated LazyFileReplay into open_file(), exposed get_object_path() for fragment loading
• lib/examples/lazy_replay_demo.rs - Usage example

Original prompt

below is a transcript of some things I want you to do. go ahead and do them. if I left questions unanswered then outline them for me and wait for me to answer them, unless you can figure it out yourself based on the context

---

I need to implement. A way of replaying the files lazily. So, I'm going to create an object that takes as arguments. A read seek the repo and a set of fragment hashes. Now, the way it's going to work is when you start to read from it. If it's an unchanged fragment you read from the previous read, seek input. If it's an added fragment, you just output that added fragment. And then it's a if it's a deleted fragment, you have to advance. Old reader. By that number of bits. But then, how do you seek backwards is? But I'm not really understanding, so it's like. If you haven't unchanged fragment of 100 bytes. You read 100 bytes from the old stream. If you hadn't added fragment of 100 bytes, you just output that added at fragment. If you have a deleted fragment of 100 bytes? You, I guess, then move on to.

I guess next time you have an unchanged fragment. The old file is already Advanced.

So, maybe you do with a deleted fragment? Exactly what it says, but then, how do you like read in the middle of a stream? Like?

Like, how do you read like? If you seek backwards, and it happens to be in the middle of a deleted fragment. How do you, uh? What do you do?

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[danhab99]

confirm that the replayer is capable of seeking forward and backward, prove it in a unit test

[danhab99]

implement the replayer in the repo open file function

[Copilot]

confirm that the replayer is capable of seeking forward and backward, prove it in a unit test

Added comprehensive unit test test_seeking_forward_and_backward that proves the LazyFileReplay reader can seek both forward and backward.

The test validates:

• Forward seeking with SeekFrom::Start
• Backward seeking to earlier positions
• Relative seeking with SeekFrom::Current (both directions)
• Seeking from end with SeekFrom::End
• Multiple back-and-forth seeks
• Reading correct data after each seek operation
• Proper handling of fragment boundaries

Test uses a simple file structure (A's, B's, C's) with DELETED fragments to demonstrate that seeking works correctly even when fragments are skipped.

Commit: f150d94

[Copilot]

implement the replayer in the repo open file function

Integrated LazyFileReplay into Repo::open_file().

The function now returns a lazy reader that:

• Pre-loads ADDED fragment data at construction (eliminating lifetime issues)
• Reads UNCHANGED fragments on-demand from parent file
• Handles DELETED fragments during position calculations
• Recursively uses lazy readers for parent files

Files opened via open_file() are no longer eagerly reconstructed into memory. Instead, they use the LazyFileReplay reader for true lazy, seekable access.

Key changes:

• Removed lifetime parameter from LazyFileReplay (now owns all necessary data)
• Pre-loads fragment data in indexed Vec at construction
• Updated open_file() to return Box<dyn ReadSeek> wrapping LazyFileReplay
• All tests pass, including the seeking test

Commit: 2e787fb