Skip to content

Conversation

@JC3
Copy link

@JC3 JC3 commented Apr 30, 2025

Cache file hashes across freight cache runs for apt backend. It seems to work. Attempt to address #123.

The infrastructure for this already existed, it's just that it stored the cache files in the always-freshly-created work directory, so the previous run's cache data never carried over. This essentially just moves those files into $VARLIB instead of $DISTCACHE.

For a 23GB registry, freight cache times are decreased from 9m 30s to about 7 seconds.

TODO:

  • Remove some temporary debug output.
  • Fix: Remove cached hashes only for $DIST.
  • Implement $CACHE != "on".
  • Address failed CI checks - they seem to have been pre-existing.
  • Further testing (e.g. by modifying existing packages, verifying hashes are correct, etc)

Implementation notes:

  • Stores cached hashes in /var/lib/freight/.apt/ (or wherever $VARLIB is).
  • Can't store next to cached *-control files because info about where those are is unavailable at the point of hash generation.
  • They will be removed by freight-clear-cache.
  • Cached data will only be used if $CACHE is "on".

Jason Cipriani added 2 commits April 30, 2025 12:43
Adds some quotes to satisfy CI.
Essentially moves hash cache files from $DISTCACHE to $VARLIB, where
they will remain across `freight cache` runs.

See issue freight-team#123 and PR freight-team#143.
@JC3 JC3 force-pushed the jc3-fix-hash-cache branch from ac6b865 to 63a79a3 Compare April 30, 2025 16:45
@JC3 JC3 marked this pull request as ready for review April 30, 2025 16:48
@JC3 JC3 changed the title WIP: Cache file hashes for apt backend. Fix file hash caching for apt backend. Apr 30, 2025
@yegorich yegorich merged commit 55b6110 into freight-team:master May 2, 2025
2 checks passed
@JC3 JC3 mentioned this pull request May 2, 2025
@JC3 JC3 deleted the jc3-fix-hash-cache branch May 2, 2025 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants