feat: add CLI and Helm configuration for CDC and S3

[kalbasit]

This commit exposes content-defined chunking (CDC) and S3 storage settings to users:

• Added CLI flags for enabling CDC and configuring chunk size boundaries (min/avg/max).
• Added CLI flags for S3 storage configuration (bucket, endpoint, region, etc.).
• Updated the Helm chart with new values and ConfigMap templates for CDC and S3.
• Provided a complete CDC configuration example in config.example.yaml.

Part of #322

[kalbasit]

• style: apply nix fmt to documentation and project files #671
• feat: add CLI and Helm configuration for CDC and S3 #670 👈 (View in Graphite)
• feat: integrate CDC with cache and implement chunk GC #669
• feat: add S3 storage support for chunks #668
• refactor: extract S3 test helpers for reuse #676
• refactor: move S3 configuration to its own package #675
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[gemini-code-assist]

Summary of Changes

Hello @kalbasit, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the application's configurability by integrating Content-Defined Chunking (CDC) and S3 storage options. Users can now enable and fine-tune CDC for deduplication and specify S3 as a storage backend for chunks, all manageable through new CLI flags and updated Helm chart values. This change aims to provide greater flexibility in deployment and storage management, particularly for experimental deduplication features.

Highlights

• Content-Defined Chunking (CDC) Configuration: Introduced CLI flags (--cache-cdc-enabled, --cache-cdc-min, --cache-cdc-avg, --cache-cdc-max) and Helm chart values for enabling and configuring CDC, allowing for experimental deduplication of NAR files.
• S3 Storage Configuration for Chunks: Added CLI flags for S3 bucket, region, endpoint, access keys, and path style, enabling S3 as a backend for storing content-defined chunks.
• Helm Chart Integration: Updated charts/ncps/values.yaml and charts/ncps/templates/configmap.yaml to expose CDC and S3 storage settings through Helm, facilitating Kubernetes deployments.
• Configuration Example: Provided a complete CDC configuration example in config.example.yaml to guide users on how to utilize the new deduplication feature.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces configuration for Content-Defined Chunking (CDC) and S3 storage through both CLI flags and Helm chart values. The changes are well-structured, exposing the new functionality consistently across different configuration methods. My review notes the added validation in pkg/ncps/serve.go for S3 configuration, which is a positive improvement. The observed code duplication is acceptable based on project guidelines. Overall, this is a good addition.

[kalbasit]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces configuration for Content-Defined Chunking (CDC) and S3 storage via CLI flags and Helm chart values. The changes are well-structured, adding the necessary flags, updating the Helm chart templates and values, and providing an example in config.example.yaml.

My main feedback is regarding code duplication in pkg/ncps/serve.go. The new getChunkStorageBackend function repeats logic from existing storage configuration functions, which has resulted in missing validation for conflicting storage settings and incomplete S3 configuration. I've left a comment with a suggestion to fix this.

[kalbasit]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces configuration for Content-Defined Chunking (CDC) and S3 storage through both CLI flags and Helm chart values. The changes are well-structured, particularly the refactoring in pkg/ncps/serve.go which centralizes storage configuration logic into a new getStorageConfig function. This improves code clarity and maintainability.

My review includes a couple of suggestions for improvement:

• Removing redundant validation for S3 configuration.
• Adding validation for CDC chunk size parameters to prevent potential issues with negative inputs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.16%. Comparing base (7137646) to head (bac7d75).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #670   +/-   ##
=======================================
  Coverage   79.16%   79.16%           
=======================================
  Files           2        2           
  Lines         456      456           
=======================================
  Hits          361      361           
  Misses         83       83           
  Partials       12       12           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dosubot]

Related Documentation

No published documentation to review for changes on this repository.

Write your first living document

^{How did I do? Any feedback?}