feat: add S3 storage support for chunks

[kalbasit]

This commit adds a new S3-compatible storage backend for content-defined chunks:

• Implemented s3Store using the MinIO Go SDK.
• Support for Get, Put, HasChunk, and Delete operations on S3.
• Added integration tests for S3 storage (runnable with MinIO).
• Transparently handles path-style and virtual-hosted style buckets.

Part of #322

[gemini-code-assist]

Summary of Changes

Hello @kalbasit, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the storage capabilities by introducing an S3-compatible backend for content-defined chunks. This allows the system to store and retrieve data chunks from cloud storage services, providing a more distributed and resilient storage option. The implementation includes all necessary CRUD operations for chunks and is thoroughly tested with integration tests.

Highlights

• New S3 Storage Backend: Introduced a new S3-compatible storage backend for content-defined chunks, enabling scalable and robust storage solutions.
• MinIO Go SDK Integration: The S3 storage implementation leverages the MinIO Go SDK for interacting with S3-compatible services.
• Core Chunk Operations: Full support for essential chunk operations including Get, Put, HasChunk, and Delete has been implemented for the S3 backend.
• Flexible Bucket Addressing: The S3 store transparently handles both path-style and virtual-hosted style buckets, providing flexibility in S3 endpoint configurations.
• Integration Tests: Comprehensive integration tests have been added for the S3 storage, designed to be runnable with a MinIO instance to ensure functionality and reliability.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[kalbasit]

• style: apply nix fmt to documentation and project files #671
• feat: add CLI and Helm configuration for CDC and S3 #670
• feat: integrate CDC with cache and implement chunk GC #669
• feat: add S3 storage support for chunks #668 👈 (View in Graphite)
• refactor: extract S3 test helpers for reuse #676
• refactor: move S3 configuration to its own package #675
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[gemini-code-assist]

Code Review

This pull request introduces S3 storage support for chunks, which is a great feature. The implementation is clean and includes integration tests. I've identified a couple of areas for improvement: one related to making the S3 endpoint URL parsing more robust, and another concerning a race condition in the PutChunk method, which is acceptable for now given the current application context and documentation as technical debt. My comments provide specific suggestions to address these points.

[kalbasit]

/gemini review

[kalbasit]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces S3-compatible storage for content-defined chunks, which is a great addition. The implementation in pkg/storage/chunk/s3.go is well-structured, using the MinIO Go SDK and incorporating locking to prevent race conditions during chunk uploads. The integration tests are comprehensive and cover the main use cases, including race conditions.

I have a couple of suggestions to improve maintainability and robustness in the S3 store implementation. One is to avoid a hardcoded TTL for locks, and the other is to make the error handling for non-existent chunks more consistent. Please see the detailed comments.