refactor(cli): unify storage configuration for export command #7280

McKnight22 · 2025-11-22T04:18:19Z

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Summary (mandatory):

This PR introduces export command support for Fs, S3, OSS, GCS and Azblob.

Details:

This PR refactors the export command to use the unified ObjectStoreConfig from the common module instead of introducing duplicated code logic for separate storage type in src/cli/src/data/export.rs.

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

I have written the necessary rustdoc comments.
I have added the necessary unit tests and integration tests.
This PR requires documentation updates.
API changes are backward compatible.
Schema or data changes are backward compatible.

src/cli/src/common/object_store.rs

src/cli/src/data/storage_export.rs

Copilot

Pull request overview

This PR refactors the export command to use a unified storage configuration approach, eliminating code duplication and adding support for multiple cloud storage backends (S3, OSS, GCS, Azure Blob) alongside the existing filesystem storage.

Key changes:

Introduced a new storage_export module with a trait-based design for different storage backends
Replaced individual storage flags and configuration parameters with a unified ObjectStoreConfig
Added comprehensive test coverage for all storage backend configurations

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/common/base/src/secrets.rs	Modified Display implementation for SecretString to return empty string when secret is empty
src/cli/src/error.rs	Removed unused `S3ConfigNotSet` error variant
src/cli/src/data/storage_export.rs	New module implementing storage backend abstraction with trait-based design for Fs, S3, OSS, GCS, and Azblob
src/cli/src/data/export.rs	Refactored to use unified storage config, removed duplicated operator building logic, added unit tests for all backends
src/cli/src/data.rs	Added storage_export module declaration
src/cli/src/common/object_store.rs	Added accessor methods for storage connection configs and split build methods for individual backends
src/cli/src/common.rs	Exported new connection type aliases for use in storage backends
src/cli/Cargo.toml	Added common-test-util dependency for testing
Cargo.lock	Updated dependency tree with common-test-util

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/cli/src/common/object_store.rs

src/common/base/src/secrets.rs

src/cli/src/data/storage_export.rs

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

WenyXu

Rest LGTM

src/cli/src/data/storage_export.rs

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/cli/src/data/storage_export.rs

src/cli/src/common/object_store.rs

src/cli/src/data/storage_export.rs

WenyXu · 2025-12-11T07:14:48Z

@McKnight22 Do you have any plans to add integration tests in the future?
Testing the CLI part might be a bit challenging, we may need to build a debug DB binary in CI and run CLI-based tests for the export/import functions. Given that we already inject credentials for various object storages (e.g., GT_AZBLOB_CONTAINER, GT_GCS_BUCKET) in our CI, this could help cover different backends.

WenyXu

Rest LGTM

McKnight22 · 2025-12-15T04:50:09Z

@McKnight22 Do you have any plans to add integration tests in the future? Testing the CLI part might be a bit challenging, we may need to build a debug DB binary in CI and run CLI-based tests for the export/import functions. Given that we already inject credentials for various object storages (e.g., GT_AZBLOB_CONTAINER, GT_GCS_BUCKET) in our CI, this could help cover different backends.

I can take it.

fengjiachun

Rest LGTM

- Utilize ObjectStoreConfig to unify storage configuration for export command - Support export command for Fs, S3, OSS, GCS and Azblob - Fix the Display implementation for SecretString always returned the string "SecretString([REDACTED])" even when the internal secret was empty. Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

- Change the encapsulation permissions of each configuration options for every storage backend to public access. Signed-off-by: McKnight22 <tao.wang.22@outlook.com> Co-authored-by: WenyXu <wenymedia@gmail.com>

- Update the implementation of ObjectStoreConfig::build_xxx() using macro solutions Signed-off-by: McKnight22 <tao.wang.22@outlook.com> Co-authored-by: WenyXu <wenymedia@gmail.com>

- Introduce config validation for each storage type Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

- Enable trait-based polymorphism for storage type handling (from inherent impl to trait impl) - Extract helper functions to reduce code duplication Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

- Improve SecretString handling and validation (Distinguishing between "not provided" and "empty string") - Add validation when using filesystem storage Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

- Refactor storage field validation with macro Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

- support GCS Application Default Credentials (like GKE, Cloud Run, or local development with ) in export (Enabling ADC without validating or to be present) (Making optional in GCS validation (defaults to https://storage.googleapis.com)) Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

This commit refactors the validation logic for object store configurations in the CLI to leverage clap features and reduce boilerplate. Key changes: - Update wrap_with_clap_prefix macro to use clap's requires attribute. This ensures that storage-specific options (e.g., --s3-bucket) are only accepted when the corresponding backend is enabled (e.g., --s3). - Simplify FieldValidator trait by removing the is_provided method, as dependency checks are now handled by clap. - Introduce validate_backend! macro to standardize the validation of required fields for enabled backends. - Refactor ExportCommand to remove explicit validation calls (validate_s3, etc.) and rely on the validation within backend constructors. - Add integration tests for ExportCommand to verify build success with S3, OSS, GCS, and Azblob configurations. Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

- Use macros to simplify storage export implementation Signed-off-by: McKnight22 <tao.wang.22@outlook.com> Co-authored-by: WenyXu <wenymedia@gmail.com>

- Rollback StorageExport trait implementation to not using macro for better code clarity and maintainability - Introduce format_uri helper function to unify URI formatting logic - Fix OSS URI path bug inherited from legacy code Signed-off-by: McKnight22 <tao.wang.22@outlook.com> Co-authored-by: WenyXu <wenymedia@gmail.com>

- Remove unnecessary async_trait Signed-off-by: McKnight22 <tao.wang.22@outlook.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me>

McKnight22 requested a review from a team as a code owner November 22, 2025 04:18

github-actions bot added size/L docs-not-required This change does not impact docs. labels Nov 22, 2025

WenyXu reviewed Nov 24, 2025

View reviewed changes

src/cli/src/common/object_store.rs Outdated Show resolved Hide resolved

src/cli/src/common/object_store.rs Show resolved Hide resolved

src/cli/src/data/storage_export.rs Show resolved Hide resolved

WenyXu requested a review from Copilot November 24, 2025 06:45

Copilot started reviewing on behalf of WenyXu November 24, 2025 06:46 View session

Copilot finished reviewing on behalf of WenyXu November 24, 2025 06:49

Copilot AI reviewed Nov 24, 2025

View reviewed changes

github-actions bot added size/XL and removed size/L labels Nov 28, 2025

McKnight22 force-pushed the pr-refactor_cli_export branch 2 times, most recently from 899028c to fa6898c Compare November 30, 2025 05:36

McKnight22 requested a review from WenyXu December 1, 2025 02:01

WenyXu reviewed Dec 4, 2025

View reviewed changes

src/cli/src/data/storage_export.rs Outdated Show resolved Hide resolved

WenyXu requested a review from Copilot December 4, 2025 03:13

Copilot started reviewing on behalf of WenyXu December 4, 2025 03:14 View session

WenyXu requested a review from fengjiachun December 4, 2025 03:15

github-actions bot added docs-required This change requires docs update. and removed docs-not-required This change does not impact docs. labels Dec 4, 2025

sunng87 mentioned this pull request Dec 4, 2025

Update docs for refactor(cli): unify storage configuration for export command GreptimeTeam/docs#2248

Open

Copilot finished reviewing on behalf of WenyXu December 4, 2025 03:17

WenyXu reviewed Dec 4, 2025

View reviewed changes

src/cli/src/data/storage_export.rs Outdated Show resolved Hide resolved

Copilot AI reviewed Dec 4, 2025

View reviewed changes

src/cli/src/data/storage_export.rs Outdated Show resolved Hide resolved

src/cli/src/data/storage_export.rs Show resolved Hide resolved

src/cli/src/common/object_store.rs Outdated Show resolved Hide resolved

src/cli/src/data/storage_export.rs Outdated Show resolved Hide resolved

WenyXu reviewed Dec 4, 2025

View reviewed changes

src/cli/src/data/storage_export.rs Outdated Show resolved Hide resolved

src/cli/src/data/storage_export.rs Outdated Show resolved Hide resolved

McKnight22 requested a review from WenyXu December 10, 2025 03:53

fengjiachun reviewed Dec 11, 2025

View reviewed changes

src/cli/src/data/storage_export.rs Outdated Show resolved Hide resolved

WenyXu approved these changes Dec 15, 2025

View reviewed changes

McKnight22 requested a review from fengjiachun December 15, 2025 05:56

fengjiachun approved these changes Dec 15, 2025

View reviewed changes

McKnight22 and others added 12 commits December 16, 2025 13:36

refactor(cli): unify storage configuration for export command

f8bdf7c

- Change the encapsulation permissions of each configuration options for every storage backend to public access. Signed-off-by: McKnight22 <tao.wang.22@outlook.com> Co-authored-by: WenyXu <wenymedia@gmail.com>

refactor(cli): unify storage configuration for export command

569ee44

- Update the implementation of ObjectStoreConfig::build_xxx() using macro solutions Signed-off-by: McKnight22 <tao.wang.22@outlook.com> Co-authored-by: WenyXu <wenymedia@gmail.com>

refactor(cli): unify storage configuration for export command

02219cf

- Introduce config validation for each storage type Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

refactor(cli): unify storage configuration for export command

25ce123

- Enable trait-based polymorphism for storage type handling (from inherent impl to trait impl) - Extract helper functions to reduce code duplication Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

refactor(cli): unify storage configuration for export command

c289db0

- Improve SecretString handling and validation (Distinguishing between "not provided" and "empty string") - Add validation when using filesystem storage Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

refactor(cli): unify storage configuration for export command

309ddeb

- Refactor storage field validation with macro Signed-off-by: McKnight22 <tao.wang.22@outlook.com>

refactor(cli): unify storage configuration for export command

2c012c9

- Use macros to simplify storage export implementation Signed-off-by: McKnight22 <tao.wang.22@outlook.com> Co-authored-by: WenyXu <wenymedia@gmail.com>

refactor(cli): unify storage configuration for export command

c3f1904

- Remove unnecessary async_trait Signed-off-by: McKnight22 <tao.wang.22@outlook.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me>

McKnight22 force-pushed the pr-refactor_cli_export branch from 8b30f17 to c3f1904 Compare December 16, 2025 05:36

WenyXu added this pull request to the merge queue Dec 18, 2025

Merged via the queue into GreptimeTeam:main with commit 72eede8 Dec 18, 2025
43 checks passed

refactor(cli): unify storage configuration for export command #7280

refactor(cli): unify storage configuration for export command #7280

Uh oh!

Conversation

McKnight22 commented Nov 22, 2025 • edited by WenyXu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Summary (mandatory):

Details:

PR Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

WenyXu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WenyXu commented Dec 11, 2025

Uh oh!

WenyXu left a comment

Choose a reason for hiding this comment

Uh oh!

McKnight22 commented Dec 15, 2025

Uh oh!

fengjiachun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

McKnight22 commented Nov 22, 2025 •

edited by WenyXu

Loading