feat(app-deployments): improve app:create performance with v2 format

[adambenhassen]

Background

App deployment uploads can be slow when deploying many documents, especially when most documents haven't changed between versions. Currently, every document is re-uploaded on each deployment, even if it already exists from a previous version

Description

Implements a new v2 storage format for app deployments that enables delta uploads, only uploading documents that don't already exist. This dramatically improves deployment speed for incremental updates.

Notable Changes

• CLI, when V2 format is defined, queries existing hashes before upload and skips documents that already exist.
• V2 format requires SHA256 hashes and validates hash matches document content (prevents hash collisions) both in CLI and server-side.
• Clickhouse and S3 now insert in parallel (not actually in parallel, but it promises to run at the same time)
• Added --showTiming flag to display per-batch timing breakdown in CLI.
• Added S3_UPLOAD_CONCURRENCY environment variable and set default to 100 (previously 30)

Backwards Compatibility

• Added --format flag to app:create (v1 default, v2 for delta uploads)
• CDN logic tries v2 key first, falls back to v1 key

Benchmarks

All changes were benchmarked with the script provided in load-tests/app-deployments.

1000 Documents

Scenario	Docs Uploaded	Parse	Validate	Coords	ClickHouse	S3	Total
V1	1000	17ms	401ms	45ms	1790ms	8274ms	9.1s
V2 Initial	1000	10ms	276ms	48ms	936ms	7040ms	7.6s
V2 Delta	50 (950 skipped)	1ms	25ms	5ms	51ms	323ms	376ms

Format Comparison

Feature	V1	V2 (SHA256)
Hash format	Any alphanumeric	SHA256 only
Hash validation	Format only	Format + content match
S3 key	app/{target}/{name}/{version}/{hash}	app-v2/{target}/{name}/{hash}
Cross-version dedup	No	Yes
Delta uploads	No	Yes

Closes CONSOLE-1582

[gemini-code-assist]

Summary of Changes

Hello @adambenhassen, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant performance improvement for app deployments by implementing a new 'v2' storage format. This new format allows for efficient delta uploads, where only new or changed documents are uploaded, and enables cross-version deduplication, reducing storage and transfer overhead. The changes span across the CLI, API, and CDN worker, ensuring a seamless and faster deployment experience, especially for applications with many documents and frequent updates.

Highlights

• New V2 Storage Format for App Deployments: Introduced a new 'v2' storage format that enables delta uploads and cross-version deduplication for app deployment documents, significantly improving performance for incremental updates. This format requires SHA256 hashes and includes content validation.
• CLI Enhancements for V2 Format: The CLI's app:create command now supports a --format flag (defaulting to 'v1', with 'v2' for delta uploads). When using 'v2', the CLI queries existing document hashes and skips re-uploading unchanged documents. A --showTiming flag was also added to display detailed performance breakdowns.
• Parallel Data Ingestion: Improved the ingestion pipeline by implementing parallel insertion of documents into ClickHouse and S3, further boosting deployment speed. The default S3_UPLOAD_CONCURRENCY environment variable was increased to 100.
• CDN Backward Compatibility: The CDN logic has been updated to first attempt fetching documents using the new 'v2' key format and gracefully fall back to the 'v1' key format if the 'v2' key is not found, ensuring compatibility with existing deployments.
• Enhanced Hash Validation: The 'v2' format enforces strict SHA256 hash validation, including a check that the provided hash matches the document's content, preventing hash collisions and ensuring data integrity. The 'v1' format retains its legacy behavior of accepting any alphanumeric hash.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new V2 storage format for app deployments, significantly improving performance through delta uploads and parallel processing. The changes include new CLI flags for format selection and timing, robust SHA256 hash validation, and optimized parallel uploads to S3 and ClickHouse. Extensive integration tests have been added to cover the new functionality, and a benchmark script with a detailed README provides clear performance comparisons. The changes are well-implemented and effectively address the stated performance goals. There is a minor style guide adherence opportunity regarding environment variable access in a utility script and a backend service provider.

[github-actions]

🚀 Snapshot Release (alpha)

The latest changes of this PR are available as alpha on npm (based on the declared changesets):

Package	Version	Info
@graphql-hive/apollo	0.47.2-alpha-20260127033018-133d4453a1a8cfd3aecb6ddaaaafc92a5447294b	npm ↗︎ unpkg ↗︎
@graphql-hive/cli	0.58.0-alpha-20260127033018-133d4453a1a8cfd3aecb6ddaaaafc92a5447294b	npm ↗︎ unpkg ↗︎
@graphql-hive/core	0.20.2-alpha-20260127033018-133d4453a1a8cfd3aecb6ddaaaafc92a5447294b	npm ↗︎ unpkg ↗︎
@graphql-hive/envelop	0.40.4-alpha-20260127033018-133d4453a1a8cfd3aecb6ddaaaafc92a5447294b	npm ↗︎ unpkg ↗︎
@graphql-hive/yoga	0.47.2-alpha-20260127033018-133d4453a1a8cfd3aecb6ddaaaafc92a5447294b	npm ↗︎ unpkg ↗︎
hive	9.2.0-alpha-20260127033018-133d4453a1a8cfd3aecb6ddaaaafc92a5447294b	npm ↗︎ unpkg ↗︎

[github-actions]

📚 Storybook Deployment

The latest changes are available as preview in: https://pr-7565.hive-storybook.pages.dev

[github-actions]

🐋 This PR was built and pushed to the following Docker images:

Targets: build

Platforms: linux/amd64

Image Tag: 133d4453a1a8cfd3aecb6ddaaaafc92a5447294b

[github-actions]

💻 Website Preview

The latest changes are available as preview in: https://pr-7565.hive-landing-page.pages.dev