perf: improve performance and optimizations in AutoMQ feature module #3113
+78,753
−16,004
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
…MQ#2515) refactor(early returning): remove unnecessary `else` block (`if` block is already returning)
…#2498) Add unit tests for WALConfirmOffsetCalculator
…e issue (AutoMQ#2514) feat: Add --catchup-topic-prefix for performance testing
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
Signed-off-by: Shichao Nie <niesc@automq.com>
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
…to… (AutoMQ#2535) fix(test): increase timeout for write completion in AbstractObjectStorageTest
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
…utoMQ#2542) Signed-off-by: Ning Yu <ningyu@automq.com>
…2548) refactor(s3stream/object-wal): complete appends sequentially (AutoMQ#2426) * chore: add todos * refactor(s3stream/object-wal): sequentially succeed * refactor(s3stream/object-wal): drop discontinuous objects during recovery * test: introduce `MockObjectStorage` * test: test sequentially succeed * refactor: record endOffset in the object path * feat: different version of wal object header * refactor: adapt to the new object header format * feat: recover from the trim offset * test: test recover continuous objects from trim offset * test: test marshal and unmarshal wal object header * test: fix tests * test: test recover from discontinuous objects * test: test recover from v0 and v1 objects * style: fix lint --------- Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
…ok (AutoMQ#2553) * feat(docker): add Spark Iceberg Docker setup and demonstration notebook * docs(readme): add Quick Start guide for AutoMQ Table Topic with Docker Compose
…Q#2555) * fix(workflow): remove conditional image build for main branch * fix(workflow): specify image tag for Spark Iceberg Docker build * fix(workflow): update Docker image tag for Spark Iceberg build
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
* fix(README): add star automq module * fix(README): adjust copy
* feat(bitnami): init and cp bitnami kafka container 3.9.0 * refactor(bitnami): rename Bitnami container files for consistency
AutoMQ#2561) feat(container): automq kafka container features and patch file for upgrade
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
…m… (AutoMQ#2570) fix(container): update Dockerfile and scripts for AutoMQ packaging improvements
* feat(docker): add GitHub Actions workflow for bitnami chart Docker image release * feat(docker): add GitHub Actions workflow for bitnami chart Docker image release * fix(docker): update image tag format for automq-bitnami in release workflow and temporarily remove the latest tag until the AutoMQ Docker Compose is refactored into Bitnami Docker Compose.
…ent (AutoMQ#2562) * feat(helm): add demo-values.yaml and update README for AutoMQ deployment * fix(demo/readme): fix demo-values.yaml and change readme description * fix(README): update Helm chart references to use 'kafka' * feat(values): update demo-values.yaml and README for AutoMQ deployment * fix(demo): image tag * fix(readme): bitnami helm chart version * fix(readme): bitnami helm chart version
AutoMQ#2574) fix(docker): update Dockerfile for AutoMQ while github action packaging installations and permissions
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
* perf(tool/perf): reduce the record header size Signed-off-by: Ning Yu <ningyu@automq.com> * style: fix lint Signed-off-by: Ning Yu <ningyu@automq.com> --------- Signed-off-by: Ning Yu <ningyu@automq.com>
…test (AutoMQ#2573) * Add documentation for --consumers-during-catchup parameter This adds documentation comments for the new parameter that allows controlling the percentage of consumers activated during catch-up reads. * Add --consumers-during-catchup parameter to PerfConfig Introduces a new parameter to control what percentage of consumers should be activated during catch-up read scenarios. Default is 100%. Includes validation to ensure the value is between 0 and 100. * Enhance ConsumerService to support partial consumer activation Modifies resetOffset() and resume() methods to accept a percentage parameter, allowing only a subset of consumers to be activated during catch-up reads. Maintains backwards compatibility with existing method signatures. * Update PerfCommand to use consumers-during-catchup parameter Updates the catch-up read logic to use the new parameter for controlling the percentage of consumers that should be activated during catch-up reads. * Fix various code style issues for whitespace and line breaks to comply with project standards * fixed the two Checkstyle issues again 😭 * Implements a new --consumers-during-catchup parameter to control what percentage of topics have active consumers during catch-up reading phases. This parameter allows users to simulate realistic scenarios where only a subset of topics experience catch-up reads simultaneously. * removes unnecessary pause
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
Revert "fix(s3wal): handle hollow after trim (AutoMQ#3058)" This reverts commit efd674a.
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
…ts (AutoMQ#3067) Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
* Enhance Code of Conduct with new community guidelines (AI transparency, privacy, integrity) * Update Slack link in README.md Signed-off-by: KRISHNA DAS MEENA <mek127142@gmail.com> * Revise Code of Conduct for clarity and inclusivity Updated the Code of Conduct to enhance clarity and inclusivity, adding new standards and examples of acceptable and unacceptable behavior. Signed-off-by: KRISHNA DAS MEENA <mek127142@gmail.com> * Update CODE_OF_CONDUCT.md Signed-off-by: KRISHNA DAS MEENA <mek127142@gmail.com> --------- Signed-off-by: KRISHNA DAS MEENA <mek127142@gmail.com>
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
…Q#3088) - [Bug] the nextRecord is not cleaned up when directly calling #next without calling #hasNext - The bug won't happen in production because each #next is called after #hasNext
* defined constants to hold the new configuration * defined config * added fields and initialization in the constructor * made the iceberg configurable, allowing user control over snapshot * remove oops extra char moment * missing spaces and trailing spaces fixes * indentation for method call chaining aligned with indentation guidelines * unit tests: configurable iceberg snapshot expiration for tablecoordinator * remove: unused imports * test file: applied spotless * fixed: failing unit tests for tablecoordinator * spotless applied on test module
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
…MQ#3089) * test(TableCoordinatorTest): enhance commit status machine tests and improve snapshot expiration handling * fix(TableCoordinator): correct spelling of 'Committed' in status transition methods * test(TableCoordinatorTest): refactor snapshot expiration tests and improve verification logic * chore: update table topic config Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Gezi-lzq <lzqtxwd@gmail.com> --------- Signed-off-by: Gezi-lzq <lzqtxwd@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
…method for improved clarity (AutoMQ#3109) Signed-off-by: Robin Han <hanxvdovehx@gmail.com>
lobsangshakya
requested review from
1sonofqiu,
Gezi-lzq,
superhx and
woshigaopp
as code owners
lobsangshakya
changed the title
lobsangshakya
changed the title
lobsangshakya
changed the title
lobsangshakya
changed the title
lobsangshakya
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces performance improvements and optimizations in the feature/performance-opt branch. Key changes include:
Impact:
These changes improve the overall efficiency of AutoMQ without affecting existing functionality. Unit tests and basic validation have been performed to ensure stability.
Notes for reviewers:
Focus on the refactored logic in the feature module.
Verify that existing workflows continue to work as expected.
Some files were reorganized for better structure; no functional changes were made to unrelated components.