SAMZA-2286: Integrate MetadataStore#putAll for improved startup time. #1125
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnishimura
force-pushed
the
samza-2286-integrate-metadatastore-putall
branch
from
f999e0a to
b22b96e
Compare
Contributor
Author
|
@shanthoosh @bharathkk - Please take a look when you get a chance. |
dnishimura
force-pushed
the
samza-2286-integrate-metadatastore-putall
branch
from
b22b96e to
75eafe8
Compare
Contributor
Author
|
Rebased with the latest master branch ⬆️ |
shanthoosh
approved these changes
Nov 7, 2019
Contributor
shanthoosh
left a comment
shanthoosh
left a comment
There was a problem hiding this comment.
LGTM.
It would be great if we can test this patch with a sample yarn job(e.g samza-hello-samza) and quantify the reduction in the write-latency.
| changelogEntriesToStore.put(taskName, valueSerde.toBytes(changeLogPartitionId)); | ||
| } else { | ||
| LOG.debug("Deleting the TaskName: {}", taskName); | ||
| metadataStore.delete(taskName); |
Contributor
There was a problem hiding this comment.
Not related to this change. But it would be beneficial to add a deleteAll API in MetadataStore (synonymous to putAll API). deleteAll API would simplify the iterative delete API invocations. If you agree, then it would be great to create a follow-up ticket and add a todo here.
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR integrates
MetadataStore#putAllwithTaskAssignmentManager,TaskPartitionAssignmentManager,ChangelogStreamManagerandZkJobCoordinatorwhen writing assignments to the coordinator stream. For a Kafka-based coordinator stream, theputAllmethod allows multiple messages to be produced before calling flush. We have observed in preliminary benchmark tests that this provides an order of magnitude improvement when callingputAllvs.putfor each entry. The impact is faster startup times.@shanthoosh - please take a look since you have the most context.