Add new framing chunk types without checksums #155
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds two new chunk types to the Snappy framing format: compressed data without a checksum, and uncompressed data without a checksum. These types are identical to their existing counterparts except they do not contain a CRC-32C checksum. Essentially, this makes including checksums for each data chunk optional rather than required. In some use cases, computing the CRC-32C checksums for the data chunks in the Snappy framing format ends up dominating execution time. Eliminating the checksums provides massive 2.5x performance improvements in our uses of Snappy for compressing address trace data prior to storing to disk. Existing readers of the Snappy framing format would be expected to fail up front on an unknown chunk type when encountering the new types, until updated to handle them, which should be a simple coding change.
Author
|
@pwnall PTAL |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds two new chunk types to the Snappy framing format: compressed data
without a checksum, and uncompressed data without a checksum. These
types are identical to their existing counterparts except they do not
contain a CRC-32C checksum. Essentially, this makes including
checksums for each data chunk optional rather than required.
In some use cases, computing the CRC-32C checksums for the data chunks
in the Snappy framing format ends up dominating execution time.
Eliminating the checksums provides massive 2.5x performance
improvements in our uses of Snappy for compressing address trace data
prior to storing to disk.
Existing readers of the Snappy framing format would be expected to
fail up front on an unknown chunk type when encountering the new
types, until updated to handle them, which should be a simple coding
change.