fix: allow dag import of 1MiB chunks wrapped in dag-pb#11185
Merged
Conversation
IPIP-499's unixfs-v1-2025 profile uses 1MiB chunks. with --raw-leaves=false, protobuf wrapping pushes blocks slightly over 1MiB. the previous 1MiB SoftBlockLimit rejected these blocks on dag import. raise SoftBlockLimit to 2MiB to match the bitswap spec, which requires implementations to support blocks up to 2MiB. - raise SoftBlockLimit to 2MiB per the bitswap spec - update error messages and help text - bump boxo to main with ipfs/boxo#1101 (raised ChunkSizeLimit/BlockSizeLimit, 256-byte overhead budget) - update sharness tests for 2MiB boundary - add test/cli boundary tests for block put, dag put, dag import, ipfs add (raw and wrapped leaves), and bitswap exchange including regression tests for the libp2p message size hard limit
Member
Author
|
Merging when CI finishes to unblock 0.40.0-rc1. This is a simple fix with low risk: existing users are not affected, we are not changing any defaults, and if someone starts producing data with >1MiB && <2MiB chunks old software is able to fetch 2MiB blocks over bitswap just fine (real limit is slightly below 4MiB). Most of the diff is regression tests we were missing: boundary tests for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is mostly a bunch of tests + small fix for
dag import.IPIP-499's (#11148)
unixfs-v1-2025profile uses 1MiB chunks. with--raw-leaves=false, protobuf wrapping pushes blocks slightly over 1MiB. the previous 1MiBSoftBlockLimitrejected these blocks ondag import.This PR raises
SoftBlockLimitto 2MiB to match the bitswap spec, which requires implementations to support blocks up to 2MiB.SoftBlockLimitto 2MiB per the bitswap specblock putetc errors. user can override and force with--allow-big-blockChunkSizeLimit/BlockSizeLimit, 256-byte overhead budget)test/cliboundary tests forblock put,dag put,dag import,ipfs add(rawanddag-pbwrapped leaves), and bitswap exchange including regression tests for the libp2p message size hard limit (4MiB minus size of CID-related overhead)