Skip to content

fix: allow dag import of 1MiB chunks wrapped in dag-pb#11185

Merged
lidel merged 1 commit intomasterfrom
fix-big-blocks
Feb 6, 2026
Merged

fix: allow dag import of 1MiB chunks wrapped in dag-pb#11185
lidel merged 1 commit intomasterfrom
fix-big-blocks

Conversation

@lidel
Copy link
Member

@lidel lidel commented Feb 6, 2026

This is mostly a bunch of tests + small fix for dag import.

IPIP-499's (#11148) unixfs-v1-2025 profile uses 1MiB chunks. with --raw-leaves=false, protobuf wrapping pushes blocks slightly over 1MiB. the previous 1MiB SoftBlockLimit rejected these blocks on dag import.

This PR raises SoftBlockLimit to 2MiB to match the bitswap spec, which requires implementations to support blocks up to 2MiB.

  • raise SoftBlockLimit to 2MiB per the bitswap spec
    • this is soft limit in CLI when someone tries to import with chunk size bigger than the limit. block put etc errors. user can override and force with --allow-big-block
    • update error messages and help text
  • bump boxo (raised ChunkSizeLimit/BlockSizeLimit, 256-byte overhead budget)
  • update sharness tests for 2MiB boundary
  • add test/cli boundary tests for block put, dag put, dag import, ipfs add (raw and dag-pb wrapped leaves), and bitswap exchange including regression tests for the libp2p message size hard limit (4MiB minus size of CID-related overhead)

IPIP-499's unixfs-v1-2025 profile uses 1MiB chunks. with
--raw-leaves=false, protobuf wrapping pushes blocks slightly over 1MiB.
the previous 1MiB SoftBlockLimit rejected these blocks on dag import.

raise SoftBlockLimit to 2MiB to match the bitswap spec, which requires
implementations to support blocks up to 2MiB.

- raise SoftBlockLimit to 2MiB per the bitswap spec
- update error messages and help text
- bump boxo to main with ipfs/boxo#1101 (raised ChunkSizeLimit/BlockSizeLimit,
  256-byte overhead budget)
- update sharness tests for 2MiB boundary
- add test/cli boundary tests for block put, dag put, dag import,
  ipfs add (raw and wrapped leaves), and bitswap exchange including
  regression tests for the libp2p message size hard limit
@lidel lidel marked this pull request as ready for review February 6, 2026 22:45
@lidel lidel requested a review from a team as a code owner February 6, 2026 22:45
@lidel
Copy link
Member Author

lidel commented Feb 6, 2026

Merging when CI finishes to unblock 0.40.0-rc1.

This is a simple fix with low risk: existing users are not affected, we are not changing any defaults, and if someone starts producing data with >1MiB && <2MiB chunks old software is able to fetch 2MiB blocks over bitswap just fine (real limit is slightly below 4MiB).

Most of the diff is regression tests we were missing: boundary tests for block put, dag put, dag import, ipfs add (raw and wrapped leaves), bitswap exchange between two nodes, and regression tests at the libp2p message size hard limit (~4MiB), so we catch regressions in case go-libp2p ships a breaking change in the future.

@lidel lidel merged commit c6702ea into master Feb 6, 2026
21 checks passed
@lidel lidel deleted the fix-big-blocks branch February 6, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant