feat(config): add Import.* for CID Profiles from IPIP-499 #11148

lidel · 2026-01-17T02:12:39Z

Closes Implement modern CID profile from IPIP-499 #11071
This PR is wiring up IPIP-499: UnixFS CID Profiles specs#499
- Depends on feat: add --dereference-symlinks flag for recursive symlink resolution go-ipfs-cmds#315
- Depends on feat(unixfs): configurable CID Profiles from IPIP-499 boxo#1088

Users can now pin down import settings to get reproducible CIDs across software updates.

Apply a profile: ipfs config profile apply unixfs-v1-2025

Two profiles available:

unixfs-v1-2025: modern CIDv1 with improved defaults
unixfs-v0-2015 (alias legacy-cid-v0): best-effort legacy CIDv0 behavior

New config options for fine-tuning:

Import.UnixFSHAMTDirectorySizeEstimation: controls HAMT threshold estimation (links, block, disabled)
Import.UnixFSDAGLayout: choose balanced or trickle layout

New ipfs add flags:

--dereference-symlinks: resolve all symlinks to target content (replaces deprecated --dereference-args)
--empty-dirs / -E: include empty directories
--hidden / -H: include hidden files

Removes test-cid-v1 and test-cid-v1-wide profiles.

Tests cover profile determinism, HAMT threshold boundaries, and symlink handling.

Related: IPIP-499, depends on boxo#1088 and go-ipfs-cmds#315

implements IPIP-499: add config options for controlling UnixFS DAG determinism and introduces `unixfs-v1-2025` and `unixfs-v0-2015` profiles for cross-implementation CID reproducibility. changes: - add Import.* fields: HAMTDirectorySizeEstimation, SymlinkMode, DAGLayout, IncludeEmptyDirectories, IncludeHidden - add validation for all Import.* config values - add unixfs-v1-2025 profile (recommended for new data) - add unixfs-v0-2015 profile (alias: legacy-cid-v0) - remove deprecated test-cid-v1 and test-cid-v1-wide profiles - wire Import.HAMTSizeEstimationMode() to boxo globals - update go.mod to use boxo with SizeEstimationMode support ref: https://specs.ipfs.tech/ipips/ipip-0499/

add CLI flags for controlling file collection behavior during ipfs add: - `--dereference-symlinks`: recursively resolve symlinks to their target content (replaces deprecated --dereference-args which only worked on CLI arguments). wired through go-ipfs-cmds to boxo's SerialFileOptions. - `--empty-dirs` / `-E`: include empty directories (default: true) - `--hidden` / `-H`: include hidden files (default: false) these flags are CLI-only and not wired to Import.* config options because go-ipfs-cmds library handles input file filtering before the directory tree is passed to kubo. removed unused Import.UnixFSSymlinkMode config option that was defined but never actually read by the CLI. also: - wire --trickle to Import.UnixFSDAGLayout config default - update go-ipfs-cmds to v0.15.1-0.20260117043932-17687e216294 - add SYMLINK HANDLING section to ipfs add help text - add CLI tests for all three flags ref: ipfs/specs#499

add comprehensive test suite for UnixFS CID determinism per IPIP-499: - verify exact HAMT threshold boundary for both estimation modes: - v0-2015 (links): sum(name_len + cid_len) == 262144 - v1-2025 (block): serialized block size == 262144 - verify HAMT triggers at threshold + 1 byte for both profiles - add all deterministic CIDs for cross-implementation testing also wires SizeEstimationMode through CLI/API, allowing Import.UnixFSHAMTSizeEstimation config to take effect. bumps boxo to ipfs/boxo@6707376 which aligns HAMT threshold with JS implementation (uses > instead of >=), fixing CID determinism at the exact 256 KiB boundary.

Previously, resolving symlinks required two flags: - --dereference-args: resolved symlinks passed as CLI arguments - --dereference-symlinks: resolved symlinks inside directories Now --dereference-symlinks handles both cases. Users only need one flag to fully dereference symlinks when adding files to IPFS. The deprecated --dereference-args still works for backwards compatibility but is no longer necessary.

- update boxo to ebdaf07c (nil filter fix, thread-safety docs) - simplify changelog for IPIP-499 section - shorten test names, move context to comments

lidel · 2026-01-20T02:27:06Z

I may add more tests or improve code, but its ready for initial review, to course correct-early.

gammazero

All code looks good, and it looks like all tests cases are covered.

config/import.go

test/cli/cid_profiles_test.go

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>

add test that confirms kubo uses balanced layout (all leaves at same depth) rather than balanced-packed (varying depths). creates 45MiB file to trigger multi-level DAG and walks it to verify leaf depth uniformity. includes trickle subtest to validate test logic can detect varying depths. supports CAR export via DAG_LAYOUT_CAR_OUTPUT env var for test vectors.

switches to ipfs/boxo@6141039 changes since 5cf22196ad0b: - refactor(unixfs): use arithmetic for exact block size calculation - refactor(unixfs): unify size tracking and make SizeEstimationMode immutable - feat(unixfs): optimize SizeEstimationBlock and add mode/mtime tests also clarifies that directory sharding globals affect both `ipfs add` and MFS.

- add UnixFSDataType() helper to directly check UnixFS type via protobuf - refactor threshold tests to use exact +1 byte calculations instead of +1 file - verify directory type directly (ft.TDirectory vs ft.THAMTShard) instead of inferring from link count - clean up helper function signatures by removing unused cidLength parameter

remove duplicate profile threshold tests from add_test.go since they are fully covered by the data-driven tests in cid_profiles_test.go. changes: - improve test names to describe what threshold is being tested - add inline documentation explaining each test's purpose - add byte-precise helper IPFSAddDeterministicBytes for threshold tests - remove ~200 lines of duplicated test code from add_test.go - keep non-profile tests (pinning, symlinks, hidden files) in add_test.go

…s-2025 # Conflicts: # docs/examples/kubo-as-a-library/go.mod # docs/examples/kubo-as-a-library/go.sum # go.mod # go.sum # test/dependencies/go.mod # test/dependencies/go.sum

…s-2025

lidel · 2026-01-31T01:28:08Z

Triage note:

found MFS bug durig final review while writing missing tests, Import.* are not correctly respected by ipfs files (we had no tests for this)
will fix next week before starting 0.40 RC1

make MFS commands (files cp, files write, files mkdir, files chcid) respect Import.CidVersion and Import.HashFunction config settings when CLI options are not explicitly provided. also add tests for: - files write respects Import.UnixFSRawLeaves=true - single-block file: files write produces same CID as ipfs add - updated comments clarifying CID parity with ipfs add

… MFS `ipfs files` commands now respect these Import.* config options: - UnixFSChunker: configures chunk size for `files write` - UnixFSDirectoryMaxLinks: triggers HAMT sharding in `files mkdir` - UnixFSHAMTDirectorySizeEstimation: controls size estimation mode previously, MFS used hardcoded defaults ignoring user config. changes: - config/import.go: add UnixFSSplitterFunc() returning chunk.SplitterGen - core/node/core.go: pass chunker, maxLinks, sizeEstimationMode to mfs.NewRoot() via new boxo RootOption API - core/commands/files.go: pass maxLinks and sizeEstimationMode to mfs.Mkdir() and ensureContainingDirectoryExists(); document that UnixFSFileMaxLinks doesn't apply to files write (trickle DAG limitation) - test/cli/files_test.go: add tests for UnixFSDirectoryMaxLinks and UnixFSChunker, including CID parity test with `ipfs add --trickle` related: boxo@54e044f1b265

…irectorySizeThreshold wire remaining HAMT config options to MFS root: - Import.UnixFSHAMTDirectoryMaxFanout via mfs.WithMaxHAMTFanout - Import.UnixFSHAMTDirectorySizeThreshold via mfs.WithHAMTShardingSize add CLI tests: - files mkdir respects Import.UnixFSHAMTDirectoryMaxFanout - files mkdir respects Import.UnixFSHAMTDirectorySizeThreshold - config change takes effect after daemon restart add UnixFSHAMTFanout() helper to test harness update boxo to ac97424d99ab90e097fc7c36f285988b596b6f05

problem: `ipfs files write` in CIDv1 directories wrapped single-block files in dag-pb even when raw-leaves was enabled, producing different CIDs than `ipfs add --raw-leaves` for the same content. fix: boxo now collapses single-block ProtoNode wrappers (with no metadata) to RawNode in DagModifier.GetNode(). files with mtime/mode stay as dag-pb since raw blocks cannot store UnixFS metadata. also fixes sparse file writes where writing past EOF would lose data because expandSparse didn't update the internal node pointer. updates boxo to v0.36.1-0.20260203003133-7884ae23aaff updates t0250-files-api.sh test hashes to match new behavior

- fix typo in files write help text - update boxo with CI fixes (gofumpt, race condition in test)

…s-2025 # Conflicts: # docs/examples/kubo-as-a-library/go.mod # docs/examples/kubo-as-a-library/go.sum # go.mod # go.sum # test/dependencies/go.mod # test/dependencies/go.sum

includes binary content types fix: gzip, zip, vnd.ipld.car, vnd.ipld.raw, vnd.ipfs.ipns-record

includes refactor of maxLinks check in addLinkChild (review feedback).

skip '@helia/mfs - should have the same CID after creating a file' test until helia implements IPIP-499 (tracking: ipfs/helia#941) the test fails because kubo now collapses single-block files to raw CIDs while helia explicitly uses reduceSingleLeafToSelf: false changes: - run aegir directly instead of helia-interop binary (binary ignores --grep flags) - cache node_modules keyed by @helia/interop version from npm registry - skip npm install on cache hit (matches ipfs-webui caching pattern)

lidel · 2026-02-03T21:30:34Z

.github/workflows/interop.yml

+      - name: Install @helia/interop
+        if: steps.helia-cache.outputs.cache-hit != 'true'
+        run: npm install @helia/interop
+      # TODO(IPIP-499): Remove --grep --invert workaround once helia implements IPIP-499


ℹ️ @achingbrain fyi I'm skipping that one test for now. while I wrote this is blocked until IPIP-499 is in Helia, seems that helia already bent backwards to simulate buggy behavior from Kubo (reduceSingleLeafToSelf: false). So maybe once 0.40.0-rc1 is tagged helia could switch to updated kubo and flip the flag and things will pass again (without waiting for IPIP-499)?

lidel · 2026-02-03T23:21:06Z

@gammazero small changes since your last review:

updated to latest feat(unixfs): configurable CID Profiles from IPIP-499 boxo#1088 with small fixes there:
- check fix and new doc.go documenting MFS semantics
- Updated boxo with maxLinks check refactor
Fixed helia-interop CI: skipping one test until helia implements IPIP-499 (raw leaf collapsing)
Merged latest master

All CI checks passing. If no concerns I will merge tomorrow to unblock RC1.

…s-2025 # Conflicts: # docs/examples/kubo-as-a-library/go.mod # docs/examples/kubo-as-a-library/go.sum # go.mod # go.sum # test/dependencies/go.mod # test/dependencies/go.sum

includes latest upstream changes from boxo main

gammazero

All additional changes look good.

lidel force-pushed the feat/ipip-499-unixfs-2025 branch 2 times, most recently from bf5578b to d79f7de Compare January 17, 2026 04:55

lidel force-pushed the feat/ipip-499-unixfs-2025 branch from d79f7de to 01b1ce0 Compare January 17, 2026 04:59

lidel mentioned this pull request Jan 18, 2026

fix: skip socket files in ipfs add to prevent errors #11149

Draft

lidel added 2 commits January 19, 2026 06:13

lidel mentioned this pull request Jan 20, 2026

feat(unixfs): configurable CID Profiles from IPIP-499 ipfs/boxo#1088

Open

chore: update boxo and improve changelog

088f1a7

- update boxo to ebdaf07c (nil filter fix, thread-safety docs) - simplify changelog for IPIP-499 section - shorten test names, move context to comments

lidel marked this pull request as ready for review January 20, 2026 02:26

lidel requested a review from a team as a code owner January 20, 2026 02:26

gammazero approved these changes Jan 21, 2026

View reviewed changes

config/import.go Outdated Show resolved Hide resolved

test/cli/cid_profiles_test.go Outdated Show resolved Hide resolved

lidel and others added 3 commits January 22, 2026 01:25

chore: update boxo to 5cf22196

25ed654

chore: apply suggestions from code review

00fb95d

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>

This was referenced Jan 24, 2026

IPIP-499: UnixFS CID Profiles ipfs/specs#499

Open

Support IPIP-499 UnixFS CID Profiles ipfs/helia#941

Open

lidel added 5 commits January 27, 2026 21:51

Merge remote-tracking branch 'origin/master' into feat/ipip-499-unixf…

9670bfe

…s-2025 # Conflicts: # docs/examples/kubo-as-a-library/go.mod # docs/examples/kubo-as-a-library/go.sum # go.mod # go.sum # test/dependencies/go.mod # test/dependencies/go.sum

chore: update to rebased boxo and go-ipfs-cmds PRs

800cba9

lidel force-pushed the feat/ipip-499-unixfs-2025 branch from 3e4059b to 800cba9 Compare January 28, 2026 00:17

docs: add HAMT threshold fix details to changelog

360df63

lidel mentioned this pull request Jan 30, 2026

feat(rpc): Content-Type headers and IPNS record get/put #11067

Merged

Merge remote-tracking branch 'origin/master' into feat/ipip-499-unixf…

05225ca

…s-2025

lidel self-assigned this Jan 30, 2026

lidel marked this pull request as draft February 1, 2026 21:49

lidel added 2 commits February 2, 2026 02:00

lidel force-pushed the feat/ipip-499-unixfs-2025 branch from a152109 to 340b0ad Compare February 3, 2026 00:38

lidel force-pushed the feat/ipip-499-unixfs-2025 branch from 340b0ad to c2d414f Compare February 3, 2026 00:44

chore(test): use Go 1.22+ range-over-int syntax

32b629c

lidel marked this pull request as ready for review February 3, 2026 15:32

lidel added 4 commits February 3, 2026 18:33

chore: update boxo to c6829fe26860

7fdb2c2

- fix typo in files write help text - update boxo with CI fixes (gofumpt, race condition in test)

Merge remote-tracking branch 'origin/master' into feat/ipip-499-unixf…

85d9c20

…s-2025 # Conflicts: # docs/examples/kubo-as-a-library/go.mod # docs/examples/kubo-as-a-library/go.sum # go.mod # go.sum # test/dependencies/go.mod # test/dependencies/go.sum

chore: update go-ipfs-cmds to 192ec9d15c1f

44793f5

includes binary content types fix: gzip, zip, vnd.ipld.car, vnd.ipld.raw, vnd.ipfs.ipns-record

chore: update boxo to 0a22cde9225c

6bb4c1a

includes refactor of maxLinks check in addLinkChild (review feedback).

lidel force-pushed the feat/ipip-499-unixfs-2025 branch from af565e3 to eca0b5d Compare February 3, 2026 19:21

lidel force-pushed the feat/ipip-499-unixfs-2025 branch from eca0b5d to a018d14 Compare February 3, 2026 20:11

lidel commented Feb 3, 2026

View reviewed changes

lidel added 3 commits February 4, 2026 05:52

Merge remote-tracking branch 'origin/master' into feat/ipip-499-unixf…

2701715

…s-2025 # Conflicts: # docs/examples/kubo-as-a-library/go.mod # docs/examples/kubo-as-a-library/go.sum # go.mod # go.sum # test/dependencies/go.mod # test/dependencies/go.sum

chore: update boxo to 1e30b954

4999924

includes latest upstream changes from boxo main

chore: update go-ipfs-cmds to 1b2a641ed6f6

e69b33f

lidel force-pushed the feat/ipip-499-unixfs-2025 branch from 92e51ed to e69b33f Compare February 4, 2026 05:43

gammazero approved these changes Feb 4, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(config): add Import.* for CID Profiles from IPIP-499 #11148

feat(config): add Import.* for CID Profiles from IPIP-499 #11148

Uh oh!

lidel commented Jan 17, 2026 •

edited

Loading

Uh oh!

lidel commented Jan 20, 2026

Uh oh!

gammazero left a comment

Uh oh!

Uh oh!

Uh oh!

lidel commented Jan 31, 2026

Uh oh!

lidel Feb 3, 2026 •

edited

Loading

Uh oh!

lidel commented Feb 3, 2026

Uh oh!

gammazero left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

feat(config): add Import.* for CID Profiles from IPIP-499 #11148

Are you sure you want to change the base?

feat(config): add Import.* for CID Profiles from IPIP-499 #11148

Uh oh!

Conversation

lidel commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lidel commented Jan 20, 2026

Uh oh!

gammazero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lidel commented Jan 31, 2026

Uh oh!

lidel Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel commented Feb 3, 2026

Uh oh!

gammazero left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lidel commented Jan 17, 2026 •

edited

Loading

lidel Feb 3, 2026 •

edited

Loading