ipfs add: parameter to control the width of created UnixFS DAG

[lidel]

TLDR

There is no CLI / RPC / Config option to control max DAG width in Kubo when creating a UnixFS DAG.

Users who want to fully control the final CID need to be able to control this.

Details

and iiuc the implicit defaults are different for files and directories:

• For HAMT-sharded directories, the DefaultShardWidth here is 256
  • my rudimentary understanding is this is fanout from Unixfs spec but has to be confirmed – see Caveats section below
• For files, DefaultLinksPerBlock here is ~174

ability to pass a static dag width for files could be exposed via ipfs add --help the same way as --chunker and --hash are, but it requires PR against boxo and kubo.

Proposed design

• Add CLI/RPC parameter dag-width ipfs add --dag-width=N with default being defined in config at Import.UnixFSDAGWidth, similar to Import.UnixFSChunker.
• Make default in legacy-cid-v0 profile match current default (174?)
• Consider going with higher value in test-cid-v1 – perhaps 1024, matching Storacha
• This very likely requires changes to boxo to expose ability to override mentioned Default..

Acceptance criteria / caveats

• keep current defaults if possible.
• setting legacy-cid-v0 profile should ensure current behavior does not change
  • we may need separate Import.UnixFSDAGWidth and Import.UnixFSHAMTWidth just to make this profile possible.
• confirm HAMT-directories continue to work with legacy software
  • confirm custom dag width is compatible with HAMT-directories, namely, what is the relation between dag width parameter here and the fanout of HAMTShard UnixFS node type. if 256 is a hard requirement, keep it. If not, make Import.UnixFSHAMTWidth follow Import.UnixFSDAGWidth unless both are set to different values by the user

Related work

• feat(config): ipfs add and Import options for controling UnixFS DAG Width #10774
  • feat(unixfs/mfs): support MaxLinks and MaxHAMTFanout boxo#906
  • feat(ipld/unixfs): DagModifier: allow specifying MaxLinks per file boxo#898

[hsanjuan]

Given that the tendency now is to try to come to a sane set of defaults, do we want to expose this?

Users who want to fully control the final CID need to be able to control this.

There must be an interesting story behind this?

[2color]

Given that the tendency now is to try to come to a sane set of defaults, do we want to expose this?

The context in which this came up is this discussion. If we change the defaults here to match current Helia/Storacha defaults, this may confuse users with data already onboarded and merkleised.

By allowing this config to be configurable, we can ensure CID equivalency for the same input across implementations.

[lidel]

Implemented in #10774, ETA Kubo 0.35.0-rc1