ipld/unixfs/io: MaxLinks and MaxHAMTFanout for directories#897
Closed
ipld/unixfs/io: MaxLinks and MaxHAMTFanout for directories#897
Conversation
Contributor
Author
|
(This is a supporting PR to being able to set dag-width in Kubo). |
hsanjuan
force-pushed
the
feat/max-links-in-directories
branch
2 times, most recently
from
2a0a3f5 to
fb96ebb
Compare
hsanjuan
changed the title
hsanjuan
force-pushed
the
feat/max-links-in-directories
branch
2 times, most recently
from
288fddd to
596680d
Compare
gammazero
approved these changes
Mar 28, 2025
Contributor
gammazero
left a comment
gammazero
left a comment
There was a problem hiding this comment.
Logic all looks good, tests too. Could only find some trivial code nits.
lidel
previously requested changes
Mar 28, 2025
Member
lidel
left a comment
lidel
left a comment
There was a problem hiding this comment.
Slight concern that WithMaxLinks alone does not allow users to hardcode current implicit defaults in this and other implementations, which is 174 or 1024 for Basic and 256 for HAMT fanout – comment inline.
WithMaxWidth() -------------- The new option sets effectively a maximum width to the directories. Currently, on a dynamic directory, the switching to-from Basic to HAMT is controlled by HAMTShardingSize, and the width of the HAMT shards by DefaultShardWidth. When WithMaxLinks() is specified, the switching is additionally controlled by the number of links exceeding the limit. In that case, MaxLinks is used as ShardWidth. The directory can only be converted back to BasicDirectory when the total number of links is below the limit. Backwards compatibility is kept and tests have been added. Note that when setting MaxLinks to a high number, it is possible that we still suffer automatic conversion to HAMT before hitting MaxLinks, if the estimated directory size is above 256KiB (as before). WithStat() ---------- Allows to set Stat on a new directory. WithCidBuilder() --------------- Allows to set the CidBuilder on a new directory.
hsanjuan
force-pushed
the
feat/max-links-in-directories
branch
from
596680d to
562f0e5
Compare
hsanjuan
dismissed
lidel’s stale review
New review requested. Changes addressed.
hsanjuan
changed the title
Contributor
Author
|
Let's continue on #906, as it has additionally touched the unixfs part a bit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR paves the way to support controlling and limiting the max number of links a directory can have by supporting option-based limitations on the unixfs directory implementation.
The result is that we can set a maximum number of links in Dynamic folders that will trigger conversion to HAMT, which has in turn it's own Fanout option.
This PR has a follow up at #906 .