IPIP-0524: Remove cross-codec conversion from HTTP Gateways by achingbrain · Pull Request #524 · ipfs/specs

achingbrain · 2025-11-27T11:07:26Z

The Accept-header driven translation between various IPLD formats is lossy, complicated, and often has us jumping through weird hoops to massage data into an acceptable format. Maybe we can do without it?

Simplifying the spec will make creating server implementations easier and the translation can still be done by clients if they require it.

Refs:

bug: path-gateway requiring ipld data model conversions (PB/JSON/CBOR) gateway-conformance#200
test!(ipip-524/v0.10): codec mismatch returns 406 gateway-conformance#254
- GO implementation:
  - feat(gateway): IPIP-0524 + AllowCodecConversion config option boxo#1077
    - feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option kubo#11090
- JS impl.
  - ?

The Accept-header driven translation between various IPLD formats is lossy, complicated, and often has us jumping through weird hoops to massage data into an acceptable format. Maybe we can do without it? Simplifying the spec will make creating server implementations easier and the translation can still be done by clients if they require it. Refs: ipfs/gateway-conformance#200

github-actions · 2025-11-27T11:09:01Z

🚀 Build Preview on IPFS ready

🔎 Commit: d4999d4
🔏 CID bafybeicuw3qh2nyrxtpxttsueuu7zl4yhy7oorhmrubwvjxhxgjbumjhba
📦 Preview:

Update tests to reflect IPIP-0524 which removes codec conversions from the gateway spec. Gateways now return 406 Not Acceptable when the requested format doesn't match the block's codec. - Replace TestDagPbConversion with TestCodecMismatchReturns406 - Remove codec conversion tests (dag-pb→dag-json, dag-cbor→dag-json) - Fix TestPathing to request matching codec format - Add body verification for DAG-CBOR traversal test - Clarify that HTML rendering remains in spec (unlike conversions) Implementations supporting optional codec conversions for backward compatibility are free to skip the 406 tests. Ref: ipfs/specs#524

Add AllowCodecConversion to gateway.Config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. When true, conversions between codecs are performed for backward compatibility. Codec conversion tests moved here from gateway-conformance since conversions are now an optional implementation feature, not a spec requirement. Gateway-conformance now tests for 406 responses. Ref: ipfs/specs#524 Ref: ipfs/gateway-conformance#254

Wire up boxo's AllowCodecConversion config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. Clients should fetch raw blocks (`?format=raw`) and convert client-side. Ref: ipfs/specs#524 Ref: ipfs/boxo#1077 Ref: ipfs/gateway-conformance#254

lidel

Small clarification question inline, but other than that looks sensible, removes complexity related to a niche feature which can be implemented in userland thanks to ?format=raw.

Cleaned up conformance in:

ipfs/gateway-conformance#254

GO implementation in Boxo/Kubo allows opt-in to previous behavior, just in case anyone depended on this niche feature from IPLD era:

ipfs/boxo#1077
- ipfs/kubo#11090

lidel · 2025-12-03T01:14:51Z

src/http-gateways/path-gateway.md

+- [application/json](https://www.iana.org/assignments/media-types/application/json) – Validates block data as JSON before returning it as-is. Invalid JSON produces 406 Not Acceptable.
+- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – Validates block data as CBOR before returning it as-is. Invalid CBOR produces 406 Not Acceptable.


@achingbrain how strong you feel about doing actual JSON or CBOR parsing here, when plain version is requested?

The main purpose here, in my mind, is not data validation, but to maximize interop ensure correct Content-Type in response, as there are tools which will make request but wont parse response if its not application/json etc.

Could we simplify spec here noting that if requested CID codec is matching [dag-]json or [dag-]cbor, we dont parse payload, but return the same response as raw but with specific vanilla content type.

It makes implementation simpler (no need to deal with CBOR/JSON parsing edge cases), reduces cost of running gateway (no JSON/CBOR parsing), but we still have the interop benefit of returning vanilla JSON/CBOR when necessary by a third-party system.

how strong you feel about doing actual JSON or CBOR parsing here, when plain version is requested?

The intention was to apply the robustness principle and let the client know what they have requested is not possible.

That is, if they use the accept header to say they want application/json, and the server responds with a content-type that says yes no problem, this is application/json, then they parse it and it turns out to not be application/json then the content-type turns out to be a guess and it doesn't really mean anything.

In reality I'm not sure how achievable this is because even our own tools are not that strict.

The HTTP spec also says:

In practice, resource owners do not always properly configure their origin server to provide the correct Content-Type for a given representation.

..so perhaps it is ok for it just to be a guess.

I'm ok to remove this.

Relaxed language to allow optimization where dag-cbor validation MAY be skipped when CID codec matches the requested format and the response is equivalent to format=raw with a different Content-Type.

This way a simple barebones gateway implementations won't have to know what dag-cbor is if they do not not support path traversal and only support direct /ipfs/cid

…-translation

soften path-gateway.md wording: validation MAY be skipped when CID codec matches the requested format, since the response is equivalent to format=raw with a different Content-Type.

restructure to match idiomatic IPIP format: expand User benefit with cross-library interop argument, add Compatibility sub-headings (json independence, UnixFS exception, opt-in, implementation-defined), add Security section, move test list into Test fixtures. clarify `application/json` Accept handling for UnixFS files in path gateway spec.

add missing 406 Not Acceptable subsection to the Response Status Codes, remove "or converted to DAG-CBOR/DAG-JSON" from the Response Payload section to align with IPIP-0524 removing server-side codec conversions.

* test: IPIP-0524 codec mismatch returns 406 Update tests to reflect IPIP-0524 which removes codec conversions from the gateway spec. Gateways now return 406 Not Acceptable when the requested format doesn't match the block's codec. - Replace TestDagPbConversion with TestCodecMismatchReturns406 - Remove codec conversion tests (dag-pb→dag-json, dag-cbor→dag-json) - Fix TestPathing to request matching codec format - Add body verification for DAG-CBOR traversal test - Clarify that HTML rendering remains in spec (unlike conversions) Implementations supporting optional codec conversions for backward compatibility are free to skip the 406 tests. Ref: ipfs/specs#524 * fix: remove remaining codec conversion tests - Replace TestNativeDag explicit DAG-JSON format with matching format - Replace cache tests using format=dag-json/json with format=raw - Add body verification for raw block responses IPIP-0524 removes codec conversions from gateway spec, so tests should not expect 200 for cross-codec requests. * docs: add v0.10.0 changelog entry for IPIP-524 --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>

* feat(gateway): add AllowCodecConversion config option Add AllowCodecConversion to gateway.Config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. When true, conversions between codecs are performed for backward compatibility. Codec conversion tests moved here from gateway-conformance since conversions are now an optional implementation feature, not a spec requirement. Gateway-conformance now tests for 406 responses. Ref: ipfs/specs#524 Ref: ipfs/gateway-conformance#254 * ci: update gateway-conformance to e17586f4 * fix(gateway): 406 error now tells you how to get the data you need when codec conversion is disabled (IPIP-0524) and you request ?format=dag-json for a dag-pb block, the 406 response now suggests fetching the raw block with ?format=raw and converting client-side. covers dag-pb directories, dag-pb files, and raw blocks requested with ?format=dag-json or ?format=dag-cbor. plain ?format=json and Accept: application/json continue to serve the default response, so existing HTTP clients are not affected. * chore: update changelog with PR link for IPIP-0524 * fix(gateway): show only native codec download link in HTML preview only show the matching format download link (dag-json or dag-cbor) in the HTML preview page when AllowCodecConversion is disabled, preventing users from clicking links that would return 406. when AllowCodecConversion is enabled, both links are shown as before. * ci: switch gateway-conformance to v0.10

…11090) * feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option Wire up boxo's AllowCodecConversion config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. Clients should fetch raw blocks (`?format=raw`) and convert client-side. Ref: ipfs/specs#524 Ref: ipfs/boxo#1077 Ref: ipfs/gateway-conformance#254 * chore: update boxo for improved 406 codec conversion error boxo now returns an actionable hint when codec conversion is rejected: suggests fetching raw block with ?format=raw and converting client-side. * chore: bump boxo and gateway-conformance to v0.10 * docs: add IPLD Logical Format note to AllowCodecConversion

lidel

Status update: all implementation PRs are now merged.

spec (this PR): added 406 Not Acceptable to response status codes section, removed stale "or converted to DAG-CBOR/DAG-JSON" wording from response payload (2bb0197)
gateway-conformance: ipfs/gateway-conformance#254 merged, released as https://github.com/ipfs/gateway-conformance/releases/tag/v0.10.0 -- codec conversion tests now expect 406
boxo: ipfs/boxo#1077 merged into main -- adds opt-in Config.AllowCodecConversion flag for users to make migration easier, 406 response with actionable error hint, and conditional download links in HTML preview
kubo: ipfs/kubo#11090 merged -- exposes Gateway.AllowCodecConversion in config, bumps boxo and gateway-conformance to v0.10, ships in ipfs/kubo#11008
js does not require changes, as it never implemented conversions i think
- @achingbrain to double check you may want to switch helia/verified fetch etc to use gateway-conformance@v0.10

This spec PR is the last piece. Ready for final feedback + tests in Kubo master / 0.40.0-rc1 later this week.

If no concerns, we will merge this once 0.40.0-rc1 ships.

add ?format= usage breakdown from ipfs.io/dweb.link public gateways showing that no real-world traffic depends on cross-codec conversion. rename IPIP to use clearer terminology.

lidel · 2026-02-06T17:38:26Z

Added (d4999d4) real-world traffic data from ipfs.io and dweb.link public gateways (24h sample, Feb 2026) showing that 100% of ?format=json requests target dag-pb or `raw blocks (UnixFS files), with zero requests relying on cross-codec conversion.

That is to say this IPIP should have zero real world impact, as literally nobody in past 24h depended on IPLD's cross-codec conversions.

achingbrain mentioned this pull request Nov 27, 2025

bug: path-gateway requiring ipld data model conversions (PB/JSON/CBOR) ipfs/gateway-conformance#200

Open

achingbrain marked this pull request as ready for review November 27, 2025 13:51

achingbrain requested review from a team and lidel as code owners November 27, 2025 13:51

This was referenced Dec 1, 2025

Protocol stewardship and improvements — IPFS/2025 ipshipyard/roadmaps#16

Closed

UX parity with with boxo/gateway for GA ipfs/service-worker-gateway#891

Open

lidel mentioned this pull request Dec 3, 2025

test!(ipip-524/v0.10): codec mismatch returns 406 ipfs/gateway-conformance#254

Merged

lidel mentioned this pull request Dec 3, 2025

feat(gateway): IPIP-0524 + AllowCodecConversion config option ipfs/boxo#1077

Merged

lidel mentioned this pull request Dec 3, 2025

feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option ipfs/kubo#11090

Merged

2 tasks

lidel reviewed Dec 3, 2025

View reviewed changes

achingbrain mentioned this pull request Dec 5, 2025

Decoding with simple type 23 does not throw? ipld/js-dag-cbor#131

Open

lidel self-assigned this Dec 10, 2025

lidel mentioned this pull request Dec 15, 2025

Release 0.40 ipfs/kubo#11008

Open

57 tasks

lidel added 5 commits February 5, 2026 18:46

Merge remote-tracking branch 'origin/main' into feat/ipip-remove-ipld…

8e1b715

…-translation

docs: clarify IPIP-0524 validation is optional for matching codecs

b1aae97

soften path-gateway.md wording: validation MAY be skipped when CID codec matches the requested format, since the response is equivalent to format=raw with a different Content-Type.

Merge branch 'main' into feat/ipip-remove-ipld-translation

4d8b03b

docs: add 406 status code section, remove stale conversion wording

2bb0197

add missing 406 Not Acceptable subsection to the Response Status Codes, remove "or converted to DAG-CBOR/DAG-JSON" from the Response Payload section to align with IPIP-0524 removing server-side codec conversions.

lidel approved these changes Feb 6, 2026

View reviewed changes

docs: add real-world traffic data, clarify title

d4999d4

add ?format= usage breakdown from ipfs.io/dweb.link public gateways showing that no real-world traffic depends on cross-codec conversion. rename IPIP to use clearer terminology.

lidel changed the title ~~IPIP-0524: Remove IPLD translation from IPFS HTTP Gateway~~ IPIP-0524: Remove cross-codec conversion from HTTP Gateways Feb 6, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IPIP-0524: Remove cross-codec conversion from HTTP Gateways#524

IPIP-0524: Remove cross-codec conversion from HTTP Gateways#524
achingbrain wants to merge 7 commits intomainfrom
feat/ipip-remove-ipld-translation

achingbrain commented Nov 27, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 27, 2025 •

edited

Loading

Uh oh!

lidel left a comment •

edited

Loading

Uh oh!

lidel Dec 3, 2025

Uh oh!

achingbrain Dec 5, 2025

Uh oh!

lidel Feb 5, 2026 •

edited

Loading

Uh oh!

lidel left a comment •

edited

Loading

Uh oh!

lidel commented Feb 6, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		- [application/json](https://www.iana.org/assignments/media-types/application/json) – Validates block data as JSON before returning it as-is. Invalid JSON produces 406 Not Acceptable.
		- [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – Validates block data as CBOR before returning it as-is. Invalid CBOR produces 406 Not Acceptable.

Conversation

achingbrain commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Build Preview on IPFS ready

Uh oh!

lidel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

achingbrain Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

lidel Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

achingbrain commented Nov 27, 2025 •

edited

Loading

github-actions bot commented Nov 27, 2025 •

edited

Loading

lidel left a comment •

edited

Loading

lidel Feb 5, 2026 •

edited

Loading

lidel left a comment •

edited

Loading

lidel commented Feb 6, 2026 •

edited

Loading