Odoc v3 rules

[jonludlam]

This PR contains a rewrite of the odoc rules in dune to unlock some of the features of odoc v3.

The focus so far has been to build a replacement the rules behind both dune build @doc and dune build @doc-new which were almost entirely separately implemented. Instead, this PR introduces a unified approach which satisfies the use cases of both of the previous targets. Of the new odoc v3 features, I've not yet implemented images and other artifacts nor source rendering.

The code in this PR has been entirely generated by Claude Code. I did not write a single line of it, though I have spent a long time looking over it!

What's new for dune build @doc?

Building the alias @doc will build the docs for the non-vendored packages found in the current dune project, as before. However, there are also some new additions:

Errors

Errors are only reported for code in the current project. This should make it much easier to fix all documentation errors. For example, in dune as of this PR, running dune build @doc reports:

File "otherlibs/dune-build-info/src/build_info.mli", line 1, characters 8-13:
Warning: Failed to resolve reference unresolvedroot(V1) Couldn't find "V1"
File "otherlibs/dune-rpc/private/dbus_address.mli", line 10, characters 4-33:
Warning: '6': bad heading level (0-5 allowed).
File "_odoc/stdune/stdune/stdune.odoc":   
Warning: Hidden fields in type 'Stdune.User_message.t'
File "_odoc/stdune/stdune/stdune.odoc":
Warning: Hidden fields in type 'Stdune.Code_error.t'
File "otherlibs/stdune/src/string.mli", line 72, characters 31-40:
Warning: Failed to resolve reference unresolvedroot(escape) Couldn't find "escape"

In contrast, building this with the rules from dune 3.20 give 2 more unfixable warnings, and building @doc-new gives about 500 lines of output telling you all the errors in all the packages that dune depends on!

This should play nicely with the pre-existing option to enable warnings-as-errors in odoc, via an env stanza in dune:

(env
 (_
  (odoc
   (warnings fatal))))

Sidebar

Documentation will be built by default with a global sidebar, which covers all of the packages present in your build tree. This can optionally be a per-package sidebar via an option in dune-workspace:

(env
 (_
  (odoc
   (sidebar per-package))))

Hierarchical docs

Documentation pages (mld files) may be arranged into a tree structure. This builds upon work earlier in the year by @panglesd. The syntax is:

(documentation
  (package mypkg)
  (files
    (glob_files_rec (*.mld with_prefix .))))

Links to ocaml.org

Where your documentation references modules, types, values, or anything else in libraries from your opam switch, these will now be correctly resolved, and in the HTML these will be turned into links to the relevant documentation on ocaml.org. Note that this is best effort and there are no guarantees that this will work correctly, as the docs on ocaml.org might be created with a different set of dependencies than are used in your project.

Extra package dependencies

If you wish to have your documentation link to modules or mlds in other packages, this is now possible via the new field in dune-project:

(documentation
  (depends <pkg>))

This, again, is building upon earlier work by @panglesd.

Changes

The alias @doc-json now builds json files in a separate path from the HTML: _build/default/_doc/_json

New aliases @doc-full and @doc-json-full

Building the new alias @doc-full will build the documentation for all relevant packages and libraries. This includes all private libraries in the project, all libraries from the opam switch used by the local packages, and any extra packages that have been specified via the documentation dependencies field in dune-project. This is the replacement for the @doc-new alias. The output will be found in _build/default/_doc/_html_full

There is also a new target @doc-json-full which builds the json files for all of the same packages and libraries as @doc-full. The output of this is found in _build/default/_doc/_json_full

Sherlodoc rule

There is a new rule to create the file _build/default/_doc/_sherlodoc/db.marshal - this is a complete sherlodoc db that indexes all of the files that are produced as part of @doc-full and can be queried using the sherlodoc CLI. For example, in the dune repo:

# dune build _build/default/_doc/_sherlodoc/db.marshal
# sherlodoc search --db _build/default/_doc/_sherlodoc/db.marshal 'Signal.t -> int' 
val Core.Signal.ascending : t -> t -> int
val Async_unix.Signal.ascending : t -> t -> int
val Core.Signal.Replace_polymorphic_compare.compare : t -> t -> int
val Core_thread.wait_signal : Core.Signal.t list -> int
val Stdune.Signal.to_int : t -> int
...

Caveats / TODOs

Currently this does not work with dune package management. The main piece of work to do would be to implement the module package_discovery.ml for dune pkg mode. The job of this module is to figure out which opam package contains each library. For dune-built packages, of course, this is trivial as the names of the libraries are derived from the name of the package. For packages built in other ways, this isn't necessarily the case, and this module looks up information from the opam switch to figure the mapping out.

Caching mostly works, but there is an odoc bug that needs to be fixed to make it work perfectly.

Questions

• Is it really the best approach to have the four different aliases (@doc, @doc-full, @doc-json and @doc-json-full)?

[jonludlam]

I have a blog post about the experience of writing this with Claude.

[jonludlam]

Also, this PR is built on top of #11716

[avsm]

I'm just trying this out by vendoring a bunch of libraries, and I notice that the build environment for the docs needs all the dependency of an opam package. For example, I depend on angstrom in my build, but dune build @doc:

> dune build @doc
File "/Users/avsm/src/git/knot/aoah-vendor2/_opam/lib/angstrom/META", line 1, characters 0-0:
Error: Library "angstrom-async" not found.
-> required by library "angstrom.async" in

(same for angstrom-lwt-unix and angstrom-unix, etc)

[jonludlam]

Angstrom is a slightly odd library in that it installs some dummy libraries that have dependencies that aren't installed. Presumably at some point the opam packages were split into separate lwt/async packages to avoid the dependency problems, and these libraries have been left behind for backward compatibility: angstrom.unix, angstrom.lwt-unix and angstrom.async, which depend on angstrom-unix, angstrom-lwt-unix and angstrom-async respectively.

For the @doc target I think we'll be able to fix this, but for the @doc-full target I think we'll need to install all of the dependencies. I'll have a go.

[avsm]

Makes sense! I'm also seeing some module references aren't being substituted in, but only in standalone mld files when I do the dune build @doc. I put together a monorepo to make this easier to reproduce after the holidays here https://tangled.org/anil.recoil.org/monopam-odocv3-dune-test (setup.sh will do the right pins and stuff) and notes on how I built it at https://anil.recoil.org/notes/aoah-2025-22