jsdom fetching and resource loader rewrite based on undici

[domenic]

Hi Undici community,

I've recently completed a big rewrite of jsdom's fetching and resource loading functionality on top of undici. The PR contains full details on the motivations and API changes from a jsdom perspective.

I want to thank the Undici maintainers for being quite responsive to my bug reports, which help make this feasible.

I would welcome review from the community on my usage of Undici and if there are any better ways to get around the rough edges I encountered. I'll excerpt a part of my PR description that might be of interest:

This work stresses the Undici system significantly, in particular, by creating a custom Dispatcher. It feels like Undici was aspirationally designed to allow custom Dispatchers, but not very many have been written by people other than the Undici authors.

The biggest friction point is that Undici seems to be in the middle of some sort of transition from an old (undocumented) handler API to a new handler API, but this transition is incomplete. Undici itself will often assume your dispatcher works with the old API, and blow up if it only works with the new API. (#4771 and #4780 were some cases I was able to minimize.)

We mitigate this by using the Undici-internal WrapHandler, which generates a wrapper so that jsdom's dispatcher only needs to call the new-API handlers. However, this is a bit scary since WrapHandler is not exported by the undici package's main module, and so is arguably not part of the public API.

There's also a lot that needs to be done carefully when implementing one's own Dispatcher, especially if that Dispatcher doesn't just decorate an existing Dispatcher. The "controller" API in particular is not documented, but implementing it correctly is crucial to avoiding errors when Undici calls into your dispatcher. It would be nice if we could just use Undici's existing controllers, but they are only given to use in onRequestStart, which is often behind an async hop, whereas we have several cases where we want to allow early aborting. This means creating facade controllers that eventually forward to real Undici controllers.

Another issue that occurred is that Unidici does not accept a simple "URL" option for its dispatcher, but instead a combination origin + path + query (with the latter two possibly overlapping). This cannot be used to represent file: URLs, which people often want to fetch with jsdom. So we have jsdom's dispatcher take an optional url string in its opaque option, which if present, overrides origin + path + query. This feels inelegant, and I wish Undici would just accept URLs, but I guess Undici is focused on HTTP(S), and not other cases like data: and file:.

In general, I remain uncertain if we've fully implemented the Dispatcher contract correctly. But, the test suite is reasonably exhaustive, so it's probably good enough for now...

In terms of files from the PR that are of interest to the Undici community:

• jsdom-dispatcher.js: our custom dispatcher. It handles file: and data: URLs independently from undici, then calls into a given "base" dispatcher (by default getGlobalDispatcher()) for HTTP(S) and WebSocket fetches. Along the way it does a lot of jsdom-specific stuff around CORS and cookie handling.
• request-interceptor.js: our helper function to let people write a Undici interceptor that goes Request → Promise<Response>. This illustrates some of the complexities around controllers and handlers that I discuss above, with multiple layers of wrapping (createSignalHandler + sendStreamResponse) necessary to get the semantics I want to present to our users.
• xhr-utils.js illustrates how, once we have the custom jsdom dispatcher in place, calling it from another part of the codebase is mostly straightforward. (Except the adapter having to deal with abort signals vs. Undici controllers...)

Thanks again for a wonderful library!