Virtual File System for Node.js

[mcollina]

A first-class virtual file system module (node:vfs) with a provider-based architecture that integrates with Node.js's fs module and module loader.

Key Features

• Provider Architecture - Extensible design with pluggable providers:

  • MemoryProvider - In-memory file system with full read/write support
  • SEAProvider - Read-only access to Single Executable Application assets
  • VirtualProvider - Base class for creating custom providers

• Standard fs API - Uses familiar writeFileSync, readFileSync, mkdirSync instead of custom methods

• Mount Mode - VFS mounts at a specific path prefix (e.g., /virtual), clear separation from real filesystem

• Module Loading - require() and import work seamlessly from virtual files

• SEA Integration - Assets automatically mounted at /sea when running as a Single Executable Application

• Full fs Support - readFile, stat, readdir, exists, streams, promises, glob, symlinks

Example

const vfs = require('node:vfs');
const fs = require('node:fs');

// Create a VFS with default MemoryProvider
const myVfs = vfs.create();

// Use standard fs-like API
myVfs.mkdirSync('/app');
myVfs.writeFileSync('/app/config.json', '{"debug": true}');
myVfs.writeFileSync('/app/module.js', 'module.exports = "hello"');

// Mount to make accessible via fs module
myVfs.mount('/virtual');

// Works with standard fs APIs
const config = JSON.parse(fs.readFileSync('/virtual/app/config.json', 'utf8'));
const mod = require('/virtual/app/module.js');

// Cleanup
myVfs.unmount();

SEA Usage

When running as a Single Executable Application, bundled assets are automatically available:

const fs = require('node:fs');

// Assets are automatically mounted at /sea - no setup required
const config = fs.readFileSync('/sea/config.json', 'utf8');
const template = fs.readFileSync('/sea/templates/index.html', 'utf8');

Public API

const vfs = require('node:vfs');

vfs.create([provider][, options])  // Create a VirtualFileSystem
vfs.VirtualFileSystem              // The main VFS class
vfs.VirtualProvider                // Base class for custom providers
vfs.MemoryProvider                 // In-memory provider
vfs.SEAProvider                    // SEA assets provider (read-only)

---

Disclaimer: I've used a significant amount of Claude Code tokens to create this PR. I've reviewed all changes myself.

[nodejs-github-bot]

Review requested:

• @nodejs/single-executable
• @nodejs/test_runner

[github-actions]

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @avivkeller.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

[Ethan-Arrowood]

Nice! This is a great addition. Since it's such a large PR, this will take me some time to review. Will try to tackle it over the next week.

[avivkeller]

Can we use path.join?

[avivkeller]

Shouldn't we import this at the top level / lazy load it at the top level?

[avivkeller]

Suggested change

	restore: restoreFS,
	restore: ctx.restore,

nit

[avivkeller]

Can we use path.join?

[avivkeller]

Let's use path.join

[avivkeller]

path.join?

[avivkeller]

Suggested change

	}).catch((err) => {
	}, (err) => {

[avivkeller]

Primordials?

[avivkeller]

Suggested change

	}).catch((err) => {
	}, (err) => {

[avivkeller]

Left an initial review, but like @Ethan-Arrowood said, it'll take time for a more in depth look

[joyeecheung]

It's nice to see some momentum in this area, though from a first glance it seems the design has largely overlooked the feedback from real world use cases collected 4 years ago: https://github.com/nodejs/single-executable/blob/main/docs/virtual-file-system-requirements.md - I think it's worth checking that the API satisfies the constraints that users of this feature have provided, to not waste the work that have been done by prior contributors to gather them, or having to reinvent it later (possibly in a breaking manner) to satisfy these requirements from real world use cases.

[codecov]

Codecov Report

❌ Patch coverage is 87.53333% with 748 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.67%. Comparing base (b864049) to head (99c94eb).
⚠️ Report is 21 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/vfs/providers/sea.js	63.02%	159 Missing ⚠️
lib/internal/vfs/module_hooks.js	81.83%	123 Missing and 2 partials ⚠️
lib/internal/vfs/provider.js	80.82%	107 Missing ⚠️
lib/internal/vfs/providers/memory.js	86.66%	105 Missing and 1 partial ⚠️
lib/internal/vfs/file_handle.js	89.57%	55 Missing and 2 partials ⚠️
lib/internal/vfs/providers/real.js	86.70%	50 Missing ⚠️
lib/internal/vfs/watcher.js	92.97%	35 Missing and 3 partials ⚠️
lib/internal/vfs/file_system.js	96.53%	36 Missing ⚠️
lib/internal/vfs/streams.js	88.05%	19 Missing ⚠️
lib/internal/vfs/router.js	82.79%	16 Missing ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #61478      +/-   ##
==========================================
- Coverage   89.74%   89.67%   -0.08%     
==========================================
  Files         675      690      +15     
  Lines      204601   210640    +6039     
  Branches    39325    40148     +823     
==========================================
+ Hits       183616   188883    +5267     
- Misses      13273    14043     +770     
- Partials     7712     7714       +2     

Files with missing lines	Coverage Δ
lib/internal/bootstrap/realm.js	96.00% <100.00%> (+<0.01%)	⬆️
lib/internal/vfs/errors.js	100.00% <100.00%> (ø)
lib/vfs.js	100.00% <100.00%> (ø)
src/node_builtins.cc	76.00% <100.00%> (-0.34%)	⬇️
lib/internal/vfs/fd.js	97.61% <97.61%> (ø)
lib/internal/test_runner/mock/mock.js	98.58% <97.36%> (-0.15%)	⬇️
lib/internal/main/embedding.js	89.09% <92.20%> (+1.59%)	⬆️
lib/internal/vfs/sea.js	87.23% <87.23%> (ø)
lib/internal/vfs/stats.js	93.84% <93.84%> (ø)
lib/internal/vfs/router.js	82.79% <82.79%> (ø)
... and 9 more

... and 37 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jimmywarting]

And why not something like OPFS aka whatwg/fs?

const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.mount('/app', rootHandle) // to make it work with fs
fs.readFileSync('/app/config.json')

OR

const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })

fs.readFileSync('sandbox:/config.json')

fs.createVirtual seems like something like a competing specification

[mcollina]

And why not something like OPFS aka whatwg/fs?

I generally prefer not to interleave with WHATWG specs as much as possible for core functionality (e.g., SEA). In my experience, they tend to perform poorly on our codebase and remove a few degrees of flexibility. (I also don't find much fun in working on them, and I'm way less interested in contributing to that.)

On an implementation side, the core functionality of this feature will be identical (technically, it's missing writes that OPFS supports), as we would need to impact all our internal fs methods anyway.

If this lands, we can certainly iterate on a WHATWG-compatible API for this, but I would not add this to this PR.

[juliangruber]

Small prior art: https://github.com/juliangruber/subfs

[Qard]

I also worked on this a bit on the side recently: Qard@73b8fc6

That is very much in chaotic ideation stage with a bunch of LLM assistance to try some different ideas, but the broader concept I was aiming for was to have a VirtualFileSystem type which would actually implement the entire API surface of the fs module, accepting a Provider type to delegate the internals of the whole cluster of file system types to a singular class managing the entire cluster of fs-related types such that the fs module could actually just be fully converted to:

module.exports = new VirtualFileSystem(new LocalProvider())

I intended for it to be extensible for a bunch of different interesting scenarios, so there's also an S3 provider and a zip file provider there, mainly just to validate that the model can be applied to other varieties of storage systems effectively.

Keep in mind, like I said, the current state is very much just ideation in a branch I pushed up just now to share, but I think there are concepts for extensibility in there that we could consider to enable a whole ecosystem of flexible storage providers. 🙂

Personally, I would hope for something which could provide both read and write access through an abstraction with swappable backends of some variety, this way we could pass around these virtualized file systems like objects and let an ecosystem grow around accepting any generalized virtual file system for its storage backing. I think it'd be very nice for a lot of use cases like file uploads or archive management to be able to just treat them like any other readable and writable file system.

[jimmywarting]

Personally, I would hope for something which could provide both read and write access through an abstraction with swappable backends of some variety, this way we could pass around these virtualized file systems like objects and let an ecosystem grow around accepting any generalized virtual file system for its storage backing. I think it'd be very nice for a lot of use cases like file uploads or archive management to be able to just treat them like any other readable and writable file system.

just a bit off topic... but this reminds me of why i created this feature request:
Blob.from() for creating virtual Blobs with custom backing storage

Would not lie, it would be cool if NodeJS also provided some type of static Blob.from function to create virtual lazy blobs. could live on fs.blobFrom for now...

example that would only work in NodeJS (based on how it works internally)

const size = 26

const blobPart = BlobFrom({
  size,
  stream (start, end) {
    // can either be sync or async (that resolves to a ReadableStream)
    // return new Response('abcdefghijklmnopqrstuvwxyz'.slice(start, end)).body
    // return new Blob(['abcdefghijklmnopqrstuvwxyz'.slice(start, end)]).stream()
    
    return fetch('https://httpbin.dev/range/' + size, {
      headers: {
        range: `bytes=${start}-${end - 1}`
      }
    }).then(r => r.body)
  }
})

blobPart.text().then(text => {
  console.log('a-z', text)
})

blobPart.slice(-3).text().then(text => {
  console.log('x-z', text)
})

const a = blobPart.slice(0, 6)
a.text().then(text => {
  console.log('a-f', text)
})

const b = a.slice(2, 4)
b.text().then(text => {
  console.log('c-d', text)
})

x-z xyz
a-z abcdefghijklmnopqrstuvwxyz
a-f abcdef
c-d cd

An actual working PoC (I would not rely on this unless it became officially supported by nodejs core - this is a hack)

const blob = new Blob()
const symbols = Object.getOwnPropertySymbols(blob)
const blobSymbol = symbols.map(s => [s.description, s])
const symbolMap = Object.fromEntries(blobSymbol)
const {
  kHandle,
  kLength,
} = symbolMap

function BlobFrom ({ size, stream }) {
  const blob = new Blob()
  if (size === 0) return blob

  blob[kLength] = size
  blob[kHandle] = {
    span: [0, size],

    getReader () {
      const [start, end] = this.span
      if (start === end) {
        return { pull: cb => cb(0) }
      }

      let reader

      return {
        async pull (cb) {
          reader ??= (await stream(start, end)).getReader()
          const {done, value} = await reader.read()
          cb(done ^ 1, value)
        }
      }
    },

    slice (start, end) {
      const [baseStart] = this.span

      return {
        span: [baseStart + start, baseStart + end],
        getReader: this.getReader,
        slice: this.slice,
      }
    }
  }

  return blob
}

currently problematic to do: new Blob([a, b]), new File([blobPart], 'alphabet.txt', { type: 'text/plain' })

also need to handle properly clone, serialize & deserialize, if this where to be sent of to another worker - then i would transfer a MessageChannel where the worker thread asks main frame to hand back a transferable ReadableStream when it needs to read something.

but there are probably better ways to handle this internally in core with piping data directly to and from different destinations without having to touch the js runtime? - if only getReader could return the reader directly instead of needing to read from the ReadableStream using js?

[jasnell]

ergonomically it might make sense to allow this to be string | VirtualProvider where string is a constrained set of default values like 'memory'.

[jasnell]

This should explain whether multiple VirtualFilesystem instances are possible to create and how the potentially interact, if at all.

[jasnell]

From an ergonomics perspective, it might be worth exploring whether create should have an auto mount option where mounting happens on creation...

const myVfs = vfs.create('memory', {
  mount: '/virtual',
});

[jasnell]

I assume unmount() is idempotent.

[jasnell]

Also, this should probably show an example of mounting on a Windows system. Does it use drive letters?

[jasnell]

Mentioning worker threads here raises a question: should VirtualFileSystem be cloneable/transferable such that we can do..

parentPort.postMessage(vfs);

So that the VirtualFileSystem instance can be shared/transferred across thread boundaries? Doesn't need to be done now but it's worth considering for later.

[jasnell]

This should probably discuss what happens when you're overlaying on a file system that is case-sensitive vs. one that is case-insensitive.

There's also a question about encodings of file/directory names. The clarification would be: in overlay mode, does handling of the file names inherit the behavior of the overlaid file system or overwrite it?

[jasnell]

As a security backup, in case a malicious module decided to create a virtual file system secretly to try to intercept writes to critical paths, it might make sense to support a process.on('vfs-mount') and process.on('vfs-unmount') set of events. An application can be notified when a path is overlaid.

[jasnell]

Suggested change

	myVfs.writeFileSync('/data/config.json', '{}');
	myVfs.writeFileSync('/data/config.json', JSON.stringify({}));

[jasnell]

Suggested change

	myVfs.writeFileSync('/data/config.json', '{"source": "vfs"}');
	myVfs.writeFileSync('/data/config.json', JSON.stringify({source: 'vfs'}));

[jasnell]

It's worth adding a short comment here that #openFile will not throw and if it fails the stream will be destroyed.

[jasnell]

Unnecessary whitespace change.

[jasnell]

I'd extend this to give some warning about it meaning that some paths on disk might not be shadowed, or some might actually be unexpectedly shadowed with a simple example. I just know someone is going to hit this case and it's going to come up as a bug report .