nil dereference in `createWindowsContainerDocument` when starting container with process isolation

[slonopotamus]

Steps to reproduce

1. Windows 11 version 10.0.26100.4652 (though this also reproes on Windows Server 2022 and likely others)
2. Run containerd (I'm using 2.1.3) as administrator
3. Run dockerd (I'm using 28.3.2) as administrator with --containerd=npipe:////./pipe/containerd-containerd arg
4. Put containerd-shim-runhcs-v1.exe on PATH (I both tried 0.13.0 and b8f90a0, current main)
5. docker run --rm -it --isolation process hello-world (bug doesn't reproduce with Hyper-V isolation)

Expected

hello-world container runs

Actual

docker: Error response from daemon: failed to create task for container: failed to create shim task: ttrpc: closed.

If we look at containerd logs, we can find that runhcs shim has crashed because of nil dereference in createWindowsContainerDocument

hcsshim/internal/hcsoci/hcsdoc_wcow.go

Line 364 in b8f90a0

for _, ml := range coi.mountedWCOWLayers.MountedLayerPaths {

time="2025-07-28T13:56:22.150202100+03:00" level=info msg="connecting to shim efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e" address="\\\\.\\pipe\\ProtectedPrefix\\Administrators\\containerd-shim-moby-efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e-pipe" namespace=moby protocol=ttrpc version=2
time="2025-07-28T13:56:22.159944000+03:00" level=info msg="created network namespace for container" cid=efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e netID=173ff9bb-e2f6-403f-b1ab-78b8300e4d92 spanID=43415fc36303d6cd traceID=326e43f10bed4374ec04964ba766bbcd
time="2025-07-28T13:56:22.176953800+03:00" level=info msg="added network endpoint to namespace" endpointID=38d7cce3-1a2e-409e-9abc-57171f652367 netID=173ff9bb-e2f6-403f-b1ab-78b8300e4d92 spanID=43415fc36303d6cd traceID=326e43f10bed4374ec04964ba766bbcd
time="2025-07-28T13:56:22.155974300+03:00" level=info msg=Span bundle="C:\\ProgramData\\containerd\\state\\io.containerd.runtime.v2.task\\moby\\efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e" checkpoint= duration=20.9795ms endTime="2025-07-28T13:56:22.176953800+03:00" name=Create parentSpanID=32b1da37d01ee87a parentcheckpoint= spanID=43415fc36303d6cd startTime="2025-07-28T13:56:22.155974300+03:00" stderr= stdin="\\\\.\\pipe\\containerd-efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e-efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e-stdin" stdout="\\\\.\\pipe\\containerd-efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e-efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e-stdout" terminal=true tid=efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e traceID=326e43f10bed4374ec04964ba766bbcd
time="2025-07-28T13:56:22.155448900+03:00" level=info msg=Span duration=21.5049ms endTime="2025-07-28T13:56:22.176953800+03:00" name=containerd.task.v2.Task.Create parentSpanID=0000000000000000 spanID=32b1da37d01ee87a spanKind=server startTime="2025-07-28T13:56:22.155448900+03:00" traceID=326e43f10bed4374ec04964ba766bbcd
time="2025-07-28T13:56:22.201531600+03:00" level=info msg="shim disconnected" id=efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e namespace=moby
time="2025-07-28T13:56:22.201531600+03:00" level=info msg="cleaning up after shim disconnected" id=efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e namespace=moby
time="2025-07-28T13:56:22.201531600+03:00" level=error msg="failed to delete task" error="ttrpc: closed" id=efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e
time="2025-07-28T13:56:22.201531600+03:00" level=info msg="cleaning up dead shim" id=efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e namespace=moby
time="2025-07-28T13:56:22.244148900+03:00" level=warning msg="warnings while cleaning up dead shim" id=efd303ec02ac3f2092951c21bc609baa72bc1cb62a0447188f99d70aa93f453e namespace=moby warnings="time=\"2025-07-28T13:56:22+03:00\" level=warning msg=\"found shim panic logs during delete\" log=\"panic: runtime error: invalid memory address or nil pointer dereference\\n[signal 0xc0000005 code=0x0 addr=0x10 pc=0xf8aca8]\\n\\ngoroutine 30 [running]:\\ngithub.com/Microsoft/hcsshim/internal/hcsoci.createWindowsContainerDocument({0x12f50d8, 0xc0002fce70}, 0xc000316080)\\n\\thcsshim/internal/hcsoci/hcsdoc_wcow.go:364 +0x11a8\\ngithub.com/Microsoft/hcsshim/internal/hcsoci.CreateContainer({0x12f50d8, 0xc0002fce70}, 0x11f5768?)\\n\\thcsshim/internal/hcsoci/create.go:257 +0xead\\nmain.createContainer({0x12f50d8, 0xc0002fce70}, {0xc0000b38c0, 0x40}, {0xc0000b602a, 0x1d}, {0x0, 0x0}, 0xc0001f37a0, 0x0, ...)\\n\\thcsshim/cmd/containerd-shim-runhcs-v1/task_hcs.go:171 +0x3f4\\nmain.newHcsTask({0x12f50d8, 0xc0002fce70}, {0x12ee1a0, 0xc00008ea80}, 0x0, 0x1, 0xc0000e6240, 0xc0001f37a0)\\n\\thcsshim/cmd/containerd-shim-runhcs-v1/task_hcs.go:222 +0x410\\nmain.newHcsStandaloneTask({0x12f50d8, 0xc0002fce70}, {0x12ee1a0, 0xc00008ea80}, 0xc0000e6240, 0xc0001f37a0)\\n\\thcsshim/cmd/containerd-shim-runhcs-v1/task_hcs.go:108 +0x4a5\\nmain.(*service).createInternal(0xc000080840, {0x12f50d8, 0xc0002fce70}, 0xc0000e6240)\\n\\thcsshim/cmd/containerd-shim-runhcs-v1/service_internal.go:188 +0xd3e\\nmain.(*service).Create(0xc000080840, {0x12f50d8?, 0xc0002fc3c0?}, 0xc0000e6240)\\n\\thcsshim/cmd/containerd-shim-runhcs-v1/service.go:152 +0x4d7\\ngithub.com/containerd/containerd/api/runtime/task/v2.RegisterTaskService.func2({0x12f50d8, 0xc0002fc3c0}, 0xc000097f00)\\n\\thcsshim/vendor/github.com/containerd/containerd/api/runtime/task/v2/shim_ttrpc.pb.go:46 +0x8c\\ngithub.com/Microsoft/hcsshim/pkg/octtrpc.ServerInterceptor.func1({0x12f50d8, 0xc0002fc1b0}, 0xc000097f00, 0xc0002f5ee8?, 0xc00008eae0)\\n\\thcsshim/pkg/octtrpc/interceptor.go:115 +0x29c\\ngithub.com/containerd/ttrpc.(*serviceSet).unaryCall(0xc00008eab0, {0x12f50d8, 0xc0002fc1b0}, 0xc00008eae0, 0xc000267110, {0xc00019cd80, 0x207, 0x240})\\n\\thcsshim/vendor/github.com/containerd/ttrpc/services.go:75 +0xdc\\ngithub.com/containerd/ttrpc.(*serviceSet).handle.func1()\\n\\thcsshim/vendor/github.com/containerd/ttrpc/services.go:118 +0x158\\ncreated by github.com/containerd/ttrpc.(*serviceSet).handle in goroutine 29\\n\\thcsshim/vendor/github.com/containerd/ttrpc/services.go:111 +0x147\\n\"\ntime=\"2025-07-28T13:56:22+03:00\" level=error msg=Span duration=2.8333ms endTime=\"2025-07-28T13:56:22.240311500+03:00\" error=\"A virtual machine or container with the specified identifier does not exist.\" errorCode=Unknown name=HcsOpenComputeSystem parentSpanID=58222812854e6a96 spanID=8f2d0fd0829d32d4 startTime=\"2025-07-28T13:56:22.237478200+03:00\" traceID=ca8eb732ce657b4b4bd7859f9801da6a\ntime=\"2025-07-28T13:56:22+03:00\" level=info msg=Span duration=18.9654ms endTime=\"2025-07-28T13:56:22.241580700+03:00\" name=delete parentSpanID=0000000000000000 spanID=58222812854e6a96 startTime=\"2025-07-28T13:56:22.222615300+03:00\" traceID=ca8eb732ce657b4b4bd7859f9801da6a\n"

[jterry75]

Best guess:

hcsshim/internal/hcsoci/resources_wcow.go

Line 44 in 379a2ae

coi.mountedWCOWLayers = mountedLayers

this is the only place that the mountedWCOWLayers are assigned before they are iterated in that loop. So it must be that something has changed when calling allocateWindowsResources from the hcsoci path from Docker. By chance did docker start setting the coi.Spec.Root.Path property? That would cause that if branch not to run and therefore never allocate the mountedWCOWLayers

[thaJeztah]

I see it's set when not using HypeV, and there was a change last Year in moby/moby@e8f4bfb, to set it to the correct path when using containerd snapshotters.

I do notice there's a check in the "local" client (i.e., don't use containerd as runtime) on Windows that checks for the field to be empty; not sure if there's a similar check when using containerd as runtime; https://github.com/moby/moby/blob/9a97f59e6e2d51818f6b8cf8589199fd6714ebc7/daemon/internal/libcontainerd/local/local_windows.go#L253-L255

[slonopotamus]

I see it's set when not using HypeV

As I said in initial bugreport, the issue only happens without HyperV.

[jterry75]

@helsaawy - Could you PTAL here plz?