containerd volume ownership results in access denied by HostProcessContainer #621
Description
I'm trying to develop a csi driver which loads an oci image with a writable layer for use in CI builds. To accomplish this I use ctr binary to do a pull operation and then an image mount operation.
During initial prototyping this worked fine where I ran (as Administrator) containerd and the ctr commands in foreground terminals. However, when trying to run the csi driver as a HostProcessContainer the ctr image mount ... command fails with an access denied message on a volume.
I really have no idea what windows is doing behind the scenes for the snapshotter etc but I'm a bit perplexed at what would cause the hpc to not have permissions to the volume seemingly being created behind the scenes over the containerd pipe.
This is an on-prem setup using rke2 so containerd is being launched behind the scenes by rke2 service which runs as service runs under LocalSystem.
# note, as kubelet continues to retry the volumeID in the log changes every time
2025-11-04T15:01:50.767444500-05:00 csi-driver {"host":"na01wkubw01d11","level":"error","message":"handler error - driver: EphemeralInlineContainerDOciDriver method: NodePublishVolume error: {\"code\":1,\"stdout\":\"sha256:4616cf4228a0bf956c4348505b73c51f87aee27e707cf534ee08a090c12def7c\\n\",\"stderr\":\"ctr: failed to set volume mount path for layer C:\\\\var\\\\lib\\\\rancher\\\\rke2\\\\agent\\\\containerd\\\\io.containerd.snapshotter.v1.windows\\\\snapshots\\\\1396: failed to bind target \\\"\\\\\\\\\\\\\\\\?\\\\\\\\Volume{334f42c6-91e6-4b41-8420-8877bd1abed4}\\\\\\\\\\\" to root \\\"c:\\\\\\\\var\\\\\\\\lib\\\\\\\\kubelet\\\\\\\\pods\\\\\\\\f028b309d27e-421e-b951-d7efb6bc6562\\\\\\\\volumes\\\\\\\\kubernetes.io~csi\\\\\\\\oci\\\\\\\\mount\\\": Access is denied.\\n\",\"timeout\":false}","service":"democratic-csi","timestamp":"2025-11-04T20:01:50.781Z"}
securityContext:
windowsOptions:
hostProcess: true
runAsUserName: NT AUTHORITY\SYSTEM
- https://github.com/democratic-csi/democratic-csi/blob/next/src/driver/ephemeral-inline-containerd-oci/index.js#L331-L382
- https://kubernetes.slack.com/archives/C0SJ4AFB7/p1762285428827269
Note, none of this would really be required if the new oci image volume type in k8s supported mounting as rw and it was somehow possible to tell/hint the snapshotter how large to make the scratch layer in windows (I need more space than the seeming default of 20GB to do the builds).
OR
If it was possible to hint via a pod label or whatever how large the scratch layer should be for the running Pod.
OR
NTFS/ReFS had some sort of CoW features like zfs et al where I could just create a rw snapshot of a whole folder on the host.
OR
Any other ideas? Everywhere I turn I keep running into walls :(
I want to use k8s containers as CI build agents (currently using with both bamboo and gh runners). I have some fairly large git repos that I don't want to:
- pull the whole repo for every build (network traffic and build delay)
- use N * X storage on the host where N = number of concurrent builds and X = repo size
So I need to cache the repo once (oci seems great for this, as over time I can update the image with more recent git data to lower the git delta required to pull each build) and have a way to create a rw scratch area for every build where I don't copy the full content for every build.