Try to support EROFS merged block device feature by adding VMDK support to imago / libkrun

[hsiangkao]

EROFS snapshotter generates one blob for each OCI layer and supports the flattened filesystem feature (also called the fsmerge feature), which uses a tiny metadata-only image to reference all layer blobs so that only one EROFS mount is needed instead of mounting each layer one by one.

The merged mount looks like below:

mount -t erofs /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/7/fsmeta.erofs \
  -odevice=/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs,\
    device=/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/2/layer.erofs,\
    ...
    device=/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/7/layer.erofs
  /run/containerd/io.containerd.mount-manager.v1.bolt/t/335/1

For RunC, it's almost the end of the story since EROFS supports file-backed mounts, but for VM-based containers likenerdbox, multiple block devices still need to be passed in, which is rather ineffective if the total number of layers is large.

However, EROFS also supports a feature called flatten device, which means you could use one-single block device that consists of a collection of a tiny metadata-only blob + layer 0 + layer 1 + ... + layer n-1.

A standard way is preferred to support that is and VMDK format seems to be the answer (since VMDK is the only popular one which supports this, unlike qcow2), and QEMU already supports VMDK in this way.

For example the following VMDK spec can be used to launch wordpress with fsmerge-enabled EROFS snapshotter:

# Disk DescriptorFile
version=1
CID=fffffffe
parentCID=ffffffff
createType="twoGbMaxExtentFlat"

# Extent description
RW 4312 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/112/fsmeta.erofs" 0
RW 176520 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/92/layer.erofs" 0
RW 16 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/93/layer.erofs" 0
RW 549528 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/94/layer.erofs" 0
RW 8 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/95/layer.erofs" 0
RW 97736 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/96/layer.erofs" 0
RW 8 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/97/layer.erofs" 0
RW 16 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/98/layer.erofs" 0
RW 22960 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/99/layer.erofs" 0
RW 16 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/100/layer.erofs" 0
RW 87736 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/101/layer.erofs" 0
RW 40 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/102/layer.erofs" 0
RW 16 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/103/layer.erofs" 0
RW 16 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/104/layer.erofs" 0
RW 106720 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/105/layer.erofs" 0
RW 88224 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/106/layer.erofs" 0
RW 24 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/107/layer.erofs" 0
RW 16 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/108/layer.erofs" 0
RW 336 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/109/layer.erofs" 0
RW 135208 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/110/layer.erofs" 0
RW 24 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/111/layer.erofs" 0
RW 16 FLAT "/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/112/layer.erofs" 0


# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.geometry.cylinders = "1260"
ddb.geometry.heads = "16"
ddb.geometry.sectors = "63"
ddb.adapterType = "ide"

[hsiangkao]

hi @dmcgowan, both imago and libkrun changes are landed upstream:
containers/libkrun#474

Could we address containerd erofs fsmerge feature itself if possible? That is a simple change..
Or is there some real concern on your side about that containerd change?