Turning on composefs breaks pivot from old bootimages

[Prashanth684]

Updated description:

Any bootimage with a libostree older than v2023.4 will be too old to know how to create composefs images when upgrading. And so once we upgrade, if composefs = yes in prepare-root.conf, ostree-prepare-root in the new initramfs will fail because it sees that composefs should be enabled and yet no composefs image exists. We currently set composefs = maybe in prepare-root.conf because of this: https://github.com/coreos/rhel-coreos-config/blob/2c3f8527babb935d8d71b505bc764dbab8163b69/manifest-el9-shared.yaml#L28-L34

This ticket tracks being able to switch it back to yes.

Original description follows.

---

Attached the full secondboot journal.
journal.txt

The specific error is:

Dec 09 17:13:23 localhost systemd[1]: Mounted /sysroot.
Dec 09 17:13:23 localhost systemd[1]: Starting OSTree Prepare OS/...
Dec 09 17:13:23 localhost ostree-prepare-root[675]: Resolved OSTree target to: /sysroot/ostree/deploy/fedora-coreos/deploy/8bd789c8d973d9910e8cc1fc477dd20ac5e975e0663085eeb06a95bfcdba5199.0
Dec 09 17:13:23 localhost ostree-prepare-root[675]: sysroot.readonly configuration value: 1 (fs writable: 1)
Dec 09 17:13:23 localhost ostree-prepare-root[675]: ostree-prepare-root: composefs: failed to mount: No such file or directory
Dec 09 17:13:23 localhost systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
Dec 09 17:13:23 localhost systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
Dec 09 17:13:23 localhost systemd[1]: Failed to start OSTree Prepare OS/.

[jlebon]

This doesn't look like the first boot of the machine. Which AMI is this BTW? Is this from the CoreOS pipeline, or somewhere else? Let's try to reproduce it outside of OpenShift (presumably by booting the AMI, doing an rpm-ostree rebase or bootc switch and rebooting?).

[Prashanth684]

Yes this is not the first boot. This log is from the e2e test that is run on OKD release nightlies, where the Openshift installer boots the machine with fcos AMis and then on second boot, pivots to scos.

[Prashanth684]

When i tried booting with the AMI that the Openshift installer uses - 39.20231101.3.0, I see this issue. But when booting with a newer AMI - 40.20240602.3.0, I don't see the issue anymore and the rebase works. openshift/installer#8640 updates the AMI for fcos and also tests the okd scos e2e.

[jlebon]

So the issue here is that any bootimage with a libostree older than v2023.4 will be too old to know how to create composefs images when upgrading. And so once we upgrade, ostree-prepare-root in the new initramfs fails because it sees that composefs should be enabled and yet no composefs image exists.

This should be fixed by ostreedev/ostree#3353, which would allow ratcheting in composefs by letting ostree-prepare-root tolerate missing composefs images.

But clearly, longer-term we want to be able to rely on composefs just working the first time. This would definitely be fixed as we keep rolling out the bootimages update effort (openshift/enhancements#1496), but we may decide to do a shorter-term tactical fix for it if we're blocked on it.

[jlebon]

For now, let's back out composefs: #1700

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale