Bootc and OSTree: Modernizing Linux System Deployment

The ability to roll back is only one part of transactional update. I used to run Mint with BTRFS and Snapper, and the absence of atomic, transactional system updates is the reason I switched to Fedora Kinoite.

With transactional system update, you atomically switch from the old system configuration to the new system configuration, without the running system passing through an intermediate state, where some of the files come from the old configuration, and some come from the new.

Kinoite installs system updates in the background, without disturbing the normal operation of the system. If an update fails, instead of entering a state where your system is trashed, and the user needs to perform an emergency rollback operation, nothing happens. The partially downloaded update data is deleted without any drama. If the update succeeds, then the changes are not applied until the user performs an action to accept the changes, either by rebooting, or by issuing a CLI command (as described by the OP).

The only reason Red Hat needed to invent this very complex mechanism was because RH does not officially have a COW-snapshot capable filesystem in its enterprise distro. [...] But RH removed Btrfs from RHEL and Btrfs was the only GPL COW filesystem, so core infrastructure in the distro means no COW on RH. [...] With no COW FS, RH had to invent a way to do transactional updates without filesystem support. Result, OStree. Git, but for binaries.

The timeline here is a bit off: OSTree was first created in 2011, long before SLES supported btrfs at all in 2014 or its on-disk format was even declared stable in 2013. So RH wasn't really invented that because of removing btrfs or similar.

Yes I know, Fedora put it back, but the key thing is, it only uses Btrfs only for compression so that Flatpak looks less horrendously inefficient.

Fedora never "put it back". It was never removed and later promoted to default for a wide variety of reasons, which included compression but also other advantages over the previous default of LVM. Compression was not actually enabled at this time, not until the next release, where it was enabled for the entire volume.

You can run an immutable OS, reboot for updates, and if you need, disable it, go in and fix the system, and then turn it back on again.

You can actually do this with OSTree with one command.

(And yes, everyone developing FOSS uses Git, but almost nobody understands Git. Embedding something you don't understand in your OS design is a VERY BAD PLAN.)

OSTree is "Git for operating systems" in that Git is the most well-known version control system that uses some form of blob addressing. The actual design and UX is extremely different, so most of Git's...unique complexities don't really carry over. (Really, a lot of that complexity comes from Git's underlying model being pretty simple but also far too low-level to be an adequate VCS, and the high-level bits on top don't mesh with it all that well. OSTree is comparatively just the lower-level side, because it's not supposed to be used like a high-level VCS anyway.)

It is so mind-shreddingly complicated that you can't do package management any more, you can't touch the underlying FS. So you need a whole new set of layers on top: virtual directories on top of the main virtual directory, and some bits with extra pseudo-filesystems layered on top of that to make some bits read-write.

I think you're conflating a bunch of different bits here? Essentially:

• One of the main goals here was more generally for systems to, in some way, be reproducible. Mutterings around this go back quite a ways:

  Only local configuration or state is stored per-system, while the vendor operating system is pulled in from the same, immutable, shared snapshot.

  If you want to achieve this on top of btrfs, you'd also need a similar set of additions, like storing the base image in a separate snapshot from the local changes and then overlaying them on boot. (I am moderately sure Poettering also had an article on specifically utilizing btrfs for this, but I can't find it now unfortunately.)

  All this is to say that the goals are a bit grander than just what transactional-updates provides.

• "Installing packages" is just adding them on top of the base OS commit. This isn't that complex and mostly boils down to a few commands; the actual complexity tends to be around stuff like "when do you run pre-/post-install scripts" (rpm-ostree spends much of its time on this), which would apply to other solutions that start from a clean base state like A/B partitioning as well. Lack of this is why SteamOS wipes out packages you installed on update.

• I'm not really sure what the "virtual directories" or "extra pseudo-filesystems" is supposed to be referring to?

The only reason Red Hat needed to invent this very complex mechanism was because RH does not officially have a COW-snapshot capable filesystem in its enterprise distro.

I'm not sure of your thinking here, COW fs is not the be-all, end-all of managing a distro and I'm not sure it works adequately. This stuff comes out of CoreOS and the container space where OpenShift uses OCI container overlays and rpm-ostree to handle updates. It works but can be annoying as you manage pre-install configs with ignition files and post-install changes through MachineConfigs or jam either into custom images, but it does work (Jack Sparrow meme goes here).

For OpenSUSE are you refering to snapper or transactional updates? AFAICT transactional-updates does not support applying changes live (rpm-ostree does). I've been using Fedora Silverblue with distrobox for a while, but rpm-ostree is very slow.

Maybe I should try OpenSUSE again (IIRC when I tried it long ago there were unrelated reasons I couldn't use it, e.g. it didn't have subpixel anti-aliasing, but meanwhile I got 4K monitors)

LVM can do snapshots, and could for decades.

Also, as far as I remember, reiser4 is also COW.