Rolling my own immutable distro

vk6flab, 23 days ago

My semi-immutable OS is based around a Debian installation where every application is installed in a separate Docker container.

When you launch the application, it volume mounts an appropriate directory that contains only the data related to that application.

Chrome for example launches with a single subdirectory inside ~/Downloads, so each instance can only see its own directory.

I can also test compilation of random repositories inside a container, without affecting the underlying OS.

The OS itself has only got a minimal Debian and Docker installed.

Been using it for several years. I can’t recall when I last rebooted it.

AbidanYre, 23 days ago

So, basically Rancher?

HubertManne, 23 days ago

now I have to look up rancher. sounded like qubeos to me.

AbidanYre, 23 days ago

From Suse. It looks like V1 and V2 are maintenance only these days.

github.com/rancher/os

Looking at the rancher GitHub it looks like they can’t decide which direction they want to go.

boredsquirrel, 23 days ago

That sounds… strange? I think Flatpak is way more resource efficient, as separate docker containers will not share a single library.

But yes, I manage some Debian workstations and the first thing I did after manually updating them to Debian 12 was

• debloat (also all the GNOME stuff)
• install all apps as Flatpaks
• setup automatic updates

Bitrot, 23 days ago

VanillaOS uses something similar with ABroot + OCI images, except for the flip flopping at every reboot.

BrianTheeBiscuiteer, 23 days ago

That was the main inspiration for me. If I do understand their setup, “testing” a change of some kind would require an explicit rollback. If a reboot in that system meant I lost those changes then that’s actually what I’m looking for.

skilltheamps, 23 days ago

I encourage you to go to town with whatever crazy setup you come up.

I just want to note that the reboot-to-update mechanism also has its positive sides, as ancient as it may seem (we do not succumb to windows level backwardness, because that fails to reap the benefits despite requiring so many reboots). Namely, you get atomic updates, hence the name “fedora atomic” for example. That means you have no transient periods where your OS is running in an inconsistent state. Like when you update a traditional distro, the new files/libraries/binaries/kernel-modules do not match anymore what is in RAM, including the currently running kernel. That leads to stuff like the nvidia driver / cuda not working until reboot, running applications failing to load a library they need now etc… The vast majority of times this is no huge problem, but in theory the only way of maintaining a system with it never running in basically undefined state is with atomic udpates.

nossaquesapao, 23 days ago

I get it, but I don’t understand why fedora has to reboot 2 times when I update using the gui. What would be the problem with updating before shutting down, and rebooting only once? Fedora reboots, updates, then reboots again.

yala, 23 days ago

I’ve been on Fedora Silverblue for over two years now and I’ve never experienced such a thing. Would you mind elaborating? Like, does the system force itself off after the first reboot without any input? Or, instead, do you notice that the installed package is not installed and therefore succumb to another reboot by your own admission? Which, to be honest, seems like some broken interaction*.

nossaquesapao, 23 days ago

I probably didn’t explain it very well, sorry.

When I update using the gnome software manager, it only downloads the packages and wait for a reboot to install them. When I click the option to reboot ans install, it boots to a software update screen, and when it finishes, it reboots again to the system.

yala, 23 days ago

Thank you for the elaboration! Quick question;

When I click the option to reboot ans install, it boots to a software update screen

You use the word “boots” here. Then,

it reboots again to the system.

Now, you use reboots accompanies with “again”. Therefore, I will assume that the previous “boots” meant to convey a reboot. As for my actual question, does your system do a full reboot the first time? So, is it technically possible to access GRUB in-between and does it ask you to fill in the LUKS-password (if you’ve enabled FDE) etc?

Finally, is this on Fedora Silverblue? Or just plain ol’ classic Fedora?

nossaquesapao, 23 days ago

It’s on main fedora, and yes, I go through grub need to input the encryption password twice. I can record it, if it makes things more clear. I’m having a weird difficulty explaining this.

yala, 23 days ago

It’s on main fedora

That’s were the disparity of our experiences stem from 😉. I’m aware that regular Fedora behaves peculiar for updates. IIRC, the reasoning involved something about ensuring that updates happened offline for the sake of system integrity. Thankfully, Fedora Atomic doesn’t have that issue 😊. Thank you clarifying!

nossaquesapao, 23 days ago

Sorry for the mess, and thank you for taking your time to answer me

yala, 23 days ago

No need to apologize fam. Thank you for being patient with my questions 😜.

equivocal, 23 days ago

IIRC main Fedora used to not do this until some update crashed people’s sessions including the update process which left their install in an unbootable state.

The ostree based versions like Silverblue avoid this by their updates not touching the running system and instead creating a new folder structure with the updates applied that will be booted into on next boot.

winterayars, 23 days ago

Basically what it’s doing is booting to an alternate OS configuration to do the install. It’s way easier to just reboot again rather than tear down the installer environment and go into a normal one. That’s basically a reboot in all but name. It’s annoying to have to enter your encryption passphrase twice, though.

I feel like a lot of Linux behaviors tell me most Linux people don’t encrypt their data, which tbh should not only be the default but should be difficult to opt out of. Apple actually does this one right. Encryption is just the way it works.

nossaquesapao, 23 days ago

That’s exactly what happens. I get it now

boredsquirrel, 23 days ago

OpenSUSE microOS/ microOS Desktop (Aeon, Kalpa) does this.

They use a complete “changes go to the next system” thing also using BTRFS.

But they dont use OSTree so the system is fundamentally flawed.

Advantages of ostree are

• complete transparency over package changes rpm-ostree db diff
• complete transparency over /etc changes (the upstream is in /usr/etc and can be reset, see here
• the OS is always based on a complete upstream remote, your local system does not matter at all. You can rebase, reset etc without being dependent on anything on the local OS.

Example: I could rebase from Fedora OSTree to CentOS OSTree. They are working on bootc images, which are bootable OCI images and in theory only one step away from uBlue-like distribution.

If you do anything relying on local package management like OpenSUSE does, you can snapshot between changes but still mess up.

So I would always base off OSTree.

What I dont get though is the reliance on reboots and images. OSTree works on all filesystems and doesnt need images, it is simply like a Git repo.

So what I would change is, to enable random local changes with a flag –direct and simply apply the changes live. I mean, that is what DNF and all the distros do too.

Only if you need a kernel upgrade you do stuff with a reboot. Version upgrades are also WAY better than the unstable mess on standard Fedora or other distros.

So track everything with OSTree, allow resets, rebases etc, but dont force all the image stuff. This is the reason why rpm-ostree takes so long and is so inefficient compared no DNF.

Just using OSTree you could only install RPMs, use a nonwheel user, SELinux confined users and have a secure and slim system.

I dont know if I miss something here. Android is rootless but the base OS is still immutable and uses A/B root, so writing only happens to the inactive partition. I dont know if immutability is some core security feature.

Rpm-ostree is really good as an allrounder, but I think a bit overkill. It does support installing packages live, but this does the same action afaik and just swaps the OS image without a reboot.

MrSpandex, 23 days ago

Have you tried rpm-ostree apply-live? coreos.github.io/rpm-ostree/apply-live/

boredsquirrel, 23 days ago

Yes but it only works for installs and failed for the one install I tried.

Nothing like just using dnf on the current system and tracking everything with OSTree

boredsquirrel, 23 days ago (edited 23 days ago)

But to OPs actual ideas:

I can use BTRFS to hold data for the rootfs in three different subvolumes (at minimum): root-A, root-B, root-Z.

That is basically rpm-ostree or BTRFS snapshots, I dont see the point yet

root-Z is my golden image and it represents what I want root to look like after reboot.

So like the upstream ostree remote or OCI image? I think you have a big thought flaw here

root-A and root-B are the active and passive instances of rootfs, but which one is active will flip-flop after every reboot.

On every reboot they flip flop? Why??

So if I boot with A, B gets replaced with the contents of Z. This means all changes you do are removed after a reboot. rpm-ostree and ostree admin both have this feature for testing but the use case is small.

If you have an imahe Z, this is like the uBlue main image, or the Fedora OSTree remote. It is the updated vanilla thing.

Not like on OpenSUSE microOS where you at most have some vanilla BTRFS snapshot from directly after the install, but the vanilla, tested, stable base set of packages.

If you replace the stuff with that always, it is like an rpm-ostree reset but always, and with a local image.

I see the benefit of having a local reset image, as internet is not always available.

But a reset really is only needed when an update breaks things, as the base is immutanle. So no.

In the meantime I can do whatever I want with A.

So you have one testing persistent image? Or is this only temporary?

Not sure how I’ll update Z (chroot or “promote” the active subvol to be Z) but without an update every reboot is an automatic rollback.

This has little sense and honestly rpm-ostree has ephemeral changes only on the live system that will vanish when rebooting.

I dont know the use case really. We are currently working on a change proposal to fix the permissions so changing the OS is pretty privileged.

The software stores handle the system updates but dont show RPMs for installation anymore. Most people will never touch the system.

Or if they do, the system is reset to the base on every update and the changeset is permanently reapplied, every time anew. You are always rebasing off upstream, your installed OS is literally not important.

Its just the diffs that are calculated and changed.

bjoern_tantau, 23 days ago

BTRFS snapshots are relatively cheap. I have about 10 or 20 or so on my systems. Snapper manages that for me.

Also I’d say that you only want to roll back for catastrophic errors like the system not booting. If you discover after 8 upgrades that a package that was updated on the first upgrade has some critical bug you wouldn’t roll back everything else to fix the issue. You would just downgrade that single package to a known good version.

yala, 23 days ago

Most immutable systems I’ve seen require a reboot in order to apply system changes.

It depends on the type of change you’re applying and which specific immutable/atomic/composable distro we’re talking about. In the case of GuixSD and NixOS, this is simply (mostly) false. For the others, if you’re going to change the base-system/image, then a reboot is required. Though, it’s important to mention that often enough systemctl soft-reboot suffices. And even then, some of them (like Fedora Atomic) offer functionality (e.g. through –apply-live) to apply the changes directly to the working system. And we’re not even talking about the latest innovations (like https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html that might be able to provide a revolutionary solution to this problem altogether.

Also, even immutable OS I’ve seen wants you to opt-in to a rollback. Rarely do I see the full effects of installing a package or altering a config immediately. By the time I notice an issue maybe it’s too late to rollback to before the change or maybe I’ve done a few other things since and I don’t want to rollback everything. I would much prefer to make “rolling forward” or persisting changes to be a very conscious process.

You seem to have false notions on how this works on atomic distros, or at least it seems your understanding only applies to a small subset of them. E.g. responsible use of GuixSD and/or NixOS completely circumvents this from the get-go. Even Fedora Atomic offers (relatively) sane management of rollbacks.

I started messing with BTRFS and I think I’ve come up with a process that will get me what I want, no matter the distro. Please poke holes in my idea. So I think I can use BTRFS to hold data for the rootfs in three different subvolumes (at minimum): root-A, root-B, root-Z. root-Z is my golden image and it represents what I want root to look like after reboot. root-A and root-B are the active and passive instances of rootfs, but which one is active will flip-flop after every reboot. So if I boot with A, B gets replaced with the contents of Z. In the meantime I can do whatever I want with A. Not sure how I’ll update Z (chroot or “promote” the active subvol to be Z) but without an update every reboot is an automatic rollback.

I don’t understand how this solves your (conceived) issue with rollbacks. Furthermore, I also wonder what kinda changes you’re applying to your system on a daily basis. Like, do you really feel the need to poke at /usr every day? Aren’t your needs regarding installing software already (mostly) solved through AppImage/Nix/Flatpak/Snap/Brew etc?

BrianTheeBiscuiteer, 23 days ago

I’m not planning to alter the system daily so, admittedly, this is a bespoke, non-trivial process to handle an uncommon use case. In general I haven’t run into the kind of issues that immutable distros proport to fix. I would say this is moreso an OCD friendly approach to OS management. I’m also hoping this setup will basically force me into using Ansible more and manual tweaks less.

I feel Guix and NixOS are a bit more in a league of their own due to their declarative nature. I’m on the fence if I want to go that far. Again, I’ll admit my knowledge of these systems is based on docs and I’ll probably have much different thoughts getting hands on.

And my goal is to rely on Flatpak and containers but if that was the answer then all the immutable distros out there are about as overbaked as my idea.

yala, 23 days ago

Thank you for the reply!

I’m not planning to alter the system daily so, admittedly, this is a bespoke, non-trivial process to handle an uncommon use case.

Honestly, I think you’re underselling the potential importance this has; I wouldn’t be surprised if a lot of users that have tried Fedora Silverblue, openSUSE Aeon or Vanilla OS were simply put off by how long it took them to get going post-installation compared to any other distros they’re used to. So, I do believe it’s a noble endeavor. However, I’m actually more in favor of educating people on the paradigm shift that comes with an atomic distro.

In general I haven’t run into the kind of issues that immutable distros proport to fix.

That’s excellent! If I may ask, which distro(s) do you usually run on your systems?

I would say this is moreso an OCD friendly approach to OS management.

Hehe, great way to put it.

I’m also hoping this setup will basically force me into using Ansible more and manual tweaks less.

Interesting to see you mention Ansible. I’ll get to this in a moment.

I feel Guix and NixOS are a bit more in a league of their own due to their declarative nature.

Exactly.

I’m on the fence if I want to go that far.

I believe you’re ripe for it; if you want to force yourself into using Ansible, then you might as well commit to GuixSD or NixOS instead.

Again, I’ll admit my knowledge of these systems is based on docs and I’ll probably have much different thoughts getting hands on.

That’s fine. We’re all continuously educating ourselves.

And my goal is to rely on Flatpak and containers

Interesting! Like, 100% rely on those two only? So not even installing software directly to base system*? This is actually kinda similar to openSUSE’s current implementation for its atomic distros**.

but if that was the answer then all the immutable distros out there are about as overbaked as my idea.

I’m not entirely sure if I understood you correctly. But I agree that Flatpak and containers can basically be achieved on any distro out there. No need to go immutable/atomic/composable if you just want Flatpak+containers.