The traditional desktop delivery model is based on a large number of distributed PCs executing the operating system and the desktop applications. Managing traditional desktop environments is incredibly challenging and expensive. Tasks like installations, configuration changes, and security measures require time-consuming procedures and dedicated deskside support. Users in the Free and Open Source Software (FOSS) community have always been keen to implement strategies for the delivery of the configuration of their systems so that they (1) can share it with the community, and (2) distribute it across multiple workstations or servers. Dotfiles and Ansible repositories are usual buzzwords for those users to respectively version specific user configurations, deliver initial system-level setup to freshly installed OSes and update existing ones. Alternatively, and especially in desktop enterprise environments leveraging multi-user desktop configuration, the delivery of the user-specific configuration can be managed through technologies like Kerberos, LDAP and Active Directory. However, the latter still needs a specific centralized model that is either not always doable and optimal or even over-killing for small configurations like single-user workstations.

Similar issues apply in the Cloud-native, and IoT field, where a company would deliver the Operating System driving their application (e.g., Kubernetes, automotive software on top of RHEL for edge, …) to the customers, and those customers should be able to modify this configuration as well, leading to the need of a complex solution that can guarantee robustnees, reliability, and transparency to the users across upgrades.

However, in the IoT and Cloud-native scenarios, what is usually needed is delivering upgrades and general configurations across devices that may expose a predictable surface of changes.

Developers can assume that the users of a configuration or software delivery tool can cluster the devices to manage, treating them primarily as stateless entities. On top of those stateless entities, the end-users will implement their specific use cases and setup by exploiting other persistent-layer models and systems like etcd¹ in Kubernetes and the Origin Kubernetes Distribution (OKD)².

OKD is based on Fedora CoreOS³ and implements a mechanism for the delivery of the configuration and software versions based on:

1. the libOSTree⁴ eco-system;
2. the Butane/Ignition⁵ technology for the provisioning of the cluster-specific initial configuration of machines;
3. the Machine Config Operator (MCO) to deliver further user configuration layers and to manage the desired state of the ostree, i.e., responsible for triggering upgrades by means of rebasing to a new base os container image.

The resources managed by the MCO (MachineConfig) represent the main surface of change an end-user should be aware of. The MCO operates by watching at high-level configuration resources pushed by the user as Kubernetes objects and supporting a subset of the configurations one can apply to nodes of a Kubernetes cluster.

In the desktop scenario, we often see different Ansible and Dotfiles repositories that are useful in particular for the initial provisioning of the machines’ configuration, but they are complex for their future maintenance: upgrades can be distributed differently, the users can modify one laptop’s /etc or /home/$user content, leading any Ansible playbook unable to guarantee idempotency and the robustness discussed above and actually breaking this distribution of high-level entities to each workstation they need to manage.

In general, the challenges one can face in managing multiple full-desktop GNU/Linux systems are:

1. Distributing the same packages to a set of systems;
2. Controlling and synchronizing system upgrades across multiple devices;
3. Distributing common configurations;
4. Allowing specific local changes for some machines;
5. Distributing sensitive content;
6. Portability of the home content, distribution, and synchronization of a subset of data across devices;

Combining the Cloud-native approach of OKD and Fedora CoreOS, and using rpm-ostree⁶, the Fedora community delivers desktop distributions like Fedora Silverblue⁷ and Kinoite⁸. This article leverages those technologies to expose how points 1 to 4 can be solved via rpm-ostree and the recent OSTree native container feature.

rpm-ostree⁶ is the hybrid image-package system driving the delivery of Fedora CoreOS (FCOS) and CentOS Stream CoreOS for OKD and RHEL CoreOS for the Red Hat Openshift Container Platform (OCP).

Under the hood, it uses libOSTree as a base image format and layers the possibility of using RPMs on both the client and server side, sharing code with the dnf project, specifically libdnf.

libOSTree is a system for versioning updates of Linux-based operating systems. It is the “Git for the operating system”. It operates in userspace, and works on top of any Linux file system. At its core is a Git-like content-addressed object store with branches (or “refs”) to track meaningful file system trees within the store. Similarly, one can check out or commit to these branches. It is used by endless OS, Flatpak, Fedora, CentOS, and the GNOME continuous project for continuous delivery of GNOME components.⁹

The main features of systems based on libOSTree and valuable for this article are:

• a strict rule to organize the file-system layout in a standard way based on the Filesystem Hierarchy Standard (FHS)¹⁰;
• transactional upgrades and rollback: any upgrade can be thought of similarly to the checkout of a set of commits in a git repository, excluding the machine-specific content in /var. The only writable directories are /etc and /var. When an upgrades occurs, the host-specific configuration content stored in /etc is passed through a 3-way merge: the content in /etc is compared with the one in /usr/etc and only the non-modified content is reconciled with the one in the upgrading /usr/etc one.
• CoreOS layering¹¹ and ostree native containers: rpm-ostree inherits the work in ostree-rs-ext¹² to implement the “container native ostree” functionality. This feature elevates Open Container Initiative (OCI) images to be natively supported as a transport mechanism for bootable operating systems.

The following sections will present how points 1-4 can be achieved with the above mentioned technologies. The article will go from building the base os image up to manually migrating an existing Fedora Workstation installation not handled via libOSTree, to an ostree, dual-boot, ostree-native-container-image-based Fedora Kinoite installation.

Note that, at the time of writing, dual-boot with Fedora Silverblue/Kinoite and custom partitioning are considered non-supported features. Any specificity of the sections regarding the migration is just having the means of inspiring others looking for a way to safely migrate by maintaining their current OS installation to an OSTree-based OS and for the sake of learning more about the file-system layout and the ostree repository management.

The idea is to (1) customize Fedora Kinoite via container-based OSTree, (2) install side-by-side to a Fedora Workstation with no use of Anaconda, (3) have a personal transport to deliver configuration and installed software to the managed workstations.

Points 5 and 6 will be covered in the future.

Wrapping up a CI to build the OSTree native container images (via GH actions)⌗

Building the Base OS Image⌗

Note: this is still WIP. In particular, no caching mechanism is being proposed, leading to higher build times.

The workstation-ostree-config repository¹³ is the source of truth of how Fedora Silverblue, Kinoite, and other flavors are built and distributed to end-users. In particular, the definition of the base ostree is described via treefiles¹⁴: this is the main one for Fedora Kinoite.

Building an OSTree native container image with rpm-ostree for these repositories is doable as in the following cli:

rpm-ostree compose image --initialize --format registry fedora-kinoite.yaml ${{ env.IMAGE_REGISTRY }}/${{ env.REGISTRY_USER }}/${{ env.IMAGE_NAME }}:rawhide-${VERSIONED_TAG}

Therefore, it is straightforward to build a GitHub action that periodically builds the base os image for Fedora Kinoite rawhide: it’s worth noting that the GitHub action will push an image with a tag that carries the HEAD commit in the workstation-ostree-config repo and the build time: not a guarantee for univocity but good enough to keep more versions of the images without triggering the layers garbage collection on quay.io when storing a new image.

After pushing to a versioned tag: e.g., rawhide-2022-12-12T0523-f50cab0125d736c8a5bec44b1cea1969c49777c8, one can retag the image to the generic tag.

skopeo copy docker://quay.io/${{ env.REGISTRY_USER }}/fedora-kinoite:rawhide-${VERSIONED_TAG} docker://quay.io/${{ env.REGISTRY_USER }}/${{ env.IMAGE_NAME }}:rawhide

Building the Content OS⌗

The base os image built in the previous section will hopefully be provided officially soon¹⁵. It should (and will) also involve automated tests and caching to improve build time.

The Content OS, instead, is the actual final image that one could use to automatically deliver content for a set of managed systems.

It’s built by using layering, by defining a Containerfile and using a container build engine like Docker, Podman or Buildah to build and ship it in quay.io as well.

The Containerfile is organized to:

1. Install packages not delivered by default in Fedora Kinoite;
2. Execute specific configuration: as per what is described and linked above, one should prefer delivering the configuration on /usr, but some software don’t allow that yet, as for example zsh that will pivot at most from /etc the configuration files;
3. Deliver specific systemd units with the logic to ensure software installation and configuration that need to be executed in an actual booted system is executed at the proper time(s).

The CI is still trivial and consists of a step to define a versioned tag as before, one to build and the last one to push the image.

Finally, a choice that could seem like an anti-pattern in the real Containers scenarios is leveraged. As we want to deliver updates by upgrading a container image manifest, it’s not worth synthesizing in a single layer all the packages installation as mostly done for the application delivery use cases.

Instead, we should distribute the installation of packages across different layers so that packages that need upgrades will change only some small chunked layers and not a single big one. This strategy is already exploited by default for building the base os image. Still, one should take care of a layered image to reduce the network overhead, the upgrade time, the OSTree, and the required registry storage size.

Installing without a boot image and dual-boot with Fedora Workstation⌗

Usually, OSTree and the other technologies covered above are meant to maintain a system over upgrades and changes. The installation is delegated to other systems: Fedora Silverblue and Kinoite can be installed via Anaconda¹⁶, Fedora CoreOS provides the coreos-installer; OKD and OCP use ad-hoc boot images and then rebase the OSTree to a container image shipped by the OKD/OCP payload¹⁷. However, for Fedora Kinoite and Silverblue, there is no supported way to maintain another system (e.g., dual-booting with an already existing Fedora Workstation), and a custom partitioning scheme is not recommended.

The purpose of this section is to go through a manual, from-scratch installation that uses a custom partitioning and provides dual booting with another Fedora Workstation installation previously installed and based on the btrfs file-system.

A reference point to succeed with this installation has been the anaconda source code related to rpm-ostree.

Custom partitioning⌗

The starting disk layout was like the following

fedora aleskandro_kinoite # lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 111.8G  0 disk 
├─sda1   8:1    0   600M  0 part /boot/efi
├─sda2   8:2    0     1G  0 part /boot
└─sda3   8:3    0 110.2G  0 part /
                                 /home
sdd      8:48   0 119.2G  0 disk
└─sdd1   8:49   0 119.2G  0 part
zram0  252:0    0     8G  0 disk [SWAP]

• sda3 and sdd1 are set as btrfs-based raid0 for data and raid1 for metadata;
• sda1 is the EFI partition;
• sda2 is the Feodora Workstation /boot ext4 partition.

Subvolumes for the btrfs FS are like in the following:

fedora aleskandro_kinoite # btrfs subvolume list /
ID 256 gen 3210681 top level 5 path home  ## << This is the home directory that will be shared by both Fedora Workstation and the custom Fedora Kinoite
ID 257 gen 3210631 top level 5 path root ## << root partition for Fedora Workstation
fedora aleskandro_kinoite # 

Creating the new subvolumes⌗

To support a new ostree-based installation of Fedora Kinoite, let’s define two new subvolumes:

su
mkdir -p /mnt/btrfs
mount -o subvolid=5 /dev/sda3 /mnt/btrfs
cd /mnt/btrfs
btrfs subvolume create fedora
btrfs subvolume create var

The subvolume fedora will store the root FS with ostree. The var file system will be used for the /var related to the stateroot we will create later for Fedora Kinoite. OStree supports having more distributions by means of (git-like) branches. Each stateroot should use a different /var.

fedora aleskandro_kinoite # btrfs subvolume list /
ID 256 gen 3210681 top level 5 path home
ID 257 gen 3210631 top level 5 path root
ID 82239 gen 3210681 top level 5 path var
ID 82245 gen 3210648 top level 5 path fedora
fedora aleskandro_kinoite # 

Initialize the ostree⌗

cd /mnt/btrfs/fedora
ostree admin init-fs --sysroot=$(pwd) $(pwd) # Initialize the ostree with a repo and a base root tree
ostree container image pull ostree/repo ostree-unverified-registry:quay.io/aleskandrox/fedora-kinoite:rawhide-layered # Pull the custom image ref
ostree admin --sysroot=$(pwd) os-init fedora-kinoite # Initialize the stateroot deployment https://ostree.readthedocs.io/en/stable/manual/deployment/
ostree container image deploy --sysroot $(pwd) --stateroot fedora-kinoite --imgref ostree-unverified-registry:quay.io/aleskandrox/fedora-kinoite:rawhide-layered

Create other files and prepare for chroot⌗

Some files, especially the ones in the /var directory that maintain the machine state and are not versioned by ostree, need to be created manually. Also, we will want to chroot into the system to finalize some configurations.

Mount the deployed sysroot:⌗

mkdir -p /mnt/fedora
mount --rbind /mnt/btrfs/fedora/ostree/deploy/fedora-kinoite/deploy/*.0 /mnt/fedora
mount --make-rslave /mnt/fedora
mount --bind /mnt/btrfs/fedora/boot /mnt/fedora/boot
mount -o ro --bind /mnt/btrfs/fedora/ostree/deploy/fedora-kinoite/deploy/*.0/usr /mnt/fedora/usr/ # read-only mount for <sysroot>/usr, as if we are in the final system

Mounting the transient file systems⌗

cd /mnt/fedora
mount -t proc /proc ./proc
mount --rbind /sys ./sys
mount --make-rslave ./sys/
mount --rbind /dev ./dev/
mount --make-rslave ./dev
mount --bind /run ./run
mount --make-slave ./run
mount --bind /mnt/btrfs/fedora/ sysroot
mount -o subvol=var /dev/sda3 ./var
mkdir -p mnt/btrfs
mount -o subvolid=5 /dev/sda3 mnt/btrfs

Creating the /var initial content⌗

for d in /var/home /var/roothome /var/opt /var/srv /var/usrlocal \
          /var/mnt /var/media /var/spool /var/spool/mail; do
    systemd-tmpfiles --create --boot --root=/mnt/fedora/ --prefix=${d}
done

chroot⌗

chroot .

From now on, one can act in the final system to finalize the configuration and prepare the actual boot.

Setting fstab⌗

The next fstab example takes into account the btrfs layout defined above to mount both the previous Fedora workstation (oldroot) and the new system.

UUID=f2ed5a7a-6787-494d-b383-ab2b44ad4724               /var/                   btrfs   noatime,ssd,discard=async,space_cache,commit=60,subvol=var      0 0
UUID=f2ed5a7a-6787-494d-b383-ab2b44ad4724               /var/home               btrfs   noatime,ssd,discard=async,space_cache,commit=60,subvol=home     0 0
UUID=f2ed5a7a-6787-494d-b383-ab2b44ad4724               /var/mnt/oldroot        btrfs   noatime,ssd,discard=async,space_cache,commit=60,subvol=root     0 0
UUID=f2ed5a7a-6787-494d-b383-ab2b44ad4724               /var/mnt/btrfs          btrfs   noatime,ssd,discard=async,space_cache,commit=60,subvolid=5      0 0
UUID=0c418d3c-586f-4341-badc-e0fa8785eae0               /var/mnt/oldroot/boot   ext4    defaults,x-systemd-requires-mounts-for=/var/mnt/oldroot         1 2
UUID=409E-3A31                                          /boot/efi               vfat    umask=0077,shortname=winnt,x-systemd-requires-mounts-for=/boot  0 2

/var is the specific subvolume created for the new install, /var/home is the home partition previously available for Fedora Workstation and /var/mnt/oldroot{,/oldboot} mounts the root of the previous Fedora Workstation installation. Finally, /boot/efi is to be mounted after the /boot default bind mount is done by Fedora Kinoite.

Create /etc/default/grub and the grub.cfg for Fedora kinoite⌗

GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=false
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_HIDDEN_TIMEOUT=0
GRUB_SAVEDEFAULT=true

During the ostree deployment, a grub.cfg file is created at /mnt/btrfs/boot/loader/grub.cfg. In the following snippet, we conclude the boot configuration by:

• creating a symbolic link to it in /boot/grub2/grub.cfg for use in the EFI grub.cfg for dual-booting;
• creating the /boot/efi dir to mount the EFI partition at system boot
• setting the default blsdir parameter for the bootloader entries¹⁸ in the grubenv file.
• creating a symbolic link in /mnt/btrfs/ostree to the /boot/ostree folder in the fedora subvolume. In fact, the root partition that the instance grub2-mkconfig executed by ostree will use as the base for the linux and initramfs location is the top level subvolume in the btrfs partition: the link will make grub2 able to load the kernel and the initramfs.
• setting the default kernel pararmeters to make the initramfs aware of the partition for root pivoting.

mkdir -p /boot/{grub2,efi}
ln -s ../loader/grub.cfg /boot/grub2/grub.cfg
grub2-editenv /boot/grub2/grubenv set blsdir=/fedora/boot/loader/entries
grub2-editenv /boot/grub2/grubenv set kernelopts="root=UUID=f2ed5a7a-6787-494d-b383-ab2b44ad4724 rootflags=subvol=fedora rw mitigations=off debug
ln -s fedora/boot/ostree /mnt/btrfs/ostree

Dual booting with Fedora Workstation⌗

Two Grub instances are available by default with the Fedora workstation installation using UEFI. One has its configuration under /boot/grub2/grub.cfg and can be kept specific to Fedora Workstation, another is in /boot/efi/EFI/fedora/grub.cfg and can be re-configured to allow choosing which second grub.cfg to chain to.

The following grub.cfg is key to minimize the interference between the OSTree managed bootloader configuration for Fedora Kinoite and the ones managed by Fedora Workstation (e.g., via dnf when upgrading the kernel).

# /boot/efi/EFI/fedora/grub.cfg:

set timeout=30

menuentry 'Fedora Kinoite' {
        insmod btrfs
        set prefix=(hd0,gpt3)/fedora/boot/grub2
        export $prefix
        configfile $prefix/grub.cfg
}

menuentry 'Fedora Workstation' {
        search --no-floppy --fs-uuid --set=dev 0c418d3c-586f-4341-badc-e0fa8785eae0
        set prefix=($dev)/grub2
        export $prefix
        configfile $prefix/grub.cfg
}

We’ve allowed the UEFI first grub2 configuration to provide a menu rather than immediately chaining to the next grub.cfg, so that one can choose the second grub2 configfile to load. The system will chain to the Fedora Kinoite grub2 configuration set in the previous section when selecting the Fedora Kinoite menu entry.

The second grub2 config for Fedora Kinoite will use the entries available at /boot/loader/entries to fulfill the entries managed by ostree with no interference with the upgrade system.

Creating the user and setting the root password⌗

useradd -m -d /var/home/aleskandro_kinoite -s /bin/zsh aleskandro -G
passwd aleskandro
passwd root

Fixing the SELinux labeling⌗

Based on the environment you are running on, some SELinux labeling can be lost or set bad:

loadpolicy -i
chcon -t shadow_t /etc/shadow

restorecon -vRF -T 0 -x /etc
restorecon -vRF -T 0 -x /var
restorecon -vRF -T 0 -x /var/home/aleskandro_kinoite # Only the new one

Rebooting⌗

Rebase again⌗

Rebasing again will allow rpm-ostree to conclude the configuration and wrap the previously lower-level ostree configuration we made.

rpm-ostree rebase ostree-unverified-registry:quay.io/aleskandrox/fedora-kinoite:rawhide-layered

What’s next⌗

Not every challenge that a desktop Linux user can face has been resolved via OSTree native container images, but a very cool and fun step towards a desired state vs current state feedback loop appraoch to the confiuguration management of personal (and not only) devices is posible today.

Some challenges still remain to be resolved or to pass through best-practices approaches like the ones from CoreOS (if not already done by someone else upstream in the community). Some examples are:

• Delivering sensitive content
• Automatically handling initial provisioning of users and similar system config via other tools (ignition?)
• Portbility of homes (systemd-homed?)
• Deliver Desktop Environment and application configuration

The work powered by the thoughts in this article is at https://github.com/aleskandro/my-ostree-config.