Fedora CoreOS is designed to be an immutable operating system, meaning many critical system directories—like /lib/modules—are read-only. This poses a challenge when you need to load custom kernel modules that aren’t shipped with the default kernel package.

In this post, you’ll learn how to use OverlayFS to compose a writable version of /lib/modules, allowing custom modules to be loaded at boot without modifying the underlying system.

Why OverlayFS?#

OverlayFS is a union filesystem that lets you layer a writable directory over a read-only one. While it’s common in container environments, it also works great for CoreOS when you want to “extend” immutable paths like /lib/modules.

Directory Layout#

Let’s assume you’ll place your custom kernel modules in:

  • Writable modules: /var/opt/modules
  • Overlay work directory: /var/opt/modules.wd

OverlayFS is designed to simulate a writable filesystem view (the overlay) by combining:

  • a read-only lower layer (e.g., /lib/modules)
  • a read-write upper layer (e.g., /var/opt/modules)
  • and a work directory, where temporary and metadata operations are stored

The kernel uses the work directory to handle special filesystem operations1 that involve changes across layers, such as:

  • Copy-up operations: When a file from the lower layer is modified, it is first copied to the upper layer. This operation’s metadata is tracked using the workdir.
  • Whiteouts: If a file in the lower layer is “deleted” in the overlay view, a special marker is created in the upperdir. That marker is managed using the workdir.
  • Rewrites and renames: These require coordination and temporary state, which the workdir tracks.

Without a valid workdir, the kernel cannot complete these operations safely, and the overlay mount will fail.

Automounting the OverlayFS at Boot#

To mount an overlay at /lib/modules, add the following line to your /etc/fstab:

overlay /lib/modules overlay lowerdir=/lib/modules,upperdir=/var/opt/modules,workdir=/var/opt/modules.wd,x-systemd.after=var.mount,x-systemd.before=systemd-modules-load.service 0 0

What this does:#

  • lowerdir=/lib/modules: The original read-only directory
  • upperdir=/var/opt/modules: Where you place custom .ko files
  • workdir=/var/opt/modules.wd: Required by OverlayFS to function
  • x-systemd.after=var.mount: Ensures /var is available first
  • x-systemd.before=systemd-modules-load.service: Ensures this overlay is mounted before loading the modules

Create the folders#

Make sure to create the directories for the overlay:

# mkdir -p /var/opt/{modules,modules.wd}

The resulting directory structure will look like this:

               +--------------------------+
               | /lib/modules (Overlay)  |
               +-----------▲--------------+
                           |
     +---------------------+---------------------+
     |                                           |
+------------+                           +---------------+
| Upperdir   | <= /var/opt/modules       | Lowerdir      | 
| (Writable) |     /lib/modules (r/o) => | (Immutable)   |
+------------+                           +---------------+
     |
+-------------+
| Workdir     | <= /var/opt/modules.wd
+-------------+

The kernel sees /lib/modules as a merged view of the lower and upper directories. Files in the upperdir take precedence.

Autoloading Custom Kernel Modules#

Fedora CoreOS uses systemd-modules-load.service to load modules at boot based on configuration files located in:

/lib/modules-load.d/
/usr/lib/modules-load.d/
/usr/local/lib/modules-load.d/
/etc/modules-load.d/
/run/modules-load.d/

To autoload your custom module (e.g., my_module.ko), create a file named /etc/modules-load.d/my_module.conf with the following content:

my_module

Make sure the module file itself is located in the corresponding /var/opt/modules/<kernel-version>/extra/ path (or any valid module path under the overlay), and that depmod is run if necessary to update module dependencies.

Restoring SELinux Labels#

After mounting the overlay, SELinux labels may need to be corrected to allow systemd-modules-load to function properly with any modules you added in /opt/modules.

Run this command after the overlay is active:

restorecon -vRF /lib/modules/

This will recursively restore the correct SELinux contexts, ensuring the system can read the modules.

Wrapping Up#

With OverlayFS, you can bypass the immutability of CoreOS for very specific, safe customizations like loading your own kernel modules—without breaking the integrity of the base OS. This approach is great for testing, development, and niche hardware support.

References#

  1. A practical look into overlayfs ↩︎