machine-id

Read only root FS:

• with preinit script (before real /sbin/init)
  • tmpfs overlay on /etc mounted
  • bind mount persistent /mnt/state/machine-id on /etc/machine-id.
• first boot works well
• after reboot systemd complains:

systemd[1]: Starting Commit a transient machine-id on disk...
systemd-machine-id-setup[611]: /etc/machine-id is not on a temporary file system.
systemd-machine-id-commit.service: Main process exited, code=exited, status=1/FAILURE
systemd-machine-id-commit.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Commit a transient machine-id on disk.

SYSTEMD-MACHINE-ID-COMMIT(1)

It runs because of conditions defined in unit file:

systemd-machine-id-commit.service

[Unit]
DefaultDependencies=no
Conflicts=shutdown.target
Before=shutdown.target
After=local-fs.target first-boot-complete.target
ConditionPathIsReadWrite=/etc
ConditionPathIsMountPoint=/etc/machine-id
 
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=systemd-machine-id-setup --commit

This unit is designed to be run when /etc/machine-id is in transient state (e.g. bind mounted to tmpfs by systemd), so ConditionPathIsMountPoint=/etc/machine-id is fulfilled in our case.

As workarund ConditionFirstBoot=yes is added:

/etc/systemd/system/systemd-machine-id-commit.service.d/bind-mount-fix.conf

[Unit]
ConditionPathIsReadWrite=|/etc
ConditionPathIsReadWrite=|/mnt/state/machine-id
ConditionFirstBoot=yes
ConditionPathIsMountPoint=/etc/machine-id

Second approach is persisting /etc/machine-id on system which is bootstrapped from immutable /usr, with tmpfs on / #39438:

/etc/systemd/system/systemd-machine-id-commit.service.d/bind-mount-fix.conf

[Unit]
ConditionPathIsReadWrite=
ConditionPathIsReadWrite=/var/mutable/etc/
ConditionPathIsMountPoint=
ConditionPathIsMountPoint=/var/mutable/etc/machine-id
 
[Service]
ExecStart=
ExecStart=systemd-machine-id-setup --commit --root=/var/mutable/

More about problem: Various use-cases for first boot/machine-id are broken #14131