fix(systemd): drop local-fs.target ordering from rechunker-group-fix.service

[Danathar]

The bug

rechunker-group-fix.service ships from here into bluefin and bluefin-lts, and it declares two things that cannot both be true:

• Before=systemd-sysusers.service
• Wants= / After=local-fs.target

Walking the stock edges between those two points shows why:

rechunker-group-fix.service   Before=  systemd-sysusers.service
systemd-sysusers.service      Before=  systemd-tmpfiles-setup-dev.service
systemd-tmpfiles-setup-dev    Before=  local-fs-pre.target
local-fs.target               After=   local-fs-pre.target
rechunker-group-fix.service   After=   local-fs.target             ← closes the loop

So every boot of a consuming image opens with an ordering cycle.

Why the symptom is a coin toss

systemd breaks a cycle by deleting whichever job in the loop it reaches first, and that choice varies per boot and per image. Depending on the victim you get:

Job deleted	What the user sees
systemd-udevd	90 s device timeouts, /var never mounts, no home directories
systemd-ask-password-console.path	LUKS password agent never starts — encrypted volumes never come up after the initrd hands over
systemd-sysusers	missing system groups
local-fs-pre.target	assorted mount fallout

One defect, several bug reports that look unrelated: projectbluefin/bluefin-lts#585, projectbluefin/bluefin-lts#466, projectbluefin/bluefin-lts#391 and #918.

Scope correction: this PR was originally opened for projectbluefin/bluefin-lts#628 (LUKS hang on the current :stable). We now think that is probably a separate bug in the initrd — the Fedora 7.1.8 kernel's CONFIG_MODPROBE_PATH points at /usr/bin/modprobe, which CentOS Stream 10 userspace doesn't ship, so dm-crypt can't autoload — but that is not yet confirmed on the reporter's machine (details and pending test). The cycle below is still real on that reporter's machine — their journal shows it — but it is not what stops their boot, and this PR should not be expected to fix #628.

Confirmed on real hardware since this PR was opened — the reporter in bluefin-lts#628 posted a journal from their own machine showing the cycle with yet another victim (the ESP's fsck job):

boot-efi.mount: Found ordering cycle on systemd-fsck@...832D-6772.service/start; has dependency on
  local-fs-pre.target/start, systemd-tmpfiles-setup-dev.service/start, systemd-sysusers.service/start,
  rechunker-group-fix.service/start, local-fs.target/start, boot-efi.mount/start
boot-efi.mount: Job systemd-fsck@...832D-6772.service/start deleted to break ordering cycle

The change

1. Order the unit the way its neighbours are ordered. Match systemd-sysusers.service and bootc's own bootc-sysusers-shadow-sync.service:

After=systemd-remount-fs.service
After=bootc-sysusers-shadow-sync.service
Before=systemd-sysusers.service

No local-fs.target edges at all. The systemd-remount-fs.service anchor answers the "is /etc writable yet?" question raised in the #921 review — it is the same precondition sysusers itself relies on to write /etc/passwd.

2. Drop the trailing ExecStart=systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev. With the unit now running before local-fs-pre.target, /var and /tmp are not mounted yet and that pass exits 65 on every single boot — this is the rechunker-group-fix.service failure line in every ujust report, and what @fabsah and @jlbauss were seeing. Nothing is lost by removing it: systemd-tmpfiles-setup.service runs the identical command later, after local-fs.target and after systemd-sysusers.service, i.e. after the groups repaired here exist.

3. Document the boundary in docs/skills/submodule-boundary.md, specifically why a downstream drop-in cannot fix this. systemd cannot reset After= or Wants= from a drop-in, and After= with an empty value is a no-op for dependencies (verified with systemctl show -p After on systemd 259). So the bluefin-lts#527 / bluefin-lts#603 style drop-ins only manage to add Before=local-fs-pre.target, producing a tighter cycle than the one they were meant to break.

Supersedes #921 — same core edge removal, but without the unrelated riders it was blocked on, and adding the systemd-remount-fs.service anchor, the tmpfiles removal, and a transaction-level test. Happy to have this folded into #921 instead if that's preferred.

Regression coverage

tests/test_rechunker_group_fix.bats adds three tests:

1. Static ordering contract — required edges present, no local-fs*.target edges, no systemd-tmpfiles ExecStart.
2. Transaction-level check — systemd-analyze verify --root=<fixture> default.target over a minimal fixture of the stock early-boot units (systemd-remount-fs, systemd-sysusers, systemd-tmpfiles-setup-dev, systemd-udevd, local-fs-pre.target, local-fs.target, sysinit.target). This lets systemd itself compute the boot transaction, and asserts no ordering cycle / deleted to break output.
3. Guard against a dead fixture — re-adds After=local-fs.target to a copy of the fixture and asserts systemd does report the cycle, so test 2 cannot silently pass on a broken fixture.

Run on Ubuntu 24.04 / bats 1.10 / systemd 255 (the CI runner environment):

# with this branch
ok 8  rechunker-group-fix.service: ordering contract matches systemd-sysusers.service
ok 9  rechunker-group-fix.service: systemd computes no ordering cycle for the boot transaction
ok 10 rechunker-group-fix.service: ordering fixture detects the local-fs.target cycle

# with the unit from main
not ok 8  rechunker-group-fix.service: ordering contract matches systemd-sysusers.service
not ok 9  rechunker-group-fix.service: systemd computes no ordering cycle for the boot transaction
# local-fs-pre.target: Found ordering cycle on systemd-tmpfiles-setup-dev.service/start
# local-fs-pre.target: Job systemd-tmpfiles-setup-dev.service/start deleted to break ordering cycle starting with local-fs-pre.target/start

Validation performed

Tests and linters

• bats tests/test_rechunker_group_fix.bats — 10/10 pass (Ubuntu 24.04 container, systemd 255). Tests 8–9 fail against the unit on main.
• shellcheck on the script and the bats file; just check; git diff --check.
• scripts/check-doc-links.sh, check-skill-index.sh, generate_skill_index.py --check — all clean.

Against the real consumer image

ghcr.io/projectbluefin/bluefin-lts:stable (amd64 sha256:33ba155b…, kernel 7.1.8-200.fc44, systemd 257-33.el10), in a container with an Anaconda-style /etc/fstab + /etc/crypttab for a LUKS root. Ran systemd-analyze verify --generators=yes over default.target, multi-user.target, graphical.target, cryptsetup.target and rechunker-group-fix.service:

Image	ordering cycle lines
as shipped	16
with this unit dropped in	0

As shipped, one of those cycles runs through cryptsetup.target and deletes the LUKS password agent, which would leave any encrypted volume that still needs unlocking after switch-root without a prompt:

cryptsetup.target: Found ordering cycle on systemd-ask-password-console.path/start; has dependency on
  plymouth-start.service/start, systemd-udevd.service/start, systemd-sysusers.service/start,
  rechunker-group-fix.service/start, local-fs.target/start, zfs-mount.service/start,
  zfs-import.target/start, zfs-import-cache.service/start, cryptsetup.target/start
cryptsetup.target: Job systemd-ask-password-console.path/start deleted to break ordering cycle starting with cryptsetup.target/start

For comparison, the bluefin-lts#527/#603 drop-in measured in the same image: 14 cycle lines, versus 10 for the same anchor set without it — including Job local-fs.target/start deleted.

Not performed

• No real boot test. Nothing here boots a VM or a LUKS machine. The evidence is the systemd transaction solver (systemd-analyze verify) run against the real image's unit set, plus the bats fixture. A lab boot of a consuming image — ideally an upgrade from a legacy rechunked image with a LUKS root, as in bluefin-lts#585 — is still needed before promotion, per the warning in the unit header.
• pre-commit run --all-files and actionlint were not run locally (not installed). The individual hook scripts listed above were run directly.

---

Closes #918
Refs projectbluefin/bluefin-lts#628 (scope corrected — not fixed by this PR)
Refs projectbluefin/bluefin-lts#585 projectbluefin/bluefin-lts#466 projectbluefin/bluefin-lts#391

🤖 Generated with Claude Code

https://claude.ai/code/session_0114TUzBMEH13zPa2VVYwLdW

[castrojo]

Reviewed diff and verified ordering contract. Dropping Wants=/After=local-fs.target removes the boot ordering cycle on systemd-sysusers. Fixes projectbluefin/bluefin-lts#628 and common#918.

[castrojo]

LGTM, approving as second review to satisfy ruleset.

Duplicate review

[Danathar]

Scope correction

This PR was opened as the fix for projectbluefin/bluefin-lts#628. It most likely isn't.

What changed: the reporter posted console output from the failing boot. It fails inside the initrd, before any of the units this PR touches ever run:

device-mapper: table: 252:0 crypt: unknown target type

Current theory (not yet confirmed): the Fedora 7.1.8-200.fc44 kernel is built with CONFIG_MODPROBE_PATH="/usr/bin/modprobe", but CentOS Stream 10 only ships /usr/sbin/modprobe, so the kernel can't autoload dm-crypt. The reporter has been asked to test a kernel-arg workaround; write-up here: projectbluefin/bluefin-lts#628 (comment). If it holds up, that fix belongs in bluefin-lts, not this repo.

What this means for this PR:

• The ordering cycle it removes is still real — the same reporter's journal shows rechunker-group-fix.service in a cycle that deleted the ESP's fsck job.
• fix(systemd): rechunker-group-fix creates local-fs ordering cycle on upgrades #918, bluefin-lts#585, Update curated.yaml: added Ecosia browser in the browser section #466 and feat(hive): add bonedigger lifecycle bot and hive-status-sync #391 still stand.
• I've edited the description to drop Fixes #628 and the "root cause of test(scripts): add bats coverage for ublue-motd and profile.d scripts #628" wording, so this doesn't get merged on a false premise or auto-close that issue.

[Danathar]

This is actually a little interesting...if true its exactly the kind of thing that could bite on an LTS version because it ships something different.

[castrojo]

LGTM: verified drop of Wants=/After=local-fs.target eliminates the ordering cycle on systemd-sysusers. CI checks green.