fix(systemd): remove rechunker local-fs ordering cycle

[castrojo]

Summary

• remove the Wants=local-fs.target and After=local-fs.target edges from rechunker-group-fix.service
• preserve ordering after bootc-sysusers-shadow-sync.service and before systemd-sysusers.service
• add a regression contract test and document the ordering invariant

The local-fs edges created an ordering cycle during upgrades from legacy rechunked images. The bridge unit must repair gshadow state before systemd-sysusers.service, without pulling in or waiting for local-fs.target.

Closes #918

Validation

• bash -n system_files/shared/usr/bin/rechunker-group-fix — passed
• git diff --check — passed
• bats tests/test_rechunker_group_fix.bats — not run: bats is not installed in the environment
• systemd-analyze verify ... — not run: systemd-analyze is not installed in the environment
• just check — not run: just is not installed in the environment
• pre-commit — not run: pre-commit is not installed in the environment

[castrojo]

This needs human review.

[hanthor]

LGTM. Clean, targeted fix with good test coverage.

What I checked:

• ✅ Wants=local-fs.target and After=local-fs.target removed from the service unit
• ✅ Inline comment explains the ordering cycle concisely
• ✅ Documentation in submodule-boundary.md updated to explain the constraint
• ✅ BATS test does not create a local-fs ordering edge enforces the contract
• ✅ Test also asserts the remaining ordering directives are preserved (DefaultDependencies, Before=sysusers, After=bootc-sysusers-shadow-sync)

Minor note: The lab-testing SKILL.md addition about PR candidate preflight is unrelated to the systemd fix. Not a blocker — it is a useful doc addition, but mixing it in makes the PR slightly harder to review.

[castrojo]

Deferred to the human queue during backlog triage — needs lab boot validation, not a desk review.

This removes the local-fs.target ordering edges from an early-boot unit whose own header warns:

Users WILL experience black screens and systems will NOT boot if this script malfunctions. Please test this properly and always make sure this works

The ordering-cycle diagnosis in #918 looks correct, and the added bats guard against reintroducing the edge is a good regression net. The part that cannot be settled by reading the diff is the runtime behaviour of ExecStart=systemd-tmpfiles --create --remove --boot once the unit no longer orders after local-fs.target — worth confirming it does not act on /var paths before those mounts are in place.

Also still DIRTY and needs a rebase.

Notably this PR also adds a lab-testing preflight note about workflows that report a PR SHA while actually booting the plain :testing image. That caveat should be honoured when validating this one: confirm the booted digest actually contains this commit before trusting a green boot.

[hanthor]

The systemd change looks right, but the PR carries two unrelated riders — one of which regresses a generated file

The core change is sound and the new test is not vacuous

Merged into main and run, the suite is clean and matches baseline exactly (same three pre-existing failure groups as unmodified main, no new ones), and the new case passes:

$ bats tests/test_rechunker_group_fix.bats
1..8
ok 1 rechunker-group-fix: appends missing group to empty gshadow
...
ok 8 rechunker-group-fix.service: does not create a local-fs ordering edge

The assertion is real — it greps the unit for ^(Wants|Requires|After)=local-fs\.target$ and for the three edges that must survive, so reverting the unit change fails it.

Blocker 1 — the PR regresses docs/skills/index.json and index.md by 15 days and one skill

These are generated files (scripts/generate_skill_index.py; the header literally says "do not hand-edit either file"). main is at:

$ head -2 docs/skills/index.json
{
  "generated_at": "2026-08-18",
$ grep -n "^Generated" docs/skills/index.md
6:Generated: 2026-08-18 · schema 1.0 · 40 skills

This PR sets them backwards to 2026-08-03 / 39 skills. That is why the branch is dirty:

$ git merge pr921
CONFLICT (content): Merge conflict in docs/skills/index.json
CONFLICT (content): Merge conflict in docs/skills/index.md

The date bump was stale on arrival and is the only reason this PR has conflicts at all. python3 scripts/generate_skill_index.py --check passes on main today; drop these two files from the diff (or rerun --write after rebasing) and the PR merges clean.

Blocker 2 — an unrelated second change the title does not mention

The title is fix(systemd): remove rechunker local-fs ordering cycle, but the diff also adds a new "PR candidate preflight" section to docs/skills/lab-testing/SKILL.md about verifying that a lab workflow's image digest resolves to the PR candidate. That is a useful paragraph and has nothing to do with rechunker ordering. Please split it out so the systemd change can be judged on its own.

Question on the fix itself (not a blocker, but please answer before merge)

docs/skills/submodule-boundary.md on main says these edges were added deliberately in common#530, and this PR deletes that provenance line along with the edges. The unit's ExecStart list writes to /etc:

ExecStart=bash -c 'touch /etc/gshadow && chmod 600 /etc/gshadow'
ExecStart=bash -c 'rm /etc/gshadow'
ExecStart=systemd-sysusers
ExecStart=rechunker-group-fix
ExecStart=systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev

With DefaultDependencies=no and local-fs.target removed, the only remaining ordering is After=bootc-sysusers-shadow-sync.service / Before=systemd-sysusers.service. The file header is explicit that "Users WILL experience black screens and systems will NOT boot if this script malfunctions", so: what guarantees /etc is the real, mounted, writable /etc at that point rather than a pre-mount one? The PR body asserts an ordering cycle but does not quote one. A systemd-analyze verify output, or the actual Found ordering cycle on … journal lines from an upgrade off a legacy rechunked image, would settle it — and per docs/skills/lab-testing/SKILL.md this is exactly the class of change that wants a boot-ordering VM test.

---

Generated by Claude Code