fix: rust-governor lock leaks to the sccache daemon and wedges every later build

[kazmosahebi]

The rust-governor shim's build lock can be captured permanently by the sccache daemon, after which every cargo invocation on the machine blocks forever with no timeout and no diagnostic.

Mechanism

~/.local/bin/cargo serialises builds on a per-user flock (shim lines 107-110):

exec 9>"$LOCK"
if ! flock -n 9; then
    notify_wait
    flock 9
fi

The comment above it (line 104) states: "The fd survives exec, so the lock lives exactly as long as the build and releases on any exit, crash included."

That holds only if no long-lived descendant inherits fd 9. A flock belongs to the open file description, so any daemon started during a governed build keeps the lock alive after the build itself exits.

sccache is exactly such a daemon. When hyperi-sccache.service is not running, cargo auto-spawns sccache inside the locked scope; it inherits fd 9, daemonises, and holds the lock until it is killed.

Trigger

hyperi-sccache.service ships disabled (with preset: enabled), so on a fresh SOE machine the first Rust build spawns sccache ad-hoc and the lock is captured immediately. The shim already anticipates the service (line 8: "sccache-hosted compiles are captured separately: hyperi-sccache.service carries Slice=rust-build.slice") - but nothing enables or starts it.

Symptom

A build that looks alive but does nothing:

• 0% CPU, no rustc children
• wchan is do_wait; its only child is flock 9
• /proc/locks shows a holder PID that no longer exists

Observed while running a local quality gate over a Rust workspace:

$ stat -c '%i' /run/user/1001/hyperi-rust-govern-1001.lock
107
$ rg ":107 " /proc/locks
7: FLOCK  ADVISORY  WRITE 3078836 00:3f:107 0 EOF
7: -> FLOCK  ADVISORY  WRITE 3078951 00:3f:107 0 EOF
$ ps -p 3078836          # holder is dead
$ ls -l /proc/3078858/fd/9
... -> /run/user/1001/hyperi-rust-govern-1001.lock   # sccache holds it

Killing sccache releases the lock, but the respawned daemon re-inherits fd 9 within seconds, so the wedge returns on the next cold build.

Impact

hyperi-ci check --quick over the same Rust workspace, same machine, same commit:

	elapsed	avg CPU
lock captured	2670 s (44.5 min)	1%
after fix	32.75 s	95%

An 81x difference that presents as "Rust builds are slow" or "the CI hook is unusable", with nothing in any log to indicate a lock is involved. It affects every Rust build on the box, not just hyperi-ci. In this case it was misattributed to disk pressure for some time.

The macOS path does not have this bug

The non-flock branch (lines 113-130) stamps the holder's PID into the lock directory and clears it when kill -0 fails, so staleness is detected there. The Linux flock path has no equivalent, precisely because it assumes fd lifetime equals process lifetime.

Suggested fix

1. Enable and start hyperi-sccache.service in the role. It is already written and already carries Slice=rust-build.slice; it just is not enabled. This removes the trigger.

2. Bound the wait, as defence in depth. Any long-lived daemon spawned during a governed build reproduces this, so removing sccache as the trigger does not remove the class. flock -w <timeout> 9 degrades a captured lock into a bounded delay plus a warning naming the holder, instead of a permanent hang. A build that runs unserialised is better than one that never runs.

Workaround

pkill -x sccache
systemctl --user enable --now hyperi-sccache.service

Then confirm the service's fd 9 is a socket rather than the lock file, and that /proc/locks no longer lists the lock inode.

Not verified

• Whether daemons other than sccache (rust-analyzer, a test harness leaving a background process) trigger the same capture. The mechanism says they would; only sccache was observed.
• Whether the role intends the service to be enabled and this is a deployment regression, or whether it was always opt-in.

[catinspace-au]

The trigger described here is gone, but the hazard is not, so this stays worth fixing -- just for a different reason than the title says.

What changed underneath: sccache_service.yml now enables AND starts hyperi-sccache.service, so sccache is spawned by systemd and never inherits fd 9 from a build. That kills the observed path.

What is still true, and is the actual problem: hyperi-rust-govern IS ~/.local/bin/cargo on every SOE box (rust_governor_enabled defaults true), and the wait was a bare flock 9 -- no timeout, no diagnostic. ANY daemon started inside a governed build still captures the lock for its own lifetime, and every later cargo command then hangs forever with nothing in the log. The script header already warns about this for cargo run, which is the same class.

The fix takes the view that serialisation is an optimisation, not a correctness rule: wait HYPERI_RUST_GOVERN_LOCK_WAIT (30 min default, configurable), then build unserialised and name the holder. Degrading beats hanging. The macOS mkdir branch had the same unbounded shape and gets the same bound.

Two things found next to it and fixed in the same commit:

• all four XDG_RUNTIME_DIR sites derived the uid from ansible_facts['user_uid'], which is the CONNECTING user -- so on a split-user machine the systemd --user calls targeted an account with no session
• the sccache enable and start both carried failed_when: false with no deploy_warnings append, so a failure said nothing at all. That is exactly the silence that would let this wedge come back unnoticed.

Not reproduced: the original 81x timing measurement. The verdict rests on reading the shim and on a timeout test that degrades at 4s instead of hanging.

Fix in #70.

[catinspace-au]

Landed in 06035f1 (#70). The lock is bounded on both the flock and mkdir branches, the XDG_RUNTIME_DIR uid comes from actual_user, and the sccache enable/start now warn instead of failing silently.