sched: bound pinned wake scans by outstanding home holds

docs/planning/green-program/aarch64-testing/3F-PR2B-2026-09-07.md

@@ -0,0 +1,263 @@
+# Slice 3f PR 2b — pinned wake scan bound, 2026-09-07
+
+Branch: `sched/3f-pr2b-pinned-wake-scan-bound`. Starting HEAD:
+`58af1a7fb63e144586b6eef06e47be178eb175d5`. Code and ratchets:
+`cab966e76c46d9a6eae343ba8bb5bf564d1ee24b`. Subsequent changes in this round
+are evidence and documentation. Evidence directory:
+`docs/planning/green-program/aarch64-testing/serials/3f-pr2b/`.
+
+## Premise and citation refresh
+
+PLAN-3F-CHAIN sections 0.2 and 2.1–2.5 were read in the supplied scratchpad.
+Its line numbers were re-derived from starting HEAD, then from the code
+commit. PR 942 is the merge `70aec756`; the starting revision also includes
+the subsequent work named in the brief: 947, 949, 950, 951, 953 and 962.
+The implementation citations below refer to `cab966e7`, not the plan's
+`72ad554d`. The source bytes are unchanged by the evidence commit.
+
+| Subject | Starting HEAD citation | Code-commit citation |
+| --- | --- | --- |
+| Ungated cumulative hold static | `kernel/src/task/scheduler.rs:447` | `kernel/src/task/scheduler.rs:447` |
+| Ungated delivery static | `kernel/src/task/scheduler.rs:493` | `kernel/src/task/scheduler.rs:499` |
+| Scheduler entry calls | `kernel/src/task/scheduler.rs:2348`, `kernel/src/task/scheduler.rs:2879` | `kernel/src/task/scheduler.rs:2358`, `kernel/src/task/scheduler.rs:2889` |
+| Hold / cumulative writer | `kernel/src/task/scheduler.rs:4747`, `kernel/src/task/scheduler.rs:4748` | `kernel/src/task/scheduler.rs:4757`, `kernel/src/task/scheduler.rs:4770` |
+| Derived waiting predicate | `kernel/src/task/scheduler.rs:4783` | `kernel/src/task/scheduler.rs:4800` |
+| Delivery / entry guard | `kernel/src/task/scheduler.rs:4829`, `kernel/src/task/scheduler.rs:4830` | `kernel/src/task/scheduler.rs:4843`, `kernel/src/task/scheduler.rs:4845` |
+| Delivery increment | `kernel/src/task/scheduler.rs:4842` | `kernel/src/task/scheduler.rs:4856` |
+| Normal-context census emitter | `kernel/src/task/scheduler.rs:572` | `kernel/src/task/scheduler.rs:578` |
+
+At starting HEAD, `kernel/src/task/scheduler.rs:4830` reads
+`PINNED_HOME_CPU_UNAVAILABLE.load(Ordering::Relaxed) == 0`. The static,
+delivery function, and `schedule` are ungated; `schedule_deferred_requeue`
+has an architecture gate but no feature gate. The cumulative writer at
+`kernel/src/task/scheduler.rs:4748` increments without retiring holds.
+Consequently, a single hold arms the thread scan on subsequent scheduler
+entries in the shipped no-feature profile. The predicate additionally
+examines ready queues, CPU current/previous/pending slots and, on aarch64,
+deferred requeue. This is a source-shape cost finding, not a cycle measurement.
+
+The production census call is `kernel/src/main_aarch64.rs:1446`; the sampled
+call is `kernel/src/task/strand_oracle.rs:478`, whose file is boot-tests gated
+at `kernel/src/task/strand_oracle.rs:3`. The production wrapper remains
+`kernel/src/task/kthread.rs:70`, calling `spawn_on_cpu` at
+`kernel/src/task/kthread.rs:79`; the definition is now
+`kernel/src/task/scheduler.rs:5572`. The source search found that wrapper
+call and no production call to the wrapper. The pin's per-CPU-state contract
+is still `kernel/src/task/thread.rs:627`, and `CPU_PINS_STAMPED` remains at
+`kernel/src/task/thread.rs:667`. Thus PR 4b remains the planned first
+production worker pin; the bound is useful before that exposure.
+
+PR 2b stays between PR 2 and PR 4a in the chain. PR 2 handles placement at
+five rescue/steal sites; this change handles hold credit balance and scan
+entry. Keeping it separate from PR 4b keeps the first production pin and its
+census acceptance distinct from this accounting change.
+
+## Implementation and scope
+
+`kernel/src/task/scheduler.rs:453` defines an inline-const array sized by
+`MAX_CPUS`, as already used at `kernel/src/task/scheduler.rs:402` and
+`kernel/src/task/percpu_stack_oracle.rs:328`. The hold resolves the home
+unconditionally and bounds the index before its saturating increment at
+`kernel/src/task/scheduler.rs:4764`. Delivery reads this CPU's slot before
+entering the loop, then saturating-decrements beside the delivered increment
+at `kernel/src/task/scheduler.rs:4857`.
+
+The brief explicitly requires retiring discarded holds. This extends the
+plan's proposed residue handling: after completing the thread pass, the
+same function retires remaining credits at `kernel/src/task/scheduler.rs:4867`.
+At that point eligible wakes have been enqueued. A remaining credit is a
+duplicate hold or a wake requeued, terminated, unpinned or removed elsewhere.
+The scheduler lock excludes concurrent holds during the pass. This does not
+remove thread state or consume a wake; no side list or sweep of other CPUs is
+introduced. There are two saturating lowering sites in the delivery function,
+for delivery and completed-pass discard, rather than the plan's delivery-only
+site. Saturating arithmetic avoids unsigned wraparound.
+
+The new array is a *reachability* counter, not a cumulative failure census
+field. Its aggregate is published as a separate bracketed line by the existing
+census emitter at `kernel/src/task/scheduler.rs:580`, using the existing
+serial macro call. The cumulative failure literal remains the one scored at
+`docker/qemu/run-aarch64-boot-test-strict.sh:307` and
+`docker/qemu/run-aarch64-prod-profile-boot-test.sh:175`; no failure count is
+subtracted to make a boot pass.
+
+With no local credit, the new entry work is the CPU lookup, one relaxed load
+and a comparison. A hold adds the home lookup, bounds comparison and relaxed
+atomic update. Delivery adds a relaxed atomic update per eligible wake and a
+load/update at completed-pass discard. Existing queue insertion remains in
+place. No new lock, allocation or formatting is added to hold/delivery paths.
+The additional formatting is in the existing normal-context census emitter.
+
+`scheduler.rs` is not Tier 2. Neither `context_switch.rs` nor `per_cpu.rs` was
+edited; a Tier-2 change would be permitted only where its own defect lives.
+The source change touches one kernel file, with no Tier-1 or Tier-2 edits.
+The pre-existing one-shot hold marker remains; no hot-path logging was added.
+
+## Ratchets and forced oracle
+
+`tests/loopback_pump_structure.rs:4850` discovers the counter from the hold's
+writer and checks the array shape, per-CPU guard dominating loop entry,
+delivery-branch decrement and completed-pass discard. Rename-together is
+GREEN. The extracted after-probe doubles its model CPU capacity from 8 to 16,
+using the inline-const array, and compiles with `-Dwarnings`.
+
+`serials/3f-pr2b/03-mutations-oracle.txt` records RED for guard deletion,
+replacing the guard with the cumulative counter, decrement removal (including
+the complete statement), non-saturating decrement, increment removal,
+unconditional decrement and discard removal. Mutations are in-memory strings;
+none is installed in the kernel source before a gate. The validator also
+rejects the unmodified starting revision.
+
+The forced oracle in `tests/loopback_pump_structure.rs:4918` extracts the
+production hold, predicate and delivery methods into isolated scheduler state,
+following the PR 2 extraction harness at
+`tests/loopback_pump_structure.rs:4307`. A walk meter is inserted into the
+extracted loop. This is a host execution probe, not a new live boot-tests
+kthread and does not exercise cross-CPU hardware dispatch. Its intentional hold stays
+out of the boot's cumulative failure counters; it does not subtract an oracle
+delta from them.
+
+The before/after readings in `serials/3f-pr2b/03-mutations-oracle.txt` are:
+
+```text
+PINNED_SCAN_ORACLE baseline=true held_delta=1 delivered_delta=1 outstanding_home=0 peer_entries=128 peer_walk_delta=128 after_entries=128 after_walk_delta=128 verdict=RED
+PINNED_SCAN_ORACLE baseline=false held_delta=1 delivered_delta=1 outstanding_home=0 peer_entries=128 peer_walk_delta=0 after_entries=128 after_walk_delta=0 verdict=PASS
+```
+
+The baseline's outstanding field is an unused harness slot, since that
+revision has no outstanding counter. In the after leg it is the source's
+actual accounting slot: home rises to 1, peer remains 0, delivery increments
+by 1, home returns to 0, and the cumulative hold count remains 1 during the
+128 subsequent entries. Five additional dispositions cover requeue,
+termination, pin clear, removal and duplicate holds.
+
+## Gate record
+
+The first standalone structure run was 65/66, attributable to adding a second
+serial macro call in `emit_pinned_placement_census`. Its pinned logging ledger
+expected one. The emitter was corrected to one call emitting two lines,
+without relaxing the ledger. The targeted correction passed, then the
+standalone full suite was rerun because source changed: 66/66 at `cab966e7`.
+This was a development correction, not a replay of a failed boot.
+
+The initial and final ARM builds passed; their only warning-shaped output
+was the accepted nightly-2025-06-24 `core` build-std future-incompatibility
+notice, per 559, 945 and
+`docs/planning/green-program/gates/CRITICAL-PATH-DEBT-PR1-2026-09-06.md`.
+No project warning was suppressed. `rustfmt --check --edition 2021
+kernel/src/task/scheduler.rs` and `git diff --check` exited 0.
+
+| Command at `cab966e7` | Result | Transcript |
+| --- | --- | --- |
+| `bash scripts/run-structure-tests.sh` after emitter correction | 66/66 suites, exit 0 | `serials/3f-pr2b/06-structure-final.txt` |
+| `bash docker/qemu/run-aarch64-boot-test-strict.sh 3` | 3/3 boots, exit 0 | `serials/3f-pr2b/09-strict.txt` |
+| `bash docker/qemu/run-aarch64-service-sequence-gate.sh --boots 2` | max 2/2 and cortex-a72 2/2, exit 0 | `serials/3f-pr2b/10-service.txt` |
+
+The seven ARM boot serials were independently checked for the new counter,
+one passing pin-guard oracle per boot and absent first-hold markers; see
+`serials/3f-pr2b/12-arm-census-audit.txt`. The new counter was 0 in 36
+samples (12 strict, 24 service). The gate transcripts carry their source
+revision; raw serial bytes have adjacent revision records.
+
+The first Beast preflight timed out `context_restore_structure` at 300 seconds,
+with host load 15.62 (15.50/9.52 over the longer windows); 65 other suites had
+completed. Before a kernel build or QEMU boot, the lane's launcher and its
+identified descendants were stopped by PID. The timeout transcript is
+`serials/3f-pr2b/13-x86-preflight-timeout.txt`. As directed, the gate was
+relaunched once with `BREENIX_STRUCTURE_SUITE_TIMEOUT_SECS=900`. No structure
+skip was set. The built-in 600-second suite retry was interrupted as part of
+stopping that first preflight, not counted as a completed run.
+
+The production gate was the final gate invocation and final Mac kernel build,
+after strict and both service profiles completed. The separate Beast retry
+continued concurrently; its build cannot overwrite the Mac kernel. The brief's
+single production invocation supersedes the older plan's three production boots.
+
+Production passed its 66/66 structure preflight, built the shipped no-feature
+kernel, and reached bsshd with the futex oracle seam absent (exit 0), recorded
+in `serials/3f-pr2b/14-production.txt`. This was one invocation and one boot.
+The raw production census at
+`serials/3f-pr2b/production-serial.txt:176` and
+`serials/3f-pr2b/production-serial.txt:177` reads:
+
+```text
+[PINNED_HOME_CPU_UNAVAILABLE:count=0:publish_discarded=0:hold_pen_migrated=0:delivered=0:migration_refused=0:stack_home_conflict=0]
+[PINNED_HOLDS_OUTSTANDING:count=0]
+```
+
+The production serial contains 0 pin-guard oracle lines and 0 first-hold
+markers, checked in `serials/3f-pr2b/production-revision.txt`.
+The original cumulative literal and the reachability line are separate:
+retiring a credit does not erase a historical failure.
+
+**Round stopped: UNATTRIBUTED x86 preflight red, tracked in issue 963.**
+The retry finished at 65/66 suites. `context_restore_structure` passed in
+368 seconds, but
+`tests/gate_qmp_backstop_structure.rs::missing_socket_and_hung_dump_are_partial_and_bounded`
+failed at `tests/gate_qmp_backstop_structure.rs:474` with `hang must reach dump`.
+The test reported socat present, accepted the timeout result and elapsed
+bound, then failed its actual dump-command receipt assertion. The failing
+suite and `docker/qemu/lib/gate-qmp-backstop.sh` have no changes from
+`58af1a7f` in this branch. This is not one of the plan's pre-adjudicated boot
+signatures (555 softirq, 536 timer_delay, 576 EL1 NULL-PC, 586
+loopback_wake_test_child:15). It is not attributed to 871: the failure log
+contains no SCSI/IO failure signature.
+
+The x86 gate exited 1 before its kernel build and QEMU boot. **x86 kernel
+compilation and boot are unverified in this round.** The userspace preparation
+completed; the retry repeated its cached preparation because the setup check
+still found no `userspace/programs/x86_64/*.elf`. This is not kernel build
+evidence. No third gate invocation was made, no criterion was weakened, and
+no unrelated fixture change was folded into PR 2b. Evidence is
+`serials/3f-pr2b/16-x86-retry.txt`,
+`serials/3f-pr2b/17-x86-qmp-failure.txt` and
+`serials/3f-pr2b/beast-preflight-logs.tar.gz`.
+
+Follow-up is [issue 963](https://github.com/ryanbreen/breenix/issues/963).
+Repair or attribute that fixture's failed dump leg, then resume x86 acceptance.
+The branch is submitted as a draft with this gate red, not merge-ready.
+Mac and Beast lane QEMU process counts were checked at 0; cleanup used only
+identified lane PIDs, with no name-based killing.
+
+## Chips and claim discipline
+
+| Chip | Before | Removed | After | Scope |
+| --- | --- | --- | --- | --- |
+| PR 2b claimed chips | 0 | 0 | 0 | Scan accounting only; no worker pin or issue credit |
+
+claim-lint: python3 scripts/claim-lint.py -> exit 0
+claim-lint: python3 scripts/claim-lint.py --commit-msg .tmp/code-commit.txt -> exit 0
+
+These are the checks immediately before `cab966e7`; the transcript is
+`serials/3f-pr2b/08-code-claim-lint.txt`, and its message is retained as
+`serials/3f-pr2b/code-commit-message.txt`.
+
+## NOT claimed
+
+- That the scan is eliminated: a CPU with an outstanding hold still scans.
+- That the residue described in plan section 2.3 is bounded in wall-clock time:
+  discard reconciliation requires that home CPU to enter the scheduler.
+- That cost was measured in cycles: this round measures source-extracted walks
+  and enumerates operations, without a cycle-count experiment.
+- That PR 2b is required before PR 4a: its prerequisite relationship is to
+  PR 4b, the planned first production pin.
+- That the extracted oracle establishes hardware dispatch, SGI delivery or a live
+  boot's forced pinned-wake recovery. Those are outside its model.
+
+
+claim-lint: python3 scripts/claim-lint.py -> exit 0
+claim-lint: python3 scripts/claim-lint.py --commit-msg .tmp/docs-commit.txt -> exit 0
+
+These are the evidence-commit checks; its message is retained in
+`serials/3f-pr2b/docs-commit-message.txt`. The tree lint checks changed text
+hunks; compressed logs and TSV files are outside its supported extensions.
+The final handoff rechecks that the kernel and ratchet blobs match `cab966e7`
+and re-derives the quoted source lines after the documentation commit.
+
+Git initially normalized CRLF serial bytes while staging the evidence. The
+same evidence commit was amended with a directory-local `.gitattributes`
+(`*.txt -text`), and the committed blob checksums were checked against the
+raw working files and `SHA256SUMS`. This changed no kernel or test source.
+
+claim-lint: python3 scripts/claim-lint.py -> exit 0
+claim-lint: python3 scripts/claim-lint.py --commit-msg .tmp/docs-commit.txt -> exit 0

docs/planning/green-program/aarch64-testing/serials/3f-pr2b/.gitattributes

@@ -0,0 +1,2 @@
+# Preserve captured serial and transcript bytes in Git.
+*.txt -text