fix(ci): repair the Node 24 release breakage and close every CI/CD false negative - #169
Conversation
Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: #168
Node >= 22.12 adds a 'module.exports' named export to the import() namespace of a CommonJS module whose exports cjs-module-lexer cannot infer statically. command-stream's src/$.cjs is such a module, so on Node 24 use-m sees two keys, skips its default-unwrap and returns the raw namespace. Every release script then destructured undefined and died with '$ is not a function', which is what failed the release jobs of runs 33734680882/33734680890, 33740815350/33740815303 and 33746569769/33746569750 on main. scripts/load-command-stream.mjs normalises all observed namespace shapes and throws an actionable error naming the received keys instead. Verbose output is gated behind START_DEBUG and off by default.
The pipeline never linted its own workflows, so two actionlint findings and 27 zizmor high-severity findings were false negatives (issue #168). - add .github/workflows/workflows.yml (actionlint 1.7.7 via Docker so the shellcheck/pyflakes integration runs, plus zizmor-action), .github/zizmor.yml and .github/actionlint.yaml, matching the CI/CD templates - fix 14 template-injection findings: github.base_ref, github.head_ref and github.event.inputs.* now reach run: blocks through env vars - hash-pin the three third-party actions (oven-sh/setup-bun, dtolnay/rust-toolchain, peter-evans/create-pull-request v7 -> v8.1.1); the rust-toolchain pin keeps 'toolchain: stable' explicit because the hash hides the branch default - add persist-credentials: false to all 24 read-only checkouts, with a comment on the 5 writer checkouts explaining why they keep credentials
The Security workflow ran CodeQL, dependency-review and secret scanning but never audited the resolved dependency graphs, so two high-severity advisories sat undetected in js/package-lock.json (issue #168, false negatives). - new cargo-audit job (taiki-e/install-action, cargo-audit@0.22.2) - new npm-audit job (npm audit --package-lock-only --audit-level=high) - both wired into the pipeline-status gate - bump brace-expansion/flatted/js-yaml via overrides; audit is now clean
js/eslint.config.mjs has js/ as its ESLint base path, so `eslint .` run from js/ silently skipped scripts/ - the release automation, i.e. exactly the code that broke in runs 33746569750/33746569769 (issue #168, false negative). - root eslint.config.mjs re-exports the js/ rules with the repo as base path - root .prettierrc/.prettierignore for the same reason - js/package.json gains lint:scripts + format:check:scripts, both in `check` - js.yml lint job runs the two new checks - 3 real defects surfaced: undefined AbortController/setTimeout globals and an unnecessary escape in verify-release-badge.mjs; plus formatting fixes
Adds the raw material the analysis is based on: full logs of the two failing runs and of the two green workflows on the same commit, per-job JSON, the earlier failing pushes of the same day, the run history of main, snapshots of the three CI/CD templates and of the hive-mind best-practices document, and actionlint/zizmor output before and after this PR. ANALYSIS.md reconstructs the timeline, lists requirements R1-R11 from the issue, gives the root cause of each problem and the fix applied, records the two candidate 'false positives' that were deliberately left alone, and surveys existing tooling.
Pins RC-1 to the precise source location: baseUse's metadataKeys allow-list in use-m does not include Node's synthetic 'module.exports' CommonJS-namespace marker, so the callable default is never unwrapped on Node >= 22.12. Snapshots the published bundle the failing runs executed, since the CDN URL is unpinned. Upstream reports (R7): - use-m#72 already open; added this incident, the reproduction, the source location and a one-line diff - js template#151 had been closed by a PR that changed only .gitkeep while all 8 call sites remained; added the evidence and reopened it - rust template#147 and python template#64 filed: workflows.yml runs actionlint but not zizmor, so unpinned actions and persisted credentials go undetected
Each of the gaps found in issue #168 was invisible to the pipeline; without a structural test they can silently return. Adds six mirrored invariants to both suites (js/test/ci-workflow-invariants.js, rust/tests/ci_workflow_invariants.rs): - workflows.yml exists and runs actionlint and zizmor, with both configs present - no untrusted github context interpolated inside a run: block - persist-credentials: false on every checkout in a non-writer job - third-party actions pinned to a 40-char commit hash - security.yml has cargo-audit and npm-audit jobs - js.yml lints and format-checks the repository-level scripts/ directory Each new assertion was mutation-tested: reverting a pin, dropping a persist-credentials line and pasting github.head_ref into a run: block each fail the expected test in both suites.
First CI run of the new Workflows job surfaced two things a local run had not: - The rhysd/actionlint Docker image bundles shellcheck; a bare local binary does not, so the first local run reported 0 findings. Reproduced locally with shellcheck on PATH and fixed all 7: 4x SC2086 (unquoted $GITHUB_PATH and $GITHUB_OUTPUT redirections) and 2x SC2126 (grep | wc -l -> grep -c). - zizmor-action defaults to inputs: '.', so it audited the template snapshots archived under dev/log/ - other projects' workflows - and failed the build on their findings. Scoped to .github/workflows; those findings are reported in the templates' own repositories instead.
…fragment The 1000-line refactoring limit fired on the third-party use-m snapshot archived under dev/log/ as investigation evidence, failing the lint job of both pipelines. That file cannot be refactored without destroying the evidence, so dev/log now sits alongside node_modules and target in the checker's exclusion list - the same boundary eslint.config.mjs and .prettierignore already draw. Adds mirrored tests (js/test/check-file-size.js, rust/tests/check_file_size.rs) covering both directions, and the rust/changelog.d fragment the changelog job requires for the new Rust test file.
The dev/log exemption matched a forward-slash pattern against relative(), which yields dev\\log on Windows, so the Bun-on-windows-latest leg failed while Linux passed. Paths are now normalised to forward slashes before both matching and reporting, and both suites assert it.
Working session summaryJavaScript, Rust, Workflows and Broken Link Checker are all green on This summary was automatically extracted from the AI working session output. |
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $22.220226📊 Context and tokens usage:Claude Opus 5: (4 sub-sessions)
Total: (8.2K new + 390.4K cache writes + 27.7M cache reads) input tokens, 177.7K output tokens, $22.220227 cost 🤖 Models used:
📎 Log file uploaded as Gist (8661KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
🔄 Auto-restart 1/5Reason: CI failures detected Starting new session to address the issues. Auto-restart-until-mergeable mode is active. This run will stop after 5 restart iterations in total. |
… findings CodeQL failed pull request #169 with two high-severity alerts. 1. js/redos in dev/log/.../upstream/use-m-8.15.1-use.js - the verbatim third-party bundle archived as incident evidence. It is not our code, is never executed here and cannot be edited without destroying the evidence, so .github/codeql/codeql-config.yml excludes dev/log from the analysis, the same boundary eslint.config.mjs, .prettierignore and scripts/check-file-size.mjs already draw. 2. js/incomplete-url-substring-sanitization in scripts/format-release-notes.mjs: 'currentBody.includes("img.shields.io")' also matches a URL that merely mentions the host in its path or query. containsPackageVersionBadge() now parses each markdown image target and compares the host exactly. Mirrored invariant tests (js/test/ci-workflow-invariants.js, rust/tests/ci_workflow_invariants.rs) fail if the CodeQL job stops passing a config-file or the config stops excluding dev/log; both were mutation tested. 5 new badge-detection tests cover the URLs the substring check used to accept.
CodeQL js/insecure-randomness (alerts #2 and #40): the isolated-user suffix and the UUID fallback came from Math.random, so the only unguessable part of a session id or of a system account name was predictable. The Rust mirror had the same defect through a time-seeded xorshift that no CodeQL query covers - a false negative found by comparing the two suites (issue #168). - js/src/lib/args-parser.js: crypto.randomUUID, falling back to a crypto.randomBytes-formatted v4 UUID instead of Math.random. - js/src/lib/user-manager.js: randomBase36() over crypto.randomInt. - rust/src/lib/user_manager.rs: random_base36() over uuid::Uuid::new_v4 (getrandom), with rejection sampling so the base36 mapping stays uniform; simple_random() removed. Mirrored regression tests: the JS one pins Math.random and fails on the old implementation (mutation tested); the Rust one checks the same spread and that the hand-rolled generator is gone.
Every external command that interpolated data into a shell string is now spawned with an argument vector, in both runtimes: - failure-handler: which/npm/gh/gh-upload-log ran through a shell with the failing command's own name and the log path interpolated into the string (CodeQL js/shell-command-injection-from-environment). - createIssue quoted the title and body by escaping `"` only, so a backtick or $(...) in the failing command was executed by the shell, and the body's newlines reached GitHub as the two characters "\n". The Rust mirror never used a shell, so its escaping only corrupted the issue text; it is gone. - ExecutionStore.execClink wrapped the query in single quotes, which a `'` in any recorded value escaped from (CodeQL js/shell-command-constructed-from-input). - version-and-commit escaped quotes into a command-stream template, which already quotes interpolated values. - Host checks now parse the URL (or rely on parse_git_url, which anchors on the github.com host) instead of testing for a substring (CodeQL js/incomplete-url-substring-sanitization). - experiments: read the log with fs instead of `cat`, and escape backslashes before quotes (CodeQL js/unnecessary-use-of-cat, js/incomplete-sanitization). Regression tests in both runtimes drive the real code path through a fake `gh`/`clink` on PATH and assert on the recorded argv; each was verified to fail against the previous implementation. Refs #168
Replacing PATH in-process also redirected the `gh` calls of the tests running beside it in the same binary (cargo runs them as threads), which made can_create_issue see the fake binary and clobbered the recorded argv; and the cfg(unix) module left its imports unused on Windows, which -Dwarnings rejected. The fake PATH is now set only on the re-executed helper process.
CodeQL rust/cleartext-logging #43 (high) was raised on a line this pull request added, so it gated the CodeQL check-run. Every one of these assertions is about the shape of the generated username, session name or UUID, not its value, so the failure message now states the expectation and omits the value. The four pre-existing sibling alerts in the same two test files get the same treatment.
Working session summaryThe blocking alert is fixed and PR #169 is green and out of draft. Root cause of the last failure: the GHAS aggregate Fix ( Also updated: Verification: This summary was automatically extracted from the AI working session output. |
🔄 Auto-restart-until-mergeable Log 1/5This log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $12.330716📊 Context and tokens usage:Claude Opus 5: (3 sub-sessions)
Total: (5.0K new + 249.0K cache writes + 14.4M cache reads) input tokens, 103.9K output tokens, $12.330716 cost 🤖 Models used:
📎 Log file uploaded as Gist (13484KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
🎉 Auto-mergedThis pull request has been automatically merged by hive-mind.
Auto-merged by hive-mind with --auto-merge flag |
Fixes #168.
What was broken
Both release pipelines on
mainfailed at 2026-09-03 08:41 UTC and stayed red for three consecutive pushes (Rust 33746569750, JavaScript 33746569769), with:Nothing in the repository caused it. The last green run was 2026-08-10; the failing commit differs only in docs and unrelated Rust code, and the same commit passed Security and Broken Link Checker. The release scripts load
use-mfrom the unpinnedhttps://unpkg.com/use-m/use.js, so a newuse-mreached the runner on its own.Root cause.
use-m'sbaseUseunwraps a CommonJS default only when theimport()namespace has no key outside a hard-codedmetadataKeysallow-list. Node ≥ 22.12 adds a synthetic'module.exports'export to CommonJS namespaces it cannot statically analyse — whichcommand-stream'ssrc/$.cjsis — and'module.exports'is not in that set. Soawait use('command-stream')returns the raw namespace andconst { $ } = …destructuresundefined. Reproduced on the same tree:The release jobs run on Node
24.x.How to reproduce
node experiments/issue-168-use-m-cjs-interop.mjs # on Node 20 vs Node 24The fix
scripts/load-command-stream.mjsresolves either namespace shape — probingloaded,loaded.default,loaded['module.exports'],loaded.default.default— and when none carries a callable member throws an error naming the keys it actually observed, instead of the opaque$ is not a function. All 8 affected scripts use it:Debug output is gated behind the repository's existing
START_DEBUG/RUNNER_DEBUG/ACTIONS_STEP_DEBUGconvention and is off by default.Regression test:
js/test/load-command-stream.mjs, 8 tests, including the exact Node-24 namespace shape that caused the outage.False negatives found while auditing the rest of the pipeline
The issue asks for all false positives, false negatives, warnings and errors. Four gaps let real defects through:
analysis/zizmor-before.txt).github/workflows/workflows.ymlrunning both, plus.github/zizmor.ymland.github/actionlint.yaml. 9 template-injection sites moved toenv:;oven-sh/setup-bun,peter-evans/create-pull-requestanddtolnay/rust-toolchainhash-pinned;persist-credentials: falseon all 24 read-only checkouts. Now 0 and 0.js/package-lock.jsoncargo-auditandnpm-auditjobs insecurity.yml; advisories patched viaoverrides. Both clean.scripts/was excluded from ESLint and Prettier —js/eslint.config.mjshasjs/as its base path, soeslint .run fromjs/silently skipped the release automation, i.e. exactly the code that broke.eslint.config.mjs/.prettierrcre-exporting thejs/rules with the repo as base path, wired intobun run checkand intojs.yml. 3 real defects surfaced: undefinedAbortController(check-web-archive.mjs:92) andsetTimeout(publish-to-crates.mjs:139), and an invalid regex escape inverify-release-badge.mjs:74.js/test/ci-workflow-invariants.jsandrust/tests/ci_workflow_invariants.rs. Each was mutation-tested: reverting a pin, dropping apersist-credentialsline and pastinggithub.head_refinto arun:block each fail the expected test in both suites.Two further false negatives showed up in the verification itself, once the new job ran on a real runner:
rhysd/actionlintDocker image bundles shellcheck; a bare local binary does not, so the first local run reported 0. With shellcheck onPATHall 7 findings reproduced and are fixed: 4 × SC2086 (unquoted$GITHUB_PATH/$GITHUB_OUTPUTredirections) and 2 × SC2126 (grep … | wc -l→grep -c).zizmor-actiondefaults toinputs: ., so it audited the template snapshots archived underdev/log/— other projects' workflows — and failed the build on their findings. It is now scoped to.github/workflows; those findings are reported in the templates' own repositories instead.A false positive from this repository's own tooling
scripts/check-file-size.mjsenforces a 1000-line refactoring limit across the whole tree, so the archived third-party snapshotdev/log/…/upstream/use-m-8.15.1-use.js(1575 lines) failed thelintjob of both pipelines. The file cannot be refactored without destroying the evidence it preserves, sodev/logjoinsnode_modules,targetand friends in the exclusion list — the same boundaryeslint.config.mjsand.prettierignorealready draw. The first version of that exemption was itself platform-dependent (relative()yieldsdev\logon Windows, so the Bun-on-windows-latestleg failed while Linux passed); paths are now normalised to forward slashes before both matching and reporting. Covered by 5 mirrored tests injs/test/check-file-size.jsandrust/tests/check_file_size.rs.Other than that, no false positives. Two candidates were examined and deliberately left alone —
pipeline-statuskeepsif: always()(atimeout-minuteskill reports ascancelled, and the gate must still fail the run; the repo's own invariants encode this on purpose), and the localnpm auditENOLOCKis a bun-node_modulesartefact that does not occur on a fresh CI checkout. Both are documented in the analysis.The failing
CodeQLcheckThe check reported as failing is the aggregate
CodeQLcheck-run from GitHub Advanced Security, not the per-languageCodeQL (…)workflow jobs — those were green throughout, which is why nothing about it appears in the workflow logs. That check-run fails a PR when code scanning reports a new alert in code the PR changed. There were two:js/redosdev/log/…/upstream/use-m-8.15.1-use.jsdev/log/archives third-party evidence verbatim, and the vendoreduse-mbundle contains a polynomial-backtracking regex. Editing archived evidence would destroy the evidence; the finding belongs upstream..github/codeql/codeql-config.ymlwithpaths-ignore: dev/log, wired in viaconfig-file:— the same boundaryeslint.config.mjs,.prettierignoreandcheck-file-size.mjsalready draw. Guarded by a new mirrored invariant test in both suites.js/incomplete-url-substring-sanitizationscripts/format-release-notes.mjscurrentBody.includes('img.shields.io')also matcheshttps://example.invalid/img.shields.ioand?u=img.shields.io, so release notes could be mistaken for already-formatted.containsPackageVersionBadge()parses every markdown image target withnew URL()and compares the host exactly. 5 unit tests, including all three look-alike URLs.…and the 23 pre-existing alerts
The issue asks for all problems, so the rest of the backlog was triaged too. Each alert had a real defect behind it, and each fix was applied in both runtimes:
js/insecure-randomness(2). Session ids and isolation usernames came fromMath.random(). Nowcrypto.randomUUID/randomBytes(16)/randomInt(36). The Rust mirror had the same defect as a false negative — a time-seeded xorshift that no CodeQL query flags, and that returns identical suffixes for processes started in the same millisecond — replaced byUuid::new_v4()bytes with rejection sampling.js/shell-command-injection-from-environment/js/shell-command-constructed-from-input(6).failure-handlerranwhich,npm view,ghandgh-upload-logthrough a shell with the failing command's own name and the log path interpolated in, andexecClinkbuiltclink '<query>' --db …, which a single quote in any recorded command escaped from. All now spawn with an argument vector.js/incomplete-sanitization(4).createIssueescaped only"before interpolating into agh issue createshell string — so a backtick or$(…)in the failing command was executed, andbody.replace(/\n/g,'\\n')sent every newline to GitHub as the two characters\n. The Rust mirror never used a shell, so its identical escaping only corrupted the issue text. Both gone. Also a redundant escape into a command-stream template (which quotes interpolations itself) and an escape-the-quote-but-not-the-backslash in an experiment.js/incomplete-url-substring-sanitization(2),js/unnecessary-use-of-cat(1).bugsUrl.includes('github.com')dropped in both runtimes (parseGitUrl/parse_git_urlanchor on the host themselves), a test assertion now compares the whole URL, andcatreplaced byfs.readFileSync.New regression tests drive the real code paths through a fake
gh/clinkonPATHand assert on the recorded argv; every one was mutation-verified against the previous implementation.rust/cleartext-logging— 5 fixed, 5 false positives. Alert #43 (rust/tests/user_manager.rs:92, high) was raised on a line this PR added, so it gated theCodeQLcheck-run: the assertion interpolated a generated username into its failure message. Every one of these test assertions is about the shape of the generated name or UUID, never its value, so the messages now state the expectation and omit the value; the four pre-existing sibling alerts in the same two test files (#18-#21) got the same treatment. The remaining five, all insrc/, flag printingexecution_record.uuidand generated isolation usernames. Those are the handles a user needs to--attach,--resumeor--statusa run - printing them is the feature, and nothing authenticates on them. They are deliberately not suppressed in code: renaming variables to dodge the heuristic would be gaming it, and excluding the rule incodeql-config.ymlwould also hide future genuine findings. The right mechanism is a per-alert "Dismiss -> False positive" in the Security tab, which changes repository state rather than this PR, so it is left for a maintainer. None of them gate this PR.Upstream reports
link-foundation/use-m'module.exports'tometadataKeysjs-…-template.gitkeepwhile all 8 call sites remained — commented with the evidence and reopenedrust-…-templatedtolnay/rust-toolchain@stablepython-…-templatepersist-credentials;pypa/gh-action-pypi-publish@release/v1on a mutable branch in the publish jobTemplate comparison
Full trees of all three templates are snapshotted in the evidence folder. Adopted from them:
workflows.yml,.github/zizmor.yml,.github/actionlint.yaml, hash-pinning,persist-credentials: false, dependency-audit jobs. Already present and matching the hive-mind best practices: per-job concurrency (cancellablecheck-*vs non-cancellablemain-writer-…), explicitpermissions:,timeout-minuteseverywhere, and thepipeline-statusaggregate gate. Gaps in the templates worth upstreaming the other way: no test-count parity gate, no file-size gate, no workflow-invariant tests.Evidence and analysis
dev/log/issues/168/pulls/169/— full logs of both failing runs and of the two green workflows on the same commit, per-job JSON, the earlier failing pushes, the run history ofmain, template snapshots, actionlint/zizmor before/after, and theuse-mbundle the failing runs executed.ANALYSIS.mdreconstructs the timeline, lists requirements R1–R11, and gives the root cause and fix for each.Verification
bun run check(lint + format, bothjs/andscripts/) — passactionlint(with shellcheck onPATH) — 0 findings;zizmor --min-confidence medium— 0 findingscargo fmt --checkandcargo clippy --all-targets -- -D warnings— cleancargo audit(71 crates) andnpm audit --audit-level=high— 0 vulnerabilities