Poller efficiency: scope PR conflict scans to PRs with tasks, add REST fallback - #43
Poller efficiency: scope PR conflict scans to PRs with tasks, add REST fallback#43bborn wants to merge 2 commits into
Conversation
…T fallback
A GM poller exhausted the shared GitHub GraphQL budget and took three boxes
down with it. GitHub bills REST and GraphQL from separate buckets, and each
bucket is per USER, not per token — every box authenticating as the same user
shares one budget. The poller was listing all 40 open PRs *including* the
`mergeable` field (which forces GitHub to compute a merge commit per PR) on a
*/2 cron, then discarding ~95% of that work because only 1-2 of those PRs had
a TaskYou task. That produced 140,719 rate-limit errors and a 178MB log.
modules/github/gh-api.mjs — budget-aware helpers, injectable `gh` executor:
- listOpenPRs/findPRForBranch never request `mergeable`, and fall back to
REST on rate-limit errors only (REST spells the branch `.head.ref`, not
`.headRefName`). Any other error still throws so real breakage stays visible.
- getMergeableState is single-PR only; "dirty" is REST's CONFLICTING, and
"unknown" (GitHub still computing) is NOT conflicting — we don't wait on it.
- scanConflictsForTaskBranches inverts the scan: intersect one cheap PR list
with local ty branches, then pay for mergeability on the survivors, capped
at 25. 40 open PRs + 2 task branches = 3 API calls instead of ~40.
- shouldScan/recordScanAttempt gate on the ATTEMPT, not the success — an
earlier gate stamped the clock only after a scan succeeded, so a failing
scan retried every cycle, reproducing the runaway it was meant to prevent.
- checkRateLimits probes GraphQL with a real query. `gh api rate_limit`
reported graphql 5000/5000 while a real query was already rejected.
modules/github/pr-conflict-scan.mjs — CLI over the above, reading task branches
from local ty state (zero API), with an attempt-stamped state file.
Defaults shipped alongside:
- poller crons move to the */10 floor, and now get REWRITTEN rather than
skipped, so boxes still carrying a */2 line are fixed on the next setup run
- modules/common/rotate-log.sh runs before each poll (one box had 487MB
across four unrotated logs)
- setup.sh suggests SSH git remotes; git over SSH costs zero API quota
- /gm-doctor Check 9 reports both buckets, probing GraphQL for real
Verified live against taskyou/taskyou-os with the real gh CLI: the GraphQL and
REST paths returned identical PR lists and mergeability, and a real scan made
1 list call + 1 mergeability call for 3 task branches against 2 open PRs.
18 unit tests (fake gh executor, no network) and qa/run-qa.sh both pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STtuCwGbnuoj7tSRhkDP3M
…ranch Both sides added a "Check 9" to /gm-doctor. Kept both, in landing order: main's Claude Auth Health stays Check 9, GitHub API Budget becomes Check 10, and the summary table lists both rows. Also extended this branch's log rotation to main's new auth-monitor cron: it writes to ~/scripts/claude-auth-monitor.log every 30 minutes and nothing was pruning it, which is the same shape as the 487MB of unrotated poll logs. The cron line is now rewritten rather than skipped, matching the poller crons, so boxes provisioned by #42 pick up rotation on the next setup run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STtuCwGbnuoj7tSRhkDP3M
Post-merge re-verification
PR is now Everything below was re-run against the live GitHub API on the merged tree, not carried over from the pre-merge run:
Reference box: |
Why
A GM poller exhausted the shared GitHub GraphQL budget and took three boxes down with it.
mergeable— a field that makes GitHub compute a merge commit per PR — on a*/2cron, filtered to CONFLICTING, and only then asked whether a TaskYou task mapped to each one. ~95% of that spend was discarded. Result on one box: 140,719 rate-limit errors and a 178MB log.gh api rate_limitlies about GraphQL: it reportedgraphql 5000/5000while a real query returned "API rate limit already exceeded".What's in it
modules/github/gh-api.mjs— zero-dep helpers with an injectableghexecutor:listOpenPRs/findPRForBranch— never requestmergeable; fall back to REST only on rate-limit errors. REST spells the branch.head.ref, not.headRefName. A 404, a bad token, or DNS failure still throws, so real breakage stays visible.getMergeableState— single-PR only (REST can't compute mergeability in a list).dirtyis REST'sCONFLICTING;unknownmeans GitHub is still computing and is treated as not conflicting — we don't poll waiting for it.scanConflictsForTaskBranches— the inversion: one cheap PR list, intersected in memory with local ty branches, then mergeability on the survivors only, capped at 25. On 40 open PRs with 2 task branches that's 3 API calls instead of ~40, feature intact.shouldScan/recordScanAttempt— gate on the attempt, not the success. The earlier gate stamped the clock only after a scan succeeded, so a failing scan retried every cycle, reproducing the runaway it was meant to prevent.checkRateLimits— probes GraphQL with a realquery{viewer{login}}and believes the probe over the counter.modules/github/pr-conflict-scan.mjs— CLI over the above. Task branches come from localtystate (zero API); optionalCONFLICT_HOOKruns per conflicting PR.Defaults shipped alongside:
*/10floor, and are now rewritten rather than skipped, so boxes still carrying a*/2line get fixed on the next setup run.modules/common/rotate-log.shruns before each poll (one box had 487MB across four unrotated logs).setup.shnow suggests SSH git remotes — git over SSH costs zero API quota./gm-doctorCheck 9: GitHub API Budget — reports both buckets, probing GraphQL for real, and flags crons tighter than*/10and oversized logs.Verification
Live, against
taskyou/taskyou-oswith the realghCLI (GraphQL primary path forced to fail exactly as a burst budget does, to exercise the fallback):identical: true).MERGEABLEfromgh pr viewand from.mergeable_state).Plus 18 unit tests (
cd modules/github && node --test, fake executor, no network) covering the cap, theunknowncase, non-rate-limit errors not falling back,.head.refvs.headRefName, and the attempt-based gate — andqa/run-qa.sh: 29 passed, 0 failed.Follow-ups
agent-poll.mjs; they need the wiring snippet applied (and their*/2crons rewritten) — that's server-side work this repo can't do from here.agent-poll.mjsitself into this repo so it stops drifting per box.🤖 Generated with Claude Code
https://claude.ai/code/session_01STtuCwGbnuoj7tSRhkDP3M