Skip to content

Poller efficiency: scope PR conflict scans to PRs with tasks, add REST fallback - #43

Open
bborn wants to merge 2 commits into
mainfrom
task/5205-poller-efficiency-scope-ci-conflict-scan
Open

Poller efficiency: scope PR conflict scans to PRs with tasks, add REST fallback#43
bborn wants to merge 2 commits into
mainfrom
task/5205-poller-efficiency-scope-ci-conflict-scan

Conversation

@bborn

@bborn bborn commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Why

A GM poller exhausted the shared GitHub GraphQL budget and took three boxes down with it.

  • GitHub bills REST and GraphQL from separate buckets (5,000 req/hr vs 5,000 pts/hr), and each bucket is per user, not per token — three boxes holding three different tokens that authenticate as the same user share one budget.
  • The conflict check listed all open PRs including mergeable — a field that makes GitHub compute a merge commit per PR — on a */2 cron, filtered to CONFLICTING, and only then asked whether a TaskYou task mapped to each one. ~95% of that spend was discarded. Result on one box: 140,719 rate-limit errors and a 178MB log.
  • gh api rate_limit lies about GraphQL: it reported graphql 5000/5000 while a real query returned "API rate limit already exceeded".

Note on scope. The runaway lives in agent-poll.mjs on the GM boxes; that file has never existed in this repo (the reference box at 5.161.251.109 was unreachable from this session — SSH timed out, so the fixes are implemented from the task's spec rather than ported line-by-line). This PR lands the corrected logic here as a tested, importable module so future GMs get it by default, plus the guardrails that would have caught it. modules/github/README.md has a Wiring into an existing poller snippet for the boxes still running the old inline block.

What's in it

modules/github/gh-api.mjs — zero-dep helpers with an injectable gh executor:

  • listOpenPRs / findPRForBranch — never request mergeable; fall back to REST only on rate-limit errors. REST spells the branch .head.ref, not .headRefName. A 404, a bad token, or DNS failure still throws, so real breakage stays visible.
  • getMergeableState — single-PR only (REST can't compute mergeability in a list). dirty is REST's CONFLICTING; unknown means GitHub is still computing and is treated as not conflicting — we don't poll waiting for it.
  • scanConflictsForTaskBranchesthe inversion: one cheap PR list, intersected in memory with local ty branches, then mergeability on the survivors only, capped at 25. On 40 open PRs with 2 task branches that's 3 API calls instead of ~40, feature intact.
  • shouldScan / recordScanAttempt — gate on the attempt, not the success. The earlier gate stamped the clock only after a scan succeeded, so a failing scan retried every cycle, reproducing the runaway it was meant to prevent.
  • checkRateLimits — probes GraphQL with a real query{viewer{login}} and believes the probe over the counter.

modules/github/pr-conflict-scan.mjs — CLI over the above. Task branches come from local ty state (zero API); optional CONFLICT_HOOK runs per conflicting PR.

Defaults shipped alongside:

  • Poller crons move to the */10 floor, and are now rewritten rather than skipped, so boxes still carrying a */2 line get fixed on the next setup run.
  • modules/common/rotate-log.sh runs before each poll (one box had 487MB across four unrotated logs).
  • setup.sh now suggests SSH git remotes — git over SSH costs zero API quota.
  • /gm-doctor Check 9: GitHub API Budget — reports both buckets, probing GraphQL for real, and flags crons tighter than */10 and oversized logs.

Verification

Live, against taskyou/taskyou-os with the real gh CLI (GraphQL primary path forced to fail exactly as a burst budget does, to exercise the fallback):

  • GraphQL list and REST fallback returned identical PR lists (identical: true).
  • Mergeability agreed across both paths (MERGEABLE from gh pr view and from .mergeable_state).
  • A real scan over 3 task branches against 2 open PRs made 1 list call + 1 mergeability call, and the gate correctly suppressed the immediate re-run.

Plus 18 unit tests (cd modules/github && node --test, fake executor, no network) covering the cap, the unknown case, non-rate-limit errors not falling back, .head.ref vs .headRefName, and the attempt-based gate — and qa/run-qa.sh: 29 passed, 0 failed.

Follow-ups

  • The live GM boxes still run the old inline block in agent-poll.mjs; they need the wiring snippet applied (and their */2 crons rewritten) — that's server-side work this repo can't do from here.
  • Worth considering: fold agent-poll.mjs itself into this repo so it stops drifting per box.

🤖 Generated with Claude Code

https://claude.ai/code/session_01STtuCwGbnuoj7tSRhkDP3M

bborn and others added 2 commits September 1, 2026 08:05
…T fallback

A GM poller exhausted the shared GitHub GraphQL budget and took three boxes
down with it. GitHub bills REST and GraphQL from separate buckets, and each
bucket is per USER, not per token — every box authenticating as the same user
shares one budget. The poller was listing all 40 open PRs *including* the
`mergeable` field (which forces GitHub to compute a merge commit per PR) on a
*/2 cron, then discarding ~95% of that work because only 1-2 of those PRs had
a TaskYou task. That produced 140,719 rate-limit errors and a 178MB log.

modules/github/gh-api.mjs — budget-aware helpers, injectable `gh` executor:
  - listOpenPRs/findPRForBranch never request `mergeable`, and fall back to
    REST on rate-limit errors only (REST spells the branch `.head.ref`, not
    `.headRefName`). Any other error still throws so real breakage stays visible.
  - getMergeableState is single-PR only; "dirty" is REST's CONFLICTING, and
    "unknown" (GitHub still computing) is NOT conflicting — we don't wait on it.
  - scanConflictsForTaskBranches inverts the scan: intersect one cheap PR list
    with local ty branches, then pay for mergeability on the survivors, capped
    at 25. 40 open PRs + 2 task branches = 3 API calls instead of ~40.
  - shouldScan/recordScanAttempt gate on the ATTEMPT, not the success — an
    earlier gate stamped the clock only after a scan succeeded, so a failing
    scan retried every cycle, reproducing the runaway it was meant to prevent.
  - checkRateLimits probes GraphQL with a real query. `gh api rate_limit`
    reported graphql 5000/5000 while a real query was already rejected.

modules/github/pr-conflict-scan.mjs — CLI over the above, reading task branches
from local ty state (zero API), with an attempt-stamped state file.

Defaults shipped alongside:
  - poller crons move to the */10 floor, and now get REWRITTEN rather than
    skipped, so boxes still carrying a */2 line are fixed on the next setup run
  - modules/common/rotate-log.sh runs before each poll (one box had 487MB
    across four unrotated logs)
  - setup.sh suggests SSH git remotes; git over SSH costs zero API quota
  - /gm-doctor Check 9 reports both buckets, probing GraphQL for real

Verified live against taskyou/taskyou-os with the real gh CLI: the GraphQL and
REST paths returned identical PR lists and mergeability, and a real scan made
1 list call + 1 mergeability call for 3 task branches against 2 open PRs.
18 unit tests (fake gh executor, no network) and qa/run-qa.sh both pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STtuCwGbnuoj7tSRhkDP3M
…ranch

Both sides added a "Check 9" to /gm-doctor. Kept both, in landing order:
main's Claude Auth Health stays Check 9, GitHub API Budget becomes Check 10,
and the summary table lists both rows.

Also extended this branch's log rotation to main's new auth-monitor cron: it
writes to ~/scripts/claude-auth-monitor.log every 30 minutes and nothing was
pruning it, which is the same shape as the 487MB of unrotated poll logs. The
cron line is now rewritten rather than skipped, matching the poller crons, so
boxes provisioned by #42 pick up rotation on the next setup run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STtuCwGbnuoj7tSRhkDP3M
@bborn

bborn commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Post-merge re-verification

origin/main (#42, Claude auth monitor) was merged in as 66cc735. Both sides had added a Check 9 to /gm-doctor; the resolution keeps both in landing order — main's Claude Auth Health stays Check 9, GitHub API Budget becomes Check 10, and the summary table lists both rows. The merge also extends this branch's log rotation to main's new claude-auth-monitor cron, which writes every 30 minutes and had nothing pruning it.

PR is now MERGEABLE / CLEAN, CI qa green.

Everything below was re-run against the live GitHub API on the merged tree, not carried over from the pre-merge run:

Check Result
GraphQL list vs REST fallback (GraphQL forced to fail as an exhausted budget does) identical PR lists — identical: true
REST branch parsing (.head.ref, not .headRefName) branch populated, matched GraphQL
Mergeability via both paths on this PR MERGEABLE from gh pr view and from .mergeable_state
Non-rate-limit error (nonexistent repo) still throws; isRateLimitError = false — not papered over
The inversion, end to end 2 gh calls total: 1 list (no mergeable field) + 1 mergeability on the single intersected candidate
Gate ordering recordScanAttempt + saveState run before any API spend (pr-conflict-scan.mjs:148-151)
Both buckets probed truthfully REST 5000/5000; GraphQL confirmed by a real query{viewer{login}}, not by the counter

modules/github: 18/18 unit tests pass. qa/run-qa.sh: 29 passed, 0 failed. bash -n clean on setup.sh and rotate-log.sh.

Reference box: agents@5.161.251.109 is still unreachable — SSH to port 22 times out (independently re-confirmed this session). The REST fallback here is therefore written from the task spec and validated against live GitHub, rather than diffed line-by-line against that box's agent-poll.mjs. That remains the one unverified-by-comparison item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant