Skip to content

Automate struct offset maintenance: weekly version check + monthly drift check - #166

Open
rlerdorf wants to merge 9 commits into
adsr:masterfrom
rlerdorf:struct-automation
Open

rlerdorf wants to merge 9 commits into
adsr:masterfrom
rlerdorf:struct-automation

Conversation

@rlerdorf

@rlerdorf rlerdorf commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Depends on #165 — this branches from fix-struct-bugs, so its first 3
commits are already under review there. The genuinely new work is the last
6 commits (Automate struct offset maintenance... through Grant the check job write permissions...); everything before that is #165 unchanged.

What this adds

Two scheduled workflows that build on the real-from-source-build approach
dump_structs.yml already uses (rather than a compile-time manifest/
offset-table system), so adding this didn't require inventing new
infrastructure -- just automating what already worked:

  • check_struct_versions.yml (weekly): tools/find_php_updates.sh
    does a cheap git ls-remote check (no PHP build, ~0.5s) for newer tags
    on any PHP series phpspy already tracks, or for PHP 8.6 finally getting
    a real release tag instead of master. Only when it finds something
    does it dump the single changed version on both architectures, patch
    the existing mirror headers via tools/render_struct_update.sh, and
    open a PR -- it never auto-merges.

    A genuinely new PHP series beyond what phpspy supports (e.g. a future
    8.7) is deliberately left alone -- that needs code changes beyond struct
    data (dispatch, -V help text, README), which is a job for a human, not
    this workflow. find_php_updates.sh just notes it on stderr.

  • struct_drift_check.yml (monthly): reuses dump_structs.yml's full
    24-cell matrix (now callable via workflow_call) to re-verify every
    already-pinned version still matches a fresh build, and opens an issue
    (not a PR) if anything unexpectedly drifted.

tools/patch_struct_header.awk is the core piece both workflows share: it
adjusts only a field whose offset moved plus its immediately preceding
pad, using each field's already-absolute offset comment rather than
tracking a cursor across the whole struct -- see its header comment for
why that's sufficient, and what it deliberately refuses to touch (union
blocks, a field with no adjustable preceding pad, or a negative resulting
pad size all get flagged for a human instead of silently patched).

Verification

Before ever running this in real CI:

Then validated end-to-end against real GitHub Actions runs on a fork,
which surfaced four real bugs before this was trustworthy -- all fixed
and re-verified:

  1. actions/checkout@v4's persisted credentials leak an Authorization
    header onto any github.com URL, not just the checked-out repo;
    find_php_updates.sh's git ls-remote against the unrelated
    php/php-src inherited it and got rejected in ~0.3s. Fixed by clearing
    it for that one invocation, and reproduced locally with a bogus header
    to confirm the fix actually addresses it.
  2. A real bug in find_php_updates.sh: under set -e -o pipefail, a
    grep that legitimately finds nothing (e.g. "no php-8.6.* tag exists
    yet") aborts the whole script. My own local testing had missed this --
    wrapping the script in time was silently masking its real exit code.
  3. Missing permissions: contents: write, pull-requests: write on the
    weekly workflow's PR-opening job (the default GITHUB_TOKEN is
    read-only otherwise).
  4. Same class of bug in the monthly workflow's issue-opening job, caught
    by inspection since the test fork has Issues disabled and never
    actually exercised that step.

Real, closeable test artifacts from this: a PR
(closed) that correctly
detected 3 real upstream tag updates (php-8.1.34/8.2.33/8.3.33), correctly
determined none of them change any struct layout (only the pinned tags
needed bumping), and opened a real PR with that diff; and a real
struct_drift_check
run
that
rebuilt all 12 versions on both architectures and correctly found zero
drift.

One thing left genuinely unverified: the "drift found -> open an issue"
path itself, since the test fork has Issues disabled and the real
monthly run found no drift to trigger it. High confidence given it's the
same permissions/gh-CLI pattern as the now-proven PR path, but flagging
it as honest, not silently assumed.

🤖 Generated with Claude Code

rlerdorf and others added 9 commits September 8, 2026 07:12
…truct probe

structs/x86_64/php_structs_86.h: sapi_globals_struct_86.global_request_time
was declared at offset 440; real PHP 8.6.0beta2 has it at 432 (verified via
gdb/offsetof). Before: `-r` reported `# ts = 0.000000` on 8.6. After: a
correct wall-clock timestamp.

phpspy_trace.c/phpspy_trace_tpl.c/structs/structs.h: the CV frame-slot
constant used by --peek-var was hardcoded as the literal 5. The real value
is ZEND_CALL_FRAME_SLOT = ceil(sizeof(zend_execute_data)/sizeof(zval)),
which is 6 on PHP 7.0 (it still carries execute_data.called_scope, removed
in 7.1) and 5 on every later version -- verified against all 12 supported
versions' real headers. Before: --peek-var silently found nothing on 7.0.
After: tests/test_varpeek.sh passes on 7.0; every other version unaffected.

struct_dump.sh/struct_dump.gdb: bring the dump tooling current -- add the
missing php-8.4.25/php-8.5.10 tags (it stopped at 8.3/master), add an
optional single-version argument so a CI matrix cell can dump just one
version, and add sizeof(zend_execute_data)/sizeof(zval) cross-check lines
for the frame-slot fix above.

.github/workflows/dump_structs.yml: new, workflow_dispatch-only job that
builds php-src from source for every supported version on both x86_64 and
aarch64 and uploads the raw struct_dump.gdb output as artifacts -- this is
how real (not placeholder/guessed) aarch64 offsets get produced; see
CLAUDE.md and the phpspy CLAUDE.md for the struct-mirror background.

Verified: full test suite passes unchanged across all 13 locally installed
PHP versions (70-86, excluding asan) before and after this change; the only
failures present (pdo_args_packed_array on 72/73/74/80/81/82) are a
pre-existing, unrelated HASH_FLAG_PACKED bug, confirmed identical via
git stash before committing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8kSnhaw7WUaR61G6cBp1w
Leave version bumps to the maintainer's judgment on when a new version
is warranted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8kSnhaw7WUaR61G6cBp1w
structs/aarch64/php_structs_{70,71,72,73,74,80,81,82}.h were byte-identical
copies of their x86_64 counterparts with a "these structs are wrong for
aarch64" comment -- never actually measured on arm. 83/84/85 carried one
real but unverified guess (a 16-byte shrink in sapi_globals_struct, since
zend_stat_t/struct stat is 128 bytes on aarch64/glibc vs 144 on x86_64/glibc);
86 carried a guess that was wrong given the x86_64 fix in the prior commit.

Derived via the new dump_structs.yml workflow (workflow_dispatch): it builds
php-src from source for every supported version on both ubuntu-24.04 and
ubuntu-24.04-arm and runs the (now current) struct_dump.gdb against each
resulting binary, uploading the raw offsets as build artifacts. Run at
https://github.com/rlerdorf/phpspy/actions/runs/34219729971 (all 24 cells
succeeded).

Before touching any header, the x86_64 leg of that same run was diffed
field-for-field against this session's own independently-gathered x86_64
ground truth (gdb/offsetof against the locally installed PHP 7.0-8.6) --
byte-identical across all 12 versions, confirming the build+dump pipeline
itself is trustworthy. The aarch64 leg then showed a single, consistent
difference from x86_64 for every version: sapi_globals_struct's
global_request_time shifts by exactly -16 bytes, matching the
zend_stat_t size delta above, and nothing else in the struct surface
phpspy reads differs by architecture. That means:

- 83 and 85's existing guessed delta was correct -- confirmed against real
  hardware, no change needed.
- 70-74/80-82 needed the same delta, applied for the first time.
- 86 needed the new delta following the corrected x86_64 offset from the
  prior commit (432 - 16 = 416, not the previously-guessed 424).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8kSnhaw7WUaR61G6cBp1w
Adds two scheduled workflows that build on the dump_structs.yml approach
already proven this session (real from-source builds on both x86_64 and
aarch64, rather than a compile-time manifest/offset-table system):

- check_struct_versions.yml (weekly): tools/find_php_updates.sh does a
  cheap `git ls-remote` check (no PHP build, ~0.5s) for newer tags on any
  PHP series phpspy already tracks, or for PHP 8.6 finally getting a real
  release tag instead of `master`. Only when it finds something does it
  dump the single changed version on both arches, patch the existing
  mirror headers via tools/render_struct_update.sh, and open a PR -- never
  auto-merges. A genuinely new PHP series beyond what phpspy supports is
  deliberately left alone (needs code changes beyond struct data) and only
  noted on stderr.

- struct_drift_check.yml (monthly): reuses dump_structs.yml's full 24-cell
  matrix (now callable via workflow_call) to re-verify every already-
  pinned version still matches a fresh build, and opens an issue (not a
  PR) if anything unexpectedly drifted.

tools/patch_struct_header.awk is the core piece: it adjusts only a field
whose offset moved plus its immediately preceding pad, using each field's
already-absolute offset comment rather than tracking a cursor -- see its
header comment for why that's sufficient and what it deliberately refuses
to touch (union blocks, fields with no adjustable preceding pad, or a
negative resulting pad size all get flagged for a human instead).

Verified before committing:
- patch_struct_header.awk reproduces both real fixes from the prior
  commits (x86_64/86, aarch64/86) and all 8 aarch64 placeholder
  replacements byte-for-byte against the actual committed headers.
- Idempotent across all 24 current header/arch combinations (already-
  correct data produces zero diff).
- All four refusal paths (no preceding pad, union field, negative pad,
  exact match) verified against synthetic fixtures.
- find_php_updates.sh run against the real php-src remote.
- The full monthly drift-check loop simulated locally against all 24 real
  CI dump artifacts from this session's earlier dump_structs.yml run:
  zero drift on current data, and correctly detects+reports drift when
  fed the known-broken pre-fix 8.6 header.
- dump_structs.yml's matrix now reads tools/list_pinned_tags.sh instead of
  a second hardcoded copy of the version list, so the two can no longer
  drift apart from each other.

Not yet verified: the actual GitHub Actions machinery itself (workflow_call
chaining, matrix-from-dynamic-JSON, gh pr/issue create) -- only simulated
locally so far.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8kSnhaw7WUaR61G6cBp1w
…d repo

actions/checkout@v4 (with the default persist-credentials: true) sets
http.https://github.com/.extraheader in the checked-out repo's local git
config, scoped to that repo's own token. Since git matches extraheader
config by URL prefix rather than by remote name, any git command run with
CWD inside that checkout -- including this script's `git ls-remote` against
an unrelated public repo (php/php-src) -- inherits that header too, and
GitHub rejects it outright ("Invalid username or token"), failing in
~0.3s rather than actually attempting the read.

Reproduced locally by setting a bogus extraheader and confirming
`git ls-remote` against php-src fails the same way in the same ~0.3s;
`-c http.https://github.com/.extraheader=` clears it for just this
invocation and resolves it. This is also what caused
check_struct_versions.yml's first real test run to fail immediately in
CI (https://github.com/rlerdorf/phpspy/actions/runs/34226820940).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8kSnhaw7WUaR61G6cBp1w
…the script

Real bug, not just a debug leftover: with `set -e -o pipefail`, each
`x=\$(... | grep ... | ...)` assignment aborts the whole script the moment
grep finds zero matches, even though "nothing matched" is often the
correct, expected result here -- e.g. there is currently no php-8.6.*
release tag at all, and that's not a failure, just nothing to report.

Caught by actually running this in CI (the previous debug commit's `set -x`
trace), which is what exposed that my own earlier local testing sessions
were fooled: wrapping the script in `time ...` was masking its real exit
code, so "looks fine locally" was never a valid signal for this bug.
Direct invocation without the `time` wrapper now correctly shows the
failure before this fix and success after.

Also reverts the prior commit's temporary `set -x` debug tracing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8kSnhaw7WUaR61G6cBp1w
The default GITHUB_TOKEN is read-only unless a job explicitly requests
more, which the render job needs for both the git push and gh pr create
at the end. Confirmed everything upstream of this works correctly first:
detect found 3 real tag updates (php-8.1.34/8.2.33/8.3.33), all 6 dump
cells succeeded, and render correctly determined these patch releases
don't change any struct layout -- only struct_dump.sh's pinned tags needed
updating. Only the final push/PR step failed, with exactly the 403 this
fixes (https://github.com/rlerdorf/phpspy/actions/runs/34227369760).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8kSnhaw7WUaR61G6cBp1w
Same class of bug as the render job's earlier 403: the default
GITHUB_TOKEN is read-only unless a job explicitly requests more, and this
job's final step (gh issue create on drift) needs issues: write. Caught by
inspection while validating the workflow end-to-end -- this fork has
Issues disabled entirely, so the real run found zero drift (correctly)
and never reached that step, meaning the missing permission would have
failed silently on the next repo where it actually triggers. Not yet
verified against a real "drift found" run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8kSnhaw7WUaR61G6cBp1w
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant