Skip to content

Preserve documentation and hook string semantics - #262

Open
sirreal wants to merge 15 commits into
masterfrom
fix-254
Open

Preserve documentation and hook string semantics#262
sirreal wants to merge 15 commits into
masterfrom
fix-254

Conversation

@sirreal

@sirreal sirreal commented Jul 23, 2026

Copy link
Copy Markdown
Member

Fixes #254.

What changed

  • Stop applying namespace cleanup to every exported string.
  • Normalize AST names at expression and metadata boundaries, preserving established JSON for defaults, constants, references, and call metadata.
  • Export literal hook names with PHP string semantics.

Why

The PHP-Parser 5 compatibility cleanup treated documentation and hook literals as identifiers. It removed meaningful backslashes. Simply deleting that cleanup also caused broad JSON churn such as false becoming \false.

Keep syntax normalization where it belongs. Leave text alone.

Side effects

  • Printing single-segment names without the leading backslash restores Method_Call_Reflector's factory-function class mapping, which had been dead since the PHP-Parser upgrade: get_current_screen()->add_help_tab() now exports uses.methods[].class as WP_Screen (likewise wp_get_theme()WP_Theme, _get_list_table()WP_List_Table, and the other mapped factories). Pinned by test.

Testing

  • npm run test:phpunit — 28 tests, 158 assertions; 11 pre-existing PHPUnit deprecation warnings.
  • php tests/prep-diff-test.php
  • Full wp-includes export: expected documentation escapes remain; no stray global prefixes in defaults, constant values, references, or call-class metadata.

Update: documentation text is now exported as authored

The final commits settle the remaining question from review: what happens to a backslash that an author actually typed inside documentation prose.

Documentation text fields are now exported byte-for-byte as authored, backslashes included:

  • description and long_description
  • tag content, description, refers, and link

So a docblock that says {@see \parse_blocks()} or @see \json_last_error() exports with the leading \ intact, instead of having it silently removed. Previously the export stripped it, which corrupted references to global functions in namespaced code and made the exported text differ from the source.

Deliberate deviation: type fields keep the global-prefix strip

Type fields are not treated as documentation text. Tag types and argument type still have the global namespace prefix stripped.

This is intentional, not an oversight. The vendored phpDocumentor (Type\Collection::expand() in reflection-docblock) prefixes every non-keyword type with \ while resolving a docblock. By the time the parser sees a type, the author's original spelling is gone: @param Foo and @param \Foo both arrive as \Foo. There is no information left to preserve. Dropping the strip would not restore author intent — it would stamp a synthetic \ onto essentially every typed parameter in the corpus, churning the JSON for no gain.

A test pins this boundary: an author writing @param \Global_Prefixed_Type still exports Global_Prefixed_Type.

The governing principle: documentation text is preserved as authored; resolver output is normalized.

Corpus verification

Regenerated the full export over WordPress 7.0.4 wp-includes (1039 PHP files) and diffed against the merge base (66f26b0). Both sides used identical vendor trees (composer.lock is unchanged by this PR; all 59 packages byte-identical) and the same PHP engine, normalized with the corrected prep-diff from #279.

Whole PR vs. merge base — 102 hunks, 109 changed values, every one accounted for:

Count Category
28 Documentation text gains an author-written backslash — {@see \parse_blocks()}, refers: \json_last_error(), {@see \mb_convert_encoding()}, \utf8_decode()
72 Expression strings drop a synthetic backslash — \truetrue, \nullnull, \compact(compact(, \COOKIEHASHCOOKIEHASH, \PHP_INT_MAXPHP_INT_MAX
9 uses.methods[].class factory mapping restored — wp_get_theme()WP_Theme, get_current_screen()WP_Screen
0 Unclassified

The 72 expression changes break down as hook arguments (26), uses.methods[].class call expressions (27), property defaults (15), and constant values (4). These, and the 9 factory mappings, are this PR's stated purpose — see "What changed" and "Side effects" above.

Isolating the final as-authored commit (0b4d163..edac50a) — 20 hunks, 22 changed values, 100% documentation text. No hook field, expression, type, uses, or extends value moves.

Zero changes corpus-wide to type, types, extends, implements, aliases, or hook name — confirming the types deviation above holds in practice: no synthetic \ is stamped onto any type in 1039 files.

Warning

This PR has no automated corpus verification. The numbers above are the only corpus evidence — please review them rather than looking for a green check.

The "Corpus Diff" CI workflow does not exist on this branch; it is still unmerged in #284. And once it does land, running it against fix-254 as-is would under-report these changes anyway: this branch's prep-diff.php normalizes documentation text away, so the doc-text changes that are the point of this PR would be silently cancelled out on both sides of the diff and reported as ~0 hunks. #279 fixes exactly that, which is why the numbers above were produced with #279's prep-diff instead.

Testing (updated)

  • Full PHPUnit suite on PHP 7.4: 167 tests, 463 assertions, all passing. (The three Error processing file lines in the output are assertions from the Blueprint error-handling fixtures, not failures.)
  • CI green on PHP 7.4 and 8.4.

sirreal added 13 commits August 14, 2026 11:44
`Hook_Reflector::getName()` short-circuited on `Scalar\String_` nodes and
returned the interpreted string value. Escape sequences were therefore
resolved, so `do_action( "\x09tab" )` exported a literal tab and
`do_action( "\xC0 bad" )` exported a raw 0xC0 byte. That byte is not valid
UTF-8, `json_encode()` returns `false` for it, and `wp parser export`
silently produced no JSON at all for the whole run.

Route string nodes through `Pretty_Printer` like every other expression.
The printer returns php-parser's `rawValue` attribute, which is the
source-verbatim spelling, and `cleanupName()` strips the quotes.

Also check `json_encode()` for failure in `Command::_get_phpdoc_data()` and
fail loudly with `json_last_error_msg()` instead of writing an empty file.
The pretty printer inherits an override that returns PHP-Parser's `rawValue`
attribute so escape sequences are not interpreted. PHP-Parser sets that
attribute to the body of a doc string, without the delimiters, so
`apply_filters( 'f', <<<EOT ... EOT, 2 )` exported its argument as a bare
`body` string with embedded newlines instead of PHP source.

Print heredoc and nowdoc nodes with the default printer, which reproduces
the `<<<LABEL ... LABEL` form and does not interpret escape sequences in doc
strings either.
`pName_FullyQualified()` prints single-segment fully-qualified names without
the leading backslash regardless of namespace context, so inside a namespaced
file the printed form denotes a namespaced symbol rather than the global one.
This is an accepted limitation because the parser targets global-namespace
WordPress core code.
The global namespace prefixes are stripped from inline `{@link}` and
`{@see}` references after the DocBlock text has been rendered, so the
stripping also reached into rendered code regions and silently deleted
the backslash an author had written in a verbatim code sample.

Carve out `<code>` regions before stripping, the same way `fix_newlines()`
protects the newlines in those regions, so code samples are exported as
they were written.
The quote-stripping pattern required a body free of quote characters, so
a hook name that contained one, like `do_action( "it's" );`, was exported
with the quotes that surround it in the source.

Match the opening quote and require the same quote at the end, allowing
the body to hold the other quote character or an escaped copy of the
delimiter. Only that pair is stripped; the body keeps its source spelling,
so `do_action( 'it\'s' );` exports as `it\'s`. Concatenated expressions
still fall through to the dynamic-name handling below.
`Method_Call_Reflector::_getClassMapping()` maps a handful of WordPress factory
functions to the class they return, so that `get_current_screen()->add_help_tab()`
is exported as a use of `WP_Screen::add_help_tab()`. The lookup never matched
before this branch, because the printed receiver carried a leading backslash
that the mapping keys do not have. Pin the restored behavior with a test.
Inline `{@see}`/`{@link}` references in prose and the reference tokens
of `@see`/`@link` tags are documentation text: the export should carry
the author's spelling, backslash included. Pin that, and pin the
boundary on the other side: `@param` types pass through phpDocumentor's
type resolution, which prefixes every non-keyword type with a synthetic
`\`, so the author's spelling is unrecoverable there and the synthetic
prefix must keep being stripped.
Stop stripping global namespace prefixes from documentation text:
inline `{@see}`/`{@link}` references in descriptions and tag content,
and the reference tokens of `@see`/`@link` tags. These fields carry the
author's own words — phpDocumentor hands them over verbatim — so any
leading backslash present was written by hand and any rewrite imposes
a spelling the author didn't choose. The consumer that renders these
references (the developer.wordpress.org theme re-parses the inline tags
at render time) owns display policy.

This removes the code-region protection machinery along with the strip:
with nothing rewriting the text, code samples and inline code spans are
verbatim by construction, and the existing tests for them become plain
regression pins.

Tag types and argument types keep the strip: those values pass through
phpDocumentor's type resolution, which synthesizes a leading `\` on
every non-keyword type regardless of how the author spelled it, so the
prefix there is resolver output, not authorship.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Some characters are stripped from documentation

1 participant