0.5.0 — degrade deeply nested HTML instead of raising, convert content on access

[baraline]

Deeply nested HTML made markdownify raise RecursionError from inside model_validate, so a get_ticket call could fail with a bare builtin, and one bad body took its whole page down with it. Fixing that opened a seam that three rounds of adversarial review then worked through; the last round is what replaced the markup scan with the parser rather than an imitation of it.

Changes

• RecursionError no longer escapes the package. Both conversion directions wrap parser faults in a new GlpiContentError (a GlpiError leaf, __cause__ preserved), so except GlpiError catches them.

• Deep nesting degrades instead of raising, and the ceiling is discovered rather than guessed. from_transport attempts the conversion and answers the RecursionError by stripping the document to its text. It never truncates: every character of prose the converting path would have produced also appears in the degraded rendering.

  Attempting it replaced a fixed MAX_HTML_DEPTH = 200 bound that was wrong in both directions. Too low — the budget is not 1000 frames but whatever is left of the stack when the conversion starts, which belongs to the caller, so a bound fixed in advance had to assume the worst and flattened everything between 200 and the real cliff of ~494. Measured, a 300- and a 400-level body now come back as Markdown with their links, emphasis and lists intact where they used to come back as plain text, with no error to notice. And too fragile — predicting the depth meant reproducing html.parser's idea of the tree, and three review rounds found seven ways for the estimate to land under the real depth, each one sending a document into the very RecursionError the bound existed to prevent. Trying the conversion cannot be wrong about whether the conversion fits.

  Costs, measured: a body too deep to convert now pays the failed attempt first (2.0–2.6× at 600 and 5000 levels), and the outcome depends on the caller's remaining stack, so the same body can convert from one call site and degrade from a deeper one. Ordinary bodies got faster (0.87–0.90×) because the scan they used to pay for on every read is gone.

• The depth scan reads html.parser's events instead of imitating its dispatch. Three rounds of adversarial review found one unbounded depth under-count after another in the pattern that stood in for the parser, each fix breeding the next. The last round alone found five: a comment closing on --\s*> rather than only -->, </ script> ending raw text, <![IGNORE[ opening a marked section, </ div foo> being a bogus comment rather than an end tag, and <a href=/> leaving an element open because the unquoted value swallows the /. Each made a document measure one level deep where the real tree was hundreds, so '<a href=/>' * 494 cleared the ceiling and raised. Two further findings were exponential backtracking, where a 39-byte body took 20.8 s.

  The depth prediction those defects were found in is gone entirely (above), but the same pattern also backed the void-tag rewrite and the degraded renderer, which inherited every misreading and the backtracking. Both now come from one pass of an html.parser subclass, which cannot be wrong about html.parser. Not a new dependency or a new risk: markdownify builds its tree with bs4 and bs4 builds it with this same parser, so every pathology it has was already in the pipeline. The exponential shapes are now flat (20.8 s → 0.12 ms).

• Rich text converts on access, not during validation. The six read models store the wire value in content_html / description_html and expose content / description as a cached_property. A list read converts nothing, and an unconvertible body affects only the record whose body is read. Write models stay eager, so caller Markdown is still checked where it was supplied.

• A body using both spellings of <br> no longer loses everything after the second. bs4 records an auto-closed <br> by name in already_closed_empty_element, so a later <br /> closes against the stale entry and stays open, and convert_br discards its children. <p>line1<br>line2</p><p>para2<br />line4</p> converted to 'line1 \nline2\n\npara2'. Void tags are written bare before conversion; <img> and <hr> lost text the same way.

• GlpiContentError now survives the write path. Outbound conversion runs in a PlainSerializer, and pydantic-core catches whatever a serializer raises and re-raises PydanticSerializationError — a ValueError, not a GlpiError, with __cause__ and __context__ both None. So on every create_*/update_* carrying a body, except GlpiError did not fire. The fault is stashed and restored around model_dump with its cause intact; a serialisation failure that is not content becomes GlpiValidationError rather than being mislabelled.

• GlpiModel no longer swallows aliased payload keys, or lets a field shadow itself. _capture_unknown_fields ran before pydantic resolved aliases and diverted the wire's content into extra_payload, leaving the body None with an HTTP 200 and no warning. A payload spelling one field two ways now keeps only the spelling pydantic would use — previously the loser became a model extra that model_dump emitted over the real field, so the attribute reported one body and the object's own dump reported the other.

• A document html.parser refuses now degrades to text instead of raising. <![FOO[ makes _markupbase raise AssertionError on the interpreters where the keyword is unknown — 3.10 through 3.12.11 as measured, no longer 3.12.14 — and bs4 re-raises it as ParserRejectedMarkup. Where that happens the caller used to get an error and none of their words, and now gets their text.

• sys.setrecursionlimit and a larger thread stack both rejected, and for the same reason: the recursion limit is a counter rather than a measurement of the stack, so a deeper thread still needs the process-global limit raised to use it. The reasoning lives in the module docstring and is enforced by an AST guard in the raise-site audit.

• The content tests assert the parity promise rather than one interpreter's output. html.parser's reading of a malformed construct is not stable across CPython patch releases — measured, 3.12.3, 3.12.11 and 3.12.14 disagree about an unterminated <script>, a comment with no -->, an end tag carrying a quoted >, and whether <![FOO[ raises at all. Six tests had written down the literal output of the build they were authored on, which is what had CI red on this branch. The expectation is now computed from the converting path: both paths read the same parser, so they move together, and the relation asserted is the one that was ever true — the degraded rendering is a superset, not an equal.

• Tests and docs. The content round-trip suite asserts the read-model list is complete by discovery rather than claiming it in prose (both guards mutation-tested). test_from_env exports GLPI_SERVER_TIMEZONE, which it needed since 0.4.3. Four live probes that justify the ticket-status typing are kept beside it. New user-guide section on rich-text conversion; GlpiContentError and the content helpers in the API reference. Version bumped to 0.5.0 across pyproject.toml, __version__ and the nine skill stamps.

Breaking

MAX_HTML_DEPTH is removed from glpi_python_client.content.conversion. It was added earlier in this same unreleased cycle and appears in no published release, so no released API changes.

"content" is no longer a field on the six read models. Reading .content is unchanged, but model_dump() now emits content_html holding HTML where it emitted content holding Markdown, and "content" not in GetTicket.model_fields. The validation alias accepts both spellings, so GLPI payloads and GetTicket(content=...) both still populate the field.

Verification

All gates green, and CI green on Python 3.10, 3.11, 3.12, 3.13 and 3.14. 1343 tests, 97.45% coverage, mypy strict, ruff, unasync_build.py --check, zero-warning Sphinx build. Also run locally against CPython 3.12.3 and 3.12.11, since the malformed-construct handling differs between them.

The converter was fuzzed over an alphabet carrying every construct all three review rounds raised — including -- >, </ script>, <![IGNORE[ and runs of whitespace and quotes inside a tag, whose absence is why the earlier corpus could not have found the last round's defects. 257000 documents with 0 non-GlpiError escapes, 0 prose losses and 0 crashes, on top of the 470000 documents that verified the depth scan against a ground-truth bs4 walk before it was removed. 10 MB single-tag, 200k-attribute and 50 MB-prose inputs all handled.

End to end, a page containing a 600-deep body returns all its tickets, with only that one degraded. Live integration against GLPI 11, re-run after the parser rewrite: 44 passed, 4 skipped.

Known limitations, not fixed here

• model_copy(update={"content": ...}) updates nothing, because content is now a cached_property and the value lands in model extras. Named outright in the module docstring and the changelog, since a caller redacting a body that way gets an object whose .content still holds the original. Rewrite content_html, or rebuild through model_validate.

• A cached_property does not invalidate. Assigning to content_html after content has been read leaves the cached Markdown in place, and so does model_copy(update={"content_html": ...}). Treat a read model as immutable once validated.

• A fenced code block loses its language tag. markdownify drops the class="language-python" that fenced_code emits, so ```python round-trips to a bare fence. A limitation of the library pair, not of an extension list.

• How a malformed body degrades depends on the CPython patch level. html.parser changed its reading of an unterminated <script>, a comment with no --> and an end tag carrying a quoted > between 3.12.3 and 3.12.14. The scan tracks the parser rather than a snapshot of it, so the depth decision stays correct on every version — but the exact text a broken construct contributes is the interpreter's, not this package's. Well-formed content is unaffected.

• httpx can still raise outside the taxonomy on a real write — a lone surrogate as UnicodeEncodeError, a non-JSON extra_payload value as TypeError — because the package's TransportRecorder never JSON-encodes, so the suite structurally cannot see it. Tracked separately from this PR.

🤖 Generated with Claude Code

[baraline]

The adversarial review of the depth scan came back after this PR was opened and found a real defect, fixed in 7c20758. Worth recording what it was, because probing it turned up three more of the same kind.

The degraded path was printing a tag it had failed to parse. "<div>" * 300 + "<p title=don't>Le serveur ne repond plus.</p>" came back as "<p title=don't>Le serveur ne repond plus." — raw markup in the one output whose whole promise is text, from a body an apostrophe is enough to produce.

The cause: I had taken the attribute rules from the HTML5 spec rather than from CPython's tolerant parser, which is what markdownify actually builds its tree with. Measured against html.parser's own event stream:

• A quote opens a value only as the first character after the =, so <p title=don't> carries the value don't, and an unquoted value may contain <.
• tagfind_tolerant runs a tag name to whitespace, / or > — so <style=> is an element named style= and never enters raw-text mode, and a self-closed <script/> does not either. Reading either as raw text swallowed the rest of the document: "<style=>" + "<div>" * 600 measured 1 level against a real 601, reached markdownify and raised — an unbounded under-count of exactly the kind the ceiling exists to prevent.
• An attribute name may contain <, so after a quoted value the parser keeps scanning to the next >.
• A declaration is text on neither path only when it is closed; a <!weird at end of input is flushed as text by close(), and dropping it lost the tail of the body.

Re-fuzzed with all of those shapes in the corpus: 15000 documents, zero depth under-counts in either direction, no document losing text beyond the character-reference case already documented, and 4000 deep documents built from the same malformed shapes with none reaching markdownify. Parity with markdownify was measured for each new case instead of inferred. Suite is now 1312 tests, coverage 97.50%, conversion.py still at 100%.

One caveat on the review itself: it was cut short by a spend limit. The strip-fidelity dimension reported and its findings are handled; depth-scan, pydantic-surface, error-taxonomy and docs-and-tests never returned, so those four dimensions have had no independent adversarial pass.

[baraline]

Round two of the review ran the four dimensions that were cut short, then the two verifiers that died on a spend limit. 12 findings, 14 adversarial verifications, 12 confirmed, 2 refuted. All of them are now addressed in e4eef06 and ea7798e.

The two that mattered

Both were unbounded depth under-counts — the one direction the ceiling exists to prevent — and both also deleted prose from the degraded path at any depth. Both came from taking a rule from the HTML5 spec instead of from the function that implements it.

• parse_endtag falls back to rawdata.find(">"), so an end tag skips nothing. CPython's own comment concedes the case: "this is not 100% correct, since we might have things like </tag attr=">">". Reading one with attribute rules made '</x a="><div>">' * 600 measure 0 against a real 600, and raise.
• locatestarttagend_tolerant reaches a quoted value only through an attribute name, and a name may itself begin with =. So <div ="<p> opens no value and ends at the first >, where treating any = before a quote as a value indicator made '<div ="' + "<p>" * 600 measure 1 against a real 600.

The attribute pattern is now a sequence of attributes rather than a run of permitted characters, and end tags have their own branch. The name in that sequence carries the parser's "starts after a quote, whitespace or /" rule, which turned out to be load-bearing twice: it is what makes <div =" a name, and what stops the pattern backtracking catastrophically — my first attempt omitted it and a 400-byte '<div a="' * 50 never finished.

The rest

• GlpiContentError did not survive the write path. Outbound conversion runs in a PlainSerializer, and pydantic-core re-raises everything as PydanticSerializationError — a ValueError, not a GlpiError, with __cause__ and __context__ both None. On every create_*/update_* carrying a body, except GlpiError did not fire and the underlying fault was unrecoverable. C1's whole point, gone on half the surface. Now restored around model_dump with its cause intact; a serialisation failure that is not content becomes GlpiValidationError rather than being mislabelled.
• One field spelled two ways shadowed itself in model_dump — the attribute reported one body and the object's own dump reported the other. The redundant spelling is dropped before pydantic resolves anything, and content_html is now the first alias choice, so a dump carrying both round-trips back to the raw body.
• O(n²) on a body of nothing but <div a=" (32 KB in 6.6 s). No > anywhere means no element anywhere, answered in constant time now. The remaining non-linear shape is documented rather than papered over, because the obvious fix is worse: bounding the attribute repetition makes a tag past the bound fail to match, which is an under-count, unbounded again once such tags nest.
• Numbers that did not reproduce. The cliff is 494 (495 for <ul><li>), not 492. A 200-level document peaks at 412 frames, not 403. _strip_tags runs at 2.2–11.5 MB/s depending on tag density, not 4–8, and is 2.3x–18x faster than the call it stands in for, not "roughly ten times". The call-site frame counts described a call site this branch deleted: conversion no longer happens inside the fetch, so it is 5 frames below the caller reading .content — the same 5 via model_validate or client.get_ticket — and 9 on the write path. The void set is copied from bs4.builder, not bs4.builder._htmlparser.
• A test that claimed a protection it could not give. READ_CONTENT_MODELS said naming the six read models explicitly made a forgotten seventh "a failure here". A hand-written list cannot notice what is missing from it. Two discovery guards now walk the class tree, and both were mutation-tested against the mistake they exist to catch.

Refuted, correctly: the claim that the setrecursionlimit AST guard misses the getattr/sys.__dict__/aliased-import spellings, and one docs claim.

Left as documented behaviour

model_copy(update={"content": ...}) updates nothing — content is a cached_property, so the value lands in model extras. Someone redacting a body that way gets an object whose .content still holds the original. Named outright in the module docstring and the changelog rather than fixed, since the fix would mean intercepting model_copy.

Verification

Re-fuzzed with end tags carrying attributes and name-less =" in the corpus — what round two's corpus lacked: 15000 documents, 0 depth under-counts against the tree that is actually converted, 0 losing a prose word, plus 3000 deep documents with none reaching markdownify. Gates: 1348 unit tests, coverage 97.53%, mypy strict, ruff, unasync --check, zero-warning docs, and 44 passed / 4 skipped against the live GLPI 11 preprod instance.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.99571% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.31%. Comparing base (6815f17) to head (3e906d3).

Files with missing lines	Patch %	Lines
glpi_python_client/content/conversion.py	94.39%	6 Missing ⚠️
glpi_python_client/models/_base.py	97.95%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #37      +/-   ##
==========================================
+ Coverage   97.24%   97.31%   +0.06%     
==========================================
  Files          80       80              
  Lines        2614     2826     +212     
==========================================
+ Hits         2542     2750     +208     
- Misses         72       76       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.