Make HTML/XML tokenization a renderer-native branch/scanning benchmark

[isomorphisms]

Missing substrate and sequencing — 2026-08-27

The first logical document-tree contract is #41; this issue must not assume that tree already exists. Follow isomorphisms/idric-arm-thumb#31 before implementation. Tokenization, tree construction, traversal, style/layout, and paint are separate acceptance stages. HTML and XML may share scanning machinery, but their recovery and tree semantics must remain distinct. Existing JVM experiments are non-authoritative evidence, not the Idriç/browser representation.

Goal

Treat HTML/XML tokenization as a first-class IB renderer workload rather than assuming complex input must immediately be handed to WebView.

IB does not need to preserve the publisher's exact browser-internal representation. A useful pipeline is:

bytes
  → tokenizer
  → temporary HTML/XML structure
  → normalized IB document IR
  → style/layout IR
  → display list
  → native paint

The tokenizer is itself a branch-heavy state machine and is therefore a good place to test the same compile-once / native-dispatch ideas already being explored for CSS selectors and renderer paint commands.

First tokenizer benchmark

Keep a deterministic semantic oracle for a deliberately narrow HTML/XML subset:

• ordinary text runs;
• <tag> / </tag> / self-closing tags;
• attributes;
• single- and double-quoted attribute values;
• & entity boundary recognition;
• whitespace and structural delimiters;
• comments as a follow-up fixture;
• UTF-8 text payloads without making full Unicode normalization part of the first benchmark.

Record exact token/event output so different implementations can be compared without tying browser semantics to one lowering strategy.

Compare implementation shapes

1. straightforward byte-at-a-time nested branching;
2. (state, character-class) → transition/action tables;
3. bulk scan of ordinary text until < / &;
4. bulk scan of names/attribute values until a structural terminator;
5. packed-word classification where appropriate;
6. SIMD/NEON only later, and only when the actual target supports it.

The important optimization is not "use a jump table everywhere." Preserve enough structure that the compiler/backend can choose between branch chains, decision tables, state transitions, word-at-a-time scans, and other native forms.

Renderer boundary

The parser/tokenizer should feed IB-owned normalized structures. Do not require a long-lived browser DOM if a smaller typed document IR is sufficient for prepaint/native rendering.

This makes the rendering path increasingly compiler-like:

HTML/XML/CSS
  → parse/tokenize once
  → normalize
  → compact semantic/layout IR
  → display list
  → CPU framebuffer / Android Surface / GPU

WebView remains an escape hatch for unsupported semantics, not the default consequence of encountering branches.

Branch-heavy browser workload inventory

• HTML/XML tokenization
  • text vs < vs &
  • opening/closing/self-closing tags
  • tag-name scanning
  • attribute-name/value parsing
  • quote modes
  • whitespace handling
  • comments / CDATA / processing instructions / doctype
• HTML/XML tree construction
  • start-tag vs end-tag behavior
  • nesting/parent selection
  • malformed-input recovery
  • special element handling
• CSS tokenization/parsing
  • identifiers, numbers, strings, delimiters
  • property/value grammar dispatch
  • functions and nested parentheses
  • at-rules
• CSS selector matching
  • tag / class / ID
  • attribute selectors
  • child >
  • descendant
  • adjacent/general sibling + / ~
  • :is(...)
  • :not(...)
  • :has(...)
  • other pseudo-classes
• CSS cascade
  • specificity comparisons
  • source order
  • !important
  • inline vs stylesheet declarations
  • inherited vs non-inherited properties
• Computed-style construction
  • defaults / initial values
  • inheritance
  • property-specific normalization
• Style invalidation after DOM changes
  • class/ID/attribute changes
  • ancestor/descendant dependencies
  • especially :has()
• Tree traversal during style/layout
  • sibling/ancestor/descendant walks
  • early exits
  • repeated node-type/property dispatch
• Layout-property dispatch
  • display type
  • positioning mode
  • overflow/clipping
  • box sizing
  • flex/grid/table/etc.
• Paint/display-list dispatch
  • text
  • backgrounds
  • borders
  • images
  • clipping
  • transforms
  • scrolling

The especially clean branch-lowering benchmarks are tokenizer state machines, selector matching, :has(), tree traversal, and paint-command dispatch.

Measurements

For each implementation keep:

• exact token/event semantics;
• bytes processed;
• tokenizer states/transitions exercised;
• branch frequency where measurable;
• time to first useful structural output / prepaint;
• wall time/throughput;
• code size and emitted native code when a backend owns the implementation.

Related

• Make IB hot paths compiler/backend optimization test targets #33 — IB hot paths as compiler/backend targets
• Make CSS selector matching a browser control-flow benchmark #35 — CSS selector matching control-flow benchmark
• Compile CSS matching and attach computed style records to DOM nodes #36 — compiled CSS matching + computed styles
• Use XML parsing as a branch-heavy lowering benchmark idric-arm-thumb#19 — XML parsing branch-heavy lowering benchmark
• Explore renderer box drawing and compact paint lowering on ARM/Thumb idric-arm-thumb#20 — renderer box drawing / compact paint lowering

Companion compiler and ARM/Thumb issues will track preserving tokenizer structure and lowering it to native target control flow.