Explore table-driven tokenizer dispatch plus bulk fast-path scanning

[isomorphisms]

Sequencing status — 2026-08-27

This remains a future architecture/benchmark issue under isomorphisms/idric-arm-thumb#31. Draft PR #40 is a bounded JVM experiment and may remain evidence, but it does not settle IB's tokenizer semantics, Idriç byte/text representation, or ARM lowering. Do not extend or integrate it as the browser implementation while the language gates are open.

Follow-up to the tokenizer/parser branchiness work: investigate a tokenizer architecture that separates ordinary-byte scanning from exceptional-byte state-machine work.

Core idea:

• classify the current byte/character into a small class;
• use something conceptually like transition[state][class] -> {new state, action} instead of a pile of nested conditionals;
• recognize that most document bytes are boring and scan whole runs until a small set of special bytes appears;
• in ordinary text, examples include <, &, and possibly NUL/newline depending on the semantic boundary;
• inside a tag, examples include whitespace, >, /, =, and quote characters;
• use word-at-a-time or SIMD scanning where profitable, with the full tokenizer/state machine handling only the exceptional bytes.

For IB, prototype this against the HTML/XML/prepaint path and benchmark it on realistic documents. Keep the semantic oracle exact: chunk boundaries, quoted attributes, entities, malformed input, NUL handling, UTF-8 boundaries, and any HTML-vs-XML differences must not change merely because the hot path is widened.

The important architectural question is whether IB can expose this structure explicitly enough that Idriç and the ARM/Thumb backend can preserve it instead of lowering it back into branch forests.

Companion issues:

• Idriç: Preserve table-driven tokenizer and bulk-scan structure through Idriç lowering Idric#27
• ARM/Thumb backend: Optimize table-driven tokenizer dispatch and scan-until-special-byte lowering idric-arm-thumb#22