Skip to content

Prototype table-driven tokenizer bulk scanning - #40

Draft
isomorphisms wants to merge 3 commits into
mainfrom
issue-38-tokenizer-bulk-scan-experiment
Draft

Prototype table-driven tokenizer bulk scanning#40
isomorphisms wants to merge 3 commits into
mainfrom
issue-38-tokenizer-bulk-scan-experiment

Conversation

@isomorphisms

Copy link
Copy Markdown
Owner

Refs #38.

This adds a deliberately narrow, test-only byte tokenizer experiment under the existing Android-prepaint JVM harness.

What it compares:

  • a scalar oracle that makes the lexical decision one byte at a time;
  • a candidate with a small byte-class table plus state × class -> action dispatch;
  • a bulk path that consumes maximal APPEND_RUN spans before returning to exceptional-byte dispatch.

The event contract is pinned explicitly and the candidate must match it across every single two-chunk split and one-byte chunking. Fixtures cover quoted attributes containing </>, entities, malformed entities, NUL, unclosed quotes, and raw UTF-8 handoff including splits inside multibyte sequences.

The long-run test also proves the candidate is actually taking the bulk path: 4096 ordinary bytes are consumed as bulk work while exceptional dispatch stays below 32 operations.

Deliberate boundaries:

  • this is a lexical experiment, not an HTML conformance tokenizer;
  • it does not integrate with renderer/prepaint production code yet;
  • UTF-8 is handed off losslessly as raw bytes rather than decoded here;
  • SIMD/word-at-a-time scanning is intentionally absent. It should be a later implementation comparison against the same oracle, not a prerequisite for validating the architecture.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant