Skip to content

common: sanitize malformed UTF-8 to U+FFFD with retry parse - #244

Open
renovys wants to merge 1 commit into
PrismML-Eng:prismfrom
renovys:pr-utf8-ufffd-fix
Open

renovys wants to merge 1 commit into
PrismML-Eng:prismfrom
renovys:pr-utf8-ufffd-fix

Conversation

@renovys

@renovys renovys commented Sep 23, 2026 •

Copy link
Copy Markdown

Problem

Chat template parsing hard-fails when the input contains malformed UTF-8 (e.g. truncated multibyte sequences from upstream tooling or IPC boundaries). The whole parse is rejected even though the rest of the input is usable.

Change

  • Add common_utf8_sanitize(): replaces each maximal invalid subpart with a single U+FFFD per the Unicode recommendation, keeps a trailing truncated sequence intact for streaming.
  • In the chat parse path, on a failed first parse, sanitize and retry once; parse AST views are then served from the sanitized copy.
  • Non-malformed inputs take the same fast path as before (single parse, no copy).

Tests

  • test-chat: regression cases for lead-byte + partial continuation runs, overlong/invalid sequences, and mixed valid/invalid payloads. Full suite passes.

Validation

Running in production on a BC-250 (gfx1013, Vulkan) service for ~7h with mixed Korean/security/code workloads — no template parse failures, no template parse failures and no regressions observed.

Chat template parsing previously choked on malformed UTF-8 input.
Add common_utf8_sanitize(), which substitutes each maximal invalid
subpart with a single U+FFFD per the Unicode recommendation, and
retry the chat parse on the sanitized input when recovery happens.

Add regression tests (test-chat).
@renovys

renovys commented Sep 27, 2026

Copy link
Copy Markdown
Author

Rebased cleanly onto the current prism base (adfffbe41, prism-b10743): git cherry-pick applies with no conflicts, and the sanitize+retry path builds and runs on a live BC-250 service. Still a single self-contained commit against prism. Ready for review when convenient.

@renovys

renovys commented Sep 30, 2026

Copy link
Copy Markdown
Author

Rebased onto the current prism (88c4bc60b). While doing so I dropped the unrelated test-backend-ops large-tile cases that had slipped into this PR, so it is now only the UTF-8 sanitize + retry-parse change and its test-chat regression tests (4 files). test-chat passes on this base (CPU build).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant