Skip to content

fix: ensure JSON output is valid UTF-8 - #3445

Closed
Wilfred wants to merge 1 commit into
LuaLS:masterfrom
Wilfred:fix-invalid-utf8-in-json-output
Closed

fix: ensure JSON output is valid UTF-8#3445
Wilfred wants to merge 1 commit into
LuaLS:masterfrom
Wilfred:fix-invalid-utf8-in-json-output

Conversation

@Wilfred

@Wilfred Wilfred commented Aug 6, 2026

Copy link
Copy Markdown

Summary

Lua strings are byte arrays, so a literal such as "\x80" can contain bytes that are not valid UTF-8. The JSON encoder previously copied those bytes unchanged, producing JSON and LSP bodies that strict UTF-8 decoders reject.

Validate strings at the JSON boundary and replace each invalid byte with U+FFFD before applying the existing JSON escapes. Valid UTF-8 is unchanged. Lua 5.1 and 5.2 keep the previous behavior because they do not provide utf8.len.

The repository itself contains a trigger in script/parser/guide.lua:

m.notNamePattern = '[^%w_\x80-\xff]'

Requesting document symbols evaluates that literal and puts its byte-string value into detail. Before this change, the response contained raw bytes 0x80 and 0xff. Afterwards it contains replacement characters and the complete response is valid UTF-8.

This intentionally fixes the serialization boundary only. Producer-specific escaping is a separate display policy and is not necessary to guarantee valid JSON output.

Related: #2983. The beautified JSON encoder used by documentation export shares the repaired string encoder and is covered by the new test.

Validation

  • 3rd/luamake/luamake: all 110 bee tests pass (3 skipped), followed by the complete LuaLS test suite.
  • End-to-end LSP session against the built server: initialize, didOpen for script/parser/guide.lua, and textDocument/documentSymbol; all 191,911 captured protocol bytes decode as UTF-8 and the trigger's detail is "[^%w_�-�]".
  • 20,000 randomized byte strings all encode to valid UTF-8.
  • Regression coverage includes valid non-ASCII text, malformed multibyte sequences, object keys, round-tripping repaired output, and beautified JSON.

Performance

A synthetic benchmark repeatedly encoded a document-symbol-shaped response with 250 symbols. Across five alternating runs, median time increased from 3.678s to 3.884s (+5.6%). Valid strings pay for the C utf8.len scan; allocation and repair occur only for invalid input.

@Wilfred Wilfred closed this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant