Skip to content

Write CLI output as UTF-8 regardless of the locale encoding - #37

Draft
imnasnainaec wants to merge 2 commits into
mainfrom
cli/utf8-stdout
Draft

Write CLI output as UTF-8 regardless of the locale encoding#37
imnasnainaec wants to merge 2 commits into
mainfrom
cli/utf8-stdout

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Closes #32.

Option 1 from the issue, plus the newline half of the same asymmetry.

The encoding fix

main() reconfigures sys.stdout and sys.stderr to UTF-8 unless they already carry it. One place, matching what --output has always done, so redirected output is byte-stable across platforms and locales.

  • A stream that is not a TextIOWrapper (a replaced stdout in embedding code) is left alone rather than guessed at.
  • errors stays strict, so this does not turn a failure into corrupted data in the export path. The coupling Cover non-BMP characters, and refuse lone surrogates with LiftWriteError #36 flagged does not bite: the only content UTF-8 cannot encode is a lone surrogate, no file can carry one into the CLI (both readers reject it), and _guarded() renders the codepoint escaped anyway. A crash here would now mean a real defect rather than a platform accident.
  • stderr gets the same treatment: error: {exc} interpolates paths, and a non-ASCII filename crashed the error path itself.

The newline fix, in the same two lines

Verifying the issue's export repro on Windows turned up a second, pre-existing difference between the two output paths: -o opens with newline="", stdout does not, and csv writes its own CRLF, so a redirected export doubled every \r. csv.reader reads that back as a blank row between every data row -- so the fix above would have made a complete-but-malformed CSV newly reachable.

export now writes through its own TextIOWrapper over the stdout byte layer, with the encoding and newline="" that --output uses, and detaches it rather than closing stdout. Redirected output is byte-identical to -o on every platform.

Measured on tests/corpus/large/sango/sango.lift, Windows 11, cp1252 stdout:

before after
export sango.lift > out.csv 202,656 bytes, 1,118 lines, not UTF-8 837,883 bytes, 4,542 lines, UTF-8
export sango.lift -o out.csv 837,883 bytes, 4,542 lines, UTF-8 unchanged

(The > out.csv byte count is now equal to -o, not the 842,425 that the doubled CRs alone would give.)

Tests

Four in tests/test_cli.py, driving the standard streams through a BytesIO under a locale codec:

  • validate text output on negative/nfd-range-ids.lift through a cp1252 stdout -- the fixture's range-parent finding names an NFD Órfão with !r, which is what crashed on main. Asserts the NFD id survives and that output reaches the summary line rather than stopping partway.
  • export through an ASCII stdout, asserting the bytes equal what -o writes for the same source. The stand-in stream spells out newline="\r\n" -- what a Windows stream does with no newline argument -- so the CRLF doubling is reproducible on Linux CI too, not just on Windows.
  • an OSError message on a non-ASCII path through an ASCII stderr.
  • a stdout replaced by io.StringIO: nothing to reconfigure, no byte layer to wrap, both commands still write to it.

Three of the four fail on main; the fourth pins behavior this change must not break.

Not affected, confirmed

--format json (both commands): json.dump defaults to ensure_ascii=True, so that output was already pure ASCII and is unchanged.


This change is Reviewable

Notes

No CHANGELOG entry: 0.1.0 has not shipped, so neither the crash nor the doubled row terminators were ever released behaviour.

imnasnainaec and others added 2 commits August 24, 2026 15:50
Every command that prints lexicon content wrote to a stdout carrying the
platform's locale encoding whenever it was not a console -- cp1252 on
Windows, ASCII under a C/POSIX locale -- so a character the codepage could
not represent raised UnicodeEncodeError and killed the command mid-output.
`validate > out.txt` died on an NFD range-element id from a committed
fixture, and `export` left a truncated file whose bytes were neither UTF-8
nor a complete export, while the same run through -o was both.

main() now reconfigures stdout and stderr to UTF-8 unless they already carry
it, leaving a stream that is not a TextIOWrapper untouched. Errors stay
strict: the only content UTF-8 cannot encode is a lone surrogate, and no file
can carry one into the CLI.

export instead builds its own wrapper over the stdout byte layer, with the
encoding and newline="" that --output has always opened with. csv writes CRLF
row terminators, and a stdout that translates newlines doubled the CR into a
blank row between every data row, so a redirected export was not the file -o
wrote even when it completed; the two are now byte-identical.

Tests drive each path through a byte sink under cp1252 and ASCII codecs,
spelling out the newline translation a Windows stream applies so the doubling
is reproducible on any platform.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.1.0 has not shipped, so redirected output crashing on the locale encoding,
and the doubled CRLF row terminators that went with it, were never released
behaviour for a `[Unreleased]` section to record a change against.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redirected CLI output crashes on characters the locale encoding cannot represent, truncating export

1 participant