Write CLI output as UTF-8 regardless of the locale encoding - #37
Draft
imnasnainaec wants to merge 2 commits into
Draft
Write CLI output as UTF-8 regardless of the locale encoding#37imnasnainaec wants to merge 2 commits into
imnasnainaec wants to merge 2 commits into
Conversation
Every command that prints lexicon content wrote to a stdout carrying the platform's locale encoding whenever it was not a console -- cp1252 on Windows, ASCII under a C/POSIX locale -- so a character the codepage could not represent raised UnicodeEncodeError and killed the command mid-output. `validate > out.txt` died on an NFD range-element id from a committed fixture, and `export` left a truncated file whose bytes were neither UTF-8 nor a complete export, while the same run through -o was both. main() now reconfigures stdout and stderr to UTF-8 unless they already carry it, leaving a stream that is not a TextIOWrapper untouched. Errors stay strict: the only content UTF-8 cannot encode is a lone surrogate, and no file can carry one into the CLI. export instead builds its own wrapper over the stdout byte layer, with the encoding and newline="" that --output has always opened with. csv writes CRLF row terminators, and a stdout that translates newlines doubled the CR into a blank row between every data row, so a redirected export was not the file -o wrote even when it completed; the two are now byte-identical. Tests drive each path through a byte sink under cp1252 and ASCII codecs, spelling out the newline translation a Windows stream applies so the doubling is reproducible on any platform. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.1.0 has not shipped, so redirected output crashing on the locale encoding, and the doubled CRLF row terminators that went with it, were never released behaviour for a `[Unreleased]` section to record a change against. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #32.
Option 1 from the issue, plus the newline half of the same asymmetry.
The encoding fix
main()reconfiguressys.stdoutandsys.stderrto UTF-8 unless they already carry it. One place, matching what--outputhas always done, so redirected output is byte-stable across platforms and locales.TextIOWrapper(a replaced stdout in embedding code) is left alone rather than guessed at.errorsstays strict, so this does not turn a failure into corrupted data in theexportpath. The coupling Cover non-BMP characters, and refuse lone surrogates with LiftWriteError #36 flagged does not bite: the only content UTF-8 cannot encode is a lone surrogate, no file can carry one into the CLI (both readers reject it), and_guarded()renders the codepoint escaped anyway. A crash here would now mean a real defect rather than a platform accident.error: {exc}interpolates paths, and a non-ASCII filename crashed the error path itself.The newline fix, in the same two lines
Verifying the issue's
exportrepro on Windows turned up a second, pre-existing difference between the two output paths:-oopens withnewline="", stdout does not, and csv writes its own CRLF, so a redirected export doubled every\r.csv.readerreads that back as a blank row between every data row -- so the fix above would have made a complete-but-malformed CSV newly reachable.exportnow writes through its ownTextIOWrapperover the stdout byte layer, with the encoding andnewline=""that--outputuses, and detaches it rather than closing stdout. Redirected output is byte-identical to-oon every platform.Measured on
tests/corpus/large/sango/sango.lift, Windows 11, cp1252 stdout:export sango.lift > out.csvexport sango.lift -o out.csv(The
> out.csvbyte count is now equal to-o, not the 842,425 that the doubled CRs alone would give.)Tests
Four in
tests/test_cli.py, driving the standard streams through aBytesIOunder a locale codec:validatetext output onnegative/nfd-range-ids.liftthrough a cp1252 stdout -- the fixture'srange-parentfinding names an NFDÓrfãowith!r, which is what crashed onmain. Asserts the NFD id survives and that output reaches the summary line rather than stopping partway.exportthrough an ASCII stdout, asserting the bytes equal what-owrites for the same source. The stand-in stream spells outnewline="\r\n"-- what a Windows stream does with no newline argument -- so the CRLF doubling is reproducible on Linux CI too, not just on Windows.io.StringIO: nothing to reconfigure, no byte layer to wrap, both commands still write to it.Three of the four fail on
main; the fourth pins behavior this change must not break.Not affected, confirmed
--format json(both commands):json.dumpdefaults toensure_ascii=True, so that output was already pure ASCII and is unchanged.This change is
Notes
No CHANGELOG entry: 0.1.0 has not shipped, so neither the crash nor the doubled row terminators were ever released behaviour.