Warn at construction when activated scripts can never divide - #337
Merged
Conversation
parser_for(locales.JA) without segmenter= built a parser that behaved exactly like a working one minus the feature: the pack activates HAN and HIRAGANA segmentation, no vocabulary covers those scripts, and no segmenter is configured, so unspaced Japanese names simply never divided -- with nothing saying why. The gap is a CONFIGURATION property, statically decidable at construction: for each activated script, does any surname entry fall wholly inside it, and is a segmenter present? If a script fails both, Parser.__post_init__ now raises a UserWarning naming the dead scripts and the exact call to pass (segmenter=locales.ja_segmenter(), with the nameparser[ja] install hint). A warning rather than an error because the inert JA registration is a pinned, deliberate property -- the pack must be safe to register without the extra -- and the rare caller who wants exactly that can filter it. The default parser (hangul covered by the census surnames), the zh pack (ships its own Han vocabulary), and any parser with a segmenter stay silent, as does the v1 facade. test_ja_pack_alone_is_inert now asserts the warning as the loud half of its claim; the property fuzzers construct through a _quiet_parser helper, since drawn segment_scripts with drawn vocabularies hit the misconfiguration legitimately and constantly -- the warning itself is pinned in test_parser.py, including that the message names only the UNCOVERED scripts under a stacked zh+ja activation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_non_interference_all_packs_combined constructs the full pack stack with ja_segmenter() only when the extra is present; the bare branch -- CI's default environment -- now trips the new construction warning, which filterwarnings=error turns into the failure the build jobs showed. The bare branch expects the warning explicitly: without the extra, the stack's hiragana activation IS unservable, and saying so is the feature. Verified in both environments (extra uninstalled: 3043 passed; reinstalled: both files green) -- the local venv's incidental namedivider install is why the gap didn't reproduce here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #337 +/- ##
=======================================
Coverage 98.48% 98.48%
=======================================
Files 41 41
Lines 2832 2845 +13
=======================================
+ Hits 2789 2802 +13
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Review fixes on the segmenterless-activation warning. parser_for re-emits construction warnings from its own frame: __post_init__'s stacklevel is sized for direct Parser(...) construction, and through parser_for's extra frame the default single-line rendering pointed into the library instead of at the exact call the message tells the user to change. The ja_segmenter hint now appears only when a Japanese script is among the dead ones -- a hangul-only gap (a from-scratch lexicon under the default policy) gets the remedies that actually apply, and every message now offers the Policy(segment_scripts=()) deactivation. Docs say 'emits' (warnings are not raised) and name the custom-lexicon reach. Test hygiene: _quiet_parser ignores this warning by MESSAGE rather than blanketing UserWarning, so a future unrelated construction diagnostic still fails the fuzz; the stacked-activation test selects its warning record by content instead of position, which pack application's own warnings could otherwise shift. Verified in both environments (extra removed: full suite green; reinstalled: green) and the attribution probed live: parser_for's warning now names the caller's file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Aug 6, 2026
#337 landed the segmenterless warning while this branch was open. Its release-log entry is rewritten in the shape the rest of the section now uses, and its usage.rst paragraph auto-merged ahead of the Decomposed text section, which is where it belongs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parser_for(locales.JA)withoutsegmenter=builds a parser that behaves exactly like a working one minus the division feature — the pack activates HAN/HIRAGANA segmentation, nothing can ever divide those scripts, and nothing says so. The gap is statically decidable at construction (per activated script: any covering surname entry? any segmenter?), soParser.__post_init__now emits aUserWarningnaming the dead scripts and the exact call to pass, with thenameparser[ja]install hint.Warning rather than error, deliberately: the inert JA registration is a pinned property (the pack must be safe to register without the extra), and the rare caller who wants exactly that can filter it. The default parser, the zh pack, the v1 facade, and any parser with a segmenter stay silent. Under a stacked
zh+jaactivation the message names only the uncovered scripts (hiragana), not the whole activation set.test_ja_pack_alone_is_inertnow asserts the warning as the loud half of its claim; the property fuzzers go through a justified_quiet_parserhelper (drawn configs hit the misconfiguration legitimately); the warning itself is pinned intest_parser.pywith four tests.Follows the API review discussion: keeping the segmenter out of the pack stands (packs are data; auto-attach would make parsing environment-dependent;
gbdtis a user choice) — the fix is making the forgotten half loud, not implicit.Test plan
filterwarnings=error; mypy, ruff, sphinx html + doctest clean-W error🤖 Generated with Claude Code