feat(core): hyphenate every language whose hyph-utf8 patterns we may carry

[yuroyami]

Resolves #207, and four defects found on the way (#615, #616, #617, #618).

What changes

Data (#207). tools/generate_hyphenation.py writes one Kotlin object for each hyph-utf8 set from a checkout of hyphenation/tex-hyphen. It carries the patterns and, for the first time, the upstream exception lists, both unmodified, under a header that quotes the copyright and the licence text and names the upstream commit. That gives 71 sets for more than 60 languages, up from 7. The seven sets bundled before regenerate with identical patterns.

Left out, with the reason recorded in the script: GPL-only or LGPL-only sets (Czech, Macedonian, Indonesian, upstream Serbian, Armenian, Latvian), sets with no stated licence (Romanian, mn-cyrl-x-lmc), empty sets (Arabic, Persian, Hebrew, Vietnamese), and Thai and Ethiopic, which break lines with no visible hyphen. A set that upstream adds later fails the script until someone decides on it. Serbian is still covered by the LPPL sh-cyrl and sh-latn sets.

Selection. English uses the full hyph-en-us set instead of about 60 hand-picked patterns. forLanguage now reads the rest of a BCP 47 tag:

• British-spelling regions get hyph-en-gb.
• 1901 picks traditional German, with the Swiss set for CH and LI.
• polyton picks polytonic Greek.
• The script picks Latin or Cyrillic Serbian, and Serbian with no script gets both sets, which share no letters.
• x-classic, x-liturgic and x-school pick the private-use sets.

Only the sets a book asks for are loaded.

Memory. The trie now lives in flat arrays with sorted edges. Measured retained heap on the JVM: Hungarian (63k patterns) goes from about 15 MB with a map per node to 1.7 MB, German from 9.3 MB to 1.1 MB. A test keeps Hungarian under 4 MB.

Fixes

• A book in a language without bundled patterns hyphenates with the English patterns #615: a declared language with no bundled set no longer gets English breaks. Only a chapter with no language, or an English one, uses English patterns.
• Hyphenation points shift after a letter whose lower case is two characters, such as Turkish İ #616: lower-casing is one character for one, so a Turkish İ no longer shifts every later break.
• A word that holds a combining mark is never hyphenated #617: words with combining marks (Indic vowel signs, decomposed accents) are hyphenated, and no break goes before a mark.
• A word that touches punctuation or holds an apostrophe is never hyphenated #618: punctuation next to a word (Krankenhaus,, «Bonjour») no longer stops it hyphenating, and an apostrophe or joiner may stand inside a word (l’université).

Tests

• HyphenationGolden holds words for every set, real ones plus pseudo-words built from pattern letters taken from across each file. The expected breaks come from the script's independent implementation, which uses a dictionary lookup over the upstream files, not the trie or the Kotlin strings.
• Mutation checks: dropping one Hungarian chunk fails the golden test. Reverting to lowercase(), removing the mark filter, or restoring the old EPUB word check and English fallback each fail their tests.
• The EPUB tests that compare German against English now sweep a range of page widths. At a single width the English side passed without proving anything.

Gate

• All 11 JVM suites of the CONTRIBUTING gate pass: core 450, pdf 853, epub 917, native-renderer 221, skia 31, difftest 6, kitepdf 32, media 48, net 18, compose-viewer 469, javascript 205. No failures.
• The EPUB sweep passed with every book's page count unchanged.
• This Linux host has no mutool, so the PDF differential ran as a KitePDF-only smoke pass: 58 pages, 0 render failures, no mean-error number. Nothing here touches PDF rendering.
• checkKotlinAbi could not run here (no Android SDK). The public JVM surface of Hyphenator matches the committed dump member for member. The new constructor and trieBytes are internal.

Notes for review

• Source size: the pattern data grows from about 0.5 MB to 3.1 MB of string constants in kitepdf-core. The largest sets are Hungarian (528 KB), the three German sets (about 270 KB each) and Classical Latin (216 KB). If that is too much for the JS or iOS bundles, the rarely selected sets (de-1901, de-ch-1901, la-x-classic, cu) are the first candidates to move into an optional module.
• CONTRIBUTING says to work on main; this session was told to push to a branch, so this arrives as a PR.

---

Generated by Claude Code