feat: integrate Lucene analysis chain into full-text indexing and querying

[lukashornych]

Refs: #258

[novoj]

Review of the P5 analyzer prototype. Every claim below was verified against real Lucene 9.12.3 (the exact chains this PR builds, driven programmatically), and the full test suite of both new test classes was executed locally — see the verification note at the end. The design decisions P5 called "expensive to reverse" (NFC at the boundary, three slots, mode declared at registration, surface forms emitted) are all implemented and correct; nothing below disputes the architecture. These are the items worth addressing before P1 builds on this.

---

1. Accent-insensitive typing is ~82 % solved, with a systematic failure class the test set doesn't cover — the one finding that affects P1's design

The folding order (ASCIIFoldingFilter after CzechStemFilter) was chosen so that stemming sees the accented endings it switches on. That works — but the query side then suffers the mirror problem: a query typed without accents never gets its ending stripped, because CzechStemmer's ending tables are written with accented characters.

Measured over a 30-lemma / 83-word Czech e-commerce vocabulary (each accented form vs. its accent-stripped typing):

variant	accent-typed query matches	inflection forms converge	false merges
this PR (fold after stemmer)	68/83 (82 %)	29/30	0
fold before stemmer	83/83 (100 %)	16/30	0

The 15 misses are not random — they are one morphological class plus a vowel-shift case:

• every adjective genitive/locative plural in -ých/-ích typed bare: pánských → pansk but panskych → panskych (same for černých, dětských, kožených, žlutých, bílých, šedých, velkých, malých, stříbrných, kuchyňských, dřevěných) — so damske boty works while damskych bot misses, and the 3-edit distance puts it beyond a fuzzy lane's reach;
• stůl → stol vs. stul → stul; dřevěný → dreven vs. dreveny → drevn.

The PR's only cross-form assertion (černá/cerna) happens to sit in the passing 82 %, so the suite currently reads as "accent-insensitivity: solved".

Suggested action: (a) add a test documenting the known-miss class (e.g. panskych vs. pánských) so the limitation is visible and pinned rather than discovered in production; (b) carry the consequence into the P1 design — AnalyzedTerm.surfaceForm() already exists precisely so the index can add a folded surface-form lane that catches these queries; that decision needs to be made before the term-dictionary layout freezes. Neither pure folding order fixes this (fold-before destroys inflection convergence, 16/30).

2. Polish gets no diacritics folding at all

BuiltInAnalyzers maps pl to a bare PolishAnalyzer ("left as-is" in the javadoc table, with no reason recorded). Measured: 4/4 common cases diverge completely — żółty/zolty, świeży/swiezy, książka/ksiazka, łóżko/lozko share no term. The argument that justified folding Czech ("typing without accents is the norm on e-commerce") applies verbatim to Polish keyboards (ł, ż, ó…). Either wrap PolishAnalyzer in DiacriticsFoldingAnalyzerWrapper too, or record in the javadoc why Polish is exempt — right now the asymmetry reads as an oversight, not a decision.

3. generic doesn't fold, slovak does — same chain otherwise

slovak = DiacriticsFoldingAnalyzerWrapper(new TokenizingAnalyzer()), generic = TokenizingAnalyzer bare. For the generic fallback there is no stemmer whose input folding could corrupt — the one reason folding is ever deferred doesn't exist here, so folding the generic chain is free accent-insensitivity for every unknown language. If the asymmetry is deliberate (e.g. "generic must be the identity-ish baseline"), one javadoc sentence would settle it.

4. close() races with lookups — cleared instance map can be silently repopulated

FulltextAnalyzerRegistry.getAnalyzer checks assertNotClosed() at entry but calls instances.computeIfAbsent(...) at the end. A lookup that passes the assert while close() runs concurrently can (a) insert a new FulltextAnalyzer after close() already iterated and cleared the map — that analyzer is never closed and its CloseableThreadLocal hardRefs leak for the process lifetime, which is exactly what the class javadoc says close() exists to prevent — or (b) receive a shared instance whose components are being released mid-use. The window is narrow (catalog close during concurrent write/query) but the fix is cheap: e.g. re-check closed inside the computeIfAbsent mapping function and after insertion (closing + removing the freshly created instance when the flag flipped), or guard close()/creation with a shared lock.

5. The streaming path allocates a String per token anyway

AnalyzedTermConsumer exists so the write path can consume terms "without materializing a list" — but analyze(...) computes normalizedText.substring(startOffset, endOffset) for every token before invoking the callback, whether or not the consumer wants the surface form. Since the callback already receives both offsets, the surface form is derivable; consider handing the consumer the normalized text once (or an interface where surfaceForm is produced lazily), so an indexing consumer that only wants term + offsets pays zero per-token allocations beyond termAttribute.toString().

6. Offsets and surfaceForm index into the normalized text, not the caller's original

Correctly documented and tested — flagging it because it is a delayed trap: any future highlighter (P4) slicing the stored attribute value with these offsets will cut at wrong positions whenever the stored value wasn't NFC to begin with. Worth a sentence in AnalyzedTermConsumer's javadoc too (it documents the offsets but the consumer is the type P4 will actually implement), and worth remembering that the highlighter must normalize before slicing.

7. Public register(String, Supplier<org.apache.lucene.analysis.Analyzer>) in an exported package, with a non-transitive requires

io.evitadb.index.fulltext.analysis is exported and register(...) takes a Lucene Analyzer supplier, but module-info (correctly) declares requires org.apache.lucene.core without transitive. Consequence: any JPMS consumer that wants to register a custom analyzer must add its own requires org.apache.lucene.core and manage the Lucene version alignment itself. That may be the right call (custom analyzers are an expert seam), but it should be a decision: either requires transitive org.apache.lucene.core (leaks Lucene into evita_engine's API surface for everyone), or keep as-is and say so in register's javadoc so the first consumer isn't surprised by an unreadable-class error. Also note the frozen-version comment in the root pom applies doubly here — a consumer compiling against a different Lucene must not be able to shift analysis behaviour.

8. REPORTED_UNKNOWN_LANGUAGES is process-static while the registry is per-catalog

Minor: the once-per-language warning set in BuiltInAnalyzers is a static, so the warning fires once per JVM, not per catalog — catalog B silently inherits catalog A's suppression, and test runs pollute each other within a fork. If the once-per-catalog semantics matter for operators (each catalog's log telling its own story), the set belongs on the registry instance; if once-per-JVM is intended, a word in the javadoc avoids the next reader "fixing" it.

---

9. Add the WordWithNumberSplitFilter to this PR — closing the P5 filter catalog for good

P5 §4.6 point 2 gave this a "take up, off by default" verdict: a token beginning or ending with a digit is split into its numeric and textual parts, both added at the same position alongside the original (123xyz → 123xyz + 123 + xyz) — production-proven for EANs and catalog numbers. It is deliberately absent from the built-in chains (position-multiplying filters have no business in a default profile), but the filter itself should ship in this PR as a composable, off-by-default TokenFilter with its own tests, so the analysis package closes complete instead of being reopened after merge. The per-field switch that enables it arrives later with the analyzer parameters in the schema step — no wiring into BuiltInAnalyzers is expected here.

---

Verified independently, no action needed (recorded so the next reviewer doesn't re-litigate them):

• All per-language test expectations (terms, offsets, position increments) reproduce exactly against real Lucene 9.12.3 — including mluvim[6,13,+2]/voln[16,22,+2], muž/muži/muže → muh×3, Schaltflächen/Schaltflaechen → schaltflach×2.
• requires org.apache.lucene.core is genuinely necessary (jar --describe-module on analysis-common 9.12.3: the requires is non-transitive) — the P5 document's claim that analysis-common brings core along is wrong and the PR is right; the prototype doc will be corrected.
• DiacriticsFoldingAnalyzerWrapper.wrapTokenStreamForNormalization is wired correctly: AnalyzerWrapper.normalize applies the delegate's normalization first, then the folding, so the fuzzy/prefix normalization path lowercases before folding as intended.
• Lucene cannot reach evita_java_driver: the driver's dependency closure (evita_api, evita_query, evita_common, grpc-shared) never touches evita_engine.
• attributeContains/FilterIndex and their NFD keys are untouched by the diff.

Scope triage against the P5 plan (decided with the plan owner; recorded so none of it reads as an oversight):

• Moved into this PR: the word/number split filter — finding 9 above.
• Deferred to the schema-design step, deliberately — no action expected here: analyzer parameters (custom stop words, optional-filter switches). AnalyzerAssignment carrying names only is correct for this PR.
• Discarded outright: the keyword-marker protection (SetKeywordMarkerFilter). Exact-match values (catalog numbers, EANs) will be modelled as separate attributes that are not fulltext-indexed, served by attributeContains and prospectively by the trigram SUBSTRING capability (#1454); embedded codes still match themselves because analysis is symmetric on both sides. This also makes the "ASCIIFoldingFilter does not honour KeywordAttribute" gap recorded in DiacriticsFoldingAnalyzerWrapper's javadoc moot — please reword that note to point at the decision rather than at a pending fix, so nobody "fixes" it later.
• Discarded outright: the DiacriticFilter parity list (P5 §10.4) and the Slovak variant-B (Czech-stemmer) measurement — no supported migration path from the legacy stack, and no demonstrated upside over the shipped Slovak baseline, respectively.
• Remaining plan items (not this PR): the real-attribute-value tokenization review (P5 §10.3) now gates P1's dictionary-layout decision, and the unknown-language-fallback user documentation lands with the schema step.

Local verification run: PR head 80b8c74 built in a clean worktree; mvn -pl evita_test/evita_functional_tests -am test -P unitAndFunctional -Dtest='FulltextAnalyzerTest,FulltextAnalyzerRegistryTest' → BUILD SUCCESS, 46 tests, 0 failures, 0 errors (both classes, all @Nested groups). lucene-analysis-stempel:9.12.3 resolves fine from the repository mirror.

[novoj]

Second review, head 3eecee81b. Everything below was reproduced by running the shipped code — a clean build of the branch (mvn -pl evita_test/evita_functional_tests -am test -P unitAndFunctional -Dtest='io.evitadb.index.fulltext.analysis.*Test' → BUILD SUCCESS, 94 tests, 0 failures) plus probes driving FulltextAnalyzerRegistry/FulltextAnalyzer against the compiled evita_engine classes and the pinned Lucene 9.12.3 artifacts.

First, the previous review's finding 1 is fixed, and fixed properly. All thirteen pairs I named as the systematic miss class now meet — panskych→pánských, stul→stůl, dreveny→dřevěný, damskych bot→dámské boty. The M7 apparatus behind it holds up under attack: the coverage invariant sweeps 980,763 Hunspell headwords with 0 uncovered, query-side fan-out measures avg 1.05–2.33 terms/token (max 7, against a MAX_VARIANTS of 19), 1.6 M random folded words produced zero overflow throws, and a 210-call hostile-input sweep raised nothing. Findings 2, 5 and 6 are resolved; finding 9 was closed better than it was asked — measured against WordDelimiterGraphFilter rather than shipped.

Five items below.

---

1. Blocking — the Romanian query chain does not stop the words the index chain stops

romanianIndexChain() runs CommaBelowNormalizationFilter before its StopFilter; romanianSearchChain() omits it. Lucene 9.12.3's Romanian stop list is written entirely in cedilla — 0 of 233 entries use comma-below, 24 use cedilla — while modern orthography and every Romanian keyboard layout produce comma-below. Measured through the shipped chains:

input	INDEX chain	SEARCH chain
și (U+0219, "and")	[]	[si]
şi (U+015F, legacy)	[]	[]
ești	[]	[esti]
niște	[]	[niste, nist]
mașină și rochie	[masin, roch]	[masina, masin, si, rochie, roch]

A correctly typed Romanian query therefore contributes the term si, which no Romanian document can ever hold. query-design.md conjuncts query terms, so this either empties the strict set or silently forces relaxation on the most ordinary Romanian phrasing. No user error is involved — the correct spelling is the one that breaks.

Fix — one line, counterfactual verified. Put CommaBelowNormalizationFilter after the lowercase filter in romanianSearchChain(), exactly where that filter's own javadoc says it belongs ("before the stop filter and the stemmer, which are the components whose cedilla-written data it exists to feed"):

TokenStream stream = new CommaBelowNormalizationFilter(new LowerCaseFilter(source));
stream = new StopFilter(stream, RomanianAnalyzer.getDefaultStopSet());
stream = new ASCIIFoldingFilter(stream);
stream = new VariantStemFilter(stream, new RomanianVariantStemmer());

With that in front, the two sides agree exactly: și → [] on both, mașină și rochie → [masina, masin, rochie, roch]. RomanianAnalysisFixture carries no stop words at all, so LanguageAnalyzerPairRecallTest cannot commit this failure — the fix needs a test that can.

2. Blocking — 14 MB of third-party dictionaries committed with no provenance, licence or NOTICE

evita_test/evita_functional_tests/src/test/resources/fulltext/hunspell/ adds pl_PL.dic 4.5 MB, cs_CZ.dic 3.5 MB, sk_SK.dic 3.3 MB, ro_RO.dic 2.1 MB plus four .aff files — ≈14 MB, permanently in git history. There is no README, no licence file, no upstream URL and no revision; evita_engine/NOTICE covers only PolishSnowballStemmer.

ro_RO.aff states its terms in-file: # Copyright Terms: GPL 2.0/LGPL 2.1/MPL 1.1 tri-license. cs_CZ, pl_PL and sk_SK carry no copyright or licence statement whatsoever — which is the problem rather than a reassurance, since their provenance is now unrecorded.

The plan flags this itself and still lists it as open: p5-analyzers.md §5.3 — "the licence … have to be verified for the specific dictionary, not estimated" — and open question P5-2 is still "the origin and licence of the Slovak Hunspell dictionary". Four dictionaries landed ahead of that answer. Mitigating: evita_functional_tests sets maven.deploy.skip=true and its test-jar carries classes only, so nothing reaches a published artifact — but the repository is public, and merging makes this irreversible.

Fix. Before merge, a README.md beside the files naming each dictionary's upstream project, version and licence, plus matching NOTICE entries. If any of the four turns out incompatible with BUSL distribution, §5.3's own escape hatch applies: load them from a configured path instead of committing them — these sweeps are developer instruments, not gates that must run everywhere.

3. The Polish chains pay Stempel's full cost for a 182-word stop list — can this be avoided?

PolishSnowballStemmer was vendored specifically to drop Stempel ("Stempel dropped — no rule table to fork over … PolishAnalyzer stays referenced for its stop set alone, so the stempel jar stays"). But PolishAnalyzer.getDefaultStopSet() initialises PolishAnalyzer$DefaultsHolder, whose static block loads both stopwords.txt and stemmer_20000.tbl (2.2 MB in the jar) into static final Trie DEFAULT_TABLE — verified by disassembling the class.

Measured on JDK 21, four runs: first call 246 / 274 / 328 / 406 ms; retained heap delta 6.06 / 6.13 / 6.09 / 6.06 MB; DEFAULT_TABLE = org.egothor.stemmer.MultiTrie2, static, never released. So both Polish chains permanently retain ~6 MB and burn ~0.3 s for a ~1.2 kB word list that nothing else uses. FulltextAnalysisChainBenchmark cannot see it — the table is a static and the census is deliberately taken unprimed.

Yes, it can be avoided, and cheaply. Copy the 182 stop words into an evitaDB resource and build the CharArraySet from it (it is Apache-2.0 from Lucene, so one NOTICE entry covers the copy — the same pattern already used for PolishSnowballStemmer). That removes the lucene-analysis-stempel dependency, its requires line in module-info.java, the 519 kB jar and the 6 MB. Nothing else in the branch touches stempel: PolishAnalyzer is referenced for the stop set only, and org.apache.lucene.analysis.pl is the only stempel package imported.

While there: p5-analyzers.md §5.2 (Polish) was never superseded — it still recommends Stempel and describes the 2.1 MB table as the reason instances are lazy. §5.3 got a proper Superseded (2026-09-15) block; §5.2 needs the same.

4. The close() / lookup race still leaks an open analyzer (previous finding 4, half-fixed)

bc0e951ad added assertNotClosed() inside the computeIfAbsent mapping function. The other half — re-checking after insertion and closing/removing the fresh instance when the flag flipped — was not implemented, so the window is now exactly the duration of definition.factory().get(), i.e. the most expensive part (a built-in Polish chain: 250–400 ms, see item 3).

Repro — a registered analyzer whose factory sleeps stands in for a slow built-in chain:

FulltextAnalyzerRegistry reg = new FulltextAnalyzerRegistry(
    (type, loc) -> Optional.of(AnalyzerAssignment.uniform("slow")));
reg.register("slow", () -> { sleep(400); return new TokenizingAnalyzer(); });

Thread t = new Thread(() -> got.complete(reg.getIndexAnalyzer("P", Locale.ENGLISH)));
t.start();
sleep(150);        // A is inside the factory; `closed` is still false
reg.close();       // B closes the registry mid-construction

FulltextAnalyzer a = got.get();
a.getTerms("hello world");   // succeeds - the analyzer was never closed
// instances map is empty, so no future close() can ever reach it

Output:

returned analyzer OPEN and usable after close(): true
instances map after close(): {}
=> analyzer reachable by any future close(): false
LEAK CONFIRMED

That is precisely what the class javadoc says close() exists to prevent — the CloseableThreadLocal hardRefs then hold the stream components for the process lifetime. There is no test for the race.

Related, measured separately and worth one line of its own: a FulltextAnalyzer handle that outlives its registry throws a raw org.apache.lucene.store.AlreadyClosedException out of analyze() — a Lucene type escaping into evita_engine's error surface. analyze() catches IOException only.

5. Javadoc and records that now contradict the shipped code

Seven statements a reader would act on and be wrong. Worth fixing in this PR, since each one was true when written and is the failure mode that costs the next reviewer an afternoon.

• TokenizingAnalyzer — still says it "is the Slovak analyzer, because Lucene ships no SlovakAnalyzer", and that a Slovak Hunspell dictionary's "provenance and licence are not settled yet". Slovak now has slovakIndexChain() with its own SlovakStemmer, and this PR commits sk_SK.dic. Doubly stale.
• DiacriticsFoldingAnalyzerWrapper — says folding "is on by default for Czech and Slovak"; the wrapper is now used by Czech alone, and four languages fold inside their own chains. Its Known gap paragraph still reads as a pending fix ("unless this filter is replaced by a keyword-aware equivalent") — this is the rewording the last review asked for explicitly, so it points at the decision that discarded keyword-marker protection rather than at work somebody will "finish".
• CommaBelowNormalizationFilter — "Index side only. … the distinction never survives to matter there" is falsified by item 1; it survives in the stop filter, which is the one component that still reads the raw spelling.
• romanianSearchChain() — "its cedilla-written entries are matched by the stop words users actually type" is the same falsified claim, stated as the reason for the omission.
• BuiltInAnalyzers — the four chain factories say variants are emitted "at its own position"; the class javadoc and VariantStemFilter say "all at one position", which is what the code does and what a query builder must assume.
• PolishSnowballStemmer — "this class should be deleted in favour of it [once the pinned Lucene carries it]". That deletion changes the index-side stemmer, i.e. it is a catalog reindex, not a cleanup — and the same paragraph records that the vendored source is Lucene main generated from a polish.sbl in no tagged Snowball release, so the released algorithm may differ. Say both at the site, next to the invitation.
• p5-word-number-split-comparison.md — §1 "Reproduce" claims the §4 table is printed by shouldReportSplitterComparison, and §8 "Sources" cites LegacyWordWithNumberSplitFilter.java "in the same package". Neither exists on this branch; the SK/PL/RO record's "Test disposition" says they were deleted. The recipe does not reproduce and the two legacy columns are no longer regenerable — either restore the instrument or mark the table as a historical measurement with the commit it was taken at.

[novoj]

The analysis chain has no char filter, and a real corpus shows what that costs

Measured while building P1's term dictionary (K3) over a production CMS catalog — 972,611 Czech articles whose body is associated data of type String. The body is HTML carrying a JSON payload in data- attributes, and the chain here (StandardTokenizer → LowerCase → Stop → ASCIIFolding) has no CharFilter, so all of it is indexed as terms.

Both arms, whole corpus — as shipped, and with Lucene's HTMLStripCharFilter in front of the analyzer:

	as shipped	HTML stripped	Δ
distinct terms	5,549,814	1,143,244	−79.4 %
postings	284,165,619	176,999,724	−37.7 %
body tokens	665,396,803	191,981,382	−71.1 %
dictionary + postings heap (JOL)	1.01 GiB	515 MiB	−49.1 %
bulk build	14.2 min	6.4 min	−54.9 %

The body is 64.8 % markup by code point: four in five distinct terms are markup, and half the structure's heap goes on them. That lands on two design points — phase-1 cost is the sum of the posting lengths of the expanded terms (p1-index-core.md §5.2), and the impact sidecar spends one byte per (field, term, document). Both pay per document for terms that can never usefully match, and both are reachable by ordinary prefix and typo expansion.

The widest posting lists

Post-analysis terms (folded, stemmed) with df over 972,611 articles, each resolved back to the surface forms that produced it over a 9,859-body sample drawn evenly across the corpus.

As shipped — the payload's vocabulary clusters just under 61 %:

p                955,974   98.3 %   <p>
dat              607,186   62.4 %   data / date   (Czech "dát" < 2 %)
dynamik          591,067   60.8 %   dynamic
quot             590,316   60.7 %   &quot;
fals/class/valu  ~590,150  60.7 %   false / class / value

With HTML stripped — ordinary Czech prose:

ze    703,538          rok   416,851   mel   387,763  (měl)
lt    370,229  (let)   jedn  360,392   rekl  298,330  (řekl)
lid   286,771  (lidé)  cesk  274,573   cl    268,071  (celý)

Correction to an earlier version of this comment, which claimed lt was a double-encoded HTML entity surviving the filter. That was wrong. The dictionary holds stems, and a short stem is not readable as text: lt is the stem of let/letech/letos ("years"), cl is celý, muh is můžete/muž. Measured on the same sample: the literal token lt survives stripping in 0.00 % of bodies, while < occurs in 41.6 % of them, entirely inside tag attribute values, and is removed in full. Stripping is sufficient on this corpus — there is no entity residue.

Two gaps, if markup handling is to be configurable

1. No char filter exists in io.evitadb.index.fulltext.analysis. HTMLStripCharFilter is in lucene-analysis-common, already a pinned dependency here, so the markup half is a small addition. There is nothing for extracting selected paths out of a structured payload.
2. AnalyzerAssignmentResolver#resolveAnalyzers(entityType, locale) is per (entity type, locale), not per attribute. Sufficient for a markup stripper, which is a no-op on plain text and can run over every field. Not sufficient for a payload-path extractor: here title, the lead paragraph and the body share one collection and one locale, and only the body is structured.

Instrument: spike/fulltext/FulltextDictionarySpike — every report it prints names which arm produced it.