Pass delimited tokens through verbatim (S_delim); release 0.5.0

[shigarov]

Brings S_delim in line with the formal model and jRegTab 0.5.0, and cuts the 0.5.0 release.

The change

S_delim = (δ, S_atom) decomposes the cell text into substrings s_k ∈ Σ* and applies S_atom to each as-is. pyRegTab trimmed every token and silently dropped empty ones, making the delimited specification the odd one out — atomic and compound segments already receive their text raw.

Splitting is now verbatim in both places it happens: n substrings always derive n items, numbered 0..n-1, surrounding whitespace intact and empty substrings included. "a, b" yields "a" and " b"; "a,,b" yields three items.

Motive: the explode stratum of ATBench (Auto-Tables, VLDB 2023) in regtab-eval-on-atbench has ground truth produced by pandas str.split(','), which keeps token whitespace and empty strings. Forced trimming made ~41 of 48 cases inexpressible.

⚠️ BREAKING (semantics)

Trimming is now opt-in via the atom's string extractor:

(VAL){","}        →  (VAL=TRIM){","}

Tasks 45 and 55 relied on the implicit trimming and were fixed that way, in both the RTL and the ATP suite. Their expected recordsets are unchanged — no fixture under tests/fixtures/ was touched.

RTL syntax, the grammar and ATP→RTL serialization are untouched. Canonical forms changed only for task_045/task_055, because of the added =TRIM in the patterns themselves.

Not a two-line deletion

Unlike jRegTab, pyRegTab tracks a byte span per item. split_with_spans now reports the span of the raw token and the leading-whitespace offset is gone. Item indices also become contiguous on the standalone delimited path, which is observable through POS filters (FilterTerm::PosExact/PosOffset/PosRange).

The whole behavioural change is one helper — both call sites go through it.

Corpus

Re-pinned to jRegTab v0.5.0 (035ff1a) and synced byte-for-byte (all 333 files verified against upstream; raw CRLF payload in task_099 and tabs in task_101 intact, no whitespace drift in the other 300):

• 152 positive pairs, including the new delim_raw
• the new normative conformance/semantic/ section — item 5 of the contract

tests/test_semantic_conformance.py executes that section (port of RtlSemanticConformanceTest). Note its defaults differ from the task suite: expected.csv is header-less and matched positionally against the schema the pattern produced, so implementation-invented attribute names stay out of the contract — task_runner.load_match_options must not be reused there.

Tests

src/matcher.rs gains a #[cfg(test)] module (the matcher had none): whitespace retention, empty tokens, edge empty tokens, contiguous indices, span arithmetic under a non-zero base, byte-exactness on multibyte text, and the =TRIM opt-in — plus end-to-end cases through match_atp.

Checked to be discriminating: with the old trimming temporarily restored, all 10 new tests fail and the 4 pre-existing ones stay green.

cargo test --no-default-features    14 passed   (was 4)
pytest tests -q                   1918 passed   (was 1908)
cargo clippy --all-targets -D warnings   clean
tools/check_grammar_sync.py              OK

Differential run against jRegTab v0.5.0, executed locally: 750/750 variants identical.

Release 0.5.0

Version bumped in Cargo.toml, Cargo.lock, pyproject.toml and __init__.py; grammar/UPSTREAM re-pinned to v0.5.0 (RTL.g4 byte-identical, sha256 unchanged); the manual differential CI job now builds jregtab@v0.5.0; README/docs parity statements updated to 0.5.0 ≙ jRegTab 0.5.0.

Tagging v0.5.0 after merge triggers the wheel build and PyPI publish.

Plan document: plans/S_DELIM_RAW_SPLIT.md.

🤖 Generated with Claude Code