Pass delimited tokens through verbatim (S_delim); release 0.5.0

.github/workflows/ci.yml

@@ -69,7 +69,7 @@ jobs:
         uses: actions/checkout@v4
         with:
           repository: regtab/jregtab
-          ref: v0.4.1
+          ref: v0.5.0
           token: ${{ secrets.JREGTAB_TOKEN }}
           path: jregtab
       - uses: actions/setup-java@v4

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "pyregtab"
-version = "0.4.0"
+version = "0.5.0"
 edition = "2021"
 description = "Native core of pyRegTab: RTL compiler, ATP matcher and table interpreter"
 license = "MIT"

README.md

@@ -16,8 +16,8 @@ and interprets the match into a relational **recordset**:
 TableSyntax → RtlCompiler/TablePattern → AtpMatcher → TableInterpreter → Recordset
 ```
 
-**pyRegTab 0.4.0 ≙ jRegTab 0.4.1** (same API, same semantics, same test
-corpus; jRegTab 0.4.1 changes only the Java build over 0.4.0), including the
+**pyRegTab 0.5.0 ≙ jRegTab 0.5.0** (same API, same semantics, same test
+corpus), including the
 embedded RTL DSL `pyregtab.dsl` — a port of jRegTab's `ru.icc.regtab.dsl`
 (added upstream in jRegTab 0.3.0). Python-side extras on top of the Java API:
 `AtpMatcher.match_many` (parallel batch matching), `Recordset.to_pandas()`,
@@ -165,8 +165,7 @@ Rust (`pyregtab._core`, built with [PyO3](https://pyo3.rs) and
 smoke test against the native core alone. Differential testing against the
 Java reference (`tools/differential.py` + `tools/RecordsetDumpMain.java`)
 compares recordsets cell-by-cell on all 750 task variants — zero
-mismatches against jRegTab v0.4.0 (whose Java sources are unchanged in
-v0.4.1).
+mismatches against jRegTab v0.5.0.
 
 ## IDE support
 

conformance/README.md

@@ -14,8 +14,14 @@ conformance/
 ├── positive/
 │   ├── <id>.rtl               — RTL source (UTF-8, no BOM, LF, trailing newline)
 │   └── <id>.expected.rtl      — canonical form: serialize(compile(<id>.rtl))
-└── negative/
-    └── <name>.rtl             — must be rejected with a compile error
+├── negative/
+│   └── <name>.rtl             — must be rejected with a compile error
+└── semantic/
+    └── <case>/                — execution semantics, see below
+        ├── pattern.rtl
+        ├── input.csv
+        ├── expected.csv
+        └── options.json       — optional
 ```
 
 **Byte-exactness caveat:** RTL string literals may contain raw CR/CRLF bytes as
@@ -39,11 +45,67 @@ Any RTL implementation must satisfy, for this corpus:
 4. Every `negative/<name>.rtl` is **rejected** with a compile error
    (`RtlCompileException` in Java, `RtlCompileError` in Python). Reporting the error
    position is recommended but not normative.
+5. For every `semantic/<case>/`, matching `pattern.rtl` against `input.csv` and
+   interpreting the result yields a recordset equal to `expected.csv`.
 
 Byte-equality of canonical forms transitively guarantees that two implementations build
-the same ATP without comparing object graphs across languages.
-
-In jRegTab the contract is executed by `ru.icc.regtab.conformance.RtlConformanceTest`;
+the same ATP without comparing object graphs across languages. Items 1–4 stop there,
+though: two implementations can agree on the canonical form of a pattern and still
+*execute* it differently. Item 5 closes that gap for the behaviours where it matters.
+
+## Semantic cases
+
+A case is a directory under `semantic/` holding:
+
+| File | Role |
+|---|---|
+| `pattern.rtl` | the RTL pattern (UTF-8, no BOM, LF, trailing newline) |
+| `input.csv` | the table to match — no header row, `,` delimiter, `"` quotes, UTF-8 |
+| `expected.csv` | the expected recordset, same CSV dialect |
+| `options.json` | optional; `attributeOrder`, `recordOrder` (`STRICT`\|`FLEXIBLE`), `expectedHasHeader` |
+
+Cell text is taken from the CSV **verbatim** — quoting is what makes leading and trailing
+spaces significant, so `"a, b"` is one cell whose text is `a, b`, and `""` is an empty
+cell. Readers must not strip surrounding whitespace.
+
+By default `expected.csv` has **no header row**: its columns are matched positionally
+against the schema the pattern produced, in record order. This keeps attribute names
+invented by the implementation out of the contract. A case whose pattern names its
+attributes (via `AVP`) may set `"expectedHasHeader": true` and put those names in the
+first row.
+
+Cases are maintained by hand, like `negative/` — the generator never writes here.
+Keep each case minimal and focused on one rule, so that a failure names the rule.
+
+## Semantics of S_delim
+
+The canonical form cannot reveal this rule — both spellings below serialize
+identically — so it is pinned by `semantic/` cases. The semantics of the delimited
+content specification `S_delim = (δ, S_atom)` (`def:delimited-content-spec`):
+
+- The input text is split on every occurrence of `δ`, keeping trailing empty fields
+  (Java `String.split(…, -1)`, Python `str.split(δ)`).
+- Each substring `sₖ ∈ Σ*` is passed to `S_atom` **verbatim**. Implementations must not
+  trim substrings and must not drop empty ones: `n` substrings always derive `n` items,
+  numbered `0..n-1`. `"a, b"` therefore yields `"a"` and `" b"`; `"a,,b"` yields
+  `"a"`, `""`, `"b"`.
+- Whitespace removal is opt-in, expressed by the atom's string extractor `ξ`:
+  `(VAL=TRIM){","}` (or `=NORM`). The extractor applies to each substring separately.
+- The same rules apply to a delimited specification nested in a compound one.
+
+Positive case `delim_raw` pins both forms syntactically; the executable checks are
+`semantic/delim_raw_tokens` (token whitespace survives),
+`semantic/delim_empty_tokens` (empty tokens derive items),
+`semantic/delim_trim` (`=TRIM` opts into trimming) and
+`semantic/compound_delim_raw` (the same rules for a delimited segment nested in a
+compound specification).
+
+> Changed in jRegTab 0.5.0. Earlier versions trimmed each substring and silently
+> dropped empty ones; patterns relying on that must add `=TRIM` to the delimited atom.
+
+In jRegTab items 1–4 of the contract are executed by
+`ru.icc.regtab.conformance.RtlConformanceTest` and item 5 by
+`ru.icc.regtab.conformance.RtlSemanticConformanceTest`;
 `ConformanceCorpusFreshnessTest` additionally guards the committed files against drift
 from the task test suite.
 
@@ -58,15 +120,17 @@ mvn test-compile org.codehaus.mojo:exec-maven-plugin:3.5.0:java \
     -Dexec.classpathScope=test
 ```
 
-Commit the result. Negative cases are maintained by hand — when adding a new error
-branch to the grammar or compiler, add a case here.
+Commit the result. Negative and semantic cases are maintained by hand — when adding a
+new error branch to the grammar or compiler, add a `negative/` case; when changing or
+clarifying how a construct *executes*, add a `semantic/` one.
 
 ## Evolving RTL
 
 Any change to the RTL language follows this order:
 
 1. Change the grammar `RTL.g4` (the normative specification) in jRegTab.
-2. Add/extend corpus cases (positive with canonical forms, negative for new error branches).
+2. Add/extend corpus cases (positive with canonical forms, negative for new error
+   branches, semantic for new or changed execution behaviour).
 3. Implement in the jRegTab compiler; CI (`conformance` job) must be green.
 4. Downstream implementations update their pinned upstream commit, sync the corpus copy,
    and implement the change; their conformance suite must be green.

conformance/UPSTREAM

@@ -1,3 +1,3 @@
-commit: 7a9b78974cc88c872bc994d6d38c69e810828ea6
-tag: v0.4.1
+commit: 035ff1a139e885e4cea85aa66a33e89a6b30f8c9
+tag: v0.5.0
 path: conformance/

conformance/VERSION

@@ -1,2 +1,2 @@
-generated: 2026-07-07
+generated: 2026-08-26
 sources: RtlTask001..150 + curated extras

conformance/positive/delim_raw.expected.rtl

@@ -0,0 +1 @@
+[ [ (VAL : CL*->REC){","} ] [ (VAL = TRIM : CL*->REC){","} ] ]

conformance/positive/delim_raw.rtl

@@ -0,0 +1 @@
+[ [(VAL : CL*->REC){','}] [(VAL=TRIM : CL*->REC){','}] ]

conformance/positive/task_045.expected.rtl

@@ -1 +1 @@
-<ANCH(1)> [ [ !BLANK? VAL ] [ !BLANK? (VAL : (SR & C0)->REC(1)){","} ] ]+
+<ANCH(1)> [ [ !BLANK? VAL ] [ !BLANK? (VAL = TRIM : (SR & C0)->REC(1)){","} ] ]+

conformance/positive/task_045.rtl

@@ -1 +1 @@
-[ [!BLANK? VAL] [!BLANK? (VAL : SR&C0->REC(1)){','}] ]+
+[ [!BLANK? VAL] [!BLANK? (VAL=TRIM : SR&C0->REC(1)){','}] ]+

conformance/positive/task_055.expected.rtl

@@ -1 +1 @@
-[ [ VAL : CL*->REC "," (VAL){","} ] ]+
+[ [ VAL : CL*->REC "," (VAL = TRIM){","} ] ]+

conformance/positive/task_055.rtl

@@ -1 +1 @@
-[ [VAL: CL*->REC ',' (VAL){','}] ]+
+[ [VAL: CL*->REC ',' (VAL=TRIM){','}] ]+

conformance/semantic/compound_delim_raw/expected.csv

@@ -0,0 +1,2 @@
+"a1"," b1"," c1"
+"a2"," b2"," c2"

conformance/semantic/compound_delim_raw/input.csv

@@ -0,0 +1,2 @@
+"a1, b1, c1"
+"a2, b2, c2"

conformance/semantic/compound_delim_raw/pattern.rtl

@@ -0,0 +1 @@
+[ [VAL: CL*->REC ',' (VAL){','}] ]+

conformance/semantic/delim_empty_tokens/expected.csv

@@ -0,0 +1,6 @@
+"k1","a"
+"k1",""
+"k1","b"
+"k2","c"
+"k2","d"
+"k2",""

conformance/semantic/delim_empty_tokens/input.csv

@@ -0,0 +1,2 @@
+"k1","a,,b"
+"k2","c,d,"

conformance/semantic/delim_empty_tokens/pattern.rtl

@@ -0,0 +1 @@
+[ [!BLANK? VAL] [!BLANK? (VAL : SR&C0->REC(1)){','}] ]+

conformance/semantic/delim_raw_tokens/expected.csv

@@ -0,0 +1,4 @@
+"k1","a"
+"k1"," b"
+"k2","c "
+"k2","d"

conformance/semantic/delim_raw_tokens/input.csv

@@ -0,0 +1,2 @@
+"k1","a, b"
+"k2","c ,d"

conformance/semantic/delim_raw_tokens/pattern.rtl

@@ -0,0 +1 @@
+[ [!BLANK? VAL] [!BLANK? (VAL : SR&C0->REC(1)){','}] ]+

conformance/semantic/delim_trim/expected.csv

@@ -0,0 +1,4 @@
+"k1","a"
+"k1","b"
+"k2","c"
+"k2","d"

conformance/semantic/delim_trim/input.csv

@@ -0,0 +1,2 @@
+"k1","a, b"
+"k2","c ,d"

conformance/semantic/delim_trim/pattern.rtl

@@ -0,0 +1 @@
+[ [!BLANK? VAL] [!BLANK? (VAL=TRIM : SR&C0->REC(1)){','}] ]+

docs/index.md

@@ -109,4 +109,4 @@ Requires **Python 3.10+**; binary wheels for Windows, Linux, and macOS.
 ---
 
 !!! note "Status"
-    Current release: **0.4.0** (feature parity with jRegTab 0.4.1) · License: **MIT** · [PyPI](https://pypi.org/project/pyregtab/) · [GitHub](https://github.com/regtab/pyregtab)
+    Current release: **0.5.0** (feature parity with jRegTab 0.5.0) · License: **MIT** · [PyPI](https://pypi.org/project/pyregtab/) · [GitHub](https://github.com/regtab/pyregtab)

docs/model/atp.md

@@ -196,6 +196,12 @@ If the cell text decomposes as `s₁ · δ · s₂ · δ · … · δ · sₙ`,
 applied independently to each `sₖ`, deriving one item per substring.  This is
 used, for example, when a single cell contains a comma-separated list of values.
 
+Each `sₖ ∈ Σ*` is passed to `S_atom` **verbatim**: substrings are not trimmed, and an
+empty substring — produced by adjacent, leading or trailing delimiters — derives an item
+with an empty string value rather than being discarded.  Hence `n` substrings always
+derive exactly `n` items.  Whitespace removal is the job of the atom's string extractor
+`ξ` (`=TRIM`, `=NORM`), which is applied to each substring in turn.
+
 ??? note "API mapping — DelimitedContentSpec"
     ```python
     DelimitedContentSpec.of(atomSpec, delimiter)
@@ -385,7 +391,7 @@ to cell `c`:
 
 - An *atomic* spec is used directly, deriving one item from the raw cell text.
 - A *delimited* spec splits the cell text by its delimiter and derives one item per
-  substring.
+  substring, passing each substring on unmodified.
 - A *compound* spec parses the cell text according to its delimiter structure and
   derives items from each component substring.
 - A *conditional* spec evaluates its condition against `c` and applies the

docs/rtl-reference.md

@@ -170,11 +170,11 @@ When the cell body contains **only** a condition and nothing else, `?` must be o
 Examples from the test suite:
 
 ```rtl
-[ [!BLANK? VAL] [!BLANK? (VAL : SR&C0->REC(1)){','}] ]+
+[ [!BLANK? VAL] [!BLANK? (VAL=TRIM : SR&C0->REC(1)){','}] ]+
 ```
 
 *(Task 45 — both cells of each row are guarded as non-blank; the second is also a delimited
-cell.)*
+cell, whose tokens are trimmed explicitly.)*
 
 ```rtl
 [ [BLANK] [] ]?
@@ -310,6 +310,30 @@ Splits the cell text by `"sep"` and derives one item per token.
 *(Task 45 — a cell like `"a,b,c"` yields three VAL items, each forming a record bound to the
 row key via `SR & C0`.)*
 
+**Splitting is verbatim.** Tokens reach the atom exactly as the split produced them, the
+same way an atomic or compound cell receives its raw text:
+
+- surrounding whitespace is **kept** — `"a, b"` yields `"a"` and `" b"`, not `"a"` and `"b"`;
+- empty tokens are **kept** — `"a,,b"` yields three items, the middle one an empty string,
+  and a trailing separator (`"a,b,"`) likewise yields a trailing empty item.
+
+This matches `pandas.Series.str.split`, which makes patterns over exploded columns
+expressible without post-processing.
+
+To trim, ask for it — add a [string extractor](#atomic--contspec) to the delimited atom.
+It is applied to each token separately:
+
+```rtl
+[(VAL=TRIM){','}]            // "a, b"    -> "a", "b"
+[(VAL=NORM){','}]            // "a,  b c" -> "a", "b c"
+```
+
+!!! warning "Changed in 0.5.0"
+
+    Before 0.5.0 every token was trimmed and empty tokens were silently dropped.
+    Patterns that relied on this need `=TRIM` (or `=NORM`) added to the delimited atom:
+    `(VAL){","}` → `(VAL=TRIM){","}`. Atomic and compound specifications are unaffected.
+
 ### Compound
 
 ```

grammar/UPSTREAM

@@ -1,4 +1,4 @@
-commit: 7a9b78974cc88c872bc994d6d38c69e810828ea6
-tag: v0.4.1
+commit: 035ff1a139e885e4cea85aa66a33e89a6b30f8c9
+tag: v0.5.0
 path: src/main/antlr4/ru/icc/regtab/rtl/RTL.g4
 sha256: 4fffbdf3f2dcb13935b8f062b5c6c55321de08cf130a4a28fa0845f5b48d68c0