Audit cleanup

.github/workflows/tests.yml

@@ -36,6 +36,12 @@ jobs:
 
       - uses: actions/checkout@v4
 
+      # Unit tests for run_test.py / run_coverage.py: stdlib-only Python,
+      # so the runner's own interpreter will do, and one OS is enough.
+      - name: Test the test tooling
+        if: runner.os == 'Linux'
+        run: python3 -m unittest discover -s tools -p 'test_*.py'
+
       - name: Install pixi
         id: pixi
         run: |

ARCHITECTURE.md

@@ -2,8 +2,9 @@
 
 Two public entry points, one shared front end:
 
-- **`Regex[pattern]`** — a single pattern, matched with Python-style
-  leftmost-first semantics.
+- **`Regex[pattern, flags]`** — a single pattern, matched with Python-style
+  leftmost-first semantics (`flags` defaults to none; it is compiled as a
+  leading inline group, `Regex._pat`).
 - **`RegexSet[patterns]`** — a multi-pattern database in the shape of
   Intel Hyperscan: scan once, report every pattern that matches and where.
 
@@ -573,9 +574,9 @@ Three constraints shape the code more than anything else:
 Every engine is differentially tested against the tagged Pike reference
 across LCG-generated inputs at chunk-boundary-adjacent lengths, including
 bytes ≥ 0x80. Set semantics are ground-truthed against CPython via
-`tools/set_oracle.py` — including `sweep_ctx`, a context-preserving variant
-that is sound for anchors and lookaround where the naive region-bounded
-sweep is not. Streaming is checked by exhaustive block/stream equivalence
+`tools/set_oracle.py` — `sweep_ctx`, a context-preserving sweep that is
+sound for anchors and lookaround where a naive region-bounded sweep is
+not. Streaming is checked by exhaustive block/stream equivalence
 over every 2- and 3-way chunk split.
 
 Benches live in `bench/bench.mojo` (single pattern) and

CLAUDE.md

@@ -8,7 +8,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 pixi run test          # run all tests (incremental: skips unchanged green files; --all forces)
 pixi run bench         # single-pattern (`Regex`) benchmark suite
 pixi run bench_all     # run all benchmarks
-pixi run coverage      # line coverage of emberregex/ over the suite (runs in the `cov` env, conda-forge LLVM; inline-aware counters by default, --gcov for the plain gcov pass; --missing, --opt-level, see run_coverage.py)
+pixi run coverage      # line coverage of emberregex/ over the suite (runs in the `cov` env, conda-forge LLVM; inline-aware counters; --missing, --opt-level, see run_coverage.py)
 pixi run coverage --check-baseline   # the CI gate: fail if coverage falls below coverage-baseline.json
 pixi run coverage --update-baseline  # rewrite that baseline from this run (commit it, and say why in the PR)
 pixi build             # build the conda package (pixi-build-mojo -> lib/mojo/emberregex.mojoc)
@@ -67,7 +67,7 @@ Engine selection happens at compile time via `comptime if` branches:
 
 **Eager DFA** (`static_dfa.mojo`) — the default DFA engine when `can_use_dfa` is true and `group_count == 0` (with captures the same tables serve the search verbs through `_use_dfa_span`; `match()` does not use them). Subset construction runs at **compile time** over the comptime NFA (byte-equivalence classes bound the per-state work); the transition table (`num_states x 256` ids in the narrowest element type that fits them — `Int8`/`Int16`/`Int32`, `edfa_id_dtype` — packed into a comptime string literal, `static_bytes.mojo`, which lowers to one static `c"..."` global; an `Array` global costs O(n²) to translate) and per-state match/EOL flag bytes materialize as constant data in the binary. The runtime engine is a pure table walk: no lazy construction, no hashing, no `raises` path, no runtime NFA copy in `__init__`. Handles the same three start contexts (pos 0 / after `\n` / mid-line) and EOL flags as the lazy DFA. Three structural passes run at comptime, in order: the DFA is **Hopcroft-minimized** over its byte classes (`_minimize`, initial partition by full flag byte, so states with different EOL flags never merge); states are then permuted so match states occupy ids `[0, num_match_states)` (the per-byte match test is an integer compare, not a flags load) followed by the word-conditional match states; and states that self-loop on all but ≤ 2 bytes (e.g. the `.*` state of `.*x`) are **accelerated** — the walkers SIMD-scan to the next exit byte instead of stepping the table (states carrying `EOL_AT_NEWLINE` or a word-conditional flag are excluded to keep per-byte match tracking), as are **regions** of start states whose rows agree outside a sparse exit set (the look-behind-split restart states of a `\b` pattern). Word boundaries: a state keeps the anchor as a pending member plus the word class of the byte that led to it, resolves it against the next byte's class on the transition, and carries `EDFA_MATCH_IF_WORD` / `_NONWORD` flags the walkers check before consuming (end of input is non-word). Patterns whose determinization exceeds `EDFA_STATE_CAP` (128) states are detected at compile time and stay on the lazy DFA.
 
-The classic table is leftmost-LONGEST (its states are sets), which is exactly what `match()` — Python `fullmatch`, a language-membership question — needs. The search-family verbs (`search`/`finditer`/`findall`/`replace`/`split`) run on a second table instead: the **leftmost-first DFA** (`static_lfdfa.mojo`, `build_lf_dfa`), whose states are priority-ORDERED lists of NFA states (DFS order of the epsilon closure, `out1` before `out2`) with truncation at MATCH and a `restart` bit that folds the unanchored start in as the lowest-priority threads. One unanchored forward walk (`lfdfa_find_end`, the same `edfa_walk_from` walker in the unanchored start states) yields Python's leftmost-first END directly — lazy quantifiers ride this lane too, `<.*?>` stops at its first `>` — and the **reverse DFA** (`static_rdfa.mojo`, `rdfa_find_start`) walks back from that end, never below the previous match end, for the start. The LF table reuses `_edfa_finish` (minimization, match-state permutation, acceleration) and the Sheng masks. A lazy pattern whose LF determinization overflows goes to the backtracker, never the lazy DFA (whose leftmost-longest walk is the wrong engine for `.*?`); a greedy one whose classic table fits but whose LF table overflows runs its search verbs on the backtracker too. The lazy DFA (`_use_lazy_dfa`) backs only patterns whose CLASSIC determinization overflowed, and it re-runs `_lf_end_at` for the leftmost-first end. Before the unanchored scan, `_lf_next_match` tries the first prefilter candidate **anchored** when a cheap anchored engine exists for the shape (`_lf_anchored_classic`: the classic table, when its longest end is the leftmost-first end — one greedy loop at most; `_lf_anchored_sbt`: the backtracker, for lazy patterns whose loops are all simple): a success needs no reverse walk, a failure hands the next candidate to the scan (one attempt per match, so the lane stays linear). When the lane has no filter/Teddy prefix but the pattern carries a **required inner literal** of ≥ 2 bytes (`extract_inner_literal`: `\w+\.txt` must contain ".txt"), `_use_rev_literal` memmems it first (`simd_find_literal_rare`): no occurrence means no match — one SIMD pass instead of the scan — and a comptime-bounded pre-literal gap moves the scan start to `lit_pos - max_offset` (Rust regex's ReverseSuffix/ReverseInner, effects (a)+(b); no leftward walking, so no backscan guard is needed). Tables travel as string literals (see `static_bytes.mojo`), so they are always static data; `EDFA_TABLE_MIN_BYTES` padding is kept for the element-count contract only.
+The classic table is leftmost-LONGEST (its states are sets), which is exactly what `match()` — Python `fullmatch`, a language-membership question — needs. The search-family verbs (`search`/`finditer`/`findall`/`replace`/`split`) run on a second table instead: the **leftmost-first DFA** (`static_lfdfa.mojo`, `build_lf_dfa`), whose states are priority-ORDERED lists of NFA states (DFS order of the epsilon closure, `out1` before `out2`) with truncation at MATCH and a `restart` bit that folds the unanchored start in as the lowest-priority threads. One unanchored forward walk (`edfa_match_at` over the LF table, whose start states are the unanchored ones) yields Python's leftmost-first END directly — lazy quantifiers ride this lane too, `<.*?>` stops at its first `>` — and the **reverse DFA** (`static_rdfa.mojo`, `rdfa_find_start`) walks back from that end, never below the previous match end, for the start. The LF table reuses `_edfa_finish` (minimization, match-state permutation, acceleration) and the Sheng masks. A lazy pattern whose LF determinization overflows goes to the backtracker, never the lazy DFA (whose leftmost-longest walk is the wrong engine for `.*?`); a greedy one whose classic table fits but whose LF table overflows runs its search verbs on the backtracker too. The lazy DFA (`_use_lazy_dfa`) backs only patterns whose CLASSIC determinization overflowed, and it re-runs `_lf_end_at` for the leftmost-first end. Before the unanchored scan, `_lf_next_match` tries the first prefilter candidate **anchored** when a cheap anchored engine exists for the shape (`_lf_anchored_classic`: the classic table, when its longest end is the leftmost-first end — one greedy loop at most; `_lf_anchored_sbt`: the backtracker, for lazy patterns whose loops are all simple): a success needs no reverse walk, a failure hands the next candidate to the scan (one attempt per match, so the lane stays linear). When the lane has no filter/Teddy prefix but the pattern carries a **required inner literal** of ≥ 2 bytes (`extract_inner_literal`: `\w+\.txt` must contain ".txt"), `_use_rev_literal` memmems it first (`simd_find_literal_rare`): no occurrence means no match — one SIMD pass instead of the scan — and a comptime-bounded pre-literal gap moves the scan start to `lit_pos - max_offset` (Rust regex's ReverseSuffix/ReverseInner, effects (a)+(b); no leftward walking, so no backscan guard is needed). Tables travel as string literals (see `static_bytes.mojo`), so they are always static data; `EDFA_TABLE_MIN_BYTES` padding is kept for the element-count contract only.
 
 **Lazy DFA** (`dfa.mojo`) — fallback DFA engine for patterns that blow the comptime state cap. Builds DFA states on demand from NFA epsilon closures and caches transitions in a 256-entry table per state. Single-pass O(n), no capture overhead. Handles simple line anchors directly: BOL/BOL_MULTILINE resolved in epsilon closure, EOL/EOL_MULTILINE checked at `\n` positions and end-of-input via precomputed flags. At `DFA_STATE_CAP` (4096) runtime states it clears the state cache and continues the walk (the current state is re-interned from its NFA set, the three start states are rebuilt); it only raises `DFA_STATE_CAP` — sending callers to the Pike VM — once it has cleared `MIN_CACHE_CLEARS` (3) times and is still consuming fewer than `MIN_BYTES_PER_STATE` (10) input bytes per state minted since the last clear.
 
@@ -119,7 +119,7 @@ current per-file durations.
   test can drift onto a different engine and keep passing — it just quietly
   stops testing what its filename claims. Pin with `_strategy.use_*`,
   `_use_lf_dfa`, `_use_onepass`, `_use_dfa_span`, `_SHENG_CAP`,
-  `accel_nib_states`, or the engine's own runtime state (`clear_count`).
+  `accel.nib_states`, or the engine's own runtime state (`clear_count`).
 - **Pin exclusively, never with a disjunction.**
   `assert_true(use_teddy or use_eager_dfa)` passes on either arm and therefore
   cannot catch a lane change. Use `comptime if HAS_FAST_BYTE_SHUFFLE:` with the
@@ -162,7 +162,7 @@ loops at 35-70 us per element op, spread over every lane. Rules:
   Int64]().unsafe_load[width=256]()` / `.unsafe_store(vec)` (a `List[Int]`
   holds 64-bit lanes). `List(fill=, length=)` + vector stores replaced 256
   appends per state and the per-cell `Array` copies
-  (`_edfa_finish`, `edfa_table_str`, `sheng_masks_str`, the set lanes).
+  (`_edfa_finish`, `table_bytes`, `sheng_masks_str`, the set lanes).
 - **A materialized table is a string literal, never an `Array`.**
   A comptime `Array` a walker `materialize`s becomes a global whose
   LLVM initializer is folded one `insertvalue` per element — O(n²): a

MULTIPATTERN_PLAN.md

@@ -37,7 +37,8 @@ list at the end is deliberate and specific, not a summary of intent.
   slots give leftmost SOM for free from the generation counter.
   `scan_spans` filters the stream to per-id leftmost non-overlapping
   spans. Verified by differentials between the two independent
-  implementations plus `tools/set_oracle.py::sweep_som`.
+  implementations plus `tools/set_oracle.py::sweep_som` (since replaced
+  by the context-preserving `sweep_ctx_som`).
   **Not done: SOM horizon modes (5.3)** — they are an offset-WIDTH
   tradeoff in stream state, and our offsets are plain `Int`, so there is
   nothing to trade until stream-state size is itself a problem.

README.md

@@ -75,7 +75,7 @@ mojo -I /path/to/emberregex your_file.mojo
 
 ## API Reference
 
-`Regex[pattern]` takes the pattern as a compile-time string literal. All parsing and NFA construction happen during compilation.
+`Regex[pattern, flags]` takes the pattern as a compile-time string literal and optional [`RegexFlags`](#flags). All parsing and NFA construction happen during compilation.
 
 ### Matching
 
@@ -160,7 +160,9 @@ var parts = re.split("one, two; three   four")
 
 ### Flags
 
-Pass flags as a second parameter, or use inline flag syntax in the pattern:
+Pass `RegexFlags` as a second parameter, or use inline flag syntax in the
+pattern. The parameter is sugar for the inline form: `Regex[p, flags]`
+compiles `(?flags)p` (after any leading `(*UTF8)` verb).
 
 ```mojo
 from emberregex import Regex, RegexFlags
@@ -173,16 +175,27 @@ re.match("HELLO").matched  # True
 var re2 = Regex["(?i)hello"]()
 re2.match("HeLLo").matched  # True
 
-# Multiline: ^ and $ match at \n boundaries
-var re3 = Regex["(?m)^\\w+"]()
-var lines = re3.findall("foo\nbar\nbaz")
-# lines: ["foo", "bar", "baz"]
+# Combined flags: `|` the values (same as inline `(?im)`)
+comptime IM = RegexFlags(RegexFlags.IGNORECASE) | RegexFlags(
+    RegexFlags.MULTILINE
+)
+var re3 = Regex["^[a-z]+", IM]()
+var lines = re3.findall("foo\nBAR\nbaz")
+# lines: ["foo", "BAR", "baz"]
 
 # Dotall: . matches \n
 var re4 = Regex["(?s)a.b"]()
 re4.match("a\nb").matched  # True
 ```
 
+| Flag | Inline | Effect |
+| --- | --- | --- |
+| `RegexFlags.IGNORECASE` | `(?i)` | case-insensitive matching |
+| `RegexFlags.MULTILINE` | `(?m)` | `^` and `$` also match at `\n` boundaries |
+| `RegexFlags.DOTALL` | `(?s)` | `.` also matches `\n` |
+| `RegexFlags.VERBOSE` | `(?x)` | whitespace and `#` comments in the pattern are ignored |
+| `RegexFlags.UNICODE` | `(?u)`, `(*UTF8)` | UTF-8 mode: `.` and classes match one codepoint (see below) |
+
 Bare inline flag groups like `(?i)` must appear **before any pattern
 content** (Python's rule — `a(?i)b` is a compile error). To apply flags
 to part of a pattern, use a scoped group: `a(?i:b)` or `(?-i:...)`.

bench/bench_compare.py

@@ -22,7 +22,7 @@
 NUMBER   = 10000  # calls per repetition (sum of many runs → less variance)
 BAR_COLS = 20     # width of the speedup bar
 
-# Must match comptime ITERS_PER_CALL in bench_static.mojo.
+# Must match comptime ITERS_PER_CALL in bench.mojo.
 # Each Mojo call() invocation runs the function this many times; divide to get per-call µs.
 MOJO_ITERS_PER_CALL = 100
 
@@ -44,15 +44,15 @@ def section(title):
     print(f"{'─'*72}")
 
 
-def _run_mojo_task(task: str) -> dict[str, float]:
-    """Run a pixi bench task and parse its markdown table output."""
+def run_mojo_static_benchmarks() -> dict[str, float]:
+    """Run `pixi run bench` (bench/bench.mojo) and parse its markdown table."""
     pixi_cmd = shutil.which("pixi")
     if pixi_cmd is None:
-        print(f"  [warning] pixi not found in PATH — skipping {task}")
+        print("  [warning] pixi not found in PATH — skipping bench")
         return {}
 
     result = subprocess.run(
-        [pixi_cmd, "run", task],
+        [pixi_cmd, "run", "bench"],
         capture_output=True, text=True,
         cwd=os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."),
     )
@@ -83,15 +83,10 @@ def _run_mojo_task(task: str) -> dict[str, float]:
     return timings
 
 
-def run_mojo_static_benchmarks() -> dict[str, float]:
-    """Run bench_static.mojo (Regex) via pixi."""
-    return _run_mojo_task("bench")
-
-
-def speedup_bar(ratio: float) -> str:
+def speedup_bar(ratio: float, cols: int = BAR_COLS) -> str:
     """Return a coloured bar string representing the speedup ratio."""
-    filled = min(int(ratio / 10.0 * BAR_COLS), BAR_COLS) if ratio <= 10 else BAR_COLS
-    bar = "█" * filled + "░" * (BAR_COLS - filled)
+    filled = min(int(ratio / 10.0 * cols), cols) if ratio <= 10 else cols
+    bar = "█" * filled + "░" * (cols - filled)
     if ratio >= 1.0:
         return f"\033[32m{bar}\033[0m"   # green = faster
     else:
@@ -108,20 +103,25 @@ def _ratio_str(ratio: float) -> str:
 def print_comparison(
     py: dict[str, float],
     static: dict[str, float],
+    labels=("Python", "Static", "Py/Stat  Bar (10x=full)"),
+    widths=(9, 9, 14, 50),  # baseline col, Regex col, missing-ratio dash, rule
+    summary="Python vs Static  — faster",
+    bar_cols=BAR_COLS,
 ):
-    """Print the two-column comparison table."""
+    """Print the two-column comparison table; ratio = baseline ÷ Regex."""
+    base_w, ours_w, dash_w, rule_w = widths
     all_names = list(py.keys())
     if not all_names:
-        print("  No Python results collected.")
+        print(f"  No {labels[0]} results collected.")
         return
 
     col_name = max(max(len(n) for n in all_names), 34)
 
     header = (
-        f"  {'Benchmark':<{col_name}}  {'Python':>9}  "
-        f"{'Static':>9}  {'Py/Stat':>7}  Bar (10x=full)"
+        f"  {'Benchmark':<{col_name}}  {labels[0]:>{base_w}}  "
+        f"{labels[1]:>{ours_w}}  {labels[2]}"
     )
-    sep = "  " + "─" * (col_name + 50)
+    sep = "  " + "─" * (col_name + rule_w)
 
     print()
     print(header)
@@ -133,36 +133,36 @@ def print_comparison(
         py_us   = py[name]
         stat_us = static.get(name)
 
-        stat_str = f"{stat_us:>9.3f}" if stat_us is not None else f"{'—':>9}"
+        stat_str = f"{stat_us:>{ours_w}.3f}" if stat_us is not None else f"{'—':>{ours_w}}"
 
         if stat_us is not None and stat_us > 0:
             ratio = py_us / stat_us
             ratio_str = _ratio_str(ratio)
-            bar = speedup_bar(ratio)
+            bar = speedup_bar(ratio, bar_cols)
             if ratio >= 1.0:
                 faster += 1
             else:
                 slower += 1
         else:
-            ratio_str = f"{'—':>14}"
-            bar = speedup_bar(0)
+            ratio_str = f"{'—':>{dash_w}}"
+            bar = speedup_bar(0, bar_cols)
             if stat_us is None:
                 missing += 1
 
         print(
-            f"  {name:<{col_name}}  {py_us:>9.3f}  "
+            f"  {name:<{col_name}}  {py_us:>{base_w}.3f}  "
             f"{stat_str}  {ratio_str}  {bar}"
         )
 
     print(sep)
     print(
-        f"  Python vs Static  — faster: {faster}  |  slower: {slower}"
+        f"  {summary}: {faster}  |  slower: {slower}"
         + (f"  |  no data: {missing}" if missing else "")
     )
 
 
 # ---------------------------------------------------------------------------
-# Python re benchmark suite  (names must match bench_static.mojo BenchIds)
+# Python re benchmark suite  (names must match bench.mojo BenchIds)
 # ---------------------------------------------------------------------------
 
 def run_python_benchmarks() -> dict[str, float]: