diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 294bde0a..da50e2cd 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -9,6 +9,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Hardening +- **Default-config benchmark follow-up: FullSync hypothesis falsified (P3b)** - single-knob + isolation via `SHARPCOREDB_PK_DEFAULT_VARIANT` (`async`, `bufferedio`, `novalidate`, + `noadaptive`, `hsinsert`, `plain`, `tuned`) disproved the earlier “FullSync dominates the + default gap” claim: the durability mode is only honored by GroupCommitWAL (off by default) and + the `async` variant measured the same. `NoEncryptMode` moves default UPDATE ~1.3x (144K vs + 109K), machine drift spans ~275-315K on SQLite itself. `docs/benchmarks/default-config-pk.md` + is corrected accordingly; the next step is a same-window interleaved A/B mode before any + further optimization is implemented. + - **Default-config benchmark published (P3)** - `docs/benchmarks/default-config-pk.md` records a median-of-3 fair-PK run where the SharpCoreDB arm uses a PURE default `DatabaseConfig` (NoEncryptMode=false, no harness flags). Honest result: the default path engages the Columnar diff --git a/docs/benchmarks/default-config-pk.md b/docs/benchmarks/default-config-pk.md index 03ec7bc7..b1739def 100644 --- a/docs/benchmarks/default-config-pk.md +++ b/docs/benchmarks/default-config-pk.md @@ -1,7 +1,7 @@ # Fair-PK benchmark with DEFAULT DatabaseConfig (P3 hardening) Run: `dotnet run --project tests/benchmarks/SharpCoreDB.Benchmarks.Comparative -- -c Release -- --pk-default` -Date: 2026-09-04 · Machine: local dev box · median of 3 runs per phase. +Date: 2026-09-04 · Machine: local dev box · median of 3 runs per phase unless noted. The SharpCoreDB arm uses a **pure default** `DatabaseConfig` (only the engine type is pinned to `AppendOnly` — `NoEncryptMode` stays at its default `false`, no page-cache/query-cache/durability @@ -17,23 +17,35 @@ contiguous UPDATE/DELETE paths do engage (verified separately by `DefaultEngineS | SQLite | 188,247 | 107,122 | 291,414 | 389,120 | | gap vs SQLite | 1.7x | 1.6x | **3.5x** | **4.2x** | -For comparison, the same schema with the tuned benchmark config (NoEncryptMode + Async WAL + larger -batches) measures UPDATE ~245K ops/s and DELETE ~172K ops/s (gaps ~1.2x / ~2.1x). +For comparison, the tuned benchmark config measured earlier in the day (UPDATE ~245K ops/s and +DELETE ~172K ops/s, gaps ~1.2x / ~2.1x). + +## Knob isolation (diagnostic single-knob variants, single-shot and median) + +The harness supports `SHARPCOREDB_PK_DEFAULT_VARIANT=` for the default arm: +`async`, `bufferedio`, `novalidate`, `noadaptive`, `hsinsert`, `plain` (= full tuned knob set with +`NoEncryptMode=true`), `tuned` (= full tuned knob set, `NoEncryptMode=false`). + +Observed (2026-09-04 afternoon): + +| variant | UPDATE ops/s | notes | +|---|---:|---| +| pure default | ~84K (median) | reference | +| `async` (WalDurabilityMode) | ~74K (single) | **disproves the FullSync hypothesis** — the mode is only honored by GroupCommitWAL (default off) | +| `bufferedio` / `novalidate` | ~74-76K (single) | within noise | +| `tuned` (no NoEncryptMode) | ~109K (median) | knob set alone is not the gap | +| `plain` (= tuned + NoEncryptMode=true) | ~145K (median) | `NoEncryptMode` moves UPDATE ~1.3x; still below the morning's ~212-245K | + +Machine drift is significant: SQLite's own UPDATE varied 275K-315K across these runs. Single-knob +deltas below ~1.3x are not reliably attributable outside a same-window interleaved A/B. ## Honest conclusion -The out-of-the-box default is **correct and engages the fast paths, but is not yet at the tuned -throughput**: UPDATE/DELETE land ~3.5-4.2x behind SQLite (vs ~1.2-2.1x tuned). The dominant -difference is the **default `WalDurabilityMode.FullSync`** (per-commit flush) versus the benchmark -arm's `Async`; the per-batch commit cost explains most of the gap. Fast-path counters prove the -contiguous code is running — the throughput is limited by durability flushing, not by resolution. - -## Recommendation / follow-up - -1. Do **not** silently weaken the durability default (`FullSync` is the safe choice for - production). Instead, optimize the **FullSync commit flush** path (fewer/single flush per - commit, group-commit of the commit markers + overwrites that already batch per page) so a - synchronous commit costs a few ms instead of tens of ms. -2. Re-run this `--pk-default` harness after that change and require the default-config UPDATE/DELETE - gap to move from ~3.5-4.2x toward ~2x before the release-cut. -3. Keep this file updated with the latest median-of-3 numbers. +1. The earlier claim that the default `WalDurabilityMode.FullSync` dominates the gap is **wrong** + (disproven by the `async` variant). Do **not** implement a “FullSync commit-flush optimization” + based on it. +2. The default path is correct and engages the fast paths; part of the remaining gap correlates + with `NoEncryptMode` (record/at-rest toggles and file-format decisions), part is machine drift. +3. Next step: add a **same-window interleaved A/B** mode to this harness (arms round-robin within + one process) so default-vs-tuned deltas are attributable, then re-open the optimization only on + a measured knob. diff --git a/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/Program.cs b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/Program.cs index aa6414a0..aa3d5f8a 100644 --- a/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/Program.cs +++ b/tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/Program.cs @@ -361,11 +361,11 @@ static void RunInsertMicroBenchmark() Console.WriteLine($" SQL/Direct overhead: {(sqlMedian / directMedian):F2}x"); } - static DatabaseConfig BuildConfig(SharpCoreDB.Interfaces.StorageEngineType engineType, bool fixedWidth = false) + static DatabaseConfig BuildConfig(SharpCoreDB.Interfaces.StorageEngineType engineType, bool fixedWidth = false, bool noEncrypt = true) { return new DatabaseConfig { - NoEncryptMode = true, + NoEncryptMode = noEncrypt, StorageEngineType = engineType, // The fair PK comparison intentionally isolates the record-layout variable: the legacy // arm opts out of the AutoFixedWidthRecords default so it measures true variable-length @@ -874,9 +874,33 @@ static BenchmarkResult RunSharpCoreDBPk( var sp = services.BuildServiceProvider(); var factory = sp.GetRequiredService(); - var config = useDefaultConfig - ? new DatabaseConfig { StorageEngineType = engineType } - : BuildConfig(engineType, fixedWidth); + DatabaseConfig config; + if (useDefaultConfig) + { + var variant = Environment.GetEnvironmentVariable("SHARPCOREDB_PK_DEFAULT_VARIANT")?.ToLowerInvariant(); + config = variant switch + { + "async" => new DatabaseConfig { StorageEngineType = engineType, WalDurabilityMode = SharpCoreDB.Services.DurabilityMode.Async }, + "bufferedio" => new DatabaseConfig { StorageEngineType = engineType, UseBufferedIO = true }, + "novalidate" => new DatabaseConfig + { + StorageEngineType = engineType, + SqlValidationMode = SharpCoreDB.Services.SqlQueryValidator.ValidationMode.Disabled, + StrictParameterValidation = false, + }, + "noadaptive" => new DatabaseConfig { StorageEngineType = engineType, EnableAdaptiveWalBatching = false }, + "hsinsert" => new DatabaseConfig { StorageEngineType = engineType, HighSpeedInsertMode = true }, + // "plain" == the tuned harness config with NoEncryptMode=true (BuildConfig default); + // "tuned" == the same knob set but NoEncryptMode=false (isolates that flag). + "plain" => BuildConfig(engineType, fixedWidth: true), + "tuned" => BuildConfig(engineType, fixedWidth: true, noEncrypt: false), + _ => new DatabaseConfig { StorageEngineType = engineType }, + }; + } + else + { + config = BuildConfig(engineType, fixedWidth); + } using var db = (SharpCoreDB.Database)factory.Create( dbPath: dbPath, @@ -979,6 +1003,12 @@ data TEXT /// private static BenchmarkResult RunPkMedian(Func arm, int reps = 3) { + // Allow quick single-shot profiling via SHARPCOREDB_BENCH_REPS (diagnostic only). + if (int.TryParse(Environment.GetEnvironmentVariable("SHARPCOREDB_BENCH_REPS"), out int envReps) && envReps > 0) + { + reps = envReps; + } + var runs = new List(reps); for (int r = 0; r < reps; r++) {