feat: safe scans, live feedback, tuning advice and relationship-shaped seeding by machado144 · Pull Request #43 · AxeForging/seedstorm

machado144 · 2026-09-17T16:08:05Z

Safe reads on real databases

Every read (counts, compare, snapshot, introspect, relationships) runs read-only with a 2s lock timeout; Ctrl+C / cancel stops the query on the server (MySQL KILL QUERY)
Unknown counts are ?, never 0; mirror never fills an unknown target
Connections can be marked production: writes need --allow-production (CLI) or the typed label (web)
Postgres: FKs read from pg_constraint, partitioned tables counted once and seeded inside their bounds

Feedback and reliability

Web run strip on every page: phase, progress, elapsed, running / quiet / reconnecting / lost; leave a page and come back to a live run
Failures name side, phase and table and list what completed (target · write · orders: …); a panic in one job never stops serve
CLI: progress lines for counting/introspection/clone, 10s connect timeout, exit codes 1 / 70 / 130; TUI seed/gaps show rows, rate and ETA
Workspace draws the graph first and fills cached counts after; form settings remembered per connection (destructive toggles never)

Tuning

seedstorm tune and the Recommend dialog (vCPU, memory, storage type and size) with reasons and a disk check; --workers auto
Runs lower writers when the server has few free connections; replica targets refused, shared-server pairs flagged

Relationship shapes

snapshot / introspect / compare --relationships, Analyze relationships in the workspace (edge badges), Compare relationships (drift) — index gate, 60s per key, cheapest first
Profiles gain relationships: (version 2); seed --shape-rows, mirror --shape-like-source; no parent above max, exact NULL share, achieved shape logged after the run

RELATIONSHIP               SOURCE                TARGET               STATUS
orders.user_id → users     5.15 / 15 / 34 · 19%  2.00 / 2 / 2 · 0%    differs

Compatibility

Without relationships:, --seed output is byte-identical to main (6 recorded files)
Counts files without relationships stay version: 1; existing flags and JSON fields unchanged

Bugs found while building this (each reproduced by a test first)

Job streams dropped events when a job wrote faster than the browser read; the stream now refills gaps from the job's own list
A run capped the connection pool it was handed and never gave it back; concurrent runs now share it and the last one out restores it
Datetime keys collided on MySQL, which rounds fractional seconds on insert
MySQL read_only no longer refuses a mirror: only super_read_only (or a Postgres standby) blocks every write
Two evals passed for the wrong reason: the full-disk one now requires the database to report a full disk, the estimate one no longer races Postgres statistics

Tests

Unit, integration (both engines: read safeguards, fault injection, shapes, memory bounds), e2e (5 new journeys)
New loadsim CI job: Cloud SQL-shaped containers (1 vCPU/629MB, 2 vCPU/7.5GB) asserting outcomes, not timings

- Read-only scans with server-side timeouts, MySQL KILL QUERY on cancel - Production flag: writes need the label typed back (web, CLI) - Panics fail one run, not the process; failures name side, phase, table - Unknown counts are never zero; partitioned tables and FKs to non-key columns seed correctly - Run strip with reattach, quiet/lost states; TUI and CLI progress - Workspace draws before counting; settings survive navigation

- tune command, --workers auto and a Recommend dialog: writers and generators from detected limits plus size and disk, with reasons - Seeds lower writers when the server lacks free connections - Snapshot counts from the workspace; calibrate another database from a file - Full disk and dropped connections are explained in run errors - Resource-limited evals shaped like small managed instances

- relations.Scan: per-FK degree histogram, percentiles, index gate, estimates, per-edge timeout - snapshot v2 with relationships; snapshot/compare/introspect --relationships - compare shows per-relationship drift

- workspace job measures shapes; graph edges show avg/max as each finishes - compare page relationships step with per-key drift - counts exports include relationships on request

- mirror plan notes when source and target share a server - a read-only replica target is refused before planning - relationship scans say whether they read a replica

- profiles gain relationships (version 2); import them from a counts file - degree dealer keeps each parent within max; exact NULL share; --shape-rows - mirror --shape-like-source and web option; achieved shape logged after runs

- command reference: tune, --relationships, --shape-rows, production flags, exit codes - profiles: relationships section; development: loadsim, fault injection, CI table - CI: loadsim job; manual loadsim-measure workflow

github-actions · 2026-09-17T16:08:15Z

✅ PR title follows the required format

Current title: feat: safe scans, live feedback, tuning advice and relationship-shaped seeding

MySQL rounds fractional seconds on insert, so keys in adjacent seconds could collide in the database while the key guard saw them as distinct.

- require no space left on device in the database log and no OOM kill - silence MySQL driver retries while loadsim containers start

Postgres reports inserts to its statistics asynchronously: the eval now waits for them, checks they are used as an estimate, then resets them and checks the exact fallback.

MySQL read_only still lets a user with SUPER write, so it now warns; super_read_only and a Postgres standby are refused.

- a job writing faster than the browser reads no longer loses events: the stream refills gaps from the job's own list - a run restores the connection limit of the pool it was handed

Two runs on one pool (two web tabs on one connection) each reserve their connections; the last one out restores what the pool allowed before.

A mirror shaped like its source dropped keys it cannot shape (junction keys, key columns) without saying so: on Keycloak 38 of 67. Each is now reported with its reason, in the CLI log, the job log and the plan.

- integration: a mirror on a schema with junction keys, key columns and a self-reference must account for every measured key (mutation checked) - unit: applied + reported covers each measured key exactly once, each with a reason - a self-reference now says so instead of reporting as not measured

A parent table larger than the key pool took every child onto the sampled parents, which raised max far above the shape. Children are now dealt a round at a time per sample, and the run says the pool is smaller than the table.

600k parents and 1.6M children: the shape lands close (avg 3 → 3.4, max 8 → 10, without children 10% → 22%) and the run says the pool is smaller than the table. Docs now give these numbers instead of promising exact.

machado144 added 9 commits September 17, 2026 16:54

feat: measure and compare relationship shapes

1c5eaa6

- relations.Scan: per-FK degree histogram, percentiles, index gate, estimates, per-edge timeout - snapshot v2 with relationships; snapshot/compare/introspect --relationships - compare shows per-relationship drift

feat: analyze and compare relationships in the web UI

ebc3b0a

- workspace job measures shapes; graph edges show avg/max as each finishes - compare page relationships step with per-key drift - counts exports include relationships on request

feat: warn about shared servers and refuse replica targets

7b60c88

- mirror plan notes when source and target share a server - a read-only replica target is refused before planning - relationship scans say whether they read a replica

feat: seed foreign keys with measured relationship shapes

6c2b3bc

- profiles gain relationships (version 2); import them from a counts file - degree dealer keeps each parent within max; exact NULL share; --shape-rows - mirror --shape-like-source and web option; achieved shape logged after runs

docs: document safe reads, tuning, relationships and loadsim

28855df

- command reference: tune, --relationships, --shape-rows, production flags, exit codes - profiles: relationships section; development: loadsim, fault injection, CI table - CI: loadsim job; manual loadsim-measure workflow

docs: benchmark relationship shapes and small-instance writers

e22f780

test: describe loadsim evals without numbered references

03c8da4

machado144 added 11 commits September 17, 2026 18:21

fix: compare datetime keys as MySQL stores them

41d53f9

MySQL rounds fractional seconds on insert, so keys in adjacent seconds could collide in the database while the key guard saw them as distinct.

test: prove the full-disk eval filled the disk

79cb27d

- require no space left on device in the database log and no OOM kill - silence MySQL driver retries while loadsim containers start

test: make the estimate fallback eval independent of stats timing

c4969fc

Postgres reports inserts to its statistics asynchronously: the eval now waits for them, checks they are used as an estimate, then resets them and checks the exact fallback.

fix: refuse only targets that refuse every write

db96400

MySQL read_only still lets a user with SUPER write, so it now warns; super_read_only and a Postgres standby are refused.

fix: keep job streams and borrowed pools intact

1f83388

- a job writing faster than the browser reads no longer loses events: the stream refills gaps from the job's own list - a run restores the connection limit of the pool it was handed

style: format the loadsim measure record

b808188

fix: share a pool's limit between concurrent runs

0910ef0

Two runs on one pool (two web tabs on one connection) each reserve their connections; the last one out restores what the pool allowed before.

feat: name the relationships a schema cannot shape

e778325

A mirror shaped like its source dropped keys it cannot shape (junction keys, key columns) without saying so: on Keycloak 38 of 67. Each is now reported with its reason, in the CLI log, the job log and the plan.

fix: deal shapes in rounds when the parent pool is a sample

06fa548

A parent table larger than the key pool took every child onto the sampled parents, which raised max far above the shape. Children are now dealt a round at a time per sample, and the run says the pool is smaller than the table.

test: pin shaped seeding above the key-pool limit

778b59d

600k parents and 1.6M children: the shape lands close (avg 3 → 3.4, max 8 → 10, without children 10% → 22%) and the run says the pool is smaller than the table. Docs now give these numbers instead of promising exact.

machado144 merged commit 00515b9 into main Sep 17, 2026
10 checks passed

Navigation Menu

feat: safe scans, live feedback, tuning advice and relationship-shaped seeding - #43

feat: safe scans, live feedback, tuning advice and relationship-shaped seeding#43
machado144 merged 20 commits into
mainfrom
feat/safe-scans-feedback-shapes

machado144 commented Sep 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

machado144 commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Safe reads on real databases

Feedback and reliability

Tuning

Relationship shapes

Compatibility

Bugs found while building this (each reproduced by a test first)

Tests

Uh oh!

github-actions Bot commented Sep 17, 2026

✅ PR title follows the required format

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

machado144 commented Sep 17, 2026 •

edited

Loading