refactor(db): replace the kysely Migrator with Flyway - #401
refactor(db): replace the kysely Migrator with Flyway#401nourshoreibah wants to merge 6 commits into
Conversation
This PR contains a database migration(no newly added migration files in this PR) It will be applied to the production database automatically when this PR merges, before the new lambda code is deployed. Please confirm before requesting review:
|
|
Database Types Check Complete The database schema files were modified, but the regenerated TypeScript types are identical to the existing ones. No changes were needed and the type definitions are already up to date. |
1 similar comment
|
Database Types Check Complete The database schema files were modified, but the regenerated TypeScript types are identical to the existing ones. No changes were needed and the type definitions are already up to date. |
Aurora DSQL has no advisory locks and allows one DDL statement per transaction. Kysely's PostgresAdapter takes a pg_advisory_xact_lock on every run and declares supportsTransactionalDdl, so Migrator wraps the whole run in one transaction. Neither is configurable through Migrator options, so the migrator has to go before the engine can. Flyway is swapped in against RDS first, on its own, so a failure here is unambiguous. It covers both engines from one migration corpus, and it replaces machinery we were maintaining: per-migration checksums instead of the bespoke sha256 schema-comment fingerprint, and `flyway validate` alongside the git-based immutability guard. - migrations are renamed to Flyway's V<version>__<description> form. Pure renames, no content change - flyway.sh is the single place Flyway is configured. It resolves the connection exactly as src/config.ts does and runs the pinned image when no flyway is on PATH, so docker stays the only prerequisite - production is adopted with baselineOnMigrate at the last kysely-era version rather than by replaying the baseline - testkit stamps flyway_schema_history instead of kysely_migration; checksums are left NULL, which Flyway treats as "not mine to validate" - the schema-comment fingerprint stays, but Flyway does not write it, so `db fingerprint` does and the Makefile and CI call it before generating types - the immutability guard is now keyed by version rather than by path, which is what production's schema history actually pins - dsql-lint runs advisory-only on newly added migrations. The corpus still contains the plpgsql that a later migration dropped, so linting all of it could only ever fail Verified against PostgreSQL 16: a from-scratch `flyway migrate` produces a schema byte-identical to applying the same files directly (pg_dump diff), the run is idempotent, `flyway validate` passes, a pre-Flyway schema is adopted without re-running anything, an empty target is refused, and the generated types are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#400 gained V20260907213524__project_rollup_bump_reports_hit before it merged, so it arrived in the pre-Flyway naming form. Rename it and move the adoption baseline up to it -- production has already applied it, so Flyway must baseline over it rather than re-run it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b62c895 to
9648c4c
Compare
|
Database Types Check Complete The database schema files were modified, but the regenerated TypeScript types are identical to the existing ones. No changes were needed and the type definitions are already up to date. |
Review of #401 found the adoption baseline is not just a one-time marker but a permanent floor, and that the deploy guard traded away a protection the old one had. Both fail silently, which is the worst shape for this. - BASELINE_VERSION is a floor, not a marker. Flyway reports any migration at or below it "Below Baseline" and never runs it: nothing pending, validate passes, migrate exits 0, column absent. CI builds from an empty schema where baselining never fires, so a PR scaffolded earlier the same day merges green and production quietly misses the change. `checks` now rejects a new migration whose version is not strictly above BASELINE_VERSION. It skips versions already on main so a rename is never mistaken for a new migration. - The out-of-order step called that case "Harmless". True above the baseline, where outOfOrder applies the migration late; false at or below it, where it is never applied at all. Reworded, and the hard failure above covers the rest. Same correction in db/README.md. - baselineOnMigrate is now off unless asked for. It only ever fires against a populated schema with no history table, which on production means a database we do not recognise -- a restored snapshot or a clone -- and adopting one silently is how you migrate the wrong thing. The deploy workflow gains an adopt_baseline input for the one-time cutover; compose sets it for local dev, where a pre-Flyway database should just be adopted. - The deploy guards used `jq -e` inside `if`. Malformed output makes jq exit non-zero, which reads as "the dangerous condition is absent" -- the opposite of the documented safe fallthrough. The preflight now validates the JSON once and fails on it, and the guards compare plain strings. - testkit applies the migration files directly, so nothing substituted Flyway's ${async} placeholder: the first migration written per the README would break `npm run reset`, `make db-reset` and every lambda's ensureSchema() with a postgres syntax error, while CI stayed green because CI goes through Flyway. testkit substitutes the placeholders itself now, and throws on an unknown one rather than silently deleting it. - The docker fallback leaned on `--network=host`, which Docker Desktop ignores unless it is switched on, and on `localhost`, which can resolve to ::1 inside the container before the host's IPv4. Added --add-host=host.docker.internal and a rewrite of loopback hostnames. Verified on PostgreSQL 16: the below-baseline skip reproduces and the new check catches it while leaving an above-baseline migration alone; adoption is refused by default and works with the opt-in; a fresh empty database still migrates in full; the guard behaves correctly across healthy/empty/untracked/adopting/ drifted/malformed; ${async} now applies through both testkit and Flyway and an unknown placeholder throws; schema still byte-identical and types unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unconditional rewrite broke Auto-regenerate DB Types: that workflow runs postgres on the runner itself, bound to loopback, so host.docker.internal resolves to the docker0 gateway and the connection is refused. On a native daemon --network=host is real and localhost is already correct; it is only Docker Desktop, which ignores --network=host unless switched on, that needs the gateway. Gate the rewrite on the daemon actually being Desktop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Database Types Check Complete The database schema files were modified, but the regenerated TypeScript types are identical to the existing ones. No changes were needed and the type definitions are already up to date. |
Stacked on #400. PR 2 of the Aurora DSQL migration — the migrator swap, still entirely on RDS, so a failure here is unambiguous.
Why
Aurora DSQL has no advisory locks and allows one DDL statement per transaction. Kysely's
PostgresAdaptertakes apg_advisory_xact_lockon every run and declaressupportsTransactionalDdl = true, soMigratorwraps the whole run in one transaction. Neither is configurable throughMigratoroptions, so the migrator has to go before the engine can.Flyway also covers both engines from one migration corpus, and it deletes machinery we were maintaining by hand.
What changed
V<version>__<description>form. Pure renames — the diffstat shows 0 changed lines for all 8.db/flyway.shis the single place Flyway is configured. It resolves the connection exactly assrc/config.tsdoes (including the RDS CA bundle) and runs the pinned image when noflywayis onPATH, so docker stays the only prerequisite. Credentials go through the environment, never argv.baselineOnMigratewithbaselineVersionpinned at the last kysely-era migration.testkitstampsflyway_schema_historyinstead ofkysely_migration. Checksums are leftNULL, which Flyway treats as "not mine to validate" — reimplementing its CRC32 would be a second thing to keep in step for no benefit.npm run fingerprintdoes, and the Makefile and CI call it before generating types.dsql-lintruns advisory-only on newly added migrations, with${async}substituted first (it parses SQL, and${async}is not SQL).Verified against PostgreSQL 16
flyway migrateproduces a schema byte-identical to applying the same files directly — thepg_dump --schema-onlydiff is emptyflyway validatepasses;flyway infoshows nothing pendingtestkitround-trips: rebuild leaves 8 history rows, the fingerprint verifies,resetDatadoes not truncate the history table, rollups stay consistentflyway_schema_historyexcluded from codegenDeviations from the plan, and why
org.flywaydb:flyway-database-dsqldoes not exist as an installable artifact. It is not on Maven Central and not bundled in the Flyway CLI 13.5.0. It lives only as source inflyway/flyway-community-db-support(added 2026-08-05, parent pinned to Flyway 10.26.0 while the CLI is 13.x). AWS'ssoftware.amazon.dsql:aurora-dsql-flyway-support2.0.0 is published, but its README declares itself deprecated in favour of the module you cannot install. This is a PR 3 decision, not a PR 2 blocker — nothing here needs it — so the DSQL jars are not added to the image yet.executeInTransaction=falsesidecar. Our baseline creates 7 tables plus indexes in one file, and the corpus still contains the plpgsql thatV20260906215733drops — so PR 3 needs a squashed DSQL baseline, not a replay of history.SERIAL→CREATE SEQUENCE+INT DEFAULT nextvalrewrite moved to PR 3. In PR 2 it would break this PR's own acceptance criterion (schema identical to the kysely-built one) and silently breakTRUNCATE ... RESTART IDENTITYin testkit, whose compensating fix the plan puts in PR 3.flyway validatecannot replace the git immutability guard. CI migrates a fresh database, so there are no prior checksums to violate; the only PR-time protection is the git one. Kept and improved rather than deleted.kysely-codegen. Moved todevDependencies.dsql-lintgating is advisory. Linting the whole corpus can only ever fail, because history contains statements a later migration already dropped.Two plan verification items are now settled empirically:
INT ... DEFAULT nextval(seq)withCREATE SEQUENCE ... CACHE 1passesdsql-lintclean, and${async}placeholders break its parser.Note
The
checksandmigratejobs now pull the Flyway image (~392 MB). Worth caching if it shows up in CI times.🤖 Generated with Claude Code