Skip to content

refactor(db): replace the kysely Migrator with Flyway - #401

Draft
nourshoreibah wants to merge 6 commits into
mainfrom
feat/flyway-migrator
Draft

refactor(db): replace the kysely Migrator with Flyway#401
nourshoreibah wants to merge 6 commits into
mainfrom
feat/flyway-migrator

Conversation

@nourshoreibah

Copy link
Copy Markdown
Collaborator

Stacked on #400. PR 2 of the Aurora DSQL migration — the migrator swap, still entirely on RDS, so a failure here is unambiguous.

Why

Aurora DSQL has no advisory locks and allows one DDL statement per transaction. Kysely's PostgresAdapter takes a pg_advisory_xact_lock on every run and declares supportsTransactionalDdl = true, so Migrator wraps the whole run in one transaction. Neither is configurable through Migrator options, so the migrator has to go before the engine can.

Flyway also covers both engines from one migration corpus, and it deletes machinery we were maintaining by hand.

What changed

  • Migrations renamed to Flyway's V<version>__<description> form. Pure renames — the diffstat shows 0 changed lines for all 8.
  • db/flyway.sh is the single place Flyway is configured. It resolves the connection exactly as src/config.ts does (including the RDS CA bundle) and runs the pinned image when no flyway is on PATH, so docker stays the only prerequisite. Credentials go through the environment, never argv.
  • Production is adopted, not replayed: baselineOnMigrate with baselineVersion pinned at the last kysely-era migration.
  • testkit stamps flyway_schema_history instead of kysely_migration. Checksums are left NULL, which Flyway treats as "not mine to validate" — reimplementing its CRC32 would be a second thing to keep in step for no benefit.
  • The schema-comment fingerprint stays but Flyway does not write it, so npm run fingerprint does, and the Makefile and CI call it before generating types.
  • The immutability guard is keyed by version rather than by path, which is what production's schema history actually pins. It now catches content edits, renames and deletions separately.
  • dsql-lint runs advisory-only on newly added migrations, with ${async} substituted first (it parses SQL, and ${async} is not SQL).

Verified against PostgreSQL 16

  • from-scratch flyway migrate produces a schema byte-identical to applying the same files directly — the pg_dump --schema-only diff is empty
  • re-running is a no-op; flyway validate passes; flyway info shows nothing pending
  • a pre-Flyway schema (tables, no history table) is adopted by baselining and re-runs nothing
  • an empty target is refused by the deploy preflight; a tracked-but-drifted one is too
  • testkit round-trips: rebuild leaves 8 history rows, the fingerprint verifies, resetData does not truncate the history table, rollups stay consistent
  • generated types are unchanged, with flyway_schema_history excluded from codegen
  • the new immutability guard passes this PR and rejects an edit, a rename and a deletion

Deviations from the plan, and why

  1. org.flywaydb:flyway-database-dsql does not exist as an installable artifact. It is not on Maven Central and not bundled in the Flyway CLI 13.5.0. It lives only as source in flyway/flyway-community-db-support (added 2026-08-05, parent pinned to Flyway 10.26.0 while the CLI is 13.x). AWS's software.amazon.dsql:aurora-dsql-flyway-support 2.0.0 is published, but its README declares itself deprecated in favour of the module you cannot install. This is a PR 3 decision, not a PR 2 blocker — nothing here needs it — so the DSQL jars are not added to the image yet.
  2. The DSQL module does not auto-split DDL and DML into separate transactions. The plan recorded that it does. Its README says the opposite: Flyway commits a whole file as one transaction, so each migration file needs exactly one DDL statement, or an executeInTransaction=false sidecar. Our baseline creates 7 tables plus indexes in one file, and the corpus still contains the plpgsql that V20260906215733 drops — so PR 3 needs a squashed DSQL baseline, not a replay of history.
  3. The SERIALCREATE SEQUENCE + INT DEFAULT nextval rewrite moved to PR 3. In PR 2 it would break this PR's own acceptance criterion (schema identical to the kysely-built one) and silently break TRUNCATE ... RESTART IDENTITY in testkit, whose compensating fix the plan puts in PR 3.
  4. flyway validate cannot replace the git immutability guard. CI migrates a fresh database, so there are no prior checksums to violate; the only PR-time protection is the git one. Kept and improved rather than deleted.
  5. kysely could not be removed entirely — it is a required peer dependency of kysely-codegen. Moved to devDependencies.
  6. dsql-lint gating is advisory. Linting the whole corpus can only ever fail, because history contains statements a later migration already dropped.

Two plan verification items are now settled empirically: INT ... DEFAULT nextval(seq) with CREATE SEQUENCE ... CACHE 1 passes dsql-lint clean, and ${async} placeholders break its parser.

Note

The checks and migrate jobs now pull the Flyway image (~392 MB). Worth caching if it shows up in CI times.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

This PR contains a database migration

(no newly added migration files in this PR)

It will be applied to the production database automatically when this PR merges, before the new lambda code is deployed. Please confirm before requesting review:

  • Applied and tested locally. cd apps/backend && make migrate, then run the affected lambda's tests (cd apps/backend/lambdas/<name> && npm test). make show-migrations shows what applied.
  • Safe for the code that is live right now. During the deploy window -- and indefinitely if the deploy fails -- the currently deployed lambdas run against your new schema. Additive changes (CREATE TABLE, nullable ADD COLUMN, CREATE INDEX) are fine in one PR. DROP COLUMN, renames, ADD COLUMN NOT NULL with no default, and new UNIQUE/CHECK/FOREIGN KEY constraints need two merged PRs -- see the expand/contract rules in apps/backend/db/README.md.
  • Kept separate from unrelated changes. A migration PR should ideally contain the migration, the code that needs it, and nothing else. It changes production state, it is the one thing here that redeploying cannot roll back, and a reviewer should be able to see the whole schema change without scrolling past unrelated work.
  • No already-merged migration was edited. Fix an old migration by adding a new one; there is no down.

shared/types/db-types.d.ts is regenerated and pushed to this branch automatically -- don't hand-edit it. Expect one red migrations-fresh check before that commit lands.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Database Types Check Complete

The database schema files were modified, but the regenerated TypeScript types are identical to the existing ones.

No changes were needed and the type definitions are already up to date.

1 similar comment
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Database Types Check Complete

The database schema files were modified, but the regenerated TypeScript types are identical to the existing ones.

No changes were needed and the type definitions are already up to date.

Base automatically changed from feat/rollups-in-application-code to main September 8, 2026 02:44
@nourshoreibah nourshoreibah added the no-review The PR review bot won't run label Sep 8, 2026
nourshoreibah and others added 3 commits September 7, 2026 22:48
Aurora DSQL has no advisory locks and allows one DDL statement per
transaction. Kysely's PostgresAdapter takes a pg_advisory_xact_lock on every
run and declares supportsTransactionalDdl, so Migrator wraps the whole run in
one transaction. Neither is configurable through Migrator options, so the
migrator has to go before the engine can.

Flyway is swapped in against RDS first, on its own, so a failure here is
unambiguous. It covers both engines from one migration corpus, and it replaces
machinery we were maintaining: per-migration checksums instead of the bespoke
sha256 schema-comment fingerprint, and `flyway validate` alongside the
git-based immutability guard.

- migrations are renamed to Flyway's V<version>__<description> form. Pure
  renames, no content change
- flyway.sh is the single place Flyway is configured. It resolves the
  connection exactly as src/config.ts does and runs the pinned image when no
  flyway is on PATH, so docker stays the only prerequisite
- production is adopted with baselineOnMigrate at the last kysely-era version
  rather than by replaying the baseline
- testkit stamps flyway_schema_history instead of kysely_migration; checksums
  are left NULL, which Flyway treats as "not mine to validate"
- the schema-comment fingerprint stays, but Flyway does not write it, so
  `db fingerprint` does and the Makefile and CI call it before generating types
- the immutability guard is now keyed by version rather than by path, which is
  what production's schema history actually pins
- dsql-lint runs advisory-only on newly added migrations. The corpus still
  contains the plpgsql that a later migration dropped, so linting all of it
  could only ever fail

Verified against PostgreSQL 16: a from-scratch `flyway migrate` produces a
schema byte-identical to applying the same files directly (pg_dump diff), the
run is idempotent, `flyway validate` passes, a pre-Flyway schema is adopted
without re-running anything, an empty target is refused, and the generated
types are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#400 gained V20260907213524__project_rollup_bump_reports_hit before it merged,
so it arrived in the pre-Flyway naming form. Rename it and move the adoption
baseline up to it -- production has already applied it, so Flyway must baseline
over it rather than re-run it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Database Types Check Complete

The database schema files were modified, but the regenerated TypeScript types are identical to the existing ones.

No changes were needed and the type definitions are already up to date.

nourshoreibah and others added 3 commits September 7, 2026 23:18
Review of #401 found the adoption baseline is not just a one-time marker but a
permanent floor, and that the deploy guard traded away a protection the old one
had. Both fail silently, which is the worst shape for this.

- BASELINE_VERSION is a floor, not a marker. Flyway reports any migration at or
  below it "Below Baseline" and never runs it: nothing pending, validate passes,
  migrate exits 0, column absent. CI builds from an empty schema where
  baselining never fires, so a PR scaffolded earlier the same day merges green
  and production quietly misses the change. `checks` now rejects a new migration
  whose version is not strictly above BASELINE_VERSION. It skips versions
  already on main so a rename is never mistaken for a new migration.

- The out-of-order step called that case "Harmless". True above the baseline,
  where outOfOrder applies the migration late; false at or below it, where it is
  never applied at all. Reworded, and the hard failure above covers the rest.
  Same correction in db/README.md.

- baselineOnMigrate is now off unless asked for. It only ever fires against a
  populated schema with no history table, which on production means a database
  we do not recognise -- a restored snapshot or a clone -- and adopting one
  silently is how you migrate the wrong thing. The deploy workflow gains an
  adopt_baseline input for the one-time cutover; compose sets it for local dev,
  where a pre-Flyway database should just be adopted.

- The deploy guards used `jq -e` inside `if`. Malformed output makes jq exit
  non-zero, which reads as "the dangerous condition is absent" -- the opposite
  of the documented safe fallthrough. The preflight now validates the JSON once
  and fails on it, and the guards compare plain strings.

- testkit applies the migration files directly, so nothing substituted Flyway's
  ${async} placeholder: the first migration written per the README would break
  `npm run reset`, `make db-reset` and every lambda's ensureSchema() with a
  postgres syntax error, while CI stayed green because CI goes through Flyway.
  testkit substitutes the placeholders itself now, and throws on an unknown one
  rather than silently deleting it.

- The docker fallback leaned on `--network=host`, which Docker Desktop ignores
  unless it is switched on, and on `localhost`, which can resolve to ::1 inside
  the container before the host's IPv4. Added --add-host=host.docker.internal
  and a rewrite of loopback hostnames.

Verified on PostgreSQL 16: the below-baseline skip reproduces and the new check
catches it while leaving an above-baseline migration alone; adoption is refused
by default and works with the opt-in; a fresh empty database still migrates in
full; the guard behaves correctly across healthy/empty/untracked/adopting/
drifted/malformed; ${async} now applies through both testkit and Flyway and an
unknown placeholder throws; schema still byte-identical and types unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unconditional rewrite broke Auto-regenerate DB Types: that workflow runs
postgres on the runner itself, bound to loopback, so host.docker.internal
resolves to the docker0 gateway and the connection is refused. On a native
daemon --network=host is real and localhost is already correct; it is only
Docker Desktop, which ignores --network=host unless switched on, that needs the
gateway. Gate the rewrite on the daemon actually being Desktop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Database Types Check Complete

The database schema files were modified, but the regenerated TypeScript types are identical to the existing ones.

No changes were needed and the type definitions are already up to date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-review The PR review bot won't run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant