From e674cb0fa6a92b84ae642df93f05bc017b6c2fa5 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 28 Sep 2026 16:09:01 +0000 Subject: [PATCH 1/2] docs: match the user guide to Snapshots, Utils, and the dialect list The left rail is Compare / Editor / Utils / Snapshots, not Schema Sync plus a SQL Editor sidebar. History split into Snapshots (Lokee) and Applies (migration runs). README lists all 14 SQL dialects. Co-authored-by: huy.ph1988 --- README.md | 13 +-- docs/USER_GUIDE.md | 183 +++++++++++++++++++++++++++--------- docs/releases/UNRELEASED.md | 15 +++ 3 files changed, 159 insertions(+), 52 deletions(-) diff --git a/README.md b/README.md index 8b80bfb9..83c1393d 100644 --- a/README.md +++ b/README.md @@ -2,10 +2,10 @@ # Fox Schema -**Compare schemas · generate migrations · run SQL — across 10 dialects.** +**Compare schemas · generate migrations · run SQL — across 14 SQL dialects.** -Install once, then open the local web UI (`foxschema`) for **Schema Sync**, the -**SQL Editor**, and optional **Workflow**. Self-host with Docker when you need a server. +Install once, then open the local web UI (`foxschema`) for **Compare**, the +**SQL Editor**, **Utils**, **Snapshots**, and optional **Workflow**. Self-host with Docker when you need a server. [foxschema.com](https://foxschema.com) · [Install](docs/INSTALL.md) · [User guide](docs/USER_GUIDE.md) · [Workflow](docs/WORKFLOW.md) · [Publish](docs/PUBLISH.md) · [Contributing](CONTRIBUTING.md) @@ -128,10 +128,11 @@ Open **http://localhost:3210**. Guide: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). ## Supported dialects -PostgreSQL · MySQL · MariaDB · SQL Server · Azure SQL · Oracle · IBM Db2 · -SQLite · ClickHouse · Amazon Redshift +PostgreSQL · CockroachDB · YugabyteDB · MySQL · MariaDB · TiDB · SQL Server · +Azure SQL · Oracle · IBM Db2 · SQLite · DuckDB · ClickHouse · Amazon Redshift -One product — Docker image includes Db2 on linux/amd64. +MongoDB and Redis appear in the connection list (settings only — no schema +compare). One product — Docker image includes Db2 on linux/amd64. ## Package notes (npm) diff --git a/docs/USER_GUIDE.md b/docs/USER_GUIDE.md index cecfd478..f6d061e2 100644 --- a/docs/USER_GUIDE.md +++ b/docs/USER_GUIDE.md @@ -12,9 +12,11 @@ SQL to make one match the other. This guide is for **using** Fox Schema — no c - [Read the diff](#read-the-diff) - [Generate & apply a migration](#generate--apply-a-migration) - [SQL Editor](#sql-editor) +- [Utilities](#utilities) - [Workflow](#workflow) - [Access control](#access-control) -- [History](#history) +- [Snapshots](#snapshots-schema-history) +- [Applies](#applies-migration-runs) - [Troubleshooting](#troubleshooting) ## What Fox Schema is for @@ -73,6 +75,11 @@ The first time you open the UI, Fox Schema may show a short **welcome wizard** a for your email so you can get product updates (new dialects, releases). It is optional — use **Skip for now** if you prefer. It only appears once per install. +You land on **Home**: continue last compare / last query, **Snapshots**, **Utilities**, +and saved connections (grouped by dialect). **⌘K** / **Ctrl+K** searches workspaces and +recents. The left rail is **Compare**, **Editor**, **Utils**, **Access**, **Workflow**, +**Snapshots**, then **Creds** and **Applies** at the bottom. + Fox also sets up an **encryption key** that protects the database passwords you save. The CLI creates one under your user data directory; Docker auto-generates one on the `/data` volume (or use `APP_ENCRYPTION_KEY` in `.env`). @@ -82,19 +89,28 @@ The CLI creates one under your user data directory; Docker auto-generates one on ## Connect a database -1. Click **Add connection** (or the connection dropdown → new). -2. Pick the **type** (PostgreSQL, MySQL, SQL Server, Oracle, Db2, …). -3. Fill in host, port, database, username, and password. Optionally a schema. +1. Click **Creds** on the left rail (or a connection chip → new). +2. Pick the **type**. Saved credentials are **grouped by dialect**; search by name. + PostgreSQL, MySQL, MariaDB, SQL Server, Azure SQL, Oracle, Db2, SQLite, DuckDB, + ClickHouse, Redshift, CockroachDB, YugabyteDB, and TiDB are first-class SQL. + MongoDB and Redis appear in the list (settings only — no schema compare). +3. Fill in host, port, database, username, and password. Schema is optional on + PostgreSQL (the form matches the engine). SQLite and DuckDB are **file paths** + — use **Browse…**; they have no password. 4. **Test** the connection, then save it. Passwords are encrypted — they're stored safely and never shown back to your browser. -Do this for both the database you're comparing **from** (source) and the one you're -comparing **to** (target). +Do this for both the database you're comparing **from** (Original) and the one you're +comparing **to** (Target). ## Run a comparison -1. Choose an **Original Server** connection and a **Target** connection. -2. Click **Compare**. +1. On the left rail, open **Compare**. +2. Choose an **Original Server** connection and a **Target** connection. +3. Click **Compare**. + +A **Same DB** warning appears only after **both** sides are picked and they name the +same database and schema. Two empty chips are not “the same database”. Fox Schema reads both schemas and builds the diff. You can narrow what it looks at (tables only, views, functions, etc.) with the scope filter. @@ -137,7 +153,7 @@ Use the **SQL Editor** to run ad-hoc queries and inspect data (separate from sch compare / migrate). It lives in the same local web UI you open with `foxschema`. 1. Open Fox Schema (`foxschema` or the Desktop shortcut). -2. In the top toolbar, click **SQL Editor** (next to Schema Sync). +2. On the left rail, click **Editor** (next to **Compare**). 3. Under **Destinations**, check one or more saved connections — the same SQL runs against every checked server (handy for comparing data across environments). 4. Type SQL in the editor. Multiple statements are fine; use the **statement strip** @@ -156,10 +172,17 @@ compare / migrate). It lives in the same local web UI you open with `foxschema`. **Skip trigger cols** (on by default) ignores audit fields such as `createdAt` / `updatedBy`. The destination grid shows a **Sync** column (on by default for all differing rows) so you can include or exclude individual rows before migrate. + On a **partial** page (not page 1, a next page exists, or the result was + truncated), rows that appear only on the other side stay **unresolved** — they + are not colored as missing/extra, because they may simply be on another page. 7. **Data migrate (≤500 row ops)** — with Compare on, **Data migrate** appears. Rows match by the **Keys** you check (PK/unique columns are marked; you can pick - any shared column, e.g. compare by name only). **Sync all** re-checks every + any shared column, e.g. compare by name only — name Keys are fine for alignment + and **Add-only** migrate). **Edit** and **Delete** require the table’s unique + key (primary key, or a non-partial unique index) to be **in the SELECT and + checked** — otherwise a name column would UPDATE/DELETE every matching row on + the destination, including rows you never saw. **Sync all** re-checks every differing row without changing your Add / Edit / Delete choices. Migrate only runs when **both** grids show the **full** result on **page 1** (no next page) — otherwise “missing on this page” is not “missing from the table” and Delete could @@ -188,34 +211,10 @@ Tips: **Edit table** shows each index’s fragmentation % for every dialect (physical or estimated probe; SQLite / DuckDB / ClickHouse / Redshift list indexes when no native % exists). Paste custom SELECT if the default fails, and use the wrench - to insert rebuild/reorg/optimize/REINDEX SQL when useful. -- **Utilities → Index Management** (SQL Editor sidebar) — pick a credential, load - all indexes grouped by table, filter by table/index name or minimum - fragmentation %, fetch fragmentation in batch, then defragment selected indexes - or all filtered rows. In **Edit table**, the index form opens under the selected - index (no jump to a form at the bottom of the section). -- **Utilities → Clone Table** — archive a huge table as `name_1` / next free - `name_N` (or a fixed starting number), then recreate an empty table with the - original name and columns so apps keep working. Toggle **Keep indexes** and - **Foreign keys (auto)** for the new table; Insert SQL or Apply. Inbound FKs - from other tables still point at the archive until you update them — then you - can drop history safely. -- **Utilities → Query files** / **Files** sidebar — import **CSV/TSV** - (delimiter: comma, tab, semicolon, pipe, or custom), **JSON** (array or - NDJSON), or **fixed-width text** (column start/length offsets). - **Destination** choices: - - **New temp SQLite workspace** (default) — short-lived `Files: …` - credential; add more files later with **Add table to existing Files - workspace** so several tables share one temp DB. - - **Import into saved credential** — create a table on a checked server - (Postgres, MySQL, SQL Server, DB2, Oracle, DuckDB, etc.) using chunked - multi-row `INSERT` bulk loads. - Large pastes/files upload in **chunks** (disk-backed session). Open the - **Files** sidebar to list workspace tables, click one to re-select that DB - and load a sample SELECT, delete one workspace, or clear all. **Replace - previous file imports** (off by default) deletes earlier `Files:` workspaces - when you create a new one; **Replace table if it exists** applies when - adding to a workspace or credential. Temp DBs expire after about 24 hours. + to insert rebuild/reorg/optimize/REINDEX SQL when useful. The index form opens + under the selected index (no jump to a form at the bottom of the section). + Index Management, Clone Table, and Query files live in **Utils** — see + [Utilities](#utilities). - **Data peek** — two ways in: - **Schema:** hold **Cmd** (macOS) or **Ctrl** (Windows/Linux) and click a table, view or MQT to see its rows without writing a query. @@ -399,12 +398,57 @@ read-write (the file is opened that way on purpose). **ClickHouse** grid row editing is blocked; other dialects that cannot apply a given write show a clear error on that connection’s result cell. -Switch back to **Schema Sync** anytime to compare and migrate schemas. +Switch back to **Compare** anytime to compare and migrate schemas. + +## Utilities + +Left rail **Utils** — a workspace of its own, not a SQL Editor sidebar. Pick one +saved credential at the top, then a tool: + +**Maintenance** + +- **Index Management** — indexes grouped by table; filter by name or minimum + fragmentation %; fetch fragmentation in batch (`POST /schema/index-fragmentation-batch`); + defragment selected indexes or all filtered rows. +- **Clone Table** — archive a huge table as `name_1` / next free `name_N` (or a + fixed starting number), then recreate an empty table with the original name and + columns so apps keep working. Toggle **Keep indexes** and **Foreign keys (auto)** + for the new table; Insert SQL or Apply. Inbound FKs from other tables still point + at the archive until you update them. + +**Insights** (estimated where the engine has no physical figure) + +- **Connection Pool**, **User Connections**, **System Info**, **Table & Index Size**. + DuckDB reports worker threads, buffer memory, and database-file blocks (no + per-table bytes — row counts are estimates). SQLite / DuckDB have no server pool + or multi-user sessions. + +**Access** + +- **DB users & grants** — same catalog as the Access workspace + (`POST /schema/db-access`). GRANT / REVOKE still needs **Grant privileges**. + +**Files** + +- **Query files** — import **CSV/TSV** (comma, tab, semicolon, pipe, or custom), + **JSON** (array or NDJSON), or **fixed-width text** (column start/length offsets). + **Destination** choices: + - **New temp SQLite workspace** (default) — short-lived `Files: …` credential; + add more files later so several tables share one temp DB. + - **Import into saved credential** — create a table on a saved server using + chunked multi-row `INSERT` bulk loads. + Large pastes/files upload in **chunks** (disk-backed session). List workspace + tables, click one to load a sample SELECT in the Editor, delete one workspace, + or clear all. **Replace previous file imports** (off by default) deletes earlier + `Files:` workspaces when you create a new one; **Replace table if it exists** + applies when adding to a workspace or credential. Temp DBs expire after about + 24 hours. **Insert SQL** from Clone Table / the table blueprint writes into the + Editor tab even if the Editor is not on screen. ## Workflow Optional workspace for scheduled and triggered jobs (SQL, HTTP, files, email/SMS) -beside Schema Sync and the SQL Editor. The designer lives in the Fox Schema UI; +beside Compare and the SQL Editor. The designer lives in the Fox Schema UI; a **separate engine process** runs the jobs. Developer / ops runbook: [WORKFLOW.md](WORKFLOW.md). Env vars: @@ -439,7 +483,7 @@ API (`POST /schema/db-access`): - **Access** workspace (needs any `access.*` tab permission, including **Open Access**). -- **Utilities → Database Access** (needs **Use utilities**). +- **Utils → DB users & grants** (needs **Use utilities**). Either family may load the catalog. Running GRANT / REVOKE still needs **Grant privileges** (`editor.grant`). SQLite / DuckDB have no GRANT catalog; @@ -473,14 +517,61 @@ Granting: and TiDB it also emits `SET DEFAULT ROLE ALL`, and on MariaDB `SET DEFAULT ROLE`, because a granted role is otherwise inactive at login. -## History - -Every migration you apply is recorded — status, target, the exact script, the -pre-migration snapshot, and per-object results. Open **History** to review or -re-inspect past runs. No passwords are stored in history. +## Snapshots (schema history) + +Left rail **Snapshots**. This tracks versions of **one** database (not a second +live connection). It is not the same list as **Applies** (migration runs) at the +bottom of the rail. + +1. Pick a saved credential and **Take first snapshot** (or snapshot again after a + live change). Applying a Compare migration also records a version. +2. The **graph** stays on screen while it reloads; use **Graph** to hide it. +3. **Original** and **Target** work like Compare: Original is a version; Target is + the live database or another version. Changing the pickers does not hide the + graph. +4. **Compare versions** is a preview. Nothing writes the live database until you + press **Update** / **Revert**. +5. **Revert** always runs against the **live** database this history was captured + from, snapshots first, and **appends** a new version (it never rewrites the + version you picked). Tick objects, or **Select all**. Nothing ticked means + nothing runs. A plan that would destroy data needs an extra confirmation; a + **blocked** plan is refused. The button says **Update** when the plan only + adds (the database has fallen behind) and **Revert** when it rolls back. +6. **Force migrate…** applies a stored version to a **different** database. It + picks its own version and target (not whatever the graph is showing). You must + confirm “this is not the history’s database”; if the plan destroys data you + also confirm that. Those two acknowledgements are separate — agreeing to data + loss is not agreeing to target another database. + +View / routine / trigger “sameness” uses the **same rules as Compare** +(`normalizeDefinitionText` in `@foxschema/sql`): whitespace and a trailing `;` +are formatting; keyword and identifier case is folded; this history’s schema +qualifier is ignored (`app.orders` = `orders`); **string-literal case is kept**. +The definition Lokee stores is still the captured text — revert builds DDL from +it. After upgrading, the first capture of an existing history may record **one +extra version** for objects whose hash changed under the new rule; later versions +appear only for real changes. + +SQLite and DuckDB file credentials can be picked from a Browse dialog on the +machine running Fox Schema. + +## Applies (migration runs) + +Bottom of the left rail, **Applies**. Every Schema Compare migration you apply is +recorded — status, target, the exact script, the pre-migration snapshot, and +per-object results. No passwords are stored. Data migrate has its own history in +the Editor. ## Troubleshooting +**UI looks disconnected / API returns 403 "This origin is not allowed".** In +`npm run dev`, Vite binds every address and prints a **Network:** URL. Dev +allows Origins on this machine's own literal IPs at ports **5173**, **5199**, +**3210**, and **3211** — not an arbitrary hostname (DNS rebinding). Opening +`http://:5173` works; `http://evil.com:5173` does not. In +production, UI and API share one origin; for a split hostname set +`FOX_ALLOWED_ORIGINS`. See [DEPLOYMENT.md](DEPLOYMENT.md#origin-policy). + **"Connection failed" / timeout.** Check host, port, and that the database accepts connections from where Fox Schema runs (in Docker, `localhost` means *inside the container* — use the host's IP or a service name, not `localhost`, to reach a DB on your machine). diff --git a/docs/releases/UNRELEASED.md b/docs/releases/UNRELEASED.md index ac216725..9ffc7d62 100644 --- a/docs/releases/UNRELEASED.md +++ b/docs/releases/UNRELEASED.md @@ -3,6 +3,21 @@ Notes for the next release. At ship time, rename this file to that version's `RELEASE_.md` and use it as the GitHub Release body. +## Workspaces + +- **Snapshots** is its own left-rail workspace (schema history / Lokee). **Applies** + at the bottom of the rail is migration-run history. They are not the same list. +- **Utils** is its own workspace (Index Management, Clone Table, insights, Query + files, DB users & grants) — not a SQL Editor sidebar. +- Saved credentials are grouped by dialect; Home continues last compare / query. + +## Origin policy + +`npm run dev` allows this machine's own literal IPs on ports 5173 / 5199 / 3210 / +3211, so a Vite **Network:** URL works. Arbitrary hostnames are still refused. +Production stays same-origin; set `FOX_ALLOWED_ORIGINS` for a split hostname. +See [DEPLOYMENT.md](../DEPLOYMENT.md#origin-policy). + ## Schema history (Lokee): definitions are compared the way Compare compares them Compare and Lokee used to decide "is this view / routine / trigger the same?" From 2d8a1eb78051050c7918376507bbd5f1a5e5c875 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 28 Sep 2026 16:09:03 +0000 Subject: [PATCH 2/2] docs: origin policy, E2E sweep pitfalls, dialectFamily, workflow save Record FOX_ALLOWED_ORIGINS and the literal-IP dev allowlist, the E2E rate-limit helper and suite timeouts, DIALECTS.md as tracked, and assertKnownFromPorts at workflow save. Point agent memory at the Utils workspace instead of the SQL Editor sidebar. Co-authored-by: huy.ph1988 --- .cursor/rules/sql-editor-agent-memory.mdc | 4 ++-- .env.example | 7 +++++++ CONTRIBUTING.md | 24 ++++++++++++++++++++++- docs/ARCHITECTURE.md | 16 ++++++++++----- docs/CODE_MAP.md | 15 ++++++++------ docs/DEPLOYMENT.md | 23 ++++++++++++++++++++++ docs/WORKFLOW.md | 8 ++++++++ 7 files changed, 83 insertions(+), 14 deletions(-) diff --git a/.cursor/rules/sql-editor-agent-memory.mdc b/.cursor/rules/sql-editor-agent-memory.mdc index fc573f00..337a1f68 100644 --- a/.cursor/rules/sql-editor-agent-memory.mdc +++ b/.cursor/rules/sql-editor-agent-memory.mdc @@ -34,9 +34,9 @@ FoxSchema = schema diff/migration + SQL Editor. Destinations checklist, Schema e ### 1. Data peek (Cmd/Ctrl-click + result FK) - **UI**: `DataPeekPanel.tsx` — modal grids; Esc closes. Each panel: WHERE / ORDER BY / LIMIT (debounced auto-apply + blur/Enter/Apply) + drag resize + arrange grip. Mounted from `SqlEditorView` (not only Schema). - **Index fragmentation (Edit table)**: `dialect-index-fragmentation.ts` + `POST /schema/index-fragmentation`; shown on `TableBlueprintModal` index rows. **Every dialect** has a default probe (`query: true`) — physical where available, otherwise estimated / catalog listing (null %). Custom SELECT fallback (`index_name`, `fragmentation_percent`). Defrag SQL via wrench (REBUILD / REORG / OPTIMIZE / REINDEX / VACUUM by dialect). -- **Utilities → Index Management**: SQL Editor left sidebar **Utilities** section → `IndexManagementModal` (not toolbar). Credential dropdown, indexes grouped by table, filter table/%, `POST /schema/index-fragmentation-batch`, defragment selected or filtered via `executeSql`. +- **Utils workspace** (`features/utilities/UtilitiesView.tsx`, left rail **Utils**): Index Management, Clone Table, insights, **DB users & grants**, Query files. Not a SQL Editor sidebar — `SqlEditorView` returns `null` for sidebar ids `utilities` / `files`. Credential chip at the top; tools dock as panes. `POST /schema/index-fragmentation-batch`; defragment via `executeSql`. **Insert SQL** from Clone Table / the table blueprint uses `insertAtCursor` fallback so SQL lands in the Editor tab even when Editor is unmounted. - **Edit index inline**: `TableBlueprintModal` opens the index form under the selected existing/pending index (or an Add-index row) and scrolls it into view — not a form stuck at the section bottom. -- **Clone Table (archive & recreate)**: Utilities sidebar + Schema **Clone** → `CloneTableModal`. SQL in `generateCloneTableSql` / `nextArchiveTableName` / `generateRenameTableSql` (`tableBlueprintSql.ts`): rename live → `name_N`, free schema-unique index/FK names on archive when needed, `CREATE TABLE` empty live twin, optional indexes + outbound FKs. Inbound FKs stay on archive (warned in UI). +- **Clone Table (archive & recreate)**: Utils workspace → `CloneTableModal`. SQL in `generateCloneTableSql` / `nextArchiveTableName` / `generateRenameTableSql` (`tableBlueprintSql.ts`): rename live → `name_N`, free schema-unique index/FK names on archive when needed, `CREATE TABLE` empty live twin, optional indexes + outbound FKs. Inbound FKs stay on archive (warned in UI). - **Triggers**: - Schema explorer Cmd/Ctrl-click TABLE / VIEW / MQT → `openDataPeek`. - Editor result FK cell (rust underline) → `openDataPeekFromFk` (`ResultsPanel` / `foreignKeyLinksForSql`). diff --git a/.env.example b/.env.example index 10d17934..5f6a4d2f 100644 --- a/.env.example +++ b/.env.example @@ -48,6 +48,12 @@ AUTH_REQUIRED=false # SSO_MICROSOFT_CLIENT_SECRET= # SSO_MICROSOFT_TENANT=common +# ── Browser origins (optional) ── +# Comma-separated origins allowed to call the API with cookies. When set, this +# is the entire allowlist. Leave unset: production is same-origin (Docker / +# foxschema open); npm run dev allows this machine's own IPs on 5173/5199/3210/3211. +# FOX_ALLOWED_ORIGINS=https://fox.example.com + # ── First-open email subscriber wizard (optional, public before login) ── # Shown once per install on first UI boot. Skip for now dismisses without posting. # Left unset, subscribe still dismisses locally (fine for local dev). @@ -63,6 +69,7 @@ AUTH_REQUIRED=false # Engine-only (required at engine boot; 32 bytes hex or base64): # FOXFLOW_ENCRYPTION_KEY= # FOXFLOW_DB_PATH=/data/workflow-engine.sqlite +# FOXFLOW_FILES_DIR=/data/workflow-files # PORT=8081 # HOST=127.0.0.1 # Optional public webhook listener (keep the engine API on loopback): diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 51e5631f..20967ed7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -98,6 +98,20 @@ npm run test:e2e:workflow `bash scripts/seed/reset-all.sh` (full `down -v` + up + reseed). Re-running against a mutated target accumulates corruption and produces false failures. +`apps/e2e/scripts/run-all.mjs` is the sweep. Dialect-matrix suites (utilities, +database-access, access-assistant) get a wall-clock budget of **120s × configured +dialects** (minimum 600s). SQL Editor (twelve files) and revert-edge cases get +**600s**; everything else **300s**. A killed suite is logged as `runner timeout`, +not a mysterious empty FAIL. + +Lokee capture / revert / force-migrate and `POST /schema/db-access` share a +**20 requests / minute** limiter. The suites run as one user, so History + Revert ++ revert-edges back-to-back used to toast “Snapshot failed” and wait 20–30s for a +version that was never recorded. New clicks go through +`apps/e2e/src/helpers/rate-limited.ts`: wait for **that** HTTP response, honour +`Retry-After` on 429, fail immediately on any other non-2xx. Do not click those +buttons with a bare Playwright `click` and then wait for the UI to look settled. + ## Testing expectations - Engine logic (compare, generator) → unit tests in `packages/sql`; drivers/providers → `packages/db`. @@ -118,9 +132,13 @@ Each dialect spans both packages: `sql-dialect.ts` / `settings.ts` under (settings, adapter, provider, sql-dialect). The exact contract — required vs. optional hooks, cross-cutting invariants (casing, index/FK naming, DROP ordering), and per-dialect gotchas — is documented in -**packages/sql/src/providers/DIALECTS.md** (local, gitignored), +[`packages/sql/src/providers/DIALECTS.md`](packages/sql/src/providers/DIALECTS.md) +(tracked; allowlisted in `.gitignore`), with the step-by-step checklist in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). Read `DIALECTS.md` before touching any `*.sql-dialect.ts` or `sql-generator.module.ts`. +Wire-compatible relatives (MariaDB/TiDB, Cockroach/Yugabyte/Redshift, Azure SQL) +must go through `dialectFamily()` in `packages/sql/src/providers/provider-settings.ts` +— do not re-list `d === 'mysql' || d === 'mariadb' || …`. ## Conventions worth knowing @@ -129,6 +147,10 @@ A few rules that have bitten people before (the full set is in [CLAUDE.md](CLAUD - **The compare key is not a SQL identifier.** `obj.tableName` is the uppercased match key from `compare.module.ts` — use `source?.name` / `targetTable?.name` for real DDL (native casing; case-sensitive on MySQL). +- **Use `dialectFamily()`, not a handwritten family chain.** MariaDB/TiDB follow + MySQL; CockroachDB/YugabyteDB/Redshift follow Postgres; Azure SQL follows SQL + Server (`packages/sql/src/providers/provider-settings.ts`). Monaco Format, access + SQL, and file-import batch sizes all go through it. - **The app's metadata-DB migrations are append-only** — never edit a shipped migration in `packages/server/src/database/schema.ts`; add a new one. - **Never store database passwords client-side or in history.** Saved connections diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 959a1822..5500b791 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -6,10 +6,11 @@ Stable orientation for FoxSchema. For where a change belongs see ## What this is Database schema **diff & migration** tool. Compare a source schema against a target, -generate dialect-native migration SQL, and deploy it. Primary target is DB2; Postgres, -MySQL, SQL Server, Oracle, SQLite, MariaDB, Azure SQL, ClickHouse, and Redshift are also -implemented. Distributions: **CLI** (`foxschema` via npm/Homebrew) and **Docker** -(single amd64 image with Db2). +generate dialect-native migration SQL, and deploy it. Fourteen SQL dialects: +Postgres (and CockroachDB / YugabyteDB / Redshift), MySQL (and MariaDB / TiDB), +SQL Server / Azure SQL, Oracle, Db2, SQLite, DuckDB, ClickHouse. MongoDB and Redis +carry connection settings only. Distributions: **CLI** (`foxschema` via npm/Homebrew) +and **Docker** (single amd64 image with Db2). ## Commands @@ -144,7 +145,12 @@ core's. Each SQL dialect has three layers, split across `packages/sql/src/providers/` (dialect + settings) and `packages/db/src/providers/` (adapter + provider). -`packages/sql` lists 14 SQL dialects (MongoDB and Redis carry settings only): +Wire-compatible relatives share a family (`dialectFamily()` in +`packages/sql/src/providers/provider-settings.ts`): MariaDB / TiDB → `mysql`; +CockroachDB / YugabyteDB / Redshift → `postgres`; Azure SQL → `sqlserver`. Use +that helper instead of listing dialects by hand (Format / Monaco language, access +SQL, file-import batch size). `packages/sql` lists 14 SQL dialects (MongoDB and +Redis carry settings only): | File | Interface | Registry | |------|-----------|----------| diff --git a/docs/CODE_MAP.md b/docs/CODE_MAP.md index a90124c8..ae7db45e 100644 --- a/docs/CODE_MAP.md +++ b/docs/CODE_MAP.md @@ -65,6 +65,8 @@ flags (`workflow`, `enterprise.channels`). ``` interfaces/ The shared vocabulary: TableSchema, TableDiff, MigrationStep. providers/ One folder per dialect — settings, SqlDialect, and Access SQL (`*.user-sql.ts`, `*.access-sql.ts`). + `dialectFamily()` in `provider-settings.ts` maps wire-compatible + relatives onto mysql / postgres / sqlserver. cores/ Connection strings, and shaping catalog rows into TableSchema. modules/ One folder per domain, named to match the frontend feature that consumes it. @@ -74,7 +76,7 @@ modules/ One folder per domain, named to match the frontend feature |---|---| | `dialect` | The `SqlDialect` contract, the registry, type mapping, capability flags | | `sql-text` | Statement splitting and SQL templating | -| `schema-diff` | Comparing two schemas, and browsing one | +| `schema-diff` | Comparing two schemas, and browsing one. `normalizeDefinitionText` is shared with Lokee | | `migrations` | Generating DDL, ordering drops, validating a plan | | `lokee-weave` | Content-addressed schema versioning and revert | | `sql-editor` | FoxScript parsing, code cells, SELECT aliasing, the SQL subset | @@ -96,7 +98,8 @@ api/ The HTTP server itself: Fastify setup, route tree, security platform/ Cross-cutting infrastructure used by every feature. contracts/ ActorContext and ServiceError - guards/ origin policy, rate limit, idempotency, target locks + guards/ origin policy (`FOX_ALLOWED_ORIGINS`, literal LAN IPs in dev), + rate limit, idempotency, target locks http/ request/response types, router, Fastify binding, responses db/ connection resolution crypto/ secret encryption @@ -186,12 +189,12 @@ the page-epoch guard, bookmarks and recents, SQL variables. | `admin` | User and role administration | | `auth` | Sign-in, SSO buttons, onboarding | | `connections` | Connection modal (login method: password / Windows / LDAP), credential manager, database settings | -| `lokee-weave` | Schema history graph and version compare | -| `migrations` | Migration run history | +| `lokee-weave` | Schema history graph and version compare (Snapshots workspace) | +| `migrations` | Migration run history (Applies on the rail) | | `object-detail` | Detail panel for a single schema object | | `schema-diff` | Diff rendering shared by compare and history | -| `sql-editor` | SQL editor, results grid, data peek, utilities | -| `utilities` | Clone table, index management, server insights | +| `sql-editor` | SQL editor, results grid, data peek (Index/Clone/Query files live in `utilities`) | +| `utilities` | Own workspace: clone table, index management, server insights, query files, DB users & grants | | `workflow` | Workflow designer (canvas, inspector, triggers, SQL and script editors), runs, variables, credentials and engine settings — through the engine proxy, plus linking saved connections to workflows | ## Where does my change go? diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 2947073b..8ea5a3bd 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -40,6 +40,7 @@ Db2) that serves both the UI and the API on one configurable port (default **321 - [Choosing a port](#choosing-a-port) - [Where app data lives](#where-app-data-lives) - [Access: single-user vs. multi-user + SSO](#access-single-user-vs-multi-user--sso) +- [Origin policy](#origin-policy) - [Cloud platforms](#cloud-platforms) - [Database drivers](#database-drivers) - [Building the image](#building-the-image) @@ -104,6 +105,7 @@ docker compose -f docker-compose.app.yml up -d | `ALLOW_HOST_CLOUD_CREDENTIALS` | off | When `true`, cloud secret resolve may use the host IAM/ADC chain without saved user credentials. **Keep off** on multi-user hosts. | | `SSO_*` | — | OAuth for Google / Microsoft / GitHub (see below). | | `NODE_ENV` | `production` | Set in the image; enforces that `APP_ENCRYPTION_KEY` is present. | +| `FOX_ALLOWED_ORIGINS` | — | Comma-separated browser origins allowed to call the API with cookies. When set, it is the entire allowlist. See [Origin policy](#origin-policy). | > The app also reads `APP_USER_EMAIL` (only for the `v2` key scheme). > Update checks default to the npm `foxschema` registry feed (see below). @@ -258,6 +260,27 @@ hostnames). **Always terminate TLS** (via your reverse proxy or platform) for any internet-facing deployment — Fox Schema handles database credentials. +## Origin policy + +The API holds database credentials and can run migrations, so only named browser +origins may call it with cookies (`packages/server/src/platform/guards/origin-policy.ts`). +The allowlist is explicit, not “any localhost”. + +| Mode | Who may call | +|------|----------------| +| `FOX_ALLOWED_ORIGINS` set | **Only** those comma-separated origins (scheme + host + port). Wins over everything else. | +| Production, unset | The origin this process is served from, plus same-origin `fetch` (`Origin` matching this request's host). Docker / `foxschema open` work without extra config. | +| `npm run dev` | This machine's **literal** addresses (`localhost`, `127.0.0.1`, `[::1]`, and `os.networkInterfaces()` IPs) on ports **5173**, **5199**, **3210**, **3211**. Hostnames other than `localhost` are refused — DNS rebinding can point `evil.com` at 127.0.0.1, and Vite (`allowedHosts: true`) would serve it. | + +A missing `Origin` is allowed (curl, health checks). A refused Origin is **403** +with `This origin is not allowed to call the Fox Schema API.` — that is why a +LAN Vite URL used to look like a blank / disconnected UI. + +```bash +# Split UI hostname in production +FOX_ALLOWED_ORIGINS=https://fox.example.com,https://fox.example.com:443 +``` + ## Cloud platforms The image is a standard single-port web server, so it runs anywhere containers do: diff --git a/docs/WORKFLOW.md b/docs/WORKFLOW.md index d865f69d..aa381ce2 100644 --- a/docs/WORKFLOW.md +++ b/docs/WORKFLOW.md @@ -36,6 +36,14 @@ The proxy is the only door the designer uses. Webhook and API-endpoint **ingress is never fronted** by a FoxSchema session — those routes authenticate their own callers (`apps/workflow-server/src/app.ts`). +Saving a workflow runs the same checks the executor will: every pipe must be +usable, and every edge `fromPort` must be a port the source pipe declares +(`assertKnownFromPorts` in `packages/workflow-engine/src/runtime/ports.ts`, +called from `apps/workflow-server/src/routes/workflows.ts`). An unknown port is +refused at save (`pipeline p: edge split → out: unknown fromPort "eu"`), not +discovered when a scheduled run fails. Pipes without static metadata skip the +check (Goto). + Without `WORKFLOW_ENGINE_TOKEN`, the engine API stays open (loopback dev only) and FoxSchema's internal resolve route returns 503, so saved connections cannot be used by workflows.