Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .cursor/rules/sql-editor-agent-memory.mdc
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ FoxSchema = schema diff/migration + SQL Editor. Destinations checklist, Schema e
### 1. Data peek (Cmd/Ctrl-click + result FK)
- **UI**: `DataPeekPanel.tsx` — modal grids; Esc closes. Each panel: WHERE / ORDER BY / LIMIT (debounced auto-apply + blur/Enter/Apply) + drag resize + arrange grip. Mounted from `SqlEditorView` (not only Schema).
- **Index fragmentation (Edit table)**: `dialect-index-fragmentation.ts` + `POST /schema/index-fragmentation`; shown on `TableBlueprintModal` index rows. **Every dialect** has a default probe (`query: true`) — physical where available, otherwise estimated / catalog listing (null %). Custom SELECT fallback (`index_name`, `fragmentation_percent`). Defrag SQL via wrench (REBUILD / REORG / OPTIMIZE / REINDEX / VACUUM by dialect).
- **Utilities → Index Management**: SQL Editor left sidebar **Utilities** section → `IndexManagementModal` (not toolbar). Credential dropdown, indexes grouped by table, filter table/%, `POST /schema/index-fragmentation-batch`, defragment selected or filtered via `executeSql`.
- **Utils workspace** (`features/utilities/UtilitiesView.tsx`, left rail **Utils**): Index Management, Clone Table, insights, **DB users & grants**, Query files. Not a SQL Editor sidebar — `SqlEditorView` returns `null` for sidebar ids `utilities` / `files`. Credential chip at the top; tools dock as panes. `POST /schema/index-fragmentation-batch`; defragment via `executeSql`. **Insert SQL** from Clone Table / the table blueprint uses `insertAtCursor` fallback so SQL lands in the Editor tab even when Editor is unmounted.
- **Edit index inline**: `TableBlueprintModal` opens the index form under the selected existing/pending index (or an Add-index row) and scrolls it into view — not a form stuck at the section bottom.
- **Clone Table (archive & recreate)**: Utilities sidebar + Schema **Clone** → `CloneTableModal`. SQL in `generateCloneTableSql` / `nextArchiveTableName` / `generateRenameTableSql` (`tableBlueprintSql.ts`): rename live → `name_N`, free schema-unique index/FK names on archive when needed, `CREATE TABLE` empty live twin, optional indexes + outbound FKs. Inbound FKs stay on archive (warned in UI).
- **Clone Table (archive & recreate)**: Utils workspace → `CloneTableModal`. SQL in `generateCloneTableSql` / `nextArchiveTableName` / `generateRenameTableSql` (`tableBlueprintSql.ts`): rename live → `name_N`, free schema-unique index/FK names on archive when needed, `CREATE TABLE` empty live twin, optional indexes + outbound FKs. Inbound FKs stay on archive (warned in UI).
- **Triggers**:
- Schema explorer Cmd/Ctrl-click TABLE / VIEW / MQT → `openDataPeek`.
- Editor result FK cell (rust underline) → `openDataPeekFromFk` (`ResultsPanel` / `foreignKeyLinksForSql`).
Expand Down
7 changes: 7 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@ AUTH_REQUIRED=false
# SSO_MICROSOFT_CLIENT_SECRET=
# SSO_MICROSOFT_TENANT=common

# ── Browser origins (optional) ──
# Comma-separated origins allowed to call the API with cookies. When set, this
# is the entire allowlist. Leave unset: production is same-origin (Docker /
# foxschema open); npm run dev allows this machine's own IPs on 5173/5199/3210/3211.
# FOX_ALLOWED_ORIGINS=https://fox.example.com

# ── First-open email subscriber wizard (optional, public before login) ──
# Shown once per install on first UI boot. Skip for now dismisses without posting.
# Left unset, subscribe still dismisses locally (fine for local dev).
Expand All @@ -63,6 +69,7 @@ AUTH_REQUIRED=false
# Engine-only (required at engine boot; 32 bytes hex or base64):
# FOXFLOW_ENCRYPTION_KEY=
# FOXFLOW_DB_PATH=/data/workflow-engine.sqlite
# FOXFLOW_FILES_DIR=/data/workflow-files
# PORT=8081
# HOST=127.0.0.1
# Optional public webhook listener (keep the engine API on loopback):
Expand Down
24 changes: 23 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,20 @@ npm run test:e2e:workflow
`bash scripts/seed/reset-all.sh` (full `down -v` + up + reseed). Re-running against a
mutated target accumulates corruption and produces false failures.

`apps/e2e/scripts/run-all.mjs` is the sweep. Dialect-matrix suites (utilities,
database-access, access-assistant) get a wall-clock budget of **120s × configured
dialects** (minimum 600s). SQL Editor (twelve files) and revert-edge cases get
**600s**; everything else **300s**. A killed suite is logged as `runner timeout`,
not a mysterious empty FAIL.

Lokee capture / revert / force-migrate and `POST /schema/db-access` share a
**20 requests / minute** limiter. The suites run as one user, so History + Revert
+ revert-edges back-to-back used to toast “Snapshot failed” and wait 20–30s for a
version that was never recorded. New clicks go through
`apps/e2e/src/helpers/rate-limited.ts`: wait for **that** HTTP response, honour
`Retry-After` on 429, fail immediately on any other non-2xx. Do not click those
buttons with a bare Playwright `click` and then wait for the UI to look settled.

## Testing expectations

- Engine logic (compare, generator) → unit tests in `packages/sql`; drivers/providers → `packages/db`.
Expand All @@ -118,9 +132,13 @@ Each dialect spans both packages: `sql-dialect.ts` / `settings.ts` under
(settings, adapter, provider, sql-dialect). The exact contract — required vs.
optional hooks, cross-cutting invariants (casing, index/FK naming, DROP ordering),
and per-dialect gotchas — is documented in
**packages/sql/src/providers/DIALECTS.md** (local, gitignored),
[`packages/sql/src/providers/DIALECTS.md`](packages/sql/src/providers/DIALECTS.md)
(tracked; allowlisted in `.gitignore`),
with the step-by-step checklist in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
Read `DIALECTS.md` before touching any `*.sql-dialect.ts` or `sql-generator.module.ts`.
Wire-compatible relatives (MariaDB/TiDB, Cockroach/Yugabyte/Redshift, Azure SQL)
must go through `dialectFamily()` in `packages/sql/src/providers/provider-settings.ts`
— do not re-list `d === 'mysql' || d === 'mariadb' || …`.

## Conventions worth knowing

Expand All @@ -129,6 +147,10 @@ A few rules that have bitten people before (the full set is in [CLAUDE.md](CLAUD
- **The compare key is not a SQL identifier.** `obj.tableName` is the uppercased
match key from `compare.module.ts` — use `source?.name` / `targetTable?.name` for
real DDL (native casing; case-sensitive on MySQL).
- **Use `dialectFamily()`, not a handwritten family chain.** MariaDB/TiDB follow
MySQL; CockroachDB/YugabyteDB/Redshift follow Postgres; Azure SQL follows SQL
Server (`packages/sql/src/providers/provider-settings.ts`). Monaco Format, access
SQL, and file-import batch sizes all go through it.
- **The app's metadata-DB migrations are append-only** — never edit a shipped
migration in `packages/server/src/database/schema.ts`; add a new one.
- **Never store database passwords client-side or in history.** Saved connections
Expand Down
13 changes: 7 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

# Fox Schema

**Compare schemas · generate migrations · run SQL — across 10 dialects.**
**Compare schemas · generate migrations · run SQL — across 14 SQL dialects.**

Install once, then open the local web UI (`foxschema`) for **Schema Sync**, the
**SQL Editor**, and optional **Workflow**. Self-host with Docker when you need a server.
Install once, then open the local web UI (`foxschema`) for **Compare**, the
**SQL Editor**, **Utils**, **Snapshots**, and optional **Workflow**. Self-host with Docker when you need a server.

[foxschema.com](https://foxschema.com) · [Install](docs/INSTALL.md) · [User guide](docs/USER_GUIDE.md) · [Workflow](docs/WORKFLOW.md) · [Publish](docs/PUBLISH.md) · [Contributing](CONTRIBUTING.md)

Expand Down Expand Up @@ -128,10 +128,11 @@ Open **http://localhost:3210**. Guide: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md).

## Supported dialects

PostgreSQL · MySQL · MariaDB · SQL Server · Azure SQL · Oracle · IBM Db2 ·
SQLite · ClickHouse · Amazon Redshift
PostgreSQL · CockroachDB · YugabyteDB · MySQL · MariaDB · TiDB · SQL Server ·
Azure SQL · Oracle · IBM Db2 · SQLite · DuckDB · ClickHouse · Amazon Redshift

One product — Docker image includes Db2 on linux/amd64.
MongoDB and Redis appear in the connection list (settings only — no schema
compare). One product — Docker image includes Db2 on linux/amd64.

## Package notes (npm)

Expand Down
16 changes: 11 additions & 5 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@ Stable orientation for FoxSchema. For where a change belongs see
## What this is

Database schema **diff & migration** tool. Compare a source schema against a target,
generate dialect-native migration SQL, and deploy it. Primary target is DB2; Postgres,
MySQL, SQL Server, Oracle, SQLite, MariaDB, Azure SQL, ClickHouse, and Redshift are also
implemented. Distributions: **CLI** (`foxschema` via npm/Homebrew) and **Docker**
(single amd64 image with Db2).
generate dialect-native migration SQL, and deploy it. Fourteen SQL dialects:
Postgres (and CockroachDB / YugabyteDB / Redshift), MySQL (and MariaDB / TiDB),
SQL Server / Azure SQL, Oracle, Db2, SQLite, DuckDB, ClickHouse. MongoDB and Redis
carry connection settings only. Distributions: **CLI** (`foxschema` via npm/Homebrew)
and **Docker** (single amd64 image with Db2).

## Commands

Expand Down Expand Up @@ -144,7 +145,12 @@ core's.

Each SQL dialect has three layers, split across `packages/sql/src/providers/`
(dialect + settings) and `packages/db/src/providers/` (adapter + provider).
`packages/sql` lists 14 SQL dialects (MongoDB and Redis carry settings only):
Wire-compatible relatives share a family (`dialectFamily()` in
`packages/sql/src/providers/provider-settings.ts`): MariaDB / TiDB → `mysql`;
CockroachDB / YugabyteDB / Redshift → `postgres`; Azure SQL → `sqlserver`. Use
that helper instead of listing dialects by hand (Format / Monaco language, access
SQL, file-import batch size). `packages/sql` lists 14 SQL dialects (MongoDB and
Redis carry settings only):

| File | Interface | Registry |
|------|-----------|----------|
Expand Down
15 changes: 9 additions & 6 deletions docs/CODE_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ flags (`workflow`, `enterprise.channels`).
```
interfaces/ The shared vocabulary: TableSchema, TableDiff, MigrationStep.
providers/ One folder per dialect — settings, SqlDialect, and Access SQL (`*.user-sql.ts`, `*.access-sql.ts`).
`dialectFamily()` in `provider-settings.ts` maps wire-compatible
relatives onto mysql / postgres / sqlserver.
cores/ Connection strings, and shaping catalog rows into TableSchema.
modules/ One folder per domain, named to match the frontend feature
that consumes it.
Expand All @@ -74,7 +76,7 @@ modules/ One folder per domain, named to match the frontend feature
|---|---|
| `dialect` | The `SqlDialect` contract, the registry, type mapping, capability flags |
| `sql-text` | Statement splitting and SQL templating |
| `schema-diff` | Comparing two schemas, and browsing one |
| `schema-diff` | Comparing two schemas, and browsing one. `normalizeDefinitionText` is shared with Lokee |
| `migrations` | Generating DDL, ordering drops, validating a plan |
| `lokee-weave` | Content-addressed schema versioning and revert |
| `sql-editor` | FoxScript parsing, code cells, SELECT aliasing, the SQL subset |
Expand All @@ -96,7 +98,8 @@ api/ The HTTP server itself: Fastify setup, route tree, security

platform/ Cross-cutting infrastructure used by every feature.
contracts/ ActorContext and ServiceError
guards/ origin policy, rate limit, idempotency, target locks
guards/ origin policy (`FOX_ALLOWED_ORIGINS`, literal LAN IPs in dev),
rate limit, idempotency, target locks
http/ request/response types, router, Fastify binding, responses
db/ connection resolution
crypto/ secret encryption
Expand Down Expand Up @@ -186,12 +189,12 @@ the page-epoch guard, bookmarks and recents, SQL variables.
| `admin` | User and role administration |
| `auth` | Sign-in, SSO buttons, onboarding |
| `connections` | Connection modal (login method: password / Windows / LDAP), credential manager, database settings |
| `lokee-weave` | Schema history graph and version compare |
| `migrations` | Migration run history |
| `lokee-weave` | Schema history graph and version compare (Snapshots workspace) |
| `migrations` | Migration run history (Applies on the rail) |
| `object-detail` | Detail panel for a single schema object |
| `schema-diff` | Diff rendering shared by compare and history |
| `sql-editor` | SQL editor, results grid, data peek, utilities |
| `utilities` | Clone table, index management, server insights |
| `sql-editor` | SQL editor, results grid, data peek (Index/Clone/Query files live in `utilities`) |
| `utilities` | Own workspace: clone table, index management, server insights, query files, DB users & grants |
| `workflow` | Workflow designer (canvas, inspector, triggers, SQL and script editors), runs, variables, credentials and engine settings — through the engine proxy, plus linking saved connections to workflows |

## Where does my change go?
Expand Down
23 changes: 23 additions & 0 deletions docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Db2) that serves both the UI and the API on one configurable port (default **321
- [Choosing a port](#choosing-a-port)
- [Where app data lives](#where-app-data-lives)
- [Access: single-user vs. multi-user + SSO](#access-single-user-vs-multi-user--sso)
- [Origin policy](#origin-policy)
- [Cloud platforms](#cloud-platforms)
- [Database drivers](#database-drivers)
- [Building the image](#building-the-image)
Expand Down Expand Up @@ -104,6 +105,7 @@ docker compose -f docker-compose.app.yml up -d
| `ALLOW_HOST_CLOUD_CREDENTIALS` | off | When `true`, cloud secret resolve may use the host IAM/ADC chain without saved user credentials. **Keep off** on multi-user hosts. |
| `SSO_*` | — | OAuth for Google / Microsoft / GitHub (see below). |
| `NODE_ENV` | `production` | Set in the image; enforces that `APP_ENCRYPTION_KEY` is present. |
| `FOX_ALLOWED_ORIGINS` | — | Comma-separated browser origins allowed to call the API with cookies. When set, it is the entire allowlist. See [Origin policy](#origin-policy). |

> The app also reads `APP_USER_EMAIL` (only for the `v2` key scheme).
> Update checks default to the npm `foxschema` registry feed (see below).
Expand Down Expand Up @@ -258,6 +260,27 @@ hostnames).
**Always terminate TLS** (via your reverse proxy or platform) for any internet-facing
deployment — Fox Schema handles database credentials.

## Origin policy

The API holds database credentials and can run migrations, so only named browser
origins may call it with cookies (`packages/server/src/platform/guards/origin-policy.ts`).
The allowlist is explicit, not “any localhost”.

| Mode | Who may call |
|------|----------------|
| `FOX_ALLOWED_ORIGINS` set | **Only** those comma-separated origins (scheme + host + port). Wins over everything else. |
| Production, unset | The origin this process is served from, plus same-origin `fetch` (`Origin` matching this request's host). Docker / `foxschema open` work without extra config. |
| `npm run dev` | This machine's **literal** addresses (`localhost`, `127.0.0.1`, `[::1]`, and `os.networkInterfaces()` IPs) on ports **5173**, **5199**, **3210**, **3211**. Hostnames other than `localhost` are refused — DNS rebinding can point `evil.com` at 127.0.0.1, and Vite (`allowedHosts: true`) would serve it. |

A missing `Origin` is allowed (curl, health checks). A refused Origin is **403**
with `This origin is not allowed to call the Fox Schema API.` — that is why a
LAN Vite URL used to look like a blank / disconnected UI.

```bash
# Split UI hostname in production
FOX_ALLOWED_ORIGINS=https://fox.example.com,https://fox.example.com:443
```

## Cloud platforms

The image is a standard single-port web server, so it runs anywhere containers do:
Expand Down
Loading
Loading