Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .cursor/rules/sql-editor-agent-memory.mdc
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ FoxSchema = schema diff/migration + SQL Editor. Destinations checklist, Schema e
### 1. Data peek (Cmd/Ctrl-click + result FK)
- **UI**: `DataPeekPanel.tsx` — modal grids; Esc closes. Each panel: WHERE / ORDER BY / LIMIT (debounced auto-apply + blur/Enter/Apply) + drag resize + arrange grip. Mounted from `SqlEditorView` (not only Schema).
- **Index fragmentation (Edit table)**: `dialect-index-fragmentation.ts` + `POST /schema/index-fragmentation`; shown on `TableBlueprintModal` index rows. **Every dialect** has a default probe (`query: true`) — physical where available, otherwise estimated / catalog listing (null %). Custom SELECT fallback (`index_name`, `fragmentation_percent`). Defrag SQL via wrench (REBUILD / REORG / OPTIMIZE / REINDEX / VACUUM by dialect).
- **Utils workspace** (`features/utilities/UtilitiesView.tsx`, left rail **Utils**): Index Management, Clone Table, insights, **DB users & grants**, Query files. Not a SQL Editor sidebar — `SqlEditorView` returns `null` for sidebar ids `utilities` / `files`. Credential chip at the top; tools dock as panes. `POST /schema/index-fragmentation-batch`; defragment via `executeSql`. **Insert SQL** from Clone Table / the table blueprint uses `insertAtCursor` fallback so SQL lands in the Editor tab even when Editor is unmounted.
- **Utils workspace** (`features/utilities/UtilitiesView.tsx`, left rail **Utils**): Index Management, **Backup & Restore** (writes commands; runs only a confirmed server-side SQL backup; lists recorded backups; never runs a restore), Clone Table, insights, **DB users & grants**, Query files. Not a SQL Editor sidebar — `SqlEditorView` returns `null` for sidebar ids `utilities` / `files`. Credential chip at the top; tools dock as panes. `POST /schema/index-fragmentation-batch`; defragment via `executeSql`. **Insert SQL** from Clone Table / the table blueprint uses `insertAtCursor` fallback so SQL lands in the Editor tab even when Editor is unmounted.
- **Edit index inline**: `TableBlueprintModal` opens the index form under the selected existing/pending index (or an Add-index row) and scrolls it into view — not a form stuck at the section bottom.
- **Clone Table (archive & recreate)**: Utils workspace → `CloneTableModal`. SQL in `generateCloneTableSql` / `nextArchiveTableName` / `generateRenameTableSql` (`tableBlueprintSql.ts`): rename live → `name_N`, free schema-unique index/FK names on archive when needed, `CREATE TABLE` empty live twin, optional indexes + outbound FKs. Inbound FKs stay on archive (warned in UI).
- **Triggers**:
Expand Down
13 changes: 10 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,13 @@ buttons with a bare Playwright `click` and then wait for the UI to look settled.
and [`apps/cli/src/tui/__tests__/README.md`](apps/cli/src/tui/__tests__/README.md)
(they document the mock seams and some real timing gotchas).
- Real cross-dialect behavior → the E2E suite.
Selectors: every control has a `data-testid`. Use `byTestId(...)` from
`apps/e2e/src/helpers/test-ids.ts` (typed against the generated catalog).
After adding a control, run `npm run test-ids` so `docs/testing/TEST_IDS.md`
and `apps/e2e/src/generated/test-ids.ts` stay in sync. How to write a test:
[docs/testing/WRITING_E2E.md](docs/testing/WRITING_E2E.md). An MCP server over
the catalog (`.mcp.json` → `scripts/test-ids/mcp-server.mjs`) helps agents
find IDs. CI comments each PR that changes the catalog.

Add or update tests with your change; a PR that changes behavior without tests will
be asked for them.
Expand All @@ -132,13 +139,13 @@ Each dialect spans both packages: `sql-dialect.ts` / `settings.ts` under
(settings, adapter, provider, sql-dialect). The exact contract — required vs.
optional hooks, cross-cutting invariants (casing, index/FK naming, DROP ordering),
and per-dialect gotchas — is documented in
[`packages/sql/src/providers/DIALECTS.md`](packages/sql/src/providers/DIALECTS.md)
(tracked; allowlisted in `.gitignore`),
[`packages/sql/src/providers/DIALECTS.md`](packages/sql/src/providers/DIALECTS.md),
with the step-by-step checklist in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
Read `DIALECTS.md` before touching any `*.sql-dialect.ts` or `sql-generator.module.ts`.
Wire-compatible relatives (MariaDB/TiDB, Cockroach/Yugabyte/Redshift, Azure SQL)
must go through `dialectFamily()` in `packages/sql/src/providers/provider-settings.ts`
— do not re-list `d === 'mysql' || d === 'mariadb' || …`.
— do not re-list `d === 'mysql' || d === 'mariadb' || …`. Backup/restore commands
are `<d>.backup.ts` registered in `packages/sql/src/modules/utilities/backup.registry.ts`.

## Conventions worth knowing

Expand Down
11 changes: 7 additions & 4 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,10 @@ packages/sql/ @foxschema/sql — dialect knowledge (pure, browser-safe
src/modules/ One folder per domain: dialect, sql-text, schema-diff,
migrations, lokee-weave, sql-editor, access, utilities
src/providers/ 14 SQL dialects, each with settings + sql-dialect
(+ optional *.user-sql.ts for account DDL)
(MongoDB and Redis carry settings only)
(+ optional *.user-sql.ts for account DDL,
*.backup.ts for backup/restore commands)
(MongoDB and Redis carry settings only; they still
have backup command builders)
src/cores/ Connection strings, catalog rows → TableSchema,
connection auth (password / Windows NTLM / Db2 LDAP)

Expand Down Expand Up @@ -177,7 +179,7 @@ only. Key hooks: `dropForeignKeyStatement`, `dropIndexStatement`, `dropTriggerSt
`createTriggerStatement`, `preDropTableStatements`, `createViewStatement`, `alterViewStatement`,
`wrapCreateSequence`, `dropTableStatement`, `dropViewStatement`, `dropSequenceStatement`,
`dropFunctionStatement`, `dropProcedureStatement`. Full hook map + fallback behavior +
per-dialect gotchas live in `packages/sql/src/providers/DIALECTS.md` (local, gitignored).
per-dialect gotchas live in `packages/sql/src/providers/DIALECTS.md` (tracked).

Version-aware DDL: `SchemaProvider.detectVersion?()` → stored in Zustand as
`targetServerVersion` → flows into `SchemaMapping` → dialect drop hooks use it. Oracle pre-23c
Expand Down Expand Up @@ -261,7 +263,8 @@ Rendering on the hot paths (what to keep when editing these components):

1. Create the dialect files in `packages/sql/src/providers/<name>/` and the driver files in `packages/db/src/providers/<name>/`
2. Register in `provider-settings.ts`, `adapter-registry.ts`, `provider-registry.ts`, `modules/dialect/registry.ts`
(and `modules/access/user-sql.registry.ts` / `modules/access/access-sql.registry.ts` when the engine has account or GRANT SQL)
(and `modules/access/user-sql.registry.ts` / `modules/access/access-sql.registry.ts` when the engine has account or GRANT SQL).
Backup/restore commands: `<d>.backup.ts` in `modules/utilities/backup.registry.ts`. A `history` query, where the engine records its backups, lets the panel list them; a server-side SQL backup can be run from the panel, a restore never is.
3. Add the dialect name to the `Dialect` union **and** the `DIALECTS` array in
`packages/sql/src/providers/provider-settings.ts` (`dialect-registry.test.ts`
fails until they match `PROVIDER_SETTINGS`). Nothing in `apps/web` needs
Expand Down
7 changes: 5 additions & 2 deletions docs/CODE_MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ flags (`workflow`, `enterprise.channels`).
```
interfaces/ The shared vocabulary: TableSchema, TableDiff, MigrationStep.
providers/ One folder per dialect — settings, SqlDialect, and Access SQL (`*.user-sql.ts`, `*.access-sql.ts`).
Backup/restore commands are `*.backup.ts` (registered in `modules/utilities/backup.registry.ts`).
`dialectFamily()` in `provider-settings.ts` maps wire-compatible
relatives onto mysql / postgres / sqlserver.
cores/ Connection strings, and shaping catalog rows into TableSchema.
Expand All @@ -81,7 +82,7 @@ modules/ One folder per domain, named to match the frontend feature
| `lokee-weave` | Content-addressed schema versioning and revert |
| `sql-editor` | FoxScript parsing, code cells, SELECT aliasing, the SQL subset |
| `access` | Permission intent, effective access, GRANT/REVOKE and account DDL (facades; emitters live in `providers/<dialect>`) |
| `utilities` | DBA queries: server insights, index fragmentation |
| `utilities` | DBA queries: server insights, index fragmentation, backup/restore command builders |

`dialect` and `sql-text` are the foundations: the other folders build on
them, never the reverse.
Expand Down Expand Up @@ -119,6 +120,7 @@ database/ The metadata store and its migrations.
|---|---|
| `access` | Database permission inspection and DBA utilities |
| `admin` | Install-wide settings, secrets, cloud credentials |
| `backup` | Per-user backup defaults (`GET`/`PUT /api/backup-settings`). Commands themselves are built in `@foxschema/sql` (`modules/utilities/backup.ts` + `providers/<d>/*.backup.ts`); the panel runs only a server-side SQL backup, after confirmation, and lists the backups the server recorded (`backupHistoryQuery`); restores are never run. |
| `auth` | Login, sessions, SSO |
| `authorization` | Role permissions (RBAC) and the permission guard |
| `compare` | Schema comparison |
Expand Down Expand Up @@ -195,7 +197,7 @@ the page-epoch guard, bookmarks and recents, SQL variables.
| `object-detail` | Detail panel for a single schema object |
| `schema-diff` | Diff rendering shared by compare and history |
| `sql-editor` | SQL editor, results grid, data peek (Index/Clone/Query files live in `utilities`) |
| `utilities` | Own workspace: clone table, index management, server insights, query files, DB users & grants |
| `utilities` | Own workspace: clone table, index management, backup & restore commands, server insights, query files, DB users & grants |
| `workflow` | Workflow designer (canvas, inspector, triggers, SQL and script editors), runs, variables, credentials and engine settings — through the engine proxy, plus linking saved connections to workflows |

## Where does my change go?
Expand All @@ -210,6 +212,7 @@ the page-epoch guard, bookmarks and recents, SQL variables.
| Reusable UI or helper | `apps/web/src/frontend/shared/` |
| Calling an API endpoint | use `api` from `@/shared/api/client` — never `fetch` directly |
| SQL for permissions or accounts | `packages/sql/src/modules/access/` (facade) + `packages/sql/src/providers/<dialect>/*.user-sql.ts` / `*.access-sql.ts`. Db2 OS-user docker steps: `buildDb2OsUserInstructions`. |
| Backup / restore commands for an engine | `packages/sql/src/providers/<dialect>/*.backup.ts` registered in `modules/utilities/backup.registry.ts`. Saved folder/format defaults: `packages/server/src/features/backup/`. |
| A dialect capability the app must branch on | `packages/sql/src/modules/capabilities/` (`dialect-features.ts`) and `packages/sql/src/modules/dialect/` |
| Guard or cross-cutting HTTP concern | `packages/server/src/platform/` |
| Workflow pipe, runtime, or store | `packages/workflow-engine/` — HTTP process in `apps/workflow-server/` ([WORKFLOW.md](WORKFLOW.md)) |
Expand Down
66 changes: 63 additions & 3 deletions docs/USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,16 @@ Docker / Homebrew / locked-down servers get **Copy command** (or use

## First run

The first time you open the UI, Fox Schema may show a short **welcome wizard** asking
for your email so you can get product updates (new dialects, releases). It is optional —
use **Skip for now** if you prefer. It only appears once per install.
Every install asks you to **sign in**. The first time, the page is **Create
your account** — the first account on this install is its administrator
(email and password). From another machine — including through a reverse
proxy — setup also asks for a one-time code printed in the server log
(`docker logs <container>` on Docker). After that, only an administrator can
add people; there is no self-registration.

After you sign in, Fox Schema may show a short **welcome wizard** asking for
your email so you can get product updates (new dialects, releases). It is
optional — use **Skip for now** if you prefer. It only appears once per install.

You land on **Home**: continue last compare / last query, **Snapshots**, **Utilities**,
and saved connections (grouped by dialect). **⌘K** / **Ctrl+K** searches workspaces and
Expand Down Expand Up @@ -463,6 +470,34 @@ saved credential at the top, then a tool:
columns so apps keep working. Toggle **Keep indexes** and **Foreign keys (auto)**
for the new table; Insert SQL or Apply. Inbound FKs from other tables still point
at the archive until you update them.
- **Backup & Restore** — Fox Schema writes the backup and restore commands
for this connection. The amber banner says *where* they run, because that is
what the folder means:
- **Your machine** (`pg_dump`, `mysqldump`, `sqlite3`, SqlPackage, `mongodump`,
`redis-cli --rdb`) — the file is written where you paste and run the command.
- **Database server** (SQL Server `BACKUP DATABASE`, Oracle Data Pump, Db2,
ClickHouse, CockroachDB, DuckDB `EXPORT DATABASE`) — the folder is a path,
DIRECTORY object, or allowed disk *on that server*.
- **Cloud** (Redshift snapshots) — there is no file to place.

Commands never contain the password (the notes say how the tool asks for it).
Folder, format, scope, compression, and “only this schema” can be **Save as my
default for** that engine (`GET` / `PUT /api/backup-settings`, per signed-in
user). Table filters and the generated file name are not saved. SQL commands
(SQL Server, CockroachDB, ClickHouse, DuckDB) offer **Open in SQL Editor**;
shell tools you copy. Names with spaces are quoted so a pasted command does
not split.

Where the backup is SQL the database server runs (SQL Server, CockroachDB,
ClickHouse, DuckDB), **Run backup now** runs it on this connection after a
confirmation that names the file and the connection. It needs **Change
schema** (`editor.ddl`), because the server treats BACKUP as a schema change.
Where the server records its backups (SQL Server's `msdb`, Db2's
`DB_HISTORY`, ClickHouse's `system.backups`), **List backups on this server**
shows them newest first; **Restore this** points the restore command at that
backup, and **Restore the newest instead** puts it back. A restore is never
run from here: it replaces a database, so it stays a command you open in the
SQL Editor and run yourself.

**Insights** (estimated where the engine has no physical figure)

Expand Down Expand Up @@ -537,6 +572,26 @@ Either family may load the catalog. Running GRANT / REVOKE still needs
**Grant privileges** (`editor.grant`). SQLite / DuckDB have no GRANT catalog;
ClickHouse has no permission builder yet.

The Access workspace opens on **Permissions** (a principal list). Pick a user
or role, then **Account** / **Grants** / **Effective**:

- **Account** — identity, membership, add / drop.
- **Grants** — one view: an object × privilege grid that opens on what that
principal **holds now** (EXECUTE on procedures and functions included), a line
under it listing what it also holds that the grid cannot show, and the
database- and schema-wide grants to edit those. A new tick is GRANT; clearing a held box is REVOKE. Boxes that
already match the catalog do not appear in the SQL. Privileges the grid cannot
show (CONNECT on the database, USAGE on a schema, DENY, unknown verbs) are
never touched — ticking SELECT on one table does not revoke the rest of the
role. When a catalog row or a grant is missing a schema (MySQL by-name
grants), the grid still matches on the object name.
- **Effective** — what they can actually do, including privileges inherited
through roles.

**Users** is CREATE / ALTER / DROP USER and ROLE. **Diff** is a separate tab:
you write a desired grant set, load the live catalog, and copy reconciliation
SQL (including DENY gaps). Fox Schema never applies Diff SQL from that screen.

What the catalog shows:

- **Roles and groups** are listed apart from users on every engine that has
Expand Down Expand Up @@ -667,6 +722,11 @@ re-enter the passwords.
the `/data` volume — make sure you didn't remove it (`docker compose down -v` deletes
volumes). See [DEPLOYMENT.md](DEPLOYMENT.md).

**Signed out while the UI was open.** A session unused for **8 hours** ends,
whatever its 7-day expiry (`FOX_SESSION_IDLE_HOURS`; `0` turns the idle limit
off). From the profile menu, **Sign out other sessions** ends every session
except this one — useful if you left a browser open elsewhere.

**Workflow engine down / Run refused.** The designer is in the UI; jobs run in a
separate process. Check **Workflow → Engine** health. New installs default to
**Disabled** — set **Enabled** and Save. `FOXFLOW_ENCRYPTION_KEY` is required at
Expand Down
37 changes: 35 additions & 2 deletions docs/releases/UNRELEASED.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ Notes for the next release. At ship time, rename this file to that version's

- **Snapshots** is its own left-rail workspace (schema history / Lokee). **Applies**
at the bottom of the rail is migration-run history. They are not the same list.
- **Utils** is its own workspace (Index Management, Clone Table, insights, Query
files, DB users & grants) — not a SQL Editor sidebar.
- **Utils** is its own workspace (Index Management, **Backup & Restore**, Clone
Table, insights, Query files, DB users & grants) — not a SQL Editor sidebar.
- Saved credentials are grouped by dialect; Home continues last compare / query.

## A development server stays on this machine
Expand Down Expand Up @@ -153,3 +153,36 @@ GitLab, Bitbucket and Azure DevOps over HTTPS with an access token.
- Remotes are `https://` only; on a shared server, private and internal
addresses are refused at connect time (including IPv6 spellings of them).

## Backup & Restore

**Utils → Backup & Restore** writes the engine's backup and restore commands
for the credential at the top of the workspace.

- The amber banner says where the command runs (your machine, the database
server, or a cloud snapshot) — that is what the folder field means.
- Passwords stay out of the command. Folder / format / scope can be saved as
*your default for this engine* (`GET` / `PUT /api/backup-settings`).
- SQL commands (SQL Server, CockroachDB, ClickHouse, DuckDB) open in the
Editor; shell tools you copy.
- Where the backup is server-side SQL, **Run backup now** runs it after a
confirmation (needs Change schema). SQL Server, Db2 and ClickHouse list the
backups they recorded; pick one and the restore reads it. Restores still only
open in the SQL Editor.

See [USER_GUIDE.md](../USER_GUIDE.md#utilities).

## Access: Grants show what is held

**Access → Permissions → Grants** opens on the privileges that principal
already holds. The SQL is only the difference (new ticks GRANT, cleared boxes
REVOKE). EXECUTE on procedures and functions is read too, so those rows open
ticked. Privileges the grid cannot show — CONNECT, schema USAGE, DENY — are
listed under it and not revoked as a side-effect of ticking one table. Permission **Diff** is a
separate tab (desired state vs catalog, including DENY).

## Sessions

A session unused for 8 hours ends (`FOX_SESSION_IDLE_HOURS`; `0` turns this
off), whatever its 7-day expiry. **Sign out other sessions** is on the profile
menu.

4 changes: 3 additions & 1 deletion docs/security/security-process.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,9 @@ Tag push v*

**Why not `--omit=dev` for the high check?** The soft warning runs without `--omit=dev` so developers see the full picture in the artifact report.

**Lockfile note:** `package-lock.json` is currently gitignored. The workflow generates it with `npm install --package-lock-only --ignore-scripts`. For more reliable and faster audits, commit the lockfile. Remove `package-lock.json` from `.gitignore` and run `npm install` once locally.
**Lockfile note:** `package-lock.json` is committed (see `.gitignore`). The
audit job runs `npm ci --ignore-scripts` so it uses that graph rather than
resolving fresh. Do not delete the lockfile from the repo.

### `secret-scan.yml`

Expand Down
Loading
Loading