diff --git a/.obvious/codebase-map.md b/.obvious/codebase-map.md new file mode 100644 index 0000000..99e9599 --- /dev/null +++ b/.obvious/codebase-map.md @@ -0,0 +1,22 @@ +# Codebase Map — FlatFilers/Examples + +Three independent demo apps, no shared code. Depth cap 2. + +| Path | What it is | +|---|---| +| `S3-upload/` | Express demo: Flatfile CSV importer that uploads to Amazon S3 via presigned URLs | +| `S3-upload/index.js` | Server; `GET /` presigns an S3 `putObject` URL and renders it with the license into the view. Config block (AWS creds, license) lives inline here — but see obvious.md: env vars are what actually work | +| `S3-upload/views/index.ejs` | Importer page; loads Flatfile + PapaParse from CDNs | +| `S3-upload/public/` | Static assets: `importer.js` (client launch logic), `index.css`, `robots-clean.csv` sample data | +| `mongodb-schema-importer/` | Express demo: generates a Flatfile importer config from MongoDB documents and/or Mongoose schemas | +| `mongodb-schema-importer/index.js` | Server; `GET /` merges DB-derived field keys with Mongoose schema config, `GET /populate-defaults` seeds 4 robot documents. Reads gitignored `.env.js` | +| `mongodb-schema-importer/schemas.js` | Mongoose `Robot` schema — edit this to change the generated importer fields | +| `mongodb-schema-importer/configFromSchema.js` | Mongoose schema → Flatfile field/validator config (required, enum→select, regex, numeric, boolean→checkbox) | +| `mongodb-schema-importer/views/`, `public/` | EJS importer page + static client assets (`importer.js`, `index.css`) | +| `vue-demo/` | Vue 2 SPA demo of the Flatfile importer | +| `vue-demo/src/` | `main.js` bootstrap, `App.vue` shell, `components/Welcome.vue` (launch button, raw-output pane, reads gitignored `env.js`) | +| `vue-demo/env.example.js` | Template for gitignored `env.js` (license + importer fields) | +| `vue-demo/public/` | `index.html` shell, `robots-clean.csv` sample data | +| `.obvious/` | Agent contract: this map, `obvious.md`, `config.yml`, `skills/local-dev/SKILL.md` | + +Gitignored local files you must create before running: `mongodb-schema-importer/.env.js`, `vue-demo/env.js`, `mongodb-schema-importer/data/` (mongod dbpath). diff --git a/.obvious/config.yml b/.obvious/config.yml new file mode 100644 index 0000000..42e1bf0 --- /dev/null +++ b/.obvious/config.yml @@ -0,0 +1,6 @@ +# Repo policy for Obvious workers. +# Auto-detected from GitHub repo settings (FlatFilers/Examples) on 2026-08-24. +defaultBranch: master +# Repo allows merge commits, squash, and rebase; squash is the default choice. +mergeMethod: squash +deleteBranchOnMerge: false diff --git a/.obvious/obvious.md b/.obvious/obvious.md new file mode 100644 index 0000000..7ebc8bf --- /dev/null +++ b/.obvious/obvious.md @@ -0,0 +1,122 @@ +# FlatFilers/Examples — Agent Guide + +Monorepo of three independent Flatfile demo applications. There is **no root +package.json, no shared build, and no CI** — each top-level directory is a +standalone app with its own dependencies, lockfile, and start command. Always +`cd` into the app directory before running anything. + +## Stack + +| App | Stack | Package manager | Port | External services | +|---|---|---|---|---| +| `S3-upload/` | Node.js, Express 4, EJS, aws-sdk v2 | npm (`package-lock.json`) | 9000 | AWS S3 (presign), Flatfile license | +| `mongodb-schema-importer/` | Node.js, Express 4, EJS, Mongoose 5, mongodb driver 3.0 | npm (`package-lock.json`; `yarn.lock` also present) | 9000 | MongoDB (local), Flatfile license | +| `vue-demo/` | Vue 2, Vue CLI 3 (webpack 4) | yarn classic 1.x (`yarn.lock`) | 8080 | Flatfile license | + +- Runtime: Node.js 20 (verified). Plain JavaScript — no TypeScript, no test suites, no Docker/Compose. +- **`S3-upload` and `mongodb-schema-importer` both hardcode port 9000** (`app.listen(9000)` in each `index.js`). Run one at a time; stop the other first. +- Verified toolchain in the sandbox snapshot: Node 20.20.2, npm 10.8.2, yarn 1.22.22, mongod 4.4.29, Playwright Chromium (screenshots). + +## Commands + +### S3-upload (Express, port 9000) + +```bash +cd S3-upload +npm install +AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=us-east-1 npm start +# → http://localhost:9000 +``` + +Credentials go in **env vars, not the `config` object in `index.js`**: the S3 +client is constructed before `aws.config.update()` runs, so the in-code config +never takes effect and the page renders empty (`CredentialsError`). Dummy +AWS-shaped values are enough for local page rendering — SigV4 presigning is a +local computation — but real credentials are needed for actual uploads. + +### mongodb-schema-importer (Express + MongoDB, port 9000) + +```bash +cd mongodb-schema-importer +npm install +# create .env.js (gitignored) — see below +npm run mongod # mongod --dbpath=data --port 28015 (needs mongod ≤ 4.x on PATH) +npm start # → http://localhost:9000 +curl http://localhost:9000/populate-defaults # seeds 4 robots +``` + +`.env.js` (gitignored, must be created): + +```js +module.exports = { + URL: 'mongodb://localhost:28015', + DBNAME: 'flatfile-demo', + COLLECTION: 'robots', + FF_LICENSE: '' +} +``` + +**MongoDB version constraint:** the pinned `mongodb@~3.0.8` driver speaks the +legacy OP_QUERY wire protocol. MongoDB ≥ 5.1 rejects it +(`UnsupportedOpQueryCommand`) and the app crashes on `/populate-defaults`. +Use **mongod 4.4.x** (installed at `/home/user/.local/bin/mongod`, symlinked +to `/usr/local/bin/mongod` in the snapshot; needs `libssl1.1` on Ubuntu 24.04). + +### vue-demo (Vue 2 dev server, port 8080) + +```bash +cd vue-demo +yarn install # yarn classic 1.x (yarn.lock v1) +cp env.example.js env.js # gitignored; fill in license + fields +NODE_OPTIONS=--openssl-legacy-provider yarn serve # → http://localhost:8080 +``` + +`NODE_OPTIONS=--openssl-legacy-provider` is **required on Node ≥ 17** (webpack 4 +MD4 hashing). Without it the dev server dies with +`error:0308010C:digital envelope routines::unsupported`. + +### Lint / tests + +- `cd vue-demo && yarn lint` — the only lint target in the repo; passes (`No lint errors found!`). +- No test suites or typechecks exist in any app; every `npm test` script is a stub that exits 1. + +## Codebase map + +See [codebase-map.md](codebase-map.md) for the folder-level overview. + +## Local Verification Summary + +Verified 2026-08-24 in sandbox `i517zqa9z31yfh4v3rhj1` (Node 20.20.2): + +| Check | Result | +|---|---| +| `S3-upload` `npm start` + `GET /` | HTTP 200, page renders with presigned S3 URL (dummy env creds) | +| `mongodb-schema-importer` `npm run mongod` + `npm start` | mongod 4.4.29 up on 28015; app up on 9000 | +| `GET /populate-defaults` | HTTP 200, "Inserted 4 Robots" | +| `GET /` (DB-driven) | HTTP 200; DB_config fields `name, nick, helmet, color, id` derived from seeded collection; Mongoose Robot schema config (6 fields) rendered | +| `vue-demo` `yarn serve` | Compiled successfully; HTTP 200 on 8080; HMR active | +| `vue-demo` `yarn lint` | Pass — no lint errors | +| Browser screenshots | `s3-upload-home.png`, `mongo-importer-home.png`, `vue-demo-home.png` (Playwright Chromium, in `/home/user/evidence/`) | + +Primary user flows exercised: (1) mongo app — seed collection then render the +importer page whose field list is generated from live DB documents; (2) S3-upload — +render page with a freshly presigned S3 upload URL; (3) vue-demo — SPA boots, +"Import robots" button + sample CSV link present, HMR connected. + +## Sandbox snapshot + +- Snapshot/template ID: `gap0pvit10wten6lqhs1:default` (sandbox `i517zqa9z31yfh4v3rhj1`) +- Captured: 2026-08-24T21:29:58.777Z +- State at capture: all three apps' `node_modules` installed; `.env.js` and `env.js` present (gitignored); mongod 4.4.29 running on 28015 with seeded `robots` collection; `mongodb-schema-importer` app on 9000; `vue-demo` dev server on 8080; git tree clean. + +## Known limitations + +- Flatfile license keys are local placeholders. The in-browser importer widget + (external SaaS) cannot fully launch without a real license — the page, config + generation, and server flows all work; the widget launch does not. + TODO(confirm): obtain a Flatfile license key to exercise the full import flow. +- `S3-upload` needs real AWS credentials for actual S3 uploads (presigning works with dummies). +- EJS pages load Flatfile/PapaParse scripts from unpkg/cdnjs — browser needs network access. +- `mongodb-schema-importer` ships both `package-lock.json` and `yarn.lock`; running + `npm install`/`npm ci` there rewrites `yarn.lock` — restore it with + `git checkout -- mongodb-schema-importer/yarn.lock` before committing. diff --git a/.obvious/skills/local-dev/SKILL.md b/.obvious/skills/local-dev/SKILL.md new file mode 100644 index 0000000..d2ff0a0 --- /dev/null +++ b/.obvious/skills/local-dev/SKILL.md @@ -0,0 +1,72 @@ +--- +name: local-dev +description: Bring up the three Flatfile demo apps in FlatFilers/Examples locally (Express apps, local mongod 4.4, Vue dev server) and verify the primary flows +--- + +# Local Dev — FlatFilers/Examples + +Durable record of the onboarding run (2026-08-24, sandbox `i517zqa9z31yfh4v3rhj1`, +Node 20.20.2). Restoring the snapshot `gap0pvit10wten6lqhs1:default` gives you +this state pre-built: deps installed, configs created, mongod seeded, two apps running. + +## Prerequisites (baked into the snapshot) + +- Node 20 + npm; yarn classic 1.22.x (`npm i -g yarn@1.22.22`) — vue-demo needs yarn, not npm. +- **mongod 4.4.29** at `/home/user/.local/bin/mongod`, symlinked to `/usr/local/bin/mongod`. + On a fresh Ubuntu 24.04 box: install `libssl1.1` (bionic deb from security.ubuntu.com), + then the `mongodb-linux-x86_64-ubuntu2004-4.4.29` tarball. **Do not use mongod ≥ 5.1** — + the pinned `mongodb@~3.0.8` driver uses OP_QUERY and MongoDB 8.x rejects it. +- Playwright + Chromium (global, `/home/user/.npm-global`) for screenshots; system deps + via `sudo playwright install-deps chromium`. + +## Bring-up sequence + +```bash +cd /home/user/work/Examples + +# 1. MongoDB (dbpath is gitignored, port 28015 — matches .env.js) +cd mongodb-schema-importer && mkdir -p data && nohup mongod --dbpath=data --port 28015 --bind_ip 127.0.0.1 > /tmp/mongod.log 2>&1 & cd .. + +# 2. mongodb-schema-importer app (port 9000) +cd mongodb-schema-importer && nohup npm start > /tmp/mongo-app.log 2>&1 & cd .. +curl -s http://localhost:9000/populate-defaults # seed 4 robots → "Inserted 4 Robots" + +# 3. vue-demo dev server (port 8080) — legacy OpenSSL flag is mandatory on Node ≥ 17 +cd vue-demo && NODE_OPTIONS=--openssl-legacy-provider nohup yarn serve > /tmp/vue-demo.log 2>&1 & cd .. + +# 4. S3-upload (port 9000 — conflicts with step 2; stop it first: kill the node PID from `ss -ltnp | grep :9000`) +cd S3-upload && AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY AWS_REGION=us-east-1 nohup npm start > /tmp/s3-upload.log 2>&1 & cd .. +``` + +## Required local files (all gitignored) + +- `mongodb-schema-importer/.env.js` → `{ URL: 'mongodb://localhost:28015', DBNAME: 'flatfile-demo', COLLECTION: 'robots', FF_LICENSE: '' }` +- `vue-demo/env.js` → copy of `env.example.js` with placeholder license + fields + +## Verification checklist (all passed 2026-08-24) + +1. `curl -w '%{http_code}' http://localhost:9000/` → 200; app log shows + `Connected successfully to server` and `Found the following robots traits: ['name','nick','helmet','color','id']`. +2. `curl http://localhost:9000/populate-defaults` → `Inserted 4 Robots`. +3. `curl -w '%{http_code}' http://localhost:8080/` → 200; vue log shows `Compiled successfully`. +4. `cd vue-demo && yarn lint` → `No lint errors found!`. +5. S3-upload `GET /` → 200 with `var signedRequest = "https://s3.amazonaws.com/..."` in the HTML. +6. Screenshots via `NODE_PATH=/home/user/.npm-global/lib/node_modules node /home/user/evidence/shot.js `. + +## Gotchas learned the hard way + +- **S3-upload credentials**: set `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`/`AWS_REGION` + env vars. The inline `config` object in `index.js` is applied *after* `new aws.S3()`, + so it never reaches the client — with no env vars the page renders empty (HTTP 200, 0 bytes). +- **Port 9000 is hardcoded** in both Express apps. Check for a listener before starting + (`ss -ltnp | grep :9000`); kill by PID — never `pkill -f "node index.js"` (it matches your own shell). +- **vue-demo on Node ≥ 17**: `NODE_OPTIONS=--openssl-legacy-provider` or webpack 4 dies with + `digital envelope routines::unsupported`. +- **npm in mongodb-schema-importer rewrites `yarn.lock`** (npm syncs it when present). + Run `git checkout -- mongodb-schema-importer/yarn.lock` before committing anything. +- **mongod 8.x breaks the app**: `Unsupported OP_QUERY command: insert` → assertion crash + on `/populate-defaults`. If `data/` was created by a newer mongod, wipe it before + starting 4.4 (`rm -rf mongodb-schema-importer/data && mkdir -p mongodb-schema-importer/data`). +- Long-running shell commands cap at ~300s — start each service in its own short command. +- The Flatfile importer widget itself needs a real license key (external SaaS); everything + server-side works with placeholders.