AI Interview Performance Coach

A deployable web app: pick the roles you're targeting, speak an interview answer, get an ML-scored breakdown plus concrete, coach-style edits that would push the answer higher.

Stack: Python · FastAPI · scikit-learn · XGBoost · sentence-transformers · DistilBERT · faster-whisper · React · TypeScript · Tailwind · Docker.

Auth: passwordless email-OTP, JWT in an httpOnly cookie. Onboarding: dark-themed flow asking users to pick ≥5 roles from a 96-role catalog spanning 12 tracks. Practice questions on Home filter to their selection. Deploy: single container — FastAPI serves the built React app and the API.

Headline results

Filled in once a real-labelled holdout is collected (see data/raw/real/README.md). All numbers below are reported on the real holdout, never on synthetic val.

Model	MAE ↓	RMSE ↓	R² ↑	Macro-F1 ↑
XGBoost (engineered features)	—	—	—	—
DistilBERT (fine-tuned, regression head)	—	—	—	—
Ensemble (Ridge over both)	—	—	—	—

Inter-rater Cohen's κ on the real holdout: — (target ≥ 0.5).

Repo layout

• src/ — ML pipeline (data, features, models, STT, feedback generator).
• api/ — FastAPI service.
  • api/auth.py + api/routes/auth.py — OTP flow + JWT cookie + onboarding preferences.
  • api/catalog.py + api/routes/catalog.py — role/track catalog driven by data/roles.yaml.
• web/ — Vite + React + TS + Tailwind frontend. Dark theme by default with scroll-reveal animations and a live mic-waveform recorder.
• data/roles.yaml — 36-role taxonomy. Add roles here, no DB migration needed.
• scripts/ — runnable entry points (generate_synthetic.py, train_baseline.py, eval_holdout.py, label_real.py, make_fixture_data.py).
• Dockerfile + docker-compose.yml + .env.example — single-container deploy.

Role catalog

12 tracks · 96 roles · 93 questions — all defined in data/roles.yaml and the per-track files under data/questions/:

Track	#	Roles
Engineering	10	Backend, Frontend, Full-Stack, iOS, Android, DevOps/SRE, Security, ML, Data, QA
Data & ML	7	Data Scientist, Data Analyst, Research Scientist, Quant, BI Analyst, NLP/GenAI, Applied ML
Product	7	PM Consumer, PM B2B, PM Platform, PM Growth, Technical PM, Program Manager, Strategy/BizOps
Marketing & Growth	6	Growth Marketer, Performance Marketing, Brand Manager, Content/SEO, Product Marketing, Marketing Analyst
Design & Research	6	Product Designer, UX Designer, UI/Visual, UX Researcher, Design Systems Lead, Service Designer
Mechanical & Hardware	10	Mechanical, Mech Design, Electrical, Hardware, Aerospace, Robotics, Manufacturing, Civil/Structural, Materials, Biomedical
Business & Management	12	Management Consultant, Strategy Consultant, Ops Manager, Business Analyst, Project Manager, Chief of Staff, Biz Dev, Account Executive, Customer Success, Supply Chain, HRBP, Recruiter
Finance	10	Financial Analyst, IB Analyst, PE Associate, VC Associate, Corp Dev, FP&A, Audit/Accountant, Tax, Risk, Wealth Management
Law & Compliance	7	Corporate Lawyer, Litigation, Compliance Officer, IP/Patent, Contracts Manager, Regulatory Affairs, Privacy Counsel
Healthcare & Life Sciences	9	Physician, Nurse, Clinical Researcher, Medical Affairs, Biotech R&D, Public Health Analyst, Healthcare Admin, Pharma Sales, Clinical Trials Manager
Economics & Policy	6	Economist, Policy Analyst, Behavioral Economist, Market Research, Public Sector Researcher, Academic Researcher
Education	6	K-12 Teacher, Higher Ed Instructor, Instructional Designer, EdTech PM, L&D / Corporate Trainer, School Administrator

Each role carries its own keyword list (merged with the parent-track keywords for feedback scoring) and a curated list of question IDs from the track's question bank.

Calibration honesty note

The ML model is trained on the original 5 tracks (swe, ds, pm, marketing, ux). On the 7 newer tracks the model still produces a score and the rule-based feedback (STAR, fillers, impact, rationale, comparison, learning, pace) works fully — those are universal. The score itself is out-of-distribution and less calibrated until you collect a real holdout on the new domains and retrain. Adding labelled data for a new track is a YAML/CSV exercise; the pipeline doesn't care which track a row is in.

Local dev

# 1. Python deps
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,audio,llm]"
python -m spacy download en_core_web_sm

# 2. Train a baseline on fixture data (gives you a working /analyze).
python scripts/make_fixture_data.py
python -m src.data.split
python scripts/train_baseline.py

# 3. Run backend (terminal 1)
source .venv/bin/activate && uvicorn api.main:app --reload

# 4. Run frontend (terminal 2)
cd web && npm install && npm run dev

Open http://localhost:5173. Sign in with any email; with SMTP_HOST unset the OTP is returned in the API response so you can paste it straight in.

Deploy (single container)

# 1. Generate a secret
python -c "import secrets; print(secrets.token_urlsafe(32))"

# 2. Configure env
cp .env.example .env
# edit .env — set JWT_SECRET, COOKIE_SECURE=true, ALLOWED_ORIGINS, SMTP_*

# 3. Build & run
docker compose up --build

The container exposes port 8000. FastAPI serves the React build (/, /login, /record/*, etc.) and the API (/api/*, but since they share the same origin in production, the proxy isn't needed — the frontend calls /api/* and FastAPI routes it directly).

Works on any container host:

• Fly.io: fly launch (it'll detect the Dockerfile), then fly secrets set JWT_SECRET=... SMTP_HOST=....
• Render: new Web Service, point at the Dockerfile, paste env vars.
• Railway: new service from repo, env vars in the dashboard.
• VPS: docker compose up -d behind nginx/Caddy with a TLS cert; set COOKIE_SECURE=true.

Required env vars

Var	Why
JWT_SECRET	Signs the session cookie. Generate a fresh one — don't ship the example.
COOKIE_SECURE=true	Set when behind HTTPS so the cookie has Secure.
ALLOWED_ORIGINS	Comma-separated origins your frontend will be served from.
SMTP_HOST / SMTP_PORT / SMTP_USER / SMTP_PASS / SMTP_FROM	Real email. With Gmail use an App Password. With unset SMTP_HOST the OTP prints to the server console.
DATABASE_URL	Defaults to SQLite under a Docker volume. Swap to postgresql+psycopg://… for Postgres.

Auth + onboarding

• No passwords. Sign in by email → 6-digit OTP → JWT in an httpOnly, SameSite=Lax, Secure (in prod) cookie.
• OTPs are hashed with pbkdf2_sha256; codes expire after 10 minutes; resending is rate-limited (60s cooldown).
• First-time onboarding is a one-question-at-a-time wizard (web/src/pages/Onboarding.tsx):
  1. Name — full legal name.
  2. Preferred name — what we greet you with (defaults to first name).
  3. Degree — Bachelor's / Master's / PhD / Other.
  4. Specialisation — free-text major or course.
  5. Resume upload — PDF / DOCX / TXT. The parser (api/resume.py) feeds a context-aware extractor that produces structured SkillEvidence objects: each evidence item carries the canonical skill, its category, the section it came from, the surrounding sentence, an importance tier (primary / secondary / incidental) and a confidence score. Section detection covers profile, experience, projects, skills, education, certifications.
  6. Role suggestions — every recommendation comes with a confidence label (strong / moderate / weak), the matched skills, the high-weight skills you're missing, evidence snippets from your CV, a one-line reason, and a warning if the match is weak. See "Role matching" below.
  7. Pick more roles — the full 12-track catalog with a minimum of 5 total selections.
• _onboarding_step is computed server-side from the User row so the wizard can resume mid-flow on any device.
• After onboarding, /auth/me returns full profile (full_name, preferred_name, degree, specialisation, resume_uploaded, onboarding_step="complete").
• /auth/me, /attempts, /history, /auth/preferences, /onboarding/*, /dashboard, /rewrite all require auth. /catalog/* and /questions/* are public; /analyze is currently public.

Role matching (context-aware)

The recommender is not a keyword-overlap counter. It runs as four small modules:

• api/skill_taxonomy.py — 113 canonical skills. Each has a category, regex patterns, and (where ambiguous) requires_any_context + blocks_if_context rules. The same surface form can map to different canonicals: regression near scikit-learn / F1 / confusion matrix becomes ml_regression; regression near Selenium / Jira / bug tracking becomes qa_regression. If neither context fits, the match is dropped — under-extraction beats misclassification.
• api/skill_extractor.py — sentence-aware extractor that returns SkillEvidence (skill, category, source section, context sentence, confidence, importance). Context terms are matched with word boundaries so uat doesn't trigger inside evaluated.
• api/role_profiles.py — explicit weighted profile per role (high / medium / low / bonus) plus required_categories and anti_signals. QA requires actual testing-category evidence; mobile requires explicit swift / kotlin / Xcode; frontend has an anti-signal that drops it when the CV is dominated by ML libraries.
• api/role_matcher.py — weighted scoring with importance multipliers (primary evidence in the Experience section scores fuller than an incidental skills-list mention), required-category gating, anti-signal penalty, and an explanation generator.

Each suggestion the UI receives includes: score, confidence_label, matched_skills, missing_high_skills, evidence_snippets (the actual CV sentences), reason, optional warning, and matched_categories. The wizard's suggestion card surfaces all of it — confidence pill, matched-skill chips (green), missing-key-skill chips (grey), and 1–2 evidence quotes.

Pinned by tests/test_role_matcher.py:

Test case	Expected
"Trained scikit-learn regression / classification baselines, evaluated with F1"	ml_regression ✓, qa_regression ✗, qa-test-engineer not in suggestions
"Performed regression testing with Selenium, tracked bugs in Jira"	qa_regression ✓, ml_regression ✗, qa-test-engineer in suggestions
Mixed CV (React + Python + PyTorch + scikit-learn + NLP)	AI/ML role above frontend-engineer
React-only CV	frontend-engineer ranks first
Linux + Docker + K8s + Terraform + AWS + Python	devops-sre or backend-engineer ranks first
React + TypeScript + CSS only	android-engineer / ios-engineer absent (no mobile signal)

Personalised dashboard

GET /dashboard (api/routes/dashboard.py) is the new / for onboarded users. It surfaces:

• Greeting using the user's preferred name + a streak ("4-day streak").
• Quick stats: total attempts, mean score, current streak, Δ vs baseline (latest score - last 5).
• Top strength + Top focus area computed across saved attempts' feature dicts, normalising each rubric dimension to 0–1 (e.g., pace peaks around 130–170 wpm).
• A last-7-days area chart of mean score by day.
• Per-dimension bars (Structure / Specificity / Impact / Clarity / Pace / Relevance) so the user sees where they're rising and where they're flat.
• A suggested next question drawn from their selected roles — prefers ones they haven't attempted; falls back to one to retry.

/practice keeps the existing question browser (search + 2-column grid + track filter); /history and /record/:qid are unchanged.

Rubric

Each answer is scored on 5 dimensions, each 0/1/2; overall = sum (0–10).

Dimension	0	1	2
Clarity	Rambling	Mostly understandable	Crisp, easy to follow
Structure	None	Loose STAR	Clear Situation/Task/Action/Result
Specificity	Vague	Some specifics	Names, numbers, dates, technologies
Relevance	Off-topic	Partial	Directly answers the question
Impact	No outcome	Qualitative	Quantified ("reduced X by 30%")

Label mapping: 0–3 weak · 4–5 average · 6–8 strong · 9–10 excellent.

Coaching feedback

The feedback generator (src/feedback/generator.py) is deterministic and explainable — no LLM. For each dimension it emits:

• strength — already at the top end.
• improvement — concrete next-level edit (fires on otherwise-good dimensions when the overall score isn't in the top tier — the "what would push you from 6.7 to 10" insight).
• warning / critical — clear shortcomings.

Dimensions covered: length, fillers, STAR structure, impact (with metric count), specificity (with named-tool detection), relevance (vs gold answer), rationale, comparison/tradeoff, learning, pace, pauses. Messages include concrete numbers ("166 words", "2 role-keyword hits", "similarity 0.42") rather than vague thresholds.

Gold-answer checklist

Alongside the per-dimension feedback, every answer gets a checklist of the 8 universal components a top-tier behavioural answer contains: named system, quantified setup, first-person action, full STAR, decision rationale, comparison/tradeoff, quantified outcome, reflection. The UI renders each item with a ✓ or ✗ so the user knows exactly which components are missing — no more opaque "similarity 0.35".

Upgraded answer (LLM rewrite)

POST /rewrite returns an LLM-rewritten version of the user's transcript at a target word count, addressing the missing checklist items. Designed not to invent facts — if the user was vague ("the database"), the rewrite either keeps the wording or substitutes a clearly-bracketed placeholder like [name a specific tool, e.g. Postgres].

Enable it by setting OPENAI_API_KEY on the backend before starting uvicorn:

export OPENAI_API_KEY=sk-...
# optional: pick a different model
export REWRITE_MODEL=gpt-4o-mini   # default
uvicorn api.main:app --reload

Without the key the endpoint returns 503 — Rewrite isn't configured on this server and the UI surfaces a friendly hint to the operator. The auth dependency still gates the endpoint, and the feature is per-user.

Test

pytest                        # 38/38 — unit + auth + onboarding wizard + dashboard + catalog + rewrite + role-matcher disambiguation + e2e API
cd web && npx tsc --noEmit    # TypeScript typecheck
cd web && npx vite build      # Production bundle

Honest scope notes

• Synthetic training data is GPT-generated; the only evaluation that counts is the human-labelled real holdout.
• The DistilBERT fine-tune is wired but on small data may not beat the XGBoost baseline. scripts/train_transformer.py prints a side-by-side and tells you which to ship.
• On macOS with Apple Silicon, torch can hit a pin_memory warning on MPS during inference; the baseline path is unaffected.
• The live mic-level waveform is purely cosmetic; authoritative scoring still comes from server-side Whisper after stop.

Plan

See a-really-strong-ml-declarative-cookie.md for the full milestone plan and the off-ramps.