🛟 Production Support AI Assistant

A multi-agent system that triages production issues, investigates the code, and reports the root cause.

"When I submit the Create User form, the page says it was successful,
 but the user is not saved in the database."
                          │
                          ▼
              ┌───────────────────────┐
              │     Triage Agent      │  extracts clues → infers layers
              └───────────┬───────────┘  → selects the MINIMUM agent set
                          │
        ┌─────────────────┼─────────────────┐
        ▼                 ▼                 ▼
   ┌─────────┐      ┌──────────┐      ┌──────────┐      (config, reviewer,
   │ UI Agent│      │API Agent │      │ DB Agent │       debugger: NOT run)
   └────┬────┘      └────┬─────┘      └────┬─────┘
        └─────────────────┼─────────────────┘
                          ▼
              ┌───────────────────────┐
              │   Reporting Agent     │ → root cause · fix · validation
              └───────────────────────┘

A multi-agent system that investigates production issues reported against a real, runnable sample web application. A triage agent extracts the technical clues from each report, infers the affected layer(s), and runs only the specialists the evidence justifies — creating and registering a new specialist when the issue's domain is not yet covered. Those agents read the actual source code to gather evidence, and a reporting agent delivers a structured root-cause report with a concrete fix and validation steps.

Built with LangGraph (orchestration), LangChain + Anthropic Claude (agents), Streamlit (UIs), FastAPI + SQLite (the sample application), and the GitHub MCP server (optional remote repository inspection).

No API credits? The same multi-agent workflow runs inside a Claude Project on a normal Claude subscription — see claude_project/README.md. The sample application and the full test suite never needed an API key at all; only the agents did.

---

1. The business problem

When a production issue is reported — "the form said success but the user was never saved" — the failure could live in any layer of the stack:

• the UI (swallowed an error, showed a false success),
• the API/service (accepted the request but mishandled it),
• the database (wrong SQL, missing commit, broken constraint),
• or configuration/infrastructure (wrong path, port, env var, token).

Manually walking all four layers is slow and error-prone. This project automates the first pass: triage the symptom, dispatch the right specialists in parallel, gather code-level evidence, and produce an actionable report an engineer can act on immediately.

2. System architecture

Production issue entered in Streamlit (app.py)
   ↓
Triage Agent  ──  extracts technical clues, infers the affected layer(s),
                  selects the MINIMUM set of agents from the registry
   ↓  (parallel fan-out — ONLY the selected agents run)
   ├── UI Agent                       (sample_app/frontend/)
   ├── API Agent                      (sample_app/api/)
   ├── Database Agent                 (sample_app/database/)
   ├── Configuration/Infra Agent      (env vars, paths, ports, tokens)
   ├── Code Reviewer Agent            (broad sweep, no layer implicated)
   ├── Debugger Agent                 (cross-cutting crash/trace)
   └── Dynamic Specialist             (agents triage created at runtime)
   ↓  (fan-in — waits for every dispatched specialist)
Reporting Agent
   ↓
Structured Production Support Report in Streamlit

Nothing runs by default. An agent executes only when triage names it and records the evidence that makes it necessary. If no specialist is warranted, routing goes straight to the reporting agent.

Each specialist is a tool-using (ReAct) agent: it does not magically know the code. It calls read-only tools to list, read, and search source files, and must cite file:line evidence for every claim.

The sample application it supports

A small but complete User Management app with the classic three tiers:

Streamlit frontend  ──HTTP──▶  FastAPI service  ──SQL──▶  SQLite database
sample_app/frontend/           sample_app/api/            sample_app/database/

• Frontend (sample_app/frontend/app.py): a Create User form (name, email, role) with explicit success/failure messages, plus a live users table. Talks to the API over HTTP (httpx) at API_BASE_URL.
• API (sample_app/api/): FastAPI endpoints — POST /users (201, 409 on duplicate email, 422 on invalid input), GET /users, GET /users/{id} (404), GET /health. Pydantic validation in schemas.py. The route owns the transaction: repositories execute SQL, the route commits.
• Database (sample_app/database/): SQLite schema (models.py), connection/path handling and init_db() (db.py), and a repository layer (repository.py) with typed errors.

LangGraph workflow

production_support/graph.py wires the state machine:

• START → triage: a structured-output LLM call fills a TriageDecision. That decision is the routing contract:

  {
    "issue_summary": "The users endpoint fails when creating a user.",
    "technical_clues": {
      "error_messages": ["HTTP 500"],
      "affected_feature": "create user",
      "expected_behavior": "201 with the created user",
      "actual_behavior": "500 Internal Server Error",
      "technologies": ["FastAPI"]
    },
    "affected_layers": ["api"],
    "selected_agents": ["api_agent"],
    "routing_reason": {
      "api_agent": "The report explicitly mentions an API endpoint returning HTTP 500."
    },
    "confidence": "high"
  }

• triage → specialists: a conditional edge maps selected_agents onto nodes through the registry and returns them as a list, so exactly those nodes — and no others — run in parallel in one superstep. An empty selection returns ["reporting_agent"], so no specialist runs at all. Unknown or hallucinated agent names are dropped rather than triggering a run-everything fallback.

• specialists → reporting_agent: the reporting node is registered with defer=True, so it runs exactly once, after every dispatched branch.

• State (production_support/state.py) gives each built-in specialist its own findings key (ui_findings, api_findings, …) so parallel writes never collide. All dynamic specialists run inside one node that writes dynamic_findings once.

3. The agents

Agent	Node	Selected when
Triage	agents/triage.py	Always — it is the router. Extracts clues, infers layers, selects the minimum agent set, and owns dynamic agent creation.
UI Agent	agents/ui_agent.py	The symptom is client-side: form handling, payload construction, response/error handling, rendering.
API Agent	agents/api_agent.py	The symptom names an endpoint, status code, or server error; covers routes, validation, exception handling, and the transaction boundary.
Database Agent	agents/database_agent.py	The request demonstrably succeeded but data is absent or wrong; covers schema, SQL, bindings, commits, constraints.
Config/Infra Agent	agents/config_agent.py	The symptom involves env vars, ports, URLs, credentials, paths, or deployment settings.
Code Reviewer Agent	agents/code_reviewer_agent.py	A broad request with no localized symptom. Carried over from the original Code Crew reviewer.
Debugger Agent	agents/debugger_agent.py	A concrete crash or trace that does not map onto one layer. Carried over from the original Code Crew debugger.
Dynamic Specialist	agents/dynamic_specialist.py	One or more registry agents created at runtime were selected; runs them concurrently.
Reporting Agent	agents/reporting_agent.py	Always, last. No tools. Merges triage + the findings of the agents that actually ran.

Every specialist — built-in or dynamically created — runs through the same read-only mechanism, production_support/agents/_base.py::run_specialist: identical tool loadout and prompt assembly, differing only in skill text and where the findings land.

The original Code Crew reviewer and debugger were not deleted; their fixed coordinator → reviewer → debugger pipeline was. They are now ordinary registry specialists that triage selects only when the evidence calls for them, exactly like every other agent.

3a. The agent registry

config/agent_registry.json is the catalog triage selects from, managed by production_support/registry.py. Each entry records the agent's name, domain, trigger keywords, LangGraph node, findings key, required tools, status (built-in or dynamic), and the path to its SKILL.md.

The catalog is injected into the triage prompt, so routing is grounded in agents that actually exist — and a name that is not in the registry can never schedule a node.

Dynamic specialist creation

When an issue names a technology no registered agent covers — Python/FastAPI internals, JWT/authentication, Docker/deployment, React, a security vulnerability — triage may propose a new specialist. The proposal is guarded before anything is written (registry.should_create_agent): it is refused if the name already exists, if the trigger keywords overlap an existing specialist's domain by two or more terms, or if the scope is undefined. Refusal means the existing agent is reused, and the decision is logged either way.

The two generalists (code_reviewer_agent, debugger_agent) are marked "generalist": true and are excluded from that overlap check, because they match every crash or review-shaped report by design — counting them as prior coverage would make it impossible to ever create a specialist. For the same reason the triage skill treats them as a last resort: routing a clearly domain-specific issue to a generalist instead of proposing a specialist is a routing failure, not reuse.

When a proposal is accepted, the system:

1. generates skills/<name>/SKILL.md from the spec, using the same eight-part structure as the built-in skills;
2. appends the entry to config/agent_registry.json with a timestamp and the creation reason;
3. makes it immediately selectable — it runs in this same workflow via the shared dynamic_specialist node, no graph recompilation needed;
4. logs why it was created, surfaced in the CLI and Streamlit UI;
5. persists it, so future issues in that domain route to it directly.

Dynamically created agents are read-only by construction. They receive the same inspection-only tool set as built-ins (no write, commit, or shell tools exist to hand them), and the generated skill states the constraint explicitly. A generated agent can read code and recommend changes; it cannot modify a repository, create commits, or execute destructive commands.

4. Skills (skills/*/SKILL.md)

Every agent has a SKILL.md that is loaded at runtime and becomes the body of its system prompt (production_support/skills_loader.py). They are the agents' operating manuals, not decorative docs. Each one defines:

• Role and Scope & Boundaries — what the agent owns and must not touch.
• Files to Inspect — the concrete paths for its layer.
• Investigation Process — numbered steps (each specialist is told to search_code("bug_enabled") because demo bugs hide behind those flags).
• Evidence Requirements — every claim needs a file:line citation.
• Output Format — the exact section structure of its findings.
• Escalation Conditions — when to hand the trail to another layer (e.g. API agent: "write succeeds in the handler but data missing on read → flag the database layer").
• Prohibited Behavior — no unsupported root-cause claims, never read .env, never modify anything.

5. How agents access code

Two read-only tool families, clearly separated:

Local inspection (production_support/tools/local_fs.py) — always available. Three tools scoped to the project root: list_project_files, read_project_file (numbered lines), search_code (regex). Paths are resolved and checked with is_relative_to so ../ and absolute paths are rejected, and a deny-list blocks .env, .git, *.db, and bytecode — the agents can never read secrets.

GitHub MCP (production_support/tools/github_mcp.py) — used only when a repository is supplied. Mechanism:

1. GITHUB_PERSONAL_ACCESS_TOKEN is loaded from .env.
2. make_github_client() creates a MultiServerMCPClient connected to https://api.githubcopilot.com/mcp/ over streamable HTTP.
3. The X-MCP-Readonly: true header prevents any repository modification.
4. get_github_tools() converts the server's tools into LangChain BaseTools that agents call to read repository files and issues.

If a repo is given but the token is missing or the server unreachable, the specialists degrade gracefully to local tools (with a visible note) instead of failing the run.

6. Setup

cd "Multi agent reviewer - copy"
python3 -m pip install -r requirements.txt
cp .env.example .env      # then edit .env

.env configuration:

Variable	Required	Purpose
ANTHROPIC_API_KEY	yes	powers every agent
GITHUB_PERSONAL_ACCESS_TOKEN	optional	read-only GitHub MCP inspection
API_BASE_URL	optional	where the frontend finds the API (default http://127.0.0.1:8000)
DATABASE_PATH	optional	SQLite file (default sample_app/users.db)
BUG_* flags	optional	demo bug switches (see §8)

Never commit .env; it is git-ignored and only .env.example is tracked.

7. Running everything

Sample app — API (initializes the SQLite schema on startup):

python3 -m uvicorn sample_app.api.main:app --port 8000

Sample app — frontend (second terminal):

python3 -m streamlit run sample_app/frontend/app.py

Production-support assistant UI (third terminal):

python3 -m streamlit run app.py

CLI alternative:

python3 main.py "The create-user form says success but no user is saved"

Tests (no API keys needed):

python3 -m pytest -q

8. Demonstrable example issues

The sample app is correct by default. Each demo bug is a real, flag-gated code path (sample_app/config.py::bug_enabled) so a symptom can be reproduced live and then diagnosed by the agents from actual code evidence.

Demo 1 — UI issue (BUG_UI_SILENT_FAILURE)

BUG_UI_SILENT_FAILURE=1 python3 -m streamlit run sample_app/frontend/app.py

Stop the API, submit the form → the page still says "User created!". Report to the assistant: "The form always says the user was created, even when the server is down, and no user ever appears." Expected outcome: triage routes to ui; the UI agent cites the flag-gated try/except Exception: pass around the POST in sample_app/frontend/app.py and the unconditional st.success(...), and recommends checking resp.raise_for_status() before reporting success.

Demo 2 — API issue (BUG_API_NO_COMMIT)

BUG_API_NO_COMMIT=1 python3 -m uvicorn sample_app.api.main:app --port 8000
curl -s -X POST localhost:8000/users -H 'content-type: application/json' \
     -d '{"name":"Ada","email":"ada@example.com","role":"editor"}'   # → 201
curl -s localhost:8000/users                                          # → []

Report: "When I submit the Create User form, the page says it was successful, but the user is not saved in the database." Expected outcome: triage selects ui_agent + api_agent (cross-layer symptom — the false success is rendered by one layer, the lost write executed by another) and leaves the rest unselected; the API agent finds the skipped conn.commit() in sample_app/api/routes.py::create_user — 201 is built from the inserted row, but the uncommitted transaction is discarded when the per-request connection closes.

Demo 3 — Database issue (BUG_DB_WRONG_ROLE)

BUG_DB_WRONG_ROLE=1 python3 -m uvicorn sample_app.api.main:app --port 8000

Create a user with role viewer → it is stored as admin. Report: "Every new user is created with the admin role no matter what role I pick." (optionally --category database) Expected outcome: the database agent cites sample_app/database/repository.py::create_user, where the role parameter is overwritten with the literal "admin" before the INSERT binding.

Bonus — Config issue (BUG_CONFIG_DB_PATH_MISMATCH)

BUG_CONFIG_DB_PATH_MISMATCH=1 python3 -m uvicorn sample_app.api.main:app --port 8000

Every request 500s with no such table: users. The config agent traces sample_app/database/db.py: request-time connections resolve to a different database file than the one init_db() prepared.

Example final report (abridged, from Demo 2)

# Production Support Report

## Reported Issue
The Create User form reports success, but the user is not saved.

## Affected Layer(s)
API (confirmed), UI (cleared)

## Triage Decision
Routed to ui + api: the false success is rendered by the frontend, but the
missing write is executed by the API.

## Investigation Evidence
- Agent: API Agent
- Files inspected: sample_app/api/routes.py, sample_app/database/repository.py
- Findings: create_user returns 201 built from the inserted row
  (routes.py:47), but conn.commit() (routes.py:42) is skipped when
  BUG_API_NO_COMMIT is enabled; the per-request connection then closes
  uncommitted (routes.py:26), discarding the row.

## Root Cause
The POST /users route completes the insert but never commits the
transaction; SQLite rolls back when the connection closes, so the API's 201
response describes a row that no longer exists.

## Recommended Fix
Ensure conn.commit() executes unconditionally on the success path of
create_user in sample_app/api/routes.py (remove the flag-gated skip).

## Validation Steps
1. POST /users → expect 201.
2. GET /users → the created user must appear.
3. Run: python3 -m pytest sample_app/tests/test_api.py::test_create_then_list

## Status / Confidence
Resolved — High. The cited code path fully explains the reported symptom.

Routing examples

These show the selection behavior directly — run any of them with python3 main.py "<issue>" and read the "SELECTED AGENTS" / "NOT SELECTED" block it prints.

Reported issue	Agents that run	Agents that do not
"The API returns 500 when creating a user"	api_agent	everything else
"The submit button does not send the form"	ui_agent	everything else
"The API returns success but the user is not saved"	database_agent	everything else
"The form shows success but the data is missing"	ui_agent, api_agent, database_agent	config, reviewer, debugger
"We changed API_BASE_URL and the app can't reach the service"	config_agent	everything else
"FastAPI crashes with TypeError: object is not callable"	a Python specialist (selected if registered, otherwise created)	unrelated layer agents

9. Project layout

app.py                      Streamlit assistant UI
main.py                     CLI entry point
llm.py                      shared chat-model factory
config/agent_registry.json  the agent catalog triage selects from
production_support/         triage, specialists, registry, graph, state, tools
skills/                     per-agent SKILL.md files (loaded as system prompts)
sample_app/                 the supported application (frontend / api / database / tests)
tests/                      workflow tests (selection, registry, skills, tool safety)

9a. Tests

python3 -m pytest -q runs everything without API keys. Alongside the sample app's API/database/bug-flag tests, the workflow tests drive the real compiled graph with a fake structured-output router, so triage, the registry lookup, the conditional edge, and the fan-in are all genuinely exercised — only the LLM call is stubbed. They assert that an API-only issue runs only api_agent, a UI-only issue only ui_agent, a database-only issue only database_agent, a cross-layer issue exactly the necessary combination, an empty selection no specialists at all, a Python issue creates and runs a Python specialist (with a real registry entry and SKILL.md written), and a proposal overlapping an existing agent is refused so the existing agent is reused.

10. Limitations & future improvements

• No memory or checkpointing — each analysis is stateless; LangGraph checkpointers could add resumable/persistent investigations.
• Read-only agents — they recommend fixes but never apply them; a human-approved "apply fix" step would close the loop.
• Single application scope — the built-in skills encode the sample app's architecture; supporting another app means writing new SKILL.md files (or letting triage create specialists for it).
• Dynamic agents are generated, not reviewed — a created specialist is read-only and structurally sound, but its investigation steps are only as good as the spec triage wrote. Treat config/agent_registry.json and new skills/ entries as code: review them, and delete any that prove unhelpful.
• Registry growth is unbounded — the keyword-overlap guard prevents near-duplicates, but nothing prunes agents that stop being useful.
• Routing quality depends on the report — a vague issue description yields low confidence and a wider agent set; the clues the reporter gives are what triage has to work with.
• No runtime telemetry — agents reason from source code only; feeding logs/metrics/traces would strengthen evidence.
• LLM cost/latency — a multi-layer run makes several tool-using LLM calls; findings are not cached between runs.
• GitHub MCP breadth — remote inspection quality depends on the hosted MCP server's tool surface and the PAT's scopes.