Multi-agent verification. Real MCP servers. Grounded answers, not guesses.
Most "AI codebase assistant" projects are a single LLM call with some retrieved context stapled on. RepoSense is built around a different idea:
An agent should say "I don't know" when it genuinely doesn't know — and a second agent should be checking the first one's work before anything reaches you.
- 🔍 Searches before it speaks — checks a vector database of pre-indexed repo content first, falls back to live GitHub calls if needed
- 🧠 Verifies its own claims — a dedicated Fact Checker agent re-validates every draft answer against real retrieved evidence
- 🔧 Built on open protocols — custom MCP servers expose GitHub access and vector search as standalone, reusable tools
- 📊 Measured, not vibes-based — a golden evaluation set with LLM-as-judge scoring, ~93–100% accuracy
- ⚡ Streams live — watch the agent's reasoning trace (tool calls, node transitions) in real time as it works
flowchart TD
A[👤 User Question] --> B[🔬 Researcher Agent]
B -->|"1️⃣ tries first"| C[(🗄️ Qdrant<br/>Vector Search)]
B -->|"2️⃣ falls back to"| D[🐙 Live GitHub API<br/>files · issues · repo info]
C --> E[📝 Analyst Agent]
D --> E
E -->|drafts an answer| F[✅ Fact Checker Agent]
F -->|"re-verifies against<br/>actual retrieved evidence"| G[📡 Streamed Answer<br/>+ Live Reasoning Trace]
style A fill:#1C1C2E,stroke:#10B981,color:#fff
style B fill:#1C1C2E,stroke:#3B82F6,color:#fff
style C fill:#1C1C2E,stroke:#DC244C,color:#fff
style D fill:#1C1C2E,stroke:#F55036,color:#fff
style E fill:#1C1C2E,stroke:#3B82F6,color:#fff
style F fill:#1C1C2E,stroke:#10B981,color:#fff
style G fill:#1C1C2E,stroke:#10B981,color:#fff
Three agents run in sequence for every question, each with a distinct job:
| Agent | Role |
|---|---|
| 🔬 Researcher | Searches Qdrant first, falls back to live GitHub tools. Hard-capped tool-call limit with a graceful fallback if it can't find an answer. |
| 📝 Analyst | Drafts an answer using only what the Researcher actually retrieved — no outside knowledge. |
| ✅ Fact Checker | Independently re-checks the draft against the Researcher's real evidence trail. Flags or rewrites anything unsupported. |
| Layer | Tool |
|---|---|
| Agent orchestration | LangGraph |
| Tool protocol | Model Context Protocol (MCP) — 2 custom servers |
| LLM inference | Groq (openai/gpt-oss-120b) |
| Vector search | Qdrant + fastembed (local ONNX embeddings, zero API cost) |
| Backend | FastAPI + Server-Sent Events streaming |
| Frontend | Next.js, multi-turn memory, live collapsible reasoning trace |
| Package management | uv (Python), npm (frontend) |
It catches its own mistakes. During testing, the Fact Checker caught the Researcher fabricating a Rust implementation detail for a repo that's actually written in C++ — and flagged it instead of letting it through.
It's measured, not just demoed. A golden evaluation set with automated LLM-as-judge scoring covers both answerable and intentionally unanswerable questions, landing around 93–100% accuracy depending on the run.
Vector similarity search always returns its closest matches — even when none of them are actually relevant. Early on, this caused the Fact Checker to see irrelevant chunks from an unrelated repo and conclude an honest "I don't know" must be wrong, then confidently fabricate an answer instead.
Fix: added a similarity score threshold to the search tool, so it explicitly returns "no relevant results" instead of always handing back its top-k matches regardless of quality. The kind of failure mode that's easy to miss until you actually test edge cases.
- Only a small set of pre-ingested repos get full RAG-quality answers; others fall back to live GitHub calls (slower, no semantic search)
- No on-demand ingestion yet — asking about a new repo doesn't add it to the vector database automatically
- Runs entirely on free-tier infrastructure (Groq, Qdrant Cloud, Koyeb, Vercel), so occasional rate limits under heavy use
# Backend
cd backend
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt
# set GROQ_API_KEY, GITHUB_TOKEN, QDRANT_URL, QDRANT_API_KEY in .env
uv run main.py
# Frontend
cd frontend
npm install
npm run dev- On-demand ingestion for repos not yet indexed
- Linter/static-analysis MCP server — validate suggested fixes, not just describe them
- Multi-repo comparison queries
Built by Ashutosh Kumar Vishwakarma