A self-hostable RAG assistant that answers questions from your organisation's own content, plus an evaluation service that measures whether those answers are any good. Built around two things most RAG deployments skip: data privacy and observability.
Development/research project, not production software. No authentication in the laptop stack, no HA, no enterprise monitoring.
The two deployments are separate. Nothing bridges them — there is no tunnel, VPN, or route from a laptop to the AWS models, and that is deliberate.
| Local (laptop) | AWS private | |
|---|---|---|
| Runs on | Docker Compose on your machine | EC2 demo instance + opt-in GPU instance |
| Answer model | OpenAI gpt-5-mini (cloud API) |
Qwen/Qwen3.5-9B on vLLM, in your VPC |
| Judge model | OpenAI gpt-5.2 (cloud API) |
Qwen/Qwen3.8-27B-FP8 on vLLM, in your VPC |
| Embeddings | self-hosted TEI container | self-hosted TEI container |
| Confidential corpus | refused by the data-policy gate | allowed |
| Cost when idle | free | roughly a coffee/month; GPU billed only while up |
| Start here | Local quick start | AWS private deployment |
A privacy demo requires the AWS deployment. The laptop stack has no local LLM option, so its only generation path is a vendor API — the policy gate correctly refuses a confidential corpus there. Full rationale: private model slate plan.
# macOS
brew install orbstack # or Docker Desktop / Podman
brew install just uv # task runner + Python env managerLinux: install Docker your distribution's usual way, then just and uv.
Nothing else is installed on the host. Embedding inference (TEI serving
Qwen/Qwen3-Embedding-0.6B) is a Compose service.
Needs: Docker, ~4GB RAM, ~2GB disk for models and data.
git clone https://github.com/gittycat/ragbench.git
cd ragbench
just setup # local venv used by `just show-config` and the eval recipesCredentials are files under secrets/, mounted at /run/secrets/<NAME> by
Compose and read at startup by Pydantic Settings. Environment variables with the
same names are ignored — this follows
OWASP secrets-management guidance.
mkdir -p secrets
echo -n "ragbench_admin" > secrets/POSTGRES_SUPERUSER
echo -n "$(openssl rand -hex 24)" > secrets/POSTGRES_SUPERPASSWORD
echo -n "ragbench_app" > secrets/RAG_SERVER_DB_USER
echo -n "$(openssl rand -hex 24)" > secrets/RAG_SERVER_DB_PASSWORD
echo -n "sk-..." > secrets/OPENAI_API_KEY # default active modelsAdd secrets/ANTHROPIC_API_KEY only if you switch an active model to Anthropic.
config.yml defines every available model; the active block picks which are
used. The checked-in defaults work as-is once an OpenAI key exists:
active:
inference: gpt5-mini # OpenAI — answers questions
embedding: qwen3-embed # self-hosted TEI, no host install
eval: gpt5-2 # OpenAI — evaluation judge
reranker: minilm-l6 # local cross-encoderSwap in any name from the models section. The qwen35-9b and qwen38-27b-judge
entries are AWS-only — they point at a non-routable placeholder until just llm-up
writes the real VPC address on the demo instance.
Breaking change: the active embedding model fixes
vector_store.dimensionand thedocument_chunks.embeddingcolumn type. Changing it invalidates every stored vector, and there are no migrations for it — you mustdocker compose down -vand re-ingest. Read getting running first.
just init # pre-fetch the reranker + warm TEI weights (~1.2GB, few minutes, once)
just up # start webapp, rag-server, task-worker, evals, postgres, tei- Web app: http://localhost:8000
- RAG API: http://localhost:8001
- Eval API: http://localhost:8002
Skipping just init is safe but the first just up spends ~200s downloading
embedding weights before tei reports healthy.
just down # stop containers, keep data
docker compose down -v # stop and DELETE the database and document storeFull procedure, pricing and teardown: docs/guide/12-private-aws-demo.md. Infrastructure detail: infra/README.md.
Prerequisites: an AWS account per infra/README.md, Node.js for CDK, and a selected environment — every recipe refuses to run without one:
setenv demo # selects AWS_ENV + AWS_PROFILE together; `setenv none` clears itjust ecr-push # 1. build arm64 images, push to ECR
just aws-bake # 2. bake the golden AMI (polls until AVAILABLE, prints elapsed)
just aws-up # 3. deploy RagbenchDemoStack, prints the demo URLThat gives you a CPU-only demo instance still calling cloud models. For private inference and judging, add the opt-in GPU stack:
just llm-up # 4. spot g6e.xlarge (L40S), both vLLM servers, up to 30 min coldjust llm-up waits for both private /health endpoints, then rewrites only the
two base_url values in the demo instance's config.yml over SSM Run Command.
The GPU has no public ingress and no laptop-reachable route. Then set
active.inference: qwen35-9b and active.eval: qwen38-27b-judge for the run.
Self-hosted is not free, and an unpriced model is dropped from cost scoring rather than counted as $0 — which silently reweights the headline score. Measure throughput and publish an explicit rate:
just llm-price <instance-usd-per-hour> <inference-tok/s> <judge-tok/s>Export the printed MODEL_PRICE_OVERRIDES in the shell that starts rag-server
and evals.
Tear down in reverse — the GPU is the expensive part, so kill it first.
just llm-down # restores the placeholder base_urls, destroys RagbenchLlmStack
just aws-down # destroys RagbenchDemoStackConfirm both CloudFormation stacks are gone before calling the demo complete.
just deploy server # base stack + Caddy TLS reverse proxy + bearer auth
just deploy cloud # base stack, pulling pre-built registry images
just deploy-down server # tear the same combination down| Command | What it does |
|---|---|
just |
list every recipe, grouped |
just up / just down / just logs |
start, stop, tail the local stack |
just build |
rebuild all images |
just show-config |
print the resolved active configuration |
just eval <args> |
run an evaluation |
just eval-compare <args> |
compare two runs |
just demo-check |
fail loudly if vector search has silently degraded to BM25-only |
just test-unit / just test-integration |
run tests |
- AWS private mode keeps a confidential corpus inside a VPC you control: both the answer model and the judge run on your own vLLM instance, so no corpus text reaches a vendor API.
- PII masking for cloud models — opt-in via
pii.enabledinconfig.yml, covering queries, chat history, retrieved context, session titles, and document ingestion. Identifiers are token-masked on the way out and restored on the way back. - A data-policy gate refuses to evaluate a confidential corpus with a judge outside the allowed execution boundary, and fails closed on a missing boundary.
Quality (accuracy, groundedness, relevance) and operations (cost, latency) are both measured. The built-in evaluation service runs automated assessments against public datasets and your own golden Q&A, distilled into five dashboard metrics:
- Retrieval Relevance — are we finding the right content?
- Faithfulness — is the answer grounded in retrieved context?
- Answer Completeness — does it cover all key points?
- Answer Relevance — does it address the question asked?
- Response Latency — is it fast enough?
These let you pick the model and setting combination that fits your data and constraints, instead of guessing.
- Backend: Python, FastAPI, PostgreSQL (pg_textsearch for BM25)
- Frontend: SvelteKit, Tailwind CSS, DaisyUI
- RAG pipeline: Docling, LlamaIndex
- Vector store: pgvector + pgvectorscale StreamingDiskANN, in the same PostgreSQL
- Search: hybrid BM25 + vector, fused with RRF
- LLM: OpenAI or Anthropic locally; private vLLM (Qwen3.5-9B / Qwen3.8-27B-FP8) on AWS
- Embeddings: self-hosted HuggingFace Text Embeddings Inference (TEI)
- Infrastructure: Docker Compose locally, AWS CDK for the demo stacks
- Operator guide — running, configuring, and tuning. Built around the tuning loop: measure a baseline, change one thing, re-measure, decide whether it helped. Covers building an evaluation set from your own documents, an experiment cookbook, privacy verification, and what the evaluations do and don't prove.
- Private AWS demo — start, price, and tear down the GPU.
- Internal documentation — architecture, RAG pipeline, retrieval, APIs, database, configuration, testing, CI/CD, and design rationale.
- Suggestions — known defects and improvement proposals.
New here? Read what this does, then getting running.
Developing on it? Prerequisites, just recipes, and testing are in
docs/internal/development.md.
Developed using Claude Code (Anthropic) as the primary coding assistant. OpenAI GPT and Google Gemini models are also used to explore alternative implementations. All code is reviewed, tested (TDD), and validated for correctness and security.
Built on the shoulders of a multitude of great open source software. MIT License