RAGu: Retrieval-Augmented Generation utility

Domain-agnostic regulatory RAG platform. Retrieves from a corpus of real U.S. statute text across federal, state, and local jurisdictions, returns cited answers grounded in source documents, and abstains when the corpus doesn't cover the question. Swap a JSON profile to navigate a different regulatory domain with no code changes.

Built on LangGraph (orchestration), Voyage AI (embeddings), Cohere (reranking), and Anthropic Claude (generation via Citations API). Served as a FastAPI endpoint.

Architecture

procure.py          Download regulation text from .gov sources
    ↓                   ↓ saves to
app/sources/        Plain-text statute files on disk
    ↓
ingest.py           Chunk → Embed (Voyage) → Store
    ↓
ragu.db             SQLite: chunks + FTS5 + vectors + metadata
    ↓
graph.py            LangGraph DAG:
                      retrieve → rerank → enough_relevance? → generate
                                                ↓ no
                                             abstain
    ↓
serve.py            FastAPI: POST /query → { answer, citations, abstained }

Quick start

python start.py

start.py handles the full pipeline:

1. Detects the active profile (from database, env var, or auto-select)
2. Installs dependencies from requirements.txt if missing
3. Checks for API keys in environment, prompts for any that are missing
4. Runs procurement (downloads source documents, skips existing files)
5. Runs ingestion (embeds new/changed sources, skips unchanged)
6. Starts the API server on port 8000

On first run, procurement fetches ~20 source documents and ingestion embeds ~100 chunks. Subsequent runs skip everything that hasn't changed and start the server in seconds.

Profile selection on startup

The database remembers which profile built it. On startup:

Condition	Behavior
DB exists with profile record	Uses the recorded profile
DB exists without profile record	Refuses to guess — asks you to specify
No DB exists	Auto-selects the first profile alphabetically (first run)
RAGU_PROFILE set, matches DB	Proceeds
RAGU_PROFILE set, doesn't match DB	Refuses — prevents cross-domain contamination

After the first successful run, python start.py remembers the profile and no env var is needed.

Running steps individually

python procure.py                 # download sources → app/sources/
python procure.py --force         # re-download all sources
python procure.py --source src-ct-ch268   # fetch one specific source

python ingest.py                  # embed new/changed sources
python ingest.py --force          # re-embed everything
python ingest.py --scope fl-state # limit to one jurisdiction

python serve.py                   # start the API server directly

Profiles

Everything domain-specific lives in a single JSON profile in profiles/. The core pipeline is domain-agnostic — swap the profile, get a different regulatory navigator with no code changes.

RAGU_PROFILE=efoil python start.py    # eFoil watercraft regulations
RAGU_PROFILE=drones python start.py   # FAA drone (UAS) regulations

Each profile defines:

Field	Purpose
jurisdictions	Jurisdiction tree (federal → state → local)
sources	Source registry with URLs, parsers, citation templates
scoping.entities	Scoping entities (water bodies, airspace zones, etc.)
scoping.entity_group_label	UI label for the scoping dropdown ("Water Bodies", "Operating Areas")
concepts	Domain-specific keyword tags for retrieval filtering
prompts	System prompt, abstention message, disclaimer
settings	Grounding threshold, retrieve/rerank K, generation model
examples	Sample queries for the frontend

To add a new domain, create a profile JSON matching the schema in app/profile.py and populate its sources.

Included profiles

Profile	Domain	Jurisdictions	Sources
efoil	eFoil watercraft regulations	6 (USCG, NY, CT, FL, LGPC, NYC DEP)	29
drones	U.S. drone (UAS) regulations	8 (FAA, TX, FL, VA, OR, MI, NYC)	16

Schema

The database schema (app/schema.sql) is domain-agnostic. Domain-specific types and categories are defined in the profile JSON, not enforced by CHECK constraints in the schema. Table structure:

Table	Purpose
jurisdictions	Jurisdiction tree with parent references
scoping_entities	Scoping entities (water bodies, airspace, etc.)
scoping_authorities	Many-to-many: scoping entity → jurisdictions
source_documents	Provenance, drift detection, conditional GET metadata
chunks	Embedded text chunks with full provenance chain
chunks_fts	FTS5 mirror for the keyword half of hybrid retrieval
query_cache	TTL-based response cache
query_history	Persistent query log across sessions
active_profile	Records which profile built the database

Migration

To migrate a database created before the RAGu rebrand (v0.0 → v0.1):

python migrate_v01.py              # default: ragu.db
python migrate_v01.py --db old.db  # custom path

The migration renames water_bodies → scoping_entities and water_body_authorities → scoping_authorities, drops the old domain-specific CHECK constraints, and preserves all existing data. Safe to run twice — detects whether migration is needed.

API keys

Three required, one optional. Set in environment (e.g. Codespace secrets) or enter interactively when start.py prompts.

Key	Service	Used by	Get it
VOYAGE_API_KEY	Voyage AI (embeddings)	ingest, query	https://dash.voyageai.com/api-keys
COHERE_API_KEY	Cohere (reranking)	query	https://dashboard.cohere.com/api-keys
ANTHROPIC_API_KEY	Anthropic (generation)	query	https://console.anthropic.com/settings/keys
NY_OPENLEG_KEY	NY Senate Open Legislation	procure	https://legislation.nysenate.gov/

eCFR and all other .gov sources require no key.

Rate limiting (Voyage free tier)

Without a payment method on Voyage, the free tier is 3 RPM / 10K TPM. The embedder automatically throttles and retries on rate-limit errors. First-time ingestion takes ~45 minutes at this rate. To remove the throttle, add a payment method to Voyage (still free for 200M tokens) and set:

export VOYAGE_RPM_LIMIT=0

API endpoints

Method	Path	Description
POST	/query	Ask a question, get a cited answer
POST	/export	Same as /query but returns a PDF
GET	/health	Liveness check + chunk count
GET	/jurisdictions	Available jurisdictions
GET	/water-bodies	Scoping entities + authority mappings
GET	/profile	Active profile metadata
GET	/defaults	Current pipeline settings
GET	/history	Persistent query history
GET	/docs	Swagger UI (auto-generated)

POST /query

curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "Do I need to register my efoil in Florida?",
    "scope": ["fl-state", "us-uscg"]
  }'

Or use water_body instead of scope for automatic jurisdiction resolution:

{ "question": "Do I need a license for my efoil on Long Island Sound?",
  "water_body": "wb-lis" }

Response:

{
  "answer": "Based on Florida statutes...",
  "citations": [
    { "title": "FL Stat. §328.48", "cited_text": "...", "start": 201, "end": 711 }
  ],
  "abstained": false,
  "scope_used": ["fl-state", "us-uscg"]
}

LangGraph DAG

g.add_edge(START, "retrieve")
g.add_edge("retrieve", "rerank")
g.add_conditional_edges("rerank", enough_relevance,
    {"generate": "generate", "abstain": "abstain"})
g.add_edge("generate", END)
g.add_edge("abstain", END)

Three inference calls per query:

1. Voyage — embed the question for vector similarity
2. Cohere — cross-encoder rerank for precision
3. Claude — grounded generation via Citations API

The grounding threshold gates the pipeline after rerank: if the best retrieval cosine score is below grounding_threshold (in the profile), the system abstains rather than guessing. The Cohere rerank_top score is captured in state and the API response for observability.

Evaluation

22-case eval set (16 answerable, 6 abstain) with a three-phase harness:

python eval/eval_harness.py            # Phase A+B (Voyage + Cohere, no Claude)
python eval/eval_harness.py --generate # + Phase C (adds Claude calls)
python eval/eval_harness.py --refresh  # re-collect Phase A data

Phase A results cache to eval/phase_a_cache.json — subsequent runs sweep instantly with zero API calls.

Current numbers (eFoil profile)

Retrieval recall@20: 0.75 (4 of 16 answerable cases missing expected citations — a chunking/embedding problem, not a threshold problem).

Cosine sweep (current gate):

threshold	answer_retention	abstain_recall	balanced
0.20	0.75	0.00	0.38
0.35	0.75	0.67	0.71
0.40	0.75	0.83	0.79
0.50	0.62	1.00	0.81

Cohere rerank_top sweep (candidate gate):

threshold	answer_retention	abstain_recall	balanced
0.08	0.75	0.50	0.62
0.20	0.69	0.83	0.76
0.30	0.69	0.83	0.76

Cosine separates better on this corpus (0.81 vs 0.76 balanced accuracy). The ceiling on answer_retention is retrieval recall — improving chunking or embeddings upstream is the highest-leverage fix.

Files

start.py           Bootstrap: profile detect → keys → procure → ingest → serve
procure.py         Download regulation source documents
ingest.py          Chunk, embed, store in SQLite (with auto-discovery)
graph.py           LangGraph RAG pipeline (Voyage + Cohere + Claude)
serve.py           FastAPI server (query, export, history, settings)
migrate_v01.py     One-time schema migration (v0.0 → v0.1)
requirements.txt   Dependencies
app/
  __init__.py      DB_PATH constant
  sources.py       Profile loader → legacy data structures
  profile.py       Pydantic profile schema + validation + DB persistence
  db.py            SQLite connection + schema init
  schema.sql       Domain-agnostic relational + retrieval schema
  fetch.py         Tiered HTTP fetching with SSL fallback
  extract.py       HTML / PDF / text extraction + cleanup
  chunk.py         Structure-aware legal text chunking
  embed.py         Voyage AI embeddings + rate-limit retry
  pipeline.py      Ingestion pipeline with content-hash skip + profile seeding
  retrieve.py      Hybrid BM25+vector retrieval, scope filtering
  sources/         Regulation text files (populated by procure.py)
eval/
  eval_harness.py  Three-phase eval: retrieve+rerank → sweep → generate
  eval_dataset.jsonl  22-case seed dataset (answerable + abstain)
  verify_corpus.py Corpus quality checker
profiles/
  efoil.json       eFoil watercraft regulatory domain profile
  drones.json      U.S. drone (UAS) regulatory domain profile

Notes

• Statutory and regulatory text is public domain.
• This is a research and triage tool, not legal advice.
• Ingestion is idempotent: unchanged sources are skipped via content hash.
• The SSL fallback in fetch.py handles .gov sites whose cert chains aren't trusted in containerized environments (e.g. Codespaces).
• Query history persists across sessions.