fastsearch

🌏 中文版: README.zh-CN.md

A single-binary, external hybrid-search engine that treats managed Postgres (pgvector) as the source of truth. It runs full-text / vector / hybrid retrieval over traceable document chunks (parsed by docparse-rs, or plain text / markdown) and carries page+bbox citations end-to-end to the answer layer — purpose-built for retrieval and grounding in AI Agents / RAG.

👉 Building an Agent? Start with Using fastsearch in an Agent (the four faces · RAG recipe · MCP · multi-tenant ACL · comparison with alternatives).

The key edge over ParadeDB: it runs on any managed Postgres (RDS / Supabase / Neon) — it only needs pgvector + logical replication, and requires no shared_preload_libraries native extension.

Release status

Surface	Repository version	Publication status (checked 2026-08-31)
Rust server / CLI / crates	0.2.0-rc.1	Release candidate; build from this repository
TypeScript SDK	0.3.0	npm currently serves 0.2.0; 0.3.0 is pending publication
Python SDK	0.2.0	Not yet published on PyPI; install from ./clients/python

The example deliberately uses the repository-local TypeScript 0.3.0 package so its shared ingestion helpers are tested before registry publication. See CHANGELOG.md for compatibility and upgrade notes.

Architecture

raw document → REST upload → ObjectStore + Postgres ingest job → independent worker
                                                        ↓ job-scoped chunks
docparse chunks / text files → Postgres (source of truth: chunk + metadata + ACL + pgvector)
                       │ logical-replication CDC (pgoutput, idempotent, LSN resumable)
                       ▼
   fastsearch engine (single binary, stateless multi-replica, derived index rebuildable)
     · BM25 inverted index (Tantivy/mmap)   · vector (brute-force / HNSW+u8 quant / pgvector direct, filter-aware)
     · fusion (RRF / normalized / weighted) · per-document ACL enforced server-side (cannot be bypassed)
     · citation traceability (page+bbox+section) + resolve_citation deep links · multimodal
   Four faces: CLI · library · REST · MCP

Five-minute quickstart

The CLI is a thin REST client of the server (like the Typesense/Qdrant/Algolia CLIs). Start a server, then the CLI talks to it — feeding a folder uploads chunks and you get full hybrid search back.

cargo build -p fastsearch-server -p fastsearch-cli
# 1) start the server (truth source / indexing / embedding / persistence all live here)
FASTSEARCH_DATA=./data FASTSEARCH_KEYS="dev=:public" ./target/debug/fastsearch-server &   # REST :8642
# 2) CLI as client (--server/--key, or env FASTSEARCH_SERVER/FASTSEARCH_KEY)
# Feed it a folder (recurses .md/.txt; markdown headings become breadcrumbs), then search
./target/debug/fastsearch index-dir --server http://localhost:8642 --key dev --collection kb ./my-docs
./target/debug/fastsearch search    --server http://localhost:8642 --key dev --collection kb --query "gross margin" --json

For docparse / PDF / REST / MCP / Python usage, see the Agent usage guide.

With DATABASE_URL and an object store configured, raw documents can enter the asynchronous job surface directly:

curl -sS -X POST http://localhost:8642/v1/documents -H 'X-API-Key: dev' \
  -F 'collection=kb' -F 'wait=auto' -F 'file=@README.md;type=text/markdown'

The default document limit is 32MiB (FASTSEARCH_MAX_DOCUMENT_BYTES); larger sources should use the source_uri form field. Worker credentials are configured separately with FASTSEARCH_WORKER_KEYS. Start the bundled worker as a separate process; it shares the server's database and object-store configuration:

DATABASE_URL=postgres://... FASTSEARCH_SERVER=http://localhost:8642 \
FASTSEARCH_WORKER_KEY=worker FASTSEARCH_OBJECT_DIR=./objects \
  cargo run -p fastsearch-ingest-worker --bin fastsearch-ingest-worker

Modules (workspace crates)

crate	Responsibility
fastsearch-core	Document model, query/filter AST, fusion (RRF / normalized / weighted), citations, ACL
fastsearch-text	Tantivy BM25 + CJK (jieba) + filtering + highlighting/facets + ACL
fastsearch-vector	In-process vector backends: deterministic brute-force/binary/TurboQuant and opt-in HNSW; filter-aware. The engine can also route directly to pgvector.
fastsearch-embed	Embedder trait + configurable HTTP backend (Ollama / OpenAI-compatible)
fastsearch-pg	Postgres source of truth: DDL, Chunk↔row mapping, doc-level replace write path, pgvector direct query
fastsearch-sync	CDC apply: pgoutput decode + idempotency + LSN checkpoint + replace semantics
fastsearch-engine	Orchestration: ingest→CDC→index→full-text / vector / hybrid search→citations + deep pagination + rebuild + media resolution
fastsearch-eval	Relevance evaluation: golden set + nDCG/recall/MRR + CI regression gate
fastsearch-server	REST (axum) + API-key auth + ACL cannot be bypassed + upload/job/worker protocol + metrics/rate-limit/audit + media gateway + CDC lifecycle
fastsearch-ingest-adapter	Shared feature-gated docparse → core adapter for CLI and worker; default builds keep only lightweight profile types
fastsearch-ingest-worker	Independent fenced job consumer: ObjectStore fetch → parse/chunk → job-scoped server publication
fastsearch-mcp	The fourth face: MCP exposing search/citation, chunk writes, and capability-gated document ingestion/status tools
fastsearch-cli	fastsearch binary: thin REST client of the server (no embedded engine). index / index-dir (feed a folder) / search / similar / ingest (client-side multi-format parse: PDF/DOCX/HTML/MD/CSV/XLSX/PPTX/SRT/EML/image + OCR + table recognition) / eval — see Ingestion & parsing
clients/{python,ts}	Zero-dependency SDKs + LangChain / LlamaIndex adapters. Registry and source-install details are recorded above and in the TypeScript / Python READMEs.

End-to-end usable: ingest/CDC → index → three search modes (keyword / vector / hybrid) → hits with citations, ACL enforced and unbypassable. All four faces in place.

Build & test

cargo test --workspace                                    # all green (PG integration runs when DATABASE_URL is set)
cargo clippy --workspace --all-targets -- -D warnings     # zero warnings
cargo fmt --all --check
DATABASE_URL=postgres://... cargo test -p fastsearch-pg   # PG integration (CI uses the pgvector/pgvector image + wal_level=logical)

Documentation

• Using fastsearch in an Agent (developer usage guide)
• Ingestion & parsing — multi-format ingest, OCR, table recognition, build tiers, model setup
• Connecting document ingestion (non-Rust callers: parser → field mapping → /v1/index; SDK adapter helpers)
• Architecture cheat-sheet / commands / invariants (CLAUDE.md)
• Module breakdown & spec index
• Requirements analysis · Product design
• Deployment · Capacity & SLO

License

Apache-2.0; see LICENSE and NOTICE. CI generates the distributable THIRD-PARTY-NOTICES.md from Cargo metadata. Tokenization dictionaries come from jieba-rs (MIT, with embedded dict) — no share-alike obligation (e.g. CC-BY-SA); shipping the MIT attribution is sufficient (see the license review).