Semantic search engine for Bialystok University of Technology (PB).
Natural-language Polish queries over the pb.edu.pl family of websites. Full hybrid retrieval pipeline: TF-IDF, BM25, dense embeddings with mmlw, RRF fusion, cross-encoder re-ranking, LLM intent classification with metadata filtering, and a RAG answer box with citations. Six selectable retrieval configurations are exposed in the backend API and on a Streamlit evaluation dashboard.
- What the project does
- Concepts in plain words
- The six retrieval configurations
- Architecture overview
- Quick start, top to bottom
- Evaluation pipeline
- GPU acceleration
- Using a local LLM with Ollama
A user types a Polish question. The backend does the following.
First, it classifies the intent of the query into one of six labels using an LLM. Free Groq tier by default; Gemini, OpenAI, or local Ollama are also supported. Labels are:
rekrutacjafor recruitmentplan_zajecfor class schedules and academic calendarkontaktfor contact info, deans' offices, career officeregulaminfor regulations, rector ordinances, statutewydarzeniafor events, conferences, open daysogolnefor general or unclassifiable
Second, it translates the intent into a metadata filter (categories and domains) so the search restricts itself to relevant subdomains. For example regulamin restricts to BIP documents; plan_zajec includes faculty schedule pages plus BIP and USOSweb for organisational ordinances.
Third, it runs a dense vector search using mmlw embeddings (768d) against Qdrant.
Fourth, it runs a BM25 lexical search in parallel.
Fifth, it fuses both ranked lists with Reciprocal Rank Fusion (RRF).
Sixth, it re-ranks the top fused candidates with a cross-encoder (BAAI/bge-reranker-v2-m3).
Seventh, it returns top results plus an LLM-generated answer with citations referring back to source documents (RAG).
The retrieval pipeline is exposed via /search?q=...&config=.... Six different config values let you compare a TF-IDF baseline against the full stack on the same query set.
The Polish-tuned model sdadas/mmlw-retrieval-roberta-base turns each document chunk and each query into a 768-dimensional vector. Similar meanings produce similar vectors. Qdrant stores the vectors and finds the nearest neighbours for the query.
A classic word-frequency-based scoring function. Good when the query and the document literally share rare words: numbers, proper nouns, names. Does not understand synonyms.
Dense embeddings catch semantic similarity (good for "how do I enroll" matching "recruitment process"). BM25 catches literal word matches (good for "Zarzadzenie 56/2026"). Reciprocal Rank Fusion combines both ranked lists: each document's final score is the sum of 1 / (k + rank) from each list. The default is k = 30.
After RRF gives the top 50 candidates, a transformer-based cross-encoder (BAAI/bge-reranker-v2-m3) reads each (query, document) pair jointly and assigns a precise relevance score. Slower than embeddings (a few seconds on CPU, milliseconds on GPU), but much sharper at putting the truly best result first.
An LLM (Groq llama-3.1-8b-instant by default) classifies the query into one of six labels using a system prompt with six few-shot examples. The label maps to a FilterSpec(categories, domains) which is applied as a Qdrant payload filter and as an in-memory filter on BM25 results. Without an LLM key the system falls back to ogolne (no filter) and still works as a hybrid plus re-ranker pipeline.
The same LLM produces a short Polish answer constrained to the top-5 retrieved chunks, with inline citations like [1], [2]. The system prompt forbids hallucinating beyond the provided context. The frontend renders the answer at the top of the results list.
Plain sklearn TfidfVectorizer plus cosine similarity. Used purely as a baseline in the six-configuration comparison, never in production retrieval.
- Recall@5 (binary hit-rate): does any of the top-5 results carry a
relevant=1label? - MRR (Mean Reciprocal Rank): average of
1 / rank-of-first-relevant. Punishes low rankings. An MRR of 0.50 means the first relevant result is on average at position 2. - nDCG@5 (Normalised Discounted Cumulative Gain): weights higher positions more heavily.
Pass ?config=<name> to /search. The same six are used as baselines on the evaluation dashboard.
| # | config |
What it does | Used in production? |
|---|---|---|---|
| 1 | tfidf |
TF-IDF cosine only, dedup by URL, no rerank, no intent | baseline only |
| 2 | bm25 |
BM25 only, dedup, no rerank, no intent | baseline only |
| 3 | dense |
mmlw plus Qdrant only, dedup, no rerank, no intent | baseline only |
| 4 | hybrid |
BM25 plus dense plus RRF fusion, dedup, no rerank, no intent | baseline only |
| 5 | hybrid_rerank |
Hybrid plus cross-encoder re-ranker, dedup, no intent | strong fallback when intent is off |
| 6 | full |
Hybrid plus rerank plus LLM intent classification plus metadata filter plus optional RAG answer box | default |
Every config also applies URL-level deduplication: instead of returning five chunks from the same PDF, the backend returns five distinct documents with the best score per URL.
[Scrapy spiders] [scripts/import_bip_pdfs.py]
kandydaci.py (kandydacipb.edu.pl) reads bip_pdfs/*.pdf
pb_main.py (pb.edu.pl + subdomains) pdfplumber -> text
biurokarier.py (biurokarier.pb.edu.pl) category from filename
eksperci.py (eksperci.pb.edu.pl)
plany.py (degra.wi + wi/we/wb/wiz/wa/wm)
HTML + PDF schedules + DOCX exam sessions
bip.py STUB (robots.txt forbids crawling)
| |
v v
output/<spider>.jsonl output/bip_manual.jsonl
\ /
v v
[token_chunker.py]
mmlw tokenizer, 400 token / 50 overlap chunks
respects existing `category` (e.g. from import scripts)
|
v
output/*_token_chunked.jsonl
|
[indexer/pipeline.py]
mmlw 768d -> Qdrant collection pb_docs (Cosine)
BM25Okapi -> output/bm25_index.pkl
TfidfVectorizer -> output/tfidf_index.pkl
^
|
[FastAPI: GET /search?q=...&config=<config>]
1. classify_intent(q) (Groq | Gemini | OpenAI | Ollama)
2. intent -> FilterSpec(categories, domains)
3. dense: "zapytanie: " + q -> mmlw -> Qdrant top-100
4. BM25: tokenize(q) -> rank-bm25 top-100
5. RRF (k=30) -> top-50
6. CrossEncoder(bge-reranker-v2-m3) -> dedup by URL -> top-N
7. (config=full) generate_answer(top-5) -> RAG answer + citations
^
| fetch + CORS *
[React + Vite frontend on :5173] [Swagger UI on :8000/docs]
[scripts/collect_eval_results.py]
For each of 6 configs, ask /search for each of 76 eval queries.
Propagates manual `relevant` labels from prior eval/results.json.
Writes eval/results_<config>.json.
[scripts/auto_label_results.py]
For every result without a `relevant` label, ask Groq LLM if
(query, intent, url, title) is relevant. Cached by (query, url).
[scripts/eval_metrics.py] CLI table comparing the 6 configs
[scripts/dashboard.py] Streamlit dashboard, sidebar config picker
Run these commands in order. Windows examples use .venv\Scripts\python.exe; on macOS or Linux replace with .venv/bin/python.
git clone https://github.com/dambeeto/SmartSearchPB
cd SmartSearchPB
python -m venv .venv.venv\Scripts\python.exe -m pip install -r requirements.txtThis pulls torch CPU through the sentence-transformers dependency chain. If you already had torch CPU installed previously and want to switch to GPU, do the swap in step 1b. Otherwise skip step 1b.
If you have an NVIDIA GPU with a recent driver and want fast indexing, replace the CPU torch with the CUDA build. You must uninstall the CPU one first:
.venv\Scripts\python.exe -m pip uninstall -y torch torchvision torchaudio
.venv\Scripts\python.exe -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
.venv\Scripts\python.exe scripts\verify_cuda.pyThe verification script prints the detected GPU and runs a small embedding batch. Expected output ends with OK - shape (4, 768), device used: cuda.
If you stay on CPU, no environment variable is needed. If you go GPU, prepend SMARTSEARCH_DEVICE=cuda to the indexer command later (or set SMARTSEARCH_DEVICE=cuda in PowerShell).
cd frontend
npm install
cd ..Requires Node.js 20.19+ or 22.12+. If you have an older Node, skip the React UI and use the Swagger UI instead (built-in, see step 10).
Copy the template and edit it:
copy .env.example .env
notepad .envPick one provider and fill it in:
INTENT_LLM_PROVIDER=groq
GROQ_API_KEY=<your key from https://console.groq.com/keys>
Groq is free as of 08-06-2026. Without a key the system still works as a hybrid plus re-ranker pipeline, but intent classification falls back to ogolne and the RAG answer box is skipped.
If you do not want to use any cloud LLM, see section 9 for a fully local setup with Ollama.
If indexer/scraper/output/ has old *.jsonl or *.pkl files from a previous run, wipe them so the new crawl writes fresh data:
del indexer\scraper\output\*.jsonl
del indexer\scraper\output\*.pklLikewise if Qdrant has an old collection volume:
docker compose down -vIf this is a clean clone, skip this step.
docker compose up -dThis starts the Qdrant container on ports 6333 (HTTP) and 6334 (gRPC). Persists data in ./qdrant_storage/.
cd indexer\scraper
..\..\.venv\Scripts\python.exe -m scrapy crawl kandydaci
..\..\.venv\Scripts\python.exe -m scrapy crawl pb_main
..\..\.venv\Scripts\python.exe -m scrapy crawl biurokarier
..\..\.venv\Scripts\python.exe -m scrapy crawl eksperci
..\..\.venv\Scripts\python.exe -m scrapy crawl plany
cd ..\..The plany spider downloads HTML, PDF and DOCX (exam-session schedules from wi.pb.edu.pl, faculty class schedules from we/wb/wiz/wa/wm.pb.edu.pl, the WI plans index on degra.wi.pb.edu.pl).
The bip spider exists but is a stub. BIP forbids all bots (Disallow: /) and even spoofing the Googlebot user agent does not work because BIP also filters by IP. Use the manual PDF import in step 6b instead.
If you have downloaded BIP PDFs (rector ordinances, senate resolutions, statute) into a bip_pdfs/ folder, import them:
.venv\Scripts\python.exe scripts\import_bip_pdfs.pyThis produces indexer/scraper/output/bip_manual.jsonl with category regulamin or rekrutacja inferred from filenames. The category lookup uses Zarzadzenie*, Uchwala*, Statut*, Regulamin* for regulamin and ZasadyRekrutacji* for rekrutacja. You can override per file by adding a bip_pdfs/bip_index.json with {filename: {url, title, category}} entries.
cd indexer\chunkers
..\..\.venv\Scripts\python.exe token_chunker.py
cd ..\..Reads every *.jsonl in the scraper output, splits text into 400-token chunks with 50-token overlap using the mmlw tokenizer, writes *_token_chunked.jsonl. Category is preserved if already set (BIP), otherwise inferred from the URL.
CPU:
.venv\Scripts\python.exe indexer\pipeline.pyGPU:
set SMARTSEARCH_DEVICE=cuda
.venv\Scripts\python.exe indexer\pipeline.pyOn CPU expect 30 to 60 minutes for a 3000-chunk corpus. On a GTX or RTX class GPU it usually finishes in 10 to 15 seconds. The pipeline wipes the previous Qdrant collection, uploads new vectors in batches of 256, then writes bm25_index.pkl and tfidf_index.pkl.
.venv\Scripts\python.exe -m uvicorn backend.app.main:app --reloadThe lifespan handler loads mmlw, the cross-encoder, the Qdrant client and both BM25 and TF-IDF pickles into app.state. Startup takes 30 to 60 seconds with both models loaded into RAM (about 4 GB). Once you see Application startup complete, the API listens on http://localhost:8000/.
Sanity checks:
curl http://localhost:8000/
curl "http://localhost:8000/search?q=rekrutacja&limit=3"
curl "http://localhost:8000/search?q=sesja+egzaminacyjna+informatyka&config=full&limit=3"
curl "http://localhost:8000/search?q=zarzadzenie+rektora+stypendium&config=hybrid_rerank&limit=3"Option A: Swagger UI (always available, no Node required).
Open http://localhost:8000/docs in any browser. Expand GET /search, click "Try it out", fill in q, choose config (one of tfidf, bm25, dense, hybrid, hybrid_rerank, full), set limit and include_answer, click "Execute". The response shows the full JSON with metadata (per-result rank_dense, rank_bm25, intent label, citations).
Option B: React frontend.
cd frontend
npm run devOpens http://localhost:5173/. Single search box, results with the highest-confidence hit on top, answer box at the top of results when config=full and an LLM key is set.
docker compose downPlus close the uvicorn and npm run dev terminals.
The system ships a 76-query evaluation set in eval/queries.json, labelled across the six intents. The pipeline below produces metrics for all six retrieval configurations and a Streamlit comparison dashboard.
.venv\Scripts\python.exe scripts\generate_eval_queries.pyUses Groq to generate 76 queries (about 13 per intent). Writes eval/queries.json. Re-run only if you want a fresh query set.
.venv\Scripts\python.exe scripts\collect_eval_results.pyFor each of the six configs and each of the 76 queries, asks /search for the top 5 results. Saves eval/results_<config>.json. Also propagates relevant: 0/1 labels from a previous eval/results.json by (query, url). Takes 10 to 20 minutes (re-ranker and intent calls dominate for hybrid_rerank and full).
.venv\Scripts\python.exe scripts\auto_label_results.pyFor every result that does not have a relevant label yet, asks Groq to judge if (query, intent, url, title) is a good match. Cached by (query, url) so each unique URL is judged once and the label is propagated to all six config files. Manual labels (without an auto_labeled: true flag) are never overwritten. Takes 5 to 10 minutes.
You can replace this step with manual labelling: open each eval/results_<config>.json and add "relevant": 0 or "relevant": 1 to each result. The LLM is a useful approximation but a careful human is more reliable.
.venv\Scripts\python.exe scripts\eval_metrics.pyPrints a table comparing all six configs (MRR, Recall@5, nDCG@5). Pass a config name (full, hybrid_rerank, ...) to also get the per-intent breakdown for that config:
.venv\Scripts\python.exe scripts\eval_metrics.py full.venv\Scripts\python.exe -m streamlit run scripts\dashboard.pyOpens http://localhost:8501/. Sidebar lets you pick one of the six configs (the rest of the page re-renders for the selection). The top of the page shows the comparison table and a bar chart for all six configs at once. The query explorer at the bottom expands each query to show the top-5 results.
The pipeline and the backend pick up the right device automatically via embedding_utils.resolve_device():
- Honour
SMARTSEARCH_DEVICEif set tocuda,cpu, ormps. - Otherwise use
cudaif available, elsemps(Apple Silicon), elsecpu.
The pipeline batches embedding generation (INDEXER_BATCH_SIZE, default 32 on CUDA) and uses torch.set_num_threads(TORCH_NUM_THREADS) on CPU for parallelism.
To go GPU on Windows with an NVIDIA card, follow step 1b in the quick start. After installing the CUDA build of torch:
set SMARTSEARCH_DEVICE=cuda
.venv\Scripts\python.exe indexer\pipeline.pyThe cross-encoder in the backend also moves to CUDA automatically. To force CPU even when CUDA is present:
set SMARTSEARCH_DEVICE=cpuIf you do not want to use any cloud LLM (no Groq, no Gemini, no OpenAI), you can run a local LLM with Ollama. It exposes an OpenAI-compatible endpoint on http://localhost:11434/v1, and the backend supports it out of the box as a fourth provider.
Download the installer from https://ollama.com and install. Linux: curl -fsSL https://ollama.com/install.sh | sh. Verify it runs:
ollama --versionollama pull llama3.1:8bOther reasonable choices for Polish intent classification and short RAG answers: qwen2.5:7b, gemma2:9b. Larger models give better answers but use more VRAM and run slower.
ollama serveLeave this terminal running. The Ollama API is now reachable on http://localhost:11434.
Edit .env (no API key required):
INTENT_LLM_PROVIDER=ollama
OLLAMA_MODEL=llama3.1:8b
OLLAMA_BASE_URL=http://localhost:11434/v1
Restart the backend. On startup it will print:
Klasyfikacja intencji + RAG: ollama / llama3.1:8b (lokalny endpoint: http://localhost:11434/v1)
The / endpoint will show intent.provider = "ollama" and intent.api_key_configured = true (Ollama needs no key). Everything else (intent classification, RAG answer box) works the same as with Groq.
Note: smaller local models tend to be weaker classifiers than Groq's llama-3.1-8b-instant (which is hosted on faster hardware and tuned for instruction following). Expect slightly lower intent accuracy. Re-rank quality is unaffected; that runs entirely on local mmlw plus bge-reranker models and does not use the LLM.
MIT. Student project for educational purposes; no warranty.