System Architecture

System Architecture

This page describes the high-level architecture of the IBF-SLM Application.

Overview

The IBF-SLM Application is a web-based platform built around three core pipelines: data ingestion and validation, reasoning-ready corpus preparation, and model evaluation. A FastAPI backend serves both a browser-facing dashboard and a REST API, backed by a PostgreSQL database.

┌─────────────────────────────────────────────────────────────────┐
│                          Client Layer                           │
│              Browser  ·  REST API consumers                     │
└───────────────────────────────┬─────────────────────────────────┘
                                │ HTTPS
┌───────────────────────────────▼─────────────────────────────────┐
│                        FastAPI Application                      │
│                                                                 │
│   /auth        /dashboard      /data       /corpus   /eval      │
│                                                                 │
│   ┌──────────┐  ┌───────────┐  ┌────────────────────────────┐  │
│   │   Auth   │  │ Dashboard │  │       Core Pipelines       │  │
│   │  Router  │  │  Router   │  │  Ingestion · Corpus · Eval │  │
│   └──────────┘  └───────────┘  └────────────────────────────┘  │
└───────────┬─────────────────────────────────┬───────────────────┘
            │ SQLAlchemy (async)              │ File I/O
┌───────────▼───────────┐         ┌───────────▼───────────────────┐
│     PostgreSQL DB     │         │        Data Storage           │
│                       │         │  Raw · Annotated · Evaluated  │
│  users                │         │  datasets (JSON / CSV)        │
│  datasets             │         └───────────────────────────────┘
│  annotations          │
│  evaluations          │
└───────────────────────┘

Layers

Client Layer

Users interact via a browser-rendered dashboard (Jinja2 templates) or programmatically via the REST API. Authentication uses httponly JWT cookies; API clients use Bearer tokens.

Application Layer

The FastAPI application is the single entry point. It is structured into routers grouped by domain:

Router	Responsibility
auth	Registration, login, logout, JWT issuance
dashboard	Aggregated views, user stats
data	Dataset upload, validation, versioning
corpus	Annotation management, 5-score classification
eval	Model evaluation runs, result storage

Database Layer

PostgreSQL stores all persistent application state. SQLAlchemy 2 with asyncpg provides async ORM access. Alembic manages schema migrations.

Core tables:

Table	Description
users	Authenticated user accounts
datasets	Uploaded raw forecasting datasets
annotations	Human and model-generated reasoning annotations
evaluations	Stored model evaluation results and scores

Data Storage Layer

Raw and processed files (JSON, CSV) are stored on the local filesystem or object storage. The pipeline reads from and writes to versioned directories:

data/
├── raw/           # Ingested source datasets
├── annotated/     # Corpus after annotation
└── evaluated/     # Model outputs and scores

Core Pipelines

1. Data Ingestion and Validation

Accepts forecast event data (extreme weather reports, impact assessments). Validates schema, deduplicates records, and stores to the datasets table and data/raw/.

2. Reasoning-Ready Corpus Preparation

Transforms validated records into annotated reasoning examples. Annotators assign a 5-score classification across impact dimensions. Outputs structured JSONL suitable for SLM fine-tuning or evaluation.

3. Model Evaluation

Runs registered small language models against the prepared corpus. Scores outputs across predefined metrics and stores results to the evaluations table for comparison in the dashboard.

Authentication and Security

• Passwords hashed with bcrypt
• Sessions managed via HS256 JWT tokens stored in httponly, SameSite cookies
• Protected routes validate the token on every request; invalid or missing tokens redirect to /login

Technology Stack

Component	Technology
Web framework	FastAPI
Database	PostgreSQL 16
ORM	SQLAlchemy 2 (async)
DB driver	asyncpg
Migrations	Alembic
Templating	Jinja2
Auth	bcrypt + python-jose
Runtime	Python 3.11+
Container (DB)	Docker Compose

Deployment

In production the application runs behind a reverse proxy (e.g. Nginx) with TLS termination. The FastAPI process is managed by a process supervisor (e.g. systemd or Docker). See the Deployment Guide for details.