Segment App

Research workstation for interactive 3D medical-image segmentation. Not a medical device. No automated diagnosis. See Disclaimer.

An end-to-end platform for researchers working with volumetric medical imaging (CT, MRI, X-ray, 2D slices). The web app combines a multi-planar Cornerstone3D viewer, an interactive segmentation studio backed by remote SAM-Med3D / MedSAM-2 inference, quantitative measurements, radiomics, experiment-level provenance, and a research-only LLM copilot.

Highlights

Three-viewport MPR viewer (axial / sagittal / coronal) with crosshair sync, brush, W/L, zoom/pan, cine, labelmap overlay
3D (SAM-Med3D) and 2D (MedSAM-2) segmentation via pluggable remote HTTP adapters
One-click quantitative measurements (volume, surface area, diameter, statistics) and MONAI radiomics
Automatic experiment tracking: every run is persisted with inputs, params, and outputs
LLM copilot (OpenAI-compatible) with five whitelisted tools — refuses to diagnose
DICOM and NIfTI I/O, with DICOM-SEG export via highdicom
Hardened production stack: Caddy reverse proxy, gunicorn, non-root containers, request-size and per-client rate limits, security headers, JSON logs
Single npm run dev for local development, single docker compose up for production

Install & run (TL;DR)

Already have Python 3.13, Node 20+, and uv? Run this from the repo root and open the URLs printed at the end:

macOS / Linux

cd backend && uv sync --extra dev && cp .env.example .env && cd ..
(cd frontend && npm install)
npm run dev

Windows (PowerShell)

cd backend; uv sync --extra dev; copy .env.example .env; cd ..
cd frontend; npm install; cd ..
npm run dev

Then open:

App: http://localhost:3000
API docs: http://localhost:8000/docs

Open backend/.env in your editor to fill in the AI / LLM keys — see Environment variables. If you only want to try the viewer and upload features, the app starts without any keys; the segmentation and copilot tabs will simply report that the upstream is unreachable.

For the full step-by-step (prerequisites, what each script does, how to debug one side in isolation) keep reading.

Architecture

┌──────────────────────────── Browser ────────────────────────────┐
│  Next.js 16  +  React 19  +  TypeScript  +  Tailwind 4         │
│  ┌────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────┐  │
│  │TopBar  │  │LeftPanel     │  │RightPanel    │  │Bottom    │  │
│  │Tools   │  │Studies/Cases │  │AI / Controls │  │Measure,  │  │
│  │        │  │              │  │              │  │Logs, LLM │  │
│  └────────┘  └──────────────┘  └──────────────┘  └──────────┘  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  CS3D Viewer Grid: Axial │ Sagittal │ Coronal  + overlay │  │
│  └──────────────────────────────────────────────────────────┘  │
└──────────────────────────┬─────────────────────────────────────┘
                           │  REST  /typed fetch
┌──────────────────────────▼─────────────────────────────────────┐
│  FastAPI  (gunicorn + uvicorn workers, non-root, tini PID-1)   │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────┐  │
│  │ files    │ │ studies  │ │ segment. │ │ measure. │ │ llm  │  │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────┘  │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────┐  │
│  │annotat.  │ │ radiom.  │ │experim.  │ │  models  │ │ auth │  │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────┘  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ Services: storage · dicom · nifti · ai · llm · radiomics │  │
│  │           · measurement · export · maskops              │  │
│  └──────────────────────────────────────────────────────────┘  │
│       SQLite (WAL) ─── original/ nifti/ masks/ exports/ logs/ │
└──────────┬──────────────────────────┬──────────────────────────┘
           │                          │
           ▼                          ▼
   Remote AI (HTTP)            OpenAI-compatible LLM
   · SAM-Med3D (3D)            · Research-only tools
   · MedSAM-2  (2D)            · 5 whitelisted tools

The AI engine is a remote HTTP service — the app never loads the model itself. Swap SAM_MED3D_API_URL / MEDSAM2_API_URL to point at your own inference endpoint. See backend/app/services/ai_external.py and medsam_external.py.

For the full architecture spec, including the tech-stack pinning table and the list of explicitly rejected packages, see the source tree under backend/app/ and frontend/src/ — the code is the source of truth.

Quickstart (development)

Prerequisites

Tool	Version	Install
Python	3.13	https://www.python.org/downloads/
Node.js	20+	https://nodejs.org
uv (Python package manager)	latest	`pip install uv` · or https://docs.astral.sh/uv/
Git	any	https://git-scm.com

Verify your toolchain:

python --version   # Python 3.13.x
node --version     # v20.x or newer
uv --version       # uv 0.x

No GPU required. The model never runs in this repo — inference is delegated to a remote HTTP endpoint that you configure in backend/.env.

1. Clone

git clone https://github.com/<your-username>/segment-app.git
cd segment-app

2. Backend (FastAPI)

The backend lives in backend/. uv creates an isolated virtualenv in backend/.venv/ and installs everything from pyproject.toml.

macOS / Linux / WSL

cd backend
uv sync --extra dev
cp .env.example .env
cd ..

Windows (PowerShell)

cd backend
uv sync --extra dev
copy .env.example .env
cd ..

Open backend/.env in any editor. The minimum to start the app is nothing — but to actually run segmentation you will need to fill in the variables listed in Environment variables:

Variable	Required for
`SAM_MED3D_API_URL` + `SAM_MED3D_API_KEY`	3D segmentation tab
`MEDSAM2_API_URL` + `MEDSAM2_API_KEY`	2D segmentation tab
`LLM_BASE_URL` + `LLM_API_KEY`	LLM Copilot tab

The viewer, upload, measurements, and experiments work without any keys.

3. Frontend (Next.js)

macOS / Linux / WSL

cd frontend
npm install
cd ..

Windows (PowerShell)

cd frontend
npm install
cd ..

4. Run

From the repo root:

npm run dev

This single command starts both servers in one terminal, prefixes their output with [backend] / [frontend], and shuts them both down on Ctrl+C. It also detects if either is already running and reuses it.

When it is ready you will see:

[dev] backend:  http://localhost:8000
[dev] frontend: http://localhost:3000

Open:

Workstation: http://localhost:3000
API docs (dev only): http://localhost:8000/docs

Useful variants

npm run dev:backend    # only FastAPI on :8000 (good for API debugging)
npm run dev:frontend   # only Next.js on :3000 (good for UI work)

npm run lint           # ruff + ESLint
npm run typecheck      # tsc --noEmit
npm run test           # pytest

To stop everything: Ctrl+C. On Windows the script taskkills the child Node/Python processes for a clean shutdown.

Common first-run gotchas

uv: command not found — install uv (pip install uv) and reopen the terminal.
npm install fails behind a corporate proxy — set npm config set proxy http://... and https-proxy first.
Port 8000 or 3000 already in use — the dev script detects this and warns; free the port or change it in scripts/dev.js.
backend/.env not picked up — the file must live directly in backend/, not the repo root. The dev launcher passes the cwd through.
AI segmentation fails with HTTP 401/403 — double-check the API keys in backend/.env and restart npm run dev (the backend only reads the env on startup).

Production deploy

The repo ships a complete production stack: docker-compose.yml builds the backend (gunicorn + uvicorn workers, non-root) and the frontend (Next.js 16 standalone) and runs them behind a Caddy reverse proxy with auto-TLS, HSTS, and gzip/zstd.

# 1. Configure secrets — never commit this file
cp .env.example .env
# edit .env: HOSTNAME, CORS_ORIGINS, SAM_MED3D_*, MEDSAM2_*, LLM_*

# 2. Build and start
docker compose up -d --build

# 3. Verify
docker compose ps                                    # all 3 services healthy
curl -fsS https://segment.example.com/healthz
curl -fsS https://segment.example.com/readyz

Persistent data lives in the named volume segment-data (DB + nifti + masks + exports + logs). Back it up at /var/lib/docker/volumes/segment-app_segment-data/_data/ or extend the compose file with a backup service.

For bare-metal / VM deploys without Docker:

npm run build      # builds the frontend (backend is interpreted)
npm run start      # starts gunicorn + node .next/standalone

See the Hardening checklist below for the production guarantees already in the box.

Environment variables

All variables are optional in development; in production the startup validator fails fast if the required ones are missing.

Backend (`backend/.env` or compose env)

Variable	Purpose	Default
`DATA_DIR`	SQLite + nifti/original/masks/exports/logs	`backend/app/data`
`ENVIRONMENT`	`development` / `production`	`development`
`LOG_LEVEL`	`INFO` / `DEBUG` / `WARNING`	`INFO`
`CORS_ORIGINS`	JSON list of allowed origins	`["http://localhost:3000"]`
`MAX_UPLOAD_SIZE_MB`	Reject larger with 413	`2048`
`ENABLE_RATE_LIMIT`	Toggle per-client rate limit	`true`
`RATE_LIMIT_PER_MINUTE`	Per-client limit (per process)	`120`
`TRUST_FORWARDED_HEADERS`	Set `true` behind a known proxy	`false`
`SAM_MED3D_API_URL`	Remote 3D segmentation endpoint	—
`SAM_MED3D_API_KEY`	Bearer token for SAM-Med3D	—
`SAM_MED3D_TIMEOUT_SEC`	HTTP timeout	`120`
`MEDSAM2_API_URL`	Remote 2D segmentation endpoint	—
`MEDSAM2_API_KEY`	Bearer token for MedSAM-2	—
`MEDSAM2_TIMEOUT_SEC`	HTTP timeout	`60`
`LLM_BASE_URL`	OpenAI-compatible base URL	`https://api.openai.com/v1`
`LLM_API_KEY`	LLM bearer token	—
`LLM_MODEL`	Model name	`gpt-4o-mini`
`LLM_TIMEOUT_SEC`	HTTP timeout	`60`

Frontend (`frontend/.env.local`, optional)

Variable	Purpose	Default
`NEXT_PUBLIC_API_BASE`	Backend base URL	`http://localhost:8000`

Runtime tuning (compose / bare-metal)

Variable	Default	Notes
`HOST`	`0.0.0.0`	Backend bind address
`PORT`	`8000`	Backend port
`WORKERS`	`2`	Gunicorn workers
`FRONTEND_PORT`	`3000`	Frontend port

API surface

FastAPI auto-generates OpenAPI docs at /docs in development. The top-level routers are:

Router	Prefix	Purpose
`files`	`/api/files`	Multipart upload (DICOM / NIfTI) + streaming download
`studies`	`/api/studies`	Studies, cases, datasets; list / create / search
`segmentation`	`/api/segmentation`	Trigger 2D/3D segmentation, fetch masks
`annotations`	`/api/annotations`	Save / list annotations per volume
`measurements`	`/api/measurements`	Quantitative measurements on a mask
`radiomics`	`/api/radiomics`	MONAI radiomics feature extraction
`llm`	`/api/llm`	Chat completions with whitelisted tools
`experiments`	`/api/experiments`	List / inspect experiment runs (provenance)
`models`	`/api/models`	Model registry and capabilities
`auth`	`/api/auth`	Stub for the deferred auth layer

/healthz (liveness) and /readyz (readiness) are exposed for orchestrators.

Project layout

segment-app/
├── README.md                 ← you are here
├── LICENSE                   ← MIT
├── CONTRIBUTING.md
├── SECURITY.md
├── CHANGELOG.md
├── .env.example              ← copy to .env, fill in keys
├── .gitignore
├── package.json              ← root scripts (dev / build / start / test / lint)
├── docker-compose.yml        ← production stack
├── Caddyfile                 ← reverse proxy (HSTS, gzip, TLS)
│
├── medsam2_server.ipynb      ← notebook for the 2D inference endpoint
│
├── backend/                  ← FastAPI + SQLModel
│   ├── app/
│   │   ├── main.py
│   │   ├── config.py
│   │   ├── db.py
│   │   ├── models.py
│   │   ├── schemas.py
│   │   ├── middleware.py
│   │   ├── logging_config.py
│   │   ├── core/             ← exceptions, ids
│   │   ├── services/         ← storage, dicom, nifti, ai, llm, radiomics, …
│   │   └── routers/          ← files, studies, segmentation, …
│   ├── tests/                ← pytest
│   ├── data/                 ← gitignored: nifti/ original/ masks/ exports/ logs/
│   └── pyproject.toml
│
├── frontend/                 ← Next.js 16 + React 19 + CS3D
│   ├── src/
│   │   ├── app/              ← App Router (layout + page)
│   │   ├── components/       ← shell, viewer, segmentation, measurements, llm, …
│   │   ├── hooks/            ← useSegmentation, useUpload, useModels
│   │   ├── stores/           ← Zustand: workspace, viewer, segmentation, log
│   │   └── lib/              ← api.ts (typed fetch), types.ts, cornerstone-init.ts
│   ├── next.config.ts
│   ├── tailwind.config.ts
│   └── package.json
│
├── scripts/
│   ├── dev.js                ← concurrent dev launcher
│   └── start.js              ← production launcher (gunicorn + node)

Hardening checklist

Everything in this list is already wired in the box.

What is not in this hardening pass (and why)

Authentication — explicitly deferred. The data API assumes a trusted-network deployment. Add an OIDC layer or reverse-proxy auth in front of the API before exposing it publicly.
Multi-replica rate limiting — the current RateLimitMiddleware is per-process. For multiple replicas, swap the storage to Redis (the interface is the same).
Persistent telemetry — Sentry / OpenTelemetry are intentionally excluded from the MVP. JSON logs are shippable to any aggregator as-is.
PHI redaction — out of scope for research-only use. Do not load identifiable patient data.

Development

Root scripts (run from the repo root):

npm run dev                 # start backend + frontend in dev mode
npm run dev:backend         # backend only
npm run dev:frontend        # frontend only

npm run build               # build frontend + sanity-check backend deps
npm run start               # production launcher (gunicorn + node)
npm run start:backend       # gunicorn only
npm run start:frontend      # node .next/standalone only

npm run lint                # frontend (eslint) + backend (ruff) + mypy
npm run typecheck           # frontend (tsc --noEmit)
npm run test                # backend pytest
npm run test:frontend       # frontend vitest

Conventions:

Backend — Python 3.13, uv for deps, ruff for lint/format, mypy --strict for types, pytest + pytest-asyncio for tests.
Frontend — TypeScript strict, ESLint (next config), Tailwind 4, vitest + Testing Library for tests.
Commits — short imperative subject (≤72 chars), reference the affected router/service in the body when relevant.

Testing

# Backend
cd backend
uv run pytest
uv run ruff check app/ tests/
uv run mypy app/

# Frontend
cd frontend
npx tsc --noEmit
npm run lint
npm run build
npm run test

Disclaimer

This software is for research and educational use only.

It is not a medical device. It does not provide a diagnosis, a prognosis, or a treatment recommendation. It is not FDA-cleared, CE-marked, or approved by any regulatory authority for clinical use.

The LLM copilot is configured to refuse to interpret imaging or suggest a diagnosis. If a clinician has a question, the copilot will refer them to a qualified radiologist or the appropriate care pathway.

Do not upload data containing Protected Health Information (PHI) to a deployment that is not on a network you control and trust. The researchers running this software are responsible for IRB approval, data-handling compliance, and any clinical decisions made using its outputs.

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for setup, coding conventions, and the PR process. By submitting a pull request you agree to license your contribution under the project's MIT license.

Security

For responsible disclosure of vulnerabilities, see SECURITY.md. Do not file public issues for security problems.

License

MIT. Dependencies retain their own licenses (MONAI: Apache-2.0, scikit-image: BSD, highdicom: MIT, Cornerstone3D: MIT).

Navigation Menu

Repository files navigation

Segment App

Table of contents

Install & run (TL;DR)

Architecture

Quickstart (development)

Prerequisites

1. Clone

2. Backend (FastAPI)

3. Frontend (Next.js)

4. Run

Useful variants

Common first-run gotchas

Production deploy

Environment variables

Backend (`backend/.env` or compose env)

Frontend (`frontend/.env.local`, optional)

Runtime tuning (compose / bare-metal)

API surface

Project layout

Hardening checklist

What is not in this hardening pass (and why)

Development

Testing

Disclaimer

Contributing

Security

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github		.github
backend		backend
frontend		frontend
scripts		scripts
.env.example		.env.example
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
Caddyfile		Caddyfile
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
docker-compose.yml		docker-compose.yml
medsam2_server_colab.ipynb		medsam2_server_colab.ipynb
package.json		package.json
sam_med3d_server_colab.ipynb		sam_med3d_server_colab.ipynb

Latest commit

History

Folders and files

Repository files navigation

Segment App

Table of contents

Install & run (TL;DR)

Architecture

Quickstart (development)

Prerequisites

1. Clone

2. Backend (FastAPI)

3. Frontend (Next.js)

4. Run

Useful variants

Common first-run gotchas

Production deploy

Environment variables

Backend (backend/.env or compose env)

Frontend (frontend/.env.local, optional)

Runtime tuning (compose / bare-metal)

API surface

Project layout

Hardening checklist

What is not in this hardening pass (and why)

Development

Testing

Disclaimer

Contributing

Security

License

About

Topics

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Backend (`backend/.env` or compose env)

Frontend (`frontend/.env.local`, optional)