A Rust-based backend workflow engine for reliable async processing.
FlowForge demonstrates how real backend systems handle background workflows, event routing, retry policies, dead-letter handling, durable persistence, structured tracing, metrics, and operational visibility — built with Rust, Axum, Tokio, SQLx, and PostgreSQL.
FlowForge is a production-style workflow engine MVP. It accepts workflow events over a REST API, stores them durably in PostgreSQL, and processes them asynchronously via a Tokio background worker. Failed workflows are retried automatically up to a configurable limit; permanently failed workflows are moved to a dead-letter state for investigation.
The system is designed to be observable: every meaningful state transition is logged with structured tracing, and a /metrics endpoint surfaces real-time counts and average processing time without requiring an external observability platform.
FlowForge was built to explore reliability patterns that appear in real backend systems: durable persistence, async workers, retry limits, dead-letter handling, audit trails, and operational visibility.
In production systems, background jobs fail. Networks time out. Downstream services return 503s. ML inference pipelines encounter schema mismatches. A naive system drops these jobs and loses data. A reliable system:
- Stores the job durably before attempting it.
- Marks it as in-progress atomically to prevent double-processing.
- Retries it on transient failures with a bounded attempt count.
- Records the failure reason for debugging.
- Moves permanently failed jobs to a dead-letter state for human review.
- Surfaces all of this through metrics and structured logs.
FlowForge models all of these behaviors across multiple workflow types — from insurance claim submissions to ML job queuing — to demonstrate the pattern is domain-agnostic.
┌─────────────────────────────────────────────────────────────┐
│ HTTP Client │
└───────────────────────────┬─────────────────────────────────┘
│ REST API
▼
┌─────────────────────────────────────────────────────────────┐
│ Axum HTTP Server │
│ POST /workflows GET /workflows GET /metrics /health │
└───────────────────────────┬─────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Services Layer │
│ workflow_service.rs metrics_service.rs │
└───────────────────────────┬─────────────────────────────────┘
│
┌─────────────┴─────────────┐
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────────────────┐
│ PostgreSQL (SQLx) │ │ Tokio Background Worker │
│ workflow_events │◄────│ poll → claim → process → │
│ workflow_attempts │ │ complete / retry / dead-letter │
└─────────────────────┘ └─────────────────────────────────┘
The HTTP server and background worker share a single PgPool. The worker uses an optimistic-locking pattern (WHERE status IN ('pending', 'failed')) to claim workflows atomically without advisory locks.
| Layer | Technology |
|---|---|
| Language | Rust (2021 edition) |
| HTTP Framework | Axum 0.7 |
| Async Runtime | Tokio 1 |
| Database | PostgreSQL 16 |
| DB Access | SQLx 0.7 (compile-time verified) |
| Serialization | Serde + serde_json |
| Logging | tracing + tracing-subscriber |
| Config | dotenvy |
| Containers | Docker Compose |
| CI | GitHub Actions |
- REST API — create, list, retrieve, and inspect workflow events
- Durable storage — every workflow is persisted to PostgreSQL before processing begins
- Background worker — Tokio task polls for eligible workflows on a configurable interval
- Retry policies — configurable
max_attemptsper workflow; failed workflows are retried automatically - Dead-letter handling — workflows that exhaust retries are moved to
dead_letteredfor investigation - Audit trail —
workflow_attemptstable records every individual attempt with timestamps and errors - Structured logging — every state transition is logged with tracing fields (workflow ID, type, attempt)
- Metrics endpoint — real-time counts by status and average processing time without Prometheus
- Deterministic processing simulation — each workflow type has predictable behavior for testing
- Dockerized — one
docker compose upgets you a running PostgreSQL instance - CI pipeline — GitHub Actions runs fmt, clippy, and cargo test on every push
┌──────────┐
POST / │ pending │
workflows └────┬─────┘
│ worker claims it
▼
┌──────────────┐
│ processing │
└──────┬───────┘
│
┌─────────┴──────────┐
│ success │ failure
▼ ▼
┌───────────┐ ┌──────────┐
│ completed │ │ failed │◄──── retry (if attempts < max_attempts)
└───────────┘ └─────┬────┘
│ attempts >= max_attempts
▼
┌───────────────┐
│ dead_lettered │
└───────────────┘
| Workflow Type | Behavior |
|---|---|
user.created |
Always succeeds on first attempt |
document.uploaded |
Always succeeds on first attempt |
claim.submitted |
Always succeeds on first attempt |
notification.send |
Fails on attempt 1 (503 from provider), succeeds on retry |
payment.review_requested |
Fails on even-numbered attempts, succeeds on odd |
ml_job.queued |
Always fails — exhausts retries and becomes dead_lettered |
Retry rules:
- Each workflow starts with
attempts = 0and a defaultmax_attempts = 3. - The worker atomically increments
attemptswhen claiming a workflow. - On failure: if
attempts < max_attempts, status →failed(eligible for retry). - On failure: if
attempts >= max_attempts, status →dead_lettered. - The error reason is stored in
last_erroron every failure. - Every attempt writes a row to
workflow_attemptsfor audit purposes.
| Method | Path | Description |
|---|---|---|
| GET | /health |
Service liveness check |
| POST | /workflows |
Submit a new workflow event |
| GET | /workflows |
List all workflow events (newest first) |
| GET | /workflows/:id |
Get a single workflow event with full payload |
| GET | /workflows/:id/status |
Get status, attempts, errors, and timestamps |
| GET | /metrics |
JSON metrics snapshot |
POST /workflows request body:
{
"workflow_type": "claim.submitted",
"payload": {
"claim_id": "CLM-1001",
"customer_id": "CUS-204",
"amount": 2500
},
"max_attempts": 3
}GET /metrics response:
{
"total_workflows": 10,
"pending": 2,
"processing": 1,
"completed": 5,
"failed": 1,
"dead_lettered": 1,
"average_processing_time_ms": 120.5
}# Health check
curl http://localhost:8080/health
# Submit a claim workflow
curl -X POST http://localhost:8080/workflows \
-H "Content-Type: application/json" \
-d '{
"workflow_type": "claim.submitted",
"payload": {
"claim_id": "CLM-1001",
"customer_id": "CUS-204",
"amount": 2500
}
}'
# Submit a notification that will fail once then succeed
curl -X POST http://localhost:8080/workflows \
-H "Content-Type: application/json" \
-d '{
"workflow_type": "notification.send",
"payload": {
"user_id": "USR-42",
"channel": "email",
"template": "welcome"
}
}'
# Submit an ML job that will always fail and dead-letter
curl -X POST http://localhost:8080/workflows \
-H "Content-Type: application/json" \
-d '{
"workflow_type": "ml_job.queued",
"payload": {
"model": "fraud-detector-v2",
"input_schema_version": "0.9"
}
}'
# List all workflows
curl http://localhost:8080/workflows
# Get a specific workflow
curl http://localhost:8080/workflows/YOUR-UUID-HERE
# Check status of a workflow
curl http://localhost:8080/workflows/YOUR-UUID-HERE/status
# View metrics
curl http://localhost:8080/metricsPrerequisites: Docker, Rust (1.75+), and sqlx-cli.
# 1. Clone the repo
git clone https://github.com/SilasDarko/FlowForge.git
cd FlowForge
# 2. Start PostgreSQL
docker compose up -d
# 3. Copy and configure environment
cp .env.example .env
# 4. Run migrations
sqlx migrate run
# 5. Start the server
cargo runThe server starts at http://127.0.0.1:8080. The background worker begins polling immediately.
# Install sqlx-cli
cargo install sqlx-cli --no-default-features --features rustls,postgres
# Run all pending migrations
sqlx migrate run
# Revert the last migration
sqlx migrate revertMigrations live in migrations/ and are also applied automatically on server startup via sqlx::migrate!.
cargo testTests in tests/workflow_tests.rs cover:
- Deterministic processing simulation for all workflow types
- Retry logic:
notification.sendsucceeds on the second attempt - Dead-letter behavior:
ml_job.queuedexhausts retries - Payment review deterministic failure pattern
- Failure messages are non-empty for known failure types
No live database is required to run these tests.
# 1. Submit an ML job
curl -X POST http://localhost:8080/workflows \
-H "Content-Type: application/json" \
-d '{"workflow_type": "ml_job.queued", "payload": {"model": "fraud-v2"}}'
# 2. Note the returned UUID
# 3. Watch the status change across poll cycles
watch -n 1 'curl -s http://localhost:8080/workflows/YOUR-UUID/status | python3 -m json.tool'
# 4. After 3 attempts, status becomes dead_lettered
# 5. View full metrics
curl http://localhost:8080/metricsServer logs show every state transition in real time:
INFO flowforge: Processing workflow workflow_id=... workflow_type=ml_job.queued attempt=1
WARN flowforge: Workflow processing failed – will retry workflow_id=... reason=ML inference failed
INFO flowforge: Processing workflow workflow_id=... workflow_type=ml_job.queued attempt=2
WARN flowforge: Workflow processing failed – will retry workflow_id=...
INFO flowforge: Processing workflow workflow_id=... workflow_type=ml_job.queued attempt=3
WARN flowforge: Workflow exhausted retries – moving to dead_lettered workflow_id=...
Structured logs — Every log line is a key-value record that can be parsed by Datadog, CloudWatch, or any structured logging pipeline:
INFO Workflow created workflow_id=<uuid> workflow_type=claim.submitted
INFO Processing workflow workflow_id=<uuid> attempt=1 max_attempts=3
INFO Workflow processing succeeded workflow_id=<uuid>
WARN Workflow processing failed – will retry workflow_id=<uuid> reason=...
WARN Workflow exhausted retries – moving to dead_lettered workflow_id=<uuid>
Set RUST_LOG=debug to see worker poll cycles. Set RUST_LOG=trace for maximum verbosity including SQLx queries.
Metrics — Poll GET /metrics from a cron job, uptime monitor, or dashboard. The response is plain JSON with no external dependencies.
flowforge/
├── src/
│ ├── main.rs # Entry point: config, pool, worker spawn, server bind
│ ├── lib.rs # Library crate root (exposes modules for tests)
│ ├── config.rs # Environment-based configuration
│ ├── db.rs # Connection pool initialization and migration runner
│ ├── error.rs # AppError type with IntoResponse for Axum
│ ├── models.rs # WorkflowEvent, WorkflowStatus, request/response types
│ ├── routes/
│ │ ├── mod.rs # Router construction
│ │ ├── health.rs # GET /health
│ │ ├── workflows.rs # POST /workflows, GET /workflows, GET /workflows/:id
│ │ └── metrics.rs # GET /metrics
│ ├── services/
│ │ ├── mod.rs
│ │ ├── workflow_service.rs # Database operations for workflows
│ │ └── metrics_service.rs # Metrics aggregation query
│ └── worker/
│ ├── mod.rs
│ └── processor.rs # Poll → claim → simulate → apply outcome loop
├── migrations/
│ ├── 20240101000001_create_workflow_events.sql
│ └── 20240101000002_create_workflow_attempts.sql
├── tests/
│ └── workflow_tests.rs # Unit tests for processing logic
├── .github/workflows/ci.yml # GitHub Actions: fmt, clippy, test, build
├── docker-compose.yml
├── .env.example
├── .gitignore
└── Cargo.toml
These are intentionally excluded from the MVP to keep the scope clean and demonstrable. They represent realistic production hardening steps:
| Improvement | Why |
|---|---|
| Kafka ingestion | Decouple producers from the processing engine at scale |
| Prometheus + Grafana | Time-series metrics, alerting, SLA dashboards |
| OpenTelemetry tracing | Distributed trace context across services |
| Exponential backoff | Configurable retry delay strategy per workflow type |
| Admin UI | Browse, filter, and requeue dead-lettered workflows |
FlowForge was built to demonstrate backend engineering depth across async workers, durable event storage, retry logic, and operational visibility — patterns that apply equally to insurance, fintech, AI platforms, and general distributed systems.