sqlite-jobq

A durable job queue that needs nothing but SQLite. No Redis, no broker, no daemon — one file on disk and a Python API.

from sqlite_jobq import JobQueue

q = JobQueue("jobs.db")
q.enqueue({"url": "https://example.com"}, priority=5)

job = q.claim(worker="worker-1")
if job:
    try:
        fetch(job.payload["url"])
    except Exception as exc:
        q.fail(job.id, str(exc))     # retried with exponential backoff
    else:
        q.complete(job.id)

Why

Small projects often need a queue but not a broker. A crawler, a batch of uploads, a nightly report — a few thousand jobs at most, on a single box. SQLite handles that comfortably and survives restarts, which is the part in-memory queues get wrong.

Features

• atomic claim: two workers never take the same job

• leases with heartbeat; a worker that dies returns its job to the queue

• retries with exponential backoff and jitter, then a dead-letter status

• priorities and delayed jobs

• no dependencies, Python 3.9+

Install

pip install sqlite-jobq

Worker

from sqlite_jobq import JobQueue, Worker

def handle(payload):
    print(payload)

with JobQueue("jobs.db") as q:
    worker = Worker(q, handle, name="worker-1")
    worker.install_signal_handlers()
    worker.run()

CLI

sqlite-jobq --db jobs.db enqueue '{"url": "https://example.com"}'
sqlite-jobq --db jobs.db stats
sqlite-jobq --db jobs.db worker --max-jobs 10
sqlite-jobq --db jobs.db reap        # return expired leases

Notes

The database is opened in WAL mode, so readers do not block the writer. Claims run inside BEGIN IMMEDIATE, which takes the write lock before selecting a candidate — that is what makes the claim atomic without a separate lock table.

License

MIT