Skip to content

feat(refid): add raw-SQL sqlite backend - #177

Closed
sthanikan2000 wants to merge 3 commits into
mainfrom
feat/refid-raw-sql-backends
Closed

feat(refid): add raw-SQL sqlite backend#177
sthanikan2000 wants to merge 3 commits into
mainfrom
feat/refid-raw-sql-backends

Conversation

@sthanikan2000

@sthanikan2000 sthanikan2000 commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds github.com/OpenNSW/core/refid/sqlite, mirroring refid/postgres's shape (DefaultTableName, Option, WithTableName, New, Migrate, Next) via database/sql + the pure-Go modernc.org/sqlite driver — no CGO. Next is the same single atomic upsert-and-increment statement as postgres, just SQLite dialect.
  • New pins the passed *sql.DB to a single connection (db.SetMaxOpenConns(1)): SQLite allows only one writer at a time, and without this, concurrent Next calls can each open their own connection and race into a SQLITE_BUSY ("database is locked") error instead of one simply queueing behind the other. Since Next is one fast statement with no transaction held across calls, this costs negligible throughput.
  • Documents the SQLite backend and the bring-your-own-Store escape hatch in the README.

Why

A pure-Go SQLite option avoids imposing a CGO build requirement on every downstream service for an optional backend, and gives local development and tests a real, on-disk backend with no external service dependency — tests run unconditionally in-process, unlike the Postgres backend's POSTGRES_TEST_DSN-gated integration test.

This PR is purely additive on top of #176 (stacked on that branch, per its updated description) — refid/postgres keeps working unchanged throughout. Base is refactor/refid-store-interface, not main; retarget once #176 merges.

Test plan

  • go build ./..., go vet ./..., golangci-lint run ./... all clean in refid/
  • go test -race ./... passes in refid/, including a 50-goroutine concurrent-Next-no-duplicates test (initially caught the SQLITE_BUSY issue SetMaxOpenConns(1) fixes, before the fix was added)

@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: ed03dc14-5383-4e2d-b5e0-0f00143ab492


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sthanikan2000
sthanikan2000 force-pushed the refactor/refid-store-interface branch from 2045f46 to b5e1731 Compare September 2, 2026 06:37
@sthanikan2000
sthanikan2000 force-pushed the feat/refid-raw-sql-backends branch from ef7a644 to 9880eca Compare September 2, 2026 06:41
@sthanikan2000 sthanikan2000 self-assigned this Sep 2, 2026

@ginaxu1 ginaxu1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL these issues:

  1. New mutates the caller's pool (refid/sqlite/store.go)
    db.SetMaxOpenConns(1) applies to every user of that *sql.DB, not just Next. A caller that shares the handle (normal SQLite usage: one file, one pool) has all queries serialized. Worse, a held *sql.Tx occupies the only connection, so Generate/Next waits forever. The package comment says this backend "makes no requirements on the caller's *sql.DB connection-pool configuration"; the constructor does the opposite.

Suggested fix: do not change pool settings. Guard Next with a mutex so in-process concurrent Generate cannot hit SQLITE_BUSY. For lock waits on the file (other connections or processes), set busy_timeout on every connection, e.g. document opening with _pragma=busy_timeout(5000) in the DSN (a one-shot PRAGMA busy_timeout on db.Exec only affects that one pooled connection). Keep the 50-goroutine test; it should still pass with the mutex and no SetMaxOpenConns. Add a test that Begin()s a Tx on the same *sql.DB, then calls Next with a short context deadline: it must not hang.

  1. Multi-process Store contract is unmet
    refid.Store requires concurrent safety "across multiple process instances sharing the same backing store". A second process (or a second sql.Open on the same file) still gets SQLITE_BUSY because busy_timeout is never set. SetMaxOpenConns(1) only serializes one *sql.DB.

Suggested fix: same DSN/pragma as (1). If this backend is single-process (dev/tests only), say that on the package and do not claim it satisfies the multi-process clause. Production multi-instance deployments stay on postgres.

@sthanikan2000

Copy link
Copy Markdown
Collaborator Author

@ginaxu1 Good catch, both fixed in f84ee67.

  1. Removed db.SetMaxOpenConns(1)New no longer touches the pool. Next is now guarded by an in-process mutex instead, so concurrent calls queue instead of racing into SQLITE_BUSY. Added the Begin()-a-Tx-then-Next test you suggested; confirmed it actually catches the old bug before restoring the fix.
  2. Agreed we can't fully close this for SQLite — docs now say plainly this backend covers single-process use only, recommend _busy_timeout in the DSN for cross-connection/process waiting, and point multi-instance deployments at refid/postgres instead.

Checked #176 too — neither point applies there. Postgres's New never touches the pool, and its single atomic statement is already safe across processes by construction (real row-level locking, not a whole-file lock).

Base automatically changed from refactor/refid-store-interface to main September 2, 2026 08:55
Adds github.com/OpenNSW/core/refid/sqlite, mirroring the postgres
backend's shape (DefaultTableName, Option, WithTableName, New,
Migrate, Next) via database/sql + the pure-Go modernc.org/sqlite
driver (no CGO) — a lighter build/deploy footprint than a CGO-based
driver for a shared library consumed by many services. Next is a
single atomic upsert-and-increment statement, same as postgres, just
SQLite dialect (numbered placeholders, unqualified DO UPDATE columns,
no TIMESTAMPTZ).

New pins the passed *sql.DB to a single connection
(db.SetMaxOpenConns(1)): SQLite allows only one writer at a time, and
without this, two concurrent Next calls can each open their own
connection and race into a "database is locked" (SQLITE_BUSY) error
instead of one simply queueing behind the other. Since Next is a
single fast statement (no transaction held across calls), this costs
negligible throughput for the low-volume counters this backend targets.

Tests run unconditionally in-process against a real on-disk file via
t.TempDir() — no service container or environment gating needed,
unlike postgres's POSTGRES_TEST_DSN-gated integration test.

go.mod: adds modernc.org/sqlite as a new direct dependency.
Adds the SQLite subsection to Database Setup and updates the Features
bullet to mention both bundled backends.
…in-process mutex instead

New used to call db.SetMaxOpenConns(1), silently mutating a *sql.DB
that may be shared with the rest of the caller's application: every
other query against that handle got serialized through one
connection, and a *sql.Tx held open elsewhere on the same db would
starve Next of its only connection, blocking Generate/Next forever.
This also contradicted the package's own doc comment, which claimed
New made no requirements on the caller's connection-pool
configuration.

New no longer touches the pool at all. Next is instead guarded by an
in-process sync.Mutex on the Store, which is enough to stop concurrent
Next calls made through one Store from racing each other into
SQLITE_BUSY, without affecting anything else sharing the same *sql.DB.

That mutex only covers this one Store instance within this one
process: it can't see a transaction opened directly against the same
*sql.DB elsewhere, or a second process sharing the same file. Rather
than try to fully solve that (SQLite has no per-row locking to build
on, only a whole-database file lock), the package doc is now explicit
that this backend satisfies refid.Store's concurrency contract for a
single process out of the box, documents _busy_timeout as the DSN
setting a caller can add for cross-connection/cross-process waiting,
and points multi-instance deployments at refid/postgres instead, which
has no equivalent limitation.

Adds a regression test that holds a transaction open directly against
the same *sql.DB and confirms Next still returns promptly rather than
hanging — verified against the old SetMaxOpenConns(1) code that it
actually catches this (fails after a 2s timeout instead of passing).
@sthanikan2000
sthanikan2000 force-pushed the feat/refid-raw-sql-backends branch from f84ee67 to 63d6335 Compare September 2, 2026 08:55

@ginaxu1 ginaxu1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@sthanikan2000

Copy link
Copy Markdown
Collaborator Author

#181 Duplicate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants