feat(refid): add raw-SQL sqlite backend - #177
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2045f46 to
b5e1731
Compare
ef7a644 to
9880eca
Compare
ginaxu1
left a comment
There was a problem hiding this comment.
PTAL these issues:
- New mutates the caller's pool (refid/sqlite/store.go)
db.SetMaxOpenConns(1) applies to every user of that *sql.DB, not just Next. A caller that shares the handle (normal SQLite usage: one file, one pool) has all queries serialized. Worse, a held *sql.Tx occupies the only connection, so Generate/Next waits forever. The package comment says this backend "makes no requirements on the caller's *sql.DB connection-pool configuration"; the constructor does the opposite.
Suggested fix: do not change pool settings. Guard Next with a mutex so in-process concurrent Generate cannot hit SQLITE_BUSY. For lock waits on the file (other connections or processes), set busy_timeout on every connection, e.g. document opening with _pragma=busy_timeout(5000) in the DSN (a one-shot PRAGMA busy_timeout on db.Exec only affects that one pooled connection). Keep the 50-goroutine test; it should still pass with the mutex and no SetMaxOpenConns. Add a test that Begin()s a Tx on the same *sql.DB, then calls Next with a short context deadline: it must not hang.
- Multi-process Store contract is unmet
refid.Store requires concurrent safety "across multiple process instances sharing the same backing store". A second process (or a second sql.Open on the same file) still gets SQLITE_BUSY because busy_timeout is never set. SetMaxOpenConns(1) only serializes one *sql.DB.
Suggested fix: same DSN/pragma as (1). If this backend is single-process (dev/tests only), say that on the package and do not claim it satisfies the multi-process clause. Production multi-instance deployments stay on postgres.
|
@ginaxu1 Good catch, both fixed in f84ee67.
Checked #176 too — neither point applies there. Postgres's |
Adds github.com/OpenNSW/core/refid/sqlite, mirroring the postgres backend's shape (DefaultTableName, Option, WithTableName, New, Migrate, Next) via database/sql + the pure-Go modernc.org/sqlite driver (no CGO) — a lighter build/deploy footprint than a CGO-based driver for a shared library consumed by many services. Next is a single atomic upsert-and-increment statement, same as postgres, just SQLite dialect (numbered placeholders, unqualified DO UPDATE columns, no TIMESTAMPTZ). New pins the passed *sql.DB to a single connection (db.SetMaxOpenConns(1)): SQLite allows only one writer at a time, and without this, two concurrent Next calls can each open their own connection and race into a "database is locked" (SQLITE_BUSY) error instead of one simply queueing behind the other. Since Next is a single fast statement (no transaction held across calls), this costs negligible throughput for the low-volume counters this backend targets. Tests run unconditionally in-process against a real on-disk file via t.TempDir() — no service container or environment gating needed, unlike postgres's POSTGRES_TEST_DSN-gated integration test. go.mod: adds modernc.org/sqlite as a new direct dependency.
Adds the SQLite subsection to Database Setup and updates the Features bullet to mention both bundled backends.
…in-process mutex instead New used to call db.SetMaxOpenConns(1), silently mutating a *sql.DB that may be shared with the rest of the caller's application: every other query against that handle got serialized through one connection, and a *sql.Tx held open elsewhere on the same db would starve Next of its only connection, blocking Generate/Next forever. This also contradicted the package's own doc comment, which claimed New made no requirements on the caller's connection-pool configuration. New no longer touches the pool at all. Next is instead guarded by an in-process sync.Mutex on the Store, which is enough to stop concurrent Next calls made through one Store from racing each other into SQLITE_BUSY, without affecting anything else sharing the same *sql.DB. That mutex only covers this one Store instance within this one process: it can't see a transaction opened directly against the same *sql.DB elsewhere, or a second process sharing the same file. Rather than try to fully solve that (SQLite has no per-row locking to build on, only a whole-database file lock), the package doc is now explicit that this backend satisfies refid.Store's concurrency contract for a single process out of the box, documents _busy_timeout as the DSN setting a caller can add for cross-connection/cross-process waiting, and points multi-instance deployments at refid/postgres instead, which has no equivalent limitation. Adds a regression test that holds a transaction open directly against the same *sql.DB and confirms Next still returns promptly rather than hanging — verified against the old SetMaxOpenConns(1) code that it actually catches this (fails after a 2s timeout instead of passing).
f84ee67 to
63d6335
Compare
|
#181 Duplicate |
Summary
github.com/OpenNSW/core/refid/sqlite, mirroringrefid/postgres's shape (DefaultTableName,Option,WithTableName,New,Migrate,Next) viadatabase/sql+ the pure-Gomodernc.org/sqlitedriver — no CGO.Nextis the same single atomic upsert-and-increment statement as postgres, just SQLite dialect.Newpins the passed*sql.DBto a single connection (db.SetMaxOpenConns(1)): SQLite allows only one writer at a time, and without this, concurrentNextcalls can each open their own connection and race into aSQLITE_BUSY("database is locked") error instead of one simply queueing behind the other. SinceNextis one fast statement with no transaction held across calls, this costs negligible throughput.Storeescape hatch in the README.Why
A pure-Go SQLite option avoids imposing a CGO build requirement on every downstream service for an optional backend, and gives local development and tests a real, on-disk backend with no external service dependency — tests run unconditionally in-process, unlike the Postgres backend's
POSTGRES_TEST_DSN-gated integration test.This PR is purely additive on top of #176 (stacked on that branch, per its updated description) —
refid/postgreskeeps working unchanged throughout. Base isrefactor/refid-store-interface, notmain; retarget once #176 merges.Test plan
go build ./...,go vet ./...,golangci-lint run ./...all clean inrefid/go test -race ./...passes inrefid/, including a 50-goroutine concurrent-Next-no-duplicates test (initially caught theSQLITE_BUSYissueSetMaxOpenConns(1)fixes, before the fix was added)