refactor(refid)!: replace gorm postgres backend with raw-SQL Store/Reservation - #176
refactor(refid)!: replace gorm postgres backend with raw-SQL Store/Reservation#176sthanikan2000 wants to merge 3 commits into
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
A shared refid/internal/sqlident.Validate replaces the table-name regex that used to live inline in postgres_store.go. Both the postgres and sqlite backends call it before interpolating a caller-supplied table name into raw SQL, so the one security-relevant check can't drift between backends. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…servation Renames the storage interface from SequenceStore to Store, and its single method from Next to Reserve. Reserve no longer persists a counter by itself: it returns a candidate value plus a Reservation the caller must resolve via Commit or Rollback. This lets a caller defer consuming the counter until its own downstream use of the generated ID has actually succeeded, so a failure after Generate no longer permanently burns/gaps a reserved value. Registry.Generate now returns (string, Reservation, error) for every call, including formats with no sequence segment (a no-op Reservation) or more than one (an aggregating Reservation that resolves each in turn). Adds ErrReservationConflict for the case where a concurrent Commit for the same scope key wins the race. Replaces the gorm-based postgres_store.go with github.com/OpenNSW/core/refid/postgres, using database/sql + the pgx stdlib driver directly (no ORM). Reserve reads the current counter with no lock held; Commit performs a compare-and-swap write (`... WHERE counter = <value read at Reserve time>`), so nothing holds a database row lock open across the caller's own downstream work between Reserve and Commit/Rollback. New's panic-on-invalid-table-name behavior from the old NewPostgresStore is gone: New now returns an error like every other constructor in this repo, matching Migrate's existing behavior for the same invalid input. go.mod: drops gorm.io/gorm and gorm.io/driver/postgres (no longer used by anything in the module) and promotes github.com/jackc/pgx/v5 (was already an indirect dependency via gorm) to direct. BREAKING CHANGE: SequenceStore, Next, NewPostgresStore, and the other postgres_store.go exports are gone. Registry.Generate's signature changed from (string, error) to (string, Reservation, error). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Updates the Quickstart, Database Setup, and Error Handling sections for the new refid.Store/Reservation API and the raw-SQL postgres backend, and adds a Deferred-Commit Reservations section explaining the Commit/Rollback usage pattern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fee19b4 to
2045f46
Compare
There was a problem hiding this comment.
PTAL these issues:
- Optimistic CAS plus "save the record, then Commit" breaks uniqueness
Reserve does an unlocked SELECT, so two concurrent Generate calls for the same scope key receive the same candidate. The documented pattern then lets both callers persist that ID before either Commit runs. One Commit wins, the other returns ErrReservationConflict, but the duplicate row is already written. If the caller's ID column is unique, the loser retries Generate, still reads the uncommitted counter, and can loop on the same ID until the winner happens to Commit. A crash or error after save and before Commit is worse: the next Generate reissues an ID that already exists.
Suggested fix: pick one of these, not a hybrid.
- Pessimistic reservation (preserves both uniqueness and no-gaps): Reserve opens a transaction, SELECT ... FOR UPDATE (or INSERT the scope row then lock it), and holds that transaction on the Reservation. Commit writes the counter and commits the tx. Rollback rolls the tx back. Concurrent Generate for the same scope key waits instead of minting a duplicate. Document that the DB connection is held until the reservation is resolved, and keep that window small.
- Consume at issuance (preserves uniqueness, accepts gaps): atomically increment in Reserve the way Next used to (INSERT ... ON CONFLICT DO UPDATE SET counter = counter + 1 ... RETURNING), and make Reservation a no-op or a handle that cannot un-consume. Downstream save failure may gap a value; that is the correct tradeoff for a unique reference ID.
Do not keep unlocked read + CAS after the caller has already stored the ID. ErrReservationConflict is not a valid recovery path once the ID has been persisted.
- reservation.Commit marks resolved before the write succeeds (refid/postgres/store.go)
resolved is set to true, then Exec runs. A transient DB error returns failure but leaves the reservation unretryable: a later Commit is a silent no-op, and a deferred Rollback does nothing. Combined with (1), the caller has a saved record and an unconsumed counter.
Suggested fix: set resolved only after a successful write. On ErrReservationConflict, mark it resolved (this reservation is dead; retry Generate). On any other error, leave it unresolved so Commit can be retried. Commit after Rollback must return an error, not nil. Rollback after a successful Commit stays a no-op. Guard resolved with a mutex so Commit and Rollback cannot race.
- Generate leaks reservations when a later segment fails (refid/registry.go)
The render loop returns immediately on the first render error and drops any Reservations already taken. For a format with two sequence segments, a failure on the second (overflow, cancelled context, DB error) never calls Rollback on the first. That is a lock/connection leak the moment a Store holds resources across Reserve (including the pessimistic design in (1), and today's memStore).
Suggested fix: accumulate reservations as they are returned; on render error, Rollback each one (errors.Join is fine), then return the original error. Add a test with two sequence segments where the second Reserve fails and assert the first counter is unconsumed.
- Schema rename will break any existing refid_sequences table
Migrate is CREATE TABLE IF NOT EXISTS, so an existing table keeps updated_at. The new upsert interpolates created_at, which is not on the old table, so Commit fails at runtime. There is no ALTER/migration, and the column rename is not needed for the CAS write.
Suggested fix: keep updated_at, set it on INSERT and on the CAS UPDATE, and do not introduce created_at. If you truly need a new column, ship a real migration, not CREATE TABLE IF NOT EXISTS
Summary
SequenceStoretoStore, and its single method fromNexttoReserve.Reserveno longer persists a counter by itself — it returns a candidate value plus aReservationthe caller must resolve viaCommitorRollback.Registry.Generatenow returns(string, Reservation, error)for every call, including formats with nosequencesegment (a no-opReservation) or more than one (an aggregatingReservationthat resolves each in turn).ErrReservationConflict, returned byReservation.Commitwhen a concurrent commit for the same scope key wins the race.postgres_store.gowithgithub.com/OpenNSW/core/refid/postgres, usingdatabase/sql+ thepgxstdlib driver directly (no ORM).go.moddropsgorm.io/gorm/gorm.io/driver/postgresentirely and promotesgithub.com/jackc/pgx/v5(previously an indirect dependency via gorm) to direct.refid/internal/sqlident, a shared table-name validator used by the new backend.Why
Deferred-commit means a caller can defer consuming a reserved counter value until its own downstream use of the generated ID (e.g. saving a record) has actually succeeded. Today's
Nextpersists the increment immediately insideGenerate, so a downstream failure after a successfulGeneratecall permanently burns/gaps that value. Separately, the gorm dependency was replaced with raw SQL to keep the module's dependency footprint minimal.Every commit in this PR leaves
refidfully functional with a working Postgres backend — the interface redesign and the backend swap land together in one commit rather than across a gap, specifically so there's no point in this history where the package has no usable backend.This is a breaking change to
Registry.Generate's signature and to theStoreinterface. Confirmed there are zero consumers of this package anywhere else in the monorepo, so no other module is affected.A follow-up PR (stacked on this branch) adds a SQLite backend as a pure feature addition on top.
Test plan
go build ./...,go vet ./...,golangci-lint run ./...all clean inrefid/go test -race ./...passes inrefid/TestStore_Integration,TestStore_CommitConflict) against a realpostgres:16-alpinecontainerrefid_test.go's in-memorymemStoretest double rewritten to implementStore/Reservation(per-scope-key lock — a global lock would deadlock a format with twosequencesegments; covered byTestGenerate_MultipleSequenceSegments)TestGenerate_RollbackLeavesCounterUnconsumed🤖 Generated with Claude Code