Skip to content

refactor(refid)!: replace gorm postgres backend with raw-SQL Store/Reservation - #176

Open
sthanikan2000 wants to merge 3 commits into
mainfrom
refactor/refid-store-interface
Open

refactor(refid)!: replace gorm postgres backend with raw-SQL Store/Reservation#176
sthanikan2000 wants to merge 3 commits into
mainfrom
refactor/refid-store-interface

Conversation

@sthanikan2000

@sthanikan2000 sthanikan2000 commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Renames the storage interface from SequenceStore to Store, and its single method from Next to Reserve. Reserve no longer persists a counter by itself — it returns a candidate value plus a Reservation the caller must resolve via Commit or Rollback.
  • Registry.Generate now returns (string, Reservation, error) for every call, including formats with no sequence segment (a no-op Reservation) or more than one (an aggregating Reservation that resolves each in turn).
  • Adds ErrReservationConflict, returned by Reservation.Commit when a concurrent commit for the same scope key wins the race.
  • Replaces the gorm-based postgres_store.go with github.com/OpenNSW/core/refid/postgres, using database/sql + the pgx stdlib driver directly (no ORM). go.mod drops gorm.io/gorm/gorm.io/driver/postgres entirely and promotes github.com/jackc/pgx/v5 (previously an indirect dependency via gorm) to direct.
  • Adds refid/internal/sqlident, a shared table-name validator used by the new backend.
  • Updates the README for the new API and backend.

Why

Deferred-commit means a caller can defer consuming a reserved counter value until its own downstream use of the generated ID (e.g. saving a record) has actually succeeded. Today's Next persists the increment immediately inside Generate, so a downstream failure after a successful Generate call permanently burns/gaps that value. Separately, the gorm dependency was replaced with raw SQL to keep the module's dependency footprint minimal.

Every commit in this PR leaves refid fully functional with a working Postgres backend — the interface redesign and the backend swap land together in one commit rather than across a gap, specifically so there's no point in this history where the package has no usable backend.

This is a breaking change to Registry.Generate's signature and to the Store interface. Confirmed there are zero consumers of this package anywhere else in the monorepo, so no other module is affected.

A follow-up PR (stacked on this branch) adds a SQLite backend as a pure feature addition on top.

Test plan

  • go build ./..., go vet ./..., golangci-lint run ./... all clean in refid/
  • go test -race ./... passes in refid/
  • Ran the postgres integration tests (TestStore_Integration, TestStore_CommitConflict) against a real postgres:16-alpine container
  • refid_test.go's in-memory memStore test double rewritten to implement Store/Reservation (per-scope-key lock — a global lock would deadlock a format with two sequence segments; covered by TestGenerate_MultipleSequenceSegments)
  • New test: TestGenerate_RollbackLeavesCounterUnconsumed

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: bfab6f02-d95f-48ea-8de2-3e5482c76f45


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

sthanikan2000 and others added 3 commits September 1, 2026 09:42
A shared refid/internal/sqlident.Validate replaces the table-name
regex that used to live inline in postgres_store.go. Both the postgres
and sqlite backends call it before interpolating a caller-supplied
table name into raw SQL, so the one security-relevant check can't
drift between backends.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…servation

Renames the storage interface from SequenceStore to Store, and its
single method from Next to Reserve. Reserve no longer persists a
counter by itself: it returns a candidate value plus a Reservation the
caller must resolve via Commit or Rollback. This lets a caller defer
consuming the counter until its own downstream use of the generated ID
has actually succeeded, so a failure after Generate no longer
permanently burns/gaps a reserved value.

Registry.Generate now returns (string, Reservation, error) for every
call, including formats with no sequence segment (a no-op Reservation)
or more than one (an aggregating Reservation that resolves each in
turn). Adds ErrReservationConflict for the case where a concurrent
Commit for the same scope key wins the race.

Replaces the gorm-based postgres_store.go with
github.com/OpenNSW/core/refid/postgres, using database/sql + the pgx
stdlib driver directly (no ORM). Reserve reads the current counter
with no lock held; Commit performs a compare-and-swap write
(`... WHERE counter = <value read at Reserve time>`), so nothing holds
a database row lock open across the caller's own downstream work
between Reserve and Commit/Rollback. New's panic-on-invalid-table-name
behavior from the old NewPostgresStore is gone: New now returns an
error like every other constructor in this repo, matching Migrate's
existing behavior for the same invalid input.

go.mod: drops gorm.io/gorm and gorm.io/driver/postgres (no longer used
by anything in the module) and promotes github.com/jackc/pgx/v5 (was
already an indirect dependency via gorm) to direct.

BREAKING CHANGE: SequenceStore, Next, NewPostgresStore, and the other
postgres_store.go exports are gone. Registry.Generate's signature
changed from (string, error) to (string, Reservation, error).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Updates the Quickstart, Database Setup, and Error Handling sections
for the new refid.Store/Reservation API and the raw-SQL postgres
backend, and adds a Deferred-Commit Reservations section explaining
the Commit/Rollback usage pattern.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@sthanikan2000
sthanikan2000 force-pushed the refactor/refid-store-interface branch from fee19b4 to 2045f46 Compare September 1, 2026 04:14
@sthanikan2000 sthanikan2000 changed the title refactor(refid): replace SequenceStore/Next with deferred-commit Store/Reserve refactor(refid)!: replace gorm postgres backend with raw-SQL Store/Reservation Sep 1, 2026

@ginaxu1 ginaxu1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL these issues:

  1. Optimistic CAS plus "save the record, then Commit" breaks uniqueness
    Reserve does an unlocked SELECT, so two concurrent Generate calls for the same scope key receive the same candidate. The documented pattern then lets both callers persist that ID before either Commit runs. One Commit wins, the other returns ErrReservationConflict, but the duplicate row is already written. If the caller's ID column is unique, the loser retries Generate, still reads the uncommitted counter, and can loop on the same ID until the winner happens to Commit. A crash or error after save and before Commit is worse: the next Generate reissues an ID that already exists.

Suggested fix: pick one of these, not a hybrid.

  • Pessimistic reservation (preserves both uniqueness and no-gaps): Reserve opens a transaction, SELECT ... FOR UPDATE (or INSERT the scope row then lock it), and holds that transaction on the Reservation. Commit writes the counter and commits the tx. Rollback rolls the tx back. Concurrent Generate for the same scope key waits instead of minting a duplicate. Document that the DB connection is held until the reservation is resolved, and keep that window small.
  • Consume at issuance (preserves uniqueness, accepts gaps): atomically increment in Reserve the way Next used to (INSERT ... ON CONFLICT DO UPDATE SET counter = counter + 1 ... RETURNING), and make Reservation a no-op or a handle that cannot un-consume. Downstream save failure may gap a value; that is the correct tradeoff for a unique reference ID.
    Do not keep unlocked read + CAS after the caller has already stored the ID. ErrReservationConflict is not a valid recovery path once the ID has been persisted.
  1. reservation.Commit marks resolved before the write succeeds (refid/postgres/store.go)
    resolved is set to true, then Exec runs. A transient DB error returns failure but leaves the reservation unretryable: a later Commit is a silent no-op, and a deferred Rollback does nothing. Combined with (1), the caller has a saved record and an unconsumed counter.

Suggested fix: set resolved only after a successful write. On ErrReservationConflict, mark it resolved (this reservation is dead; retry Generate). On any other error, leave it unresolved so Commit can be retried. Commit after Rollback must return an error, not nil. Rollback after a successful Commit stays a no-op. Guard resolved with a mutex so Commit and Rollback cannot race.

  1. Generate leaks reservations when a later segment fails (refid/registry.go)
    The render loop returns immediately on the first render error and drops any Reservations already taken. For a format with two sequence segments, a failure on the second (overflow, cancelled context, DB error) never calls Rollback on the first. That is a lock/connection leak the moment a Store holds resources across Reserve (including the pessimistic design in (1), and today's memStore).

Suggested fix: accumulate reservations as they are returned; on render error, Rollback each one (errors.Join is fine), then return the original error. Add a test with two sequence segments where the second Reserve fails and assert the first counter is unconsumed.

  1. Schema rename will break any existing refid_sequences table
    Migrate is CREATE TABLE IF NOT EXISTS, so an existing table keeps updated_at. The new upsert interpolates created_at, which is not on the old table, so Commit fails at runtime. There is no ALTER/migration, and the column rename is not needed for the CAS write.

Suggested fix: keep updated_at, set it on INSERT and on the CAS UPDATE, and do not introduce created_at. If you truly need a new column, ship a real migration, not CREATE TABLE IF NOT EXISTS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants