Skip to content

chore(blobs): reconcile orphaned GCS objects #27

Description

@Aparnap2

Objective

Add the storage reconciliation/sweeper that was the remaining scope of the original #16 blob work. #16's transport and GCS content path are already delivered; this issue owns the orphan-reconciliation invariant.

Core invariant

Every GCS object in the managed ClaimOps document namespace must have a corresponding live DB document record. Objects without a live record must be surfaced and safely reconciled according to an explicit retention/grace policy.

Scope

  • Enumerate objects under the managed document prefix through a narrow BlobStore/listing port.
  • Resolve each object against the authoritative tenant/document metadata in PostgreSQL.
  • Detect orphaned objects without a live DB record.
  • Apply an explicit grace period so an object created immediately before DB/outbox commit cannot be deleted prematurely.
  • Emit structured logs and stable metrics for scans, orphans, deletes, delete failures, and skipped/grace-protected objects.
  • Keep deletion bounded/idempotent and fail safely on storage/database outages.
  • Preserve tenant isolation; do not log document contents or sensitive metadata.
  • Add deterministic unit tests with a fake blob lister/deleter.
  • Add localgcp integration coverage against the running Storage emulator.
  • Document the invariant, grace policy, and operational recovery procedure.

Non-goals

  • Parser selection or document extraction.
  • Changes to Pub/Sub event schemas.
  • Changing document admission/integrity semantics.
  • Bulk deletion without a persisted/observable reconciliation decision.

Acceptance

  • make check green.
  • LocalGCP storage integration test proves orphan detection and safe deletion.
  • Newly-created objects inside the grace window are not deleted.
  • Referenced/live objects are never deleted.
  • Delete failures are observable and do not terminate the entire sweep.
  • Rerunning a sweep is idempotent.
  • No raw document/PII/secret logging.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions