feat(#941): Prevent duplicate market-watcher jobs after failover

[chiomailekuba]

Summary

Closes #941

This PR implements a production-ready, durable, lease-coordinated market-watcher job processing system to prevent duplicate job execution and duplicate watcher notifications across worker failovers, timeouts, restarts, and concurrent executions.

Changes Made

1. Durable Database Model (market_watcher_jobs):

   • Added marketWatcherJobs table in src/db/schema.ts with unique constraint on jobKey (${marketId}:${eventType}:${eventRef}) serving as the database-level idempotency boundary.
   • Tracks atomic leaseToken (UUID), leaseUntil expiration timestamp, monotonic attempt counter, execution timestamps, watchersNotified count, and status transitions (pending, running, retryable, succeeded, failed).
   • Added DDL migration drizzle/migrations/0029_market_watcher_jobs.sql.

2. Lease Coordination & Failover Recovery (MarketWatcherJobCoordinator):

   • claimJob: Uses atomic conditional UPDATE to acquire leases only if job is pending, ready for retry (nextAttemptAt <= now), or if a previous worker's lease expired (leaseUntil < now).
   • markSucceeded: Commits completion conditioned on id, leaseToken, and running status. If a timed-out worker attempts to commit after a failover worker has taken over the lease, the stale commit is safely rejected.
   • markFailed: Bounded retries with exponential backoff delay calculation (retryDelayMs). When attempts reach maxAttempts, transitions to terminal failed status and prevents further queue re-enqueues.
   • recoverExpiredLeases: Reclaims stale or abandoned in-flight leases back to retryable for failover processing.
   • enqueueMarketWatcherJob: Idempotent enqueue helper ensuring duplicate events do not create multiple database records or redundant queue jobs.

3. Queue & Worker Architecture:

   • Exported marketWatcherQueue and marketWatcherQueueName ("market-watcher-jobs") in src/queue/index.ts.
   • Implemented MarketWatcherWorker in src/workers/marketWatcherWorker.ts with concurrency support, BullMQ event hooks, failure logging, and graceful shutdown handling.

4. Metrics & Observability:

   • Registered Prometheus metrics in src/metrics/registry.ts: marketWatcherJobRunsTotal, marketWatcherJobRetriesTotal, marketWatcherLeaseConflictsTotal, and marketWatcherNotificationsTotal.
   • Structured logging with correlation IDs and error descriptions without leaking sensitive data.

5. Test Coverage:

   • Added tests/marketWatcherJobService.test.ts (14 unit tests) covering deterministic job keys, bounded exponential backoff, idempotency, lease claiming, failover recovery, stale commit rejection, retry delays, and retry exhaustion.
   • Added tests/marketWatcherWorker.test.ts verifying worker lifecycle, idempotency, and graceful stop.
   • Updated tests/marketWatchers.test.ts for clean test execution.

Acceptance Criteria Verification

• Deterministic behavior for valid, invalid, duplicate, and boundary-case inputs.
• Authorization, validation, and state-transition invariants remain strictly enforced.
• Retries, partial failure, and concurrent execution cannot produce duplicate notifications or inconsistent state.
• Focused tests cover success, rejection, boundary, and regression scenarios.
• Existing callers remain fully compatible.
• Structured logs and Prometheus metrics make failovers and retries diagnosable without exposing sensitive data.

[drips-wave]

@chiomailekuba Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits