feat(redis): simplify tracked writes with client clocks - #140
Conversation
BREAKING CHANGE: node-redis registered script methods and direct Lua consumers must append client timestamps; semantic Redis request shapes and frame v1 remain unchanged.
lan17
left a comment
There was a problem hiding this comment.
Deep review of the client-clock protocol (10 finder angles + adversarial verification + gap sweep; tsc and the 519-test unit suite pass locally). The core protocol verified sound: invalidation math, fence coverage on every read path, adapter dispatch against real node-redis/GLIDE semantics, cluster routing, retry idempotency, migration steps, and benchmark claims. Inline comments below; the corrupt-watermark hardening items (wrong-type key, >2^53 numeric parse) are tracked separately.
lan17
left a comment
There was a problem hiding this comment.
Follow-up to the review discussion: concrete fix suggestions for the corrupt-watermark hardening items (strict shared integer grammar with a MAX_SAFE cap, and WRONGTYPE self-heal). All three are small and independent; together they make every corruption class route to the script's existing repair path.
lan17
left a comment
There was a problem hiding this comment.
Two follow-ups from a coverage pass over the final head (97.6% statements / 95% branches; unit 528 + integration 125/125 green locally): inline below. One process question rather than a code comment: does test:integration gate PRs in CI, or is it a manual step? The wrong-type-watermark and persistent-watermark behaviors are covered only there — FakeRedis structurally cannot represent either state — so the unit tier is blind to regressions in those two safety paths if integration is manual-only. The remaining uncovered lines are pre-existing convenience paths (local-cache.ts get/put-without-config, RedisCache.getResult's disabled-layer return) and constructor validation branches — reviewed, fine to leave.
|
On the integration-gating question: yes. The PR-triggered CI workflow runs |
## Summary Add opt-in stale-on-error recovery from the existing Redis frame, on top of the native client-clock protocol from #140. - `F = ttlSec[CacheLayer.REMOTE]` remains the ordinary Redis freshness boundary. - `M = staleOnErrorMaxAgeSec` is the absolute stale-recovery age ceiling and requested physical retention. - One initial `GET` or tracked `MGET(value, watermark)` either serves a fresh frame, retains an `F..M` raw candidate, or misses without a candidate. - After the source rejects, a resolved error classifier decides whether DialCache may use the retained candidate. - Recovery never rereads Redis, republishes the value, or replaces the original source rejection when it cannot serve. Closes #117 ## Configuration and public API ```ts import { CacheLayer, DialCache, DialCacheKeyConfig, FallbackTimeoutError, } from "dialcache"; const dialcache = new DialCache({ // Instance default: the broad transient-infrastructure policy. shouldAttemptStaleRecovery: (error) => error instanceof FallbackTimeoutError || isRetriableDatabaseError(error), }); const getUser = dialcache.cached((id: string) => db.fetchUser(id), { keyType: "user_id", useCase: "GetUser", cacheKey: (id) => id, defaultConfig: new DialCacheKeyConfig({ ttlSec: { [CacheLayer.REMOTE]: 300 }, // F: ordinary Redis freshness staleOnErrorMaxAgeSec: 3_600, // M: absolute recovery ceiling }), }); // A per-use-case classifier REPLACES the instance and built-in policies for // that use case; include the timeout case when it should stay eligible. const getEntitlements = dialcache.cached((id: string) => db.fetchEntitlements(id), { keyType: "user_id", useCase: "GetEntitlements", cacheKey: (id) => id, shouldAttemptStaleRecovery: (error) => error instanceof FallbackTimeoutError, defaultConfig: new DialCacheKeyConfig({ ttlSec: { [CacheLayer.REMOTE]: 300 }, staleOnErrorMaxAgeSec: 3_600, }), }); ``` New API surface: - `DialCacheKeyConfig.staleOnErrorMaxAgeSec?: number` - `DialCacheConfig.shouldAttemptStaleRecovery?: StaleRecoveryPredicate` - per-use-case `shouldAttemptStaleRecovery` on `cached()` and `getOrLoad()` options - root-exported `StaleRecoveryPredicate`, `StaleRecoveryOutcome`, and `StaleRecoveryMetricLabels` - optional `DialCacheMetricsAdapter.staleRecovery(labels)` and `observeStaleRecoveryValueAge(labels, seconds)` hooks Classifier precedence — highest wins, and an override replaces the levels below it rather than composing with them: 1. per-use-case `shouldAttemptStaleRecovery` on `cached()` / `getOrLoad()` options 2. `DialCacheConfig.shouldAttemptStaleRecovery` 3. built-in: `error instanceof FallbackTimeoutError` only The intended split: put the broad policy for transient, retriable infrastructure failures on the instance, and use a per-use-case override where particular data needs a stricter policy. Predicates should deny authoritative domain outcomes — auth/permission/entitlement failures, revocation, deletion or not-found, validation, and programmer errors — where a stale value would be wrong rather than merely old. Predicates must synchronously return a boolean; a throw, non-boolean, or thenable fails closed, and an accidental rejecting thenable is consumed. `cached()` snapshots its selected predicate at registration; `getOrLoad()` resolves it per invocation. Disabled calls remain true pass-through and never invoke it. Omitting `M` keeps recovery off; a sparse runtime overlay that omits it inherits the configured default, and an explicit `0` disables an inherited value. A positive policy requires `0 < F < M <= 31,536,000` seconds. Invalid static policy throws before registration; invalid runtime `M` records `config_resolution`, disables only recovery for that invocation, and preserves otherwise-valid ordinary Redis policy. `DialCacheKeyConfig.disabled()` sets `M` to `0`. ## Execution ```mermaid flowchart TD A["One native GET or tracked MGET; classify raw frame with application Date.now()"] -->|"0 <= age < F"| B["Deserialize and return fresh"] A -->|"F <= age < M"| C["Record ordinary miss; retain raw candidate"] A -->|"missing, future, fenced, invalid, or age >= M"| D["Ordinary miss without candidate"] A -->|"read error, timeout, or fresh deserialization failure"| E["Fallback; recovery forbidden"] C --> F["Call source"] D --> F F -->|success| G["Return source value and attempt normal publication"] F -->|rejection denied by classifier| H["Throw exact source rejection"] F -->|eligible rejection| I["Use only retained candidate"] I -->|"age < M before and after lazy load"| J["Return candidate without publication"] I -->|missing, expired, future, or load failure| H E -->|source rejection| H ``` The candidate stays serialized/compressed until the source call settles. On an eligible rejection, DialCache checks `0 <= age < M`, lazily deserializes/decompresses, checks `M` again after that potentially asynchronous work, and only then returns it. Equality at `F` is logically stale; equality at `M` is unavailable. A successful source refresh wins and writes a newly timestamped frame. Recovery does not write or extend Redis, populate process-local cache, schedule shadow validation, or emit a shadow-age observation. Request-local caching may memoize the returned reference only within the active outer `enable()` scope. Existing coalescing shares the whole read/source/recovery decision; `coalesce: false` gives each caller an independent candidate and source attempt. ## Snapshot and freshness semantics The retained candidate is the initial Redis snapshot: - tracked reads atomically apply the value and watermark observed by the initial primary-routed `MGET`; - an invalidation completed before that read fences the candidate; - invalidation, refresh, deletion, expiry, or eviction after the read does not revoke or replace the in-memory bytes; - tracked and untracked recovery therefore use the same one-read model. For tracked use cases this opt-in path can weaken the usual strict freshness guarantee when invalidation races with the source attempt. Use cases that cannot tolerate that bounded relaxation should leave recovery disabled or deny the error in their classifier. `F` and `M` bound Redis serving only. Request-local and process-local layers occur earlier and keep their own scope/TTL lifetimes. A frame can be nearly `F` old when it warms process-local cache and then receive a full local TTL, so `ttlSec.local <= F` does not make `F` a strict end-to-end age limit; disable earlier layers when that is required. ## Redis, time, and compatibility There is no Redis envelope or key change: frame v1, `:dialcache-frame-v1`, `DialCacheRedisClient`, and `RedisReadRequest` remain unchanged. Reads use native `GET`/primary-routed `MGET`, writes use native complete-frame `SET`, and invalidation remains the only Lua operation. No Redis `TIME` or clock-offset estimation is added. Writers request physical retention through `M` when enabled and `F` otherwise. Tracked values keep the existing one-hour physical TTL cap and `tracked_ttl_clamped` signal from #140. The configured logical `M` is not clipped, but a tracked frame may physically disappear before reaching it. Core now treats every decoded frame's real writer `createdAtMs` as authoritative for ordinary logical `F`, including untracked reads. Custom Redis clients that returned a constant timestamp must return the actual epoch-millisecond frame stamp before upgrading. Roll out readers while `M` is omitted or `0`, upgrade the complete fleet, and only then enable positive `M`. Once a writer retains through `M`, do not reintroduce a pre-feature reader until affected keys have expired or been removed; older readers trust physical presence and can serve `F..M` as fresh. Application-process clock skew can move the boundary early or late. Future-dated frames fail closed and use the existing future-offset observation. Durations and deadlines remain monotonic. ## Metrics New series, exposed by both bundled backends: | Backend | Metric | Type | Labels / tags | Emitted | | --- | --- | --- | --- | --- | | Prometheus | `dialcache_stale_recovery_counter` | counter | `cache_namespace`, `use_case`, `key_type`, `outcome` | once per classifier-authorized recovery check | | Prometheus | `dialcache_stale_recovery_value_age_histogram` | histogram, buckets `[1, 5, 15, 60, 300, 900, 3600, 10800, 43200, 86400, 259200, 604800]` s (shared with shadow value age) | `cache_namespace`, `use_case`, `key_type`, `outcome` | only alongside `served`; value = return-time age in seconds | | Datadog | `dialcache.stale_recovery.count` | count | `cache_namespace`, `use_case`, `key_type`, `outcome` | once per classifier-authorized recovery check | | Datadog | `dialcache.stale_recovery.value_age` | histogram or distribution, per the adapter's existing `observationMetricType` option | `cache_namespace`, `use_case`, `key_type`, `outcome` | only alongside `served` | Label reference: - `cache_namespace` — `DialCacheConfig.namespace` (default `urn`); present on every DialCache metric. - `use_case` / `key_type` — the registered use case and key type; never the cache key or id. - `outcome` — the bounded root-exported `StaleRecoveryOutcome` union: - `served` — the retained candidate was returned to the caller (`age < M` held both before and after lazy deserialization); - `miss` — no candidate was retained by the initial read, or the candidate's age reached `M` (or its stamp became future/invalid) by check time; - `deserialization_error` — a candidate within age failed deserialize/decompress. - On the value-age series the `outcome` label is structurally always `served` — it exists so both outcome series share one label schema. - Neither series carries a `layer` label: recovery reuses the caller-serving initial read, so there is no separate layer to attribute. - Adapter authors receive the backend-neutral camel-case object `StaleRecoveryMetricLabels { cacheNamespace, useCase, keyType, outcome }` and map it to their backend's naming, as the bundled adapters do. The backing `DialCacheMetricsAdapter.staleRecovery(labels)` and `observeStaleRecoveryValueAge(labels, seconds)` hooks are optional — existing custom adapters keep compiling, and omitting a hook skips only that observation, never recovery itself. A rejection the classifier denies emits no stale-recovery outcome: the counter measures authorized checks, not all source failures. No raw error, exception message, or cache key ever enters a label. Recovery adds no second ordinary `request`, `observeGet`, `miss`, cache-read error, or Redis command — the initial read is the one caller-serving telemetry trail. Existing fallback error/duration telemetry still records the source rejection even when recovery serves, so a served recovery stays visibly paired with the failure that caused it. Changes to existing series, visible on upgrade even where recovery stays off (all carry their usual `cache_namespace`, `use_case`, `key_type` labels; `layer`/`error`/`in_fallback` as noted): - **`dialcache_miss_counter` / hit rate, at `layer="remote"`** — logical `F` is now enforced from every frame's real `createdAtMs`, including untracked reads, so physically present but logically old frames that previously served now count as misses. Keyspaces that relied on physical-TTL slack will see a one-time remote miss-rate rise. With recovery enabled, a retained `F..M` candidate also records an ordinary miss even when recovery later serves it — during incidents, treat `staleRecovery{outcome="served"}` as its own population rather than expecting it in hit-rate math. `dialcache_request_counter` and the get-duration histogram are unchanged in population: still exactly one per caller-serving read. - **`dialcache_future_timestamp_offset_histogram` (`layer="remote"` and shadow layers)** — its population widened: previously it observed only tracked frames, but every serving read now validates the stamp, so future-dated untracked frames are observed (and fail closed) as well. The Prometheus help string changed accordingly. - **`dialcache_error_counter` with `error="tracked_ttl_clamped"`, `layer="remote"`, `in_fallback="false"`** — enabling `M` above one hour on a tracked use case makes every dispatched tracked write request TTL `M` and get clamped, so this existing configuration signal fires per write for such configs (see the Redis section above). Alert rules keyed on the error counter without an `error` label filter will absorb this as a steady rate. - **`dialcache_error_counter` with `error="config_resolution"`** — an invalid runtime `M` overlay records one per invocation while recovery is disabled for that invocation and ordinary reads continue. ## Cost The Redis outage-path cost drops from two payload reads to one. The tradeoff moves to Node memory: one raw candidate is retained per distinct in-flight key through the source attempt; same-key coalesced followers share it. The benchmark includes a delayed, high-cardinality, incompressible-payload scenario rather than relying on the highly compressed default fixture. ## Validation Node 22.22.0: - typecheck - 611 unit tests with 98.18% statement coverage - ESM/CJS build and packed TypeScript consumers - 139 live Redis/Valkey integration tests passed; 2 unavailable-cluster cases skipped after the expected connection timeout - stale-on-error benchmark semantic assertions passed - exactly 1.00 adapter read per independent recovery flight and one read shared by 500 coalesced callers - delayed 128 × 64 KiB raw-candidate benchmark observed +8.00 MiB external memory while retained and returned to baseline after recovery BREAKING CHANGE: Ordinary Redis reads now enforce logical age from each frame's real `createdAtMs`, including untracked reads. Deploy new readers before enabling physical `M` retention.
Summary
Finish the client-clock simplification on top of
main:SETcontaining a complete frame-v1 value stamped once with writerDate.now();MGET(value, watermark), with a missing watermark treated as zero; andTIMEis unused.This PR contains no rollout gate. Old and new tracked protocols are gated and drained externally rather than allowed to coexist.
Why
The previous tracked write paired a placeholder
SETwith a Lua stamp that read Redis server time. Same-clock fencing was exact, but the write path carried two commands, per-write Lua, placeholder/nonce pairing state, and a split-pair failure mode: a sustained stamp failure (a denied command, a proxy rejectingEVALSHA) still landed every pairedSET, replacing readable values with unreadable placeholders and shifting full tracked traffic to the source within one TTL horizon. This PR trades exact server-clock fencing for a documented client-clock buffer: every write is one atomic nativeSETthat either lands servable or does not exist, and the placeholder, nonce, stamp-reply, and script-recovery machinery is deleted. The cost is stated rather than hidden — invalidation correctness now depends on thefutureBufferMssizing and watermark durability requirements below.Architecture
GET;createdAtMsis informational and does not gate servingMGET(value, watermark); serving and initial-shadow reads requirecreatedAtMs > watermarkand not future-datedSET PX, stamped once by the adapterA malformed string watermark misses. Native
MGETreturnsnilfor a wrong-type member, so a wrong-type watermark has the same read behavior as an absent one. The next explicit invalidation repairs a wrong-type watermark; other script read errors surface and cannot bypass the monotonic update.The value envelope, timestamp field, Redis key derivation, watermark key/value encoding, and tracked primary-read routing are unchanged.
Correctness
This is a read-time invalidation fence. A write behind an active watermark may be stored, but it remains a tracked miss. A tracked invocation that reaches the Redis path does not publish its fallback directly to process-local cache; a later validated Redis hit may warm it. Local-only, remote-policy-disabled, and ramped-down paths retain their existing local-cache behavior.
Production invalidation requires:
Dmax: maximum elapsed time from invalidation sampling until a stale pre-mutationSETcan become visible in Redis, including source visibility, fallback, serialization, bounded client queue/reconnect delay, network transit, and Redis execution;S: maximum writer-clock lead over the invalidator; andM: operational margin.For example: fills visible within 2 s, fleet clock skew at most 1 s, and a 1 s margin call for
futureBufferMs ≈ 4000.An unbounded offline queue or retry path makes a finite
Dmaximpossible. Future-dated tracked frames fail closed on serving and initial-shadow reads and emit the optional future-timestamp-offset observation. Confirmation reads observe but retain the frame for payload comparison; untracked reads do not gate serving on the informational timestamp.Only invalidation maintains watermarks. Their TTL is:
Persistent watermarks remain persistent. Each dispatched tracked write whose configured TTL is capped emits
error="tracked_ttl_clamped"and attempts the write with the capped TTL. Losing a watermark removes its read-time fence, so production must usenoevictionor an equivalent guarantee and alert on memory pressure, rejected writes, and evictions. Watermark loss also includes async-replication failover — a promoted replica may lack the newest watermark — so choose replication and failover guarantees accordingly; DialCache does not issueWAIT.Performance and simplification
The tracked write path drops from
SET + stamp Luato oneSET:The maintained live benchmark exercised 6,750 sequential writes across 100 B, 10 KiB, 100 KiB, and 1 MiB payloads. Every size produced exactly
1.0 SET/op,0.0 script/op, and0.0 TIME/op. Absolute latency remains environment-dependent; the benchmark validates command shape and sequential latency, not saturated throughput.Tracked reads intentionally remain
MGET(value, watermark): correctness needs an atomic primary snapshot, and invalidated misses still transfer the full payload before Node applies the fence. Standalone GLIDE retains its one-commandBatchso tracked reads cannot follow a replica-read preference.Compatibility and rollout
Wire formats remain compatible, but old tracked state is not safe to carry across the protocol transition because old watermark lifetimes were derived for the old writer.
Before enabling this release for a namespace:
Waiting for natural expiry is safe only with all traffic disabled, bounded remaining TTLs for both tracked values and watermarks, no persistent watermark, and a wait covering the old value and future-buffer-derived watermark lifetimes.
Source compatibility changes:
DialCacheRedisClient.write()returnsvoid, andRedisWriteRequestno longer acceptswatermarkKey;DialCacheRedisPlaceholderLostErrorare removed;dialcacheRedisScriptsandDialCacheNodeRedisScriptsare removed; ordinary node-redis clients require no DialCache script registration;ValkeyGlideRuntimeno longer requiresClusterBatch;fill_blockedis removed fromShadowValidationOutcome;tracked_ttl_clampedis added toMetricErrorKind, so exhaustive switches andRecord<MetricErrorKind, ...>values must add it; andValidation
corepack pnpm check: typecheck, 529 unit tests with coverage, ESM/CJS build, and packed-consumer testscorepack pnpm test:integration: 125 passed, 0 skipped, across Redis 6.2 and Valkey 8 with node-redis and GLIDE, including the GLIDE Cluster suiteSETand zero Lua/TIMEcalls per operation through 1 MiBorigin/main: branch remains based directly onmainCloses #139
BREAKING CHANGE: DialCacheRedisClient.write() now returns void and RedisWriteRequest no longer accepts watermarkKey. The placeholder/stamp protocol and related exports, dialcacheRedisScripts/DialCacheNodeRedisScripts registration facade, ValkeyGlideRuntime.ClusterBatch requirement, and fill_blocked outcome are removed; MetricErrorKind adds tracked_ttl_clamped; and the Prometheus future-timestamp histogram bucket schema changes. Node-redis clients should be created without DialCache script registration.