diff --git a/README.md b/README.md index 65fac64..ef5d955 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ [![Codecov](https://codecov.io/gh/lan17/DialCache/branch/main/graph/badge.svg)](https://codecov.io/gh/lan17/DialCache) [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/lan17/DialCache/badge)](https://scorecard.dev/viewer/?uri=github.com/lan17/DialCache) -Fine-grained TypeScript caching with explicit enabled contexts, request-local memoization, process-local and Redis TTL caching, stable key construction, runtime rollout controls, request coalescing, adapter-based observability, and Redis watermark-based targeted invalidation. +Fine-grained TypeScript caching with explicit enabled contexts, request-local memoization, process-local and Redis TTL caching, opt-in stale-on-error recovery, stable key construction, runtime rollout controls, request coalescing, adapter-based observability, and Redis watermark-based targeted invalidation. ## Contents @@ -17,7 +17,7 @@ Fine-grained TypeScript caching with explicit enabled contexts, request-local me - [Keys, ids, and extra dimensions](#keys-ids-and-extra-dimensions) - [Runtime config and ramp controls](#runtime-config-and-ramp-controls) - [Cache layers](#cache-layers) - - [Request-local cache](#request-local-cache) · [Process-local cache](#process-local-cache) · [Redis-backed TTL cache](#redis-backed-ttl-cache) · [Remote read deadlines](#remote-read-deadlines-and-async-liveness) · [Serialization](#serialization) · [Compression](#compression) · [Shadow validation](#shadow-validation) + - [Request-local cache](#request-local-cache) · [Process-local cache](#process-local-cache) · [Redis-backed TTL cache](#redis-backed-ttl-cache) · [Stale on source error](#stale-on-source-error) · [Remote read deadlines](#remote-read-deadlines-and-async-liveness) · [Serialization](#serialization) · [Compression](#compression) · [Shadow validation](#shadow-validation) - [Cached-value ownership](#cached-value-ownership) - [Targeted invalidation and watermarks](#targeted-invalidation-and-watermarks) - [Request coalescing](#request-coalescing) @@ -82,9 +82,9 @@ request-local cache -> process-local cache -> Redis cache -> fallback function - Results from the lower chain are memoized request-locally when that layer is enabled. - Process-local hits return immediately. - Process-local misses try Redis and populate the process-local cache on a Redis hit. -- Redis misses call the fallback and attempt to write one complete Redis frame. An active untracked process-local layer may publish that fallback result directly. For tracked keys, invocations that reach the Redis read/write path suppress direct process-local publication after fallback until a later validated Redis hit can warm it; local-only, remote-policy-disabled, and ramped-down paths remain governed by their local policy and may publish locally. +- Redis misses call the fallback and attempt to write one complete Redis frame. When stale-on-error is opted in, the initial Redis read may retain a logically expired frame as a recovery candidate. An eligible fallback rejection can return that snapshot without another Redis read. An active untracked process-local layer may publish successful fallback results directly. For tracked keys, invocations that reach the Redis read/write path suppress direct process-local publication after fallback until a later validated Redis hit can warm it; local-only, remote-policy-disabled, and ramped-down paths remain governed by their local policy and may publish locally. - Selected Redis keys can execute non-serving [shadow work](#shadow-validation) that validates hits and fills semantic misses, even before Redis is allowed to serve callers. -- Redis read failures and timeouts are logged, counted in metrics, and fail open without attempting a second Redis operation. Redis write failures also fail open. `invalidateRemote` requires a configured Redis client; missing configuration and Redis failures are logged, counted, and rethrown so callers do not assume invalidation succeeded. +- Initial Redis read failures and timeouts are logged, counted in metrics, and fail open without attempting stale recovery. Redis write failures also fail open. `invalidateRemote` requires a configured Redis client; missing configuration and Redis failures are logged, counted, and rethrown so callers do not assume invalidation succeeded. - Cache-key construction and config-provider failures also fail open and run the fallback uncached. - A missing effective process-local/Redis TTL disables that layer by policy; a configured TTL with no ramp defaults to 100%. Disabled layers record a disabled reason and fall through to the next layer/fallback. @@ -125,6 +125,7 @@ Use `cached(fn, options)` for an extracted, reusable function. The wrapped calla | `serializer` | when the return type is not statically JSON-compatible | Per-function `Serializer` for Redis values (see [Serialization](#serialization)). | | `shadowComparator` | no | Synchronous application-level equality for shadow validation; defaults to Node's strict deep equality. | | `trackForInvalidation` | no (default `false`) | Opts this use case's Redis entries into watermark-based targeted invalidation. | +| `shouldAttemptStaleRecovery` | no | Synchronous source-error classifier for this use case; replaces the instance or built-in classifier (see [Stale on source error](#stale-on-source-error)). | | `fallbackTimeoutMs` | no (default `60_000`) | Fallback deadline in milliseconds, at most 2,147,483,647; `null` disables it (see [Fallback deadlines](#fallback-deadlines)). | `cached()` validates `useCase` at registration: a duplicate within one `DialCache` instance throws `UseCaseIsAlreadyRegisteredError`. Both APIs reject the internal name `watermark` with `UseCaseNameIsReservedError`. @@ -151,7 +152,7 @@ const profile = await dialcache.getOrLoad( ); ``` -The options match `cached()` except that the direct `key` replaces the `cacheKey` selector. `defaultConfig` and `fallbackTimeoutMs` are validated and snapshotted for each invocation. Outside an enabled scope, DialCache invokes `load` directly without constructing a key or resolving runtime policy. +The options match `cached()` except that the direct `key` replaces the `cacheKey` selector. `defaultConfig`, `fallbackTimeoutMs`, and the selected stale-recovery classifier are validated and captured for each invocation. Outside an enabled scope, DialCache invokes `load` directly without constructing a key, resolving runtime policy, or invoking the classifier. `getOrLoad()` does not register its `useCase`, so repeated calls should reuse one stable, deployment-defined name such as `"BuildProfile"`. Keep it bounded: never derive `useCase` from a user, request, id, or other high-cardinality input because it is part of both cache identity and metrics labels. Put those values in `key` instead. @@ -206,24 +207,25 @@ Instance-wide behavior is set through the `DialCache` constructor: | `localMaxSize` | `10_000` | Global process-local entry cap; `0` disables process-local storage. Nonnegative safe integer. | | `shadowMaxInFlight` | `1` | Maximum scheduled or active shadow jobs per `DialCache` instance, including uncancellable underlying work. Positive safe integer; excess work is dropped without queuing. | | `cacheConfigProvider` | none | Resolves runtime config per enabled invocation as a sparse overlay on the function's `defaultConfig`; `null` applies no overrides. | +| `shouldAttemptStaleRecovery` | only `FallbackTimeoutError` | Synchronous instance-default source-error classifier. A per-use-case callback replaces it. | | `metrics` | disabled | A `DialCacheMetricsAdapter` (see [Metrics](#metrics)). | | `logger` | `console` | Receives operational cache failures and opted-in confirmed shadow mismatch warnings (`debug`, `warn`, `error`). Synchronous throws and rejections from returned promises or thenables are isolated without being awaited. | -Per-invocation cache policy is a `DialCacheKeyConfig`: per-layer `ttlSec` and `ramp` maps keyed by `CacheLayer.LOCAL` (process-local) and `CacheLayer.REMOTE` (Redis), a `requestLocal` boolean, a `coalesce` boolean (see [Request coalescing](#request-coalescing)), an optional `remoteReadTimeoutMs`, and an optional `shadow` group. `ShadowConfig` contains the independent shadow `ramp` percentage plus the default-off `logMismatches` control. +Per-invocation cache policy is a `DialCacheKeyConfig`: per-layer `ttlSec` and `ramp` maps keyed by `CacheLayer.LOCAL` (process-local) and `CacheLayer.REMOTE` (Redis), a `requestLocal` boolean, a `coalesce` boolean (see [Request coalescing](#request-coalescing)), an optional `staleOnErrorMaxAgeSec`, an optional `remoteReadTimeoutMs`, and an optional `shadow` group. `ShadowConfig` contains the independent shadow `ramp` percentage plus the default-off `logMismatches` control. Every cached definition or `getOrLoad()` invocation can provide an optional per-use-case `defaultConfig`. It is the baseline policy, and the `cacheConfigProvider` result is a sparse field-level overlay on that baseline. For cache enablement fields, precedence is runtime config, then `defaultConfig`, then DialCache's disabled baseline. For the remote-read deadline, precedence is runtime `remoteReadTimeoutMs`, `defaultConfig.remoteReadTimeoutMs`, `redis.readTimeoutMs`, then the 50 ms library default. -The disabled baseline sets `requestLocal` to false, leaves the process-local and Redis TTLs unset, and sets `shadow.ramp` to 0% with mismatch logging false. A shared layer with no effective TTL is disabled by policy. When a shared layer has an effective TTL but no effective ramp, its ramp defaults to 100%. Shadow work remains disabled unless `shadow.ramp` is explicitly greater than zero. +The disabled baseline sets `requestLocal` to false, leaves the process-local and Redis TTLs and stale-on-error maximum unset, and sets `shadow.ramp` to 0% with mismatch logging false. A shared layer with no effective TTL is disabled by policy. When a shared layer has an effective TTL but no effective ramp, its ramp defaults to 100%. Shadow work remains disabled unless `shadow.ramp` is explicitly greater than zero. -`DialCacheKeyConfig` preserves an omitted `requestLocal` as `undefined` so the overlay can distinguish omission from an explicit `false`; the effective value still defaults to false after resolution. An omitted `coalesce` is preserved the same way, and its effective value defaults to true, so request coalescing stays on unless a use case explicitly opts out. +`DialCacheKeyConfig` preserves an omitted `requestLocal` as `undefined` so the overlay can distinguish omission from an explicit `false`; the effective value still defaults to false after resolution. An omitted `coalesce` is preserved the same way, and its effective value defaults to true, so request coalescing stays on unless a use case explicitly opts out. An omitted `staleOnErrorMaxAgeSec` inherits an earlier value in a sparse runtime overlay and otherwise leaves recovery off; `0` explicitly disables an inherited stale policy. -A provider result of `null` (or defensive `undefined`) applies no overrides. An empty `DialCacheKeyConfig` and omitted runtime fields also inherit the baseline. Top-level fields, cache-layer leaves, and leaves inside `shadow` merge independently; an explicit `false` logging flag overrides an inherited `true`. Use explicit values to override inherited policy: `requestLocal: false` disables request-local caching and a layer ramp of `0` disables that shared layer. `DialCacheKeyConfig.disabled()` is the complete new-cache-invocation kill switch in one call: request-local and shadow work off, shadow logging off, and both shared layers ramped to 0. It leaves `coalesce` unset: with every layer off there is no in-flight sharing to disable, and a use case ramped back up at runtime coalesces again unless it explicitly opts out. It does not cancel already-admitted work, and explicit maintenance operations such as `invalidateRemote()` remain available. To stop new cache-invocation Redis reads and fills while preserving other runtime settings, explicitly set both `ramp.remote` and `shadow.ramp` to `0`; the remote ramp alone stops serving but does not override an inherited nonzero shadow ramp. +A provider result of `null` (or defensive `undefined`) applies no overrides. An empty `DialCacheKeyConfig` and omitted runtime fields also inherit the baseline. Top-level fields, cache-layer leaves, and leaves inside `shadow` merge independently; an explicit `false` logging flag overrides an inherited `true`. Use explicit values to override inherited policy: `requestLocal: false` disables request-local caching, `staleOnErrorMaxAgeSec: 0` disables stale recovery, and a layer ramp of `0` disables that shared layer. `DialCacheKeyConfig.disabled()` is the complete new-cache-invocation kill switch in one call: request-local, stale recovery, and shadow work off, shadow logging off, and both shared layers ramped to 0. It leaves `coalesce` unset: with every layer off there is no in-flight sharing to disable, and a use case ramped back up at runtime coalesces again unless it explicitly opts out. It does not cancel already-admitted work, and explicit maintenance operations such as `invalidateRemote()` remain available. To stop new cache-invocation Redis reads and fills while preserving other runtime settings, explicitly set both `ramp.remote` and `shadow.ramp` to `0`; the remote ramp alone stops serving but does not override an inherited nonzero shadow ramp. -DialCache validates `defaultConfig` when `cached()` registers a definition and whenever `getOrLoad()` is invoked: TTLs must be positive safe integers no greater than 31,536,000 seconds (a fixed 365-day duration), remote-read deadlines must be positive safe integers within their documented limit, layer and shadow ramps must be finite percentages from 0 to 100, layer maps and `shadow` must be objects, and `requestLocal`, `coalesce`, and `shadow.logMismatches` must be booleans when present. Invalid defaults are rejected immediately. +DialCache validates `defaultConfig` when `cached()` registers a definition and whenever `getOrLoad()` is invoked: TTLs must be positive safe integers no greater than 31,536,000 seconds (a fixed 365-day duration), a positive stale-on-error maximum must be a safe integer in the same range and strictly greater than the remote TTL, remote-read deadlines must be positive safe integers within their documented limit, layer and shadow ramps must be finite percentages from 0 to 100, layer maps and `shadow` must be objects, and `requestLocal`, `coalesce`, and `shadow.logMismatches` must be booleans when present. `0` is the one valid non-positive stale maximum and explicitly disables recovery. Invalid defaults are rejected immediately. Each registration or one-shot invocation captures an immutable internal snapshot of `defaultConfig`; mutating the supplied config or its maps later does not change that operation's baseline. Runtime policy changes belong in the provider's returned overlay. -Runtime TTL and ramp leaves are used as supplied instead of falling back to valid default leaves. A TTL outside the same 1-to-31,536,000-second range disables that layer with `invalid_ttl`; a nonnumeric, non-finite, or out-of-range ramp disables it with `invalid_ramp`. Valid ramps include both `0` and `100`. Other layers can still run, and invalid leaves also record a `config_resolution` error so provider garbage is alertable separately from intentional ramp-downs. A malformed runtime config object, layer-map shape, `requestLocal` value, `coalesce` value, explicit `remoteReadTimeoutMs`, or removed top-level `shadowRamp` fails config resolution for the invocation, records `config_error`, and executes the fallback uncached without attempting Redis. The public `DialCacheKeyConfig` constructor and static defaults likewise reject `shadowRamp` immediately; migrate it to `shadow.ramp`. +Runtime TTL and ramp leaves are used as supplied instead of falling back to valid default leaves. A TTL outside the same 1-to-31,536,000-second range disables that layer with `invalid_ttl`; a nonnumeric, non-finite, or out-of-range ramp disables it with `invalid_ramp`. Valid ramps include both `0` and `100`. Other layers can still run, and invalid leaves also record a `config_resolution` error so provider garbage is alertable separately from intentional ramp-downs. A malformed runtime config object, layer-map shape, `requestLocal` value, `coalesce` value, explicit `remoteReadTimeoutMs`, or removed top-level `shadowRamp` fails config resolution for the invocation, records `config_error`, and executes the fallback uncached without attempting Redis. An invalid runtime `staleOnErrorMaxAgeSec` is narrower: DialCache records `config_resolution`, disables recovery for that invocation, and preserves an otherwise valid fresh Redis policy. The public `DialCacheKeyConfig` constructor and static defaults likewise reject `shadowRamp` immediately; migrate it to `shadow.ramp`. An invalid runtime `shadow.ramp` does not affect the cache result or disable an otherwise valid Redis policy. If normal traversal reaches an otherwise shadow-eligible Redis path, DialCache skips shadow work and records a `config_resolution` error. An invalid runtime `shadow.logMismatches` likewise preserves the cache result, Redis policy, shadow result, and shadow metric while suppressing the warning. DialCache validates this diagnostic leaf only after the metrics hook, exact-key cohort, and capacity gates admit shadow work, then records one remote `config_resolution` error for that admitted resolution. @@ -246,6 +248,8 @@ const dialcache = new DialCache({ }, // Can be changed by the provider at runtime for this use case. remoteReadTimeoutMs: 35, + // Sparse override of the maximum retained age; use 0 to turn recovery off. + staleOnErrorMaxAgeSec: 1_800, }); } return null; // apply no overrides; use the cached function's baseline @@ -259,6 +263,8 @@ const getUser = dialcache.cached((userId: string) => db.fetchUser(userId), { defaultConfig: new DialCacheKeyConfig({ // Omitted ramps default to 100% because these layers have TTLs. ttlSec: { [CacheLayer.LOCAL]: 30, [CacheLayer.REMOTE]: 300 }, + // Keep Redis data for up to one hour and serve it only after a source error. + staleOnErrorMaxAgeSec: 3_600, }), }); ``` @@ -402,9 +408,11 @@ Awaiting those public promises does not drain detached shadow work. Shadow sched Neither adapter owns additional resources or native script handles, so the application simply closes the underlying client after draining work. -Reads use native `GET` for untracked entries and one atomic `MGET` for each tracked value-and-watermark pair. The adapters validate and decode the returned frame in the Node process. Tracked reads are deliberately routed to primaries so a lagging replica cannot hide an invalidation watermark. After a tracked read settles, DialCache samples `readerNowMs = Date.now()` once. On caller-serving reads and initial shadow observations, a frame with `createdAtMs > readerNowMs` fails closed as a miss before deserialization and emits the bounded offset observation described under [Metrics](#metrics). A confirmation read still observes a future offset, but payload equality—not a second wall-clock sample—decides whether the original observation was superseded. Untracked reads do not consult the informational frame timestamp for serving; Redis physical TTL remains their normal serving lifetime. +Reads use native `GET` for untracked entries and one atomic `MGET` for each tracked value-and-watermark pair. The adapters validate and decode the returned frame in the Node process. Tracked reads are deliberately routed to primaries so a lagging replica cannot hide an invalidation watermark. After a read settles, DialCache evaluates the frame against the observing application's `Date.now()`. Without stale-on-error, the read accepts only nonnegative ages strictly below the effective remote TTL `F`. With a positive recovery maximum `M`, that same initial read is bounded by `M`: ages below `F` deserialize and serve normally, while ages from `F` through strictly below `M` remain raw as a possible source-error recovery candidate. Future-dated frames fail closed before deserialization and emit the bounded offset observation described under [Metrics](#metrics). A shadow confirmation read still observes a future offset but retains the payload only for supersession comparison; it can never serve that frame. -Writes are native too, so the payload never crosses the Redis-to-Lua boundary. Each bundled adapter samples the writer process's `Date.now()` once immediately before dispatch and issues one `SET valueKey frame PX cacheTtlMs` containing the complete version-1 frame. Tracked and untracked writes have the same adapter request and command shape; only tracked reads include the watermark key. Core caps a tracked Redis value's physical TTL at one hour, while untracked Redis and local TTLs retain their configured limits. Each dispatched write whose configured tracked TTL exceeds that cap emits `error="tracked_ttl_clamped"`; the write is attempted with the capped TTL. The write never reads, creates, or extends a watermark. A frame written behind an active watermark remains physically present but is a tracked miss until its `createdAtMs` is greater than the watermark. Same-key writes are ordinary Redis last-writer-wins operations, with no Lua, pipeline, or transaction on the write path. +Writes are native too, so the payload never crosses the Redis-to-Lua boundary. Each bundled adapter samples the writer process's `Date.now()` once immediately before dispatch and issues one `SET valueKey frame PX cacheTtlMs` containing the complete version-1 frame. Tracked and untracked writes have the same adapter request and command shape; only tracked reads include the watermark key. With stale-on-error enabled, the requested physical TTL is `M` instead of `F`. `M` remains the configured logical recovery ceiling even when it exceeds one hour. Core separately caps every tracked Redis value's physical TTL at one hour, so such a tracked candidate may be evicted by expiry before it reaches logical age `M`; untracked Redis and local TTLs retain their configured limits. Each dispatched tracked write whose requested TTL exceeds that cap emits `error="tracked_ttl_clamped"`. The write never reads, creates, or extends a watermark. A frame written behind an active watermark remains physically present but is a tracked miss until its `createdAtMs` is greater than the watermark. Same-key writes are ordinary Redis last-writer-wins operations, with no Lua, pipeline, or transaction on the write path. + +The network shape remains one top-level Redis command and one round trip per semantic read (`GET` or `MGET`) and one `SET` per write. Stale recovery reuses the frame returned by that initial command and never adds a second Redis read, including after a source rejection. Retaining a raw candidate instead consumes process memory until that source attempt settles, once per distinct in-flight key (same-key coalesced callers share it). DialCache does not call Redis `TIME` or maintain a Redis-clock offset. Use the maintainer benchmarks below to measure the target Redis/Valkey version, payload distribution, and high-cardinality in-flight memory exposure. Native commands retain Redis's wrong-type behavior. An untracked `GET` surfaces `WRONGTYPE`; tracked `MGET` represents a wrong-type member as a missing value. A wrong-type tracked value is therefore a semantic miss and may be replaced with a valid complete frame after fallback. A wrong-type watermark is indistinguishable from an absent watermark to `MGET`, so tracked reads apply the zero baseline; it does not prevent or alter native value writes. The next explicit invalidation replaces that wrong-type key with a valid string watermark. Other script read failures remain errors and cannot bypass the monotonic update. @@ -414,13 +422,59 @@ Invalidation is the only remaining Lua operation. Both adapters dispatch it as ` Command-restricted Redis ACLs must allow native `GET`, `MGET`, and `SET`, plus `EVALSHA` and `EVAL` for invalidation recovery. If script-invoked commands are checked separately, the invalidation script needs `GET`, `SET`, and `PTTL`. Redis `TIME`, `MULTI`, `EXEC`, `WATCH`, `UNLINK`, and `SCRIPT LOAD` are not used by DialCache. The integration matrix covers Redis 6.2 and Valkey 8. +#### Stale on source error + +Stale-on-error is an opt-in Redis policy with two ages: + +- `F = ttlSec.remote` is the logical fresh lifetime. Every ordinary Redis read enforces `F`, regardless of whether recovery is enabled. +- `M = staleOnErrorMaxAgeSec` is the configured logical recovery-age ceiling. When configured, Redis requests physical retention through `M`, but ordinary reads still treat the frame as a miss at age `F`. `M` is not clipped for tracked use cases; their Redis value TTL is separately capped at one hour, so a retained frame can physically disappear before reaching `M`. + +`F` and `M` bound only Redis serving. Request-local and process-local hits occur earlier in the chain and follow their own scope or TTL lifetimes. A Redis frame can be nearly `F` old when it warms the process-local cache and then receive a full local TTL, so setting the local TTL to `F` or lower does not make `F` a strict end-to-end age limit. Disable those earlier layers when that global bound is required. + +With a positive `M`, the initial Redis command admits frames up to `M` and classifies them in the Node process. A frame with `0 <= age < F` is deserialized and served as a normal hit. A frame with `F <= age < M` records the ordinary Redis miss but retains its raw payload while DialCache calls the source of truth. A missing, future-dated, watermark-fenced, or `age >= M` frame is a miss with no retained candidate. A fresh frame that fails initial deserialization is likewise not eligible to be reconsidered as stale. + +If the source rejects, DialCache first applies a synchronous `shouldAttemptStaleRecovery(error)` classifier. Precedence is per-use-case `cached()`/`getOrLoad()` option, then the `DialCache` instance option, then the built-in policy. The built-in policy authorizes only `error instanceof FallbackTimeoutError`, whether the error came from this invocation's deadline or propagated from a nested/source operation. An override replaces the lower-precedence policy rather than composing with it, so an application override that should preserve timeout recovery must include that case itself. Custom predicates should narrowly admit transient, retriable infrastructure failures. They should deny authoritative domain outcomes such as auth, permission, entitlement, or revocation failures; deletion or not-found results; and validation or programmer errors. Use a per-use-case classifier when particular data requires a stricter policy than the instance default. A supplied policy must be a function; DialCache validates it at instance construction, cached-wrapper registration, or `getOrLoad()` invocation. During classification, a callback throw, thenable, or non-boolean result fails closed; DialCache consumes a rejecting thenable, logs the classifier failure, and preserves the original source rejection. Calls outside an enabled context never invoke the classifier. + +When the classifier returns `true`, DialCache consults only the frame retained from the initial read and issues no additional Redis command. It requires `0 <= Date.now() - createdAtMs < M` both before and after lazy deserialization/decompression, so crossing `M` during an asynchronous serializer load cannot serve. A valid candidate suppresses the source rejection for that caller. A missing or expired candidate, or a candidate that cannot deserialize/decompress, rethrows the exact original source rejection. + +```ts +import { CacheLayer, DialCacheKeyConfig, FallbackTimeoutError } from "dialcache"; + +const getUser = dialcache.cached((userId: string) => db.fetchUser(userId), { + keyType: "user_id", + useCase: "GetUserWithStaleRecovery", + cacheKey: (userId) => userId, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, // F: normal reads serve for 60 seconds + staleOnErrorMaxAgeSec: 300, // M: recovery requires age < 5 minutes + }), + // This replaces the built-in classifier, so retain its timeout case explicitly. + shouldAttemptStaleRecovery: (error) => + error instanceof FallbackTimeoutError || isRetriableDatabaseError(error), +}); +``` + +Omitting `staleOnErrorMaxAgeSec` keeps recovery off and inherits a configured default when used in a sparse runtime overlay. An explicit `0` disables it. A positive `M` requires an enabled remote TTL and must satisfy `0 < F < M <= 31_536_000`; invalid static defaults throw, while an invalid runtime overlay records `config_resolution`, disables only stale recovery for that invocation, and leaves valid ordinary Redis reads active. The effective config snapshot is fixed for the invocation: its initial read uses that snapshot's `F`, `M`, and remote-read deadline even if the provider changes while the source call is in flight. `cached()` captures its selected classifier when the wrapper is registered; `getOrLoad()` captures one for each invocation. + +The retained frame is a snapshot. For a tracked key, the initial primary-routed `MGET` atomically applies the watermark that existed with the value at read time, so an invalidation completed before that read fences the candidate. An invalidation completed after the read does not revoke the already-retained bytes. Concurrent refreshes, deletions, expiry, and eviction after the initial read are likewise not observed for either tracked or untracked candidates. Opting a tracked use case into stale recovery therefore permits recovery to weaken its usual strict freshness guarantee during this source-error path; applications that cannot tolerate that behavior should leave recovery disabled or deny the error in their classifier. + +Recovery does not write Redis, populate the process-local cache, schedule shadow validation, or emit the shadow value-age observation. When request-local caching is enabled, the recovered value is memoized only in that outer `enable()` scope. This prevents an outage response from becoming a new shared cache value. + +Default coalescing applies to the whole sequence, so same-key followers share one initial read, one retained candidate, one source rejection, and one recovery decision. With `coalesce: false`, each concurrent caller instead performs its own initial read, retains its own candidate, and runs its own source call and independent fallback deadline; request-local memoization can still serve later sequential calls after a recovered value settles. High-cardinality delayed source failures can therefore retain one raw payload per distinct in-flight key until the source settles. + +Each classifier-authorized recovery check emits exactly one optional `staleRecovery` outcome: `served`, `miss`, or `deserialization_error`. It does not add another ordinary `request`, `observeGet`, `miss`, or cache-read error: the initial Redis operation is the one caller-serving telemetry trail. `served` additionally emits the optional stale-recovery value-age observation, measured at actual return time in seconds; `miss` and `deserialization_error` emit no age. Existing fallback-duration and fallback-error telemetry still records the source rejection even when recovery serves. Recovery never adds a raw exception or key to metric labels. + +This feature keeps the existing frame-v1 Redis keys; it does not create a second stale key or change the TypeScript shape of `DialCacheRedisClient` or `RedisReadRequest`. Its timestamp behavior is nevertheless breaking for custom clients: ordinary untracked reads previously treated `createdAtMs` as informational, while they now require the real frame timestamp to satisfy logical `F` (and recovery uses it for `M`). Custom clients that returned a constant must return a valid epoch-millisecond stamp before upgrading. Roll out the DialCache version that enforces logical `F` everywhere before enabling writers that retain values to `M`. A pre-feature reader trusts physical expiry and could otherwise serve a retained frame normally between `F` and `M`. Once any write uses physical `M`, treat that keyspace as a downgrade barrier until every such key has expired or been explicitly removed; disabling recovery on current readers is safe because they still enforce `F`, but reintroducing an older reader is not. + +The `F` and `M` comparisons follow the [application-process clock contract](#targeted-invalidation-and-watermarks) used by all serving timestamps. Choose both ages with the deployment's observed clock skew in mind. Redis physical expiry bounds key storage and whether a read can acquire a candidate; it does not revoke a snapshot already retained by the process. That snapshot remains eligible only while the return-time check satisfies `age < M`, even if Redis expires, deletes, or evicts the key after the initial read. + #### Remote read deadlines and async liveness DialCache bounds every active Redis read. The effective timeout is resolved per use case and per invocation: runtime `remoteReadTimeoutMs`, then `defaultConfig.remoteReadTimeoutMs`, then optional instance `redis.readTimeoutMs`, then 50 ms. Values must be positive safe integers no greater than 2,147,483,647. There is no unbounded escape hatch for remote reads. When the deadline expires, DialCache aborts the optional `RedisReadContext.signal`, records one `cache_read_timeout` error, logs a `RedisReadTimeoutError`, and starts the source fallback. Late read fulfillment or rejection is consumed and ignored. A read failure or timeout never triggers a post-fallback Redis write; an untracked active process-local miss may retain the source value, while a tracked key suppresses local publication because the failed read did not establish watermark safety. -Same-key followers share the leader's remaining remote-read budget. The timer covers only the semantic Redis read, not config resolution, serializer load, fallback work, Redis writes, or invalidation. `fallbackTimeoutMs` starts separately when the source fallback begins. +Same-key followers share the leader's remaining remote-read budget. The timer covers the one semantic Redis read, not config resolution, serializer load, fallback work, Redis writes, or invalidation. `fallbackTimeoutMs` starts separately when the source fallback begins. With stale-on-error enabled, the initial read may retain a raw candidate through that fallback; recovery reuses it and creates no second Redis-read budget. The bundled node-redis adapter passes the signal through per-command options, which can remove queued work where supported. Aborting after dispatch does not unsend a command or prove that Redis stopped executing it. GLIDE's current adapter commands have no per-invocation signal, so a read may continue after DialCache has fallen back. Keep client-native connection, retry, queue, and response budgets in place; they bound underlying resource lifetime while DialCache's deadline bounds caller wait time. @@ -443,7 +497,7 @@ bytes 11... serialized payload (optionally zstd-compressed; see Compression) Adapters build complete frames in the Node process with `encodeRedisFrame` and decode them with Node buffer primitives. The version-1 value-envelope format, Redis value-key derivation, and decimal watermark encoding are unchanged. That wire compatibility does not make old tracked state safe to carry across the protocol cutover: old watermark lifetimes were derived for the old write protocol. -Redis physical TTL remains authoritative for normal expiry; the frame timestamp supports future-frame rejection and shadow value-age observability. `payload` is produced by the operation's serializer, or by `JsonSerializer` by default. Custom serializers can return either `string` or `Buffer`. Payloads stored raw keep their exact serialized bytes: strings are stored as UTF-8 and Buffers byte-for-byte without base64 expansion, except that binary output beginning with a [compression envelope byte](#compression) (`0x00`–`0x02`) gains a one-byte escape prefix on the wire. Payloads at or above the compression threshold may instead be stored as a zstd envelope (see [Compression](#compression)), so wire bytes for large values are not the serializer's output. Adapters return the frame payload as-is; the envelope — including restoring a compressed string's representation before `serializer.load` — is interpreted by the core above them. +Redis physical TTL bounds how long the stored key remains available to future reads. A completed read owns its returned frame, so later expiry, deletion, or eviction cannot revoke that in-process snapshot. The frame timestamp enforces logical `F`/`M` age, future-frame rejection, and shadow value-age observability. `payload` is produced by the operation's serializer, or by `JsonSerializer` by default. Custom serializers can return either `string` or `Buffer`. Payloads stored raw keep their exact serialized bytes: strings are stored as UTF-8 and Buffers byte-for-byte without base64 expansion, except that binary output beginning with a [compression envelope byte](#compression) (`0x00`–`0x02`) gains a one-byte escape prefix on the wire. Payloads at or above the compression threshold may instead be stored as a zstd envelope (see [Compression](#compression)), so wire bytes for large values are not the serializer's output. Adapters return the frame payload as-is; the envelope — including restoring a compressed string's representation before `serializer.load` — is interpreted by the core above them. DialCache uses native `JSON.stringify` and `JSON.parse` by default. There is no runtime validation pass, so the default adds no traversal beyond JSON serialization itself. A top-level `undefined` result is supported with an internal sentinel. @@ -565,12 +619,12 @@ The detached job uses this bounded algorithm: 3. If `C0` is non-null, obtain `S`, deserialize an isolated snapshot of `C0`, and run the default or custom semantic comparator. Any non-null `C0` is observation-only: DialCache never repairs or overwrites it, including when deserialization fails. 4. If `C0` and `S` match semantically, emit `match` without another Redis read. 5. Otherwise, reread Redis directly in the same mode as `C1`, bypassing request-local and process-local cache. -6. If `C1` is missing under the normal value/watermark protocol or differs byte-for-byte from `C0`, emit `superseded`; if it is identical, emit `mismatch`. A future-dated tracked `C1` records its offset but remains available for this payload comparison, so a reader-clock step does not change the verdict. +6. If `C1` is missing under the normal value/watermark protocol or differs byte-for-byte from `C0`, emit `superseded`; if it is identical, emit `mismatch`. A future-dated `C1` records its offset but remains available for this payload comparison, so a reader-clock step does not change the verdict. 7. If the confirmation read fails or reaches its Redis-read deadline, emit `confirmation_error`. Here a semantic miss means the Redis read returned `null`; it does not include a non-null payload that later fails deserialization. A physical tracked frame rejected by its watermark, timestamp domain, or future-time check is therefore a miss and may be replaced by the fill. A caller fallback rejection or timeout never becomes accepted `S` and never starts the fill. -Both detached Redis reads use the effective `remoteReadTimeoutMs` and the key's normal protocol. Every semantic-miss fill uses the same serializer, TTL, complete-frame SET, and client-clock timestamp semantics as an ordinary fill. Tracked `C0` and `C1` reads remain watermark-aware and are routed to primaries by the bundled adapters. The fill itself never reads or mutates the watermark. Untracked reads use the ordinary one-key read route, which has no shadow-specific primary guarantee. Strings compare exactly, Buffers compare by bytes, and string/Buffer pairs compare by their UTF-8 bytes. DialCache does not deserialize `C1`, compare it with `S`, or chase another version. +Both detached Redis reads use the effective `remoteReadTimeoutMs` and the key's normal native `GET` or `MGET` protocol. The initial `C0` observation enforces the logical remote TTL `F`; the non-serving `C1` confirmation bypasses logical age solely to determine whether the original payload bytes were superseded. Every semantic-miss fill uses the same serializer, complete-frame SET, and client-clock timestamp semantics as an ordinary fill; when stale-on-error is active, it requests physical retention through `M` while serving reads still enforce `F`. Tracked `C0` and `C1` reads remain watermark-aware and are routed to primaries by the bundled adapters. The fill itself never reads or mutates the watermark. Untracked reads use the ordinary one-key read route, which has no shadow-specific primary guarantee. Strings compare exactly, Buffers compare by bytes, and string/Buffer pairs compare by their UTF-8 bytes. DialCache does not deserialize `C1`, compare it with `S`, or chase another version. The detached scheduler, Redis-read deadline timers, and overall shadow deadline timer are unreferenced, so they do not keep an otherwise idle process alive. Detachment is asynchronous work on the Node event loop, not a worker thread: synchronous source, serializer, or comparator work can still occupy the event loop after the request path has been released. @@ -604,7 +658,7 @@ The byte caps apply before logger framing or escaping, so they do not guarantee Detached Redis reads, serializer loads/dumps, payload sizes, and read/write errors use the existing layer label with `layer="remote_shadow"`. This distinguishes non-serving Redis cost from caller-path `layer="remote"` telemetry without adding a metric or label. The established `observeGet{layer="remote"}` boundary includes caller-path deserialization, while `observeGet{layer="remote_shadow"}` ends when the deadline-bounded Redis read result settles; detached serializer work and any later raw-client settlement are outside that timer. The request-path read that supplied a served `C0` keeps `layer="remote"`, and a ramped-down caller keeps `disabled{layer="remote", reason="ramped_down"}`. No `disabled{layer="remote_shadow"}` event is emitted for ineligible or dropped work; `dropped` remains the terminal shadow outcome. Confirmation reads use the same `remote_shadow` value, with `superseded` or `confirmation_error` describing their role. -The command amplification is bounded: a selected served hit adds one SoT read and adds `C1` only for a semantic mismatch candidate; a selected ramped-down hit adds detached `C0`, reuses the caller's existing SoT read, and likewise adds `C1` only for a candidate; a selected ramped-down miss adds detached `C0` and at most one fill in the key's existing mode. `superseded` means only that the original observation could not be confirmed. `mismatch` means the exact `C0` payload survived another Redis read after the SoT disagreement; it is not a cross-system atomic snapshot or a guarantee that the mismatch persists. For an untracked key it is also not proof of primary freshness or invalidation safety. +The command amplification is bounded: a selected served hit adds one SoT read and adds `C1` only for a semantic mismatch candidate; a selected ramped-down hit adds detached `C0`, reuses the caller's existing SoT read, and likewise adds `C1` only for a candidate; a selected ramped-down miss adds detached `C0` and at most one fill in the key's existing mode. `superseded` means only that the original observation could not be confirmed. `mismatch` means the exact `C0` payload survived another Redis read after the SoT disagreement; it is not a cross-system atomic snapshot or a guarantee that the mismatch persists. For an untracked key it is also not proof of invalidation safety because no watermark participates. The initial `C0` read and later fill are not atomic. The fill is a normal overwrite, not a compare-and-set or write-if-still-missing operation: another writer can populate Redis after the semantic miss and be overwritten by the shadow fill. For tracked keys, the next atomic value-and-watermark read still rejects a frame whose client timestamp is at or before the watermark, so size `futureBufferMs` to cover the complete SoT, serialization, client queue, network, write interval, and fleet clock-skew budget when stale-serving protection matters. An untracked shadow fill has no such read fence and retains the ordinary TTL-based last-writer-wins contract; because it is detached, an older accepted source value may be written after a concurrent source mutation and remain until expiry. Shadow mode never repairs a frame returned as a non-null semantic `C0`; a physical frame rejected by normal tracked-read semantics is a miss and may be overwritten with a fresh TTL. Shadow work never invalidates, evicts local state, or changes the value returned to the caller. @@ -669,7 +723,7 @@ The internal `:dialcache-frame-v1` suffix identifies values written with DialCac A cached Redis value whose writer-provided `createdAtMs` is older than or equal to the watermark is a tracked miss. `invalidateRemote(keyType, id, futureBufferMs)` proposes the invalidating process's `Date.now()` plus the buffer, and Lua keeps the greater of that proposal and the existing watermark. A tracked read obtains value and watermark in one primary-routed `MGET`; a missing watermark is the zero baseline, malformed or out-of-range decimal state fails closed, and `createdAtMs <= watermark` misses. Redis also returns `nil` for a wrong-type member of `MGET`, so a wrong-type watermark has the same zero-baseline behavior as an absent one until the next explicit invalidation repairs it. Native `MGET` must transfer the full stored frame before Node can apply the fence verdict, so a future window can repeatedly transfer a large fenced payload. Fallback still returns normally and writes one complete replacement frame even when that frame remains behind the watermark. A tracked invocation that reaches the Redis read/write path does not publish its fallback directly to process-local cache; a later validated Redis hit may warm it. Local-only, remote-policy-disabled, and ramped-down paths remain governed by local policy, while request-local memoization remains unconditional. A ramped-out invocation without shadow work does not consult Redis; selected tracked shadow reads remain watermark-aware. -All serving timestamps come from application-process epoch clocks; DialCache does not call Redis `TIME`, estimate an offset, or compensate for skew. Participating application nodes therefore need external clock synchronization and monitoring. Healthy managed node pools commonly stay close, but Kubernetes does not guarantee a maximum offset, and pauses or NTP faults can be much larger than normal millisecond-scale skew. A tracked writer ahead of a reader produces fail-closed misses until the reader clock catches up; untracked reads do not gate serving on the informational stamp. Operation durations and deadlines continue to use the monotonic `performance.now()` clock. +All serving timestamps come from application-process epoch clocks; DialCache does not call Redis `TIME`, estimate an offset, or compensate for skew. Participating application nodes therefore need external clock synchronization and monitoring. Healthy managed node pools commonly stay close, but Kubernetes does not guarantee a maximum offset, and pauses or NTP faults can be much larger than normal millisecond-scale skew. Relative clock differences shift logical expiry early or late, while frames dated after a reader clock fail closed until that clock catches up. Operation durations and deadlines continue to use the monotonic `performance.now()` clock. Watermarks are invalidation state, not disposable cache entries. The Redis deployment must preserve them for their derived TTL: use `noeviction` or an equivalent guarantee for deployments that rely on the read-time fence, and choose persistence and failover guarantees appropriate to the application's consistency requirements. If a watermark is lost, a tracked read treats it as zero and may serve an existing frame that the lost watermark had fenced. Alert on memory headroom and rejected writes under `noeviction`; if another eviction policy is used, also alert on `evicted_keys`. Redis replication is asynchronous by default, and DialCache does not issue `WAIT` or provide strong consistency across failover. @@ -707,7 +761,7 @@ Coalescing only applies when at least one cache layer is active and the use case Because coalescing is keyed by the selected or direct key, concurrent calls with the same key share the leader's execution. Any function argument or captured value omitted from the key must be safe to share this way; include inputs such as locale, auth context, or cancellation behavior when they can change the returned value or whether the underlying loader should run separately. -The per-use-case `coalesce` boolean (default true) turns this sharing off. `coalesce: false` in a `defaultConfig` or runtime overlay disables both scopes: same-key concurrent callers each perform their own layer reads with their own full remote-read budget, their own fallback with an independent [fallback deadline](#fallback-deadlines), and their own cache writes — request-local and process-local publication is last-writer-wins, every Redis write is one complete-frame last-writer-wins `SET`, and tracked Redis reads later apply their usual watermark fence. Request-local memoization of settled values still serves later sequential calls in the same scope. Use it when the key intentionally omits per-caller inputs that must not be shared, or when callers must not inherit a leader's failure or `FallbackTimeoutError`. Disabling coalescing reintroduces the thundering-herd exposure described above, emits `request`/`miss`/latency metrics once per caller instead of once per flight, never emits `dialcache_coalesced_counter`, and keeps `getCoalescingState()` idle for that use case. With shadow work enabled, each un-coalesced caller may attempt to schedule detached validation; same-key shadow deduplication and `shadowMaxInFlight` still bound admitted jobs and drop the excess, but source reads are no longer combined. +The per-use-case `coalesce` boolean (default true) turns this sharing off. `coalesce: false` in a `defaultConfig` or runtime overlay disables both scopes: same-key concurrent callers each perform their own layer reads with their own full remote-read budget, their own fallback with an independent [fallback deadline](#fallback-deadlines), and their own cache writes — request-local and process-local publication is last-writer-wins, every Redis write is one complete-frame last-writer-wins `SET`, and tracked Redis reads later apply their usual watermark fence. Request-local memoization of settled values still serves later sequential calls in the same scope. Use it when the key intentionally omits per-caller inputs that must not be shared, or when callers must not inherit a leader's failure, `FallbackTimeoutError`, or stale-recovery result. Disabling coalescing reintroduces the thundering-herd exposure described above, emits `request`/`miss`/latency metrics once per caller instead of once per flight, never emits `dialcache_coalesced_counter`, and keeps `getCoalescingState()` idle for that use case. With shadow work enabled, each un-coalesced caller may attempt to schedule detached validation; same-key shadow deduplication and `shadowMaxInFlight` still bound admitted jobs and drop the excess, but source reads are no longer combined. ### Fallback deadlines @@ -739,11 +793,11 @@ try { } ``` -The timer starts only when the fallback begins, including after a remote-read deadline has elapsed. When coalescing is enabled (the default), same-key followers share the process or request-local leader's remaining budget and receive its `FallbackTimeoutError`; pass-through invocations where every layer is disabled, and callers whose use case disables `coalesce`, have independent timers. Cache hits create no fallback timer. Calls that were initially outside an enabled context remain true pass-through and are not timed out, even when the operation configures `fallbackTimeoutMs`. +The timer starts only when the fallback begins, including after a remote-read deadline has elapsed. When coalescing is enabled (the default), same-key followers share the process or request-local leader's remaining budget and timeout outcome; pass-through invocations where every layer is disabled, and callers whose use case disables `coalesce`, have independent timers. A timeout produces `FallbackTimeoutError`, which the built-in stale-recovery classifier authorizes when the initial Redis read retained a candidate within `M`; otherwise callers receive the error. Cache hits create no fallback timer. Calls that were initially outside an enabled context remain true pass-through and are not timed out, even when the operation configures `fallbackTimeoutMs`. Deadline delivery requires the JavaScript event loop to make progress. It cannot preempt a synchronous fallback prefix or other event-loop blocking, so rejection can arrive later than the configured duration; when control returns, DialCache checks the monotonic deadline before accepting the result. The deadline timer remains referenced until the fallback settles or times out. Consequently, an abandoned enabled fallback can keep an otherwise idle short-lived process alive until that deadline; shutdown code should drain outstanding DialCache work rather than discarding its promises. -Timing out rejects the DialCache chain and clears its flight normally. A later fallback resolution is ignored, so that timed-out invocation cannot become the accepted `S` for a shadow fill or proceed to ordinary serializer, Redis, or local-cache publication. The underlying function is not canceled and may continue its own I/O or side effects; give the source operation its own native timeout or `AbortSignal` whenever possible. `fallbackTimeoutMs: null` disables this guard and makes finite fallback settlement entirely application-owned. Use the `null` escape hatch only after intentionally accepting that liveness risk. It does not create an unbounded detached shadow operation: [shadow validation](#shadow-validation) still uses a 60-second whole-job budget. +Timing out rejects the source attempt with `FallbackTimeoutError`; the DialCache chain either serves an authorized retained candidate or rejects with that exact error, then clears its flight normally. A later fallback resolution is ignored, so that timed-out invocation cannot become the accepted `S` for a shadow fill or proceed to ordinary serializer, Redis, or local-cache publication. The underlying function is not canceled and may continue its own I/O or side effects; give the source operation its own native timeout or `AbortSignal` whenever possible. `fallbackTimeoutMs: null` disables this guard and makes finite fallback settlement entirely application-owned. Use the `null` escape hatch only after intentionally accepting that liveness risk. It does not create an unbounded detached shadow operation: [shadow validation](#shadow-validation) still uses a 60-second whole-job budget. Timeout failures retain the bounded metrics classification `error="fallback"` with `in_fallback="true"`; the typed error provides the timeout details without adding high-cardinality labels. @@ -808,7 +862,9 @@ The Prometheus adapter emits: | `dialcache_coalesced_counter` | Counter | `cache_namespace`, `use_case`, `key_type`, `scope` | Coalesced requests split by `request_local` or `process` scope | | `dialcache_shadow_validation_counter` | Counter | `cache_namespace`, `use_case`, `key_type`, `outcome` | Sampled Redis shadow-job outcomes | | `dialcache_shadow_value_age_histogram` | Histogram | `cache_namespace`, `use_case`, `key_type`, `outcome` | Age in seconds of the validated Redis value at shadow verdict time, recorded for `match` and `mismatch` | -| `dialcache_future_timestamp_offset_histogram` | Histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Positive offset in seconds for a valid tracked frame dated after the observing process clock | +| `dialcache_future_timestamp_offset_histogram` | Histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Positive offset in seconds for a valid frame dated after the observing process clock | +| `dialcache_stale_recovery_counter` | Counter | `cache_namespace`, `use_case`, `key_type`, `outcome` | Classifier-authorized stale-recovery checks: `served`, `miss`, or `deserialization_error` | +| `dialcache_stale_recovery_value_age_histogram` | Histogram | `cache_namespace`, `use_case`, `key_type`, `outcome` | Actual return-time age in seconds of a retained value, recorded only for `served` | | `dialcache_compression_counter` | Counter | `cache_namespace`, `use_case`, `key_type`, `layer`, `outcome` | Payload compression outcomes: writes record `compressed`, `below_threshold`, `not_smaller`, or `write_over_limit`; reads record `decompressed`, `fallback_raw`, or `read_over_limit` | | `dialcache_get_timer` | Histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Cache get latency in seconds | | `dialcache_fallback_timer` | Histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Elapsed time until the underlying function settles or timeout rejection is delivered | @@ -818,11 +874,11 @@ The Prometheus adapter emits: | `dialcache_compression_ratio_histogram` | Histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Compressed-to-original payload size ratio for compressed writes | | `dialcache_compression_timer` | Histogram | `cache_namespace`, `use_case`, `key_type`, `layer`, `operation` | Payload compression and decompression latency in seconds | -The future-timestamp histogram uses dedicated buckets from millisecond-scale skew through multi-hour clock faults. It records one positive offset after a valid tracked frame is decoded. Caller-serving and initial shadow reads then reject that frame; confirmation reads retain it only for payload-equality classification. Invalid or non-finite timestamp values miss without entering histogram sums. The same future frame can be observed repeatedly. Alert against the deployment's allocated skew budget, not every millisecond-level sample. +The future-timestamp histogram uses dedicated buckets from millisecond-scale skew through multi-hour clock faults. It records one positive offset after a valid frame is decoded. Caller-serving and initial shadow reads then reject that frame; confirmation reads retain it only for payload-equality classification. Invalid or non-finite timestamp values miss without entering histogram sums. The same future frame can be observed repeatedly. Alert against the deployment's allocated skew budget, not every millisecond-level sample. `policy_disabled` means that a process-local or Redis layer has no effective TTL after runtime overlays are applied. It is an intentional policy outcome, including the default when `defaultConfig` is omitted, rather than a configuration-loading failure. -Every metric carries `cache_namespace`, including disabled-context, key-construction, coalescing, shadow-validation, and invalidation paths that do not have a constructed key. Its value is `DialCacheConfig.namespace`, defaulting to `urn`. The `layer` label is `request_local`, `local` (process-local), `remote` (caller-serving Redis), or `remote_shadow` (detached, non-serving Redis work); `noop` means no cache layer was reached. Detached reads, serializer work, payload sizes, and Redis read/write errors use `remote_shadow`, while the dedicated bounded shadow `outcome` records the terminal job result. The bounded `scope` label on `dialcache_coalesced_counter` distinguishes request-local from instance-scoped single-flight work. `scope="process"` coordinates calls only within one `DialCache` instance; separate instances in the same process do not share in-flight state. +Every metric carries `cache_namespace`, including disabled-context, key-construction, coalescing, shadow-validation, stale-recovery, and invalidation paths that do not have a constructed key. Its value is `DialCacheConfig.namespace`, defaulting to `urn`. The `layer` label is `request_local`, `local` (process-local), `remote` (caller-serving Redis), or `remote_shadow` (detached, non-serving Redis work); `noop` means no cache layer was reached. Detached reads, serializer work, payload sizes, and Redis read/write errors use `remote_shadow`, while the dedicated bounded shadow `outcome` records the terminal job result. Stale recovery reuses caller-serving Redis work from the initial read, so its dedicated counter and value-age histogram need no `layer` label. The bounded `scope` label on `dialcache_coalesced_counter` distinguishes request-local from instance-scoped single-flight work. `scope="process"` coordinates calls only within one `DialCache` instance; separate instances in the same process do not share in-flight state. ### Datadog @@ -874,7 +930,9 @@ The Datadog adapter emits exact increments of `1` for counters and preserves sec | `dialcache.coalesced.count` | Count | `cache_namespace`, `use_case`, `key_type`, `scope` | Coalesced requests by sharing scope | | `dialcache.shadow.count` | Count | `cache_namespace`, `use_case`, `key_type`, `outcome` | Sampled Redis shadow-job outcomes | | `dialcache.shadow.value_age` | Distribution or histogram | `cache_namespace`, `use_case`, `key_type`, `outcome` | Age in seconds of the validated Redis value at shadow verdict time, recorded for `match` and `mismatch` | -| `dialcache.future_timestamp_offset` | Distribution or histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Positive offset in seconds for a valid tracked frame dated after the observing process clock | +| `dialcache.future_timestamp_offset` | Distribution or histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Positive offset in seconds for a valid frame dated after the observing process clock | +| `dialcache.stale_recovery.count` | Count | `cache_namespace`, `use_case`, `key_type`, `outcome` | Classifier-authorized stale-recovery checks: `served`, `miss`, or `deserialization_error` | +| `dialcache.stale_recovery.value_age` | Distribution or histogram | `cache_namespace`, `use_case`, `key_type`, `outcome` | Actual return-time age in seconds of a retained value, recorded only for `served` | | `dialcache.compression.count` | Count | `cache_namespace`, `use_case`, `key_type`, `layer`, `outcome` | Payload compression outcomes: writes record `compressed`, `below_threshold`, `not_smaller`, or `write_over_limit`; reads record `decompressed`, `fallback_raw`, or `read_over_limit` | | `dialcache.get.duration` | Distribution or histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Cache get latency in seconds | | `dialcache.fallback.duration` | Distribution or histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Elapsed time until the underlying function settles or timeout rejection is delivered | @@ -908,7 +966,7 @@ These values are defined by the backend-neutral core and are identical for every ### Custom adapters -For other telemetry backends, implement `DialCacheMetricsAdapter` and pass the adapter through `new DialCache({ metrics })`. Every backend-neutral label object exposes the logical namespace as camel-case `cacheNamespace`; adapters should map it to their backend's `cache_namespace` label/tag. This field is present even when no key or cache layer was reached. Implement the optional `shadowValidation` method to enable shadow work as well as record its outcomes; omitting it leaves all shadow work disabled even when `shadow.ramp` is nonzero or mismatch logging is enabled. The optional `observeShadowValueAge` method records the validated value's age in seconds for `match` and `mismatch` outcomes; omitting it skips only that observation without affecting shadow eligibility. The optional `observeFutureTimestampOffset` method receives existing bounded cache labels plus the exact positive offset in seconds; omitting it does not change tracked read decisions: serving and initial-shadow reads still miss, while confirmation reads still retain the frame for payload comparison. Every metrics callback is fire-and-forget: DialCache isolates synchronous throws and consumes rejections from returned promises or thenables, but never awaits or drains observer work. Omit `metrics` to disable metrics. +For other telemetry backends, implement `DialCacheMetricsAdapter` and pass the adapter through `new DialCache({ metrics })`. Every backend-neutral label object exposes the logical namespace as camel-case `cacheNamespace`; adapters should map it to their backend's `cache_namespace` label/tag. This field is present even when no key or cache layer was reached. Implement the optional `shadowValidation` method to enable shadow work as well as record its outcomes; omitting it leaves all shadow work disabled even when `shadow.ramp` is nonzero or mismatch logging is enabled. The optional `observeShadowValueAge` method records the validated value's age in seconds for `match` and `mismatch` outcomes; omitting it skips only that observation without affecting shadow eligibility. The optional `observeFutureTimestampOffset` method receives existing bounded cache labels plus the exact positive offset in seconds; omitting it does not change read decisions: serving and initial-shadow reads still miss, while confirmation reads still retain the frame for payload comparison. The optional `staleRecovery` method records one bounded terminal outcome for each classifier-authorized recovery check; omitting it disables only that observation, not recovery itself. The optional `observeStaleRecoveryValueAge` method records the actual return-time age in seconds only when recovery serves; omitting it skips only that observation. Every metrics callback is fire-and-forget: DialCache isolates synchronous throws and consumes rejections from returned promises or thenables, but never awaits or drains observer work. Omit `metrics` to disable metrics. ## Maintainers @@ -932,6 +990,16 @@ pnpm benchmark:redis-write The command builds `dist`, then runs sequential native writes at 100 B, 10 KiB, 100 KiB, and 1 MiB payloads. It reports `SET`, script, and `TIME` calls per operation, server-side `SET` cost from `INFO commandstats`, and client-side p50/p95 latency. Semantic assertions require exactly one `SET`, zero scripts, and zero `TIME` calls per write. Because operations are sequential, the benchmark validates command shape and single-operation latency; it does not measure saturated concurrent throughput or maximum write capacity. Like the cache-path benchmark it is a maintainer tool, is not part of the published package, and applies no timing threshold — absolute numbers depend on the machine, engine, and load, so compare runs only within one environment. Scale iteration counts with `DIALCACHE_BENCH_WRITE_SCALE`. +### Stale-on-error benchmark + +With Redis reachable at `REDIS_URL`, exercise the native-read design and a representative compressible payload: + +```bash +pnpm benchmark:stale-on-error +``` + +The benchmark warms isolated keys, verifies that physical retention uses `M`, and reports fresh end-to-end hits, native reads of a retained frame, end-to-end stale recovery, and same-key coalesced recovery. It asserts exactly one adapter read per stale-recovery flight. A separate high-cardinality scenario holds delayed source calls open with distinct incompressible raw payloads, then reports process memory before retention, while every candidate is retained, and after recovery. Run the built script with `node --expose-gc scripts/benchmark-stale-on-error.mjs` for less noisy memory snapshots. It snapshots `INFO commandstats` and network byte counters around each scenario without resetting shared server statistics, and reports command, server-CPU, network, and client-throughput signals per operation. Semantic assertions cover compression, exact source/recovery/read counts, and returned values; timing and memory remain informational with no pass/fail threshold. Override work sizes with `DIALCACHE_BENCH_STALE_ITERATIONS`, `DIALCACHE_BENCH_STALE_FANOUT`, `DIALCACHE_BENCH_STALE_PAYLOAD_BYTES`, `DIALCACHE_BENCH_STALE_MEMORY_KEYS`, `DIALCACHE_BENCH_STALE_MEMORY_PAYLOAD_BYTES`, and `DIALCACHE_BENCH_STALE_MEMORY_SOURCE_DELAY_MS`. + ### Releasing Publishing starts by manually running the `Release` workflow from current `main`. After the package checks pass, Semantic Release selects the next version from Conventional Commits since the highest stable `vX.Y.Z` tag. While the package is pre-1.0, breaking changes bump minor — their `BREAKING CHANGE:` footers still drive full release notes without forcing 1.0.0 — `feat` bumps minor, and every other normal PR-title type (`fix`, `perf`, `docs`, `style`, `refactor`, `test`, `build`, `chore`, `ci`, and `revert`) bumps patch. The highest required bump wins. Major bumps return when 1.0.0 is cut; `release.config.mjs` implements this table and must change together with this section. diff --git a/package.json b/package.json index 8d6ddcb..0ad4da2 100644 --- a/package.json +++ b/package.json @@ -78,6 +78,7 @@ "scripts": { "benchmark:request-local": "pnpm build && node scripts/benchmark-request-local.mjs", "benchmark:redis-write": "pnpm build && node scripts/benchmark-redis-write.mjs", + "benchmark:stale-on-error": "pnpm build && node scripts/benchmark-stale-on-error.mjs", "build": "tsup src/index.ts src/datadog.ts src/node-redis.ts src/prometheus.ts src/redis-protocol.ts src/valkey-glide.ts --format esm,cjs --dts --clean", "check": "pnpm typecheck && pnpm test && pnpm build && pnpm test:package", "typecheck": "tsc --noEmit", diff --git a/scripts/benchmark-stale-on-error.mjs b/scripts/benchmark-stale-on-error.mjs new file mode 100644 index 0000000..f4b56f4 --- /dev/null +++ b/scripts/benchmark-stale-on-error.mjs @@ -0,0 +1,625 @@ +// Maintainer benchmark for the stale-on-error Redis path. It uses the public +// node-redis adapter against a live Redis, keeps payload I/O native, and +// reports client latency plus INFO commandstats/network deltas. Results have +// no pass/fail timing threshold; compare runs only on the same environment. +// +// Requires Redis, e.g.: docker run --rm -p 6379:6379 redis:6.2 +// Usage: pnpm benchmark:stale-on-error (REDIS_URL to override) +// For less noisy memory deltas: pnpm build && node --expose-gc scripts/benchmark-stale-on-error.mjs +// Optional sizing: DIALCACHE_BENCH_STALE_ITERATIONS, +// DIALCACHE_BENCH_STALE_FANOUT, DIALCACHE_BENCH_STALE_PAYLOAD_BYTES, +// DIALCACHE_BENCH_STALE_MEMORY_KEYS, +// DIALCACHE_BENCH_STALE_MEMORY_PAYLOAD_BYTES, and +// DIALCACHE_BENCH_STALE_MEMORY_SOURCE_DELAY_MS. +import assert from "node:assert/strict"; +import { randomBytes } from "node:crypto"; +import { performance } from "node:perf_hooks"; + +import { createClient } from "redis"; + +import { + CacheLayer, + DialCache, + DialCacheKey, + DialCacheKeyConfig, +} from "../dist/index.js"; +import { createNodeRedisDialCacheClient } from "../dist/node-redis.js"; + +const redisUrl = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; +const iterations = readPositiveInteger("DIALCACHE_BENCH_STALE_ITERATIONS", 200); +const fanout = readPositiveInteger("DIALCACHE_BENCH_STALE_FANOUT", 500); +const payloadBytes = readPositiveInteger("DIALCACHE_BENCH_STALE_PAYLOAD_BYTES", 64 * 1024); +const memoryKeyCount = readPositiveInteger("DIALCACHE_BENCH_STALE_MEMORY_KEYS", 128); +const memoryPayloadBytes = readPositiveInteger( + "DIALCACHE_BENCH_STALE_MEMORY_PAYLOAD_BYTES", + 64 * 1024, +); +const memorySourceDelayMs = readPositiveInteger( + "DIALCACHE_BENCH_STALE_MEMORY_SOURCE_DELAY_MS", + 250, +); +const freshAgeSec = 60; +const logicalFreshAgeSec = 1; +const staleMaxAgeSec = 60; +const redisReadTimeoutMs = 2_000; +const sourceStartTimeoutMs = redisReadTimeoutMs + 1_000; +const namespace = `dialcache-stale-benchmark-${process.pid}-${Date.now()}`; +const keyType = "benchmark_id"; +const id = "shared"; +const sourceError = new Error("benchmark source unavailable"); +const payloadPattern = "dialcache-stale-on-error-compressible-payload-"; +const payload = { + id, + // Repeated text intentionally exercises the default zstd path instead of + // benchmarking an unrealistically tiny raw JSON value. + body: payloadPattern.repeat( + Math.ceil(payloadBytes / payloadPattern.length), + ).slice(0, payloadBytes), +}; +const memoryPayload = randomBytes(memoryPayloadBytes); +// Avoid the single-byte escape used for raw payloads whose first byte overlaps +// a compression marker. This keeps the representative Redis frame exactly ten +// bytes larger than the random serializer payload. +memoryPayload[0] = 0xff; +const rawBufferSerializer = { + dump(value) { + assert(Buffer.isBuffer(value), "the memory benchmark serializer expects a Buffer"); + return Buffer.from(value); + }, + load(value) { + assert(Buffer.isBuffer(value), "the memory benchmark serializer expects binary Redis data"); + return Buffer.from(value); + }, +}; + +const redis = createClient({ + url: redisUrl, + disableOfflineQueue: true, + commandsQueueMaxLength: 1_000, + socket: { connectTimeout: 2_000 }, +}); +redis.on("error", () => undefined); + +try { + await redis.connect(); +} catch (error) { + console.error( + `Could not reach Redis at ${redisUrl}; start one first, e.g. docker run --rm -p 6379:6379 redis:6.2`, + ); + throw error; +} + +const nativeAdapter = createNodeRedisDialCacheClient(redis); +let adapterReadCalls = 0; +const adapter = { + ...nativeAdapter, + read(...args) { + adapterReadCalls += 1; + return nativeAdapter.read(...args); + }, +}; +const staleOutcomes = new Map(); +const compressionOutcomes = new Map(); +const noOpMetrics = { + request() {}, + miss() {}, + disabled() {}, + error() {}, + invalidation() {}, + coalesced() {}, + shadowValidation() {}, + staleRecovery({ outcome }) { + staleOutcomes.set(outcome, (staleOutcomes.get(outcome) ?? 0) + 1); + }, + compression({ outcome }) { + compressionOutcomes.set(outcome, (compressionOutcomes.get(outcome) ?? 0) + 1); + }, + observeGet() {}, + observeFallback() {}, + observeSerialization() {}, + observeSize() {}, + observeStoredSize() {}, + observeCompressionRatio() {}, + observeCompression() {}, +}; +const dialcache = new DialCache({ + namespace, + shouldAttemptStaleRecovery: () => true, + redis: { + client: adapter, + readTimeoutMs: redisReadTimeoutMs, + compression: { thresholdBytes: 1_024, level: 3 }, + }, + metrics: noOpMetrics, + logger: { debug() {}, warn() {}, error() {} }, +}); + +let freshSourceCalls = 0; +let freshWarmed = false; +const freshUseCase = "BenchmarkFreshHit"; +const loadFresh = dialcache.cached( + async () => { + freshSourceCalls += 1; + if (freshWarmed) { + throw new Error("fresh benchmark unexpectedly reached the source"); + } + return payload; + }, + { + keyType, + useCase: freshUseCase, + cacheKey: () => id, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: freshAgeSec }, + }), + }, +); + +let staleSourceCalls = 0; +let staleSourceMode = "warm"; +let staleSourceGate; +const staleUseCase = "BenchmarkStaleRecovery"; +const loadStale = dialcache.cached( + async () => { + staleSourceCalls += 1; + if (staleSourceMode === "warm") { + return payload; + } + if (staleSourceMode === "gated-rejection") { + await staleSourceGate.promise; + } + throw sourceError; + }, + { + keyType, + useCase: staleUseCase, + cacheKey: () => id, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: logicalFreshAgeSec }, + staleOnErrorMaxAgeSec: staleMaxAgeSec, + }), + }, +); + +let memorySourceCalls = 0; +let memorySourceMode = "warm"; +let memorySourceGate; +const memoryUseCase = "BenchmarkHighCardinalityMemory"; +const memoryIds = Array.from( + { length: memoryKeyCount }, + (_, index) => `memory-${index}`, +); +const loadMemory = dialcache.cached( + async () => { + memorySourceCalls += 1; + if (memorySourceMode === "warm") { + return memoryPayload; + } + await memorySourceGate.promise; + throw sourceError; + }, + { + keyType, + useCase: memoryUseCase, + cacheKey: (memoryId) => memoryId, + serializer: rawBufferSerializer, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: logicalFreshAgeSec }, + staleOnErrorMaxAgeSec: staleMaxAgeSec, + }), + }, +); + +const freshValueKey = redisValueKey(namespace, keyType, id, freshUseCase); +const staleValueKey = redisValueKey(namespace, keyType, id, staleUseCase); +const memoryValueKeys = memoryIds.map((memoryId) => + redisValueKey(namespace, keyType, memoryId, memoryUseCase) +); + +try { + assert.deepEqual(await dialcache.enable(async () => await loadFresh()), payload); + freshWarmed = true; + assert.deepEqual(await dialcache.enable(async () => await loadStale()), payload); + staleSourceMode = "rejection"; + + const storedBytes = await redis.strLen(staleValueKey); + const serializedBytes = Buffer.byteLength(JSON.stringify(payload)); + const physicalTtlMs = await redis.pTTL(staleValueKey); + assert(storedBytes > 10, "the stale benchmark frame must contain a payload"); + assert( + storedBytes < serializedBytes / 2, + "the representative payload should be materially compressed on Redis", + ); + assert( + physicalTtlMs <= staleMaxAgeSec * 1_000 + && physicalTtlMs >= staleMaxAgeSec * 1_000 - 5_000, + `stale-on-error writes must retain the frame near M; observed ${physicalTtlMs} ms`, + ); + + const rows = []; + rows.push(await measureScenario({ + name: "fresh end-to-end hit", + operations: iterations, + run: async () => { + for (let index = 0; index < iterations; index += 1) { + assert.deepEqual(await dialcache.enable(async () => await loadFresh()), payload); + } + }, + })); + assert.equal(freshSourceCalls, 1, "fresh hits must not reach the source after warmup"); + + // The writer and reader are this process, so waiting beyond F makes the + // retained frame a deterministic logical miss while Redis keeps it to M. + await wait(logicalFreshAgeSec * 1_000 + 100); + assert((await redis.pTTL(staleValueKey)) > 0, "the logically stale frame must remain retained"); + + rows.push(await measureRetainedScenario({ + name: "retained-frame native adapter read", + operations: iterations, + run: async () => { + for (let index = 0; index < iterations; index += 1) { + assert.notEqual( + await adapter.read({ valueKey: staleValueKey }), + null, + "the adapter must return the retained frame for core age filtering", + ); + } + }, + })); + + const recoveryOutcomesBefore = staleOutcomes.get("served") ?? 0; + const recoverySourceCallsBefore = staleSourceCalls; + const endToEndRecovery = await measureRetainedScenario({ + name: "end-to-end stale recovery", + operations: iterations, + run: async () => { + for (let index = 0; index < iterations; index += 1) { + assert.deepEqual(await dialcache.enable(async () => await loadStale()), payload); + } + }, + }); + rows.push(endToEndRecovery); + assert.equal( + endToEndRecovery.adapterReadCalls, + iterations, + "each stale-recovery flight must issue exactly one native Redis read", + ); + assert.equal(staleSourceCalls - recoverySourceCallsBefore, iterations); + assert.equal((staleOutcomes.get("served") ?? 0) - recoveryOutcomesBefore, iterations); + + staleSourceGate = deferred(); + staleSourceMode = "gated-rejection"; + const coalescedSourceCallsBefore = staleSourceCalls; + const coalescedOutcomesBefore = staleOutcomes.get("served") ?? 0; + const coalesced = await measureRetainedScenario({ + name: "coalesced stale recovery", + operations: fanout, + run: async () => { + const pending = Array.from( + { length: fanout }, + () => dialcache.enable(async () => await loadStale()), + ); + let sourceStartError; + try { + await waitFor( + () => staleSourceCalls > coalescedSourceCallsBefore, + sourceStartTimeoutMs, + ); + } catch (error) { + sourceStartError = error; + } finally { + // Do not strand the fanout if Redis or the benchmark assertion fails. + staleSourceGate.resolve(); + } + if (sourceStartError !== undefined) { + await Promise.allSettled(pending); + throw sourceStartError; + } + const values = await Promise.all(pending); + for (const value of values) { + assert.deepEqual(value, payload); + } + }, + }); + rows.push(coalesced); + assert.equal( + staleSourceCalls - coalescedSourceCallsBefore, + 1, + "same-key coalescing must share one source rejection", + ); + assert.equal( + (staleOutcomes.get("served") ?? 0) - coalescedOutcomesBefore, + 1, + "same-key coalescing must share one recovery decision", + ); + assert.equal( + coalesced.adapterReadCalls, + 1, + "same-key coalescing must share one native Redis read", + ); + + const memoryCompressionBefore = compressionOutcomes.get("compressed") ?? 0; + const memoryWarmSourceCallsBefore = memorySourceCalls; + const warmedMemoryValues = await Promise.all( + memoryIds.map((memoryId) => + dialcache.enable(async () => await loadMemory(memoryId)) + ), + ); + for (const value of warmedMemoryValues) { + assert.deepEqual(value, memoryPayload); + } + assert.equal( + memorySourceCalls - memoryWarmSourceCallsBefore, + memoryKeyCount, + "each high-cardinality key must be warmed from the source", + ); + assert.equal( + (compressionOutcomes.get("compressed") ?? 0) - memoryCompressionBefore, + 0, + "random memory-benchmark payloads must remain raw rather than compressing", + ); + const memoryStoredBytes = await redis.strLen(memoryValueKeys[0]); + assert.equal( + memoryStoredBytes, + memoryPayloadBytes + 10, + "the raw binary Redis frame must contain only its ten-byte protocol header beyond the payload", + ); + + await wait(logicalFreshAgeSec * 1_000 + 100); + assert( + (await redis.pTTL(memoryValueKeys[0])) > 0, + "the high-cardinality frames must remain retained after becoming logically stale", + ); + + memorySourceGate = deferred(); + memorySourceMode = "gated-rejection"; + const memoryRecoverySourceCallsBefore = memorySourceCalls; + const memoryOutcomesBefore = staleOutcomes.get("served") ?? 0; + let memoryBefore; + let memoryRetained; + await collectGarbageIfAvailable(); + memoryBefore = process.memoryUsage(); + const highCardinality = await measureRetainedScenario({ + name: "high-cardinality delayed stale recovery", + operations: memoryKeyCount, + retainedValueKey: memoryValueKeys[0], + run: async () => { + const pending = memoryIds.map((memoryId) => + dialcache.enable(async () => await loadMemory(memoryId)) + ); + let sourceStartError; + try { + await waitFor( + () => memorySourceCalls - memoryRecoverySourceCallsBefore >= memoryKeyCount, + Math.max(sourceStartTimeoutMs, 10_000), + ); + // Hold every distinct source call open so each flight retains its own + // raw candidate before the process-level memory snapshot. + await wait(memorySourceDelayMs); + await collectGarbageIfAvailable(); + memoryRetained = process.memoryUsage(); + } catch (error) { + sourceStartError = error; + } finally { + memorySourceGate.resolve(); + } + if (sourceStartError !== undefined) { + await Promise.allSettled(pending); + throw sourceStartError; + } + const values = await Promise.all(pending); + for (const value of values) { + assert.deepEqual(value, memoryPayload); + } + }, + }); + await collectGarbageIfAvailable(); + const memoryAfter = process.memoryUsage(); + rows.push(highCardinality); + assert.equal( + highCardinality.adapterReadCalls, + memoryKeyCount, + "each distinct high-cardinality flight must issue exactly one native Redis read", + ); + assert.equal( + memorySourceCalls - memoryRecoverySourceCallsBefore, + memoryKeyCount, + "each distinct high-cardinality flight must reach its own source call", + ); + assert.equal( + (staleOutcomes.get("served") ?? 0) - memoryOutcomesBefore, + memoryKeyCount, + "each distinct high-cardinality flight must serve its retained candidate", + ); + assert.notEqual( + memoryRetained, + undefined, + "the delayed-source memory snapshot must be captured", + ); + + const serverInfo = parseInfo(await redis.sendCommand(["INFO", "server"])); + console.log( + `Stale-on-error benchmark — ${redisUrl} (${serverInfo.redis_version ?? "unknown engine"})`, + ); + console.log( + `payload JSON=${serializedBytes.toLocaleString("en-US")} B, stored frame=${storedBytes.toLocaleString("en-US")} B, F=${logicalFreshAgeSec}s, M=${staleMaxAgeSec}s`, + ); + console.table(rows.map((row) => ({ + scenario: row.name, + operations: row.operations, + "elapsed (ms)": row.elapsedMs.toFixed(2), + "ops/sec": Math.round((row.operations / row.elapsedMs) * 1_000).toLocaleString("en-US"), + "adapter reads": row.adapterReadCalls, + "adapter reads/op": perOperation(row.adapterReadCalls, row.operations), + "GET/op": perOperation(row.getCalls, row.operations), + "server us/op": perOperation(row.serverUsec, row.operations), + "net in B/op": perOperation(row.netInputBytes, row.operations), + "net out B/op": perOperation(row.netOutputBytes, row.operations), + }))); + console.log( + `Node memory snapshots — ${memoryKeyCount.toLocaleString("en-US")} distinct flights held for ${memorySourceDelayMs.toLocaleString("en-US")} ms with ${memoryPayloadBytes.toLocaleString("en-US")} B incompressible raw payloads`, + ); + console.table([ + memorySnapshotRow("before flights", memoryBefore, memoryBefore), + memorySnapshotRow("all sources delayed", memoryRetained, memoryBefore), + memorySnapshotRow("after recovery", memoryAfter, memoryBefore), + ]); + console.log( + `Memory deltas are process-level observations that include Redis client buffers, promises, and source-call state; GC ${typeof globalThis.gc === "function" ? "was requested before the baseline" : "was not exposed"}. No memory or timing threshold is applied.`, + ); + console.log( + "Semantic assertions passed. INFO deltas are observational and include small snapshot-query overhead; no timing threshold is applied.", + ); +} finally { + await redis.del([freshValueKey, staleValueKey, ...memoryValueKeys]).catch(() => undefined); + await redis.quit().catch(() => redis.disconnect()); +} + +async function measureScenario({ name, operations, run }) { + const before = await redisSnapshot(redis); + const adapterReadsBefore = adapterReadCalls; + const start = performance.now(); + await run(); + const elapsedMs = performance.now() - start; + const scenarioAdapterReadCalls = adapterReadCalls - adapterReadsBefore; + const after = await redisSnapshot(redis); + const getCalls = commandDelta(before, after, "get", "calls"); + let serverUsec = 0; + for (const command of ["get", "mget", "set", "evalsha", "eval"]) { + serverUsec += commandDelta(before, after, command, "usec"); + } + return { + name, + operations, + elapsedMs, + adapterReadCalls: scenarioAdapterReadCalls, + getCalls, + serverUsec, + netInputBytes: after.netInputBytes - before.netInputBytes, + netOutputBytes: after.netOutputBytes - before.netOutputBytes, + }; +} + +async function measureRetainedScenario(options) { + try { + return await measureScenario(options); + } catch (error) { + const remainingTtlMs = await redis.pTTL( + options.retainedValueKey ?? staleValueKey, + ).catch(() => null); + if (remainingTtlMs !== null && remainingTtlMs <= 0) { + throw new Error( + `Stale-on-error benchmark exhausted its M=${staleMaxAgeSec}s retention window during "${options.name}"; reduce the configured iteration, fanout, or payload size`, + { cause: error }, + ); + } + throw error; + } +} + +async function collectGarbageIfAvailable() { + if (typeof globalThis.gc === "function") { + globalThis.gc(); + // Give native Redis and compression buffers one turn to release their + // backing stores before a second collection stabilizes the snapshot. + await wait(0); + globalThis.gc(); + } +} + +function memorySnapshotRow(name, snapshot, baseline) { + return { + snapshot: name, + "rss MiB": mebibytes(snapshot.rss), + "rss delta MiB": mebibytes(snapshot.rss - baseline.rss), + "heap MiB": mebibytes(snapshot.heapUsed), + "heap delta MiB": mebibytes(snapshot.heapUsed - baseline.heapUsed), + "external MiB": mebibytes(snapshot.external), + "external delta MiB": mebibytes(snapshot.external - baseline.external), + }; +} + +function mebibytes(bytes) { + return (bytes / (1024 * 1024)).toFixed(2); +} + +function commandDelta(before, after, command, field) { + return (after.commands[command]?.[field] ?? 0) - (before.commands[command]?.[field] ?? 0); +} + +async function redisSnapshot(client) { + const commandInfo = String(await client.sendCommand(["INFO", "commandstats"])); + const statsInfo = parseInfo(await client.sendCommand(["INFO", "stats"])); + const commands = {}; + for (const line of commandInfo.split("\n")) { + const match = /^cmdstat_([a-z0-9_-]+):calls=(\d+),usec=(\d+)/.exec(line.trim()); + if (match !== null) { + commands[match[1]] = { calls: Number(match[2]), usec: Number(match[3]) }; + } + } + return { + commands, + netInputBytes: Number(statsInfo.total_net_input_bytes ?? 0), + netOutputBytes: Number(statsInfo.total_net_output_bytes ?? 0), + }; +} + +function parseInfo(raw) { + const values = {}; + for (const line of String(raw).split("\n")) { + const separator = line.indexOf(":"); + if (separator > 0 && line[0] !== "#") { + values[line.slice(0, separator)] = line.slice(separator + 1).trim(); + } + } + return values; +} + +function redisValueKey(cacheNamespace, cacheKeyType, cacheId, useCase) { + const key = new DialCacheKey({ + namespace: cacheNamespace, + keyType: cacheKeyType, + id: cacheId, + useCase, + }); + return `${key.urn}:dialcache-frame-v1`; +} + +function deferred() { + let resolve; + const promise = new Promise((resolvePromise) => { + resolve = resolvePromise; + }); + return { promise, resolve }; +} + +async function waitFor(predicate, timeoutMs) { + const deadlineMs = performance.now() + timeoutMs; + while (performance.now() < deadlineMs) { + if (predicate()) { + return; + } + await wait(1); + } + throw new Error(`Timed out after ${timeoutMs} ms waiting for the benchmark source call`); +} + +function wait(milliseconds) { + return new Promise((resolve) => setTimeout(resolve, milliseconds)); +} + +function perOperation(value, operations) { + return (value / operations).toFixed(2); +} + +function readPositiveInteger(name, fallback) { + const raw = process.env[name]; + if (raw === undefined) { + return fallback; + } + const parsed = Number(raw); + if (!Number.isSafeInteger(parsed) || parsed <= 0) { + throw new RangeError(`${name} must be a positive safe integer`); + } + return parsed; +} diff --git a/scripts/test-package.mjs b/scripts/test-package.mjs index 2979455..bc57389 100644 --- a/scripts/test-package.mjs +++ b/scripts/test-package.mjs @@ -79,6 +79,44 @@ async function verifyPackedInvalidation({ createAdapter, label, redisProtocol }) } } `; +const packedStaleRecoveryCheckSource = String.raw` +async function verifyPackedStaleRecovery(root, label) { + let readCalls = 0; + const sourceError = new Error("packed source unavailable"); + const cache = new root.DialCache({ + shouldAttemptStaleRecovery: (error) => error === sourceError, + redis: { + client: { + read: async () => { + readCalls += 1; + return { + payload: JSON.stringify({ source: "cached" }), + createdAtMs: Date.now() - 2_000, + }; + }, + write: async () => undefined, + invalidate: async () => undefined, + }, + }, + }); + const load = cache.cached(async () => { + throw sourceError; + }, { + keyType: "id", + useCase: "PackedStaleRecovery", + cacheKey: () => "123", + defaultConfig: new root.DialCacheKeyConfig({ + ttlSec: { [root.CacheLayer.REMOTE]: 1 }, + ramp: { [root.CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 60, + }), + }); + const value = await cache.enable(() => load()); + if (readCalls !== 1 || value.source !== "cached") { + throw new Error("The packed " + label + " stale recovery did not retain one Redis snapshot"); + } +} +`; const rootConsumer = `import { CacheLayer, DialCache, @@ -117,6 +155,9 @@ const rootConsumer = `import { type ShadowConfig, type ShadowValidationMetricLabels, type ShadowValidationOutcome, + type StaleRecoveryMetricLabels, + type StaleRecoveryOutcome, + type StaleRecoveryPredicate, } from "dialcache"; // @ts-expect-error The unused MissingKeyConfigError class was removed instead of deprecated. import { MissingKeyConfigError } from "dialcache"; @@ -198,6 +239,19 @@ const shadowMetrics: DialCacheMetricsAdapter = { void outcome; }, }; +const staleMetrics: DialCacheMetricsAdapter = { + ...metrics, + staleRecovery: (labels: StaleRecoveryMetricLabels) => { + const outcome: StaleRecoveryOutcome = labels.outcome; + void outcome; + }, + observeStaleRecoveryValueAge: (labels: StaleRecoveryMetricLabels, seconds: number) => { + const outcome: StaleRecoveryOutcome = labels.outcome; + const ageSeconds: number = seconds; + void outcome; + void ageSeconds; + }, +}; const shadowOutcomes: Readonly> = { match: true, mismatch: true, @@ -213,6 +267,19 @@ const shadowOutcomes: Readonly> = { dropped: true, }; void shadowOutcomes; +const staleRecoveryOutcomes: Readonly> = { + served: true, + miss: true, + deserialization_error: true, +}; +const staleRecoveryLabels: StaleRecoveryMetricLabels = { + cacheNamespace: "consumer-cache", + useCase: "Load", + keyType: "id", + outcome: "served", +}; +void staleRecoveryOutcomes; +void staleRecoveryLabels; const metricLayers: Readonly> = { [CacheLayer.LOCAL]: true, [CacheLayer.REMOTE]: true, @@ -227,11 +294,27 @@ const shadowCacheConfig: DialCacheConfig = { shadowMaxInFlight: 2, }; const shadowCache = new DialCache(shadowCacheConfig); +const staleRecoveryPredicate: StaleRecoveryPredicate = (error) => error instanceof Error; +const invalidAsyncStaleRecoveryConfig: DialCacheConfig = { + // @ts-expect-error Stale-recovery predicates must return a boolean synchronously. + shouldAttemptStaleRecovery: async () => true, +}; +const staleCacheConfig: DialCacheConfig = { + namespace: "consumer-stale-cache", + metrics: staleMetrics, + shouldAttemptStaleRecovery: staleRecoveryPredicate, +}; +const staleCache = new DialCache(staleCacheConfig); const shadowConfig: ShadowConfig = { ramp: 50, logMismatches: true, }; const shadowKeyConfig = new DialCacheKeyConfig({ shadow: shadowConfig }); +const staleKeyConfig = new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 300, +}); +const staleRecoveryMaxAgeSec: number | undefined = staleKeyConfig.staleOnErrorMaxAgeSec; const dogStatsDClient: DatadogDogStatsDClient = { increment: () => undefined, histogram: () => undefined, @@ -273,6 +356,7 @@ const load = cache.cached(async (id: string) => id, { cacheKey: (id) => id, fallbackTimeoutMs: 1_000, shadowComparator: stringShadowComparator, + shouldAttemptStaleRecovery: staleRecoveryPredicate, defaultConfig: new DialCacheKeyConfig({ ttlSec: { [CacheLayer.LOCAL]: 60, [CacheLayer.REMOTE]: 60 }, ramp: { [CacheLayer.LOCAL]: 100, [CacheLayer.REMOTE]: 100 }, @@ -294,6 +378,7 @@ const inlineAsync: Promise<{ readonly id: string }> = cache.getOrLoad( { ...inlineOptionsFor("InlineAsync"), shadowComparator: (cachedValue, sourceValue) => cachedValue.id === sourceValue.id, + shouldAttemptStaleRecovery: staleRecoveryPredicate, }, ); @@ -580,6 +665,12 @@ void coalesceFlag; void structuralConfigProvider; void shadowCache; void shadowKeyConfig; +void staleMetrics; +void staleCache; +void staleCacheConfig; +void staleRecoveryPredicate; +void staleKeyConfig; +void staleRecoveryMaxAgeSec; void requestLocalCoalescingLabels; void cacheMetricLabels; void invalidationMetricLabels; @@ -669,6 +760,7 @@ void rootHasNoRandomRampSampler; void datadogMetrics; void datadogClassAdapter; void missingObservationType; +void invalidAsyncStaleRecoveryConfig; `; const integrationConsumer = `import * as valkeyGlide from "@valkey/valkey-glide"; import { DialCache } from "dialcache"; @@ -808,6 +900,7 @@ await import("dialcache/valkey-glide"); await import("dialcache/datadog"); const redisProtocol = await import("dialcache/redis-protocol"); ${packedInvalidationCheckSource} +${packedStaleRecoveryCheckSource} if (typeof nodeRedis.createNodeRedisDialCacheClient !== "function") { throw new Error("The packed ESM node-redis adapter export is missing"); } @@ -820,6 +913,7 @@ await verifyPackedInvalidation({ label: "ESM node-redis", redisProtocol, }); +await verifyPackedStaleRecovery(root, "ESM"); console.log("${nodeInvalidationMarker}"); const fallbackTimeoutError = new root.FallbackTimeoutError("PackageRuntime", 1000); if (!(fallbackTimeoutError instanceof root.DialCacheError) || fallbackTimeoutError.timeoutMs !== 1000) { @@ -947,6 +1041,7 @@ const esmDisabledOverlay = root.DialCacheKeyConfig.disabled(); if ( esmDisabledOverlay.requestLocal !== false || esmDisabledOverlay.coalesce !== undefined + || esmDisabledOverlay.staleOnErrorMaxAgeSec !== 0 || esmDisabledOverlay.shadow?.ramp !== 0 || esmDisabledOverlay.shadow.logMismatches !== false || esmDisabledOverlay.ramp[root.CacheLayer.LOCAL] !== 0 @@ -988,6 +1083,42 @@ const inlineSecond = await overlayCache.enable(() => ); if (inlineCalls !== 1 || inlineSecond !== inlineFirst) { throw new Error("The packed ESM runtime did not execute getOrLoad through the cache chain"); +} +let ancientTimestampReadCalls = 0; +let ancientTimestampFallbackCalls = 0; +const ancientTimestampCache = new root.DialCache({ + redis: { + client: { + read: async () => { + ancientTimestampReadCalls += 1; + return { + payload: JSON.stringify({ source: "cached" }), + createdAtMs: 1, + }; + }, + write: async () => undefined, + invalidate: async () => undefined, + }, + }, +}); +const ancientTimestampLoad = ancientTimestampCache.cached(async () => { + ancientTimestampFallbackCalls += 1; + return { source: "fallback" }; +}, { + keyType: "id", + useCase: "PackedAncientUntrackedTimestamp", + cacheKey: () => "123", + defaultConfig: new root.DialCacheKeyConfig({ + ttlSec: { [root.CacheLayer.REMOTE]: 60 }, + }), +}); +const ancientTimestampValue = await ancientTimestampCache.enable(() => ancientTimestampLoad()); +if ( + ancientTimestampReadCalls !== 1 + || ancientTimestampFallbackCalls !== 1 + || ancientTimestampValue.source !== "fallback" +) { + throw new Error("The packed ESM runtime did not reject an ancient untracked frame timestamp"); }`, ], { cwd: workspace }, @@ -1057,7 +1188,7 @@ console.log("${observerIsolationMarker}");`, let payload = Buffer.alloc(4 * 1024 * 1024, 1); const payloadReference = new WeakRef(payload); const redis = { - read: async () => ({ payload, createdAtMs: 1 }), + read: async () => ({ payload, createdAtMs: Date.now() }), write: async () => undefined, invalidate: async () => undefined, }; @@ -1144,6 +1275,7 @@ require("dialcache/valkey-glide"); require("dialcache/datadog"); const redisProtocol = require("dialcache/redis-protocol"); ${packedInvalidationCheckSource} +${packedStaleRecoveryCheckSource} if (typeof nodeRedis.createNodeRedisDialCacheClient !== "function") { throw new Error("The packed CommonJS node-redis adapter export is missing"); } @@ -1284,6 +1416,7 @@ const cjsDisabledOverlay = root.DialCacheKeyConfig.disabled(); if ( cjsDisabledOverlay.requestLocal !== false || cjsDisabledOverlay.coalesce !== undefined + || cjsDisabledOverlay.staleOnErrorMaxAgeSec !== 0 || cjsDisabledOverlay.shadow?.ramp !== 0 || cjsDisabledOverlay.shadow.logMismatches !== false || cjsDisabledOverlay.ramp[root.CacheLayer.LOCAL] !== 0 @@ -1293,6 +1426,7 @@ if ( } void (async () => { await cjsNodeInvalidationCheck; + await verifyPackedStaleRecovery(root, "CommonJS"); let calls = 0; const overlayCache = new root.DialCache({ cacheConfigProvider: () => new root.DialCacheKeyConfig({ diff --git a/src/config.ts b/src/config.ts index 5b89c81..0a2d264 100644 --- a/src/config.ts +++ b/src/config.ts @@ -10,6 +10,8 @@ export enum CacheLayer { export type Awaitable = T | Promise; export type LayerConfig = Partial>; +/** Synchronously classifies whether one source rejection may use a retained Redis candidate. */ +export type StaleRecoveryPredicate = (error: unknown) => boolean; /** Per-use-case runtime policy for detached Redis shadow work. */ export interface ShadowConfig { @@ -40,6 +42,17 @@ export class DialCacheKeyConfig { * independent fallback deadline, and its own cache writes. */ readonly coalesce?: boolean; + /** + * Exclusive logical Redis-frame age ceiling in seconds. A value retained by + * the initial read may be returned after an eligible source rejection only + * while its age is less than this value; an age exactly at the ceiling is a + * miss. Omission disables recovery by default and inherits in runtime + * overlays; zero explicitly disables an inherited policy. A positive value + * requires a smaller positive remote TTL and may not exceed 31,536,000 + * seconds (365 days). Tracked values retain their separate one-hour physical + * TTL cap. + */ + readonly staleOnErrorMaxAgeSec?: number; /** * Maximum time DialCache waits for a remote read before failing open to the * source of truth. Overrides the instance default for this use case. @@ -52,6 +65,7 @@ export class DialCacheKeyConfig { shadow?: ShadowConfig; requestLocal?: boolean; coalesce?: boolean; + staleOnErrorMaxAgeSec?: number; remoteReadTimeoutMs?: number; }) { if (config === null || typeof config !== "object" || Array.isArray(config)) { @@ -78,6 +92,11 @@ export class DialCacheKeyConfig { if (config.coalesce !== undefined) { this.coalesce = config.coalesce; } + // Like ttlSec/ramp leaves, validation is deferred to static-default capture + // or runtime resolution so malformed runtime policy can fail open narrowly. + if (config.staleOnErrorMaxAgeSec !== undefined) { + this.staleOnErrorMaxAgeSec = config.staleOnErrorMaxAgeSec; + } if (config.remoteReadTimeoutMs !== undefined) { assertValidDeadlineMs(config.remoteReadTimeoutMs, "DialCache remoteReadTimeoutMs"); this.remoteReadTimeoutMs = config.remoteReadTimeoutMs; @@ -98,15 +117,16 @@ export class DialCacheKeyConfig { } /** - * The explicit cache-invocation kill switch: request-local caching and - * shadow work off, with both shared layers ramped to 0. As a provider - * overlay it disables every inherited path instead of relying on field - * omission. It does not cancel admitted work or disable explicit + * The explicit cache-invocation kill switch: request-local caching, stale + * recovery, and shadow work off, with both shared layers ramped to 0. As a + * provider overlay it disables every inherited path instead of relying on + * field omission. It does not cancel admitted work or disable explicit * maintenance operations. */ static disabled(): DialCacheKeyConfig { return new DialCacheKeyConfig({ requestLocal: false, + staleOnErrorMaxAgeSec: 0, shadow: { ramp: 0, logMismatches: false, @@ -149,6 +169,18 @@ export type Logger = Pick; export interface DialCacheConfig { readonly cacheConfigProvider?: CacheConfigProvider; + /** + * Instance default for deciding whether a source rejection may use a + * retained Redis value. Must be synchronous. Per-use-case policy overrides + * and replaces this callback; omission admits only DialCache's + * FallbackTimeoutError. Throws, thenables, and non-boolean results deny + * recovery without replacing the source rejection. Custom predicates should + * narrowly admit transient, retriable infrastructure failures and deny + * authoritative outcomes such as auth, permission, entitlement, revocation, + * deletion, not-found, validation, and programmer errors. Use per-use-case + * overrides when particular data requires a stricter policy. + */ + readonly shouldAttemptStaleRecovery?: StaleRecoveryPredicate; /** * Logical namespace used in cache keys, invalidation identity, ramp sampling, * and metrics. Defaults to "urn". May not contain `{` or `}`. diff --git a/src/datadog.ts b/src/datadog.ts index b8182e7..471a780 100644 --- a/src/datadog.ts +++ b/src/datadog.ts @@ -9,6 +9,7 @@ import type { InvalidationMetricLabels, SerializationMetricLabels, ShadowValidationMetricLabels, + StaleRecoveryMetricLabels, } from "./metrics.js"; export type DatadogObservationMetricType = "histogram" | "distribution"; @@ -34,6 +35,7 @@ export interface DatadogMetricsOptions { type CacheTag = "cache_namespace" | "use_case" | "key_type" | "layer"; type DatadogTags = Record; +type OutcomeMetricLabels = ShadowValidationMetricLabels | StaleRecoveryMetricLabels; type Observation = (name: string, value: number, tags: DatadogTags) => void; const DEFAULT_NAMESPACE = "dialcache"; @@ -50,6 +52,8 @@ const METRIC_SUFFIXES = { shadowValidation: "shadow.count", shadowValueAge: "shadow.value_age", futureTimestampOffset: "future_timestamp_offset", + staleRecovery: "stale_recovery.count", + staleRecoveryValueAge: "stale_recovery.value_age", compression: "compression.count", get: "get.duration", fallback: "fallback.duration", @@ -122,17 +126,25 @@ export class DatadogDialCacheMetrics implements DialCacheMetricsAdapter { } shadowValidation(labels: ShadowValidationMetricLabels): void { - return this.increment(this.metricNames.shadowValidation, shadowValidationTags(labels)); + return this.increment(this.metricNames.shadowValidation, outcomeTags(labels)); } observeShadowValueAge(labels: ShadowValidationMetricLabels, seconds: number): void { - this.observe(this.metricNames.shadowValueAge, seconds, shadowValidationTags(labels)); + this.observe(this.metricNames.shadowValueAge, seconds, outcomeTags(labels)); } observeFutureTimestampOffset(labels: CacheMetricLabels, seconds: number): void { this.observe(this.metricNames.futureTimestampOffset, seconds, cacheTags(labels)); } + staleRecovery(labels: StaleRecoveryMetricLabels): void { + return this.increment(this.metricNames.staleRecovery, outcomeTags(labels)); + } + + observeStaleRecoveryValueAge(labels: StaleRecoveryMetricLabels, seconds: number): void { + this.observe(this.metricNames.staleRecoveryValueAge, seconds, outcomeTags(labels)); + } + compression(labels: CompressionMetricLabels): void { this.increment(this.metricNames.compression, { ...cacheTags(labels), outcome: labels.outcome }); } @@ -189,7 +201,7 @@ function cacheTags(labels: CacheMetricLabels): Record { }; } -function shadowValidationTags(labels: ShadowValidationMetricLabels): DatadogTags { +function outcomeTags(labels: OutcomeMetricLabels): DatadogTags { return { cache_namespace: labels.cacheNamespace, use_case: labels.useCase, diff --git a/src/dialcache.ts b/src/dialcache.ts index 9afbf11..f2abe94 100644 --- a/src/dialcache.ts +++ b/src/dialcache.ts @@ -8,6 +8,7 @@ import { type CacheConfigProvider, type DialCacheConfig, type Logger, + type StaleRecoveryPredicate, } from "./config.js"; import { DialCacheContext, getOrCreateRequestLocalCache, type RequestLocalCache } from "./context.js"; import { FallbackTimeoutError, UseCaseIsAlreadyRegisteredError, UseCaseNameIsReservedError } from "./errors.js"; @@ -37,9 +38,10 @@ import { deterministicShadowRampSample } from "./internal/ramp.js"; import { RedisCache, type FutureFramePolicy } from "./internal/redis-cache.js"; import { fetchKeyConfig, - resolveLayerConfigResult, + resolveRemoteLayerConfigResult, type LayerConfigResolution, type ResolvedLayerConfig, + type ResolvedRemoteLayerConfig, } from "./internal/runtime-config.js"; import { shadowMismatchLogDetails } from "./internal/shadow-log-json.js"; @@ -126,6 +128,18 @@ interface CacheOperationOptionsBase { * This is stable use-case behavior, not runtime rollout configuration. */ readonly shadowComparator?: ShadowComparator; + /** + * Overrides the DialCache-instance source-error classifier for this use + * case, replacing both the instance and built-in policies. Must be + * synchronous. Returning true authorizes recovery from the Redis candidate + * retained by the initial read; every other result fails closed without + * replacing the source rejection. Custom predicates should narrowly admit + * transient, retriable infrastructure failures and deny authoritative + * outcomes such as auth, permission, entitlement, revocation, deletion, + * not-found, validation, and programmer errors. Use this override for data + * that requires a stricter policy than the instance default. + */ + readonly shouldAttemptStaleRecovery?: StaleRecoveryPredicate; /** * Monotonic deadline applied once an initially enabled invocation starts its * fallback, in milliseconds. Must be at most 2,147,483,647. Defaults to 60 @@ -235,7 +249,7 @@ type ShadowValidationStart = /** The caller-owned, fallback-deadline-bounded SoT operation. */ readonly source: Promise; /** Valid remote policy retained even though its serving ramp excluded this key. */ - readonly remoteConfig: ResolvedLayerConfig; + readonly remoteConfig: ResolvedRemoteLayerConfig; /** Includes synchronous SoT work that ran before shadow admission. */ readonly startedAtMs: number | null; }; @@ -245,7 +259,7 @@ type ShadowValidationRunStart = | { readonly kind: "redis"; readonly source: Promise; - readonly remoteConfig: ResolvedLayerConfig; + readonly remoteConfig: ResolvedRemoteLayerConfig; }; const DEFAULT_LOCAL_MAX_SIZE = 10_000; @@ -253,6 +267,8 @@ const DEFAULT_FALLBACK_TIMEOUT_MS = 60_000; const DEFAULT_SHADOW_MAX_IN_FLIGHT = 1; const defaultConfigProvider: CacheConfigProvider = () => null; const defaultLogger: Logger = console; +const defaultStaleRecoveryPredicate: StaleRecoveryPredicate = + (error) => error instanceof FallbackTimeoutError; export class DialCache { private readonly context = new DialCacheContext(); @@ -263,6 +279,7 @@ export class DialCache { private readonly logger: Logger; private readonly redisCache: RedisCache | null; private readonly metrics: DialCacheMetricsAdapter | null; + private readonly staleRecoveryPredicate: StaleRecoveryPredicate; private readonly shadowMaxInFlight: number; private readonly shadowFlights = new Map(); private readonly processFlights = new Map(); @@ -297,13 +314,16 @@ export class DialCache { this.namespace = namespace; this.logger = safeLogger(config.logger ?? defaultLogger); this.metrics = safeMetrics(config.metrics ?? null); + this.staleRecoveryPredicate = resolveStaleRecoveryPredicate( + config.shouldAttemptStaleRecovery, + defaultStaleRecoveryPredicate, + ); this.shadowMaxInFlight = shadowMaxInFlight; - this.localCache = new LocalCache(this.configProvider, localMaxSize); + this.localCache = new LocalCache(localMaxSize); this.redisCache = config.redis === undefined ? null : new RedisCache({ - configProvider: this.configProvider, redis: config.redis, metrics: this.metrics, }); @@ -350,6 +370,10 @@ export class DialCache { const defaultConfig = snapshotDefaultConfig(options.defaultConfig); const fallbackTimeoutMs = resolveFallbackTimeoutMs(options.fallbackTimeoutMs); const shadowComparator = resolveShadowComparator(options.shadowComparator); + const staleRecoveryPredicate = resolveStaleRecoveryPredicate( + options.shouldAttemptStaleRecovery, + this.staleRecoveryPredicate, + ); this.registerUseCase(options.useCase); return (...args: Parameters): Promise> => @@ -362,6 +386,7 @@ export class DialCache { defaultConfig, fallbackTimeoutMs, shadowComparator, + staleRecoveryPredicate, ); } @@ -375,6 +400,10 @@ export class DialCache { const defaultConfig = snapshotDefaultConfig(options.defaultConfig); const fallbackTimeoutMs = resolveFallbackTimeoutMs(options.fallbackTimeoutMs); const shadowComparator = resolveShadowComparator(options.shadowComparator); + const staleRecoveryPredicate = resolveStaleRecoveryPredicate( + options.shouldAttemptStaleRecovery, + this.staleRecoveryPredicate, + ); this.assertUseCaseIsNotReserved(options.useCase); return this.executeCacheOperation( @@ -384,6 +413,7 @@ export class DialCache { defaultConfig, fallbackTimeoutMs, shadowComparator, + staleRecoveryPredicate, ); } @@ -394,6 +424,7 @@ export class DialCache { defaultConfig: DialCacheKeyConfig | null, fallbackTimeoutMs: number | null, shadowComparator: ShadowComparator, + staleRecoveryPredicate: StaleRecoveryPredicate, ): Promise { const rawFallback = async (): Promise => await load(); const noLayerLabels = { @@ -465,6 +496,7 @@ export class DialCache { keyConfig, fallback, shadowValidation, + staleRecoveryPredicate, ); } } @@ -474,6 +506,7 @@ export class DialCache { keyConfig, fallback, shadowValidation, + staleRecoveryPredicate, CacheLayer.LOCAL, ); } @@ -557,6 +590,7 @@ export class DialCache { keyConfig: DialCacheKeyConfig, fallback: () => Promise, shadowValidation: ShadowValidationPlan, + staleRecoveryPredicate: StaleRecoveryPredicate, ): Promise { const run = async (): Promise => { const start = performance.now(); @@ -573,6 +607,7 @@ export class DialCache { keyConfig, fallback, shadowValidation, + staleRecoveryPredicate, REQUEST_LOCAL_CACHE_LAYER, ); requestLocalCache.set(key.urn, value); @@ -589,6 +624,7 @@ export class DialCache { keyConfig: DialCacheKeyConfig | null, fallback: () => Promise, shadowValidation: ShadowValidationPlan, + staleRecoveryPredicate: StaleRecoveryPredicate, fallbackMetricLayer: MetricLayer, ): Promise { // This predicate is the single home of the default: omission means on in @@ -603,6 +639,7 @@ export class DialCache { localLayer.config, fallback, shadowValidation, + staleRecoveryPredicate, ); return coalesce ? await this.singleFlightProcess(key, run) : await run(); } @@ -645,7 +682,7 @@ export class DialCache { remote, fallback, shadowValidation, - remoteLayer.config, + staleRecoveryPredicate, ); }; return coalesce ? await this.singleFlightProcess(key, run) : await run(); @@ -657,6 +694,7 @@ export class DialCache { localConfig: ResolvedLayerConfig, fallback: () => Promise, shadowValidation: ShadowValidationPlan, + staleRecoveryPredicate: StaleRecoveryPredicate, ): Promise { const local = this.readLocalWithResolvedConfig(key, localConfig); if (local.status === "hit") { @@ -690,6 +728,7 @@ export class DialCache { remoteLayer, fallback, shadowValidation, + staleRecoveryPredicate, ); } @@ -707,7 +746,7 @@ export class DialCache { remote, fallback, shadowValidation, - remoteLayer.config, + staleRecoveryPredicate, ); } @@ -724,7 +763,7 @@ export class DialCache { key: DialCacheKey, keyConfig: DialCacheKeyConfig | null, local: CacheGetResult | null, - remoteConfig: ResolvedLayerConfig, + remoteConfig: ResolvedRemoteLayerConfig, fallbackLabels: CacheMetricLabels, fallback: () => Promise, shadowValidation: ShadowValidationPlan, @@ -757,7 +796,7 @@ export class DialCache { remote: RemoteCacheGetResult, fallback: () => Promise, shadowValidation: ShadowValidationPlan, - resolvedRemoteConfig?: ResolvedLayerConfig, + staleRecoveryPredicate: StaleRecoveryPredicate, ): Promise { if (remote.status === "hit") { if (local.status === "miss") { @@ -782,17 +821,50 @@ export class DialCache { return value; } - const remoteErrored = remote.status === "disabled" && remote.reason === "config_error"; - const remoteWriteConfig = remote.status === "miss" ? remote.config : remoteErrored ? resolvedRemoteConfig : undefined; - const fallbackLayer = remote.status === "miss" || remoteErrored ? CacheLayer.REMOTE : CacheLayer.LOCAL; - const value = await this.callFallback(labelsFor(key, fallbackLayer), fallback); - const skipCacheWrite = (remote.status === "miss" || remote.status === "disabled") && remote.skipCacheWrite === true; + const remoteConfigErrored = remote.status === "disabled" && remote.reason === "config_error"; + const remoteWriteConfig = remote.status === "miss" || remote.status === "retained" + ? remote.config + : undefined; + const fallbackLayer = remote.status === "miss" || remote.status === "retained" || remoteConfigErrored + ? CacheLayer.REMOTE + : CacheLayer.LOCAL; + const staleRecoveryMaxAgeSec = remote.status === "retained" + || (remote.status === "miss" && remote.reason === "cache_miss") + ? remote.config.staleOnErrorMaxAgeSec + : null; + const retainedFrame = remote.status === "retained" ? remote.frame : null; + let value: T; + try { + value = await this.callFallback(labelsFor(key, fallbackLayer), fallback); + } catch (fallbackError) { + if ( + staleRecoveryMaxAgeSec !== null + && this.shouldAttemptStaleRecovery(staleRecoveryPredicate, fallbackError) + ) { + try { + const recovered = await redisCache.recoverRetainedCandidate( + key, + retainedFrame, + staleRecoveryMaxAgeSec, + ); + if (recovered.status === "hit") { + return recovered.value; + } + } catch (recoveryError) { + // Recovery is subordinate to the source rejection and must never + // replace it, including for a custom serializer that violates its + // declared contract in an unexpected way. + this.logger.warn("Error using retained Redis value during stale recovery", recoveryError); + } + } + throw fallbackError; + } // A tracked fallback was not validated against a watermark after the source // call. If this invocation reached the Redis write path, let a later // authoritative Redis hit populate local regardless of write success. - const suppressLocalWrite = skipCacheWrite + const suppressLocalWrite = (remote.status === "disabled" && remote.skipCacheWrite === true) || (remoteWriteConfig !== undefined && key.trackForInvalidation); - if (!skipCacheWrite && remoteWriteConfig !== undefined) { + if (remoteWriteConfig !== undefined) { try { await redisCache.put(key, value, remoteWriteConfig); } catch (error) { @@ -805,6 +877,28 @@ export class DialCache { return value; } + private shouldAttemptStaleRecovery( + predicate: StaleRecoveryPredicate, + fallbackError: unknown, + ): boolean { + let result: unknown; + try { + result = predicate(fallbackError); + } catch (predicateError) { + this.logger.warn("DialCache stale recovery predicate threw; recovery was denied", predicateError); + return false; + } + if (typeof result !== "boolean") { + // The public contract is synchronous. Consume an accidental rejecting + // thenable without awaiting it, and fail closed without delaying the + // original source rejection. + void settleUnexpectedThenable(result); + this.logger.warn("DialCache stale recovery predicate returned a non-boolean; recovery was denied"); + return false; + } + return result; + } + private scheduleShadowValidation( redisCache: RedisCache, key: DialCacheKey, @@ -947,8 +1041,16 @@ export class DialCache { operationFinished = true; maybeRelease(); }; - const readShadowFrame = (futureFramePolicy: FutureFramePolicy): Promise => { - const read = redisCache.startPayloadReadForShadow(key, readTimeoutMs, futureFramePolicy); + const readShadowFrame = ( + maxAgeSec: number | null, + futureFramePolicy: FutureFramePolicy, + ): Promise => { + const read = redisCache.startPayloadReadForShadow( + key, + maxAgeSec, + readTimeoutMs, + futureFramePolicy, + ); pendingRedisReads.add(read.settled); void read.settled.then(() => { pendingRedisReads.delete(read.settled); @@ -986,11 +1088,11 @@ export class DialCache { return "timeout"; } - let shadowFillConfig: ResolvedLayerConfig | null = null; + let shadowFillConfig: ResolvedRemoteLayerConfig | null = null; if (start.kind === "redis") { let frame: DecodedRedisFrame | null; try { - frame = await readShadowFrame("reject"); + frame = await readShadowFrame(start.remoteConfig.ttlSec, "reject"); } catch { return "redis_error"; } @@ -1089,7 +1191,7 @@ export class DialCache { let confirmationFrame: DecodedRedisFrame | null; try { - confirmationFrame = await readShadowFrame("retain"); + confirmationFrame = await readShadowFrame(null, "retain"); } catch { return "confirmation_error"; } @@ -1208,11 +1310,13 @@ export class DialCache { private async resolveRemoteLayerConfig(key: DialCacheKey, keyConfig: DialCacheKeyConfig | null) { try { - const result = resolveLayerConfigResult({ + const result = resolveRemoteLayerConfigResult({ config: keyConfig, key, - layer: CacheLayer.REMOTE, }); + if (result.staleOnErrorConfigError === true) { + this.recordError(key, CacheLayer.REMOTE, "config_resolution"); + } if (result.status === "disabled") { this.metrics?.disabled({ ...labelsFor(key, CacheLayer.REMOTE), reason: result.reason }); this.recordInvalidLeaf(key, CacheLayer.REMOTE, result.reason); @@ -1229,20 +1333,20 @@ export class DialCache { private async readRemoteWithResolvedConfig( redisCache: RedisCache, key: DialCacheKey, - layerConfig: ResolvedLayerConfig, + layerConfig: ResolvedRemoteLayerConfig, readTimeoutMs: number, ): Promise> { try { return await redisCache.getWithResolvedConfig(key, layerConfig, readTimeoutMs); } catch (error) { this.logger.warn("Error getting value from Redis cache", error); - return { status: "error", operation: "read" }; + return { status: "error" }; } } - private async putLocalFailOpen(key: DialCacheKey, value: T, config?: { readonly ttlSec: number }): Promise { + private async putLocalFailOpen(key: DialCacheKey, value: T, config: { readonly ttlSec: number }): Promise { try { - await this.localCache.put(key, value, config); + this.localCache.put(key, value, config); } catch (error) { this.logger.warn("Error putting value in local cache", error); this.recordError(key, CacheLayer.LOCAL, "cache_write"); @@ -1405,6 +1509,7 @@ function snapshotDefaultConfig(config: DialCacheKeyConfig | null | undefined): D const shadowConfig = config.shadow; const requestLocal = config.requestLocal; const coalesce = config.coalesce; + const staleOnErrorMaxAgeSec = config.staleOnErrorMaxAgeSec; const remoteReadTimeoutMs = config.remoteReadTimeoutMs; if (requestLocal !== undefined && typeof requestLocal !== "boolean") { throw new TypeError("DialCache defaultConfig requestLocal must be a boolean"); @@ -1422,6 +1527,7 @@ function snapshotDefaultConfig(config: DialCacheKeyConfig | null | undefined): D ramp: rampConfig, ...(requestLocal === undefined ? {} : { requestLocal }), ...(coalesce === undefined ? {} : { coalesce }), + ...(staleOnErrorMaxAgeSec === undefined ? {} : { staleOnErrorMaxAgeSec }), ...(remoteReadTimeoutMs === undefined ? {} : { remoteReadTimeoutMs }), ...(shadowConfig === undefined ? {} : { shadow: shadowConfig }), }); @@ -1450,6 +1556,31 @@ function snapshotDefaultConfig(config: DialCacheKeyConfig | null | undefined): D } } + if (snapshot.staleOnErrorMaxAgeSec !== undefined) { + const maxAgeSec = snapshot.staleOnErrorMaxAgeSec; + if (typeof maxAgeSec !== "number") { + throw new TypeError("DialCache defaultConfig staleOnErrorMaxAgeSec must be a number"); + } + if (maxAgeSec !== 0 && !isSupportedCacheTtlSec(maxAgeSec)) { + throw new RangeError( + `DialCache defaultConfig staleOnErrorMaxAgeSec must be a nonnegative safe integer no greater than ${MAX_CACHE_TTL_SEC}`, + ); + } + if (maxAgeSec > 0) { + const remoteTtlSec = snapshot.ttlSec[CacheLayer.REMOTE]; + if (remoteTtlSec === undefined) { + throw new RangeError( + "DialCache defaultConfig staleOnErrorMaxAgeSec requires ttlSec.remote", + ); + } + if (maxAgeSec <= remoteTtlSec) { + throw new RangeError( + "DialCache defaultConfig staleOnErrorMaxAgeSec must be greater than ttlSec.remote", + ); + } + } + } + if (snapshot.shadow !== undefined) { if (snapshot.shadow.ramp !== undefined) { if (typeof snapshot.shadow.ramp !== "number") { @@ -1548,6 +1679,9 @@ function safeMetrics(metrics: DialCacheMetricsAdapter | null): DialCacheMetricsA callObserver(() => metrics.shadowValidation!(labels)), } : {}), + staleRecovery: (labels) => callObserver(() => metrics.staleRecovery?.(labels)), + observeStaleRecoveryValueAge: (labels, seconds) => + callObserver(() => metrics.observeStaleRecoveryValueAge?.(labels, seconds)), observeShadowValueAge: (labels, seconds) => callObserver(() => metrics.observeShadowValueAge?.(labels, seconds)), observeFutureTimestampOffset: (labels, seconds) => @@ -1581,12 +1715,25 @@ function resolveShadowComparator( return comparator ?? isDeepStrictEqual; } +function resolveStaleRecoveryPredicate( + predicate: StaleRecoveryPredicate | undefined, + fallback: StaleRecoveryPredicate, +): StaleRecoveryPredicate { + if (predicate === undefined) { + return fallback; + } + if (typeof predicate !== "function") { + throw new TypeError("DialCache shouldAttemptStaleRecovery must be a function"); + } + return predicate; +} + // Frame stamps and the observation both use application-process epoch clocks. -// Core rejects tracked frames that are future-dated when read, but the reader -// clock can step backward before a detached shadow verdict, so clamp that age -// to zero. A custom client that violates the decode contract can hand over a -// non-finite stamp; recording it would permanently poison backend histogram -// sums, so the observation is skipped instead. +// Core rejects future frames before they can serve, but a confirmation read +// may retain one solely for payload supersession comparison. Clamp that +// diagnostic age to zero. A custom client that violates the decode contract +// can hand over a non-finite stamp; recording it would permanently poison +// backend histogram sums, so the observation is skipped instead. function shadowValueAgeSeconds(createdAtMs: number): number | undefined { const ageSeconds = (Date.now() - createdAtMs) / 1000; if (!Number.isFinite(ageSeconds)) { @@ -1615,6 +1762,7 @@ async function settleUnexpectedThenable(value: unknown): Promise { try { await Promise.resolve(value); } catch { - // Comparators are synchronous; consume accidental async rejection safely. + // Synchronous extension points may accidentally return a rejected thenable; + // consume it without letting that rejection affect cache control flow. } } diff --git a/src/index.ts b/src/index.ts index 6fa2b67..7a8d269 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,5 +1,12 @@ export { CacheLayer, DialCacheKeyConfig } from "./config.js"; -export type { CacheConfigProvider, DialCacheConfig, LayerConfig, Logger, ShadowConfig } from "./config.js"; +export type { + CacheConfigProvider, + DialCacheConfig, + LayerConfig, + Logger, + ShadowConfig, + StaleRecoveryPredicate, +} from "./config.js"; export { DialCacheContext } from "./context.js"; export type { CacheMetricLabels, @@ -18,6 +25,8 @@ export type { SerializationMetricLabels, ShadowValidationMetricLabels, ShadowValidationOutcome, + StaleRecoveryMetricLabels, + StaleRecoveryOutcome, } from "./metrics.js"; export { DialCacheError, diff --git a/src/internal/cache-result.ts b/src/internal/cache-result.ts index 9abae50..df53ca7 100644 --- a/src/internal/cache-result.ts +++ b/src/internal/cache-result.ts @@ -1,20 +1,33 @@ -import type { ResolvedLayerConfig } from "./runtime-config.js"; +import type { ResolvedLayerConfig, ResolvedRemoteLayerConfig } from "./runtime-config.js"; import type { DisabledReason } from "../metrics.js"; import type { DecodedRedisFrame } from "../redis-client.js"; export type CacheGetResult = | { readonly status: "hit"; readonly value: T } - | { readonly status: "miss"; readonly config: ResolvedLayerConfig; readonly skipCacheWrite?: boolean } + | { readonly status: "miss"; readonly config: ResolvedLayerConfig } | { readonly status: "disabled"; readonly reason: DisabledReason; readonly skipCacheWrite?: boolean; }; +export type RedisCacheMissReason = "cache_miss" | "deserialization_error"; + export type RedisCacheGetResult = | { readonly status: "hit"; readonly value: T; readonly frame: DecodedRedisFrame } - | Exclude, { readonly status: "hit" }>; + | { + /** A valid F..M frame retained only as a possible source-error fallback. */ + readonly status: "retained"; + readonly frame: DecodedRedisFrame; + readonly config: ResolvedRemoteLayerConfig; + } + | { + readonly status: "miss"; + readonly config: ResolvedRemoteLayerConfig; + readonly reason: RedisCacheMissReason; + }; export type RemoteCacheGetResult = | RedisCacheGetResult - | { readonly status: "error"; readonly operation: "read" }; + | Extract, { readonly status: "disabled" }> + | { readonly status: "error" }; diff --git a/src/internal/local-cache.ts b/src/internal/local-cache.ts index dd12443..9020d1b 100644 --- a/src/internal/local-cache.ts +++ b/src/internal/local-cache.ts @@ -2,19 +2,16 @@ import { performance } from "node:perf_hooks"; import { LRUCache } from "lru-cache"; -import { CacheLayer, type CacheConfigProvider, type DialCacheKeyConfig } from "../config.js"; +import { CacheLayer, type DialCacheKeyConfig } from "../config.js"; import type { DialCacheKey } from "../key.js"; import type { CacheGetResult } from "./cache-result.js"; import { cacheTtlSecToMs } from "./duration.js"; import { - fetchKeyConfig, resolveLayerConfigResult, type LayerConfigResolution, type ResolvedLayerConfig, } from "./runtime-config.js"; -export type Fallback = () => Promise; - interface LocalEntry { readonly value: T; } @@ -22,10 +19,7 @@ interface LocalEntry { export class LocalCache { private readonly cache: LRUCache> | null; - constructor( - private readonly configProvider: CacheConfigProvider, - maxSize: number, - ) { + constructor(maxSize: number) { this.cache = maxSize === 0 ? null @@ -40,33 +34,6 @@ export class LocalCache { }); } - async get(key: DialCacheKey, fallback: Fallback): Promise { - const result = await this.getIfPresentResult(key); - if (result.status === "hit") { - return result.value; - } - - const value = await fallback(); - if (result.status === "miss") { - await this.put(key, value, result.config); - } - return value; - } - - async getIfPresent(key: DialCacheKey): Promise { - const result = await this.getIfPresentResult(key); - return result.status === "hit" ? result.value : undefined; - } - - async getIfPresentResult(key: DialCacheKey, keyConfig?: DialCacheKeyConfig | null): Promise> { - const layerConfig = await this.resolveLayerConfig(key, keyConfig); - if (layerConfig.status === "disabled") { - return layerConfig; - } - - return this.getWithResolvedConfig(key, layerConfig.config); - } - getWithResolvedConfig(key: DialCacheKey, layerConfig: ResolvedLayerConfig): CacheGetResult { const hit = this.cache?.get(key.urn) as LocalEntry | undefined; @@ -77,33 +44,21 @@ export class LocalCache { return { status: "miss", config: layerConfig }; } - async resolveLayerConfig( + resolveLayerConfig( key: DialCacheKey, - keyConfig?: DialCacheKeyConfig | null, - ): Promise { - // Chain callers pass the once-resolved config; standalone callers omit it and we fetch. - const config = keyConfig === undefined ? await fetchKeyConfig(this.configProvider, key) : keyConfig; + keyConfig: DialCacheKeyConfig | null, + ): LayerConfigResolution { return resolveLayerConfigResult({ - config, + config: keyConfig, key, layer: CacheLayer.LOCAL, }); } - async put(key: DialCacheKey, value: T, config?: { readonly ttlSec: number }): Promise { - const ttlSec = config?.ttlSec ?? await this.resolveLocalTtlSec(key); - if (ttlSec === null) { - return; - } - + put(key: DialCacheKey, value: T, config: { readonly ttlSec: number }): void { // lru-cache expires when age > ttl, while DialCache historically expired // when its integer-millisecond clock reached the configured boundary. - const ttlMs = cacheTtlSecToMs(ttlSec) - 1; + const ttlMs = cacheTtlSecToMs(config.ttlSec) - 1; this.cache?.set(key.urn, { value }, { size: 1, ttl: ttlMs }); } - - private async resolveLocalTtlSec(key: DialCacheKey): Promise { - const layerConfig = await this.resolveLayerConfig(key); - return layerConfig.status === "enabled" ? layerConfig.config.ttlSec : null; - } } diff --git a/src/internal/redis-cache.ts b/src/internal/redis-cache.ts index 2f48fcb..0ff271e 100644 --- a/src/internal/redis-cache.ts +++ b/src/internal/redis-cache.ts @@ -1,6 +1,6 @@ import { performance } from "node:perf_hooks"; -import { CacheLayer, type CacheConfigProvider, type DialCacheKeyConfig } from "../config.js"; +import { CacheLayer } from "../config.js"; import { RedisReadTimeoutError } from "../errors.js"; import { invalidationPrefix, redisClusterHashTag, type DialCacheKey } from "../key.js"; import { @@ -9,6 +9,7 @@ import { type DialCacheMetricsAdapter, type MetricErrorKind, type MetricLayer, + type StaleRecoveryOutcome, } from "../metrics.js"; import type { DecodedRedisFrame, DialCacheRedisClient, RedisCachePayload } from "../redis-client.js"; import { JsonSerializer, type Serializer } from "../serializer.js"; @@ -22,7 +23,7 @@ import { } from "./compression.js"; import { assertValidDeadlineMs, withMonotonicDeadline } from "./deadline.js"; import { cacheTtlSecToMs, MAX_TRACKED_REDIS_VALUE_TTL_MS } from "./duration.js"; -import { fetchKeyConfig, resolveLayerConfigResult, type ResolvedLayerConfig } from "./runtime-config.js"; +import type { ResolvedRemoteLayerConfig } from "./runtime-config.js"; export interface RedisConfig { /** @@ -46,7 +47,6 @@ export interface RedisConfig { } interface RedisCacheOptions { - readonly configProvider: CacheConfigProvider; readonly redis: RedisConfig; readonly metrics: DialCacheMetricsAdapter | null; } @@ -60,12 +60,20 @@ interface StartedRedisRead { export type FutureFramePolicy = "reject" | "retain"; +type RedisStaleRecoveryResult = + | { readonly status: "hit"; readonly value: T } + | { readonly status: "miss" }; + +type FrameAgeResult = + | { readonly status: "valid"; readonly ageMs: number } + | { readonly status: "future" } + | { readonly status: "invalid" }; + const defaultSerializer = new JsonSerializer(); const REDIS_FRAME_KEY_SUFFIX = ":dialcache-frame-v1"; const DEFAULT_REMOTE_READ_TIMEOUT_MS = 50; export class RedisCache { - private readonly configProvider: CacheConfigProvider; private readonly defaultSerializer: Serializer; private readonly compression: Required | null; private readonly client: DialCacheRedisClient; @@ -85,7 +93,6 @@ export class RedisCache { throw new TypeError("RedisConfig.watermarkTtlSec was removed; watermark lifetime is derived by DialCache"); } - this.configProvider = options.configProvider; this.defaultSerializer = options.redis.serializer ?? defaultSerializer; this.compression = resolveCompressionConfig(options.redis.compression); this.metrics = options.metrics; @@ -101,36 +108,26 @@ export class RedisCache { this.client = options.redis.client; } - async get(key: DialCacheKey): Promise { - const result = await this.getResult(key); - return result.status === "hit" ? result.value : undefined; - } - - async getResult(key: DialCacheKey, keyConfig?: DialCacheKeyConfig | null): Promise> { - const layerConfig = await this.resolveRemoteLayerConfig(key, keyConfig); - if (layerConfig.status === "disabled") { - return layerConfig; - } - - return await this.getWithResolvedConfig( - key, - layerConfig.config, - keyConfig?.remoteReadTimeoutMs ?? this.readTimeoutMs, - ); - } - async getWithResolvedConfig( key: DialCacheKey, - layerConfig: ResolvedLayerConfig, + layerConfig: ResolvedRemoteLayerConfig, readTimeoutMs = this.readTimeoutMs, ): Promise> { + const freshAgeMs = cacheTtlSecToMs(layerConfig.ttlSec); + const maximumAgeMs = cacheTtlSecToMs( + layerConfig.staleOnErrorMaxAgeSec ?? layerConfig.ttlSec, + ); const metricLayer = CacheLayer.REMOTE; const start = performance.now(); this.recordMetric((metrics) => metrics.request(labelsFor(key, metricLayer))); try { let frame: DecodedRedisFrame | null; try { - frame = await this.startPayloadRead(key, readTimeoutMs, metricLayer, false).result; + frame = await this.startRawPayloadRead( + key, + readTimeoutMs, + false, + ).result; } catch (error) { this.recordError( key, @@ -141,7 +138,20 @@ export class RedisCache { } if (frame === null) { this.recordMetric((metrics) => metrics.miss(labelsFor(key, metricLayer))); - return { status: "miss", config: layerConfig }; + return { status: "miss", config: layerConfig, reason: "cache_miss" }; + } + + // Classify the raw value/watermark snapshot from one application-clock + // sample after the bounded read settles. M is the absolute ceiling and F + // remains the ordinary serving boundary. + const frameAge = this.frameAge(key, frame, metricLayer); + if (frameAge.status !== "valid" || frameAge.ageMs >= maximumAgeMs) { + this.recordMetric((metrics) => metrics.miss(labelsFor(key, metricLayer))); + return { status: "miss", config: layerConfig, reason: "cache_miss" }; + } + if (frameAge.ageMs >= freshAgeMs) { + this.recordMetric((metrics) => metrics.miss(labelsFor(key, metricLayer))); + return { status: "retained", config: layerConfig, frame }; } try { @@ -149,14 +159,55 @@ export class RedisCache { return { status: "hit", value, frame }; } catch { this.recordMetric((metrics) => metrics.miss(labelsFor(key, metricLayer))); - return { status: "miss", config: layerConfig }; + return { status: "miss", config: layerConfig, reason: "deserialization_error" }; } } finally { - // Preserve the established caller-serving boundary: Redis read plus load. + // Preserve the caller-serving boundary: Redis read plus any ordinary + // fresh-value load. A retained candidate is loaded only after the source + // rejects and is not another Redis operation. this.recordMetric((metrics) => metrics.observeGet(labelsFor(key, metricLayer), elapsedSeconds(start))); } } + /** + * Use the raw F..M frame retained by the initial read. No Redis command is + * issued here. The age is checked both before and after the potentially + * asynchronous serializer so the value is below M when it actually serves. + */ + async recoverRetainedCandidate( + key: DialCacheKey, + frame: DecodedRedisFrame | null, + maxAgeSec: number, + ): Promise> { + if (frame === null) { + this.recordStaleRecovery(key, "miss"); + return { status: "miss" }; + } + + const maximumAgeMs = cacheTtlSecToMs(maxAgeSec); + const initialAge = this.frameAge(key, frame, CacheLayer.REMOTE); + if (initialAge.status !== "valid" || initialAge.ageMs >= maximumAgeMs) { + this.recordStaleRecovery(key, "miss"); + return { status: "miss" }; + } + + let value: T; + try { + value = await this.deserializePayload(key, frame.payload, CacheLayer.REMOTE); + } catch { + this.recordStaleRecovery(key, "deserialization_error"); + return { status: "miss" }; + } + + const servingAge = this.frameAge(key, frame, CacheLayer.REMOTE); + if (servingAge.status !== "valid" || servingAge.ageMs >= maximumAgeMs) { + this.recordStaleRecovery(key, "miss"); + return { status: "miss" }; + } + this.recordStaleRecovery(key, "served", servingAge.ageMs / 1_000); + return { status: "hit", value }; + } + /** * Decode the retained Redis payload again for detached semantic comparison, * recording it separately from caller-serving Redis work. @@ -173,11 +224,13 @@ export class RedisCache { */ startPayloadReadForShadow( key: DialCacheKey, + maxAgeSec: number | null, readTimeoutMs: number, futureFramePolicy: FutureFramePolicy, ): StartedRedisRead { return this.startMeasuredPayloadRead( key, + maxAgeSec === null ? null : cacheTtlSecToMs(maxAgeSec), readTimeoutMs, REMOTE_SHADOW_CACHE_LAYER, true, @@ -185,25 +238,21 @@ export class RedisCache { ); } - async put(key: DialCacheKey, value: T, config?: { readonly ttlSec: number }): Promise { - const ttlSec = config?.ttlSec ?? await this.resolveRemoteTtlSec(key); - if (ttlSec === null) { - return; - } - await this.putWithLayer(key, value, ttlSec, CacheLayer.REMOTE); + async put(key: DialCacheKey, value: T, config: ResolvedRemoteLayerConfig): Promise { + await this.putWithLayer(key, value, retentionTtlSecFor(config), CacheLayer.REMOTE); } /** Populate a detached Redis miss using the caller's resolved policy snapshot. */ async putForShadow( key: DialCacheKey, value: T, - config: { readonly ttlSec: number }, + config: ResolvedRemoteLayerConfig, shouldWrite: () => boolean, ): Promise { await this.putWithLayer( key, value, - config.ttlSec, + retentionTtlSecFor(config), REMOTE_SHADOW_CACHE_LAYER, shouldWrite, ); @@ -297,10 +346,24 @@ export class RedisCache { private startPayloadRead( key: DialCacheKey, + maxAgeMs: number | null, readTimeoutMs: number, metricLayer: MetricLayer, unrefTimer: boolean, futureFramePolicy: FutureFramePolicy = "reject", + ): StartedRedisRead { + const read = this.startRawPayloadRead(key, readTimeoutMs, unrefTimer); + return { + result: read.result.then((frame) => + this.validateFrameAge(key, frame, maxAgeMs, metricLayer, futureFramePolicy)), + settled: read.settled, + }; + } + + private startRawPayloadRead( + key: DialCacheKey, + readTimeoutMs: number, + unrefTimer: boolean, ): StartedRedisRead { const abortController = new AbortController(); const pending = Promise.resolve().then(() => @@ -319,10 +382,8 @@ export class RedisCache { timeoutError: () => new RedisReadTimeoutError(key.useCase, readTimeoutMs), unrefTimer, }); - const result = bounded.then((frame) => - this.validateTrackedFrame(key, frame, metricLayer, futureFramePolicy)); return { - result, + result: bounded, settled: pending.then( () => undefined, () => undefined, @@ -332,6 +393,7 @@ export class RedisCache { private startMeasuredPayloadRead( key: DialCacheKey, + maxAgeMs: number | null, readTimeoutMs: number, metricLayer: MetricLayer, unrefTimer: boolean, @@ -339,7 +401,14 @@ export class RedisCache { ): StartedRedisRead { const start = performance.now(); this.recordMetric((metrics) => metrics.request(labelsFor(key, metricLayer))); - const read = this.startPayloadRead(key, readTimeoutMs, metricLayer, unrefTimer, futureFramePolicy); + const read = this.startPayloadRead( + key, + maxAgeMs, + readTimeoutMs, + metricLayer, + unrefTimer, + futureFramePolicy, + ); const result = read.result.then( (frame) => { if (frame === null) { @@ -361,18 +430,34 @@ export class RedisCache { return { result, settled: read.settled }; } - private validateTrackedFrame( + private validateFrameAge( key: DialCacheKey, frame: DecodedRedisFrame | null, + maxAgeMs: number | null, metricLayer: MetricLayer, futureFramePolicy: FutureFramePolicy, ): DecodedRedisFrame | null { - if (frame === null || !key.trackForInvalidation) { - return frame; + if (frame === null) { + return null; } - if (!Number.isSafeInteger(frame.createdAtMs) || frame.createdAtMs < 0) { + const age = this.frameAge(key, frame, metricLayer); + if (age.status === "future") { + return futureFramePolicy === "reject" ? null : frame; + } + if (age.status === "invalid") { return null; } + return maxAgeMs === null || age.ageMs < maxAgeMs ? frame : null; + } + + private frameAge( + key: DialCacheKey, + frame: DecodedRedisFrame, + metricLayer: MetricLayer, + ): FrameAgeResult { + if (!Number.isSafeInteger(frame.createdAtMs) || frame.createdAtMs < 0) { + return { status: "invalid" }; + } const readerNowMs = Date.now(); if (frame.createdAtMs > readerNowMs) { @@ -383,9 +468,9 @@ export class RedisCache { offsetSeconds, )); } - return futureFramePolicy === "reject" ? null : frame; + return { status: "future" }; } - return frame; + return { status: "valid", ageMs: readerNowMs - frame.createdAtMs }; } private async deserializePayload( @@ -420,20 +505,6 @@ export class RedisCache { return key.serializer ?? this.defaultSerializer; } - private async resolveRemoteLayerConfig(key: DialCacheKey, keyConfig?: DialCacheKeyConfig | null) { - const config = keyConfig === undefined ? await fetchKeyConfig(this.configProvider, key) : keyConfig; - return resolveLayerConfigResult({ - config, - key, - layer: CacheLayer.REMOTE, - }); - } - - private async resolveRemoteTtlSec(key: DialCacheKey): Promise { - const layerConfig = await this.resolveRemoteLayerConfig(key); - return layerConfig.status === "enabled" ? layerConfig.config.ttlSec : null; - } - private recordMetric(record: (metrics: DialCacheMetricsAdapter) => void): void { if (this.metrics === null) { return; @@ -448,6 +519,30 @@ export class RedisCache { private recordError(key: DialCacheKey, layer: MetricLayer, kind: MetricErrorKind): void { this.recordMetric((metrics) => metrics.error({ ...labelsFor(key, layer), error: kind, inFallback: false })); } + + private recordStaleRecovery( + key: DialCacheKey, + outcome: StaleRecoveryOutcome, + valueAgeSeconds?: number, + ): void { + const labels = { + cacheNamespace: key.namespace, + useCase: key.useCase, + keyType: key.keyType, + outcome, + } as const; + this.recordMetric((metrics) => metrics.staleRecovery?.(labels)); + if (valueAgeSeconds !== undefined) { + this.recordMetric((metrics) => metrics.observeStaleRecoveryValueAge?.( + labels, + valueAgeSeconds, + )); + } + } +} + +function retentionTtlSecFor(config: ResolvedRemoteLayerConfig): number { + return config.staleOnErrorMaxAgeSec ?? config.ttlSec; } function payloadSize(payload: string | Buffer): number { diff --git a/src/internal/redis-payload.ts b/src/internal/redis-payload.ts index c857b63..d77ab83 100644 --- a/src/internal/redis-payload.ts +++ b/src/internal/redis-payload.ts @@ -53,9 +53,9 @@ function decodeRedisPayload(raw: Buffer): RedisCachePayload { /** * Encode a serializer payload into a servable DialCache Redis frame. * - * Writes stamp a client-clock `createdAtMs`. Core rejects tracked frames dated - * after the reader's clock and uses decoded timestamps for shadow value-age - * observations, so stamp real client time, not a constant. + * Writes stamp a client-clock `createdAtMs`. Core rejects serving frames dated + * after the reader's clock, enforces logical age, and uses decoded timestamps + * for shadow value-age observations, so stamp real client time, not a constant. */ export function encodeRedisFrame(payload: RedisCachePayload, createdAtMs: number): Buffer { if (!isValidRedisTimestampMs(createdAtMs)) { diff --git a/src/internal/runtime-config.ts b/src/internal/runtime-config.ts index fdabb70..9f10f68 100644 --- a/src/internal/runtime-config.ts +++ b/src/internal/runtime-config.ts @@ -15,25 +15,43 @@ export interface ResolvedLayerConfig { readonly ramp: number; } -export type LayerConfigResolution = - | { readonly status: "enabled"; readonly config: ResolvedLayerConfig } +/** Remote-only policy resolved against the same invocation snapshot as its TTL. */ +export interface ResolvedRemoteLayerConfig extends ResolvedLayerConfig { + readonly staleOnErrorMaxAgeSec: number | null; +} + +export type LayerConfigResolution = + | { readonly status: "enabled"; readonly config: Config } | { readonly status: "disabled"; readonly reason: "ramped_down"; /** Valid policy retained even though its ramp excluded this key. */ - readonly config: ResolvedLayerConfig; + readonly config: Config; } | { readonly status: "disabled"; readonly reason: Exclude; }; +/** + * A malformed optional stale policy is diagnostic-only: the valid remote layer + * remains available with recovery disabled. + */ +type RemoteLayerConfigResolution = LayerConfigResolution & { + readonly staleOnErrorConfigError?: true; +}; + interface ResolveLayerConfigOptions { readonly config: DialCacheKeyConfig | null; readonly key: DialCacheKey; readonly layer: CacheLayer; } +interface ResolveRemoteLayerConfigOptions { + readonly config: DialCacheKeyConfig | null; + readonly key: DialCacheKey; +} + export async function fetchKeyConfig( configProvider: CacheConfigProvider, key: DialCacheKey, @@ -46,11 +64,6 @@ export async function fetchKeyConfig( return mergeKeyConfig(defaultConfig, runtimeConfig); } -export function resolveLayerConfig(options: ResolveLayerConfigOptions): ResolvedLayerConfig | null { - const resolution = resolveLayerConfigResult(options); - return resolution.status === "enabled" ? resolution.config : null; -} - export function resolveLayerConfigResult(options: ResolveLayerConfigOptions): LayerConfigResolution { const config = options.config; if (config === null) { @@ -91,6 +104,48 @@ export function resolveLayerConfigResult(options: ResolveLayerConfigOptions): La : { status: "disabled", reason: "ramped_down", config: { ttlSec, ramp } }; } +export function resolveRemoteLayerConfigResult( + options: ResolveRemoteLayerConfigOptions, +): RemoteLayerConfigResolution { + const resolution = resolveLayerConfigResult({ + ...options, + layer: CacheLayer.REMOTE, + }); + const configuredMaxAge: unknown = options.config?.staleOnErrorMaxAgeSec; + if (!("config" in resolution)) { + if ( + resolution.reason === "policy_disabled" + && configuredMaxAge !== undefined + && configuredMaxAge !== 0 + ) { + return { ...resolution, staleOnErrorConfigError: true }; + } + return resolution; + } + + if (configuredMaxAge === undefined || configuredMaxAge === 0) { + return { + ...resolution, + config: { ...resolution.config, staleOnErrorMaxAgeSec: null }, + }; + } + if ( + !isSupportedCacheTtlSec(configuredMaxAge) + || configuredMaxAge <= resolution.config.ttlSec + ) { + return { + ...resolution, + config: { ...resolution.config, staleOnErrorMaxAgeSec: null }, + staleOnErrorConfigError: true, + }; + } + + return { + ...resolution, + config: { ...resolution.config, staleOnErrorMaxAgeSec: configuredMaxAge }, + }; +} + function mergeKeyConfig( defaultConfig: DialCacheKeyConfig | null, runtimeConfig: DialCacheKeyConfig | null | undefined, @@ -112,6 +167,9 @@ function mergeKeyConfig( const remoteReadTimeoutMs = overlay?.remoteReadTimeoutMs !== undefined ? overlay.remoteReadTimeoutMs : defaultConfig?.remoteReadTimeoutMs; + const staleOnErrorMaxAgeSec = overlay?.staleOnErrorMaxAgeSec !== undefined + ? overlay.staleOnErrorMaxAgeSec + : defaultConfig?.staleOnErrorMaxAgeSec; const shadow = mergeShadowConfig(defaultConfig?.shadow, overlay?.shadow); return new DialCacheKeyConfig({ @@ -119,6 +177,7 @@ function mergeKeyConfig( ramp: mergeLayerConfig(defaultConfig?.ramp, overlay?.ramp, "ramp"), ...(requestLocal === undefined ? {} : { requestLocal }), ...(coalesce === undefined ? {} : { coalesce }), + ...(staleOnErrorMaxAgeSec === undefined ? {} : { staleOnErrorMaxAgeSec }), ...(remoteReadTimeoutMs === undefined ? {} : { remoteReadTimeoutMs }), ...(shadow === undefined ? {} : { shadow }), }); diff --git a/src/metrics.ts b/src/metrics.ts index 2bda225..925955d 100644 --- a/src/metrics.ts +++ b/src/metrics.ts @@ -24,6 +24,11 @@ export type ShadowValidationOutcome = | "confirmation_error" | "timeout" | "dropped"; +/** Bounded terminal outcomes for an attempted stale-on-error Redis recovery. */ +export type StaleRecoveryOutcome = + | "served" + | "miss" + | "deserialization_error"; /** * Bounded compression outcomes. Writes record compressed, below_threshold, * not_smaller, or write_over_limit (serialized form exceeds the decompression @@ -106,6 +111,13 @@ export interface ShadowValidationMetricLabels { readonly outcome: ShadowValidationOutcome; } +export interface StaleRecoveryMetricLabels { + readonly cacheNamespace: string; + readonly useCase: string; + readonly keyType: string; + readonly outcome: StaleRecoveryOutcome; +} + export interface DialCacheMetricsAdapter { request(labels: CacheMetricLabels): void; miss(labels: CacheMetricLabels): void; @@ -128,14 +140,26 @@ export interface DialCacheMetricsAdapter { */ observeShadowValueAge?(labels: ShadowValidationMetricLabels, seconds: number): void; /** - * Positive offset in seconds when a decoded tracked Redis frame is dated - * after the observing process's epoch clock. Serving and initial shadow reads - * fail closed as misses; a shadow confirmation retains the frame solely for + * Positive offset in seconds when a decoded Redis frame is dated after the + * observing process's epoch clock. Serving and initial shadow reads fail + * closed as misses; a shadow confirmation retains the frame solely for * payload comparison. This is a workload-shaped diagnostic, not proof of * clock skew. Optional so existing custom adapters keep compiling. */ observeFutureTimestampOffset?(labels: CacheMetricLabels, seconds: number): void; // Optional so existing custom adapters keep compiling without changes. + staleRecovery?(labels: StaleRecoveryMetricLabels): void; + /** + * Age in seconds of the Redis value when stale-on-error recovery serves it: + * the observing process's epoch clock minus the retained frame's + * `createdAtMs`, clamped at zero. Emitted only alongside the terminal + * `served` outcome. Frames are stamped with the writer application's epoch + * clock, so cross-process skew makes this coarse operational evidence rather + * than a precise measurement. Optional so existing custom adapters keep + * compiling without changes. + */ + observeStaleRecoveryValueAge?(labels: StaleRecoveryMetricLabels, seconds: number): void; + // Optional so existing custom adapters keep compiling without changes. compression?(labels: CompressionMetricLabels): void; observeGet(labels: CacheMetricLabels, seconds: number): void; observeFallback(labels: CacheMetricLabels, seconds: number): void; diff --git a/src/prometheus.ts b/src/prometheus.ts index e3ee92e..2f0a658 100644 --- a/src/prometheus.ts +++ b/src/prometheus.ts @@ -11,6 +11,7 @@ import type { InvalidationMetricLabels, SerializationMetricLabels, ShadowValidationMetricLabels, + StaleRecoveryMetricLabels, } from "./metrics.js"; export interface PrometheusMetricsOptions { @@ -26,7 +27,8 @@ type ErrorLabels = CounterLabels | "error" | "in_fallback"; type SerializationLabels = CounterLabels | "operation"; type InvalidationLabels = "cache_namespace" | "key_type" | "layer"; type CoalescedLabels = "cache_namespace" | "use_case" | "key_type" | "scope"; -type ShadowValidationLabels = "cache_namespace" | "use_case" | "key_type" | "outcome"; +type OutcomeLabels = "cache_namespace" | "use_case" | "key_type" | "outcome"; +type OutcomeMetricLabels = ShadowValidationMetricLabels | StaleRecoveryMetricLabels; type CompressionLabels = CounterLabels | "outcome"; interface BaseCollectorConfig { @@ -87,9 +89,11 @@ export class PrometheusDialCacheMetrics implements DialCacheMetricsAdapter { private readonly errorCounter: Counter; private readonly invalidationCounter: Counter; private readonly coalescedCounter: Counter; - private readonly shadowValidationCounter: Counter; - private readonly shadowValueAgeHistogram: Histogram; + private readonly shadowValidationCounter: Counter; + private readonly shadowValueAgeHistogram: Histogram; private readonly futureTimestampOffsetHistogram: Histogram; + private readonly staleRecoveryCounter: Counter; + private readonly staleRecoveryValueAgeHistogram: Histogram; private readonly compressionCounter: Counter; private readonly getTimer: Histogram; private readonly fallbackTimer: Histogram; @@ -114,6 +118,8 @@ export class PrometheusDialCacheMetrics implements DialCacheMetricsAdapter { this.shadowValidationCounter = counter(registry, collectors.shadowValidationCounter); this.shadowValueAgeHistogram = histogram(registry, collectors.shadowValueAgeHistogram); this.futureTimestampOffsetHistogram = histogram(registry, collectors.futureTimestampOffsetHistogram); + this.staleRecoveryCounter = counter(registry, collectors.staleRecoveryCounter); + this.staleRecoveryValueAgeHistogram = histogram(registry, collectors.staleRecoveryValueAgeHistogram); this.compressionCounter = counter(registry, collectors.compressionCounter); this.getTimer = histogram(registry, collectors.getTimer); this.fallbackTimer = histogram(registry, collectors.fallbackTimer); @@ -162,11 +168,11 @@ export class PrometheusDialCacheMetrics implements DialCacheMetricsAdapter { } shadowValidation(labels: ShadowValidationMetricLabels): void { - this.shadowValidationCounter.inc(shadowValidationLabels(labels)); + this.shadowValidationCounter.inc(outcomeLabels(labels)); } observeShadowValueAge(labels: ShadowValidationMetricLabels, seconds: number): void { - this.shadowValueAgeHistogram.observe(shadowValidationLabels(labels), seconds); + this.shadowValueAgeHistogram.observe(outcomeLabels(labels), seconds); } observeFutureTimestampOffset(labels: CacheMetricLabels, seconds: number): void { @@ -176,6 +182,14 @@ export class PrometheusDialCacheMetrics implements DialCacheMetricsAdapter { this.futureTimestampOffsetHistogram.observe(cacheLabels(labels), seconds); } + staleRecovery(labels: StaleRecoveryMetricLabels): void { + this.staleRecoveryCounter.inc(outcomeLabels(labels)); + } + + observeStaleRecoveryValueAge(labels: StaleRecoveryMetricLabels, seconds: number): void { + this.staleRecoveryValueAgeHistogram.observe(outcomeLabels(labels), seconds); + } + compression(labels: CompressionMetricLabels): void { this.compressionCounter.inc({ ...cacheLabels(labels), outcome: labels.outcome }); } @@ -222,7 +236,7 @@ function cacheLabels(labels: CacheMetricLabels): Record { }; } -function shadowValidationLabels(labels: ShadowValidationMetricLabels): Record { +function outcomeLabels(labels: OutcomeMetricLabels): Record { return { cache_namespace: labels.cacheNamespace, use_case: labels.useCase, @@ -285,10 +299,23 @@ function collectorConfigs(prefix: string) { futureTimestampOffsetHistogram: { type: "histogram", name: `${prefix}dialcache_future_timestamp_offset_histogram`, - help: "Positive offset in seconds of tracked Redis frames dated after the observing DialCache process clock.", + help: "Positive offset in seconds of Redis frames dated after the observing DialCache process clock.", labelNames: ["cache_namespace", "use_case", "key_type", "layer"], buckets: FUTURE_TIMESTAMP_OFFSET_BUCKETS, }, + staleRecoveryCounter: { + type: "counter", + name: `${prefix}dialcache_stale_recovery_counter`, + help: "DialCache stale-on-error Redis recovery outcomes.", + labelNames: ["cache_namespace", "use_case", "key_type", "outcome"], + }, + staleRecoveryValueAgeHistogram: { + type: "histogram", + name: `${prefix}dialcache_stale_recovery_value_age_histogram`, + help: "Age in seconds of Redis values served by DialCache stale-on-error recovery.", + labelNames: ["cache_namespace", "use_case", "key_type", "outcome"], + buckets: VALUE_AGE_BUCKETS, + }, compressionCounter: { type: "counter", name: `${prefix}dialcache_compression_counter`, diff --git a/src/redis-client.ts b/src/redis-client.ts index aa934b1..780656d 100644 --- a/src/redis-client.ts +++ b/src/redis-client.ts @@ -66,12 +66,13 @@ export type RedisCachePayload = string | Buffer; * header's creation time. The payload is the serializer output, possibly * still wrapped in a compression envelope that DialCache core interprets * above the adapter (see the `dialcache/redis-protocol` module doc). All - * frames carry application-clock time supplied by the writer. Tracked serving - * and initial-shadow reads reject frames dated after the reading process's - * clock before deserialization; confirmation reads retain them for payload - * comparison, and untracked reads treat the stamp as informational. DialCache - * also uses `createdAtMs` for shadow value-age observability. Tracked watermark - * fencing already happened inside the decoder. + * frames carry application-clock time supplied by the writer. Caller-serving + * reads apply their logical age ceiling and reject frames dated after the + * reading process's clock before deserialization; stale-on-error may retain a + * raw frame between its fresh and maximum ages. Confirmation reads may retain + * a frame solely for payload comparison. DialCache also uses `createdAtMs` for + * shadow and stale-recovery value-age observability. Tracked watermark fencing + * already happened inside the decoder. */ export interface DecodedRedisFrame { readonly payload: RedisCachePayload; @@ -151,9 +152,9 @@ export interface DialCacheRedisClient { * * A non-null frame is transferred to DialCache. A returned Buffer payload * must remain stable and must not be mutated, pooled, or reused after this - * method settles; DialCache may retain it beyond the request for - * best-effort shadow deserialization. Adapters that recycle response - * storage must return a dedicated Buffer. + * method settles; DialCache may retain it for source-error recovery or + * best-effort shadow work. Adapters that recycle response storage must + * return a dedicated Buffer. */ read(request: RedisReadRequest, context?: RedisReadContext): Awaitable; /** @@ -162,11 +163,9 @@ export interface DialCacheRedisClient { * * All writes are one native `SET valueKey frame PX cacheTtlMs` whose * frame comes from `encodeRedisFrame` with a client-clock `createdAtMs`. - * DialCache uses a tracked frame's decoded `createdAtMs` for future-time - * rejection and uses decoded timestamps for shadow value-age observations, - * so writers must stamp real client time, not a constant. Untracked serving - * remains governed by Redis TTL and does not reject on the informational - * timestamp. + * DialCache uses every frame's decoded `createdAtMs` for future-time + * rejection and logical-age enforcement, and for shadow value-age + * observations, so writers must stamp real client time, not a constant. * * Tracked and untracked writes use the same complete-frame SET. Core caps a * tracked value's physical TTL at one hour. Under the documented clock-skew diff --git a/src/serializer.ts b/src/serializer.ts index c57334f..626044a 100644 --- a/src/serializer.ts +++ b/src/serializer.ts @@ -7,8 +7,9 @@ export interface Serializer { * Async implementations must settle within an application-defined deadline. * The serialized input is borrowed and immutable. Implementations must not * mutate a Buffer passed here because DialCache may retain the exact payload - * and call load again for best-effort shadow validation; copy it first if - * mutation is required. Repeated loads of the same payload must be independent. + * for stale recovery or call load again for best-effort shadow validation; + * copy it first if mutation is required. Repeated loads of the same payload + * must be independent. */ load(value: string | Buffer): Awaitable; } diff --git a/test/datadog.test.ts b/test/datadog.test.ts index 5f7a538..3c92dbc 100644 --- a/test/datadog.test.ts +++ b/test/datadog.test.ts @@ -11,6 +11,7 @@ import { type MetricErrorKind, type MetricLayer, type ShadowValidationOutcome, + type StaleRecoveryOutcome, } from "../src/index.js"; import { NO_CACHE_LAYER, @@ -120,6 +121,12 @@ const SHADOW_VALIDATION_OUTCOMES: Readonly dropped: true, }; const shadowValidationOutcomes = Object.keys(SHADOW_VALIDATION_OUTCOMES) as ShadowValidationOutcome[]; +const STALE_RECOVERY_OUTCOMES: Readonly> = { + served: true, + miss: true, + deserialization_error: true, +}; +const staleRecoveryOutcomes = Object.keys(STALE_RECOVERY_OUTCOMES) as StaleRecoveryOutcome[]; const COMPRESSION_OUTCOMES: Readonly> = { compressed: true, below_threshold: true, @@ -160,6 +167,21 @@ describe("Datadog metrics adapter", () => { keyType: "user_id", outcome: "match", }); + metrics.staleRecovery({ + cacheNamespace: cacheLabels.cacheNamespace, + useCase: "LoadUser", + keyType: "user_id", + outcome: "served", + }); + metrics.observeStaleRecoveryValueAge( + { + cacheNamespace: cacheLabels.cacheNamespace, + useCase: "LoadUser", + keyType: "user_id", + outcome: "served", + }, + 90.5, + ); metrics.observeShadowValueAge( { cacheNamespace: cacheLabels.cacheNamespace, @@ -213,6 +235,18 @@ describe("Datadog metrics adapter", () => { value: 1, tags: { cache_namespace: "users", use_case: "LoadUser", key_type: "user_id", outcome: "match" }, }, + { + method: "increment", + name: "dialcache.stale_recovery.count", + value: 1, + tags: { cache_namespace: "users", use_case: "LoadUser", key_type: "user_id", outcome: "served" }, + }, + { + method: "distribution", + name: "dialcache.stale_recovery.value_age", + value: 90.5, + tags: { cache_namespace: "users", use_case: "LoadUser", key_type: "user_id", outcome: "served" }, + }, { method: "distribution", name: "dialcache.shadow.value_age", @@ -278,6 +312,15 @@ describe("Datadog metrics adapter", () => { }, 60, ); + metrics.observeStaleRecoveryValueAge( + { + cacheNamespace: cacheLabels.cacheNamespace, + useCase: cacheLabels.useCase, + keyType: cacheLabels.keyType, + outcome: "served", + }, + 90, + ); metrics.observeFutureTimestampOffset(cacheLabels, 0.006); expect(client.calls.map(({ method, name, value }) => ({ method, name, value }))).toEqual([ @@ -289,6 +332,7 @@ describe("Datadog metrics adapter", () => { { method: observationMetricType, name: "service.cache.compression.ratio", value: 0.04 }, { method: observationMetricType, name: "service.cache.compression.duration", value: 0.05 }, { method: observationMetricType, name: "service.cache.shadow.value_age", value: 60 }, + { method: observationMetricType, name: "service.cache.stale_recovery.value_age", value: 90 }, { method: observationMetricType, name: "service.cache.future_timestamp_offset", value: 0.006 }, ]); }); @@ -327,6 +371,14 @@ describe("Datadog metrics adapter", () => { outcome, }); } + for (const outcome of staleRecoveryOutcomes) { + metrics.staleRecovery({ + cacheNamespace: cacheLabels.cacheNamespace, + useCase: cacheLabels.useCase, + keyType: cacheLabels.keyType, + outcome, + }); + } for (const outcome of compressionOutcomes) { metrics.compression({ ...cacheLabels, outcome }); } @@ -364,6 +416,18 @@ describe("Datadog metrics adapter", () => { outcome, })), ); + expect( + client.calls + .filter(({ name }) => name === "dialcache.stale_recovery.count") + .map(({ tags }) => tags), + ).toEqual( + staleRecoveryOutcomes.map((outcome) => ({ + cache_namespace: cacheLabels.cacheNamespace, + use_case: cacheLabels.useCase, + key_type: cacheLabels.keyType, + outcome, + })), + ); expect( client.calls .filter(({ name }) => name === "dialcache.compression.count") @@ -584,15 +648,23 @@ describe("Datadog metrics adapter", () => { it("enforces Datadog's 200-character final metric-name limit", () => { const client = new RecordingDogStatsDClient(); - const longestValidNamespace = "a".repeat(176); - const tooLongNamespace = "a".repeat(177); + const longestValidNamespace = "a".repeat(175); + const tooLongNamespace = "a".repeat(176); const metrics = new DatadogDialCacheMetrics({ client, namespace: longestValidNamespace, observationMetricType: "distribution", }); - metrics.observeFutureTimestampOffset(cacheLabels, 1); + metrics.observeStaleRecoveryValueAge( + { + cacheNamespace: cacheLabels.cacheNamespace, + useCase: cacheLabels.useCase, + keyType: cacheLabels.keyType, + outcome: "served", + }, + 1, + ); expect(client.calls[0]?.name).toHaveLength(200); expect( diff --git a/test/dialcache-config-ramp.test.ts b/test/dialcache-config-ramp.test.ts index eb0d8ae..a885b06 100644 --- a/test/dialcache-config-ramp.test.ts +++ b/test/dialcache-config-ramp.test.ts @@ -44,6 +44,12 @@ describe("DialCache runtime config and ramp controls", () => { expect(new DialCacheKeyConfig({ coalesce: true }).coalesce).toBe(true); }); + it("preserves stale-on-error omission and explicit disable values", () => { + expect(new DialCacheKeyConfig({}).staleOnErrorMaxAgeSec).toBeUndefined(); + expect(new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 0 }).staleOnErrorMaxAgeSec).toBe(0); + expect(new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 3_600 }).staleOnErrorMaxAgeSec).toBe(3_600); + }); + it("preserves shadow omission and explicit kill-switch values", () => { expect(new DialCacheKeyConfig({}).shadow).toBeUndefined(); expect(new DialCacheKeyConfig({ shadow: {} }).shadow).toEqual({}); @@ -77,13 +83,14 @@ describe("DialCache runtime config and ramp controls", () => { it("captures an immutable default policy snapshot when the use case is registered", async () => { const suppliedDefault = new DialCacheKeyConfig({ - ttlSec: { [CacheLayer.LOCAL]: 60 }, - ramp: { [CacheLayer.LOCAL]: 100 }, + ttlSec: { [CacheLayer.LOCAL]: 60, [CacheLayer.REMOTE]: 60 }, + ramp: { [CacheLayer.LOCAL]: 100, [CacheLayer.REMOTE]: 100 }, shadow: { ramp: 25, logMismatches: true, }, coalesce: false, + staleOnErrorMaxAgeSec: 3_600, }); const observedDefaults: Array = []; const dialcache = new DialCache({ @@ -103,6 +110,7 @@ describe("DialCache runtime config and ramp controls", () => { suppliedDefault.ttlSec[CacheLayer.LOCAL] = 0; suppliedDefault.ramp[CacheLayer.LOCAL] = 0; + (suppliedDefault as { staleOnErrorMaxAgeSec?: number }).staleOnErrorMaxAgeSec = 0; const mutableShadow = suppliedDefault.shadow as { ramp?: number; logMismatches?: boolean; @@ -119,6 +127,7 @@ describe("DialCache runtime config and ramp controls", () => { expect(observedDefaults[1]).toBe(observedDefaults[0]); expect(observedDefaults[0]?.ttlSec[CacheLayer.LOCAL]).toBe(60); expect(observedDefaults[0]?.ramp[CacheLayer.LOCAL]).toBe(100); + expect(observedDefaults[0]?.staleOnErrorMaxAgeSec).toBe(3_600); expect(observedDefaults[0]?.shadow).toEqual({ ramp: 25, logMismatches: true, @@ -360,6 +369,75 @@ describe("DialCache runtime config and ramp controls", () => { RangeError, `no greater than ${MAX_CACHE_TTL_SEC}`, ], + [ + "negative stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: -1, + }), + RangeError, + "nonnegative safe integer", + ], + [ + "fractional stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 60.5, + }), + RangeError, + "nonnegative safe integer", + ], + [ + "non-finite stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: Number.NaN, + }), + RangeError, + "nonnegative safe integer", + ], + [ + "unsafe stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: Number.MAX_SAFE_INTEGER + 1, + }), + RangeError, + "nonnegative safe integer", + ], + [ + "over-maximum stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: MAX_CACHE_TTL_SEC + 1, + }), + RangeError, + `no greater than ${MAX_CACHE_TTL_SEC}`, + ], + [ + "stale-on-error max age equal to the remote TTL", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 60, + }), + RangeError, + "must be greater than ttlSec.remote", + ], + [ + "positive stale-on-error max age without a remote TTL", + new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 3_600 }), + RangeError, + "requires ttlSec.remote", + ], + [ + "stale-on-error max age below the remote TTL", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 30, + }), + RangeError, + "must be greater than ttlSec.remote", + ], ["negative ramp", new DialCacheKeyConfig({ ramp: { [CacheLayer.LOCAL]: -1 } }), RangeError, "between 0 and 100"], ["over-100 ramp", new DialCacheKeyConfig({ ramp: { [CacheLayer.LOCAL]: 101 } }), RangeError, "between 0 and 100"], ["non-finite ramp", new DialCacheKeyConfig({ ramp: { [CacheLayer.LOCAL]: Number.POSITIVE_INFINITY } }), RangeError, "between 0 and 100"], @@ -405,6 +483,12 @@ describe("DialCache runtime config and ramp controls", () => { TypeError, "must be a boolean", ], + [ + "wrong-type stale-on-error max age", + new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: "3600" as unknown as number }), + TypeError, + "must be a number", + ], ["primitive config", 42 as unknown as DialCacheKeyConfig, TypeError, "must be an object"], ["array config", [] as unknown as DialCacheKeyConfig, TypeError, "must be an object"], [ @@ -457,6 +541,35 @@ describe("DialCache runtime config and ramp controls", () => { })).not.toThrow(); }); + it("accepts disabled and exact-maximum static stale-on-error policy", () => { + const dialcache = new DialCache(); + + expect(() => dialcache.cached(async () => "value", { + keyType: "item_id", + useCase: "DisabledStaticStaleOnErrorWithoutRemote", + cacheKey: () => "000", + defaultConfig: new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 0 }), + })).not.toThrow(); + expect(() => dialcache.cached(async () => "value", { + keyType: "item_id", + useCase: "DisabledStaticStaleOnError", + cacheKey: () => "123", + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 0, + }), + })).not.toThrow(); + expect(() => dialcache.cached(async () => "value", { + keyType: "item_id", + useCase: "MaximumStaticStaleOnError", + cacheKey: () => "456", + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: MAX_CACHE_TTL_SEC - 1 }, + staleOnErrorMaxAgeSec: MAX_CACHE_TTL_SEC, + }), + })).not.toThrow(); + }); + it.each([ ["a primitive", 42], ["an array", []], @@ -487,6 +600,7 @@ describe("DialCache runtime config and ramp controls", () => { it("returns the explicit kill-switch overlay from DialCacheKeyConfig.disabled()", () => { expect(DialCacheKeyConfig.disabled()).toEqual(new DialCacheKeyConfig({ requestLocal: false, + staleOnErrorMaxAgeSec: 0, shadow: { ramp: 0, logMismatches: false, @@ -575,6 +689,112 @@ describe("DialCache runtime config and ramp controls", () => { expect(second.calls).toBe(2); }); + it.each([ + ["null", null], + ["negative", -1], + ["fractional", 60.5], + ["NaN", Number.NaN], + ["infinite", Number.POSITIVE_INFINITY], + ["far over maximum", Number.MAX_SAFE_INTEGER], + ["unsafe", Number.MAX_SAFE_INTEGER + 1], + ["over maximum", MAX_CACHE_TTL_SEC + 1], + ["equal to fresh TTL", 60], + ["below fresh TTL", 30], + ["wrong type", "3600"], + ] as const)( + "disables only stale recovery for an invalid runtime max age ($0)", + async (_name, configuredMaxAge) => { + const redis = new FakeRedis(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: configuredMaxAge as unknown as number, + }), + }); + let calls = 0; + const getUser = dialcache.cached(async (userId: string) => ({ userId, calls: ++calls }), { + keyType: "user_id", + useCase: `InvalidRuntimeStaleMaxAge${String(_name)}`, + cacheKey: (userId) => userId, + }); + + const first = await dialcache.enable(async () => await getUser("123")); + const second = await dialcache.enable(async () => await getUser("123")); + + expect(second).toEqual(first); + expect(calls).toBe(1); + expect(redis.getCalls).toBe(2); + expect(redis.setCalls).toBe(1); + }, + ); + + it.each([ + [ + "an explicit stale-on-error zero", + () => new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 0 }), + 1, + ], + [ + "an invalid stale-on-error maximum", + () => new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 1 }), + 1, + ], + ["the complete disabled overlay", () => DialCacheKeyConfig.disabled(), 0], + ] as const)( + "does not recover a retained stale value after $0", + async (_name, disabledOverlay, expectedRemoteReads) => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-08-02T12:00:00.000Z")); + try { + const useCase = `RetainedStaleRuntimeDisable${expectedRemoteReads}`; + const redis = new FakeRedis(); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const source = vi.fn<() => Promise<{ readonly id: string; readonly version: number }>>() + .mockResolvedValueOnce({ id: "123", version: 1 }) + .mockRejectedValueOnce(sourceError); + let runtimeConfig = new DialCacheKeyConfig({}); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => runtimeConfig, + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 10, + }), + }); + const valueKey = `${new DialCacheKey({ + keyType: "user_id", + id: "123", + useCase, + }).urn}:dialcache-frame-v1`; + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ + id: "123", + version: 1, + }); + expect(redis.ttlMs(valueKey)).toBe(10_000); + await vi.advanceTimersByTimeAsync(2_000); + runtimeConfig = disabledOverlay(); + const readsBeforeDisabledCall = redis.getCalls; + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(redis.getCalls - readsBeforeDisabledCall).toBe(expectedRemoteReads); + expect(redis.setCalls).toBe(1); + expect(redis.ttlMs(valueKey)).toBe(8_000); + expect(source).toHaveBeenCalledTimes(2); + } finally { + vi.useRealTimers(); + } + }, + ); + it("applies runtime config changes to subsequent calls", async () => { // Given a provider whose config can change without redeploying the cached function. let runtimeConfig: DialCacheKeyConfig | null = DialCacheKeyConfig.enabled(60); diff --git a/test/dialcache-local.test.ts b/test/dialcache-local.test.ts index 0065281..227e782 100644 --- a/test/dialcache-local.test.ts +++ b/test/dialcache-local.test.ts @@ -504,10 +504,12 @@ describe("DialCache local-only MVP", () => { const dialcache = new DialCache({ logger }); const localCache = (dialcache as unknown as { readonly localCache: { - put: (key: DialCacheKey, value: unknown, config?: { readonly ttlSec: number }) => Promise; + put: (key: DialCacheKey, value: unknown, config: { readonly ttlSec: number }) => void; }; }).localCache; - vi.spyOn(localCache, "put").mockRejectedValueOnce(new Error("local write failed")); + vi.spyOn(localCache, "put").mockImplementationOnce(() => { + throw new Error("local write failed"); + }); let calls = 0; const getUser = dialcache.cached(async (userId: string) => ({ userId, calls: ++calls }), { keyType: "user_id", diff --git a/test/dialcache-logger.test.ts b/test/dialcache-logger.test.ts index ebb3509..8f8554a 100644 --- a/test/dialcache-logger.test.ts +++ b/test/dialcache-logger.test.ts @@ -119,10 +119,12 @@ describe("DialCache logger isolation", () => { const dialcache = new DialCache({ logger }); const localCache = (dialcache as unknown as { readonly localCache: { - put: (key: DialCacheKey, value: unknown, config?: { readonly ttlSec: number }) => Promise; + put: (key: DialCacheKey, value: unknown, config: { readonly ttlSec: number }) => void; }; }).localCache; - vi.spyOn(localCache, "put").mockRejectedValueOnce(new Error("local write failed")); + vi.spyOn(localCache, "put").mockImplementationOnce(() => { + throw new Error("local write failed"); + }); const getUser = dialcache.cached(async () => ({ source: "fallback" }), { keyType: "user_id", useCase: "ThrowingLoggerLocalWrite", diff --git a/test/dialcache-metrics.test.ts b/test/dialcache-metrics.test.ts index 23879dc..42b76dd 100644 --- a/test/dialcache-metrics.test.ts +++ b/test/dialcache-metrics.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it, vi } from "vitest"; import { CacheLayer, DialCache, + DialCacheKey, DialCacheKeyConfig, type CacheMetricLabels, type CoalescedMetricLabels, @@ -14,8 +15,9 @@ import { type SerializationMetricLabels, type Serializer, type ShadowValidationMetricLabels, + type StaleRecoveryMetricLabels, } from "../src/index.js"; -import { FakeRedis } from "./fake-redis.js"; +import { encodeFrame, FakeRedis } from "./fake-redis.js"; class RecordingMetrics implements DialCacheMetricsAdapter { readonly events: Array<{ readonly name: string; readonly labels: Record; readonly value?: number }> = []; @@ -44,6 +46,14 @@ class RecordingMetrics implements DialCacheMetricsAdapter { this.record("coalesced", labels); } + staleRecovery(labels: StaleRecoveryMetricLabels): void { + this.record("staleRecovery", labels); + } + + observeStaleRecoveryValueAge(labels: StaleRecoveryMetricLabels, seconds: number): void { + this.record("staleRecoveryValueAge", labels, seconds); + } + observeGet(labels: CacheMetricLabels, seconds: number): void { this.record("get", labels, seconds); } @@ -77,6 +87,13 @@ const remoteOnly = () => ramp: { [CacheLayer.REMOTE]: 100 }, }); +const staleRemoteOnly = () => + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 10, + }); + const tick = (): Promise => new Promise((resolve) => setTimeout(resolve, 0)); describe("DialCache observability metrics", () => { @@ -97,6 +114,8 @@ describe("DialCache observability metrics", () => { coalesced: vi.fn(() => thenable), shadowValidation: vi.fn(() => thenable), observeFutureTimestampOffset: vi.fn(() => thenable), + staleRecovery: vi.fn(() => thenable), + observeStaleRecoveryValueAge: vi.fn(() => thenable), observeGet: vi.fn(() => thenable), observeFallback: vi.fn(() => thenable), observeSerialization: vi.fn(() => thenable), @@ -135,6 +154,21 @@ describe("DialCache observability metrics", () => { outcome: "match", } satisfies ShadowValidationMetricLabels); isolatedMetrics.observeFutureTimestampOffset?.(labels, 0.001); + isolatedMetrics.staleRecovery?.({ + cacheNamespace: "urn", + useCase: "RejectingMetricsThenable", + keyType: "user_id", + outcome: "served", + } satisfies StaleRecoveryMetricLabels); + isolatedMetrics.observeStaleRecoveryValueAge?.( + { + cacheNamespace: "urn", + useCase: "RejectingMetricsThenable", + keyType: "user_id", + outcome: "served", + } satisfies StaleRecoveryMetricLabels, + 60, + ); isolatedMetrics.observeGet(labels, 0); isolatedMetrics.observeFallback(labels, 0); isolatedMetrics.observeSerialization({ ...labels, operation: "dump" }, 0); @@ -142,7 +176,7 @@ describe("DialCache observability metrics", () => { expect(then).not.toHaveBeenCalled(); await tick(); - expect(then).toHaveBeenCalledTimes(12); + expect(then).toHaveBeenCalledTimes(14); }); it("includes the configured cache namespace on every metric path", async () => { @@ -412,6 +446,43 @@ describe("DialCache observability metrics", () => { expect(events(metrics, "error", { useCase: "DisabledByPolicy" })).toHaveLength(0); }); + it("reports invalid stale-on-error policy without disabling fresh Redis", async () => { + const metrics = new RecordingMetrics(); + const redis = new FakeRedis(); + const dialcache = new DialCache({ + metrics, + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 60, + }), + }); + let calls = 0; + const getUser = dialcache.cached(async (userId: string) => ({ userId, calls: ++calls }), { + keyType: "user_id", + useCase: "InvalidStaleOnErrorPolicy", + cacheKey: (userId) => userId, + }); + + const first = await dialcache.enable(async () => await getUser("123")); + const second = await dialcache.enable(async () => await getUser("123")); + + expect(second).toEqual(first); + expect(calls).toBe(1); + expect(redis.getCalls).toBe(2); + expect(redis.setCalls).toBe(1); + expect(events(metrics, "error", { + useCase: "InvalidStaleOnErrorPolicy", + layer: CacheLayer.REMOTE, + error: "config_resolution", + inFallback: false, + })).toHaveLength(2); + expect(events(metrics, "disabled", { + useCase: "InvalidStaleOnErrorPolicy", + layer: CacheLayer.REMOTE, + })).toHaveLength(0); + }); + it("labels cache errors separately from fallback errors", async () => { // Given cache and fallback errors carry caller-defined names containing dynamic identifiers. const metrics = new RecordingMetrics(); @@ -470,6 +541,250 @@ describe("DialCache observability metrics", () => { ); }); + it("records one complete telemetry trail when stale recovery serves a retained value", async () => { + const metrics = new RecordingMetrics(); + const redis = new FakeRedis(); + const useCase = "StaleRecoveryServedMetrics"; + const staleValue = { userId: "123", version: 1 }; + const key = new DialCacheKey({ keyType: "user_id", id: "123", useCase }); + redis.setRaw( + `${key.urn}:dialcache-frame-v1`, + encodeFrame(staleValue, Date.now() - 2_000), + 10_000, + ); + const source = vi.fn(async () => { + throw new Error("source unavailable"); + }); + const logger = { debug: vi.fn(), warn: vi.fn(), error: vi.fn() }; + const dialcache = new DialCache({ + metrics, + redis: { client: redis, readTimeoutMs: 1_000 }, + logger, + shouldAttemptStaleRecovery: () => true, + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleRemoteOnly(), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(staleValue); + + expect(source).toHaveBeenCalledOnce(); + expect(events(metrics, "error", { useCase })).toEqual([ + { + name: "error", + labels: { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + layer: CacheLayer.REMOTE, + error: "fallback", + inFallback: true, + }, + }, + ]); + const remoteLabels = { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + layer: CacheLayer.REMOTE, + }; + expect(events(metrics, "fallback", { useCase })).toEqual([ + { name: "fallback", labels: remoteLabels, value: expect.any(Number) }, + ]); + expect(events(metrics, "miss", { useCase })).toEqual([ + { name: "miss", labels: remoteLabels }, + ]); + expect(events(metrics, "request", { useCase })).toEqual([ + { name: "request", labels: remoteLabels }, + ]); + expect(events(metrics, "get", { useCase })).toEqual([ + { name: "get", labels: remoteLabels, value: expect.any(Number) }, + ]); + expect(events(metrics, "staleRecovery", { useCase })).toEqual([ + { + name: "staleRecovery", + labels: { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + outcome: "served", + }, + }, + ]); + expect(events(metrics, "staleRecoveryValueAge", { useCase })).toEqual([ + { + name: "staleRecoveryValueAge", + labels: { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + outcome: "served", + }, + value: expect.any(Number), + }, + ]); + expect(events(metrics, "staleRecoveryValueAge", { useCase })[0]?.value).toBeGreaterThanOrEqual(2); + expect(logger.warn).not.toHaveBeenCalled(); + }); + + it("does not record stale value age when recovery misses", async () => { + const metrics = new RecordingMetrics(); + const redis = new FakeRedis(); + const useCase = "StaleRecoveryMissMetrics"; + const sourceError = new Error("source unavailable"); + const dialcache = new DialCache({ + metrics, + redis: { client: redis, readTimeoutMs: 1_000 }, + shouldAttemptStaleRecovery: () => true, + }); + const getUser = dialcache.cached(async (): Promise<{ readonly userId: string }> => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleRemoteOnly(), + }); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(events(metrics, "staleRecovery", { useCase })).toEqual([ + { + name: "staleRecovery", + labels: { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + outcome: "miss", + }, + }, + ]); + expect(events(metrics, "staleRecoveryValueAge", { useCase })).toHaveLength(0); + }); + + it("does not record stale value age when recovery deserialization fails", async () => { + const metrics = new RecordingMetrics(); + const redis = new FakeRedis(); + const useCase = "StaleRecoveryDeserializationErrorMetrics"; + const sourceError = new Error("source unavailable"); + const key = new DialCacheKey({ keyType: "user_id", id: "123", useCase }); + redis.setRaw( + `${key.urn}:dialcache-frame-v1`, + encodeFrame({ userId: "123" }, Date.now() - 2_000), + 10_000, + ); + const serializer: Serializer<{ readonly userId: string }> = { + dump: async (value) => JSON.stringify(value), + load: async () => { + throw new Error("cannot deserialize retained value"); + }, + }; + const dialcache = new DialCache({ + metrics, + redis: { client: redis, readTimeoutMs: 1_000 }, + shouldAttemptStaleRecovery: () => true, + }); + const getUser = dialcache.cached(async (): Promise<{ readonly userId: string }> => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleRemoteOnly(), + serializer, + }); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(events(metrics, "staleRecovery", { useCase })).toEqual([ + { + name: "staleRecovery", + labels: { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + outcome: "deserialization_error", + }, + }, + ]); + expect(events(metrics, "staleRecoveryValueAge", { useCase })).toHaveLength(0); + }); + + it("samples served stale value age after asynchronous deserialization completes", async () => { + vi.useFakeTimers({ toFake: ["Date"] }); + const clockStart = new Date("2026-08-29T12:00:00.000Z"); + vi.setSystemTime(clockStart); + + try { + const metrics = new RecordingMetrics(); + const redis = new FakeRedis(); + const useCase = "StaleRecoveryReturnTimeValueAgeMetrics"; + const staleValue = { userId: "123" }; + const key = new DialCacheKey({ keyType: "user_id", id: "123", useCase }); + redis.setRaw( + `${key.urn}:dialcache-frame-v1`, + encodeFrame(staleValue, clockStart.getTime() - 2_000), + 10_000, + ); + let markLoadStarted!: () => void; + const loadStarted = new Promise((resolve) => { + markLoadStarted = resolve; + }); + let releaseLoad!: () => void; + const loadGate = new Promise((resolve) => { + releaseLoad = resolve; + }); + const serializer: Serializer = { + dump: async (value) => JSON.stringify(value), + load: async () => { + markLoadStarted(); + await loadGate; + return staleValue; + }, + }; + const dialcache = new DialCache({ + metrics, + redis: { client: redis, readTimeoutMs: 1_000 }, + shouldAttemptStaleRecovery: () => true, + }); + const getUser = dialcache.cached(async (): Promise => { + throw new Error("source unavailable"); + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleRemoteOnly(), + serializer, + }); + + const result = dialcache.enable(async () => await getUser()); + await loadStarted; + expect(events(metrics, "staleRecoveryValueAge", { useCase })).toHaveLength(0); + + vi.setSystemTime(clockStart.getTime() + 3_000); + releaseLoad(); + + await expect(result).resolves.toEqual(staleValue); + expect(events(metrics, "staleRecoveryValueAge", { useCase })).toEqual([ + { + name: "staleRecoveryValueAge", + labels: { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + outcome: "served", + }, + value: 5, + }, + ]); + } finally { + vi.useRealTimers(); + } + }); + it("classifies config, Redis write, and serializer failures by stable operation", async () => { const metrics = new RecordingMetrics(); const logger = { debug: vi.fn(), warn: vi.fn(), error: vi.fn() }; @@ -689,9 +1004,11 @@ describe("DialCache observability metrics", () => { const writeFailure = new DialCache({ metrics, logger }); const writeLocalCache = (writeFailure as unknown as { - readonly localCache: { put: () => Promise }; + readonly localCache: { put: () => void }; }).localCache; - vi.spyOn(writeLocalCache, "put").mockRejectedValueOnce(new Error("local write failed")); + vi.spyOn(writeLocalCache, "put").mockImplementationOnce(() => { + throw new Error("local write failed"); + }); const writeValue = writeFailure.cached(async (id: string) => id, { keyType: "user_id", useCase: "LocalWriteErrorClassification", diff --git a/test/dialcache-observability-internals.test.ts b/test/dialcache-observability-internals.test.ts index d6ec4a4..74ce725 100644 --- a/test/dialcache-observability-internals.test.ts +++ b/test/dialcache-observability-internals.test.ts @@ -1,10 +1,10 @@ import { describe, expect, it } from "vitest"; import { CacheLayer, DialCacheKey, DialCacheKeyConfig } from "../src/index.js"; -import { LocalCache } from "../src/internal/local-cache.js"; -import { RedisCache } from "../src/internal/redis-cache.js"; -import { fetchKeyConfig, resolveLayerConfig } from "../src/internal/runtime-config.js"; -import { encodeFrame, FakeRedis } from "./fake-redis.js"; +import { + fetchKeyConfig, + resolveRemoteLayerConfigResult, +} from "../src/internal/runtime-config.js"; const key = (defaultConfig: DialCacheKeyConfig | null = DialCacheKeyConfig.enabled(60)) => new DialCacheKey({ keyType: "user_id", id: "123", useCase: "ObservabilityInternals", defaultConfig }); @@ -20,6 +20,7 @@ describe("DialCache observability internal compatibility paths", () => { ramp: 20, logMismatches: true, }, + staleOnErrorMaxAgeSec: 3_600, }); const cases = [ { @@ -39,6 +40,7 @@ describe("DialCache observability internal compatibility paths", () => { ramp: 80, logMismatches: true, }, + staleOnErrorMaxAgeSec: 3_600, }), }, { @@ -48,6 +50,7 @@ describe("DialCache observability internal compatibility paths", () => { shadow: { logMismatches: false, }, + staleOnErrorMaxAgeSec: 0, }), expected: new DialCacheKeyConfig({ requestLocal: true, @@ -58,10 +61,11 @@ describe("DialCache observability internal compatibility paths", () => { ramp: 20, logMismatches: false, }, + staleOnErrorMaxAgeSec: 0, }), }, { - runtime: new DialCacheKeyConfig({ shadow: {} }), + runtime: new DialCacheKeyConfig({ shadow: {}, staleOnErrorMaxAgeSec: 7_200 }), expected: new DialCacheKeyConfig({ requestLocal: true, coalesce: false, @@ -71,6 +75,7 @@ describe("DialCache observability internal compatibility paths", () => { ramp: 20, logMismatches: true, }, + staleOnErrorMaxAgeSec: 7_200, }), }, ]; @@ -96,72 +101,47 @@ describe("DialCache observability internal compatibility paths", () => { expect(merged?.ramp[CacheLayer.LOCAL]).toBe(50); }); - it("keeps LocalCache get/getIfPresent compatibility while exposing disabled reads", async () => { - // Given a local cache with enabled config and a second key with no config. - const cache = new LocalCache(async () => null, 10); - const enabledKey = key(); - const disabledKey = key(null); - let calls = 0; + it("keeps invalid stale-on-error policy diagnostic-only in remote resolution", () => { + const remoteKey = key(); - // When get() populates a value and getIfPresent() reads it back. - const first = await cache.get(enabledKey, async () => ({ calls: ++calls })); - const hit = await cache.getIfPresent<{ calls: number }>(enabledKey); - const disabled = await cache.getIfPresentResult(disabledKey); - - // Then compatibility helpers still behave like the pre-metrics API, and disabled state is explicit. - expect(first).toEqual({ calls: 1 }); - expect(hit).toEqual({ calls: 1 }); - expect(disabled).toEqual({ status: "disabled", reason: "policy_disabled" }); - expect(calls).toBe(1); - }); - - it("keeps RedisCache get compatibility and skips writes when remote config is disabled", async () => { - // Given a Redis cache with one valid stored frame and one key without remote config. - const redis = new FakeRedis(); - const redisCache = new RedisCache({ - configProvider: async () => null, - redis: { client: redis }, - metrics: null, + expect(resolveRemoteLayerConfigResult({ + config: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 3_600, + }), + key: remoteKey, + })).toEqual({ + status: "enabled", + config: { ttlSec: 60, ramp: 100, staleOnErrorMaxAgeSec: 3_600 }, }); - const enabledKey = key(); - const disabledKey = new DialCacheKey({ - keyType: "user_id", - id: "456", - useCase: "ObservabilityInternals", - defaultConfig: new DialCacheKeyConfig({ - ttlSec: { [CacheLayer.LOCAL]: 60 }, - ramp: { [CacheLayer.LOCAL]: 100 }, + expect(resolveRemoteLayerConfigResult({ + config: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 60, }), + key: remoteKey, + })).toEqual({ + status: "enabled", + config: { ttlSec: 60, ramp: 100, staleOnErrorMaxAgeSec: null }, + staleOnErrorConfigError: true, }); - redis.setRaw(`${enabledKey.urn}:dialcache-frame-v1`, encodeFrame({ source: "redis" })); - - // When the compatibility get() reads Redis and put() sees disabled remote config. - const hit = await redisCache.get<{ source: string }>(enabledKey); - await redisCache.put(disabledKey, { source: "fallback" }); - - // Then get() unwraps the value and the disabled remote write is skipped. - expect(hit).toEqual({ source: "redis" }); - expect(redis.values.has(`${disabledKey.urn}:dialcache-frame-v1`)).toBe(false); - }); - - it("preserves runtime-config edge behavior used by metrics", async () => { - // Given a config with an omitted ramp. - const missingRamp = new DialCacheKeyConfig({ ttlSec: { [CacheLayer.LOCAL]: 60 }, ramp: {} }); - - // When runtime config is resolved through compatibility paths. - const noConfig = await resolveLayerConfig({ - config: null, - key: key(null), - layer: CacheLayer.LOCAL, + expect(resolveRemoteLayerConfigResult({ + config: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 0, + }), + key: remoteKey, + })).toEqual({ + status: "enabled", + config: { ttlSec: 60, ramp: 100, staleOnErrorMaxAgeSec: null }, }); - const noRamp = await resolveLayerConfig({ - config: missingRamp, - key: key(), - layer: CacheLayer.LOCAL, + expect(resolveRemoteLayerConfigResult({ + config: new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 3_600 }), + key: remoteKey, + })).toEqual({ + status: "disabled", + reason: "policy_disabled", + staleOnErrorConfigError: true, }); - - // Then absent policy stays disabled and an omitted ramp defaults to 100%. - expect(noConfig).toBeNull(); - expect(noRamp).toEqual({ ttlSec: 60, ramp: 100 }); }); }); diff --git a/test/dialcache-redis.test.ts b/test/dialcache-redis.test.ts index 69663e1..951e8bd 100644 --- a/test/dialcache-redis.test.ts +++ b/test/dialcache-redis.test.ts @@ -225,10 +225,12 @@ describe("DialCache Redis TTL layer", () => { expect(metrics.miss).toHaveBeenCalledOnce(); }); - it("serves a future-dated untracked frame without consulting the reader clock", async () => { + it("rejects a future-dated untracked frame using the reader clock", async () => { + const nowMs = 1_700_000_000_000; + const nowSpy = vi.spyOn(Date, "now").mockReturnValue(nowMs); const cachedValue = { source: "redis" }; const redis: DialCacheRedisClient = { - read: vi.fn(async () => ({ payload: JSON.stringify(cachedValue), createdAtMs: Number.MAX_SAFE_INTEGER })), + read: vi.fn(async () => ({ payload: JSON.stringify(cachedValue), createdAtMs: nowMs + 1_250 })), write: vi.fn(async () => undefined), invalidate: vi.fn(async () => undefined), }; @@ -250,18 +252,21 @@ describe("DialCache Redis TTL layer", () => { }), serializer, }); - const nowSpy = vi.spyOn(Date, "now").mockImplementation(() => { - throw new Error("untracked reads must not consult the reader clock"); - }); - - await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(cachedValue); + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ source: "fallback" }); - expect(nowSpy).not.toHaveBeenCalled(); - expect(serializer.load).toHaveBeenCalledOnce(); - expect(serializer.dump).not.toHaveBeenCalled(); - expect(fallback).not.toHaveBeenCalled(); - expect(observeFutureTimestampOffset).not.toHaveBeenCalled(); - expect(metrics.miss).not.toHaveBeenCalled(); + expect(nowSpy).toHaveBeenCalledOnce(); + expect(serializer.load).not.toHaveBeenCalled(); + expect(fallback).toHaveBeenCalledOnce(); + expect(observeFutureTimestampOffset).toHaveBeenCalledWith( + { + cacheNamespace: "urn", + useCase: "RedisUntrackedFutureFrame", + keyType: "user_id", + layer: CacheLayer.REMOTE, + }, + 1.25, + ); + expect(metrics.miss).toHaveBeenCalledOnce(); }); it.each([ diff --git a/test/dialcache-shadow-confirmation.test.ts b/test/dialcache-shadow-confirmation.test.ts index bba47b9..68f3665 100644 --- a/test/dialcache-shadow-confirmation.test.ts +++ b/test/dialcache-shadow-confirmation.test.ts @@ -55,7 +55,6 @@ function deferred(): Deferred { type ReadStep = () => RedisCachePayload | null | Promise; -const SCRIPTED_FRAME_CREATED_AT_MS = 1_700_000_000_000; const MAX_TRACKED_REDIS_VALUE_TTL_MS = 60 * 60 * 1_000; class ScriptedRedis implements DialCacheRedisClient { @@ -63,7 +62,7 @@ class ScriptedRedis implements DialCacheRedisClient { readonly contexts: Array = []; readonly write = vi.fn(async (_request: RedisWriteRequest): Promise => undefined); readonly invalidate = vi.fn(async (_request: RedisInvalidationRequest): Promise => undefined); - frameCreatedAtMs = SCRIPTED_FRAME_CREATED_AT_MS; + frameCreatedAtMs = Date.now(); constructor(private readonly steps: ReadStep[]) {} @@ -75,7 +74,10 @@ class ScriptedRedis implements DialCacheRedisClient { throw new Error("Unexpected Redis read"); } const payload = await step(); - return payload === null ? null : { payload, createdAtMs: this.frameCreatedAtMs }; + if (payload === null) { + return null; + } + return { payload, createdAtMs: this.frameCreatedAtMs }; } } @@ -296,6 +298,45 @@ describe("DialCache Redis shadow confirmation", () => { expect(redis.invalidate).not.toHaveBeenCalled(); }); + it("confirms identical C1 bytes after the served C0 crosses its freshness age", async () => { + const nowMs = 1_700_000_000_000; + let readerNowMs = nowMs; + const nowSpy = vi.spyOn(Date, "now").mockImplementation(() => readerNowMs); + try { + const payload = JSON.stringify({ id: "123", version: 1 }); + const redis = new ScriptedRedis([() => payload, () => payload]); + redis.frameCreatedAtMs = nowMs - 999; + const metrics = new RecordingMetrics(); + const dialcache = createCache(redis, metrics); + const getUser = dialcache.cached(async () => { + readerNowMs = nowMs + 2; + return { id: "123", version: 2 }; + }, { + ...trackedOptions( + "ShadowConfirmationCrossesFreshAge", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 10, + shadow: { ramp: 100 }, + }), + ), + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ + id: "123", + version: 1, + }); + await waitForShadowEvents(metrics, 1); + + expect(metrics.shadowEvents.map(({ outcome }) => outcome)).toEqual(["mismatch"]); + expectTrackedReads(redis, 2); + } finally { + nowSpy.mockRestore(); + } + }); + it("does not log confirmed mismatches when logging is omitted", async () => { const payload = JSON.stringify({ id: "private-id", version: 1 }); const redis = new ScriptedRedis([() => payload, () => payload]); @@ -625,13 +666,13 @@ describe("DialCache Redis shadow confirmation", () => { it("confirms the same C1 payload when the reader clock steps backward after accepting C0", async () => { const nowMs = 1_700_000_000_000; + const payload = JSON.stringify({ id: "123", version: 1 }); + const redis = new ScriptedRedis([() => payload, () => payload]); + redis.frameCreatedAtMs = nowMs - 1_000; const nowSpy = vi.spyOn(Date, "now") .mockReturnValueOnce(nowMs) .mockReturnValue(nowMs - 2_000); try { - const payload = JSON.stringify({ id: "123", version: 1 }); - const redis = new ScriptedRedis([() => payload, () => payload]); - redis.frameCreatedAtMs = nowMs - 1_000; const metrics = new RecordingMetrics(); const dialcache = createCache(redis, metrics); const getUser = dialcache.cached(async () => ({ id: "123", version: 2 }), { @@ -689,7 +730,7 @@ describe("DialCache Redis shadow confirmation", () => { return payload; }, ]); - redis.frameCreatedAtMs = nowMs - 90_000; + redis.frameCreatedAtMs = nowMs - 30_000; const metrics = new RecordingMetrics(); const dialcache = createCache(redis, metrics); const getUser = dialcache.cached(async () => ({ id: "123", version: 2 }), { @@ -702,7 +743,7 @@ describe("DialCache Redis shadow confirmation", () => { expect(metrics.shadowEvents.map(({ outcome }) => outcome)).toEqual(["mismatch"]); expect(metrics.shadowAgeEvents).toHaveLength(1); - expect(metrics.shadowAgeEvents[0]?.seconds).toBe(90); + expect(metrics.shadowAgeEvents[0]?.seconds).toBe(30); expect(metrics.shadowAgeEvents[0]?.labels).toMatchObject({ useCase: "ShadowMismatchValueAge", keyType: "user_id", @@ -1073,6 +1114,52 @@ describe("DialCache Redis shadow confirmation", () => { expectTrackedReads(redis, 1); }); + it("never serves retained stale data when remote serving is ramped down", async () => { + const stalePayload = JSON.stringify({ id: "123", source: "stale-cache" }); + const redis = new ScriptedRedis([() => stalePayload]); + redis.frameCreatedAtMs = Date.now() - 120_000; + const metrics = new RecordingMetrics(); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const source = vi.fn(async () => { + throw sourceError; + }); + const dialcache = createCache(redis, metrics); + const getUser = dialcache.cached(source, { + ...trackedOptions("ShadowDarkRetainedStale", new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + ramp: { [CacheLayer.REMOTE]: 0 }, + staleOnErrorMaxAgeSec: 3_600, + shadow: { ramp: 100 }, + })), + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + await waitForShadowEvents(metrics, 1); + + expect(source).toHaveBeenCalledOnce(); + expectTrackedReads(redis, 1); + expect(redis.write).not.toHaveBeenCalled(); + expect(redis.invalidate).not.toHaveBeenCalled(); + expect(metrics.shadowEvents).toEqual([{ + cacheNamespace: "urn", + useCase: "ShadowDarkRetainedStale", + keyType: "user_id", + outcome: "source_error", + }]); + expect(metrics.ordinaryEvents.filter(({ name, labels }) => + name === "request" && labels.layer === REMOTE_SHADOW_CACHE_LAYER + )).toHaveLength(1); + expect(metrics.ordinaryEvents.filter(({ name, labels }) => + name === "get" && labels.layer === REMOTE_SHADOW_CACHE_LAYER + )).toHaveLength(1); + expect(metrics.ordinaryEvents.filter(({ name, labels }) => + name === "disabled" + && labels.layer === CacheLayer.REMOTE + && labels.reason === "ramped_down" + )).toHaveLength(1); + }); + it("does not misclassify a source-propagated FallbackTimeoutError as its own timeout", async () => { const redis = new ScriptedRedis([() => JSON.stringify({ id: "123", source: "cache" })]); const metrics = new RecordingMetrics(); @@ -1210,7 +1297,7 @@ describe("DialCache Redis shadow confirmation", () => { it.each([ { name: "tracked", tracked: true }, { name: "untracked", tracked: false }, - ])("fills a clean $name dark Redis miss and attributes the read and write to remote_shadow", async ({ + ])("retains a clean $name dark Redis miss through M and attributes the work to remote_shadow", async ({ name, tracked, }) => { @@ -1223,7 +1310,12 @@ describe("DialCache Redis shadow confirmation", () => { useCase: `ShadowDarkMissFill${name}`, cacheKey: () => "123", trackForInvalidation: tracked, - defaultConfig: remoteConfig(0), + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + ramp: { [CacheLayer.REMOTE]: 0 }, + staleOnErrorMaxAgeSec: 3_600, + shadow: { ramp: 100 }, + }), }); await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123" }); @@ -1239,7 +1331,7 @@ describe("DialCache Redis shadow confirmation", () => { } expect(redis.write).toHaveBeenCalledOnce(); expect(redis.write).toHaveBeenCalledWith(expect.objectContaining({ - cacheTtlMs: 60_000, + cacheTtlMs: 3_600_000, value: JSON.stringify({ id: "123" }), })); expect(Object.hasOwn(redis.write.mock.calls[0]?.[0] ?? {}, "watermarkKey")).toBe(false); diff --git a/test/dialcache-shadow-validation.test.ts b/test/dialcache-shadow-validation.test.ts index dfe1d30..57e5751 100644 --- a/test/dialcache-shadow-validation.test.ts +++ b/test/dialcache-shadow-validation.test.ts @@ -331,7 +331,7 @@ describe("DialCache Redis shadow validation", () => { id: "123", useCase, payload: JSON.stringify({ id: "123", version: 1 }), - createdAtMs: nowMs - 120_000, + createdAtMs: nowMs - 50_000, }); const dialcache = createShadowCache(redis, metrics); const getUser = dialcache.cached(async () => ({ id: "123", version: 2 }), { @@ -344,7 +344,7 @@ describe("DialCache Redis shadow validation", () => { expect(metrics.shadowEvents[0]?.outcome).toBe("mismatch"); expect(metrics.shadowAgeEvents).toHaveLength(1); - expect(metrics.shadowAgeEvents[0]?.seconds).toBe(120); + expect(metrics.shadowAgeEvents[0]?.seconds).toBe(50); expect(metrics.shadowAgeEvents[0]?.labels).toMatchObject({ useCase, outcome: "mismatch" }); } finally { nowSpy.mockRestore(); @@ -391,7 +391,7 @@ describe("DialCache Redis shadow validation", () => { } }); - it("skips a non-finite value-age observation from an untracked custom client", async () => { + it("rejects a non-finite untracked timestamp before shadow validation", async () => { const redis = new FakeRedis(); const metrics = new RecordingMetrics(); const useCase = "ShadowNonFiniteValueAge"; @@ -408,7 +408,8 @@ describe("DialCache Redis shadow validation", () => { return frame === null ? null : { ...frame, createdAtMs: Number.POSITIVE_INFINITY }; }); const dialcache = createShadowCache(redis, metrics); - const getUser = dialcache.cached(async () => cachedValue, { + const source = vi.fn(async () => cachedValue); + const getUser = dialcache.cached(source, { keyType: "user_id", useCase, cacheKey: () => "123", @@ -416,11 +417,13 @@ describe("DialCache Redis shadow validation", () => { }); expect(await dialcache.enable(async () => await getUser())).toEqual(cachedValue); - await waitForShadowEvents(metrics, 1); + await nextImmediate(); - expect(metrics.shadowEvents[0]?.outcome).toBe("match"); + expect(source).toHaveBeenCalledOnce(); + expect(metrics.shadowEvents).toEqual([]); expect(metrics.shadowAgeEvents).toEqual([]); expect(metrics.futureTimestampEvents).toEqual([]); + expect(redis.setCalls).toBe(1); }); it("re-deserializes the retained payload instead of comparing a caller-mutated hit", async () => { diff --git a/test/dialcache-stale-on-error.test.ts b/test/dialcache-stale-on-error.test.ts new file mode 100644 index 0000000..dc8b633 --- /dev/null +++ b/test/dialcache-stale-on-error.test.ts @@ -0,0 +1,1181 @@ +import { performance } from "node:perf_hooks"; + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +import { + CacheLayer, + DialCache, + DialCacheKey, + DialCacheKeyConfig, + type CachedOptions, + type DecodedRedisFrame, + type DialCacheMetricsAdapter, + type RedisReadContext, + type RedisReadRequest, + type Serializer, +} from "../src/index.js"; +import { MARKER_ZSTD_UTF8 } from "../src/internal/compression.js"; +import { RedisCache } from "../src/internal/redis-cache.js"; +import { decodeFrame, encodeFrame, FakeRedis } from "./fake-redis.js"; + +const FRESH_TTL_SEC = 1; +const MAX_AGE_SEC = 10; +const SOURCE_UNAVAILABLE = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); +const allowStaleRecovery = (): boolean => true; + +class RecordingRedis extends FakeRedis { + readonly readRequests: RedisReadRequest[] = []; + readonly readContexts: Array = []; + + override async read( + request: RedisReadRequest, + context?: RedisReadContext, + ): Promise { + this.readRequests.push(request); + this.readContexts.push(context); + return await super.read(request); + } +} + +class HangingReadRedis extends FakeRedis { + readonly readRequests: RedisReadRequest[] = []; + readonly readContexts: Array = []; + + constructor(private readonly hangOnCall: number) { + super(); + } + + override async read( + request: RedisReadRequest, + context?: RedisReadContext, + ): Promise { + this.readRequests.push(request); + this.readContexts.push(context); + if (this.readRequests.length === this.hangOnCall) { + return await new Promise(() => undefined); + } + return await super.read(request); + } +} + +interface Deferred { + readonly promise: Promise; + resolve(value: T): void; + reject(error: unknown): void; +} + +function deferred(): Deferred { + let resolve!: (value: T) => void; + let reject!: (error: unknown) => void; + const promise = new Promise((resolvePromise, rejectPromise) => { + resolve = resolvePromise; + reject = rejectPromise; + }); + return { promise, resolve, reject }; +} + +function staleConfig(options: { readonly local?: boolean; readonly requestLocal?: boolean } = {}): DialCacheKeyConfig { + return new DialCacheKeyConfig({ + ttlSec: { + ...(options.local ? { [CacheLayer.LOCAL]: 60 } : {}), + [CacheLayer.REMOTE]: FRESH_TTL_SEC, + }, + ramp: { + ...(options.local ? { [CacheLayer.LOCAL]: 100 } : {}), + [CacheLayer.REMOTE]: 100, + }, + ...(options.requestLocal ? { requestLocal: true } : {}), + staleOnErrorMaxAgeSec: MAX_AGE_SEC, + }); +} + +function redisValueKey(useCase: string, id = "123", trackForInvalidation = false): string { + const key = new DialCacheKey({ keyType: "user_id", id, useCase, trackForInvalidation }); + return `${key.urn}:dialcache-frame-v1`; +} + +function watermarkKey(id = "123"): string { + return `{urn:user_id:${id}}#watermark`; +} + +function seedStale(redis: FakeRedis, useCase: string, value: unknown, trackForInvalidation = false): void { + redis.setRaw( + redisValueKey(useCase, "123", trackForInvalidation), + encodeFrame(value, Date.now() - 2_000), + MAX_AGE_SEC * 1_000, + ); +} + +function expectNativeReadCount(redis: RecordingRedis | HangingReadRedis, count: number): void { + expect(redis.readRequests).toHaveLength(count); + expect(redis.readRequests.every((request) => !Object.hasOwn(request, "maxAgeMs"))).toBe(true); +} + +function recordingMetrics(): { + readonly metrics: DialCacheMetricsAdapter; + readonly staleRecovery: ReturnType; + readonly shadowValidation: ReturnType; +} { + const staleRecovery = vi.fn(); + const shadowValidation = vi.fn(); + return { + staleRecovery, + shadowValidation, + metrics: { + request: vi.fn(), + miss: vi.fn(), + disabled: vi.fn(), + error: vi.fn(), + invalidation: vi.fn(), + shadowValidation, + staleRecovery, + observeGet: vi.fn(), + observeFallback: vi.fn(), + observeSerialization: vi.fn(), + observeSize: vi.fn(), + }, + }; +} + +function setupDefaultStaleUseCase unknown>( + useCase: string, + source: Fn, + options: { readonly defaultConfig?: DialCacheKeyConfig } = {}, +) { + const redis = new RecordingRedis(); + const recordedMetrics = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics: recordedMetrics.metrics, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + // Every fixture source is JSON-compatible; the generic helper cannot retain + // that conditional-type proof across all callers. + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: options.defaultConfig ?? staleConfig(), + } as CachedOptions); + return { redis, dialcache, getUser, staleRecovery: recordedMetrics.staleRecovery }; +} + +function rejectionReason(result: PromiseSettledResult): unknown { + if (result.status !== "rejected") { + throw new Error("Expected rejection"); + } + return result.reason; +} + +describe("DialCache stale-on-error recovery", () => { + beforeEach(() => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-08-02T12:00:00.000Z")); + const clockOriginMs = Date.now(); + vi.spyOn(performance, "now").mockImplementation(() => Date.now() - clockOriginMs); + }); + + afterEach(() => { + vi.useRealTimers(); + vi.restoreAllMocks(); + }); + + it("serves the raw stale candidate retained by the initial Redis read without publication", async () => { + const useCase = "StaleRecoveryServed"; + const staleValue = { id: "123", version: 1 }; + const source = vi.fn((): { readonly id: string; readonly version: number } => { + throw SOURCE_UNAVAILABLE; + }); + const { redis, dialcache, getUser, staleRecovery } = setupDefaultStaleUseCase(useCase, source); + seedStale(redis, useCase, staleValue); + const ttlBefore = redis.ttlMs(redisValueKey(useCase)); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(staleValue); + + expect(source).toHaveBeenCalledOnce(); + expectNativeReadCount(redis, 1); + expect(redis.setCalls).toBe(0); + expect(redis.ttlMs(redisValueKey(useCase))).toBe(ttlBefore); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith({ + cacheNamespace: "urn", + useCase, + keyType: "user_id", + outcome: "served", + }); + }); + + it.each([ + { boundary: "F - 1 ms", ageMs: FRESH_TTL_SEC * 1_000 - 1, expectedStatus: "hit" }, + { boundary: "F", ageMs: FRESH_TTL_SEC * 1_000, expectedStatus: "retained" }, + { boundary: "M - 1 ms", ageMs: MAX_AGE_SEC * 1_000 - 1, expectedStatus: "retained" }, + { boundary: "M", ageMs: MAX_AGE_SEC * 1_000, expectedStatus: "miss" }, + ] as const)("classifies an initial frame at $boundary as $expectedStatus", async ({ + boundary, + ageMs, + expectedStatus, + }) => { + const useCase = `StaleRecoveryInitialBoundary${boundary.replaceAll(/[^A-Za-z0-9]/g, "")}`; + const retained = { id: "123", version: 1 }; + const redis = new RecordingRedis(); + const redisCache = new RedisCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics: null, + }); + const key = new DialCacheKey({ keyType: "user_id", id: "123", useCase }); + redis.setRaw( + redisValueKey(useCase), + encodeFrame(retained, Date.now() - ageMs), + // Keep the frame physically present beyond M so exact-M coverage proves + // logical classification rather than FakeRedis expiry. + (MAX_AGE_SEC + 1) * 1_000, + ); + + const result = await redisCache.getWithResolvedConfig(key, { + ttlSec: FRESH_TTL_SEC, + ramp: 100, + staleOnErrorMaxAgeSec: MAX_AGE_SEC, + }); + + expect(result.status).toBe(expectedStatus); + if (result.status === "hit") { + expect(result.value).toEqual(retained); + } else if (result.status === "retained") { + expect(result.frame.createdAtMs).toBe(Date.now() - ageMs); + } else { + expect(result.reason).toBe("cache_miss"); + } + expectNativeReadCount(redis, 1); + expect(redis.ttlMs(redisValueKey(useCase))).toBeGreaterThan(MAX_AGE_SEC * 1_000); + }); + + it("fails closed on a future-dated frame without retaining it for recovery", async () => { + const useCase = "StaleRecoveryFutureFrame"; + const source = vi.fn(async () => { + throw SOURCE_UNAVAILABLE; + }); + const { redis, dialcache, getUser, staleRecovery } = setupDefaultStaleUseCase(useCase, source); + redis.setRaw( + redisValueKey(useCase), + encodeFrame({ id: "123", version: 1 }, Date.now() + 1), + MAX_AGE_SEC * 1_000, + ); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(SOURCE_UNAVAILABLE); + + expect(source).toHaveBeenCalledOnce(); + expectNativeReadCount(redis, 1); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "miss" })); + }); + + it("keeps feature-off writes at the fresh TTL and never performs a recovery read", async () => { + const useCase = "StaleRecoveryFeatureOff"; + const source = vi.fn<() => Promise<{ readonly id: string; readonly version: number }>>() + .mockResolvedValueOnce({ id: "123", version: 1 }) + .mockRejectedValueOnce(SOURCE_UNAVAILABLE); + const { redis, dialcache, getUser, staleRecovery } = setupDefaultStaleUseCase(useCase, source, { + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + }), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123", version: 1 }); + expect(redis.ttlMs(redisValueKey(useCase))).toBe(1_000); + await vi.advanceTimersByTimeAsync(1_000); + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(SOURCE_UNAVAILABLE); + expectNativeReadCount(redis, 2); + expect(redis.setCalls).toBe(1); + expect(staleRecovery).not.toHaveBeenCalled(); + }); + + it("does not deserialize a retained candidate when the source succeeds", async () => { + const useCase = "StaleRecoverySourceSuccess"; + const sourceValue = { id: "123", version: 2 }; + const source = vi.fn(async () => sourceValue); + const redis = new RecordingRedis(); + const { metrics, staleRecovery } = recordingMetrics(); + const serializer: Serializer = { + dump: vi.fn(async (value) => JSON.stringify(value)), + load: vi.fn(async () => { + throw new Error("retained candidate should stay raw while the source succeeds"); + }), + }; + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + serializer, + }); + seedStale(redis, useCase, { id: "123", version: 1 }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toBe(sourceValue); + + expect(source).toHaveBeenCalledOnce(); + expectNativeReadCount(redis, 1); + expect(redis.setCalls).toBe(1); + expect(redis.ttlMs(redisValueKey(useCase))).toBe(10_000); + expect(serializer.load).not.toHaveBeenCalled(); + const refreshedFrame = decodeFrame(redis.raw(redisValueKey(useCase))); + expect(refreshedFrame.createdAtMs).toBe(Date.now()); + expect(JSON.parse(refreshedFrame.payload as string)).toEqual(sourceValue); + expect(staleRecovery).not.toHaveBeenCalled(); + }); + + it("does not deserialize or record recovery when the classifier denies a retained candidate", async () => { + const useCase = "StaleRecoveryClassifierDenied"; + const retained = { id: "123", version: 1 }; + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const redis = new RecordingRedis(); + const { metrics, staleRecovery } = recordingMetrics(); + const denyRecovery = vi.fn(() => false); + const serializer: Serializer = { + dump: vi.fn(async (value) => JSON.stringify(value)), + load: vi.fn(async () => retained), + }; + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async (): Promise => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + serializer, + shouldAttemptStaleRecovery: denyRecovery, + }); + seedStale(redis, useCase, retained); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(denyRecovery).toHaveBeenCalledOnce(); + expect(denyRecovery).toHaveBeenCalledWith(sourceError); + expect(serializer.load).not.toHaveBeenCalled(); + expect(serializer.dump).not.toHaveBeenCalled(); + expect(staleRecovery).not.toHaveBeenCalled(); + expectNativeReadCount(redis, 1); + }); + + it("rejects with the original source error when a retained frame reaches the exact maximum age", async () => { + const useCase = "StaleRecoveryCrossesMaximumDuringSource"; + const redis = new RecordingRedis(); + const valueKey = redisValueKey(useCase); + redis.setRaw( + valueKey, + encodeFrame({ id: "123", version: 1 }, Date.now() - 9_000), + 20_000, + ); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const sourceStarted = deferred(); + const sourceGate = deferred<{ readonly id: string; readonly version: number }>(); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const result = Promise.allSettled([dialcache.enable(async () => await getUser())]); + await sourceStarted.promise; + await vi.advanceTimersByTimeAsync(1_000); + sourceGate.reject(sourceError); + const [settled] = await result; + + expect(rejectionReason(settled!)).toBe(sourceError); + expectNativeReadCount(redis, 1); + expect(redis.ttlMs(valueKey)).toBe(19_000); + expect(redis.setCalls).toBe(0); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "miss" })); + }); + + it("uses one runtime policy snapshot for the initial read and delayed recovery", async () => { + const useCase = "StaleRecoveryRuntimePolicySnapshot"; + const redis = new RecordingRedis(); + const staleValue = { id: "123", version: 1 }; + seedStale(redis, useCase, staleValue); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const sourceStarted = deferred(); + const sourceGate = deferred<{ readonly id: string; readonly version: number }>(); + let freshTtlSec = 1; + let maxAgeSec = 3; + let remoteReadTimeoutMs = 25; + const cacheConfigProvider = vi.fn(async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: freshTtlSec }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: maxAgeSec, + remoteReadTimeoutMs, + })); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + }); + + const result = Promise.allSettled([dialcache.enable(async () => await getUser())]); + await sourceStarted.promise; + freshTtlSec = 4; + maxAgeSec = 20; + remoteReadTimeoutMs = 75; + sourceGate.reject(sourceError); + const [settled] = await result; + + expect(settled).toEqual({ status: "fulfilled", value: staleValue }); + expect(cacheConfigProvider).toHaveBeenCalledOnce(); + expectNativeReadCount(redis, 1); + expect(redis.readContexts.map((context) => context?.timeoutMs)).toEqual([25]); + }); + + it("applies the current runtime fresh age to an existing retained frame", async () => { + const useCase = "StaleRecoveryRuntimeFreshAge"; + const redis = new RecordingRedis(); + const retainedValue = { id: "123", version: 1 }; + redis.setRaw( + redisValueKey(useCase), + encodeFrame(retainedValue, Date.now() - 3_000), + MAX_AGE_SEC * 1_000, + ); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const source = vi.fn(async () => { + throw sourceError; + }); + let freshTtlSec = 4; + const cacheConfigProvider = vi.fn(async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: freshTtlSec }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: MAX_AGE_SEC, + })); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(retainedValue); + expect(source).not.toHaveBeenCalled(); + + freshTtlSec = 2; + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(retainedValue); + expect(source).toHaveBeenCalledOnce(); + + freshTtlSec = 4; + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(retainedValue); + + expect(source).toHaveBeenCalledOnce(); + expect(cacheConfigProvider).toHaveBeenCalledTimes(3); + expectNativeReadCount(redis, 3); + }); + + it("caps tracked stale retention at one hour without creating a watermark", async () => { + const useCase = "StaleRecoveryTrackedRetentionCap"; + const redis = new RecordingRedis(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 } }); + const getUser = dialcache.cached(async () => ({ id: "123" }), { + keyType: "user_id", + useCase, + cacheKey: () => "123", + trackForInvalidation: true, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 7_200, + }), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123" }); + + expect(redis.ttlMs(redisValueKey(useCase, "123", true))).toBe(60 * 60 * 1_000); + expect(redis.ttlMs(watermarkKey())).toBe(-2); + expect(redis.readRequests).toEqual([ + { valueKey: redisValueKey(useCase, "123", true), watermarkKey: watermarkKey() }, + ]); + }); + + it("does not clamp the configured tracked logical maximum age to the physical one-hour cap", async () => { + const useCase = "StaleRecoveryTrackedLogicalMaximum"; + const redis = new RecordingRedis(); + const retained = { id: "123", version: 1 }; + redis.setRaw( + redisValueKey(useCase, "123", true), + encodeFrame(retained, Date.now() - 3_700_000), + 10_000, + ); + redis.setRaw(watermarkKey(), "0", 10_000); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const getUser = dialcache.cached(async (): Promise => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + trackForInvalidation: true, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 7_200, + }), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(retained); + expectNativeReadCount(redis, 1); + }); + + it.each([ + ["object", Object.freeze({ code: "SOURCE_OBJECT" })], + ["null", null], + ["undefined", undefined], + ] as const)("preserves an arbitrary %s rejection when recovery misses", async (_name, sourceError) => { + const useCase = `StaleRecoveryIdentity${_name}`; + const source = vi.fn(async () => { + throw sourceError; + }); + const { redis, dialcache, getUser, staleRecovery } = setupDefaultStaleUseCase(useCase, source); + + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expectNativeReadCount(redis, 1); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "miss" })); + }); + + it("never attempts recovery after the initial Redis read fails", async () => { + const useCase = "StaleRecoveryInitialReadError"; + const redis = new RecordingRedis(); + redis.failGet = true; + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + logger: { debug: vi.fn(), error: vi.fn(), warn: vi.fn() }, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async () => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests).toHaveLength(1); + expect(staleRecovery).not.toHaveBeenCalled(); + }); + + it("serves the retained candidate when Redis becomes unavailable during the source attempt", async () => { + const useCase = "StaleRecoveryRedisUnavailableAfterRead"; + const redis = new RecordingRedis(); + const retained = { id: "123", version: 1 }; + seedStale(redis, useCase, retained); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + logger: { debug: vi.fn(), error: vi.fn(), warn: vi.fn() }, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async () => { + redis.failGet = true; + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(retained); + + expect(redis.readRequests).toHaveLength(1); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "served" })); + }); + + it("serves the retained candidate after its Redis key expires during the source attempt", async () => { + const useCase = "StaleRecoveryRedisExpiryAfterRead"; + const redis = new RecordingRedis(); + const retained = { id: "123", version: 1 }; + redis.setRaw( + redisValueKey(useCase), + encodeFrame(retained, Date.now() - 2_000), + 500, + ); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const sourceStarted = deferred(); + const sourceGate = deferred<{ readonly id: string; readonly version: number }>(); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 10 }, + metrics, + logger: { debug: vi.fn(), error: vi.fn(), warn: vi.fn() }, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const result = dialcache.enable(async () => await getUser()); + await sourceStarted.promise; + await vi.advanceTimersByTimeAsync(500); + expect(redis.ttlMs(redisValueKey(useCase))).toBe(0); + sourceGate.reject(sourceError); + + await expect(result).resolves.toEqual(retained); + expect(redis.readRequests).toHaveLength(1); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "served" })); + }); + + it("does not retry Redis when the initial read times out", async () => { + const useCase = "StaleRecoveryInitialReadTimeout"; + const redis = new HangingReadRedis(1); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 10 }, + metrics, + logger: { debug: vi.fn(), error: vi.fn(), warn: vi.fn() }, + }); + const getUser = dialcache.cached(async () => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const result = Promise.allSettled([dialcache.enable(async () => await getUser())]); + await vi.advanceTimersByTimeAsync(10); + const [settled] = await result; + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests).toHaveLength(1); + expect(redis.readContexts[0]?.signal.aborted).toBe(true); + expect(staleRecovery).not.toHaveBeenCalled(); + }); + + it("serves stale after the fallback deadline and ignores the late source result", async () => { + const useCase = "StaleRecoveryFallbackTimeout"; + const redis = new RecordingRedis(); + seedStale(redis, useCase, { id: "123", version: 1 }); + const sourceGate = deferred<{ readonly id: string; readonly version: number }>(); + const sourceStarted = deferred(); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 100 }, metrics }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + fallbackTimeoutMs: 10, + defaultConfig: staleConfig(), + }); + + const result = dialcache.enable(async () => await getUser()); + await sourceStarted.promise; + await vi.advanceTimersByTimeAsync(10); + + await expect(result).resolves.toEqual({ id: "123", version: 1 }); + expect(redis.readRequests).toHaveLength(1); + expect(redis.setCalls).toBe(0); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "served" })); + + sourceGate.resolve({ id: "123", version: 2 }); + await vi.advanceTimersByTimeAsync(0); + expect(redis.setCalls).toBe(0); + }); + + it("classifies recovery deserialization failure and never retries a normal deserialization miss", async () => { + const recoveryUseCase = "StaleRecoveryDeserializeError"; + const normalUseCase = "StaleRecoveryInitialDeserializeError"; + const redis = new RecordingRedis(); + seedStale(redis, recoveryUseCase, { id: "123" }); + redis.setRaw( + redisValueKey(normalUseCase), + encodeFrame({ id: "123" }, Date.now()), + MAX_AGE_SEC * 1_000, + ); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const serializer: Serializer<{ readonly id: string }> = { + dump: vi.fn(async (value) => JSON.stringify(value)), + load: vi.fn(async () => { + throw new Error("cannot decode"); + }), + }; + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const recover = dialcache.cached(async (): Promise<{ readonly id: string }> => { + throw sourceError; + }, { + keyType: "user_id", + useCase: recoveryUseCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + serializer, + }); + const initialFailure = dialcache.cached(async (): Promise<{ readonly id: string }> => { + throw sourceError; + }, { + keyType: "user_id", + useCase: normalUseCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + serializer, + }); + + const [recoverySettled] = await Promise.allSettled([dialcache.enable(async () => await recover())]); + expect(rejectionReason(recoverySettled!)).toBe(sourceError); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ + useCase: recoveryUseCase, + outcome: "deserialization_error", + })); + + const readsBeforeInitialFailure = redis.readRequests.length; + const [initialSettled] = await Promise.allSettled([ + dialcache.enable(async () => await initialFailure()), + ]); + expect(rejectionReason(initialSettled!)).toBe(sourceError); + expect(redis.readRequests).toHaveLength(readsBeforeInitialFailure + 1); + expect(staleRecovery).toHaveBeenCalledTimes(1); + expect(staleRecovery).not.toHaveBeenCalledWith(expect.objectContaining({ useCase: normalUseCase })); + }); + + it("rejects with the source error when asynchronous recovery deserialization crosses M", async () => { + const useCase = "StaleRecoveryDeserializeCrossesMaximum"; + const redis = new RecordingRedis(); + const retained = { id: "123", version: 1 }; + redis.setRaw( + redisValueKey(useCase), + encodeFrame(retained, Date.now() - 9_000), + 20_000, + ); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const loadStarted = deferred(); + const loadGate = deferred(); + const serializer: Serializer = { + dump: vi.fn(async (value) => JSON.stringify(value)), + load: vi.fn(async () => { + loadStarted.resolve(); + return await loadGate.promise; + }), + }; + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async (): Promise => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + serializer, + }); + + const result = Promise.allSettled([dialcache.enable(async () => await getUser())]); + await loadStarted.promise; + await vi.advanceTimersByTimeAsync(1_000); + loadGate.resolve(retained); + const [settled] = await result; + + expect(rejectionReason(settled!)).toBe(sourceError); + expectNativeReadCount(redis, 1); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "miss" })); + }); + + it("serves the initial tracked snapshot when invalidation races with the source attempt", async () => { + const useCase = "StaleRecoveryInvalidatedDuringSource"; + const redis = new RecordingRedis(); + seedStale(redis, useCase, { id: "123", version: 1 }, true); + redis.setRaw(watermarkKey(), "0", MAX_AGE_SEC * 1_000 + 60_000); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const sourceStarted = deferred(); + const sourceGate = deferred<{ readonly id: string; readonly version: number }>(); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + trackForInvalidation: true, + defaultConfig: staleConfig(), + }); + + const result = dialcache.enable(async () => await getUser()); + await sourceStarted.promise; + await dialcache.invalidateRemote("user_id", "123"); + sourceGate.reject(sourceError); + + await expect(result).resolves.toEqual({ id: "123", version: 1 }); + expect(redis.readRequests).toHaveLength(1); + expect(redis.readRequests.every(({ watermarkKey: key }) => key === watermarkKey())).toBe(true); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "served" })); + }); + + it("blocks a tracked candidate invalidated before the initial Redis snapshot", async () => { + const useCase = "StaleRecoveryInvalidatedBeforeRead"; + const redis = new RecordingRedis(); + seedStale(redis, useCase, { id: "123", version: 1 }, true); + redis.setRaw(watermarkKey(), String(Date.now()), MAX_AGE_SEC * 1_000 + 60_000); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async (): Promise<{ readonly id: string; readonly version: number }> => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + trackForInvalidation: true, + defaultConfig: staleConfig(), + }); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(redis.readRequests).toHaveLength(1); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "miss" })); + }); + + it("serves the initial candidate when Redis is refreshed during the source attempt", async () => { + const useCase = "StaleRecoveryRefreshedDuringSource"; + const redis = new RecordingRedis(); + const initial = { id: "123", version: 1 }; + const refreshed = { id: "123", version: 2 }; + seedStale(redis, useCase, initial); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const sourceStarted = deferred(); + const sourceGate = deferred(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const result = dialcache.enable(async () => await getUser()); + await sourceStarted.promise; + redis.setRaw(redisValueKey(useCase), encodeFrame(refreshed), MAX_AGE_SEC * 1_000); + sourceGate.reject(sourceError); + + await expect(result).resolves.toEqual(initial); + expect(redis.readRequests).toHaveLength(1); + expect(JSON.parse(decodeFrame(redis.raw(redisValueKey(useCase))).payload as string)).toEqual(refreshed); + }); + + it("recovers cached undefined without starting shadow validation", async () => { + const useCase = "StaleRecoveryUndefinedNoShadow"; + const redis = new RecordingRedis(); + redis.setRaw( + redisValueKey(useCase, "123", true), + encodeFrame("__dialcache_json_undefined_v1__", Date.now() - 2_000), + MAX_AGE_SEC * 1_000, + ); + redis.setRaw(watermarkKey(), "0", MAX_AGE_SEC * 1_000 + 60_000); + const { metrics, staleRecovery, shadowValidation } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getOptional = dialcache.cached(async (): Promise => { + throw new Error("source unavailable"); + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + trackForInvalidation: true, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: MAX_AGE_SEC, + shadow: { ramp: 100 }, + }), + }); + + await expect(dialcache.enable(async () => await getOptional())).resolves.toBeUndefined(); + await vi.advanceTimersByTimeAsync(0); + + expect(redis.readRequests).toHaveLength(1); + expect(redis.setCalls).toBe(0); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "served" })); + expect(shadowValidation).not.toHaveBeenCalled(); + }); + + it("applies a lowered runtime recovery maximum to an existing retained frame immediately", async () => { + const useCase = "StaleRecoveryLoweredRuntimeMaximum"; + const redis = new RecordingRedis(); + redis.setRaw( + redisValueKey(useCase), + encodeFrame({ id: "123", version: 1 }, Date.now() - 5_000), + 10_000, + ); + let maxAgeSec = 10; + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: maxAgeSec, + }), + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(async (): Promise<{ readonly id: string; readonly version: number }> => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123", version: 1 }); + maxAgeSec = 3; + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expectNativeReadCount(redis, 2); + }); + + it("does not resurrect or extend a frame after raising the runtime recovery maximum", async () => { + const useCase = "StaleRecoveryRaisedRuntimeMaximum"; + const redis = new RecordingRedis(); + let maxAgeSec = 3; + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + let sourceCalls = 0; + const source = vi.fn(async (): Promise<{ readonly id: string; readonly version: number }> => { + if (++sourceCalls === 1) { + return { id: "123", version: 1 }; + } + throw sourceError; + }); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: maxAgeSec, + }), + shouldAttemptStaleRecovery: allowStaleRecovery, + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123", version: 1 }); + expect(redis.ttlMs(redisValueKey(useCase))).toBe(3_000); + await vi.advanceTimersByTimeAsync(3_000); + maxAgeSec = 10; + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expectNativeReadCount(redis, 2); + expect(redis.setCalls).toBe(1); + }); + + it("coalesces recovery and does not populate process-local cache", async () => { + const useCase = "StaleRecoveryProcessCoalescing"; + const source = vi.fn(async () => { + throw SOURCE_UNAVAILABLE; + }); + const { redis, dialcache, getUser, staleRecovery } = setupDefaultStaleUseCase(useCase, source, { + defaultConfig: staleConfig({ local: true }), + }); + seedStale(redis, useCase, { id: "123", version: 1 }); + const localCache = (dialcache as unknown as { + readonly localCache: { put: (...args: unknown[]) => void }; + }).localCache; + const localPut = vi.spyOn(localCache, "put"); + + const values = await dialcache.enable(async () => await Promise.all([getUser(), getUser(), getUser()])); + + expect(source).toHaveBeenCalledOnce(); + expect(redis.readRequests).toHaveLength(1); + expect(values[1]).toBe(values[0]); + expect(values[2]).toBe(values[0]); + expect(localPut).not.toHaveBeenCalled(); + expect(staleRecovery).toHaveBeenCalledTimes(1); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(values[0]); + expect(source).toHaveBeenCalledTimes(2); + expect(redis.readRequests).toHaveLength(2); + expect(localPut).not.toHaveBeenCalled(); + }); + + it("memoizes a recovered reference only within the active request-local scope", async () => { + const useCase = "StaleRecoveryRequestLocal"; + const source = vi.fn(async () => { + throw new Error("source unavailable"); + }); + const { redis, dialcache, getUser } = setupDefaultStaleUseCase(useCase, source, { + defaultConfig: staleConfig({ requestLocal: true }), + }); + seedStale(redis, useCase, { id: "123", version: 1 }); + + const [first, second] = await dialcache.enable(async () => { + const firstValue = await getUser(); + const secondValue = await getUser(); + return [firstValue, secondValue] as const; + }); + + expect(second).toBe(first); + expect(source).toHaveBeenCalledOnce(); + expect(redis.readRequests).toHaveLength(1); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(first); + expect(source).toHaveBeenCalledTimes(2); + expect(redis.readRequests).toHaveLength(2); + }); + + it("decompresses a retained value during stale recovery", async () => { + const useCase = "StaleRecoveryCompressed"; + const retained = { id: "123", blob: "compressible stale payload ".repeat(1_024) }; + let available = true; + const source = vi.fn(async () => { + if (available) { + return retained; + } + throw SOURCE_UNAVAILABLE; + }); + const { redis, dialcache, getUser, staleRecovery } = setupDefaultStaleUseCase(useCase, source); + + await expect(dialcache.enable(async () => await getUser())).resolves.toBe(retained); + const stored = decodeFrame(redis.raw(redisValueKey(useCase))).payload; + expect(Buffer.isBuffer(stored) && stored[0]).toBe(MARKER_ZSTD_UTF8); + + available = false; + await vi.advanceTimersByTimeAsync(2_000); + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(retained); + + expectNativeReadCount(redis, 2); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "served" })); + }); + + it("contains a corrupt compression envelope and preserves the source rejection", async () => { + const useCase = "StaleRecoveryCorruptCompression"; + const source = vi.fn(async (): Promise<{ readonly id: string }> => { + throw SOURCE_UNAVAILABLE; + }); + const { redis, dialcache, getUser, staleRecovery } = setupDefaultStaleUseCase(useCase, source); + redis.setRaw( + redisValueKey(useCase), + encodeFrame( + Buffer.concat([Buffer.from([MARKER_ZSTD_UTF8]), Buffer.from("not a zstd frame")]), + Date.now() - 2_000, + 1, + ), + MAX_AGE_SEC * 1_000, + ); + + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(SOURCE_UNAVAILABLE); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "deserialization_error" })); + expectNativeReadCount(redis, 1); + }); + + it("runs independent stale recovery chains when coalescing is disabled", async () => { + const useCase = "StaleRecoveryCoalescingDisabled"; + const retained = { id: "123", version: 1 }; + const source = vi.fn(async () => { + throw new Error("source unavailable"); + }); + const { redis, dialcache, getUser, staleRecovery } = setupDefaultStaleUseCase(useCase, source, { + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: MAX_AGE_SEC, + coalesce: false, + }), + }); + seedStale(redis, useCase, retained); + + const values = await dialcache.enable(async () => await Promise.all([getUser(), getUser()])); + + expect(values).toEqual([retained, retained]); + expect(source).toHaveBeenCalledTimes(2); + expectNativeReadCount(redis, 2); + expect(staleRecovery).toHaveBeenCalledTimes(2); + }); +}); diff --git a/test/dialcache-stale-recovery-policy.test.ts b/test/dialcache-stale-recovery-policy.test.ts new file mode 100644 index 0000000..40cc38a --- /dev/null +++ b/test/dialcache-stale-recovery-policy.test.ts @@ -0,0 +1,402 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; + +import { + CacheLayer, + DialCache, + DialCacheKey, + DialCacheKeyConfig, + FallbackTimeoutError, + type CachedOptions, + type GetOrLoadOptions, + type Serializer, +} from "../src/index.js"; +import { encodeFrame, FakeRedis } from "./fake-redis.js"; + +const FRESH_TTL_SEC = 1; +const MAX_AGE_SEC = 10; + +type CachedValue = { readonly id: string; readonly version: number }; +type StaleRecoveryPredicate = (error: unknown) => boolean; + +function staleConfig(): DialCacheKeyConfig { + return new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: MAX_AGE_SEC, + }); +} + +function redisValueKey(useCase: string): string { + const key = new DialCacheKey({ + keyType: "user_id", + id: "123", + useCase, + trackForInvalidation: false, + }); + return `${key.urn}:dialcache-frame-v1`; +} + +function seedStale(redis: FakeRedis, useCase: string): void { + redis.setRaw( + redisValueKey(useCase), + encodeFrame({ id: "123", version: 1 }, Date.now() - 2_000), + MAX_AGE_SEC * 1_000, + ); +} + +function createDialCache( + redis: FakeRedis, + shouldAttemptStaleRecovery?: StaleRecoveryPredicate, +): DialCache { + return new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + logger: { + debug: vi.fn(), + error: vi.fn(), + warn: vi.fn(), + }, + ...(shouldAttemptStaleRecovery === undefined ? {} : { shouldAttemptStaleRecovery }), + }); +} + +function cachedRejecting( + dialcache: DialCache, + useCase: string, + sourceError: unknown, + shouldAttemptStaleRecovery?: StaleRecoveryPredicate, +) { + const source = vi.fn(async (): Promise => { + throw sourceError; + }); + const options = { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + ...(shouldAttemptStaleRecovery === undefined ? {} : { shouldAttemptStaleRecovery }), + } satisfies CachedOptions; + return { source, getUser: dialcache.cached(source, options) }; +} + +describe("DialCache stale-recovery error policy", () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("validates the instance predicate at construction", () => { + expect(() => + new DialCache({ + shouldAttemptStaleRecovery: "yes" as unknown as StaleRecoveryPredicate, + }), + ).toThrow(TypeError); + }); + + it("validates use-case predicates when each API captures them", () => { + const dialcache = new DialCache(); + const invalidPredicate = "yes" as unknown as StaleRecoveryPredicate; + const source = async (): Promise => ({ id: "123", version: 1 }); + + expect(() => + dialcache.cached(source, { + keyType: "user_id", + useCase: "InvalidCachedStaleRecoveryPolicy", + cacheKey: () => "123", + shouldAttemptStaleRecovery: invalidPredicate, + }), + ).toThrow(TypeError); + expect(() => + dialcache.getOrLoad(source, { + keyType: "user_id", + useCase: "InvalidGetOrLoadStaleRecoveryPolicy", + key: "123", + shouldAttemptStaleRecovery: invalidPredicate, + }), + ).toThrow(TypeError); + }); + + it("denies an ordinary source rejection by default", async () => { + const useCase = "DefaultDeniedStaleRecovery"; + const sourceError = new Error("source unavailable"); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis); + const { getUser } = cachedRejecting(dialcache, useCase, sourceError); + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(redis.getCalls).toBe(1); + }); + + it("allows any FallbackTimeoutError through the built-in policy", async () => { + const useCase = "DefaultTimeoutStaleRecovery"; + const timeout = new FallbackTimeoutError("NestedUseCase", 25); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis); + const { getUser } = cachedRejecting(dialcache, useCase, timeout); + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ + id: "123", + version: 1, + }); + + expect(redis.getCalls).toBe(1); + }); + + it("uses the instance predicate when the use case has no override", async () => { + const useCase = "InstanceStaleRecoveryPolicy"; + const sourceError = new Error("source unavailable"); + const predicate = vi.fn((error: unknown) => error === sourceError); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis, predicate); + const { getUser } = cachedRejecting(dialcache, useCase, sourceError); + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ + id: "123", + version: 1, + }); + + expect(predicate).toHaveBeenCalledOnce(); + expect(predicate).toHaveBeenCalledWith(sourceError); + }); + + it("lets a use-case predicate allow recovery over an instance denial", async () => { + const useCase = "UseCaseAllowsStaleRecovery"; + const sourceError = new Error("source unavailable"); + const instancePredicate = vi.fn(() => false); + const useCasePredicate = vi.fn(() => true); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis, instancePredicate); + const { getUser } = cachedRejecting(dialcache, useCase, sourceError, useCasePredicate); + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ + id: "123", + version: 1, + }); + + expect(useCasePredicate).toHaveBeenCalledOnce(); + expect(useCasePredicate).toHaveBeenCalledWith(sourceError); + expect(instancePredicate).not.toHaveBeenCalled(); + }); + + it("lets a use-case predicate deny recovery over an instance allowance", async () => { + const useCase = "UseCaseDeniesStaleRecovery"; + const sourceError = new Error("source unavailable"); + const instancePredicate = vi.fn(() => true); + const useCasePredicate = vi.fn(() => false); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis, instancePredicate); + const { getUser } = cachedRejecting(dialcache, useCase, sourceError, useCasePredicate); + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(useCasePredicate).toHaveBeenCalledOnce(); + expect(useCasePredicate).toHaveBeenCalledWith(sourceError); + expect(instancePredicate).not.toHaveBeenCalled(); + }); + + it("lets an explicit instance predicate replace the built-in timeout policy", async () => { + const useCase = "InstanceDeniesTimeoutStaleRecovery"; + const timeout = new FallbackTimeoutError("NestedUseCase", 25); + const predicate = vi.fn(() => false); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis, predicate); + const { getUser } = cachedRejecting(dialcache, useCase, timeout); + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(timeout); + + expect(predicate).toHaveBeenCalledOnce(); + expect(predicate).toHaveBeenCalledWith(timeout); + }); + + it.each([ + { + name: "throws", + predicate: vi.fn(() => { + throw new Error("predicate failed"); + }) as StaleRecoveryPredicate, + }, + { + name: "returns a non-boolean", + predicate: vi.fn(() => "yes") as unknown as StaleRecoveryPredicate, + }, + { + name: "returns a thenable", + predicate: vi.fn(() => Promise.resolve(true)) as unknown as StaleRecoveryPredicate, + }, + ])("fails closed and preserves the source rejection when the predicate $name", async ({ predicate }) => { + const useCase = `DefensiveStaleRecoveryPolicy${predicate.name}`; + const sourceError = new Error("source unavailable"); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis); + const { getUser } = cachedRejecting(dialcache, useCase, sourceError, predicate); + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(predicate).toHaveBeenCalledOnce(); + expect(predicate).toHaveBeenCalledWith(sourceError); + }); + + it("consumes a rejecting predicate thenable while failing closed without stale recovery", async () => { + const useCase = "RejectingThenableStaleRecoveryPolicy"; + const sourceError = new Error("source unavailable"); + const predicateError = new Error("predicate failed asynchronously"); + let markThenableConsumed!: () => void; + const thenableConsumed = new Promise((resolve) => { + markThenableConsumed = resolve; + }); + const then = vi.fn(( + _onFulfilled: ((value: boolean) => unknown) | null | undefined, + onRejected: ((reason: unknown) => unknown) | null | undefined, + ) => { + onRejected?.(predicateError); + markThenableConsumed(); + }); + const predicate = vi.fn(() => ({ then })) as unknown as StaleRecoveryPredicate; + const load = vi.fn((): CachedValue => ({ id: "123", version: 1 })); + const serializer = { + dump: (value: CachedValue) => JSON.stringify(value), + load, + } satisfies Serializer; + const redis = new FakeRedis(); + const dialcache = createDialCache(redis); + const source = vi.fn(async (): Promise => { + throw sourceError; + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + shouldAttemptStaleRecovery: predicate, + serializer, + }); + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + await thenableConsumed; + + expect(predicate).toHaveBeenCalledOnce(); + expect(predicate).toHaveBeenCalledWith(sourceError); + expect(then).toHaveBeenCalledOnce(); + expect(then.mock.calls[0]?.[1]).toEqual(expect.any(Function)); + expect(load).not.toHaveBeenCalled(); + expect(redis.getCalls).toBe(1); + }); + + it("snapshots a cached function's predicate at registration", async () => { + const useCase = "CachedPredicateSnapshot"; + const sourceError = new Error("source unavailable"); + const registeredPredicate = vi.fn(() => false); + const replacementPredicate = vi.fn(() => true); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis); + const source = vi.fn(async (): Promise => { + throw sourceError; + }); + const options = { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + shouldAttemptStaleRecovery: registeredPredicate, + } satisfies CachedOptions; + const getUser = dialcache.cached(source, options); + (options as { shouldAttemptStaleRecovery: StaleRecoveryPredicate }).shouldAttemptStaleRecovery = + replacementPredicate; + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(registeredPredicate).toHaveBeenCalledOnce(); + expect(replacementPredicate).not.toHaveBeenCalled(); + }); + + it("resolves a getOrLoad predicate independently for each invocation", async () => { + const useCase = "GetOrLoadPredicatePerInvocation"; + const sourceError = new Error("source unavailable"); + const deny = vi.fn(() => false); + const allow = vi.fn(() => true); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis); + const load = vi.fn(async (): Promise => { + throw sourceError; + }); + const options = { + keyType: "user_id", + useCase, + key: "123", + defaultConfig: staleConfig(), + shouldAttemptStaleRecovery: deny, + } satisfies GetOrLoadOptions; + seedStale(redis, useCase); + + await expect(dialcache.enable(async () => await dialcache.getOrLoad(load, options))).rejects.toBe(sourceError); + + (options as { shouldAttemptStaleRecovery: StaleRecoveryPredicate }).shouldAttemptStaleRecovery = allow; + await expect(dialcache.enable(async () => await dialcache.getOrLoad(load, options))).resolves.toEqual({ + id: "123", + version: 1, + }); + + expect(deny).toHaveBeenCalledOnce(); + expect(allow).toHaveBeenCalledOnce(); + expect(load).toHaveBeenCalledTimes(2); + }); + + it("does not invoke the predicate outside an enabled context", async () => { + const useCase = "DisabledContextStaleRecoveryPolicy"; + const sourceError = new Error("source unavailable"); + const predicate = vi.fn(() => true); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis); + const { source, getUser } = cachedRejecting(dialcache, useCase, sourceError, predicate); + seedStale(redis, useCase); + + await expect(getUser()).rejects.toBe(sourceError); + + expect(source).toHaveBeenCalledOnce(); + expect(predicate).not.toHaveBeenCalled(); + expect(redis.getCalls).toBe(0); + }); + + it("invokes the policy once for a coalesced leader", async () => { + const useCase = "CoalescedStaleRecoveryPolicy"; + const sourceError = new Error("source unavailable"); + const predicate = vi.fn(() => true); + let releaseSource!: () => void; + const sourceGate = new Promise((resolve) => { + releaseSource = resolve; + }); + const redis = new FakeRedis(); + const dialcache = createDialCache(redis); + const source = vi.fn(async (): Promise => { + await sourceGate; + throw sourceError; + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + shouldAttemptStaleRecovery: predicate, + }); + seedStale(redis, useCase); + + const pending = dialcache.enable(async () => await Promise.all([getUser(), getUser(), getUser()])); + await vi.waitFor(() => expect(source).toHaveBeenCalledOnce()); + releaseSource(); + const values = await pending; + + expect(predicate).toHaveBeenCalledOnce(); + expect(predicate).toHaveBeenCalledWith(sourceError); + expect(values[1]).toBe(values[0]); + expect(values[2]).toBe(values[0]); + expect(redis.getCalls).toBe(1); + }); +}); diff --git a/test/prometheus.test.ts b/test/prometheus.test.ts index 808b69f..21e0e21 100644 --- a/test/prometheus.test.ts +++ b/test/prometheus.test.ts @@ -16,6 +16,7 @@ import { type DisabledReason, type MetricErrorKind, type ShadowValidationOutcome, + type StaleRecoveryOutcome, } from "../src/index.js"; import { PrometheusDialCacheMetrics, createPrometheusDialCacheMetrics } from "../src/prometheus.js"; import { FakeRedis } from "./fake-redis.js"; @@ -82,6 +83,11 @@ const SHADOW_VALIDATION_OUTCOMES: Readonly timeout: true, dropped: true, }; +const STALE_RECOVERY_OUTCOMES: Readonly> = { + served: true, + miss: true, + deserialization_error: true, +}; interface IncompatibleCollectorCase { readonly schemaPart: string; @@ -182,6 +188,21 @@ describe("Prometheus metrics adapter", () => { keyType: labels.keyType, outcome: "match", }); + metrics.staleRecovery({ + cacheNamespace: labels.cacheNamespace, + useCase: labels.useCase, + keyType: labels.keyType, + outcome: "served", + }); + metrics.observeStaleRecoveryValueAge( + { + cacheNamespace: labels.cacheNamespace, + useCase: labels.useCase, + keyType: labels.keyType, + outcome: "served", + }, + 90, + ); metrics.observeShadowValueAge( { cacheNamespace: labels.cacheNamespace, @@ -258,6 +279,15 @@ describe("Prometheus metrics adapter", () => { VALUE_AGE_BUCKETS, ), histogramSchema("schema_dialcache_size_histogram", ["cache_namespace", "use_case", "key_type", "layer"], SIZE_BUCKETS), + counterSchema( + "schema_dialcache_stale_recovery_counter", + ["cache_namespace", "use_case", "key_type", "outcome"], + ), + histogramSchema( + "schema_dialcache_stale_recovery_value_age_histogram", + ["cache_namespace", "use_case", "key_type", "outcome"], + VALUE_AGE_BUCKETS, + ), histogramSchema( "schema_dialcache_stored_size_histogram", ["cache_namespace", "use_case", "key_type", "layer"], @@ -277,6 +307,20 @@ describe("Prometheus metrics adapter", () => { outcome: "match", }); + const staleRecoveryValueAge = families.find( + ({ name }) => name === "schema_dialcache_stale_recovery_value_age_histogram", + ); + const staleRecoveryValueAgeSum = staleRecoveryValueAge?.values.find( + ({ metricName }) => metricName === "schema_dialcache_stale_recovery_value_age_histogram_sum", + ); + expect(staleRecoveryValueAgeSum?.value).toBe(90); + expect(staleRecoveryValueAgeSum?.labels).toEqual({ + cache_namespace: labels.cacheNamespace, + use_case: labels.useCase, + key_type: labels.keyType, + outcome: "served", + }); + const futureTimestampOffset = families.find( ({ name }) => name === "schema_dialcache_future_timestamp_offset_histogram", ); @@ -428,6 +472,39 @@ describe("Prometheus metrics adapter", () => { ); }); + it("exports every bounded stale-recovery outcome without adding cache identity or layer labels", async () => { + const registry = new Registry(); + const metrics = new PrometheusDialCacheMetrics({ registry, prefix: "stale_" }); + const labels = { + cacheNamespace: "users", + useCase: "PrometheusStaleRecovery", + keyType: "user_id", + } as const; + const outcomes = Object.keys(STALE_RECOVERY_OUTCOMES) as StaleRecoveryOutcome[]; + + for (const outcome of outcomes) { + metrics.staleRecovery({ ...labels, outcome }); + } + + for (const outcome of outcomes) { + await expect( + sumMetric(registry, "stale_dialcache_stale_recovery_counter", { + cache_namespace: labels.cacheNamespace, + use_case: labels.useCase, + key_type: labels.keyType, + outcome, + }), + ).resolves.toBe(1); + } + + const family = ((await registry.getMetricsAsJSON()) as unknown as MetricFamily[]).find( + ({ name }) => name === "stale_dialcache_stale_recovery_counter", + ); + expect(family?.values.map(({ labels: emitted }) => Object.keys(emitted))).toEqual( + outcomes.map(() => ["cache_namespace", "use_case", "key_type", "outcome"]), + ); + }); + it("exports every bounded compression outcome without rewriting labels", async () => { const registry = new Registry(); const metrics = new PrometheusDialCacheMetrics({ registry, prefix: "compression_" }); diff --git a/test/redis-real.integration.test.ts b/test/redis-real.integration.test.ts index 54690d1..7d13152 100644 --- a/test/redis-real.integration.test.ts +++ b/test/redis-real.integration.test.ts @@ -262,6 +262,189 @@ describe.each(engines)("DialCache Redis protocol on $name", ({ image }) => { }); }); + it.each([false, true])( + "retains a logically stale value through its maximum age and recovers it after source rejection (tracked=%s)", + async (trackForInvalidation) => { + if (client === undefined || admin === undefined) { + throw new Error("Redis test clients did not start"); + } + const namespace = `real-stale-${kind}-${trackForInvalidation ? "tracked" : "untracked"}`; + const useCase = "RealStaleOnError"; + const id = "123"; + const key = new DialCacheKey({ + namespace, + keyType: "item_id", + id, + useCase, + trackForInvalidation, + }); + const valueKey = `${key.urn}:dialcache-frame-v1`; + const sourceValue = { id, version: 1 }; + const sourceError = new Error("source unavailable"); + const source = vi.fn<() => Promise>() + .mockResolvedValueOnce(sourceValue) + .mockRejectedValueOnce(sourceError); + const redisRead = vi.fn(client.adapter.read.bind(client.adapter)); + const redisClient: DialCacheRedisClient = { + read: redisRead, + write: client.adapter.write.bind(client.adapter), + invalidate: client.adapter.invalidate.bind(client.adapter), + }; + const dialcache = new DialCache({ + namespace, + redis: { client: redisClient, readTimeoutMs: 10_000 }, + shouldAttemptStaleRecovery: () => true, + }); + const getItem = dialcache.cached(source, { + keyType: "item_id", + useCase, + cacheKey: () => id, + trackForInvalidation, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 60, + }), + }); + + await expect(dialcache.enable(async () => await getItem())).resolves.toEqual(sourceValue); + const retainedTtlMs = await admin.pTTL(valueKey); + expect(retainedTtlMs).toBeGreaterThan(55_000); + expect(retainedTtlMs).toBeLessThanOrEqual(60_000); + + await admin.set( + valueKey, + encodeFrame(JSON.stringify(sourceValue), 0, Date.now() - 2_000), + { PX: 60_000 }, + ); + const ttlBeforeRecovery = await admin.pTTL(valueKey); + + await expect(dialcache.enable(async () => await getItem())).resolves.toEqual(sourceValue); + + expect(source).toHaveBeenCalledTimes(2); + expect(redisRead).toHaveBeenCalledTimes(2); + expect(await admin.pTTL(valueKey)).toBeLessThanOrEqual(ttlBeforeRecovery); + }, + ); + + it("does not recover a tracked stale value fenced by invalidation", async () => { + if (client === undefined || admin === undefined) { + throw new Error("Redis test clients did not start"); + } + const namespace = `real-stale-invalidated-${kind}`; + const useCase = "RealStaleOnErrorInvalidated"; + const id = "123"; + const key = new DialCacheKey({ + namespace, + keyType: "item_id", + id, + useCase, + trackForInvalidation: true, + }); + const valueKey = `${key.urn}:dialcache-frame-v1`; + const sourceValue = { id, version: 1 }; + const sourceError = new Error("source unavailable"); + const source = vi.fn<() => Promise>() + .mockResolvedValueOnce(sourceValue) + .mockRejectedValueOnce(sourceError); + const redisRead = vi.fn(client.adapter.read.bind(client.adapter)); + const redisClient: DialCacheRedisClient = { + read: redisRead, + write: client.adapter.write.bind(client.adapter), + invalidate: client.adapter.invalidate.bind(client.adapter), + }; + const dialcache = new DialCache({ + namespace, + redis: { client: redisClient, readTimeoutMs: 10_000 }, + shouldAttemptStaleRecovery: () => true, + }); + const getItem = dialcache.cached(source, { + keyType: "item_id", + useCase, + cacheKey: () => id, + trackForInvalidation: true, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 60, + }), + }); + + await expect(dialcache.enable(async () => await getItem())).resolves.toEqual(sourceValue); + await admin.set( + valueKey, + encodeFrame(JSON.stringify(sourceValue), 0, Date.now() - 2_000), + { PX: 60_000 }, + ); + await dialcache.invalidateRemote("item_id", id); + + await expect(dialcache.enable(async () => await getItem())).rejects.toBe(sourceError); + expect(source).toHaveBeenCalledTimes(2); + expect(redisRead).toHaveBeenCalledTimes(2); + }); + + it("serves the retained tracked snapshot when invalidation arrives during the source attempt", async () => { + if (client === undefined || admin === undefined) { + throw new Error("Redis test clients did not start"); + } + const namespace = `real-stale-invalidation-race-${kind}`; + const useCase = "RealStaleOnErrorInvalidationRace"; + const id = "123"; + const key = new DialCacheKey({ + namespace, + keyType: "item_id", + id, + useCase, + trackForInvalidation: true, + }); + const valueKey = `${key.urn}:dialcache-frame-v1`; + const sourceValue = { id, version: 1 }; + const sourceError = new Error("source unavailable"); + const sourceStarted = deferred(); + const releaseSource = deferred(); + const source = vi.fn(async (): Promise => { + sourceStarted.resolve(undefined); + await releaseSource.promise; + throw sourceError; + }); + const redisRead = vi.fn(client.adapter.read.bind(client.adapter)); + const redisClient: DialCacheRedisClient = { + read: redisRead, + write: client.adapter.write.bind(client.adapter), + invalidate: client.adapter.invalidate.bind(client.adapter), + }; + const dialcache = new DialCache({ + namespace, + redis: { client: redisClient, readTimeoutMs: 10_000 }, + shouldAttemptStaleRecovery: () => true, + }); + const getItem = dialcache.cached(source, { + keyType: "item_id", + useCase, + cacheKey: () => id, + trackForInvalidation: true, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 60, + }), + }); + await admin.set( + valueKey, + encodeFrame(JSON.stringify(sourceValue), 0, Date.now() - 2_000), + { PX: 60_000 }, + ); + + const pending = dialcache.enable(async () => await getItem()); + await sourceStarted.promise; + await dialcache.invalidateRemote("item_id", id); + releaseSource.resolve(undefined); + + await expect(pending).resolves.toEqual(sourceValue); + expect(source).toHaveBeenCalledOnce(); + expect(redisRead).toHaveBeenCalledOnce(); + }); + it("compresses values above the threshold and stores small values byte-identical", async () => { if (client === undefined || admin === undefined) { throw new Error("Redis test clients did not start");