Skip to content

feat: add stale-on-error Redis recovery - #121

Merged
lan17 merged 4 commits into
mainfrom
agent/stale-on-error
Aug 30, 2026
Merged

feat: add stale-on-error Redis recovery#121
lan17 merged 4 commits into
mainfrom
agent/stale-on-error

Conversation

@lan17

@lan17 lan17 commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Summary

Add opt-in stale-on-error recovery from the existing Redis frame, on top of the native client-clock protocol from #140.

  • F = ttlSec[CacheLayer.REMOTE] remains the ordinary Redis freshness boundary.
  • M = staleOnErrorMaxAgeSec is the absolute stale-recovery age ceiling and requested physical retention.
  • One initial GET or tracked MGET(value, watermark) either serves a fresh frame, retains an F..M raw candidate, or misses without a candidate.
  • After the source rejects, a resolved error classifier decides whether DialCache may use the retained candidate.
  • Recovery never rereads Redis, republishes the value, or replaces the original source rejection when it cannot serve.

Closes #117

Configuration and public API

import {
  CacheLayer,
  DialCache,
  DialCacheKeyConfig,
  FallbackTimeoutError,
} from "dialcache";

const dialcache = new DialCache({
  // Instance default: the broad transient-infrastructure policy.
  shouldAttemptStaleRecovery: (error) =>
    error instanceof FallbackTimeoutError || isRetriableDatabaseError(error),
});

const getUser = dialcache.cached((id: string) => db.fetchUser(id), {
  keyType: "user_id",
  useCase: "GetUser",
  cacheKey: (id) => id,
  defaultConfig: new DialCacheKeyConfig({
    ttlSec: { [CacheLayer.REMOTE]: 300 }, // F: ordinary Redis freshness
    staleOnErrorMaxAgeSec: 3_600,         // M: absolute recovery ceiling
  }),
});

// A per-use-case classifier REPLACES the instance and built-in policies for
// that use case; include the timeout case when it should stay eligible.
const getEntitlements = dialcache.cached((id: string) => db.fetchEntitlements(id), {
  keyType: "user_id",
  useCase: "GetEntitlements",
  cacheKey: (id) => id,
  shouldAttemptStaleRecovery: (error) => error instanceof FallbackTimeoutError,
  defaultConfig: new DialCacheKeyConfig({
    ttlSec: { [CacheLayer.REMOTE]: 300 },
    staleOnErrorMaxAgeSec: 3_600,
  }),
});

New API surface:

  • DialCacheKeyConfig.staleOnErrorMaxAgeSec?: number
  • DialCacheConfig.shouldAttemptStaleRecovery?: StaleRecoveryPredicate
  • per-use-case shouldAttemptStaleRecovery on cached() and getOrLoad() options
  • root-exported StaleRecoveryPredicate, StaleRecoveryOutcome, and StaleRecoveryMetricLabels
  • optional DialCacheMetricsAdapter.staleRecovery(labels) and observeStaleRecoveryValueAge(labels, seconds) hooks

Classifier precedence — highest wins, and an override replaces the levels below it rather than composing with them:

  1. per-use-case shouldAttemptStaleRecovery on cached() / getOrLoad() options
  2. DialCacheConfig.shouldAttemptStaleRecovery
  3. built-in: error instanceof FallbackTimeoutError only

The intended split: put the broad policy for transient, retriable infrastructure failures on the instance, and use a per-use-case override where particular data needs a stricter policy. Predicates should deny authoritative domain outcomes — auth/permission/entitlement failures, revocation, deletion or not-found, validation, and programmer errors — where a stale value would be wrong rather than merely old.

Predicates must synchronously return a boolean; a throw, non-boolean, or thenable fails closed, and an accidental rejecting thenable is consumed. cached() snapshots its selected predicate at registration; getOrLoad() resolves it per invocation. Disabled calls remain true pass-through and never invoke it.

Omitting M keeps recovery off; a sparse runtime overlay that omits it inherits the configured default, and an explicit 0 disables an inherited value. A positive policy requires 0 < F < M <= 31,536,000 seconds. Invalid static policy throws before registration; invalid runtime M records config_resolution, disables only recovery for that invocation, and preserves otherwise-valid ordinary Redis policy. DialCacheKeyConfig.disabled() sets M to 0.

Execution

flowchart TD
  A["One native GET or tracked MGET; classify raw frame with application Date.now()"] -->|"0 <= age < F"| B["Deserialize and return fresh"]
  A -->|"F <= age < M"| C["Record ordinary miss; retain raw candidate"]
  A -->|"missing, future, fenced, invalid, or age >= M"| D["Ordinary miss without candidate"]
  A -->|"read error, timeout, or fresh deserialization failure"| E["Fallback; recovery forbidden"]

  C --> F["Call source"]
  D --> F
  F -->|success| G["Return source value and attempt normal publication"]
  F -->|rejection denied by classifier| H["Throw exact source rejection"]
  F -->|eligible rejection| I["Use only retained candidate"]
  I -->|"age < M before and after lazy load"| J["Return candidate without publication"]
  I -->|missing, expired, future, or load failure| H
  E -->|source rejection| H
Loading

The candidate stays serialized/compressed until the source call settles. On an eligible rejection, DialCache checks 0 <= age < M, lazily deserializes/decompresses, checks M again after that potentially asynchronous work, and only then returns it. Equality at F is logically stale; equality at M is unavailable.

A successful source refresh wins and writes a newly timestamped frame. Recovery does not write or extend Redis, populate process-local cache, schedule shadow validation, or emit a shadow-age observation. Request-local caching may memoize the returned reference only within the active outer enable() scope. Existing coalescing shares the whole read/source/recovery decision; coalesce: false gives each caller an independent candidate and source attempt.

Snapshot and freshness semantics

The retained candidate is the initial Redis snapshot:

  • tracked reads atomically apply the value and watermark observed by the initial primary-routed MGET;
  • an invalidation completed before that read fences the candidate;
  • invalidation, refresh, deletion, expiry, or eviction after the read does not revoke or replace the in-memory bytes;
  • tracked and untracked recovery therefore use the same one-read model.

For tracked use cases this opt-in path can weaken the usual strict freshness guarantee when invalidation races with the source attempt. Use cases that cannot tolerate that bounded relaxation should leave recovery disabled or deny the error in their classifier.

F and M bound Redis serving only. Request-local and process-local layers occur earlier and keep their own scope/TTL lifetimes. A frame can be nearly F old when it warms process-local cache and then receive a full local TTL, so ttlSec.local <= F does not make F a strict end-to-end age limit; disable earlier layers when that is required.

Redis, time, and compatibility

There is no Redis envelope or key change: frame v1, :dialcache-frame-v1, DialCacheRedisClient, and RedisReadRequest remain unchanged. Reads use native GET/primary-routed MGET, writes use native complete-frame SET, and invalidation remains the only Lua operation. No Redis TIME or clock-offset estimation is added.

Writers request physical retention through M when enabled and F otherwise. Tracked values keep the existing one-hour physical TTL cap and tracked_ttl_clamped signal from #140. The configured logical M is not clipped, but a tracked frame may physically disappear before reaching it.

Core now treats every decoded frame's real writer createdAtMs as authoritative for ordinary logical F, including untracked reads. Custom Redis clients that returned a constant timestamp must return the actual epoch-millisecond frame stamp before upgrading. Roll out readers while M is omitted or 0, upgrade the complete fleet, and only then enable positive M. Once a writer retains through M, do not reintroduce a pre-feature reader until affected keys have expired or been removed; older readers trust physical presence and can serve F..M as fresh.

Application-process clock skew can move the boundary early or late. Future-dated frames fail closed and use the existing future-offset observation. Durations and deadlines remain monotonic.

Metrics

New series, exposed by both bundled backends:

Backend Metric Type Labels / tags Emitted
Prometheus dialcache_stale_recovery_counter counter cache_namespace, use_case, key_type, outcome once per classifier-authorized recovery check
Prometheus dialcache_stale_recovery_value_age_histogram histogram, buckets [1, 5, 15, 60, 300, 900, 3600, 10800, 43200, 86400, 259200, 604800] s (shared with shadow value age) cache_namespace, use_case, key_type, outcome only alongside served; value = return-time age in seconds
Datadog dialcache.stale_recovery.count count cache_namespace, use_case, key_type, outcome once per classifier-authorized recovery check
Datadog dialcache.stale_recovery.value_age histogram or distribution, per the adapter's existing observationMetricType option cache_namespace, use_case, key_type, outcome only alongside served

Label reference:

  • cache_namespaceDialCacheConfig.namespace (default urn); present on every DialCache metric.
  • use_case / key_type — the registered use case and key type; never the cache key or id.
  • outcome — the bounded root-exported StaleRecoveryOutcome union:
    • served — the retained candidate was returned to the caller (age < M held both before and after lazy deserialization);
    • miss — no candidate was retained by the initial read, or the candidate's age reached M (or its stamp became future/invalid) by check time;
    • deserialization_error — a candidate within age failed deserialize/decompress.
  • On the value-age series the outcome label is structurally always served — it exists so both outcome series share one label schema.
  • Neither series carries a layer label: recovery reuses the caller-serving initial read, so there is no separate layer to attribute.
  • Adapter authors receive the backend-neutral camel-case object StaleRecoveryMetricLabels { cacheNamespace, useCase, keyType, outcome } and map it to their backend's naming, as the bundled adapters do.

The backing DialCacheMetricsAdapter.staleRecovery(labels) and observeStaleRecoveryValueAge(labels, seconds) hooks are optional — existing custom adapters keep compiling, and omitting a hook skips only that observation, never recovery itself. A rejection the classifier denies emits no stale-recovery outcome: the counter measures authorized checks, not all source failures. No raw error, exception message, or cache key ever enters a label.

Recovery adds no second ordinary request, observeGet, miss, cache-read error, or Redis command — the initial read is the one caller-serving telemetry trail. Existing fallback error/duration telemetry still records the source rejection even when recovery serves, so a served recovery stays visibly paired with the failure that caused it.

Changes to existing series, visible on upgrade even where recovery stays off (all carry their usual cache_namespace, use_case, key_type labels; layer/error/in_fallback as noted):

  • dialcache_miss_counter / hit rate, at layer="remote" — logical F is now enforced from every frame's real createdAtMs, including untracked reads, so physically present but logically old frames that previously served now count as misses. Keyspaces that relied on physical-TTL slack will see a one-time remote miss-rate rise. With recovery enabled, a retained F..M candidate also records an ordinary miss even when recovery later serves it — during incidents, treat staleRecovery{outcome="served"} as its own population rather than expecting it in hit-rate math. dialcache_request_counter and the get-duration histogram are unchanged in population: still exactly one per caller-serving read.
  • dialcache_future_timestamp_offset_histogram (layer="remote" and shadow layers) — its population widened: previously it observed only tracked frames, but every serving read now validates the stamp, so future-dated untracked frames are observed (and fail closed) as well. The Prometheus help string changed accordingly.
  • dialcache_error_counter with error="tracked_ttl_clamped", layer="remote", in_fallback="false" — enabling M above one hour on a tracked use case makes every dispatched tracked write request TTL M and get clamped, so this existing configuration signal fires per write for such configs (see the Redis section above). Alert rules keyed on the error counter without an error label filter will absorb this as a steady rate.
  • dialcache_error_counter with error="config_resolution" — an invalid runtime M overlay records one per invocation while recovery is disabled for that invocation and ordinary reads continue.

Cost

The Redis outage-path cost drops from two payload reads to one. The tradeoff moves to Node memory: one raw candidate is retained per distinct in-flight key through the source attempt; same-key coalesced followers share it. The benchmark includes a delayed, high-cardinality, incompressible-payload scenario rather than relying on the highly compressed default fixture.

Validation

Node 22.22.0:

  • typecheck
  • 611 unit tests with 98.18% statement coverage
  • ESM/CJS build and packed TypeScript consumers
  • 139 live Redis/Valkey integration tests passed; 2 unavailable-cluster cases skipped after the expected connection timeout
  • stale-on-error benchmark semantic assertions passed
  • exactly 1.00 adapter read per independent recovery flight and one read shared by 500 coalesced callers
  • delayed 128 × 64 KiB raw-candidate benchmark observed +8.00 MiB external memory while retained and returned to baseline after recovery

BREAKING CHANGE: Ordinary Redis reads now enforce logical age from each frame's real createdAtMs, including untracked reads. Deploy new readers before enabling physical M retention.

lan17 added a commit that referenced this pull request Aug 7, 2026
## Summary

Replace read-side Lua with native Redis commands and decode DialCache's
frame in TypeScript:

- untracked reads use `GET`
- tracked reads use one atomic, primary-routed `MGET` for the value and
watermark
- write and invalidation remain Lua-backed; a watermark-fenced tracked
write now atomically unlinks the stale value it rejects
- node-redis registers only the three mutation scripts, and GLIDE owns
only the three mutation script handles
- custom adapters can reuse the public `decodeRedisFrame` and
`decodeTrackedRedisFrame` helpers

This removes the Redis-to-Lua payload materialization and `string.sub`
copy on every hit while preserving the semantic
`DialCacheRedisClient.read()` boundary.

## Read architecture

| Adapter / mode | Untracked | Tracked | Primary guarantee |
| --- | --- | --- | --- |
| node-redis standalone | `GET` | `MGET` | standalone connection |
| node-redis Cluster | `GET` | raw `MGET` | `sendCommand(..., false,
...)` routes to the slot primary |
| GLIDE standalone | `GET` | one-command `Batch(false).mget(...)` |
standalone batches execute on the primary even with replica reads
configured; `MGET` itself is atomic |
| GLIDE Cluster | `GET` | custom-command `MGET` | explicit
`primarySlotKey` route |

The shared decoder:

- validates the frame version and minimum length
- preserves missing/short/unsupported frames as clean misses
- parses integer and fractional legacy watermarks with the same accepted
grammar as Lua
- rejects values whose Redis-created timestamp is at or before the
watermark
- preserves unsupported payload encodings as
`DialCacheRedisPayloadEncodingError`
- returns binary payloads through a zero-copy `Buffer.subarray()` view

Tracked value and watermark reads retain one atomic snapshot, with both
values returned by a single `MGET`. Their existing shared Cluster hash
tag remains required; mismatched tags still fail with `CROSSSLOT`.

## Breaking change

- `READ_CACHE_SCRIPT` and `READ_TRACKED_CACHE_SCRIPT` are removed from
`dialcache/redis-protocol`.
- `dialcacheRedisScripts.dialcacheRead` and
`dialcacheRedisScripts.dialcacheReadTracked` are removed from
`dialcache/node-redis`.
- Custom node-redis wrappers must expose native `get` / `sendCommand`;
`legacyMode` clients are unsupported because neither their callback
surface nor `.v4` view exposes the complete
native-command-plus-custom-script contract.
- The GLIDE helper requires GLIDE 2.x, a direct official `GlideClient`
or `GlideClusterClient`, and the same module namespace that created it.
Forwarding wrappers should implement `DialCacheRedisClient` directly
because their topology cannot be inferred safely.
- Official node-redis clients and direct GLIDE 2.x clients passed
through the documented helpers keep the same application-facing call
shape, so those consumers can bump the package without code changes.
- Redis keys, frame format, and invalidation behavior are unchanged. A
tracked write rejected by an active future watermark still returns
`false`, but now also unlinks the stale value key. No data migration or
cache flush is required.
- The fenced-write cleanup requires `UNLINK` (Redis 4.0+ or compatible
Valkey) and permission for scripts to invoke it. With a
command-restricted ACL that denies `UNLINK`, the write fails open as
`cache_write` and leaves the stale value for a later cleanup or expiry.

`BREAKING CHANGE:` the four deprecated read-Lua exports and
registrations above are removed; node-redis adapters require the
promise-mode native-command surface; the GLIDE helper requires a direct
GLIDE 2.x client from the supplied runtime; and the fenced-write cleanup
requires Redis `UNLINK` support plus ACL permission. Under the
repository's release configuration, this change should release as
`v1.0.0`.

## Adapter behavior changes

- The node-redis factory now requires native `get` and `sendCommand`
methods in addition to the three registered mutation methods.
- The GLIDE factory declares an optional `@valkey/valkey-glide ^2.0.0`
peer, validates `Batch` support eagerly, and classifies standalone
versus cluster behavior from the supplied runtime's client identities
before allocating scripts. Its standalone non-atomic primary batch
avoids consuming caller-owned `WATCH` state.
- Redis `MGET` returns `null` for wrong-type members. A tracked
wrong-type value is therefore a clean miss and may be repaired with a
valid DialCache frame after fallback succeeds, while a wrong-type
watermark prevents the tracked write from succeeding. An untracked `GET`
still surfaces `WRONGTYPE`. Real-engine tests cover both repair and
repeated fail-open behavior, including metrics.
- The public read contract now specifies frame decoding, miss and
watermark rules, atomic authoritative snapshots, and returned-buffer
ownership. Shared decoders validate leaf reply types; adapters retain
only client-specific envelope validation.

## Benchmark

The benchmark harness and JSON results were intentionally kept outside
the repository. Methodology:

- Redis 6.2.22 and Valkey 8.1.8
- Node 22.22.0, node-redis 4.7.1, GLIDE 2.4.2
- binary payloads of 100 B, 1 KiB, 10 KiB, 100 KiB, and 1 MiB
- fresh untracked hit, fresh tracked hit, and invalidated tracked miss
- three alternating rounds, one command in flight, loopback Docker
- median throughput, latency, Redis `INFO commandstats` execution time,
and network bytes

At 1 MiB, native fresh-hit throughput improved 15-45% across the two
engines and adapters. Server-reported command execution time per logical
read fell 95-98%. Small 100 B / 1 KiB end-to-end results were mostly
flat/noisy while reported command time still fell about 80-90%; the
notable small-case regression was Redis/node-redis's 100 B tracked hit
at about -10% throughput. These loopback, one-in-flight results are
directional rather than production-capacity measurements.

Representative Redis 6.2 + node-redis medians:

| 1 MiB scenario | Lua ops/s | Native ops/s | Lua server us/read |
Native server us/read | Lua -> native p50 |
| --- | ---: | ---: | ---: | ---: | ---: |
| untracked hit | 230 | 269 | 719.8 | 32.6 | 3.718 ms -> 2.955 ms |
| tracked hit | 217 | 259 | 713.7 | 31.2 | 3.630 ms -> 3.016 ms |
| invalidated tracked miss | 1,762 | 284 | 361.6 | 31.0 | 0.566 ms ->
2.949 ms |

The invalidated-miss row is the main tradeoff: Lua returns only a null
reply, while native `MGET` transfers the stale frame before TypeScript
rejects it. At 1 MiB this changes roughly 3-5 response bytes into about
1.05 MB. Across both engines and adapters, invalidated-miss throughput
fell 77-84% at 1 MiB (46-58% at 100 KiB), even though server-reported
command time still fell 91-94%.

The benchmark intentionally measured the read itself and therefore
includes that full transfer. In the application path, the first
completed fallback that reaches a still-fenced tracked write now
atomically unlinks the stale value, bounding subsequent transfers for
that entry. This is only a partial mitigation: a read failure or timeout
never reaches the write-side cleanup, so the stale payload can continue
to transfer or time out until another completed read cleans it up or its
TTL expires.

## Scope

This branch is updated onto the current `v0.15.0` read contract,
including the untracked-cache shadowing changes from
#122. It deliberately does not
include the server-time / maximum-age behavior proposed in
#121. That work can be evaluated
separately against this read path and its benchmark tradeoffs.

## Validation

- `corepack pnpm typecheck`
- `corepack pnpm test` - 424 tests, coverage thresholds passed
- `corepack pnpm build`
- `corepack pnpm test:package` - including real node-redis and GLIDE
standalone and Cluster consumer types, plus packed ESM/CommonJS absence
checks for all four removed APIs
- `corepack pnpm test:integration` - 113 tests across Redis 6.2, Valkey
8, and Redis Cluster
- tracked wrong-type value repair and repeated wrong-type watermark
fail-open behavior exercised end to end across both adapters and both
standalone engines
- stale tracked frames exercise the real decoder and record a remote
miss, request/get/fallback timing, and no read error across both
adapters and both standalone engines
- fenced tracked writes prove stale-value unlinking while preserving the
exact watermark and its TTL trajectory
- cluster `SCRIPT FLUSH` recovery proves mutation scripts repopulate
every master and a subsequent identical read is a cache hit
- GLIDE package tests compile against the supported 2.0.0 floor and
exercise separate module instances plus packed ESM/CommonJS error
identity
- focused GLIDE primary/replica probe and three-node Cluster probe
- `git diff --check`
@lan17
lan17 force-pushed the agent/stale-on-error branch from a2f55ce to 388027c Compare August 19, 2026 23:35
@lan17
lan17 force-pushed the agent/stale-on-error branch from bd122cb to 439335c Compare August 25, 2026 06:27

@lan17 lan17 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review: stale-on-error recovery

Verdict: core semantics are sound; nothing here blocks the serving path. The F/M boundary math (strict 0 <= age < limit, shared by serving, recovery, and shadow C0 through the single validateFrameAge), recovery isolation (recovered values provably never reach Redis, the local LRU, or shadow — the writers at dialcache.ts:819/1065/1277 are all unreachable from the recovery return), original-rejection preservation, coalescing of the full read/source/recovery chain, exactly-one-staleRecovery-outcome accounting, and the tracked reread's watermark fencing all held up under adversarial verification. Typecheck and all 588 unit tests pass on the branch.

15 findings as inline comments — 4 medium, 11 low. The mediums cluster on observability and validation depth, not data correctness:

  • silent age-gate rejections (redis-cache.ts:474) — the only frame-rejection class with zero diagnostic;
  • README:452's "telemetry remains unchanged" vs the intentional double-emission into layer="remote";
  • M never validated against the tracked 1h cap (runtime-config.ts:137) — per-write tracked_ttl_clamped error noise plus a silently truncated recovery window;
  • the unreachable pre-metric throw in recoverWithResolvedConfig (redis-cache.ts:167) — the one zero-outcome recovery exit.

Verified non-issues (they look like bugs but aren't)

  • Dropping finishRedisChain's resolvedRemoteConfig parameter removed dead code: on main, the remoteErrored ? resolvedRemoteConfig arm could never execute — the call sites that passed the parameter only ever received hit/miss/error results.
  • Invalid-stamp frames classifying as recovery-eligible cache_miss is consistent with the documented taxonomy (README:612 groups timestamp-domain with watermark/future-time rejections); the excluded "invalid reply/encoding" class is thrown protocol errors, which already bypass recovery via the status: "error" arm.
  • The recovery read's inFallback: false on errors is correct per README:950's cache-plumbing-versus-application definition.

Below-the-cut minors (verified, not inlined)

  • The three benchmark scripts now share five copy-pasted helpers with no common module: deferred, readPositiveInteger, noOpMetrics (already drifted — the new copy has five members the old lacks), the cmdstat parse loop, and the connect boilerplate. A scripts/benchmark-lib.mjs stops the drift.
  • measureScenario's redis parameter has exactly one possible argument (every call site passes the module-level client), and four of its five per-command counters are stored but never read.
  • safeMetrics wraps staleRecovery in the 6-line presence-preserving spread copied from shadowValidation, but nothing gates on its presence (sole consumer is metrics.staleRecovery?.(...)); the one-line callObserver(() => metrics.staleRecovery?.(labels)) already used for compression in the same literal is behavior-identical.

Methodology: 9 independent finder angles → 20 adversarial verifiers (one per deduped candidate; 3 candidates refuted and withheld) → gap sweep. Two findings were validated by implementation: the reason-enum alternative to skipStaleRecovery compiles clean and passes all stale-on-error tests, flipping that failure mode from fail-open to fail-closed.

Comment thread src/internal/redis-cache.ts Outdated
Comment thread README.md Outdated
Comment thread src/internal/runtime-config.ts
Comment thread src/internal/redis-cache.ts Outdated
Comment thread src/internal/redis-cache.ts Outdated
Comment thread README.md Outdated
Comment thread scripts/benchmark-stale-on-error.mjs
Comment thread scripts/benchmark-stale-on-error.mjs Outdated
Comment thread test/dialcache-stale-on-error.test.ts
Comment thread test/dialcache-stale-on-error.test.ts Outdated
@lan17

lan17 commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

Follow-up on the below-the-cut scripts/benchmark-lib.mjs suggestion: I am not extracting a shared benchmark module in this PR. The overlap is pairwise rather than coherent across all three scripts (deferred and environment parsing in one pair; Redis connection and command-stat parsing in another), and their options and metrics semantics differ. A module now would create an abstraction around coincidences.

I will take the local simplifications, including the unused measureScenario parameter and counters. Cross-script extraction can wait until there is a stable common surface.

@lan17
lan17 force-pushed the agent/stale-on-error branch from 439335c to 9ae561d Compare August 25, 2026 19:51
BREAKING CHANGE: Ordinary Redis reads now treat every decoded frame createdAtMs as serving-authoritative. Custom Redis clients must return real epoch-millisecond writer timestamps, and deployments must roll out new readers before enabling physical M retention.
@lan17
lan17 force-pushed the agent/stale-on-error branch from 9ae561d to aa9365f Compare August 26, 2026 00:40

@lan17 lan17 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: stale-on-error direction

Overall, the implementation is sound and the F/M design is the right abstraction: F remains the ordinary freshness boundary, M is a retained recovery ceiling, the source stays authoritative while healthy, and stale data is considered only after a source rejection. The reread is a good correctness tradeoff because it rechecks age, invalidation/watermark state, eviction, and concurrent refresh at serving time. I did not find a serving-path correctness bug in the current head.

1. API safety: let use cases restrict which source errors qualify

The main concern is that every source rejection currently qualifies for stale recovery. That includes not only availability failures, but also domain failures such as NotFound-after-delete, authorization/permission errors, validation errors, cancellation/AbortError, and programmer/invariant failures.

For a generic cache library, I think the API should provide a stable per-use-case predicate, e.g.:

shouldServeStale?: (error: unknown) => boolean

and gate recovery on it. This belongs with stable use-case behavior (similar to shadowComparator), not runtime rollout config. Predicate failure/throw should fail closed and must never replace the original source rejection.

Keeping the current "all rejections" behavior as the default is defensible because positive M is already explicit opt-in, but if we do that the README should call out the domain-error implication prominently. My preference is to add the predicate before merge.

2. Observability: distinguish stale from concurrently refreshed recovery

staleRecovery{outcome="served"} can mean either:

  • an actually stale retained value was returned, or
  • another writer refreshed Redis while the source call was in flight, and the recovery reread returned that fresh value.

The latter behavior is correct and is one of the advantages of rereading, but the metric loses that distinction. Consider either splitting the outcome into served_fresh / served_stale, or adding a recovery-age observation. A recovery-age histogram is probably the more useful operational signal.

I would also add one explicit test for the concurrent-refresh case: initial read sees stale -> source blocks -> another writer stores fresh -> source rejects -> recovery returns the newly fresh value.

3. Docs: call out local TTL semantics

Stale-on-error controls Redis serving only. A process-local hit returns before Redis/source are consulted, so if ttlSec.local > ttlSec.remote, local cache can return something older than F without a source error. That is existing DialCache behavior, but it is easy to misread this feature as making F a global freshness boundary.

A short note such as "configure local TTL <= F when F must be the use case's global freshness boundary" would make this explicit.

Direction / composition

I would keep this feature as an availability fallback, not turn it into a circuit breaker. The intended composition is:

Redis read using F -> source / circuit breaker -> rejection -> Redis reread using M

A breaker protects the source; DialCache provides the retained fallback. A future runtime stale-first / degraded-mode policy could be useful for operator-driven load shedding, but it should be a separate explicit feature because it changes source-call semantics and can reduce outage Redis traffic from two payload reads to one.

The rest of the implementation looks strong: recovery does not republish into Redis/process-local/shadow, the original rejection wins on failed recovery, the required miss-reason gate fails closed, config is resolved once per invocation, and the createdAtMs compatibility / readers-first rollout hazard is now documented correctly.

@lan17

lan17 commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Following up on the latest review: I agree that stale recovery should fail closed by default rather than treating every source rejection as eligible.

I propose modeling this as one resolved policy with the following precedence:

per-use-case shouldAttemptStaleRecovery
→ DialCache-instance shouldAttemptStaleRecovery
→ built-in marker-error classifier

The marker should be named something like StaleRecoveryEligibleError, rather than ShouldServeStale: it only authorizes one recovery reread, which may miss or may return a concurrently refreshed fresh value.

The predicates should override, not augment, the lower-level policy. That lets a use case either tighten or broaden the instance default. Both existing structural gates still apply: recovery requires a configured positive M, a definitive initial Redis cache_miss, and an eligible source rejection. A predicate that throws, returns a non-boolean, or accidentally returns a promise should fail closed and must never replace the original source rejection.

This is stable application behavior, so the per-use-case option belongs alongside shadowComparator, not in runtime DialCacheKeyConfig. The "global" setting should be an instance default on DialCacheConfig, not process-global state.

The marker must remain application-visible. Calls outside an enabled cache context are true pass-through, and failed recovery must continue to rethrow the exact original error unchanged; DialCache should not unwrap or otherwise special-case it.

I recommend that DialCache's own FallbackTimeoutError remain eligible in the built-in policy because it represents exactly the availability failure this feature is intended to cover. A global or per-use-case override can still veto it.

lan17 commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Agree with the proposed precedence and fail-closed predicate semantics, but I would simplify the built-in fallback and avoid introducing a new StaleRecoveryEligibleError marker for now.

My preference:

per-use-case shouldAttemptStaleRecovery
→ DialCache-instance shouldAttemptStaleRecovery
→ built-in: error instanceof FallbackTimeoutError

Rationale: DialCache can classify its own timeout error with certainty, but arbitrary source errors belong to application-specific taxonomies. A DialCache-specific marker would couple deep source code to cache policy and encourage wrapping/subclassing errors just to communicate eligibility. That also works against the useful invariant that failed recovery rethrows the exact original rejection unchanged.

Applications that know their infrastructure errors can express that naturally at the instance level, e.g. FallbackTimeoutError || DatabaseUnavailableError || CircuitOpenError, while a per-use-case predicate can tighten or broaden that policy. First-defined-wins/override semantics are the right choice so a use case can explicitly veto an instance default.

I also agree the predicate should be synchronous and defensive: throw, non-boolean return, or Promise return => fail closed, and never replace the original source rejection.

I’d only add a marker later if we discover a concrete ergonomics need for deep source code to opt itself into recovery without central policy knowing its error taxonomy.

lan17 commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

One more design change after thinking through the outage behavior: I think we should drop the recovery reread and retain the initial eligible stale candidate instead.

Proposed flow:

initial Redis read
  age < F       -> return fresh
  F <= age < M  -> retain candidate in memory, treat as ordinary miss
  unavailable   -> ordinary miss with no recovery candidate

source succeeds -> return/write source value
eligible source rejection + retained candidate -> return candidate
otherwise -> rethrow original rejection

The reason is load behavior. During the sustained source outage this feature exists to survive, the current common path is:

Redis GET/MGET -> observe retained stale value
source -> fails
Redis GET/MGET again -> usually observe the same retained stale value

So the recovery mechanism can roughly double Redis read QPS and transfer the same payload twice exactly while another dependency is failing. Once a use case has explicitly opted into stale-on-error with M, it has already declared that a value of age < M is acceptable after an eligible source failure. I don't think picking up a concurrent refresh is worth making a second Redis read part of the normal outage path.

This also makes the mechanism simpler: M controls whether the initial read may retain a recovery candidate, F controls whether that candidate can serve normally, and the source-error classifier controls whether the retained candidate may be used.

The real semantic tradeoff is invalidation. A tracked candidate could be invalidated while the source call is in flight, and without the reread we would not observe that newer watermark before returning the retained value. I think that is acceptable for this feature: stale-on-error is explicitly opt-in per use case and is already a deliberate bounded-consistency relaxation during an availability failure. We should document that a retained recovery candidate reflects the Redis/invalidation snapshot from the initial read and may cross an invalidation that races with the source attempt.

If we decide that trackForInvalidation must remain a hard fence even during stale recovery, then tracked keys are the one case where a reread is justified. But my preference is to keep one coherent single-read recovery model rather than making outage traffic behavior depend on tracked vs untracked keys.

Combined with the earlier error-classification change, the model becomes:

one Redis read -> fresh return OR retain F..M candidate
                    ↓
                  source
              success | eligible failure
                 ↓             ↓
             fresh wins   retained candidate

This gives the feature much better failure amplification characteristics: one Redis read per request rather than potentially two during the outage.

Retain the initial F..M Redis frame instead of rereading after source failure, classify eligible errors with instance and per-use-case predicates, and record served value age.

Document tracked snapshot semantics and cover the one-read memory tradeoff in tests and benchmarks.
@lan17

lan17 commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Implemented the agreed follow-up as a new commit on this PR branch.

  • Added shouldAttemptStaleRecovery with per-use-case → instance → built-in precedence. The built-in admits only FallbackTimeoutError; overrides replace lower policy and fail closed on throws, non-booleans, or thenables.
  • Replaced the recovery reread with a single-read retained-snapshot model. The initial read serves < F, retains raw F..M, and recovery rechecks M before and after lazy deserialization.
  • Made tracked and untracked recovery use the same snapshot semantics. An invalidation before the initial tracked MGET fences; a racing invalidation after that snapshot does not revoke the retained bytes.
  • Kept configured M as the logical ceiling while preserving the existing one-hour tracked physical TTL cap.
  • Removed unreachable stale-recovery read_error / read_timeout outcomes and added served-value-age observations in core, Prometheus, and Datadog.
  • Updated the PR and issue design docs, public API/package checks, real Redis integration coverage, and the benchmark's high-cardinality raw-candidate memory case.

Validation on Node 22.22.0:

  • typecheck, build, and packed ESM/CJS consumers
  • 604 unit tests with 98.11% statement coverage
  • 139 live Redis/Valkey integration tests passed; 2 unavailable-cluster cases skipped after the expected timeout
  • benchmark assertions passed with one adapter read per independent recovery flight; 128 × 64 KiB retained candidates produced the expected +8.00 MiB external-memory observation and returned to baseline after recovery

lan17 commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Test coverage looks very strong now, especially around the actual state-machine/race semantics rather than just line coverage. The suite covers fresh vs retained behavior, source success, source failure, M crossing during the source and during async deserialization, Redis failure/expiry after the initial snapshot, refresh-after-read, invalidation before and after the tracked snapshot, compression/corruption, request/process coalescing, no shared-cache publication, runtime-policy changes, and the full classifier precedence/fail-closed behavior. The real Redis/Valkey integration coverage for tracked/untracked recovery and invalidation races is also valuable.

I don't see a test-coverage blocker. Two small additions would make the contract even tighter:

  1. Pin the initial-read boundary table explicitly, especially exact M. Ideally cover:
age = F - 1ms -> fresh
age = F       -> retained
age = M - 1ms -> retained
age = M       -> unavailable / no candidate

There are already tests that imply most of this (including candidates crossing M later), but direct boundary tests make the public strict < F / < M contract harder to accidentally change.

  1. Assert classifier denial never deserializes the retained candidate. Use a spy/custom serializer and a retained F..M frame, have shouldAttemptStaleRecovery return false, and verify serializer.load is never called and no staleRecovery outcome is emitted. That pins an important outage-performance property: denied/domain failures shouldn't spend CPU decompressing/deserializing potentially large retained payloads.

Both are polish rather than blockers; overall the recovery test matrix is in very good shape.

@lan17

lan17 commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Added both suggested tests in a separate tests-only commit.

  • The initial-read matrix now asserts the Redis classifier directly: F - 1 ms -> hit, F -> retained, M - 1 ms -> retained, and M -> miss. Each frame remains physically present through M + 1s, so the exact-M row proves logical expiry rather than fake-Redis key expiry.
  • The classifier-denial case uses a retained candidate plus a custom serializer and a false per-use-case override over an allowing instance predicate. It asserts the original source rejection, one classifier call, no serializer.load or dump, no staleRecovery outcome, and one Redis read.

Validation on Node 22.22.0 is green locally: 50 focused tests, 607 full unit tests, typecheck, build, packed ESM/CJS consumers, and 139 live Redis/Valkey integration tests passed with the 2 expected unavailable-cluster skips.

@lan17
lan17 marked this pull request as ready for review August 30, 2026 02:32
@lan17
lan17 merged commit 365786f into main Aug 30, 2026
7 checks passed
@lan17
lan17 deleted the agent/stale-on-error branch August 30, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add opt-in stale-on-error recovery from retained Redis values

1 participant