feat: add stale-on-error Redis recovery - #121
Conversation
## Summary Replace read-side Lua with native Redis commands and decode DialCache's frame in TypeScript: - untracked reads use `GET` - tracked reads use one atomic, primary-routed `MGET` for the value and watermark - write and invalidation remain Lua-backed; a watermark-fenced tracked write now atomically unlinks the stale value it rejects - node-redis registers only the three mutation scripts, and GLIDE owns only the three mutation script handles - custom adapters can reuse the public `decodeRedisFrame` and `decodeTrackedRedisFrame` helpers This removes the Redis-to-Lua payload materialization and `string.sub` copy on every hit while preserving the semantic `DialCacheRedisClient.read()` boundary. ## Read architecture | Adapter / mode | Untracked | Tracked | Primary guarantee | | --- | --- | --- | --- | | node-redis standalone | `GET` | `MGET` | standalone connection | | node-redis Cluster | `GET` | raw `MGET` | `sendCommand(..., false, ...)` routes to the slot primary | | GLIDE standalone | `GET` | one-command `Batch(false).mget(...)` | standalone batches execute on the primary even with replica reads configured; `MGET` itself is atomic | | GLIDE Cluster | `GET` | custom-command `MGET` | explicit `primarySlotKey` route | The shared decoder: - validates the frame version and minimum length - preserves missing/short/unsupported frames as clean misses - parses integer and fractional legacy watermarks with the same accepted grammar as Lua - rejects values whose Redis-created timestamp is at or before the watermark - preserves unsupported payload encodings as `DialCacheRedisPayloadEncodingError` - returns binary payloads through a zero-copy `Buffer.subarray()` view Tracked value and watermark reads retain one atomic snapshot, with both values returned by a single `MGET`. Their existing shared Cluster hash tag remains required; mismatched tags still fail with `CROSSSLOT`. ## Breaking change - `READ_CACHE_SCRIPT` and `READ_TRACKED_CACHE_SCRIPT` are removed from `dialcache/redis-protocol`. - `dialcacheRedisScripts.dialcacheRead` and `dialcacheRedisScripts.dialcacheReadTracked` are removed from `dialcache/node-redis`. - Custom node-redis wrappers must expose native `get` / `sendCommand`; `legacyMode` clients are unsupported because neither their callback surface nor `.v4` view exposes the complete native-command-plus-custom-script contract. - The GLIDE helper requires GLIDE 2.x, a direct official `GlideClient` or `GlideClusterClient`, and the same module namespace that created it. Forwarding wrappers should implement `DialCacheRedisClient` directly because their topology cannot be inferred safely. - Official node-redis clients and direct GLIDE 2.x clients passed through the documented helpers keep the same application-facing call shape, so those consumers can bump the package without code changes. - Redis keys, frame format, and invalidation behavior are unchanged. A tracked write rejected by an active future watermark still returns `false`, but now also unlinks the stale value key. No data migration or cache flush is required. - The fenced-write cleanup requires `UNLINK` (Redis 4.0+ or compatible Valkey) and permission for scripts to invoke it. With a command-restricted ACL that denies `UNLINK`, the write fails open as `cache_write` and leaves the stale value for a later cleanup or expiry. `BREAKING CHANGE:` the four deprecated read-Lua exports and registrations above are removed; node-redis adapters require the promise-mode native-command surface; the GLIDE helper requires a direct GLIDE 2.x client from the supplied runtime; and the fenced-write cleanup requires Redis `UNLINK` support plus ACL permission. Under the repository's release configuration, this change should release as `v1.0.0`. ## Adapter behavior changes - The node-redis factory now requires native `get` and `sendCommand` methods in addition to the three registered mutation methods. - The GLIDE factory declares an optional `@valkey/valkey-glide ^2.0.0` peer, validates `Batch` support eagerly, and classifies standalone versus cluster behavior from the supplied runtime's client identities before allocating scripts. Its standalone non-atomic primary batch avoids consuming caller-owned `WATCH` state. - Redis `MGET` returns `null` for wrong-type members. A tracked wrong-type value is therefore a clean miss and may be repaired with a valid DialCache frame after fallback succeeds, while a wrong-type watermark prevents the tracked write from succeeding. An untracked `GET` still surfaces `WRONGTYPE`. Real-engine tests cover both repair and repeated fail-open behavior, including metrics. - The public read contract now specifies frame decoding, miss and watermark rules, atomic authoritative snapshots, and returned-buffer ownership. Shared decoders validate leaf reply types; adapters retain only client-specific envelope validation. ## Benchmark The benchmark harness and JSON results were intentionally kept outside the repository. Methodology: - Redis 6.2.22 and Valkey 8.1.8 - Node 22.22.0, node-redis 4.7.1, GLIDE 2.4.2 - binary payloads of 100 B, 1 KiB, 10 KiB, 100 KiB, and 1 MiB - fresh untracked hit, fresh tracked hit, and invalidated tracked miss - three alternating rounds, one command in flight, loopback Docker - median throughput, latency, Redis `INFO commandstats` execution time, and network bytes At 1 MiB, native fresh-hit throughput improved 15-45% across the two engines and adapters. Server-reported command execution time per logical read fell 95-98%. Small 100 B / 1 KiB end-to-end results were mostly flat/noisy while reported command time still fell about 80-90%; the notable small-case regression was Redis/node-redis's 100 B tracked hit at about -10% throughput. These loopback, one-in-flight results are directional rather than production-capacity measurements. Representative Redis 6.2 + node-redis medians: | 1 MiB scenario | Lua ops/s | Native ops/s | Lua server us/read | Native server us/read | Lua -> native p50 | | --- | ---: | ---: | ---: | ---: | ---: | | untracked hit | 230 | 269 | 719.8 | 32.6 | 3.718 ms -> 2.955 ms | | tracked hit | 217 | 259 | 713.7 | 31.2 | 3.630 ms -> 3.016 ms | | invalidated tracked miss | 1,762 | 284 | 361.6 | 31.0 | 0.566 ms -> 2.949 ms | The invalidated-miss row is the main tradeoff: Lua returns only a null reply, while native `MGET` transfers the stale frame before TypeScript rejects it. At 1 MiB this changes roughly 3-5 response bytes into about 1.05 MB. Across both engines and adapters, invalidated-miss throughput fell 77-84% at 1 MiB (46-58% at 100 KiB), even though server-reported command time still fell 91-94%. The benchmark intentionally measured the read itself and therefore includes that full transfer. In the application path, the first completed fallback that reaches a still-fenced tracked write now atomically unlinks the stale value, bounding subsequent transfers for that entry. This is only a partial mitigation: a read failure or timeout never reaches the write-side cleanup, so the stale payload can continue to transfer or time out until another completed read cleans it up or its TTL expires. ## Scope This branch is updated onto the current `v0.15.0` read contract, including the untracked-cache shadowing changes from #122. It deliberately does not include the server-time / maximum-age behavior proposed in #121. That work can be evaluated separately against this read path and its benchmark tradeoffs. ## Validation - `corepack pnpm typecheck` - `corepack pnpm test` - 424 tests, coverage thresholds passed - `corepack pnpm build` - `corepack pnpm test:package` - including real node-redis and GLIDE standalone and Cluster consumer types, plus packed ESM/CommonJS absence checks for all four removed APIs - `corepack pnpm test:integration` - 113 tests across Redis 6.2, Valkey 8, and Redis Cluster - tracked wrong-type value repair and repeated wrong-type watermark fail-open behavior exercised end to end across both adapters and both standalone engines - stale tracked frames exercise the real decoder and record a remote miss, request/get/fallback timing, and no read error across both adapters and both standalone engines - fenced tracked writes prove stale-value unlinking while preserving the exact watermark and its TTL trajectory - cluster `SCRIPT FLUSH` recovery proves mutation scripts repopulate every master and a subsequent identical read is a cache hit - GLIDE package tests compile against the supported 2.0.0 floor and exercise separate module instances plus packed ESM/CommonJS error identity - focused GLIDE primary/replica probe and three-node Cluster probe - `git diff --check`
a2f55ce to
388027c
Compare
bd122cb to
439335c
Compare
lan17
left a comment
There was a problem hiding this comment.
Deep review: stale-on-error recovery
Verdict: core semantics are sound; nothing here blocks the serving path. The F/M boundary math (strict 0 <= age < limit, shared by serving, recovery, and shadow C0 through the single validateFrameAge), recovery isolation (recovered values provably never reach Redis, the local LRU, or shadow — the writers at dialcache.ts:819/1065/1277 are all unreachable from the recovery return), original-rejection preservation, coalescing of the full read/source/recovery chain, exactly-one-staleRecovery-outcome accounting, and the tracked reread's watermark fencing all held up under adversarial verification. Typecheck and all 588 unit tests pass on the branch.
15 findings as inline comments — 4 medium, 11 low. The mediums cluster on observability and validation depth, not data correctness:
- silent age-gate rejections (
redis-cache.ts:474) — the only frame-rejection class with zero diagnostic; - README:452's "telemetry remains unchanged" vs the intentional double-emission into
layer="remote"; Mnever validated against the tracked 1h cap (runtime-config.ts:137) — per-writetracked_ttl_clampederror noise plus a silently truncated recovery window;- the unreachable pre-metric throw in
recoverWithResolvedConfig(redis-cache.ts:167) — the one zero-outcome recovery exit.
Verified non-issues (they look like bugs but aren't)
- Dropping
finishRedisChain'sresolvedRemoteConfigparameter removed dead code: on main, theremoteErrored ? resolvedRemoteConfigarm could never execute — the call sites that passed the parameter only ever received hit/miss/error results. - Invalid-stamp frames classifying as recovery-eligible
cache_missis consistent with the documented taxonomy (README:612 groups timestamp-domain with watermark/future-time rejections); the excluded "invalid reply/encoding" class is thrown protocol errors, which already bypass recovery via thestatus: "error"arm. - The recovery read's
inFallback: falseon errors is correct per README:950's cache-plumbing-versus-application definition.
Below-the-cut minors (verified, not inlined)
- The three benchmark scripts now share five copy-pasted helpers with no common module:
deferred,readPositiveInteger,noOpMetrics(already drifted — the new copy has five members the old lacks), the cmdstat parse loop, and the connect boilerplate. Ascripts/benchmark-lib.mjsstops the drift. measureScenario'sredisparameter has exactly one possible argument (every call site passes the module-level client), and four of its five per-command counters are stored but never read.safeMetricswrapsstaleRecoveryin the 6-line presence-preserving spread copied fromshadowValidation, but nothing gates on its presence (sole consumer ismetrics.staleRecovery?.(...)); the one-linecallObserver(() => metrics.staleRecovery?.(labels))already used forcompressionin the same literal is behavior-identical.
Methodology: 9 independent finder angles → 20 adversarial verifiers (one per deduped candidate; 3 candidates refuted and withheld) → gap sweep. Two findings were validated by implementation: the reason-enum alternative to skipStaleRecovery compiles clean and passes all stale-on-error tests, flipping that failure mode from fail-open to fail-closed.
|
Follow-up on the below-the-cut I will take the local simplifications, including the unused |
439335c to
9ae561d
Compare
BREAKING CHANGE: Ordinary Redis reads now treat every decoded frame createdAtMs as serving-authoritative. Custom Redis clients must return real epoch-millisecond writer timestamps, and deployments must roll out new readers before enabling physical M retention.
9ae561d to
aa9365f
Compare
lan17
left a comment
There was a problem hiding this comment.
Review: stale-on-error direction
Overall, the implementation is sound and the F/M design is the right abstraction: F remains the ordinary freshness boundary, M is a retained recovery ceiling, the source stays authoritative while healthy, and stale data is considered only after a source rejection. The reread is a good correctness tradeoff because it rechecks age, invalidation/watermark state, eviction, and concurrent refresh at serving time. I did not find a serving-path correctness bug in the current head.
1. API safety: let use cases restrict which source errors qualify
The main concern is that every source rejection currently qualifies for stale recovery. That includes not only availability failures, but also domain failures such as NotFound-after-delete, authorization/permission errors, validation errors, cancellation/AbortError, and programmer/invariant failures.
For a generic cache library, I think the API should provide a stable per-use-case predicate, e.g.:
shouldServeStale?: (error: unknown) => booleanand gate recovery on it. This belongs with stable use-case behavior (similar to shadowComparator), not runtime rollout config. Predicate failure/throw should fail closed and must never replace the original source rejection.
Keeping the current "all rejections" behavior as the default is defensible because positive M is already explicit opt-in, but if we do that the README should call out the domain-error implication prominently. My preference is to add the predicate before merge.
2. Observability: distinguish stale from concurrently refreshed recovery
staleRecovery{outcome="served"} can mean either:
- an actually stale retained value was returned, or
- another writer refreshed Redis while the source call was in flight, and the recovery reread returned that fresh value.
The latter behavior is correct and is one of the advantages of rereading, but the metric loses that distinction. Consider either splitting the outcome into served_fresh / served_stale, or adding a recovery-age observation. A recovery-age histogram is probably the more useful operational signal.
I would also add one explicit test for the concurrent-refresh case: initial read sees stale -> source blocks -> another writer stores fresh -> source rejects -> recovery returns the newly fresh value.
3. Docs: call out local TTL semantics
Stale-on-error controls Redis serving only. A process-local hit returns before Redis/source are consulted, so if ttlSec.local > ttlSec.remote, local cache can return something older than F without a source error. That is existing DialCache behavior, but it is easy to misread this feature as making F a global freshness boundary.
A short note such as "configure local TTL <= F when F must be the use case's global freshness boundary" would make this explicit.
Direction / composition
I would keep this feature as an availability fallback, not turn it into a circuit breaker. The intended composition is:
Redis read using F -> source / circuit breaker -> rejection -> Redis reread using M
A breaker protects the source; DialCache provides the retained fallback. A future runtime stale-first / degraded-mode policy could be useful for operator-driven load shedding, but it should be a separate explicit feature because it changes source-call semantics and can reduce outage Redis traffic from two payload reads to one.
The rest of the implementation looks strong: recovery does not republish into Redis/process-local/shadow, the original rejection wins on failed recovery, the required miss-reason gate fails closed, config is resolved once per invocation, and the createdAtMs compatibility / readers-first rollout hazard is now documented correctly.
|
Following up on the latest review: I agree that stale recovery should fail closed by default rather than treating every source rejection as eligible. I propose modeling this as one resolved policy with the following precedence: The marker should be named something like The predicates should override, not augment, the lower-level policy. That lets a use case either tighten or broaden the instance default. Both existing structural gates still apply: recovery requires a configured positive This is stable application behavior, so the per-use-case option belongs alongside The marker must remain application-visible. Calls outside an enabled cache context are true pass-through, and failed recovery must continue to rethrow the exact original error unchanged; DialCache should not unwrap or otherwise special-case it. I recommend that DialCache's own |
|
Agree with the proposed precedence and fail-closed predicate semantics, but I would simplify the built-in fallback and avoid introducing a new My preference: Rationale: DialCache can classify its own timeout error with certainty, but arbitrary source errors belong to application-specific taxonomies. A DialCache-specific marker would couple deep source code to cache policy and encourage wrapping/subclassing errors just to communicate eligibility. That also works against the useful invariant that failed recovery rethrows the exact original rejection unchanged. Applications that know their infrastructure errors can express that naturally at the instance level, e.g. I also agree the predicate should be synchronous and defensive: throw, non-boolean return, or Promise return => fail closed, and never replace the original source rejection. I’d only add a marker later if we discover a concrete ergonomics need for deep source code to opt itself into recovery without central policy knowing its error taxonomy. |
|
One more design change after thinking through the outage behavior: I think we should drop the recovery reread and retain the initial eligible stale candidate instead. Proposed flow: The reason is load behavior. During the sustained source outage this feature exists to survive, the current common path is: So the recovery mechanism can roughly double Redis read QPS and transfer the same payload twice exactly while another dependency is failing. Once a use case has explicitly opted into stale-on-error with This also makes the mechanism simpler: The real semantic tradeoff is invalidation. A tracked candidate could be invalidated while the source call is in flight, and without the reread we would not observe that newer watermark before returning the retained value. I think that is acceptable for this feature: stale-on-error is explicitly opt-in per use case and is already a deliberate bounded-consistency relaxation during an availability failure. We should document that a retained recovery candidate reflects the Redis/invalidation snapshot from the initial read and may cross an invalidation that races with the source attempt. If we decide that Combined with the earlier error-classification change, the model becomes: This gives the feature much better failure amplification characteristics: one Redis read per request rather than potentially two during the outage. |
Retain the initial F..M Redis frame instead of rereading after source failure, classify eligible errors with instance and per-use-case predicates, and record served value age. Document tracked snapshot semantics and cover the one-read memory tradeoff in tests and benchmarks.
|
Implemented the agreed follow-up as a new commit on this PR branch.
Validation on Node 22.22.0:
|
|
Test coverage looks very strong now, especially around the actual state-machine/race semantics rather than just line coverage. The suite covers fresh vs retained behavior, source success, source failure, M crossing during the source and during async deserialization, Redis failure/expiry after the initial snapshot, refresh-after-read, invalidation before and after the tracked snapshot, compression/corruption, request/process coalescing, no shared-cache publication, runtime-policy changes, and the full classifier precedence/fail-closed behavior. The real Redis/Valkey integration coverage for tracked/untracked recovery and invalidation races is also valuable. I don't see a test-coverage blocker. Two small additions would make the contract even tighter:
There are already tests that imply most of this (including candidates crossing M later), but direct boundary tests make the public strict
Both are polish rather than blockers; overall the recovery test matrix is in very good shape. |
|
Added both suggested tests in a separate tests-only commit.
Validation on Node 22.22.0 is green locally: 50 focused tests, 607 full unit tests, typecheck, build, packed ESM/CJS consumers, and 139 live Redis/Valkey integration tests passed with the 2 expected unavailable-cluster skips. |
Summary
Add opt-in stale-on-error recovery from the existing Redis frame, on top of the native client-clock protocol from #140.
F = ttlSec[CacheLayer.REMOTE]remains the ordinary Redis freshness boundary.M = staleOnErrorMaxAgeSecis the absolute stale-recovery age ceiling and requested physical retention.GETor trackedMGET(value, watermark)either serves a fresh frame, retains anF..Mraw candidate, or misses without a candidate.Closes #117
Configuration and public API
New API surface:
DialCacheKeyConfig.staleOnErrorMaxAgeSec?: numberDialCacheConfig.shouldAttemptStaleRecovery?: StaleRecoveryPredicateshouldAttemptStaleRecoveryoncached()andgetOrLoad()optionsStaleRecoveryPredicate,StaleRecoveryOutcome, andStaleRecoveryMetricLabelsDialCacheMetricsAdapter.staleRecovery(labels)andobserveStaleRecoveryValueAge(labels, seconds)hooksClassifier precedence — highest wins, and an override replaces the levels below it rather than composing with them:
shouldAttemptStaleRecoveryoncached()/getOrLoad()optionsDialCacheConfig.shouldAttemptStaleRecoveryerror instanceof FallbackTimeoutErroronlyThe intended split: put the broad policy for transient, retriable infrastructure failures on the instance, and use a per-use-case override where particular data needs a stricter policy. Predicates should deny authoritative domain outcomes — auth/permission/entitlement failures, revocation, deletion or not-found, validation, and programmer errors — where a stale value would be wrong rather than merely old.
Predicates must synchronously return a boolean; a throw, non-boolean, or thenable fails closed, and an accidental rejecting thenable is consumed.
cached()snapshots its selected predicate at registration;getOrLoad()resolves it per invocation. Disabled calls remain true pass-through and never invoke it.Omitting
Mkeeps recovery off; a sparse runtime overlay that omits it inherits the configured default, and an explicit0disables an inherited value. A positive policy requires0 < F < M <= 31,536,000seconds. Invalid static policy throws before registration; invalid runtimeMrecordsconfig_resolution, disables only recovery for that invocation, and preserves otherwise-valid ordinary Redis policy.DialCacheKeyConfig.disabled()setsMto0.Execution
The candidate stays serialized/compressed until the source call settles. On an eligible rejection, DialCache checks
0 <= age < M, lazily deserializes/decompresses, checksMagain after that potentially asynchronous work, and only then returns it. Equality atFis logically stale; equality atMis unavailable.A successful source refresh wins and writes a newly timestamped frame. Recovery does not write or extend Redis, populate process-local cache, schedule shadow validation, or emit a shadow-age observation. Request-local caching may memoize the returned reference only within the active outer
enable()scope. Existing coalescing shares the whole read/source/recovery decision;coalesce: falsegives each caller an independent candidate and source attempt.Snapshot and freshness semantics
The retained candidate is the initial Redis snapshot:
MGET;For tracked use cases this opt-in path can weaken the usual strict freshness guarantee when invalidation races with the source attempt. Use cases that cannot tolerate that bounded relaxation should leave recovery disabled or deny the error in their classifier.
FandMbound Redis serving only. Request-local and process-local layers occur earlier and keep their own scope/TTL lifetimes. A frame can be nearlyFold when it warms process-local cache and then receive a full local TTL, sottlSec.local <= Fdoes not makeFa strict end-to-end age limit; disable earlier layers when that is required.Redis, time, and compatibility
There is no Redis envelope or key change: frame v1,
:dialcache-frame-v1,DialCacheRedisClient, andRedisReadRequestremain unchanged. Reads use nativeGET/primary-routedMGET, writes use native complete-frameSET, and invalidation remains the only Lua operation. No RedisTIMEor clock-offset estimation is added.Writers request physical retention through
Mwhen enabled andFotherwise. Tracked values keep the existing one-hour physical TTL cap andtracked_ttl_clampedsignal from #140. The configured logicalMis not clipped, but a tracked frame may physically disappear before reaching it.Core now treats every decoded frame's real writer
createdAtMsas authoritative for ordinary logicalF, including untracked reads. Custom Redis clients that returned a constant timestamp must return the actual epoch-millisecond frame stamp before upgrading. Roll out readers whileMis omitted or0, upgrade the complete fleet, and only then enable positiveM. Once a writer retains throughM, do not reintroduce a pre-feature reader until affected keys have expired or been removed; older readers trust physical presence and can serveF..Mas fresh.Application-process clock skew can move the boundary early or late. Future-dated frames fail closed and use the existing future-offset observation. Durations and deadlines remain monotonic.
Metrics
New series, exposed by both bundled backends:
dialcache_stale_recovery_countercache_namespace,use_case,key_type,outcomedialcache_stale_recovery_value_age_histogram[1, 5, 15, 60, 300, 900, 3600, 10800, 43200, 86400, 259200, 604800]s (shared with shadow value age)cache_namespace,use_case,key_type,outcomeserved; value = return-time age in secondsdialcache.stale_recovery.countcache_namespace,use_case,key_type,outcomedialcache.stale_recovery.value_ageobservationMetricTypeoptioncache_namespace,use_case,key_type,outcomeservedLabel reference:
cache_namespace—DialCacheConfig.namespace(defaulturn); present on every DialCache metric.use_case/key_type— the registered use case and key type; never the cache key or id.outcome— the bounded root-exportedStaleRecoveryOutcomeunion:served— the retained candidate was returned to the caller (age < Mheld both before and after lazy deserialization);miss— no candidate was retained by the initial read, or the candidate's age reachedM(or its stamp became future/invalid) by check time;deserialization_error— a candidate within age failed deserialize/decompress.outcomelabel is structurally alwaysserved— it exists so both outcome series share one label schema.layerlabel: recovery reuses the caller-serving initial read, so there is no separate layer to attribute.StaleRecoveryMetricLabels { cacheNamespace, useCase, keyType, outcome }and map it to their backend's naming, as the bundled adapters do.The backing
DialCacheMetricsAdapter.staleRecovery(labels)andobserveStaleRecoveryValueAge(labels, seconds)hooks are optional — existing custom adapters keep compiling, and omitting a hook skips only that observation, never recovery itself. A rejection the classifier denies emits no stale-recovery outcome: the counter measures authorized checks, not all source failures. No raw error, exception message, or cache key ever enters a label.Recovery adds no second ordinary
request,observeGet,miss, cache-read error, or Redis command — the initial read is the one caller-serving telemetry trail. Existing fallback error/duration telemetry still records the source rejection even when recovery serves, so a served recovery stays visibly paired with the failure that caused it.Changes to existing series, visible on upgrade even where recovery stays off (all carry their usual
cache_namespace,use_case,key_typelabels;layer/error/in_fallbackas noted):dialcache_miss_counter/ hit rate, atlayer="remote"— logicalFis now enforced from every frame's realcreatedAtMs, including untracked reads, so physically present but logically old frames that previously served now count as misses. Keyspaces that relied on physical-TTL slack will see a one-time remote miss-rate rise. With recovery enabled, a retainedF..Mcandidate also records an ordinary miss even when recovery later serves it — during incidents, treatstaleRecovery{outcome="served"}as its own population rather than expecting it in hit-rate math.dialcache_request_counterand the get-duration histogram are unchanged in population: still exactly one per caller-serving read.dialcache_future_timestamp_offset_histogram(layer="remote"and shadow layers) — its population widened: previously it observed only tracked frames, but every serving read now validates the stamp, so future-dated untracked frames are observed (and fail closed) as well. The Prometheus help string changed accordingly.dialcache_error_counterwitherror="tracked_ttl_clamped",layer="remote",in_fallback="false"— enablingMabove one hour on a tracked use case makes every dispatched tracked write request TTLMand get clamped, so this existing configuration signal fires per write for such configs (see the Redis section above). Alert rules keyed on the error counter without anerrorlabel filter will absorb this as a steady rate.dialcache_error_counterwitherror="config_resolution"— an invalid runtimeMoverlay records one per invocation while recovery is disabled for that invocation and ordinary reads continue.Cost
The Redis outage-path cost drops from two payload reads to one. The tradeoff moves to Node memory: one raw candidate is retained per distinct in-flight key through the source attempt; same-key coalesced followers share it. The benchmark includes a delayed, high-cardinality, incompressible-payload scenario rather than relying on the highly compressed default fixture.
Validation
Node 22.22.0:
BREAKING CHANGE: Ordinary Redis reads now enforce logical age from each frame's real
createdAtMs, including untracked reads. Deploy new readers before enabling physicalMretention.