Skip to content

feat(redis): simplify tracked writes with client clocks - #140

Merged
lan17 merged 6 commits into
mainfrom
codex/client-clock-redis-timestamps
Aug 25, 2026
Merged

feat(redis): simplify tracked writes with client clocks#140
lan17 merged 6 commits into
mainfrom
codex/client-clock-redis-timestamps

Conversation

@lan17

@lan17 lan17 commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Summary

Finish the client-clock simplification on top of main:

  • every tracked and untracked Redis value write is one native SET containing a complete frame-v1 value stamped once with writer Date.now();
  • tracked writes never read, create, or extend watermarks;
  • tracked Redis value TTLs are capped at one hour, while invalidation-owned watermarks live for at least two hours and long enough to outlive the future fence;
  • tracked reads retain the primary-routed atomic MGET(value, watermark), with a missing watermark treated as zero; and
  • invalidation is the only remaining Lua operation. Redis TIME is unused.

This PR contains no rollout gate. Old and new tracked protocols are gated and drained externally rather than allowed to coexist.

Why

The previous tracked write paired a placeholder SET with a Lua stamp that read Redis server time. Same-clock fencing was exact, but the write path carried two commands, per-write Lua, placeholder/nonce pairing state, and a split-pair failure mode: a sustained stamp failure (a denied command, a proxy rejecting EVALSHA) still landed every paired SET, replacing readable values with unreadable placeholders and shifting full tracked traffic to the source within one TTL horizon. This PR trades exact server-clock fencing for a documented client-clock buffer: every write is one atomic native SET that either lands servable or does not exist, and the placeholder, nonce, stamp-reply, and script-recovery machinery is deleted. The cost is stated rather than hidden — invalidation correctness now depends on the futureBufferMs sizing and watermark durability requirements below.

Architecture

Path Result
Untracked read Native GET; createdAtMs is informational and does not gate serving
Tracked read Primary atomic MGET(value, watermark); serving and initial-shadow reads require createdAtMs > watermark and not future-dated
Any Redis value write One complete-frame SET PX, stamped once by the adapter
Invalidation Monotonic Lua with one stable invalidator timestamp reused across retries

A malformed string watermark misses. Native MGET returns nil for a wrong-type member, so a wrong-type watermark has the same read behavior as an absent one. The next explicit invalidation repairs a wrong-type watermark; other script read errors surface and cannot bypass the monotonic update.

The value envelope, timestamp field, Redis key derivation, watermark key/value encoding, and tracked primary-read routing are unchanged.

Correctness

This is a read-time invalidation fence. A write behind an active watermark may be stored, but it remains a tracked miss. A tracked invocation that reaches the Redis path does not publish its fallback directly to process-local cache; a later validated Redis hit may warm it. Local-only, remote-policy-disabled, and ramped-down paths retain their existing local-cache behavior.

Production invalidation requires:

futureBufferMs >= Dmax + S + M
  • Dmax: maximum elapsed time from invalidation sampling until a stale pre-mutation SET can become visible in Redis, including source visibility, fallback, serialization, bounded client queue/reconnect delay, network transit, and Redis execution;
  • S: maximum writer-clock lead over the invalidator; and
  • M: operational margin.

For example: fills visible within 2 s, fleet clock skew at most 1 s, and a 1 s margin call for futureBufferMs ≈ 4000.

An unbounded offline queue or retry path makes a finite Dmax impossible. Future-dated tracked frames fail closed on serving and initial-shadow reads and emit the optional future-timestamp-offset observation. Confirmation reads observe but retain the frame for payload comparison; untracked reads do not gate serving on the informational timestamp.

Only invalidation maintains watermarks. Their TTL is:

max(existing TTL, 2 hours, watermark - invalidatedAtMs + 1 hour + 1 minute)

Persistent watermarks remain persistent. Each dispatched tracked write whose configured TTL is capped emits error="tracked_ttl_clamped" and attempts the write with the capped TTL. Losing a watermark removes its read-time fence, so production must use noeviction or an equivalent guarantee and alert on memory pressure, rejected writes, and evictions. Watermark loss also includes async-replication failover — a promoted replica may lack the newest watermark — so choose replication and failover guarantees accordingly; DialCache does not issue WAIT.

Performance and simplification

The tracked write path drops from SET + stamp Lua to one SET:

  • one command/reply instead of two;
  • no placeholder, nonce, stamp reply state, write batch, transaction, write-side Lua, or stamp script-cache recovery;
  • payload bytes still cross the network once; and
  • less fixed client, Redis, replication, and AOF work at high write rates.

The maintained live benchmark exercised 6,750 sequential writes across 100 B, 10 KiB, 100 KiB, and 1 MiB payloads. Every size produced exactly 1.0 SET/op, 0.0 script/op, and 0.0 TIME/op. Absolute latency remains environment-dependent; the benchmark validates command shape and sequential latency, not saturated throughput.

Tracked reads intentionally remain MGET(value, watermark): correctness needs an atomic primary snapshot, and invalidated misses still transfer the full payload before Node applies the fence. Standalone GLIDE retains its one-command Batch so tracked reads cannot follow a replica-read preference.

Compatibility and rollout

Wire formats remain compatible, but old tracked state is not safe to carry across the protocol transition because old watermark lifetimes were derived for the old writer.

Before enabling this release for a namespace:

  1. Stop and drain every old writer and invalidator, including fallbacks, shadow work, Redis client queues, and other in-flight operations that can still write tracked state.
  2. Purge every tracked value, complete or placeholder, and every watermark in the affected namespace. A full namespace flush is simplest when Redis is dedicated; untracked complete values may remain.
  3. Enable the new release only after the purge completes.

Waiting for natural expiry is safe only with all traffic disabled, bounded remaining TTLs for both tracked values and watermarks, no persistent watermark, and a wait covering the old value and future-buffer-derived watermark lifetimes.

Source compatibility changes:

  • DialCacheRedisClient.write() returns void, and RedisWriteRequest no longer accepts watermarkKey;
  • placeholder/stamp protocol exports and DialCacheRedisPlaceholderLostError are removed;
  • dialcacheRedisScripts and DialCacheNodeRedisScripts are removed; ordinary node-redis clients require no DialCache script registration;
  • ValkeyGlideRuntime no longer requires ClusterBatch;
  • fill_blocked is removed from ShadowValidationOutcome;
  • tracked_ttl_clamped is added to MetricErrorKind, so exhaustive switches and Record<MetricErrorKind, ...> values must add it; and
  • the Prometheus future-timestamp histogram uses a new skew-oriented bucket schema, which is incompatible with a same-name collector registered with the former default buckets.

Validation

  • corepack pnpm check: typecheck, 529 unit tests with coverage, ESM/CJS build, and packed-consumer tests
  • corepack pnpm test:integration: 125 passed, 0 skipped, across Redis 6.2 and Valkey 8 with node-redis and GLIDE, including the GLIDE Cluster suite
  • live Redis write benchmark: 6,750 writes, one SET and zero Lua/TIME calls per operation through 1 MiB
  • independent multi-pass adversarial review (10 finder angles, verification, gap sweep) produced 15 findings — 2 corruption-class correctness, 1 shadow-telemetry, 2 doc-contract, plus operability and cleanup items; 14 applied in 8068f46, 1 simplification declined (timestamp assert kept as defense in depth)
  • refreshed origin/main: branch remains based directly on main

Closes #139

BREAKING CHANGE: DialCacheRedisClient.write() now returns void and RedisWriteRequest no longer accepts watermarkKey. The placeholder/stamp protocol and related exports, dialcacheRedisScripts/DialCacheNodeRedisScripts registration facade, ValkeyGlideRuntime.ClusterBatch requirement, and fill_blocked outcome are removed; MetricErrorKind adds tracked_ttl_clamped; and the Prometheus future-timestamp histogram bucket schema changes. Node-redis clients should be created without DialCache script registration.

BREAKING CHANGE: node-redis registered script methods and direct Lua consumers must append client timestamps; semantic Redis request shapes and frame v1 remain unchanged.
@lan17 lan17 changed the title feat(redis): use client clocks for timestamps feat(redis): simplify tracked writes with client clocks Aug 24, 2026

@lan17 lan17 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review of the client-clock protocol (10 finder angles + adversarial verification + gap sweep; tsc and the 519-test unit suite pass locally). The core protocol verified sound: invalidation math, fence coverage on every read path, adapter dispatch against real node-redis/GLIDE semantics, cluster routing, retry idempotency, migration steps, and benchmark claims. Inline comments below; the corrupt-watermark hardening items (wrong-type key, >2^53 numeric parse) are tracked separately.

Comment thread src/dialcache.ts Outdated
Comment thread src/internal/redis-cache.ts
Comment thread src/dialcache.ts
Comment thread src/internal/redis-cache.ts Outdated
Comment thread README.md Outdated
Comment thread src/node-redis.ts Outdated
Comment thread src/internal/redis-cache.ts
Comment thread src/node-redis.ts
Comment thread test/fake-redis.ts Outdated
Comment thread scripts/test-package.mjs Outdated

@lan17 lan17 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up to the review discussion: concrete fix suggestions for the corrupt-watermark hardening items (strict shared integer grammar with a MAX_SAFE cap, and WRONGTYPE self-heal). All three are small and independent; together they make every corruption class route to the script's existing repair path.

Comment thread src/internal/redis-scripts.ts Outdated
Comment thread src/internal/redis-scripts.ts Outdated
Comment thread src/internal/redis-payload.ts Outdated

@lan17 lan17 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two follow-ups from a coverage pass over the final head (97.6% statements / 95% branches; unit 528 + integration 125/125 green locally): inline below. One process question rather than a code comment: does test:integration gate PRs in CI, or is it a manual step? The wrong-type-watermark and persistent-watermark behaviors are covered only there — FakeRedis structurally cannot represent either state — so the unit tier is blind to regressions in those two safety paths if integration is manual-only. The remaining uncovered lines are pre-existing convenience paths (local-cache.ts get/put-without-config, RedisCache.getResult's disabled-layer return) and constructor validation branches — reviewed, fine to leave.

Comment thread src/dialcache.ts
Comment thread src/internal/redis-payload.ts Outdated
@lan17

lan17 commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

On the integration-gating question: yes. The PR-triggered CI workflow runs pnpm test:integration inside the test job, and the active protect-main ruleset requires that test status with strict/up-to-date enforcement. The preceding head ran all 125 integration tests with zero CI skips across Redis 6.2/Valkey 8 and node-redis/GLIDE. Keeping WRONGTYPE and persistent-watermark behavior in real integration coverage rather than expanding FakeRedis is therefore merge-gated; the new head is running the same required job now.

@lan17
lan17 merged commit e5d88e2 into main Aug 25, 2026
6 checks passed
@lan17
lan17 deleted the codex/client-clock-redis-timestamps branch August 25, 2026 05:48
lan17 added a commit that referenced this pull request Aug 30, 2026
## Summary

Add opt-in stale-on-error recovery from the existing Redis frame, on top
of the native client-clock protocol from
#140.

- `F = ttlSec[CacheLayer.REMOTE]` remains the ordinary Redis freshness
boundary.
- `M = staleOnErrorMaxAgeSec` is the absolute stale-recovery age ceiling
and requested physical retention.
- One initial `GET` or tracked `MGET(value, watermark)` either serves a
fresh frame, retains an `F..M` raw candidate, or misses without a
candidate.
- After the source rejects, a resolved error classifier decides whether
DialCache may use the retained candidate.
- Recovery never rereads Redis, republishes the value, or replaces the
original source rejection when it cannot serve.

Closes #117

## Configuration and public API

```ts
import {
  CacheLayer,
  DialCache,
  DialCacheKeyConfig,
  FallbackTimeoutError,
} from "dialcache";

const dialcache = new DialCache({
  // Instance default: the broad transient-infrastructure policy.
  shouldAttemptStaleRecovery: (error) =>
    error instanceof FallbackTimeoutError || isRetriableDatabaseError(error),
});

const getUser = dialcache.cached((id: string) => db.fetchUser(id), {
  keyType: "user_id",
  useCase: "GetUser",
  cacheKey: (id) => id,
  defaultConfig: new DialCacheKeyConfig({
    ttlSec: { [CacheLayer.REMOTE]: 300 }, // F: ordinary Redis freshness
    staleOnErrorMaxAgeSec: 3_600,         // M: absolute recovery ceiling
  }),
});

// A per-use-case classifier REPLACES the instance and built-in policies for
// that use case; include the timeout case when it should stay eligible.
const getEntitlements = dialcache.cached((id: string) => db.fetchEntitlements(id), {
  keyType: "user_id",
  useCase: "GetEntitlements",
  cacheKey: (id) => id,
  shouldAttemptStaleRecovery: (error) => error instanceof FallbackTimeoutError,
  defaultConfig: new DialCacheKeyConfig({
    ttlSec: { [CacheLayer.REMOTE]: 300 },
    staleOnErrorMaxAgeSec: 3_600,
  }),
});
```

New API surface:

- `DialCacheKeyConfig.staleOnErrorMaxAgeSec?: number`
- `DialCacheConfig.shouldAttemptStaleRecovery?: StaleRecoveryPredicate`
- per-use-case `shouldAttemptStaleRecovery` on `cached()` and
`getOrLoad()` options
- root-exported `StaleRecoveryPredicate`, `StaleRecoveryOutcome`, and
`StaleRecoveryMetricLabels`
- optional `DialCacheMetricsAdapter.staleRecovery(labels)` and
`observeStaleRecoveryValueAge(labels, seconds)` hooks

Classifier precedence — highest wins, and an override replaces the
levels below it rather than composing with them:

1. per-use-case `shouldAttemptStaleRecovery` on `cached()` /
`getOrLoad()` options
2. `DialCacheConfig.shouldAttemptStaleRecovery`
3. built-in: `error instanceof FallbackTimeoutError` only

The intended split: put the broad policy for transient, retriable
infrastructure failures on the instance, and use a per-use-case override
where particular data needs a stricter policy. Predicates should deny
authoritative domain outcomes — auth/permission/entitlement failures,
revocation, deletion or not-found, validation, and programmer errors —
where a stale value would be wrong rather than merely old.

Predicates must synchronously return a boolean; a throw, non-boolean, or
thenable fails closed, and an accidental rejecting thenable is consumed.
`cached()` snapshots its selected predicate at registration;
`getOrLoad()` resolves it per invocation. Disabled calls remain true
pass-through and never invoke it.

Omitting `M` keeps recovery off; a sparse runtime overlay that omits it
inherits the configured default, and an explicit `0` disables an
inherited value. A positive policy requires `0 < F < M <= 31,536,000`
seconds. Invalid static policy throws before registration; invalid
runtime `M` records `config_resolution`, disables only recovery for that
invocation, and preserves otherwise-valid ordinary Redis policy.
`DialCacheKeyConfig.disabled()` sets `M` to `0`.

## Execution

```mermaid
flowchart TD
  A["One native GET or tracked MGET; classify raw frame with application Date.now()"] -->|"0 <= age < F"| B["Deserialize and return fresh"]
  A -->|"F <= age < M"| C["Record ordinary miss; retain raw candidate"]
  A -->|"missing, future, fenced, invalid, or age >= M"| D["Ordinary miss without candidate"]
  A -->|"read error, timeout, or fresh deserialization failure"| E["Fallback; recovery forbidden"]

  C --> F["Call source"]
  D --> F
  F -->|success| G["Return source value and attempt normal publication"]
  F -->|rejection denied by classifier| H["Throw exact source rejection"]
  F -->|eligible rejection| I["Use only retained candidate"]
  I -->|"age < M before and after lazy load"| J["Return candidate without publication"]
  I -->|missing, expired, future, or load failure| H
  E -->|source rejection| H
```

The candidate stays serialized/compressed until the source call settles.
On an eligible rejection, DialCache checks `0 <= age < M`, lazily
deserializes/decompresses, checks `M` again after that potentially
asynchronous work, and only then returns it. Equality at `F` is
logically stale; equality at `M` is unavailable.

A successful source refresh wins and writes a newly timestamped frame.
Recovery does not write or extend Redis, populate process-local cache,
schedule shadow validation, or emit a shadow-age observation.
Request-local caching may memoize the returned reference only within the
active outer `enable()` scope. Existing coalescing shares the whole
read/source/recovery decision; `coalesce: false` gives each caller an
independent candidate and source attempt.

## Snapshot and freshness semantics

The retained candidate is the initial Redis snapshot:

- tracked reads atomically apply the value and watermark observed by the
initial primary-routed `MGET`;
- an invalidation completed before that read fences the candidate;
- invalidation, refresh, deletion, expiry, or eviction after the read
does not revoke or replace the in-memory bytes;
- tracked and untracked recovery therefore use the same one-read model.

For tracked use cases this opt-in path can weaken the usual strict
freshness guarantee when invalidation races with the source attempt. Use
cases that cannot tolerate that bounded relaxation should leave recovery
disabled or deny the error in their classifier.

`F` and `M` bound Redis serving only. Request-local and process-local
layers occur earlier and keep their own scope/TTL lifetimes. A frame can
be nearly `F` old when it warms process-local cache and then receive a
full local TTL, so `ttlSec.local <= F` does not make `F` a strict
end-to-end age limit; disable earlier layers when that is required.

## Redis, time, and compatibility

There is no Redis envelope or key change: frame v1,
`:dialcache-frame-v1`, `DialCacheRedisClient`, and `RedisReadRequest`
remain unchanged. Reads use native `GET`/primary-routed `MGET`, writes
use native complete-frame `SET`, and invalidation remains the only Lua
operation. No Redis `TIME` or clock-offset estimation is added.

Writers request physical retention through `M` when enabled and `F`
otherwise. Tracked values keep the existing one-hour physical TTL cap
and `tracked_ttl_clamped` signal from
#140. The configured logical `M`
is not clipped, but a tracked frame may physically disappear before
reaching it.

Core now treats every decoded frame's real writer `createdAtMs` as
authoritative for ordinary logical `F`, including untracked reads.
Custom Redis clients that returned a constant timestamp must return the
actual epoch-millisecond frame stamp before upgrading. Roll out readers
while `M` is omitted or `0`, upgrade the complete fleet, and only then
enable positive `M`. Once a writer retains through `M`, do not
reintroduce a pre-feature reader until affected keys have expired or
been removed; older readers trust physical presence and can serve `F..M`
as fresh.

Application-process clock skew can move the boundary early or late.
Future-dated frames fail closed and use the existing future-offset
observation. Durations and deadlines remain monotonic.

## Metrics

New series, exposed by both bundled backends:

| Backend | Metric | Type | Labels / tags | Emitted |
| --- | --- | --- | --- | --- |
| Prometheus | `dialcache_stale_recovery_counter` | counter |
`cache_namespace`, `use_case`, `key_type`, `outcome` | once per
classifier-authorized recovery check |
| Prometheus | `dialcache_stale_recovery_value_age_histogram` |
histogram, buckets `[1, 5, 15, 60, 300, 900, 3600, 10800, 43200, 86400,
259200, 604800]` s (shared with shadow value age) | `cache_namespace`,
`use_case`, `key_type`, `outcome` | only alongside `served`; value =
return-time age in seconds |
| Datadog | `dialcache.stale_recovery.count` | count |
`cache_namespace`, `use_case`, `key_type`, `outcome` | once per
classifier-authorized recovery check |
| Datadog | `dialcache.stale_recovery.value_age` | histogram or
distribution, per the adapter's existing `observationMetricType` option
| `cache_namespace`, `use_case`, `key_type`, `outcome` | only alongside
`served` |

Label reference:

- `cache_namespace` — `DialCacheConfig.namespace` (default `urn`);
present on every DialCache metric.
- `use_case` / `key_type` — the registered use case and key type; never
the cache key or id.
- `outcome` — the bounded root-exported `StaleRecoveryOutcome` union:
- `served` — the retained candidate was returned to the caller (`age <
M` held both before and after lazy deserialization);
- `miss` — no candidate was retained by the initial read, or the
candidate's age reached `M` (or its stamp became future/invalid) by
check time;
- `deserialization_error` — a candidate within age failed
deserialize/decompress.
- On the value-age series the `outcome` label is structurally always
`served` — it exists so both outcome series share one label schema.
- Neither series carries a `layer` label: recovery reuses the
caller-serving initial read, so there is no separate layer to attribute.
- Adapter authors receive the backend-neutral camel-case object
`StaleRecoveryMetricLabels { cacheNamespace, useCase, keyType, outcome
}` and map it to their backend's naming, as the bundled adapters do.

The backing `DialCacheMetricsAdapter.staleRecovery(labels)` and
`observeStaleRecoveryValueAge(labels, seconds)` hooks are optional —
existing custom adapters keep compiling, and omitting a hook skips only
that observation, never recovery itself. A rejection the classifier
denies emits no stale-recovery outcome: the counter measures authorized
checks, not all source failures. No raw error, exception message, or
cache key ever enters a label.

Recovery adds no second ordinary `request`, `observeGet`, `miss`,
cache-read error, or Redis command — the initial read is the one
caller-serving telemetry trail. Existing fallback error/duration
telemetry still records the source rejection even when recovery serves,
so a served recovery stays visibly paired with the failure that caused
it.

Changes to existing series, visible on upgrade even where recovery stays
off (all carry their usual `cache_namespace`, `use_case`, `key_type`
labels; `layer`/`error`/`in_fallback` as noted):

- **`dialcache_miss_counter` / hit rate, at `layer="remote"`** — logical
`F` is now enforced from every frame's real `createdAtMs`, including
untracked reads, so physically present but logically old frames that
previously served now count as misses. Keyspaces that relied on
physical-TTL slack will see a one-time remote miss-rate rise. With
recovery enabled, a retained `F..M` candidate also records an ordinary
miss even when recovery later serves it — during incidents, treat
`staleRecovery{outcome="served"}` as its own population rather than
expecting it in hit-rate math. `dialcache_request_counter` and the
get-duration histogram are unchanged in population: still exactly one
per caller-serving read.
- **`dialcache_future_timestamp_offset_histogram` (`layer="remote"` and
shadow layers)** — its population widened: previously it observed only
tracked frames, but every serving read now validates the stamp, so
future-dated untracked frames are observed (and fail closed) as well.
The Prometheus help string changed accordingly.
- **`dialcache_error_counter` with `error="tracked_ttl_clamped"`,
`layer="remote"`, `in_fallback="false"`** — enabling `M` above one hour
on a tracked use case makes every dispatched tracked write request TTL
`M` and get clamped, so this existing configuration signal fires per
write for such configs (see the Redis section above). Alert rules keyed
on the error counter without an `error` label filter will absorb this as
a steady rate.
- **`dialcache_error_counter` with `error="config_resolution"`** — an
invalid runtime `M` overlay records one per invocation while recovery is
disabled for that invocation and ordinary reads continue.

## Cost

The Redis outage-path cost drops from two payload reads to one. The
tradeoff moves to Node memory: one raw candidate is retained per
distinct in-flight key through the source attempt; same-key coalesced
followers share it. The benchmark includes a delayed, high-cardinality,
incompressible-payload scenario rather than relying on the highly
compressed default fixture.

## Validation

Node 22.22.0:

- typecheck
- 611 unit tests with 98.18% statement coverage
- ESM/CJS build and packed TypeScript consumers
- 139 live Redis/Valkey integration tests passed; 2 unavailable-cluster
cases skipped after the expected connection timeout
- stale-on-error benchmark semantic assertions passed
- exactly 1.00 adapter read per independent recovery flight and one read
shared by 500 coalesced callers
- delayed 128 × 64 KiB raw-candidate benchmark observed +8.00 MiB
external memory while retained and returned to baseline after recovery

BREAKING CHANGE: Ordinary Redis reads now enforce logical age from each
frame's real `createdAtMs`, including untracked reads. Deploy new
readers before enabling physical `M` retention.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use client clocks for Redis value and watermark timestamps

1 participant