feat(quotas): Support shadow (dual) writes for the quotas Redis pool by dmajere · Pull Request #6294 · getsentry/relay

dmajere · 2026-08-06T21:08:45Z

Summary

Reintroduces dual ("shadow") writes for the quotas Redis pool so quota data can be duplicated to a second Redis/Valkey deployment during a migration. This capability existed previously as the MultiWrite client (#4064) but was lost during the async Redis migration (#4552) and formally removed from config (#4656). This re-implements it on the current async (AsyncRedisClient) stack, scoped to quotas.

Config

Add an optional shadow list of cluster nodes next to cluster_nodes on the quotas pool:

redis:
  project_configs:
    cluster_nodes:
      - redis://rc-memorystore-relay-projectconfigs.sentry.:6379
  quotas:
    cluster_nodes:
      - redis://rc-memorystore-relay-quotas.sentry.:6379
    shadow:
      - redis://relay-quotas.valkey.sentry.:6379

When shadow is set, every command executed against the pool is sent to the primary and, best-effort, to the shadow cluster.

Implementation

relay-config: optional shadow: Option<Vec<String>> on the cluster Redis config, threaded through RedisConfigRef and build_redis_config.
relay-redis: MultiWrite { primary, secondaries } variant on AsyncRedisClient and AsyncRedisConnection. The ConnectionLike impl dispatches the command to the primary and all shadows concurrently (via futures::join/join_all), returns only the primary's result, and logs + swallows shadow errors. Connection acquisition for shadows is best-effort so a down shadow never blocks the primary path. stats() reports the primary only; retain() (pool maintenance) fans out to all pools. Added relay-log as an optional dep under the impl feature.
relay-server: create_async_redis_client builds the shadow cluster (same options, metrics tag quotas_shadow) and wraps the pool via multi_write when shadow is set. Script preloading needs no change — SCRIPT LOAD is fanned out through the same connection, so the rate-limiting Lua script is loaded on the primary and shadow at startup.

Behavior / trade-offs (fail-open)

Shadow failures never affect live rate limiting — errors are logged and dropped.
If the shadow is unavailable at startup its SCRIPT LOAD is skipped; later EVALSHA on the shadow returns NOSCRIPT (logged/dropped) until the next relay restart re-primes it. Acceptable for a migration shadow.

Tests

Added config parse/serialize tests for the shadow field.
relay-config / relay-redis / relay-quotas build & tests pass; relay-server builds with and without processing; clippy + fmt clean.

Adds an optional `shadow` list of cluster nodes to a Redis pool config. When set, every command is fanned out to the primary and, best-effort, to the shadow cluster(s) concurrently. Only the primary result is returned and shadow errors are logged and swallowed (fail-open). This is used to dual-write quota data while migrating between Redis deployments. - config: add `shadow: Option<Vec<String>>` to cluster Redis config - relay-redis: add `MultiWrite` variant to `AsyncRedisClient`/`AsyncRedisConnection` with concurrent fan-out on the write path - service: build the shadow client and wrap the quotas pool via `multi_write`

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9b96fa6. Configure here.}

cursor · 2026-08-06T21:21:40Z

+                    }
+                }
+                primary_result
+            }),


Shadow waits gate primary latency

High Severity

req_packed_command and req_packed_commands use futures::join so the caller only returns after every shadow command finishes. A slow or timing-out shadow therefore delays the primary result by up to response_timeout (default 30s), coupling quota latency to shadow health despite comments claiming otherwise.

Additional Locations (1)

relay-redis/src/real.rs#L382-L405

^{Reviewed by Cursor Bugbot for commit 9b96fa6. Configure here.}

Since its a wait for all, latency of write is latency of the slowest + cycles for managing context switches etc.
this should not be a big issue since we plan to similarly performant clusters.

Dav1dde

Implementation/dispatch looks good.

Mostly need a change to how the configuration works, I'd mirror to multi write approach of the client, like the original PR also had. That also means double writes work independently of whether it is a cluster connection or a single node connection.

And please do a human de-slopify of the comments, other people actually read this and I see shadows everywhere now.

Dav1dde · 2026-08-11T14:49:17Z

+        ///
+        /// This is used to dual-write quota data while migrating between Redis deployments.
+        #[serde(default, skip_serializing_if = "Option::is_none")]
+        shadow: Option<Vec<String>>,


Why are you using shadow here but in the Redis client itself you model it via MultiWrite?

Dav1dde · 2026-08-11T14:51:39Z

+    /// Commands executed against the returned client are sent to the primary and, best-effort, to
+    /// every shadow client. Only the primary result is returned.


Result is returned from where? I think you want to say that the commands sent to the secondaries are fire and forget, it just reads weird, because this function definitely doesn't fail.

Dav1dde · 2026-08-11T14:51:53Z

+    pub fn multi_write(
+        primary: AsyncRedisClient,
+        secondaries: Vec<AsyncRedisClient>,
+    ) -> Result<Self, RedisError> {


Why did you make this fallible?

followed same code pattern as in above single function.

Dav1dde · 2026-08-11T14:53:44Z

+                secondaries,
+            } => {
+                // Acquiring the primary connection must succeed, otherwise the whole operation
+                // fails, just like a non-shadowed client.


This shadow terminology really should just be secondary, that's what it is called in code.

Also some of the comments are redundant with the code like:

// The recursive `get_connection` calls are boxed to break the infinitely sized // future that async recursion would otherwise create.

Dav1dde · 2026-08-11T14:54:10Z

+            AsyncRedisClient::MultiWrite { secondaries, .. } => {
+                write!(
+                    f,
+                    "AsyncRedisPool::MultiWrite({} shadows)",


Suggested change

"AsyncRedisPool::MultiWrite({} shadows)",

"AsyncRedisPool::MultiWrite({} secondaries)",

Dav1dde · 2026-08-12T09:14:41Z

+                let secondaries = futures::future::join_all(
+                    secondaries
+                        .iter_mut()
+                        .map(|secondary| secondary.req_packed_command(cmd)),


This also sends reads to the secondaries while only writes need to be sent. This is relevant for project config requests (technically we can just never configure the multi write for project configs, but if everything is backed by a single redis it's a bit of a footgun).

I think the old impl had the same issue.

Not sure how to compat that, maybe just checking for some well known read commands (GET, MGET, EXISTS) is enough.

dmajere requested a review from a team as a code owner August 6, 2026 21:08

cursor Bot reviewed Aug 6, 2026

View reviewed changes

Comment thread relay-redis/src/real.rs

dmajere force-pushed the feat/quotas-redis-shadow-write branch from 0ce08ab to 9b96fa6 Compare August 6, 2026 21:17

cursor Bot reviewed Aug 6, 2026

View reviewed changes

Dav1dde requested changes Aug 11, 2026

View reviewed changes

Dav1dde reviewed Aug 12, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(quotas): Support shadow (dual) writes for the quotas Redis pool - #6294

feat(quotas): Support shadow (dual) writes for the quotas Redis pool#6294
dmajere wants to merge 1 commit into
masterfrom
feat/quotas-redis-shadow-write

dmajere commented Aug 6, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Aug 6, 2026

Uh oh!

dmajere Aug 6, 2026

Uh oh!

Dav1dde left a comment

Uh oh!

Dav1dde Aug 11, 2026

Uh oh!

Dav1dde Aug 11, 2026

Uh oh!

Dav1dde Aug 11, 2026

Uh oh!

dmajere Aug 12, 2026

Uh oh!

Dav1dde Aug 11, 2026

Uh oh!

Dav1dde Aug 11, 2026

Uh oh!

Dav1dde Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		/// Commands executed against the returned client are sent to the primary and, best-effort, to
		/// every shadow client. Only the primary result is returned.

	"AsyncRedisPool::MultiWrite({} shadows)",
	"AsyncRedisPool::MultiWrite({} secondaries)",

Uh oh!

Conversation

dmajere commented Aug 6, 2026

Summary

Config

Implementation

Behavior / trade-offs (fail-open)

Tests

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 6, 2026

Choose a reason for hiding this comment

Shadow waits gate primary latency

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dav1dde left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants