OFREP ETag can result in race condition weakness.

[JamieSinn]

Problem

By using the ETag as the cache differentiator, you can run into some ugly consequences based on the reliability/consistency of your underlying datastore. This is not specific to any one implementation or another.

Detail

ETag just assigns whether or not the value has changed since the last request, and has no bearing on when, or the order of what lifecycle operation configs have been changed. This can have major impacts, as illustrated later on.

Below is a sample progression of the versions of a single configuration that were made extremely close together. In this case, each configuration is different (hence the different ETags) and was updated in sequence.

flowchart LR

CN[Config 0<br/>ETag=Z]
C1[Config 1<br/>ETag=A]
C2[Config 2<br/>ETag=B]
C3[Config 3<br/>ETag=C]
CP[Config N...]

CN -->|T=-2...| C1
C1 -->|T=0.1| C2
C2 -->|T=0.2| C3
C3 -->|T=0.3| CP

Loading

This update process means that you are relying on your backing datastore to be guaranteed to write the configurations in order, every time, and ensure that there are no out of order writes.

If there are out of order writes (or another race condition in something persay the cache filling, etc) you may get an unknown final configuration at T=0.5.

For the purposes of this, the reason (SSE, Polling, etc) for the given refetches is irrelevant as they're a complexity that can make this problem worse, or better depending on vendor implementation. The OFREP spec does not define how these messages should be handled/sent out based on multiple changes to the underlying configuration, and thus we cannot speculate on a given design.

Example Issues

Example 1 - Not an issue

Let's pose a situation here with the following criteria:

• An OFREP provider has been initialized with Config 0, an arbitrary configuration before the series of edits here, with an ETag of Z.
• There was a race condition in the underlying store for the configuration, and the configuration being served is Config 2 incorrectly.
• The OFREP provider makes a fetch to retrieve the current config at T=0.35 sending the If-None-Match=Z and retrieves Config 2.

This results in the provider comparing the ETags and validating them to be different, and accepting Config 2 as the valid config. For all intents and purposes to the provider, this is accurate and has no issues. But this means the provider has missed the latest available update. Not the fault of OFREP here.

Example 2 - Unintended Rollback of the config

Let's add a small modification to this:

• An OFREP provider has been initialized with Config 2, as it initialized/fetched the config at the exact moment it was written, with an ETag of B.
• There was a race condition in the underlying store for the configuration, and shortly after initialization the final config written resulted in being Config 1.

This has a much more drastic impact. To the provider, we're now invalidating the current configuration with an invalid out of date one. But because the ETag is "different" there is no reason to suspect/reject the configuration.

The reality here, is that the configuration was rolled back in time, with potentially unknown impact based on the usage/plan to roll out the config.

Proposed Solution - LastModified Headers

By introducing the LastModified Header as an optional/equally valid cache invalidation header option, you now add the timestamp of the originating file/change to the validation path.

Let's revisit the previous situations but with the concept of validating against a LastModified Header.

flowchart LR

CN[Config 0<br/>LastModified=T<br/>ETag=Z]
C1[Config 1<br/>LastModified=T+0.1<br/>ETag=A]
C2[Config 2<br/>LastModified=T+0.2<br/>ETag=B]
C3[Config 3<br/>LastModified=T+0.3<br/>ETag=C]
CP[Config N...]

CN -->|T=-2...| C1
C1 -->|T=0.1| C2
C2 -->|T=0.2| C3
C3 -->|T=0.3| CP

Loading

Example 1 - Not an issue

Let's pose a situation here with the following criteria:

• An OFREP provider has been initialized with Config 0, an arbitrary configuration before the series of edits here, with a LastModified of T.
• The OFREP provider makes a fetch to retrieve the current config at T=0.35 with a If-Modified-Since=T header and retrieves Config 2
• There was a race condition in the underlying store for the configuration, and the configuration being served is Config 2 incorrectly

This is valid, and does not drastically impact differently than the ETag situation.

Example 2 - Unintended Rollback of the config

• An OFREP provider has been initialized with Config 2, as it initialized/fetched the config at the exact moment it was written, with a LastModified of T+0.2.
• There was a race condition in the underlying store for the configuration, and shortly after initialization the final config written resulted in being Config 1.
• The provider fetches for the configuration with an If-Modified-Since=T+0.2 header and receives a 304, and does not update the config.

Proposal

This ultimately is a safer, and more stable way of returning configuration which at its core is naturally a timeseries based datatype.
The recommendation here is to expand the OFREP spec to make the LastModified header an equally valid cache invalidation header to the bulk evaluation endpoint.

This is a defense in depth approach, and is something very real that I've personally had to deal with and solve for with the DevCycle SDKs and Cloudflare's cache not properly being globally invalidated in time when relating to successive changes.