cloudwait

cloudwait is a high-performance Rust HTTP/1.1 reverse proxy for AI inference APIs that may take a long time to produce response headers or body bytes.

It behaves like a normal reverse proxy for ordinary routes, and adds configurable "chunk-wait" handling for selected endpoints such as Anthropic, OpenAI-compatible, and other AI streaming APIs.

What it does

• Proxies HTTP/1.1 requests using precise request and response boundary parsing.
• Listens on TCP sockets and Unix domain sockets.
• Connects to TCP or Unix upstreams.
• Supports HTTP and HTTPS TCP upstreams.
• Supports custom CA roots and opt-in invalid certificate acceptance per upstream.
• Routes by downstream Host header.
• Tunnels WebSocket upgrades as raw bidirectional streams after a successful 101 response.
• Handles special AI routes with upstream header wait, fallback headers, chunked downstream streaming, and periodic keepalives.
• Supports JSON-style and SSE-style keepalive payloads.
• Supports fixed local responses for selected routes without opening an upstream connection.
• Uses TOML configuration only.

Why this exists

Some clients, tunnels, and intermediaries close idle HTTP connections when an upstream inference server is slow to send response headers or the first body bytes. For configured special routes, cloudwait can keep the downstream connection alive by sending a valid chunked response and periodic keepalive chunks while it waits for the upstream response.

Normal routes are not modified this way. They are proxied with standard HTTP/1.1 request and response framing.

Quick start

Prerequisites:

• Rust 1.82 or newer.
• A reachable HTTP/HTTPS upstream.

git clone <your-repo-url>
cd cloudwait
cp config.example.toml config.toml
$EDITOR config.toml
cargo run --release -- config.toml

If no config path is passed, cloudwait reads ./config.toml.

Send a request through the proxy:

curl -v http://127.0.0.1:8080/v1/messages

Configuration overview

The config file is TOML. See config.example.toml for a fully commented example.

Top-level settings:

max_header_bytes = 65536
io_buffer_bytes = 16384
io_timeout = "30s"
request_header_timeout = "10s"
max_connections = 1024

Routing settings:

[proxy]
default_upstream = "main"

[proxy.host_upstreams]
"api.example.com" = "main"
"local.example.test" = "local"

Listeners:

[[listen]]
type = "tcp"
addr = "0.0.0.0:8080"

[[listen]]
type = "unix"
path = "/tmp/cloudwait.sock"
remove_existing = true
mode = 0o660

Upstreams:

[upstreams.main]
type = "tcp"
scheme = "https"
addr = "api.example.com:443"
server_name = "api.example.com"
connect_timeout = "10s"
custom_ca_pem = "/etc/ssl/certs/api-example-ca.pem"
accept_invalid_certs = false

[upstreams.local]
type = "unix"
scheme = "http"
path = "/tmp/upstream.sock"

Notes:

• TCP upstreams can use scheme = "http" or scheme = "https".
• HTTPS upstreams require server_name for SNI and certificate validation.
• Unix upstreams must use scheme = "http".
• accept_invalid_certs = true should only be used for controlled internal or test upstreams.

Special chunk-wait routes

Special routes are configured with [[special_routes]]. They are useful for AI inference endpoints that may delay headers or stream sparse body chunks.

Common endpoints:

• /v1/messages
• /v1/responses
• /v1/chat/completions
• /v1/completions

Example:

[[special_routes]]
paths = ["/v1/messages", "/v1/responses", "/v1/chat/completions", "/v1/completions"]
match = "prefix"
methods = "*"
normalize_path = true
upstream = "main"
rewrite_http_code_5xx = 200
header_timeout = "110s"
upstream_wait_timeout = "15m"
pre_body_keepalive_interval = "30s"
fallback_kind = "json"

[special_routes.json]
keepalive_hex = "0a"
fallback_header = [
  "HTTP/1.1 200 OK",
  "Content-Type: application/json",
  "Transfer-Encoding: chunked",
]

[special_routes.sse]
comment = ""
json_response = "error_event"
json_error_event = "error"
fallback_header = [
  "HTTP/1.1 200 OK",
  "Content-Type: text/event-stream",
  "Cache-Control: no-cache",
  "Transfer-Encoding: chunked",
]

Important behavior:

• The full downstream request is forwarded upstream before the proxy waits for upstream response headers.
• If upstream headers arrive before header_timeout, the proxy forwards the upstream body as a downstream chunked response.
• If upstream headers do not arrive before header_timeout, the proxy sends the configured fallback response header downstream, then keeps waiting for upstream until upstream_wait_timeout.
• upstream_wait_timeout also caps the wait for the first upstream body payload after response headers arrive. The default is 15m.
• Downstream special-route responses use Transfer-Encoding: chunked when they can carry a body.
• Content-Length is stripped from rewritten downstream special-route responses.
• Keepalive chunks are sent while waiting for the first upstream body payload and between delayed upstream body chunks.
• Query strings are ignored for route matching.

Method matching

methods can be:

methods = "*"
methods = "GET,POST"
methods = "HEAD"

Path matching

match can be:

match = "exact"
match = "prefix"

When normalize_path = true, the request path is normalized before matching. Query strings are ignored for matching.

JSON keepalives

JSON-style keepalives use raw bytes encoded as hex:

[special_routes.json]
keepalive_hex = "0a"

0a is a newline byte. On the wire, special-route responses are chunked, so the downstream bytes include chunk framing around the keepalive payload.

SSE keepalives

SSE mode is selected when:

• the downstream request Accept header includes text/event-stream, or
• the downstream request has Content-Type: application/json and its first top-level stream field is true, or
• the upstream response Content-Type is text/event-stream, or
• fallback_kind = "sse" is configured and the header timeout fallback fires.

Default SSE keepalive is an empty comment:

:

That payload is sent inside HTTP chunked framing. For example, payload bytes :\n\n are sent as:

3\r\n:\n\n\r\n

Configure SSE keepalive as a comment:

[special_routes.sse]
comment = "keepalive"

Or as an event/data heartbeat:

[special_routes.sse]
event = "keepalive"
data = ""

Do not combine comment with event or data.

For JSON request bodies, stream intent is detected incrementally while the body is forwarded upstream. The detector only considers the first top-level stream field; nested fields, string contents, non-JSON bodies, and later duplicate stream fields do not force SSE mode.

When a stream-intent request receives an upstream Content-Type: application/json response, json_response controls whether that JSON is converted to SSE or forwarded as JSON before any fallback header is sent:

[[special_routes]]
rewrite_http_code_5xx = 200

[special_routes.sse]
json_response = "error_event"
json_error_event = "error"

json_response = "error_event" is the default. It rewrites the downstream response as text/event-stream and emits the upstream JSON body as one SSE event. The default event name is error:

event: error
data: {"error":"bad"}

json_response = "raw" forwards the upstream JSON headers and body when no fallback header has been sent yet. If the fallback header has already been sent, the proxy keeps the committed SSE response shape and still emits the upstream JSON body as the configured SSE error event.

rewrite_http_code_5xx optionally rewrites any matched special-route response header whose status is HTTP 500-599. It applies to upstream response headers, fallback headers, fixed local responses, and proxy-generated 5xx responses, and it is not overridden by JSON or SSE response shaping.

CR and LF characters are removed from json_error_event and from the upstream JSON body before emitting the SSE frame. Special-route requests that already request SSE or have JSON request bodies are forwarded upstream without Accept-Encoding, so JSON-to-SSE conversion receives a text JSON body rather than compressed bytes.

Fixed local responses

A special route can return a local response immediately without opening an upstream connection:

[[special_routes]]
paths = ["/"]
match = "exact"
methods = "HEAD"
fixed_response_header = [
  "HTTP/1.1 200 OK",
  "Content-Type: text/plain",
]
fixed_response_body = ""

The proxy adds Content-Length from fixed_response_body. Do not include Content-Length or Transfer-Encoding in fixed_response_header.

Docker deployment

The deploy directory contains:

• build-arm64.sh: builds a static ARM64 musl binary at deploy/app.
• compose.yml: runs the binary from a scratch container with a read-only filesystem and dropped capabilities.
• config.toml: deployment-oriented commented config.

Build the static ARM64 binary:

./deploy/build-arm64.sh
file deploy/app

The default target is aarch64-unknown-linux-musl. The build expects an ARM64 musl linker, exposed as aarch64-linux-musl-gcc unless CC_aarch64_unknown_linux_musl is overridden.

Run with Docker Compose:

cd deploy
docker compose up --build

The Compose file mounts:

• ./app as /app.
• ./config.toml as /config.toml.
• ./run as /tmp/run for Unix sockets.

By default, the deployed process runs:

/app /config.toml

Logging and tracing

Enable info logs with RUST_LOG=info:

RUST_LOG=info cargo run --release -- config.toml

For Docker Compose, run a one-off service with the environment variable passed into the container:

cd deploy
docker compose run --rm --service-ports -e RUST_LOG=info app

Or add RUST_LOG=info under the service environment section in deploy/compose.yml for normal docker compose up runs.

Info logs include connection-scoped events with the connection id first, for example:

id=12 client_connection_open ...
id=12 special_fallback_header_sent ...
id=12 special_keepalive_sent ...
id=12 client_connection_closed ...

Use the id= field to correlate client, upstream, fallback, keepalive, and response-modification events for one request path.

Logs are intended for debugging behavior such as:

• whether a request matched a special route.
• whether upstream response headers arrived before header_timeout.
• whether fallback headers were sent.
• whether JSON or SSE keepalives were selected.
• whether original upstream response headers were rewritten for downstream delivery.

Do not put secrets in URLs. The proxy avoids logging query strings for route diagnostics, but request paths and hostnames can still be operationally sensitive.

Testing

Run the standard verification suite:

cargo fmt -- --check
cargo clippy --all-targets -- -D warnings
cargo test

Run benchmarks:

cargo bench --bench proxy_bench

Useful test areas:

• tests/http_tests.rs: HTTP framing and parsing behavior.
• tests/proxy_normal_tests.rs: normal proxy behavior and chunked streaming.
• tests/proxy_special_tests.rs: special-route fallback, keepalive, JSON, and SSE behavior.
• tests/security_resilience_tests.rs: request-smuggling and malformed-framing defenses.
• tests/tls_websocket_tests.rs: TLS upstream and WebSocket upgrade behavior.
• tests/logging_tests.rs: info-level trace behavior and log redaction expectations.

Security notes

cloudwait is designed to avoid common HTTP/1.1 desynchronization hazards:

• rejects duplicate Content-Length request framing.
• rejects Content-Length with Transfer-Encoding.
• rejects unsupported or repeated transfer codings.
• rejects request framing nominated through the Connection header.
• validates chunk-size lines, chunk-data terminators, and chunk trailers before forwarding unsafe bytes.
• rejects framing headers inside chunk trailers.
• enforces maximum header bytes for request and response parsing.
• uses explicit I/O timeouts and a global active-connection limit.

Operational recommendations:

• Keep accept_invalid_certs = false unless testing or using a controlled internal upstream.
• Keep max_header_bytes, request_header_timeout, io_timeout, upstream_wait_timeout, and max_connections sized for your environment.
• Run cargo audit as part of release checks.
• Treat RUST_LOG=debug logs as potentially sensitive in production.

Limitations

• Downstream protocol support is HTTP/1.1 over TCP or Unix sockets.
• HTTPS support is for upstream TCP connections, not downstream TLS termination.
• Unix upstreams support HTTP only.
• Special-route fallback headers must use chunked transfer encoding and must not include Content-Length.

License

Private