Prevent oversized Event Hub batches and unbounded buffering

[erica vellanoweth (ericavella)]

Summary

• bump the release version to 3.3.30
• prevent oversized Event Hub batches by checking the projected batch body size before dequeuing the next event
• cap queued telemetry at 256 MiB in addition to the existing event-count limit
• requeue failed batches only while capacity remains and explicitly report dropped events
• update transmission accounting only after a send succeeds and expose buffered-byte diagnostics

Root cause

Graph500 repeatedly launches a small Scale=10, EdgeFactor=16 workload until the profile timeout. Each short iteration emits a burst of metrics plus relatively large process-detail events. On the observed ARM64 runs, this created sustained telemetry pressure and repeatedly exposed an error in EventHubTelemetryChannel.

The channel previously appended the next event before checking its 700 KB body threshold. That boundary-crossing event could push the encoded AMQP batch beyond Event Hubs' 1 MiB limit. Event Hubs rejected the batch, VirtualClient requeued it, and new Graph500 telemetry continued accumulating behind the permanently invalid batch. The resulting retry backlog eventually exhausted guest memory.

EventPipe diagnostics confirmed this was not a CBS/token-renewal stall: the original reproduction contained 5,849 size-limit failures on telemetry-events, with rejected encoded sizes of 1,048,581-1,382,895 bytes and no authorization or timeout errors.

The batching defect is architecture-independent. ARM64 reproduced the host failure more reliably because the observed workload/event cadence created the invalid batch composition and retry backlog quickly; the fix does not special-case Graph500 or ARM64.

Design

The change deliberately leaves the Event Hubs SDK retry policy intact rather than imposing a separate channel-level send timeout. Memory safety comes from bounding the queue by bytes, while the batching correction prevents the confirmed permanently unsendable batch. REST/APIM transmissions remain single-event sends.

Validation

• EventHubTelemetryChannelTests: 7 passed, covering AMQP batch splitting, byte-bounded buffering, failure/requeue accounting, concurrent buffering, and REST single-event behavior
• cleaned implementation ed6bfdeb3 (current head adds test coverage only): Ubuntu 22.04 ARM64 Standard_D4ps_v6, experiment 20f00cae-f663-4436-8b47-8802995cf539, 10-minute Graph500 Dev Event Hub run, exit code 0, 1,249 publishes, 0 publish errors and 0 oversized batches
• earlier 30-minute post-fix run 6e88d2d4-1e34-4fd9-b09f-5463b4b98944: 4,966 publishes and 0 oversized batches
• three-hour bounded-buffer reproduction cfaa1fd4-4ac8-46b1-978f-b22bff4c547b: completed with stable memory and exposed the original 5,849 oversized-batch failures used for the RCA

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[erica vellanoweth (ericavella)]

Post-fix VM validation is complete on PR head d449c8c77. Ubuntu 22.04 ARM64 Graph500 experiment 6e88d2d4-1e34-4fd9-b09f-5463b4b98944 ran for 30 minutes against Dev Event Hub and exited 0. The EventPipe trace recorded 4,966 publish attempts with zero oversized-message errors; the prior failure began around four minutes and repeated every five seconds. Two transient owner-close events were retried successfully by the Event Hubs SDK.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[erica vellanoweth (ericavella)]

Why this shows up with Graph500

Graph500 is unusually effective at exposing this channel defect because the profile repeatedly launches short Graph500 processes in a tight loop. Each iteration publishes relatively large ProcessDetails and Graph500.ProcessDetails records, including process metadata, output, and event context. That produces a continuous stream of sizable records on telemetry-events.

The batching bug was composition-dependent: the channel accumulated events below its 700 KB body threshold, then dequeued and appended one more event before checking the limit. A large Graph500 process record could therefore push the encoded AMQP batch from a valid size to 1.05-1.38 MB. Event Hubs rejected that batch, VC requeued the same records, and subsequent Graph500 iterations continued adding telemetry behind them. The repeated rejection/backlog—not a hung Graph500 child process—eventually consumed guest memory in the unfixed implementation.

Why ARM64 reproduces it more reliably

The batching code itself is architecture-independent; this can happen on x64 if the same event-size and throughput conditions occur. ARM64 is the reliable trigger for this profile because its observed Graph500 telemetry pressure is substantially higher: approximately 285 metrics/sec in the failing ARM64 runs versus approximately 153 metrics/sec in successful x64 runs. The rapid ARM64 iteration/event cadence reaches the problematic batch composition earlier and grows the retry backlog faster, which is why the guest typically disappeared after roughly 114-144 minutes.

Changing the VM SKU or Ubuntu version does not remove the defect. We reproduced it on Ubuntu 22 ARM64 using Standard_D4ps_v6, while the last historical ARM64 success used Standard_D4ps_v5. The fix is therefore to prevent construction of an oversized batch and bound failed-send memory—not to special-case Graph500 or ARM64.