Comma3 Openpilot Stream

A third recorded stream for the comma 3X: the video openpilot's model reads, judged by comma's own shipped model.

comma.ai ran a public video-compression challenge. This is not an entry to it. Reading the challenge made clear which larger problem it stands in for, and this work went at that problem instead: the path from a comma device's cameras to comma's training cluster.

Everything below was measured on public data with comma's own shipped model. Every number is unrounded and reproducible with the code in this repository. What is not established is listed as plainly as what is.

How this started

I came across the challenge while following what comma has been shipping. The task is a sandbox: shrink one 1344x760 clip below a byte budget and score it with a fixed metric on the whole frame. That is a homework exercise, and the leaderboard answers it by tuning an encoder to one clip.

The question the sandbox stands in for is different. A fleet of devices records video that the training cluster mostly never sees, and the one stream that does arrive is too small for the model. I did not enter the sandbox. I went one level up, read the data path in openpilot's source, and measured a proposal against comma's own model rather than against the challenge metric.

What the data path does today

A stock comma 3X records 151.92 MB/min of video and automatically uploads 1.92 MB/min, which is 1.264 percent. The rest leaves only on request. In uploader.py the automatic path admits exactly two files, qlog and qcamera.ts, and has no fallback branch for anything else.

The one video stream that does leave, qcamera, covers the narrow road camera only. Scaled uniformly from 1928 to 526 pixels wide, the region the driving model reads arrives as 406 by 203 encoded pixels where the model consumes 512 by 256: 1.59x short. The wide camera has no such stream at all, and the same treatment would give it 174 by 87, 8.66x short.

On 2026-08-12 comma announced chestnut, an external GPU dock and a driving model with, in their words, 30x more parameters and 100x more FLOPs than the on-device one. Inference compute went up. The data path did not change.

Left: what a stock comma 3X writes per minute against what it uploads on its own. Right: the model's 512x256 canvas, the 406x203 that qcamera carries for that region, and the 174x87 the wide camera would get under the same scaling.

The proposal

Spend the encoder's bits on the region that openpilot's own get_warp_matrix projects into, instead of spreading them evenly across the frame. The region follows from the camera intrinsics and the pitch and yaw limits in calibrationd, so a device computes it from what it already knows. Nothing is drawn by hand.

Left: the narrow camera on a device with a separate wide camera. Red is what gets softened before encoding; the yellow box is the canvas at level calibration, the dotted box everything the canvas can reach within the calibration limits. Right: the public clip's own configuration, one camera serving both model inputs, so the same frame is also warped through the double field of view canvas and far less of it is free.

Then record that region as its own stream: cropped to the model's 512x256 canvas, scaled inside the encoder where downscaling is free, at 5 Hz, which is the rate the published training input uses.

How it was judged

Not by a proxy metric on the whole frame. The car does not look at the whole frame; modeld warps a 512x256 canvas out of each camera. So model_probe.py reproduces modeld's preprocessing from compile_modeld.py (nearest-neighbour warp, the chroma matrix, the four-luma-phase packing) and runs the shipped driving_supercombo.onnx on original and compressed video. The measure is how far the model's own outputs move.

Controls on the probe: identical input twice moves the output by exactly 0.0; one frame step of ordinary driving moves it by 0.2834.

One frame warped onto the model's canvas, from the original, after a flat encode and after the masked one, both at 256 kbit/s and 526x330. The bottom row is the absolute luma difference from the uncompressed canvas: lane paint and car edges light up under flat.

Result

24 clips from the public archive, 100 frame pairs each, x265 at 256 kbit/s, 526x330, with big_img taken from the uncompressed original, as a separate wide sensor supplies it on every device comma currently sells.

arm	mean output deviation
flat, no mask	1.02598
mask from the warp region	0.71796
control: same area moved to the corner	1.28737
control: uniform blur over the whole frame	1.33465
control: inverse mask, softening what the model reads	4.30571

The mask beats flat on 23 of 24 clips, ratio of means -30.02 percent. It beats the corner decoy on 24 of 24. Both non-inverse controls are worse than doing nothing, which is what separates this from "blurring helps": the gain is in where the bits go.

Mean deviation per arm over 24 clips, each dot one clip. Only the mask lands below flat. The two controls that soften the same amount somewhere else land above it.

Each point is one clip; below the diagonal the mask beat flat. Left, big_img from the untouched original: 23 of 24. Right, one camera serving both inputs: 19 of 24, and the gain is small.

At the corners of the legal calibration range the ratio of mask to flat runs from 0.7278 at level to 0.9110 at the worst corner. Weaker, never reversed.

Ratio of mask to flat at level and at the four corners of the pitch and yaw limits. Below 1.0 everywhere.

Per output head, every actionable head improves: pose -32.4 percent, plan -29.6, lane lines -25.0, lead probability -22.7, road edges -22.5, meta -15.4, desire -15.4, lead -10.5, lane line probability -10.4. The gain is not carried by one head.

Change in deviation per output head at level calibration, 8 clips of 60 pairs. Every head moves less with the mask, the driving heads most.

Cost of the stream, measured against qcamera encoded with its own settings in the same run: qcamera 1.937 MB/min at deviation 0.90943; the canvas stream at 96 kbit/s 0.730 MB/min at 0.91229, a tie in fidelity for 37.7 percent of the bytes; at 192 kbit/s 1.456 MB/min at 0.78350, 13.8 percent more faithful for 75.2 percent of the bytes. Both road cameras at 96 kbit/s cost 1.461 MB/min against 1.937 for the single camera covered today.

Bytes per minute against deviation for qcamera as shipped and for the canvas stream at two bitrates, same clips, same run. The dotted line is both road cameras at 96 kbit/s.

What is not established

• comma2k19 has no wide-camera footage. The second model input is the same narrow frame through the sbigmodel matrix, taken from the original. The configuration is close to a two-camera device, not identical to one. On the public clip's own single-camera configuration, where the narrow frame must also serve the wide canvas, only 17.9 percent of the frame is free to soften and the effect is -4.69 percent, 19 of 24 clips (results/third_mode_onecam.json).
• The softening runs before the encoder. openpilot's V4L2 wrapper sets no QP or ROI control, so that is the only path available today. An encoder with a QP map would do it cheaper.
• Nothing here measures training outcomes. It measures how far the deployed model's answer moves, which is a proxy for that and not the thing itself.
• There is no openpilot patch. Feeding a 20 fps configured encoder session at 5 fps is an untested driver state that cannot be settled from source, only on a device.
• On a comma 3X's narrow camera the static calibration envelope leaves no margin: at the pitch limit the canvas reaches past the sensor edge. The headroom on that device is on the wide camera and in following the live calibration rather than the envelope.

A side finding while building the probe: two legitimate decode paths for the same file (yuv420_to_rgb in the challenge's frame_utils.py, and PyAV's rgb24) disagree on 81.85 percent of subpixels, so challenge scores taken on different machines are not strictly comparable.

Reproducing it

pip install numpy onnx onnxruntime scipy av

ffmpeg on PATH with libx265 and libx264. CPU only.

Two directories beside this README, or wherever WARPBITS_ROOT points:

openpilot/   checkout of github.com/commaai/openpilot at 7cf55c3b7a2d9bcee87821e413fa322866f64c5b
clips/       *.mkv from the challenge's public archive

git clone --depth 1 --filter=blob:none --no-checkout https://github.com/commaai/openpilot.git
cd openpilot && git sparse-checkout init --cone
git sparse-checkout set openpilot/common openpilot/selfdrive/modeld && git checkout

The repository nests a directory of the same name inside itself; openpilot.common resolves to openpilot/openpilot/common. That is the expected layout.

Clips come from the archive that the challenge's download_and_remux.sh fetches from huggingface.co/datasets/commaai/comma2k19, remuxed the same way:

ffmpeg -y -f hevc -framerate 20 -r 20 -i <segment>.hevc -c copy clips/<n>.mkv

Then:

python -m warpbits.third_mode --clips 24 --pairs 100 --wide-untouched
python -m warpbits.final_check --clips 8 --pairs 60
python -m warpbits.figures --clip 15

The last one needs matplotlib and rebuilds figures/ from results/ and one clip.

results/ holds the runs the tables above were read from. Running the commands above writes into the same files, so copy them aside first if you want to keep the originals.

Files

• warpbits/warp_roi.py computes the readable region and the mask from openpilot's own transformations. Depends on openpilot and numpy only; usable on its own.
• warpbits/model_probe.py reproduces modeld's preprocessing and runs the shipped model.
• warpbits/third_mode.py is the experiment and its three controls.
• warpbits/final_check.py is the calibration sweep and the per-head breakdown.
• warpbits/figures.py draws figures/ from the results and one clip.
• results/ the measured runs: third_mode.json, third_mode_onecam.json, final_check.json, canvas_stream.json.

comma2k19 and openpilot are MIT licensed by comma.ai. This code is Apache-2.0, copyright 2026 Slava Sexton; see LICENSE and NOTICE.