feat(cli): dimos data — resumable cloud uploads

[spomichter]

Adds dimos data — resumable uploads of mem2 recordings (and any file) to Dimensional cloud, over the /v1/data multipart API.

$ dimos data upload                    # newest recording in RECORDINGS_DIR
$ dimos data upload --since 1h         # everything recorded in the last hour
$ dimos data upload flight.log --kind log --robot drone-1
$ dimos data ls / pull ID / status ID / quota

• Resumable: create is keyed on sha256; the server's part listing is the only resume state. Re-run after any failure (kill -9 included) and only missing parts transfer. No local state files.
• mem2-aware: manifest auto-extracted from the session .db's _streams table (open schema, transports pass through untouched); live sessions (recently-written .db) are skipped with a clear message.
• Compression: lz4-frame, streaming file→file into an OS tempdir, deleted the moment the upload ends — nothing persists on the machine. --no-compress for disk-tight robots.
• Auth: rides dimos login end to end — endpoint from GlobalConfig.dimos_cloud_url, key from the stored credential; 401 surfaces as "run dimos login", no preflight round-trips.
• Config: dimos_upload_retries / dimos_upload_compress on GlobalConfig; per-call overrides via CloudDataConfig.

Tested: unit suite with an in-memory contract fake (compress/manifest round-trip, resume-only-missing-parts, live-session guard, pull sha-verify+decompress, auth errors) + live against production: upload→ls→quota→idempotent re-send→pull byte-exact→kill-mid-upload→resume→complete, generic --kind log, bad-key handling.

Streaming ingest (continuous WebTransport uplink) is deliberately out of scope — lands with Paul's transport; the client here is the post-run half.

[codecov]

Codecov Report

❌ Patch coverage is 76.22549% with 97 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/cli/dimos.py	15.18%	67 Missing ⚠️
dimos/cloud/data.py	79.85%	22 Missing and 6 partials ⚠️
dimos/cli/cloud.py	50.00%	1 Missing ⚠️
dimos/cloud/test_data.py	99.46%	0 Missing and 1 partial ⚠️

@@            Coverage Diff             @@
##             main    #3548      +/-   ##
==========================================
+ Coverage   76.13%   76.41%   +0.27%     
==========================================
  Files        1228     1247      +19     
  Lines      119174   120941    +1767     
  Branches    10684    10933     +249     
==========================================
+ Hits        90731    92413    +1682     
- Misses      25342    25419      +77     
- Partials     3101     3109       +8     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	70.91% <76.22%> (+0.36%)	⬆️
OS-ubuntu-latest	72.66% <76.22%> (+0.33%)	⬆️
Py-3.10	72.65% <76.22%> (+0.33%)	⬆️
Py-3.11	72.65% <76.22%> (+0.33%)	⬆️
Py-3.12	72.65% <76.22%> (+0.33%)	⬆️
Py-3.13	72.65% <76.22%> (+0.33%)	⬆️
Py-3.14	72.65% <76.22%> (+0.33%)	⬆️
Py-3.14t	72.65% <76.22%> (+0.32%)	⬆️
SelfHosted-Large	29.78% <20.58%> (+0.03%)	⬆️
SelfHosted-Linux	35.73% <20.58%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/cli/test_cloud.py	100.00% <ø> (ø)
dimos/core/global_config.py	86.66% <100.00%> (+1.56%)	⬆️
dimos/cli/cloud.py	84.78% <50.00%> (-0.33%)	⬇️
dimos/cloud/test_data.py	99.46% <99.46%> (ø)
dimos/cloud/data.py	79.85% <79.85%> (ø)
dimos/cli/dimos.py	58.72% <15.18%> (-6.36%)	⬇️

... and 51 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[spomichter]

Addressed the review + a ponytail audit pass, all in ea4a508/b64ddb0:

1. uv.lock: correction — the +lz4 hunk is in [[tool.mypy.overrides]], not dependencies; lz4>=4.4.5 was already a runtime dep on main (line 155), so no lock change is owed. (Also tidied the override's insertion point.) Branch base equals current main; the web-job red was a flake, rerunning.
2. EXDEV in pull(): fixed — download stages in TemporaryDirectory(dir=out.parent), so the final rename is same-filesystem and atomic.
3. tmpfs blow-up: fixed both sides — upload stages beside the source recording, pull beside the destination; nothing ever lands in /tmp. Regression test asserts both staging dirs.
4. Timeouts: every request (_req, _put, _download, and cli/cloud.py's device-flow calls) now carries timeout=global_config.dimos_http_timeout (default 60s); _put closes its response; URLError/TimeoutError surface as clean messages. Regression test asserts the timeout is passed.

Audit cuts in the same commit: hashlib.file_digest + shutil.copyfileobj replace all hand-rolled chunk loops (the _CHUNK module constant is deleted, not promoted — no knob left to hide), config collapses to base_url/api_key with retries/compress read from GlobalConfig at call time, functools.partial for part sends, CLI closures flattened. Net −45 lines.

[leshy]

mem2 uploads are ideally psql databases btw, there was a person on trial specing it out

[greptile-apps]

Greptile Summary

This change adds resumable cloud upload and verified pull workflows for recordings and files. Two failures remain: a server-controlled LZ4 filename can produce a destination outside the recordings directory after suffix removal, and a recording removed during upload discovery causes the command to terminate with a traceback.

Confidence Score: 2/5

The pull destination is not fully confined after LZ4 suffix transformation, and upload discovery does not tolerate recordings disappearing during filesystem enumeration.

Two independently reproduced failures remain, including one that allows a server-controlled filename to target a path outside the recordings directory.

Files Needing Attention: dimos/cloud/data.py; dimos/cli/dimos.py

Security Review

The pull workflow accepts the server-provided filename ...lz4, then removes its suffix to produce ... The resulting default destination resolves to the parent of the recordings directory before extraction attempts its final replacement.

T-Rex Logs

What T-Rex did

• T-Rex produced a proof for a posted P1 finding tied to a focused CloudData pull reproduction, supported by a reproduction source and the run output.
• T-Rex produced a proof for a posted P1 finding related to the focused upload discovery-to-stat race, accompanied by a focused reproduction source and a stack trace.
• T-Rex produced a proof for a posted P1 finding with no artifacts attached.
• T-Rex produced a general contract validation proof showing a runtime script that mocks CloudData transport and exercises normal and malicious LZ4 pulls, along with its output log.
• T-Rex produced a general contract validation proof documenting the discovery-to-stat race reproduction script and the accompanying output log that shows a FileNotFoundError traceback.

_{Ran code and verified through T-Rex}

Comments Outside Diff (2)

1. General comment

   pull accepts ...lz4 and constructs an unconfined default destination

   • Bug
     • Path('...lz4').name is accepted, and removing .lz4 yields ... With no explicit destination, the constructed output is RECORDINGS_DIR / '..', which resolves to the parent directory rather than remaining within recordings. The actual LZ4 extraction path was executed and Path.replace() attempted this target; it raised OSError: [Errno 16] Device or resource busy in the test environment.
   • Cause
     • The filename validation rejects only empty names, . and .. before suffix removal, but does not validate the post-removesuffix('.lz4') filename/component.
   • Fix
     • Derive the output basename first, then reject empty, . and .. results (and construct the destination only from that validated value).

   _{Ran code and verified through T-Rex}

2. General comment

   Upload discovery race escapes concise CLI error handling

   • Bug
     • dimos data upload --since can terminate with an uncaught FileNotFoundError when a .db recording vanishes after RECORDINGS_DIR.glob() discovers it but before p.stat() runs.
   • Cause
     • recordings_since() performs unguarded p.stat() in target discovery, and upload() calls that helper before its try/except (RuntimeError, OSError) per-target upload loop.
   • Fix
     • Handle vanished paths during latest_recording() and recordings_since() discovery (for example, skip FileNotFoundError), or extend command-level handling around target discovery to emit a concise error.

   _{Ran code and verified through T-Rex}

_{Reviews (5): Last reviewed commit: "review: atomic lz4 extraction (never tru..." | Re-trigger Greptile}

[spomichter]

@greptile review and update rating as needed

[greptile-apps]

Suffix transformation escapes confinement

A server filename of ...lz4 passes the basename check, but name.removesuffix(".lz4") becomes ... With the default destination, this constructs RECORDINGS_DIR / "..", which resolves outside the recordings directory before the staged extraction is replaced. Validate the filename after removing the LZ4 suffix and reject empty, . and .. output names before constructing out.

Artifacts

Focused CloudData pull reproduction source

• Mocks CloudData request and download behavior, then executes ordinary and `...lz4` LZ4 pulls against an isolated recordings directory; it reaches the vulnerable destination construction and replace path.

CloudData pull reproduction output

• Captured command output shows a normal confined pull succeeding and `...lz4` producing `recordings/..`, which resolves outside recordings and fails during replace.

_{Ran code and verified through T-Rex}

[greptile-apps]

Discovery errors bypass handling

If a recording disappears after it is found by glob() but before recordings_since() calls stat(), the resulting FileNotFoundError escapes before the per-file try/except block. dimos data upload --since therefore terminates with a traceback rather than the command's concise failure handling. Skip vanished files during discovery or handle discovery errors at the command boundary.

Artifacts

Focused upload discovery-to-stat race reproduction source

• The script deletes a discovered recording just before stat and calls the actual upload command function; takeaway: it deterministically exercises the claimed race.

Upload command output when a discovered recording vanishes

• Captured execution exited 1 with FileNotFoundError from recordings_since through upload before the guarded loop; takeaway: the CLI emits a traceback instead of concise failure handling.

_{Ran code and verified through T-Rex}