Added an experimental dandi image for soak-testing an unreleased fix

[CodyCBakerPhD]

Soak-test branch for dandi-cli#1910, the fix for dandi/dandi-cli#1907. Not for merge as-is.

The nightly ingest re-downloads every dandiset because dandi download -e refresh compares the local mtime against the archive's record with a 1e-6 s tolerance, and the runner's incoming root truncates mtimes to whole seconds. Measured on the runner:

wrote 1700000000.000500  read back 1700000000.000000  delta 0.000500 s
wrote 1700000000.500000  read back 1700000000.000000  delta 0.500000 s
worst round-trip deviation: 0.500000 s

dandi-cli#1910 passes an explicit 2 s tolerance. Verified against that branch: current dandi re-downloads at 1 s and 2 s granularity, the branch skips, and a genuine change is still caught (size delta either direction, or a re-upload 3 s or more later).

What this adds

dandi.Dockerfile takes a DANDI_SPEC build arg naming the pip requirement for dandi. It defaults to the PyPI release, so :latest is built exactly as before. git is installed only when the spec names a VCS ref, leaving the default build with no apt work at all.

experimental_dandi_image.yml builds and publishes dandi-cli:experimental-<tag> from that spec, on push to a claude/** branch or on demand with a different spec. Before pushing the image it asserts the fix is present in the built bytes and runs dispatch's suite inside it. Tags always carry the experimental- prefix so an image built from unmerged upstream code cannot be mistaken for the supported one.

How the soak test reaches a scheduled run

Scheduled Ingest runs data-ingest-runner's workflow from that repo's main, so it accepts no per-run overrides. What it does read is that repo's TASK_FORCE_REF variable, naming the branch of this repo to check out. So this branch's DEFAULT_DANDI_IMAGE is the lever, and it is marked do-not-merge.

To run it: set TASK_FORCE_REF in data-ingest-runner to this branch and wait for the nightly, or trigger Manual Ingest there with task_force_ref set to it. Judge the result on the Run dispatch wall time and dandi's per-asset STATUS column, where skipped / no change is the fix working and done with a full byte count is the bug. Then clear TASK_FORCE_REF and revert DEFAULT_DANDI_IMAGE.

Once a released dandi carries the fix, none of this is needed: :latest picks it up on the next image build. The DANDI_SPEC arg and the workflow are worth keeping for the next candidate fix; the DEFAULT_DANDI_IMAGE line is not.

Prompts

So, I noticed all the scheduled runs lately have taken ~15 minutes, implying to me they aren't skipping processing properly. We shouldn't be reconverting or reuploading anything unless there are new sessions added for a lab/project, or that labs/projects conversion script has changed since last run - any idea whats up?

note a different session made pr 27

(but IDK if its useful)

yes, dig into the mtime thing and fix it

feel free to provide a prompt to give to a sesion on the dandi-cli to fix that if there is an issue too

do we need any of this if we just fix the DANDI thing upstream?

ok can you try using the branch on dandi/dandi-cli#1908? Is there an easy way to do that and make sure it resolves our issue?

only thing is 1908 is A LOT more than I'd like as a solution. Any way to test competing PRs,like 1909 and 1910?

forget all that then; can you cut an experimentally tagged dandi image for branch on 1910 and use that in live scheduled runner tests to ensure it actually resolves the issue?

---

Generated by Claude Code

[CodyCBakerPhD]

it worked!