allow enforcing a maximum scene length for the stitcher

[owencking]

New Feature Summary

As of swt-detection v9.0, we have a high degree of control over the number of representatives. This allows us to get frequent representatives for highly dynamic scenes, like credits. That's great!

However, some highly dynamic scenes can last for a long time. A 60s credits sequence with representatives every 2s, yields 30 representatives. For OCR/captioning workflows, this situation causes unpredictable VRAM requirements. We need some way to impose a limit for downstream VRAM demands.

It would be helpful to enforce a maximum scene length (expressed in milliseconds) for scenes with text, such that, for any would-be scene greater than that max length, the scene is split into (approximately) equal length segments each of which is less than that max.

For example, if a credits sequence lasts 180 seconds and would ordinarily have a 3-minute long scene, but there is a maximum scene length of 50 seconds, then the scene will be split into 4 scenes each of approximately 45 seconds.

Related

No response

Alternatives

No response

Additional context

This feels slightly hacky (though not too bad), but I can't figure out another way to avoid having to process an indefinitely large number of representatives in a single call downstream.

[keighrim]

@owencking I see the problem, but I'm not sure if the stitcher is the right place for a fix.

The root problem is hardware (VRAM) restriction in LLM apps, and I'd rather not fix that by reshaping the SWT annotation. Splitting a genuinely long credits sequence into arbitrary short scenes distorts the temporal segmentation for other downstream consumers that don't know about this particular practice/contract. Namely, a downstream resource limit shouldn't rewrite an upstream annotation.

I think the fix might belong at the SDK consumption boundary, where frames are extrtacted from a TimeFrame and batched into prompts, for a few reasons:

• the real constraint is image count, not scene duration; a duration cap is an indirect proxy (a 180s scene is ~90 images at reps-every-2s but ~18 at every-10s), so I'd cap on target frames, not milliseconds.
• and even a frame count isn't a portable number: the real cost is per-token, and an image's token count varies with its resolution and the model's tiling, hence it's not feasible to safely/correctly estimate memory footage only from the input text/image (or even audio). So rather than hardcode a number, the SDK should count actual tokenized "size" (probe the model once, count tokens per input) and size batches to available VRAM adaptively.
• crucially, that batching results has to stay internal: the number of forward passes can vary by GPU, but the output MMIF syntax/structure must not. So "images per forward pass" is GPU-adaptive implementation detail and invisible, while "output TextDocuments per TimeFrame" is a separate, deterministic I/O design decision.

That last point opens a new question, do we want one TextDocument per TimeFrame with concatenated LLM responses vs. multiple grounded TextDocuments, the answer might be task-dependent (OCR vs captioning) and also needs the frames-with-sources grounding (mmif-python #384).

I'm sketching this at the SDK level and will follow up with a concrete proposal (might transfer this issue when I have a better idea what goes to where). Keeping the stitcher's segmentation as-is in the meantime.

[keighrim]

So at a meeting today we discussed the computing and downstream issues of long scenes. We considered either smart batching of input images or limiting the number of images per batch to produce multiple annotations.

We decided on the second option. The immediate next step is to add a "max image cap" to the SDK runtime parameters. This provides manual control over the batch size instead of relying on complex and brittle auto-calculation logic.

For scenes exceeding this cap, the SDK will create multiple TextDocument annotations aligned with the same TimeFrame, differentiated by their origins tinme points. We recognize and accept that this violates the implicit assumption of a single TextDocument per TimeFrame, but it was never explicit anyway.

(transferring the issue to sdk repo)