Feature Request: Expose a safe MTMD memory-usage probe for non-causal image limits

[ardan-bkennedy]

Problem

llama.cpp commit 552f18f912a32ea86edf82e2b76431cb7131538d changed llama-server and mtmd-cli to inspect an mmproj before normal initialization. For projectors using non-causal image attention, they cap the effective image token maximum to the text context's physical batch size:

const int n_ubatch = llama_n_ubatch(lctx);
auto usage = mtmd_get_memory_usage(mmproj_path, mparams);

if (usage.use_non_causal && usage.image_max_tokens > n_ubatch) {
    mparams.image_max_tokens = n_ubatch;
    mparams.image_min_tokens = std::min(mparams.image_min_tokens, n_ubatch);
}

Other apps needs the same pre-initialization information. A global image_max_tokens = n_ubatch workaround is incorrect because causal projectors can process image tokens in chunks and should not have their supported resolution reduced.

Current upstream API gap

mtmd_get_memory_usage is unsuitable for C and FFI consumers:

• It is declared outside MTMD's extern "C" block.
• It returns mtmd_memory_usage by value using the C++ ABI.
• mtmd_memory_usage contains std::map<ggml_backend_dev_t, size_t>.
• Failure is represented by an empty/default result rather than an explicit status.
• The header labels the API unstable and says it may change without deprecation.

Yzma uses Go/libffi and must not mirror a compiler- and standard-library-specific std::map layout or bind a mangled C++ symbol.

Capability needed upstream

A supported upstream interface must allow a binding to probe an mmproj before normal MTMD initialization and obtain:

• the effective image_max_tokens;
• whether default image decoding uses non-causal attention;
• an explicit success/failure result.

The API and returned data must be C-compatible and exported from the installed/shared MTMD interface. Kronk does not require the per-device byte map. If upstream keeps that map in the public result, it needs a C representation, explicit ownership rules, and a corresponding release function.

The exact API design belongs to llama.cpp. Viable designs include a small projector-capability probe or a C-compatible replacement for mtmd_get_memory_usage.

Duplicate and related-work search

No standalone llama.cpp issue or active implementation PR was found for this exact API gap as of 2026-10-02.

There is already an exact request by JamePeng in the only comment on #29773. It proposes replacing the exposed std::map with a C-compatible entry array and explicit ownership/status handling. Any new upstream tracker should reference that comment rather than presenting the problem as previously unreported.

Related upstream work:

• mtmd: cap max_image to ubatch for non_causal models #29773 introduced the non-causal n_ubatch clamp and the two required probe values.
• mtmd: fix mtmd_get_memory_usage #24867 fixed and optimized mtmd_get_memory_usage, but did not make it C-compatible.
• mtmd_decode_use_non_causal(ctx, nullptr) is public after a context exists, but Kronk needs the information before its normal initialization so it can set the effective image limits correctly.

Search terms already checked included mtmd_get_memory_usage, mtmd_memory_usage, image_max_tokens, use_non_causal, n_ubatch, extern C, C-compatible, and FFI across llama.cpp issues and pull requests.

Source links

• Upstream behavior: 552f18f
• Merged implementation PR: mtmd: cap max_image to ubatch for non_causal models #29773
• Existing C-API request: mtmd: cap max_image to ubatch for non_causal models #29773 (comment)
• Earlier probe work: mtmd: fix mtmd_get_memory_usage #24867
• Yzma binding request: Expose a safe MTMD memory-usage probe for non-causal image limits hybridgroup/yzma#436
• Kronk implementation tracker: Cap non-causal MTMD image tokens to n_ubatch once yzma exposes the probe ardanlabs/kronk#1007
• Current declaration: https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/mtmd.h