Module llama cpp Roadmap

Navigation: Home > Modules

Roadmap-Hinweis: Vage Bullets ohne Akzeptanzkriterien in Checkbox-Tasks überführen. Format: - [ ] <Task> (Target: <Q/Jahr>).

llama_cpp Plugin Roadmap

Current Status

v2.2.0 — Real LlamaWrapper inference wired in behind THEMIS_LLM_ENABLED. generate(), embed(), exportLoRA, and importLoRA all delegate to LlamaWrapper when a non-empty model path is provided and the macro is set. Stub mode (empty path / CI without model) is preserved as a transparent fallback.

Completed ✅

• THEMIS_LLM_PLUGIN() export macro
• LlamaCppPlugin : ILLMPlugin — full interface (generate, RAG, embed, LoRA, stats)
• loadModel / unloadModel with stub mode and real LlamaWrapper initialisation
• Thread-safe LoRA registry (std::mutex)
• getCapabilities() — supports_lora, supports_embeddings, plugin_version
• getMemoryStats() / getPerformanceStats() — JSON
• themis_llm_create / themis_llm_destroy C-linkage entry points
• 50 unit tests (LlamaCppPluginFocusedTests, groups A–N)
• Plugin manifest + CMake registration
• Streaming token output via InferenceRequest::stream_callback (v2.1.0)
• generateStream(request, callback) convenience method (v2.1.0)
• generateBatch(requests) batch inference method (v2.1.0)
• LlamaCppPluginRegistrar — PluginManager hot-plug integration (v2.1.0)
• getCapabilities().supports_streaming = true (v2.1.0)
• getCapabilities().supports_batching = true (v2.1.0)
• Real llama.cpp inference via LlamaWrapper behind THEMIS_LLM_ENABLED (v2.2.0)
• Real embedding vectors via LlamaWrapper::embed() (v2.2.0)
• exportLoRA / importLoRA delegated to LlamaWrapper (v2.2.0)
• tests/CMakeLists.txt updated — registrar + deps added for N1–N6 (v2.2.0)
• Gap-Closure Wave 2 (2026-08-31): generate() STUB/SIMULATION NOTE completed with Removal Plan field; all 7 STUB blocks in llama_cpp_plugin.cpp and 2 in llama_cpp_registrar.cpp have complete documentation templates (Purpose, Activation, Production Delta, Removal Plan)

In Progress

(none — all previously in-progress items are now complete)

Planned Features

• Function/tool calling (Target: Q4 2026)
• Per-request cancellation token (Target: Q4 2026)

Implementation Phases

Phase 1 — Design / API Contract ✅

• ILLMPlugin interface reviewed; all methods implemented

Phase 2 — Core Implementation ✅

• LlamaCppPlugin stub with load/generate/embed/LoRA lifecycle
• Real LlamaWrapper delegation behind THEMIS_LLM_ENABLED (v2.2.0)

Phase 3 — Error Handling & Edge Cases ✅

• generate() returns error when model not loaded
• embed() returns empty when model not loaded
• Thread-safe LoRA registry (duplicate id replacement)
• generateStream() swallows callback exceptions; increments error_count_
• generateBatch() propagates per-request errors without aborting the batch
• Stub fallback when LlamaWrapper::loadModel() fails (file not found, etc.) (v2.2.0)

Phase 4 — Tests ✅

• 50 unit tests across groups A–N
• 3 group-O structured-error tests (O1–O3): generate() without model loaded
• Registrar link fixed in tests/CMakeLists.txt (v2.2.0)

Phase 5 — Performance / Hardening ✅

• Real llama.cpp inference benchmark (benchmarks/bench_llama_cpp_inference.cpp; stub path exercised in CI; 6 benchmark scenarios) (Target: Q3 2026)
• Concurrency test P1: 8 threads × 10 generate() calls — no race, no deadlock (Target: Q3 2026)
• Concurrency test P2: 4 threads concurrent generateBatch(5) — correct response count (Target: Q3 2026)
• Concurrency test P3: interleaved loadLoRA() + generate() from 6 threads — all succeed (Target: Q3 2026)

Phase 6 — Documentation & Acceptance ✅

• README, CHANGELOG, ROADMAP, ARCHITECTURE, FUTURE_ENHANCEMENTS, AUDIT, SECURITY

Phase 7 — Security & Concurrency Hardening ✅ (v2.4.0 — 2026-08-09)

• inference_count_ / error_count_ converted to std::atomic<uint64_t> — lock-free reads (A1)
• stream_retry_count_ added as std::atomic<uint64_t> — exposed in getPerformanceStats() (A1)
• generateRAG() data-race fixed: shared state (model_loaded_, context_length_) snapshotted under mutex at entry (A2)
• generateStream() stream-callback retry: up to 3 attempts for transient exceptions; bad_alloc non-retryable; stream_retry_count_ incremented per transient retry (D1)
• thread_join_no_timeout findings triaged as false positives; no helper retained because the module has no owned join sites (D2)
• importLoRA GGUF magic-bytes check (0x47 0x47 0x55 0x46) + 2 GB size bound — fail-closed (B2)
• loadModel() opt-in model-file integrity gate via verify_model_digest + expected_model_digest config keys (B3)
• setPolicyFn(PolicyFn) — pluggable inference policy hook; generate() / generateRAG() gate on denial (B4)
• LlamaCppPluginRegistrar::initFromServerConfig(server_config) — server-startup integration point; reads config["llm"]["model_path"] (C1)
• defaultReloadCallback() fixed — calls loadModel() when path present; returns true in stub mode (C2)
• LLCPG-1..4 release gate benchmarks added (TTFT, batch-embed, LoRA-load P99, regression baseline) (E1)
• Tests Groups U (concurrency, 4), V (security, 6), W (registrar integration, 8), X (retry/join, 3) — 21 new tests (Q3 2026)

Production Readiness Checklist

• Unit tests present (89 tests: groups A–X)
• Stub mode for CI without model file
• Thread-safe LoRA registry
• Capabilities correctly reported
• context_length read from config JSON (n_ctx/context_length keys, fallback 4096)
• ModelInfo::context_length populated from config on loadModel()
• generateRAG() uses RAGContextAssembler — no naive document concatenation
• InferenceRequest::max_tokens capped by RAGContextAssembler::computeMaxTokens()
• generateStream() honours callback with 3-attempt transient-exception retry
• generateBatch() preserves request order in response vector
• LlamaCppPluginRegistrar provides PluginManager hot-plug integration
• LlamaCppPluginRegistrar::initFromServerConfig() provides server-startup integration point
• Real llama.cpp inference wired in (THEMIS_LLM_ENABLED)
• Real embeddings via LlamaWrapper::embed() with L2 normalisation
• exportLoRA / importLoRA delegated to LlamaWrapper; importLoRA GGUF-validated before delegation
• Concurrency hardening verified: 8-thread generate(), 4-thread generateBatch(), 6-thread LoRA+generate() race — all pass (P1–P3)
• inference_count_ / error_count_ / stream_retry_count_ are std::atomic<uint64_t> — lock-free reads
• generateRAG() shared-state data-race eliminated (mutex snapshot at entry)
• supports_function_call = true; tool-call stub synthesised in test/stub mode; tools forwarded through bridge path (S1–S3)
• Per-request cancellation token (InferenceRequest::cancellation_token); pre-inference check returns success=false / "Request cancelled" (T1–T2)
• Model file integrity check: opt-in via "verify_model_digest": true + "expected_model_digest" config keys
• LoRA adapter integrity: GGUF magic bytes + 2 GB size bound validated in importLoRA
• Inference policy gate: setPolicyFn(fn) pluggable hook; denial returns success=false
• LLCPG-1..4 release gate benchmarks present (TTFT, batch-embed throughput, LoRA P99, regression baseline)

Known Issues & Limitations

• generateBatch() is sequential; true parallel batch requires real llama.cpp.
• Stub mode is active when compiled without THEMIS_LLM_ENABLED or when the model path is empty / the file does not exist.

Breaking Changes

v2.1.0 — getCapabilities().plugin_version changed from "2.0.0" to "2.1.0". getPluginVersion() similarly returns "2.1.0".

Latente Symbole (Unused-Functions-Audit)

Stand: 2026-08-09 – Quelle: src/UNUSED_FUNCTIONS_REPORT.md

🧪 NUR_TESTS (implementiert, kein Produktions-Aufrufer)

• LlamaCppPlugin – LLM-Plugin-Implementierung für llama.cpp; vollständig implementiert (generate, embed, generateRAG, generateStream, generateBatch, LoRA-Lifecycle, Memory/Performance-Stats, Policy-Gate, Security-Validation). 89 Unit-Tests + Benchmark vorhanden.

✅ Produktionslücke geschlossen (v2.4.0): LlamaCppPluginRegistrar::initFromServerConfig(server_config) ist als sauberer Server-Startup-Integrationspunkt implementiert. Der Server-Startup-Code kann LlamaCppPluginRegistrar::initFromServerConfig(config) aufrufen, um LlamaCppPlugin in den LLMPluginManager zu registrieren, wenn config["llm"]["model_path"] gesetzt ist. defaultReloadCallback() delegiert nun korrekt an loadModel() statt eines Stub-Kommentars.

Program Execution Model — Wave Context

This module is a contributing module in the program-level Wave A → B → C → D execution model. It does not own a primary wave deliverable but must remain release_critical-green throughout all waves and must deliver Wave D operability improvements in Q1 2027. See ../../ROADMAP.md for the full wave model and exit criteria.

Wave D Contribution for llama_cpp

• Deliver or validate distributed tracing, high-cardinality stress coverage, exporter reliability, and operator remediation hints as applicable to this module (Target: Q1 2027)
• Contribute to or validate long-duration soak test coverage for this module's primary paths (Target: Q1 2027)
• Ensure runbook coverage for operator-critical scenarios in this module (Target: Q1 2027)

Cross-Wave Requirements

• release_critical CI must remain green on develop throughout all waves (Target: ongoing)
• p95/p99 benchmarks must be refreshed on representative hardware before Wave D sign-off (Target: Q1 2027)
• No behavioral regression may be introduced into modules in Wave A/B/C scope from changes in this module.

Program-Level Success Criteria (contribution)

• This module's distributed/acceleration paths fail closed (Target: Q1 2027)
• Benchmark-backed p95/p99 baselines exist on representative hardware (Target: Q1 2027)
• Operator-critical paths have diagnostics, alerts, and runbooks (Target: Q1 2027)

---

ThemisDB 1.9.0-beta · Home · Module-Index · GitHub · Issues