Module ingestion Roadmap

Navigation: Home > Modules

Ingestion Module Roadmap

Current Status

Production ingestion runtime exists across multi-source connectors, orchestration controls, validation/quality gating, and workflow-driven extraction support.

In Progress

• [~] hardening connector parity and deterministic fallback across mixed capability environments (Target: Q3 2026)
• [~] benchmark stabilization for ingestion throughput, extraction, and quality-judge hot paths (Target: Q3 2026)
• [~] diagnostics consistency for connector, validation, and workflow incident classes (Target: Q3 2026)
  • Canonical retry taxonomy bridge added in ingestion_api_contract.h (toRetryTimeoutSource, toRetryExhaustionReason) with focused assertions in test_ingestion_contract_hardening_focused.cpp (Delivered: 2026-08-09)
• [~] route filesystem ingestion through WorkflowEngine profile execution path when engine is configured (Target: Q4 2026)
  • [~] extend parity path: workflow_engine_mode=filesystem now enables workflow-based execution for plugin/object-storage sources that provide filesystem-backed locations (Target: Q4 2026)
  • source-integrity remediation for workflow routing core completed: compile-breaking token corruption in workflow_engine.cpp and malformed method scoping/call tokens in ingestion_manager.cpp removed; workflow dispatch path is source-validated again (Delivered: 2026-09-07)
• [~] Phase D: Embedding & Index Version Governance (Target: Q4 2026)
  • Specification Document: src/ingestion/EMBEDDING_VERSION_GOVERNANCE.md (2026-09-24 CREATED)
  • Core Ingestion Changes:
    • Reindex decision engine triggered on embedding model/dimension/chunking changes
    • Dual-read query-time comparison (read current index, compare against candidate index during canary)
    • Safe embedding model migration without requiring full re-ingestion
  • Index Manifest V1 Schema: src/index/INDEX_MANIFEST_V1_SCHEMA.json for RocksDB persistence
  • Acceptance Criteria:
    • Reindex triggers fire correctly on all 4 conditions (model_id, embedding_dim, chunking_profile_id, schema major version)
    • Dual-read comparison produces bit-identical validation results
    • Index manifest RocksDB persistence validated with data durability tests
  • CMake Feature Gate: THEMIS_EMBEDDING_VERSION_GOVERNANCE enables canary deployment pattern
  • CI Gate: .github/workflows/gate-pr-rag-version.yml validates version governance on ingestion changes

Planned Features

Short-term (3-6 months)

• tighten deterministic behavior under high-volume mixed-source ingestion workloads (Delivered: Wave D 2026-Q4)
• extend stress coverage for checkpoint/quarantine/retry edge scenarios (Delivered: Wave D — tests/ingestion/test_ingestion_highcardinality_stress.cpp)
• improve operator-facing diagnostics for connector/validation/quality incidents (Delivered: Wave D — docs/operability/RUNBOOK_INGESTION_PIPELINE.md)

Mid-term (6-12 months)

• [~] re-baseline p95/p99 envelopes for ingestion control-plane operations (Target: Q1 2027)
• [~] broaden benchmark depth for quality-judge and extraction-intensive workflows (Target: Q1 2027)
• [~] harden long-running reliability under sustained ingest pressure (Target: Q1 2027)

Implementation Phases

Phase 1: Design / API Contract

• freeze connector/control/quality/workflow contracts for active major line (Delivered: Q3 2026)
• define explicit error taxonomy for connector, validation, and workflow failure classes (Delivered: Q3 2026)
  • Contract header: include/ingestion/ingestion_api_contract.h
  • Error codes: INGESTION_BUFFER_FULL, INGESTION_SCHEMA_INVALID, INGESTION_DUPLICATE_KEY, INGESTION_QUOTA_EXCEEDED, INGESTION_TIMEOUT, INGESTION_BATCH_ROLLBACK, INGESTION_BACKEND_UNAVAILABLE, INGESTION_CONNECTOR_FAILED, INTERNAL_ERROR

Phase 2: Core Implementation

• [~] complete hardening for connector intake and orchestration internals (Target: Q4 2026)
• [~] align validation/quality/workflow behavior to bounded runtime contracts (Target: Q4 2026)
• [~] move IngestionManager filesystem main-path execution to profile-driven workflow engine routing with fail-closed processing errors (Target: Q4 2026)

Phase 3: Error Handling and Edge Cases

• [~] standardize fail-safe behavior for unsupported/degraded connector scenarios (Target: Q4 2026)
• [~] unify diagnostics across retry/quarantine/validation/quality incidents (Target: Q4 2026)

Phase 4: Tests

• expand focused regressions for mixed connector and workflow edge scenarios (Delivered: Q3 2026)
• extend deterministic stress fixtures for high-throughput ingestion operations (Delivered: Q3 2026)
  • Test file: tests/ingestion/test_ingestion_contract_hardening_focused.cpp
  • Test cases: INCH-01..INCH-16 (write-path, batch atomicity, schema validation, back-pressure)
  • kIngestionContractSeed = 42; all tests self-contained, no external I/O

Phase 5: Performance and Hardening

• lock benchmark-backed release gates for ingestion hot paths (Delivered: Q3 2026)
• validate p95/p99 and throughput behavior against release baselines (Delivered: Q3 2026)
  • Benchmark file: benchmarks/ingestion/bench_ingestion_release_gates.cpp
  • Gates: INRG-01..INRG-06 (write ≥500k/s, batch ≤5ms, schema ≤100µs, buffer ≤10µs, backpressure ≤50µs, quota ≤50µs)
  • kIngestionCanonicalSeed = 42; Repetitions(5)

Phase 6: Documentation and Acceptance

• core ingestion module docs aligned to source-verifiable behavior
• roadmap/future planning separated from historical changelog entries
• Phase 1-6 Wave 3B Category-D closure delivered (Q3 2026)
  • Contract header, 16 focused tests (INCH-01..16), 6 release-gate benchmarks (INRG-01..06)

Production Readiness Checklist

• core ingestion surfaces documented and source-verified
• module-level security and failure behavior documented
• benchmark mapping documented in performance expectations
• [~] remaining hardening tasks closed for connector/control/workflow edge paths
• [~] release benchmark stabilization complete

Known Issues and Limitations

• runtime behavior depends on connector capability, runtime flags, and workflow configuration.
• selected connector and workflow edge scenarios need continued hardening.
• benchmark breadth should continue expanding for advanced ingestion scenarios.

Breaking Changes

No breaking ingestion contract planned. Any contract-breaking change requires migration notes and changelog entry before merge.

Program Execution Model — Wave Context

This module is a contributing module in the program-level Wave A → B → C → D execution model. It does not own a primary wave deliverable but must remain release_critical-green throughout all waves and must deliver Wave D operability improvements in Q1 2027. See ../../ROADMAP.md for the full wave model and exit criteria.

Wave D Contribution for ingestion

• Deliver or validate distributed tracing, high-cardinality stress coverage, exporter reliability, and operator remediation hints as applicable to this module (Delivered: 2026-09-16 — tests/ingestion/test_ingestion_highcardinality_stress.cpp, IN-BM-01..04 benchmark gates)
• Contribute to or validate long-duration soak test coverage for this module's primary paths (Delivered: 2026-09-16 — tests/integration/test_ingestion_pipeline_soak.cpp)
• Ensure runbook coverage for operator-critical scenarios in this module (Delivered: 2026-09-16 — docs/operability/RUNBOOK_INGESTION_PIPELINE.md)

Cross-Wave Requirements

• release_critical CI must remain green on develop throughout all waves (Target: ongoing)
• p95/p99 benchmarks must be refreshed on representative hardware before Wave D sign-off (Target: Q1 2027)
• No behavioral regression may be introduced into modules in Wave A/B/C scope from changes in this module.

Program-Level Success Criteria (contribution)

• This module's distributed/acceleration paths fail closed (Delivered: Wave D 2026-09-16)
• Benchmark-backed p95/p99 baselines exist on representative hardware (Delivered: IN-BM-01..04 in benchmarks/ingestion/bench_ingestion_dedicated_gates.cpp)
• Operator-critical paths have diagnostics, alerts, and runbooks (Delivered: docs/operability/RUNBOOK_INGESTION_PIPELINE.md)

---

ThemisDB 1.9.0-beta · Home · Module-Index · GitHub · Issues