Module query Roadmap

Navigation: Home > Modules

Roadmap-Hinweis: Vage Bullets ohne Akzeptanzkriterien in Checkbox-Tasks ueberfuehren. Format: - [ ] <Task> (Target: <Q/Jahr>).

Query Module Roadmap

Current Status

Production-ready multi-model query stack with parser, optimizer, execution, federation, caching, and compatibility layers in active use.

Wave Alignment (see root ROADMAP.md § Program Execution Model):

• Wave A (Q3–Q4 2026): Query planning determinism, timeout behavior consistency, cancellation semantics, distributed write stress
• Wave A Exit Criteria: Deterministic chaos evidence (query timeout, cancellation) + release-critical CI GREEN + federated execution baselines
• Wave B (Q3–Q4 2026): Distributed execution baselines, hybrid planner (ANN+graph), benchmark gates, parallel optimization
• Wave B Exit Criteria: Hybrid planner single-shard stable + federated execution p95/p99 on representative hardware
• Tier 2 Functional Completeness: Query planning is foundational for all workloads

Hybrid Retrieval Rollout Readiness: 65% 🟢 (issue #5468).

• Phase A (single-shard exact): ✅ Ready with error-path fixes (return-value checks, exception handling).
• Phase B (thread-safety): ✅ COMPLETE 2026-08-08 — QueryOptimizer per_query_cost_model_mutex_, advisor_cost_model_mutex_, adaptive_init_flag_ added; PlanCache stats atomics + deadline propagation wired; 9 tests in test_query_optimizer_thread_safety.cpp.
• Phase B (hybrid planner): 🟡 Q3 2026 — ANN + graph hybrid planner single-shard scope in progress.
• Phase C (parallel optimization): 🟡 Q3 2026+ (thread-safety gate now passed).
• Rollout risk detail: ai_working/HYBRID_RETRIEVAL_ROLLOUT_PLAN.md §7

In Progress

• [~] Query hardening wave for safety, resilience, and predictable performance (Target: Q3 2026)
  • Complete remaining performance/regression benchmark gates for vectorized and federated paths (Target: Q3 2026)
  • Continue reliability hardening for cancellation, limits, and distributed query failure behavior (Target: Q3 2026)
  • Canonical retry metadata envelope integrated into federation timeout paths (retry_count, exhaustion reason, timeout source, correlation id) with focused coverage in test_query_federation_hardness.cpp (Delivered: 2026-08-09)
• [~] approved next implementation block: support Graph Phase 3 with query-planner and optimizer hardening before follow-on scheduling/load-balancing work (Target: Q3 2026)
• [~] AQL LLM Integration Consolidation — Phase 1-4 (2026-06-18 → ongoing)
  • Formalize dependency contract between src/query/ (Query Engine) and src/aql/ (LLM Integration)
  • Define canonical parser validation pipeline and SLA for LLM-generated AQL
  • Phase 1: Define integration boundary (12 hrs) ✅ 2026-06-18
    • Created src/query/AQL_LLM_INTEGRATION_CONTRACT.md (canonical specification)
    • Updated src/query/ARCHITECTURE.md with LLM integration section
    • Updated src/aql/ARCHITECTURE.md with dependency documentation
  • Phase 2: Wire parser validation + metrics (20 hrs) ✅ 2026-06-18
    • ✅ validateAQLWithParser() implemented in llm_aql_handler.cpp:1553
    • ✅ translateNLToAQL() calls validation with retry-on-error logic
    • ✅ Created integration test suite: test_aql_llm_integration.cpp (16 test cases)
    • ✅ Added Prometheus metrics instrumentation for validation tracking
    • ✅ Verified Prometheus counters/histograms bound to LLMMetricsCollector
    • Updated src/aql/ROADMAP.md to cross-reference consolidation work
  • Phase 3: Consolidate documentation (12 hrs) ✅ 2026-08-05
    • ✅ Created AQL_LLM_INTEGRATION_PHASE3_PARSER_CHANGES.md (3h, comprehensive parser evolution doc)
    • ✅ Created AQL_LLM_INTEGRATION_PHASE3_METRICS.md (3h, metrics interpretation guide)
    • ✅ Created AQL_LLM_INTEGRATION_PHASE3_API_CONTRACT.md (2h, public API definition)
    • ✅ Created AQL_LLM_INTEGRATION_MIGRATION_GUIDE.md (2h, user adoption guide)
    • ✅ Updated AQL_CONSOLIDATION_INDEX.md with Phase 3 links (1h)
    • ✅ All cross-references verified; no broken links (1h, QA validation)
  • Phase 4: Validation SLA performance tests (20 hrs) ✅ COMPLETE (2026-08-05)
    • ✅ Created test_aql_validation_performance.cpp (8 performance test cases)
    • ✅ Tests verify SLA: ≤500ms per parse, ≥100 q/s throughput, <50ms error enrichment
    • ✅ Registered in tests/query/CMakeLists.txt with performance tier/labels
    • ✅ Build verification: All dependencies resolved (fmt, boost, spdlog, etc.)
    • ✅ SLA validation: All 8 tests pass; performance targets confirmed met
    • ✅ Detailed results: PHASE_6B_SLA_VALIDATION_REPORT.md
  • Full detailed roadmap: AQL_CONSOLIDATION_AUDIT_2026_06_18.md
• AQL Mutations Language Extension — Phase 1-5 Complete (v2.0.0, 2026-07-15)
  • INSERT, UPDATE, REPLACE, REMOVE, UPSERT statements implemented and tested
  • Transaction block integration (BEGIN...COMMIT) with atomic multi-statement batching
  • Full detailed roadmap + delivery evidence: AQL_MUTATIONS_ROADMAP.md
  • Phase 1: Parser & Tokenizer Enhancement ✅ 2026-07-15
  • Phase 2: Safety & Semantic Validation ✅ 2026-07-15
  • Phase 3: Translation & Execution Plan ✅ 2026-07-15
  • Phase 4: Transaction Support & Atomicity ✅ 2026-07-15
  • Phase 5: Testing, Performance & Documentation ✅ 2026-07-15
  • Open: migration guide (Target: Q4 2026), security audit (Target: Q4 Week 1), production checklist final sign-off
• [~] AQL v2.0.0 Remaining Work — Geospatial + FTS wiring (Target: Q3–Q4 2026)
  • Full language roadmap: AQL_V2_0_0_COMPLETE_ROADMAP.md
  • DDL (CREATE/DROP COLLECTION/INDEX/VIEW) — parser + executor + 32 tests delivered 2026-07-22
  • Geospatial parser Phase 1 COMPLETE 2026-07-27: ST_* already work in FILTER/SORT/RETURN via qe_evalFunction; 26 tests in test_aql_st_predicates.cpp; Phase 2 (optimizer hints) next (Target: Q3 2026)
  • FTS query enhancement (phrase/proximity queries; ≤100ms on 100K documents) (Target: Q3–Q4 2026)
    • SEARCH/PHRASE/NEAR/STARTS_WITH/BOOST/ANALYZER tokens added to lexer (2026-08-09)
    • FtsPredType enum, FtsPredicateNode, SearchClauseNode structs added to aql_parser.h (2026-08-09)
    • parseSearchClause() implemented; wired into parseQuery() (2026-08-09)
    • search_clause field added to Query struct (2026-08-09)
    • Executor backend wiring (FTS index lookup, scoring) (Target: Q4 2026) — delivered 2026-09-03 (Block 2)
    • Phrase/proximity executor behavior is now source-covered with focused FtsExecutor tests (exact phrase, case-insensitive phrase, bounded proximity, boolean reuse, timeout, OOM) and production-backed matching over posting-list positions (2026-09-09)
    • Benchmark harness now exercises the real FtsExecutor on an on-disk 10K/100K corpus in benchmarks/rag/bench_fts_phase_b.cpp instead of the former synthetic token simulation (2026-09-09)
    • Performance gate: ≤100ms on 100K documents (Target: Q4 2026) — local benchmark evidence captured 2026-09-09 with BM_FtsPhraseQuery/100000 p95_ms=1.3638 and BM_FtsProximityQuery/100000 p95_ms=1.53029
    • [~] Broad baseline refresh automation now includes the FTS benchmark artifact path on Build: Benchmarks run 34345454063; representative-hardware capture remains open. (Target: Q1 2027)
  • [~] Cross-feature integration tests (1000+ tests, zero v1.x regressions) (Target: Q4 2026)

Phase 2 — Performance & Scalability Readiness (Target: 2026-09-30)

Performance and scalability optimization with measurable, gated performance improvements validated against Wave 7 gates.

Graph/Query Optimization Backlog

Core Goal: Deliver query optimization features with measurable latency/throughput improvements validated against Wave 7 gates.

1. Query Plan-Cache Implementation (Target: Q3 2026)

Scope: src/query/optimizer/ (plan caching subsystem)

• Implement query plan caching mechanism (Target: Q3 2026) ✅ COMPLETE

  • Design: LRU cache for query plans (size: tunable, default 1000 plans)
  • Key: normalized query text + parameter types
  • Value: compiled execution plan
  • Invalidation: on schema changes, statistics updates
  • Delivery: plan_cache.cpp (existing, verified production-ready)

• Add plan reuse validation (Target: Q3 2026) ✅ COMPLETE

  • Detect plan invalidation conditions (schema DDL, index drops)
  • Implement cache invalidation with precise granularity (table-level)
  • Log cache invalidations for diagnostics

• Performance gate: 10%+ latency improvement (Target: Q3 2026) ✅ COMPLETE

  • Measure: 10%+ latency improvement for repeated queries
  • Validate: Regression tests PG1, PG2 in test_query_optimizer_regression.cpp
  • Baseline: read p99≤200µs, batch p99≤5ms

• Add stress tests (Target: Q3 2026) ✅ COMPLETE

  • Cache coherency under concurrent queries (CMX-style tests)
  • Cache invalidation under concurrent DDL (test STRESS1_CacheCoherencyConcurrentDDL)
  • Cache eviction under plan explosion (test PC5_CacheLRUEviction)
  • Delivery: test_query_optimizer_regression.cpp (33 total test cases)

2. Cost-Model Refinement (Target: Q3 2026) ✅ COMPLETE

Scope: src/query/optimizer_cost_model_enhancements.h/.cpp

• Improve cardinality estimation (Target: Q3 2026) ✅ COMPLETE

  • Implement histogram-based estimation (ColumnHistogram struct)
  • Use table statistics (row count, column selectivity)
  • Add correlation awareness (ColumnCorrelation struct)
  • Delivery: optimizer_cost_model_enhancements.h/.cpp

• Implement cost-based join ordering (Target: Q3 2026) ✅ COMPLETE

  • Compare: nested-loop vs. hash-join vs. merge-join costs
  • Select: lowest-cost join order
  • Validate: execution stats match estimates (tests CM1-CM3)
  • Delivery: optimizer_cost_model.cpp (existing joint cost estimation methods)

• Add estimate validation (Target: Q3 2026) ✅ COMPLETE

  • Log estimate vs. actual cardinality (EstimateValidation struct)
  • Compute estimation error metrics (computeMAPE, computeP95Error)
  • Flag systematic underestimation/overestimation (tests EV3, EV4)
  • Delivery: CostModelEnhancements static methods + tests

• Create diagnostics (Target: Q3 2026) ✅ COMPLETE

  • Cardinality estimates logged in CostModelEnhancements::recordEstimate
  • Warning: large estimation errors (>50%, flagged as WARN level)
  • Systematic bias detection with recommendations
  • Delivery: optimizer_cost_model_enhancements.cpp logging integration

3. Cache-Efficiency Improvements (Target: Q3 2026)

Scope: src/storage/buffer_pool/, src/query/ (cache-aware optimizations)

• Profile and optimize buffer-pool hit rate (Target: Q3 2026)

  • Measure: cache hit rate before/after optimization
  • Implement: cache-aware prefetching (sequential access patterns)
  • Optimize: LRU eviction policy tuning

• [~] Implement cache-aware data layout (Target: Q3 2026)

  • Co-locate frequently-joined columns (reduce cache misses)
  • Minimize cache line waste (column grouping)
  • Validate: latency improvement under scan workloads

• Add cache-pressure telemetry (Target: Q3 2026)

  • Track: cache miss rate, eviction rate, pressure level
  • Implement: adaptive eviction (age-based, frequency-based)
  • Alert: high pressure conditions

• [~] Gate on performance (Target: Q3 2026)

  • Validate: no regression under scan workloads
  • Document: cache efficiency metrics in ROADMAP.md

4. Resource-Pooling & Load-Balancing (Target: Q3 2026)

Scope: src/scheduler/, src/network/ (thread pools, connection pools)

• [~] Consolidate thread-pool management (Target: Q3 2026)

  • Audit: existing thread pools across modules
  • Implement: centralized pool manager (if not present)
  • Standardize: pool sizing, queue depths, overflow handling

• [~] Implement adaptive pool sizing (Target: Q3 2026)

  • Measure: queue depth, wait times, CPU utilization
  • Adjust: pool size based on workload pressure
  • Implement: gradual scaling (avoid thrashing)

• [~] Add connection pool rebalancing (Target: Q3 2026)

  • Distribute: connections evenly across backend shards
  • Rebalance: on shard failure/recovery
  • Validate: no starvation under uneven load

• [~] Gate on throughput (Target: Q3 2026)

  • Validate: write latency stable under peak load
  • Document: pool sizing recommendations

Performance Regression Gates (Target: 2026-09)

1. Execute Wave-7 Full Suite:

   • [~] bench_w7a_release_critical_signoff (RCS-01..08) (Target: 2026-09)
   • [~] bench_w7b_endurance_soak (SOK-01..08) (Target: 2026-09)
   • [~] bench_w7c_degradation_fault_recovery (DFR-01..08) (Target: 2026-09)
   • [~] bench_w7d_guardrails_variance_operability (GVO-01..08) (Target: 2026-09)

2. Validate Gate Status: (Target: 2026-09)

   • [~] GATE-W7-01: Read p99 ≤ 200µs ✓ (Target: 2026-09)
   • [~] GATE-W7-02: Write ≥ 80k ops/s ✓ (Target: 2026-09)
   • [~] GATE-W7-03: Range p99 ≤ 500µs ✓ (Target: 2026-09)
   • [~] GATE-W7-04: Batch p99 ≤ 5ms ✓ (Target: 2026-09)
   • [~] GATE-W7-05/06: Self-check counters ✓ (Target: 2026-09)

3. Baseline Variance Report: (Target: 2026-09)

   • [~] Use: benchmarks/wave7/report_variance_w7.py (Target: 2026-09)
   • [~] Measure: p50, p95, p99, max latencies (Target: 2026-09)
   • [~] Document: variance envelope and drift tolerance (Target: 2026-09)

Repeatable Under-Load Results (Target: 2026-09)

1. Endurance Benchmark Suite: (Target: 2026-09)

   • [~] Duration: 8+ hour soak tests (Target: 2026-09)
   • [~] Workload: Mix of read/write/range/batch operations (Target: 2026-09)
   • [~] Load: Realistic production intensity (Target: 2026-09)

2. Stability Measurements: (Target: 2026-09)

   • [~] Track: p95/p99 latency over time (Target: 2026-09)
   • [~] Detect: performance drift or degradation (Target: 2026-09)
   • [~] Alert: significant variance (>10% change) (Target: 2026-09)

3. Archive Results: (Target: 2026-09)

   • [~] Save: benchmark data for release sign-off (Target: 2026-09)
   • [~] Document: latency/throughput envelopes (Target: 2026-09)
   • [~] Baseline: for regression detection (Target: 2026-09)

Phase 2 Exit Criteria

• [~] Query plan-cache: ✓ implemented, 10%+ improvement gated (Target: 2026-09)
• [~] Cost-model: ✓ cardinality estimation validated, joins optimized (Target: 2026-09)
• [~] Cache-efficiency: ✓ hit rate improved, prefetching added (Target: 2026-09)
• [~] Resource pooling: ✓ consolidated, adaptive sizing functional (Target: 2026-09)
• [~] Wave 7 gates: ✓ all 6 PASS, no regressions (Target: 2026-09)
• [~] 130+ new unit/integration tests created and passing (Target: 2026-09)
• [~] Performance envelopes documented and baselined (Target: 2026-09)

Deliverables

1. Query optimization implementations (plan-cache, cost-model, cache-efficiency, pooling)
2. 130+ focused tests (CACHE-01..20, OPT-01..30, POOL-01..20, E2E-QUERY-01..40)
3. Wave-7 baseline validation report
4. Performance envelope documentation
5. Updated src/query/ROADMAP.md with delivery status

Notes

• Prioritize improvements with measurable Wave-7 gate validation
• No optimization without baseline measurement
• Document all performance tuning assumptions
• If gates regress, investigate and remediate before declaring phase complete

---

Planned Features

Wave 9 Block 3 Delivery (2026-08-26)

• W9-10: 7 HIGH gaps closed — catch_all_swallow, memory_leak, null_dereference, todo_as_productionlogic, uncaught_exception, unchecked_result, string_concat_loop (Target: Q3 2026) ✅ COMPLETE
• W9-11: AQL FunctionCall compat shim deprecation warning added; callers documented; W10-C: compat shim removed — no callers existed; canonical SimilarityCall/ProximityCall paths are sole active dispatch (Target: Q4 2026) ✅ COMPLETE
• W9-12: Hybrid ANN+graph planner planAnnGraphHybrid() delivered in tensor_aware_query_optimizer.cpp; AnnFrontdoor + IKnowledgeGraph::neighbours() + RRF fusion; 500ms gate; 14 regression tests (Target: Q3 2026) ✅ COMPLETE

Hybrid Retrieval Rollout Gates (issue #5468)

• [~] Phase A gate: fix 50% of return-value check gaps (340 → 170) in optimizer (Target: Q3 2026)
• [~] Phase A gate: fix 50% of exception-handling gaps (180 → 90) in optimizer (Target: Q3 2026)
• Phase A ctest gate: test_query_planner_fallback with degraded-mode injection (tests/query/test_query_planner_fallback.cpp: degraded mode + query_planner_fallback_total check present)
• Phase B gate: fix thread-safety gaps in parallel plan optimization (140 → 56) (COMPLETE 2026-08-08: QueryOptimizer per_query_cost_model_mutex_, advisor_cost_model_mutex_, adaptive_init_flag_ added; PlanCache stats atomics + deadline propagation wired; test_query_optimizer_thread_safety.cpp — 9 tests)
• Phase B gate: hybrid planner (ANN + graph) enabled with single-shard scope (planAnnGraphHybrid delivered W9-12 2026-08-26)
• Phase B gate: query_planner_fallback_total Prometheus metric wired (src/query/query_optimizer.cpp:177)
• Phase C gate: parallel optimization enabled after thread-safety gate passed (Target: Q3 2026+)

Short-term (3-6 months)

• Replace makeNotImplemented()/stub-backed process-mining AQL gaps with context-backed trace, pattern-match, and ideal-comparison execution in src/query/functions/process_mining_functions.cpp (Target: Q4 2026) — delivered 2026-08-31
• Replace the three generic runtime-error paths in src/query/functions/ethics_functions.cpp (ETHICS_GET_ARGUMENTS, ETHICS_FIND_SIMILAR_DILEMMAS, ETHICS_TRAVERSE_CHAIN) with collection/graph/vector-backed execution over FunctionContext scanners (Target: Q4 2026) — delivered 2026-08-31
• AQL Mutations — INSERT/UPDATE/REPLACE/REMOVE/UPSERT for data manipulation (UPSERT/INSERT/UPDATE/REMOVE/REPLACE nodes already in include/query/aql_parser.h; DML in parse loop)
• Harden optimizer decision quality under skewed statistics and changing workloads (Target: Q4 2026)
• Expand federated query failure handling with deterministic partial-result policies (Target: Q4 2026)
• Strengthen query resource-limit enforcement diagnostics and operator-facing observability (Target: Q4 2026)

Mid-term (6-12 months)

• Advance approximate query processing from baseline implementations to production-suitable coverage (Target: Q1 2027)
• Extend ML-assisted optimization with strict fallback guarantees and reproducibility checks (Target: Q1 2027)
• Expand continuous-query backpressure and persistence hardening for long-lived subscriptions (Target: Q1 2027)

Implementation Phases

Phase 1: Safety and Access Hardening

• Keep parser/translation safety checks complete for edge-case query inputs (Target: Q3 2026)
• Ensure collection/access validation paths are enforced consistently across all execute entry points (Target: Q3 2026)

Phase 2: Optimizer and Planning Hardening

• Improve plan-selection robustness under stale or partial statistics (Target: Q4 2026)
• Add deterministic regression packs for rewrite, cost, and adaptive plan switches (Target: Q4 2026)
• sequence optimizer/cache hardening ahead of broader runtime and federation work in the current execution block (Target: Q3 2026)

Phase 3: Federation and Distributed Query Hardening

• Expand cross-cluster/federated timeout and retry envelopes with bounded memory behavior (Target: Q4 2026)
• Validate shard routing and partial-failure semantics under fault-injection (Target: Q4 2026)

Phase 4: Runtime and Performance Hardening

• Re-baseline vectorized execution performance and memory envelopes on representative datasets (Target: Q1 2027)
• Tighten JIT fallback and equivalence checks for hot-query compilation paths (Target: Q1 2027)

Phase 5: Documentation and Release Readiness

• Keep query docs source-aligned with explicit sourcecode verification evidence per update cycle (Target: ongoing)
• Keep completed roadmap items exclusively in changelog (Target: ongoing)

Production Readiness Checklist

• Status: Tracking in progress
• Nachweise: query focused tests, federation tests, optimizer tests, performance suites
• Hinweis: Abgeschlossene Arbeit wird ausschliesslich in CHANGELOG dokumentiert.

Known Issues and Limitations

• Some long-horizon performance guarantees for federation/vectorized paths still require broader benchmark evidence.
• Advanced approximate/ML optimizer paths need additional production hardening and fallback verification.
• Continuous-query scale and backpressure behavior needs further validation under sustained load.

Breaking Changes

• Query public APIs and language compatibility paths remain additive-first in active major lines.
• Any future behavioral change requiring client migration must be versioned and documented in changelog/migration notes.

Program Execution Model — Wave Context

This module is a contributing module in the program-level Wave A → B → C → D execution model. It does not own a primary wave deliverable but must remain release_critical-green throughout all waves and must deliver Wave D operability improvements in Q1 2027. See ../../ROADMAP.md for the full wave model and exit criteria.

Wave D Contribution for query

• Deliver or validate distributed tracing, high-cardinality stress coverage, exporter reliability, and operator remediation hints as applicable to this module — tests/query/test_query_highcardinality_stress.cpp (HighCardinalityQueryParsing, ConcurrentPlannerStress, ComplexJoinQueryStress) delivered; docs/operability/RUNBOOK_QUERY_ENGINE.md (5 scenarios: PlannerOOM, ExecutorDeadlock, PlanCacheCorruption, SlowQuery, JoinExplosion) delivered (Delivered: 2026-09-16)
• Contribute to or validate long-duration soak test coverage for this module's primary paths — tests/integration/test_query_engine_soak.cpp (QuerySoak_ParsePlanExecuteThroughput ≥2000 q/s, QuerySoak_PlannerStability, QuerySoak_ConcurrentExecutionReliability) delivered (Delivered: 2026-09-16)
• Ensure runbook coverage for operator-critical scenarios in this module — docs/operability/RUNBOOK_QUERY_ENGINE.md with 5 incident classes and log patterns (Delivered: 2026-09-16)

Cross-Wave Requirements

• release_critical CI must remain green on develop throughout all waves (Target: ongoing)
• p95/p99 benchmarks must be refreshed on representative hardware before Wave D sign-off (Target: Q1 2027)
• No behavioral regression may be introduced into modules in Wave A/B/C scope from changes in this module.

Program-Level Success Criteria (contribution)

• This module's distributed/acceleration paths fail closed — existing error handling confirmed; [QUERY:PlannerOOM], [QUERY:ExecutorDeadlock], [QUERY:JoinExplosion] runbook paths documented (Delivered: 2026-09-16)
• Benchmark-backed p95/p99 baselines exist on representative hardware — benchmarks/query/bench_query_dedicated_gates.cpp (QE-BM-01..04: parse p95, plan p95, execute p95, concurrent throughput) delivered (Delivered: 2026-09-16)
• Operator-critical paths have diagnostics, alerts, and runbooks — docs/operability/RUNBOOK_QUERY_ENGINE.md with 5 scenarios, log patterns, and remediation tables (Delivered: 2026-09-16)

---

ThemisDB 1.9.0-beta · Home · Module-Index · GitHub · Issues