doc: design decoupled coordination - #38696
Draft
aljoscha wants to merge 17 commits into
Draft
Conversation
Define catalog authority for maintained lifecycle and separate it from request-scoped query execution. Record the ownership and readability boundaries needed for independent clients, while leaving implementation mechanisms open and full multi-adapter operation outside this deliverable. Include an append-only implementation log for work across sessions.
Make sink export installation an implication of the committed catalog diff, not a sequencer-side lifecycle call. Create exports after source and table inputs, then initialize their default read policies. Existing read holds protect inputs through installation, and SQL success still waits for it. Extend sink compatibility coverage with no-op creation and rename operations. Independent review found no blocking issue. CI Clippy and formatting passed before a documentation push superseded the run. Runtime validation remains pending, and local builds were blocked by missing artifacts and tools. This is an incremental ownership migration. Cluster-side catalog following, MV/index creation implications, and direct committed-update tests remain open.
Separate index reconstruction from bootstrap orchestration so committed catalog implications can reuse it without inheriting bootstrap timestamp selection or installation. Keep cache validation, dependency ordering, and CREATE INDEX behavior unchanged. This is preparatory work, not index lifecycle decoupling. Runtime cache reads and committed index installation remain to be implemented. Validation: adapter cargo check and Rust formatting passed. Independent review found no issues. Full formatting and lint were blocked by local tooling and dependency build failures. Runtime tests and PR CI are pending.
Make index-add implications acquire compatible cached plans or reconstruct from catalog definitions, then install after cluster and storage creation. Retain precommit optimization and success-only session notices, without a sequencer installation closure. Reuse the existing read-hold handover. Expose best-effort expression-cache reads through its task and validate cached dependencies against committed state and available compute imports. Filter session notices for optimizer dependencies dropped during planning. Extend cache-handle tests and add cache-disabled SQL restart coverage for index use, EXPLAIN, notices, and drop cleanup. Same-batch external catalog application remains a test gap until a production subscriber exists. Adapter and cache-test compilation, Rust and Python formatting, and Ruff passed. Full local formatting and lint are blocked by missing tools. Runtime validation is delegated to the draft PR CI and remains pending.
Register materialized view storage collections from committed implications before dependent sinks and indexes acquire read holds. Share descriptor construction with bootstrap to preserve initial visibility and replacement shard ownership. Apply read policies through the deferred batch. Keep compute installation and write enablement in the sequencer for this incremental step. Extend the existing MV test with a dependent index. Adapter cargo check and Rust/Python formatting pass. Independent review found no issue. Full local formatting and lint are blocked by missing tools and a doctest dependency build. Regular PR CI is pending.
Reuse bootstrap MV optimization and notice rendering as a catalog-based planning boundary for compute implications. Preserve cache policy, SQL typing, item versions, and timestamp selection. Compute installation and input read protection remain unchanged. Extend cache-disabled restart coverage to MV results, stored EXPLAIN plans, and continued maintenance after restart. Validation: adapter cargo check and Rust/Python formatting passed. Independent review found no issue. Full formatting and lint are blocked by missing tools and an OpenSSL dependency build. Runtime validation and regular PR CI are pending.
Keep a REFRESH AT CREATION materialized view unexecuted across restart, change its input after creation, and verify reconstruction uses the committed refresh timestamp rather than current input state. Extend the existing cache-disabled restart workflow without a test-only coordinator API. Record the input-protection obstacle to moving runtime MV installation. Broader creator holds can change historical readability and do not cover all access paths exposed by catalog changes. Validation: Python formatting and Ruff passed. Full local formatting and lint are blocked by missing tools and an OpenSSL dependency build. Runtime coverage is pending regular PR CI.
Record explicit catalog bounds as the compaction authority for maintained collections. Define the catalog transaction, application, and recovery boundaries while leaving representation, granularity, and publication cadence to implementation and measurement. Document delegated advancement as an alternative and explain the tradeoff between ongoing catalog publication and owner-backed DDL admission. Preserve the exploration log and append the agreed decision. Documentation checks passed, including preservation of earlier log entries. Full formatting and lint remain blocked by missing tools and the Python doctest OpenSSL dependency build. No runtime behavior changes.
aljoscha
force-pushed
the
decoupled-coordination
branch
from
September 7, 2026 12:02
9705b3c to
d1afd3d
Compare
aljoscha
changed the title
aljoscha
force-pushed
the
decoupled-coordination
branch
7 times, most recently
from
September 7, 2026 14:31
b77723e to
be76324
Compare
Protect all logical collection inputs so recovery can reconstruct from catalog SQL without preserving optimizer decisions. Record the storage boundaries, progress-driven protection, and accepted retention cost. Automatic creation timestamps account for every logical input. Reject explicit historical refresh requests incompatible with committed compaction permission. Existing objects retain their promised results and convert only once their remaining recovery needs are protected. The conversion mechanism and rollout policy remain open. Document integrity checks passed, with independent review of the recovery boundary. Full formatting and lint are blocked by missing local tools and an OpenSSL dependency build. No runtime recovery experiments or retention measurements were run.
aljoscha
force-pushed
the
decoupled-coordination
branch
2 times, most recently
from
September 8, 2026 14:15
42629f6 to
48a42d1
Compare
Expand unmaterialized views through their relational name references and raw HIR reads, independently of optimization and available indexes. Stop at persisted collection outputs and logs, preserving exact version IDs. This establishes dependency discovery for logical recovery protection without changing admission, read holds, or runtime installation. Add catalog-boundary tests for eliminated and implicit inputs, shared view subgraphs, index independence, MV boundaries, and collection versions. Adapter library and test compilation pass. Rust formatting passes. Full formatting and lint are blocked by missing tools and an OpenSSL dependency build. Runtime validation remains pending in draft PR CI.
aljoscha
force-pushed
the
decoupled-coordination
branch
from
September 8, 2026 14:28
48a42d1 to
565e789
Compare
…y tests Strip validation churn from the design's implementation log: pending CI, passed formatting checks, local tooling gaps, and review outcomes are all reconstructible from the PR and buried the findings and decisions. Update the session prompt so future handoffs record only findings, decisions, open questions, and the next step, and note which local checks work. Reduce the logical_collection_inputs tests to the promises that matter: name resolution survives planning-time elimination, raw-HIR reads count, indexes do not affect traversal, and upstream MV outputs are leaves. The dropped version test mutated catalog maps directly, and the replacement-MV case cannot run in the debug catalog because it has no shard metadata for the target.
Focus implementation on observable recovery, lifecycle, and query-client outcomes in environments initialized under the new protection rules. Defer existing-environment conversion without weakening its result preservation contract. Direct implementers to the active milestone and retain rejected alternatives as decision context rather than pending work.
Account for committed compaction permission independently of retention policies and execution holds. Preserve it during initialization, shared shard version changes, and drops, and apply monotonic advances in batches. Recovery uses protected readability rather than policy frontiers, and initialization stops if a fencing compare fails. This is the storage consumer boundary for decoupled coordination. Durable catalog production, maintained admission, and MV recovery wiring remain unimplemented, so SQL-created collections do not yet use these bounds. Five focused storage tests cover persist advancement, recovery, aliases, and drops. Recovery regressions were verified with the fixes removed. Formatting, cheap lint checks, and all-target checks through adapter and compute-client pass locally. Broad validation remains with draft PR CI.
Include logical collection inputs in MV refresh preparation and creation timestamp selection, even when planning or optimization removes their reads. Preserve early holds and acquire missing planner-introduced inputs before installation, rejecting explicit refreshes that those inputs cannot support. Carry query-scoped name references separately from statement dependencies so replacement targets do not constrain input readability. Add SQL coverage for eliminated inputs, automatic timestamps, refresh-at-creation, replacement targets, and implicit SQL-function reads. This prepares catalog-backed recovery protection but does not implement durable bounds or maintained requirements. Admission currently uses held readability. Validation: bin/fmt, cargo check for mz-adapter and mz-sqllogictest with all targets, and selected Cargo, formatting, and Python-doc lint checks passed. SQL execution and broader validation are delegated to draft PR CI.
aljoscha
force-pushed
the
decoupled-coordination
branch
from
September 8, 2026 16:16
b471041 to
540dceb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Make the catalog authoritative for maintained storage and compute lifecycle,
independent of the adapter that initiated a change. Query-local execution stays
on a separate fast protocol. The design records the ownership, read-protection,
recovery, and query-ordering boundaries for incremental implementation.
Description
Sink export creation now follows committed catalog additions instead of a
sequencer-side lifecycle call. Exports are installed after source/table inputs
and before default read-policy initialization. Existing read holds protect the
inputs throughout installation, and SQL success still waits for installation.
Bootstrap retains its separate recovery path.
This is an incremental step, not completion of decoupling. Controllers still
live in the adapter. Index/MV creation implications, cluster-side catalog
following, independent query clients, and catalog-authorized compaction remain
open. Full multi-adapter deployment and transparent query failover are not this
deliverable.
Validation
Extended
test/testdrive/kafka-sinks.tdwith no-op sink creation and renameoperations before its existing output verification. Independent review found no
blocking issue. Direct committed-update installation still needs coverage beyond
the SQL harness, including same-batch MV/sink creation.
CI validation is in progress. Local validation was limited by missing build
artifacts and tools. This PR remains a draft until the changed boundaries and
the broader design outcome are validated.
Replaces #38694 under the outcome-focused name Decoupled coordination.