[server] ZooKeeper Removal and Embedded Raft Metadata Backend

[XuQianJin-Stars]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Fluss currently depends on an external Apache ZooKeeper ensemble for cluster coordination, metadata persistence, sequence IDs, TabletServer liveness, dynamic config, and ACLs. Clients never talk to ZooKeeper (bootstrap.servers + Admin RPC only), but every server deployment still needs a healthy ZK quorum.

This extra stateful system is a major operational burden:

• A production Fluss cluster cannot start without ZooKeeper (quorum, tick time, session timeout, SASL/ACL, backups).
• ZK failures cascade into coordinator leadership, TabletServer registration, and metadata availability.
• Container, Kubernetes, edge, and single-node developer setups pay for a second service that exists only for coordination.
• Extra network hops and a second failure domain sit on the metadata path.

This matches the published architecture direction: ZooKeeper is transitional and should be replaced by KvStore for metadata and Raft for coordination. Kafka KRaft and similar embedded-journal designs show the same class of change is viable when the consensus group stays small and the data plane stays out of the quorum.

Goals

1. Default deploy is Fluss-only: CoordinatorServers form a Raft group; no ZooKeeper process is required.
2. A supported dual-backend window so existing ZK clusters can keep running and migrate with tools, not a flag day.
3. No change to client APIs, log/KV format, or bootstrap.servers.
4. A later major (Phase 6) removes ZK mode after soak. That deletion is not part of the first release of this work.

Tracking FIP: FIP-52.

Solution

Replace ZooKeeper with an embedded Apache Ratis Raft quorum on CoordinatorServers, plus a RocksDB-backed metadata KvStore.

• Only coordinators are Raft voters. TabletServers are observers: they discover a coordinator from coordinator.seed.nodes, register/heartbeat over RPC, and consume a pull-based FetchMetadata stream.
• Consensus is KRaft-style commit-apply: the store is mutated only after a log entry is committed. The Raft term is the coordinator epoch.
• Backend selection is exclusive:
  • zookeeper.address only → ZooKeeper (deprecated)
  • coordinator.seed.nodes only → Raft
  • both keys → fail fast
  • neither key → Raft (default)
• Public client APIs, binary log/KV format, and bootstrap.servers stay unchanged. New operator CLI: fluss raft * and fluss zk-to-raft-*.
• This issue / FIP covers Phases 1–5 (dual-support, testing, migration tools, docs, Raft as default). Phase 6 (delete ZooKeeper mode and the shaded ZK dependency) is out of the first merge.

Related materials

• FIP-52: FIP-52
• Implementation branch: feature/zookeeper-removal-raft
• User docs: website/docs/install-deploy/deploying-with-raft.md, website/docs/install-deploy/migrate-zk-to-raft.md

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!