Exchange storage key DKG artifacts over Iroh

[huitseeker]

Stacked on #2433.

This replaces manual public file exchange with one shared Iroh document. One operator starts dkg board from the genesis file, threshold, and epoch, then sends the private ticket to each validator through the trusted bootstrap channel. Each operator runs dkg run with that ticket, the genesis file, its signing key, a private work directory, and an output directory.

The runners exchange signed public artifacts, resume after restarts, and write local storage key bundles. Operators compare the shared public output, keep each secret share private, and start validators only after every runner succeeds. An opt-in Compose check runs the full three-validator flow.

Changelog

[[entry]]
scope       = "validator"
impact      = "added"
description = "Adds an Iroh bulletin board for exchanging storage key DKG artifacts."

[Mirko-von-Leipzig]

I think broadly speaking this would benefit from an explanation/goal and having the code model that.

[Mirko-von-Leipzig]

I'll need to dig into the actual DKG ceremony in more detail first; I'd also like to know what alternatives there are to a central board, or how this would operate for a live network key rotation. I imagine we would want something like this to share txs for new incoming validators etc.

This is beyond the purvey of this PR, but I need to just wrap my head around things.

[sergerad]

create_with_network writes document-id.hex, board-format, and upload-secrets/ as separate steps. A crash between any two leaves a directory that fails on restart with the misleading "predates participant-scoped uploads" error and must be discarded.

Consider grouping all three into a single board-meta/ directory written via the existing publish_directory helper (temp dir + rename), after creating the Iroh document. The rename becomes the single commit point: if board-meta/ exists, its contents are guaranteed complete; if not, re-initialize (an orphaned namespace in the docs store from a pre-rename crash is harmless since its ID never left the process).

[huitseeker]

Done.

[sergerad]

The board persists its identity as separate hand-encoded files (board-format, document-id.hex, upload-secrets/participant-N.hex), each with its own read/decode/validate path (require_current_board_format, decode_fixed_hex, load_or_create_upload_secrets). This module already uses serde+TOML for registration.toml — the same pattern would collapse all of this into one struct:

#[derive(Serialize, Deserialize)]
struct BoardState {
    format: u32,                      // replaces the board-format marker file
    document_id: NamespaceId,         // iroh types implement serde directly
    #[serde(with = "hex::serde")]
    upload_secrets: Vec<[u8; 32]>,
}

written as a single board-state.toml (0600, via tempfile::NamedTempFile::persist). That deletes the custom format-check and hex plumbing, and as a bonus fixes the non-atomic initialization: one file, one atomic rename, so a crash can no longer leave a half-initialized directory that errors misleadingly on restart.

The BoardTicket string format is worth keeping hand-rolled — it's a user-facing copy-paste credential, and DocTicket inside it is already iroh's self-encoding ticket type.

[huitseeker]

Thanks. I kept the existing file representation and grouped it under an atomically published board-meta/ directory. That fixes partial initialization without adding another serialization schema, while retaining the specific validation for each field.

[sergerad]

SecretKey implements FromStr (hex), so the read path doesn't need decode_fixed_hex. It intentionally has no Display, so the write side keeps the explicit hex::encode(secret.to_bytes()):

if path.exists() {
    let text = fs_err::read_to_string(&path)?;
    return text.trim().parse::<SecretKey>().context("invalid Iroh endpoint secret");
}
let secret = SecretKey::generate();
// tempfile + persist() instead of write_new_file: atomic, so a crash
// mid-write can't leave a partial secret file that fails on every restart.
let mut file = tempfile::NamedTempFile::new_in(data_directory)?;
// set 0o600 before writing
file.write_all(hex::encode(secret.to_bytes()).as_bytes())?;
file.persist(&path)?;
Ok(secret)

[huitseeker]

Done

[sergerad]

Could we implement iroh_tickets::Ticket for BoardTicket instead of hand-parsing? https://docs.rs/iroh-tickets/latest/iroh_tickets/trait.Ticket.html

The colon-split/hex FromStr reimplements what the Ticket trait (from iroh-tickets, already a transitive dep) provides: derive serde on the struct, set KIND = "miden-storage-key-dkg-board", and get the postcard+base32 prefixed token with Display/FromStr for free — the same way DocTicket itself is encoded. The read-only-capability and nonzero-participant checks move into the decode_bytes impl.

[huitseeker]

Done

[sergerad]

nit: these could be parsed earlier in the stack right?

[huitseeker]

Yes. run_validator now parses the ticket immediately after reading the nonempty file.

[sergerad]

This is manually reimplementing what compose already does — supervising processes, propagating failures, tearing down. Should we split this into services using the same anchor pattern the file already uses for x-validator:

x-dkg-runner: &dkg-runner
  profiles: ["storage-key-dkg"]
  image: ${MIDEN_VALIDATOR_IMAGE:-miden-validator}
  volumes: [node-data:/data]
  entrypoint: ["/bin/sh", "-c"]
  command:
    - |
      until [ -s "/data/storage-key-dkg-check/board-tickets/participant-$${PARTICIPANT}.ticket" ]; do sleep 1; done
      exec miden-validator dkg run --board-file ... --signing-key.hex "$${SIGNING_KEY}" ...

services:
  storage-key-dkg-board:    # runs `dkg board`
  storage-key-dkg-runner-1: { <<: *dkg-runner, environment: { PARTICIPANT: 1, SIGNING_KEY: "0101…" } }
  storage-key-dkg-runner-2: { <<: *dkg-runner, environment: { PARTICIPANT: 2, SIGNING_KEY: "0303…" } }
  storage-key-dkg-runner-3: { <<: *dkg-runner, environment: { PARTICIPANT: 3, SIGNING_KEY: "0404…" } }
  storage-key-dkg-check:    # depends_on the 3 runners with service_completed_successfully;
                            # body shrinks to just the cmp assertions

[sergerad]

Also we need to consider whether we should update run-node.sh to allow for a setup using dkg/iroh.

[huitseeker]

The DKG process is actually synchronous and requires participants to be online and active — it's a hard limitation. The storage-key-dkg-check runs through docker compose run --rm. It must return one result and leave no helper services behind. Keeping the board and runners in that job does both.

run-node.sh starts long-running node services. Due to that synchronous limitation, I've left this alone: the DKG is a one-time ceremony and it's arguable whether it should run on each node start (e.g. it won't work for restarts, see above). If local setup needs one entry point, I'd add a run-dkg.sh command that finishes before run-node.sh starts the node, in a followup.

[Mirko-von-Leipzig]

I"m going to move this to draft until we decide if we need a p2p solution for this. While it would be nice to have, this is a lot of code to add and we may not need it - we'll drive manual for a while and see if this is required.