docs: template conformance, sanctioned shapes, exported-symbol coverage (refs #64)

docs/README.md

@@ -50,6 +50,32 @@ This directory holds DART's **official documentation** (tracked in git). It targ
 7. **Testing** — the test list with each test's intent; coverage number; `go vet` / `-race` results; how to run; sources of any golden / cross-validated values.
 8. **Limitations & TODO**.
 
+### Sanctioned document shapes
+
+The 8-section template above is the default **package** shape. Three shapes
+are sanctioned; every document's shape is assigned here (unlisted documents
+default to `package`). The machine-readable block is the enforcement source
+for `scripts/check-docs.sh` — keep it in sync with the prose.
+
+<!-- DOC-SHAPES
+package: Overview; Concepts; Public API; Invariants & Guarantees; Concurrency & Call Permissions; Stability Contract; Testing; Limitations & TODO
+cmd: Overview; Usage; Behavior & guarantees (with Concurrency & lifecycle and Stability Contract as subsections); Testing; Limitations & TODO
+multi: Overview; per-package API sections; Concurrency & Call Permissions; Stability Contract; Testing; Limitations & TODO
+assignments: docs/dart.md=cmd; docs/observability.md=multi
+heading-aliases: Concepts <=> Wire form; Determinism / Stability Contract <=> Stability Contract
+-->
+
+- **package** (default): the 8 required sections in §"Required sections".
+- **cmd** (assigned: `docs/dart.md`): a binary document — `Usage` replaces
+  the API section; Concurrency and Stability may live as subsections of
+  "Behavior & guarantees".
+- **multi** (assigned: `docs/observability.md`): one document covering
+  several packages uses per-package API sections instead of unified
+  Concepts/Public-API/Invariants sections.
+- **Heading aliases**: the Concepts section may be titled by its domain
+  content (e.g. peer.md's "Wire form"); the Stability Contract may be titled
+  "Determinism / Stability Contract".
+
 ### What "Testing" must record
 
 "Testing" is not a single line saying "tests exist". It must let the reader judge **trustworthiness**: list the specific property each test guards, the coverage number, and whether `vet`/`-race` pass, plus reproduction commands. If a function is left uncovered, state why explicitly.

docs/chunk.md

@@ -46,6 +46,10 @@ func (c Config) BlocksPerChunk() int64
 Constants `MiB` and `GiB` are provided for convenience. All grid methods below
 assume a valid Config.
 
+`ErrInvalidConfig` is the sentinel `Validate` returns for a malformed `Config`
+(`BlockSize>0`, `ChunkSize>0`, `ChunkSize%BlockSize==0` required); wrap-check
+with `errors.Is`.
+
 ### 3.2 Grid math
 
 ```go
@@ -111,6 +115,11 @@ stripped (a URL can smuggle one in via a percent-encoded `%1F` in the path).
 Note that cross-field collisions are the only case: within one namespace the
 fixed 8-byte chunk-index suffix forces equal objectIDs anyway.
 
+`UnitSep` (0x1F) is the field separator `ChunkKey` mixes between fields; it is
+why `namespace` is rejected at engine construction when it contains 0x1F and
+why derived object identities have it stripped (`stripUnitSep`): the separator
+must never appear inside a field, or serialization would not be injective.
+
 ### 3.4.1 `func IsDigest(s string) bool`
 
 The shared digest recognizer (`<algorithm>:<hex>`, lowercase alnum algorithm,

docs/cluster.md

@@ -155,6 +155,18 @@ address — see §3.2). Only the peer itself can say what its ID is. Exchanging
 rosters also means one reachable neighbour is enough to find the whole cluster,
 which is what makes a truncated DNS answer or a partial seed list survivable.
 
+### 3.9.1 `DynamicProvider` methods
+
+| Method | Contract |
+|---|---|
+| `Self() Member` | the configured self member (constant after construction). |
+| `Current() *View` / `Subscribe() (<-chan *View, func())` | the `Provider` interface: latest view / view-change feed (unsubscribe via the returned func). |
+| `Run(ctx)` | the background loop: periodic `Refresh` on `RefreshInterval` until `ctx` is canceled. Start it before serving traffic so the first view is in place. |
+| `Refresh(ctx) *View` | one synchronous gather + publish cycle; also callable on demand. |
+| `Learn(members ...Member)` | hearsay ingest: invalid IDs are rejected (§3.3.1) and reported via `OnError`; self entries ignored; valid members are learned immediately. Safe for concurrent use. |
+| `LearnPeer(id, addr string)` | records an inbound contact from `id` at `addr` — refreshes that member's liveness clock (only *direct* contact does) and adopts the newest address. |
+| `Roster() []Member` | the full member list we answer roster fetches with (self included). |
+
 ### Adding and forgetting are not symmetric
 
 - **Adding** happens immediately, on hearsay. Being wrong costs a request sent to a

docs/dart.md

@@ -149,6 +149,28 @@ but immediate and readiness-aware (see [k8s.md](./k8s.md)). `-discover`'s help
 text lists exactly what the running binary was linked with.
 
 
+### 3.7 Concurrency & lifecycle
+
+All servers (client, peer, admin) run concurrently; every write to the
+operator's `out` stream is serialized through a locked writer. Handler
+lifetime follows the node-level admission-gate contract — nothing is closed
+under a live handler (docs/node.md §3.1, ADR-0003). The binary's version is
+stamped at build time (`-ldflags "-X main.version=..."`) and reported by
+`-version` and on startup.
+
+### 3.8 Stability contract
+
+- The flag set, defaults, and their validation rules are the operator-facing
+  contract; changing a default is a breaking change for existing deployments
+  and must be called out in the changelog.
+- Wire-visible behavior (range semantics, the registry-mirror path set, peer
+  headers) is governed by the Stability Contract sections of the engine,
+  registry, and peer package documents — this binary adds no wire rules of
+  its own beyond the wiring documented here.
+- The discovery-scheme set is a property of the binary (`dns`+`static` for
+  `dart`, plus `k8s` for `dart-k8s`); embedding applications choose it via
+  `node.Run`'s scheme registration.
+
 ## 4. Testing
 
 - **Results**: `go vet` clean; `go test` all pass; `go test -race` clean.

docs/engine.md

@@ -251,7 +251,9 @@ Together these bound abrupt node death: the dial timeout makes the first affecte
 read fail in ~1 s, a dial failure opens the circuit on that single observation, and
 every later read skips the departed peer entirely.
 
-Metrics: `dart_hedge_total{event=fired|primary_won|backup_won}` and
+`NewMetrics(r *metrics.Registry) *Metrics` allocates and registers the
+engine's metric set; `Engine` accepts it via `Options.Metrics` (a nil one
+disables engine metrics). Metrics: `dart_hedge_total{event=fired|primary_won|backup_won}` and
 `dart_peer_failover_total`. Comparing `backup_won` against `fired` shows whether
 hedging is paying for itself; `dart_peer_failover_total` rising marks peers
 actually going away. `primary_won`/`backup_won` are recorded **only when a hedge

docs/fetch.md

@@ -91,6 +91,11 @@ and close; the fetcher's `Header` is applied first, then the per-request
 is proxying verbatim. This backs the engine's passthrough fallback for
 Range-ignoring origins. Time complexity is O(1); no body bytes are buffered.
 
+`DefaultMaxFlight` bounds how long a shared flight may run before a later call
+starts a new one (the fix for issue #4's permanently-poisoned cache key: a
+stalled flight expires instead of blocking the key forever). Joiners of a
+flight wait inline, one worker goroutine per flight (#60).
+
 ### 3.4 `func FetchBlock(ctx, f Fetcher, url string, blockSize, blockIndex, size int64) (Range, error)`
 
 Fetches one block: `start = blockIndex*blockSize`, `end = start+blockSize-1`.

docs/observability.md

@@ -36,6 +36,7 @@ err := r.Render(w) // Prometheus text format
 | `Gauge` | `Set(float64)`, `Value() float64` | atomic float bits |
 | `Histogram` | `Observe(float64)` | cumulative `le` buckets + `_sum`/`_count`; bounds sorted/de-duplicated; `+Inf` implicit |
 | `Registry` | `NewCounter/NewGauge/NewHistogram`, `NewGaugeFunc/NewCounterFunc`, `Render(io.Writer)` | `HELP`/`TYPE` emitted once per metric name |
+| `ErrInvalidName` | `Error() string` | returned by the `New*` constructors when a metric or label name is not a valid Prometheus name (`[a-zA-Z_:][a-zA-Z0-9_:]*` / label `[a-zA-Z_][a-zA-Z0-9_]*`) |
 
 `NewGaugeFunc`/`NewCounterFunc` register a metric whose value is **sampled at
 scrape time** by a callback. That is how state owned elsewhere — cache occupancy,

docs/peer.md

@@ -81,7 +81,9 @@ type Server struct {
 }
 ```
 
-An `http.Handler` for the wire form above.
+An `http.Handler` for the wire form above: `ServeHTTP` dispatches
+`/peer/v1/block` (and rejects invalid `X-DART-Hop` values, §2); unknown paths
+get 404. Safe for concurrent use, as any handler must be.
 
 ### 3.3 `type Client` / `func NewClient() *Client`
 
@@ -161,7 +163,7 @@ func (b *Breaker) Allow(addr string) bool
 func (b *Breaker) RecordSuccess(addr string)
 func (b *Breaker) RecordFailure(addr string)      // soft: spends one unit of budget
 func (b *Breaker) RecordHardFailure(addr string)  // definitive: opens at once
-func (b *Breaker) State(addr string) BreakerState   // closed | open | half-open
+func (b *Breaker) State(addr string) BreakerState   // closed | open | half-open (String() prints those words)
 func (b *Breaker) Healthy(addr string) bool         // usable now (no probe reserved)
 func (b *Breaker) OpenCount() int
 var ErrCircuitOpen = errors.New("peer: circuit open")
@@ -269,6 +271,8 @@ const HeaderPeerAddr = "X-DART-Peer-Addr"
 type Roster struct { Epoch string; Members []RosterMember }
 type RosterMember struct { ID, Addr string; Weight float64 }
 type RosterServer struct { NodeID string; Src func() Roster; Learn func(id, addr string) }
+// RosterServer.ServeHTTP answers GET RosterPath with Src() as JSON, and feeds
+// the fetcher's self-identification headers to Learn (hearsay).
 func (c *Client) FetchRoster(ctx, addr, selfID, selfAddr string) (Roster, string, error)
 ```
 

docs/registry.md

@@ -94,6 +94,10 @@ func BlobDigest(path string) (string, bool)
   (`/v2/<name>/blobs/uploads/...`), an empty name or digest, an extra path
   segment after the digest, a non-lowercase algorithm, or a leading/trailing
   separator in the algorithm.
+- `Mirror.ServeHTTP` is the single dispatch point: blob paths go to the engine
+  (when the engine declines — e.g. the origin cannot serve ranges — it answers
+  via a direct passthrough proxy to the upstream), everything else passes
+  through. It is safe for concurrent use, as any `http.Handler` must be.
 - Blob responses carry `Docker-Content-Digest` (echoed from the path, which *is*
   the digest) and `Content-Type: application/octet-stream`.
 - Pass-through uses `httputil.ReverseProxy` with `SetXForwarded`, and rewrites
@@ -130,6 +134,9 @@ rather than plain Basic:
 ```go
 func LoadCredentials(path string) (map[string]Credential, error)
 func NewAuthTransport(base http.RoundTripper, creds map[string]Credential) *AuthTransport
+// AuthTransport.RoundTrip attaches/caches/exchanges the bearer token per
+// request (singleflight per (host, scope), conditional drop-on-rejection)
+// and otherwise delegates to base. Safe for concurrent use.
 ```
 
 **Why a RoundTripper rather than threading a credential through the read path.**
@@ -227,7 +234,39 @@ automatically if the mirror is unavailable, so a DART outage degrades to a direc
 pull rather than a failed one. Add `-peers`/`-self-id` to enable P2P between
 nodes (see docs/dart.md).
 
-## 7. Testing
+## 7. Concurrency & Call Permissions
+
+- `Mirror.ServeHTTP` is safe for concurrent use, as any `http.Handler` must
+  be; it holds no per-request mutable state of its own — blob serving defers
+  to the engine's own concurrency contract (docs/engine.md §5).
+- `AuthTransport.RoundTrip` is safe for concurrent use: the token cache is
+  mutex-guarded; token exchanges are singleflight-shared per (registry host,
+  path-derived scope) — the cache and inflight maps key on `tokenKey(host,
+  scope)`, the challenge realm is trusted as delivered (§5);
+  a stored cache entry is never mutated, and drop-on-rejection is conditional
+  on the rejected value (`dropTokenIf`), so a concurrent fresh store always
+  survives.
+- The pass-through reverse proxy is stateless between requests.
+- Call order: `New` first; there is no `Close` — the mirror owns no resources
+  beyond its transport.
+
+## 8. Stability Contract
+
+- **Breaking**: widening or narrowing the path set `BlobDigest` accepts as
+  cacheable. The classifier must stay exactly aligned with `chunk.IsDigest`
+  (§3.4.1 of docs/chunk.md); changing either side without the other splits
+  cache identity between the mirror and content-addressed clients.
+- **Breaking**: the blob response contract — `Docker-Content-Digest` echoed
+  from the path, `Content-Type: application/octet-stream`, and range
+  semantics inherited from the engine (200/206/416, Content-Length framed).
+- **Contract (assumption-backed)**: the trust model of §5 — the trusted
+  read-only origin (A1) and the realm-as-delivered rule — is part of this
+  stability contract. Weakening it is a T2/T3-triggered change requiring an
+  ADR (docs/adr/README.md).
+- Pass-through behavior (Host rewrite, X-Forwarded-* via `SetXForwarded`) is
+  observable to upstreams and treated as stable.
+
+## 9. Testing
 
 - **Results**: `go vet` clean; `go test` all pass; `go test -race` clean.
 - **Coverage**: **86.4%** of statements.
@@ -296,7 +335,7 @@ In `cmd/dart`:
 | `TestBuildRegistryMirror` | `-registry` mounts the mirror; a bad upstream fails the build without leaking the cache-dir lock |
 | **`TestBuildRegistryAuth`** | **a token-demanding private upstream is served end-to-end; bad/missing credential files fail the build without leaking the lock** |
 
-## 8. Limitations & TODO
+## 10. Limitations & TODO
 
 - **No per-request credential**: authentication is per upstream, not per client.
   A client-supplied token is forwarded on the **pass-through** path but not used

docs/store.md

@@ -108,7 +108,7 @@ no-op stub.
 ### 3.4 Two budgets: `Tiered` (owned / borrowed) with TinyLFU admission
 
 ```go
-type Class uint8            // Owned | Borrowed
+type Class uint8            // Owned | Borrowed; String() prints "owned"/"borrowed" for logs/metrics
 type ClassStore interface {
     Store
     PutClass(k BlockKey, data []byte, c Class) (admitted bool, err error)

docs/tracker.md

@@ -13,7 +13,7 @@ Building the tree over **all** Ready members means a node's parent may be a
 member that is not reading the object at all, so it has to fetch-on-behalf just
 to pass bytes along. Building it over the **readers** makes every parent a node
 that actually wants the data (it either holds it or is already fetching it),
-which is the design's active-reader-set optimization (§4.2, §6.2).
+which is the design's active-reader-set optimization (docs/hashring.md §2/§3).
 
 Three properties make this safe and cheap:
 
@@ -90,6 +90,11 @@ POST /tracker/v1/join    {"file","node","ttlMs"} -> {"epochS","readers","ttlMs"}
 POST /tracker/v1/leave   {"file","node"}         -> 204
 ```
 
+The join body is `JoinRequest{File, Node, TTLMs}`: `File` is the opaque object
+key, `Node` the reader's stable cluster ID, and `TTLMs` the requested lease in
+milliseconds — 0 means the tracker default, and client-supplied values are
+clamped to the configured range (see §2; duration-overflow guarded).
+
 ```go
 mux := (&tracker.Server{R: reg}).Handler()   // serve
 c := tracker.NewClient()                     // 2s timeout: control plane fails fast
@@ -100,7 +105,29 @@ err = c.Leave(ctx, addr, file, node)
 Non-POST is `405`; malformed JSON or missing `file`/`node` is `400`. Bodies are
 capped at 64 KiB.
 
-## 4. Engine integration
+## 4. Invariants & Guarantees
+
+- **Tick freeze**: the published reader set changes only at tick boundaries
+  (recomputation is lazy — on activity, at most once per tick; there is no
+  background goroutine). Between ticks the topology and `epochS` are stable,
+  so TCP connections to readers do not churn.
+- **`epochS` bumps only when the frozen set actually changes** — joins that
+  refresh an existing lease, and leaves of absent readers, never bump it.
+- **Deterministic wire form**: the frozen set is sorted by node ID; every
+  observer of the same registry state reads the same epoch and reader list.
+- **Membership by liveness only**: a reader is in the set iff its lease is
+  unexpired at freeze time. `Leave` deletes the lease immediately; for a file
+  with remaining readers the published set follows at the next freeze, but
+  removing the **last** lease deletes the whole file entry, so `Readers` then
+  reports `(nil, 0)` without waiting for a tick (the tick-freeze guarantee
+  covers joins and lease expiries, not this deletion edge). An idle entry is
+  forgotten after `IdleGrace`.
+- **No collision with placement keys**: the file key uses the sentinel chunk
+  index -1, which no real chunk's `ChunkKey` input can carry.
+- **Client-supplied TTLs are clamped** to the configured lease range;
+  arithmetic is duration-overflow guarded.
+
+## 5. Engine integration
 
 `engine.Options.TrackerRegistry` (this node's local tracker) and
 `TrackerClient` (to reach remote trackers) enable the feature; leaving both nil
@@ -115,14 +142,14 @@ keeps all-member routing. The engine then:
 unreachable, or the reader set has fewer than two usable members, routing falls
 back to all Ready members.
 
-## 5. Concurrency & Call Permissions
+## 6. Concurrency & Call Permissions
 
 - `Registry` is safe for concurrent use (single mutex); `Readers`/`Join` return
   copies, never internal slices. Verified with `-race`.
 - `Client` is safe for concurrent use.
 - The engine's reader-set cache is mutex-guarded and holds only IDs.
 
-## 6. Stability Contract
+## 7. Stability Contract
 
 - The JSON shapes and paths (`/tracker/v1/join`, `/leave`) are the tracker wire
   protocol; changing them is a protocol change.
@@ -131,7 +158,7 @@ back to all Ready members.
 - `epochS` must change only when the set changes (readers rely on it to detect
   topology changes cheaply).
 
-## 7. Testing
+## 8. Testing
 
 - **Results**: `go vet` clean; `go test` all pass; `go test -race` clean.
 - **Coverage**: **89.5%** of statements.
@@ -172,7 +199,7 @@ Engine-side (in `internal/engine`):
 | `TestTreeNodesFallsBackToAllMembers` | no tracker, or a single reader, falls back to all-member routing |
 | `TestReaderSetTreeEndToEnd` | 3 nodes sharing one tracker all read correct bytes |
 
-## 8. Limitations & TODO
+## 9. Limitations & TODO
 
 - **Tracker liveness**: a tracker that dies is replaced by HRW on the next
   membership change, and `S` rebuilds from renewals — but in-flight reads during