fix(balancer): stop routing new miners to a node whose Ghost Core is dead (#778) - #796
Merged
Merged
Conversation
…dead (#778) `/health` has reported `healthy: false` correctly since #771, but nothing acted on it. The balancer's data model had no health field, the peer list was filtered only on mesh-gossip recency — which is independent of Ghost Core — and `load_balancer.rs` contained zero occurrences of `healthy`, `core_reachable` or `health`. So a node that knew its own Core was dead kept advertising itself and kept taking new miners. ## The failure is inverted, not merely unhandled A node whose Core is dead sheds miners. Fewer miners means lower utilisation. Utilisation is exactly what the balancer optimises for — so the node becomes a MORE attractive destination the worse it gets. vm8 sat in that state for 2h15m on 2026-08-24 while `ghostd` crash-looped 260 times, gossiping `peer_count: 7` throughout. ## What changed - `/api/internal/pool-nodes` now publishes `this_node.core_healthy`, read from the same probe `/health` uses (`core_health_reachable`, split out of `get_health` so both share one signal). - The balancer diverts to an eligible peer when its own Core is unreachable, in BOTH routing paths — `should_proxy` (utilisation) and `fallback_pick_by_count` (early boot, no capacity reported yet). ⚠ The first draft of this patched only the fallback, which is the path that almost never runs in production. ## The two defaults are deliberately OPPOSITE - **ghost-pool** treats an unwired probe as UNHEALTHY, matching `get_health` exactly: "If nothing wired a probe, this node cannot demonstrate it can reach Ghost Core, and saying `healthy` on that basis is what hid vm7's outage." - **The translator** treats an ABSENT field as healthy. Absence there means the colocated ghost-pool predates the field, not that Core is down — assuming the worst would make every translator divert all of its miners the moment it out-ran the pool binary in a rolling deploy. ## Degraded beats unreachable The check sits AFTER candidate selection, so it can only divert to a peer that already passed the eligibility filter. An unhealthy node with no eligible peer keeps serving locally rather than routing to nothing — the "what if every node is unhealthy" hazard the issue flags. ## Tests Three, and the pair is what gives them teeth: the local node is IDLE (0 miners) against a busier peer (5), so every utilisation rule says "keep it". an_unreachable_core_diverts_regardless_of_utilisation -> diverts a_healthy_idle_node_keeps_its_miners -> does NOT divert an_unreachable_core_with_no_peer_still_serves_locally -> serves locally ⚠ Verified they FAIL when the divert is neutered (1 failed / 64 passed), and pass when restored (65/65). Without the healthy-case twin, the first test would pass on a balancer that diverted unconditionally. ⚠ NOT addressed: peer health is still not gossiped, so this closes only the LOCAL case — suggestion 2 of the issue. A node cannot yet exclude an unhealthy PEER, which needs a `PoolPeerInfo` field and a gossip change. Refs #778
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/healthhas reportedhealthy: falsecorrectly since #771, but nothing acted on it. Thebalancer's data model had no health field, the peer list was filtered only on mesh-gossip
recency — which is independent of Ghost Core — and
load_balancer.rscontained zerooccurrences of
healthy,core_reachableorhealth. So a node that knew its own Core wasdead kept advertising itself and kept taking new miners.
The failure is inverted, not merely unhandled
A node whose Core is dead sheds miners. Fewer miners means lower utilisation. Utilisation is
exactly what the balancer optimises for — so the node becomes a MORE attractive destination the
worse it gets. vm8 sat in that state for 2h15m on 2026-08-24 while
ghostdcrash-looped 260times, gossiping
peer_count: 7throughout.What changed
/api/internal/pool-nodesnow publishesthis_node.core_healthy, read from the same probe/healthuses (core_health_reachable, split out ofget_healthso both share one signal).paths —
should_proxy(utilisation) andfallback_pick_by_count(early boot, no capacityreported yet). ⚠ The first draft of this patched only the fallback, which is the path that
almost never runs in production.
The two defaults are deliberately OPPOSITE
get_healthexactly: "Ifnothing wired a probe, this node cannot demonstrate it can reach Ghost Core, and saying
healthyon that basis is what hid vm7's outage."ghost-pool predates the field, not that Core is down — assuming the worst would make every
translator divert all of its miners the moment it out-ran the pool binary in a rolling deploy.
Degraded beats unreachable
The check sits AFTER candidate selection, so it can only divert to a peer that already passed
the eligibility filter. An unhealthy node with no eligible peer keeps serving locally rather
than routing to nothing — the "what if every node is unhealthy" hazard the issue flags.
Tests
Three, and the pair is what gives them teeth: the local node is IDLE (0 miners) against a
busier peer (5), so every utilisation rule says "keep it".
⚠ Verified they FAIL when the divert is neutered (1 failed / 64 passed), and pass when
restored (65/65). Without the healthy-case twin, the first test would pass on a balancer that
diverted unconditionally.
⚠ NOT addressed: peer health is still not gossiped, so this closes only the LOCAL case —
suggestion 2 of the issue. A node cannot yet exclude an unhealthy PEER, which needs a
PoolPeerInfofield and a gossip change.Refs #778