Skip to content

fix(balancer): stop routing new miners to a node whose Ghost Core is dead (#778) - #796

Merged
defenwycke merged 1 commit into
mainfrom
fix/778-balancer-sees-core-health
Aug 31, 2026
Merged

fix(balancer): stop routing new miners to a node whose Ghost Core is dead (#778)#796
defenwycke merged 1 commit into
mainfrom
fix/778-balancer-sees-core-health

Conversation

@defenwycke

Copy link
Copy Markdown
Contributor

/health has reported healthy: false correctly since #771, but nothing acted on it. The
balancer's data model had no health field, the peer list was filtered only on mesh-gossip
recency — which is independent of Ghost Core — and load_balancer.rs contained zero
occurrences of healthy, core_reachable or health. So a node that knew its own Core was
dead kept advertising itself and kept taking new miners.

The failure is inverted, not merely unhandled

A node whose Core is dead sheds miners. Fewer miners means lower utilisation. Utilisation is
exactly what the balancer optimises for — so the node becomes a MORE attractive destination the
worse it gets. vm8 sat in that state for 2h15m on 2026-08-24 while ghostd crash-looped 260
times, gossiping peer_count: 7 throughout.

What changed

  • /api/internal/pool-nodes now publishes this_node.core_healthy, read from the same probe
    /health uses (core_health_reachable, split out of get_health so both share one signal).
  • The balancer diverts to an eligible peer when its own Core is unreachable, in BOTH routing
    paths — should_proxy (utilisation) and fallback_pick_by_count (early boot, no capacity
    reported yet). ⚠ The first draft of this patched only the fallback, which is the path that
    almost never runs in production.

The two defaults are deliberately OPPOSITE

  • ghost-pool treats an unwired probe as UNHEALTHY, matching get_health exactly: "If
    nothing wired a probe, this node cannot demonstrate it can reach Ghost Core, and saying
    healthy on that basis is what hid vm7's outage."
  • The translator treats an ABSENT field as healthy. Absence there means the colocated
    ghost-pool predates the field, not that Core is down — assuming the worst would make every
    translator divert all of its miners the moment it out-ran the pool binary in a rolling deploy.

Degraded beats unreachable

The check sits AFTER candidate selection, so it can only divert to a peer that already passed
the eligibility filter. An unhealthy node with no eligible peer keeps serving locally rather
than routing to nothing — the "what if every node is unhealthy" hazard the issue flags.

Tests

Three, and the pair is what gives them teeth: the local node is IDLE (0 miners) against a
busier peer (5), so every utilisation rule says "keep it".

an_unreachable_core_diverts_regardless_of_utilisation   -> diverts
a_healthy_idle_node_keeps_its_miners                    -> does NOT divert
an_unreachable_core_with_no_peer_still_serves_locally   -> serves locally

⚠ Verified they FAIL when the divert is neutered (1 failed / 64 passed), and pass when
restored (65/65). Without the healthy-case twin, the first test would pass on a balancer that
diverted unconditionally.

⚠ NOT addressed: peer health is still not gossiped, so this closes only the LOCAL case —
suggestion 2 of the issue. A node cannot yet exclude an unhealthy PEER, which needs a
PoolPeerInfo field and a gossip change.

Refs #778

…dead (#778)

`/health` has reported `healthy: false` correctly since #771, but nothing acted on it. The
balancer's data model had no health field, the peer list was filtered only on mesh-gossip
recency — which is independent of Ghost Core — and `load_balancer.rs` contained zero
occurrences of `healthy`, `core_reachable` or `health`. So a node that knew its own Core was
dead kept advertising itself and kept taking new miners.

## The failure is inverted, not merely unhandled

A node whose Core is dead sheds miners. Fewer miners means lower utilisation. Utilisation is
exactly what the balancer optimises for — so the node becomes a MORE attractive destination the
worse it gets. vm8 sat in that state for 2h15m on 2026-08-24 while `ghostd` crash-looped 260
times, gossiping `peer_count: 7` throughout.

## What changed

- `/api/internal/pool-nodes` now publishes `this_node.core_healthy`, read from the same probe
  `/health` uses (`core_health_reachable`, split out of `get_health` so both share one signal).
- The balancer diverts to an eligible peer when its own Core is unreachable, in BOTH routing
  paths — `should_proxy` (utilisation) and `fallback_pick_by_count` (early boot, no capacity
  reported yet). ⚠ The first draft of this patched only the fallback, which is the path that
  almost never runs in production.

## The two defaults are deliberately OPPOSITE

- **ghost-pool** treats an unwired probe as UNHEALTHY, matching `get_health` exactly: "If
  nothing wired a probe, this node cannot demonstrate it can reach Ghost Core, and saying
  `healthy` on that basis is what hid vm7's outage."
- **The translator** treats an ABSENT field as healthy. Absence there means the colocated
  ghost-pool predates the field, not that Core is down — assuming the worst would make every
  translator divert all of its miners the moment it out-ran the pool binary in a rolling deploy.

## Degraded beats unreachable

The check sits AFTER candidate selection, so it can only divert to a peer that already passed
the eligibility filter. An unhealthy node with no eligible peer keeps serving locally rather
than routing to nothing — the "what if every node is unhealthy" hazard the issue flags.

## Tests

Three, and the pair is what gives them teeth: the local node is IDLE (0 miners) against a
busier peer (5), so every utilisation rule says "keep it".

    an_unreachable_core_diverts_regardless_of_utilisation   -> diverts
    a_healthy_idle_node_keeps_its_miners                    -> does NOT divert
    an_unreachable_core_with_no_peer_still_serves_locally   -> serves locally

⚠ Verified they FAIL when the divert is neutered (1 failed / 64 passed), and pass when
restored (65/65). Without the healthy-case twin, the first test would pass on a balancer that
diverted unconditionally.

⚠ NOT addressed: peer health is still not gossiped, so this closes only the LOCAL case —
suggestion 2 of the issue. A node cannot yet exclude an unhealthy PEER, which needs a
`PoolPeerInfo` field and a gossip change.

Refs #778
@defenwycke
defenwycke merged commit 6eea341 into main Aug 31, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant