Detect a node that dies when nothing else is left to report it

[bburda]

Pull Request

Summary

A node dies and nothing reports it. The node cannot report it itself, and the other detectors
cannot see it: qos_mismatch needs both sides present to compare profiles, orphan looks at a
topic whose publisher took the topic with it, and lifecycle_expectation only watches nodes the
operator listed as required-active.

This adds node_death, which raises GRAPH_NODE_DISAPPEARED for a node that was alive and armed
and stopped being alive, together with the suppression framework the umbrella issue asks to be
opt-in and explicit.

Alive is not membership in the entity snapshot. In runtime discovery a dead node leaves the
snapshot, so the two look the same; in manifest and hybrid discovery the manifest keeps the App
and only clears its online flag, so a detector counting membership would make a manifest node
immortal. Tracking is keyed on the stable fully qualified name rather than the app id, because an
id is recomputed each sweep and gains a namespace prefix once a bare-name collision exists, so a
live node held under its old id would be reported as gone.

The grace period is wall clock, not ticks. The entity cache is rebuilt on a debounced graph
event, so every tick between two refreshes sees the same snapshot and one absent cache generation
is counted again on each of them. At a fast tick a two-tick tolerance is shorter than one refresh
cycle, so the window has a floor in milliseconds and says so when it raises the configured value.

Suppression is opt-in. Nothing is suppressed unless suppress names it. A configured
allowlist that suppress does not name has no effect and warns that it has none. Two
suppressors ship: an operator allowlist, and clean lifecycle shutdown. A suppressor also declares
whether its veto is durable, because only a durable veto may reclaim tracker bookkeeping - a veto
that can lift later would otherwise lose a real fault for good once the condition ends.

Also in this change: the absence boundary

GRAPH_NODE_INACTIVE says a required node is not active. GRAPH_NODE_DISAPPEARED says a node is
gone. Both can be true of one node, they have different repairs, and both standing at once is
correct - this change does not remove that.

What changed is narrower. Sustained absence used to mature a violation that had not yet been
reported, so a node observed non-active for a few ticks and then killed acquired an inactive fault
built entirely from evidence gathered after it could no longer be observed. That rule existed
because no presence detector existed to own departures. Now one does, so absence continues a
violation that has already matured and no longer creates one that has not.

With one exception, which is the part worth reviewing closely: the reliability gate refuses to arm
a managed node that never reads active, and node_death only tracks armed nodes. For such a
node the handover has no receiver, so absence may still mature its violation - otherwise a
required node that comes up unconfigured and dies is reported by nothing at all. The split is
keyed on whether the node was ever armed, latched rather than recomputed, because a managed node
legitimately reads not-armed again after an ordinary deactivate while node_death is still
tracking it.

Known limitation

A second node's death, while the fault from the first is still outstanding, is added to the
fault's description but arrives as an update to an already-confirmed record: no state transition,
no freeze frame and no recording of its own. Acknowledging the fault between the two deaths avoids
this, because a report arriving after an acknowledgement is counted as a new occurrence and gets
its own evidence.

Two ways of forcing it from the detector were measured and rejected rather than skipped. Sending a
clear and a raise back to back does nothing: a single opposite-direction report cannot move the
fault manager's hysteresis latch, so the raise lands on the already-confirmed branch. Calling the
clear service instead is worse than not doing it, because it deletes the per-topic readings
captured for that fault and by default clears every symptom the correlation engine attributes to
it. A correct fix belongs in the fault manager, as an operation that re-confirms a record without
clearing it. This is documented in the package README and design doc.

Also in this change: presence ownership rests on knowledge

Two detectors divide one job, and until now the division was decided by a fact the plugin did not
have. GRAPH_NODE_DISAPPEARED is raised for a node the reliability gate has armed; the gate arms a
managed node whose lifecycle state reads active, and it also arms one whose state has never been
measured, because it deliberately treats an unknown state as "do not gate" so that a node nobody
can read is not silenced across every other detector. Those two answers are not the same fact, and
the second was being latched permanently into the flag lifecycle_expectation consults before it
lets absence mature a violation. A required node that never configured could therefore be reported
by the presence detector and, by the same latch, not reported by the only detector that could still
have described what was wrong with it.

The division now asks a second, stricter question, and the gate itself is unchanged. Ownership is
earned when the node is not managed at all or its label reads active; it is withheld while the
state is merely unread and the watcher still has re-seed attempts left; and once those attempts are
spent it is provisional rather than earned - the node is owned, because otherwise nothing would
ever report its departure, but that grant yields the moment a real label finally arrives. A grant
made on knowledge latches, because a later deactivate does not hand the node back. A grant made on
ignorance does not, because the reason for it stops being true the instant somebody else can report
the node.

One consequence is worth stating rather than leaving to be found: a node that departs during its own
warmup is reported by nobody. That is the bringup-quiesce trade-off behaving as designed, it
predates this change, and it is written down where the boundary is described.

Also in this change: the sanitizer test budget

The sanitizer jobs walk the workspace one package at a time under a single 45-minute budget for
the whole step. This package's end-to-end suite runs 24 minutes of that under instrumentation, up
from 10:38 on main, so the step stopped fitting: the TSan run was cut inside the package that
follows this one, and ASan finished the same run with 20 seconds to spare. No single scenario is
responsible - the slowest is 106 s and the rest is a 30-50 s tail across 73 targets, because these
tests wait on grace windows, respawn delays and restart loops rather than compute.

quality.yml now tests this package in a job of its own, once per sanitizer, and ci.yml does the
same over humble and lyrical, where the same cap applies to the step rather than to the job: lyrical
was cut inside this package and humble finished with three minutes to spare. The sweeps skip it. The job builds only the chain up to the package, which still includes the
integration-test package the scenarios launch from, and restores the matching sweep's ccache
instead of saving a second copy of the same objects. Instrumentation, the timeout multiplier and
the sanitizer options are unchanged from the sweep it left, and the job is not path-filtered,
because the plugin drives the gateway, the fault manager and discovery, so the changes most likely
to break it are outside its own tree.

The scenarios' own wall-clock budgets now scale with MEDKIT_TEST_TIME_SCALE, which the sanitizer
jobs already set to the factor they apply to every declared CTest timeout. A budget asserted inside
a test is invisible to that rewrite, so an instrumented graph that is slow to forget a departed node
blows one and the failure reads as a detector that never reported. Only give-up bounds scale: poll
intervals, enforced respawn delays and the sustained-observation windows keep their values, and the
scale is 1.0 whenever the variable is unset, so the normal jobs are unchanged.

---

Issue

• closes A node dies and nothing is left to report it #624

---

Type

• Bug fix
• New feature or tests
• Breaking change
• Documentation only

---

Testing

The tests were written and merged before the detector, so every scenario was watched failing for
the right reason first, and the suite is what the implementation had to satisfy rather than a
description of it.

73 ctest targets, all green, of which 42 are end-to-end against a real gateway, a real fault
manager and a real ROS graph. 23 of those scenarios are new here.

The end-to-end set covers: a process that exits, its return and the clearing that follows, a
managed node that deactivates but keeps running, a manifest node that never came online, CLI-
convention node names, two nodes sharing a bare name in different namespaces, a fast tick against
a stale cache generation, a node restarting repeatedly across a window, a gateway restart with a
death outstanding, each suppressor including the inert-allowlist warning, pruning that must not
heal a fault that is still true, and each side of the absence boundary.

Every scenario that asserts an ABSENCE gates on the plugin being armed first and uses a helper
that fails when the fault surface is unreachable, so it cannot pass against a stack where the
plugin never loaded. Two such helpers are added here, each with a test that points it at a surface
which dies mid-window and fails if the helper still passes.

The whole suite was also run with the plugin path pointed at a file that does not exist: every
scenario fails, naming the load failure.

Reviewers can check the boundary with ctest -R "node_death_boundary_e2e", the suppressors with
ctest -R "node_death_suppression_e2e", and the tracker's bound with
ctest -R "test_node_liveness_tracker".

No route is added or changed. The only public surface difference is additive content inside the
existing x-medkit-watchdog payload, which gains a node_death block.

---

Checklist

• Breaking changes are clearly described (and announced in docs / changelog if needed)
• Tests were added or updated if needed
• Docs were updated if behavior or public API changed

[Copilot]

Pull request overview

Adds a new node_death detector to the ros2_medkit_graph_watchdog plugin to raise GRAPH_NODE_DISAPPEARED when a previously-armed node disappears and no other detector can observe/report the departure, plus an opt-in suppression framework (allowlist + clean lifecycle shutdown) and supporting tracking/retention logic.

Changes:

• Introduce node_death detector with a dedicated liveness tracker keyed by stable FQN, wall-clock-based grace flooring, and bounded bookkeeping under identity churn.
• Add opt-in suppression framework via a Suppressor interface, including allowlist suppression and lifecycle clean-shutdown suppression, plus pruning semantics tied to “durable” suppressors.
• Expand unit/integration/e2e test coverage and strengthen e2e harness window assertions (silence/persistence/description-scoping).

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_suppressor.cpp	Adds unit tests for the Suppressor contract and apply_suppressors() behavior.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_node_liveness_tracker.cpp	Adds extensive unit tests for NodeLivenessTracker state machine, pruning, caps, and description ordering.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_lifecycle_shutdown_suppressor.cpp	Adds tests for lifecycle clean-shutdown suppression behavior against a real ReliabilityGate.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_lifecycle_expectation_tracker.cpp	Updates lifecycle expectation tracker tests to reflect the new “absence boundary” and armed-vs-never-armed split.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_lifecycle_expectation_integration.cpp	Updates integration scenarios and expectations around absence handling and boundary behavior with the new presence detector.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_graph_watchdog_plugin.cpp	Adds regression coverage for node_death retention/config fallback behavior and oversized integer validation.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_allowlist_suppressor.cpp	Adds unit tests for allowlist suppression matching forms and durability.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/e2e/harness.py	Adds stronger e2e assertion helpers for sustained presence and scoped “never names” checks; extends flaky server harness.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/src/graph_watchdog_plugin.cpp	Adds shared node_death prune_grace resolution and aligns retention math with detector-side validation/flooring.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/src/detectors/node_death_detector.cpp	Introduces the new node_death detector implementation (tracking, suppression, pruning, reporting/clearing guard).
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/src/detectors/lifecycle_expectation_detector.cpp	Wires reliability-gate arming into lifecycle expectation matches to support the absence-boundary split.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/suppressor.hpp	Adds the suppression interface and apply_suppressors() helper.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/node_liveness_tracker.hpp	Adds NodeLivenessTracker for armed-node disappearance tracking with cap/collapse logic and freshness ordering.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/lifecycle_shutdown_suppressor.hpp	Adds durable suppressor for clean lifecycle shutdown departures.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/lifecycle_expectation_tracker.hpp	Extends tracker model and absence behavior to incorporate “ever armed” logic and prevent absence-maturing where node_death should own departures.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/graph_watchdog_plugin.hpp	Exposes a test seam for departed-retention tick computation.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/detector.hpp	Changes DetectorContext::{raise_fault,clear_fault} to return bool to signal whether an async request was actually enqueued.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/detector_config_keys.hpp	Adds node_death grace ceiling + wall-clock miss_grace floor computation helpers.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/allowlist_suppressor.hpp	Adds allowlist suppressor implementation and matching semantics documentation.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/aggregated_fault.hpp	Updates aggregated fault emission helpers to return the underlying send/enqueue boolean.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/CMakeLists.txt	Registers new unit/integration/e2e tests and adds node_death sources to relevant test targets.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.