Learning Lift (LL) replay harness: frozen vs adapted insight policy on recurring scenarios

[hyrule-engineering-loop]

Change class

app_feature

Repos touched

• hyrule-infra

Senior role reviews

• network_architect: not approved
• systems_engineer: approved
• devops_netops: approved
• security_auditor: not approved
• finops_integrity: not approved
• virtual_lab_chaos: not approved

Source-of-truth files consulted

• hyrule-infra:README.md

Validation gates run

• ['/opt/engineering-loop/.venv/bin/python3', '-c', 'import json\nfrom pathlib import Path\nfor raw in json.loads('["docs/runbooks/learning-lift-replay-harness.md", "docs/runbooks/noc-standing-grant-rollout.md", "docs/runbooks/soc-mode-promotion.md"]'):\n path = Path(raw)\n if not path.exists():\n continue\n if not path.is_file():\n raise SystemExit(f'not a file: {raw}')\n path.read_text(encoding='utf-8')\n']: passed

Expected production impact

none

Rollback plan

Discard the generated feature worktree and branch; no production state was changed.

NOC handoff

• handoff artifact: /var/lib/engineering-loop/runs/issue_network_operations_412/handoff/noc_handoff.json
• rollback trigger: operator rejection, failed gates, or failed post-deploy checks
• expected duration: none

Post-deploy checks

• review graph state
• run documented gates

Operator notes

Closes #412

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 3660312)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 412 - Partially compliant Compliant requirements: Design the scenario format: the new runbook learning-lift-replay-harness.md defines the YAML fixture format, family_id, decision_points, and the deterministic replay contract. Wire the result into the promotion runbooks: both noc-standing-grant-rollout.md and soc-mode-promotion.md are updated to reference LL thresholds. Non-compliant requirements: Implement replay in the knowledge repo evals: no code changes are made in the knowledge repo; the PR only adds documentation. Wire the result into hyrule-knowledge insights metrics: the CLI output example in the runbook shows a placeholder null; the actual metric replacement is not implemented.

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪

🏅 Score: 35

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Incomplete Implementation The PR claims to close the ticket but only adds documentation and runbook modifications. The actual replay harness implementation in the knowledge repo (evals/replay/) and the CLI metric replacement are missing. This means the runbook promotion criteria reference a metric that does not yet exist, potentially causing confusion or blocking promotions incorrectly. # Learning Lift (LL) replay harness Reference: arXiv 2605.06717 §4.3 (Equation 4). ## What Learning Lift measures Learning Lift quantifies whether operator feedback on earlier recurring scenarios actually improves later decisions. We compute it as: LL = IDQ(adapted) − IDQ(frozen) - **IDQ(adapted)** — the Insight Decision Quality of a policy that has received the labels from the *earlier* round of the same scenario family. - **IDQ(frozen)** — the IDQ of the exact same policy version, run against the *later* decision points without any exposure to those earlier labels. Positive LL on a recurring scenario family is the strongest evidence that the learning loop (feedback → policy update → later replay) produces better timing and action choice. It justifies gate relaxation; zero or negative LL is a signal to pause promotion and investigate the feedback path. ## Scenario format A replay scenario file is a small YAML fixture. It pins a **fingerprint family** and a sequence of decision points with ground-truth labels. ```yaml replay_scenario: family_id: "bgp-flap-hotspot-northbound-weekends" description: > BGP session flap on cr1.de1 weekends. Same vendor / same RPKI-sourced prefix. Fingerprint is stable across weeks; only the exact timestamp and prefix vary. source_ledger_pattern: "ledger/insights/noc/bgp-flap-*" min_span_days: 14 decision_points: # Decision point A – the "past round" - id: dp-a timestamp: "2025-06-14T08:32:00Z" fingerprint: loop: noc posture: bgp signal_hash: "a1b2c3d4" labels: - operator: "svag" verdict: "surfaced" action: "prepare_commit_confirm" comment: "real flap, standard path" # Decision point B – the "later round", same family - id: dp-b timestamp: "2025-06-21T09:05:00Z" fingerprint: loop: noc posture: bgp signal_hash: "a1b2c3d4" labels: - operator: "svag" verdict: "surfaced" action: "prepare_commit_confirm" comment: "same signature, correct again" family_id is a human-readable stable identifier. source_ledger_pattern tells the knowledge-repo ingest where to look for the matching InsightDecisionRecord stream. decision_points are ordered. Labels from dp-a are fed only into the adapted policy; the frozen policy is the codebase version that was current at dp-a.timestamp and is never re-trained. Both policies are then evaluated on dp-b using the same evaluation harness that the 53-case deterministic fixture suite uses. Deterministic replay contract The replay harness is deterministic and fixture-first, just like the existing 53-case insight-policy regression suite: Freeze: check out the policy version pinned by the first decision point (dp-a). Call this policy_frozen. Adapt: start from policy_frozen, apply the dp-a labels as operator feedback, and run the policy-update step once. Call this policy_adapted. Evaluate both on dp-b using the identical feature vector that was extracted at dp-b.timestamp. Compute IDQ_frozen and IDQ_adapted. Report: LL = IDQ_adapted - IDQ_frozen. The harness runs in the knowledge repo (evals/replay/). It needs no live infrastructure, no credentials, and no non-deterministic LLM calls — the feature vectors and labels are the fixture. Usage in promotion runbooks SOC mode promotion After the required idq and cgs thresholds, an additional positive evidence line is expected before handoff_live: LL ≥ 0 on at least one recurring SOC scenario family (≥ 5 labels in the family, spanning ≥ 7 days). If LL is negative, promotion to handoff_live is blocked pending investigation of the feedback/update pipeline. NOC standing-grant rollout Before any NOC_STANDING_GRANT_ACTION_CLASSES expansion beyond the initial acknowledge_icinga grant, the runbook should show: LL ≥ 0 on at least one recurring NOC scenario family with the grant-relevant action class present in the labels (≥ 5 labels, spanning ≥ 7 days). Metrics output The knowledge-repo CLI reports LL alongside IDQ and CGS: cd ~/Dev/knowledge uv run hyrule-knowledge insights metrics --loop soc Sample output: loop: soc idq: 0.82 cgs: 0.71 learning_lift: 0.09 (family: bgp-flap-hotspot-northbound-weekends, n=2) label_count: 47 When no replay scenario has been evaluated yet, learning_lift is reported as null with a note to run the replay harness. The placeholder is removed once at least one family is populated. Rollout and maintenance Scenario families live in knowledge/evals/replay/scenarios/<loop>/. Each family is a YAML file. New families are added by curators; the same branch-protection rule that guards ledger/insights/ also guards evals/replay/scenarios/. The nightly knowledge-refresh job (auto-merge.yml) skips scenario files — they are human-curated fixtures, not machine-generated projections. Operators may propose new families via the standard knowledge PR flow. The PR must include the fixture, the computed LL from a local replay run, and a justification for why the fingerprint is stable enough to be considered recurring. </details> <details><summary><a href='https://github.com/AS215932/network-operations/pull/479/files#diff-d8dc3f328a02b239cf99e6f2d91d0f196f1cc33438c413ac62984e1ff93c45e2R35-R35'><strong>Premature Criteria</strong></a> The new promotion criterion for handoff_dry → handoff_live requires LL ≥ 0, but the LL metric is still a placeholder (null) and the replay harness is not implemented. Until the knowledge repo change is done, this criterion cannot be evaluated, and listing it in the runbook may mislead operators into thinking the metric is available. </summary> ```markdown | handoff_dry → handoff_live | Everything above, plus ≥ 10 dry-built handoffs labeled well-formed with accept ≥ 0.80; LL ≥ 0 on ≥ 1 recurring SOC scenario family (≥ 5 labels, ≥ 7 days) |

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 3660312
Explore these optional code suggestions:

Category	Suggestion	Impact
General	Clarify adaptation scope for multi-point families The description implies that only the labels from the first decision point are used for adaptation. In practice a family can have more than two decision points, and the adapted policy should incorporate all labels from decision points preceding the evaluation point. Revise step 2 to say "apply all labels from decision points earlier than the evaluation point" and step 3 to say "on the latest decision point (or all later points, depending on the harness configuration)". docs/runbooks/learning-lift-replay-harness.md [80-86] -1. **Freeze**: check out the policy version pinned by the first decision point - (`dp-a`). Call this `policy_frozen`. -2. **Adapt**: start from `policy_frozen`, apply the `dp-a` labels as operator - feedback, and run the policy-update step once. Call this `policy_adapted`. -3. **Evaluate both** on `dp-b` using the identical feature vector that was - extracted at `dp-b.timestamp`. Compute `IDQ_frozen` and `IDQ_adapted`. +1. **Freeze**: check out the policy version that was current at the timestamp of + the earliest decision point in the family. Call this `policy_frozen`. +2. **Adapt**: start from `policy_frozen`, apply **all labels** from decision + points that precede the evaluation point as operator feedback, and run the + policy-update step. Call this `policy_adapted`. +3. **Evaluate both** on the **latest** decision point (or on every later + decision point and aggregate) using the identical feature vector that was + extracted at that decision point's timestamp. Compute `IDQ_frozen` and + `IDQ_adapted`. 4. **Report**: `LL = IDQ_adapted - IDQ_frozen`. Suggestion importance[1-10]: 7 __ Why: The suggestion improves the description to handle families with more than two decision points, making the documentation more accurate and less misleading. It is a valid and important clarification.	Medium
	Minimum example size mismatch The example fixture contains only two decision points (n=2). The criteria later in the same document require ≥5 labels in the family, so the example could mislead curators into thinking two points satisfy the threshold. Replace the example with a family that has at least five labelled decision points. docs/runbooks/learning-lift-replay-harness.md [31-64] replay_scenario: family_id: "bgp-flap-hotspot-northbound-weekends" description: > BGP session flap on cr1.de1 weekends. Same vendor / same RPKI-sourced prefix. Fingerprint is stable across weeks; only the exact timestamp and prefix vary. source_ledger_pattern: "ledger/insights/noc/bgp-flap-*" min_span_days: 14 decision_points: - - id: dp-a - timestamp: "2025-06-14T08:32:00Z" + - id: dp-1 + timestamp: "2025-06-07T08:32:00Z" fingerprint: loop: noc posture: bgp signal_hash: "a1b2c3d4" labels: - operator: "svag" verdict: "surfaced" action: "prepare_commit_confirm" comment: "real flap, standard path" - - id: dp-b - timestamp: "2025-06-21T09:05:00Z" + - id: dp-2 + timestamp: "2025-06-14T09:05:00Z" fingerprint: loop: noc posture: bgp signal_hash: "a1b2c3d4" labels: - operator: "svag" verdict: "surfaced" action: "prepare_commit_confirm" comment: "same signature, correct again" + - id: dp-3 + timestamp: "2025-06-21T10:12:00Z" + fingerprint: + loop: noc + posture: bgp + signal_hash: "a1b2c3d4" + labels: + - operator: "svag" + verdict: "surfaced" + action: "prepare_commit_confirm" + comment: "identical pattern, operator approves" + + - id: dp-4 + timestamp: "2025-06-28T11:30:00Z" + fingerprint: + loop: noc + posture: bgp + signal_hash: "a1b2c3d4" + labels: + - operator: "svag" + verdict: "surfaced" + action: "prepare_commit_confirm" + comment: "recurring, no deviation" + + - id: dp-5 + timestamp: "2025-07-05T07:45:00Z" + fingerprint: + loop: noc + posture: bgp + signal_hash: "a1b2c3d4" + labels: + - operator: "svag" + verdict: "surfaced" + action: "prepare_commit_confirm" + comment: "consistent, good signal" + Suggestion importance[1-10]: 3 __ Why: The suggestion correctly identifies that the example has only two decision points while criteria require ≥5, but it fails to update the later references to dp-a and dp-b in the replay contract, which would become inconsistent. Therefore, the suggestion is incomplete and could introduce confusion.	Low

---

Previous suggestions

Suggestions up to commit 3660312

Category	Suggestion	Impact
General	Align label count wording The threshold for "≥ 5 labels in the family" is inconsistent with the SOC promotion criteria in soc-mode-promotion.md, which requires "≥ 5 labels, ≥ 7 days" without specifying "in the family." Clarify whether the 5-label minimum applies to the entire family or to the specific scenario family used for LL computation, and align the wording across both runbooks to avoid ambiguity. docs/runbooks/learning-lift-replay-harness.md [99-101] -- `LL ≥ 0` on at least one recurring SOC scenario family (≥ 5 labels in the - family, spanning ≥ 7 days). If `LL` is negative, promotion to `handoff_live` - is blocked pending investigation of the feedback/update pipeline. +- `LL ≥ 0` on at least one recurring SOC scenario family (≥ 5 labels across the family, spanning ≥ 7 days). If `LL` is negative, promotion to `handoff_live` is blocked pending investigation of the feedback/update pipeline. Suggestion importance[1-10]: 5 __ Why: The suggestion correctly identifies a minor inconsistency in wording between runbooks, but the existing code already implies the labels are within the family context. The improvement is marginal and does not address a functional issue, so a moderate score is appropriate.	Low
	Scope label count to family The NOC standing-grant rollout criteria reference "≥ 5 labels" but do not specify whether these must be within the same scenario family or across the entire loop. This could lead to misinterpretation when multiple families exist. Add explicit scoping to match the SOC criteria pattern, e.g., "≥ 5 labels in that family." docs/runbooks/learning-lift-replay-harness.md [108-109] - `LL ≥ 0` on at least one recurring NOC scenario family with the grant-relevant - action class present in the labels (≥ 5 labels, spanning ≥ 7 days). + action class present in the labels (≥ 5 labels in that family, spanning ≥ 7 days). Suggestion importance[1-10]: 5 __ Why: The suggestion adds clarity by explicitly scoping the label count to the family, matching the SOC pattern. However, the original text already implies this scope, so the improvement is minor and does not fix a critical issue, warranting a moderate score.	Low

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[github-actions]

Persistent review updated to latest commit 3660312