Skip to content

fix(bots): count only broadcast submits and report plan skips - #181

Merged
haydenshively merged 3 commits into
mainfrom
fix/bots-35-tick-telemetry-and-submit-outcome
Aug 28, 2026
Merged

fix(bots): count only broadcast submits and report plan skips#181
haydenshively merged 3 commits into
mainfrom
fix/bots-35-tick-telemetry-and-submit-outcome

Conversation

@haydenshively

@haydenshively haydenshively commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Supersedes #134, absorbing its load-bearing parts. Groundwork for BOTS-35 defect #1 (profit
ordering), which stacks on this in #182.

The submit-outcome bug

PendingQueue.submit already documents that it resolves false when nothing was broadcast, and that
"a caller counting real broadcasts must not count those". Both liquidators typed the dep as
Promise<void> and discarded it, so a failed send cleared the position's backoff and incremented
submitted — resetting a failing position to attempt 1 with no suppression at all.

Now honored in both bots, with a notSent counter — and the outcome is structured, because the
boolean turned out to be the wrong shape.

submit resolves false for four different causes, and they are not equivalent: tx.send_aborted,
nonce.sync_failed and queue.nonce_hole are queue-wide and say nothing about the position, while
tx.submit_failed is that position's own send being rejected. It now resolves a SubmitOutcome
carrying which:

export type SubmitOutcome = { sent: true } | { sent: false; reason: 'refused' | 'send_failed' }

Only send_failed re-arms backoff. refused leaves the position immediately retryable, which is the
point of not blaming it for a queue-wide condition.

Both reallocation bots consume the same queue and are adapted. Neither keeps per-position backoff —
one label per vault, retried next tick — so both read only the broadcast fact, with the reason logged
on reallocation.not_broadcast.

This corrects an earlier version of this PR, which recorded nothing on any false and claimed the
failure history "stands". It does not: reaching the submit branch means the backoff gate already let
the position through, so any surviving entry is necessarily expired. Retaining it suppressed nothing
and the next block re-quoted, re-simulated and re-sent — the exact loop this PR set out to stop, still
intact for send failures. Found in review; see the thread on runner/tick.ts.

It also vindicates #134's SubmitOutcome, which this PR had dropped on the argument that the existing
boolean already carried the contract. The boolean carried the broadcast fact; the caller needs the
cause.

What the bug cost, measured

The 31 Jul maturity produced 157 tx.submit_failed in the send-stage window. Joined on the label
field those resolve to six positions:

position submit_failed sent confirmed
0xe3867590 49 2 2
0x2aa923e0 47 2 2
0x37a87ca1 19 0 0
0x9b2ce248 18 1 1
0xb98e7ebf 13 1 1
0xba22ef2e 11 1 1
five others 0 1 1

Two positions absorbed 96 attempts between them and 0x37a87ca1 failed 19 times and never landed.
Without the backoff clear those six would have backed off instead of re-attempting dozens of times.
A month-old unexplained number, now closed.

Plan-skip reporting and counter identities

plan()planWithReason(), so a sizing skip reports why (plan.skipped) instead of vanishing.
TickCounters is restructured around identities documented on the type:

liquidatable === inflightSkipped + planSkipped + planned
planned      === cooledDown + backoffSkipped + noSwapPath + quoteFailed + ok + reverted
ok           === submitted + notSent

A stage added without a counter now breaks a sum instead of silently dropping a position, and the
identities are asserted in tests. tick.end is emitted even when a submit aborts the tick, with
complete: false. A sizing skip deliberately records neither backoff nor cooldown — several reasons
clear on their own as chain time advances, so suppressing a skipped position would delay
re-evaluating it exactly when it becomes viable.

Two latent bugs the skip reasons guard

  • cap_not_positivemaxRepaidNormalMode takes effectiveDebt - maxDebt as its numerator,
    which goes negative whenever a write-off pushes effective debt under maxDebt while debt > maxDebt keeps normal mode open (e.g. debt 1000, maxDebt 900, badDebt 200). The negative cap
    propagated through maxSeizeForCap into a plan with negative seizedAssets.
  • nothing_to_seize — an empty best slot built a (0, 0) plan, which isBadDebtRealization
    reads as a write-off against a still-solvent position.

Both are reachable and neither was caught by the old if (seizedAssets === 0n) check.

Scope

Blue takes the submit-outcome fix only — that bug is live on two chains (Base + Robinhood), so
closing #134 without porting it would silently drop a production fix. Deferred from #134 and not
absorbed: BlockSampler / block-cadence.ts, and blue's planWithReason + counter restructure.

Context, and what this does not fix

Three findings came out of the 31 Jul cbBTC/USDC post-mortem, kept deliberately distinct:

  1. Allowance under-approval at the simulate stage — suppression active. Not addressed here. The
    mechanism's premise is verified in source (encode-call.ts:103-107 approves balance-based,
    "over-approving by the profit margin", true only when a margin exists) and its viability predicate
    matches 13 of 13 sampled outcomes, but no reproduction exists.
  2. Submit outcome discarded at the send stage — no suppression. This PR.
  3. Backoff schedule vs the viability crossover: BACKOFF_MAX_BLOCKS=64 gives a 128s ceiling against
    a crossover at ~138s, and backoff suppressed 743 of 824 pre-crossover plan attempts. An env
    change (BACKOFF_BASE_BLOCKS=1, BACKOFF_MAX_BLOCKS=8), not code.

Sizing the prize honestly: $10,004 was notional. A liquidator's margin is only the LIF bonus,
14.98 bps at t+123s — roughly $15 gross. Across the whole maturity, $11,424 repaid at 15–25 bps
of clearing headroom is $17–29 gross split among every liquidator. This PR is justified as
correctness, not revenue; the pool scales at roughly 0.2% of notional, so the revenue case rests on
where Midnight maturity size is heading, not on this incident.

Why nothing was fillable early in the window: execution cost was dominated by oracle-versus-DEX
basis
, which decayed over about two minutes — best observed cost 24.19 bps at t+60–80 falling to
7.85 bps at t+138, on the same venue and the same collateral across every position size. Against a
LIF ramp still under 15 bps at t+123, no candidate cleared. So ordering, a viability gate, and the
backoff fix are each correct and none of them wins that particular fill.

Sizing: the write-off cap guard

Normal-mode sizing guarded the selected repay cap, but an RCF-exempt slot takes that cap from the
still-positive effectiveDebt, so a write-off pushing effective debt under maxDebt slipped through
and produced a plan that reverts on every simulation.

The contract computes the RCF numerator _position.debt - maxDebt on the post-writeoff debt
(midnight-contracts.txt:1864) and unconditionally, before the rcfThreshold exemption in the same
require can waive the cap — while normal mode is gated on the pre-writeoff originalDebt
(:1826). So the subtraction underflows to Panic 0x11 whatever the exemption says; the contract's own
":1861" comment that debt >= maxDebt in that branch does not survive a write-off.

Now guarded on the underflow condition itself, under a new writeoff_below_max_debt skip reason. The
pre-existing test for the non-exempt case had noted the exempt path "would ... produce a plan instead"
and worked around it; that is now a test rather than a comment.

Verification

  • pnpm --filter @morpho-org/{midnight,blue}-liquidation run typecheck — 0 errors
  • pnpm lint — 0 warnings; pnpm format; knip clean
  • pnpm test — 2411 passed / 12 skipped (+11 new). The 4 failing files are pre-existing and
    environmental (RPC_URL_8453 unset for the anvil fork suites); identical on stashed main, which
    had 2400 passing.
  • Per CLAUDE.md, each new behaviour was inverted to confirm its test fails, then reverted — exactly
    3 targeted failures, no others.

Linear

Part of BOTS-35

This PR must not close BOTS-35. It is groundwork only — the submit-outcome fix and the plan-skip
telemetry. None of the ticket's four acceptance criteria is met by this PR alone; #182 meets criteria 1
and 3, criterion 2's premise is retracted there, and criterion 4 needs a live maturity. See #182 for
the full criterion-by-criterion status.

Referenced, not closed

Ref BOTS-87 — porting this PR's plan-skip telemetry to blue, plus the block sampler.

🤖 Generated with Claude Code

@linear-code

linear-code Bot commented Aug 27, 2026

Copy link
Copy Markdown

BOTS-35

BOTS-87

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6be43bf6e9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bots/midnight-liquidation/src/sizing/plan.ts
Comment thread bots/midnight-liquidation/test/runner/tick.test.ts
Comment thread bots/midnight-liquidation/src/runner/tick.ts
Comment thread bots/midnight-liquidation/src/sizing/plan.ts
haydenshively and others added 3 commits August 28, 2026 08:30
Absorbs the load-bearing parts of #134, which supersedes it.

`PendingQueue.submit` already documents that it resolves `false` when nothing
was broadcast, and that "a caller counting real broadcasts must not count
those" — but both liquidators' index.ts closures typed the dep as
`Promise<void>` and discarded it. So a failed send cleared the position's
backoff and incremented `submitted`, resetting a failing position to attempt 1
with no suppression. Honor the boolean in both bots and add a `notSent`
counter. On a no-broadcast outcome the failure history now stands: backoff is
neither cleared nor recorded, since a queue refusal says nothing about the
position.

Replace midnight's `plan()` with `planWithReason()` so a sizing skip reports
why (`plan.skipped`) instead of vanishing, and restructure `TickCounters`
around documented sum identities so a stage added without a counter breaks a
sum instead of silently dropping a position. `tick.end` is now emitted even
when a submit aborts the tick, with `complete: false`.

Two of the skip reasons guard latent bugs rather than just naming outcomes:

- `cap_not_positive`: `maxRepaidNormalMode` takes `effectiveDebt - maxDebt` as
  its numerator, which goes negative whenever a write-off pushes effective debt
  under `maxDebt` while `debt > maxDebt` keeps normal mode open (debt 1000,
  maxDebt 900, badDebt 200). A negative cap propagated through
  `maxSeizeForCap` into a plan with negative `seizedAssets`.
- `nothing_to_seize`: an empty best slot built a `(0, 0)` plan, which
  `isBadDebtRealization` reads as a write-off against a solvent position.

Deferred from #134, not absorbed: `BlockSampler` / `block-cadence.ts`, and
blue's `planWithReason` + counter-identity restructure. Blue takes the
submit-outcome fix only, since that bug is live on two chains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects found by review of this PR.

The submit boolean conflated a queue-wide refusal with this position's own send
being rejected. Only the first says nothing about the position; the second says a
great deal. Reaching submit at all means the backoff gate let the position
through, so any surviving entry is necessarily expired — leaving it untouched
suppressed nothing, and the next block re-quoted, re-simulated and re-sent. That
is the loop this PR set out to stop. submit now resolves a SubmitOutcome and only
'send_failed' re-arms backoff.

Normal-mode sizing guarded the selected repay cap, but an RCF-exempt slot takes
the cap from the still-positive effectiveDebt, so a write-off that pushes
effective debt under maxDebt slipped past. The contract evaluates
'_position.debt - maxDebt' on the post-writeoff debt and unconditionally, before
the exemption in the same require (midnight-contracts.txt:1864), while normal
mode is gated on the pre-writeoff originalDebt (:1826) — so those plans revert
with Panic 0x11 every time. Guard the underflow condition itself, under a new
writeoff_below_max_debt skip reason.
Both reallocators consume the same pending queue, so widening submit's return type
broke their typecheck. Neither keeps per-position backoff — one label per vault,
retried next tick — so both only read the broadcast fact; the reason is logged on
reallocation.not_broadcast, where it tells an operator whether the queue refused
or this vault's own send was rejected.
@haydenshively
haydenshively force-pushed the fix/bots-35-tick-telemetry-and-submit-outcome branch from ad77590 to 6294bd5 Compare August 28, 2026 13:31
@haydenshively
haydenshively merged commit 298aa1b into main Aug 28, 2026
4 checks passed
@haydenshively
haydenshively deleted the fix/bots-35-tick-telemetry-and-submit-outcome branch August 28, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants