fix(bots): count only broadcast submits and report plan skips - #181
Merged
haydenshively merged 3 commits intoAug 28, 2026
Conversation
This was referenced Aug 27, 2026
haydenshively
marked this pull request as ready for review
August 28, 2026 01:59
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6be43bf6e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
spennyp
approved these changes
Aug 28, 2026
Absorbs the load-bearing parts of #134, which supersedes it. `PendingQueue.submit` already documents that it resolves `false` when nothing was broadcast, and that "a caller counting real broadcasts must not count those" — but both liquidators' index.ts closures typed the dep as `Promise<void>` and discarded it. So a failed send cleared the position's backoff and incremented `submitted`, resetting a failing position to attempt 1 with no suppression. Honor the boolean in both bots and add a `notSent` counter. On a no-broadcast outcome the failure history now stands: backoff is neither cleared nor recorded, since a queue refusal says nothing about the position. Replace midnight's `plan()` with `planWithReason()` so a sizing skip reports why (`plan.skipped`) instead of vanishing, and restructure `TickCounters` around documented sum identities so a stage added without a counter breaks a sum instead of silently dropping a position. `tick.end` is now emitted even when a submit aborts the tick, with `complete: false`. Two of the skip reasons guard latent bugs rather than just naming outcomes: - `cap_not_positive`: `maxRepaidNormalMode` takes `effectiveDebt - maxDebt` as its numerator, which goes negative whenever a write-off pushes effective debt under `maxDebt` while `debt > maxDebt` keeps normal mode open (debt 1000, maxDebt 900, badDebt 200). A negative cap propagated through `maxSeizeForCap` into a plan with negative `seizedAssets`. - `nothing_to_seize`: an empty best slot built a `(0, 0)` plan, which `isBadDebtRealization` reads as a write-off against a solvent position. Deferred from #134, not absorbed: `BlockSampler` / `block-cadence.ts`, and blue's `planWithReason` + counter-identity restructure. Blue takes the submit-outcome fix only, since that bug is live on two chains. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects found by review of this PR. The submit boolean conflated a queue-wide refusal with this position's own send being rejected. Only the first says nothing about the position; the second says a great deal. Reaching submit at all means the backoff gate let the position through, so any surviving entry is necessarily expired — leaving it untouched suppressed nothing, and the next block re-quoted, re-simulated and re-sent. That is the loop this PR set out to stop. submit now resolves a SubmitOutcome and only 'send_failed' re-arms backoff. Normal-mode sizing guarded the selected repay cap, but an RCF-exempt slot takes the cap from the still-positive effectiveDebt, so a write-off that pushes effective debt under maxDebt slipped past. The contract evaluates '_position.debt - maxDebt' on the post-writeoff debt and unconditionally, before the exemption in the same require (midnight-contracts.txt:1864), while normal mode is gated on the pre-writeoff originalDebt (:1826) — so those plans revert with Panic 0x11 every time. Guard the underflow condition itself, under a new writeoff_below_max_debt skip reason.
Both reallocators consume the same pending queue, so widening submit's return type broke their typecheck. Neither keeps per-position backoff — one label per vault, retried next tick — so both only read the broadcast fact; the reason is logged on reallocation.not_broadcast, where it tells an operator whether the queue refused or this vault's own send was rejected.
haydenshively
force-pushed
the
fix/bots-35-tick-telemetry-and-submit-outcome
branch
from
August 28, 2026 13:31
ad77590 to
6294bd5
Compare
haydenshively
deleted the
fix/bots-35-tick-telemetry-and-submit-outcome
branch
August 28, 2026 13:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #134, absorbing its load-bearing parts. Groundwork for BOTS-35 defect #1 (profit
ordering), which stacks on this in #182.
The submit-outcome bug
PendingQueue.submitalready documents that it resolvesfalsewhen nothing was broadcast, and that"a caller counting real broadcasts must not count those". Both liquidators typed the dep as
Promise<void>and discarded it, so a failed send cleared the position's backoff and incrementedsubmitted— resetting a failing position to attempt 1 with no suppression at all.Now honored in both bots, with a
notSentcounter — and the outcome is structured, because theboolean turned out to be the wrong shape.
submitresolvesfalsefor four different causes, and they are not equivalent:tx.send_aborted,nonce.sync_failedandqueue.nonce_holeare queue-wide and say nothing about the position, whiletx.submit_failedis that position's own send being rejected. It now resolves aSubmitOutcomecarrying which:
Only
send_failedre-arms backoff.refusedleaves the position immediately retryable, which is thepoint of not blaming it for a queue-wide condition.
Both reallocation bots consume the same queue and are adapted. Neither keeps per-position backoff —
one label per vault, retried next tick — so both read only the broadcast fact, with the reason logged
on
reallocation.not_broadcast.This corrects an earlier version of this PR, which recorded nothing on any
falseand claimed thefailure history "stands". It does not: reaching the submit branch means the backoff gate already let
the position through, so any surviving entry is necessarily expired. Retaining it suppressed nothing
and the next block re-quoted, re-simulated and re-sent — the exact loop this PR set out to stop, still
intact for send failures. Found in review; see the thread on
runner/tick.ts.It also vindicates #134's
SubmitOutcome, which this PR had dropped on the argument that the existingboolean already carried the contract. The boolean carried the broadcast fact; the caller needs the
cause.
What the bug cost, measured
The 31 Jul maturity produced 157
tx.submit_failedin the send-stage window. Joined on thelabelfield those resolve to six positions:
0xe38675900x2aa923e00x37a87ca10x9b2ce2480xb98e7ebf0xba22ef2eTwo positions absorbed 96 attempts between them and
0x37a87ca1failed 19 times and never landed.Without the backoff clear those six would have backed off instead of re-attempting dozens of times.
A month-old unexplained number, now closed.
Plan-skip reporting and counter identities
plan()→planWithReason(), so a sizing skip reports why (plan.skipped) instead of vanishing.TickCountersis restructured around identities documented on the type:A stage added without a counter now breaks a sum instead of silently dropping a position, and the
identities are asserted in tests.
tick.endis emitted even when a submit aborts the tick, withcomplete: false. A sizing skip deliberately records neither backoff nor cooldown — several reasonsclear on their own as chain time advances, so suppressing a skipped position would delay
re-evaluating it exactly when it becomes viable.
Two latent bugs the skip reasons guard
cap_not_positive—maxRepaidNormalModetakeseffectiveDebt - maxDebtas its numerator,which goes negative whenever a write-off pushes effective debt under
maxDebtwhiledebt > maxDebtkeeps normal mode open (e.g. debt 1000, maxDebt 900, badDebt 200). The negative cappropagated through
maxSeizeForCapinto a plan with negativeseizedAssets.nothing_to_seize— an empty best slot built a(0, 0)plan, whichisBadDebtRealizationreads as a write-off against a still-solvent position.
Both are reachable and neither was caught by the old
if (seizedAssets === 0n)check.Scope
Blue takes the submit-outcome fix only — that bug is live on two chains (Base + Robinhood), so
closing #134 without porting it would silently drop a production fix. Deferred from #134 and not
absorbed:
BlockSampler/block-cadence.ts, and blue'splanWithReason+ counter restructure.Context, and what this does not fix
Three findings came out of the 31 Jul cbBTC/USDC post-mortem, kept deliberately distinct:
mechanism's premise is verified in source (
encode-call.ts:103-107approves balance-based,"over-approving by the profit margin", true only when a margin exists) and its viability predicate
matches 13 of 13 sampled outcomes, but no reproduction exists.
BACKOFF_MAX_BLOCKS=64gives a 128s ceiling againsta crossover at ~138s, and backoff suppressed 743 of 824 pre-crossover plan attempts. An env
change (
BACKOFF_BASE_BLOCKS=1,BACKOFF_MAX_BLOCKS=8), not code.Sizing the prize honestly: $10,004 was notional. A liquidator's margin is only the LIF bonus,
14.98 bps at t+123s — roughly $15 gross. Across the whole maturity, $11,424 repaid at 15–25 bps
of clearing headroom is $17–29 gross split among every liquidator. This PR is justified as
correctness, not revenue; the pool scales at roughly 0.2% of notional, so the revenue case rests on
where Midnight maturity size is heading, not on this incident.
Why nothing was fillable early in the window: execution cost was dominated by oracle-versus-DEX
basis, which decayed over about two minutes — best observed cost 24.19 bps at t+60–80 falling to
7.85 bps at t+138, on the same venue and the same collateral across every position size. Against a
LIF ramp still under 15 bps at t+123, no candidate cleared. So ordering, a viability gate, and the
backoff fix are each correct and none of them wins that particular fill.
Sizing: the write-off cap guard
Normal-mode sizing guarded the selected repay cap, but an RCF-exempt slot takes that cap from the
still-positive
effectiveDebt, so a write-off pushing effective debt undermaxDebtslipped throughand produced a plan that reverts on every simulation.
The contract computes the RCF numerator
_position.debt - maxDebton the post-writeoff debt(
midnight-contracts.txt:1864) and unconditionally, before thercfThresholdexemption in the samerequirecan waive the cap — while normal mode is gated on the pre-writeofforiginalDebt(:1826). So the subtraction underflows to Panic 0x11 whatever the exemption says; the contract's own
":1861" comment that
debt >= maxDebtin that branch does not survive a write-off.Now guarded on the underflow condition itself, under a new
writeoff_below_max_debtskip reason. Thepre-existing test for the non-exempt case had noted the exempt path "would ... produce a plan instead"
and worked around it; that is now a test rather than a comment.
Verification
pnpm --filter @morpho-org/{midnight,blue}-liquidation run typecheck— 0 errorspnpm lint— 0 warnings;pnpm format;knipcleanpnpm test— 2411 passed / 12 skipped (+11 new). The 4 failing files are pre-existing andenvironmental (
RPC_URL_8453unset for the anvil fork suites); identical on stashedmain, whichhad 2400 passing.
CLAUDE.md, each new behaviour was inverted to confirm its test fails, then reverted — exactly3 targeted failures, no others.
Linear
Part of BOTS-35
This PR must not close BOTS-35. It is groundwork only — the submit-outcome fix and the plan-skip
telemetry. None of the ticket's four acceptance criteria is met by this PR alone; #182 meets criteria 1
and 3, criterion 2's premise is retracted there, and criterion 4 needs a live maturity. See #182 for
the full criterion-by-criterion status.
Referenced, not closed
Ref BOTS-87 — porting this PR's plan-skip telemetry to blue, plus the block sampler.
🤖 Generated with Claude Code