Major x402 fixing

growth-agent/README.md

@@ -27,24 +27,24 @@ Export it as a picture with `uv run python scripts/run_local.py --graph` (writes
 
 ## Stack
 
-| Component | Technology |
-|---|---|
-| Runtime | Python 3.11 (Scaleway Serverless Container) |
-| LLM | IONOS AI Model Hub (Llama 3.3 70B) by default — see below |
-| Social | Mastodon REST API, Bluesky AT Protocol |
-| Storage | Scaleway S3 |
-| Package manager | uv |
+| Component       | Technology                                                |
+| --------------- | --------------------------------------------------------- |
+| Runtime         | Python 3.11 (Scaleway Serverless Container)               |
+| LLM             | IONOS AI Model Hub (Llama 3.3 70B) by default — see below |
+| Social          | Mastodon REST API, Bluesky AT Protocol                    |
+| Storage         | Scaleway S3                                               |
+| Package manager | uv                                                        |
 
 ### LLM provider
 
 The provider is selected at runtime by the `LLM_PROVIDER` env var — `ionos` (default) or
 `mistral` — and each needs its matching API key. Set `LLM_MODEL` to override the provider's
 default model. Selection logic is in `agent/llm_client.py` (`LLMClient.from_env()`).
 
-| `LLM_PROVIDER` | API key env var |
-| --- | --- |
+| `LLM_PROVIDER`    | API key env var   |
+| ----------------- | ----------------- |
 | `ionos` (default) | `IONOS_API_TOKEN` |
-| `mistral` | `MISTRAL_API_KEY` |
+| `mistral`         | `MISTRAL_API_KEY` |
 
 ## Development
 
@@ -137,12 +137,12 @@ uv run python scripts/run_local.py --diagnose
 
 This shows the content queue, next scheduled drafts, LLM analysis status, and recent run logs. Log statuses:
 
-| Status | Meaning |
-|---|---|
-| `completed` | Handler ran successfully |
-| `started` | Handler was invoked but never finished (timeout or crash) |
-| `crashed` | Handler hit an unexpected error (traceback included) |
-| No log for today | Cron did not fire at all |
+| Status           | Meaning                                                   |
+| ---------------- | --------------------------------------------------------- |
+| `completed`      | Handler ran successfully                                  |
+| `started`        | Handler was invoked but never finished (timeout or crash) |
+| `crashed`        | Handler hit an unexpected error (traceback included)      |
+| No log for today | Cron did not fire at all                                  |
 
 ### 2. Run individual tasks locally
 
@@ -177,17 +177,18 @@ kill %1
 
 This runs all daily tasks (analytics, publish, pipeline refill) and weekly tasks (insights on Monday) — exactly what Scaleway executes on `0 8 * * *`.
 
-### 4. Inspect container logs in Grafana (Cockpit)
+### 4. Inspect logs
 
-Grafana gives you the actual stdout/stderr from the running container — useful when the cron fires but no S3 log is written (e.g. startup crash before `_get_storage()` succeeds).
+Grafana/Cockpit dashboards have not worked for this account — they render empty. Read the logs
+through Cockpit's Loki API instead, with the script in `scw_js`:
 
 ```bash
-# After bin/deploy.sh, retrieve the URL once:
-cd terraform
-tofu output grafana_url
+cd ../scw_js && npx tsx scripts/logs.ts growth --since 24h
 ```
 
-Open the URL and log in with your Scaleway account (IAM — no separate Grafana user needed). In Grafana: **Explore → Loki**, query: `{service_name="growth-agent"}`.
+One Cockpit token covers the whole Scaleway project, so that command reads this container's logs as
+well as every serverless function's. Setup and caveats are in
+[`scw_js/README.md`](../scw_js/README.md) → _Reading the logs_.
 
 ### Common issues
 

scw_js/README.md

@@ -144,6 +144,23 @@ The assistant's two web tools, sold per call. `search_service.ts` proxies Brave'
 
 **The SSRF defence is unchanged by payment.** A payment authorises a fetch, not a fetch of `169.254.169.254`. See the header comment of [`web_fetch_service.ts`](./web_fetch_service.ts).
 
+**Refunds, and the one exercise that tests them.** Cooperative refunds are the seller's job (`llmx402cron`, or `scripts/recover_channels.ts` on demand), and the SDK builds each one entirely from the **stored** channel record — the amount from `balance - chargedCumulativeAmount`, the candidate filter from `balance !== 0`, and the signature from `refundNonce`. All three are caches of chain state, and all three have gone stale in production. `resyncChannelState` re-reads them before every sweep, which is the only thing standing between a working refund and a permanently broken one.
+
+The failure mode that hid for months: a successful refund **deletes** the channel record, and a later deposit with the same voucher signer recreates it with `refundNonce: 0` while the chain has moved on. Every later refund is then signed against a consumed nonce and reverts (`0x164f1afe`) — and because the SDK's refund loop has no per-channel catch, one such channel blocks the whole sweep. 2.08 USDC accumulated behind it.
+
+No unit test can catch that: every test here replaces the chain with a mock that agrees with the local record by construction. The exercise that _would_ catch it needs a real chain, and it must **refund the same channel twice** — a single refund passes and proves nothing, because the nonce only goes stale after the first one. On Base Sepolia (free USDC, and the escrow contract is deployed there):
+
+```bash
+# 1. buyer: open a channel and spend on it — scw_js/notebooks/sc_llm_x402_buyer.ipynb (USE_BASE, testnet)
+# 2. seller: claim what is owed, then refund the rest
+npx tsx scripts/recover_channels.ts eip155:84532 --apply
+# 3. buyer: run the notebook again — same voucher signer, so the SAME channelId is re-funded
+# 4. seller: refund a second time. THIS is the step that used to revert.
+npx tsx scripts/recover_channels.ts eip155:84532 --apply
+```
+
+Run it before touching the refund path. It is deliberately not in CI: it needs a funded key and a network, and CI stays hermetic.
+
 ### `growth_api.ts` - Growth Agent Draft Approval
 
 API for reviewing, editing, and approving AI-generated social media drafts. Used by the Growth Agent notebooks and cron job.
@@ -207,6 +224,66 @@ npm run dev:llmx402
 npm run dev:llmx402cron
 ```
 
+## Reading the logs
+
+Not the Scaleway console, and not Grafana — both show nothing useful here. `scripts/logs.ts` queries
+Cockpit's Loki API directly:
+
+```bash
+npx tsx scripts/logs.ts                                      # which functions are logging
+npx tsx scripts/logs.ts facilitator --since 36h --grep "Settlement failed"
+npx tsx scripts/logs.ts llmx402cron --since 48h --grep "Refund sweep"
+```
+
+**It reads the whole Scaleway project, not just this package** — Cockpit is scoped per project, so
+the facilitator, analytics and comment-service logs come out of the same command. The script lives
+here only because this is where the operational scripts live.
+
+Needs `SCW_COCKPIT_LOGS_URL` and `SCW_COCKPIT_LOGS_TOKEN` in `.env`. That token is a **Cockpit**
+token with `read_only_logs` scope — `SCW_SECRET_KEY` is rejected with a 403 — created with
+`scw cockpit token create name=<name> token-scopes.0=read_only_logs region=fr-par`. Its secret is
+shown once, so save it immediately: a token whose secret is lost can only be deleted.
+
+Worth knowing before an incident: a function that has not run inside the `--since` window does not
+appear in the discovery listing at all, so widen the window before concluding anything is missing.
+
+## Alerting on log content
+
+Scaleway's built-in alerts are metric-based, and the failure that motivated this section produced no
+error metric at all — the facilitator answered HTTP 200 with `success: false` and the revert reason
+buried in the body. That is only ever visible as text in a log line, so `alerts/payments.yaml`
+defines Loki ruler rules that watch for it directly.
+
+```bash
+npx tsx scripts/alerts.ts                  # list the rule groups currently on the ruler
+npx tsx scripts/alerts.ts --push           # push alerts/payments.yaml
+npx tsx scripts/alerts.ts --delete payments
+```
+
+Needs `SCW_COCKPIT_RULES_TOKEN` in `.env` — a **separate** Cockpit token from the read-only one
+above, scoped `full_access_logs_rules`:
+`scw cockpit token create name=<name> token-scopes.0=full_access_logs_rules region=fr-par`. Kept
+separate because `logs.ts` is run casually and often and should only ever be able to read; this one
+can write and delete alerting rules.
+
+Four rules, deliberately few — see the comments at the top of `alerts/payments.yaml` for the
+reasoning on which four and why the rest were left out. One constraint worth knowing before writing
+a fifth: **Scaleway's Loki ruler caps a range-vector window at 1h.** `llmx402cron` runs every 12h,
+so a rule cannot stay pinned "firing" across the gap between runs the way a naive `[13h]` window
+would suggest — that gets rejected outright. Every rule here uses `[1h]`, which resolves after an
+hour and re-fires on the next cron run if the problem persists. Read a resolved notification as "no
+new occurrence in the last hour", not as "fixed".
+
+Push is a manual, explicit step — never wired into deploy. Alerting rules changing as a side effect
+of shipping code is its own kind of surprise, and a rule silently dropped by a deploy looks
+identical to a system that is simply quiet.
+
+**A rule that has never fired is not known to work.** Before trusting a new one, push a throwaway
+group matching a log line that occurs on every routine run (e.g. `"claimAndSettle completed"`),
+confirm the email arrives with the summary/description actually populated, then
+`--delete` it. This is exactly how `PaymentCronFailed` was proven to work in practice — pushing a
+test rule surfaced a real, previously-unnoticed `withdraw_delay_mismatch` failure on Base Sepolia.
+
 ## Deployment
 
 ```bash

scw_js/alerts/payments.yaml

@@ -0,0 +1,98 @@
+# Loki alerting rules for the x402 payment path.
+#
+# Pushed with `npx tsx scripts/alerts.ts --push`; never on deploy. Alerting config that changes as
+# a side effect of shipping code is its own kind of surprise.
+#
+# ── Why these four and not more ────────────────────────────────────────────────────────────────
+# Each rule has to mean "a human needs to do something". The failure mode of alerting is not
+# missing an alert, it is sending enough useless ones that people stop reading — which is how the
+# refund bug survived for weeks in a log line that was printed every 12 hours.
+#
+# So: no rule on `"Settlement failed"` as a whole, because most of those are a caller's bad
+# signature or their own empty allowance, not our problem. No rule on
+# `"Cached channel state had drifted"`, because the repair working is the design.
+#
+# ── Why the windows are [1h] ───────────────────────────────────────────────────────────────────
+# Scaleway's Loki ruler caps a range-vector window at 1h — a rule written as `[13h]` is rejected
+# outright (`matrix selector range 13h0m0s exceeds maximum allowed limit of 1h0m0s`), so there is
+# no way to keep an alert pinned open across the 12h gap between `llmx402cron` runs.
+#
+# The honest design given that cap: `[1h]` fires within an hour of the bad line appearing, then
+# resolves — and if the underlying problem is still there, the next cron run produces the same
+# line and the rule fires again 12h later. You get paged every time it recurs, just not
+# continuously in between. A resolved notification here means "no new occurrence in the last
+# hour", not "fixed" — read it that way, don't take it as confirmation.
+name: payments
+interval: 1m
+rules:
+  # The post-condition of the refund sweep, and the most valuable rule here: it is written against
+  # the OUTCOME, so it fires for causes nobody has thought of yet. Four unrelated bugs — a stale
+  # cached balance, a stale refundNonce, a failed refund transaction on Base, a withdrawDelay
+  # mismatch on Base Sepolia — all present identically as escrow that should have gone home.
+  - alert: RefundEscrowStuck
+    expr: |
+      sum(count_over_time({resource_name=~".+llmx402cron"} |= "still hold escrow" [1h])) > 0
+    for: 0m
+    labels:
+      severity: critical
+      area: payments
+    annotations:
+      summary: "Escrow is stuck: channels are past the refund threshold and still hold funds"
+      description: >-
+        The refund sweep ran and left money behind. This fires on the outcome, so the cause is
+        open — read the run before assuming which one it is.
+        Investigate: cd scw_js && npx tsx scripts/logs.ts llmx402cron --since 24h --grep "still hold escrow"
+        Then: npx tsx scripts/recover_channels.ts eip155:10   (dry run; --apply to recover)
+
+  # The cron's own failures. Always ours, never a caller's.
+  - alert: PaymentCronFailed
+    expr: |
+      sum(count_over_time({resource_name=~".+llmx402cron"} |~ "Refund sweep failed|claimAndSettle failed" [1h])) > 0
+    for: 0m
+    labels:
+      severity: critical
+      area: payments
+    annotations:
+      summary: "The payment cron failed a claim or refund sweep"
+      description: >-
+        Claims or refunds threw. The error reason is in the log line.
+        Investigate: cd scw_js && npx tsx scripts/logs.ts llmx402cron --since 24h --grep "failed"
+
+  # Only the subset of settle failures that are OUR problem. A bad signature or a caller's empty
+  # allowance is their business and must not page us.
+  #
+  # `refund_[a-z_]*failed` matches the family, not a list: it started as refund_simulation_failed
+  # alone and missed refund_transaction_failed, one of the four causes of the incident above, which
+  # then paged nothing until the 12-hourly cron tripped RefundEscrowStuck. withdraw_delay_mismatch
+  # stays out deliberately — it is a channel-config disagreement a buyer can cause, and the outcome
+  # rule above already catches the version of it that costs anyone money.
+  - alert: FacilitatorNeedsAttention
+    expr: |
+      sum(count_over_time({resource_name=~".+facilitator"} |= "Settlement failed" |~ "refund_[a-z_]*failed|insufficient_fee_allowance|settlement_pending" [1h])) > 0
+    for: 0m
+    labels:
+      severity: critical
+      area: payments
+    annotations:
+      summary: "Facilitator settlement needs attention (not a caller error)"
+      description: >-
+        One of: a refund simulation reverted, our fee allowance ran out, or a settlement was
+        broadcast but never confirmed (settlement_pending carries a tx hash to reconcile).
+        The decoded EVM revert is in errorMessage, which is logged and never returned over HTTP.
+        Investigate: cd scw_js && npx tsx scripts/logs.ts facilitator --since 2h --grep "Settlement failed"
+
+  # Days of lead time. claim/settle skip /verify, so this path gets no `remainingSettlements`
+  # warning from the facilitator — without this the approval simply runs out one day.
+  - alert: FeeAllowanceLow
+    expr: |
+      sum(count_over_time({resource_name=~".+llmx402cron"} |= "Fee allowance nearly exhausted" [1h])) > 0
+    for: 0m
+    labels:
+      severity: warning
+      area: payments
+    annotations:
+      summary: "Facilitator fee allowance is nearly exhausted"
+      description: >-
+        Claims will start failing with insufficient_fee_allowance once it runs out. Re-approve USDC
+        for the facilitator's spender address from the receiver wallet.
+        Investigate: cd scw_js && npx tsx scripts/logs.ts llmx402cron --since 24h --grep "allowance"