diff --git a/README.md b/README.md index 10ec44c..b71cf71 100644 --- a/README.md +++ b/README.md @@ -78,15 +78,15 @@ The stack monitors: - Disk usage and I/O - Network traffic (receive/transmit) - System uptime -- **Remote Blockchain Nodes** - Planck, Heisenberg, staging bootnode, and staging rpcnode fleets +- **Remote Blockchain Nodes** - Planck, Heisenberg, mainnet bootnode, and mainnet rpcnode fleets - Node metrics (system resources, peers, network I/O) - Substrate metrics (block production, finalization) - Mining metrics (hashrate, difficulty) - - Staging bootnode (`a1`–`a7`) and staging rpcnode (`rpc1`/`rpc2`) scrapes use Cloudflare Access headers (same `http_headers` block as senoti/quersi) -- **Subsquid / Explorer** - Planck testnet and staging mainnet fleets (`planck-subsquid-*`, `staging-subsquid-*`) + - Mainnet bootnode (`a1`–`a7`) and mainnet rpcnode (`rpc1`/`rpc2`) scrapes use Cloudflare Access headers (same `http_headers` block as senoti/quersi) +- **Subsquid / Explorer** - Planck testnet and mainnet fleets (`planck-subsquid-*`, `mainnet-subsquid-*`) - Processor Prometheus on `subsquid-proc-1` / `subsquid-mainnet-proc-1` (active-color indexer) - node_exporter on app, chain, and both DB colors (`subsquid-*.quantus.com` / `subsquid-mainnet-*.quantus.com`) - - Staging scrapes use the same Cloudflare Access headers as senoti/quersi + - Mainnet scrapes use the same Cloudflare Access headers as senoti/quersi - **Support Services** - Telemetry and monitoring infrastructure - Telemetry Host (qm-telemetry.quantus.cat) - VPS system metrics - Telemetry Backend (feed-telemetry.quantus.cat) - Application metrics @@ -359,7 +359,7 @@ When **both** Telegram (`TELEGRAM_BOT_TOKEN` + `TELEGRAM_CHAT_ID`) and `SLACK_WE - 🔴 **Other critical** → Email only - 🟡 **Warnings / non-critical** → Slack - Default receiver → Slack -- **Planck**, **staging_mainnet** (bootnode + rpcnode) → 2 min `group_wait` +- **Planck**, **mainnet** (bootnode + rpcnode) → 2 min `group_wait` - **Heisenberg** → 10 min `group_wait` If either Telegram or Slack is missing, Grafana falls back to email-only local policies (`policies.local.yml`). Contact points for whichever channels are configured are still provisioned, but routing only uses Email until both are set. @@ -372,8 +372,8 @@ Alerts are configured via provisioning files in `grafana/provisioning/alerting/` **Node Health:** - 🔴 **Node Down** - Triggers when a `*-node`, `*-chain`, or `*-substrate` scrape is down for 5+ minutes -- 🔴 **No New Blocks** - Fires when no new blocks have been produced for 7+ minutes (rule); first Telegram notification arrives ~10 min after the last block (7 min threshold + 1 min `for:` + ~2 min `group_wait`). Staging RPC nodes do not export `last_block_time`; they use a dedicated best-block height stall (`delta(...)[7m] < 1`) -- 🟡 **Low Peer Count** - Triggers when peer count drops below 2 (all non-Heisenberg chains, including staging_mainnet) +- 🔴 **No New Blocks** - Fires when no new blocks have been produced for 7+ minutes (rule); first Telegram notification arrives ~10 min after the last block (7 min threshold + 1 min `for:` + ~2 min `group_wait`). Mainnet RPC nodes do not export `last_block_time`; they use a dedicated best-block height stall (`delta(...)[7m] < 1`) +- 🟡 **Low Peer Count** - Triggers when peer count drops below 2 (all non-Heisenberg chains, including mainnet) **System Resources:** - 🔴 **Low Disk Space** - Triggers when disk usage exceeds 85% @@ -476,7 +476,7 @@ Policies are assembled at container start from `policies.production.yml` or `pol | Network | Priority | First Notification | Repeat Interval | |---------|----------|-------------------|-----------------| | **Planck** 🔴 | Highest | 2 minutes | once until resolved (`8736h`) | -| **staging_mainnet** 🔴 | Highest | 2 minutes | once until resolved (`8736h`) | +| **mainnet** 🔴 | Highest | 2 minutes | once until resolved (`8736h`) | | **Heisenberg** 🟡 | Medium | 10 minutes | once until resolved (`8736h`) | Fallback by severity (if no chain label): @@ -667,24 +667,24 @@ monitoring/ ## Included Dashboards -Dashboards are grouped by **concern**, not by network. Chain-specific views use a **Chain** dropdown (planck / heisenberg / staging bootnode / staging rpcnode). +Dashboards are grouped by **concern**, not by network. Chain-specific views use a **Chain** dropdown (planck / heisenberg / mainnet bootnode / mainnet rpcnode). ### Overview (home) **Quantus Network Overview** — first page after login: -- Chain height, last block age, and uptime for Planck, Heisenberg, staging bootnodes, and staging RPC nodes +- Chain height, last block age, and uptime for Planck, Heisenberg, mainnet bootnodes, and mainnet RPC nodes - Telemetry host status and connected nodes - Refreshes every 10 seconds **Service Status** — public-safe status for chains and support services (intended for Grafana Public Dashboard sharing): -- Chains: Planck / Heisenberg (Chain 1–2 + Node 1–2 each); Staging Bootnodes (`a1`–`a7` chain + host, fleet 30d); Staging RPC nodes (`rpc1`/`rpc2` chain + host, fleet 30d) -- Quersi; Logs (Host / Graylog); Senoti units (App / DB / MQ / Watcher / Core); Explorer (Planck) and Staging Explorer units (Indexer / API 1–2 / DB / Chain + sync); Faucet; Telemetry +- Chains: Planck / Heisenberg (Chain 1–2 + Node 1–2 each); Mainnet Bootnodes (`a1`–`a7` chain + host, fleet 30d); Mainnet RPC nodes (`rpc1`/`rpc2` chain + host, fleet 30d) +- Quersi; Logs (Host / Graylog); Senoti units (App / DB / MQ / Watcher / Core); Explorer (Planck) and Mainnet Explorer units (Indexer / API 1–2 / DB / Chain + sync); Faucet; Telemetry - Explorer DB uses `max(up)` across blue/green (only one active outside cutover; matches alerts) - Per-unit UP/DOWN, 30d availability %, and coarse success/error rates only — no host capacity, balances, or internal topology ### Chains -Chain dashboards link to each other via the **Chains** dropdown. The Chain selector is planck / heisenberg / staging bootnode / staging rpcnode. **Consensus & Mining** QPoW panels join validator metrics on the selected job's `chain` label (RPC nodes do not export QPoW). +Chain dashboards link to each other via the **Chains** dropdown. The Chain selector is planck / heisenberg / mainnet bootnode / mainnet rpcnode. **Consensus & Mining** QPoW panels join validator metrics on the selected job's `chain` label (RPC nodes do not export QPoW). | Dashboard | What it covers | |-----------|----------------| @@ -702,7 +702,7 @@ Chain dashboards link to each other via the **Chains** dropdown. The Chain selec | **Telemetry** | Telemetry VPS host + backend message feeds | | **Support Host** | Support server system metrics | | **Senoti Host** | Senoti fleet system metrics | -| **Subsquid Host** | Subsquid fleet system metrics (Fleet: Planck / staging) | +| **Subsquid Host** | Subsquid fleet system metrics (Fleet: Planck / mainnet) | | **Quersi Host** | Quersi wallet remote-config system metrics | | **Logs Host** | Logs server system metrics | @@ -711,7 +711,7 @@ Chain dashboards link to each other via the **Chains** dropdown. The Chain selec | Dashboard | What it covers | |-----------|----------------| | **Faucet** | Request rates, transfers, balance, rejections | -| **Explorer** | Subsquid sync, RPC, Node.js performance (Fleet: Planck / staging) | +| **Explorer** | Subsquid sync, RPC, Node.js performance (Fleet: Planck / mainnet) | | **Graylog** | Ingest rate, journal fill, buffer fill, indexer failures, heap | ## Customization diff --git a/grafana/dashboards/applications/explorer.json b/grafana/dashboards/applications/explorer.json index 69a0e77..3d3c25a 100644 --- a/grafana/dashboards/applications/explorer.json +++ b/grafana/dashboards/applications/explorer.json @@ -902,11 +902,11 @@ }, { "selected": false, - "text": "staging", - "value": "staging" + "text": "mainnet", + "value": "mainnet" } ], - "query": "planck,staging", + "query": "planck,mainnet", "skipUrlSync": false, "type": "custom" } diff --git a/grafana/dashboards/chains/chain-health.json b/grafana/dashboards/chains/chain-health.json index 032a192..8dd73d3 100644 --- a/grafana/dashboards/chains/chain-health.json +++ b/grafana/dashboards/chains/chain-health.json @@ -537,16 +537,16 @@ }, { "selected": false, - "text": "staging bootnode", + "text": "mainnet bootnode", "value": "bootnode" }, { "selected": false, - "text": "staging rpcnode", + "text": "mainnet rpcnode", "value": "rpcnode" } ], - "query": "planck,heisenberg,staging bootnode : bootnode,staging rpcnode : rpcnode", + "query": "planck,heisenberg,mainnet bootnode : bootnode,mainnet rpcnode : rpcnode", "skipUrlSync": false, "type": "custom" } diff --git a/grafana/dashboards/chains/consensus-mining.json b/grafana/dashboards/chains/consensus-mining.json index 4d28934..85e94bc 100644 --- a/grafana/dashboards/chains/consensus-mining.json +++ b/grafana/dashboards/chains/consensus-mining.json @@ -459,16 +459,16 @@ }, { "selected": false, - "text": "staging bootnode", + "text": "mainnet bootnode", "value": "bootnode" }, { "selected": false, - "text": "staging rpcnode", + "text": "mainnet rpcnode", "value": "rpcnode" } ], - "query": "planck,heisenberg,staging bootnode : bootnode,staging rpcnode : rpcnode", + "query": "planck,heisenberg,mainnet bootnode : bootnode,mainnet rpcnode : rpcnode", "skipUrlSync": false, "type": "custom" } diff --git a/grafana/dashboards/chains/network-peers.json b/grafana/dashboards/chains/network-peers.json index cd80fdf..f3a44b7 100644 --- a/grafana/dashboards/chains/network-peers.json +++ b/grafana/dashboards/chains/network-peers.json @@ -2629,16 +2629,16 @@ }, { "selected": false, - "text": "staging bootnode", + "text": "mainnet bootnode", "value": "bootnode" }, { "selected": false, - "text": "staging rpcnode", + "text": "mainnet rpcnode", "value": "rpcnode" } ], - "query": "planck,heisenberg,staging bootnode : bootnode,staging rpcnode : rpcnode", + "query": "planck,heisenberg,mainnet bootnode : bootnode,mainnet rpcnode : rpcnode", "skipUrlSync": false, "type": "custom" } diff --git a/grafana/dashboards/chains/node-operations.json b/grafana/dashboards/chains/node-operations.json index f0e7fd8..b425a65 100644 --- a/grafana/dashboards/chains/node-operations.json +++ b/grafana/dashboards/chains/node-operations.json @@ -1894,16 +1894,16 @@ }, { "selected": false, - "text": "staging bootnode", + "text": "mainnet bootnode", "value": "bootnode" }, { "selected": false, - "text": "staging rpcnode", + "text": "mainnet rpcnode", "value": "rpcnode" } ], - "query": "planck,heisenberg,staging bootnode : bootnode,staging rpcnode : rpcnode", + "query": "planck,heisenberg,mainnet bootnode : bootnode,mainnet rpcnode : rpcnode", "skipUrlSync": false, "type": "custom" } diff --git a/grafana/dashboards/chains/transactions.json b/grafana/dashboards/chains/transactions.json index 3142fb6..abb04ec 100644 --- a/grafana/dashboards/chains/transactions.json +++ b/grafana/dashboards/chains/transactions.json @@ -1097,16 +1097,16 @@ }, { "selected": false, - "text": "staging bootnode", + "text": "mainnet bootnode", "value": "bootnode" }, { "selected": false, - "text": "staging rpcnode", + "text": "mainnet rpcnode", "value": "rpcnode" } ], - "query": "planck,heisenberg,staging bootnode : bootnode,staging rpcnode : rpcnode", + "query": "planck,heisenberg,mainnet bootnode : bootnode,mainnet rpcnode : rpcnode", "skipUrlSync": false, "type": "custom" } diff --git a/grafana/dashboards/infrastructure/subsquid-host.json b/grafana/dashboards/infrastructure/subsquid-host.json index c13e7f6..d343f25 100644 --- a/grafana/dashboards/infrastructure/subsquid-host.json +++ b/grafana/dashboards/infrastructure/subsquid-host.json @@ -18,7 +18,7 @@ "id": 1, "options": { "mode": "markdown", - "content": "# Subsquid Fleet System Metrics\n\nCapacity and health for Subsquid hosts. Use **Fleet** (Planck testnet vs staging mainnet) and **Host** to focus a role or view All.\n\nPlanck: `subsquid-*.quantus.com`. Staging: `subsquid-mainnet-*.quantus.com`.\n\n**DB blue/green:** Outside a deploy cutover, only one of `db-blue` / `db-green` is expected UP; the other may be inactive. Alert fires only when both are down." + "content": "# Subsquid Fleet System Metrics\n\nCapacity and health for Subsquid hosts. Use **Fleet** (Planck testnet vs mainnet) and **Host** to focus a role or view All.\n\nPlanck: `subsquid-*.quantus.com`. Mainnet: `subsquid-mainnet-*.quantus.com`.\n\n**DB blue/green:** Outside a deploy cutover, only one of `db-blue` / `db-green` is expected UP; the other may be inactive. Alert fires only when both are down." } }, { @@ -1165,11 +1165,11 @@ }, { "selected": false, - "text": "staging", - "value": "staging" + "text": "mainnet", + "value": "mainnet" } ], - "query": "planck,staging", + "query": "planck,mainnet", "skipUrlSync": false, "type": "custom" }, diff --git a/grafana/dashboards/overview/service-status.json b/grafana/dashboards/overview/service-status.json index 19d1bd3..f66568e 100644 --- a/grafana/dashboards/overview/service-status.json +++ b/grafana/dashboards/overview/service-status.json @@ -1059,12 +1059,12 @@ "id": 200, "options": { "mode": "markdown", - "content": "### Staging Bootnodes" + "content": "### Mainnet Bootnodes" } }, { "type": "stat", - "title": "Staging Bootnode a1 chain", + "title": "Mainnet Bootnode a1 chain", "gridPos": { "h": 4, "w": 3, @@ -1129,7 +1129,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a2 chain", + "title": "Mainnet Bootnode a2 chain", "gridPos": { "h": 4, "w": 3, @@ -1194,7 +1194,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a3 chain", + "title": "Mainnet Bootnode a3 chain", "gridPos": { "h": 4, "w": 3, @@ -1259,7 +1259,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a4 chain", + "title": "Mainnet Bootnode a4 chain", "gridPos": { "h": 4, "w": 3, @@ -1324,7 +1324,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a5 chain", + "title": "Mainnet Bootnode a5 chain", "gridPos": { "h": 4, "w": 3, @@ -1389,7 +1389,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a6 chain", + "title": "Mainnet Bootnode a6 chain", "gridPos": { "h": 4, "w": 3, @@ -1454,7 +1454,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a7 chain", + "title": "Mainnet Bootnode a7 chain", "gridPos": { "h": 4, "w": 3, @@ -1519,7 +1519,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a1 host", + "title": "Mainnet Bootnode a1 host", "gridPos": { "h": 4, "w": 3, @@ -1584,7 +1584,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a2 host", + "title": "Mainnet Bootnode a2 host", "gridPos": { "h": 4, "w": 3, @@ -1649,7 +1649,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a3 host", + "title": "Mainnet Bootnode a3 host", "gridPos": { "h": 4, "w": 3, @@ -1714,7 +1714,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a4 host", + "title": "Mainnet Bootnode a4 host", "gridPos": { "h": 4, "w": 3, @@ -1779,7 +1779,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a5 host", + "title": "Mainnet Bootnode a5 host", "gridPos": { "h": 4, "w": 3, @@ -1844,7 +1844,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a6 host", + "title": "Mainnet Bootnode a6 host", "gridPos": { "h": 4, "w": 3, @@ -1909,7 +1909,7 @@ }, { "type": "stat", - "title": "Staging Bootnode a7 host", + "title": "Mainnet Bootnode a7 host", "gridPos": { "h": 4, "w": 3, @@ -1974,7 +1974,7 @@ }, { "type": "stat", - "title": "Staging Bootnodes chain — 30d", + "title": "Mainnet Bootnodes chain — 30d", "gridPos": { "h": 4, "w": 12, @@ -2032,7 +2032,7 @@ }, { "type": "stat", - "title": "Staging Bootnodes host — 30d", + "title": "Mainnet Bootnodes host — 30d", "gridPos": { "h": 4, "w": 12, @@ -2099,12 +2099,12 @@ "id": 217, "options": { "mode": "markdown", - "content": "### Staging RPC nodes" + "content": "### Mainnet RPC nodes" } }, { "type": "stat", - "title": "Staging RPC rpc1 chain", + "title": "Mainnet RPC rpc1 chain", "gridPos": { "h": 4, "w": 6, @@ -2169,7 +2169,7 @@ }, { "type": "stat", - "title": "Staging RPC rpc2 chain", + "title": "Mainnet RPC rpc2 chain", "gridPos": { "h": 4, "w": 6, @@ -2234,7 +2234,7 @@ }, { "type": "stat", - "title": "Staging RPC rpc1 host", + "title": "Mainnet RPC rpc1 host", "gridPos": { "h": 4, "w": 6, @@ -2299,7 +2299,7 @@ }, { "type": "stat", - "title": "Staging RPC rpc2 host", + "title": "Mainnet RPC rpc2 host", "gridPos": { "h": 4, "w": 6, @@ -2364,7 +2364,7 @@ }, { "type": "stat", - "title": "Staging RPC chain — 30d", + "title": "Mainnet RPC chain — 30d", "gridPos": { "h": 4, "w": 12, @@ -2422,7 +2422,7 @@ }, { "type": "stat", - "title": "Staging RPC host — 30d", + "title": "Mainnet RPC host — 30d", "gridPos": { "h": 4, "w": 12, @@ -4200,7 +4200,7 @@ "id": 304, "options": { "mode": "markdown", - "content": "## Staging Explorer" + "content": "## Mainnet Explorer" } }, { @@ -4215,7 +4215,7 @@ "id": 330, "targets": [ { - "expr": "up{job=\"staging-subsquid-proc-1\"}", + "expr": "up{job=\"mainnet-subsquid-proc-1\"}", "refId": "A", "datasource": { "type": "prometheus", @@ -4280,7 +4280,7 @@ "id": 331, "targets": [ { - "expr": "up{job=\"staging-subsquid-app-1\"}", + "expr": "up{job=\"mainnet-subsquid-app-1\"}", "refId": "A", "datasource": { "type": "prometheus", @@ -4345,7 +4345,7 @@ "id": 332, "targets": [ { - "expr": "up{job=\"staging-subsquid-app-2\"}", + "expr": "up{job=\"mainnet-subsquid-app-2\"}", "refId": "A", "datasource": { "type": "prometheus", @@ -4410,7 +4410,7 @@ "id": 333, "targets": [ { - "expr": "max(up{job=~\"staging-subsquid-db-(blue|green)-1\"})", + "expr": "max(up{job=~\"mainnet-subsquid-db-(blue|green)-1\"})", "refId": "A", "datasource": { "type": "prometheus", @@ -4475,7 +4475,7 @@ "id": 335, "targets": [ { - "expr": "up{job=\"staging-subsquid-chain-1\"}", + "expr": "up{job=\"mainnet-subsquid-chain-1\"}", "refId": "A", "datasource": { "type": "prometheus", @@ -4540,7 +4540,7 @@ "id": 340, "targets": [ { - "expr": "avg_over_time(up{job=\"staging-subsquid-proc-1\"}[30d]) * 100", + "expr": "avg_over_time(up{job=\"mainnet-subsquid-proc-1\"}[30d]) * 100", "refId": "A", "datasource": { "type": "prometheus", @@ -4598,7 +4598,7 @@ "id": 341, "targets": [ { - "expr": "avg_over_time(up{job=\"staging-subsquid-app-1\"}[30d]) * 100", + "expr": "avg_over_time(up{job=\"mainnet-subsquid-app-1\"}[30d]) * 100", "refId": "A", "datasource": { "type": "prometheus", @@ -4656,7 +4656,7 @@ "id": 342, "targets": [ { - "expr": "avg_over_time(up{job=\"staging-subsquid-app-2\"}[30d]) * 100", + "expr": "avg_over_time(up{job=\"mainnet-subsquid-app-2\"}[30d]) * 100", "refId": "A", "datasource": { "type": "prometheus", @@ -4714,7 +4714,7 @@ "id": 343, "targets": [ { - "expr": "avg_over_time((max(up{job=~\"staging-subsquid-db-(blue|green)-1\"}))[30d:1m]) * 100", + "expr": "avg_over_time((max(up{job=~\"mainnet-subsquid-db-(blue|green)-1\"}))[30d:1m]) * 100", "refId": "A", "datasource": { "type": "prometheus", @@ -4772,7 +4772,7 @@ "id": 345, "targets": [ { - "expr": "avg_over_time(up{job=\"staging-subsquid-chain-1\"}[30d]) * 100", + "expr": "avg_over_time(up{job=\"mainnet-subsquid-chain-1\"}[30d]) * 100", "refId": "A", "datasource": { "type": "prometheus", @@ -4830,7 +4830,7 @@ "id": 336, "targets": [ { - "expr": "sqd_processor_sync_ratio{job=\"staging-subsquid-proc-1\"} * 100", + "expr": "sqd_processor_sync_ratio{job=\"mainnet-subsquid-proc-1\"} * 100", "refId": "A", "datasource": { "type": "prometheus", diff --git a/grafana/dashboards/overview/welcome.json b/grafana/dashboards/overview/welcome.json index 9ed4522..8df5a93 100644 --- a/grafana/dashboards/overview/welcome.json +++ b/grafana/dashboards/overview/welcome.json @@ -18,7 +18,7 @@ "id": 1, "options": { "mode": "markdown", - "content": "# Quantus Network Overview\n\nReal-time monitoring across **Planck**, **Heisenberg**, **Staging bootnodes**, and **Staging RPC nodes**. Use the **Chains** folder dashboards with the chain selector for detailed views.\n\n[quantus.com](https://www.quantus.com/)" + "content": "# Quantus Network Overview\n\nReal-time monitoring across **Planck**, **Heisenberg**, **Mainnet bootnodes**, and **Mainnet RPC nodes**. Use the **Chains** folder dashboards with the chain selector for detailed views.\n\n[quantus.com](https://www.quantus.com/)" } }, { @@ -353,7 +353,7 @@ }, { "type": "stat", - "title": "Staging Bootnodes - Chain Height", + "title": "Mainnet Bootnodes - Chain Height", "gridPos": { "h": 4, "w": 8, @@ -397,7 +397,7 @@ }, { "type": "stat", - "title": "Staging Bootnodes - Last Block", + "title": "Mainnet Bootnodes - Last Block", "gridPos": { "h": 4, "w": 8, @@ -457,7 +457,7 @@ }, { "type": "stat", - "title": "Staging Bootnodes - Uptime", + "title": "Mainnet Bootnodes - Uptime", "gridPos": { "h": 4, "w": 8, @@ -519,7 +519,7 @@ }, { "type": "stat", - "title": "Staging RPC nodes - Chain Height", + "title": "Mainnet RPC nodes - Chain Height", "gridPos": { "h": 4, "w": 8, @@ -563,8 +563,8 @@ }, { "type": "stat", - "title": "Staging RPC nodes - Last Block", - "description": "Seconds since the last QPoW block on the staging chain. RPC nodes are not validators and do not export last_block_time; this uses the bootnode series.", + "title": "Mainnet RPC nodes - Last Block", + "description": "Seconds since the last QPoW block on the mainnet chain. RPC nodes are not validators and do not export last_block_time; this uses the bootnode series.", "gridPos": { "h": 4, "w": 8, @@ -624,7 +624,7 @@ }, { "type": "stat", - "title": "Staging RPC nodes - Uptime", + "title": "Mainnet RPC nodes - Uptime", "gridPos": { "h": 4, "w": 8, diff --git a/grafana/provisioning/alerting/policies.production.yml b/grafana/provisioning/alerting/policies.production.yml index 52c070b..159372d 100644 --- a/grafana/provisioning/alerting/policies.production.yml +++ b/grafana/provisioning/alerting/policies.production.yml @@ -50,11 +50,11 @@ policies: repeat_interval: 8736h continue: false - # Staging (bootnode + rpcnode metrics use chain=staging_mainnet) + # Mainnet (bootnode + rpcnode metrics use chain=mainnet) # No New Blocks + RPC height stall → Email + Telegram - receiver: Email Notifications matchers: - - chain = staging_mainnet + - chain = mainnet - severity = critical - alertname =~ ".*No New Blocks" group_by: ["alertname", "chain"] @@ -64,7 +64,7 @@ policies: - receiver: Telegram Notifications matchers: - - chain = staging_mainnet + - chain = mainnet - severity = critical - alertname =~ ".*No New Blocks" group_by: ["alertname", "chain"] @@ -72,19 +72,19 @@ policies: repeat_interval: 8736h continue: false - # Staging — other critical → Email only + # Mainnet — other critical → Email only - receiver: Email Notifications matchers: - - chain = staging_mainnet + - chain = mainnet - severity = critical group_wait: 2m repeat_interval: 8736h continue: false - # Staging — non-critical → Slack + # Mainnet — non-critical → Slack - receiver: Slack Notifications matchers: - - chain = staging_mainnet + - chain = mainnet group_wait: 2m repeat_interval: 8736h continue: false diff --git a/grafana/provisioning/alerting/rules.yml b/grafana/provisioning/alerting/rules.yml index b9b53e8..2b3825d 100644 --- a/grafana/provisioning/alerting/rules.yml +++ b/grafana/provisioning/alerting/rules.yml @@ -269,7 +269,7 @@ groups: description: '{{ with $values.B }}RPC node {{ index $labels "instance" }} best block height has not increased in 7+ minutes (delta: {{ printf "%.0f" .Value }}) - node may be stalled or disconnected from block producers{{ else }}RPC node {{ index $labels "instance" }} best-block delta could not be measured - block height metrics returned no data{{ end }}' summary: 'No new blocks on RPC node {{ index $labels "instance" }}' labels: - chain: staging_mainnet + chain: mainnet severity: critical # System Resource Monitoring @@ -1150,10 +1150,10 @@ groups: severity: critical # =============================================== - # EXPLORER MONITORING ALERTS (Staging) + # EXPLORER MONITORING ALERTS (Mainnet) # =============================================== - uid: staging_explorer_down - title: Staging Explorer Service Down + title: Mainnet Explorer Service Down condition: C data: - refId: A @@ -1162,7 +1162,7 @@ groups: to: 0 datasourceUid: prometheus model: - expr: up{job="staging-subsquid-proc-1"} + expr: up{job="mainnet-subsquid-proc-1"} refId: A - refId: B relativeTimeRange: @@ -1222,14 +1222,14 @@ groups: execErrState: Alerting for: 3m annotations: - description: 'Staging Explorer service ({{ index $labels "instance" }}) is not responding - blockchain data indexing unavailable' - summary: "Staging Explorer Service is DOWN" + description: 'Mainnet Explorer service ({{ index $labels "instance" }}) is not responding - blockchain data indexing unavailable' + summary: "Mainnet Explorer Service is DOWN" labels: - chain: staging + chain: mainnet severity: critical - uid: staging_explorer_sync_lag - title: Staging Explorer Sync Lag + title: Mainnet Explorer Sync Lag condition: C data: - refId: A @@ -1238,7 +1238,7 @@ groups: to: 0 datasourceUid: prometheus model: - expr: sqd_processor_chain_height{job="staging-subsquid-proc-1"} - sqd_processor_last_block{job="staging-subsquid-proc-1"} + expr: sqd_processor_chain_height{job="mainnet-subsquid-proc-1"} - sqd_processor_last_block{job="mainnet-subsquid-proc-1"} refId: A - refId: B relativeTimeRange: @@ -1298,14 +1298,14 @@ groups: execErrState: Alerting for: 10m annotations: - description: '{{ with $values.B }}Staging Explorer is {{ printf "%.0f" .Value }} blocks behind chain height (threshold: >100 blocks) - check processing performance{{ else }}Staging Explorer sync lag could not be measured - processor metrics returned no data{{ end }}' - summary: "Staging Explorer Synchronization Lag" + description: '{{ with $values.B }}Mainnet Explorer is {{ printf "%.0f" .Value }} blocks behind chain height (threshold: >100 blocks) - check processing performance{{ else }}Mainnet Explorer sync lag could not be measured - processor metrics returned no data{{ end }}' + summary: "Mainnet Explorer Synchronization Lag" labels: - chain: staging + chain: mainnet severity: warning - uid: staging_explorer_high_rpc_failure_rate - title: Staging Explorer High RPC Failure Rate + title: Mainnet Explorer High RPC Failure Rate condition: C data: - refId: A @@ -1314,7 +1314,7 @@ groups: to: 0 datasourceUid: prometheus model: - expr: (rate(sqd_rpc_request_count{job="staging-subsquid-proc-1",kind="failure"}[5m]) / (rate(sqd_rpc_request_count{job="staging-subsquid-proc-1",kind="success"}[5m]) + rate(sqd_rpc_request_count{job="staging-subsquid-proc-1",kind="failure"}[5m]) + 0.001)) * 100 + expr: (rate(sqd_rpc_request_count{job="mainnet-subsquid-proc-1",kind="failure"}[5m]) / (rate(sqd_rpc_request_count{job="mainnet-subsquid-proc-1",kind="success"}[5m]) + rate(sqd_rpc_request_count{job="mainnet-subsquid-proc-1",kind="failure"}[5m]) + 0.001)) * 100 refId: A - refId: B relativeTimeRange: @@ -1374,17 +1374,17 @@ groups: execErrState: Alerting for: 5m annotations: - description: '{{ with $values.B }}Staging Explorer RPC failure rate is {{ printf "%.1f" .Value }}% (threshold: >10%) - check RPC node connectivity at {{ index $labels "url" }}{{ else }}Staging Explorer RPC failure rate could not be measured - RPC metrics returned no data{{ end }}' - summary: "Staging High RPC Failure Rate" + description: '{{ with $values.B }}Mainnet Explorer RPC failure rate is {{ printf "%.1f" .Value }}% (threshold: >10%) - check RPC node connectivity at {{ index $labels "url" }}{{ else }}Mainnet Explorer RPC failure rate could not be measured - RPC metrics returned no data{{ end }}' + summary: "Mainnet High RPC Failure Rate" labels: - chain: staging + chain: mainnet severity: warning # =============================================== - # SUBSQUID FLEET ALERTS (Staging — app, chain, db) + # SUBSQUID FLEET ALERTS (Mainnet — app, chain, db) # =============================================== - uid: staging_subsquid_app_chain_down - title: Staging Subsquid App/Chain Down + title: Mainnet Subsquid App/Chain Down condition: C data: - refId: A @@ -1393,7 +1393,7 @@ groups: to: 0 datasourceUid: prometheus model: - expr: up{job=~"staging-subsquid-(app|chain).*"} + expr: up{job=~"mainnet-subsquid-(app|chain).*"} refId: A - refId: B relativeTimeRange: @@ -1453,14 +1453,14 @@ groups: execErrState: Alerting for: 3m annotations: - description: 'Staging Subsquid {{ index $labels "job" }} ({{ index $labels "instance" }}) is down for more than 3 minutes' - summary: 'Staging Subsquid {{ index $labels "job" }} is DOWN' + description: 'Mainnet Subsquid {{ index $labels "job" }} ({{ index $labels "instance" }}) is down for more than 3 minutes' + summary: 'Mainnet Subsquid {{ index $labels "job" }} is DOWN' labels: - chain: staging + chain: mainnet severity: critical - uid: staging_subsquid_db_both_down - title: Staging Subsquid DB Both Colors Down + title: Mainnet Subsquid DB Both Colors Down condition: C data: - refId: A @@ -1469,7 +1469,7 @@ groups: to: 0 datasourceUid: prometheus model: - expr: max(up{job=~"staging-subsquid-db-(blue|green)-1"}) + expr: max(up{job=~"mainnet-subsquid-db-(blue|green)-1"}) refId: A - refId: B relativeTimeRange: @@ -1529,10 +1529,10 @@ groups: execErrState: Alerting for: 3m annotations: - description: "Both Staging Subsquid DB colors (blue and green) are down - no active database available" - summary: "Staging Subsquid DB both colors DOWN" + description: "Both Mainnet Subsquid DB colors (blue and green) are down - no active database available" + summary: "Mainnet Subsquid DB both colors DOWN" labels: - chain: staging + chain: mainnet severity: critical # =============================================== diff --git a/prometheus/prometheus.yml b/prometheus/prometheus.yml index 30817d6..b416de1 100644 --- a/prometheus/prometheus.yml +++ b/prometheus/prometheus.yml @@ -324,7 +324,7 @@ scrape_configs: - subsquid-chain-1.quantus.com labels: chain: planck -- job_name: staging-subsquid-proc-1 +- job_name: mainnet-subsquid-proc-1 honor_timestamps: true track_timestamps_staleness: false scrape_interval: 15s @@ -351,8 +351,8 @@ scrape_configs: - targets: - subsquid-mainnet-proc-1.quantus.com labels: - chain: staging -- job_name: staging-subsquid-app-1 + chain: mainnet +- job_name: mainnet-subsquid-app-1 honor_timestamps: true track_timestamps_staleness: false scrape_interval: 15s @@ -379,8 +379,8 @@ scrape_configs: - targets: - subsquid-mainnet-app-1.quantus.com labels: - chain: staging -- job_name: staging-subsquid-app-2 + chain: mainnet +- job_name: mainnet-subsquid-app-2 honor_timestamps: true track_timestamps_staleness: false scrape_interval: 15s @@ -407,8 +407,8 @@ scrape_configs: - targets: - subsquid-mainnet-app-2.quantus.com labels: - chain: staging -- job_name: staging-subsquid-db-blue-1 + chain: mainnet +- job_name: mainnet-subsquid-db-blue-1 honor_timestamps: true track_timestamps_staleness: false scrape_interval: 15s @@ -435,8 +435,8 @@ scrape_configs: - targets: - subsquid-mainnet-db-blue-1.quantus.com labels: - chain: staging -- job_name: staging-subsquid-db-green-1 + chain: mainnet +- job_name: mainnet-subsquid-db-green-1 honor_timestamps: true track_timestamps_staleness: false scrape_interval: 15s @@ -463,8 +463,8 @@ scrape_configs: - targets: - subsquid-mainnet-db-green-1.quantus.com labels: - chain: staging -- job_name: staging-subsquid-chain-1 + chain: mainnet +- job_name: mainnet-subsquid-chain-1 honor_timestamps: true track_timestamps_staleness: false scrape_interval: 15s @@ -491,7 +491,7 @@ scrape_configs: - targets: - subsquid-mainnet-chain-1.quantus.com labels: - chain: staging + chain: mainnet - job_name: planck-senoti-app-1 honor_timestamps: true track_timestamps_staleness: false @@ -719,33 +719,36 @@ scrape_configs: CF-Access-Client-Secret: files: - /etc/prometheus/secrets/cf_access_client_secret + relabel_configs: + - target_label: chain + replacement: mainnet static_configs: - targets: - - a1-qm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a1-qm-mainnet.quantus.com labels: instance: a1 - targets: - - a2-qm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a2-qm-mainnet.quantus.com labels: instance: a2 - targets: - - a3-qm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a3-qm-mainnet.quantus.com labels: instance: a3 - targets: - - a4-qm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a4-qm-mainnet.quantus.com labels: instance: a4 - targets: - - a5-qm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a5-qm-mainnet.quantus.com labels: instance: a5 - targets: - - a6-qm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a6-qm-mainnet.quantus.com labels: instance: a6 - targets: - - a7-qm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a7-qm-mainnet.quantus.com labels: instance: a7 - job_name: bootnode-node @@ -771,33 +774,36 @@ scrape_configs: CF-Access-Client-Secret: files: - /etc/prometheus/secrets/cf_access_client_secret + relabel_configs: + - target_label: chain + replacement: mainnet static_configs: - targets: - - a1-hm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a1-hm-mainnet.quantus.com labels: instance: a1 - targets: - - a2-hm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a2-hm-mainnet.quantus.com labels: instance: a2 - targets: - - a3-hm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a3-hm-mainnet.quantus.com labels: instance: a3 - targets: - - a4-hm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a4-hm-mainnet.quantus.com labels: instance: a4 - targets: - - a5-hm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a5-hm-mainnet.quantus.com labels: instance: a5 - targets: - - a6-hm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a6-hm-mainnet.quantus.com labels: instance: a6 - targets: - - a7-hm-mkML2cajSbEaIYKpdo1SFexDqOUOWVy7.quantus.com + - a7-hm-mainnet.quantus.com labels: instance: a7 - job_name: rpcnode-chain @@ -823,13 +829,16 @@ scrape_configs: CF-Access-Client-Secret: files: - /etc/prometheus/secrets/cf_access_client_secret + relabel_configs: + - target_label: chain + replacement: mainnet static_configs: - targets: - - rpc1-qm-bAhQwgqf3zXDHKv6yAmetQkOalMKi1uA1.quantus.com + - rpc1-qm-mainnet.quantus.com labels: instance: rpc1 - targets: - - rpc2-qm-bAhQwgqf3zXDHKv6yAmetQkOalMKi1uA1.quantus.com + - rpc2-qm-mainnet.quantus.com labels: instance: rpc2 - job_name: rpcnode-node @@ -855,13 +864,16 @@ scrape_configs: CF-Access-Client-Secret: files: - /etc/prometheus/secrets/cf_access_client_secret + relabel_configs: + - target_label: chain + replacement: mainnet static_configs: - targets: - - rpc1-hm-bAhQwgqf3zXDHKv6yAmetQkOalMKi1uA1.quantus.com + - rpc1-hm-mainnet.quantus.com labels: instance: rpc1 - targets: - - rpc2-hm-bAhQwgqf3zXDHKv6yAmetQkOalMKi1uA1.quantus.com + - rpc2-hm-mainnet.quantus.com labels: instance: rpc2 - job_name: 'logs-host'