ICP dashboard cycle-burn drop traced to a metrics storage outage

On 14 October 2026, forum user basvandijk confirmed that the ~58% drop in the dashboard's cycle-burn rate on 10 October came from a metrics storage outage. The real burn rate never dropped, and the graph has been backfilled. If you exported cycle-burn or block-rate data from the dashboard or ic-api between 10 and 14 October, fetch it again. No canister costs changed.

What happened to the cycle-burn series

The forum user ICPulse opened the thread on 11 October. According to the report, "On 10 October between 12:30 and 14:00 UTC" the network cycle-burn rate on the dashboard fell from "~48 B cycles/s to ~20 B/s" and stayed there. Per-subnet queries to ic-api showed that all 42 subnets dropped at the same time, each to 0.3–0.5 of its earlier level. Execution metrics did not move. Daily means for 9–11 Oct were 4,561 to 4,686 msg/s, 14.1 to 16.1 B instr/s and 188 to 189 blocks/s. No NNS proposal was executed in that window, and the burn rate had not changed during the 2cdff28 rollout on 8 Oct.

Other signs of the same fault followed:

• Nightly dips. The series fell to ~1.4 B/s for about 45 minutes at 01:00 UTC on 11 Oct. The same V-shaped dip appeared at 00:50 UTC on 12 Oct, while execution metrics stayed flat.
• Block-rate spikes. On 12 Oct the dashboard showed a finalization rate of 1,700 blocks/s at 15:40 UTC, against a normal 160 to 200. Every subnet jumped by about 10x in the same minute; for example, subnet 3zsyy went from 3.0 to 30.3.
• On-chain mismatch. ICPulse read the certified metric over 26 hours and got 51 B cycles/s, while the dashboard showed 32 B for the same window.

On 14 October basvandijk posted the cause. A node in a Kubernetes cluster went down and took one VictoriaMetrics vmstorage pod with it. The other storage nodes kept serving the metrics they held, so the graph showed only part of the total. After the node came back, the team first had to fix a metrics replication problem before they could backfill accurately. The graph now shows correct values. ICPulse replied that the backfill matches their on-chain reads "to within a few percent". The same storage fault explains why the drop was proportional on every subnet, and it also explains the nightly dips.

Who is affected and why it matters

The fault only affected people who read the off-chain metrics pipeline: tokenomics dashboards, cycle-burn trackers, analytics exports and alerts built on ic-api series. Any values cached between 10 and 14 October are wrong, and so are averages computed from them.

Canister balances, fees and the cost table in the docs were not affected. Nothing changed in the protocol. The ICP burn series on the ledger also stayed flat, and ICPulse notes that this is expected. ICP is burned when someone buys cycles, while cycles are consumed when canisters run, so the two series do not move together from day to day.

The incident also shows how fragile the public series can be. The dashboard metrics live in an off-chain time-series database, and losing one storage pod quietly cut the reported total by more than half without any error message. A certified on-chain source exists, and the thread shows how to use it. In reply to a question about moving the metrics infrastructure on-chain, basvandijk pointed to a new dashboard under construction at https://g3psz-xyaaa-aaaam-ajeza-cai.icp.net/.

What to do now

1. Fetch dashboard or ic-api series for 10–14 October again and drop any cached values from that window.
2. To check the numbers yourself, read the path /subnet/<subnet_id>/metrics with a read_state request. The IC interface specification requires the subnet form of the endpoint for this path, for example /api/v3/subnet/<effective_subnet_id>/read_state.
3. Decode the CBOR map. In the state tree, consumed_cycles_total is a map with a low part of type nat and a high part of type opt nat.
4. Compute rates from samples taken hours apart, and use the median of several reads. ICPulse saw twelve certified reads of subnet pzp6e within 17 seconds move between 268,201 T and 268,209 T, because in-flight prepaid execution is counted and later refunded.
5. Treat the value as nominal cycles. The spec says resource use is recorded at normal-schedule rates even where it is free, so the figure is not a count of cycles burned.
6. Add a sanity check: if the burn rate moves but the rate of update_transactions_total does not, suspect the data pipeline before you suspect the network.

Canisters can also call the management canister method subnet_metrics (spec PR #333 in dfinity/developer-docs, merged 7 Oct 2026). The PR describes it as EXPERIMENTAL and says it is callable by canisters only, not via ingress. The Candid interface from that PR is:

type subnet_metrics_args = record { subnet_id : principal };

type subnet_metrics_result = record {
    block_height : nat;
    num_canisters : nat;
    canister_state_bytes : nat;
    consumed_cycles_total : nat;
    update_transactions_total : nat;
    million_round_instructions_total : nat;
};

Caveats

• The only public account of the root cause is the forum reply from basvandijk. There is no separate incident report.
• On 12 October Zane wrote that within "next few weeks" the consumed_cycles_total value under /subnet/<subnet_id>/metrics will become monotonic, with outstanding prepayments no longer included. ICPulse asked whether this will use the same field or a new one, and that question has no answer yet. Monitoring code that takes deltas should be ready for this semantic change.
• Subnet splitting moves canister histories between subnets, so a per-subnet consumed_cycles_total can go down. Sum the values across all subnets for network-level rates.