Sixty Minutes of Doubt: A UK Incident Runbook for Aergo Deployments
Photo: RamaGaspar, CC BY-SA 4.0, via Wikimedia Commons
On-chain finality is binary. Operational response is not. Aergo's block producers will keep producing blocks while your incident channel fills with screenshots and speculation. The first hour is therefore less about root cause and more about converting technical signals into governed decisions that a UK auditor, regulator or board can later review.
The phrase "the chain says no" covers several distinct events: a deterministic smart contract revert, a consensus stall, an unexpected state transition, a key compromise or a bridge imbalance. They share one property. The ledger is the record, but it is not the response plan. Teams that conflate immutability with inaction tend to lose the first hour to debate.
Before the Clock Starts: Define What "No" Means
An incident runbook is only as good as its trigger. Before any anomaly occurs, agree quantitative thresholds for declaring an incident. Examples include a sustained rise in failed transactions above a baseline, a contract event that contradicts an approved state machine, an alert from a signing device, or a reconciliation break between an Aergo deployment and an off-chain system of record.
Name the roles in advance. A UK deployment should have an incident commander, a scribe and a communications lead who are separate people. The commander owns decision rights. The scribe owns the timeline. The communications lead owns the next update, not the whole story. Under the Senior Managers and Certification Regime, map these roles to named responsibilities so that containment actions are not taken by whoever happens to be online.
Minutes 0–10: Declare, Do Not Diagnose
The first ten minutes should produce a declaration, not a diagnosis. Open a single incident channel, state the observed facts, record the time and assign severity. Freeze deployments and configuration changes to the affected environment. If the anomaly involves keys, suspend signing activity for the affected addresses. If it involves a smart contract, confirm whether the contract has a pause function and who controls it.
Avoid the temptation to debug in the main channel. Technical investigation belongs in a sub-thread with a named lead. The incident channel should carry decisions, timestamps and evidence references only. This separation matters later when a regulator or auditor asks who knew what, and when.
Minutes 10–25: Contain Without Rewriting History
Aergo mainnet is not a staging environment. You cannot roll it back for convenience. Containment therefore means changing what your organisation controls, not changing what the chain has already recorded. Practical options include pausing contract functions where the design permits, revoking or rotating compromised keys, isolating a node or RPC endpoint, and halting automated jobs that read from or write to the affected deployment.
Decision rights should be explicit. Who can pause a contract? Who can rotate a key? Who can instruct a counterparty to stop relying on a particular attestation? If the answer is a committee, the committee needs a pre-agreed quorum and a fallback. The worst outcome in this window is a containment action taken without a record, because it creates a second incident: unauthorised operational change.
Minutes 25–40: Preserve Evidence Before You Fix
Fixing the issue is satisfying. Preserving the evidence is mandatory. Capture node logs, RPC request and response pairs, transaction receipts, event logs, signing device logs and the exact configuration of affected services. Take hashes of the artefacts and record them in the incident timeline. Do not delete logs to tidy up. Do not overwrite a compromised key store before its state has been imaged.
If personal data may be involved, the UK GDPR clock starts when you become aware of a personal data breach, not when your investigation concludes. The ICO expects notification within 72 hours where the threshold is met. That does not mean every on-chain anomaly is a reportable breach. It does mean the decision must be taken deliberately and recorded, including the reasoning if you decide not to notify.
Minutes 40–50: Start the Regulatory Clocks
UK teams should maintain a short list of clocks that may start during an incident. The ICO's 72-hour window is the most familiar. FCA and PRA operational resilience expectations require firms to consider whether a severe but plausible disruption affects important business services, and whether supervisory notification is appropriate. Firms with EU operations may also need to consider the Digital Operational Resilience Act. If the incident touches payment services, consumer credit or investment activity, additional notification routes may apply.
Record the regulatory decision, not just the regulatory outcome. "We considered notification and decided X because Y" is a defensible audit trail. Silence is not.
Minutes 50–60: Communicate What You Know, Not What You Hope
Stakeholder communications in the first hour should be short, factual and time-bound. State what has been observed, what has been contained and when the next update will arrive. Avoid speculation about root cause. Avoid promising a fix. Avoid attributing the incident to a vendor, a counterparty or an individual before evidence supports it.
Customers, counterparties and the board need different levels of detail, but they need the same discipline. One approved line, one next update time, one owner. This is also the moment to confirm that the incident record is being preserved for legal hold, because disputes and regulatory reviews often arrive months after the technical fix.
The Tabletop Checklist
| Minute | Action | Owner | Evidence artefact | Regulatory trigger |
|---|---|---|---|---|
| 0–10 | Declare incident and severity | Incident commander | Incident channel record | None yet |
| 0–10 | Freeze deployments and signing | Platform lead | Change log entry | None yet |
| 10–25 | Contain affected contracts or keys | Contract owner or security lead | Decision record and timestamp | Consider FCA/PRA notification |
| 25–40 | Preserve logs, receipts and configurations | Scribe and platform lead | Hash manifest and chain of custody | Start UK GDPR assessment |
| 40–50 | Record regulatory decision | Compliance lead | Decision memo | ICO 72-hour clock if applicable |
| 50–60 | Issue stakeholder update | Communications lead | Approved statement and next update time | None |
After the Hour: Handover, Not Heroics
The first hour ends with a handover to a longer incident process. Do not wait for root cause to begin the post-incident review. Immutability means the chain will still hold the evidence tomorrow. The operational response must hold its own record too. UK teams that treat the first sixty minutes as a governance exercise, rather than an engineering sprint, are the ones best placed to explain themselves when the scrutiny arrives.