Google Cloud has published its final incident report on the October 20 outage in its London region, saying backup generators at the data center hosting the europe-west2-b zone never received a signal to start when the site’s only active utility feed failed. The report, posted to the Google Cloud status page on October 27, says the primary impact was mitigated after 6 hours and 8 minutes.

Impact began at 23:00 US/Pacific on October 19, which was 07:00 BST on October 20 in London, when servers, storage and network hardware in the zone lost power. Workloads hosted entirely within europe-west2-b became unreachable, and services in the region’s other zones saw elevated errors as traffic failed over, according to Google. The final report lists 40 Google Cloud products as affected, including Compute Engine, Google Kubernetes Engine, Cloud Storage, Cloud SQL, BigQuery, Cloud Spanner, Cloud Run and Persistent Disk.

A chain of failures

According to the report, the facility was running on a single utility feed. Circuit breaker feeder B had been physically opened and locked out for corrective repairs following issues identified during a quarterly generator load test on October 14. At 22:53 US/Pacific on October 19, utility feed A failed because of an external underground cable fault.

The generators should then have taken over. Instead, the facility’s generator plant “failed to receive a signal to automatically start and take critical load,” Google wrote. The undervoltage sensing used by the programmable logic controller (PLC) that manages the switchover was wired upstream of the isolated feed B breaker, so it detected voltage that was not reaching the load and never issued a start command. Uninterruptible power supply (UPS) batteries “carried the critical load as designed until their battery reserves depleted,” the report said, and at about 23:00 the zone’s hardware lost power.

Why recovery took hours

Power returned well before services did. Standby utility power was restored at 23:54 US/Pacific, and utility power was re-established to the data halls at 00:20 on October 20. Clusters and network devices began recovering shortly afterward, but storage recovery was the main constraint: Google said the process “required a substantial period to finish prior to restoring service to customers.”

Engineers re-admitted traffic gradually from 03:45. Regional services and backend compute management systems were fully restored at 04:29, new virtual machine creation resumed at 04:46, and by 05:08 US/Pacific (13:08 BST) more than 99% of Compute Engine virtual machines were healthy, the report said.

The effects reached beyond the failed zone. As traffic shifted into the region’s two other zones, demand surges and control-plane timeouts caused HTTP 500 and 503 errors, queuing and degraded performance. Some customer failovers were unsuccessful because of capacity constraints and “difficulty in replicating the underlying storage mechanism,” and storage writes saw a transient error rate during the initial failover, according to Google.

Google has not said how many customers were affected. The Register reported on the day of the outage that Lloyds Bank, HSBC, Royal Mail, Rightmove, DPD, BT, Screwfix and Revolut were among those affected.

What Google is changing

The report lists corrective actions with completion dates or estimates:

  • Repair of the faulty cable and restoration of feed A by the utility provider, estimated for October 30, followed by a controlled transfer back to feeds A and B by October 31.
  • Reconfiguration and validation of the PLC, completed October 26, plus a comprehensive code review and multi-failure resiliency testing with vendors, estimated for October 31.
  • Full generator load transfer tests on both feeds after utility repairs, estimated for October 31.
  • Validation of PLC configurations at sites worldwide with similar topologies and manufacturers, estimated for November 15.
  • An updated emergency operating procedure for unexpected loss of the active utility source, completed October 26, which on-site staff now practice twice per shift.
  • A freeze on planned maintenance at sites with similar topologies and elevated approvals for non-standard configuration changes, estimated for October 30.

Google also said it would automate alert-to-communication for location-impacting incidents and expand platform testing for infrastructure-level events, without giving dates.

Why it matters

Customers running single-zone deployments took the full force of the outage, and the report is explicit about responsibility for those: Compute Engine zonal resources “require customers to provision failover or cross-zone recovery resources if needed.” During the incident, Google’s status updates suggested that affected customers try to migrate workloads to another zone or region if feasible.

The incident also shows the limits of spreading workloads across zones within one region. Customers with resources in the unaffected zones and in regional architectures still saw errors and failover problems as load moved, and some failovers failed for lack of capacity. Amro Al-Said Ahmad of Keele University told The Register the outage showed how cloud dependencies can cause cascading failures, arguing that resilience requires multi-region deployments, workload portability and tested disaster recovery.

The caveats

The root-cause account is Google’s own, and no independent investigation has been published. Figures also vary between sources: The Register put the start of the disruption at 06:08 UTC based on reporting on the day, while the status page header lists 38 affected services compared with 40 in the final report’s product list.

Corrections and tips: email the editor using the address on the Contact page.