Google Cloud Service Health (2024)

28 Mar 2022 08:30 PDT

Summary

On Tuesday, 22 March 2022, Google Cloud Networking experienced congestion on network infrastructure to and from the network edge locations [1] in Queretaro, Mexico, for a duration of 1 hour and 45 minutes, following multiple fiber cuts between the United States and Mexico. Affected customers may have experienced high latency, high retransmits and elevated errors.

We would like to apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.

[1] - https://cloud.google.com/vpc/docs/edge-locations

Root Cause

Google’s wide-area network is primarily comprised of infrastructure owned and operated by Google, although some physical links use fiber infrastructure which is owned and operated by third parties. Google has multiple resilient fiber paths and network points of presence serving users in Mexico.

Multiple simultaneous fiber cuts on diversely-routed paths, at the same time as a planned maintenance event on another diversely-routed path, resulted in a capacity shortfall in Google’s wide-area network between Queretaro, Mexico and the United States. This multiple failure event represents a rare case in which the resulting capacity shortfall was visible to Google's users.

Traffic was subsequently redirected to alternate links; however, there was insufficient capacity to serve all the redirected traffic, resulting in packet loss for user-facing network traffic to Google from users in Mexico.

Remediation and Prevention

Google’s automated repair mechanism detected the congestion on Tuesday, 22 March 2022, at 15:42 US/Pacific and redirected the traffic through alternate edge locations. However, the alternate link did not have sufficient capacity to serve all the redirected traffic, resulting in packet loss. The congestion was cleared when traffic was manually routed around the failed links on Tuesday, 22 March 2022, at 17:15 US/Pacific.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. However, this incident was the result of multiple concurrent failures on links which are designed to provide resilience; such failures, whilst very rare, are expected at a low frequency. We are taking the following steps to ensure we respond quickly to similar events in the future:

Improving the reliability of tooling that manually reroutes traffic in extreme multiple-failure scenarios.
Improving the diagnostic systems that help Google's Cloud Networking teams decide how to deal with extreme multiple-failure scenarios.

We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.