Increased Errors with Multiple Fastly Services by a Third Party Service Provider
Increased Errors with Multiple Fastly Services by a Third Party Service Provider
12 June 2025, 17:55 UTC
12 June 2025, 17:55 UTC
We are seeing increased errors across multiple Fastly services that utilize a common third party service provider, unrelated to Fastly's Edge Cloud Network.
Fastly has launched acute incident response practices to investigate into this issue further in an effort to reduce the impact to our customers.
12 June 2025, 19:08 UTC
12 June 2025, 19:08 UTC
As our engineers determine the scope of impact our Customer Escalation Management team will be updating the impacted components on this status post.
12 June 2025, 19:19 UTC
12 June 2025, 19:19 UTC
We're aware of an ongoing Google Cloud incident, detailed on their status page, which is affecting several Fastly services. Customers may be experiencing increased latency due to impact to our KV Store.
Specifically, those trying to access their control plane via manage.fastly.com might notice that several key Observability features are not loading, including Billing, Historical Stats, Log Explorer & Insights, Origin Inspector, Real-Time Log Streaming, and Real-time Analytics. You may also find it difficult to engage with API endpoints. Additionally, customers might see elevated errors when accessing Fastly webpages and documented resources.
Our engineers are working to restore these services as a high priority and we will provide more information shortly.
You can monitor the Google Cloud incident here:
12 June 2025, 19:35 UTC
12 June 2025, 19:35 UTC
We've confirmed that customers won't receive status post notifications through their Support Slack channels during this incident.
However, we want to assure you that your ability to request a support case through our Support Slack services remains unaffected. You can still open new support cases as needed.
12 June 2025, 20:17 UTC
12 June 2025, 20:17 UTC
Google Cloud has communicated that they have successfully deployed mitigations to the majority of their services. We are now observing a gradual recovery of Fastly services.
We will continue to monitor this situation with the highest priority until all Fastly and customer services are fully restored.
We have continued to monitor the Google Cloud Status Page for the latest information by the third party service provider.
13 June 2025, 00:22 UTC
13 June 2025, 00:22 UTC
Engineering has confirmed that Fastly services have been fully restored. Customers may have experienced increased errors across multiple Fastly services that utilize a common third party service provider, unrelated to Fastly's Edge Cloud Network from 17:55 to 21:10 UTC on the 12th of June 2025.
While our services are back to full operation, our third-party supplier, Google Cloud, is still in the process of gradually restoring a few of their services as indicated on the Google Cloud Status Page tracking the remaining actions.
This incident is resolved.
Affected customers would have experienced impact to varying degrees and to a shorter duration than as set forth above.
To offer feedback on our status page, click "Give Feedback"
Status Post, Created Date/Time: 2025-06-12 18:46:25 UTC
Note: Our Customer Escalation Management team will update the start date and time of the initial "investigating" status post upon the resolution of this incident. This update is meant to provide our customers and their end users with a potential impact window. The date and time mentioned in the message above indicates when the status post was requested by our Acute Incident Response team.
13 June 2025, 16:30 UTC
13 June 2025, 16:30 UTC
Incident Update: Full Recovery and Root Cause Identified
Current Status: Resolved
We're confirming that all services impacted during yesterday's incident have fully recovered. Our teams continuously monitored the situation and verified the stability of all affected systems.
Root Cause Analysis
Our investigation has confirmed that the increased errors and latency across our KV Store, control plane, and certain Observability features were a direct result of the Google Cloud disruption on the 12th of June 2025.
Google has publicly shared a post-mortem regarding the incident. It includes specific information about the root cause—an invalid automated quota update to their API management system—along with their mitigation strategies and steps they're taking for future prevention.
Next Steps
For a comprehensive understanding of the incident's origin, mitigation, and Google's preventative measures, we encourage customers to review the official Google Cloud Status Page. This provides a full breakdown from their perspective.
Additionally, our Product and Engineering teams will thoroughly review all Google Cloud Platform (GCP) reports to determine mitigation strategies and ensure they are adopted as part of our long-term preventive measures to ensure our systems are more resilient to third-party outages of this kind in the future.
We appreciate your patience and understanding as we worked through this third-party event. Our focus remains on providing reliable and high-performance services.