Elevated Errors for Hyderabad (HYD), Manila (MNL), Bogota (BOG), Johannesburg (JNB), Singapore (SIN) for Compute Services

Incident

29 January 2025, 21:58 UTC

Elevated Errors for Hyderabad (HYD), Manila (MNL), Bogota (BOG), Johannesburg (JNB), Singapore (SIN) for Compute Services

Status: closed

Start: 28 January 2025, 04:21 UTC

End: 29 January 2025, 21:58 UTC

Duration: 1 day 17 hours 37 minutes

Affected Components:

Fastly Platform Fastly Application Asia South America Africa Bogota (BOG) API & Configuration Management Notification Center Johannesburg (JNB) Hyderabad (HYD) Manila (MNL) Osaka (ITM) Singapore (SIN) Tokyo (NRT) Tokyo (TYO) API Services Configuration Management Services

Affected Groups:

All Public Users

Investigating

28 January 2025, 04:21 UTC

We are investigating elevated errors to our Hyderabad (HYD), Manila (MNL), Bogota (BOG), Johannesburg (JNB), Singapore (SIN) POPs for Compute Services and the Fastly Application.

All other products and services are unaffected by this incident.

Identified

28 January 2025, 09:31 UTC

Our engineers have identified the contributing factor and are applying a fix to our Hyderabad (HYD), Manila (MNL), Bogota (BOG), Johannesburg (JNB), Singapore (SIN) POPs for Compute Services.

All other locations and services are unaffected.

Identified

28 January 2025, 10:27 UTC

Our engineers have identified the contributing factor and are continuing to apply the mitigation strategy to our Hyderabad (HYD), Manila (MNL), Bogota (BOG), Johannesburg (JNB), Singapore (SIN) POPs for Compute services.

All other locations and services are unaffected.

Monitoring

28 January 2025, 17:14 UTC

Engineering has deployed a fix and have confirmed a gradual recovery to our Hyderabad (HYD), Manila (MNL), Bogota (BOG), Johannesburg (JNB), Singapore (SIN) POPs and Fastly Application service. We will continue to monitor until we’ve confirmed that customer experience has been fully restored.

After further investigation, Engineering observed impact to our Tokyo (NRT), Tokyo (TYO), and Osaka (ITM) POPs.

Resolved

29 January 2025, 21:58 UTC

Engineering has confirmed that Hyderabad (HYD), Manila (MNL), Osaka (ITM), Tokyo (NRT), Tokyo (TYO), Bogota (BOG), Fastly Application, Johannesburg (JNB), Singapore (SIN) POPs and service have been fully restored. Compute customers may have experienced errors for requests that utilized geolocation as well as login issues with the Fastly Application from the 28th of January at 04:21 to 18:30 UTC.

Affected customers may have experienced impact to varying degrees and to a shorter duration than set forth above.

As the fix was deployed Engineering made additional adjustments to address impact observed to our Tokyo (NRT), Tokyo (TYO), and Osaka (ITM) POPs. Customer during this time may have experienced an increase in error rates from the 28th of January at 23:00 to the 29th of January 05:00 UTC.

Affected customers may have experienced impact to varying degrees and to a shorter duration than set forth above.

This incident is resolved.

To offer feedback on our status page, click "Give Feedback"

Status Post, Created Date/Time: 2025-01-28 09:30:53 UTC

Note: Our Customer Escalation Management team will update the start date and time of the initial "investigating" status post upon the resolution of this incident. This update is meant to provide our customers and their end users with a potential impact window. The date and time mentioned in the message above indicates when the status post was requested by our Acute Incident Response team.