We are investigating elevated errors for evaluating Fastly Alerts configured on Domain Inspector and Origin Inspector metrics.

Our engineers have identified the contributing factor and are applying a fix our Domain Inspector, Origin Inspector.

Engineering has confirmed the impact to Domain Inspector, Origin Inspector has been mitigated.

Engineering has confirmed that impact to the Domain Inspector and Origin Inspector metrics has been fully restored. Customers may have experienced degraded service for Fastly Alerts from 21:42 to 23:05 UTC.

This incident is resolved.

Fastly Engineering detected a performance impact event affecting Streaming Logs in various POPs throughout our network. Customers may have experienced a delay or discarded log messages from 16:19 to 16:38 UTC.

This incident is resolved.

We're currently investigating performance issues with our Real Time Analytics and Historical Stats services at our Columbus (CMH) Point of Presence (POP).

All other services are unaffected.

Our engineers have identified the contributing factor and are applying a fix to Real Time Analytics and Historical Stats services at our Columbus (CMH) Point of Presence (POP).

All other locations and services are unaffected.

Engineering has confirmed the impact to Real Time Analytics and Historical Stats services at our Columbus (CMH) Point of Presence (POP) has been mitigated.

Engineering has confirmed that Real Time Analytics and Historical Stats services at our Columbus (CMH) Point of Presence (POP) has been fully restored. Customers may have experienced a delay or dropped observability product data from. 01:15 to 04:05 UTC.

This incident is resolved.

Affected customers may have experienced impact to varying degrees and to a shorter duration than as set forth above.

We're currently investigating performance issues with our Origin Inspector service.

Our engineers have identified the contributing factor and are applying a fix to Origin Inspector

Engineering has deployed a fix and have confirmed a gradual recovery to Origin Inspector. We will continue to monitor until we’ve confirmed that customer experience has been fully restored.

Engineering has confirmed that our Origin Inspector service has been fully restored. Customers may have experienced missing historical Origin Inspector metrics in both the APIs and web interface from the 14th December 2023 at 00:00 to the 20th of December 2023 at 03:18 UTC.

This incident is resolved.

We're currently investigating performance issues with our Real Time Analytics and Historical Stats services only in our Lisbon (LIS) data center.

All other services and data centers are unaffected.

This event has been resolved.

We have identified the cause of elevated errors in our Streaming Logs service and are deploying a fix. 

Our network availability and all other services are unaffected by this incident 

A fix has been implemented and we are monitoring the results.

Engineering has confirmed that the degraded performance for streaming log services has been fully restored. Customers may have experienced varying degrees of log loss or delays in log delivery from 20:50 UTC to 21:50 UTC as a result of this incident.

This incident has been resolved.

Fastly has identified an issue in which customers may see error messages in the Fastly UI for S3 and Kinesis endpoints indicating that a token is expired. However, this is isolated to the Fastly logging system is intermittently hitting a rate with the AWS Security Token Service (STS) API. This intermittent error does not appear to be causing log loss, but results in an error messaging in the UI. This only affects endpoints S3 and Kinesis endpoints that are using role-based authentication. 

Fastly is currently working to resolve this intermittent error. All other locations and services are unaffected. 

Engineering has deployed a fix to mitigate rate limiting errors and have observed a gradual recovery for streaming log services. We will continue to monitor the effects of the change and will post an update once services have been fully restored.

Our investigations into previously deployed mitigation measures has verified that our customers should no longer experience log loss as a result of this incident.

We investigated into the continued reports of error messages observed within the Fastly App and identified an error in the timing when reacquiring temporary credentials. We have confirmed that the impact to streaming log services has been resolved, and we do not see log loss in connection to this error message.

We are deploying an additional fix to resolve this Fastly App UI error message for our customers. We will post an update once all remaining error messages have been fully corrected.

A fix was deployed and we have observed role-based S3 and Kinesis logging endpoints returning to normal in the Fastly UI. Services that handle little to no traffic may see the error remaining until the logging system has successfully sent a batch of logs.

We're investigating elevated errors in Historical Stats.

All other locations and services are unaffected

We're still investigating elevated errors in Historical Stats.

During this time, customers may experience 5xx errors when accessing the Historical Stats endpoint.

All other locations and services are unaffected

This issue has been identified and a fix is being implemented. 

A fix has been implemented and we are monitoring the results.

We've observed a significant decrease in 5xx errors, but are currently monitoring for logs that may have failed to deliver during this incident.

This incident has been resolved and no further impacts have been observed. 

We're investigating elevated errors in Streaming Logs. 

This issue has been identified and a fix is being implemented. 

A fix has been implemented and we are monitoring the results.

Engineering has confirmed that Streaming Logs has been fully restored. Customers sending log messages from the affected POPs would experience a similar proportion of log messages discarded from 15:20 to 15:53 UTC.

This incident is resolved.

Affected customers may have experienced impact to varying degrees and to a shorter duration than as set forth above.

We're investigating elevated errors in Streaming Logs. 

All other locations and services are unaffected

This issue has been identified and a fix is being implemented. 

A fix has been implemented and we are monitoring the results.

A fix has been implemented and we are still monitoring the results. We have seen partial improvement for the Streaming Logs. Our Dublin (DUB) POP is still experiencing elevated errors at this time.

Engineering has confirmed that Streaming Logs has been fully restored. Customers sending log messages from the affected POPs would experience a similar proportion of log messages discarded from 16:50 to 19:35 UTC.

This incident is resolved.

Affected customers may have experienced impact to varying degrees and to a shorter duration than as set forth above.