Streaming Logs
Fastly Engineering detected a performance impact event affecting Streaming Logs in various POPs throughout our network. Customers may have experienced a delay or discarded log messages from 16:19 to 16:38 UTC.
This incident is resolved.
We have identified the cause of elevated errors in our Streaming Logs service and are deploying a fix.
Our network availability and all other services are unaffected by this incident
A fix has been implemented and we are monitoring the results.
Engineering has confirmed that the degraded performance for streaming log services has been fully restored. Customers may have experienced varying degrees of log loss or delays in log delivery from 20:50 UTC to 21:50 UTC as a result of this incident.
This incident has been resolved.
Fastly has identified an issue in which customers may see error messages in the Fastly UI for S3 and Kinesis endpoints indicating that a token is expired. However, this is isolated to the Fastly logging system is intermittently hitting a rate with the AWS Security Token Service (STS) API. This intermittent error does not appear to be causing log loss, but results in an error messaging in the UI. This only affects endpoints S3 and Kinesis endpoints that are using role-based authentication.
Fastly is currently working to resolve this intermittent error. All other locations and services are unaffected.
Engineering has deployed a fix to mitigate rate limiting errors and have observed a gradual recovery for streaming log services. We will continue to monitor the effects of the change and will post an update once services have been fully restored.
Our investigations into previously deployed mitigation measures has verified that our customers should no longer experience log loss as a result of this incident.
We investigated into the continued reports of error messages observed within the Fastly App and identified an error in the timing when reacquiring temporary credentials. We have confirmed that the impact to streaming log services has been resolved, and we do not see log loss in connection to this error message.
We are deploying an additional fix to resolve this Fastly App UI error message for our customers. We will post an update once all remaining error messages have been fully corrected.
A fix was deployed and we have observed role-based S3 and Kinesis logging endpoints returning to normal in the Fastly UI. Services that handle little to no traffic may see the error remaining until the logging system has successfully sent a batch of logs.
We're investigating elevated errors in Streaming Logs.
This issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
Engineering has confirmed that Streaming Logs has been fully restored. Customers sending log messages from the affected POPs would experience a similar proportion of log messages discarded from 15:20 to 15:53 UTC.
This incident is resolved.
Affected customers may have experienced impact to varying degrees and to a shorter duration than as set forth above.
We're investigating elevated errors in Streaming Logs.
All other locations and services are unaffected
This issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
A fix has been implemented and we are still monitoring the results. We have seen partial improvement for the Streaming Logs. Our Dublin (DUB) POP is still experiencing elevated errors at this time.
Engineering has confirmed that Streaming Logs has been fully restored. Customers sending log messages from the affected POPs would experience a similar proportion of log messages discarded from 16:50 to 19:35 UTC.
This incident is resolved.
Affected customers may have experienced impact to varying degrees and to a shorter duration than as set forth above.
We're currently investigating performance issues with our Streaming Logs service.
All other services are unaffected.
Impact is currently with S3 logging endpoints in US-East. Other logging endpoints are unaffected at this time.
This issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're currently investigating performance issues with our Streaming Logs service and NGWAF on compute. Customers on NGWAF may experience some of signal telemetry dropping.
All other services are unaffected.
This issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're currently investigating an increase in discards with our Streaming Logs service.
All other services are unaffected.
This issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We have investigated performance issues with our Streaming Logs service. Customers may have experienced delay of log delivery or discarding of logs during the affected time window.
All other services were unaffected.
This incident has been resolved.
On March 7th, 2023, Rapid7 will formally retire their Logentries product. As a result, Fastly will be completely removing the associated Logentries integration on the same date. We strongly advise all customers to replace their Logentries integration with a different logging integration (https://docs.fastly.com/en/guides/integrations#_logging-endpoints) in advance of this end-of-life date.
If you have a paid plan from Rapid7, you can choose to retain your Logentries data by following Rapid7’s archiving instructions (https://docs.logentries.com/docs/aws-s3-archiving).
This product has been retired.