Agents and Module
Our teams have detected an issue that is currently impacting Signal Sciences data collectors. New events from agents may not be processed by our back end services, agents may erroneously appear to be offline, and agents may not be receiving updated rules sets.
Customers may experience also delays in data available in the Signal Sciences console.
Our engineers have identified the contributing factor and are applying a fix to our Agent Downloads service.
Engineering has deployed a fix and have confirmed a gradual recovery to our Agent Downloads service. We will continue to monitor until we’ve confirmed that customer experience has been fully restored.
Engineering has confirmed that our Agents Downloads service has been fully restored. Customers may have experienced intermittent issues viewing data in their Signal Sciences Dashboard. In addition, customers would also see a delay on request blocking or no blocking decisions made by Signal Sciences agents from as early as 16:40 UTC and an increase in performance degradation at 19:20 UTC to 22:03 UTC.
This incident is resolved.
Affected customers may have experienced impact to varying degrees and to a longer or shorter duration than as set forth above.
We are currently investigating elevated errors to our Request stats in our NGWAF Dashboard services.
Our engineers have identified the contributing factor and are applying a fix to Requests stats in our NGWAF Dashboard services.
Engineering has confirmed the impact to NGWAF Requests dashboard services has been mitigated.
Our engineers have observed a recurrence of this issue and identified the contributing factor. They are currently applying a fix to Requests stats in our NGWAF Dashboard services.
Update: Agents appeared offline within the service dashboard from 21:25 UTC to 21:50 UTC (~25 minutes), however our investigations confirmed that they were still operational.
- Engineering has confirmed the impact to our NGWAF Agent and Module services have been mitigated.
Engineers are continuing to apply mitigations for Request services within our dashboard.
Our network availability and all other services are unaffected by this incident.
Engineering has confirmed the impact to NGWAF Request dashboard services has been mitigated, we will resolve this status post after a period of monitoring.
This event has been resolved.
Our engineers have identified the contributing factor and are applying a fix to NGWAF@edge and Cloud WAF services.
All other locations and services are unaffected.
Engineering has confirmed the impact to Agents and Module, NGWAF@Edge, Cloud WAF has been mitigated.
Engineering has confirmed that Agents and Module, NGWAF@Edge, Cloud WAF has been fully restored. Customers may have experienced NGWAF agents not receiving updates from 17:58 to 18:42 UTC.
This incident is resolved.
On the 18th of December 2023 at 05:00 UTC, Fastly engineers inadvertently launched an incomplete agent release for our NGWAF services. We ask that any customer who deployed release version 4.49.1, to rollback to the previous version 4.49.0.
There is no impact to customer services as a result of this inadvertent release. Our network availability and all other services are unaffected.
Fastly is investigating possible performance impact to NGWAF Rule and IP List updates.
Customers may experience intermittent delays when applying changes to their rules and/or IP list.
All other services are unaffected.
Fastly engineers have deployed mitigation measures and services are available. Customers may continue to experience intermittent periods of delays, and Fastly will continue to monitor NGWAF rules and IP list services for stability.
All other services are unaffected.
For additional information, or to report additional impact, please contact support at https://support.fastly.com.
Fastly Engineering observed the NGWAF decisioning engine not checking new requests against customer rules. New IP flags are also delayed during this time.
All other services are unaffected.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
From 15 June 2023, 19:48 UTC to 15 June 2023, 20:20 UTC Fastly Engineering observed the NGWAF decisioning engine not check new requests against customer rules. New IP flags were also delayed during this time. All other services during this time were unaffected.
Fastly is currently experiencing issues with NGWAF GeoIP country identification. Customers may experience instances in which NGWAF labels requests with an incorrect region. For further information, please contact support@fastly.com.
This issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Fastly engineers are aware of a recent change to some package file names due to a new release process. We are currently working on a fix for this issue.
All other services are unaffected.
This issue has been identified and a fix is being implemented.
Customers seeing errors pulling from our Alpine apk repository via apk can pull .apk packages manually from our apk site: https://apk.signalsciences.net/?prefix=3.17/main/ or download site: https://dl.signalsciences.net/?prefix=sigsci-agent/4.39.1/alpine/
Cloud Foundry customers can set the SIGSCI_DISABLE_CHECKSUM_INTEGRITY_CHECK environment variable as true to disable the package checksum verification during the install process until a fix is released.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
This issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.