Our teams have detected an issue that is currently impacting Signal Sciences data collectors. New events from agents may not be processed by our back end services, agents may erroneously appear to be offline, and agents may not be receiving updated rules sets.

Customers may experience also delays in data available in the Signal Sciences console. 

Our engineers have identified the contributing factor and are applying a fix to our Agent Downloads service.

Engineering has deployed a fix and have confirmed a gradual recovery to our Agent Downloads service. We will continue to monitor until we’ve confirmed that customer experience has been fully restored.

Engineering has confirmed that our Agents Downloads service has been fully restored. Customers may have experienced intermittent issues viewing data in their Signal Sciences Dashboard. In addition, customers would also see a delay on request blocking or no blocking decisions made by Signal Sciences agents from as early as 16:40 UTC and an increase in performance degradation at 19:20 UTC to 22:03 UTC.

This incident is resolved.

Affected customers may have experienced impact to varying degrees and to a longer or shorter duration than as set forth above.

We are currently investigating elevated errors to our Request stats in our NGWAF Dashboard services.

Our engineers have identified the contributing factor and are applying a fix to Requests stats in our NGWAF Dashboard services.

Engineering has confirmed the impact to NGWAF Requests dashboard services has been mitigated.

Our engineers have observed a recurrence of this issue and identified the contributing factor. They are currently applying a fix to Requests stats in our NGWAF Dashboard services.

Update: Agents appeared offline within the service dashboard from 21:25 UTC to 21:50 UTC (~25 minutes), however our investigations confirmed that they were still operational.

• Engineering has confirmed the impact to our NGWAF Agent and Module services have been mitigated.
  

Engineers are continuing to apply mitigations for Request services within our dashboard.

Our network availability and all other services are unaffected by this incident.

Engineering has confirmed the impact to NGWAF Request dashboard services has been mitigated, we will resolve this status post after a period of monitoring.

This event has been resolved.

Our engineers have identified the contributing factor and are applying a fix to NGWAF@edge and Cloud WAF services.

All other locations and services are unaffected.

Engineering has confirmed the impact to Agents and Module, NGWAF@Edge, Cloud WAF has been mitigated.

Engineering has confirmed that Agents and Module, NGWAF@Edge, Cloud WAF has been fully restored. Customers may have experienced NGWAF agents not receiving updates from 17:58 to 18:42 UTC.

This incident is resolved.

On the 18th of December 2023 at 05:00 UTC, Fastly engineers inadvertently launched an incomplete agent release for our NGWAF services. We ask that any customer who deployed release version 4.49.1, to rollback to the previous version 4.49.0. 

There is no impact to customer services as a result of this inadvertent release. Our network availability and all other services are unaffected. 

Fastly engineers are aware of a recent change to some package file names due to a new release process. We are currently working on a fix for this issue.

All other services are unaffected.

This issue has been identified and a fix is being implemented.

Customers seeing errors pulling from our Alpine apk repository via apk can pull .apk packages manually from our apk site: https://apk.signalsciences.net/?prefix=3.17/main/ or download site: https://dl.signalsciences.net/?prefix=sigsci-agent/4.39.1/alpine/

Cloud Foundry customers can set the SIGSCI_DISABLE_CHECKSUM_INTEGRITY_CHECK environment variable as true to disable the package checksum verification during the install process until a fix is released.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Our teams have detected an issue that impacts Signal Sciences data collectors beginning at 00:53 UTC. Agents may appear offline in dashboard/API, agents will not receive rule updates; rate limit rules, templated rules and sites alerts will not take new traffic into account; new traffic will not appear in the dashboard/API. Customers may experience delays in data available in the Signal Sciences console.

This issue has been identified and a fix is being implemented. 

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Our teams have detected an issue that is currently impacting Signal Sciences data collectors. New events from agents may not be processed by our back end services, agents may erroneously appear to be offline, and agents may not be receiving updated rules sets. Customers may experience delays in data available in the Signal Sciences console.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Our teams have identified impact to the Signal Sciences Rule Builder from approximately 20:48 UTC to 21:15 UTC.

During this time new rule sets would not be pushed to agents. Customers sites would continue being protected by previous rule sets during this time.

Agents should have received updated rules sets once systems recovered.

This incident has been resolved.

We are currently investigating issues surrounding customers trying to download noarch packages via yum and receiving a 404 response. Direct downloads of these packages via https://dl.signalsciences.net/ can be used as a temporary workaround.

The issue has been identified and a fix is being implemented.

This Incident has now been resolved

We are continuing to investigate and monitor this issue as an isolated incident, please continue to send and monitor ticket updates via email (securitysupport@fastly.com) while we investigate.

Issue has been isolated and determined to be impacting customers whom have enabled SSO and have not accepted the invite sent from the console. Logging in directly from SSO will prevent customers from viewing, updating and interacting with tickets in the console. Fastly advises customers to accept the invite send from the console to resolve their issues with ticket logging and interactions.