Elevated Errors within NGWAF Dashboard

Incident

02 February 2024, 01:50 UTC

Elevated Errors within NGWAF Dashboard

Status: closed

Start: 31 January 2024, 18:09 UTC

End: 02 February 2024, 01:50 UTC

Duration: 1 day 7 hours 41 minutes

Affected Components:

Security Agents and Module Security Console & API Agent Downloads Rule and IP List updates Rule Processing Requests

Affected Groups:

All Public Users

Investigating

31 January 2024, 18:09 UTC

We are currently investigating elevated errors to our Request stats in our NGWAF Dashboard services.

Identified

31 January 2024, 18:36 UTC

Our engineers have identified the contributing factor and are applying a fix to Requests stats in our NGWAF Dashboard services.

Monitoring

31 January 2024, 19:17 UTC

Engineering has confirmed the impact to NGWAF Requests dashboard services has been mitigated.

Identified

01 February 2024, 21:10 UTC

Our engineers have observed a recurrence of this issue and identified the contributing factor. They are currently applying a fix to Requests stats in our NGWAF Dashboard services.

Update

01 February 2024, 22:21 UTC

Update: Agents appeared offline within the service dashboard from 21:25 UTC to 21:50 UTC (~25 minutes), however our investigations confirmed that they were still operational.

Engineering has confirmed the impact to our NGWAF Agent and Module services have been mitigated.

Engineers are continuing to apply mitigations for Request services within our dashboard.

Our network availability and all other services are unaffected by this incident.

Monitoring

01 February 2024, 23:42 UTC

Engineering has confirmed the impact to NGWAF Request dashboard services has been mitigated, we will resolve this status post after a period of monitoring.

Resolved

02 February 2024, 01:50 UTC

This event has been resolved.