Prism - Intermittent Network Instability Affecting Portions of Ashburn – Incident details

Experiencing partially degraded performance

Intermittent Network Instability Affecting Portions of Ashburn

Monitoring
Degraded performance
Started 20 days ago

Affected

Database Nodes

Operational from 6:50 AM to 8:27 PM, Degraded performance from 8:27 PM to 12:00 AM

Ashburn Database Functions

Operational from 6:50 AM to 8:27 PM, Degraded performance from 8:27 PM to 12:00 AM

ASH - Ashburn, US (US-East-1)

Degraded performance from 6:50 AM to 8:27 PM, Operational from 6:50 AM to 8:27 PM, Degraded performance from 8:27 PM to 8:35 AM, Operational from 8:27 PM to 8:35 AM, Degraded performance from 8:35 AM to 1:04 AM, Operational from 8:35 AM to 1:04 AM, Degraded performance from 1:04 AM to 12:00 AM

ASH-1

Operational from 6:50 AM to 1:04 AM, Degraded performance from 1:04 AM to 12:00 AM

ASH-2

Degraded performance from 6:50 AM to 12:00 AM

ASH-3

Degraded performance from 6:50 AM to 12:00 AM

Updates
  • Update
    Update

    Connection has been established, we will schedule a migration sometime this weekend for all affected nodes except ASH-1, ASH-3, ASH-7, and ASH-9

  • Monitoring
    Monitoring

    Network has stabilized a lot but still intermittently experiences issues for a select portion of users. We are currently waiting to be onboarded by DataPacket.

  • Update
    Update

    This has been postponed as Cloudflare is not currently processing LOAs due to reduced staffing over the holiday period. We expect to proceed once they resume normal operations around January 5th.

  • Identified
    Identified

    We've secured an alternative tunnel (Cosmic was too slow, so we made other arrangements) and plan to cut over tomorrow. We'll announce the specific time shortly. This should resolve all current issues, though it will serve as a temporary fix until our CDN77 cross connect is in place.

  • Monitoring
    Monitoring

    Since moving select chunks of targeted infrastructure over to Cloudflare magic transit (via CosmicGuard) on isolated infrastructure, the network has stabilized. We will continue to monitor the situation, and still plan to bring DataPacket & Cosmic into our main blend.

  • Investigating
    Investigating

    Due to sustained DDoS attacks reaching upwards of 10 Tbps against our Ashburn infrastructure, some customers may notice intermittent packet loss on certain routes (especially European ones) impacting ASH-1 through ASH-20. This is likely a result of our upstream + inline DDoS mitigation provider (GlobalSecureLayer) experiencing saturation at their scrubbing centers during the attacks (they are actually doing a fairly good job at mitigating the attacks with what's available. This is an astronomical amount of bandwidth though).

    Under normal conditions, we can reduce the blast radius by adjusting mitigation trigger thresholds for non-targets to avoid scrubbing paths. In this case, doing so would introduce unacceptable risk to overall network stability, so we’re keeping the more conservative posture in place.

    We understand that consistent connectivity is critical for many workloads. We maintain backup infrastructure in a separate datacenter, and we can move a limited/select number of servers over as a temporary mitigation for customers who are most impacted.

    To add redundancy, we’re bringing CDN77 / DataPacket online as a backup upstream, but provisioning is blocked until their team returns from the Christmas holiday in January. In parallel, we’re also in contact with CosmicGuard as an emergency failover option.

    We’ll continue to post updates as we make progress and appreciate your patience.