Duration: 6 minutes

Affected Components: VIN-52 (Formerly ASH-52)

May 29, 08:59:58 GMT+0
Investigating - We are currently investigating this incident..

May 29, 09:05:38 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours

Affected Components: FRA-1, ASH-13, ASH-28, ASH-29, FRA-5, FRA-3, FRA-2, ASH-2, FRA-6, ASH-30, SGP-1, ASH-1, ASH-5, ASH-6, ASH-3, ASH-10

Apr 5, 16:00:00 GMT+0
Identified - Hello, We will be performing emergency migrations on the following machines: **Ashburn:** ASH-1, ASH-2, ASH-3, ASH-5, ASH-6, ASH-10, ASH-13, ASH-28, ASH-29, ASH-30 **Frankfurt:** FRA-1, FRA-2, FRA-3, FRA-4, FRA-5, FRA-6 **Dallas:** DAL-3 **Singapore:** SGP-1 These machines are running legacy deployments that are incompatible with our automated infrastructure management platform. Each machine will be manually reinstalled and redeployed to bring it in line with our current stack. **This process may result in an IP address change for affected servers.** The maintenance window is scheduled for **3 hours**, though the individual impact per server will range from approximately **5 to 30 minutes** of downtime. For servers in **Ashburn** and **Frankfurt**, users have the option to migrate ahead of time through the panel. We recommend doing so at your earliest convenience so you can make any necessary adjustments, such as updating DNS records or firewall rules, on your own schedule..

Apr 5, 16:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 5, 19:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 minute

Affected Components: ASH-46

Apr 2, 04:20:37 GMT+0
Resolved - This incident has been resolved..

Apr 2, 04:19:47 GMT+0
Investigating - We are currently investigating this incident..

Duration: 36 minutes

Affected Components: NAE-11

Mar 20, 20:38:41 GMT+0
Investigating - We are currently investigating this incident. .

Mar 20, 21:14:43 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours and 44 minutes

Affected Components: ASH-3

Mar 17, 20:23:39 GMT+0
Investigating - We are currently investigating this incident..

Mar 18, 00:07:30 GMT+0
Resolved - This incident has been resolved. This outage was caused by a significant cooler failure which has since been replaced..

Duration: 58 minutes

Affected Components: ASH-4, ASH-2, ASH-5, ASH-3, ASH-6, ASH-7

Mar 14, 19:09:51 GMT+0
Investigating - We are currently investigating this incident..

Mar 14, 19:12:05 GMT+0
Identified - ASH-7 is back.

Mar 14, 19:12:49 GMT+0
Resolved - Everything is back.

Mar 14, 19:18:39 GMT+0
Identified - Down again. We're working with our upstream provider to resolve this. Estimated resolution is \~5 minutes..

Mar 14, 20:07:43 GMT+0
Resolved - This incident has been resolved. We do not expect future blips. A postmortem will be posted as soon as we have word on what went wrong from our (legacy) Ashburn upstream..

Affected Components: , SGP-1, SIN - Singapore, SG →

Mar 8, 05:47:30 GMT+0
Investigating - We are currently investigating this incident. Node SGP-1 will go offline from 10am to 11am EST tomorrow as we perform a hardware swap.

Affected Components: Ashburn Mass Offsite Storage

Feb 23, 04:57:26 GMT+0
Investigating - We are currently investigating this incident. The affected infrastructure includes nodes ASH-1 through ASH-45 and NYC-1 through NYC-18, as well as all regional assets within the Dallas (DAL), Miami (MIA), Los Angeles (LAX), and London (LON) sectors..

Duration: 1 hour and 26 minutes

Affected Components: Game Panel (Replica Aggregate), FRA-1, , FRA-5, FRA-3, FRA-2, FRA-8, FRA-7, FRA-6, FRA-3, , Billing Panel, FRA - Frankfurt, DE (EU-Central) → Core Webfront →

Feb 19, 18:03:26 GMT+0
Investigating - Our upstream provider, Hetzner, has unexpectedly blocked our Frankfurt routers, citing excessive IPv6 traffic (we don't even use IPv6). We are actively disputing this, as our Minecraft hosting services rely almost exclusively on IPv4. Our team is working quickly to resolve the situation, and we expect to have our site and the Frankfurt nodes back online within the hour. Once services are fully stabilized, we will share details regarding compensation for the downtime. Thank you for your patience..

Feb 19, 19:29:45 GMT+0
Resolved - This incident has been resolved..

Duration: 1 day, 3 hours and 58 minutes

Affected Components: ASH-4

Feb 14, 03:14:25 GMT+0
Investigating - We are currently investigating this incident..

Feb 14, 19:38:48 GMT+0
Identified - A dimm (ram) experienced a catastrophic failure and because of the \[global memory shortage\](<https://en.wikipedia.org/wiki/2024–present%5Fglobal%5Fmemory%5Fsupply%5Fshortage>) we haven't been able to yoink one from repair stock a spare node since there are none. We've scheduled a tech appointment for later today to move the drives to a separate machine. Failures like this are extremely rare and we understand how unexpected downtime can hurt communities, please make a ticket to discuss compensation Under normal circumstances, we would have been able to bring the node back online on another machine within the same location in roughly an hour. Unfortunately, we currently have no spare memory stock available in this datacenter, so the data will need to be transferred to a node in a different location (Ashburn1 (Equinix DC3) -> Ashburn2 (Coresite)), which will result in an IP change. This node was scheduled to be migrated to our new Ashburn PoP in one day, so an IP change would have occurred regardless. The failure itself is simply unfortunate timing, as it could have happened on any other day..

Feb 15, 04:22:53 GMT+0
Identified - Access to the node has been restored (spare memory was located) and \~2TB of server data is being migrated to another node for future stability..

Feb 15, 07:12:30 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours

Affected Components: FRA-2, FRA-1, , FRA-5, FRA-3, FRA-8, FRA-3, Frankfurt Database Functions, Frankfurt Mass Offsite Storage, FRA - Frankfurt, DE (EU-Central) →

Feb 13, 13:00:00 GMT+0
Identified - We will be performing network maintenance on our Frankfurt infrastructure to onboard CDN77 as an additional transit provider and migrate the remaining nodes to Prism-owned IP subnets. This change is part of a broader network upgrade and will significantly improve DDoS protection, routing reliability, and latency consistency for services hosted in Frankfurt. What’s changing * All affected services will receive new IP addresses * Prism subdomains will update automatically. No action is required if you connect via a Prism hostname * If you connect using a raw IP address, you will need to update it once the migration is complete Impact * Brief network interruptions may occur during the IP transition * No data loss is expected * Services will resume automatically once routing propagates Benefits * Improved resilience against DDoS attacks * More stable and predictable routing * Better overall network performance in Frankfurt We will email all affected users with details and updated connection information ahead of the change..

Feb 13, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 13, 15:00:00 GMT+0
Completed - Maintenance has completed successfully.

Affected Components: FRA-1, FRA-5, FRA-3

Feb 1, 22:26:14 GMT+0
Investigating - We are currently investigating latency issues in Frankfurt affecting nodes FRA-1, FRA-3, and FRA-5\. This is due to pipe saturation on our Hetzner uplinks. We are working to resolve this by deploying an additional router and tunnel in the Frankfurt POP, which we expect to be online by tomorrow..

Duration: 4 hours

Affected Components: ASH-3, ASH-44, , FRA-2, ASH-34, NYC-21, Ashburn Mass Offsite Storage, NYC-23, NYC-25, NYC-22, NYC-20, VIN-51 (Formerly ASH-51), ASH-43, NYC-26, NYC-28, , ASH-40, ASH-31, FRA-1, ASH-41, ASH-4, ASH-20, LON-3, , NYC-27, VIN-49 (Formerly ASH-49), SEG-4, NAE-12, VIN-52 (Formerly ASH-52), , DFW-4, ASH-13, NAE-16, NAE-17, VIN-48 (Formerly ASH-48), ASH-26, SGP-1, NAE-15, VIN-47 (Formerly ASH-47), NYC-18, NYC-24, NYC-19, , ASH-1, ASH-38, LAX-1, ASH-21, DAL-2, ASH-17, LON-2, DAL-1, ASH-24, ASH-27, , ASH-28, ASH-32, ASH-35, ASH-36, ASH-37, ASH-46, ASH-33, ASH-29, NYC-1, NYC-4, NYC-5, DFW-5, FRA-5, ASH-42, NYC-6, SEG-9, NYC-8, MIA-1, FRA-3, NYC-2, ASH-45, LAX-2, NYC-10, NYC-9, ASH-2, ASH-12, ASH-15, ASH-16, ASH-6, ASH-39, SYD-1, NYC-3, NYC-7, FRA-8, FRA-7, NAE-13, VIN-50 (Formerly ASH-50), DFW-6, FRA-6, ASH-30, NAE-11, FRA-3, NAE-14, ASH-22, ASH-11, ASH-9, Frankfurt Mass Offsite Storage, , , ASH-5, ASH-18, ASH-19, LON-1, , ASH-23, ASH-7, ASH-10, ASH-25, ASH-14, , LAX - Los Angeles, US (US-West) → SIN - Singapore, SG → SYD - Sydney, AU → Backup Machines → NYC - New York, US (US-East-2) → FRA - Frankfurt, DE (EU-Central) → ASH - Ashburn, US (US-East-1) → DAL - Dallas, US → LON - London, GB (EU-Northwest) → MIA - Miami, US (US-Southeast) →

Jan 31, 19:00:00 GMT+0
Identified - We will be performing a major overhaul of our backup infrastructure to significantly improve backup and restore speeds, increase overall reliability, and introduce data deduplication to reduce storage overhead. This maintenance involves migrating several hundred terabytes of existing backup data to our new storage system. Due to the scale of the migration, this process may take an extended period to complete. During this time, backups may run slower than usual or be temporarily paused, but live services will remain unaffected. We will closely monitor the migration and provide updates if there are any changes to expected behavior. Once completed, customers can expect faster backups, quicker restores, and a more resilient backup platform going forward. Thank you for your patience while we make these long-term improvements..

Jan 31, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 31, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour and 54 minutes

Affected Components: FRA-3

Jan 28, 20:59:04 GMT+0
Investigating - We are currently investigating this incident..

Jan 28, 21:25:12 GMT+0
Identified - The physical network interface overheated and died. A tech will replace it when available..

Jan 28, 22:53:17 GMT+0
Resolved - This was resolved 1 hour ago.

Duration: 3 minutes

Affected Components: Game Panel (Replica Aggregate)

Jan 25, 07:05:53 GMT+0
Investigating - We are currently investigating this incident..

Jan 25, 07:07:50 GMT+0
Investigating - Panel API is completely unreachable .

Jan 25, 07:09:18 GMT+0
Resolved - This incident has been resolved. This outage was caused by code written for an older version of the panel being merged with the main panel codebase, which caused a cascading failure as replicas rebuilt. The cluster has been rolled back..

Duration: 1 day, 3 hours and 13 minutes

Affected Components: ASH-3, ASH-43, VIN-51 (Formerly ASH-51), NAE-15, ASH-41, ASH-44, ASH-40, ASH-22, VIN-47 (Formerly ASH-47), ASH-31, ASH-20, LON-3, NAE-12, VIN-52 (Formerly ASH-52), , , DFW-4, ASH-34, ASH-13, NAE-16, NAE-17, VIN-48 (Formerly ASH-48), ASH-1, ASH-26, , VIN-49 (Formerly ASH-49), LAX-1, ASH-21, DAL-2, ASH-17, LON-2, DAL-1, ASH-24, ASH-27, ASH-28, ASH-32, ASH-35, ASH-36, ASH-37, ASH-38, ASH-46, ASH-33, ASH-29, NYC-1, NYC-4, NYC-5, DFW-5, ASH-42, NYC-6, ASH-4, SEG-9, NYC-8, MIA-1, SEG-4, NYC-2, ASH-45, LAX-2, NYC-10, NYC-9, ASH-2, ASH-12, ASH-15, ASH-16, ASH-6, ASH-39, SYD-1, , NYC-3, Ashburn Mass Offsite Storage, NYC-7, NAE-13, VIN-50 (Formerly ASH-50), DFW-6, ASH-30, NAE-11, NAE-14, ASH-11, ASH-9, , , ASH-5, ASH-18, ASH-19, LON-1, , ASH-23, ASH-7, ASH-10, ASH-25, ASH-14, , LAX - Los Angeles, US (US-West) → Backup Machines → SYD - Sydney, AU → NYC - New York, US (US-East-2) → ASH - Ashburn, US (US-East-1) → DAL - Dallas, US → LON - London, GB (EU-Northwest) → MIA - Miami, US (US-Southeast) →

Jan 21, 02:59:10 GMT+0
Investigating - Ceph doesn't seem to be happy about loosing 3 drives..

Jan 22, 06:12:40 GMT+0
Resolved - Cluster is healthy..

Duration: 5 days, 15 hours and 4 minutes

Affected Components: NYC-1, NYC-4, NYC-5, NYC-6, NYC-8, NYC-2, NYC-10, NYC-9, NYC-3, NYC-7

Jan 25, 21:40:13 GMT+0
Resolved - This incident has been resolved..

Jan 20, 06:36:33 GMT+0
Investigating - We are currently investigating this incident. A small handful of players may experience unusually high latency due to mutual peering..

Duration: 12 minutes

Affected Components: Game Panel (Replica Aggregate)

Jan 12, 03:54:56 GMT+0
Investigating - We are currently experiencing intermittent failures with our email delivery system. Some customers may encounter delays or errors when attempting to send emails through our platform. Following a very successful marketing campaign this weekend, we are being heavily rate limited by our email provider due to the increased volume of signups and communications. Our engineering team has identified the issue and is actively working on a resolution. Affected users may see temporary bounce backs or delivery delays during this time. Thankfully, this does not affect any server purchases or setup processes, as we have temporarily disabled required email verification. We will provide an update once the issue has been fully resolved. We appreciate your patience and apologize for any inconvenience..

Jan 12, 04:07:12 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour

Jan 5, 17:30:00 GMT+0
Identified - We will be migrating all services off **ASH-8** following a recent hardware failure. ASH-8 is legacy infrastructure and will be fully retired to prevent further instability. **Impact:** During the migration, affected servers will experience downtime while data is transferred and the service is brought back online. **This migration will require an IP change**. **Action required:** Please open a support ticket so we can schedule a migration time that works for you, coordinate any DNS/allowlist changes, and discuss compensation. If we do not hear from you, we will **automatically migrate your server on Monday during this maintenance window**..

Jan 5, 17:30:01 GMT+0
Identified - Maintenance is now in progress.

Jan 5, 18:30:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 minute

Affected Components: DAL-2, DAL-1, , DAL - Dallas, US →

Jan 5, 02:13:02 GMT+0
Investigating - We are currently investigating this incident..

Jan 5, 02:13:46 GMT+0
Resolved - This incident has been resolved. Thank you nullrouting..

Duration: 4 minutes

Affected Components: Game Panel (Replica Aggregate)

Jan 4, 03:22:28 GMT+0
Investigating - We are currently investigating this incident..

Jan 4, 03:26:40 GMT+0
Resolved - Traefik load balancer caching issue. Resolved..

Duration: 1 hour

Affected Components: ASH-3, ASH-13, ASH-22, ASH-20, ASH-1, ASH-26, LAX-1, ASH-21, DAL-2, ASH-17, DAL-1, ASH-24, ASH-27, , ASH-28, ASH-32, ASH-35, ASH-36, ASH-37, ASH-38, ASH-41, ASH-46, ASH-33, ASH-45, ASH-40, ASH-31, ASH-29, ASH-34, NYC-1, NYC-4, NYC-5, DFW-5, ASH-42, NYC-6, ASH-4, SEG-9, NYC-8, MIA-1, ASH-43, DFW-4, SEG-4, NYC-2, LAX-2, NYC-10, NYC-9, ASH-44, ASH-2, ASH-12, ASH-15, ASH-16, ASH-6, ASH-39, , NYC-3, Ashburn Mass Offsite Storage, NYC-7, DFW-6, ASH-30, ASH-11, ASH-9, , , ASH-5, ASH-18, ASH-19, ASH-23, ASH-7, ASH-10, ASH-25, ASH-14, , LAX - Los Angeles, US (US-West) → NYC - New York, US (US-East-2) → ASH - Ashburn, US (US-East-1) → DAL - Dallas, US → MIA - Miami, US (US-Southeast) →

Jan 3, 02:00:00 GMT+0
Identified - During this maintenance window, we will be migrating our NA database infrastructure and implementing onsite SSD caches at each location to increase backup throughput..

Jan 3, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 3, 03:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: FRA-2, FRA-1, LON-3, LON-2, , FRA-5, FRA-3, FRA-8, FRA-7, FRA-6, FRA-3, Frankfurt Mass Offsite Storage, LON-1, , FRA - Frankfurt, DE (EU-Central) → LON - London, GB (EU-Northwest) →

Jan 2, 20:00:00 GMT+0
Identified - During this maintenance window, we will be migrating our European database infrastructure and implementing onsite SSD caches at each location to increase backup throughput..

Jan 2, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 2, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 10 minutes

Affected Components: Game Panel (Replica Aggregate)

Jan 2, 10:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. Servers will remain accessible and joinable but will be unable to control during this window..

Jan 2, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 2, 10:10:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 8 minutes

Affected Components: VIN-50 (Formerly ASH-50)

Jan 2, 07:16:32 GMT+0
Investigating - We are currently investigating this incident..

Jan 2, 07:24:45 GMT+0
Resolved - This incident has been resolved..

Duration: 1 day, 15 hours and 51 minutes

Affected Components: ASH-3, ASH-44, ASH-43, ASH-40, ASH-31, ASH-41, ASH-1, ASH-4, ASH-13, ASH-17, ASH-46, ASH-42, ASH-45, ASH-2, ASH-12, ASH-15, ASH-16, ASH-6, Ashburn Database Functions, ASH-11, ASH-9, ASH-5, ASH-18, ASH-19, ASH-7, ASH-10, ASH-14

Dec 30, 06:50:03 GMT+0
Investigating - Due to sustained DDoS attacks reaching upwards of 10 Tbps against our Ashburn infrastructure, some customers may notice intermittent packet loss on certain routes (especially European ones) impacting ASH-1 through ASH-20\. This is likely a result of our upstream + inline DDoS mitigation provider (GlobalSecureLayer) experiencing saturation at their scrubbing centers during the attacks (they are actually doing a fairly good job at mitigating the attacks with what's available. This is an astronomical amount of bandwidth though). Under normal conditions, we can reduce the blast radius by adjusting mitigation trigger thresholds for non-targets to avoid scrubbing paths. In this case, doing so would introduce unacceptable risk to overall network stability, so we’re keeping the more conservative posture in place. We understand that consistent connectivity is critical for many workloads. We maintain backup infrastructure in a separate datacenter, and we can move a limited/select number of servers over as a temporary mitigation for customers who are most impacted. To add redundancy, we’re bringing CDN77 / DataPacket online as a backup upstream, but provisioning is blocked until their team returns from the Christmas holiday in January. In parallel, we’re also in contact with CosmicGuard as an emergency failover option. We’ll continue to post updates as we make progress and appreciate your patience..

Dec 30, 20:27:45 GMT+0
Monitoring - Since moving select chunks of targeted infrastructure over to Cloudflare magic transit (via CosmicGuard) on isolated infrastructure, the network has stabilized. We will continue to monitor the situation, and still plan to bring DataPacket & Cosmic into our main blend..

Jan 2, 08:35:54 GMT+0
Identified - We've secured an alternative tunnel (Cosmic was too slow, so we made other arrangements) and plan to cut over tomorrow. We'll announce the specific time shortly. This should resolve all current issues, though it will serve as a temporary fix until our CDN77 cross connect is in place..

Jan 3, 01:04:16 GMT+0
Identified - This has been postponed as Cloudflare is not currently processing LOAs due to reduced staffing over the holiday period. We expect to proceed once they resume normal operations around January 5th..

Jan 10, 22:30:24 GMT+0
Monitoring - Network has stabilized a lot but still intermittently experiences issues for a select portion of users. We are currently waiting to be onboarded by DataPacket..

Jan 15, 08:34:09 GMT+0
Monitoring - Connection has been established, we will schedule a migration sometime this weekend for all affected nodes except ASH-1, ASH-3, ASH-7, and ASH-9.

Jan 20, 06:33:45 GMT+0
Monitoring - Link established in IAD1.

Jan 31, 22:40:50 GMT+0
Resolved - This incident has been resolved..

Duration: 17 hours and 25 minutes

Dec 29, 01:14:46 GMT+0
Investigating - We are currently investigating this incident. This node is called ASH-8 in the panel but some old subdomains (which have been phased out) refer to it as `ash.2.prsm.sh `.

Dec 29, 18:39:19 GMT+0
Resolved - Spare has been racked & clients have been moved. .

Duration: 1 minute

Affected Components: DAL-2, DAL-1

Dec 29, 01:11:31 GMT+0
Investigating - We are currently investigating this incident..

Dec 29, 01:12:43 GMT+0
Resolved - This incident has been resolved..

Affected Components: ASH-4

Dec 20, 23:47:46 GMT+0
Resolved - All servers were restarted due to a docker daemon crash. There will be no further symptoms of this..

Dec 20, 23:49:13 GMT+0
Resolved - All servers have started .

Duration: 2 hours and 1 minute

Affected Components: LON-1

Dec 11, 16:02:46 GMT+0
Investigating - We are currently investigating this incident..

Dec 11, 17:55:38 GMT+0
Identified - > We are investigating a boot issue on this node that causes the system to freeze during hardware initialization before services start. Data is intact, but the server cannot currently complete startup. We are applying a kernel configuration workaround and will bring the node back online as soon as it passes testing..

Dec 11, 18:00:44 GMT+0
Monitoring - > The node is now progressing through the boot process after applying a kernel configuration fix. It is no longer stuck during hardware initialization and is moving into system startup. We will bring services online once it fully completes boot and passes verification..

Dec 11, 18:03:43 GMT+0
Resolved - Servers are starting.

Dec 12, 02:23:45 GMT+0
Resolved - One of our London nodes experienced a failure, prompting our technical team to initiate a proactive migration to a replacement machine rather than wait for repairs. During this migration, our bare metal provider's automated DDoS mitigation systems incorrectly flagged the legitimate inter-server transfer traffic as an attack (this is so incredibly stupid because the traffic never left their network in the first place; it was all LOCAL) and null-routed the original box, causing a complete service outage. Our technical team spent the better part of this morning working through the provider's ticket system to clarify that the traffic was legitimate operational activity and restore connectivity. We apologize for the disruption and appreciate your patience. We are still deciding on the SLA application here..

Duration: 27 minutes

Affected Components: , FRA-1, FRA - Frankfurt, DE (EU-Central) →

Dec 9, 01:17:23 GMT+0
Investigating - We are currently investigating this incident..

Dec 9, 01:22:09 GMT+0
Identified - Node is booting.

Dec 9, 01:44:43 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours and 29 minutes

Affected Components: LON-2, LON-1

Dec 4, 17:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. The per node impact should be about \~10 minutes as we swap some hardware..

Dec 4, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 4, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Dec 4, 20:06:49 GMT+0
Identified - Maintenance is now in progress..

Dec 4, 20:16:47 GMT+0
Completed - lon1 is up.

Dec 4, 20:21:26 GMT+0
Identified - lon2 is being worked on.

Dec 4, 20:28:40 GMT+0
Completed - Maintenance has completed successfully..

Duration: 36 minutes

Affected Components: , SIN - Singapore, SG →

Nov 29, 06:00:00 GMT+0
Identified - We are upgrading the remaining OVH bound Singapore infrastructure from EPYC 4004 to EPYC 4005 chips. This will incur an IP change..

Nov 29, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 29, 06:35:53 GMT+0
Completed - Maintenance has completed successfully & ahead of schedule :) Please check your servers to find their new IPSs..

Duration: 3 hours

Affected Components: Game Panel (Replica Aggregate)

Nov 27, 03:40:00 GMT+0
Identified - We are scheduling maintenance on our game panel to add new features and fix the bugs our clients have reported. The maintenance window is set for three hours, but the work will likely finish sooner. All servers will remain online during this time!.

Nov 27, 03:40:01 GMT+0
Identified - Maintenance is now in progress.

Nov 27, 06:40:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 6 minutes

Affected Components: ASH-18

Nov 27, 00:28:18 GMT+0
Investigating - Reboot signal has been sent. This update was issued by an automated response service..

Nov 27, 00:26:37 GMT+0
Investigating - We are currently investigating this incident..

Nov 27, 00:32:53 GMT+0
Resolved - ASH-18 is now operational! This update was created by an automated monitoring service. Notes: Replica reports as healthy..

Duration: 1 hour

Affected Components: ASH-2, ASH-11

Nov 17, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 17, 06:00:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 17, 05:00:00 GMT+0
Identified - We are moving all clients off of our last (and oldest) rented Ashburn nodes, ASH-1 and ASH-2, to owned hardware now that we have finally gotten more rack space and can restock without concerns for future expansion. This will, of course incur an IP change so you'll need to update that on your end. All subdomains created on our platform will auto update. This affects a small minority of clients in Ashburn and if you'd like to see if you're affected, go to settings and refer to the node listing..

Duration: 1 hour and 41 minutes

Affected Components: , DAL-2, DAL-1, DAL - Dallas, US →

Nov 16, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 16, 05:10:33 GMT+0
Completed - Maintenance has completed successfully..

Nov 16, 05:38:40 GMT+0
Identified - network will blip momentarily as we route outbound through GSL.

Nov 16, 05:40:34 GMT+0
Completed - Maintenance has completed successfully. There shouldn't be any future impact..

Nov 16, 04:00:00 GMT+0
Identified - Now that we have a redundant fiber connection to GSL in Equinix Dallas, we're going to turn back always on mitigation and route all connections through GSL's private backbone for lower latency and direct peering. We expect the interruption from this cutover to be brief but have made the window 1 hour incase anything goes wrong..

Duration: 20 minutes

Affected Components: Game Panel (Replica Aggregate)

Nov 7, 17:00:00 GMT+0
Identified - We will be moving our Panel to it's own dedicated machine due to capacity restraints. We expect an impact of \~10 minutes downtime during this cutover..

Nov 7, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 7, 17:20:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 4 hours and 47 minutes

Affected Components: , DAL-2, DAL-1, DAL - Dallas, US →

Oct 12, 15:00:00 GMT+0
Identified - We're migrating our servers from Prime DA4 to Equinix DA2 for direct fiber cross connects to GSL + better networking. Servers will be down for a little bit within this 3 hour window (we estimate \~1h)..

Oct 12, 15:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 12, 19:47:00 GMT+0
Completed - Maintenance has completed successfully..

Duration: 9 days, 7 hours and 19 minutes

Affected Components: LON-2, LON-1, , LON - London, GB (EU-Northwest) →

Oct 4, 15:00:00 GMT+0
Identified - We're replacing old, legacy SKUs in London with new machines. If you're on LON1 or LON2, you'll be reassigned a new static IP during this migration (\~90% of servers on these machines are using our preexisting DNS so they're fine)..

Oct 4, 16:00:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 4, 15:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 13, 20:23:23 GMT+0
Identified - Maintenance is now in progress..

Oct 13, 22:18:54 GMT+0
Completed - IPs have been updated.

Duration: 10 minutes

Affected Components: , ASH - Ashburn, US (US-East-1) →

Sep 27, 03:10:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 27, 03:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Sep 27, 03:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 15 minutes

Sep 8, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 8, 07:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Sep 8, 07:15:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 30 minutes

Affected Components: , , , DAL - Dallas, US → LAX - Los Angeles, US (US-West) → ASH - Ashburn, US (US-East-1) →

Aug 31, 15:00:00 GMT+0
Identified - During this time, we're going to be migrating our legacy client database infrastructure to our own. For most of you this will incur an ip change, and we will update legacy DNS to point to the new IPs. We only expect this disruption to take 5 minutes at max but you might need to update your configurations..

Aug 31, 15:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 31, 15:30:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours

Affected Components: , DAL - Dallas, US →

Aug 31, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 31, 04:00:00 GMT+0
Identified - We're going to be performing routine maintenance on the following nodes in Dallas (some of which are receiving a hardware upgrade!): * DAL5 * DAL7 * DAL12 If you're affected by any hardware upgrade related IP changes, we'll be sending you an email later today :).

Aug 31, 06:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: , SIN - Singapore, SG →

Aug 23, 17:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Aug 23, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 23, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 hour

Affected Components: , DAL - Dallas, US →

Jun 26, 19:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Jun 26, 20:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 26, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 3 hours

Affected Components: , LON - London, GB (EU-Northwest) →

Jun 9, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 9, 02:00:00 GMT+0
Identified - latest update: Maintenance rescheduled. Will happen the day after due to unforeseen issues with the panel earlier today. Hey everyone, Just a heads-up that we’ll be migrating our London location to Frankfurt starting tonight at 10:00 PM GMT-4\. This is part of a broader infrastructure upgrade to improve reliability and protection across Europe. We initially moved from Falkenstein to London after some issues with Neorotect + the Ryzen 9000 upgrade. Apologies for the short notice, but this also includes the replacement of our `lon1` node which has intermittently had issues (that we've fixed for the most part after the three occurrences. it just needs to be replaced), which means clients on that node will experience up to 30 minutes of downtime during the transition. What to expect: * Servers will move to our Frankfurt deployment * IPs will switch to our own subnets * GlobalSecureLayer (GSL) DDoS mitigation will go live for stronger, more consistent protection Downtime: * Most servers: _Under 1 minute_ * `lon1` servers: _Up to 15 minutes_ If you’re using a `prsm.sh` subdomain or anything that we provide, you’re good to go — DNS will update automatically. If you’re using the raw IP directly, you’ll need to update it after the migration. Thanks for bearing with us — we’ll send out confirmations once your server has been moved. — The Prism Team.

Jun 9, 05:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 54 minutes

Affected Components: Game Panel (Replica Aggregate)

May 27, 03:38:06 GMT+0
Identified - Up to DAL7 done. Servers are restarting..

May 27, 03:31:59 GMT+0
Identified - 0 to DAL5 Restarted + IPs updated. We'll start on the next wave. Panel is back..

May 27, 03:53:48 GMT+0
Completed - Completed. Connectivity validated & usages normalized. Thank you for your patience :).

May 27, 03:00:18 GMT+0
Identified - Starting maintenance, we've put the panel in maintenance mode to take a snapshot beforehand..

May 27, 03:56:04 GMT+0
Completed - Just a small subupdate, we've restarted wings and rebound some internal ports to clean up what's open on client IPs and whatnot. Panel connectivity will blip for about 30 seconds or so but servers and their connections will remain functional..

May 27, 03:00:01 GMT+0
Identified - Maintenance is now in progress.

May 27, 03:00:00 GMT+0
Identified - Hi everyone, We're in the process of moving all client servers to our own dedicated IPv4 subnet (under AS214150), transitioning away from the current Elcro-assigned IPs. This change will improve connectivity, security, and make any future migrations smoother. Most servers already use the public alias format `(node).prsm.sh`, so if you're using that (as most clients are), you'll only notice a brief blip, usually under 5 minutes, while DNS updates propagate. If you're still using the legacy raw IPs or DNS resolved versions, please make sure to update your configurations accordingly. We don’t expect any downtime beyond a few minutes, but feel free to reach out if you have any questions or concerns. Thanks! — The Prism Team.

May 27, 03:49:53 GMT+0
Identified - DAL9 and onward completed. We're monitoring everything to make sure it all looks good but servers should all be back (except on DAL5, that poor 7600 is struggling though it _should_ normalize in the next few minutes)..

Duration: 5 minutes

Affected Components: , LON - London, GB (EU-Northwest) →

May 17, 17:25:49 GMT+0
Identified - Taking a backup of all servers before putting back into prod. Est: 40 sec..

May 17, 17:28:18 GMT+0
Completed - Should work now, servers have been started and are pulling appropriate images..

May 17, 17:23:11 GMT+0
Identified - We're starting this early since the issue looks to be a bit more widespread than initially estimated. Servers have restarted..

Duration: 1 hour and 17 minutes

Affected Components: Billing Panel

May 10, 05:01:36 GMT+0
Identified - Hi everyone, our very high quality whmcs theme, lagom decided to shit the database so we're working on reinstalling it.. Lovely updates :c.

May 10, 05:33:47 GMT+0
Completed - All resolved, the billing panel is back with a hefty speed bump too since we had the opportunity to make some caching optimizations..

May 10, 07:08:25 GMT+0
Completed - Maintenance has completed successfully..

May 10, 05:51:52 GMT+0
Identified - .