[R03-DAL] Partial Outage

Incident Report for PloxHost

Postmortem

Hello,

We apologize for the late response regarding this RFO. At about 8:13 PM EST on December 1st our monitoring software alerted us of a possible outage of one of the PDUs inside of R03. This rack holds most of our Minecraft Hosting infrastructure within Dallas, and a few of our Dedicated Server hosting customers.

To rectify this issue, our team started to investigate the cause of the PDU outage. Datacenter technical staff was notified. The PDU was reading as an error state, and we determined the best solution here was to replace the entire device. At around 9:00 PM EST, the PDU has been tested by the datacenter and was placed into R03. The datacenter has started to plug in effected devices into the new PDU.

After verifying servers were back online, our team started an investigation looking into why the PDU has failed, as these devices are built to withstand years of 24x7 runtime. At this current time, we cannot determine what caused the PDU to fail. We will be running preventive maintenance in the future to replace cabling under the rack to possibly help resolve an issue like this in the issue.

We thank you for your patience. Our company is uptime driven and giving our customers the best experience is our top goal.

Our team has determined that SLA credit will be handed out to Dedicated Servers & Colocation devices. SLA credit is not available to Minecraft Hosting customers.

Thank you,
PloxHost Management & Sys. Admins

Posted Dec 11, 2021 - 01:22 UTC

Resolved

The PDU has been replaced and our team has restored accessed to servers. If you’re located on the following nodes we are still working to restore access:
- Extreme 59
- Budget 22
- Budget 36

If you’re having any further issues with your servers please contact support.
An RFO will be sent out tomorrow along with an extension to your service if you faced downtime. We apologize for the inconvenience and thank you for your patience. Have a good evening.

Posted Dec 02, 2021 - 03:32 UTC

Identified

Our internal monitoring has pointed to an issue with the PDU inside R03 in Dallas, Texas. We've dispatched techs from our data center to possibly replace the faulty PDU. We apologize for any inconvience.

Posted Dec 02, 2021 - 02:12 UTC

Investigating

We are currently investigating an outage reported in R03 in our Dallas data center.

Posted Dec 02, 2021 - 02:07 UTC

This incident affected: Datacenter Operations: Dallas, Texas (Minecraft Hosting: Dallas, Texas, VPS Hosting: Dallas, Texas, KVM Hosting: Dallas, Texas, Discord Bot Hosting: Dallas, Texas, Dedicated Server Hosting & Colocation: Dallas, Texas).