Update - We have most of our VPS infrastructure back online and are still working on restoring VPS services and adjusting a few other things. We will continue to keep this active until things are back to normal.
Feb 27, 2025 - 18:39 UTC
Update - We are continuing to monitor for any further issues.
Feb 26, 2025 - 02:00 UTC
Update - We are continuing to monitor for any further issues.
Feb 26, 2025 - 01:59 UTC
Monitoring - The Discord bot database has been restored, and the network is now online. We have completed initial testing on the node, and everything appears to be stable.
At this point, we are moving this issue to monitoring to ensure continued stability. Our team will remain on high alert, and if any further issues arise, we will address them immediately.
With the database now restored, we are shifting focus to restoring VPS services. This process is now underway, and we will provide further updates as progress is made.
Thank you for your patience throughout this process. SLA compensation will still be provided once everything is fully confirmed as resolved.
Feb 25, 2025 - 20:02 UTC
Update - The Discord bot database has been restored, and we are now working on getting the network online.
We appreciate your patience and will provide another update once we have more progress.
Feb 25, 2025 - 19:36 UTC
Update - We have another update regarding this ongoing outage:
The original Virtualizor issue affected our Discord bot database, which is why many bots are still offline. Right now, we are in the process of restoring the database, but this is a time-consuming task, and our system administrators are working as fast as possible to get everything back online.
We are also still working on bringing all VPS nodes back up. Some have already been restored, but there are still a few that remain offline. This has been a complex issue, and we know it's been frustrating. We appreciate everyone's patience as we work through it.
We want to reassure everyone that SLA compensation will be provided once everything is fully restored. This outage is being treated with the highest priority, but we also want to ensure the process is done carefully to avoid any data corruption. We will keep updating you over the coming hours.
Feb 25, 2025 - 03:43 UTC
Update - Time for a morning update! As of right now we do have the panel online. However, There still needs to be a lot of configuration that needs to be done. Once the configuration is done then we can start the restoring process.
Unfortunately, as mentioned in the previous update restoring will the longest processes as we can't restore multiple servers at a time and it has to be done one at the time. I also just want to remind everyone there will be SLA compensation for these issues.
Feb 23, 2025 - 17:04 UTC
Update - Hi everyone first and foremost we are sorry for all these issues. Our teams are working very hard to getting things back up and running. Once everything is up and running it's gonna take 48 hours to get everything restored. Unfortunately, This will be the longest part of this. But, We are working hard and this is the highest priority
Feb 23, 2025 - 01:00 UTC
Update - We are still in the process of restoring our VPS system. A new master node has been created that is completely separate from our slave node. We are currently in the process of restoring the old master node, which involves reinstalling the OS, installing the slave program, and restoring the data from our backup. It's been a bumpy road throughout this whole process. Rest assured, this is priority number one for us and we will work through the weekend to resolve this. Compensation will be provided once the services are restored.
Feb 22, 2025 - 18:04 UTC
Update - We are still continuing to work on getting things back online we apologize for the inconvenience. Once we have more info will share it with you all.
Feb 21, 2025 - 15:03 UTC
Identified - Earlier today we received a notification from our VPS panel provider (Virtualizor) that they suffered a security breach. In response to this, we have been working on reinstalling and mitigating any possible way that client data could be accessed in regards to this. As soon as we found out about this, we notified our operations team and we have been in a meeting all day in order to make sure that we are able to resolve this as fast as possible.
It doesn't seem like any customer data has been accessed by this security exploit. We will also be in contact with the vps software provider (Virtualizor) to see what they will be doing to prevent these exploits from happening again in the future.
Feb 21, 2025 - 01:20 UTC
Update - We are continuing to investigate this issue.
Feb 20, 2025 - 21:16 UTC
Investigating - We are aware of the panel and servers being down. We are currently investigating the issue and will post updated as we learn more.
Feb 20, 2025 - 19:11 UTC