First of all, my deepest apology!
Due a severe hardware failure on the server and me completly overworked we have some data loss on the server. The “Nether” is gone, and probably unrecoverable.
I’ll work with each player at loss an individual solution.
Now, what happened?
The disk where the VMs where located, broke down, and due an error on the raidcontroller – or probably me totaly overworked – the raidcontroller extended the wrong HDD and re initiated the disk where the data was to an empty raid and not the one that was supposed to be changed to. To bad ESXi does not support software raid…
The result of that heroic act is that all VMs are gone. Puff.
After rebuilding the MetaCraft machine, I noticed some weak spots in the backup system I was using, and with that, some data was not recoverable.
With that, the map “Nether” is gone.
Working since 16:00, I was able to pull my corenetwork up again.
IMPORTANT: The Raidcontroller is still rebuilding the raid. Please DO NOT go to mob/exp farms for to long, as this will create a huge load for now. I should be okay after one-two days.
So, what the steps to prevent any event like this?
The following steps are being done to to prevent similar events:
- Do a full hardware sanity check
- Replace unhealthy hardware
- Rework complete backup concept
- Rework network concept, so backup concept fits in
- Get a second backup server (Or storage in general)
- Get better monitoring
- Search for a second Sysadmin (If you know someone capable, contact schemen)
For all the people at loss, I am very sorry for this unlucky event. It really hurts due the “Raid the Nether” event we had not to long a go.
I promise to find a solution so we can compensate the loss!