Emergency Server Maintenance (HDD Replacement, completed)

[Iren I.]

Background: We need to inform you that we've discovered malfunctions in one of the HDDs in the RAID array and thus, we need to perform urgent hardware maintenance to replace it and to recover the full functionality of the disk subsystem.


Estimated Service Restoration: Unfortunately, no ETA can be currently provided. We will update this thread when new HDD is located and the server is powered off to proceed with the maintenance. Please check the current thread for updates.

We apologize for any inconvenience this may be causing you or your company. Your patience and understanding are highly appreciated.

[Iren I.]

We will be shutting down the server to perform the HDD replacement within the next 5 minutes.

We apologize for the inconveniences this maintenance is causing you and will keep you updated in this thread.

[Iren I.]

Our datacenter staff is still working on HDD replacement. We will update you as soon as we have any new information.

Thank you for your understanding.

[Evgeniy G.]

HDD has been replaced. Unfortunately we have faced new issue caused by RAID10 array degradation. Situation became true when hardware raid controller marked another member of array as failed and the whole virtual array as unusable due to a lack of members. We are still working on the issue and will update this thread as new information is available.

[Stas G.]

We have managed to assemble RAID10 virtual array and currently we are performing file system check on the server. Having done this we`ll attach new HDD and initiate RAID synchronization. As the data consistency is our main aim we will be performing it in the single user mode with disabled clients VMs. ETA for the first stage is about 1 hour , for the second stage - approximately 2 hours.

[Stas G.]

FSCK has been completed on the server, RAID synchronization process has been initiated.

We will update this thread as soon as any new information is available.

[Stas G.]

RAID rebuild is in progress on the carrier at the moment.

[Stas G.]

RAID rebuild is still running on the server.

Please accept our apologies for these inconveniences.

[Stas G.]

We are sorry for the inconveniences this incident is causing you, but the attempt of RAID rebuild was not finished successfully due to multiple ECC errors. Currently we are working on data transmission from HDDs with errors to a new disk array. This is a preventive measure aimed at keeping the data integrity of all virtual machines on the carrier.

Please follow the forum thread for updates, we highly recommend you to subscribe to it to receive instant e-mail notifications about current situation on the server.

[Valentin P.]

The data synchronization is still in progress, with the data being transferred to new hard disk drives. We once more offer our most sincere apologies for the inconveniences this incident causes you or your company.

The integrity of customer data and the fast restoration of service are our top priorities right now, and we will keep you up to date via this forum thread as new information becomes available.
Please consider subscribing to this thread, so that you could instantly receive any new updates to your email.

[Valentin P.]

The synchronization process is still ongoing, due to the large amount of data that needs to be transferred to the new medium, and the need to maintain data integrity. Once the data is synchronized, we will replace the new disk drives on the carrier.

We can only apologize once more for this incident and the inconveniences this is doubtlessly causing you. Be assured that we’re doing everything we can to get the carrier back up and running as quickly as possible.
Please consider subscribing to this thread, so that you could instantly receive any new updates to your email.

[Irina I.]

The synchronization process is still ongoing.

We will update the thread as soon as we have any new information available.

[Stas G.]

The data synchronization is still running on the carrier. We are doing our best to keep all the data of customers accounts in integrity and safety.

Please accept our most sincere apologies for these ongoing inconveniences.

[Stas G.]

The synchronization of users data from degraded array to a separate HDD has been successfully completed. Hardware rearrangement has been accomplished as well.

Currently we are working on synchronizing the data to a new environment.

Please follow this thread for new updates.

[Stas G.]

15% of users data has been copied to a new HDD.

Please be advised this procedure will take less time, as all the data is located on a properly functional HDD and copy is performed to a healthy RAID array. The main operation running at the moment is reading, as opposed to previous synchronization, and the efficiency of reading operation is a way higher than of writing one.

[Stas G.]

The copying of data is 41% complete.

[Stas G.]

56% of users data has been copied.

[Stas G.]

The copying of data is 81% complete.

[Stas G.]

Data synchronization has been completed. All virtual machines are starting one by one at the moment.

[Stas G.]

All virtual machines are up and running. Currently we are checking performance of operating systems and system software installed on each virtual machine on this carrier to make sure they operate properly. In case you experience any issues after this incident please contact our Virtual Servers department at https://www.whbsupport.com/index.php with detailed description of the problem.

We continue monitoring the carrier closely.