Server Unavailable (resolved)

[Vitaliy K.]

We need to inform you that the server has unexpectedly become unavailable.

We are investigating the reasons of this situation and working on minimizing the customer impact.

Estimated Service Restoration: Unfortunately, no ETA can be currently provided. We will update you as soon as any new information is available. Please check for the updates on the current thread.

We apologize for any inconvenience this may be causing you or your company. Your patience and understanding are highly appreciated.

[Vitaliy K.]

The server was unavailable due to kernel panic.

It's currently up and running. The virtual machines are booting up one by one at the moment.

[Vitaliy K.]

All virtual machines are back online.

We are now closing this forum thread.

[Stas G.]

We are forced to re-open this thread, as the carrier has suddenly become unavailable again.

Please accept our most sincere apologies for these repeated inconveniences. We are investigating the reasons of this situation and working on minimizing the customer impact.

[Stas G.]

We are still working on getting the server back online as soon as possible. Please subscribe to the forum thread to get instant e-mail notifications about the current situation on the carrier.

[Stas G.]

We are booting up the server now. Filesystem check is currently running on the server. As soon as it's completed the server will be back online.

[Stas G.]

FSCK has been completed. Virtual machines are currently starting one by one.

[Stas G.]

The carrier is up and running and all virtual machines are back online.

As the posts in this thread may seem a bit confusing, we would like to summarize the recent events.

First we noticed kernel panic on the server that caused its unavailability.

Then we have performed hard reboot of the system to get it back online and analyze server logs to determine the reason of panic.

As you may know, kernel panic events may occur due to hardware-related and software-related reasons.
After investigation of server logs and check of hardware sensors/status information we came to the conclusion that panic was not caused by hardware malfunction.

The carrier with all virtual machines has been booted up and server logs showed no hardware errors that could have caused this incident.

Kernel update was issued to exclude software-related reasons, but to apply it the reboot of the carrier was required.
Therefore, we have postponed reboot to schedule it in the nearest non-business time.

Unfortunately, due to the same reason the server became unavailable again and we were forced to re-open this forum thread and update the kernel in a timely manner.
At the moment the carrier is running with the latest stable kernel and all the VMs are online, nevertheless we continue monitoring the carrier to make sure the issue is resolved on a permanent basis.

[Vitaliy K.]

We were forced to reboot the server again due to multiple errors in its file system.

We will update you shortly when we have more information.

[Vitaliy K.]

The server is online, the file system has been checked on startup and the errors have been fixed.

Currently, virtual machines are booting up. At this moment, 30% of VMs are up and running.

[Vitaliy K.]

All virtual machines are up.

We continue monitoring the carrier.

[Stas G.]

We haven't detected any issues during server monitoring. The carrier is working in a stable mode, therefore we are closing this thread.