Site problems today?

[Pip]

BTW this is my submission for the photo contest. Couldn't get the car back on the road so had to get creative.

[cakM3]

[ELCID86]

Still.



Thumbs, iPhone, Tapatalk.

[cakM3]

Yep….hopefully whatever is affecting the Mafia site will be corrected soon…. not much we can do at this time….

[johnrando]

Yes, still being worked.

[danewilson77]

The latest:

Dane,

Upon further review, there appears to be an issue with the temporary file for the previously crashed table on your server.

"
zhpmafia_zhpforum.filedata
error : Can't create new tempfile: './zhpmafia_zhpforum/filedata.TMD'
status : Operation failed
"

I am attempting a different type of repair on this particular table at the moment, and I will update this ticket once that process has completed. In the interim, please let me know if you have any additional questions or concerns.


Thank you,

Keith W.
ServInt MST

"No flamesuit required"

[cakM3]

Looks like the issues I have been experiencing all day have been fixed

Thanks Dane!!

[danewilson77]

SERVINT is telling me the host of issues has been resolved. Let me know of any threads that do not behave properly.

Thanks for the patience fellas.

[Hornung418]

Was going through major withdrawl...

[danewilson77]

And now.....the rest of the story.

2015-01-15 17:01:26 EST

To ServInt SolidFire VPS customers,

Last night, January 15th, an unexpected service interruption occurred on our SolidFire platform during a scheduled maintenance that we reported would not be service affecting. For this interruption, we are extremely apologetic. We cannot say enough about how disappointed we are that anyone was affected by what was supposed to be a routine, live maintenance. We strive for all of our customers to see 100% uptime. When they do not, we investigate why and improve to make sure similar outages cannot occur in the future.

The scheduled maintenance was for the purpose of upgrading our Reston, VA SolidFire cluster to the next major system release. The maintenance was a routine update operation, being performed by SolidFire’s own staff. It was not supposed to have any impact on service. However, during the pre-upgrade process, one of the nodes in the cluster failed to reset properly. During the recovery from this failure, a second node in the cluster exited the cluster unexpectedly

This is the point where visible problems began for some of our customers, as those who were utilizing the exited node would not failover to new nodes and could no longer access the storage cluster. Because of the distribution of affected customers, we were ultimately forced to cold boot all SolidFire powered host nodes. Once we did this, all but two virtual environments (VEs) were brought back online. The two which failed were restored from backup.

In the end, the SolidFire cluster lost no data, but the unclean reboot of VEs caused several to experience data corruption when they ran filesystem checks. Customers experienced differing periods of downtime as a result of this event.

We will be working with SolidFire to determine exactly what happened and how to prevent it in the future. Once we do this, we will also determine what hardware, if any, needs to be replaced. We unequivocally do not want a repeat of this situation or anything like it.

As with any unexpected downtime, this disruption of service is covered by our Service Level Agreement. Customers who were impacted should file a support ticket requesting credit for the downtime they suffered.

Once again, we apologize profusely for the unplanned event. We are confident that we will solve the issue that resulted in the flawed maintenance and it will not be repeated.

Matthew Loschert
CTO, ServInt