Outages in places you would not expect continue from “Sandy”. 11/02/2012

Did you see pictures of the subways underwater? Well if you did then you may understand why other services may be slow in recovering, remember electric and telco services run underground as well as above especially in the great metropolitan areas.

Examples of outages due to infrastructure damage include things like area codes not having services in the midwest because of damage in New York City. I learned about this from one of my vendors in a sales/product meeting. 

Another vendor of our company sent out a detailed letter explaining how their systems failed and even how their backups were effected by power outages, and during all of this they started to off load some of their traffic to a new vendor to reduce the impact even more as a second power outage hit them.  I am copying the email I received from Zopim so others can see how quickly even the best backup plans can be destroyed.

(Advertisement)
Now is a good time to think about backup in the cloud do you have a vendor? You may want to try Backblaze. Give it a shot for a free trial.

 

I also want to say that this is not intended to be a negative piece against Zopim. In fact I applaud the quick thinking involved to look for alternative solutions when many may have just said “We will get back on line when we get back on line”. 

I want to thank the team at Zopim for giving so much information so that my clients and followers will begin to understand the need for good backups and even possible backups plans to the existing disaster recovery plans.  In the example here 2 different data centers locations that were Host  to Zopim were affected by the same event.  Proximity is very important and in Zopim’s letter they point out that they had backups in Singapore but had difficulty in the failover process. 

Please read this letter so you can get the full gist of how hard companies plan and try to provide backups but sometimes the world around is just so brutal that you can not prevent a failure.

Dear steven,

We are emailing you because you were potentially affected by the service outages we experienced on 30 and 31 October 2012. While you might have seen some of our statements, we feel that we owe you a detailed explanation.

We experienced significant outage due to the worst storm surge to have hit the USA in 200 years. As a direct result of Hurricane Sandy, power lines were damaged and significant flooding occurred at our data center resulting in total loss of power to a significant number of our servers, causing the first extended downtime to Zopim.

Zopim is and always has been prepared to deal with hardware and power failures. We have multiple servers and redundancy built into our infrastructure across multiple data centers, backed by generator power and UPS. Our uptime was 99.5% over the last year despite immense traffic and multiple hardware hiccups. In addition, critical data has always been protected and backed up at off-site locations.

However, when Hurricane Sandy hit, our hosting provider’s two data centers in New York City lost their backup power. In addition, it caused their server management system to fail. As a result, we were unable to launch additional servers in Singapore’s data center as part of our failover process.

Immediately after the first downtime incident, we started replicating parts of our service onto Amazon Web Services. This will reduce our reliance on a single hosting provider, and allows us to maintain availability in events like these. Consequently, when a second interruption to power supply occurred, we were able to maintain a lightweight version of Zopim.

Trust us when we say we understand the frustration caused by the downtime of Zopim. While you were not able to serve your customers, we were unable to serve you as well. For this, we are truly sorry. While we cannot control the forces of nature, we should have been better prepared for a catastrophic failure of this scale. We hope that this email has given you a sense of what went wrong and what we have learned.

We will be crediting your account with $10 as a token of our sincerity. Affected Trial users will also have your Trial period extended. We greatly appreciate your continued support through this crisis.

Our Deepest Apologies,
Team Zopim

Techy new and views from the techies at www.end2endsupport.com and E2 Computers in Tarpon Springs.

Looking for a way to reduce you business phone bills up to 80% ask us how RingCentral and VoIP may be a solution for your business. Or, do you want your company to sound more professional see what a hosted PBX can do for you.  Give us a call for more information and watch this video that introduces some of the features our product.