The Cloud Does not Auto-Validate Your Work

ReadWriteWeb's Rick Turoczy recently reported in "Dark Side of the Cloud" the recent incident that resulted in a loss of data for ylastic, a company that facilitates management of Amazon's AWS environments for businesses.

A few days ago, something went amiss with Ylastic's Elastic Block Stores (EBS) on AWS. Application instances were hung. It's unclear how it happened, whether it was a Ylastic issue, an AWS issue, or other, but ultimately data was lost, and Ylastic was forced to revert to a previous data snapshot. Unfortunately, the most recent valid data snapshot was 7 weeks old. Ouch. As Ylastic reported:

"Some time in the last month or so, our EBS snapshotting of this stuck volume seems to have stopped working correctly.... We have gone back and run through all the snapshots, and the last good snapshot that we have is from October 1."

The "cloud" provides an incredible opportunity for start-ups, in terms of purchasing computing and database and data storage facilities at usage-based pricing. Amazon Web Services is about as stable as you can get, when it comes to highly-reliable high-volume computational, database, and data infrastructure. But as with any IT infrastructure, be it in-house or in the cloud, it's the customer's responsibility to monitor processing results, and notice if a processing anomaly occurs. AWS monitors status in these terms: "This program is running, as we promised it would. It output this data; yes, we have the data the app has produced..." In other words, AWS is a similar to an electric utility. They keep the systems running, the processing chugging along, the storage of output intact. If a customer's application has a problem, only the customer can notice that. If the problem occurs only intermittently, and the customer isn't monitoring the processing closely -- that's when big problems occur. Use of the cloud means you are outsourcing processing and data storage to a contractor (for example, Amazon Web Services). As ylastic learned:

Our first outage and lesson learned - test those EBS snapshots religiously...

ZDNet's Phil Wainewright points out these fundamental principles in his "Back up your online data. Now." which highlights how Digital Railroad, a photo archiving and commerce site used by over 1,500 professional photographers, shut down without warning after running into financial trouble. Their creditor decided to "have all information erased from the storage devices and then sell the equipment at auction." Despite pleas from customers. Phil asks the operative question:

Does this example mean we should all stop using cloud providers and go back to the ‘good old days’ of running our own software and servers? Of course not. You’re more likely to lose everything to a disk failure on your own machines than you are to a business failure of a thid-party provider. But it’s still essential in either case to have a back-up strategy.

So, have you monitored the cloud processing you accomplished today? And, did you back up your transactions and data? And verify that those back-ups are valid? If not, now's a good time to check...

Be sure to read the next Cloud article: Yahoo's YQL Makes the Internet Your Database