Caution: Black Swans Crossing
Your company probably is not well-prepared for an IT disaster. Your company probably doesn't understand IT disasters as business continuity crises. Your CEO and board probably don't give these subjects enough attention, or look at them as fundamental, company-wide issues.
And the CIO needs to do something about all of the above.
That's the upshot of a conversation I had with Accenture's Gil Brodnitz and Gary Curtis (Brodnitz is senior executive in the Strategic IT Effectiveness practice, Curtis is global co-lead for Technology Consulting.) They authored this article on the subject with their colleague Robert Emmel; it includes some examples of unanticipated problems and also a sidebar headlined "Thirteen Questions That CEOs Need to Ask Their CIOs."
Brodnitz put disaster planning in the context of Nassim Nicholas Taleb's "black swan" theory, which holds that massive, hard-to-predict events can crash onto the scene with huge consequences.
A lot of companies, especially after 9/11, are prepared for site failures. They're in compliance with the regulators, but aren't war-gaming other scenarios that could lay them low. "People are fighting the last war," says Brodnitz. "What if the sites were fine, but all the people were gone?"
Think avian flu, and its possible impact on your IT shop. Then consider what an empty data center, whether in New York or Oregon or India, could mean for your business.
Says Curtis, "If you look at disaster and IT recovery plans, they are significantly deficient." But contingencies can be planned. He worked with a large company in Florida on preparing for a hurricane. Beyond the facility, which was built to withstand wind and water, there was discussion of getting people to the site in the wake of a regional disaster, and then getting them necessities like fresh water. "We put in the infrastructure to move skeleton crews by helicopter from as much as 200 miles away -- that should be two times the distance needed for a big regional disaster," he says. "We rehearsed it."
That kind of stuff doesn't just happen, nor is it within the domain of most CIOs. IT has the lead responsibility in architecting it, but it's a shared responsibility, and one that top corporate leadership must consider in terms of business continuity. At most companies, says Curtis, "That's not sinking in."
Worse, these problems may be ignored in an economic downturn as people face cost pressures and look for short-term fixes to keep their own jobs safe, says Brodnitz.
Accenture sees a crisis coming. "We haven't seen a major company go down because of a database failure, but it's been close," says Brodnitz. "Something like that might have to happen for this to be a board level issue. And it will happen, as software continues to become more complex."
Related: Here's a disaster-recovery story written in the wake of 9/11. It would have been a wrenching piece to report for any writer, but for me there was a personal element, and I felt like those guys were honoring my friends with their work.