Skip navigation

Tag Archives: disaster recovery

A few days before the recent British Airways (BA) catastrophic IT failure I was in Kuala Lumpur, Malaysia, giving a talk at the second ASEAN Business Continuity Conference entitled “Building a Robust ITDR Plan”.

The main thrust of this talk was that as IT is at the heart of every organisation, ITDR is at the heart of Business Continuity, and that it is up to the organisation’s top management to ensure that its ITDR plans both meet the needs of the organisation and are known to work.

It appears that BA’s ITDR plans did not work, and although we don’t know whether the plans were appropriate for BA, the possibility is that they weren’t. In any event, the failure certainly came as a nasty surprise to BA’s top management.

I was asked to provide a closing thought to my talk on “Building a Robust ITDR Plan”, and I used a quote from Georges Clemenceau, the Prime Minister of France in the First World War, to sum up my ideas. For those of you who aren’t that aware of the catastrophe suffered by France in that war, it lost a generation of young men. Out of 8 million men conscripted, 4 million were wounded and 1 in 6 killed.

Georges Clemenceau said “War is too serious a matter to entrust to military men.”

I said “ITDR is too serious a matter to entrust to technologists.”

BA will have learnt that lesson, as France did, the hard way.

Cyber and terrorist attacks currently appear to dominate Business Continuity (BC) thinking, but over the weekend we had a classic example of a good old fashioned failure of a critical IT system causing major disruption and some resulting poor incident management that compounded the problem. The company involved was British Airways (BA), and I say poor incident management because this is what the public has perceived and what BA customers experienced. No doubt there will be an internal BA investigation into what went wrong, but as a BC professional I’d love to know about three aspects of the incident and BA’s response:

  1. How long did it take from the initial failure of the system for the IT support technicians to realise that they were dealing with a major incident, who did they escalate the incident to (if anyone), were the people designated to handle major incident contactable, and was the problem compounded by the fact that BA’s IT had been outsourced to India?
  2. The system that failed is so critical to BA’s operations that it must have had a Recovery Time Objective (RTO) of minutes, or at worst, a couple of hours. To achieve this, BA should have put in place a duplicate live version of the system (Active/Active). Either BA did not have such a recovery option in place (I’m guessing that they had a replica – Active/Passive), which implies that they failed to understand the need to have a very short downtime on the system, or it had not been properly tested and failed when required.
  3. Why were the communications with customers  (people who were booked on BA flights) handled so badly? BA must have a plan to communicate with passengers, but was this dependent on the very system that failed?

For me, even before the inquest takes place, the major lesson to be learned is that the effectiveness of an organisation’s BC and incident response plans can only be assured by actually using the plans and responding to incidents. If you don’t want to find this out in response to a real incident, then you need to run realistic and regular exercises so that every aspect of your response is tested and the people involved know what to do. It doesn’t matter how good your Business Continuity Management (BCM) process is, how closely aligned to ISO 22301 it is, how good the result of the latest BC audit, or how much documentation you have. It’s your ability to respond effectively and recover in time that matters.

BA have suffered damage to their reputation , how much is yet to be seen. They will have suffered financial damage, and when the London Stock Market opens for trading we’ll see how much it has affected their share price. Maybe BA do run realistic and regular exercises. If they do, they should have identified the issues with the systems and incident response that were encountered over the weekend and acted on the lessons learned.

 

 

I have been further convinced of the need for the Business Continuity (BC) profession to get back to its fundamentals by the juxtaposition of the publication by the Business Continuity Institute (BCI) of a comprehensive list of legislation, regulations, standards and guidelines in the field of Business Continuity Management (BCM) and the experience of many business that were affected by the recent floods in the north-west of England.

Some small businesses, mainly those that operate and serve very local markets, have temporarily closed until their premises can be refurbished, but others are up and running and continuing to trade even though their premises were badly flooded. The businesses that are back up and running had implemented BC, but not in the way envisaged by the BC profession through its standards and guidelines.

These businesses had taken steps to ensure that they could recover from incidents like the recent flooding by doing such things as backing up their data, implementing cloud computing, knowing where they could obtain replacement premises and equipment, being able to redirect their telephones, and having adequate insurance cover. They are also managed by people who know how to respond to incidents, are committed to the continued success of their business, and know what needs to be recovered by when without having to read a plan.

None of these businesses had implemented a formal BCM programme, none of them had followed any guidelines, and none of them had implemented a Business Continuity Management System (BCMS) or been certified to a BCM standard.

The publication by the BCI of a comprehensive list of BCM legislation, regulations, standards and guidelines is very useful, and I’m not decrying it. But, and it is a very big but, the purpose of BC is to enable organisations to be resilient to incidents that affect their ability to operate. The people who own and run business in the north-west of England that had taken steps to ensure that they could recover from the recent flooding are practising the fundamentals of BC, and by and large have never even heard of BCM legislation, regulations, standards and guidelines.

Don’t get me wrong, there’s nothing wrong with BCM legislation, regulations, standards and guidelines, but they are not the end in itself. I sometimes think that BC professionals lose sight of this.

Despite my best efforts, I’m still unable to kill off the myth about “80% of companies without recovery plans failing within 18 months of having a disaster”. The myth comes in many statistical guises, and the latest example appears in a white paper from AVG, the online security company, which contains the quote from Touche Ross “The survival rate for companies without a disaster recovery plan is less than 10%”.

Depressingly, this quote is used by a large number of organisations that should know better, and is usually stated in the format “A Touche Ross study found that the survival rate for companies without a disaster recovery plan is less than 10%”. I have tried very hard to find this Touche Ross study, but to no avail. Touche Ross has not existed as a separate company since 1989 when it became Deloitte Touche , so this is hardly a recent study, even if it actually exists.

I have searched the Deloitte web site and cannot find any reference to the study in question, and have now made contact with Deloitte to ask if they can try and find the study, and whether or not they stand by the quote. Watch this space!

Finally, there is real concrete evidence that an organisation’s ability to recover is central to its immediate survival. Not its ability to recover after an incident, but its ability to demonstrate its recovery capability as perceived by others before any incident occurs. Business Continuity is now firmly center stage.

According to The Times, senior UK government officials “want the Co-operative Bank to be sold to a bigger player that could stabilise its IT system, which is feared to be so precarious that the bank could not cope with a serious problem.” For years I’ve been telling senior executives that not being able to demonstrate the existence of credible and tested Business Continuity arrangements could mean the difference between survival and failure, and now I can point to a real example. Business Continuity is not just for use in response to an incident – it must be demonstrable to interested parties well before any incident takes place.

Apparently, In the risk factors disclosed in its annual report, the Co-operative Bank has stated that “whilst a basic level of resilience to a significant data outage is in place, the bank does not currently have a proven end-to-end disaster recovery capability”. How many organisations can really hand on heart state that they have a proven end-to-end disaster recovery capability? Not that many.

Business Continuity has been practised in the banking industry for more than 25 years, and many of today’s accepted Business Continuity ideas and practices started in banking. Where banking leads in Business Continuity, other industries follow.

How long will it be before organisation’s in other industries are put at risk because they do not have a proven end-to-end disaster recovery capability?