ITDR is a Business not an IT issue

A few days before the recent British Airways (BA) catastrophic IT failure I was in Kuala Lumpur, Malaysia, giving a talk at the second ASEAN Business Continuity Conference entitled “Building a Robust ITDR Plan”.

The main thrust of this talk was that as IT is at the heart of every organisation, ITDR is at the heart of Business Continuity, and that it is up to the organisation’s top management to ensure that its ITDR plans both meet the needs of the organisation and are known to work.

It appears that BA’s ITDR plans did not work, and although we don’t know whether the plans were appropriate for BA, the possibility is that they weren’t. In any event, the failure certainly came as a nasty surprise to BA’s top management.

I was asked to provide a closing thought to my talk on “Building a Robust ITDR Plan”, and I used a quote from Georges Clemenceau, the Prime Minister of France in the First World War, to sum up my ideas. For those of you who aren’t that aware of the catastrophe suffered by France in that war, it lost a generation of young men. Out of 8 million men conscripted, 4 million were wounded and 1 in 6 killed.

Georges Clemenceau said “War is too serious a matter to entrust to military men.”

I said “ITDR is too serious a matter to entrust to technologists.”

BA will have learnt that lesson, as France did, the hard way.

Incident Management and IT Failure

Cyber and terrorist attacks currently appear to dominate Business Continuity (BC) thinking, but over the weekend we had a classic example of a good old fashioned failure of a critical IT system causing major disruption and some resulting poor incident management that compounded the problem. The company involved was British Airways (BA), and I say poor incident management because this is what the public has perceived and what BA customers experienced. No doubt there will be an internal BA investigation into what went wrong, but as a BC professional I’d love to know about three aspects of the incident and BA’s response:

1. How long did it take from the initial failure of the system for the IT support technicians to realise that they were dealing with a major incident, who did they escalate the incident to (if anyone), were the people designated to handle major incident contactable, and was the problem compounded by the fact that BA’s IT had been outsourced to India?
2. The system that failed is so critical to BA’s operations that it must have had a Recovery Time Objective (RTO) of minutes, or at worst, a couple of hours. To achieve this, BA should have put in place a duplicate live version of the system (Active/Active). Either BA did not have such a recovery option in place (I’m guessing that they had a replica – Active/Passive), which implies that they failed to understand the need to have a very short downtime on the system, or it had not been properly tested and failed when required.
3. Why were the communications with customers  (people who were booked on BA flights) handled so badly? BA must have a plan to communicate with passengers, but was this dependent on the very system that failed?

For me, even before the inquest takes place, the major lesson to be learned is that the effectiveness of an organisation’s BC and incident response plans can only be assured by actually using the plans and responding to incidents. If you don’t want to find this out in response to a real incident, then you need to run realistic and regular exercises so that every aspect of your response is tested and the people involved know what to do. It doesn’t matter how good your Business Continuity Management (BCM) process is, how closely aligned to ISO 22301 it is, how good the result of the latest BC audit, or how much documentation you have. It’s your ability to respond effectively and recover in time that matters.

BA have suffered damage to their reputation , how much is yet to be seen. They will have suffered financial damage, and when the London Stock Market opens for trading we’ll see how much it has affected their share price. Maybe BA do run realistic and regular exercises. If they do, they should have identified the issues with the systems and incident response that were encountered over the weekend and acted on the lessons learned.

 

 

BC Plans – Fit for Purpose?

In a survey about the experience of handling major losses undertaken Vericlaim and Alarm, more than half of respondents “rated the practical assistance offered by a BCP (Business Continuity Plan) following a major incident as one or two out of a possible score of five”. In other words, the BC Plans of the organisations responding to the survey were found to not particularly helpful when responding to a major loss!

This finding seems to have been rather under reported by the BC community who are usually so forward in explaining the importance of having a BC Plan and extolling the virtues of BC in improving resilience. Personally, I find it a damning indictment of the BC profession.

One of the things that constantly both amuses and horrifies me is how far most BC Plans are from the description given in the Business Continuity Institute’s (BCI’s) Good Practice Guidelines. This states that a BC Plan should be “…focused, specific and easy to use…”, and that the important characteristics for an effective BC Plan are that is direct, adaptable, concise, and relevant.

Over the years I have had the pleasure of see hundreds, if not thousands of BC Plans from a wide variety of organisations, and I can safely say that more than 90% of these plans do not fit in with this description. They tend to contain lots of information that is irrelevant to the purpose of responding to a major incident and seem to be written more for the benefit of the organisation’s auditors than for use by people who need to take action to reduce the impact of the incident on the organisation.

As a BC consultant, I keep trying my best to improve BC Plans, but I’m constantly being knocked back by people who tell me that all sorts of things need to be put into their BC Plans, more often than not because of an audit or review undertaken by a third party.

For far too long this situation has been allowed to continue unchallenged. It cannot do so for too much longer without the BC profession losing credibility.

 

 

Back to Business Continuity Funamentals

I have been further convinced of the need for the Business Continuity (BC) profession to get back to its fundamentals by the juxtaposition of the publication by the Business Continuity Institute (BCI) of a comprehensive list of legislation, regulations, standards and guidelines in the field of Business Continuity Management (BCM) and the experience of many business that were affected by the recent floods in the north-west of England.

Some small businesses, mainly those that operate and serve very local markets, have temporarily closed until their premises can be refurbished, but others are up and running and continuing to trade even though their premises were badly flooded. The businesses that are back up and running had implemented BC, but not in the way envisaged by the BC profession through its standards and guidelines.

These businesses had taken steps to ensure that they could recover from incidents like the recent flooding by doing such things as backing up their data, implementing cloud computing, knowing where they could obtain replacement premises and equipment, being able to redirect their telephones, and having adequate insurance cover. They are also managed by people who know how to respond to incidents, are committed to the continued success of their business, and know what needs to be recovered by when without having to read a plan.

None of these businesses had implemented a formal BCM programme, none of them had followed any guidelines, and none of them had implemented a Business Continuity Management System (BCMS) or been certified to a BCM standard.

The publication by the BCI of a comprehensive list of BCM legislation, regulations, standards and guidelines is very useful, and I’m not decrying it. But, and it is a very big but, the purpose of BC is to enable organisations to be resilient to incidents that affect their ability to operate. The people who own and run business in the north-west of England that had taken steps to ensure that they could recover from the recent flooding are practising the fundamentals of BC, and by and large have never even heard of BCM legislation, regulations, standards and guidelines.

Don’t get me wrong, there’s nothing wrong with BCM legislation, regulations, standards and guidelines, but they are not the end in itself. I sometimes think that BC professionals lose sight of this.

The Use of Information is Changing

As most people are only too well aware, the way that we find and use information is going through a radical and fundamental change, which is being driven by the Internet. What doesn’t seem to have permeated the world of Business Continuity though, is that this change is revolutionising the Business Continuity Plan.

Not too many years ago, in our house, we used to keep a telephone directory and combined bus and train timetable near our front door, close to where we had our telephone. Today, we have neither of those things, and if we want to find a telephone number or the time of a bus or train we’ll simply use the Internet, and rapidly find what we’re looking without wading through pages and pages of small print trying to decipher how the directory or timetable is organised before getting to the information that we want. We also had the depressing problem of finding out later on that we’d looked up the information in a document that was out of date, and that one of the family had inadvertently thrown away the new version and kept the old one.

Telephone directories and timetables are just two examples of documents that are being used by fewer and fewer people, and most of those are older people who find it hard to change a lifetime’s habits. Using printed documents to find information is becoming a thing of the past, as anyone who mixes with youngsters will confirm. Why then, do we persist with documents in the world of Business Continuity, what’s wrong with just finding the information that we need from the Internet?

The problems of document based Business Continuity Plans are only too well known. Unfortunately, more often than not, they are difficult to use in a crisis, contain unnecessary information, and are out of date. What we really need is something that is simple to use, delivers exactly what is required, and provides the latest information. That is an App.

An App is short for an Application, and is quite simply a piece of software designed to fulfil a particular purpose, and is downloaded by a user to a computing device from which it can be used. Apps can be used to obtain information, and when designed to provide the information required to respond to an incident, they are an ideal and powerful tool.

Don’t make the mistake of thinking that holding a Business Continuity Plan as a PDF document and making it available on the Internet via an App is the same thing as an App designed to enable someone to respond to an incident, it’s not. You don’t look up the time of a train on the Internet by opening up a PDF document and searching through it, do you?
A Business Continuity App can provide responders with clear, action orientated, and time-based direction, while allowing quick access to relevant and up to date support information. Exactly what we want to achieve.

This revolution has profound consequences for world of Business Continuity, and if you’d to find out what these are, then come and listen to me present at the BCI World Conference and Exhibition in November. The Business Continuity Plan, as a document, is dead, long live the Business Continuity App.

The Myth that Never Dies

Despite my best efforts, I’m still unable to kill off the myth about “80% of companies without recovery plans failing within 18 months of having a disaster”. The myth comes in many statistical guises, and the latest example appears in a white paper from AVG, the online security company, which contains the quote from Touche Ross “The survival rate for companies without a disaster recovery plan is less than 10%”.

Depressingly, this quote is used by a large number of organisations that should know better, and is usually stated in the format “A Touche Ross study found that the survival rate for companies without a disaster recovery plan is less than 10%”. I have tried very hard to find this Touche Ross study, but to no avail. Touche Ross has not existed as a separate company since 1989 when it became Deloitte Touche , so this is hardly a recent study, even if it actually exists.

I have searched the Deloitte web site and cannot find any reference to the study in question, and have now made contact with Deloitte to ask if they can try and find the study, and whether or not they stand by the quote. Watch this space!

Planning Horizon

What is your organisation’s Business Continuity planning horizon? By that, I mean what time-scale after an incident that causes disruption  do your Business Continuity Plans cover? A day, a week, a month, longer?

Every organisation that I’ve ever come across determines some kind of time limit, which is linked to the level of service that it plans to recover to. Without such a planning horizon, recovery plans would cover the complete resumption of the organisation back to its original state – which would be far too detailed and complex, and assume that nothing would change after the incident.

This planning horizon needs to be agreed at an early stage of the Business Continuity Management (BCM) process, before the Business Impact Analysis (BIA) is undertaken. This is because the BIA needs to concentrate on those activities that need to be recovered within the planning horizon. If this boundary hasn’t been put on the BIA, then a lot of time and effort will be wasted analysing every single activity.

So what? If everyone has a planning horizon then why mention it in a blog? Because it’s something that the Business Continuity industry chooses to keep secret. Try finding it is the ISO standard or the BCI’s Good Practice Guidelines. The idea of concentrating on the urgent activities in the BIA is there, but you won’t find anything about top management deciding on a planning horizon in the BCM Programme management sections. What’s everyone being so coy about?

Maximum Survivable Incident

I’m very pleased that I’ve managed to get my latest client, a small electronics company that actually decided by themselves to implement Business Continuity Management (BCM) rather than being told to, to think about the maximum scale of incident that it wants to plan to survive. Many organisations shy away from this issue, which makes it difficult when advising on safe separation distances for backups and recovery sites, but my client’s management team understands the issues and will be coming up with an answer.

I think that the factor that will determine the answer is the geographic spread of their staff. If there is some kind of natural or man made disaster that affects the homes and families of most of the staff then it is unlikely that they will want to come to work to help out their employer, particularly if their employer is asking them to work a significant distance from their families who may be evacuated.

If this is the case then we’re probably talking of their surviving an incident that has an effective radius of about 30km. Such an incident would take quite a catastrophic and unlikely event given that the client is nowhere near a nuclear or chemical facility, well away from the coast, and not in an earthquake zone or near an active volcano. The most likely wide spread event is a river flood, but that doesn’t usually last more than a few weeks in the UK.

Plane Crash in City

One of my favourite scenarios for an exercise is a plane crashing into a building. It is one of the only incidents where salvage may not be possible for a long period of time (because of the need to investigate a crash site) that is coupled with the potential total destruction of paper records and any backups that may have been left on site (stupid, yes, but it is very common).

Sadly, whenever I use this scenario there are always people that tell me that it’s unrealistic. In response I tell them that it’s quite a common occurrence, particularly near to airports. For all of you who think that it’s unrealistic, please read about the recent crash of a passenger jet near Lagos airport. It does happen, and for cities like London, where at least one plane a minute travels across the city on approach to the main airport (in this case Heathrow), it’s not a question of if, bur when.