Disaster Recovery Planning
I am a ‘night owl’ rather than an early riser, but finishing work at 3am was even unusual for me.
The 4th October had been a long day. I had risen at a reasonable time and started work around 8.30 am. I tend to have a few short breaks nowadays with a longer one mid afternoon which is when I tend to have lunch. This was day was no different. As usual I started work again in the evening and, with nothing on the ‘gogglebox’ of interest, this always going to be a late finish. However, I had something urgent I needed to work on for a client. It was required by Friday afternoon.
As I was in flow I just carried on, and finally finished work at just before 3am on 5th October. I had a business call scheduled for 9am so rather than sort out my external drive for daily back-up purposes I shut down, and headed for bed.
Rising around 8.15am and after completing my ablution routine I kick-started my laptop off and put the kettle on. About a minute later the electric kicks off. Odd, but after the fuse-box had tripped about three times without explanation all seemed OK. That was until I carried my coffee over to my laptop. It had switched off and no amount of trying was going to revive my trusted main tool of my trade.
I didn’t have time to dwell on the sinking feeling as I had to be on Zoom for my video call in just a few minutes. Immediately I thought ‘Get the Ipad’. As I did so I spotted in the same drawer my external hard-drive, at which point I let out an audible groan to myself. Why had I not backed my work last night?
Approximately 13 hours of work, gone and lost forever. Well until I completed it again!
I completed my video call, and had a second one scheduled immediately afterwards. The latter was a catch with a business contact, possibly over future collaboration. He was clearly busy and so I offered to re-schedule, which to my relief he agreed to do. I could now consider my laptop dilemma.
It is worth pointing out that in my line of work 95% of my whole business is on this laptop. Without measures to allow for disaster my business would be gone as of today.
As it was I had taken steps to protect myself. This is not me being smug, but I had been through a business disaster situation before and that one I shall never forget.
Let me set the scene.
It’s my first business and I am selling 90% of the asset value. I employ around 25 staff in this firm at this point (who were being transferred to another business I owned). I have signed the Sale and Purchase Agreement contract on 4th May, and am bound by the conditions of sale, the main responsibility of which is to look after the sold portfolio perfectly for a transitional period of two months whilst the cases migrate to the purchaser. Failure to do this means I lose a life changing amount of money, the sale falls through, I am no doubt sued for an obscene amount of money and everyone loses their jobs including me.
No pressure then!
At 7.57pm on Thursday 31st May 2012, I was just about to walk on to a football pitch when I received a call on my mobile from one of my staff who worked the ‘late’ shift at work. I shall change the names for confidentiality reasons, but the conversation went something like this:
Me: Hi Nicola, everything OK?
Nicola: No. Clive the server room is on fire!
After confirming everyone was alright and out of the office and that the Emergency Services had been informed I jumped in my car, still in my football kit and raced to the office.
When I arrived there twenty minutes later, I was greeted by four fire engines and my then IT Manager with his head in his hands.
I asked him: ‘Andy, we are OK aren’t we? We have a Disaster Recovery Plan in place, so we have everything covered?’
Andy looked at me and said:
‘Some of it is covered but other projects have taken precedence over other elements of it.’
That was the worst moment of my business life. Firemen were piling in to our offices with breathing apparatus as smoke bellowed from the building. I felt sick.
It crossed my mind that the last 12 years of my life was for nothing. Oh the sacrifices.
Then as I turned I saw the late-shift staff huddled together recounting their individual recollections whilst a couple of them puffed on cigarettes. At that moment it dawned on me that I could have just let down 25 people and their families. They were blissfully unaware of my internal turmoil as I calmly co-ordinated a ‘ring around’ to all other staff so we could warn and advice them in advance for the next day.
Having released the late-shift staff for the day, later that evening my IT Manager and I got to survey the damage.
By acting quickly Nicola and the others had prevented a complete catastrophe.
Out of 6 servers we lost one immediately . Unfortunately our main one.
The staff were brilliant and we got our non-geographic numbers diverted to mobiles and started communicating the situation to customers, introducers and the purchaser the following morning. Everyone was calm and followed the communication instructions implicitly, ensuring no-one panicked.
The replacement server arrived just before close of business on the Friday, but it had a fault!
It was the Queen’s Diamond Jubilee that weekend as she celebrated 60 years. There was an extended bank holiday of two days and so hardly any hardware suppliers were available.
We ended up paying through the nose for a replacement.
Two more servers failed over the coming days. But working through that weekend with my IT Manager we got everything up and running by the first working day back.
Back on track we avoided a breach and financial penalties under theTransitional Services Agreement.
Phew!
When disaster strikes you learn the flaws in your disaster recovery plan, if you have one. We scraped by. Just..
I cover Disaster Recovery Planning in my Success Program because it is essential to protecting the business you are building.
I recently completed a business audit for a firm that has since decided to retain me as an adviser. The Company had no Disaster Recovery Plan. When I spoke with the Business Owner he said that it had been on his agenda for a couple of years but he had never got around to it. He had more pressing issues like building the firm amongst other challenges. I put to him a scenario about a fire breaking out and wiping out his business. He half-laughed and commented:
‘What are the chances of that happening?’
I recounted my experience.
How quickly do you think Disaster Recovery Planning shot up his agenda?
My client now has a plan in place should the unthinkable strike.
At the time of drafting this I am still angry with myself for not taking that extra couple of minutes to back-up what I had done, whether 3am in the morning or not, there was no excuse.
Thankfully, my client who had the 5pm deadline is happy with my work, having no idea of the self-inflicted stress I had over their project. And I now look forward to spending much of the weekend catching up on the stuff I was supposed to do yesterday and today!
Those couple of minutes cost me an additional thirteen hours of work and unnecessary anxiety over my client’s deadline. But it could have been so much worse.
So, what’s your Disaster Recovery Plan look like?
Oh and, when you get it together, make sure you implement ALL of it!!
