Blog

← Back to Main Page

Another Summer Storm Season Comes to an End – Winter is Next

November 5, 2013  |  Written by Martin Triplett, Director, Process and Program Management

As we reach the one year anniversary of Hurricane Sandy’s tear through the Caribbean and eastern United States it seems fitting to discuss disaster preparedness and business continuity. Sandy wreaked havoc on the NYC metro area — forever changing lives and temporarily stunting “Gotham”. Telx was fortunate and well prepared for the event, even without access to its corporate headquarters for nearly three weeks. Sandy’s storm surges of 9-12 feet above normal hit a city better prepared for snow than tropical storms.

From my perspective it was also a teachable moment. I recently read a spy novel written by an acquaintance – it was fiction but the key takeaway for me was to have a plan… and have a back-up plan. In that framework, the life of a CIA agent is no different than operating a resilient business. There’s a need for a toolset, a ‘go bag’ of critical functions to make it through the unforeseen and come out ahead of those that are less prepared.

I want to provide insight into some attributes of what make a BC/DR plan. This is a high-level development process and by no means an exhaustive list but hopefully will spur additional thought:

  • Plan for all scenarios – be creative, write a list and don’t judge the contents until the brainstorming is complete (you can remove the hybrid vampire/zombie invasion later…on second thought - better keep it on the list). A few specific thoughts:
     
  • How will you access a datacenter with closed bridges or a disabled public transit system? A $300 investment for a few bicycles could enable a skeleton crew of employees to support the business and get home to rest & care for a family
     
  • What level of priority is your facility given for refueling of backup generators? No doubt about it – some buildings, companies, or even portions of town receive a higher priority level for fuel services during a disaster. Like many things in this life it often comes down to strength in numbers
     
  • Do you have the scale to maintain 24/7 on-site technical expertise during a disaster and leverage staff from other facilities within the same metro region?
     
  •  Rigorously Maintain Critical Infrastructure – In any technology environment and even more so for a data center service provider, the need to actively maintain and carefully track the maintenance of backup infrastructure is key.  Ensuring that your preventative maintenance program is structured enables you to more easily report on the maintenance status of all critical assets.
     
  • Assemble a team of key personnel; have back-ups; and back-ups for back-ups. Disasters seem to strike when your key people are on vacation. One in seven Telx employees is a member of our Crisis Management Team. It can be a challenge to coordinate a group of that size, but it is necessary to ensure full awareness when an event strikes
     
  • Meet at frequent intervals - Quarterly is usually sufficient if there have been no recent events. Standard agenda items can be – what happened since the last meeting; what is on the horizon (hurricanes in the summer, snowstorms in the winter, etc.); and how can we continue to improve & learn from ourselves and others
     
  • Prioritize events - Segregate events that are norms of the business from those critical events that require “all hands on deck”. There’s no need to pull the fire alarm for every event that occurs. Components within datacenters experience outages but that is, on occasion, part of the business. As for people & infrastructure, both miss a step at times; the key is taking proactive and reactive steps surrounding an event and having in place the resiliency of infrastructure that eliminates, or severely reduces, downtime
     
  • Empower a Knowledgeable Team - Ignore the bullet above and empower people with the tools, knowledge, and confidence to contact any senior leader within the firm – these senior leaders are also the sponsors of your crisis management team
     
  • Don’t be afraid to engage outside help – When establishing or enhancing a BC/DR plan, leverage vendors if needed. This can allow your business to focus on its core competency. Your company may have a great legal team but if law is not your primary business, you’re apt to seek outside counsel, right?
     
  • Capture Lessons Learned - Take the opportunity to learn from a disaster; share mistakes made – this is the most powerful form of transparency. Review these lessons learned in your next cyclical session

As stated previously, this is only a high level overview; facility design, a solid infrastructure maintenance program, and an excellent support staff provide for a “force field” of protection against unforeseen events. We’ll examine some of those topics as well as how a BC/DR program supports compliance with SSAE 16, PCI and other industry standards in future posts.