Defying Disaster
Having a disaster recovery plan for critical information assets may mean the difference between an organization’s survival and its demise. Hurricane Katrina is a grave example of just how damaging Mother Nature can be. The business community has been moved deeply by the impact of the devastating effects of Hurricane Katrina to the Gulf Coast region of the United States. The outpouring of support and relief is ongoing.
Please note: This article was authored by Matt Wenzler and Adam Montella. It is reprinted from an Automotive Industry Action Group (AIAG) publication. For the original version, please download a copy of the Defying Disaster PDF.
Aside from the obvious impact Hurricane Katrina had on the lives of those affected, business operations for the region were impacted dramatically. Many businesses were forced to put their business continuity plans and IT disaster recovery infrastructure into action. Many more companies found that the lack of a plan or failover capabilities made an already difficult task insurmountable. In a hopeless situation, the presence of a detailed business continuity plan can create hope.
Having a disaster recovery implementation for critical information assets could make the difference between an organization’s survival and its demise. In today’s competitive landscape, businesses rely heavily on access to their data. Often, this data must be available 24/7, and any interruption of that access can be catastrophic to the business.
Increasingly, this data is not only for internal use, but also is used by business partners, vendors and customers. In essence, the availability of this information becomes the definition of your company’s service and the lifeblood of your organization. Both your needs and the needs of your customers remain the same regardless of disasters or brief disruptions. According to Gartner Group, two out of five organizations go out of business within five years after the event of a disaster. In the information-hungry world we live in, reducing vulnerabilities of data access can mean life or death for your organization.
Disasters like Katrina or unexpected interruptions can cause irretrievable blows to the reputation of your organization, as well as the loss of irreplaceable, invaluable data. Establishing a business continuity strategy is an absolute requirement for any organization. Once this need is addressed, it is necessary to come up with an IT failover plan that meets the needs of your organization.
A Framework for Action
According to a 2004 survey by AT&T Corp. and the International Association of Emergency Managers, nearly onethird of U.S. companies do not have a business continuity plan in place. Large-scale disasters expose the quantity of businesses that are unprepared, while demonstrating just how damaging that lack of preparation can be. The tragedy of 9/11, the enormous power grid failures of the Midwest in 2003, and the string of hurricanes affecting the American pan handle, Florida and the Carolinas have brought business continuity and disaster recovery to the forefront of business concern.
However, it is important to note that localized disasters such as fire, and even brief disruptions such as fiber cuts, can be equally damaging. A simple Internet virus or worm spread over a single laptop can bring your operations to a grinding halt. Industry standards and regulations, such as HIPPA and Sarbanes-Oxley, also act as drivers for implementing solutions.
These regulations are designed to protect businesses by enforcing the implementation of business continuity plans. Business continuity planning provides both a framework for action, as well as peace of mind for employees. Creating and testing a plan helps the individuals working within the organization to accomplish needed tasks under the harshest conditions.
During a widespread disaster such as a hurricane, both private industry and government will be in harsh competition for the same limited supplies and resources, including food, water, computers and clean-up services. The impact of the disaster may be so widespread that local relocation may not be an option. Transportation options may be limited or non-existent for days or weeks depending on the disaster.
Components of a Good Plan
Does this scenario sound familiar? Bob is assigned the task of writing the company’s business continuity plan. A month or so later, Bob distributes the beautifully bound document to all departments, where it sits on a bookshelf collecting dust. The next time the plan is looked at is when the disaster strikes. The plan is destined to fail.
Disaster planning must involve all stakeholders in the process. Just as data recovery takes into consideration different priorities and timeframes for bringing systems back on line, different priorities and timelines exist for bringing services back to a pre-disaster level. With input from all stakeholders, the plan becomes important to everyone, not just Bob. However, your planning is still not complete. The plan is only good if every employee is aware of the plan and how to use it. Here’s another question: will the plan work?
One of the most critical steps in the planning process is to test your plan. Short of an actual disaster, the easiest and most efficient way to test your plan is through a tabletop exercise. An exercise:
- Serves as the “final exam” at the end of a planning cycle.
- Fosters communication between business units.
- Trains users on the employment of a plan and their role in a disaster.
- Provides a “no-fault” environment to identify gaps.
The time to find out if your plan will work is not when you are standing in a pile of rubble that used to be your business. Having the plan fail during an exercise is actually a good thing, as long as changes are immediately made, updates are communicated, employees are made aware, etc. and thus, begins the planning cycle.
What should your plan consist of? Unfortunately, there is no “one-size-fits-all” option in disaster planning; be aware that your plan will not address every situation that arises during the disaster. However, with a solid plan that is trained and tested, you will have the framework to make informed decisions that will save the business and provide a way for your company to return to full production as soon as possible.
Determining Readiness
Your plan is developed, personnel are trained and the plan has been tested. Then, a disaster strikes and your plan still fails. You have forgotten the most important aspect of disaster planning: a disaster leaves victims in its wake, and some of these victims may be the very personnel you were counting on. If you haven’t considered your employees’ personal and family needs during and after a disaster, they will not be there for you.
If it is an emerging disaster, allow time for your employees to address protection of their family and property. Once all is safe on the home front, they are more likely to be available for their employer. After Hurricane Andrew struck south Florida, firefighters and police in Homestead walked off the job or never showed up for work because their homes were damaged; their lives were in chaos. Once city officials brought in crews to assist in the clean up and temporary repair of their homes, they felt secure enough to return to duty.
Also take into consideration the hardship on a family if you relocate your business operation out of the area. Who will take care of that elderly parent that lives with Bob? How can Sue leave her children for a month or more on short notice? Your plan will only work if every member of your response team is familiar with it, and post-disaster expectations and roles are clearly defined.
Understand that when disaster strikes, your customers will not reduce their expectations for service. By the same token, your expectations for your vendors will only rise. Therefore, clearly understand how your partners and their preparedness will affect your business. The stability of your vendors must be a key part of your business continuity plan.
A strategy for a line of communication to your suppliers is a great first step, outlining the framework of where and how they will provide service to your company. In addition, an understanding of their plans could be a helpful component for your own strategy, revealing what adjustments must be made to increase the probability of the successful restoration of your company.
During Hurricane Katrina, companies like General Physics Corp. (GP), which provides consulting and disaster response services, were affected by the disaster themselves. With offices located throughout the Gulf States, GP became a victim of the disaster damages to facilities and employees’ homes and widespread regional shortages of supplies, food, water and lodging. The flip side to this is that the expectation of GP by its customers increased.
By having a solid plan in place, GP was able to recover, take care of its employees and meet the challenges posed by its customers. GP is currently providing direct staffing support to the City of New Orleans for its recovery process.
What About the Data?
It is clearly understood that the human aspect of continuity planning is paramount, but all of your plans could be for naught if you do not have sufficient solutions in place for the recovery of critical information assets.
First things first: ensure that you have successfully reduced your risk of an outage due to single points of failure. This means that your production environment must be built with redundancies across areas such as connectivity, power, air conditioning and fire suppression. Both physical securities such as access control and intelligent IT securities (e.g., intrusion detection systems) should be implemented. Battery backup systems and on-site power generators are necessary to avoid lengthy power outages.
Of course, all the planning in the world can be undone in a single night when a wall of water 10 feet high comes flooding into your server room. When the New Orleans levy system gave way, many companies’ server rooms were above the flood zone, but all of their communication networks were rendered useless as a result of the storm. It is during this scenario that you need to have a plan in place for recovery of your IT information and applications.
Matthias Horch, CEO for Secure-24, has a clear understanding of just how damaging these outages can be. Secure-24 provides disaster recovery services out of fully redundant data centers in Southfield, Mich. and Phoenix. “During the 2003 power outage, we had customers bringing their servers over to our datacenter in the trunks of their cars,” he says. “After Katrina, the biggest challenge was shipping, as no one was able to deliver the necessary back up tapes and data repositories from the affected areas.”
Many companies make backup tapes of their data and store them off site. “If your company can survive an outage of several weeks, then this may be all you need to do,” says Matthias. “However, most companies cannot afford to be down for that long and need something more immediate in place.”
“Something more immediate” comes in many different flavors. IT disaster recovery can consist of off-site tapes or an off-site disk-to-disk data repository. The latter provides a faster rebuild time, but neither solution allows testing of the disaster recovery environment. Testing is critical because it lets you know if your solution works and makes for a smoother transition when an actual disaster strikes. Replication of your company’s IT infrastructure in an alternative geographic location could be the defining factor for the survivability of your company.
Asynchronous and synchronous in many different flavors. IT disaster nous replication of your environment recovery can consist of off-site tapes or to a disaster recovery site allows an off-site disk-to-disk data repository. employees to quickly relocate and The latter provides a faster rebuild continue operations.
Synchronous, or mirrored sites, will provide zero downtime, but can be exceedingly costly. The more prevalent solution is an asynchronous replication. It is less expensive and can be deployed with only 20 minutes to 30 minutes of downtime for your organization. Typically, these solutions provide a software replication of each of your company’s servers to a scaled-down version of your production environment. The performance might not be as good as your standard network, but you can continue operations and rebuild from there.
Providing a “command center” environment for your critical human resources is equally important. A command center should be a safe place with all the necessities of doing business. Power, Internet and network access, and communications are all requirements of the command center. As a member benefit, AIAG offers a command center environment out of a Category V data center. This solution offers critical human resource managers all the tools they will need while managing the company’s disaster recovery.
Putting it all Together
Business continuity planning and data recovery are simply components of the more holistic concept of emergency management. Emergency management encompasses these components and much more. Regardless of where the responsibility of emergency management rests within your organization (security, IT or risk management), these managers must be responsibile for the process and the authority of senior management when implementing the plan. Taking this approach will help assure your business survives the next disaster.
