Case 1: Construction Firm

This  construction  firm  acted as  the prime contractor on numerous buildings. Headquartered in Cincinnati, OH with satellite offices throughout the Midwest, the company specialized in Class B office buildings as well as warehouse construction. In addition to its 1,000-plus employees, the firm worked with countless subcontractors for concrete and steel work, etc. Ironically, the construction firm did not own its own headquarters, but shared space with a dental office and a bank. Unfortunately for them, the dental office had a malfunction in its laser equipment, causing catastrophic smoke and water damage to the data center. This also wiped out the company's e-mail server.

All of the company's critical documents-from engagement letters to bid proposals to submitted bids-were housed on these servers. Even worse, this is where their current project plans resided so they had no way of seeing the current status of each project.  Although the firm's  project tracking  system  was accessible, most employees used it for historical data only, relying on e-mail to track work in process. Without a backup e-mail system, real-time project management was nonexistent.

There were backup tapes stored offsite, but this information couldn't be recovered fast enough to stay on schedule. By the time the e-mail system was back up and running, projects were spiraling out of control. The staff really tried to pull together, striving to bridge the huge gap in knowledge between company and personal e-mails. Some honest contractors were helpful, and shared information about their end of the project. Others exploited the situation and readjusted their take on the project, gouging the company on their services. But without any record of the transactions, and with everyone scrambling to keep their heads above water, it was tough to root out these problems.

The company could probably have survived these issues if it weren't for the loss of project schedules. These project timelines had to be reconstructed from bits of paper and aligned with subcontractor schedules. This caused a series of mistimed construction problems.

With space at a premium  at  company headquarters, all construction  materials were scheduled  to be delivered  just in time to be brought to sites. But without project schedules, all of this coordination  was lost  -  costing  the  company dearly in  loss of  efficiency and economies of scale. Even under normal operating conditions, construction sites require a complex order of sequential tasks to be performed.

For  example, rebar needs to be  installed by  the  company and checked by the building inspector before concrete  is poured.  At one site, rebar was delivered a day after the concrete was supposed  to be  poured,  so  the team had to dump the concrete unused because the rebar wasn't onsite. Typically, these complex construction operations were loosely coordinated through e-mail and phone calls, but once e-mail went down, the timelines were severely botched. Even though the firm got projects back online fairly quickly, the cost overruns due to delays put them out
of business.

Lessons Learned

Act quickly. Problems become progressively more complicated with time. Solving a problem 15 minutes after discovery is an order of magnitude simpler than the same problem left unsolved for another hour or day.

Don't rely on e-mail as a mission-critical tool. Doing so left the firm highly vulnerable. When e-mail went down, they lost the majority of their critical contact information, so they had to spend an inordinate amount of time trying to recover the context for what was happening - costing them valuable time from getting sites operational again.

Use additional online supplementary systems for critical project tracking. Consider investing in an online "situation center" to store additional information so you have a baseline to work from if all other systems are down. Choose a tool that you can use from anywhere, in case of a large or regional disaster.

Plan for catastrophic events. Business continuity hinged on backup e-mail servers. Although backup servers were in place in the  event  of a hardware failure, all  of these machines were hosted in the same location. In its desire to save money on offsite  e-mail servers,  the firm scrapped plans to house servers offsite, thinking catastrophic disaster was highly unlikely. Yet all it took was a small fire to take them down. When the  server room was destroyed, the company was operating blind, with no way to communicate.

Ensure synchronization of events .Establish a mechanism to inform all players of the key steps in each project, and provide a clear real-time set of signals for everybody to follow the sequence of their work.

Case 2: Retail Clothing Chain

Amajor  national retail clothing store targeting the Gen X  market  faced  outages  in nearly a dozen of its stores during  Hurricane  Rita.  The company executed a well thought-out plan for getting stores back up and running quickly. However, the company could not reconnect with its dispersed employees. The company had prepared for numerous contingencies. However, they hadn't anticipated a lapse in communication.  This led to a significant percentage of their primarily young workforce taking other jobs. Once the outages occurred, everyone scattered. The company sought to not only get its stores back up and running, but to reach out to help employees affected by the devastating storm.

There was no trouble prepping stores with new inventory and equipment. The company had invested in a strong catastrophic recovery plan, which involved storing and staging cash registers and overstock of product. This equipment  was shipped to affected stores  along  with dumpsters to remove damaged goods. Immediately after the storm hit, the retailer hit the ground with its recovery plan, cleaning out stores, throwing out merchandise and rolling in new point-of-sale computer equipment.

What they weren't counting on was that much of their college-age staff would quickly move on. The recovery plan called for store managers to call employees, but in many cases store managers had also evacuated the area. The company realized they needed to connect with their workforce. The company wanted to help. They wanted to offer care packages during the catastrophe. Instead, they were stuck hiring everyone from scratch.

They initially handed off the 1,500 employee phone list to their inbound call center. However, this meant many calls from customers  went unanswered - turning a regional problem into an international problem. After three days of calling they had only found 200 employees.

To tackle this challenge, they turned to an auto notification system to quickly reach employees  and equally important, receive acknowledgements back en masse. They uploaded a spreadsheet of contact names to the auto notification system, and within 40 minutes had tracked down 1,485 out of the 1,500 employees.

Instead of pulling call center agents away from their core responsibilities, like answering customer calls, they simply pressed a button to quickly contact many people. They learned what they could expect of people returning to work and when. This provided an efficient way to reach those with no e-mail or pagers in a communication blast - without swamping the call center. More importantly, they were able to find employees who were in dire straits, and deliver emergency credit cards to help ease their troubles.

This situational awareness site also prepared the retailer to engage its employees in real-time news gathering. This provides a unique means of gathering information about the situation at every store. Each store is now equipped with security cameras to monitor properties in the case of a storm or other business disruption. Store managers can also take digital photos and video and upload this to the situational awareness site.  This inside  information  gives management valuable insight to support decisions and does not leave them blind during an emergency - or at the mercy of news crews to see what is going on.

Lessons Learned

Don't omit HR from recovery. Every night your whole company goes down the elevator and goes home. If they don't return in the morning, you have no company. Use  auto  notification  communication  gear .Let employees know critical data, such as when store is reopening, where to report and how to obtain interim paychecks.  This helped the company retain a  strong workforce.

Choose communication systems that can reach people on the go.Alert employees via multiple modes of communication instead of solely relying on e-mail. Ensure  instant  acknowledgements back  from employees so you  can plan accordingly. An  auto notification system allows the company to rapidly call many people in parallel. With the push of a button, they sent the same message to all the store managers to alert them that they have 30 minutes to respond with their inventory and request allocations. At a glance at a single screen, they can see who has acknowledged  the message and who is "out of the  loop." Armed with this information, managers know whom to track down. Have a plan for how to return to normal operations.

Businesses need a plan to seamlessly return stores to normal operations. This may involve simply notifying employees that a problem is resolved, or require more complex tasks in the case of a regional evacuation. Not restoring order in the field could cost hours of prime retail selling time.

Case 3: Regional Bank

A large financial  institution  was disrupted by a virus that attacked all branch computers. Since they had a solid business continuity plan in place, along with a multi-modal communication tool, they  were able to  contact  employees outside of regular e-mail. This quick and efficient communication kept staff informed of how to operate without e-mail.

The bank let employees know servers were down, how long the outage was anticipated and what to do in the meantime. A virus had bypassed the bank's firewall and virus protection software, infecting many of its 2,000 branches. This brought down the bank network that supported loan origination and other bank services. Continuing operations in a timely manner was critical. One loan officer alone was dealing with 55 clients, three home closings, and five clients looking to lock in their mortgage rates the day the systems crashed. Even a few hours of downtime would have a direct impact on customers.

Using an auto notification system, they instantly alerted employees via  cell  phone,  home  phone,  pagers  and personal e-mail accounts, directing them to an online situation room where they could quickly access files and instructions.  Employees used the system to alert management of any unanticipated problems so corrective action could be taken immediately.

The  company  had  a  multi-faceted plan  whereby employees  used their regular e-mail during  standard business. If those systems went down, they went directly to a backup e-mail system. If the content in internal and backup systems was corrupted, employees switched over to a multi-modal non-e-mail-based mechanism for communicating. This bought them critical time until the e-mail server was back up and running. This included directing employees to a secondary system not dependent on in-house servers.

In addition, bank management had the foresight to establish a crack team ahead of time to respond to critical events. Using online tools, employees were able to quickly ask for and get creative solutions to unforeseen challenges. Members of the team visited the situational awareness site to review troubles and gather consensus for solution. This ensured employees gain critical guidance at a critical time.

Lessons Learned

Use employees as a powerful means of gathering basic information about the situation at every branch. Each employee provides "inside" information that flows into a single  log so  management has  constant situational awareness to support all decisions.

The most difficult challenge in team activities is coordination.To address this, the bank kept an accurate, real-time online version of task lists during business interruptions, checking off items as they were  completed to avoid omission of important tasks or duplication of efforts. This proved particularly important for geographically dispersed teams as a way to coordinate completion of events. This knowledge gives decision makers the certainty of full situational awareness. Streamline approvals without abandoning safeguards.
Approvals are meant as a safeguard for a business to avoid undesired, risky or wasteful behavior. In an urgent event, automate approvals so they don't interfere with speedy problem resolution. Interact with oversight. Engage  everyone  in  the mechanics of the response. This involved reducing information into something meaningful. The bank created a log of all important events in a secure online forum. Team members posted questions and received comments and responses to the log. This provided a secure, simple, well organized way for lots of people in different places at different times to communicate efficiently. Most importantly, it provided a simple way for bank executives to sign off on plans without impeding prompt action.

Add experts to the team. Business continuity experts have solved similar problems in the past and will see solutions that many novices will miss. These professionals are more likely to realize the subtleties that can have a considerable impact on problem solving. It's also critical to bring in  experts  who  can  transform  data  into  true knowledge. This may involve understanding of the legal consequences of a particular decision, the health and environmental impacts of a chemical spill or the potential reaction of trade unions to changes in work conditions. Businesses require quick, decisive recommendations from qualified experts.

Foster creativity but frame it with controls. There are powerful ways to tap into team creativity, such as building team consensus and leadership enablement. With team consensus, a manager engages trusted colleagues as a sounding board when making a decision, testing ideas and soliciting other views from associates. Generally, this team acts as a creative checkpoint, not the ultimate decision maker. Another approach is enabling leaders. Teams are authorized to delegate authority and have the latitude to adjust team goals to make the organization stronger.

Consider worst-case scenarios. It wasn't likely that the regular and  backup  e-mail  servers  would  both  be disabled.  In preparing  for  those  bad  things  that many people  don't want to  hear about,  the  bank's  business continuity professionals were well prepared for primary and secondary outages.

Conclusion

As the above examples demonstrate, we can't predict all business interruptions.  The companies  that survive, however, do so by tackling critical components of problem solving, including synchronization of resources, proper approval,  notification and  acknowledgement of  key personnel and closure  to  return businesses to  normal operations. Ensuring real-time situational awareness saves critical time and effort and reduces frustration during highly stressful situations. Moreover, it ensures more accurate decision making that can have a profound impact on the continued success of the organization. Ultimately, ensuring prompt response, knowledge flow and employee coordination are key to recovery.

Ted Collins  is  president and  CEO of  MissionMode  Solutions, Inc., developer of a collaboration and crisis management  solution designed for private companies and public safety agencies. He can be reached at (877) 833-7763.