Issue Archive: May/June 2009

Ten Universities, One Federation: Sharing BCP Costs, Software, and Best Practices

     In 2005, a handful of Ohio universities gathered together to share thoughts on business continuity planning. Within a year, seven universities had joined to form a consortium of business continuity practitioners. At the time of this writing, the consortium has grown to ten members. They work from a memorandum of understanding (MOU) to share costs, software, and best practices. In a time of shrinking budgets and state cost cutting, there is a growing interest in this collaboration model and how it works. Here, we present the history and governance of this “BCP Federation.”

History

     Prior to the BCP Federation, there were several associations or agreements already in place that paved the way for a new collaborative effort to form. One such group was the Inter-University Council of Ohio (IUC), bringing together representatives from Ohio’s 14 public universities.1 Additionally, The Ohio State University (OSU) and the University of Cincinnati had worked together to address common IT disaster recovery concerns, establishing an agreement that enabled each to recover mainframe operations at the other’s location.

     In the spring of 2005, the IUC’s insurance consortium had completed a collaboration on disaster recovery and was examining various methods to implement a business continuity program for its members. Diane Leone, director of risk management for Cleveland State University, learned of a similar program at OSU. OSU had just gone through the process of obtaining, customizing, and field-testing third-party business continuity software to assist with enterprise-wide continuity planning. When Leone reached out to John Ellinger, senior director of information technology at OSU, Ellinger recognized the opportunity for OSU to share its findings and BCP best practices with other higher-education institutions. “The IUC universities had just completed disaster recovery collaboration, and I thought BCP would be a second great opportunity to reduce costs and share risks,” said Ellinger. “Each university saves over 50 percent on the cost of the software and hosting, rather than going it alone. They also reduce the risk to starting up a BCP program at their university.”

     Representatives from several universities met informally to discuss the possibilities.2 Ellinger believed the others could take advantage of the work already done by OSU:

• Pre-configured software, customized for a higher education environment
• Tested methodology based on industry best practices
• Handouts, templates, matrices, and other planning materials
• New collaboration opportunities

     By building upon an existing foundation, it would be much simpler to jump-start or augment a university’s BCP program. The overall benefits were obvious.

     Leone believes that this collaboration “made it much less difficult and less expensive [to implement a BCP program] than any other way we had been considering.” Lura Clapper, director of risk management for Write State University, agrees: “Without the Federation’s support and experience, we would not have been able to pull off a BCP project at that point in time.”

Governance

     The group decided upon a memorandum of understanding (MOU) as the best vehicle to structure the relationship.3 David Lindstedt, director of enterprise continuity management for OSU, drafted an MOU for discussion. The group met in the fall and winter of 2005 to hammer out the details, agreeing on the following structure.

Length of Term: The MOU would be a three-year agreement. Official membership in the BCP Federation would occur on a three-year cycle. At the end of the three years, the members have the option to continue or dissolve the BCP Federation. Any member of the BCP Federation may withdraw at any time; however, payment of fees must continue for the three-year term.

Executive Governing Board (EGB): The BCP Federation established an Executive Governing Board. Each institution is equally represented on the EGB with up to two sponsors from each institution serving on the board. Each institution is entitled to one vote, and all matters are decided by majority vote (as needed). A chair of the EGB was also elected to a one- year term that rotates among institutions. No institution may chair the EGB for two consecutive years.

Change Control Board (CCB): The group established an initial baseline for the software configuration. Once the baseline was established, there would be no major re-design considerations accepted for two years. Each institution designated a local administrator for the software. These local admins constitute a Change Control Board. The board is chaired by a “super administrator” who sets up and administers the software on a cross-institution level. All changes that affect any cross-institutional users are submitted to the CCB. Each institution is entitled to one vote; all matters decided by majority vote. Changes are implemented by a super administrator.

Costs: All member institutions share software costs charged as a percentage of the institution’s total operating and non-operating revenues. This includes both the one-time purchase price of the software as well as the on-going hosting and maintenance fees. One institution volunteered to be responsible for invoicing, collecting, and making payments to the vendor. Future members of the Federation will pay either: (A) a percentage of the original purchase price and on-going maintenance (based on recalculated total cost per institution’s total revenues), or (B) the cost of additional user licenses.

     Once the MOU was in final form, it was signed by a proper representative from each participating university. The existing IUC structure provided a mechanism to accomplish this task. Since each institution’s legal department deemed it necessary to review it before signing, it took several months to finalize.

Configuration

     While OSU had configured the software for a higher education institution, the other schools needed to review the system to make sure it would work for each school’s methodology and specific organizational culture. The goal was to develop or modify existing screens, fields, and options so all institutions could use the system. Members agreed they wanted a common look, feel, and vocabulary with institution-specific customizations kept to a minimum.

     Representatives from each institution attended a two-day configuration review session. The group painstakingly reviewed each screen and field, examining every detail of the system. Duane Starkey, director of the Office of Information Technology, Business Services, at Ohio University, reflected that, “I was expecting a grueling couple of days of resolving issues unique to each institution. However, the advance preparations by the OSU staff resulted in the process going very quickly. The review sparked a lot of discussion and we had some interesting conversations about programs and methodology.”

     Surprisingly, very few changes were identified. “I was concerned,” said Lindstedt, “that we would have a lot of customized fields for each university and we would have to put significant additional security in place to support them. But the suggestions made were thoughtful ones that every institution could utilize across the platform.”

     In addition, the group had to decide what data they would share across institutions and what data would remain separated. Some decisions were obvious, such as keeping employee information confidential. Others were more thought-provoking, such as whether the schools could share vendor and equipment entries. In the end, the group decided to separate most of the data with the exception of equipment and software information. The super administrators “segmented” the data by configuring security specific to each category, a complicated task that took many hours of planning and execution.

Mishaps, Issues, and Lessons Learned

     The system went live in the summer of 2006 and while the launch went well, there were a few problems and issues to tackle. Minor problems such as poorly written data entries and cross-institution mistakes were quickly corrected. 

     A more significant issue concerned the position of super administrators. Since OSU had initially configured the software, the two super administrators were located at OSU and had exclusive experience. They were responsible for system-wide changes, security settings, general troubleshooting, data consistency and cleanup, and regular imports of data from each institution. Should a portion of their time be paid for by the Federation? And what would happen if OSU suddenly shifted their job functions? While it was not in the original MOU, the Federation decided to pay a portion of the salary for an FTE super administrator, also split by percentage of each institution’s total revenues. It was decided that at least one other university needed to train a third super administrator to ensure continuity. “Having another university assist with the super administrator responsibilities, if necessary, gave the Federation another level of back-up to ensure that the resources would be available to perform system-wide changes when necessary,” said Kim McClain, customer resources manager at Ohio University.

     Perhaps the predominant lesson learned from the experience of all the universities was that, despite the considerable effort it took to set up the software, setting up a continuity program was even harder. Paul Allen, director of business services and risk management for Miami University, noted that, “Universities are complex, highly decentralized organizations. Trying to get all the constituencies working on a single project is a significant challenge. Getting the various units to focus on something we all hope never happens is a very daunting task.”

     Many of the schools were just beginning a BCP program. Each university had to address the foundational issues com-mon to all such programs: Who will own the program? What kind of staff do we need? What methodology will we use? How will we get groups to participate? Who can we get from senior leadership to champion the program? Getting the software ready was the “easy” part; now the real work could begin.

Frequently Asked Questions

     In a time of shrinking budgets and state cost cutting, there is a growing interest in the BCP Federation. “Collaboration and sharing risk are essential in today’s public environment,” says Ellinger. Higher education institutions and other organizations from the United States and Canada have contacted the Federation to learn more about what they did and how they did it. Some commonly asked questions are as follows.

Q: What part of the institution “owns” the BCP Program?

A: As is the case across the industry, different institutions position the program in different areas. Right now, each program reports either to risk management or the CIO/IT area of its respective institution.

Q: How did you handle training for the practitioners and local admins of each institution?

A: OSU provided a customized nuts-and-bolts session for local admins as well as an optional training session to executives and plan builders about the methodology they created and shaped for their environment. All other training was left up to each university.

Q: How does each institution address end user training?

A: Some schools work individually with distinct areas to build their plan from start to finish. Most schools bring in large groups of end users for training sessions and the end users are then responsible for going out and building their own plans for their areas.

Q: How many end users are there in your BCP software?

A: 850 named users (and growing).

Q: How often does the Federation meet?

A: We aim for a quarterly teleconference and an annual in-person “summit.”

Q: Have you developed a template that would work for any school?

A: Yes and no. On the one hand, we have customized the supporting business continuity software for use in a higher education environment. On the other hand, we don’t have all the answers ready-at-hand for any given department, area, or school. In our experience (which seems to correspond to that of BCP practitioners at large), every area reacts differently to different emergencies. Each has its own responses, priorities, procedures, organizational framework and culture, and so forth. While we certainly think we have the right methodology to engage in the right planning efforts, no search and replace template with “your name here” is going to really work in a disaster.

Q: How many dedicated FTEs do you have assigned just to business continuity planning?

A: With the exception of one institution, the BCP coordinators/planners wear several different hats and are therefore not dedicated solely to business continuity. CI