About a year-and-a half ago, Land O'Lakes, didn't have an adequate business continuity program. "If you're a financial institution or an airline, business continuity is clearly critical because if your system is down, your money flow stops," Wilberts explains. "We're actually a manufacturing and marketing company, and I think that, in relation to manufacturing operations the impact of business continuity issues is probably a little different and perhaps, at least initially, less severe."
But there certainly is the potential for critical disruption to this $7 billion household name. Land O'Lakes, a farmer-owned cooperative headquartered in St. Paul, MN, does business in all 50 states, and it sells more than butter and cheese. Other products include animal feed, animal milk re-placers, crop nutrients, crop protection products, seed, and eggs. In 2002, Land O'Lakes acquired Purina Mills, doubling the size of its feed business, and shortly thereafter, business continuity and recovery issues came to the fore.
Where There's a Will…
"In the middle of that integration, while we were still running on multiple systems, we had something very simple happen-an air conditioning system failed," says Wilberts. "That air conditioning system had an alarm system, which failed along with it. And before we knew it, we had a situation where the computer got too hot and it had a hard crash, seriously damaging multiple hard drives. It took us 54 hours to recover. At that point, the leadership of our organization realized what a significant outage meant to our business."
Before the outage, "9/11 had certainly raised some awareness." External auditors had called the company's disaster recovery systems "out of date," and early Sarbanes-Oxley requirements indicated that disaster recovery "was going to be a factor," says Wilberts. But nothing drums up support like a critical issue in your own backyard. The air conditioning mishap was clearly a catalyst, creating Land O'Lakes management support for expanded disaster recovery and business continuity planning and initiatives.
Wilberts calls that support "organizational will" and says that it is one of three "key enablers" to BCP success. "Management has to make a commitment to do this," he says. "It's one of those things that everybody wants to have but nobody wants to do. You really have to have somebody somewhere in the leadership of the company saying 'This is important and we will do it.' It wasn't until the last few years that this commitment became an essential priority.'"
So once management was on board, what was next?
Wilberts says he followed traditional wisdom that says the starting point to any BCP program is a business impact analysis (BIA). "While Land O'Lakes had done a BIA, it did not create the sense of urgency to act like a real disaster experience would have. And when we did have an incident, the company realized this is what it means to us, this is what the regulatory arena is like, this is the risk arena, and we decided that we had to do it."
Who and What?
The scope of the project quickly became clear. In the early mainframe days of disaster recovery, the only question you had to answer, Wilberts says, was "How fast can you get the mainframe back?'" And "back in the 1993 - '94 timeframe," Wilberts could answer that question. "We had a plan in place and did rehearsals and we had a recovery system."
So what happened? Everything … all at once.
"Over the last 10 years, there was a complete change in the technology architecture. We went from a single IBM mainframe to hundreds of servers and a very complicated network. Then our company grew significantly. We had, over a five-year period, several acquisitions and mergers, and now we are a very different company than when we started. And between Y2K and ERP, we had barely enough resources to do what we were doing, leaving disaster recovery unattended."
Obviously, this was going to be a big job. Whose job it was wasn't quite as clear. Wilberts found himself in the middle of a "typical" debate. Historically, disaster recovery "is an IS problem," he says. "But, then IS needs the business to tell us what is needed, so it becomes a business risk management problem. And you have this ongoing differing of opinions over who owns this. Does the risk management part of the organization own this or does IS own this?" Wilberts worked with the CFO and the controller to decide who would be responsible for what. Wilberts was tapped to cover IT continuity. The controller was charged with business unit plans. But "within months, the controller took a different job, and I agreed to take on leadership of both the IT disaster recovery and the business continuity."
But Wilberts says "managing it all" certainly doesn't mean doing it all alone, and he clearly has a great deal of respect and admiration for the team. In fact, the second "key enabler" he identifies is "leadership." What Wilberts means by leadership is a core team of people to get the job done. Two Land O'Lakes veterans with more than 40 years combined company experience joined him to implement and run the business continuity program. They are Michael McKeown, director of production services, and Marsha Smith, in-house disaster recovery consultant. In addition to Smith and McKeown, there are some 30 Land O'Lakes employees who work part-time on BCP and are part of the planning team."You have to have people in place who have passion and drive," Wilberts says. "Marsha and Michael along with the broader team are the ones who really make it happen." Which brings him to the third enabler. "You need a partner," he says. By partner, he means someone who can provide a broad range of BCP services. In this case, that means a commercial alternate data center site, a local office recovery center, and consulting services.
"We are a company that is located primarily in the northern U.S., and we have one main corporate campus environment. We have one data center. So we're not an environment that has multiple facilities that we can use as an alternate data center or an office center." Land O'Lakes chose a vendor that offered a wide range of services, including consulting and alternate recovery sites. "Central to our need was a local office recovery center equipped with PCs, phones, and work space. They were able to establish this facility," Wilberts says, adding that the site will accommodate 300 Land O'Lakes employees. In addition, data center equipment is backed up by the same vendor at a hot site located in a facility in a different part of the country.
So with his three key enablers-management support, leadership, and a partner-in place, Wilberts and his team got going and have been on the fast track for nearly two years. "Once you have your enablers, it just takes resources and a lot of hard work," Wilberts says. "There is no way of getting around it. This is not easy. It has been 18 months, six recovery rehearsals, and a whole lot of learning."
Test and Test Again
Those six tests were largely the responsibility of Smith, who joined the team in September of 2003. The first test was in November. The first test was confined to four eight-hour days, seven people, and "a few critical applications," Smith says. "We recovered things that were very simple and that we had done before with our previous vendor in order to get familiar with the new partner and facilities. We learned a lot, and we learned that we had a lot of work to do."Smith and McKeown designed a testing schedule that added more critical applications and business unit participation to each exercise. "Our last few rehearsals included all of our critical applications for all the businesses," Smith says. "The last two rehearsals, in December of 2004 and February of this year, were a full recovery process with a full team of people simulating a disaster scenario and working around the clock to validate our recovery time objective delivery." The final rehearsal was the most comprehensive to date, adds McKeown. "We had over 100 business people come into the office recovery center and they practiced some of their business continuity plan activities." Those exercises required them to work through scenarios in which they had no IT systems available as well as exercises using the systems IT had recovered, he says.
"The experience and the feedback from the business was very positive," says McKeown. "They learned how important their response plans are and how much they have to do yet to establish good solid business continuity plans."
Building Functional Plans
The plans the business units tested were brand new. As IT was getting its recovery program in place, McKeown and Smith had been reaching out to business functions to help them build plans. With consulting support from its partner, plan templates were developed and training sessions were conducted. "We began with collecting key information and setting target dates for having that information," Smith says, "and then we put that information into the plan templates."Once plans were documented, business units tested them via structured walk-throughs prior to the February rehearsal. "We now need to refine the plans based on our rehearsal learning and then transition to a maintenance mode in which business units will keep their plans current and will continue rehearsing and updating them annually. "Today, Land O'Lakes has 93 business continuity plans aligned with functional teams. "We have a document repository where we keep all of the information, where it is updated, and where everyone can get their hands on it," says Smith. "We also created an external Web site where these documents are accessible in the event of a disaster."
"As Marsha was working with the consultant to define the business continuity plans, they also addressed the definition and needs of a disaster management organizational structure," McKeown says. "Not only the functional team, but the special management structure that we would need to have in place in the event of a disaster. We have an executive emergency management team defined, and we have a set of responsibilities defined for that team." In addition to the executive team, there are emergency management teams for each of the three main business units-Dairy Foods, Feed, and Seed. There are also emergency response teams involving communications, HR, and facilities management.
"Our initial focus has been the recovery of the data center and the business functions" at headquarters, Wilberts says. That leaves some 350 locations, including plants, ware-houses, distribution centers, retail outlets, manufacturing facilities, and sales offices that still need plans. These locations are all supported by the IT and network infrastructure in St. Paul. "We need to extend plans to all of these locations," Wilberts adds. Developing templates for remote locations "was not part of the first phase but it is something we will do in the next 18 months."But that's not all the team will be doing. "During phase two, we will be taking what we've learned and improving our plans," Wilberts says. That includes tightening recovery time and recovery point objectives. "Phase two has another element. We have to institutionalize our documentation and change management," Wilberts adds.But Wilberts and team are well aware their job is one that will never be done. "This is a journey," Wilberts says. "We're probably months away from where we really want to be, but that's okay because we are in a really good place and we've come a long way. We feel really, really good about what we've accomplished."
Taking a Bite Out of BCP
"One of the best decisions we made was to carve out small bites of this and take small steps," says Michael McKeown, director of production services. McKeown says this advice applies to all elements of BCP, but in particular to testing. "I drew an analogy to a Broadway play. The first time you get your players together you are not going to do a full dress rehearsal with everybody in costume and everybody knowing their lines. What we did was scope the rehearsals appropriately and not work 24 hours around the clock in our first tests. We decided that we were going to learn as we went along. That has proved very beneficial. It engaged staff and kept us from burning everyone out. It also allowed us to work through technical issues as we encountered them.
Mark Wilberts, Land O'Lakes CIO, says choosing the right partners is critical to the success of your business continuity program. Here are Wilbert's continuity and high-availability picks:
HP provided a business continuity solution for Land O'Lakes to meet their business needs for availability and compliance. This collaboration with HP led to a multiyear services contract that enabled Land O'Lakes to be consistent with its strategy to work with fewer IT vendors. The solution included Business Continuity Consulting Services that helped Land O'Lakes' various business units define and develop comprehensive business continuity plans that would be put into action in the event of a disaster. The solution also included backup facilities at two HP Recovery Centers. The first Recovery Center is located in Kennesaw, GA where Land O'Lakes receives data center and infrastructure backup, and the other is an Office Recovery Center in St. Paul, MN to accommodate 300 end-users. HP teamed with Land O'Lakes to design and construct this facility specifically for their needs.
Veritas Veritas provides backup software and solutions for Land O'Lakes.
Iron Mountain Land O'Lakes chose Iron Mountain for vital records storage.
Qwest and MCI Qwest and MCI are Land O'Lakes' picks for wide area and local network services.
Established in 1921, Land O'Lakes-a national farmer-owned cooperative-is one of America's premier dairy foods and agricultural companies. The company is a leading marketer of dairy-based products for the consumer, food service, and food ingredients markets and provides farmers, local cooperatives, and customers across the nation an extensive line of agricultural supplies, as well as state-of-the-art production and business services. Land O'Lakes does business in all 50 states and approximately 50 countries, with total annual sales exceeding $7 billion. The company is headquartered in St. Paul, MN, and operates more than 200 plants and facilities nationwide. Land O'Lakes produces a full line of quality dairy products including butter and butter blends, margarine and spreads, dairy case cheese, deli cheese, and deli meat. The LAND O'LAKES brand is recognized by more than 95 percent of U.S. consumers and the company holds number-one market shares in such product lines as branded butter and deli cheese. In the Ag Services, or farm supply, side of the business: Land O'Lakes Feed division, Land O'Lakes Purina Feed LLC, holds the nation's leading market shares in both lifestyle and livestock feeds. The company manufactures and markets high-quality feeds for a wide variety of animals from beef cattle to dairy cows, from poultry to pigs, from horses to cats, from emus to monkeys-and more. Land O'Lakes Purina Feed also is the world leader in developing and marketing animal milk replacers for young animals including calves, goat kids, and baby pigs. The LAND O'LAKES and Purina brands are feed industry leaders. Land O'Lakes seed is the fourth-largest seed company in the United States. It provides the latest conventional and biotechnology traits to farmers, ranchers, golf courses, and parks throughout the United States. Some of the seeds developed and marketed include alfalfa, corn, soybeans, canola, grain sorghum, wheat, sugarbeets, and turf grasses. Seed is marketed through the CROPLAN GENETICS, HYTEST, and Forage First brands. Land O'Lakes is also involved in the egg business through its MOARK joint venture. Branded eggs are sold under the LAND O'LAKES and Eggland's Best labels. Agriliance, an agronomy marketing venture between Land O'Lakes, Inc. and CHS Inc., is the nation's leading crop inputs marketing organization. Working through the local cooperative system, Agriliance markets crop nutrients, crop protection products, seed, information management, and technical services to producers and ranchers in all 50 states, as well as Canada and Mexico.