Disaster Recovery In Action: Barclays Capital

At 8:45 a.m. on Tuesday, Sept.11, Mike Keegan, chief administrative officer of the Americas for Barclays Capital, was on the phone with the firm's London office when he heard a bang. "It was a big bang and it sounded like something that happened in the street but I didn't think a lot of it," he says. "Then a few seconds later people were at the windows and looking up and you could just see papers and metal flying through the air."

The U.S. headquarters for Barclays Capital, an international investment bank offering retail, corporate banking and asset management services, is 222 Broadway, just .8 of a mile from the World Trade Center.

Within minutes of the first crash, Keegan says he went to see the chief executive officer of Barclays' American operation Tom Kalaris, who said the telephones were out. Not sure why but realizing the close proximity of the Verizon switching station, the two decided almost immediately to enact the first phase of the firm's disaster-recovery plan.

That fist stage called for the organization of a staging center where employees would receive instructions drawn from the firm's disaster-recovery plan. The plan, however, had to be altered almost immediately, as the staging center was supposed to be in the World Trade Center. A new staging center was quickly set up "outside the building, back towards the seaport," says Keegan.

As employees made their way towards the seaport, the second plane hit the south tower of the World Trade Center. "When the second plane hit we did go to the second phase," says Keegan. That final phase called for Barclays to communicate with their disaster-recovery vendor, Comdisco, to get its back-up facility in N.J. ready to go.

The damaged phones lines made that simple communication with Comdisco more challenging than expected. Keegan says it was decided to send an e-mail and to "put someone in a car and go there physically." Comdisco received the e-mail and accepted the declaration of a disaster, which prompted the firm to start preparing the facility for visitors. Under the plan, devised with the input of PricewaterhouseCoopers (PWC), the site would be able to receive about one third of Barclays 1,200 employees and support them for two to 14 days--two weeks, it had been decided, would be sufficient time to find alternate, longer-term space.

Lisa Reshaur, a manager with PWC, says her firm was brought in to evaluate Barclays' disaster-recovery plan and work with the firm's business and IT staff to improve upon it. "They had done some testing and things hadn't gone as well as they'd hoped," she says. "We were brought in to assess their program and enhance it."

Chief Information Officer of the Americas for Barclays Capital Nick Themelis says that after a number of unsuccessful attempts, he was able to get through to the firm's London office where a command-and-control center was established for employees to call. That was accomplished by 9:30 a.m.

Though the plan called for selected employees to head to the Comdisco site in N.J., no one envisioned a scenario in which almost every access point out of Manhattan would be closed. Themelis says that, unable to travel to the back-up site, a makeshift command-and-control center was set up at an employee's home on East 56th Street while the team plotted a course out of the city.

Employees finally started arriving at the back-up facility around 2 p.m., where their first order of business was building out desktops and attempting to establish connectivity with critical business partners like the Depository Trust Company and the Federal Reserve.

Keegan says that by 4:00 p.m., most employees had been contacted and told where to go the following morning. Technicians worked through the night so workstations would be operational when they arrived.

Themelis says that by daybreak, critical applications for funding, money transfer, government trading and derivatives were up and running.

He explains that a triangulated network, consisting of the back-up site in N.J., Barclays' systems in N.Y (which were not destroyed) and the firm's London office, almost ensured that applications would be able to access critical databases which contained identical information. "If we were to lose this (N.Y.) building, we would then have shifted the network pointers to the U.K.," says Themelis.

Almost as important as having redundant data sources for critical applications is having multiple telecom providers to ensure connectivity in case one goes down. "Where we lost a carrier, we had an alternate in all the critical situations and most of the less critical ones, so the disruptions were minimal," explains Themelis.

Difficulties arose, says Keegan, in establishing some point-to-point connections with business partners--both to other displaced firms, which were difficult to locate, and to some industry utilities that speak proprietary electronic languages. "We prepared (for a disaster) by connecting our disaster recovery site to our counter parties and clearers primary sites, but in this case everybody was in their disaster recovery sites," he says. Simply finding contact phone numbers for critical counter parties turned out to be an unenvisioned challenge.

Keegan says the firm is still working to resolve some settlement issues with its customers. Though Barclays was sending cash to its clearer on the Tuesday morning, executives found out later in the day that clients were still waiting for payments. He described the situation as a "bit of a backlog" that he hoped would be cleared up shortly.

The Sept. 11 experience, says Keegan, will be the impetus for some changes in Barclays' disaster-recovery plan, such as increasing the bandwidth between the U.S. and U.K. operations in case those systems need to exchange large amounts of information. Though this time the N.Y. systems were not destroyed and could be accessed remotely from N.J., the firm still wound up routing some information through England to connect with industry utilities--increased bandwidth will help ensure sound communication.

Also, instead of being able to only activate about one third of its staff in an emergency situation, Themelis says he would like to have the entire staff operational, whether they be "at home, online, or in a different state."

The firm has already started to reoccupy its downtown location, which was recently declared structurally sound and cleaned of debris. Keegan anticipates Barclays' entire staff will be back at the Broadway location soon.

The disaster that hit N.Y. on Sept. 11 now has firms thinking about the concept of business continuity on a whole new--and much larger--scale. "We have to start thinking about losing bigger pieces of real estate and really diversify as far away from sources (of information storage) as possible," says Themelis.

PWC Director Richard Luongo also thinks the stakes have been raised and disaster recovery will now be taken more seriously. "It seems to be a board level issue now. People are going to make this one of their key business objectives."

Reflecting on the experience and how Barclays will plan for the next challenge, Keegan says, "I think we were for the most part in pretty good shape. We have to rethink some of the communication issues--ultimately, I think, that's probably the biggest issue, but I can't imagine something on this scale ever happening again."