With the anniversary of Sept. 11 fast approaching, disaster recovery is once again taking center stage in the minds of industry observers. But have financial-services firms addressed critical issues such as backup-trading floors and geographic diversity?
"History repeats itself; that's one of the things that's wrong with history," the famous American trial lawyer, Clarence Darrow, once said. Certainly, anyone who was witness to the horror that engulfed Manhattan last September can relate to the painful truth of Darrow's astute observation.
But as we approach the one-year anniversary of this generation's Pearl Harbor, the future of the American economy may well rest in the ability of the financial-services industry to learn from the mistakes of its past. Unfortunately, if history is any guide, this will be an uphill challenge for the industry.
When terrorists detonated a bomb in the underbelly of the World Trade Center in February 1993, there was talk, across the financial-services industry, about the need to enhance disaster-recovery plans - and perhaps even build backup trading floors. But, with a few notable exceptions, that talk was mainly idle chatter. "I don't know what it takes to wake people up. The (industry basically) said, in 1993, that lightning won't strike twice. Well, it sure as hell did, didn't it - big time," says New York Board of Trade Senior Executive Vice President and Interim Chief Operating Officer Pat Gambaro.
Indeed, last September, many firms were caught unprepared when New York City was devastated by a terrorist attack that made the 1993 bombing pale in comparison. The American Stock Exchange, located in the shadow of the Twin Towers, was devoid of a backup-trading facility, and had to scramble to relocate its equities and options traders. The New York Stock Exchange, faced with a host of challenges unrelated to the physical stability of its 11 Wall Street headquarters, had to shut its doors until Sept. 17 - and subsequently faced tough questions from critics who wondered why the United States' largest equity market did not have a fully equipped disaster-recovery trading floor. Moreover, many securities firms experienced connectivity problems linked to the destruction of a Verizon communications hub, while others were second-guessed for having too many offices commingled in downtown Manhattan.
In retrospect, the fact that some securities firms were caught with their guard down is understandable. After all, who could have predicted a disaster of such mind-blowing proportions? But now, nearly one year after Sept. 11, it is both appropriate and necessary to investigate whether financial-services firms have done enough to enhance their disaster-recovery strategies.
Have firms paid enough attention to disaster recovery? Will firms sink or swim if there is another attack? And what has been done to address clear deficiencies illuminated by Sept. 11, such as the need for backup-trading floors and geographic diversity?
In truth, from an overall industry perspective, it is tough to provide concrete answers to these queries. The Securities and Exchange Commission is not expected to release its cluster of disaster-recovery regulations until later this year, and the Securities Industry Association - the industry group that has formed a Business Continuity Planning (BCP) committee - did not publish its best-practices guide by press time. In that report, the SIA expects to provide a disaster-recovery roadmap, detailing all the steps a securities firm should take to keep its business up and running in specific emergency situations. "Really, this (best-practices guide) is intended to help member firms navigate BCP - (outlining) what they should be including in their (disaster-recovery) process ... . But this is (only) an interim measure, and once the (SEC's) regulations are announced, we will work with those," says Paul Honey, chairman of the SIA's BCP committee.
Honey, who declines to comment on his role as director of global-contingency planning at Merrill Lynch, says that one of the issues the SIA will address in its best-practices guide is the need to build proper backup-trading facilities. But many firms have already taken steps to address that area.
Calling For BackUp
In the wake of Sept. 11, no single disaster-recovery issue has drawn more attention than backup-trading floors. Before last September, many securities firms deemed it prohibitively expensive to operate and maintain a redundant trading floor. But backup facilities, of one kind or another, became a necessary disaster-recovery expense in the wake of the attacks.
Some securities firms have already built disaster-recovery floors on their own, while others have outsourced trading-floor recovery to a third-party vendor. In addition, a handful of exchanges - cognizant of the high-costs tied to building a redundant trading site - have also weighed the pros and cons of sharing a disaster-recovery site with a competitor.
Perhaps no firm is as qualified to talk about what it takes to create a backup site than the New York Board of Trade - the only Manhattan-based exchange that actually built a disaster-recovery-trading facility prior to Sept. 11. Since its trading pits at 4 World Trade Center were completely destroyed by the attacks, NYBOT - an open-outcry-driven commodities market - had to quickly relocate all of its staff and member firms to its Long Island City-based backup-trading facility. Fortunately, due mainly to lessons learned from the 1993 WTC bombing, the NYBOT was prepared. "What we learned in 1993 was that in order to preserve (your) business, you had to make sure that you had someplace to go in the event of a disaster," says Gambaro.
At first, the NYBOT's board rejected the request of its IT team to build a backup facility, citing the fact that it would be too costly. But after the IT team brought in an outside consultant that predicted the NYBOT could potentially lose $350,000 per day in exchange fees if it ever had to shut down its WTC headquarters, the board changed its mind and gave the backup site a green light. The NYBOT subsequently decided to house not only its backup facility but also its backup-data center in Long Island City. However, through the years, the IT team had to regularly do battle with the board to keep the site operational, says Gambaro.
The NYBOT, he says, had to pay a one-time charge of $200,000 to build the backup-trading facility, and an extra $150,000, annually, to rent the space. But the exchange was able to keep the costs of the Long Island City site relatively low by recycling its outdated computer equipment. "Every time we put in a new piece of equipment at 4 World Trade, we moved the old equipment over to Long Island City ... . The equipment was already paid for and the maintenance was included in the original licensing agreement we signed for the computers," explains Gambaro.
To make sure that the backup facility was up to snuff, NYBOT also ran tests, on a quarterly basis, and performed software upgrades whenever it made a change to one of its operating systems at its headquarters. All the hard work and expenditures paid dividends last September, when the NYBOT moved into the Long Island City site - a facility the exchange still uses as its primary trading floor. Due to all its preparations, says Gambaro, the exchange could have gone live at its backup facility at 8:00 p.m. on Sept. 11. But, citing the industry's telecommunications problems, the NYBOT decided to hold off until Sept. 17, when it traded 140,000 lots - roughly twice the average-daily volume it was recording at 4 WTC.
Eventually, says Gambaro, the NYBOT built eight pits to accommodate trading of all its products. Moreover, not satisfied with relying on Long Island City as its one-and-only trading facility, the exchange also recently built a backup-trading floor at 39 Broadway. Not surprisingly, all of the work the exchange put into disaster recovery has not come cheap. "We have probably spent about $10 million, since Sept. 11, to accommodate our business," says Gambaro. "We have spent between $5 and $7 million at Long Island City, and about $2 or $3 million at 39 Broadway."
To ease some of that financial burden, the NYBOT is searching for a partner, such as a regional exchange, to share its Long Island City trading facility. By the first quarter of 2003, says Gambaro, the NYBOT expects to permanently move its primary-trading floor from Long Island City to the New York Mercantile Exchange's headquarters, and the exchange hopes to find a partner to share its backup costs prior to that date.
One market that will definitely not be sharing any space with the NYBOT is the NYSE. The Big Board, says an exchange spokesman, has already built a backup-trading facility "somewhere in Manhattan." What's more, while declining to provide more details about the disaster-recovery site, the spokesman says the NYSE is also thinking about building a secondary, active trading floor outside of Manhattan. "We've looked at the possibility of establishing a second active site ... (but) it is unclear what portion of our daily activity would be allocated to that second active site, if we decide to do this," says the spokesman, noting that the exchange is considering sites in Westchester County, N.Y., among other areas.
Tweet This