BCP’s Balancing Act
Synchronicity
Comment letters quickly attacked the paper as being too proscriptive, noting that such a separation would preclude financial-services institutions from performing synchronous replication. Such replication is seen by many as critical in an industry where seconds lost in processing institutional trades could mean millions of dollars in discrepancies.
Ravi Apte, chief technology officer with the American Stock Exchange, says the Amex is only interested in synchronous replication. "Given the kind of business we are in, we don't want to have a situation where we are five or 10 minutes out," he says, referring to the time delay for data replication in an asynchronous environment. "Suppose we get 50 or 100 messages a second. Being out one minute might mean losing 6,000 orders and that is not a good situation to be in." In asynchronous replication, trade information is sent to a backup data center in batches, so whatever has not been sent at the time of a disaster may be lost.
Nasdaq backs up its data in an asynchronous environment. To offset the inherent risks, the all-electronic exchange is beefing up security at its data center in Connecticut. A more secure data center reduces the chances information might be lost before being replicated in its Maryland-based backup facility 300 miles away.
Steven Randich, chief information officer with Nasdaq, contends the exchange would lose less than a second of data if its Connecticut center was destroyed. "We believe that losing a fraction of a second, given the severity of such an incident, is an acceptable risk," he says.
Lehman Brothers' Chief Technology Officer Bridget O'Connor agrees that asynchronous data replication may offer an acceptable risk/reward proposition. "We're looking to move farther out than we currently are," she says. Today, the firm maintains data centers in Jersey City and on 10th Avenue, with trading floors on 7th Avenue and Park Avenue.
"We're playing with the technology," says O'Connor. "My sense is that on the horizon will be technologies that allow you to do asynchronous replication and potentially put at risk a transaction or two. Our thinking is that we're going to push it as far as is reasonably possible."
Michael Haney, a senior analyst with Boston-based Celent Communications, says that the pushback against the 200-300 mile mark has as much to do with budgetary limitations as technological ones. He describes the minimum separation requirement as being "cost prohibitive." The suggestions in the original white paper "cost money and we're in a depressed economy," says Haney. "Financial institutions are struggling for ways to cut costs and they'd rather try and wait for profits to increase before they make these investments."
"There is no minimum industry standard," he continues, "but the standard is whatever is achievable cost-wise without driving firms into bankruptcy."
Jim Simmons, CEO with SunGard Availability Services, agrees that most firms are not in a financial position to execute on their dream BCP plan. "Considering the economic conditions, C-level executives are trying to do what's right to protect their information availability while keeping an eye on the shareholders," he says.
Marty Colburn, chief technology officer with the National Association of Securities Dealers, says that it's best to let firms work within their budgets rather than devising regulations that dictate a one-size-fits-all approach. The NASD has proposed a rule (3510) that would require all NASD-member firms to file a business-continuity plan with the self-regulatory organization.
Colburn admits, however, that the NASD will not vet the plans unless asked for guidance, at which time it will likely refer firms to experts in the BCP field.
One of those experts, says Colburn, could be SunGard.
Simmons says SunGard can offer firms a variety of flexible and scalable solutions. From offering backup trading seats on a dedicated basis to creating time-sharing arrangements where two or more firms have the right to access a backup facility.
As for the replication debate, Ken Smith - president of SunGard Planning Solutions, the consulting wing of SunGard Availability Services - explains synchronous backup has distance limitations because there are a number of steps involved in the process. First, a trade is written on a local disk drive, which can occur quickly at short distances. Then, before another transaction is recorded, that order would have to be sent, for example, to California, to be written on a backup disk, says Simmons. Once written in California, the New York site would need to receive a confirmation that the backup was successfully created.
"In computer speeds that's a horrendously long amount of time," says Smith, noting even a sub-second delay can mean the loss of hundreds of orders. "Unless someone changes basic physics, the solution is better asynchronous processing."
Smith says vendors like Hitachi, EMC, HP and Compaq are hard at work on such a solution. "If any vendor cracks that nut, they'll sell a lot of disk drives," he says.
Ultimate geographical dispersion, however, is a fact of life for firms with a global footprint. Paula Larkin, a managing director and head of corporate-business resiliency and crisis management with JPMorgan Chase, says having a presence in other countries means more BCP options.
"We cross-train our people in different regions of the world to help us get through a crisis," she says. "We push the dual-processing model so that we can do business in multiple places which provides us with significant advantages," she says.
Jerry Klawitter, vice president of business continuity with JPMorgan Chase's investment bank, says that trends like offshore outsourcing can also help geographically diversify a firm. "Offshore outsourcing, in some cases, you can dovetail with business-continuity planning by splitting operations at the same time. Sometimes you can pull those two strategies together," he says.
Building A Strong BCP Structure
Larkin says that divvying up processing on a permanent basis helps mitigate concentration risk. "It becomes business as usual," she says. You don't have to test that because it operates on a daily basis."
Many others agree the key to BCP is integrating resiliency into every function on a daily basis. Morgan's Ferris says executives should take BCP tags out of their thinking and rather "absorb that into the business-as-usual model." By its very nature, he says, creating operational diversity at the levels of real estate, technology and personnel will result in a strong model "that is difficult to take down with any single event."
Also helping to create a strong model, says Ferris, is providing centralized BCP guidance while allowing business units to create and test their own plans. At Morgan Stanley, a representative from each unit acts as a point person to the central BCP group. The two then interact as plans are vetted and finalized.
Aaron Meckler, senior vice president of corporate BCP with Wells Fargo, says his firm works much the same way. At Wells, he manages the central BCP team that establishes standards for the business lines to follow. "We can look across the enterprise for best practices and synergies," says Meckler, directing resources "much like a traffic cop."
Celent's Haney says he knows first-hand that some degree of centralization is critical. Haney says before joining Celent, he was working with a multinational financial institution whose New York office never rehearsed its BCP plan. However, the firm's Tokyo branch conducted annual testing. "A lack of consistency in firms is definitely something that needs to be addressed," he says.
Morgan's Ferris says that after financial institutions have addressed inconsistencies in their own shops, it will be time to take a look at the BCP plans critical counter-parties have in place. His firm is still in the collection phase of gathering the BCP plans of its major partners and providers, but when that is completed, the examination of those plans will begin. Such inter-firm vetting, he says, will only serve to make the industry more resilient as a whole.
Ferris explains, "Clearly if we go through a plan and it's horribly filled with holes, yes, that's going to be expressed."
Tweet This