IT CHALLENGE

THE CHALLENGE
Suresh Kumar, managing director and CIO at Pershing, wanted faster recovery time from a disaster than his third-party vendor could provide. So, he built his own mirrored-data center. Here's how he did it.

The events of Sept. 11 hadn't even happened yet, but Suresh Kumar, chief information officer of Pershing, realized that his business-continuity efforts weren't up to the speed he wanted. He had outsourced that task to a third-party vendor. Unfortunately, though, if many of the vendor's clients went into disaster-recovery mode at the same time, Kumar says it could be as long as 24 hours before things were back to normal.

That was far too long for his liking, given that Pershing, - a division of Donaldson, Lufkin & Jenrette Securities Corporation - provides global correspondent and prime-brokerage services to more than 750 institutional and retail-financial organizations, registered-investment advisors and managed-account programs.

"We were looking to see how do we bring that down to something reasonable? We needed to recover from a complete disaster in our primary-data center and be back in businesses in no more than four hours." Kumar says he was attracted to the concept of mirrored-data centers, where information is backed up instantaneously in a separate off-site facility.

So, in March 2001, he hunted around and learned that, "IBM had this technology where, at the hardware level, you could replicate the complete storage environment in real time at another site," which Pershing adopted.

The system Pershing built comprises 10 IBM TotalStorage Enterprise Storage Servers, known as Sharks, two IBM eServer z900 mainframes and IBM's Geographically Dispersed Parallel Sysplex (GDSP) and Extended Remote Copy software (XRC), which allows Pershing to copy more than 20 terabytes of data, equal to about 20 million 400-page books.

By December of last year, the system was up and running. The copying time is less than nine seconds and recovery time has been cut to a few hours, says Kumar, whose firm deals with about 36 million transactions per day, about 2,000 per second.

Kumar is not alone in his quest for faster recovery, as more and more firms are eyeing mirrored-data centers as a solution for business continuity.

Gary Foster, chief technology officer of Omgeo, needed a robust disaster-recovery system for his firm's central-matching solution, Central Trade Manager. He opted to build a mirrored-data center using technology from EMC Corporation, a major player in data-replication software. Foster says that as the industry moves to a one-day-settlement system, if a disaster takes place "and I don't have this in place, then I am out of business."

The key to Omgeo's setup is EMC's Symmetrix Remote Data Facility (SRDF) software, which maintains mirrored copies of Omgeo's production data at both an EMC remote facility and Omgeo's own remote facilities. Omgeo is also using EMC TimeFinder software to perform local backups.

Data replication is a growing area in the data-storage business. According to research firm GartnerDataquest, EMC has 54.7 percent of the data-replication market, compared to 9.2 percent for IBM and 5.3 percent for Compaq.

"Mirrored storage comes up in almost every discussion I have on (data recovery)," notes Larry Tabb, who heads the securities and investment practice at TowerGroup. That's because other storage options, such as tapes, no longer cut it. Tabb adds that more information is being communicated in real time, increasing the risk of massive data loss if something happens before a firm has a chance to back it up. Firms with near real-time replication systems "have been more successful in dealing with some of the disaster issues."

While mirrored-data centers can provide firms with relief from data loss, it can add stresses in term of costs. In Pershing's case, the system cost eight figures to build. Kumar says that while costs are one of the challenges in deployment, having a redundant system in place that mirrors existing systems means that it can also be put to use. "We want to be able to use both sites simultaneously for processing data," he says. However, he doesn't expect to be able to do that for another year or so.

Foster predicts that mirrored sites will become dominant in the investment community and, in fact, firms will eventually add a third-redundant site to the mix, likely in a distant geographic location. Higgins agrees, noting that Sept. 11 showed firms how exposed they were to disasters. Many were forced into their recovery site and then realized they had no further back up. With three centers and the ability to have them all live and contributing to the operation, if one goes down, the other two pick up the workload. What it means, though, is that IT departments need to revisit the notion of centralizing all their employees in one place, he says. Kumar says that with the "cost of bandwidth becoming inexpensive and plentiful, and CPUs becoming more and more powerful, it makes it easy for us to do these kinds of things."