Weathering Sandy: How Wall Street Got Through The Hurricane

In addition to the trail of destruction and death left behind in its wake, Hurricane Sandy shed new light on just how hard it is for Wall Street companies to prepare for the unthinkable.

The storm undoubtedly will force leaders of New York City and its suburbs to assess where their approach to hurricane preparedness worked and failed. A similar scene is playing out in the financial services sector, and in the coming weeks and months, companies will make sure their practices for future disasters are up to par.

At ConvergEx Group, which operates electronic trade-execution platforms and supports trading in more than 100 markets around the world, a strong, dispersed IT organization and the ability to keep communications up were crucial factors in the company being able to function during the storm's most chaotic hours, says Bea Ordonez, the chief operating officer of its global execution business.

"We had our IT and support people working around the clock," Ordonez says. "We had people pulling all-night shifts to make sure their sites remained operational and functional." A number of New York–area operations had to rely on generator power, and the company also has a big London office that "was key to us being able to have a fallback plan to support trading operations," she says. ConvergEx's New York City headquarters are in Midtown Manhattan, so it didn't face the physical damage that companies encountered closer to Wall Street in Lower Manhattan, where flooding caused major damage in places.

The offices of Morgan Stanley and American International Group were among the hardest hit. The area surrounding Goldman Sachs's headquarters in Lower Manhattan was flooded (see photo, previous page), but the building remained operational thanks to generator power.

Generators proved critical for many companies, as did having offices and data centers— including networking resources—in multiple locations.

"You've got to have network diversity, not only being able to physically relocate technology and people, but also how this stuff talks to each other," says Robert Flatley, CEO of CoreOne Technologies, whose company provides financial data creation, management and distribution services for hedge funds and asset managers.

"If you have a single point of failure, and that's a router or something somewhere, and that building gets taken out, you're in trouble."

CoreOne is based in one of the city's flood zones, but its Midtown office—in addition to offices on suburban Long Island and in London; Hong Kong; Bangalore, India, and Santa Clara, Calif.—enabled it to remain operational. It hosts 300 operations and sites for about 220 companies.

Some companies learned the hard way the risk of having a generator in the basement, after their backup power source got flooded. Flatley says that during his days running electronic trading for Deutsche Bank, its generator was on the roof. "A lot of these buildings in Lower Manhattan, the backup generators were in the basement," he says. "It makes no sense."

SunGard expends a huge effort thinking through disaster-recovery scenarios, since it specializes in data center services to a range of companies, including many in financial services. But even it encountered some unforeseen eventualities.

The storm, combined with a seasonal high tide, broke a levee in Carlstadt, N.J., not far from the industrial park where SunGard has a data center operation. The data center had all its equipment on a raised floor, and the flood waters never got high enough to stop operations. Nevertheless, the SunGard recovery team found itself working farther down its disaster-recovery checklists than it ever had before at the Carlstadt location.

But the team spotted a potential gap in the plan. The data center needed fuel to replenish the backup generators. The delivery truck's route would take it close to where the levee had broken and the water was the deepest. So the team mapped an alternative route to ensure the truck didn't end up mired at some low-lying intersection. Soon, two 7,500-gallon trucks were delivering fuel in shifts to keep the complex's 40,000 gallons of reserves stocked. Existing plans assumed "any flood water would recede by the time the replacement fuel was needed," but that turned out not to be the case, says Nick Magliato, chief operations officer for SunGard Availability Services. "We had to think of a path from another direction for them to come in."

On the whole, the disaster plan worked as expected and the SunGard facilities operated continuously. But even the best plans need people prepared to do a little improvisation if needed. — With Charles Babcock

As the Senior Editor of Advanced Trading, Justin Grant plays a key role in steering the magazine's coverage of the latest issues affecting the buy-side trading community. Since joining Advanced Trading in 2010, Grant's news analysis has touched on everything from the latest ... View Full Bio