ECNs Battle Blackout

Thursday's blackout sorely tested the reliability of the nations' electronic-communications networks (ECNs) -- computerized networks that match buy and sell orders in stocks. In their first major test since Sept. 11, the ECNs switched to backup generators and grappled with phone problems.

Instinet -- the largest ECN which also operates Island -- converted to backup power and was able to remain open after Thursday's 4:15 p.m. outage, reports Andrew Goldman, executive vice president at Instinet Corp. But it did cancel its after-hours crossing session.

Instinet's trading floor, located on the eighth floor of the Reuters building at Three Times Square, had several days worth of backup power, says Goldman. It "flipped over to backup without a hiccup and stayed up and running all through the close of trading and beginning Friday morning," he says.

But, "Island had a different proposition that it faced," Goldman adds. "If subscribers lost connectivity, their open orders would not be canceled, and they would be exposed to the market. We went through a controlled shutdown starting at approximately 4:30 p.m. Thursday. The sole concern was being able to ensure the integrity of the trading system for customers, and we worked very closely with them to make that happen."

The outage also affected Bloomberg TradeBook, located on 59th Street and Park Ave. It swung over to backup sites and was up and running on generators, says Kim Bang, president of Bloomberg TradeBook.

Wally Sullivan -- CEO of Pulse Trading, a Boston-based agency broker that executes orders on behalf of institutions -- agrees. "All in all, the ECNs did perform very well," but, "they were acting funny," adds Sullivan. He says that some of the market data was slow and there were crossed markets, signaling that quotes were stale. "You didn't see the integrity of the markets as far as the fast, electronic tighter spreads that you typically see," he adds.

In Sullivan's view, the ECN that had the fewest problems was Chicago-based Archipelago.

Steve Rubinow, chief technology officer at Archipelago, claims the Archipelago Exchange (ArcaEx) does have an advantage over ECNs based on the East coast. "We have a primary site in Chicago. We only had to worry about the alternative site in New York," he says. However, that is not to say that the New York site is not important. "Many of our New York customers connect to us through that site. It's a very important site for us," notes the CTO. "It has active production value."

Once the ECNs were operating on backup power, the biggest problems they experienced were phone issues.

Instinet had problems with phone lines, mainly related to Verizon and MCI, says Goldman. "The greatest lesson this taught us was the same one coming out of 9-11 -- redundant connections with phones and connectivity to backup systems. That's what gets you through these infrequent episodes," he says.

In Archipelago's case, "There were a couple of blips during the day where we weren't able to receive calls," says Rubinow. Staff had to use cell phones and Blackberrys to communicate between Chicago and New York, says the CTO. Archipelago also used a mixture of carriers, including AT&T and Cingular.

According to Bang, Bloomberg TradeBook also had problems with telephones. "We had to route telecommunications via London and, in some cases, bounce back calls to the U.S.," says Bang, who was in London at the time.

To restore connectivity with their customers, ECN employees burned the midnight oil.

Goldman says, "It was an amazing effort by the folks who stayed there through the night on Thursday so they could be up on the desk 6:00 a.m. Friday to try to work with customers. There are always heroic portraits that emerge in these situations." One employee walked 10 miles Thursday night and 10 miles back Friday morning, he says. Ivy is Editor-at-Large for Advanced Trading and Wall Street & Technology. Ivy is responsible for writing in-depth feature articles, daily blogs and news articles with a focus on automated trading in the capital markets. As an industry expert, Ivy has reported on a myriad ... View Full Bio