Nasdaq Overloaded with Capacity Issues, Defends Decimalization Delay

Can the Nasdaq cope with the capacity issues that loom ahead? Gregor Bailar, the executive vice president and CIO at the National Association of Securities Dealers, says yes, but it just needs some time. That's one thing that the industry doesn't have. Volumes are increasing at a rapid pace, market volatility translates into additional quotes, and decimalization is supposed to be just around the corner.

And while even the most wizened investors don't know if the market will rise or fall, Nasdaq's IT department understands that either way, network traffic will rise in a volatile market, because a multitude of transactions cascade from every order executed. Bailar looks in only one strategic direction--up. "The growth rates are pretty astounding," he says. Indeed, Nasdaq's rapid expansion has resulted in an outstripping of its systems, facilities and forecasts.

Capacity planning is ongoing and critical to the future of Nasdaq. "We tread the line on how to forecast the future," says Bailar, who contends that Nasdaq's track record has been pretty good. The process is to identify peaks in volume and then move to ensure that capacity is great enough to handle volumes much higher than necessary at any given moment. Last year, daily peaks were estimated in the 2 billion-shares-traded range, with "headroom" for up to 4 billion shares.

But a huge spike in the fourth quarter of last year meant a whole new blueprint had to be drawn up. Nasdaq was on a growth trajectory of 15% to 20%, "and all of a sudden, it went through the roof to 2.88 billion shares," Bailar says. "Boom! It's off the charts. Now we're in the 4 billion range, shooting for 8 billion" shares. That's well beyond its current built-in headroom. The rapid rise is forcing the exchange to ramp up network capacity fast.

Not surprisingly, Nasdaq suffered network delays and slowdowns during this period, according to investors. While the exchange says its network offers a 99.86% service-availability rate, "They haven't made it as glitch-free as it could be," says Tony Broy, CEO of Hill-Thompson Magid & Co., a market maker based in Chicago. Broy, a NASD governor, also sees network speed as a problem. "It's not as fast as it could be. When the market is going full speed ahead, Nasdaq had many more failures than should be acceptable."

In particular, the investment community points to a day in April when trading was slowed as Nasdaq tried to fix a glitch in a SelectNet trading system file. "A file in SelectNet was filling up," Hickey says, so "we added a second file online and had to suspend operations for a few seconds. As a result, there was a slow response time for five firms," and Nasdaq had to expand the pointer files that direct requests to records. "Those files have been permanently expanded," he says.

Nasdaq is trying to head off even small glitches with major overhauls. The exchange has accelerated its much-vaunted five-year plan, designed to support its new, highly scalable SuperMontage architecture. SuperMontage is Nasdaq's next-generation trading system, which will support current capabilities and new trading practices in an integrated architecture. Through an automatic execution mechanism, all quotations and orders are brought together in one window for the trader to view.

SuperMontage will replace SelectNet and the Small Order Entry System the exchange now uses. Nasdaq is banking on a multiparallel architecture from Compaq's Tandem Computer division and an offload balancing scheme-the basis of SuperMontage-to make Nasdaq's systems flexible for future volume spikes. As recently as late last year, Nasdaq thought the original expansion plan was generous in the amount of capacity headroom it would offer during the next five years.

However, Bailar has had to put the exchange's five-year plan into high gear. Nasdaq has taken a long look at each network component and "the way each box can scale," Bailar says. The goal is "to scale any component horizontally as opposed to getting bigger machines. We're outstripping the biggest machines now," he says. Multiple systems share the load, and disaster recovery is embedded in the system. "Now we have a backup to every backup," Hickey says, which includes mirroring, testing and increased redundancy.

All the systems are coordinated at the Nasdaq Data Center, which takes up 26,000 square feet on the first floor of the Nasdaq Technology Center in Trumbull, Conn., with room to grow to the second floor. Unisys and Tandem systems are at the heart of the trading system, which connects with servers at more than 600 market-maker firms. Traders connect into Nasdaq through computers called Service Delivery Platforms. The IT staff represents more than 700 employees.

Nasdaq's IT staff and vendor representatives monitor system operations in the 4,300-square-foot command center in Trumbull. In one quadrant of the room, 24 MCI WorldCom employees monitor the Enterprise-wide Network (EWN II) MCI WorldCom runs for Nasdaq. All operating data is presented in graphical form so the IT staff can easily monitor system performance and spot glitches before they become real problems. While traders focus on the spikes and dives of stock prices, IT staff vigilantly watch a system of fluctuating bars-like blips on a radar screen-that monitor the health of network performance. "The system will begin to stress at about 3.45 billion shares," Hickey says.

Adding to Nasdaq's capacity woes is the rise of electronic communication networks, which account for 30% of Nasdaq's volume, and extended trading hours. Round-the-clock trading is clearly where stock markets are heading. On Oct. 25, Nasdaq expanded its trade reporting and quotation system hours until 6:30 p.m Eastern time. It plans to offer trading in Europe, India, and Japan, pushing the exchange one step closer to becoming a round-the-clock market.

Also affecting capacity is the SEC mandated change to decimal pricing. Though the software code has been written for decimalization of some stocks-Release 1.0 of the Integrated Quotation Management System is available-Nasdaq has told the SEC that it will not be ready for the change until February 2001. At that time, Release 2.0 will finally handle volumes in the 4 billion range. That's about the same time that the SuperMontage effort should be completed.

The Nasdaq Stock Market is actually an amalgam of systems, including quotation, trade, and execution, confirmation, and support systems, that form an electronic exchange. Each of these financial systems, vital to Nasdaq's operations, features its own technology. For example, the trade and execution systems, which are responsible for switching messages as well as executing, reporting and reconciling trades, run on 40 high-end S72000 Tandem Himalaya computers-a total of seven production nodes with 108 processors located at the data center in Trumbull. The system also features more than 15 Gbits of main memory and more than 1 terabyte of disk storage. An additional eight Tandem nodes are used for software development, quality control and testing.

Unisys 5800 Clearpath eight-processor mainframes once ran Nasdaq's quotation system, which provides up-to-the-minute quotation data for Nasdaq securities and other issues. These CPUs were upgraded to Unisys 66800 systems as part of the SuperMontage initiative. The system is backed by 1,024 mbytes of memory as well as 452 gbits of cache disk storage space and 192 mbytes of solid-state storage.

Nasdaq also operates an identical and fully redundant backup center in Rockville, Md. Whatever can be done at the data center in Trumbull can be done in Rockville. Often, they operate concurrently, Hickey says. Given the recent surge in volume, Nasdaq will expand its Trumbull facilities this summer. Bailar says Nasdaq will spend more than $1 billion on new technology during the next five years.When the data center makes the move to its expanded quarters, it will have lots of new IT luxuries. For example, the floor features the 36 inches of space required to house wiring in neat "streets and avenues," to make pulling and laying wire less of a chore. A new cooling system that can chill 25,000 gallons of water comes into the center from two sources.

Nasdaq learned the importance of dual power sources in 1988 when a squirrel fried itself on an electric company's transformer and brought the exchange to its knees for a day, Hickey says. An energy system with an uninterruptible power supply ensures that the system stays up even if both sources of power are lost, and the exchange has amassed 132 tons of batteries to serve as a back-up source of power.

On a separate note, Davidson worries that plans for a global Nasdaq-with plans to create a Nasdaq Japan--will shift business away from the smaller U.S. companies as investors turn to other dealers for round-the-clock service. And the dealers also are concerned that there will be even more slowdowns as Nasdaq expands.

Bailar says Nasdaq isn't going to expand into foreign markets until it knows that its exchanges in each country can handle the volume. He and Hickey contend that many of the slowdowns result from old technology used by small traders themselves. "We've done everything on our side of the wall," he says, but some firms are still using 200-MHz systems when they should be at 400 MHz or faster. The slower systems time out before orders can go through and firms resend them. As a result, there are extra traffic and additional slowdowns on the Nasdaq network, Hickey says.

Another concern among NASD members is that plans to spin off the exchange into a separate company-which got the go-ahead from NASD members in April-and eventually go public with an IPO, could add more chaos to an already volatile situation.

To Hickey and Bailar, these moves will be good for Nasdaq and, in turn, its customers. For one thing, it will give the exchange an opportunity to compete with dot-coms for the best IT talent, Bailar says. That could lead to lower turnover among Nasdaq's employees and contractors, which now ranges from 7% to 9%. "It's hard because we're not a dot-com officially-we don't have stock options," says Bailar. "But we do have really exciting work-the people who come here are people who want to go to Mars." Providing better compensation and benefits is one way to keep IT professionals bullish on Nasdaq-and that might just be the most important cog in keeping the Nasdaq wheel turning in the face of all the upward momentum.