CIO Challenge -- VoIP

The Challenge: Managing traditional analog hoot-n-holler networks, which are critical to investment management firms' operations, can be expensive and challenging, so many CIOs are eyeing alternatives. For some, the answer is IP telephony.

David Hughes, associate vice president and manager of the network server infrastructure at First Albany Capital, was looking for a way to better manage the financial institution's hoot-n-holler network, a communications network that the Albany, N.Y.-based asset management firm relied on for linking the firm's 18 offices in 12 states. He wanted to reduce expenses, improve business continuity capabilities and make sure that the service remained high.

Hoot-n-holler networks, also known as squawk boxes, are the heart of most investment firms. They are an always-on broadcast network that pumps out information to the firm and allows multiple parties to speak to one another simultaneously. They're used for things like the morning call and to allow traders to chat during the trading day by simply pushing a button. They've been around for at least 50 years, and they still rely on old technology.

"Basically, what we had was an old analog network," Hughes says, explaining that each office had a dedicated 56k line that handled hoot-n-holler traffic. The problem, he says, is that "You've got a very costly and hard-to-manage solution that doesn't provide a whole lot of redundancy."

For example, if one of the lines into an office went down, then the office was cut off and out of the loop until the downed line was fixed. It was far too expensive to add backup lines to each office, Hughes relates. Additionally, if First Albany wanted to add an extension to the network for a new staff member, Hughes was at the mercy of his local telecom carrier. It could take days for the carrier to make the change and it could cost thousands of dollars in service fees.

Adding to the complexity was the fact that the firm used three different hoot-n-holler systems to segregate its different business lines: equities, investment banking and fixed income. Hughes had to manage three different bridges to each carrier.

In 2002, Hughes began hearing a lot of talk about voice over Internet protocol technology (VoIP) and how corporations were using it to transmit telephone calls over their data networks. The voice signals are broken into data packets and sent along the data network to the recipient, where they are converted back into a voice call.

So Hughes met with the firm's equipment vendor, Cisco Systems, which already was touting the concept of HHoIP - hoot 'n holler over Internet protocol - and building technology solutions around it. In early 2003, First Albany ran a proof of concept involving three locations, its main office and two branches, using T-1 lines. The firm was impressed with the results; today, it runs its entire hoot-n-holler network via IP.

First Albany is not alone. Many investment management firms are eyeing IP as a solution to the challenges of managing their hoot-n-holler analog networks, notes Keith O'Brien, distinguished systems engineer, Cisco. It's a general trend in the industry to move what he calls "financial voice" off legacy analog systems and onto IP-based technology that allows firms to leverage their existing data networks. O'Brien estimates that IP telephony has a 50 percent penetration on Wall Street, as firms test various ways to deploy it in their organizations. The VoIP landscape "is changing rapidly," he says.

Impressive Growth

The growth of the VoIP market appears to be hitting critical mass at the enterprise level. Scottsdale, Ariz.-based research firm In-Stat/MDR reports that VoIP penetration at corporations quadrupled last year, rising from 3 percent in 2003 to 12 percent in 2004. It expects that to nearly double to 20 percent by end of 2005. Large enterprises are the biggest adopters.

Research firm Infonetics, which is based in San Jose, Calif., predicts that the carrier VoIP equipment market will experience annual compound growth of 39 percent over the next three years, rising from $1.3 billion in yearly revenues in 2003 to $4.8 billion by 2007.

Tarun Kapoor, CEO of Pleasantville, N.Y.-based VoIP software provider Pangean Technologies, says, "The technology has really evolved." A few years ago VoIP lacked the reliability needed to run something as critical as a hoot-n-holler network. "Now," he says, "the technology is ready for prime time."

New York-based Swiss American Securities is eyeing IP telephony for its hoot-n-holler system. CIO Christian Hudson says that his firm's relative small size - Swiss American employs 150 people - prevents it from being an early adopter. "We really can't be in a position where we test that type of technology," he says.

But Hudson knows larger firms are testing it, and he's watching and waiting. "I would say that the industry is on the beginning of the IP curve," he notes. "The cost savings are real enough to have everybody sit up and take notice."

Talk Is Cheap

First Albany's Hughes says, "The cost savings were substantial enough to raise some eyebrows." The analog network cost $125,000 a year to maintain, largely due to the expense of the leased lines and any moves and changes. It cost just $25,000 to add the HHoIP capability and some switches to First Albany's existing wide area network and eliminate the leased lines. Hughes says his firm was able to use its existing hoot-n-holler hardware, so it didn't have to scrap end-terminal devices, such as turrets, hoot phones or squawk boxes. Also, he no longer has to pay carrier fees every time he wants to add or drop people from the network; rather, IT staff can make the change quickly.

It took six months to roll out the project across the entire firm, Hughes says. The biggest stumbling block was configuring the system and finding the right quality of service settings so that each individual office came through loud and clear. "One thing that you learn to watch for is echo problems," where people are hearing every other word or the speaker sounds like it is under water, he relates. Adjusting settings on the routers, however, solved those problems, he explains.

It was "trial and error," Hughes continues. "Every network, depending on its topology, is a bit different. Whatever worked for us might not work for another firm." It involves lots of testing and tweaking of the local settings on the branch routers, which is managed from the central office.

The benefits are worth the effort, Hughes asserts. Not only has First Albany saved money, it now has the redundancy and business continuity capabilities that it sought before. The dual router format it deployed provided each office with Frame Relay and NetVPN carrier services, so if one service tanks, there's a failover to the other and the branch isn't dropped from hoot-n-holler connections, as it was with the analog system.

One of the unexpected benefits is better monitoring. "Basically, we didn't have an eye into the analog network," Hughes explains. That meant it was difficult to trouble shoot. For example, if there was feedback coming from a faulty device or someone accidentally pushed a button, then the problem would have to be isolated by shutting down different network circuits running the various groups of branches. Those people would be left with no service while the head office tried to isolate the branch with the problem.

"That's not an efficient method of troubleshooting," Cisco's O'Brien observes. With an IP system, he says, "There's a lot more visibility," and IT staff can monitor the network centrally.

For firms thinking about adopting an IP-based hoot-n-holler system, First Albany's Hughes has one piece of advice: "Do as much testing as you can and try to simulate the production network as closely as possible."