Wall Street & Technology is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management

01:30 PM
Connect Directly
RSS
E-Mail
50%
50%

How BGC Partners Stays on Top of the Flood of Trade Data

BGC Partners, a global inter-dealer broker, uses Sybase's IQ column-based database to store and analyze the escalating tsunami of trade data.

"In the last five years, the amount of data being collected has exploded," notes Chris Crosby, SVP of corporate technology at BGC Partners. "Through technology we're able to capture every tick and transaction around the world. People have all this data and wonder how to use it and how to get at it."

Standard-issue database servers from Oracle, IBM and Microsoft don't cut it anymore, according to Crosby, because they cannot house nor query such enormous volumes of data.

Enter Sybase IQ, the column-based database and analytics engine BGC is beta testing and plans to put into production soon. The IQ database stores large volumes of data economically by saving only new elements of data updates to the database rather than re-saving the entire entry, Crosby explains. (For instance, in tick data, generally the price is the only thing that changes with each update.)

The column-based (rather than row-based) format of Sybase, Crosby adds, also improves efficiency and retrieval speed. "Nine times out of ten, you're not selecting everything, you're selecting very specific information," he says.

"Over the many years I've been here at BGC, we've had vendors come in and show us various tick warehouse database products," Crosby notes. "Most can run on a fast desktop PC under some trader's desk and do real-time analytics on a large portion of data throughout the day or a couple of days, but they weren't persistent and they didn't go back in history unless you separately loaded up historical information." In addition to all of these capabilities, the Sybase database can be queried with standard languages and protocols, whereas typically products that rely on complex event processing often require proprietary languages and conventions, Crosby adds.

Accelerating Analysis

A high-speed Sybase ASE engine captures BGC's fixed income transactions, batches them and sends them over to the IQ servers for storage, reports and queries, Crosby explains. "We aggregate data on the fly, so every five minutes we sum up and roll up transactions by product and by counterparty, and we store that because our management wants to know how many trades we've done in each product and receive other summarizations," Crosby says.

"We found it was difficult to query ASE and do those summaries on the fly," he continues. "When we put the same data in Sybase IQ, we found that we could run the summaries on the IQ server against the raw data in about half the time. So not only did we see data space savings, we've reduced complexity by not having an aggregation engine and we increased our speed of delivery. It's a big bang for a little buck and it streamlines overall storage."

The firm, which offers voice and electronic brokerage services in the fixed income, rates, foreign exchange, derivatives and futures markets, also plans to move historical data — every electronic trade it's placed in the past eight years — over to IQ, according to Crosby.

The primary driver behind the new database and analytics platform is to be able to perform trend analysis on behalf of customers, Crosby relates. Currently black-box trading shops tend to react more quickly to market movements than non-black-box customers, he says. "One of our goals is to find out how our customers are suffering in the wake of black-box trading and how we can help them be more competitive," Crosby adds.

Crosby estimates that when BGC goes live with the database product, it will start capturing 2 million columns a day, reaching a terabyte of data within a year.

A New Sun

BGC Partners runs the IQ database on Sun Solaris servers. It recently began using the container-based virtualization built into Solaris. "That's helped our Unix admin team quickly allow groups of developers to have their own servers — they can bring them up and take them down virtually any time and any where, which makes it extremely flexible," Crosby notes.

Sybase in late February announced version 15 of ASE and IQ, and BGC plans to upgrade to both, eventually. "Upgrading is never easy," Crosby acknowledges. "You have to test everything thoroughly to make sure you don't have any snafus, that your queries and processes that were working fine on the older version of the platform are now working well."

BGC Partners separated from its parent company Cantor Fitzgerald in 2004 and listed on Nasdaq in April 2008. Cantor, which as a tenant of the World Trade Center was devastated by the September 2001 terrorist attacks, has only four database administrators in the U.S., and two of them devote all their time to support, Crosby concedes. "When you look at the number of servers they already have to deal with and then look at rolling out a brand-new version of database software, it's an arduous task," he says.

Register for Wall Street & Technology Newsletters
Video
5 Things to Look For Before Accepting Terms & Conditions
5 Things to Look For Before Accepting Terms & Conditions
Is your corporate data at risk? Before uploading sensitive information to cloud services be sure to review these terms.