5 Steps to Turning the Market Data Crush Into an Advantage

The rise of algorithmic and high-frequency trading has simultaneously driven a seemingly insatiable thirst for real-time market data as well as an avalanche of message traffic. As a result, feed handlers must churn faster than ever to keep trading strategies and risk engines humming. But Wall Street firms are scrambling to find more efficient ways to analyze all of the readily available data.

"There's no time for human analysis," says Aite Group senior analyst Philip Lawton. "Firms have to be able to see and evaluate market exposure on an aggregate basis, and they have to recognize breaches quickly."

People assume that the more data you have, the better, adds Ralf Roth, global head of Elektron transactions and enterprise strategy at Thomson Reuters. But the more data you have, he points out, the greater the need for the right tools to read it and determine what is relevant.

Step 1: Create an Efficient Storage and Retrieval Platform

Before firms start attempting to draw hasty conclusions from their data, the most important step they can take to prepare for the crush of information is to establish an efficient means to cleanse, store and retrieve relevant market data, suggests Roth. To do this, he says, firms are looking at evolving database technologies. While data historically has been stored in vast databases with a comprehensive range of functionality, today a number of specialized database technologies have cropped up that are designed to deal with large amounts of tick data.

Conquering Big DataWith the rise of electronic trading, algorithms and now high-frequency trading, market data and messaging volumes have exploded. Our December digital issue looks at the tools and strategies capital markets firms are employing to handle the crush of data, identify trading signals, predict market movements and gain an edge.

"It's as if you tried in the past to have a vehicle that could do anything for you -- go fast on the highway and provide lots of space to transport things; you'd end up having a very generic vehicle that doesn't cater to specific tasks in an optimized fashion," Roth explains. "Now, if you want to go fast, you might say I want a Formula 1 car; or you might want an 18-wheeler that can carry a large load. Databases are the same -- people are moving from one that can do everything to one that is very good at dealing with analytics and tick data, but probably not a good choice to store HR data."

Step 2: Integrate Various Data Sets

In order to get a complete picture and more valuable analysis, organizations need to aggregate multiple data sources, looking at market data in conjunction with transaction data. "You need to combine two of the biggest data problems in the market," says Roth. In addition to storing market data, firms need to store and analyze every order sequence and every trade, and look at correlations between the different sets of data, he asserts. "If you want to look at how trades perform, you need to look at the transaction itself," he says.

Firms also should use analytics to derive new data from market data, Roth suggests. In addition to assessing the market impact of trades and how firms are performing against a trading trajectory or a benchmark, he explains, this approach will enable firms to review how they can use analytics to spot, for example, market structure trends. "It's very relevant in today's trading environment, where people are looking for all sorts of smart ideas on how to process data and extract it for themselves," Roth adds.

Step 3: Build High-Speed Analytics Capabilities

Once an efficient data management environment is established, firms must build high-speed analytics capabilities. To perform calculations in as near real time as possible, capital markets firms are employing the latest Intel-based, multicore machines as well as special purpose hardware, such as field-programmable gate array (FPGA) chips. They also are increasingly leveraging streaming technologies, including complex event processing.

StreamBase Systems just launched a business intelligence platform that connects to more than 100 streaming data sources for real-time trading risk management, network operations risk management, fraud detection and web analytics. The StreamBase BI system, which creates an in-memory data warehouse, sits atop StreamBase's CEP platform and evaluates all queries in real time as data changes, according to StreamBase CEO Mark Palmer.

Step 4: Provide Tools to Help Traders Understand the Data

Meanwhile, new data visualization technologies can help traders and portfolio managers sift through vast amounts of data, spot patterns and understand market trends more quickly, according to Thomson Reuters' Roth, who notes that data visualization has become an important part of many modern analytics solutions. Some hedge funds are now using computer-generated heat maps to filter and highlight market, pricing and other data that is relevant to their trading strategies. And new Adobe Flash-based platforms and clustering techniques, as well as more interactive performance reporting, are allowing firms to visualize massive amounts of unstructured data more efficiently and at a much lower cost than before.

Step 5: Look to the Cloud

Firms looking for more scalability, flexibility and speed increasingly are turning to cloud architectures for advanced analytics. "Some firms are now building modeling and their own analytics operations directly into the cloud," says Stephane Dubois, founder and CEO of market data provider Xignite. "They don't have to buy servers or deploy infrastructure."

To perform complex analytics on large data sets, he adds, "Historically, you would have to bring the data into the data center. Now you can pull it out locally in a cloud environment where you can build applications, and the data is already there. If you need to access terabytes of data, you don't have to deal with all the issues in processing this data."

All of this means increased speed to market. Peter N. Johnson, chief technology officer at BNY Mellon, noted at Wall Street & Technology's Capital Markets Cloud Symposium in May that deploying analytics in the cloud enabled BNY Mellon to cut provisioning from months or weeks to a few hours, "or sometimes even less."

Meanwhile, the lifespan of trading and risk models has rapidly declined, adding to the need for speed in developing and performing advanced analytics. Noting that a model typically is obsolete within three months, Xignite's Dubois says, "Firms need to ask themselves how quickly they can get the data to validate their models."

At the WS&T event, BNY Mellon's Johnson also stressed the importance of the speed-to-market advantages of the cloud. "We're focused on infrastructure as a service. There's money to be saved, and you will save on automation," he said. "But our real focus is on time to market from a development perspective."

Ultimately, Thomson Reuters' Roth contends, the best way to deal with the market data explosion lies in the combination of new database technologies, CEP and the right architecture. In addition, he asserts, analytics should be performed in a centralized place, as often the results are being used in multiple ways. "It makes a lot of sense to do the work once and have a back-end processing environment that does all the analytics work for you so you're not doing all the work twice," Roth says. "After all, you're using the same data and creating the same results but using it in different ways."

Melanie Rodier has worked as a print and broadcast journalist for over 10 years, covering business and finance, general news, and film trade news. Prior to joining Wall Street & Technology in April 2007, Melanie lived in Paris, where she worked for the International Herald ... View Full Bio