Capital Markets Outlook 2013: Demand for Deep Analytics Challenges Data Managers

Why It's Important: It's been said that data is the business on Wall Street. Without proper pricing, valuations and information on customers and counterparties, the business would grind to a halt. Today, however, the demands on the traditional data management organization are increasing: Business units are asking for better analytics, risk managers want access to more data, regulators want a clearer picture of a bank's financials, all while unstructured data continues to grow.

Where the Industry Is Now: The industry has collectively spent billions over the past few decades to get its data house in order. Numerous efforts, including those from the industry standard working group Enterprise Data Management Council, have helped improve data management, but the proliferation of unstructured data has added a layer of complexity.

"In the capital markets, organizations are spending a lot of time and money focused on the amount of data they acquire," says Dave Schuette, chief informationist at Knowledgent, an information management and analysis consultancy. "In turn, they're spending a lot of money on more advanced analytics based on that data acquisition."

Business leaders are looking for ways to incorporate analytics on vast amounts of data into everyday decision making processes. Data managers are turning to big data technologies to handle existing data volumes. And it looks like companies are willing to spend on big data. According to IDC, an industry analyst, the big data technology and services market will grow at a 39.4 percent CAGR through 2015.

Luckily, there's funding for these data initiatives. "We're seeing many more data management projects that are formally funded," says Marc Alvarez, senior director of reference data infrastructure at Interactive Data Corp. "Previously, it was more ad hoc. There's more of a serious funding strategy focused on data right now."

Focus In 2013: Big data technologies, which many firms have been testing and piloting, will likely see targeted project implementations focused on risk management, operational efficiency and new product development, says Schuette. However, there's a lot of uncertainty and skepticism around big data. The technology vendor hype cycle has confused users, lumping cloud, big data and likely a few other terms, into one group. "The hype and use of the term is overused," says Schuette. "People are using it with cloud, and it's creating more hype than reality."

[Data Center Spending Is Simply Unsustainable]

Still, fear is one of the real drivers of big data management. "One of the biggest drivers is the fear of not doing it," Schuette says. "What if a firm's competitors are doing it, and they come up with a unique product or gain a competitive advantage?"

Industry Leaders: The larger firms, with well-established data management organizations, are best prepared to incorporate the vast amounts of unstructured data that big data provides. Some large firms are setting up new Apache Hadoop infrastructures. There's also a lot of focus on managing metadata and taxonomies and semantic analytical capabilities. "That's something that needs to be invested in, and this is where a lot of special sauce is," Schuette says. "The larger firms are spending the time, effort and money to establish big data capabilities."

Technology Providers: Many well-established tech vendors, including EMC, IBM, Oracle, SAP Sybase, SAS and Teradata, offer big data products. Financial market data providers, including Bloomberg, Interactive Data and Thomson Reuters, are increasing the types of data available in their feeds.

In addition, a number of open source options are available, including Apache Hadoop, a framework for data-intensive distributed applications; R, a programming language; Cascading, a software abstraction layer for Hadoop; MongoDB, an open source NoSQL data store; and Scribe, which aggregates log data streamed in real time from a large number of servers.

Price Tag: Up-front costs for open source products are minimal, but support and other costs often add up quickly. Larger traditional database vendors offer a number of products at various price levels based on data set size, processing power and other variables.

Greg MacSweeney is editorial director of InformationWeek Financial Services, whose brands include Wall Street & Technology, Bank Systems & Technology, Advanced Trading, and Insurance & Technology. View Full Bio