Big Data 2.0: Myth vs. Reality

Challenge: Despite the big data hype, financial firms are investing in tools and analyzing a variety of data sets to gain competitive insights.

Why It's Important: Social media sites such as Twitter are creating a firehouse of data, and along with unstructured text and comments, can be analyzed. "There's an unprecedented amount of data being created every day, and that data creation is being accelerated," says Keith Cooper, CEO of Connotate, a Web extraction and monitoring company. "The people who ... can analyze it have a decided advantage over those who cannot," says Cooper.

I think people need to be aware that the optimal term is ‘science' of data. I fear that people are expecting magic." -- Neil Palmer, SunGard Consulting Services

Where The Industry Is Now: Investment firms are mining data from Web scraping companies [like Connotate and others] to track tens of thousands of public companies around the world, for real-time or historical trends in stock prices, financial ratios, management changes and more. "The technology enables the aggregation of that data to allow them to think about it, analyze it and make bets on stocks in their portfolios," says Connotate's Cooper. Hedge funds are all about using Big Data to make specific bets on individual companies. They are collecting Sentiment analysis from Twitter, StockTwits, blogs and other social media feeds, along side financial data and using it in algorithms.

[For learn more about all of the topics that will shape the business technology landscape next year, download the November Digital Issue: Capital Markets Industry Outlook 2014.]

Focus In 2014: Capital markets firms are looking at two major applications of big data, according to John Avery, partner, SunGard Consulting Services. One relates to using "extraction transform load tools" or ETL to delay the mapping and transformation of data until the point of consumptions by a data warehouse.

The second is focused on analytical capabilities that firms want to add to their portfolios. Firms want to "analyze and perform analytics on new data sets to give them some business edge," says Avery. Sentiment analysis is on the radar screen as unstructured data like news and other data can be helpful in retention and acquisition of institutional clients. In compliance, data sets are captured both electronically and by analyzing voice calls from clients inside capital markets firms, notes Avery. While this is exploratory, it's obviously very important for the sell side, he says.

Industry Leaders: The Googles and Facebooks of the world are two obvious breeding grounds for big data. "It's interesting to watch their approach and whether they would impact capital markets," says Neil Palmer, partner in SunGard Consulting Services. In capital markets, Dunn & Bradstreet, Equifax and Experian are using Big Data to make informed credit risk decisions.

Technology: Open-source Hadoop a preferred choice when it comes to big data. EMC's Map Reduce is another option. IBM and Cognos are the main players in the business intelligence (BI) space. Hadoop 2.0 is about to come out. It will integrate with other BI tools such as Tableau Software and Spotfire for visualization and Business Objects.

Price Tag: Estimates range from $25-$50K per year to analyze data behind your firewall. Once a firm starts bringing in data from outside the company, the price tag jumps significantly, estimates Cooper. Ivy is Editor-at-Large for Advanced Trading and Wall Street & Technology. Ivy is responsible for writing in-depth feature articles, daily blogs and news articles with a focus on automated trading in the capital markets. As an industry expert, Ivy has reported on a myriad ... View Full Bio