Big Data: The Next New Thing?
Big data is a hot area in venture capital circles and on the conference circuit. With data spewing out of social networking sites like Twitter and Facebook and corporate web sites VCs have been investing in analytic platforms that can analyze customer behavior, optimize digital marketing and help companies with fraud detection.
I attended a lunch panel last week as part of the O’Reilly Media Strata Summit in New York about where Wall Street and technology meet. Most of the conference was about how do you apply technology to filter and curate massive amounts of data. Isn’t this what Wall Street has been doing forever? Why is it new?
Though Wall Street firms have been focusing on data management for about 20 plus years – scrubbing data, normalizing data and setting up data warehouses to create golden copies to feed risk management systems and back-office settlement systems— this is old stuff. So what is big data?
I asked a few people attending the lunch this question and was told, “Yes, Wall Street has been doing this but now the rest of the world is catching up.”
According to Terradata, which acquired Aster Data Systems, one of the data analytics companies sponsoring the event: “Big data is a term commonly applied to large data sets where volume, variety, velocity, or complexity are beyond the ability of commonly used software tools to efficiently capture, manage, and process. Examples include Web logs, sensor networks, social media, medical records and more.”
In high frequency trading, firms can look for unusual patterns in complex data sets across asset classes and geographies. Now that the cost of servers with multi-core processors has fallen, many businesses can afford to crunch large data stores using analytics. But has the Internet democratized access to information and leveled the playing field for investors against the most savvy hedge funds and prop trading firms?
The answer is Yes, if you can afford the tools and infrastructure required to process what’s known as Big Data. One panelist was skeptical of this outcome.
Roger Ehrenberg, a managing partner and founder of IAVentures, who spoke on the panel, suggested the Internet has actually raised the barriers to entry. In fact, the cost of managing the gross amounts of data goes up, asserts Ehrenberg, who points out that it’s the synthesis of data using the tools that has become difficult. According to Ehrenberg, quant hedge funds like Renaissance Technologies, Bridgewater, 2 Sigma and AQR that have unlimited budgets for data technology and dedicated teams to clean, index, store and create the data stores, he said. “As democratization of information has exploded, the cost of extracting an information signal from that mass of data has exploded, (as well),” said Ehrenberg. It’s the synthesis of the data that has gotten so difficult,” said Ehrenberg.
That is why the topic of Big Data is on the radar screen. Also, the costs of commodity servers have come down and speed has gone up. Whether the science behind big data is new or not, the technologies of data mining, business intelligence, standard query language (SQL), and data storage, are being packaged up for consumption by all businesses and industries. And that is democratization. With new laws like Dodd-Frank Wall Street Reform and Consumer Protection Act requiring regulators to conduct surveillance on derivatives transactions, the tools of big data are going to be in demand.Ivy is Editor-at-Large for Advanced Trading and Wall Street & Technology. Ivy is responsible for writing in-depth feature articles, daily blogs and news articles with a focus on automated trading in the capital markets. As an industry expert, Ivy has reported on a myriad ... View Full Bio