Petabytes of Data? Analyzing Endless Data To Create Value

Gigabytes, terabytes, petabytes? (You lost me at gigabyte, by the way.) Financial firms are scrambling to handle terabytes and even petabytes of data. But why?

Many don't know exactly why they're ramping up their storage capabilities, other than because they've been told that using all the data that's available will provide insight that will help them make better business decisions, and ultimately lead to happier customers and a healthier bottom line.

Well, color me skeptical. Yes, diving into data can uncover some surprising things about a business. The numbers don't lie and mining data can help improve business processes in a variety of ways.

[Video: The Future Of Data Management]

For instance, before sabermetrics became popular in baseball, very few teams actually did much with their data. Baseball is one of the most data-intensive sports, and each team had volumes of stats on how players performed with, say, two strikes or with runners in scoring position or in day games versus night games. Every team had access to the same data, but very few did anything with it.

Today, almost every team in baseball uses statistical analysis to some extent to improve performance. Previously, teams didn't know what they could do with the data or how to analyze it. Thus, it provided little value to baseball franchises.

Data Heavy, Insight Light

Capital markets organizations have been data heavy and have always used data to analyze investment opportunities and, especially since the financial crisis, to monitor and uncover risks.

Today, however, firms are looking for new types of data in order to find new insights and to gain an edge on the competition. Many are considering unstructured data and information from a variety of sources -- Web logs, mobile tracking, weather and census data, for example -- to help create an advantage. In order to gather all of this data, terabytes and possibly even petabytes (or dare we say an exabyte -- 1 billion GB) of storage are going to be needed.

To have a petabyte of data is a remarkable feat, and many firms are looking to big data technologies and data analytics engines to mine all that data. But if a firm doesn't know what to do with the data it gathers, that isn't much different from the baseball teams sitting on mountains of data for decades, not knowing how to get any value out it. After all, it only took about 80 years for sabermetrics to take off and change the way the game of baseball is played.

For banks, the trick will not be to amass the largest data store. The goal will be to figure out ways to mine and analyze the sometimes-unrelated data sets to create some sort of value. As the size of the data warehouse grows, firms must invest as heavily in data analytics as they do hard disks.