Wall Street & Technology is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management

11:04 AM
Jonas Olsson, Graz
Jonas Olsson, Graz

Big Data: No Replacement for the Financial Data Warehouse

News of data warehouses’ demise is premature, argues Jonas Olsson. In fact, there is reason to believe the future for data warehouse is brighter than ever.

Some people are suggesting that with the explosion of “Big Data,” data warehouses are becoming a thing of the past, that they are the “new mainframes.” The perception is understandable, as big data is certainly sucking the oxygen out of the server room. But for anyone taking data seriously, the old maxim “perception is reality” is not acceptable, especially if you are working in the financial industry.

Jonas Olsson, Graz
Jonas Olsson, Graz

So, is there any fact-based validity to the idea that data warehouses are becoming the new legacy system, holding financial services companies back from new insights and greater efficiencies? I argue the notion is absurd, like saying there should be no rail transportation because someone invented the automobile. A closer investigation shows that data warehouses, overshadowed by excitement about the new possibilities of tapping into new kinds of data, have actually lost none of their appeal or real-world power, particularly as compliance, risk management, regulatory reporting and operational efficiency remain hot topics.

To compare data warehouses and big data, it is useful to first establish definitions for each: Big data refers to managing vast quantities – 100 terabytes or more – of unstructured data, generated from machines, social media and the like; data warehouses are designed to handle structured data, pulled from trading, customer management, portfolio accounting and other operational systems. Further, the vast majority of financial firms are not dealing with anywhere close to 100 terabytes of actual data. In fact, few organizations approach the outer limits of data warehouse capabilities – most involve no more than 15 terabytes, and those at that level could likely greatly reduce it with an improved data model.

Beyond size, big data falls short for financial firms because two of the primary drivers for technology investment are compliance and risk management, both of which require limited volumes of structured data from operational systems, and therefore are best served by data warehouses. In addition, because all data in a data warehouse is traceable, compliance departments at any moment can quickly and easily determine where and when information was derived, for an accurate audit trail, something big data is not designed to handle.

[For more on It's More About "Wide Data" Than Big Data]

When one considers these differences in evaluating the unique capabilities of each data management approach, it becomes clear that news of data warehouses’ demise is premature. In fact, there is reason to believe the future for data warehouse is brighter than ever.

We speak with many CIOs and Lead Architects and some are looking into big data technology because they feel their data warehouses are becoming too big. On closer inspection, we find that bloated data warehouses can be easily fixed by applying financial domain-specific knowledge to how the data is organized.

Even without applying domain-specific knowledge, it is my firm belief that that the real data problem for the financial industry is not vast volumes of data; it is the complexity of the data. Sure, there are times when financial firms need to store huge volumes of data – historical tick data for example. But for the vast majority of financial organizations, the real trick is to figure out how to get multiple operational systems to work with each other without losing or duplicating data. Introducing the hot new technology may be appealing, but solving the complexity puzzle effectively is better served by the data warehouse.

The story of big data is still largely one of potential (though incredibly exciting potential, most would agree) and no bank should see it as a replacement for current data warehouse initiatives that have proven their value time and time again.

Instead, financial technology professionals should continually seek out any and all solutions that could make their firms more profitable, productive, secure and compliant. This means not saying “this or that,” but instead “this and that.”

Jonas Olsson is the founder and CEO of Graz, a data warehouse provider for the financial industry.

More Commentary
A Wild Ride Comes to an End
Covering the financial services technology space for the past 15 years has been a thrilling ride with many ups as downs.
The End of an Era: Farewell to an Icon
After more than two decades of writing for Wall Street & Technology, I am leaving the media brand. It's time to reflect on our mutual history and the road ahead.
Beyond Bitcoin: Why Counterparty Has Won Support From Overstock's Chairman
The combined excitement over the currency and the Blockchain has kept the market capitalization above $4 billion for more than a year. This has attracted both imitators and innovators.
Asset Managers Set Sights on Defragmenting Back-Office Data
Defragmenting back-office data and technology will be a top focus for asset managers in 2015.
4 Mobile Security Predictions for 2015
As we look ahead, mobility is the perfect breeding ground for attacks in 2015.
Register for Wall Street & Technology Newsletters
5 Things to Look For Before Accepting Terms & Conditions
5 Things to Look For Before Accepting Terms & Conditions
Is your corporate data at risk? Before uploading sensitive information to cloud services be sure to review these terms.