Data Cleansing Products to Save Industry Time and Money

Missing data sequences, incorrect data points, multiple incompatible formats and synchronization are just a few of the common data problems that plague riskmanagement departments. Data is an essential part of the risk management process and the fundamental task of gathering, cleansing and pre-calculating data in order to calculate risk exposures and run scenarios is quite an expensive one. A team of two-to-three people working full-time on risk management data issues at financial services firms can easily run in the half million dollar per year range-not to mention the time and effort taken away from their core risk management functions that suffer as a result of the diversion. This seems to be a high price to pay for a function that has a more efficient, cost-effective solution that is slowly catching on in the risk management arena.

A relatively new and quietly emerging product is hitting the risk management scene and looking to penetrate firms small and large. Risk data products, or external providers of risk data for risk management applications, can save time and money and look to become even more popular over the next year. "This is a very new topic overall and a lot of financial institutions didn't realize that this data was available," says Debbie Williams, research director at Meridien Research and author of a recent brief on the risk data subject. "We did some market penetration numbers and I would expect to see a lot of activity in the next 12 months." She adds that risk data products are likely to be popular with both large institutions that have many different departments and data sources and smaller firms that lack the money and man power to perform their own data gathering and cleansing functions.

What are the major problems with internal risk data today? Williams says that risk data requires a great deal of manual manipulation and cleaning before it is ready for risk management applications. In order to accomplish this, people and resources are needlessly dedicated to the tedious task of getting that data ready for calculations. "In a large financial institution, the kind of data needed for risk management purposes is primarily time series data and pre-calculated things like volatilities and correlations, which are calculated internally by their own risk groups," explains Williams. "So they're pulling data from sources like Reuters or Bloomberg and importing that data into a spreadsheet or something, running calculations on it and then saving that data back out as a data set. This is very time consuming and there are often whole teams of people doing it."

Allan Malz, head of data research and sales at RiskMetrics, which markets a risk data product called DataMetrics, agrees that the task of preparing data is an often expensive and inefficient process that doesn't necessarily need to be done internally. Malz explains that analytics, or basically modeling and math, must be used to transform the market data collected and cleaned into what he calls "fundamental risk factors," which are the time series data used in generating scenarios and calculating risk measurements with specific applications. "With all of these analytics you have to know when to do them and have an appropriately defined fundamental set of risk factors. This is a separate component than just dealing with the data," says Malz.

But even before these analytical functions required to prepare risk data can be performed, the market data must be cleaned, or validated, which can also be a timely process. "Firms have teams that spend their time basically just cleaning up the data that's originally there-- missing values, bad values, all different kinds of problems," notes Williams. "So techniques are necessary for spotting the errors and determining why the errors occurred." Malz adds that validation is "the costliest and most problematic for risk managers," and is a large job in any risk department. Williams emphasizes that this task of cleaning and validating data is the same effort underway daily at hundreds of financial efforts around the world and a vendor product could cut down significantly on the time and resources spent performing those functions. "With economies of scale, if you just did cleaned and validated all the data once rather than at each and share the resource across a bunch of institutions, then the cost becomes cheaper," concludes Williams.

1 of 2