Big Data Influences More Long-Term Storage

The pursuit of big data has led to an increase of companies keeping older files they would have previously discarded, confirms Scott Gillespie, expert on regulations and compliance and senior VP of business analysis at Quadron Data Solutions.

There is a six-year retention requirement of client account data, he explains. Account records, trades, holdings, profile information, goals and objectives and so on fall into this category. There are three reasons these records, which have historically been discarded with regularity, are now starting to stick around.

1. The cost of data storage has come down dramatically.

When technology directors face decisions around deleting versus saving, the lower cost of storage makes to easier to choose the latter. This "just let it be" attitude is further supported by the complexity of what data to actually cut.

Those what-to-delete problems are two fold. First, the connectivity of data means even an old events can be tied to current data, meaning the deletion of an old file could have an adverse impact on the quality of newer information. The other is identifying the proper account data, tagged to the right account.

Gillespie provides an example that demonstrates the challenges of deleting aged records of two customer accounts, one belonging to Smith and the other to Jones:

Smith's account experiences an address change on 1/1/06, and the bank is legally required to maintain the old address for six years. Smith's suitability changes on 1/1/09, and the bank is required to maintain the old suitability information for six years. Smith's account closes on 1/1/14, and the bank must maintain the account records for six years after closing. In all that time Jones's account hasn't changed.

Aging and deleting Smith's aged records without impacting Jones's account records may realistically require cherry picking from the database. Between the cost of data storage coming down, and the risk of explaining to regulators why data was prematurely deleted, firms may be better off just holding on to all that data for a very extended period, Gillespie explains.

2. Deeper analytics require larger data sets.

The big data mega-trend is driven by the idea that having more data will help uncover more information. Analysis can uncover trends and correlations that enable market differentiation and a competitive edge. Theoretically, the more data sets a firm collects, the more information they can derive.

If a firm has a policy that says to delete things seven years and older it raises the question of big data needs. A company may find interesting information relating to client transactions and holdings over the past five years, but what if they want to then look further to the last six to ten, or even fifteen years?

In Gillespie's example, a firm is doing some big data analysis that shows a particular region is doing the most investment advisory business. To learn more they will need to know the particulars of account profiles including the demographic of clients, objectives, and how long they've been customers. This allows for deeper questions like, "Which regions and reps are doing the most Investment Advisory business and what is the demographic profile of their client base?" and "What is the changing mix of securities trades over the past five years?"

Firms are quickly realizing that if they want to track trends and perform analytics beyond the last six years they cannot delete older data. Gillespie says his clients who were deleting data only a few years ago are now holding onto it going forward, hoping to glean more from a longer time frame. "Many are recognizing the need to perform analytics and posing the question of what kind of data we need to keep," he says, "It's a big conversation in the executive offices of broker dealers."

3. Regulatory expectations are rising.

Regulators are performing risk-based exams, which are highly dependent upon ready access to client, trade and position holding data, he explains. From a data storage and usage standpoint regulators used to just have a checklist (e.g. books of records, images of account records), "now they are posting big data analytics questions and expecting firms to hand over the results of their exam."

For example, the SEC can make the following request with the expectation that it will be fulfilled by a query of warehouse data: Deliver all purchases of leveraged ETF equities over the past 24 months by clients over 55 who changed their objective to growth or speculation during the same period.

No matter that the above question will leave some firms without robust warehousing to scramble for answers. The reality is that regulators understand data is cheap to store and they should be able to get any report from a investment firm. Would it then be so unreasonable to assume future questions put forward by regulators will be aided by "outdated" information? If old account records will benefit data queries, would it not be foolish to delete it today?

These possibilities are weighing on the minds of executives, according to Gillespie. "Clients are contenting to add storage at this point," he says, adding that more and more clients are electing to hold onto older data older than even a couple years ago. "They will address it ultimately but not exactly at the minimum requirements." Becca Lipman is Senior Editor for Wall Street & Technology. She writes in-depth news articles with a focus on big data and compliance in the capital markets. She regularly meets with information technology leaders and innovators and writes about cloud computing, datacenters, ... View Full Bio