Building A Big Diverse Data Platform For Wall Street

The entire capital markets industry is buying into the big data hype, or what I'm calling business intelligence optimization (BIO). I touched on this change in wording in an earlier piece that wasn't focused on ‘big data', but the process to harvest and exploit new knowledge -- regardless of where the data resides.

In reviewing this definition, there is no panacea for raising the analytical quotient in capital markets. There is no big data analytics in a box. There are, however, components or architectural ingredients that are essential in getting value throughout the information supply chain. And, how are those insights operationalized to best impact the bottom line? There are straightforward and relatively simple approaches that offer different ways to build a platform, without the usual complexity associated with data analytics projects.

To build a successful BIO platform, start by understanding:

Data Skills: Talent is scarce in the data science/quant market to leverage and utilize these data and tools.

Standards: Standards, data quality and governance are changing rapidly and not fully formed, so deployment and management must be taken in measured steps.

Culture: Some companies may not be internally built for BIO. A data driven culture that is willing to experiment and understand there are some activities that can generate immediate ROI and others that will take some time is required. It's called continuous improvement and failing fast, and some firms will not be able to access fast insights.

Complexity: It's not easy. Firms have difficulty mapping and stepping through the fundamental factors for any project -- people, process and a clear vision for a "future state" or "next generation" platform -- and BIO is more complex than most projects.

Dedication: Let's not mince words -- it's hard. This will take dedication. The "edge" firms can gain takes hard work, time and patience. Gartner suggests that only 20% of the global 1,000 organization having a strategic focus on "information infrastructures" by 2015.

[For more on how Wall Street organizations are approaching big data challenges, read: Big Data Could Transform Global Financial Markets.]

Is there a path of least resistance? At this time and for the near-term future, a unified data architecture (UDA) is one of the best ways to build a platform that not only enables and operationalizes analytics, big data, BIO or information -- whatever term is used -- but also leverage existing IT assets. The goal is to take current systems, introduce new technology only when needed, and maintain flexibility so the company can rapidly develop applications and analytics that result in better performance. This is a hybrid ecosystem.

The components of a UDA include open source Hadoop, a discovery platform and an integrated data warehouse. Hadoop is an excellent solution for data staging, preprocessing and simple analytics among other applications. The data discovery platform should enable rapid exploration of data using a variety of analytic embedded techniques accessible by business users. And to operationalize findings and insights, a warehouse puts factors, rules or other outputs from Hadoop and the discovery platform into an integrated environment for production to be used across the organization or for a specific business project or unit.

The current combination of components are not just adding value, but also enabling speed to market. A UDA is a sound approach that injects new complimentary components, but does not compete with the fundamental architecture. Rather it enables firms to fill voids in their systems and create the capabilities to rapidly develop analytics and applications. Most importantly, it delivers business intelligence that is not just interesting, but actionable.

All the speed, analytics, insights and connectivity are nice, yet without an actionable outcome to improve the business, the former is just a piecemeal approach that is creating an expensive big data science experiment.