According to a new Gartner report around 64% of firms have either deployed or launched a Big Data initiative in 2013. Industry watchers anticipate this number will only increase in 2014 and beyond. Financial services will be at the forefront of this initial look at Big Data taking the first baby steps in a multiyear voyage.
Few industries are as data centric or as awash in data as finance. Every interaction a client has with a banking institution can potentially produce actionable data that has potential business value associated with it. Capital Markets has always had multiple sources and silos of data in all areas, the front office, back office and the mid office. From a business perspective as well as a technology impact perspective, the biggest technical trend facing executives in financial services is undoubtedly Big Data. The former is perhaps the most important factor determining the pace of adoption.
Lines of business spanning areas like consumer banking, capital markets, wealth management are now beginning to ask questions of themselves as to how they need to not just on board all this data but also how to then draw actionable insights from it within a reasonable SLA. That puts a lot of pressure on IT to deliver on this vision as it moves from bestseller to the datacenter.
The Unrelenting Pace of Technology
We are witnessing the fastest ever rate of innovation in the data space primarily as a result of explosive developments in areas like Hadoop/ MapReduce (especially with Gen 2 & YARN), the Cloud and the NoSQL ecosystems. Couple these developments with ever favorable economics on the hardware end of the spectrum which make possible very large deployments of commodity hardware - be they pools of in-memory data grids (RAM) or scale out NAS based storage, all typically running on cheaper hardware (potentially using ever cheaper Flash/SSD technologies) deploying Open Source solutions. Software defined storage products that let IT abstract and pool storage across on-premise and cloud environments are now starting to be available at a very reasonable price point as well. All these components will provide a foundation to enable the highly scalable and self healing architectures that Big Data implementations will require.
Converging trends – Social, Cloud, Mobility and Analytics (SMAC) – play a significant role in driving adoption
We would like to clarify at the outset that there is no single and universal definition of Big Data (from a size perspective). It varies from an organization, line of business or even application perspective. Industry analysts widely describe the 3 V's (Volume, velocity and variety) as the trifecta from a definition perspective. Let's add a fourth to it, Veracity – which pertains to the signal to noise ratio and the concomitant problem of unclean data.
Social, Cloud, Mobility and Analytics are converging trends and one could contend that these three are all part of the Big Data lifecycle.
Social Media enables direct communication between financial services and their retail clients. These interactions can provide a valuable way to garner feedback around specific product lines as well as a way of landing a younger clientele.
Cloud plays a role in the life cycle of development, deployment and optimization of Big Data applications. Innovation at the largest banks is often shackled by the absence of a responsive and agile infrastructure. It can take days to procure servers to host bursts of workloads that may not be feasible for existing IT departments to rapidly turn around. This problem is by no means unique to Big Data but what makes it even more relevant here is that processing needs in this specialized area are typically larger than your average IT application. Cloud Management Platforms (CMP) are beginning to provide orchestration capabilities by means of workload portability around public and private clouds. Developments around OpenStack and around DevOps are providing complementary capabilities in this evolving space.
At that point, options like Amazon EMR (Elastic Map Reduce) and other hosted Big Data solutions become very attractive not just from -
1) a time to market perspective (an expendable pool of computing capacity as well as the choice of pre-canned & pre-configured Big Data stacks are all available for one to deploy)
2) from a cost perspective (you only pay for what you use)
Mobile technologies results in the production of more data and that too at a very high velocity. This velocity is both ingress velocity ("the speed at which these feeds can be sent into your architecture") as well as egress velocity ("the speed at which actionable results need to be gleaned from this data"). In the era of BYOD, data is king.
Finally, Analytics is the first killer app for Big Data. Be it the low hanging fruit of reporting & dashboards to forecasting and predictive modeling and even Data Science. One of the biggest trends for 2014, is the enhancement of analytics capabilities to incorporate real streams of data at a humongous scale. Existing applications can now incorporate such functionality to derive real time meaning from this data.
Financial Services Use Cases
Big Data use-cases can be classified into two broad areas from a business value perspective. The first in how they help you build a competitive advantage in the Red Ocean part of your business and the second in now they help you develop blue oceans. The CIO's primary concern should be business value and competitiveness and not just keeping with the trends for the sake of it. Over the coming weeks we will explore four of these distinct use cases where firms are beginning to use Big Data related technologies. There are more but we believe these will be foundational as they touch areas where the financial industry is experiencing fundamental business shifts.
These include Risk Management (given the implications of legislation Basel-III with its focus on market liquidity risk, stress testing and capital adequacy), fraud detection and management (in areas spanning credit cards and payment networks), intelligent customer upsell in retail banking and wealth management.
The fourth bucket is the ubiquitous Investment management space where newer (and increasingly unstructured) sources of data are being pulled in and combined with existing sources as part of investment decisions without negatively impacting latency around the applications themselves. As Chief Architect of Red Hat's Financial Services Vertical, Vamsi Chemitiganti is responsible for driving Red Hat's technology vision from a client standpoint. The breadth of these areas range from Platform, Middleware, Storage to Big Data and Cloud (IaaS and PaaS). The ... View Full Bio