Big Data Overload is Reshaping the Technology Market

Today we are clearly seeing a desire for businesses to access more data, which coincides with increased availability of data. New kinds of data, and a lot of the same. Data generated by machines, sensors, geological locations of individuals, etc. All of that transfers to a pool that needs to be pumped out, cleaned, and put together in a form digestible to a business and end user. There's a broad range of latencies and velocities of data, and preferably we all want it within an instant.

Traditionally, you have a limited set of data that requires some heavy duty processing, and by limited we don't necessarily mean smaller. Financial organizations have a a huge volume, but data was clearly identified and owned by the organization, explains Yves de Montcheuil, VP of marketing at Talend, a business process integration solutions provider.

"What we're seeing is a requirement to extend this use to more data types, so that's where big data comes into play. Even data within an organization that wasn't used, for example, the data that is generated by sensors on manufacturing equipment… By combining all this internal data, we can access more external data -- social data, sentiment analysis, predictive data. It's publicly available but it's about individuals, and you have this whole thing about open data, sensor data, spending by the government, data that is made more and more openly available due to legal requirements." Increasingly, organizations want to bring this external data into the equation.

But where to put it all? Organizations are increasingly relying on data warehousing companies, they are providing the space in which we dump the data we've aggregated and cleansed. At the Teradata PARTNERS conference in October it was announced that Teradata, the global leader in data warehousing and data analytics, launched their cloud and offering data warehousing as a service (DWaaS), managed by Teradata consulting professionals. Talend is the the only data integration service given the contract in advance, and to date the default option for uploading data into the DWaaS for Teradata customers.

"We've been partners with them forever," adds Montcheuil. "Their goal is for customers who don't want to deploy Teradata on premises to have the ability to farm it out to Teradata and manage the data warehouse for them. This does not alleviate the need to bring into the warehouse all the data needed for analytics, so the customer will have to provide to Teradata access to data sources."

Should firms be concerned about farming out their data? Perhaps, but DWaaS will take time to deploy, and in the end it won't be trust or security issues that hold back adoption. Organizations will go through obsolescent cycles of old systems before replacing them, testing the waters with ad hoc projects, helping to create proof of concepts and ramp up the technologies they need.

Big data is becoming more progressive, finding ways to more mainstream IT. People are no longer experimenting, they are full stream ahead and deriving value. "Big data is becoming the center of IT," explains Montcheuil, "and it's becoming the center on a ubiquitous platform. It's no longer the application at the center, instead data applications will plug into this data system." Becca Lipman is Senior Editor for Wall Street & Technology. She writes in-depth news articles with a focus on big data and compliance in the capital markets. She regularly meets with information technology leaders and innovators and writes about cloud computing, datacenters, ... View Full Bio