Big Data: Rising In Importance But Still Challenging, New Surveys Say


Gil Press, Contributor

January 12, 2016

Source: NewVantage Partners

Two recent surveys shed new light on the state of big data, showing growing adoption but persistent data management challenges.

Management consulting firm NewVantage Partners (NPV) will release next week the results of its 4th annual Big Data Executive Survey (see my coverage of the previous rounds here and here). Survey respondents were Fortune 1000 senior business and technology executives. IT industry association CompTIA has released last month its Big Data Insights and Opportunities report, based on an online survey of 402 business and IT professionals.

Here are 6 observations, based on these two reports, about what is happening with big data in practice:

Big data adoption is rising but it’s still not in production everywhere

NVP found that 62.5% of firms have at least one instance of big data in production, up from 31.4% in 2013 and 48.2% in 2014. Only 5.4% of firms now report that they have no big data initiatives planned or underway.  51% of the companies participating in the CompTIA survey said they have some form of big data initiative in place, up from 42% in 2013.

The reputation of big data is getting stronger

69.6% of firms view big data as very important or critical to their business success, up from 54.5% in 2014 (NVP).  72% of companies that have launched some form of big data initiative say that their results have exceeded expectations (CompTIA).

Big data drivers: Insights wanted, speed required

Developing greater insights about the business and customers (37%) and speed (faster time-to-answer, faster time-to-decision, and faster time-to-market, at 29.7%), were cited as the biggest drivers of big data investments (NPV). 63% rely on data for day-to-day operations, 60% use data to better understand customers, and 59% rely on data to measure business objectives (CompTIA).

The challenge of the variety of data sources trumps volume and velocity

Firms continue to report that variety is the primary technical driver behind big data investments (40.0%), with volume (14.5%) and velocity (3.6%) lagging well behind (NVP). 45% of companies say that a high degree of their data is fragmented  and another 42% say their data fragmentation is moderate (CompTIA).

Organizational challenges: finding the right talent and leadership

42% of companies don’t have real-time analytics skills and 41% are lacking relational databases skills. As a result of poor data management and usage, 29% suffer from inefficient decision-making and inability to reach new customer segments, while 38% suffer from wasted time. Only 31% of companies are exactly where they want to be in managing and using data (CompTIA).  54% of firms have appointed a Chief Data Officer (CDO), up from 12% in 2012. And 20% report that the Chief Data Officer is the executive with primary ownership responsibility for big data initiatives (NVP).

For big data initiatives to succeed, the business and technology sides of the organization must cooperate

33.9% of firms identified business and technology cooperation as the most critical factor in business adoption, up from 23.4% in 2013, and leading all other factors by a wide margin. Strong business sponsorship was cited as the most critical factor by 23.2% of firms. By contrast, technology leadership (5.4%) and technology selection (0.0%) drew negligible responses (NPV).

This article was written by Gil Press from Forbes and was legally licensed through the NewsCred publisher network.

There is 1 comment

  • Azana Baksh - 01/13/2016 21:01
    Very informative article on Big Data. With the explosion of big data, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it’s computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. The open source HPCC Systems big data processing platform can help companies with these challenges by deriving insights from massive data sets quickly and simply. Designed by data scientists, it is a complete integrated solution from data ingestion and data processing to data delivery. Their built-in Machine Learning Library and Matrix processing algorithms can assist with business intelligence and predictive analytics. More at

Great ! Thanks for your subscription !

You will soon receive the first Content Loop Newsletter