Big Organizations Have A Love/Hate/Love Affair With Hadoop

Author

Matt Asay

October 3, 2014

CIOs are of two minds on Hadoop. They’re not convinced that it can help them tackle their data but at the same time they’re hiring Hadoop talent in droves. What gives?

The reality is that while enterprises continue to stumble in the Big Data dark, they know that Hadoop—a framework for storing and running local calculations on distributed pools of data—will almost certainly be a core part of the answer, and they’re investing heavily to ensure they don’t get left behind. The primary investment? Hiring Hadoop talent.

Hadoop’s Big Data Gravy Train

Last week I detailed how Big Data fence-sitters are increasing their big data experiments. Companies that had “no plans to invest” in big data technologies in 2012 and 2013 are finally starting to experiment in 2014:

 

Source: Gartner

At the same time, however, enterprises aren’t necessarily convinced that Hadoop—the poster child for Big Data—is the answer to their big data woes. 

According to a recent Barclays survey of 100 CIOs, for example, Hadoop still has a lot to prove. Of the 100 CIOs polled, 72 indicated that it’s “still too early to say whether Hadoop would become an important technology in their organization.”

In a separate survey of data scientists, 76% found Hadoop too difficult to program, among other complaints, causing 35% to give up on Hadoop altogether.

Of course, sometimes people use Hadoop where they shouldn’t. As Facebook’s analytics chief Ken Rudin said at Strata in 2013: “Hadoop isn’t always the best tool for what we need to do…. In reality, big data should include Hadoop and it should include relational [databases], and it should include any other technology that is suited for the task at hand.” 

High Demand For Hadoop Talent

And yet Hadoop is absolutely the right technology for a wide array of Big Data uses, and offers an array of benefits that traditional data technologies fail to deliver. This shows up pretty clearly in the job postings. As reported recently, Hadoop continues to stand out as one of the hottest job skills in the industry, paying a median salary of over $100,000 per year, according to PayScale data.

In terms of absolute jobs, Oracle still rules the roost, though it’s fading fast, according to Indeed job data:

 

Source: Indeed

But if we look at relative job growth, Hadoop has everyone beat, whether traditional systems like Teradata and Oracle or even new data technologies like NoSQL databases:

 

Source: Indeed

By some measures, Hadoop demand is up 34% since 2013. Though a few years old, McKinsey & Co.’s 2011 report on big data holds true today:

By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.

While Hadoop isn’t the only big data technology, it’s a reasonable proxy for big data, generally. And demand is high. 

True, it may be, as Twitter open source chief Chris Aniszczyk highlights, that we’re “still in the early adopter stage and most companies don’t generate enough data to warrant Hadoop scale setups,” leading CIOs to question the value they’d get from Hadoop. Maybe. But it’s also the case that few can afford to wait. Data is increasingly the primary currency for competition, and Hadoop is at the heart of it.

Lead image courtesy of Shutterstock

Great ! Thanks for your subscription !

You will soon receive the first Content Loop Newsletter