The Big Data Challenge Isnt the Needle in the Haystack. Its the Haystack.

Author

H.O. Maycotte, Contributor

January 21, 2015

A lot of the buzz around big data, analytics and insights has been focused on finding the proverbial needle in the haystack. You figure out what you want to know, and you analyze the data available to you to find the answer.

There’s only one problem. How do you know you’re asking the right question?

I contend that the real challenge isn’t finding the needle in the haystack. It’s finding and mastering the haystack itself. And I also believe that the growing ability to unify and cross-correlate data across silos is what’s going to make that mastery possible.

Here’s what I mean. A lot of the marketing and analytics software solutions available today do a great job of answering questions you have in their particular silo. For example, you can set up a BI dashboard tied to your POS software to tell you how many customers bought a particular item in a certain period of time. Or you can configure your email system to send reports on who opened a marketing email and which links they clicked on. In other words, you have a haystack of customer data and you use analytics to sort through it to find the needle — or the answer to your question.

The problem with that is that some of the most interesting insights go unnoticed, because you don’t have the ability to look at your customer data across silos (or haystacks, if you will). And being able to do that could lead you to questions you would otherwise have never thought to ask.

What if a look across all the haystacks could tell you, for example, not just that the baby-boomer females you’ve been targeting are clicking on your ads and buying your products more often than any other group — but also that they’re not spending nearly as much as the older male audience that never clicks through? That’s the kind of answer a more integrated view of data can provide, i.e., an answer to a question you probably didn’t know you had.

As marketers work with IT to build out a data-driven infrastructure, it’s critical to shape the infrastructure in a way that prioritizes discovering questions over finding answers. And to do that, you need three things:

1. Integrated data. Breaking down marketing silos is the first step in getting a 360-degree view of customers. But think, too, about the possibilities of ultimately going past that to include breaking down other silos — across marketing and IT, or across marketing and other areas like customer service or the call center. Making customer data from every marketing silo available for integrated analysis is the right place to start. Once we agree on that, let’s extend the vision further to make customer data from every organizational silo available. This is what it takes to achieve a truly comprehensive view of the customer.

2. Timely results. Discovery in the era of big data and advanced analytics is an iterative process. As more data comes into play and changes the landscape of discovery, any answer that has been hard-coded to data will quickly become obsolete — and just lead to new questions. In this environment, waiting weeks for analytics teams to come back with answers renders those answers useless. To fully exploit marketing opportunities, companies need data infrastructures that allow for real-time data explorations. Only then will it be possible to fully focus on what you really need to know, in the moment — instead of what you thought you needed to know back when you formulated your question.

3. Direct decision support. When people are waiting weeks for analytics teams to weigh in, the waiting is really only one part of the problem. The other part is having to rely on analysts — and BI experts, scientists and even developers — at all. I believe data-driven infrastructures need to make it easy for non-technical marketers and others in the workforce to independently get the data support they need for decision-making. A recent report indicated that marketing teams are looking for more actionable insights today, but rarely finding them — and I think this could be important to changing that.

As we look at bringing data together in the quest for better answers, let’s start by thinking about how we can design our data infrastructures to help us ask better questions. Let’s stop thinking about the needle in the haystack, and start thinking about the haystack.

This article was written by H.O. Maycotte from Forbes and was legally licensed through the NewsCred publisher network.

Great ! Thanks for your subscription !

You will soon receive the first Content Loop Newsletter