Data scientists: Modern-day knowledge prospectors, panning for insight.
How do we accelerate tomorrow’s business?
How can tomorrow’s businesses mine data more effectively?
How does relying on data get better results than judgment alone?
Read on for answers.
Back in the day, if you wanted to know how a system worked, you’d have to go knowledge prospecting: Search for clues, pan through the dross, work a promising seam to exhaustion.
Big-data analytics gave birth to a different kind of gold rush. To milk the analogy, it’s the strip-mining of data to generate new insights: You blow open the data and see what falls out.
Google and Amazon dynamite their data extremely successfully. What should you do before you open your big-data mine?
Google’s Flu Trends project is a prime example of data mining. Google has billions of bytes of data about its users; not just their search strings but also location details, demographics, shopping and traveling habits, etc.
The data giant used the power of machine learning to learn to predict flu epidemics. After applying millions of mathematical models to billions of data points, they found a set of less than one hundred variables that would predict where flu would strike next.
[For more on big data in healthcare, read: Scientists Save Healthcare (But They're Not From Med School)]
But here’s the rub: No one, not even the people who built the model, entirely knows how it works.
That’s the thing about big-data analytics: The analysis can expose surprising, counter-intuitive conclusions:
- For a start, the number of variables is far higher than any human analyst could comfortably visualize.
- Also, the model looks nothing like the kind that you might intuitively expect if it had been suggested by a doctor or an epidemiologist.
Rather than modeling sick days, searches for doctors’ offices, or online purchases of ultra-soft tissues, Google’s model includes all kinds of odd-seeming variables. Some, like the prevalence of searches for winter sports results, make some sense—after all, flu season falls in winter.
But other variables aren’t so easily explained; they stay in the model simply because it makes better predictions with than without.
“When you have this many variables, you have to give up an understanding of how the world works,” says Kenneth Cukier, data editor at the Economist. “Humans have limited capacity. In a small-data world we could look for patterns, but in a big-data universe we are unable to spot [them].”
Algorithms that “mine” the data for patterns are already being used in every stage of business from capacity planning to customer acquisition and retention. If mathematical models make useful predictions, does it matter if nobody understands them?
[For more on using data rather than hunches, read: 3 Keys To Monetize Big Data]
It Works! But We Don’t Know Why. Is That A Problem?
But if you’re worried by the idea of a model that you don’t understand, you’re not alone.
Dr. Tiffany Jenkins, sociologist and cultural commentator from the Institute of Ideas raises the specter of the “deification” of data—the belief that data is the be-all and end-all of decision making, when so many decisions have an ethical or moral dimension that data can’t address.
“Big data means nothing without context and interpretation,” she told me. “The central point about data is that only we—human beings, not machines—can make something of it. Only we can draw conclusions, hypotheses, and work out why it matters—or if it matters at all.”
In other words, you shouldn’t remove the human element of interpreting the results of the model.
Human Judgment Is Critical
So much rests on our interpretation. If the analysis of big data helps determine, say, whom to give life-saving treatment to, do organizations have a duty to build “social responsibility” variables into their algorithms?
With 90% of the world’s largest employers already using algorithms to screen job applications, are we facing the day when big data will determine if someone is hired or not? And if so, does this run the risk of creating cookie-cutter workforces, where the creativity born of diverse backgrounds and mindsets is lost?
The Bottom Line
Human judgment must still have the last word when it comes to spotting the new, the unexpected, and the revolutionary. But be prepared to put your faith in a model that you don’t completely understand.
In tomorrow’s business, big data can tell you more about your operations than your people alone.
Disagree? Weigh in with a comment below…
Now Read This (more from NetAppVoice)
• Scientists Save Healthcare (But They’re Not From Med School)
• Steve Wozniak: Apple, Cloud, Flash, Luck, And Open Doors
• How Are Burberry Bags Like Caesar’s Palace?
• Read more from our talented writers