When Big Data Projects Go Right


Howard Baldwin, Contributor

February 3, 2015

I wrote a couple of weeks ago about big data projects going wrong, which is always a popular topic. Anytime you can put adversity in a headline, it’s bound to increase clicks.

In the interest of equal time, and at the risk of losing those curious clicks, I want to now look at the idea of big data projects going right. Reading about adversity may be fun, but when it’s your own big data project, figuring out how to make it go right is even more important. Ready?

In Computer Business Review, James Nunns looked at five big data scenarios in five different industries, all of which were real eye-openers in terms of big data success. New York City is compiling data from multiple city agencies to determine a risk profile for buildings, which helps it prioritize inspections. Gatwick Airport is comparing data about ingress and egress of passengers in terminals with staffing levels, and using the resulting data to increase the number of people going through security per hour, but also the number of takeoffs it schedules per hour. Walmart is optimizing inventory based on seasonal and regional preferences (presumably with more accuracy than it could have done previously).

Analytics expert Loraine Lawson wrote last month in her IT Business Edge column about ways to get a big ROI for big data. Who wouldn’t want that? It basically means applying the idea of “BI for the masses” (something the industry has talked about for a long time) to “big data for the masses.” She quotes a piece by Wikibon CTO David Floyer about coupling big data insights with in-memory technologies to drive “real-time input through the front windshield” rather than the rear-view mirror. Bottom line: think about how to make big data part of operations, rather than just analysis.

Sometimes the information you really want will be beyond the scope of your operations, as Mary Branscombe notes on ZDNet when she writes about how vast big data sources really are. She makes a point that I really love: “Big data isn’t big because you have lots of it. It’s big because it covers lots of areas in which you can find insights that your regular data set – however large – doesn’t cover.”

Branscombe offers some great stories of companies that combined their data with others’ data in order to get an even grander insight than they ever could before. Case in point: the credit card company that realized it had a record of what consumers were paying at the gas pump, and took it to Esso with the offer to reveal geographically what consumers were paying and thus how they could offer the lowest prices in any particular region.

I also liked what Forbes contributor H.O. Maycotte posited a couple of weeks ago: that the big data challenge isn’t the needle in the haystack – it’s the haystack. He insightfully writes, “The problem with that is that some of the most interesting insights go unnoticed, because you don’t have the ability to look at your customer data across silos (or haystacks, if you will). And being able to do that could lead you to questions you would otherwise have never thought to ask.” I love the idea of serendipitous answers, but to get them, you need to think about integrating data on a timely basis with strong decision support tools. Basically the same thing you needed to do ten years ago, but now with more data sources.

One last suggestion: if you’re interested in what’s coming in big data – that is, what you’re likely to use it for next – check out O’Reilly’s latest edition of Big Data Now, its annual wrap-up of developments in the field, now its fourth year (available for download). Its seven themes, from cognitive augmentation and cheap sensors to open-source tools and more, will really get your mind racing about the possibilities of big data.

This article was written by Howard Baldwin from Forbes and was legally licensed through the NewsCred publisher network.

Great ! Thanks for your subscription !

You will soon receive the first Content Loop Newsletter