Big Data—the Insight Engine of the Future?
It seems that you cannot open a business-focused magazine these days without bumping into the claim that big data will forever change the way we do business, including the way we understand customers. But like many of the things we have heard over the last couple of decades about the impact of digitizing everything under the sun, there is both more and less to the claims being made for big data.
“Big data” means different things to different people but generally refers to large datasets with individual-specific information generated through transaction records and online activity such as social media and web-site click-through behavior. Other data sources include machine or sensor data and emails and other documents.
At KJT Group we use a destination metaphor to characterize the specific type of insight we are seeking (as in, “if you don’t know where you’re going, you won’t know when you get there”). Big data has been particularly useful with respect to a couple of these insight destinations; discovery and prediction (our other destinations are “explanation” and “transformation”). Discovery often involves finding the signal in the noise and working with a lot of data makes it easier to find subtle patterns that might not be detectable otherwise. For example, CVS found by analyzing reams of store-level data that customers in urban areas shop differently than those in the suburbs. The urban shoppers bought a much wider array of products, while the suburban shoppers bought mostly health and beauty products and filled their prescriptions. The resulting insight was that urban shoppers were treating their neighborhood CVS as more of a full-service drug-grocery store while suburban shoppers regarded CVS primarily as a place to fill prescriptions and buy typical “drugstore” items.
The other success for big data has been “predictive analytics.” Prediction requires that we establish reliable associations or correlations between different observations—in effect quantifying the strength of the patterns in the data. Harrah’s casinos, as just one example, have models that predict, based on a customer’s gaming losses, when an associate should intervene with drink coupons or show tickets, encouraging the customer to take a little break from gambling.
Big data has limitations, however, and presents some challenges to corporation that need to be overcome if big data is to become a true “insight engine.” Writing in The Wall Street Journal, John Jordan, a professor at the Smeal College of Business at Penn State highlighted some challenges that corporations face in implementing a big data. The first challenge is a shortage of talent with the skills to capitalize on big data. This extends to managers’ ability to grasp sophisticated statistical methods. Jordan also suggests that the sheer volume of numbers can easily overwhelm our mental capacity to process information on this scale and that data visualization techniques have not yet caught up with that need.
In the end, big data is just numbers, and numbers alone do not constitute insight. The best way to extract insight from any dataset is to start with one or more hypotheses that can be tested. Hypotheses help us separate signal from noise, and that is just as important with big data as it is with relatively small survey samples.
— David Bakken, PhD, Chief Insight Officer