From Market Researcher to “Insight Scientist”
In an article in the October, 2012 issue of Harvard Business Review, Thomas H. Davenport and D.J. Patil proclaim that “data scientist” is “the sexiest job of the 21st century.” There’s no doubt that data scientists are in demand as the digital trails we generate grow exponentially (in just the last few months I’ve added Instagram, Snapchat and a second Twitter account). “Big data” places a high premium on the ability to write code for algorithms that will clean, transform, and beat these new (and often unstructured) data streams into submission and they yield up their secrets.
Market researchers are climbing aboard the big data bandwagon (maybe out of fear of being left behind by the data revolution) and those of us with quantitative analysis skills are likewise repositioning ourselves as data scientists. However, I think we will be better-served by evolving into insight scientists rather than data scientists. While data science is a necessary condition for extracting insight from large datasets, insight science goes beyond simple interpretation of the facts to encompass understanding of the insight process which in turn enhances our ability to generate market insights.
As part of a panel discussion at ESOMAR Congress 2011 in Amsterdam, Jeff Hunter (at the time Consumer Insights Director at General Mills) remarked that almost all of the deep market insights he could recall came from qualitative research. Three things that may give qualitative research an edge can help us better understand insight creation. First, we often turn to qualitative research when the ratio of assumptions (things that might be true) to facts (things that we are pretty sure are true) is high, which increases the likelihood of learning something new. Second, we tend to impose fewer boundaries or constraints at the outset. For example, rather than insist on a random representative sample (to which we can apply standard inferential statistics), we are likely to cast a wide net that includes more individuals who might be at the extremes. This can increase the likelihood of a serendipitous finding. Finally, we often employ qualitative research when we are faced with what Frank Yates, professor of marketing and psychology at the University of Michigan, calls a “construction” decision, where the management team is trying to create or discover the ideal alternative.
In an article that appeared in Research World (“The Flavour of Insights,” Nov-Dec 2011) I suggested that there are at least four distinct types of insight and that we can boost our chances of uncovering meaningful, actionable insight by making sure our insight need is aligned with the type of insight our process will yield. Discovery is the first and possibly most basic insight type. Discovery insights reveal things that were previously unknown. Prediction, the second type of insight, arises from observing associations between different things. These associations allow us to make statements about the likelihood of a particular event given the occurrence of another event. Data science is well-suited for discovery and prediction.
Explanation, the third type of insight, depends on both discovery and prediction but is qualitatively different in that we must develop and then test hypotheses. This is the first insight type that requires us to go beyond the facts and to ask “Why?” in addition to “What?” The last type of insight is transformation. Transformational insights seldom arise from a single observation (or dataset). Most “a-ha” moments emerge from connections we make between seemingly disparate bits of knowledge.
What traits and skills will insight scientists need beyond curiosity and core market research skills? I think there is a parallel between discovery and prediction insights and what we sometimes think of as “left-brain thinking” as Daniel Pink describes it in his book A Whole New Mind: Why Right-brainers Will Rule the Future—thinking that is good at sequential processing, at handling text and numbers and that focuses on details. Data science requires this kind of thinking. Insight scientists, on the other hand, need to apply both left and right-brain modes of thinking. The right-brain way of thinking is good at processing information from different sources simultaneously; this mode of thinking specializes in context and sees the big picture (here’s a quick, fun “test” to determine if you think left or right– http://en.sommer-sommer.com/braintest/) .
Of course, insight scientists need to be good at generating hypotheses and finding or creating data to test those hypotheses. It’s worth repeating an observation made by MIT Professor John D. C. Little about a key difference in the way people in business and academia look for causes. Managers tend to look at two situations that produced different outcomes to find key differences in those two situations. Academic researchers tend to look across many different situations with varying outcomes to find underlying patterns that reveal general principles (“theories”). Insight scientists will need to be more like the academic researchers.
One more requirement: insight scientists need to become experts at data visualization. As consultant and author Dan Roam points out, we are inherently visual creatures, with well-developed visual information pathways. As a rule, market researchers know how to create charts but do not know how to visually organize and present data in a way that takes advantage of our visual information processing system. That has to change if we want to become true insight scientists.
– David Bakken, PhD, Chief Insight Officer