Countering common critics on survey data with market research skills
Don't be mislead by survey data estimating the likelihood of adoption and the relative importance of a choice attribute.
With some justification, not everyone is a fan of survey data. Periodically, a client or prospective client recites one of two quotes.
The first quote is from the founder of Apple, Steve Jobs, who said “People don’t know what they want until you show it to them. That’s why I never rely on market research. Our task is to read things that are not yet on the page.”
The second quote is from David Ogilvy.:“The trouble with market research is that people don’t think what they feel, they don’t say what they think, and they don’t do what they say.”
Those who don’t want to do data collection use the offensive play, which is the quote from Steve Jobs. The play goes like this: 'there is no sense undertaking market research because after all, people don’t know what they want'. On the other hand, support for the Ogilvy position is usually invoked once the unpalatable research findings are tabled. The defensive play switches to 'that’s all very well and good, but as we know, respondents don’t do what they say'.
“People don’t know what they want”
Rebuttal #1
Steve Jobs’ perspective is easiest to reconcile. Jobs was talking about new to the world products such as an MP3 player in 2001. He was right in that, if your product innovation is ostensibly ‘new to the world’ and the market has limited ability to conceive its application, then there is not a lot of value to be derived from asking respondents about adoption. Henry Ford was from the same school when he said, “If I had asked people what they wanted, they would have said faster horses.” However, using survey data is useful for illuminating attitudes, market sizing based on core needs, and understanding current alternatives. If Steve Jobs was talking about an invention that was a phone that incorporated an MP3 player, a camera, and GPS then perhaps the market could comprehend such a value-added bundle and more readily estimate adoption. And today, if Apple wanted to understand why users were switching to Samsung, then survey data is surely the best source of data.
Rebuttal #2
The other counter-argument is that with respect, you are not Steve Jobs! Every now and then, a visionary entrepreneur comes along, whose imagination and resources are unconstrained, and proposes something as frame-breaking as putting “a thousand songs in your pocket.”
For the rest of us – less than visionaries – research is a critical component to optimizing our efforts and investment. However, the rationale for removing insight from the process is different depending on what phase of the research project we are in.
Market research and the 5 stages of grief
I should tell you, the context in which I usually hear these comments change depending on where we are in the project. If we are at the end of the project, then the comments often reflect the seminal work by Elisabeth Kübler-Ross. Her five stages of grief model ranges from denial and anger to acceptance and action. There are a handful of alerts researchers receive when the client has slipped into denial and anger. Some of them include questions like “Who commissioned this research? Where did you get the sample from?” And that old chestnut, “You do realize people don’t always do what they say they will do.” We would agree wholeheartedly with the last statement and therefore, apply techniques that sidestep that issue.
When is survey data criticism justifiable?
As in all fields of life, there is a degree of variability in the quality of market researchers. And as is the case in most (if not all) services, there is an asymmetry of information. Meaning the seller is better informed than the buyer, and occasionally the buyer struggles to recognize poor quality researchers from good quality researchers. Here are two basic situations where criticism of findings arising from survey data could be justified.
Situation
|
Discussion
|
|---|
| Estimating the likelihood of adoption. |
The wrong answer is a dichotomous yes or no response.
The better answer is to apply a Juster Scale. In 1966, Professor F. Thomas Juster argued that verbal intentions were disguised probability statements, therefore, why not directly capture the probabilities themselves as expressed by the respondents.
Using an eleven-point scale ranging from 0-10, for likelihood to adopt, Forethought only counts those respondents scoring eight or above as a yes. According to Professor Juster, an eight is ‘very probable’ on the likelihood scale. We have found this to be a remarkably accurate indication of actual in-market adoption.
|
| Estimating the relative importance of an attribute in choice. |
The classic folly in understanding the relative importance of attributes, such as price, is to ask the stated importance of price. Respondents do two things. Firstly, they overstate the importance of price, which then leads them to also overstate their price sensitivity.
The better approach is to assemble the hypothesized list of reasons for adoption (explanatory variables), including price, and to then ask the respondent to rate each explanatory variable on a 0-10 importance scale. With the right dependent variable, such as value for money, multivariate analytics is then applied to infer the importance. We have found this to be a remarkably accurate indication of the relative importance of an attribute in choice.
|
A third scenario, usually limited to qualitative research for high involvement goods and services (that is, high perceived social risk), is where a social desirability bias creates distortions in stated choice. Similarly, the resulting mismatch between stated and actual can be averted by using inferential analysis to infer importance.
I am not sure about other research companies but, about one in five Forethought colleagues is a marketing scientist. These folks are responsible for the design and analysis of survey data. About 60% of our colleagues are consultants arranged into category verticals (banking, telecommunications, healthcare, and so on) to maximize domain knowledge.
My job is working with survey data; I love it! Pretty much the whole purpose of my professional life has been to assist businesses to predict and affect behavioral change as it relates to marketing. The primary vehicle for that prediction has been and continues to be, survey data. Forethought operates in just two areas of behavioral change: gaining and retaining market share for services businesses. With over 4,000 projects completed, I cannot recall a single instance where our predictive models have been incorrect. Without a doubt, market research insights is a game of skill.
Author: Ken Roberts, CEO of Forethought
Source: GreenBook
