Surprising market research findings may very well be true
I was recently asked to review a paper for potential acceptance at a conference. The paper's topic was the evaluation of market research reports by users.
The standout finding from this survey of market research buyers was the association between the the degree of critical evaluation and whether the findings were surprising or not. Put simply, if the findings weren't anticipated then the report was more likely to be critically evaluated and checked.
Now this could be prudent application of Twyman's law - "anything surprising or interesting is probably wrong". But Twyman was talking about deviations from well established patterns (i.e. deviations from 'laws'), in market research surprising findings are more likely to be deviations from managerial assumptions, politics or marketing mythology. And particularly findings that no one wanted (like that customers are unhappy - or even that customers ARE happy).
It seems that managers, being human, exhibit substantial confirmation bias. Even scientists have been found to check and re-check surprising findings, but less so ones that support their hypothesis. Similarly academic journal reviewers tend to accept papers with results that support existing beliefs, and reject others.
The scary thing therefore about this paper is that we already know that market research companies are more likely to present findings that conform to client beliefs (eg, new product tests are more likely to say the product could succeed, than that it is likely to fail).
Given the researchers' own confirmation bias (they want to give their clients the results they want to hear), surprising market research reports are more likely to be correct - because they were probably checked and rechecked for error. And non-surprising findings are less likely to be correct. Yet it is the surprising findings that get clients' critical evaluation.
It should be round the other way.
Footnote: I'm (very pleasantly) surprised that so many companies around the world are our sponsors and members of the R&D Initiative. Because we keep coming up with discoveries that surprise – even ourselves.
The standout finding from this survey of market research buyers was the association between the the degree of critical evaluation and whether the findings were surprising or not. Put simply, if the findings weren't anticipated then the report was more likely to be critically evaluated and checked.
Now this could be prudent application of Twyman's law - "anything surprising or interesting is probably wrong". But Twyman was talking about deviations from well established patterns (i.e. deviations from 'laws'), in market research surprising findings are more likely to be deviations from managerial assumptions, politics or marketing mythology. And particularly findings that no one wanted (like that customers are unhappy - or even that customers ARE happy).
It seems that managers, being human, exhibit substantial confirmation bias. Even scientists have been found to check and re-check surprising findings, but less so ones that support their hypothesis. Similarly academic journal reviewers tend to accept papers with results that support existing beliefs, and reject others.
The scary thing therefore about this paper is that we already know that market research companies are more likely to present findings that conform to client beliefs (eg, new product tests are more likely to say the product could succeed, than that it is likely to fail).
Given the researchers' own confirmation bias (they want to give their clients the results they want to hear), surprising market research reports are more likely to be correct - because they were probably checked and rechecked for error. And non-surprising findings are less likely to be correct. Yet it is the surprising findings that get clients' critical evaluation.
It should be round the other way.
Footnote: I'm (very pleasantly) surprised that so many companies around the world are our sponsors and members of the R&D Initiative. Because we keep coming up with discoveries that surprise – even ourselves.
posted by Byron Sharp at 10:13 PM
0 Comments:
Post a Comment
<< Home