- “Do It Yourself” (DIY) research tools can provide fast results at relatively low cost compared to full service research projects. But are they the best choice for your research needs?
- DIY is a valuable tool for short, simple surveys, if the underlying sample is sound.
- Examples of good uses for DIY surveys are testing ad creatives, gauging sentiment on current news events by PR agencies, or getting a quick read on reactions to a new product or idea.
Market research, at its heart, can be neatly summed up as “Asking the right people the right questions and understanding their answers.”
DIY is a good solution for short, fast data needs. Behind this apparent simplicity lies a multitude of skills, crafts and creativity that the researcher has taken years to master. In the light of a movement towards DIY research we need to examine how well these products might perform on these three points.
1. The Right People: Sampling
The past few years have seen a movement towards a “wisdom of the crowds” mentality and away from pure random samples or even well-constructed stratified quota samples. Importantly, the wisdom of the crowds does not state that any large enough sample of people will be just as good. The “inventor” of the Wisdom of Crowds theory, James Surowiecki differentiates between “wise” and “unwise” crowds.
Wise crowds exhibit “Diversity of Opinion” (i.e. drawn from all strata of society and having independent knowledge) and “Independence” (i.e. opinions not determined by the opinions of those around them). The reason this works is not too dissimilar to the reason random samples work! Any method used to access respondents must be examined carefully for all sources of potential bias. Of all the skills in the researcher’s toolbox this is the one that should have been prized above all others. Instead, researchers have allowed themselves to be persuaded by some clients to accept poorly designed questions or have neglected the intense training and practice required to craft a question which is 100% “fit for purpose.”
2. The Right Questions
When real products, real brands and real business decisions—possibly affecting real jobs—are at stake it is best to review the questionnaire with a research expert.
For example, here is a seemingly-simple question:
Have you bought a new car in the past 12 months?
-Yes
-No, I don’t drive
1.What is meant by ‘new’? Does this mean a car new to me or a car that was new when I bought it?”
2. “I’m the fleet buyer for a large corporation. Do you mean me personally or me at work?”
3. “I bought a new car 13 months ago so I should vote ‘no’; but the ‘no’ code also specifies that it is for people who don’t drive. Oh, what the heck I’ll check ‘in the last 12 months’.”
And so it might go on; even the seemingly simplest of questions is a minefield in the hands of the inexperienced.
Many research users are, of course, capable of framing a question correctly however the process is sometimes extensive and often time consuming. With questionnaire design it really is a case of “garbage in-garbage out.”
3. Understanding the answers
Here DIY researchers are really on their own. With potentially poor respondent selection and imperfect questioning how is the data to be interpreted? Relying on gut instinct is likely to produce errors in our own cultural setting let alone when dealing with data from a foreign country. Good researchers will draw on all their considerable expertise in analyzing data, using their experience of other brands and categories to put findings into context. Without context statistics are there to be used and abused.
When the research questions are complex and the research results will drive critical decisions, there may be a high cost to pay for bypassing the research professional and going it alone.
Market research is all about reducing risk. When used appropriately, DIY tools are an excellent choice for delivering simple data fast. When used for the wrong reasons and in the wrong situations, DIY research can add to risk.
Dynata’s Insight Platform offers flexible service and support models. From 44 offices worldwide, 3,000 Dynata experts are here to support your data to insights journey. To learn more about which research model works best for your business, contact us.