By: Melanie Courtright,
EVP Global Research Science & Data Strategy
About this series: Digital technologies have dramatically lowered the cost of research, ushering in important new opportunities to gain and apply research insights. Unfortunately, these cost structures can also lead to compromises in quality – which in turn can compromise the accuracy and reliability of research insights, and ultimately the business decisions they support. This series will present eight of the most important characteristics to look for in a quality research partner.
The first key indicator of quality is sourcing breadth and depth, a measure of methodological quality in how a research provider assembles their sample – literally the foundation of quality in the research process. The way a provider sources their sample directly affects two significant issues that in turn affect overall quality.
Quality Issue: Bias. Bias in a sample produces results that are not representative. Imagine a group of dart players trying to hit the bullseye. A sample with high bias would produce a tight grouping of darts, but they would all be off to one side of the dartboard. In survey research, high bias equates to results that are consistent, but consistently non-representative.
Bias is a potentially serious issue for online surveys. Online research is a form of nonprobability sampling, in that some elements of the population have no chance of being selected – in this case, people who aren’t online. As a result, providers must be rigorous in sourcing to ensure that the sample frame is accurately representative, drawing from a wide population reach and recruiting data sources from multiple channels.
Beyond diverse recruiting, providers who are committed to quality incentivize research samples with a variety of rewards geared to different kinds of studies. For example, in consumer research, gratuities with a value calibrated to the complexity of the survey are appropriate; participants in B2B studies are likely to be more motivated by access to the study results.
Quality Issue: Variance. Variance is a measure of representativeness over time, or consistency. To return to the dart players, a high-variance sample will scatter darts in random patterns across the dartboard. In research, high-variance samples deliver results that change unpredictably, which is especially problematic for tracker studies.
To control variance, providers need to ensure consistency in their sample and in the research experience. Achieving the first means establishing a stable, consistent recruiting effort to prevent short-term recruiting imbalances, and nurturing long-term relationships by treating sample with respect, consideration and transparency to support longevity.
To deliver a consistent, high-quality research experience, providers should ensure that survey designs are easy to navigate and appropriate for their purpose; that their technology and connectivity is robust and redundant to give sample reliable accessibility; and that their delivery team has the expertise to assist clients as needed in creating and administering well designed studies.
Evaluating providers. There are relatively reliable and objective ways to judge a provider’s commitment to responsible sourcing that minimizes these two issues.
For bias, ask for measures of representativeness. How does their sample frame compare to established, objective benchmarks in the general population such as home ownership, TV viewing, employment, illness and other important characteristics?
To evaluate variance, find out how their sample compares to the same benchmarks over time.
Sourcing sample with breadth and depth is just one of the dimensions of quality I will cover over the next weeks and months. But it is the essential foundation for research data that can be used to advance business goals with confidence.
This blog is part 1 of an 8-part blog series by Dynata. Check back next month to find out another key quality characteristic to look for in your research partner. To discover why quality data is imperative for reliable insights and confident business decisions, Contact us today!