- The majority of research participants are engaged and honest in their responses.
- Although “bad” participants usually make no material difference to survey results because there are so few of them, we must still be concerned about them, especially since their impact is increased in low-incidence projects.
- Technology enables increasingly powerful quality controls
Verification procedures in telephone and face-to-face interviewing (both at the interview and data entry stage) are usually done as a check on the truthfulness of the interviewer, not the participant. It is the interviewer who has the means, motive, and opportunity to cheat. In online surveys, it is the participant who can cheat. Why would they do that?
They might cheat to qualify for a survey and claim a reward. In fact, there is a great deal of evidence to suggest that hard-core fraudsters over-qualify for interviews. Research also suggests that a large majority of such people are physically located in places where the small rewards offered by market research are worth a great deal.
For this reason, many research agencies do not allow participants into surveys with “out-of-area” IP addresses. Dynata’s quality controls have IP verification functionality built in, as do many other proprietary systems.
Dynata prevents the same “person” doing the same survey more than once, either because they are coming from multiple panels being used on a project or exist multiple times within the same panel. Otherwise “good” participants might be tempted to cheat to get into a survey because a high reward is offered, especially if the invitation gives the game away by telling you how to qualify! Research Now SSI usually does not state the survey topic or reward until after the initial screening process, to minimize the risk of fraudulent qualification.
We know, from our own research, that large incentives do not encourage generally better response rates, so we discourage them.
There is much that panel companies can do to prevent frauds from entering questionnaires. Dynata uses the following controls, among others:
- Recruiting from trusted partnerships with loyalty programs whose memberships are verified at source.
- Feedback loop with action taken on any poor-quality respondent reported to Research Now SSI by a client
- Tracking of quality throughout a panelist’s lifetime
- Two-factor claim authentication required for reward redemption
- 24 and 72-hour claim delays
We can also establish the truthfulness of the participant by asking them something they should know repeatedly. If they provide consistent answers on age, gender, etc., we can assume they are telling the truth there, and can have confidence they are telling the truth on other questions.
Dynata also recommends using the questionnaire itself, where possible, to verify that the person is indeed who they claim to be by asking them something that they should know based on who they say they are. This is easier to do in business-to-business settings where knowledge-based questions can be asked. However, we need to use caution when creating in-survey quality control questions to reduce the risk of false positives. For example, it is very easy for a genuine participant to mis-click while moving through a survey. For this reason, Research Now SSI recommends not assuming fraudulent behavior unless we see multiple pieces of evidence of it.
Dynata constantly seeks to add new and better levels of verification for our panels. For example, establishing trust and engagement by analyzing answers to open end questions. Open end answer analysis can be particularly effective in deterring fraudsters because it is difficult to invent complex, unique, relevant answers to an open-ended question.