The challenge isn’t just removing bad respondents. It’s avoiding the removal of good ones.
Data quality is an ever-present issue in the research industry with conversations focused on how to reduce inattention and prevent fraud. As the stakes have become higher and bad actors become harder to spot with simple checks, the tools available have increased in both sophistication and quantity, all claiming to be the “fix” to your fraud problem.
Simple data cleaning methods can lead to overcleaning
In theory, the ideal outcome is to identify bad actors before they enter a survey and prevent them from affecting data quality. Simpler data cleaning methods, such as digital fingerprinting or other blanket approaches, rely on a limited set of signals rather than a complete view of respondent quality. Without a broader context, these methods can struggle to distinguish truly fraudulent activity from the behavior of authentic respondents. Often this can result in overcleaning and removing high-quality respondents in the process.
In an ideal world, the overlap of signals between good and poor-quality data would be minimal, and we could set our “threshold” to catch all fraud with minimal real respondents being impacted.
Innocent until proven otherwise – we should apply that principle to our data quality measures
Consider how the criminal justice system approaches guilt. No one wants dangerous people walking free, but wrongly convicting an innocent person is just as serious a failure, arguably worse. That’s exactly why the burden of proof is important. A single piece of circumstantial evidence isn’t enough. Evidence must be corroborated and considered in context before a verdict is reached.
Fraud detection should work the same way. A single signal (e.g., a shared IP address, an unfamiliar device, a completion time that’s slightly faster than average) is the same as circumstantial evidence. Without context about that respondent’s history, their behavior across surveys, and their actual in-survey performance, removing them from a data set isn’t a cautious choice. It’s a wrongful conviction. And the cost doesn’t disappear just because it’s invisible: it shows up in skewed data, inflated screen-out rates, and studies that are harder to fill. The burden of proof for removal needs to be high, and it needs to come from multiple signals, not a single snapshot in time.
Limited data quality signals create blind spots
The signals associated with high-quality and poor-quality respondents often overlap, making it difficult to draw a clean line between the two. When decisions are based on limited signals, such as basic digital fingerprinting and comparing that information to external databases, the risk of misclassification increases significantly. As illustrated by threshold B below, more than half of legitimate respondents could be excluded to eliminate fraud. In many cases, that means removing highly engaged respondents who would otherwise contribute valuable, accurate data. Compare that to threshold A, where the majority of poor-quality respondents are removed, but with a more reasonable impact on well-behaved respondents.
In reality, the overlap between these two groups isn’t known, and can also change depending on the signal being used to differentiate them. Because these decisions are based on a limited set of signals, they may miss the broader behavioral patterns that indicate respondent quality. As a result, high-quality respondents can be screened out before they ever have the opportunity to demonstrate their value.
Quality is nuanced and cannot be accurately assessed using only a few data points captured at a single moment in time. A point-in-time signal provides limited context and may not reflect a respondent’s historical behavior, engagement patterns, or in-survey performance. Taken together, these broader indicators provide a much more reliable view of respondent quality and should play a role in removal decisions.
Most importantly, relying on narrow signals to make quality decisions creates a significant risk of overcleaning. When overcleaning occurs, researchers may unintentionally remove not only fraudulent respondents, but also legitimate, attentive, and highly engaged participants. This can skew results, introduce bias into the dataset, artificially inflate screen out rates, lower incidence rates, and make studies more difficult to fulfill. In other words, organizations may be sacrificing valuable respondent contributions without realizing it, ultimately reducing sample quality rather than improving it.
The final unintended consequence can be a poor experience for many respondents, resulting in long-term impacts on the respondent ecosystem.
What does this look like in practice?
Consider a study targeting residents of a major metropolitan area. The data cleaning tool flagged and removed a cluster of responses originating from the same IP address range which can be a common signal used to identify potential ballot stuffing or coordinated bot activity. On the surface, the logic seems sound.
What the cleaning tool didn’t know: that IP address range belonged to a large senior living community. The respondents were real people: older adults with consistent panel histories and attentive survey-taking behavior, all sharing a single facility-managed WiFi network. Because they connected through the same router, their responses triggered a pattern the tool had been built to catch. Nearly three dozen responses from a highly engaged demographic were quietly removed from a study designed to reach exactly that audience. Quotas became harder to fill, incidence appeared artificially low, and the insights that remained underrepresented the very people the client most wanted to understand.
This kind of error isn’t rare. Shared IP addresses are everywhere: apartment buildings, universities, corporate offices, hospitals, libraries, and rural communities on satellite internet. A single IP can represent dozens of distinct, entirely legitimate individuals. Treating IP sharing as proof of fraud doesn’t make data cleaner; it makes it less representative of the real world.
The challenge goes deeper than IP addresses. Human behavior is genuinely complex. A parent who completes surveys during school pickup, a night-shift worker who responds at 3am, or a privacy-conscious participant who always uses a VPN are all patterns can look suspicious in isolation.
Against a fuller picture of who that person is across many surveys and interactions, they’re completely ordinary. Blanket cleaning rules flatten that complexity. They don’t distinguish the outlier from the fraudster, and in doing so, they eliminate exactly the kind of real, diverse respondents that make research meaningful.
The goal is to remove the right respondents, or better yet, prevent them from ever participating
We believe in protecting the quality of your study at every possible step in the process from recruitment, through response and data delivery. However, it is important to keep these ideas in mind, especially when using quality tools that do not take respondent history or behaviour into account. If the use of these tools leads to a very high screen out rate, particularly across providers and panels, it’s worth considering if the tradeoff being made is the right one and adjusting the thresholds the tool is using to better balance respondent quality with accurate insights.
It’s important to work with first-party data providers, who have far more information on respondents than a digital fingerprint captured at a single moment in time. Respondent quality is best evaluated through a comprehensive view of behavior across surveys, including historical performance, engagement patterns, and in-survey actions.
Context matters. The more complete the picture, the better organizations can identify genuinely poor-quality respondents while preserving the engaged, high-quality participants that are essential to accurate insights. This is also why sophisticated in-survey quality measures, such as QualityScore, are so valuable. By evaluating actual survey behavior rather than relying solely on external signals, they provide a more precise assessment of respondent quality and help ensure that data cleaning improves outcomes instead of simply reducing sample.
