Author’s Note

The seeds of this book were sown over fifteen years ago. I was working for NPD and was talking about some of our panel challenges with Steve Coffey, my former boss, mentor, and friend, when he remarked—rather offhandedly in my recollection—that “it’s all just one big global panel.” That comment stayed with me.

I first wrote about panels being examples of what economists call a tragedy of the commons in an article entitled Big Bang Disruption (Research Live, Feb 2014). There were dark clouds on the horizon even then, and that was before the wave of programmatic execution that now dominates online research.

I am grateful to those who provided feedback on early drafts of this e-book. Their criticisms, which I expect to hear from a broader audience, were twofold. One is that the book focuses too much on the problems and not enough on the solutions. It is an expectation in most business contexts that you don’t talk about problems without talking about solutions. I have included practical advice that individual buyers and suppliers can take to mitigate the worst of the problems. Nevertheless, for reasons I make clear in the book, I do not see any durable solutions on the horizon and am pessimistic there will be.

The second is that the rhetoric I use is excessive, and the title of the book is distasteful. I’m playing this with a “straight bat,” as the Brits might say. If the arguments are stark, it is because the writing is direct and the observations are real. I didn’t make up the term enshittification: Cory Doctorow did, to describe what happens to digital ecosystems when firms aim to maximize profits. It is coarse, but it is a perfect analogy. Guilty as charged for the poo emoji page numbers.

Importantly, though, everyone who has read the book (thus far) has agreed with my observations.

Introduction

Before I make my case, let me get my bona fides out of the way.

Education: I hold a Ph.D. with distinction in political science, with significant training in sampling theory, quantitative methods, and economic theory. For over 25 years, I’ve worked as a sampling expert for world-renowned research firms alongside commercial, academic, and government statisticians.
Sample Buying: I was one of the industry’s top buyers of sample as smartphones became pervasive in the West.
Sample Selling: I was later one of the largest sellers of programmatic sample during the era of massive API networking.
P&L and Operational Management: I’ve led global teams responsible for sample buying, sample selling, study design, and project management for six companies across five continents. I designed, programmed, and ran my first online survey in 1998.

I share this background only to establish that I am qualified to comment on the state of programmatic sampling.

And that state is dreadful.

The consequences of the industry’s underlying issues are severe. Clients are regularly receiving bad data, which leads to flawed conclusions and, ultimately, poor decisions. What is less well understood—despite nearly two decades of decline—is not just why this is happening, but why it’s still happening.

It is not the fraud that’s the problem. Don’t get me wrong: fraud is a problem. In fact, everybody knows that everybody knows it’s a problem. It is a common knowledge narrative across the industry.

But it’s not THE problem.

The real problem is that the market for programmatic sample is characterized by multiple market failures.

These failures have led to an ecosystem materially corrupted by inattentive and unrepresentative (and fraudulent) respondents who are appearing in ever larger proportions. This is happening because all respondents—including the rapidly diminishing pool of honest and conscientious ones—are regularly subjected to a punishing user experience.

This observation isn’t new either. Along with many others, I have written about it extensively for over a decade. Why, then, haven’t we fixed the problem?

Market failures, by their nature, resist solutions, most often because they require collective action, an “enforcer” (like a government), and the willingness to endure short-term pain. Demonstrably, none of these is present in the industry.
Clients either don’t know how bad things are (which is an implication of the market failure) or they don’t care, for reasons I will make clear below. The net effect is the elimination of corrective feedback to the market.
This dynamic has given rise to what noted investment strategist and narrative expert Ben Hunt would call a Common Knowledge problem. Not only does everyone on the supply side of the market know that these problems exist, everyone knows that everyone else knows it as well. But because the solutions are out of reach, the industry can only substitute an impuissant narrative for action—a perverse illusion of vigilance. We act as if talking about these issues is tantamount to addressing them, even though we’ve been having the same conversations for over a decade.

As a consequence of these factors—and it pains me on many levels to say this, not least because I personally know many of the serious, intelligent, and motivated people who are trying to fix this problem—the industry is unlikely to right this ship.

This e-book explores the numerous systemic issues plaguing the programmatic sampling industry. First, we’ll delve into the history of how these market failures emerged and the role technological advancements, like APIs, played in accelerating the decline. Then, we’ll examine the economic principles behind these failures, particularly focusing on the specific market failures that exist in the industry. We’ll also discuss the implications of these failures for the respondent experience, how they have led to what economists call a “tragedy of the commons,” and why the industry’s current trajectory is unsustainable.

Finally, we’ll consider why the market has been unable to correct itself and explore the potential—albeit limited—solutions that could be considered to mitigate these issues.

1. The Inherent, Long-Term Conflict Between Participant Utility and Market Research

At the heart of the sampling industry lies a fundamental assumption: a research participant’s ideal outcome—that which maximizes her/his utility—is to complete a study, whatever the motivation (money, interest, or something else) may be.

It is therefore in our best interest to help participants find and complete high-quality surveys. Simple enough.

Yet, this principle is in direct conflict with two inherent features of survey research:

Not everyone qualifies: Surveys typically target specific audiences, meaning many participants are turned away, which frustrates their efforts to complete a study.
Variable survey quality: The quality of surveys and the survey-taking experience vary widely, impacting participants’ engagement levels.

These features of market research have existed since time immemorial, of course. We know that, over time, they sow the seeds for a terminal event: at some point, people simply stop participating. It is precisely in the digital age, however, that these features combine to create a toxic user experience that accelerates respondent burnout and disengagement, and ultimately degrades data quality.

2. A Brief History

The programmatic sampling industry is facing a crisis rooted in its very structure. It is a classic case of good innovations leading to unforeseen negative consequences.

The roots of this issue trace back to the mid- to late 2000s. During this period, technological advancements that fueled humanity’s digital revolution dramatically improved access to data on consumer behavior, access to sample, and the efficiency of the sampling process. This progress spurred clients to rely more heavily on data-driven decision-making, leading to a period of growth for the industry.

However, this positive trajectory would eventually turn to the dark side.

Web 2.0 and the Rise of the Interactive Web

In the early days of the Internet—some 25 years ago—websites functioned as digital brochures. Online surveys offered a rare chance for users to interact with a largely static web.

Then came Web 2.0, which transformed the Internet. Technology advanced rapidly, improving reach, usability, and engagement. Most importantly, Web 2.0 made the Internet truly interactive, and in doing so, fundamentally altered the dynamics of our industry. This shift was as transformative for market research as the Reformation was for Christianity. With Web 2.0, the research industry lost its monopoly on intercession: no longer were researchers the sole mediators of conversations between brands and consumers. Consumers could now engage directly with brands. (Research departments would shortly thereafter lose their monopoly on insights budgets as well.)

Twitter, YouTube, and Facebook were revolutionary. Meanwhile, our surveys, which had remained unchanged for decades, became decidedly less engaging by comparison.

The First Great Recruitment Crash, and Chasing Yield

By the mid-2000s, the industry’s approach to engaging participants was already problematic. Unsurprisingly, this period marked the first significant collapse in panel retention. I witnessed this firsthand in 2006.

Faced with this capacity problem, the industry responded by taking two major actions:

Fishing in the Broader Ecosystem: The industry began to expand its recruitment efforts, tapping into the broader customer acquisition ecosystem. This included working with affiliate networks, co-registration partners, and—most controversially—sourcing from the infamous “river.” River sampling involved intercepting potential respondents who were browsing unrelated websites and redirecting them directly to surveys, often with minimal screening or validation. This approach aimed to expand the top of the funnel by making it as easy as possible for respondents to reach a survey, but it also introduced significant quality concerns.
Chasing Yield: In a bid to maximize yield, the industry fundamentally changed its sampling practices. Historically, recruitment had been a lengthy process. A new panelist might spend several days completing initial steps, from double opt-in to their first profile survey, before receiving a real client survey. Sampling was equally inefficient. Emails sent to large audiences often resulted in many disappointed potential respondents who were screened out due to low study incidence or timing issues. To address this, the industry developed routers. These systems ensured that once a participant clicked a link, we would show them survey after survey until s/he qualified. Routers completely reversed traditional sampling principles from finding people for surveys to finding surveys for people.

As the industry did this, people began to raise concerns about shoddy panel practices and biased data, especially as the early routers were notoriously clunky.

These developments, however, were merely the tinder for a larger fire.

The Match That Lit the Fire: API-ification of the Supply Ecosystem

As the digital economy rapidly evolved in the late 2000s and early 2010s, businesses raced to innovate, scale operations, and deliver richer user experiences. A key driver of this transformation was the rise of Application Programming Interfaces, or APIs.

APIs served as the “pipes” connecting disparate systems, enabling the seamless integration of features we now take for granted—like real-time social media feeds, third-party payment processing, and scalable cloud services.

In the sampling industry, APIs revolutionized the way respondents were delivered to surveys. What had once been a manual and cumbersome process could now be streamlined and automated. APIs made it possible to shuttle respondents to surveys programmatically, allowing machines to make real-time decisions based on a predefined set of parameters.

However, these interfaces were agnostic about data quality and long-term respondent implications. They were designed for efficiency and scalability and, crucially, were implemented under the assumption that programmatic transactions are frictionless—which they are demonstrably not.

Thus, while APIs are neither inherently dysfunctional nor intrinsically bad, they absolutely accelerated the “death march” of respondents toward disengagement. As participants were pushed through a system indifferent to their experience, their willingness to participate declined. In merely pursuing efficiency, APIs contributed to the erosion of the ecosystem’s integrity—not by design, but by neglect.

And that, readers, is where our story crosses over to the dark side. But before we dive deeper, let’s explore how economists define competitive markets and what it means for a market to “fail.”

3. Competitive Markets and Market Failure

In microeconomic theory, a well-functioning market operates under certain conditions that allow for efficient resource allocation and optimal outcomes for all participants. However, when these conditions are not met, markets can experience failures—situations where resources are misallocated, leading to suboptimal outcomes.

There are essentially four key criteria for a competitive market.

Competitive Equilibrium: A market with many buyers and sellers, where no single entity has the power to dictate terms. This balance allows for fair competition and prevents one or more firms from exercising unfair dominance or, in extreme cases, monopolistic control.
Perfect (or Sufficient) Information: Participants have access to essential information about product quality and pricing, enabling them to make informed decisions. When information is asymmetrical, some participants gain an unfair advantage.
No Externalities: Individual transactions do not impose unintended costs or benefits on others outside the transaction. In a perfectly competitive market, all costs and benefits are internalized, ensuring that market prices reflect the true value of goods and services.
Rational Behavior: All parties act in their own self-interest, making decisions that reflect the true cost and benefit of their actions. Rational behavior ensures that resources are allocated efficiently across the market.

The Old Information Asymmetry: Suppliers Hold the Cards

Even before the digital revolution, the market for research sample suffered from a persistent market failure: information asymmetry.

In the pre-digital era—before AI, APIs, smartphones, and even the Internet—suppliers in the sampling industry had a significant advantage over buyers. This information asymmetry stemmed from the difficulty buyers faced in accurately gauging the quality and value of the sample they were purchasing. The process was entirely manual: buyers would often have to rely on multiple bids from suppliers, with little transparency into what they were actually getting. This lack of transparency meant that suppliers essentially held all the cards, able to set prices with limited accountability. For decades—and even well into the 21st century—sample was often thought to be like wine: if it’s more expensive, it must be better.

The advent of digital networking in the sampling industry mitigated this problem considerably. As I predicted in 2014, APIs reduced the information asymmetry between sample buyers and suppliers by increasing price transparency and liquidity. This promoted greater pricing parity and exerted both top- and bottom-line pressure on suppliers. Yet in doing so, APIs fundamentally changed the online research landscape. This new and seemingly frictionless way of shuffling people to studies set the stage for broader systemic problems.

Kindling for the Fire

Remember: the seeds of the industry’s woes were sown well before the proliferation of APIs.

Frictionless Registration: The push for frictionless registration often resulted in weaker validation and profiling, increasing the likelihood that bad actors could enter revenue surveys undetected.
Technological Backwardness: The industry’s slow pace of innovation and resistance to change (particularly to avoid disruption to tracking and normed studies) meant that the industry’s user experience lagged the rest of the interactive digital world. Because of this, the industry paid a heavy toll in user acquisition and retention, especially among younger audiences.
Lack of Standards: The absence of standardized practices for questionnaire design and operations—even for basic elements like defining incidence rates or coding demographic variables—meant that each company essentially spoke a different language to survey respondents.

Despite these issues, the industry managed to keep things running—mainly thanks to human operators. Though they may have been less efficient, they knew how to fix things and relied on trusted suppliers to get them through tough situations. Additionally, there were limited opportunities for things to spiral out of control. Yes, respondents might bounce around in a company’s router, repeatedly asked for basic demographics, but at some point, there were no more surveys left to try.

However, these backstops disappeared with the rise of programmatic execution and the industry’s relentless push to automate and increase profitability.

With the introduction of APIs, anyone with access and sufficient technology prowess can buy and sell sample, add a bit of margin, and make money—even without their own proprietary audience or survey inventory. A company might use its own sample for proprietary studies while simultaneously rebrokering sample from another supplier to a different buyer.

When this happened, all the chickens came home to roost.

Externalities and the Cost of Turbocharging an Already Terrible User Experience

What is an externality?

In economic terms, an externality is a side effect or consequence of a commercial activity that is not reflected in the cost of the goods or services involved. Simply put, externalities are costs or benefits imposed on third parties who did not choose to incur them. These can be either positive (benefits) or negative (costs).

Pollution is the archetypal externality, and it is a fitting analogy given our topic.

Consider a factory that emits pollutants as part of its manufacturing process. These pollutants harm the health and environment of the surrounding community—people who are not involved in the transaction between the factory and its customers. The cost of this pollution is not included in the price of the factory’s products, meaning the community bears the burden without compensation. This is a negative externality because the true cost of production is not fully reflected in the market price.

In the sampling industry, the poor user experience and the resulting degradation of the respondent pool are negative externalities. The consequences include participant scarcity, unreliable data, and flawed decision-making. While these costs sometimes fall on the firms directly involved, the impact has spread across the entire market, affecting all suppliers and buyers.

Poor User Experience as the Root of the Problem

Poor user experience is widely recognized as a core issue in the sampling industry—one that underpins many of the negative externalities mentioned above. This problem manifests in several ways, each contributing to the overall degradation of data quality and market sustainability.

Survey Design Flaws: Lengthy, repetitive, confusing, or broken surveys frustrate respondents, leading to increased dropout rates and lower-quality data.
Frequent Redirects: Respondents are often shuffled between different surveys and platforms or dropped into routers without explanation, exacerbating frustration and increasing the likelihood of attrition.
Inconsistent Standards: The lack of standardization in demographics and a mishmash of operational practices result in a disjointed and grueling experience, where respondents are repeatedly asked the same basic questions.
Inadequate Data Use: Suppliers and buyers often don’t use known profile data for respondents, leading them up blind alleys to surveys for which they will never qualify, wasting their time and further increasing frustration.
Derisory Compensation: Survey participants are often poorly compensated for their time, especially after “market making” intermediaries take their cut, further diminishing their willingness to participate.

The cumulative effect of these issues is clear: a poor user experience drives respondents away, reducing the pool of high-quality participants and ultimately leading to unreliable data. This degradation of data quality threatens the sustainability of the entire market. (You might wonder why the market hasn’t self-corrected by now; I’ll address that shortly.)

APIs and a Tragedy of the Commons

The interconnected nature of the sampling ecosystem has effectively created a single, massive pool of respondents—one global panel. This dynamic has led to what economists call a “tragedy of the commons,” where individual firms, acting in their short-term self-interest, overuse and degrade this shared resource. Without sufficient investment in maintaining or improving the quality of the respondent pool, the stock of high-quality participants is depleted.

It has long been known (especially in the online era) that respondents often belong to multiple panels, but in a world where the entire supply side of the ecosystem is effectively networked, the impact on panelist and data quality has been catastrophic.

The “Enshittification” of the Market

In a properly functioning marketplace, liquidity should promote efficiency. Sample should flow freely, like water, to where it’s needed most. Market makers, operating on thin margins but high volumes, would ideally facilitate this process by providing liquidity and aiding in price discovery where needed.

But the market for sample is hardly frictionless. Indeed, the opposite is true. The sampling market exemplifies what Cory Doctorow calls the “enshittification” of digital platforms—a process where quality is systematically degraded in favor of profit. Initially, these platforms attract users with high-quality experiences, but as they prioritize profit, they introduce measures that undermine those experiences.

This concept is directly applicable to our ecosystem. Market makers create an illusion of abundant and easily accessible sample, but this masks an ugly truth: their abysmally low conversion rates (calculated as the number of respondents who complete a survey divided by the number who start) clearly show they are plowing through respondents with little regard for their experience.

And it’s not just the market makers. Anyone operating a router is inherently making trade-offs between profit and respondent experience. Once these practices are in place, they are impossible to reverse. Anyone with access to conversion data across multiple suppliers knows this to be true.

By continuously redirecting respondents into suboptimal outcomes, the industry accelerates user frustration and disengagement, thereby reducing the pool of willing, good participants and degrading the overall quality of data collected. Doctorow’s “enshittification” captures this perfectly. The relentless pursuit of growth is eroding the ecosystem and casting serious doubt on the long-term viability of the market.

OK, but what about fraud?

Everyone knows (and everyone knows that everyone knows) that fraud is rampant in the industry.

The enabling technology of the digital age facilitates fraud as a feature rather than a bug, making it increasingly difficult to detect. Researchers are playing a mad game of whack-a-mole with organized fraudsters who, through brute force, clever scripting, and now Generative AI, can easily bypass outdated defenses. This problem has been disastrously exacerbated by the industry’s naïveté, short-term commercial focus, and technological backwardness.

While I have spoken and written extensively about fraud and do not want to diminish its significance, it is an altogether different problem. Fraud is a matter of criminality, not externality. A company’s ability to manage fraud is largely within its control. Unlike market failures, which pose an existential threat to the industry, fraud is a battle that can be fought—albeit without end.

I believe the industry will improve its ability to detect and mitigate fraud, but it will remain an ongoing fight, much like it is in the broader digital ecosystem.

4. Why Isn’t the Market Correcting Itself?

A properly functioning market would address these user experience issues; however, the structure of the sampling market makes this practically impossible.

The New Information Asymmetry: Hidden Costs and a Prisoner’s Dilemma

In a properly functioning market, the costs associated with low-quality data would be transparent, driving a market correction. Higher-quality panelists would be in greater demand, and prices would adjust accordingly, incentivizing firms to invest in better panelist experiences, thereby improving data quality.

But in the sampling market, these costs are not visible to the right people.

A new kind of information asymmetry has emerged, exacerbating the problem and preventing a market correction. While the industry has been vocal about its challenges (especially fraud), there has been no widespread pain felt by clients.

Without this pain, there’s no pressure for change.

Why is this?

Here again, economics gives us the answer. Every actor in this market—suppliers, buyers, and market makers—is conflicted and faces a classic prisoner’s dilemma. Any firm that takes steps to improve quality, which typically means increasing friction and costs, risks losing revenue if others don’t do the same.

To state it plainly:

No sample seller is going to broadcast its fraud reversals or take draconian measures to cut off disengaged respondents, especially when the buyer’s survey is usually the root cause. At most, they might deprioritize that buyer.

No rebroker or market maker is going to stop bashing respondents into studies.

No sample buyer is going to admit, “Yes, we have a fraud and/or engagement problem because of our crappy surveys,” nor will they take draconian measures that reduce feasibility.

Instead, when they are sitting in front of a client, they will say, “Yes, that’s a problem for the industry generally—but not for us.”

Do clients really not know, or do they just not care?

Clients usually play a pivotal role in shaping industry standards, yet there is no sign of revolt on this issue. Why?

As I suggested above, most clients simply don’t know. Many are completely in the dark about the nuances of data collection methodologies. Corporate researchers, in particular, labor under an archaic and obsolete understanding of the sampling ecosystem, leaving them blind to subpar practices and thus unable to demand better. This ignorance effectively removes any gun-to-the-head pressure for the industry to improve quality. Clients aren’t challenging the status quo because they don’t even realize it needs to be challenged.

But suppose that clients do know what’s going on. It is common knowledge that clients make “Iron Triangle” decisions, trading off price, speed, and quality. More precisely, everybody knows that clients regularly prioritize cost-efficiency and rapid turnaround times over data quality. This preference signals to the industry that quality is negotiable, if not dispensable. Consequently, the economic demand for high-quality data is stifled by a stronger preference for cheap and fast results, perpetuating a market where standards are low.

In other words, maybe clients just don’t give a damn.

There is at least one organization with strong end-client representation that is looking at quality. CASE, a group of experts from across the industry, recently dropped a bombshell report on fraud and had an early read of this paper. We see eye to eye on the issues, and I am supportive of their work.

But whether it is a lack of pain or of will, without a significant shift in client demands—brought on by a rude awakening to the long-term impacts of compromised data quality—the industry is unlikely to self-correct.

Competition Inhibits Coordination

Even in markets with externalities, like the one for sample supply, economic theory would suggest that actors could negotiate solutions if transaction costs were low and property rights were well-defined. The idea is that, in a frictionless world, parties could theoretically bargain to correct externalities and reach an efficient outcome, regardless of who holds the rights.

But in the sampling industry, these conditions don’t apply. Transaction costs—those associated with negotiating standards, monitoring compliance, and enforcing those standards—are prohibitively high. The sheer number of actors, the complexity of their relationships, and the business implications make coordination nearly impossible—meaning externalities are left unchecked.

It’s worth explicitly acknowledging the enormity of the coordination problem. For one of the “solutions” that people bandy about is some sort of convergence on standards—as if the industry just needed to roll up its sleeves a bit further to get there. About a dozen years ago, when the industry was confronting the dual challenges of rampant smartphone adoption and nascent API-ification, I was an active participant at early meetings attended by senior people from major market research agencies where we discussed the merits of standards. About the most we could agree on was that they would be helpful and improve the user experience. But two things became clear in those meetings:

Each company was heavily invested in its own methods. Even if they were willing to cooperate, the implications of change would be vast, affecting everything from commercial agreements to operational practices to technology development.
Every person in that room was a competitor, which meant that from the outset, nobody was inclined to cooperate.

These coordination and self-interest issues are classic characteristics of market failures. Economic theory proposes solutions, but they offer only limited promise.

A Central Authority?

Resolving market failures with endemic coordination problems typically requires a central authority—either voluntary or compulsory—with the power to set standards and enforce rules that lead to more efficient and equitable outcomes.

In the case of pollution, this role is played by the government, which imposes environmental regulations backed by the force of law.

But in the absence of such “hard authority,” market participants might still voluntarily cede certain rights to a central group. This can happen through industry associations, self-regulatory organizations, or cooperative agreements where firms recognize the mutual benefits of adhering to common standards. For example, the Internet Engineering Task Force (IETF) and the Interactive Advertising Bureau (IAB) are associations whose members voluntarily adhere to agreed-upon protocols and standards.

However, these voluntary agreements are often flimsy. Their effectiveness depends on the subject matter, the complexity of the industry, and the level of support from dominant players. By way of example, Spotify, the world’s largest online music and podcasting service, dropped its IAB membership and certification in 2024. While it’s unlikely that Spotify will completely abandon IAB guidelines, it surely had a self-serving reason for leaving, one which it believes will yield a net benefit with negligible consequences. Nonetheless, it is a blow to the IAB’s credibility and sustainability.

Can the associations solve the problem?

Recently, a who’s who of market research associations has come together under the Global Data Quality Initiative. Per the GDQ’s website:

The Market Research Society (MRS UK) is coordinating efforts with Association for Qualitative Research (AQR), The Canadian Research Insights Council (CRIC), ESOMAR, Insights Association, the QRCA, The Research Society (TRS), SampleCon, and the Association of Market Research Austria (VMÖ) to address ongoing and emerging risks to data quality in the market and social research, consumer insights and analytics industry. With the goal of increasing information and building trust, each organization will lead a workstream that delivers to the global quality resources to improve the conversation and outcomes around:

The language of quality - how we refer to the different aspects of fraud, duplicates, and survey cleaning in ways that inform with accuracy and transparency
Fraud detection – tracking the prevalence of fraudulent survey completions by humans or bots and outlining best fraud detection and mitigation practices
Identification and mitigation of bias from sample frame and representativeness
Data quality in research surveys, and the resulting impact on overall quality of the data
Improvement in the research participant experience

These are undeniably positive steps. I know many of the people working on this initiative. They are highly capable and deeply understand the issues. Establishing best practices and increasing transparency will certainly help.

But ultimately, the nature of research is not black and white. Despite decades of experience, thousands of research-on-research studies, millions of metrics, and exabytes of data, what is considered best practice varies considerably. Broad standards may be necessary, but they run the risk of being so generalized that certification becomes meaningless. Even if we could define meaningful standards, enforcing them is another matter entirely. What would change from the standpoint of user experience? What would it mean to be noncompliant? Could we ever imagine formal sanctions?

I have no doubt the industry will produce instructive guidance on best practices. But history offers little reason to believe these efforts will lead to a meaningful certification process, much less a compliance mechanism—especially with the rise of new, non-traditional entrants who do “insights” work but have little connection to the insights industry’s associations.

Consequently, for all the reasons I’ve noted above, I don’t believe the industry will be able to address its market failures through these associations. Firms will need to make individual decisions with the understanding that these systemic issues will persist. Clients, too, will be left to navigate this landscape largely on their own.

Just as they do now.

5. The Illusion of Vigilance

In the sampling industry, the constant dialogue around poor user experience has created the illusion that the issue is being actively managed and addressed. However, this perception masks a deeper and more difficult reality: the narrative itself has become so entrenched that it sustains the very problem it seeks to solve.

Ben Hunt, Chief Investment Officer of Second Foundation Investment Partners, argues persuasively that narratives shape and reinforce behavior in markets, industries, and civil society. He would describe the industry’s approach to user experience as a “Common Knowledge” problem. The reasoning goes like this:

Everyone knows that poor user experience is a problem.
By virtue of how thought leadership functions in the industry, this issue has become a regular topic at conferences and in industry publications. Not only does everyone know that poor user experience is a problem, but everyone knows that everyone else knows it, too.

Here’s the catch: because this problem stems from an unaddressed market failure—where the incentives to fix it simply don’t exist—the industry has little choice but to substitute conversation for real action. Poor user experience becomes Poor User Experience!™, a shorthand for a problem everyone understands but no one solves. It is pure theater. Poor User Experience!™ is so widely accepted and understood that we don’t even need to explain it anymore. We throw the term out there—earnestly or offhandedly—to suggest that We Understand the Problem and Are Working on It!™ even though we have been having the same conversations for over a decade.

(To be fair, there are a few individuals genuinely fighting the good fight, but a quick scan of conference agendas and thought leadership pieces shows how entrenched this narrative has become.)

The real danger here lies in how this Common Knowledge reifies the narrative, actively preventing change. The industry’s collective focus on Poor User Experience!™ acts like a social cover-up for the deeper issue: an unresolved market failure. It is an acknowledgment that gives the industry a pass to substitute words for action. As long as the narrative persists, it sustains the status quo, creating yet another massive obstacle to meaningful improvement.

For reality to change, the narrative must be shattered by something so significant—so universally understood and accepted—that it forces a reevaluation of the industry’s approach. What could trigger such a shift?

High-Profile Scandal: A massive scandal involving widespread fraud or data quality failures that severely impacts the entire industry could serve as a narrative-breaking event, compelling the industry to confront the reality of its issues. Pollsters, especially, worry about this.
Technological Breakthrough: A groundbreaking technological advancement that dramatically improves survey quality and user experience could redefine industry standards, challenging the existing narrative. AI offers some promise here, particularly with conversational research that mimics human interaction. However, for AI to truly disrupt the status quo, it would need to be more than just a novel tool—it would have to fundamentally alter the way we conduct sampling, which would include sparing respondents the endless hell of router dumping and significantly enhancing engagement.
Market Disruption: A new entrant to the market with innovative practices could set a new standard for quality and efficiency, disrupting the status quo and forcing existing players to adapt. The disruption required to break the current narrative, however, would have to be substantial—introducing not just a better product, but a completely new approach that renders the old ways obsolete.

The odds of any of these happening seem long indeed.

6. Caveat Emptor: Advice for Buyers

So, what should conscientious buyers do then?

The industry’s end clients historically used brand names as a shortcut for quality. That ship sailed ages ago. No sample company—be it a rewards community, a performance marketer, a marketplace, or traditional MRAs—is immune from these problems.

Given the numerous reasons for which a durable, structural solution will remain elusive, the logical next step is for individual firms to take steps that mitigate the issues, if only for themselves.

This starts by understanding the fundamentals of the modern sample ecosystem.

Understanding the Principles

The unfortunate reality is that many clients (perhaps most) don’t understand modern sample practices, leaving them vulnerable to poor-quality data and its consequences. A good starting point is ESOMAR’s guidebook, 37 Questions to Help Buyers of Online Samples. As its name suggests, it is a comprehensive resource (perhaps overly so) that will help buyers know what questions to ask of suppliers. That said, too few suppliers provide the detailed information it requests—and when they do, it’s often too vague to be genuinely useful.

At a minimum, clients should understand the following:

Sample Origins: Know the overall structure of players in the market. Be particularly aware of which suppliers have proprietary sample and which are merely rebrokering others’ audiences.
Recruitment Practices: Learn how panel owners and rewards communities recruit new members. Specifically, focus on how they “fish” in the broader ecosystem for registrations. Understand the practices they use to verify that participants are real, profiled, attentive, and nonduplicated.
Fraud Risks and Mitigation: Understand where and how fraud can occur, be it during registration, survey completion, or incentive claiming. Recognize the practices that encourage fraud and what suppliers should be doing to combat it.
Yield Maximization and Routing: Familiarize yourself with how yield maximization and routing affect user experience. Know how both buyers and suppliers use these practices to determine where a participant is sent and how it impacts the quality of data collected.
Key Performance Indicators (KPIs): Equip yourself with the knowledge of critical KPIs used to monitor performance and compare vendors. This is essential for making informed decisions about your sampling strategies and partnerships.

Understand Supplier Practices

While understanding the generalities of the ecosystem is crucial, clients must bring this to life by understanding the specific practices of their suppliers. Surface-level marketing and vague promises are not enough. There are five key areas to dig into.

Routing and Profiling Practices: Clients must demand detailed information about how respondents are routed—not just from sample providers, but from any research agencies or data collection companies (essentially those hosting the survey) as well. What KPIs are they optimizing? How many redirects are typical? Are respondents often funneled through multiple surveys before completing one? High levels of routing can degrade the respondent experience and lead to low-quality data. Additionally, evaluate the depth and accuracy of profiling data available from suppliers and how they use this data for targeting. The management of routing and profiling directly impacts conversion rates and respondent satisfaction, both critical KPIs.
Survey Quality Assurance: Understanding the steps suppliers take to ensure survey quality is non-negotiable. This includes knowing their processes for testing surveys, ensuring they function correctly across devices, and verifying that they meet engagement standards. Will a supplier reduce the flow of sample to poorly performing studies? What does that mean for you? Poor survey quality can lead to high dropout rates and unreliable data, which are reflected in KPIs like conversion rates and panel engagement metrics.
Compensation and Participation: Examine how respondents are compensated and the frequency with which they are allowed to participate in surveys. Suppliers should provide clear information on their incentive structures, as well as how much of the respondent’s compensation is consumed by intermediaries. Inadequate compensation and over-participation can lead to respondent fatigue, lower engagement, and skewed data, affecting KPIs such as reversal rates and panel engagement metrics.
Third-Party vs. Proprietary Sample Practices: Understand how suppliers manage third-party samples compared to their proprietary panels. This includes the vetting process for third-party sources and the standards they impose to ensure data quality isn’t compromised. The balance between third-party and proprietary sample can significantly influence KPIs like conversion rates and the overall integrity of the data.
Research-on-Research and Continuous Improvement: Inquire whether suppliers are conducting research-on-research to continually refine their methods and benchmark against gold standard sources. Suppliers committed to continuous improvement are more likely to deliver high-quality data.

Identify and Monitor KPIs

KPIs are king in programmatic sampling. Algorithms route panelists in a constant effort to maximize something, be that revenue, profit, or even a combination of both.

Gone are the days when incidence and response rates are useful metrics, and one will learn next to nothing from the pure fiction known as the Panel Book. Here are the key metrics to focus on:

Conversion Rates: Conversion rates are the ratio of completes divided by starts. They are an all-encompassing statistic that essentially shows you how attentive the supplier is to participant experience. For example, a 30% conversion rate is quite good, whereas you may find market makers with conversion rates of 5%. When you have conversion rate as well as its component metrics (quota fulls, incidence, and abandon rates), then you can learn very quickly what kind of work the company does, how good its targeting is, and whether it exposes people to difficult work.

Panel Engagement Metrics: These include retention rates and frequency of participation. Many providers will have an internal measure of panelist quality as well, which is their way of assessing whether someone is a good responder. High engagement and retention rates typically correlate with higher data quality, as engaged respondents are more likely to provide thoughtful, reliable answers.

Fraud and Disengagement Detection Rates (the stuff they catch up front): While fraud is a pervasive issue, understanding how a supplier detects and manages it is crucial. Higher detection rates might indicate a proactive stance, but you’ll want to ensure that fraud is effectively handled to maintain data integrity.

Reversal Rates (the stuff they catch after the fact): This metric tracks the percentage of completed surveys that are later rejected due to fraud or poor-quality responses. High reversal rates are a warning sign that the supplier may have issues with panel management or fraud detection.

Completes by Third-Party Audiences vs. Proprietary Sample: Know the balance between proprietary and third-party sample in your supplier’s offerings. Third-party sample introduces additional risks, especially if those sources have lower standards. Ensure that third-party samples are held to the same quality checks as proprietary samples.

Write Good Surveys

Let’s simply state the obvious. A good survey is one which is adapted to the device being used, allows participants to easily summon and enter responses to questions, engages respondents throughout and is respectful of their time, and functions without breaking. Good suppliers will help you identify and correct issues.

Vigilance Is the Only Viable Path

The best defense is vigilance. By understanding the fundamentals of programmatic sampling, focusing on KPIs, and thoroughly vetting supplier practices, clients can navigate the murky waters.

While the industry itself may not correct course, individual players who adopt a caveat emptor approach can still find success and avoid the worst pitfalls. Ultimately, this path isn’t about finding perfect solutions—those don’t exist in this industry—but about mitigating risks and making informed choices. The burden of quality assurance falls on those who care enough to demand it and those who are committed to delivering it.

7. Caveat Vendor: Advice for Suppliers

Nine years ago, when I first wrote about sample suppliers blocking problematic buyers—those offering poor-quality surveys with low conversion rates—it was a rare and manual process. Today, this process is common and automated. Indeed, most vendors now have fine-grained controls that allow them to make decisions at a very granular level, evaluating not just buyers in the aggregate but individual projects and even specific quota groups. Yet, despite these advances, poor survey experiences remain too common.

Vendors can take several key steps to preserve their panels by improving these experiences.

Know and Leverage Your Full Unit Economics

Vendors commonly use some variant of Earnings per Click/Entry (EPC) to decide which survey they will send respondents to. This isn’t enough: it’s only the revenue side of the equation.

Sustainability is only possible by understanding full unit economics—both the revenue generated by sending a respondent to a survey and the potential long-term costs of replacing that panelist if they churn.

Each potential respondent comes with two key probabilities that vendors must calculate and incorporate into their decision-making:

Probability of Completing a Survey (PC): This is the likelihood that a respondent will qualify for and complete a given survey. This will be a function of a variety of factors, from demographics to the specific needs of the study to the vendor’s recent experience sending similar panelists to that survey. PC drives the expected revenue from the survey.
Probability of Imminent Attrition (PA): This is the likelihood that a respondent will stop taking surveys altogether after this experience. This is influenced by recent experiences (e.g., failed attempts, survey abandonments), the payout, time investment, and tenure on the panel. PA informs the long-term costs of losing the panelist.

Vendors can combine these probabilities to calculate the expected value and costs of routing a respondent to a survey:

Earnings Per Click/Entry (EPC): This is the total earnings divided by the number of respondents who attempted to complete a survey, and it is one of the currencies by which suppliers decide where to send sample. This starts off as an expected value function, where EPC is equal to an estimated aggregate Probability of Completing (PC) multiplied by the payout. Over time, this will become an observed value. Some companies also use Earnings Per Click per Minute (EPCM) to factor in the time a respondent is likely to spend on the survey.
Expected Replacement Cost (ERC): This is the expected cost of replacing a respondent if they quit taking surveys. This is a forward-looking calculation based on Probability of Attrition (PA) and the typical recruitment costs for a panelist with similar demographics and engagement levels. This will intrinsically be an estimate given that the actual attrition event is likely to only be observable after the project is complete. It is thus something that would need to be evaluated and simulated before implementation and monitored for accuracy and predictive power.

The goal is then to compare these values in real-time to know not just which survey is the better earner, but whether the best decision may be to not send the respondent anywhere! Every time a respondent abandons a survey, the likelihood of them never taking another survey doubles. Ignoring this fact can lead to disastrous long-term costs. An effective algorithm will allow firms to naturally put a thumb on the scale and prioritize revenue versus profitability—and they should. Variable margins can also potentially improve entries and conversion depending on the respondent and study characteristics.

Incent the Right Behaviors, and Don’t Tolerate Abuse

Pricing and costs in the industry are primarily denominated in completes. Any outcome short of that was sort of “too bad” for the respondent, regardless of time spent or how poorly the survey performed.

I wrote nearly ten years ago that suppliers need to stop tolerating abusive buyers. This is now happening. Suppliers must go a step further by variably incenting participants for beneficial actions—from providing profile data to trying to take part even if they don’t complete the study—and factor this into their costs.

Continuously Refine for Operational Excellence

It may seem unrelated, but operational excellence is a powerful lever in maximizing sampling quality and revenue. In any programmatic execution, revenue and profit are closely linked to operational speed—specifically, how quickly respondents can be matched to and complete surveys.

But speed isn’t enough. You want to be fast and smart. The operational systems that route respondents must optimize both speed and quality, reducing poor experiences by targeting the right respondents in real-time and maximally reducing friction.

Automation across the entire supply chain—algorithmic decision-making, constant tuning of demographic mapping and API performance, and real-time fraud detection—is never “one and done.” Integrations are living, breathing things and require constant maintenance. Every tenth of a second shaved off the survey-taking process can significantly improve user experience, respondent retention, and profitability. Over time, incremental gains add up. Proper automation, done right, can directly enhance the quality of data and reduce dropout rates, ensuring respondents are more likely to stay engaged in the long term.

Never Stop Learning and Innovating

Panel companies have two main assets: their audiences and the intelligence they use to optimize recruitment and deployment. Firms should be constantly tuning their models through analysis and simulation to understand the impact of survey experiences, incentives, and operational practices on panelist health, attrition rates, revenue, profitability, and long-term value—and then baking these learnings back into their AI/ML models.

Ultimately, vendors who apply these principles and consistently innovate will build a competitive advantage in both profitability and quality.

Conclusion

The programmatic sampling industry is ensnared in a web of market failures and systemic issues that thwart effective solutions. Despite the technological advancements that have revolutionized data collection and accessibility, the industry is plagued by poor user experiences and rampant fraud. Of these, poor user experiences present the greatest challenge. They are the root cause of economic externalities. The problem is exacerbated because its consequences are hidden from clients, who may themselves be making conscious tradeoffs between quality and price. The net result is the absence of any lever of demand that would normally promote a market correction. A structural problem demands a structural solution—one that compels action to address the poor user experience at scale. Yet any solution will require coordinated action on a historically unprecedented scale.

While the efforts of industry associations and technological innovations offer glimmers of hope, they will fall short of the mark unless they address the structural and collective action problems at their core. Concretely, this means clients must understand what’s going on and send the right signals to the market. It also means that the industry, collectively, must make a concerted effort to prioritize respondent experience. At minimum, this must entail the creation of rigorous standards that include transparency—and, ideally (if implausibly), certification or enforcement.

The challenges are daunting, but acknowledging them is the first step toward a sustainable sampling ecosystem. It is incumbent upon all stakeholders—suppliers, buyers, and clients alike—to recognize the gravity of these issues and work collaboratively toward solutions that ensure the long-term health and viability of the industry.

Sources

Author’s Note

Deitch, JD. “‘Big Bang’ Disruption – and Five Keys to Survival.” Research Live, 5 February 2014. https://www.research-live.com/article/features/big-bang-disruption—and-five-keys-to-survival/id/4011192

Chapter 2. A Brief History

O’Reilly, Tim. “What Is Web 2.0: Design Patterns and Business Models for the Next Generation of Software.” O’Reilly Media, 30 September 2005. http://www.oreilly.com/pub/a/web2/archive/what-is-web-20.html

Deitch, JD. “‘Big Bang’ Disruption – and Five Keys to Survival.” Research Live, 5 February 2014. On research departments losing their monopoly on insights budgets.

Grey Matter Research & Consulting. “Dirty Little Secrets of Online Panels.” 2009. https://greymatterresearch.com/online-panels/

Chapter 3. Competitive Markets and Market Failure

Deitch, JD. “Disrupting Market Research: An Update.” Greenbook, 2 December 2014. https://www.greenbook.org/insights/market-research-trends/disrupting-market-research-an-update

Hardin, Garrett. “The Tragedy of the Commons.” Science 162, no. 3859 (1968): 1243–1248.

Doctorow, Cory. “The ‘Enshittification’ of TikTok.” Wired, 23 January 2023. https://www.wired.com/story/tiktok-platforms-cory-doctorow/

Coalition for Advancing Sampling Excellence (CASE). “About.” https://case4quality.com/about

Deitch, JD. “Fraud Alert: Ghost Completes.” Research Live, 27 April 2020. https://www.research-live.com/article/opinion/fraud-alert-ghost-completes/id/5068182

Chapter 4. Why Isn’t the Market Correcting Itself?

Davis, Morton D. “Game Theory: The Prisoner’s Dilemma.” Encyclopaedia Britannica. https://www.britannica.com/science/game-theory/The-prisoners-dilemma

Coase, R. H. “The Problem of Social Cost.” Journal of Law and Economics 3 (1960): 1–44.

Podnews. “Spotify Leaves the IAB; No Longer Certified.” 22 April 2024. https://podnews.net/update/spotify-leaves-iab

Global Data Quality (GDQ). https://globaldataquality.org/ Quoted as the site read in October 2024; the list of partner associations has since grown.

Chapter 5. The Illusion of Vigilance

Hunt, Ben. “Epsilon Theory Manifesto.” Epsilon Theory, 1 June 2013. https://www.epsilontheory.com/epsilon-theory-manifesto/

Chapter 6. Caveat Emptor: Advice for Buyers

ESOMAR. Questions to Help Buyers of Online Samples (the “37 Questions”). Amsterdam: ESOMAR, 2021. https://esomar.org/code-and-guidelines/37-questions-to-help-buyers-of-online-samples

Deitch, JD. “Death to the Panel Book: Uncovering Better Sample Metrics.” Research World. https://archive.researchworld.com/death-to-the-panel-book-uncovering-better-sample-metrics/

Chapter 7. Caveat Vendor: Advice for Suppliers

Deitch, JD. “The Future of Panels: Five Investments Needed to Improve Participation and Reduce Bias.” Greenbook, 14 May 2015. https://www.greenbook.org/insights/research-methodologies/the-future-of-panels-five-investments-needed-to-improve-participation-and-reduce-bias

The Enshittification of Programmatic Sampling

Summary

The full text