March 19, 2018
Professor, Geisinger Health System
Visiting Fellow, Institute for Advanced Study in Toulouse
ABSTRACT: Randomized controlled trials (RCTs) are a “gold standard” for evaluating drugs and other interventions and policies in medicine, business, and government. Yet in several recent cases, RCTs or other empirical studies have proven controversial and sparked outrage, even when they tested treatments that were close or identical to treatments already used in practice with little objection. The “A/B Illusion” is a proposed anomaly of ethical judgment in which people view a field experiment (or more generally, any research effort) designed to study the effects of an existing or proposed practice (an “A/B test”) as more morally suspicious than a universal implementation of an untested practice (A or B). In a series of preregistered experiments with nearly 5000 participants, we found: (1) The A/B illusion can be observed with substantial effect sizes in a wide variety of domains, including medical checklists, self-driving car design, blood-pressure drugs, retirement plan defaults, poverty alleviation, and genetic testing. (2) The effect cannot be fully explained by participants holding intuitive prior beliefs about the superiority of A or B (so that an experiment may seem unfair to whoever gets the “inferior” treatment); participants being averse to any mechanism that randomly assigns people to a policy; or participants failing to imagine alternative policies to A and B prior to evaluating them in isolation. (3) Some participants object to the lack of informed consent when a policymaker runs an A/B test, but no participants object to the lack of informed consent when a policymaker unilaterally imposes either A or B. These results may help us develop ways to better explain the purpose and value of RCTs as powerful knowledge-creating tools to prospective participants and other stakeholders.
Joint work with: Michelle N. Meyer,1 Patrick R. Heck,1 Geoffrey S. Holtzman,1 Stephen M. Anderson,1 William Cai,2 Duncan J. Watts2 [ 1 Geisinger Health System; 2 Microsoft Research]