Manuscript pdf (July 18 2016)
Allan Dafoe, Baobao Zhang, & Devin Caughey
Despite random assignment, scenario-based survey experiments suffer from threats to internal validity that resemble confounding bias in observational studies. Manipulating specific features of a scenario generally changes subjects’ beliefs about background features of the scenario, possibly confounding the effects of the beliefs of interest. Describing a hypothetical country as “a democracy”, for example, makes respondents more likely to think it is wealthy, European, predominantly Christian and white, and interdependent and allied with the United States. We show how to theorize about confounding in survey experiments, and how to diagnose it using placebo tests. We also evaluate three potential solutions. Encouraging subjects not to think about specific countries does not reduce confounding. Controlling for potential confounders in the scenario reduces confounding only on the controlled and closely related attributes. The best solution is embedding a natural experiment in the scenario, a new technique that mitigates all sources of confounding.
EGAP Pre-Registration and Pre-analysis Plan ( 20150701AA; 176] 20150715AA)