Information Equivalence in Survey Experiments

Allan Dafoe, Baobao Zhang, & Devin Caughey

Survey experiments often manipulate the description of attributes in a hypothetical scenario, with the goal of learning about those attributes’ real-world effects. Such inferences rely on an underappreciated assumption: experimental conditions must be information equivalent (IE) with respect to background features of the scenario. IE is often violated because subjects, when presented with information about one attribute, update their beliefs about others too. Labeling a country “a democracy,” for example, affects subjects’ beliefs about the country’s geographic location. When IE is violated, the effect of the manipulation need not correspond to the quantity of interest (the effect of beliefs about the focal attribute). We formally define the IE assumption, relating it to the exclusion restriction in instrumental-variable analysis. We show how to predict IE violations ex ante and diagnose them ex post with placebo tests. We evaluate three strategies for achieving IE. Abstract encouragement is ineffective. Specifying background details reduces imbalance on the specified details and highly correlated details, but not others. Embedding a natural experiment in the scenario can reduce imbalance on all background beliefs, but raises other issues. We illustrate with four survey experiments, focusing on an extension of a prominent study of the democratic peace.

Journal. Paper pdf

Pre-analysis Plan for Study of Effect of Democracy

Pre-analysis Plan for Study of Effect of Childcare

Redundant links:

Manuscript pdf (March 15 2018)

OSF Page with Pre-analysis Plan

EGAP Pre-Registration and Pre-analysis Plan ([167] 20150701AA; 176] 20150715AA)

Pre-analysis plan code on GitHub