Dalia Ghanem (UC Davis)

"Testing Attrition Bias in Field Experiments"

Nov 06, 2018
from 04:10 PM to 05:30 PM

4101 Social Science and Humanities Giannini Library Conference Room


Attrition is a common threat to the internal validity of field experiments in economics.  This paper brings insights from the nonparametric identification literature in panel models to address the question of how to formally test for attrition bias in the presence of baseline outcome data.  We first conduct a systematic review of the field experiment literature in economics and find that there is no consensus on how to test for attrition bias.  Of the field experiments we reviewed:  81% test for differential attrition rates, but only 59% conduct some type of test for internal validity conditional on response status.  We then formally  demonstrate  that:  (1)  the  differential  attrition  rate  test  does  not  control  size in general, and (2) internal validity has a sharp testable restriction on the baseline outcome distribution.  We show that this restriction implies joint (as opposed to simple) tests of distributional equality, which are not widely used in the literature.  We also demonstrate  that  the  restriction  differs  depending  on  whether  the  authors’  object  of  interest is the treatment effect for respondents or for the population selected for the evaluation.  We propose randomization procedures to obtain p-values for Kolmogorov-Smirnov- and Cramer-von-Mises-type statistics of the joint hypothesis.  We further extend our framework to testing internal validity in the case of stratified randomization.  Finally, we include simulation experiments to illustrate the implications of these results for empirical practice.

Filed under: Development