Key indicators that are used for policy making often come from large-scale sample surveys, and rely on the accuracy of the data from surveys. The existing and growing need for such data has led to the considerable length of many surveys, resulting in substantial burden for individuals. There is evidence showing that the length of the survey is affecting participation in surveys, leading to greater nonresponse and increased potential for nonresponse bias. Longer surveys have also been linked to suboptimal responding due to increased burden, resulting in measurement error. Thus, long interviews can bias survey estimates and lead to misinformed decisions. Given the serious consequences and general increasing reluctance to participate in surveys, the effect of survey length has received surprisingly little attention to its measurement and reduction.
We propose a combination of a full-factorial experimental design to separate and estimate the effects of nonresponse and measurement error in both interviewer- and self-administered modes of data collection. We will also use this experiment to implement a split questionnaire design, randomly assigning respondents to receive a subset of the survey questions, and multiply imputing data for the omitted questions. Our main hypothesis is that this approach will yield estimates with less bias and even less total error compared to asking the full questionnaire. This will be the first such experimental comparison and will likely help the proliferation of this fundamentally different approach to survey design.
To evaluate the extent of the problem and implement a solution, this study has four specific aims:
1. Examine whether measurement error increases as a function of survey length;
2. Isolate the impact of survey length on nonresponse bias; and
3. Evaluate the reduction of bias and impact on mean square error from using split questionnaire design, after multiply imputing the full data for all respondents.
This study will be the first to evaluate the effect of both nonresponse and measurement error due to survey length. It will also be the first to experimentally demonstrate the potential benefits of split questionnaire design in improving survey estimates, and how it can be implemented using available tools. The long-term objective of this study is to provide empirical evidence leading to a paradigm shift in survey design that will allow for improved information on health while reducing respondent burden.
Funding:
Research Triangle Institute (RTI)
Funding Period:
10/28/2013 to 09/30/2016