ANN ARBOR — By design, surveys hinge on responses. Every survey requiring any kind of a response from participants is only as good as the people who answer. But somewhat paradoxically, the quality of a survey can also be affected by those who don’t respond through a phenomenon called nonresponse bias.

Nonresponse bias occurs when those who are unwilling or unable to take part in a survey differ from those who do in a significant enough way that the group of those who did respond to the survey is no longer representative of the larger population being studied. 

Nonresponse bias can be a significant problem for survey research, but researchers are constantly working to develop new strategies to overcome it, and one of those strategies was the subject of a recent presentation by Brady West as a part of the Insights Speaker Series at the Institute for Social Research.

In the presentation, titled “Moving Beyond Response Rates to Understand Nonresponse Bias,” west outlined alternatives to simply reporting the response rate for a given survey as a means for describing nonresponse bias in a given survey. West argued that researchers need to be careful even in how they address nonresponse bias, because a faulty definition nonresponse bias can have consequences, noting that it can be particular problem when surveys only think of potential participants as “respondents” or “non-respondents” all the time.

“If you were to survey or ask me to participate in a survey over and over again, I’d probably respond. But if you talk to my mom, she’s going hang the phone up on you every time you call her,” he said.  “Unfortunately, in [a deterministic definition of nonresponse bias], that means people who would never, ever respond are, of course not observed because they never, ever respond. So this is a theoretical definition of nonresponse bias, but it illuminates where that bias comes from.”

Alternatively, researchers can think of nonresponse bias as a mathematical issue.

“You can think of it as a stochastic phenomenon where the bias can roughly be written as the co-variance between the Y variable of interest and the person’s propensity to respond, divided by the average propensity in the survey that we’re trying to conduct,” said West. “Nonresponse bias is going to be large under this setting.”

Ultimately, West concluded that nonresponse bias must be confronted, but researchers have more tools than ever to address it.

“In all the estimates that we produce based on our surveys, there are more modern approaches to measuring and reducing bias, both quantitative and qualitative that now exist and they’re easy to implement and they’re easy to document,” he said. “We all know that survey response rates are declining in all different kinds of surveys, but it’s important to do as much as possible from a design perspective to combat these trends and not read into lower response rates as automatic evidence of non-response bias.”
West’s full Insights presentation is available on the ISR YouTube channel and below. For more on the Insights speaker series, click here.