Prospective sampling bias in COVID-19 recruitment methods: experimental evidence from a national randomized survey testing recruitment materials

Abstract Background In the context of the COVID-19 pandemic, social science research has required recruiting many prospective participants. Many researchers have explicitly taken advantage of widespread public interest in COVID-19 to advertise their studies. Leveraging this interest, however, risks creating unrepresentative samples due to differential interest in the topic. In this study, we investigate the design of survey recruitment materials with respect to the views of resultant participants. Methods Within a pan-Canadian survey (stratified random mail sampling, n = 1969), the design of recruitment invitations to prospective respondents was experimentally varied, with some prospective respondents receiving COVID-specific recruitment messages and others receiving more general recruitment messages (described as research about health and health policy). All respondents participated, however, in the same survey, allowing comparison of both demographic and attitudinal features between these groups. Results Respondents recruited via COVID-19 specific postcards were more likely to agree that COVID-19 is serious and believe that they were likely to contract COVID-19 compared to non-COVID respondents (odds = 0.71, p = 0.04; odds = 0.74, p = 0.03 respectively; comparing health to COVID-19 framed respondents). COVID-19 specific respondents were more likely to disagree that the COVID-19 threat was exaggerated compared to the non-COVID survey respondents (odds = 1.44, p = 0.02). Conclusions COVID-19 recruitment framing garnered a higher response rate, as well as a sample with greater concern about coronavirus risks and impacts than respondents who received more neutrally framed recruitment materials.