Discussion

To the best of our knowledge, the survey practitioner is left with very little guidance when it comes to making a decision on how much effort and resources should be invested in order to reach and interview potential participants. In spite of attempts to use standardised terminology to designate the various sources of survey non-response,21 the academic literature uses inhomogeneous wording for non-response due to reaching difficulties: ‘not at home’, ‘inaccessible’, ‘non-contact’ or ‘failure to deliver’.4 ,5 ,12 In addition, there is no indication regarding the amount of effort beyond which a household could qualify under one of these terms. In turn, little is known about non-response as a function of reaching effort.

Nevertheless, since the inferential paradigm of random sampling demands full completion to guarantee that survey estimates are unbiased, survey researchers are prescribed to maximise response rates.2 ,15 ,22 How high the response rate should be, however, has been answered so diversely (from 50% to 85%)22 that survey practitioners are likely to be confused. Furthermore, empirical evidence has shown that response rate is a weak predictor of non-response bias.8 ,11 ,22–26 The amount of effort that lowers non-response bias under an acceptable level is therefore unknown, but indicators of representativeness that can inform fieldwork decisions are now under development.24 The issue is aggravated by the hypothesis that measurement bias might increase with effort to reach participants: late respondents, who are less available and/or more reluctant to take the survey—but will eventually take it—might be more likely to provide answers filled with measurement errors.15

Maximum number of attempts reported to contact potential participants vary considerably: from four19 to unlimited13 ,18 and there is no typical value in between.2 ,4 ,12 ,15 ,17 ,20 Reports, however, consistently show a dramatic decrease in completion rate from one attempt to the next; first attempts have yielded 30–70% of the final sample, and more than 80% were completed by the fourth attempt.2 ,4 ,12 ,13 ,15 ,18 ,19 Our results are in line with these reports, suggesting that effort to reach and interview participants could be stopped earlier.

Hence, analyses of key variable estimates as a function of collection effort are all the more important. In our health survey, frequencies of most of the medical conditions diminished significantly as the effort increased, while sociodemographic characteristics stayed stable—except for household size, which is expectedly correlated with contact rate,7 employment status and income. This result suggests that non-participation at early stages is related to the survey topic, with healthier households participating at later stages. In contrast to the abundant literature on survey non-response, few health surveys compared late or hard-to-reach respondents to earlier or easier-to-reach respondants.10 ,18 ,27–29 Some found results in line with ours,18 ,27 although to a lesser extent, but others did not or gave equivocal results.10 ,28 ,29 Since the literature on this subject is scanty, replication of similar analyses is needed.

While percentages of conditions varied from one round to the next, the same was not true for cumulative percentages, which represent the percentages that would have been obtained if the survey was stopped at each given round. Beyond the fifth attempt, percentages varied by less than 1%. Likewise, SEs of these percentages decreased by less than 0.1% beyond the fifth wave, suggesting that maintaining the effort up to five rounds might be enough.

In the absence of a sufficient body of knowledge, the health survey practitioner might err on the safe side and take our results for granted. Indeed, there is a rationale for such results: less healthy households are likely to be interviewed at early stages of a home-based survey, since their members are more likely to be present at home and more inclined to take a survey that resonates with their current preoccupations. To some extent, such ‘unhealthy at-home’ effect can be seen as the reciprocal of the healthy worker effect.30 Indeed, in our survey and in several large surveys of the British population, being employed and being hard to reach were found to be correlated.20 In these British surveys, the participants who were easy-to-reach during daytime on weekdays were found to be less healthy than other participants; this supports an ‘unhealthy at-home’ effect. In our survey, however, increasing the reaching effort did not correspond to increasing evenings or weekend interviews: a non-significant trend (p=0.064) was found in the opposite direction.

Our results might be due, however, to the aforementioned measurement bias. Since we dealt with self-reported conditions, our trends might reflect reporting trends rather than actual condition trends. Since late respondents are less available and/or more reluctant to take the survey, they might be inclined to provide answers which alleviate the survey burden—hence depicting the household health as better than it actually is. Although we cannot rule out such bias, it is unlikely to create our results entirely: (1) trends were dramatic for some conditions, much less so for others, with no rationale for such variations under a ‘declaration bias’ hypothesis; (2) for four of seven health conditions, the lowest frequencies corresponded to rounds 4–5 or rounds 6–7 rather than rounds 8–11. Likewise, our result might reflect trends in undiagnosed diseases, especially since the survey was conducted in an underserved population. This hypothesis, however, is not supported by the trends of sociodemographic variables: late responders were more likely to be employed and to have higher income than their earlier counterparts; health insurance (or absence thereof) was fairly constant from one wave to the next.

Another limitation is the 42.4% non-response rate (29.8% refusals, 12.6% non-interviewed at the 11th wave). Both rates are in the low end of the range of recent health surveys.7 ,8 ,10 ,31 ,32 Yet, if non-respondents and respondents differ systematically by health status, our results might be biased. However, these differences cannot induce a bias for the association between survey effort and health status unless variables on which non-respondents differ from respondents are correlated to both survey effort and health status, and are so in a way that distorts measures that were used to elicit the association.12 ,15 ,22 Although similarities between the sample and the targeted population support minimal bias, such a bias cannot be excluded. Indeed, variables for which we could perform valid comparisons between our sample and the census data were scanty, mainly because our sample was probabilistic at the household level only and collected data mostly at that level, while available census data are essentially based on individual characteristics. For the same reasons, valid weighting procedures to reduce bias could not be applied to our situation.