https://orcid.org/0000-0002-6456-5460
Figures
Abstract
Background
According to the World Health Organization, 600 million cases of foodborne disease occurred in 2010. To inform risk management strategies aimed at reducing this burden, attribution to specific foods is necessary.
Objective
We present attribution estimates for foodborne pathogens (Campylobacter spp., enterotoxigenic Escherichia coli (ETEC), Shiga-toxin producing E. coli, nontyphoidal Salmonella enterica, Cryptosporidium spp., Brucella spp., and Mycobacterium bovis) in three African countries (Burkina Faso, Ethiopia, Rwanda) to support risk assessment and cost-benefit analysis in three projects aimed at increasing safety of beef, dairy, poultry meat and vegetables in these countries.
Methods
We used the same methodology as the World Health Organization, i.e., Structured Expert Judgment according to Cooke’s Classical Model, using three different panels for the three countries. Experts were interviewed remotely and completed calibration questions during the interview without access to any resources. They then completed target questions after the interview, using resources as considered necessary. Expert data were validated using two objective measures, calibration score or statistical accuracy, and information score. Performance-based weights were derived from the two measures to aggregate experts’ distributions into a so-called decision maker. The analysis was made using Excalibur software, and resulting distributions were normalized using Monte Carlo simulation.
Results
Individual experts’ uncertainty assessments resulted in modest statistical accuracy and high information scores, suggesting overconfident assessments. Nevertheless, the optimized item-weighted decision maker was statistically accurate and informative. While there is no evidence that animal pathogenic ETEC strains are infectious to humans, a sizeable proportion of ETEC illness was attributed to animal source foods as experts considered contamination of food products by infected food handlers can occur at any step in the food chain. For all pathogens, a major share of the burden was attributed to food groups of interest. Within food groups, the highest attribution was to products consumed raw, but processed products were also considered important sources of infection.
Conclusions
Cooke’s Classical Model with performance-based weighting provided robust uncertainty estimates of the attribution of foodborne disease in three African countries. Attribution estimates will be combined with country-level estimates of the burden of foodborne disease to inform decision making by national authorities.
Author summary
The World Health Organization has presented estimates of the burden of foodborne disease and attribution percentages of zoonotic foodborne pathogens to food groups (e.g., beef, vegetables). To inform food safety decision making at the national level, these estimates should be combined, extended to pathogens with human reservoirs and provided at a more detailed level (e.g., drinking raw milk as one food product in the dairy group). We present a Structured Expert Elicitation study using the same protocol as the World Health Organization (Cooke’s Classical Model) to attribute foodborne disease of selected pathogens (Campylobacter spp., enterotoxigenic Escherichia coli, Shiga-toxin producing E. coli, nontyphoidal Salmonella enterica, Cryptosporidium spp., Brucella spp., Mycobacterium bovis) to different food types and food products in the beef, dairy, poultry and vegetable groups in three African countries (Burkina Faso, Ethiopia, and Rwanda). Individual experts’ uncertainty assessments resulted in modest statistical accuracy and high information scores, that is concentrated in small ranges, suggesting overconfident assessments. Nevertheless, the optimized item-weighted decision maker was statistically accurate and informative. Cooke’s Classical Model with performance-based weighting provided robust uncertainty estimates of the attribution of foodborne disease in three African countries.
Citation: Sapp AC, Amaya MP, Havelaar AH, Nane GF (2022) Attribution of country level foodborne disease to food group and food types in three African countries: Conclusions from a structured expert judgment study. PLoS Negl Trop Dis 16(9): e0010663. https://doi.org/10.1371/journal.pntd.0010663
Editor: Paul J. Brindley, George Washington University Medical Center, UNITED STATES
Received: January 31, 2022; Accepted: July 14, 2022; Published: September 12, 2022
Copyright: © 2022 Sapp et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data are contained within the manuscript.
Funding: o ACS, MPA and AHH were funded by the UK Foreign, Commonwealth & Development Office (FCDO), the Bill & Melinda Gates Foundation and the CGIAR Research Program on Agriculture for Nutrition and Health led by the International Food Policy Research Institute through a project entitled “Urban food markets in Africa: Incentivizing food safety using a pull-push approach” (OPP1195588, The Pull-Push Project) and a project led by the Ohio State University entitled “The Assessment and Management of Risk from Non-typhoidal Salmonella, Diarrheagenic Escherichia coli and Campylobacter in Raw Beef and Dairy in Ethiopia” (OPP1195643, TARTARE) and the United States Agency for International Development (USAID) Bureau for Food Security under Agreement # AID-OAA-L-15-00003 as part of Feed the Future Innovation Lab for Livestock Systems. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Infections of humans by enteric pathogens can occur through various transmission routes including food, water, air, soil, human-to-human contact, and animal-to-human contact. Due to these various pathways, burden estimates of foodborne disease are challenging. The Foodborne Disease Burden Epidemiology Reference Group (FERG) was established by the World Health Organization (WHO) in 2007 with the aim to develop the first worldwide estimates of the burden of foodborne disease on a sub-regional level. Subregions were based on the official grouping of WHO Member States. FERG further subdivided each region into subregions based on child and adult mortality as described by [1]: very low child and adult mortality (stratum A), low child mortality and very low adult mortality (stratum B), low child mortality and high adult mortality (stratum C), high child and adult mortality (stratum D), and high child mortality and very high adult mortality (stratum E). Burkina Faso was assigned to the Africa Region (AFR), stratum D (AFRD) and Ethiopia and Rwanda to AFR, stratum E (AFRE). FERG has used Structured Expert Judgment according to Cooke’s Classical Model [2] to quantify the relative impact of foodborne disease compared to other transmission routes [3]. Factors considered by the experts included the epidemiology of the foodborne hazards (microorganism or chemical) causing the disease, seasonality, patterns of food consumption, geographic regions, as well as the ecology of the hazards. According to the World Health Organization [4], 600 million cases of foodborne disease were estimated to have occurred globally in 2010. Of these, enteric pathogens accounted for 550 million cases. The African sub-regions D and E were observed to have the highest burden of foodborne disease among all sub-regions analyzed: 13,000 and 12,000 Disability Adjusted Life Years (DALYs) per 100,000 population, respectively.
Global attribution estimates of foodborne disease to specific food groups were first published in 2017 for a limited number of zoonotic pathogens [5]. Such estimates are not yet available for many pathogens that have human reservoirs, such as Shigella spp. and Escherichia coli pathotypes. Building on these estimates, Li et al., estimated the global and sub-regional burden of pathogens in animal source foods [6]. Estimates of disease burden attributed to specific food groups, food types and food products are not yet available at country level.
We present a study on food attribution to support the objectives of three projects aiming to improve food safety in three African countries:
- The Assessment and Management of Risk from Non- typhoidal Salmonella, Diarrheagenic Escherichia coli and Campylobacter in Raw Beef and Dairy in Ethiopia (TARTARE) [7];
- Urban Food Markets in Africa (Pull-Push) [8] and
- Rwanda Enhancement for Enabling Policy Support to the Dairy Sector (RD) [9].
All projects had a similar need for foodborne disease burden data attributed to specific food products to support food safety decision making. To complement attribution estimates available from FERG, we conducted three country-specific Structures Expert Judgment studies, which are reported in this paper. Separate papers will report burden estimates, based on FERG data and the attribution estimates presented elsewhere [10,11].
Methods
Ethical statement
Approvals were obtained from the University of Florida Institutional Review Board (IRB 201901947 -Ethiopia, 202001259-Burkina Faso, 202001265-Rwanda), from the International Livestock Research Institute Institutional Research Ethics Committee (ILRI-IREC2020-06, Ethiopia and Burkina Faso) and from the University of Rwanda College of Medicine and Health Sciences Institutional Review Board (129/CHMS IRB/2020). Written consent was received from all participating experts.
Countries and scope
The three projects consider food safety in Burkina Faso, Ethiopia and Rwanda (Fig 1). Based on infant and adult mortality, Burkina Faso (2017 population 15,605,210) was assigned to the AFRD subregion and Ethiopia (2017 population 106,399,926) and Rwanda (2017 population 11,980,960) to the AFRE sub-region by FERG [12]. Table 1 presents the objectives and scope of the three projects. Different food groups and different hazards were chosen for consideration in these projects.
Hazards (in this context defined as a biological agent which may cause illness in humans) were selected for each country separately based on the goals for the individual projects as shown in Table 1. The selection of foods and hazards to be included was made jointly by project partners while designing the studies and was informed by the results from the global burden of foodborne disease study [4] and specific concerns in the target countries, as voiced by local partners. We designed a hierarchical scheme to classify foods at three levels. The first level was based on FERG and included the food groups “Beef, Ruminants’ meat, Dairy, Poultry, Vegetables, Fruits and nuts, Grains and beans, Oils and sugars, and Other foods”. Guided by the goals of the projects and input from experts in each country, food groups were subdivided in food types. For example, food types in the Dairy category included “Milk from cattle and Milk from other species” while food types in the Vegetables category included “Tomatoes, Leafy greens, Cabbage, Green pepper, Onions, and Other vegetables”. Only food types of interest for the three projects were further subdivided in food products, representing specific consumer products. These products were defined on a country basis as necessary. For example, in Ethiopia, the Milk from cattle food type included the food products “Raw, Fermented traditional (ergo), Cottage cheese traditional (ayib), Heat treated, and Other products” while in Rwanda, this food type included the food products “Raw, Fermented traditional, Fermented industrial, Heat treated and Other products”. Within each category, items were defined to be mutually exclusive and exhaustive (i.e., attribution estimates should sum to 100%). Experts were asked to provide conditional assessments for food group domains. For example, of all cases of foodborne disease due to Campylobacter, what percentage of these cases is attributable to consumption of Red meat? And for all cases attributed to Red meat, what proportion is attributable to Raw meat? For enterotoxigenic Escherichia coli (ETEC), we also elicited attribution to food groups while FERG results [5] were used for other hazards.
Point of attribution
The point of attribution for food groups, food types and food products were based on FERG who have defined this as “the point where the hazards entered the place where the foods are prepared for final consumption” [3]. For example, a person may become ill from eating a fresh green salad that was contaminated by a knife previously used to cut raw chicken. If the chicken had been contaminated with pathogens prior to entering the food preparation area, we would attribute the illness to the chicken, not the lettuce.
Cooke’s classical model
The Classical Model for Structured Expert Judgment is a method to elicit, validate and aggregate expert opinion for uncertainty quantification [2,13]. The experts are asked to quantify their uncertainty about quantities of interest, by providing subjective assessments for percentiles of the distributions. Usually the 5th, 50th and the 95th percentiles, denoting the best estimate (50th percentile), along with a credible interval, provided by the 5th and 95th percentiles. The validation is enabled by calibration questions, whose realizations are not known to experts but are known to analysts. The objective evaluation of experts’ assessments is enabled by a calibration and an information score, which are derived from the calibration questions. The calibration score measures the statistical accuracy of expert’s assessments, that is, relative to expert’s assessments, how discrepant the distribution of the realizations is to the expected relative frequencies of the realizations. We expect 5% of the realizations to fall below expert’s 5th percentiles and 5% of the realizations to fall above the 95th percentiles, and 90% of the realizations to fall between the 5th and the 95th percentiles, with 45% above and 45% below the median. The information score denotes how concentrated expert’s assessments are with respect to a background measure. A combined score, obtained by multiplying the calibration score with the information score is used to evaluate the overall performance of experts and it is also used to compute performance-based weights. Experts’ three elicited quantiles are used to construct so-called minimum informative distributions, that is, distributions which don’t assume any parametric form, nor other information apart from the three quantiles. The Classical Model enables the weighted combination of these distributions using various weights, which result in corresponding Decision Makers (DMs). Performance-based weights which result from experts’ normalized combined score lead to a Performance-Based Decision Maker. Experts’ distributions for the calibration questions can be aggregated as well, the same performance-based weights and therefore the Performance-Based Decision Maker can be evaluated using the calibration and information score, just as for any expert. Various performance-based weights can be considered in aggregating distributions, such as global weights, which are computed using all calibration questions, and item weights, which are derived for each question. Alternatively, optimized Decision Makers search for subsets of experts whose aggregated distributions lead to the best performing Performance-Based Decision Makers in terms of both the calibration and information score. Finally, equal weights can also be considered, and the performance of all resulting Decision Makers can be compared using their calibration and information score.
We note that unlike data-driven models, which rely on large number of observations, expert judgment methods typically employ, due to practical constraints, a limited number of experts. The Classical Model accepts a minimum number of 4 experts and advises for at least 6 experts to participate in a study [13]. A robustness analysis provides insights into how robust DMs performance is with respect to each expert or calibration question.
Expert identification, invitation, selection, and panels
Identification of experts was carried out by the study team for each project within each country. In total, 31 experts were identified in Ethiopia, 27 in Burkina Faso and 23 in Rwanda. Of these, 11, 12 and 9 participated in the study (Fig 2). The identified experts were sent a formal letter of invitation by email to participate in the study and, if interested, complete a Qualtrics (Qualtrics, Provo, UT) survey using a link provided in the invitation letter. The survey consisted of questions pertaining to their working knowledge and experience.
Experts with appropriate domain knowledge were selected by project principal investigators and project managers based upon the survey. Here, appropriate knowledge was defined as expertise in one or more areas of diarrheal disease in humans, zoonoses, microbial food safety, water and sanitation or veterinary public health. Selected experts were sent an email informing them of their selection into the study for their appropriate country, along with biographies of the interviewers. Selected experts were asked to return a signed consent form and were also asked to recommend colleagues not yet invited as potential participants. In addition, experts who served in one of the three studies, and who had working knowledge in one or more other study countries, were invited to the subsequent studies where appropriate.
Of the 31 identified experts in Ethiopia, 18 submitted detailed background information, 17 were selected and enrolled by consent. Of these, 6 experts withdrew consent due to time conflicts or due to COVID-19 when invited for an interview. In total, 11 experts provided estimates in Ethiopia. For Burkina Faso 27 experts were identified, 14 experts submitted detailed background information, 9 experts were selected and enrolled by consent, and 3 experts carried over from the Ethiopia study. Overall, 12 experts provided estimates in Burkina Faso. For the Rwanda study, 23 experts were identified, 13 submitted detailed background information, 7 were selected and enrolled by consent and 2 experts carried over from the Ethiopia study. For Rwanda, 9 experts provided estimates (Fig 2). Experts were not remunerated for their inputs.
Expert’s education is summarized in Table 2. In all countries, veterinary medicine (including veterinary public health) was the most frequent education, followed by biological sciences (including microbiology) in Burkina Faso and Rwanda but not in Ethiopia. Other backgrounds included agricultural sciences, environmental and food sciences, medical doctor, and public health (including epidemiology). All experts in Burkina Faso held PhD degrees, in Ethiopia there were 8 PhDs and in Rwanda 7. In all countries, most experts were affiliated with universities (6, 8, 7 in Burkina Faso, Ethiopia and Rwanda, respectively), followed by research and development institutes (6, 2 and 2). In Ethiopia, 1 expert worked for a government institute.
Expert training
A background document was prepared for each country separately and provided in advance to the experts. This document included a description of the concepts of uncertainty, variability, and probabilities, as well as an explanation of the calibration and target questions, a definition of the point of attribution, published background information and estimates on the epidemiology of the pathogens of interest. In addition, the background document detailed the procedure such as interview duration, agenda items, and an explanation on how to use Zoom software to participate in the interview and how to complete the survey instrument. All experts completed a training video on providing quantitative estimates under uncertainty (Training video for expert elicitation—YouTube).
Expert interviews
Interviews were held via the Zoom platform and had a duration of 60–90 minutes. Interviews covered welcome and introduction to the study and an interactive discussion covering the concept of uncertainty. Subsequently, experts provided assessments for the calibration questions during the interview without access to any resources. Then, interviewers walked through an example target question with the experts. After the interview, experts were allotted two weeks to research and complete answers to the target questions, which were returned by email to the interviewer. Experts were encouraged to use resources as they considered necessary while evaluating target questions.
Calibration questions
All panels included the same series of 10 calibration questions that covered three general themes regarding the continent of Africa. These questions were chosen to reflect the broad domains of the study, i.e., infectious diseases and food and referred to data from Africa, but not from any of the countries to which the study applied. We did not the experts to know any of the answers exactly, but they should be able to produce reasonable estimates based on their general background knowledge. If they were less knowledgeable in a specific domain, we expected them to acknowledge this by providing wider uncertainty intervals. Interviewers presented the questions during the interview and requested answers based solely on current expert domain knowledge. All questions were completed during the interview and answers were sent to the interviewer by email before the interview concluded. Table 3 provides some example questions, while S1 Table provides the full set of calibration questions.
If an expert participated in more than one study, their calibration assessments were re-used for all subsequent studies, reducing the burden to experts for participating in more than one study.
Target questions
Target questions consisted of combinations of hazards with food groups and food types for attribution by country. For each combination, experts were asked to attribute the incidence of foodborne disease in a typical year. Target questions differed between countries, reflecting the food-hazard combinations detailed in Table 1. Estimates were collected by an Excel spreadsheet where each individual tab represented a specific pathogen in one country. An example set of questions and the Excel instrument used to collect the data is shown in Fig 3. The instrument included two quality checks to support the experts to provide valid estimates. For each food group, type or product, the expected sum of medians was provided. The experts were instructed that the sum of the medians should be close to, but not necessarily exactly 100%. An indicator (****) was shown next to each set of uncertainty estimates. The indicator would disappear if all three estimates for a target question were completed and 5th percentile < 50th percentile < 95th percentile. Strictly increasing percentiles are a theoretical requirement of the Classical Model when eliciting continuous distributions. All experts were allotted two weeks to return their target estimates, but exceptions were granted as necessary.
Data analysis
Individual expert assessments were imported into Excalibur software [14] for each country separately and evaluated using their calibration and information score. The theoretical background is described in [13]. Experts’ assessments were combined using performance-based weights, both optimized and non-optimized, along with equal weights. The resulting Decision Makers’ performance was evaluated and the Decision Maker with the highest combined score selected.
We performed an expert and item-wise robustness analysis on the Ethiopia expert results. For expert-wise robustness analysis, each individual expert was removed from the pool of experts and the impact on the resulting item weighted optimized Decision Maker was assessed. Similarly, each calibration question (item) was removed, and the item re-weighted optimized Decision Maker’s performance were evaluated.
The item weighted optimized Decision Maker’s distributions were used to obtain uncertainty estimates for the attribution of foodborne disease in the three countries. The attribution in food groups is based on FERG, while attribution to food type and food products is based on expert judgment elicited for these three studies specifically.
100 percentiles from the Decision Maker’s uncertainty distribution were exported from Excalibur for each target question in each of the three countries. These percentiles were imported in R software version 4.1.0 [15] and 10,000 observations were jointly sampled from each target variable whose distribution is characterized by these percentiles. A normalization procedure was applied to tuples of jointly selected observations. For example, each tuple with joint samples of food groups for a given hazard was normalized to sum to 100%. The procedure ensured that, as appropriate, the resulting sample mean attribution estimates for all hazards in all food groups, types and products summed up to 100% and provided 10,000 normalized samples of the uncertainty distribution. Normalized samples were thus obtained for each corresponding target question. While the estimates for food types or food products were elicited conditional on corresponding food groups or food types, the mean and 95% credible interval of unconditional attribution estimates were calculated for further combination with foodborne disease burden estimates.
Treemaps were constructed using the treemapify package in R [16] to provide a visual summary of the mean attribution results per hazard and per country. For completeness of the treemaps, mean food group attribution results from FERG [5] were included as appropriate.
Results
The calibration and information scores of the item weights optimized Decision Maker are plotted along with individual expert’s scores in Fig 4. Calibration results were relatively similar in all three countries (Fig 4). No single expert met the threshold of a calibration score greater than 0.05, while information scores were quite high in the range of 1 to 3.5. The information score of experts in Burkina Faso was on average lower than in the other two countries. The Optimized Item Weighted Decision Maker performed best in all three countries. In all countries Decision Maker’s outperformed individual experts in terms of calibration score, which did not occur at a loss of much information. In Ethiopia, the Decision Maker’s calibration score was 0.7, displaying a significant improvement in statistical accuracy, with an information score of 1.8. For Burkina Faso, a calibration score of 0.4 and an information score of 1.1 were obtained. For Rwanda a calibration score of 0.7 and an information score of 2.0 were obtained. All Decision Makers were exceeding the threshold of 0.05. Having information scores higher than 1 implies that the Decision Makers were also informative relative to the uniform background.
Calibration score (x-axis) by information score (y-axis) for A) Ethiopia, B) Burkina Faso and C) Rwanda. Black dots depict individual experts in the study while red triangles represent the item weights optimized Decision Maker.
Robustness analysis per expert and question revealed that the results were robust with respect to the pool of experts in each country and with respect to the calibration questions. Removing individual experts or questions led to only slight changes in the performance of the Decision Maker’s.
Tables 4–8 present the attribution estimates per country. About ¾ of the foodborne burden by ETEC in Ethiopia was attributed to the food groups of interest (beef, dairy, poultry, and vegetables) (Table 4). In Burkina Faso, about 40% of the foodborne burden of ETEC was attributed to the food groups of interest (poultry and vegetables) (Table 6). Attribution of ETEC to dairy was lower in Burkina Faso (13%, Table 5) than in Ethiopia (24%, Table 4).
In Ethiopia, about 70 percent of the burden of Campylobacter, Salmonella and Shiga-toxin producing Escherichia coli (STEC) from dairy was attributed to milk from cattle (Table 5) and about ⅓ of the burden of dairy was attributed to consumption of raw cattle milk. About 60% of the burden of these pathogens in beef was attributed to red meat and about 30% to beef consumed raw. For poultry, 100% of the burden was attributed to chicken meat by the study team because meat from other poultry species is consumed very little, if at all. Of the burden of Campylobacter and Salmonella from poultry meat, 56% was attributed to chicken bought raw and prepared at home. A higher fraction of the burden of Salmonella was attributed to tomatoes than of ETEC (0.32, 0.06 respectively) which translated to a higher burden attributed to raw tomatoes (0.22, 0.04 respectively).
The attribution of the burden from vegetables to Salmonella and ETEC in Burkina Faso was similar to Ethiopia (Table 7). However, about half of the burden of Campylobacter and Salmonella from poultry meat was attributed to chicken meat consumed out of the home.
For Rwanda, most of the burden of dairy for all pathogens was attributed to cattle milk (~90 percent, Table 8). Note that the burden of M. bovis was attributed 100% to dairy by the study team as this pathogen is considered to be exclusively transmitted through milk [3,5]. The attribution to raw milk varied by pathogen with the highest proportions attributed for Brucella and Cryptosporidium (61% and 63%, respectively) and the lowest for Campylobacter (0.33). A substantial proportion of the burden from dairy was also attributed to traditionally fermented cattle milk products (~ 20%, Table 8) and even industrially fermented products (0.06–0.20).
More detailed data including those used to create figures are provided in S2 Table.
Tree maps (Figs 5–7) graphically show the results of the experts’ assessments, combined with food group attribution from FERG [5] or this study (for ETEC). Treemaps display hierarchical data as a set of nested rectangles, where the surface area is subdivided according to the proportion of disease attributed to all food groups, food types and food products included in the study. In our case, the surface area of each treemap represents all foodborne disease in a country by a hazard. The surface area is first subdivided according to the attribution to food groups. Food groups are color coded as indicated in the legend to the right of the figure. For example, 68% of all cases of illness by Brucella spp. in Rwanda were attributed to dairy (light goldenrod). Second, the surface area of dairy is further subdivided to represent the contribution of different dairy types, represented by italic font. For example, of the 68% Brucella spp. cases in Rwanda attributed to dairy, 63% were attributed to milk from cattle. Third, the surface area of milk from cattle is subdivided in five dairy products, represented by regular font. Of the 63% of Brucella spp. cases in Rwanda attributed to milk from cattle, 39% were attributed to raw milk consumption.
Discussion
Individual experts in the three countries did not provide statistically accurate assessments, as all calibration scores were lower than the significance level of 0.05. In turn, their assessments were very informative, with half of the experts’ information scores being higher than 2. This suggests that experts in the study were overconfident–they provided narrow uncertainty margins for the calibration questions which did not include the realizations at the expected relative frequency. Despite having a relatively low number of experts providing assessments, the item-weighted optimized Decision Maker’s in all three countries were statistically accurate. Moreover, the significantly improved statistical accuracy did not come at the cost of low informativeness as all Decision Maker’s information scores were above 1. These results show the power of performance-based weighting and suggest that the range of estimates provided by the experts as a group did represent the realizations to the calibration questions at the expected relative frequencies.
Even though dedicated training sessions were offered to the experts before the interviews, these apparently were not sufficient to prevent overconfident assessments. These results are in contrast with an earlier study in the US, where a large proportion of the experts individually produced well calibrated assessments [17]. An important difference between these two studies was that the US study was performed as an in-person meeting, while the current study was conducted through remote communication. Nonetheless, it needs to be emphasized that empirical evidence suggest that a large proportion of experts do not provide well calibrated assessments. An analysis on 322 experts [13] showed that only 27% of the experts’ assessments in 74 studies were calibrated, using a 0.05 threshold. By contrast, all experts in the three studies presented in this paper provided assessments leading to calibration scores smaller than 0.05. Finally, we emphasize that experts’ ability to provide valid assessments under uncertainty does not appear to correlate with years of experience or domain knowledge [18].
It is notable that the uncertainty in some attribution estimates is rather small, for example attribution of ETEC transmission to grains and beans in Ethiopia, or Campylobacter transmission to cooked red meat, also in Ethiopia. For these products, experts largely agreed on the attribution estimates. The uncertainty in other attribution estimates was higher, independent of a relatively low or high mean attribution estimate. For example, the expected attribution of milk from cattle due to Campylobacter was relatively high at 70%, and the 95% uncertainty interval ranged between 6% and 100%. On the other hand, the attribution estimates for offal due to Salmonella were rather low (36%), with an attendant large uncertainty interval ranging between 3% and 92%.
Attribution results for ETEC to food groups in Ethiopia and Burkina Faso were largely similar, even though only 2 experts provided estimates for both countries. Experts were explicitly instructed that there is no evidence that animal pathogenic ETEC strains are infectious to humans [19], but nevertheless in both countries, a sizeable proportion of ETEC illness was attributed to animal source foods. We asked the experts for their rationale behind this attribution, and they considered contamination of food products by infected food handlers can occur at any step in the food chain and may contribute to transmission of pathogens with human-only reservoirs by animal source foods. Likewise, attribution results for poultry meat and vegetables that were done in both countries were largely similar. A noteworthy exception is a higher attribution of poultry meat related illness to chicken meat consumed out of the home in Burkina Faso (~ 50%) vs. Ethiopia (~ 20%). This corresponds to a difference in the contribution of street food to chicken consumption in these countries [20,21].
To support decision making by national authorities, the attribution estimates presented in this study will be combined with national level estimates of the burden of foodborne disease from WHO FERG to present estimates of the burden of specific food groups and food products [10,11]. These data will inform priority setting of foods for control activities and can be combined with risk assessment and economic estimates to inform cost-benefit analyses.
Supporting information
S1 Table. Calibration questions used in the three African countries burden of foodborne disease source attribution expert elicitation.
https://doi.org/10.1371/journal.pntd.0010663.s001
(DOCX)
Acknowledgments
We acknowledge the inputs from experts in providing attribution estimates.
Under the grant conditions of the Bill & Melinda Gates Foundation, a Creative Commons Attribution 4.0 Generic License has already been assigned to the Author Accepted Manuscript version that might arise from this submission. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors alone.
References
- 1. Ezzati M, Lopez AD, Rodgers A, Hoorn SV, Murray CJ. Selected major risk factors and global and regional burden of disease. The Lancet. 2002;360: 1347–1360. pmid:12423980
- 2.
Cooke RM. Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford, New York: Oxford University Press; 1992.
- 3. Hald T, Aspinall W, Devleesschauwer B, Cooke R, Corrigan T, Havelaar AH, et al. World Health Organization Estimates of the Relative Contributions of Food to the Burden of Disease Due to Selected Foodborne Hazards: A Structured Expert Elicitation. PLOS ONE. 2016;11: e0145839. pmid:26784029
- 4. Havelaar AH, Kirk MD, Torgerson PR, Gibb HJ, Hald T, Lake RJ, et al. World Health Organization Global Estimates and Regional Comparisons of the Burden of Foodborne Disease in 2010. PLOS Med. 2015;12: e1001923. pmid:26633896
- 5. Hoffmann S, Devleesschauwer B, Aspinall W, Cooke R, Corrigan T, Havelaar A, et al. Attribution of global foodborne disease to specific foods: Findings from a World Health Organization structured expert elicitation. PLOS ONE. 2017;12: e0183641. pmid:28910293
- 6. Li M, Havelaar AH, Hoffmann S, Hald T, Kirk MD, Torgerson PR, et al. Global disease burden of pathogens in animal source foods, 2010. PLOS ONE. 2019;14: e0216545. pmid:31170162
- 7.
TARTARE: The Assessment and Management of Risk from Non-typhoidal Salmonella, Diarrheagenic Escherichia coli and Campylobacter in Raw Beef and Dairy in Ethiopia | CFI. [cited 3 Jan 2022]. Available: https://foodsafety.osu.edu/research/international-research-shifting-food-safety-paradigm/tartare
- 8.
Urban food markets in Africa: Incentivizing food safety using a pull-push approach | International Livestock Research Institute. [cited 3 Jan 2022]. Available: https://www.ilri.org/research/projects/urban-food-markets-africa-incentivizing-food-safety-using-pull-push-approach
- 9.
ENHANCE Gordon—Feed the Future Innovation Lab for Livestock Systems—University of Florida, Institute of Food and Agricultural Sciences—UF/IFAS. [cited 3 Jan 2022]. Available: https://livestocklab.ifas.ufl.edu/projects/enhance-gordon/
- 10. Havelaar AH, Sapp AC, Amaya MP, Grace D, Knight-Jones T, Morgan KM, et al. Burden of foodborne disease due to bacterial hazards associated with beef, dairy, poultry meat and vegetables in Ethiopia and Burkina Faso, 2017. In review.
- 11. Sapp AC, Nane GF, Amaya MP, Niyonzima E, Hatekigimana J-P, VanSickle JJ, et al. Estimates of disease burden caused by foodborne pathogens in contaminated dairy products in Rwanda, 2010. In review.
- 12. Gibb H, Devleesschauwer B, Bolger PM, Wu F, Ezendam J, Cliff J, et al. World Health Organization estimates of the global and regional disease burden of four foodborne chemical toxins, 2010: A data synthesis. F1000Research. 2015;4: 1–11. pmid:26918123
- 13.
Hanea AM, Nane GF. An In-Depth Perspective on the Classical Model. In: Hanea AM, Nane GF, Bedford T, French S, editors. Expert Judgement in Risk and Decision Analysis. Cham: Springer; 2021. pp. 225–256. https://doi.org/10.1007/978-3-030-46474-5_10
- 14.
Cooke RM. Excalibur. In: TU Delft [Internet]. [cited 2 Jan 2022]. Available: https://www.tudelft.nl/en/eemcs/the-faculty/departments/applied-mathematics/applied-probability/research/research-themes/risk/software/excalibur/excalibur
- 15.
R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2021. Available: https://www.R-project.org/
- 16. Wilkins D. treemapify: Draw Treemaps in “ggplot2.” 2021. Available: https://CRAN.R-project.org/package=treemapify
- 17. Beshearse E, Bruce BB, Nane GF, Cooke RM, Aspinall W, Hald T, et al. Attribution of Illnesses Transmitted by Food and Water to Comprehensive Transmission Pathways Using Structured Expert Judgment, United States—Volume 27, Number 1—January 2021—Emerging Infectious Diseases journal—CDC. pmid:33350907
- 18. Burgman MA, McBride M, Ashton R, Speirs-Bridge A, Flander L, Wintle B, et al. Expert Status and Performance. PLOS ONE. 2011;6: e22998. pmid:21829574
- 19. Delahoy MJ, Wodnik B, McAliley L, Penakalapati G, Swarthout J, Freeman MC, et al. Pathogens transmitted in animal feces in low- and middle-income countries. Int J Hyg Environ Health. 2018;221: 661–676. pmid:29729998
- 20.
Amenu K, Bedasa M, Wamile M, Worku H, Kasim K, Taha M, et al. Qualitative assessment of chicken and vegetable value chains in Harar and Dire Dawa, Ethiopia: Food safety perspectives. Nairobi, Kenya: International Livestock Research Institute; 2021. Report No.: ILRI Research Report 82. Available: https://cgspace.cgiar.org/handle/10568/116667
- 21.
Value chain assessment of animal source foods and vegetables in Ouagadougou, Burkina Faso: Food safety, quality and hygiene perceptions and practices. In: International Livestock Research Institute [Internet]. 16 Nov 2020 [cited 3 Jan 2022]. Available: https://www.ilri.org/publications/value-chain-assessment-animal-source-foods-and-vegetables-ouagadougou-burkina-faso-food