2.1 Ethics statement

Ethics approval was granted by the Child and Adolescent Health Service Human Research Ethics Committee (RGS2477) at Perth Children’s Hospital. Written informed consent was obtained from the parent or legal guardian of each participant.

A BN model consists of two components, 1) a DAG that qualitatively describes how variables (nodes) interact with each other, and 2) a joint conditional probability distribution that quantitatively specifies how changes in each parent variable probabilistically drive changes in their child variables. The arrows (arcs) of the DAG indicate the presence of a direct influence of predecessor (or parent) variables on their child nodes (nodes extending from other nodes). A causal BN is one in which the arcs represent influences that are causal; sometimes with minor exceptions for associations that are required, but are not a core part of the causal process being modelled. Developing a causal BN involves constructing the model structure (the DAG) that describes the problem domain of interest, and quantifying the probabilistic effect that parent variables have on their child variables as a series of conditional probability tables (CPTs), i.e., parameterisation. In this project we used domain expert knowledge and data in combination and iteratively, to build a causal BN to predict causative pathogens for childhood pneumonia, aiming to distinguish bacterial from non-bacterial cases for which the former is more likely to benefit from antibiotic use. Fig 1 shows the schematic of the BN development process.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The schematic of BN development.

The top blue row presents the development of the BN from its initial form (left) to the fourth iteration (right), and how this iteratively interacts with the data analysis (middle grey row) and the knowledge elicitation (KE, bottom green row). Everything is primarily designed to support model development; however the models themselves also support and drive data analysis and KE. In some cases, there are also direct interactions between data analysis and KE. During the KE process, survey outcomes often directly guide the design and focus of the following workshop. As the BN is developed and revised over several iterations, any data analysis, KE or modelling outcomes from an earlier iteration may inform any of these activities in the next iteration.

https://doi.org/10.1371/journal.pcbi.1010967.g001

2.2 Knowledge elicitation

Expert knowledge elicitation occurred through a series of group workshops, surveys and one-on-one meetings involving 6–8 domain experts (with backgrounds in infectious disease, clinical microbiology, emergency medicine, general paediatrics, immunology and epidemiology). The theme of the workshop and survey was often driven by the evolving modelling progress—questions arising from data analysis, development of model structure and parameterisation. In total, as shown in Fig 1, four workshops were conducted covering the development of the model structure, data interpretation, model structure validation, and validation of the application use cases. Five surveys were distributed to inform variable selection, model validation via clinical use cases, and elicitation of key model parameters. One-on-one meetings were organised when more detailed discussion was considered necessary by the modellers, such as to consolidate understanding of a particular topic discussed during a workshop. In S1 File, we provide the detailed description of the model development process (Section A, S1 File) associated with a full list of survey questions (Section B—F, S1 File).

2.3 BN structure and parameterisation

The final model structure was achieved through an iterative process of synthesising expert knowledge, data and learnings from earlier BN revisions. Parameterisaton of the model was conducted using the expectation maximisation (EM) algorithm [36], with one of the following three methods applied to each node:

1. A data-driven approach was used to derive CPTs directly from the observed distribution of relevant variables in the data, when data for those variables was sufficient.
2. An expert-driven approach was used for conceptual variables that were not or could not be recorded in the data. In this approach, the CPT was fully specified by domain expert knowledge alone and not further updated with data. For example, susceptibility to progression describes the extent to which an infected child is at risk of progressing from mild to severe manifestations of pneumonia. In the model, this susceptibility is higher in younger children, in those with chronic respiratory disease, and in those who are immunocompromised.
3. A hybrid expert-data approach was used in which expert-driven CPTs were used as a Bayesian prior (i.e., a starting point) assigned a weight of one data point and subsequently updated using available data. This method was preferred for variables that were strongly associated with other data, but could not be directly observed or mapped to any individual variable (i.e., latent), as well as for observable variables where data were sparse for certain patient subgroups. Note, the priors used can be accessed via the source model files on Open Science Framework (https://osf.io/m97vb/).

The parameterisation method for each variable was chosen based the definition of each variable and how well we were able to quantify them based on available data and/or expert knowledge. A description of the parameterisation method used for each variable is provided in the BN dictionary (S1 Table).

2.4 BN performance and evaluation

We evaluated the performance of the BN in two ways. Quantitatively, numerical evaluation ensured the model predictions were consistent with the observed data. While metrics that predict accuracy such as area under the receiver operating characteristic curve (AUROC) have been widely used, these measures are more suitable for definite classifications. Log loss (and similar probability-sensitive metrics) is considered more informative for evaluating models that generate probabilistic predictions [37] like BNs, because it rewards accurate probability estimates most highly. Log loss is calculated, for a given case, as the negative log of the model’s estimated probability of what was actually observed (i.e., −log{P(observed outcome)}) with a lower log loss indicating a better fit of the model to the observed data. To quantitatively evaluate the model predictions from a clinical decision perspective, we also presented model predictions in terms of false positives (FP), false negatives (FN), their associated rates (FPR/FNR), sensitivity and specificity under different FPR/FNR trade-off preferences.

Sensitivity analyses were conducted using variance-based sensitivity analysis (VBSA) [38, 39], to investigate how the target variable of the model is influenced by varying the assumptions for a range of key variables, especially those affected by a high degree of uncertainty around data or domain expert knowledge. More specifically, we simultaneously vary the parameters for the CPTs of a group of variables (by up to 20%) and observe how the distribution of a target variable state spreads given the variation in those parameters. Essentially, the VBSA allows us to investigate the impact on the target of the second order uncertainty that may exist in the CPT parameters (without incorporating that second order uncertainty directly into the model).

While acknowledging that overfitting cannot be properly determined until we can assess the model using external datasets, we can still gain some understanding of where overfitting may occur by identifying which part of the model is fitted most strongly to our cohort. We have adopted an approach that compares CPTs between the trained model and prior model using Kullback–Leibler (KL) divergence [40].

The approaches above are unfortunately of limited value in evaluating the causal features of a model. For example, it is common for two BNs to differ in their causal features but be identical in their statistical properties [29, 41]; such models would be assessed as equivalently good by any of the above metrics. More commonly, major differences in causal features may only lead to minor differences in predictive performance. Thus, while predictive performance can help identify significant issues (such as major missing pathways or dependencies), other approaches must be used to evaluate the individual causal features of a BN. This can be achieved in a number of ways (e.g., comparing to literature, conducting experiments, etc.); here we performed this validation with experts, by walking through the DAG structure afresh at each workshop, and by soliciting feedback on the DAG structure via followup emails. In addition, we performed qualitative expert validation of the model outputs, allowing us to assess if the model behaves in a way that meets clinician expectations and offers clinically informative predictions. We simulated three typical scenarios encountered in clinical practice, and presented BN predictions in both survey format (Survey 4) and during workshops (Workshops 3 and 4) for expert review.

We used the GeNIe BN (https://www.bayesfusion.com/) software to elicit the BNs presented here, and Netica (https://www.norsys.com/) to develop and parameterise subsequent models. Data, evaluation and sensitivity analyses were conducted using R [42] and Python [43]. Throughout the paper, we label the name of a BN variable in bold, and its state name in italics where referenced for the first time. Source models, raw evaluation results and python scripts for running the analysis can be accessed via Open Science Framework (https://osf.io/m97vb/).

3.1 Summary of model variables

The model consists of 62 variables including 14 which are background factors, six pathogen-specific variables describing the presence of pathogens in the nasopharynx, eight variables representing infection or the clinical diagnosis of infection, 25 variables representing specific signs and symptoms, five laboratory variables, and four intervention variables. In the BN dictionary, we provide information on each model variable, including identifiers (ID and variable name), what it means and how it’s mapped to data (description), how it’s discretised in the BN (states), its parents in the BN, and how it’s affected (mechanistically rather than statistically) by those parent nodes. The dictionary also notes the observational status (whether observable, latent or derived), and the parameterisation method used (data-driven, expert-driven or hybrid expert-data) for each variable. Fig 3 shows a sample of the BN dictionary; the full dictionary is provided in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Extract from the BN dictionary.

The full dictionary is provided in S1 Table.

https://doi.org/10.1371/journal.pcbi.1010967.g003

Alongside the 52 observable variables, nine latent variables were created for modelling purposes, including three concept variables that describe the epidemiological context (Fig 4, light blue nodes (12)—(14)) and six infection-relevant variables that describe the causative pathogens and the sites of the respiratory tract involved (Fig 5, white nodes (21)—(26)). These latent variables help to explicitly describe important underlying mechanisms that cannot be directly observed and thus captured by data. With respect to the choice of parameterisation method, 16 variables were exclusively trained from data (data-driven), 7 were exclusively parameterised based on expert knowledge (expert-driven), and 39 nodes were trained via the hybrid expert-data approach. One variable (current clinical phenotype (52)) was introduced as a summary of the presenting clinical phenotypes of pneumonia. This variable was derived based on a separate clustering using the EM algorithm to determine a child’s most probable pneumonia phenotype based on their clinical observations. The analysis resulted in two phenotypes, where phenotype 1 represented a more severe type which increased the probability of all observable signs and symptoms compared to phenotype 2, and this information was added as an additional column in the cohort data and subsequently used as an observed variable. We discuss the role of this variable further below §3.2.3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The epidemiological context, BN v4.5.

Due to the number of arcs emanating from Age group, a lighter shade of blue has been used to make the graph easier to read. The detailed model structure and definition for each variable are provided in S1 Table. Source model files can be accessed via Open Science Framework (https://osf.io/m97vb/).

https://doi.org/10.1371/journal.pcbi.1010967.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The infection, BN v4.5.

The detailed model structure and definition for each variable are provided in S1 Table. Source model files can be accessed via Open Science Framework (https://osf.io/m97vb/).

https://doi.org/10.1371/journal.pcbi.1010967.g005

3.2 Model description

3.2.3 The evidence.

From a clinical perspective, a child’s causative pathogen for pneumonia (25), current clinical phenotype (52) and upper airway involvement (26) are not directly observed (i.e., they are latent), and can only be inferred from clinical observations and the results of laboratory investigations. From a causal perspective, these latent variables interact to give rise to the various types of evidence including clinical symptoms, signs, and laboratory results (Fig 6). The potential causes of signs, symptoms, and laboratory results are rarely deterministic, singular or mutually exclusive. For example, cough (29) can be driven by upper airway irritation, inflammation, or mucus production (e.g., postnasal drip secondary to rhinorrhoea/rhinitis) or can be from pneumonia affecting the lower airways. Similarly, increased respiratory rate (53) can be a consequence of airway inflammation or low oxygen saturation (58) and can vary by the child’s age.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The evidence, BN v4.5.

Causative pathogen for pneumonia, current clinical phenotype and upper airway involvement interact to give rise to the various types of evidence including clinical signs, symptoms, and laboratory results. The BN dictionary (S1 Table) details how these latent variables may give rise to each type of evidence. Source model files can be accessed via Open Science Framework (https://osf.io/m97vb/).

https://doi.org/10.1371/journal.pcbi.1010967.g006

Background (dark blue) and intervention (purple) factors may also influence certain types of evidence. For example, age is an important determinant of the report of pain (e.g., headache (30), abdominal pain (47) and sore throat (32)) because children need to be old enough to communicate these symptoms. Recent antibiotic exposure (60) may reduce the probability of a successful blood culture (blood culture result (53)) even if a child is bacteremic. Knowing the type of oxygen (62) (supplemental or room air) is essential for interpreting oxygen saturation in relation to lung function. Symptoms and signs may also interact, e.g., vomiting (39) and diarrhoea (40) may lead to reduced oral intake (41) which could subsequently lead to lethargy (42).

The classification of the current clinical phenotype of pneumonia is analogous to a count-based metric for disease severity, where those with phenotype 1 had a median of 12 pneumonia-relevant signs and symptoms, compared to a median of 8 for phenotype 2. The inclusion of this variable allowed us to separate the differential influence of causative pathogen (typical bacterial or viral-like) from the severity of clinical manifestations conditional on the causative pathogen. For example, the overall probability of a child being phenotype 1 (severe) was almost identical for pneumonia involving viruses and viral-like pathogens (14.0%) and those involving typical bacterial pathogens (13.6%). For a child who had symptom onset within two days and been transferred from another hospital, the probability of phenotype 1 was much higher for viral-like pneumonia (79.4%) than typical bacterial case (42.6%). In contrast, if the child presented to ED directly (not transferred), the overall probability of phenotype 1 is lower (15.5%), however, typical bacterial pneumonia now has a more severe phenotype (23.8%) than their viral-like counterparts (12.1%).

3.3 Model evaluation and sensitivity analyses

A child’s causative pathogen for pneumonia (25) is our ideal target for prediction because it would respond (or not respond) to antibiotic use and subsequently drive the progression of disease. While the model can predict this variable, it is latent, making it challenging to evaluate using observational data alone. Hence, we instead assess the performance of the BN using the recorded clinically-confirmed bacterial pneumonia (28) as the best estimate of the causative pathogen type, acknowledging that some of these diagnoses may be incorrect. So long as the number of incorrect diagnoses is not great (less than 5% in BN v4.5 would certainly be acceptable), this will not affect our conclusions. More specifically, the assumption that node 28 is a good surrogate would not hold if ‘viral-like pneumonia (node 25)’ can also probabilistically lead to pleural effusion or positive blood culture, and in BN v4.5, these probabilities are very low: 3% and 0% respectively. Fig 7 presents the BN predicted probabilities of clinically-confirmed bacterial pneumonia for every pneumonia episode for children in the cohort, and compares these against their final diagnosis. The graphs represent four input scenarios (a-d), each providing more information to the model than its preceding scenario, which approximately replicate how information may chronologically become available in clinical practice, serving also as examples of potential time points at which the BN can be applied for decision support:

1. (a) provides the model with available basic background factors of the partipants; these can include age group, childcare days per week, chronic respiratory disease, ethnicity, influenza season, prematurity, smoker in the household, impaired immunity, influenza vaccine, pneumococcal vaccine, and previous significant infection.
2. (b) uses the information in (a) plus signs, symptoms and interventions if reported; these can include duration of symptom/s onset, rhinorrhoea, cough, respiratory rate, oxygen saturation, breathing difficulty, crackles, wheezing, (parent-reported) fever, highest temperature, chill, sweats, irritability, vomiting, diarrhoea, reduced oral intake, lethargy, chest pain, abdominal pain, other pain, earache, sore throat, headache, rash, (prior) antibiotic exposure, type of oxygen, hospital transfer, and (decision to) perform blood culture.
3. (c) uses information in (b) plus any detection (or no detection) of respiratory pathogens in the nasopharynx; these can include RSV, influenza, HMPV, parainfluenza, mycoplasma, and any typical bacteria.
4. (d) includes (c) plus all other available results; these can include C-reactive proteins (CRP), white cell count (WCC), and neutrophil proportion (of total WCC).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The performance of BN v4.5.

Predicted probabilities of clinically-confirmed bacterial pneumonia are compared with the final observations (orange if yes) for four input scenarios, with mean log loss and AUROC based on 10-fold cross-validation. For each scenario, available information from each participant was entered as input for the BN to predict the clinically-confirmed bacterial pneumonia.

https://doi.org/10.1371/journal.pcbi.1010967.g007

Graphs that show a strong overlap (such as in Fig 7a) indicate the model is only able to achieve weak discrimination between bacterial diagnoses and others. However, as further information is provided (through scenarios (b), (c) and (d)), the discrimination improves, as evidenced by both the graphs and the improving AUROC scores. In contrast, the log score does not improve over the first 3 scenarios; coupled with the improving AUROC, this suggests the initial probabilistic prediction is already quite close to the true class. Log loss does improve in the final scenario, as does AUROC again and the greatest discrimination is visible in this graph. Table 1 additionally presents the evaluation metrics for other observable targets that are also close surrogates of the (latent) causative pathogen for pneumonia (the primary target for prediction), including pleural effusion, positive blood culture result, and the detection of viruses or mycoplasma in the nasopharynx. As expected we see improved performance in predicting these variables as more observations are entered in the input scenarios, confirming that the model is likely to behave in a reasonable way when predicting the primary target.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The performance of BN v4.5 in predicting a range of evaluation targets under different input scenarios, in the form of mean log loss and AUROC based on 10-fold cross-validation.

Note when the prediction targets themselves were included as observed variables in the corresponding input scenario, we removed them from the list of input variables. For example, no prediction about RSV in nasopharynx under input scenario (c) is made (and hence shown as NA) because RSV status is already known in that scenario. Raw results and scripts for generating these metrics can be accessed via Open Science Framework (https://osf.io/m97vb/).

https://doi.org/10.1371/journal.pcbi.1010967.t001

From a clinical decision-making perspective, a model needs to provide predictions that trade-off false positive and false negative rates (FPR and FNR, respectively). More specifically, such a trade-off needs to be defined under a particular decision-making context or, in other words, alternative FPR/FNR trade-off preferences. Table 2 shows how the desirable threshold in predicting a target variable may vary greatly under different input scenarios and different FPR/FNR trade-off preferences—i.e., different relative weightings of the consequences of false positive versus false negative predictions. For example, when FPR and FNR are weighted equally, lower thresholds help to simultaneously minimise FPR and FNR when more information becomes available for input scenarios (c) and (d). Under the same input scenarios, the preferred threshold decreases further as higher weight is assigned to FNR.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The optimal performance of the BN in predicting clinically-confirmed bacterial pneumonia (28) under different input scenarios and different FPR/FNR trade-off preferences (P1-P4).

Specifically, we defined optimal based on a minimised weighted sum of FPR and FNR across the 10-fold cross-validation results. For each input scenario and FPR/FNR trade-off preference, we present here the threshold used when the optimal performance is achieved, and its associated false positives (FP), false negatives (FN), sensitivity and specificity. Note that there is a total of 25 positive observations of clinically-confirmed bacterial pneumonia in our cohort of 230 cases of X-ray confirmed pneumonia. Raw results and scripts for generating these metrics can be accessed via Open Science Framework (https://osf.io/m97vb/).

https://doi.org/10.1371/journal.pcbi.1010967.t002

We varied relevant parameters by 20% to investigate how various factors affect the predicted probability of typical bacterial pneumonia (24) which is determininistically related to the bacterial state of the causative pathogen for pneumonia (25) (Fig 8). Three groups of variables were investigated, and the target prediction is most sensitive to parameters describing the pathogenicity of the potential pathogens in causing pneumonia (white), ranging from 12% to 50%. Varying the prevalence of various pathogens in the nasopharynx caused the predicted probability of the involvement of typical bacteria to vary from 22% to 30% (yellow). Varying the assumptions regarding the latent epidemiological concepts (e.g., level of exposure) had a small influence on the probability of involvement of typical bacteria (25% to 27%) (blue).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Sensitivity analysis, BN v4.5.

Relevant parameters were varied by 20% to investigate how they affected the predicted probability of involvement of typical bacteria. This target prediction was most sensitive to the pathogenicity of potential pathogens in causing pneumonia (white), followed by the prevalence of various pathogens in nasopharynx (yellow), and was least sensitive to the latent epidemiological concepts (blue). CPTs that were varied in the pathogenicity group include those for both viral-like pneumonia (23) and typical bacterial pneumonia (24). The pathogens in nasopharynx group involve CPTs for RSV in nasopharynx (16), Influenza in nasopharynx (15), HMPV in nasopharynx (17), Parainfluenza in nasopharynx (18), Mycoplasma in nasopharynx (19) and bacteria in nasopharynx (20). And the latent concepts group involved CPTs for level of exposure (12), susceptibility to colonisation (13) and susceptibility to progression (14).

https://doi.org/10.1371/journal.pcbi.1010967.g008

The use of KL divergence analysis (S2 Table) allowed us to highlight the discrepancies between prior and posterior CPTs in particular among the hybrid expert-data variables; i.e., which expert-informed CPTs were shifted the most by the data. We noticed that both viral-like pneumonia (23) and typical bacterial pneumonia (24) rank highly among the hybrid expert-data variables. As an interesting example, given the presence of a nasopharynx infection, but negative detections for all modelled viral-like pathogens in the nasopharynx, the probability of a viral pneumonia still shifts from 11% to 98% (keeping in mind the model is being asked to predict a pneumonia of some sort). This implies a potentially more significant role for other respiratory viruses among children with X-ray confirmed pneumonia.

3.4 Demonstrative clinical scenarios

When serving as a clinical decision support tool, the available background factors, clinical observations and investigation results would be entered as model inputs for making inferences about causative pathogen for pneumonia (25) in a particular episode. We presented three scenarios to demonstrate the potential use of this model in a clinical context Figs (9)–(11). Proposed and reviewed by domain experts, the selected variables in these scenarios are not those you would specifically measure or look for when suspecting bacterial pneumonia—rather this is information commonly present when clinicians need to make a diagnosis of bacterial pneumonia and decide on interventions such as antibiotic use.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Clinical scenario 1, BN v4.5.

This use case presents an infant (age group (1)) with X-ray confirmed pneumonia, moderately high temperature (highest temperature (37)), cough (29) and runny nose (rhinorrhoea (31)). The BN predicts a low probability of typical bacterial pneumonia for this child (causative pathogen for pneumonia (25)), and a reported history of diarrhoea (40) alone does not largely affect the prediction. The target prediction remains low unless both the C-reactive proteins (55) and white cells count (56) are moderately elevated.

https://doi.org/10.1371/journal.pcbi.1010967.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Clinical scenario 2, BN v4.5.

This scenario presents a school-aged (age group (1)) child with X-ray confirmed pneumonia and with moderately high temperature (highest temperature (37)), runny nose (rhinorrhoea (31)) and high respiratory rate (53). The BN predicts a 22% probability of typical bacterial pneumonia for this child (causative pathogen for pneumonia (25)), and this prediction increases to 42% if the child has no breathing difficulty (49). Moderate and very high C-reactive proteins (55) and white cells count (56) further increase the probability of a typical bacterial cause for this pneumonia.

https://doi.org/10.1371/journal.pcbi.1010967.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Clinical scenario 3, BN v4.5.

This scenario shows a pre-school aged (age group (1)) child with X-ray confirmed pneumonia and very high temperature (highest temperature (37)) during the influenza season (9). The probability of typical bacterial pneumonia (causative pathogen for pneumonia (25)) increases if the child is under-vaccinated for pneumococcus (Pneumococcal vaccine (10)), regardless of whether RSV is detected from their nasopharynx or not (RSV in nasopharynx (16)). If fully-vaccinated, the presence of wheezing (43) will lead to a decrease in the predicted probability of typical bacterial pneumonia, and this prediction will drop to only 4% if RSV is detected in the child’s nasopharynx. Other variables involved in this scenario are crackles (44), vomiting (39) and diarrhoea (40).

https://doi.org/10.1371/journal.pcbi.1010967.g011

The BN predicted probability of typical bacterial pneumonia is 28% for the cohort of children with X-ray confirmed pneumonia (labelled in the blue box at the top right corner). For each scenario, this prediction is updated, branching in multiple possible ways that are conditional on information that may become available over time along the child’s clinical course and thus act as potential time points when the BN can be applied to make predictions for decision support. In parentheses we provide the change in the target prediction in relation to the cohort average; we use red text to indicate if the predicted probability of typical bacterial pneumonia is higher than the cohort average, and use green text to indicate predicted target probabilities which are at least 50% lower than the cohort average.

Fig 9 presents a scenario in which the BN predicts a low probability (13%) of a typical bacterial cause of pneumonia for a 4 month old infant with X-ray confirmed pneumonia with moderate signs and symptoms (i.e., highest temperature of 38.5 degrees Celcius, runny nose and cough). The predicted probability remains low unless both the CRP and WCC are high (at least 100 mg/L and 12 × 10⁹/L, respectively). In contrast, if instead a 9 year old (school-aged) child with X-ray confirmed pneumonia presents various signs and symptoms (Fig 10), the probability of typical bacterial pneumonia is predicted to be higher and similar to the cohort average. The predicted probability of typical bacterial pneumonia is almost doubled if the child has no breathing difficulty, and a moderately high CRP (40 mg/L) can be indicative for a bacterial cause. In other words, the significance of a raised CRP varies under various scenarios. The third scenario (Fig 11) presents a 3.5 year old (pre-school aged) child with X-ray confirmed pneumonia and very high temperature during the influenza season. The probability of typical bacterial pneumonia increases if the child is under-vaccinated for pneumococcus, regardless of whether RSV is detected in their nasopharynx. If fully-vaccinated, the presence of wheezing will lead to a decrease in the predicted probability of typical bacterial pneumonia, and this prediction will drop to only 4% if RSV is detected in the child’s nasopharynx.

4.1 Using latent concepts to reveal the epidemiological context

When a child has X-ray confirmed pneumonia, the causative pathogen for pneumonia usually remains unobserved, i.e., latent. Patient background factors such as basic demographics, seasonality, vaccination and comorbidities can help predict the latent causative pathogen because they often causally influence the presence of a respiratory pathogen in the child’s nasopharynx or respiratory tract which in turn predisposes to pneumonia. For example, RSV is more prevalent in the winter in temperate environments, owing to its elevated circulation in the community, and bacterial infection is more probable in a younger child due to the interplay between their developing immunity and respiratory microbiota [44, 45]. The difficulties faced by clinicians arises from the dynamic and complex interactions amongst these factors, as we saw in Fig 11 for a young child who presents to hospital in the influenza season.

We addressed this challenge by firstly organising relevant variables under a causal framework (Fig 4), and introducing several latent concepts such as the level of exposure and the susceptibility of a child to colonisation. This approach helped explain the potentially complex role of each variable in the problem domain. For example, older children may have lower susceptibility to bacterial colonisation as well as to pneumonia progression, with both leading to a reduced probability of typical bacterial pneumonia [45]. However, they are also more likely to have a high exposure through attendance at childcare or school [46], leading to an increase in the nasopharyngeal presence of all pathogens, thereby increasing the potential for subsequent infection. Being explicit about these underlying mechanisms allowed us to capture and understand the role of each factor in the context of each pneumonia episode, which in turn gave us further insight into the epidemiological context and more precise predictions of the causative pathogen for pneumonia.

4.2 Mapping clinical and laboratory evidence to explicit causal paths

Model predictions can be further improved by utilising additional information which is likely to be available shortly after presentation, namely, clinical observations and laboratory investigation results. By modelling the overlap and divergence in the pathophysiological pathways indicated by each piece of clinical and laboratory evidence (Fig 6), the causal BN approach can estimate the episode-specific probability of each causative pathogen based on any input variables available. As illustrated in clinical scenario 1 (Fig 9), while diarrhoea can be a feature of either viral or bacterial pneumonia, knowing about diarrhoea alone does not greatly affect the predicted probability of typical bacterial pneumonia. However, when additional information such as CRP and WCC becomes available, the prediction can vary from close to 0% up to 82%.

A challenge faced by clinicians is the variable predictive value of specific clinical and laboratory observations when the underlying epidemiology changes [15]. More specifically, typical bacterial pneumonia is more likely to drive elevated WCC and CRP [35], however, elevated biomarkers can also be caused by viral pathogens, especially when disease manifestation is severe, leading to reduced specificity for bacterial infection. Our BN addressed this issue by introducing a variable to summarise the clinical phenotype for each episode, which functioned to separate the influence of the causative pathogen from the clinical progression of the presenting episode based on the patient’s susceptibility to progression and the duration and severity of the presenting illness. This was illustrated in Scenario 1 (Fig 9) and Scenario 2 (Fig 10), in which the same blood results could lead to significantly different predictions of typical bacterial pneumonia for the two demographically and clinically distinct presenting episodes.

4.3 The challenge and need to better understand pathogenicity

It became apparent since early in the model development (after Workshop 1) that the pathogenicity of different organisms plays a central role in driving the probability that a pathogen is causative. There are a large number of organisms (viral and bacterial) that are capable of causing pneumonia, while there are also frequent detections of multiple potentially pathogenic organisms in the nasopharynx. Hence, it is important to understand the prevalence of relevant organisms in the respiratory tract (including the nasopharynx) as well as the probability of being causative of pneumonia given they are present there. These issues affect how the model should be structured (e.g., to what extent we should separate or group pathogens) and how we should parameterise the model (e.g., given the co-presence of more than one pathogen, which one is the more likely to be the causative agent). Many of these issues remain the subject of debate and ultimately unresolved by the scientific community [47].

We designed a series of survey questions to elicit the current understanding and knowledge of domain experts (Survey 2, 3 and 5, see S1 File, Section C, D and F). Finding the best way to frame questions proved challenging given the complexity of the problem domain. Consequently, we explored a variety of approaches based on modelling needs, facilitated by graphical illustration, data visualisation, and various question formats. This process also helped clarify the modelling requirements themselves, helping us to better manage our current imperfect understanding of the problem domain. In the areas of greatest need, sensitivity analyses were subsequently conducted to understand how much the uncertainty in our understanding could impact on the BN predicted outcomes (Fig 8).

4.4 Implications for providing decision support for antibiotic prescription

Among the 230 children enrolled in our cohort of X-ray confirmed pneumonia, 47% were exposed to recent antibiotic use prior to the point of hospital admission when baseline study data were collected. Of these, 99% eventually received antibiotics during their hospital stay (post-enrolment), although only 11% had clinically-confirmed bacterial pneumonia [35]. This suggests that clinicians rarely have the information they need to have confidence to withhold antibiotics. Applied to this cohort, the resulting BN predicted that 28% had typical bacterial pneumonia (either bacterial or viral-bacterial co-infection), indicating the potential to greatly rationalise use of antibiotics. Nevertheless, translating mathematical predictions into better clinical decision-making is not straightforward, as the the best practical use of a model is affected by its ability to predict the real world, i.e., by its performance characteristics. To this end, we showed the most desirable model output threshold for choosing an action can be very sensitive to different input scenarios and trade-off preferences (Table 2).

From a clinical decision making perspective, the choice of model threshold for guiding antibiotic use should take into account the values (utilities or preferences) that one attaches to possible consequences of their decision, e.g., how much clinicians are willing to accept false positives (i.e., over-treating a non-bacterial infection with antibiotics) in order to avoid false negatives (i.e., missing/delaying antibiotic therapy for a bacterial infection). Although satisfactory numeric predictive performance has been achieved by our BN in predicting directly observable variables, including an AUROC of 0.8 in predicting clinically-confirmed bacterial pneumonia with sensitivity 88% and specificity 66%, we can only assess the acceptability of these performance characteristics within the context of specific input scenarios and trade-off preferences (i.e., preferences over the trade-off between FPR and FNR). Looking at Table 2 on this basis, with the clinical data corresponding to FPR/FNR trade-off preference P3, in column (a) the model predicts a positive result in 202 of 205 cases that were not clinically-confirmed bacterial pneumonia. Hence, almost all the truly negative cases would receive antibiotics, suggesting little additional value over current clinical practice (recalling that in our cohort, 99% received antibiotics). Conversely, column (d) shows that, with more information in the input scenario, the model predicts a negative result for 84 of the 205 cases without clinically-confirmed bacterial pneumonia, suggesting the model may help avoid up to 41% of unnecessary antibiotic prescriptions. Of note, clinicians may also want to adjust their FPR/FNR trade-off preference based on a specific patient’s circumstances. For example, they may reduce the acceptable FNR for subgroups that are more vulnerable to potentially severe consequences of a false negative result for bacterial pneumonia (such as neonates); or they may accept a higher FNR for older children where close follow-up is possible.

The model’s behaviour in predicting the (latent) causative pathogens for pneumonia was qualitatively reviewed in various forms, including in Workshops 3 and 4 and Survey 4 (S1 File, Section E). The review outcomes from our experts from diverse domain areas provided confidence in the potential clinical value that this BN offers as a decision support tool. The three scenarios in §3.4 provide an insight into how we envisage the model outputs could support diagnostic decisions under various clinical contexts, leading to numerous questions about how model outputs should be presented. Even if causal models have good performance and produce explainable outputs, they will not be useful tools unless their outputs can be presented in a form that end-users can understand, interpret and use. The barriers to implementation and uptake that decision support tools including BNs face are widely reported [48, 49].

4.5 Study limitations and future direction

The BN needs further validation before it can be clinically implemented, particularly examining the performance of the model in additional cohorts that differ in both key contextual factors and in the availability and operational definitions of superficially equivalent variables. Examples of such differences include: whether the definition of pneumonia is distinct across studies; whether a different spectrum of respiratory infections are involved; and how the clinical epidemiology differs based on the geographically distinct transmission dynamics or variations across healthcare systems. In particular, we highlight the important distinction between variable definitions (or in this case a BN dictionary) and a data dictionary. A variable definition would (usually) not change if it has been well designed to describe a certain mechanistic process in a problem domain; such a variable could then be readily mapped to an external dataset that intends to capture observations about the very same mechanistic process. Any significant discrepancy between a variable definition and an associated data definition should lead to a review and potential revision of the model structure with respect to the original modelling purpose, with the aim to improve the adaptability of the model, the interpretation of model performance once adapted, and the development of the BN for future adaptation and implementation. Several measures were taken to address the overfitting problem, including the 10-fold cross-validation, expert-informed priors and the KL divergence assessment. We acknowledge that the strongest test of overfitting must make use of an external dataset; however, the KL divergence results provide a unique insight into where such overfitting may be present in the model, providing a clear focus for any future evaluation that assesses the model on external datasets.

It is important to note there is a distinction between the actual presence of an organism at a body site and its successful detection by sampling at that body site; we only ever know about the former via evidence of the latter. In the model, we used the detection of organisms as a surrogate for their actual presence in the nasopharynx, in order to reduce the number of latent variables. For our model development, these data were collected prospectively and PCR testing was performed systematically to identify relevant pathogens over and above the routine clinical practice, generating high quality information with little missing data. However, the quality of model predictions will not be as robust if data from routine clinical review and investigation are used alone. Future work should consider the sensitivity and specificity of the laboratory techniques used to capture this information.

The BN can be further improved using richer data with a higher level of granularity. For example, information on the severity of reported symptoms was limited. We would like to further improve the classification of pneumonia (or reduce potential mis-classification due to potential limitations of assessment techniques such as X-rays) by capturing a more comprehensive spectrum of lower respiratory tract infections such as bronchitis and bronchiolitis, and in particular how they should be explicitly mapped to other model variables. Furthermore, other comorbidities were identified but are yet to be included in the model, such as the history of neurological conditions that may affect susceptibility to colonisation and disease progression. There is also the potential to include further information on the use of intravenous fluid, intensive care management and patient progression over time, which would enable the model to predict the likely effects of specific interventions. Integrating such models into electronic medical record systems to enable efficient data capture is an important future direction, as it would significantly improve the ability to evaluate and eventually trial, maintain and use these models and, ultimately, to derive the most value from them.