Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

Prediction of perioperative transfusions using an artificial neural network

  • Steven Walczak ,

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Software, Validation, Writing – original draft, Writing – review & editing

    swalczak@usf.edu

    Affiliation School of Information, Florida Center for Cybersecurity, University of South Florida, Tampa, FL, United States of America

    ⨯
  • Vic Velanovich

    Roles Data curation, Formal analysis, Supervision, Writing – original draft, Writing – review & editing

    Affiliation Department of Surgery, Morsani College of Medicine, University of South Florida, Tampa, FL, United States of America

    ⨯

Abstract

Background

Accurate prediction of operative transfusions is essential for resource allocation and identifying patients at risk of postoperative adverse events. This research examines the efficacy of using artificial neural networks (ANNs) to predict transfusions for all inpatient operations.

Methods

Over 1.6 million surgical cases over a two year period from the NSQIP-PUF database are used. Data from 2014 (750937 records) are used for model development and data from 2015 (885502 records) are used for model validation. ANN and regression models are developed to predict perioperative transfusions for surgical patients.

Results

Various ANN models and logistic regression, using four variable sets, are compared. The best performing ANN models with respect to both sensitivity and area under the receiver operator characteristic curve outperformed all of the regression models (p < .001) and achieved a performance of 70–80% specificity with a corresponding 75–62% sensitivity.

Conclusion

ANNs can predict >75% of the patients who will require transfusion and 70% of those who will not. Increasing specificity to 80% still enables a sensitivity of almost 67%. The unique contribution of this research is the utilization of a single ANN model to predict transfusions across a broad range of surgical procedures.

Citation: Walczak S, Velanovich V (2020) Prediction of perioperative transfusions using an artificial neural network. PLoS ONE 15(2): e0229450. https://doi.org/10.1371/journal.pone.0229450

Editor: Vakhtang Tchantchaleishvili, Thomas Jefferson University, UNITED STATES

Received: September 3, 2019; Accepted: February 6, 2020; Published: February 24, 2020

Copyright: © 2020 Walczak, Velanovich. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data availability The data is part of the ACS-NSQIP PUF database maintained by the American College of Surgeons. The data is available free of charge from the American College of Surgeons for members of hospitals participating in the NSQIP. Data access queries for non-members can be directed to the Corresponding Author, Steven Walczak (swalczak@usf.edu) or Mr. David Chapel, School of Information Office Manager, dmchapel@usf.edu.

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Surgical patients frequently require transfusions both during and immediately following surgery. Transfusions have been shown to improve patient outcomes and reduce mortality [1–3], as well as length of stay and readmission [4]. However, patients requiring transfusion are often also at greater risk for postoperative morbidities and mortality [5], such as: blood borne infections, ABO incompatibility, hemolytic reactions, transfusion-associated circulatory overload, transfusion related acute lung injury, immunosuppression [6], increased risk of perioperative infection [7], and earlier recurrence of malignancies after resection [8,9]. Preoperative knowledge of likely patient transfusion requirements would improve blood resource management, inform clinicians of the likelihood for postoperative morbidities, lower cost by avoiding preoperative typing and crossing patients unlikely to need a transfusion, and could serve to further improve patient outcomes [10].

The ability to identify patients who would be most likely to receive blood transfusions could greatly help in perioperative and postoperative planning and preparation. Regression models have been previously used to predict or model operative and postoperative transfusions [10,11]. Besides regression, artificial neural networks (ANNs) are also a reliable choice for modeling and problem solving in medicine [12].

ANNs are a machine learning methodology based on modeling the neuronal activity of the human brain and have been widely used across various disciplines of medicine [13,14]. Prior research indicates that ANNs generally outperform comparative regression models [12].

ANNs have been used in prior research to predict the surgically-related transfusion needs of patients, but they have all been for a single type of operation. Prior research has used ANNs to predict the transfusion needs of surgery patients undergoing abdominal aortic aneurysm repair [15] and coronary artery bypass grafting [16] operations. Other ANN-based transfusion prediction research is also singularly focused and includes transfusion needs for patients of trauma [17], and acute myeloblastic leukemia [18].

Our research goal is to evaluate the efficacy of using a single ANN model to preoperatively predict perioperative transfusion needs of patients over a wide spectrum of operations. This will significantly extend prior research that has developed ANN prediction models that only work for a single type of operation or disease. A corollary benefit will be the examination of several variables to determine their contribution to the ANN model’s prediction accuracy and subsequent explanatory power of the variables. Although the primary emphasis of this research is to prove that a single ANN may be used for transfusion prediction across any type of surgery, the examination of the impact of the variables selected is important to clinicians [19–21] and hence why two of the selected input variables for the ANN are further examined to establish their clinical contribution to a transfusion decision.

Materials and methods

The study was approved by the institutional review board of the University of South Florida. The American College of Surgeons National Surgical Quality Improvement Program’s (NSQIP) participation use files (PUF) for the time period of 1/1/2014 to 12/31/2015 were used for the analysis. A total of 1,636,438 records of surgical procedures were obtained. The 2014 dataset was used for model development, while the 2015 dataset which was unique from the 2014 dataset was used for model validation. The validation data set was used only a single time for each version of the ANNs developed, emulating prospective use of the ANN on new out-of-sample data. Results are only reported for the 2015 validation dataset. Although retrospective, applying the ANN models trained using 2014 data on 2015 data for validation simulates the real-world ongoing generalization capabilities of the ANN models. For example, a new ANN model developed from a 2020 dataset using the method and variables described next, should have similar performance in predicting transfusion requirements of patients undergoing any type of surgery in the year 2021.

ANN models were developed using NeuralWorks® Professional II/Plus to predict the intraoperative and/or postoperative blood transfusions for patients undergoing any type of surgery. Variables for the ANN transfusion prediction model were selected following review of 36 prior research articles on predicting or administering surgical transfusions and/or surgical bleeding. The prevalence of various variables used in these prior studies is shown in Table 1. Variables were selected using a heuristic of 12.5% prevalence in the literature examined, thus a variable is used in the ANN models if it was mentioned in at least 5 prior research articles which is a 12.9% prevalence. An exception is made for diabetes, which although the diabetes variable did not meet the initial inclusion heuristic, diabetes is mentioned as a bleeding risk factor in a sufficient number of other research publications that are not specific to surgery [55–60]. The variables selected following the prior research variable selection heuristic method (as shown in Table 1) were: age, sex, body mass index (BMI), presence of diabetes, hematocrit level, platelet count, international normalization ratio (INR), and creatinine.

thumbnail
Download:
Table 1. Values used in prior research to predict surgical transfusions or bleeding.

https://doi.org/10.1371/journal.pone.0229450.t001

ANN variables should not be highly correlated [61,62] to avoid noise and overly strong output effects from the correlated variables, so a Pearson’s correlation matrix is used to validate that all the input variables are relatively uncorrelated, with the matrix shown in Table 2. Sex is a Boolean variable, diabetes is a categorical variable, and all other variables are continuous real values. Text data from the NSQIP database were converted to the appropriate type for the sex and diabetes variables. All data were normalized prior to training to prevent undue influence from variables that held much larger values than the rest of the variables: age, BMI, and platelet count.

thumbnail
Download:
Table 2. Input variable correlation matrix.

https://doi.org/10.1371/journal.pone.0229450.t002

In addition to these variables used in the transfusion prediction models, presence of bleeding disorders, co-morbidity severity as measured by the American Association of Anesthesiologists (ASA) classification, whether the operation was elective or an emergency, operation complexity as measured by the work relative value units (w RVU), operation time (in minutes), and mortality were recorded as demographic data. The demographic data is used to determine similarities and differences between transfused and non-transfused data, but again was not used as input to the ANN models.

Records for individual patient operations were eliminated from use in both the training and validation datasets if they were missing any of the ANN variable values other than INR or creatinine, with the cohort flow diagram shown in Fig 1. The output or dependent variable is the number of units transfused of either whole blood or packed red blood cells, since these are the only values reported for transfusions in the NSQIP database. The goal of the ANN output is to predict whether or not a transfusion will be needed by a specific surgery patient, thus any non-zero dependent value is used to indicate a transfusion will occur, while zero output values indicate that no transfusion will be needed by the patient.

thumbnail
Download:
Fig 1. Cohort selection flow diagram.

https://doi.org/10.1371/journal.pone.0229450.g001

Four different ANN transfusion and regression transfusion models are developed around different combinations of the heuristically selected variables, dependent on the presence or absence of INR and/or creatinine. Set 1 uses all of the variables listed: age, sex, BMI, diabetes, hematocrit, platelet count, INR, and creatinine. Set 2 use the same variable set as Set 1, but eliminates the creatinine variable. Set 3 uses the same variable set as Set 2, but also eliminates the INR variable. Set 4 uses the same variables set as Set 1, but eliminates just the INR variable. As shown in Fig 1, the number of training and validation samples for each model ranged from 297670 training samples and 343780 validation samples for Set 1 with all variables present to 615785 training samples and 721992 validation samples for Set 3, which had the fewest number of variables missing both INR and creatinine.

Supervised learning techniques of backpropagation (BP) and radial basis function (RBF) were both attempted along using varying quantities of hidden nodes per layer and different numbers of hidden layers (one or two), following recognized ANN research methodology [61–63]. BP training is selected due to its very wide usage in clinical decision support models so as to facilitate comparison of results with other ANN clinical decision making models [13,64]. RBF training is selected as an alternative training method due to its proven performance when extrapolation is required [65].

The initial architecture for each ANN model used a hidden layer composed of a number of nodes equal to the input layer, or preceding hidden layer in the case of the two hidden layer backpropagation trained model. Training occurred for one million training epochs and also required that the root-mean-square error (RMSE) for training was below 0.01, with an epoch size of 6 samples. If after one million training epochs the RMSE was still not below 0.01, then training continued using 1000 epoch training size iterations until the desired RMSE was reached. The reason that just a RMSE stopping condition is not used to stop training is due to the very large number of non-transfused patients, which caused the ANN training RMSE to go below 0.02 and sometimes 0.01 within 20–30 thousand epochs, but resulted in poor performance on the validation set due to the fact that this used one half to less than one third of the possible training samples to be evaluated by the ANN during training. The number of hidden layer nodes was then increased or decreased by 2, separately for each hidden layer, until no further improvement in performance as measured by sensitivity for a 70% specificity was detected for the validation set. Interestingly, this happened very quickly for the BP models. The best performing architecture for variable Sets 2 and 3 is a single hidden layer model with the hidden layer nodes equal to the number of input nodes. The best performing ANN architecture for variable Sets 1 and 4, containing the INR variable, is a two-hidden layer BP trained model with the first hidden layer equal to two times the number of input nodes minus 1 and the second hidden layer being half the size of the first hidden layer. All of the connection weights were always randomized anytime one of the ANN models was trained. An example of the backpropagation trained ANN model architectures for transfusion prediction with a single hidden layer for Sets 2 and 3 are shown in Fig 2. Each validation set is used only a single time for each model following training.

thumbnail
Download:
Fig 2. Sample ANN architectures for variable sets Set 2 and Set 3.

Size of the boxes representing the processing elements in the ANN indicates the size of numeric difference from zero. Hollow boxes indicate negative values and filled in boxes indicate zero or positive values. Dashed lines represent the weighted connections between processing elements, with the length of the dashes indicating positive or negative values and the color indicating relative value.

https://doi.org/10.1371/journal.pone.0229450.g002

Examining the contribution of using ANN modeling for transfusion predictions is facilitated by comparing the ANN results against a comparable logistic regression model, since regression is one of the most popular clinical decision making tools [12,66]. Because a binary decision is being modeled: whether or not a patient will require a transfusion, logistic regression is the appropriate choice for the type of regression. Regression models for each of the four data sets are developed using MatLab®.

Because both the ANN models and the logistic regression models produce a continuous value, a single cutoff value is need for each model to determine which values, those above the cutoff, represent a required transfusion (sensitivity) and which values below the cutoff represent no need for a transfusion (specificity). While it is possible to maximize sensitivity to 100% with a very small cutoff value, this also reduces the specificity to the point where a transfusion is predicted for almost all patients. Therefore, the heuristic of maximizing sensitivity while at the same time having a specificity of 70–80% is used to select cutoff values for both the ANN and regression models. The cutoff values, which started at zero, were determined using the training data output were modified by one tenths and then by one hundredths and could also be altered by 5 thousandths to move the respective specificity level towards either 70% or 80%, with the closest value at or above the respective heuristic specificity value chosen using these cutoff value guidelines. Respective model performance is determined using the level of sensitivity achieved for specific pre-defined levels of specificity on the out-of-sample validation set from surgeries performed during 2015.

Statistical methods

The nonparametric machine learning method of backpropagation trained ANNs is the primary statistical method utilized for development of the transfusion prediction models, as described above. Logistic regression models are also developed for each of the four variable sets evaluated with the ANN models.

Area under the receiver operator characteristic (AUROC) curve for each of the ANN and regression models is calculated using trapezoidal approximation. Statistical significance between the differences of the various models and AUROC values are calculated using a Chi-squared test for differences, since this test is nonparametric and does not require a normal distribution of either the data or the variances [67]. The Chi-square tests for differences are run at a 0.05 significance level with one degree of freedom.

Results

The surgical patient population demographics for the largest sets of training and validation data, Set 3, which is a superset of all other data sets, are shown in Table 3. There are no clinically significant differences between the 2014 model development cohort and 2015 model validation cohort. However, there were important clinically significant differences between the transfused and non-transfused patients for both the 2014 model development and 2015 model validation cohorts. Transfused patients were, on average, older, more likely to have diabetes mellitus and bleeding disorders, have more severe co-morbidities (as assessed by ASA class), more likely to undergo emergency operations, which were more complex (as measured by wRVU’s), and requiring longer time to complete. These patients also had higher creatinine values, INR values and PTT values. Lastly, transfused patients had a significant increase (p < .01 using a standardized t-test with known variances) in the rate of 30-day mortality compared to non-transfused patients with transfused patients being 14 times more likely to die.

thumbnail
Download:
Table 3. Surgical patient demographics.

https://doi.org/10.1371/journal.pone.0229450.t003

The backpropagation learning algorithm produced better performing ANN models than the RBF, with respect to maximizing sensitivity at the specified specificity levels. The sensitivity, specificity, and AUROC curve values for the backpropagation trained ANN models and the logistic regression models are reported in Table 4. Recall that the best performing architectures for variable sets 1 and 4 are both two hidden layer architectures and the best performing architecture for variable sets 2 and 3 are both single hidden layer architectures. Significance values in Table 4 are calculated using Chi-square at a .05 significance level. From Table 4, the Set 2 ANN had the best overall sensitivity performance at both the 70% specificity and 80% specificity levels. However the Set 3 ANN had the best AUROC values, indicating superior performance for all levels of desired specificity. The ANNs for data sets 2 and 3 both outperformed all of the regression models, however the regression models did perform better on data sets 1 and 4 over the corresponding ANN models. The best AUROC value for the ANN model using variable set 3 is statistically significantly better than all other models except for the ANN model for variable set 2 for which there was not significance difference. Specifically, the ANN model using variable set 3 was p < .01 better than the regressions models for variable sets 1 and 4 and p < .001 better than all of the other models, based on the Chi-squared test for differences.

thumbnail
Download:
Table 4. ANN and regression model results for each of the 4 data sets.

https://doi.org/10.1371/journal.pone.0229450.t004

A total of 2678 different current procedural terminology (CPT) codes, which are used to report different surgical procedures, are represented in the ANN training and validation data sets. These codes came from 531 different hospitals across the United States in 2014 and 615 hospitals in 2015. The operations represented by these CPT codes are shown in Table 5. The training set had 2498 CPT codes, of which 155 were not present in the validation set. The validation set had 2523 CPT codes of which 180 were not present in the training set. The large number of diverse CPT codes will help validate the research goal of demonstrating the efficacy and generalization of a single ANN model for predicting transfusions for patients of any operation, not just a single operation as has been done in prior research.

thumbnail
Download:
Table 5. Surgeries performed for 2014 training and 2015 validation data sets.

https://doi.org/10.1371/journal.pone.0229450.t005

Table 5 also reports the number of operations for each type of surgery in the 2015 validation set that had transfusions and also the ANN variable Set 3 sensitivity at a 70% specificity level for each operation type, which yielded the 75.4% overall sensitivity value for the ANN. Only 2 specific types of operation, out of the 56 types reported, had no transfusion and this was for a total of 3 operations out of the 721992 operations predicted by the ANN model. This demonstrates that although the transfusion prevalence may be low for some types of operations, transfusions do occur across a wide variety of operation types, which further demonstrates the need for transfusion prediction models capable of working across a large collection of surgery types. Remembering that all of the transfusion predictions made for all of the surgeries reported in the NSQIP-PUF dataset were made by a single ANN prediction model, the Set 3 validation sensitivity does vary across the different types of operation reported, but only 3 out of the 56 operation types, representing 4.3% of the operations for which a transfusion is administered, were below 50% sensitivity.

Discussion

Preoperative assessment of patients at risk for intraoperative and postoperative hemorrhage and need for blood transfusions is both a patient safety issue and a cost issue. Obviously, in cases of life-threatening intraoperative hemorrhage, blood transfusions are potentially lifesaving. However, “routine” type and cross-matching of blood is a costly and potentially wasteful endeavor. Each type and cross-match Medicare payment is $37.30 with an additional $17.96 for each additional unit cross-matched [68]. The ability to accurately classify 70–80% of patients who will not require any transfusion, along with identifying a corresponding 66–75% of patients who will need a potentially lifesaving transfusion, can reduce surgical blood costs by eliminating a large number of unneeded type and cross-matches for those patients identified as not requiring a transfusion for operations that would typically perform a type and cross-match.

The standard approach to assessing operative bleeding risk includes history, physical examination, laboratory evaluation (INR, PTT, platelet count, bleeding time), type of operation, patient co-morbidities (e.g., inherited coagulopathies, coagulopathy of liver disease), drug-induced coagulopathy (e.g., anticoagulation therapy for cardiovascular disease), and platelet dysfunction (e.g., immune thrombocytopenic purpura) [6,69]. Our patient data has also shown that age, ASA class, presence of diabetes mellitus and bleeding disorders were more associated with patients receiving transfusions. Operations on or near large vessels, those with extensive dissections, and prolonged operations are generally considered heuristics of a higher risk of bleeding and consequent need for transfusion. Our data do reflect that patients receiving transfusions were more likely to undergo more complex operations (as measured by w RVU values) and longer operations. Although useful, these heuristics have not been precise in identifying the amount a patient will bleed or who will need a transfusion [70,71]. Therefore, better predicting models would be useful and ANNs provide the opportunity to create better transfusion prediction models.

In addition to being a machine learning method able to learn solutions to arbitrarily complex problems [72], ANNs have several other advantages over regression and other statistical methods that serve to improve medical model prediction performance, including: they are nonparametric thus data is not required to fit specific prerequisite requirements and therefore surgeons do not have to have advanced statistical analysis training [72], all variable interactions do not need to be predefined since they are discovered through machine learning, and once trained they are highly resistant to noise or errors in the data for the input variables. This has been demonstrated in the current research by the fact that 180 new operations (CPT codes) were included in the results for the ANN that were not present in the training set, thus these operations were completely new to the ANN prediction model.

The principle drawback of ANN models in medicine is their black box nature [72] and subsequent difficulty in determining which input variables are most significant for clinical decision making regarding surgical transfusions. Several techniques exist for trying to determine the explanatory power of input variables, such as iteratively leaving out select variables [72,73] or summing of the connected weights [74]. The leave-one-out strategy is used in the current research to measure if INR or creatinine had any purpose or explanatory power in transfusion prediction modeling. The results displayed in Table 4 show that the data sets that removed the INR variable performed better than the other data sets, which indicates that this variable is not contributing to the determination of a future transfusion need for any specific patient. The lack of affect from the INR variable might be caused by high correlation with other variables included in the input variable set and as such acts as noise to the ANN, reducing performance [62]. A Pearson’s correlation matrix is calculated for all variables in the data set (see Table 2) and no unusually high correlations were detected, with the highest absolute correlation value being 0.209 indicating a small and acceptable level of correlation and validating the finding that INR, which had a maximum absolute correlation value of 0.147, does not contribute to transfusion predictions when used with the other indicated variables. This is in contrast to the findings from Table 3, which indicate that transfused patients would have a higher INR, however the ANN demonstrates this value is unnecessary for predicting transfusions. The fact that two hidden layer architectures performed better than single hidden layer architectures for the ANN transfusion prediction models that utilize the INR variable (variable Sets 1 and 4), indicates that the solution surface when INR is considered has additional nonlinearities [61]. Future research could examine why the introduction of an INR value creates a more complex solution surface for predicting transfusions.

The data set that removed just creatinine (Set 4) from the full set of variables (Set 1) had similar performance, with no statistically measurable difference to the full data set. The data set that removed creatinine and INR both did show improved performance and had the highest AUROC value. Future research is needed to further investigate the importance of creatinine and other different variable combinations, where variables are selected to minimize correlation and to maximize predictive performance. Identification of contributing and non-contributing variables has clinical significance in that contributing variables are further identified as key elements in the transfusion process. Non-contributing variables may successfully be removed from the decision making process, making the transfusion decision more efficient by reducing potential noise from non-contributing variables and less costly by eliminating the need to perform specimen collection and the subsequent pathology to acquire these unneeded values.

ANNs should definitely be in any clinical medicine researcher’s toolbox, as demonstrated by the results for variable sets 2 and 3. However, from Table 4, depending on the specific variables used, regression modeling may perform as well as or slightly better than ANNs. Thus, clinical researchers should utilize both ANNs as well as regression and possibly other machine learning techniques such as random forests and then compare the results of all models to select the best performing model and consequently optimize clinical outcomes.

It is interesting to note from Table 3 that postoperative deaths within 30 days of surgery occurred 14 times more frequently in patients who required transfusions. The non-transfused mortality rate was 4 per 1000 patients, while the transfused mortality rate was 56–57 per 1000 patients. The rates of postoperative morbidity were also significantly higher in transfused patients. Higher mortality and morbidity rates for transfused patients may be due to the difficulty of the operation, but the transfusions themselves may play a role [6]. The majority of the complications in the transfused patients are infectious, thus the immunosuppressive nature of transfusion may affect postoperative morbidity. The fact that transfused patients are much more likely to suffer mortality or postoperative morbidity indicates that preoperative knowledge of the likelihood for a transfusion to be needed perioperatively, could also be used by anaesthesiologists and surgeons to proactively plan treatment and antibiotic regimens for these patients in addition to improving the management of blood supply.

Bleeding and transfusion prediction models have been developed in a number of specialties, including trauma [75], orthopedic surgery [76,77], cardiac surgery [78,79], vascular surgery [80], liver transplantation [81], and otolaryngology [82]. Our study is unique in that rather than assessing transfusion need for a single type of operation, we can do it across a broad spectrum of operation types. Most prior transfusion estimation models relied on conventional statistical techniques, such as multiple logistic regression modeling. However, our studies demonstrates that the ANN modeling methodology can be superior to logistic regression when the best independent variable set is used. The ANN model using variable Set 2 accurately predicted the transfusion needs for two thirds of the surgical patients who would need one and for over 80% of those who would not require any transfusion and the ANN model using variable set 3 produced a significantly higher AUROC. If a larger false positive rate can be tolerated at a 70% specificity level, then the ANN model is able to identify over three fourths of patients for any type of surgery who will need a transfusion, assuming the pre-operative labs would include either hematocrit or hemoglobin and also platelet count. The false positive rate identifying individuals as needing a transfusion who ultimately did not receive one could have a negative effect on the blood supply, but does not negatively affect the clinical outcomes for these patients, other than a heightened vigilance for morbidity that is less likely to occur.

Others have used ANN to predict bleeding due to medical or surgical therapy. These include lower gastrointestinal hemorrhage [83], cardiopulmonary bypass [84], experimental hemorrhage model in rats [85], and radiation therapy for prostate cancer [86]. Although none of these studies examined consequent transfusion requirements. Hayn, et al. [87] used predictive modeling based on decision trees on a large dataset of multiple different types of operations to predict transfusion needs. These efforts reflect the need to better predict blood loss and consequent transfusion requirements.

Limitations and future research

There are limitations to the reported ANN transfusion prediction model. Firstly, as intraoperative hemorrhage is related to surgical trauma and surgeon skill, reducing blood loss and transfusion need is very much operator dependent. In addition, although length of operation and extent of dissection can be anticipated by the operating surgeon to a certain extent, these factors are still quite variable. There is no obvious way to include this in a preoperative prediction model. Restrictive blood transfusion practices [88] were also not able to be represented in the variables for the ANN model. Future research is needed to examine if analogous values for the actual length and difficulty of an operation may be used in the transfusion prediction ANN model and how this will affect performance of the ANN models.

Although 2678 different operations in total were evaluated, with 2523 different operations in the validation set, these are all operations that reported the values for the variables used in the ANN models. Any operation that did not record a hematocrit value, a platelet count value, age sex, BMI, or diabetes status would not have been included in either the training or validation sets for the ANN models. While the ANN demonstrated very good sensitivity for the operations evaluated, its performance on the operations that did not report one of the required variable values is undetermined. However, the lack of a hematocrit value specifically may indicate that the operation being performed had an extremely low probability of requiring a transfusion and as such it is generally safe to not evaluate transfusion requirements for these operations.

Additionally, the current ANN models predict perioperative transfusion need for up to one year following the training date. As long as there are no new surgery protocols, procedures, or technologies that impact patient bleeding and transfusions, then the current model should be able to continue predicting transfusions at the documented sensitivity and specificity levels until such change occurs. Therefore, retraining of the ANN may be unnecessary until an advancement in surgical protocol, process or technology occurs. Future research should examine how long diagnostic and prognostic ANN models may be used before re-training becomes necessary, based on decreased sensitivity and specificity values.

As mentioned, the NSQIP database used for the reported research only reports transfusion of whole blood or packed red blood cells. Other types of blood products may also be transfused, such as plasma or platelets. The platelet count variable was not removed from any of the ANN models attempting to determine the impact of variables because it is considered the primary indicator for platelet transfusions and the ANN model is designed to work for any type of transfusion. Since platelet transfusions were not reported in the NSQIP database, the validity of the model for predicting platelet transfusions in addition to whole blood and packed red blood cell transfusions could not be ascertained. Future research is needed to attain databases of surgical procedures that also report the transfusion of platelets in addition to other blood products and use this data to further evaluate the efficacy of the proposed ANN model for preoperative transfusion predictions.

Finally, since not all possible ANN architectures were evaluated for each of the backpropagation and RBF trained models, this represents another possible limitation in that a slightly different architecture could possibly improve the performance of the ANN surgical transfusion prediction models. Thus, the reported results should be interpreted as the minimum possible sensitivity achievable for the defined specificity levels and consequent AUROC values. Future research in addition to examining different variable combinations to further clarify the impact of each variable could also examine the various architectures not evaluated by the current research, specifically incrementing or decrementing the number of hidden nodes in each layer by one instead of two.

Clinical implications

The research reported in this article demonstrates the efficacy of using a single ANN model to predict the transfusion requirements of patients across a wide range of operations as shown in Table 5. This leads to the question of how the ANN results can be used in clinical practice. The NeuralWorks® Professional II/Plus shell tool used to develop the ANN transfusion prediction models comes with a feature that enables the ANN to be produced in an executable format, utilizing the C programing language, which may be run from a command line prompt. Clinicians should not be expected to know how to execute the ANN executable application and thus a user interface would be required to be produced that would either automatically acquire the requisite input variable values or prompt the clinician for missing values. The user interface would then execute the encoded ANN and translate the results into a more readable format, such as reporting a “high likelihood for a transfusion,” when the ANN predicts a transfusion occurring.

Automatic acquisition of variables could be accomplished by linking the ANN model with existing electronic medical record (EMR) systems or surgical planning systems, such as One Medical Passport® (see https://www.onemedicalpassport.com/). Prior research has shown that clinical decision support systems like ANNs may be connected and embedded with EMR systems [89]. Because of the documented improvements in clinical decision making when clinicians utilize EMR systems [90], integrating the ANN transfusion prediction decision support system with an EMR or surgical planning system would facilitate access by integrating the ANN transfusion prediction decision support tool into their workflow, which would further improve blood bank planning and surgical and post-operative preparation [91] and consequently improve patient outcomes. The ANN would be able to alert surgeons to possible unexpected complications that would lead to a transfusion even if none were anticipated and could also be used to confirm the necessity of surgeon blood orders. Ultimately though, the decision to type and cross-match and ultimately transfuse blood products is the surgeon’s and the ANN-based clinical decision support system should be used as evidence to assist surgeons in making those decisions.

Conclusion

ANNs have been shown to efficaciously provide a mechanism to preoperatively evaluate the potential for need of perioperative transfusions for individual patients across a wide range of operation types. A single ANN model has been shown to work at 66–76% sensitivity with a corresponding 80–70% specificity, with a corresponding AUROC of 0.847 to 0.858, across a very large number of different operations (2678 CPT codes). This advances the state-of-the-art where in the past ANN models have only predicted transfusions for a single type of operation. Additionally, the sensitivity and specificity performance is across data from 615 different hospitals indicating good generalization of these results for any hospital choosing to use this ANN-based preoperative transfusion model. Incorporation of other anticipated factors, such as difficulty and length of the operation, may additionally increase its predictive value.

References

  1. 1. Ferraris VA, Hochstetler M, Martin JT, Mahan A, Saha SP. Blood transfusion and adverse surgical outcomes: the good and the bad. Surgery. 2015;158(3):608–17. pmid:26032824
  2. 2. Wu WC, Smith TS, Henderson WG, Eaton CB, Poses RM, Uttley G, et al. Operative blood loss, blood transfusion, and 30-day mortality in older patients after major noncardiac surgery. Ann Surg. 2010;252(1):11–7. pmid:20505504
  3. 3. Wu WC, Trivedi A, Friedmann PD, Henderson WG, Smith TS, Poses RM, et al. Association between hospital intraoperative blood transfusion practices for surgical blood loss and hospital surgical mortality rates. Ann Surg. 2012;255(4):708–14. pmid:22367442
  4. 4. Goodnough LT, Maggio P, Hadhazy E, Shieh L, Hernandez‐Boussard T, Khari P, et al. Restrictive blood transfusion practices are associated with improved patient outcomes. Transf. 2014;54(10pt2):2753–9.
  5. 5. Rao SV, Jollis JG, Harrington RA, Granger CB, Newby LK, Armstrong PW, et al. Relationship of blood transfusion and clinical outcomes in patients with acute coronary syndromes. JAMA. 2004;292(13):1555–62. pmid:15467057
  6. 6. Brasel K. Hemorrhagic risk and blood components. In: Fischer JE (Ed.). Fischer’s Mastery of Surgery, 7th ed. Philadelphia:Wolters Kluwer, 2019, pp. 103–10.
    • 7. Houbiers JG, van de Velde CJ, van de Watering LM, Hermans J, Schreuder S, Bijnen AB, et al. Transfusion of red cells is associated with increased incidence of bacterial infection after colorectal surgery: A prospective study. Transf. 1997;37(2):126–34.
    • 8. Ejaz A, Spolverato G, Kim Y, Margonis GA, Gupta R, Amini N, et al. Impact of blood transfusion and transfusion practices on long-term outcome following hepatopancreaticobiliary surgery. J Gastrointest Surg. 2015;19(5):887–96. pmid:25707813
    • 9. Hjalgrim H, Edgren G, Rostgaard K, Reilly M, Tran TN, Titlestad KE, et al. Cancer incidence in blood transfusion recipients. J Natl Cancer Instit. 2007;99(24):1864–74.
    • 10. Ecker BL, Simmons KD, Zaheer S, Poe SL, Bartlett EK, Drebin JA, et al. Blood transfusion in major abdominal surgery for malignant tumors: a trend analysis using the national surgical quality improvement program. JAMA Surgery. 2016;151(6):518–25. pmid:26763765
    • 11. Frisch NB, Wessell NM, Charters MA, Yu S, Jeffries JJ, Silverton CD. Predictors and complications of blood transfusion in total hip and knee arthroplasty. J Arthroplasty. 2014;29(9):189–92.
    • 12. Dreiseitl S, Ohno-Machado L. Logistic regression and artificial neural network classification models: a methodology review. J Biomed Informatics. 2002;35(5–6):352–9.
    • 13. Amato F, López A, Peña-Méndez EM, Vaňhara P, Hampl A, Havel J. Artificial neural networks in medical diagnosis. J Appl Biomed. 2013;11(2):47–58.
    • 14. Lisboa PJ. A review of evidence of health benefit from artificial neural networks in medical intervention. Neural Networks. 2002;15(1):11–39. pmid:11958484
    • 15. Walczak S, Scharf JE. Reducing surgical patient costs through use of an artificial neural network to predict transfusion requirements. Decision Support Sys. 2000;30(2):125–38.
    • 16. Covin R, O'Brien M, Grunwald G, Brimhall B, Sethi G, Walczak S, et al. Factors affecting transfusion of fresh frozen plasma, platelets, and red blood cells during elective coronary artery bypass graft surgery. Archives Pathol Lab Med. 2003;127(4):415–23.
    • 17. Walczak S. Artificial neural network medical decision support tool: predicting transfusion requirements of ER patients. IEEE Transact on Info Technol in Biomed. 2005;9(3):468–74.
    • 18. Ho WH, Chang CS. Genetic-algorithm-based artificial neural network modeling for platelet transfusion requirements on acute myeloblastic leukemia patients. Expert Sys with Applic. 2011;38(5):6319–23.
    • 19. Dag A, Topuz K, Oztekin A, Bulur S, Megahed FM. A probabilistic data-driven framework for scoring the preoperative recipient-donor heart transplant survival. Decision Support Sys. 2016;86:1–12.
    • 20. Hunink MM, Weinstein MC, Wittenberg E, Drummond MF, Pliskin JS, Wong JB, et al. Decision making in health and medicine: Integrating evidence and values. Cambridge University Press; 2001.
      • 21. Walczak S, Velanovich V. Improving prognosis and reducing decision regret for pancreatic cancer treatment using artificial neural networks. Decision Support Sys. 2018;106:110–8.
      • 22. Barr PJ, Donnelly M, Cardwell C, Alam SS, Morris K, Parker M, et al. Drivers of transfusion decision making and quality of the evidence in orthopedic surgery: a systematic review of the literature. Transfus Med Reviews. 2011;25(4):304–16.
      • 23. Giordano GF Sr, Giordano GF Jr, Rivers SL, Chung GK, Mammana RB, Marco JD, et al. Determinants of homologous blood usage utilizing autologous platelet-rich plasma in cardiac operations. Ann Thor Surg. 1989;47(6):897–902.
      • 24. Karkouti K, Cohen MM, McCluskey SA, Sher GD. A multivariable model for predicting the need for blood transfusion in patients undergoing first‐time elective coronary bypass graft surgery. Transfus. 2001;41(10):1193–203.
      • 25. Krishna NM, Nagaraja PS, Singh NG, Nanjappa SN, Kumar KN, Prabhakar V, et al. Evaluation of risk scores in predicting perioperative blood transfusions in adult cardiac surgery. Ann Cardiac Anaesth. 2019;22(1):73–8.
      • 26. Litmathe J, Boeken U, Feindt P, Gams E. (2003). Predictors of homologous blood transfusion for patients undergoing open heart surgery. Thorac Cardiovasc Surgeon. 2003;51(1):17–21.
      • 27. Magovern JA, Sakert T, Benckart DH, Burkholder JA, Liebler GA, Magovern GJ Sr, et al. A model for predicting transfusion after coronary artery bypass grafting. Ann Thorac Surg. 1996;61(1):27–32. pmid:8561579
      • 28. Ranucci M, Castelvecchio S, Frigiola A, Scolletta S, Giomarelli P, Biagioli B. Predicting transfusions in cardiac surgery: the easier, the better: the Transfusion Risk and Clinical Knowledge score. Vox Sanguinis. 2009;96(4):324–32. pmid:19192264
      • 29. Williams ML, Trivedi JR, Doughtie C, Slaughter MS. Is female sex an independent risk factor for perioperative transfusion in coronary artery bypass graft surgery?. J Amer Coll Surgeons. 2011;212(3):362–6.
      • 30. Zheng F, Cammisa FP Jr, Sandhu HS, Girardi FP, Khan SN. Factors predicting hospital stay, operative time, blood loss, and transfusion in patients undergoing revision posterior lumbar spine decompression, fusion, and segmental instrumentation. Spine. 2002;27(8):818–24. pmid:11935103
      • 31. Shander A, Javidroozi M, Ozawa S, Hare GM. What is really dangerous: anaemia or transfusion?. Brit J Anaesth. 2011;107(suppl_1):i41–i59.
      • 32. Chang SS, Duong DT, Wells N, Cole EE, Smith JA, Cookson MS. Predicting blood loss and transfusion requirements during radical prostatectomy: the significant negative impact of increasing body mass index. J Urol. 2004;171(5):1861–5. pmid:15076294
      • 33. Eastridge BJ, Malone D, Holcomb JB. Early predictors of transfusion and mortality after injury: a review of the data-based literature. J Traum Acute Care Surg. 2006;60(6):S20–5.
      • 34. Nunez TC, Cotton BA. Transfusion therapy in hemorrhagic shock. Curr Opin Crit Care. 2009;15(6):536–41. pmid:19730099
      • 35. Simon TL, Alverson DC, AuBuchon J, Cooper ES. Practice parameter for the use of red blood cell transfusions: developed by the Red Blood Cell Administration Practice Guideline Development Task Force of the College of American Pathologists. Arch Path Lab Med. 1998;122(2):130–8. pmid:9499355
      • 36. Spinella PC, Holcomb JB. Resuscitation and transfusion principles for traumatic hemorrhagic shock. Blood Rev. 2009;23(6):231–40. pmid:19695750
      • 37. Ad N, Holmes SD, Massimiano PS, Spiegelstein D, Shuman DJ, Pritchard G, et al. Operative risk and preoperative hematocrit in bypass graft surgery: role of gender and blood transfusion. Cardiovasc Revasc Med. 2015;16(7):397–400. pmid:26361981
      • 38. Reeves BC, Ascione R, Chamberlain MH, Angelini GD. Effect of body mass index on early outcomes in patients undergoing coronary artery bypass surgery. J Americ Coll Cardiol. 2003;42(4):668–76.
      • 39. Stamou SC, Nussbaum M, Stiegel RM, Reames MK, Skipper ER, Robicsek F, et al. Effect of body mass index on outcomes after cardiac surgery: is there an obesity paradox?. Ann Thorac Surg. 2011;91(1):42–7. pmid:21172483
      • 40. Expert Working Group. Guidelines for red blood cell and plasma transfusion for adults and children. CMAJ: Canadian Med Assoc J. 1997;156(11):S1–S24.
      • 41. Hsu YM, Ness PM, Cushing MM. Principles of red blood cell transfusion. In Hematology: Basic Principles and Practice. 2017 (pp. 1702–14). Elsevier.
        • 42. Carson JL, Grossman BJ, Kleinman S, Tinmouth AT, Marques MB, Fung MK, et al. Red blood cell transfusion: a clinical practice guideline from the AABB. Ann Internal Med. 2012;157(1):49–58.
        • 43. Curnow J, Pasalic L, Favaloro EJ. Why Do Patients Bleed?. Surg J. 2016;2(1):e29–e43.
        • 44. Massicotte L, Sassine MP, Lenis S, Roy A. Transfusion predictors in liver transplant. Anesth & Analg. 2004;98(5):1245–51.
        • 45. Millett PJ, Porramatikul M, Chen N, Zurakowski D, Warner JJ. Analysis of transfusion predictors in shoulder arthroplasty. J Bone Joint Surg. 2006;88(6):1223–30. pmid:16757754
        • 46. Pulitano C, Arru M, Bellio L, Rossini S, Ferla G, Aldrighetti L. A risk score for predicting perioperative blood transfusion in liver surgery. Brit J Surg. 2007;94(7):860–5. pmid:17380562
        • 47. Tobian AA, Heddle NM, Wiegmann TL, Carson JL. Red blood cell transfusion: 2016 clinical practice guidelines from AABB. Transfus. 2016;56(10):2627–30.
        • 48. Jung KH, Kim SM, Choi MG, Lee JH, Noh JH, Sohn TS, et al. Preoperative smoking cessation can reduce postoperative complications in gastric cancer surgery. Gastr Canc. 2015;18(4):683–90.
        • 49. Achneck HE, Sileshi B, Parikh A, Milano CA, Welsby IJ, Lawson JH. Pathophysiology of bleeding and clotting in the cardiac surgery patient: from vascular endothelium to circulatory assist device surface. Circulation. 2010;122(20):2068–77. pmid:21098468
        • 50. Despotis GJ, Hogue CW Jr. Pathophysiology, prevention, and treatment of bleeding after cardiac surgery: a primer for cardiologists and an update for the cardiothoracic team. Am J Cardio. 1999;83(4):15–30.
        • 51. Kaufman RM, Djulbegovic B, Gernsheimer T, Kleinman S, Tinmouth AT, Capocelli KE, et al. Platelet transfusion: a clinical practice guideline from the AABB. Ann Internal Med. 2015;162(3):205–13.
        • 52. Marietta M, Facchini L, Pedrazzi P, Busani S, Torelli G. Pathophysiology of bleeding in surgery. In Transplant Proceed. 2006 (Vol. 38, No. 3, pp. 812–814). Elsevier.
          • 53. Alotaibi GS, Almodaimegh H, McMurtry MS, Wu C. Do women bleed more than men when prescribed novel oral anticoagulants for venous thromboembolism? A sex-based meta-analysis. Thrombosis Research. 2013;132(2):185–9. pmid:23932014
          • 54. Mauer AC, Khazanov NA, Levenkova N, Tian S, Barbour EM, Khalida C, et al. Impact of sex, age, race, ethnicity and aspirin use on bleeding symptoms in healthy adults. J Thrombosis Haemost. 2011;9(1):100–8.
          • 55. Albers JR, Hull SK, Wesley RM. Abnormal uterine bleeding. Amer Fam Physician. 2004;69(8):1915–34.
          • 56. De Berardis G, Lucisano G, D’Ettorre A, Pellegrini F, Lepore V, Tognoni G, et al. Association of aspirin use with major bleeding in patients with and without diabetes. JAMA. 2012;307(21):2286–94. pmid:22706834
          • 57. Lal S, Cheng B, Kaplan S, Softness B, Greenberg E, Goland RS, et al. Gingival bleeding in 6-to 13-year-old children with diabetes mellitus. Pediatr Dentistry. 2007;29(5):426–30.
          • 58. Luo JC, Leu HB, Huang KW, Huang CC, Hou MC, Lin HC, et al. Incidence of bleeding from gastroduodenal ulcers in patients with end-stage renal disease receiving hemodialysis. CMAJ. 2011;183(18):E1345–51. pmid:22083684
          • 59. Peng YL, Leu HB, Luo JC, Huang CC, Hou MC, Lin HC, et al. Diabetes is an independent risk factor for peptic ulcer bleeding: A nationwide population‐based cohort study. J Gastroent Hepat. 2013;28(8):1295–9.
          • 60. Thomsen RW, Riis A, Christensen S, Nørgaard M, Sørensen HT. Diabetes and 30-day mortality from peptic ulcer bleeding and perforation: a Danish population-based cohort study. Diabetes Care. 2006;29(4):805–10. pmid:16567819
          • 61. Walczak S, Cerpa N. Heuristic principles for the design of artificial neural networks. Information and Software Technology. 1999;41(2):107–17.
          • 62. Zhang GP. Avoiding pitfalls in neural network research. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews). 2007;37(1):3–16.
          • 63. Cruz JA, Wishart DS. Applications of machine learning in cancer prediction and prognosis. Cancer Informatics. 2006;2:59–78.
          • 64. Lisboa PJ, Taktak AF. The use of artificial neural networks in decision support in cancer: a systematic review. Neural Networks. 2006;19(4):408–15. pmid:16483741
          • 65. Park J, Sandberg IW. Approximation and radial-basis-function networks. Neural Computation. 1993;5(2):305–16.
          • 66. Mangiameli P, West D, Rampal R. Model selection for medical diagnosis decision support systems. Decision Support Sys. 2004;36(3):247–59.
          • 67. McHugh ML. The Chi-square test of independence. Biochem Med. 2013;23(2):143–9.
          • 68. Frank SM, Oleyar MJ, Ness PM, Tobian AAR. Reducing unnecessary preoperative blood orders and costs by implementing an updated institution-specific maximum surgical blood order schedule and a remote electronic blood release system. Anesthesiology 2014;121(3):501–9. pmid:24932853
          • 69. Owings JT, Grosselin RC. Bleeding and transfusion. In: Souba WW, Fink MP, Jurkovich GJ, Kaiser LR, Pearce WH, Pemberton JH, Soper NJ (Eds.). ACS Surgery: Principles and Practice 2005. New York:WebMD, Inc., 2005, pp. 50–64.
            • 70. Velanovich V. The value of routine preoperative laboratory testing in predicting postoperative complications: A multivariate analysis. Surgery. 1991;109(3 Pt 1):236–43.
            • 71. Benarroch-Gampel J, Sheffield KM, Duncan CB, Brown KM, Han Y, Townsend CM Jr, et al. Preoperative laboratory testing in patients undergoing elective, low-risk ambulatory surgery. Ann Surg. 2012;256(3):518–28. pmid:22868362
            • 72. Tu JV. Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes. J Clin Epidemiol. 1996;49(11):1225–31. pmid:8892489
            • 73. Walczak S. Evaluating medical decision making heuristics and other business heuristics with neural networks. In: Intelligent Decision Making: An AI-based Approach 2008 (pp. 259–87). Springer, Berlin, Heidelberg.
              • 74. Augasta MG, Kathirvalavakumar T. Reverse engineering the neural networks for rule extraction in classification problems. Neural Process Letters. 2012;35(2):131–50.
              • 75. El-Menyar A, Goyal P, Tilley E, Latifi R. The clinical utility of shock index to predict the need for blood transfusion and outcomes in trauma. J Surg Res. 2018;227:52–9. pmid:29804862
              • 76. Kim HJ, Park HS, Jang MJ, Koh WU, Song JG, Lee CS, et al. Predicting massive transfusion in adolescent idiopathic scoliosis patients undergoing corrective surgery: Assessment of preoperative radiographic findings. Medicine. 2018; 97(22) e10972. pmid:29851849
              • 77. Shin HJ, Min BH, Na HS. FIBTEM as a predictor of intra- and postoperative blood loss in revisional total hip arthroplasty: A prospective, observational study. Medicine. 2018; 97(22) e10929. pmid:29851830
              • 78. Meesters MI, Burtman D, van de Van PM, Boer C. Prediction of postoperative blood loss using thromboelastometry in adult cardiac surgery: Cohort study and systematic review. J Cardiothorac Vasc Anesth. 2018;32(1):141–50. pmid:29126688
              • 79. Klein AA, Collier T, Yeates J, Miles LF, Fletcher SN, Evans C, et al. The ACTA PORT-score for predicting perioperative risk of blood transfusion for adult cardiac surgery. Br J Anaesth. 2017;119(3):394–401. pmid:28969306
              • 80. Pieri M, Nardelli P, De Luca M, Landoni G, Frassoni S, Melissano G, et al. Predicting the need for intraoperative large volume blood transfusions during thoraco-abdominal aortic aneurysm repair. Eur J Vasc Endosvasc Surg. 2017;53(3):347–53.
              • 81. Eghbal MH, Samadi K, Khosravi MB, Sahmeddini MA, Ghaffaripoor S, Ghorbani M, et al. The input of preoperative variables on intraoperative blood loss and transfusion requirements during orthotopic liver transplant. Exp Clin Transplant. 2019;17(4):507–12. pmid:29025385
              • 82. Abu-Ghanem S, Warshavsky A, Carmel NN, Abu‐Ghanem Y, Abergel A, Fliss DM, et al. Predictive factors for perioperative blood transfusion in neck dissection. Larnygoscope. 2016;126(4):851–7.
              • 83. Das A, Ben-Menachem T, Cooper GS, Chak A, Sivak MV Jr, Gonet JA, et al. Prediction of outcome in acute lower-gastrointestinal haemorrhage based on an artificial neural network: internal and external validation of a predictive model. The Lancet. 2003;362(9392):1261–6.
              • 84. Huang RS, Nedelcu E, Bai Y, Wahed A, Klein K, Tint H, et al. Post-operative bleeding risk stratification in cardiopulmonary bypass patients using artificial neural network. Ann Clin Lab Sci. 2015;45(2):181–6. pmid:25887872
              • 85. Jefferson MF, Pendleton N, Mohamed S, Kirkman E, Little RA, Lucas SB, et al. Prediction of hemorrhagic blood loss with a genetic algorithm neural network. J Appl Physiol. 1998;84(1):357–61. pmid:9451657
              • 86. Tomatis S, Rancati T, Fiorino C, Vavassori V, Fellin G, Cagna E, et al. Late rectal bleeding after 3D-CRT for prostate cancer: development of a neural-network-based predictive model. Physics Med Biol. 2012;57(5):1399–412.
              • 87. Hayn D, Kreiner K, Kastner P, Breznik N, Hofmann A, Gombotz H., et al. Data driven methods for predicting blood transfusion needs in elective surgery. Stud Health Technol Inform. 2016;223:9–16. pmid:27139379
              • 88. Clevenger B, Mallett SV, Klein AA, Richards T. Patient blood management to reduce surgical risk. Br J Sur. 2015;102(11):1325–37.
              • 89. Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Americ Med Informatics Assoc. 2007;14(1):29–40.
              • 90. van Poelgeest R, van Groningen JT, Daniels JH, Roes KC, Wiggers T, Wouters MW, et al. Level of Digitization in Dutch Hospitals and the Lengths of Stay of Patients with Colorectal Cancer. J Med Sys. 2017;41(5):84.
              • 91. Feldman K, Davis D, Chawla NV. Scaling and contextualizing personalized healthcare: A case study of disease prediction algorithm integration. J Biomed Informatics. 2015;57:377–85.