Abstract
Background
Human immunodeficiency virus (HIV) viral failure occurs when antiretroviral therapy fails to suppress and sustain a person’s viral load count below 1,000 copies of viral ribonucleic acid per milliliter. For those newly diagnosed with HIV and living in a setting where healthcare resources are limited, such as a low- and middle-income country, the World Health Organization recommends viral load monitoring six months after initiation of antiretroviral treatment and yearly thereafter. Deviations from this schedule are made in cases where viral failure occurs or at the discretion of the clinician. Failure to detect viral failure in a timely fashion can lead to delayed administration of essential interventions. Clinical prediction models based on information available in the patient medical record are increasingly being developed and deployed for decision support in clinical medicine and public health. This raises the possibility that prediction models can be used to detect potential for viral failure in advance of viral measurements, particularly when those measurements occur infrequently.
Objective
Our goal is to use electronic health record data from a large HIV care program in Kenya to characterize and compare the predictive accuracy of several statistical machine learning methods for predicting viral failure at the first and second measurements following initiation of antiretroviral therapy. Predictive accuracy is measured in terms of sensitivity, specificity and area under the receiver-operator characteristic curve.
Methods
We trained and cross-validated 10 statistical machine learning models and algorithms on data from over 10,000 patients in the Academic Model Providing Access to Healthcare care program in western Kenya. These included parametric, non-parametric, ensemble, and Bayesian methods. The input variables included 50 items from the clinical record, hand picked in consultation with clinician experts. Predictive accuracy measures were calculated using 10-fold cross validation.
Results
Viral load failure rate is about 20% in this patient cohort at both the first and second measurements. Ensemble techniques generally outperformed other methods. For predicting viral failure at the first follow up measure, specificity was over 90% for these methods, but sensitivity was typically in the 50–60% range. Predictive accuracy was greater for the second follow up measure, with sensitivities over 80%. Super Learner, gradient boosting and Bayesian additive regression trees consistently outperformed other methods. For a viral failure rate of 20%, the positive predictive value for the top-performing methods is between 75 and 85%, while the negative predictive value is over 95%.
Conclusion
Evidence from this study suggests that machine learning techniques have potential to identify patients at risk for viral failure prior to their scheduled measurements. Ultimately, prognostic virologic assessment can help guide the administration of earlier targeted intervention such as enhanced drug resistance monitoring, rigorous adherence counseling, or appropriate next-line therapy switching. External validation studies should be used to confirm the results found here.
Funding source: National Institute of Allergy and Infectious Diseases
Award Identifier / Grant number: R01 AI 108441
Award Identifier / Grant number: P30 AI 42853
Award Identifier / Grant number: K24 AI 134359
Funding source: Fogarty International Center
Award Identifier / Grant number: D43 TW010050
Research funding: This work was funded by National Institute of Allergy and Infectious Diseases (R01 AI 108441, P30 AI 42853, K24 AI 134359) and Fogarty International Center (D43 TW010050).
Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
Competing interests: Authors state no conflict of interest.
Appendix
A. Additional tables
Detailed summary statistics.
| First virologic failure | Second virologic failure | ||||
|---|---|---|---|---|---|
| Total (N=10,180) | No (N=8,002) | Yes (N=2,178) | No (N=4,900) | Yes (N=1,173) | |
| Baseline age | |||||
| Median (Q1, Q3) | 39 (32, 47) | 39 (32, 47) | 39 (31, 46) | 40 (33, 49) | 40 (33, 48) |
| BMI | |||||
| Median (Q1, Q3) | 23 (21, 25) | 23 (21, 25) | 23 (21, 25) | 23 (21, 25) | 23 (21, 25) |
| Missing | 4,413 (43.3%) | 3,278 (41.0%) | 1,135 (52.1%) | 1,949 (39.8%) | 523 (44.6%) |
| Days since baseline | |||||
| Median (Q1, Q3) | 240 (160, 370) | 200 (27, 340) | |||
| Days since first VL | |||||
| Median (Q1, Q3) | 230 (140, 370) | 220 (70, 350) | |||
| Prop days on ARVs | |||||
| Median (Q1, Q3) | 0 (0, 0.84) | 0 (0, 0.85) | 0 (0, 0.76) | 0.37 (0, 0.86) | 0 (0, 0.84) |
| Prop defaulted visits | |||||
| Median (Q1, Q3) | 0 (0, 0.10) | 0 (0, 0.10) | 0 (0, 0.11) | 0 (0, 0.10) | 0 (0, 0.091) |
| Num encounters | |||||
| Median (Q1, Q3) | 6.0 (2.0, 9.0) | 6.0 (3.0, 9.0) | 5.0 (1.0, 8.0) | 7.0 (3.0, 9.0) | 6.0 (2.0, 9.0) |
| Baseline WHO stage | |||||
| Median (Q1, Q3) | 1.0 (1.0, 3.0) | 1.0 (1.0, 2.0) | 2.0 (1.0, 3.0) | 1.0 (1.0, 3.0) | 2.0 (1.0, 3.0) |
| Missing | 6,330 (62.2%) | 4,926 (61.6%) | 1,404 (64.5%) | 3,027 (61.8%) | 746 (63.6%) |
| Baseline ARV line | |||||
| First | 5,282 (52%) | 4,202 (53%) | 1,080 (50%) | 2,092 (43%) | 553 (47%) |
| Second | 125 (1%) | 88 (1%) | 37 (2%) | 50 (1%) | 18 (2%) |
| Third | 13 (0%) | 12 (0%) | 1 (0%) | 1 (0%) | 0 (0%) |
| Missing | 4,760 (46.8%) | 3,700 (46.2%) | 1,060 (48.7%) | 2,757 (56.3%) | 602 (51.3%) |
| Gender | |||||
| Female | 6,810 (67%) | 5,405 (68%) | 1,405 (65%) | 3,237 (66%) | 741 (63%) |
| Male | 3,370 (33%) | 2,597 (32%) | 773 (35%) | 1,663 (34%) | 432 (37%) |
| Baseline weight | |||||
| Median (Q1, Q3) | 62 (56,70) | 62 (56,70) | 61 (56,68) | 62 (57,70) | 63 (56,69) |
| Missing | 4,414 (43.4%) | 3,279 (41.0%) | 1,135 (52.1%) | 1,950 (39.8%) | 523 (44.6%) |
| Height | |||||
| Median (Q1, Q3) | 170 (160, 170) | 170 (160, 170) | 170 (160, 170) | 170 (160, 170) | 170 (160, 170) |
| Missing | 4,604 (45.2%) | 3,431 (42.9%) | 1,173 (53.9%) | 2,062 (42.1%) | 557 (47.5%) |
| HIV status disclosed | 774 (8%) | 618 (8%) | 156 (7%) | 167 (3%) | 57 (5%) |
| On contraceptive | 7,479 (73%) | 5,996 (75%) | 1,483 (68%) | 4,151 (85%) | 956 (82%) |
| On health cover | 3,067 (30%) | 2,428 (30%) | 639 (29%) | 1,757 (36%) | 455 (39%) |
| On TB IPT regimen | 4,123 (41%) | 3,387 (42%) | 736 (34%) | 2,503 (51%) | 526 (45%) |
| STI symptoms | 255 (3%) | 196 (2%) | 59 (3%) | 125 (3%) | 42 (4%) |
| TB symptoms | 973 (10%) | 681 (9%) | 292 (13%) | 452 (9%) | 169 (14%) |
| Referral ordered | 6,055 (59%) | 4,754 (59%) | 1,301 (60%) | 3,399 (69%) | 819 (70%) |
| Referred to PHDP | 7,777 (76%) | 6,300 (79%) | 1,477 (68%) | 4,395 (90%) | 982 (84%) |
| Needs family TX support | 1,603 (16%) | 1,277 (16%) | 326 (15%) | 909 (19%) | 179 (15%) |
| Been hospitalized | 220 (2%) | 159 (2%) | 61 (3%) | 119 (2%) | 33 (3%) |
| Alcohol consumer | 1,367 (13%) | 1,074 (13%) | 293 (13%) | 726 (15%) | 163 (14%) |
| Cigarette smoker | 463 (5%) | 360 (4%) | 103 (5%) | 232 (5%) | 57 (5%) |
| Chest X-ray | 357 (4%) | 261 (3%) | 96 (4%) | 220 (4%) | 60 (5%) |
| General exam: abnormal | 1,127 (11%) | 855 (11%) | 272 (12%) | 636 (13%) | 178 (15%) |
| Skin exam: abnormal | 598 (6%) | 420 (5%) | 178 (8%) | 312 (6%) | 96 (8%) |
| Lymph nodes exam: abnormal | 122 (1%) | 83 (1%) | 39 (2%) | 54 (1%) | 13 (1%) |
| Respiratory exam: abnormal | 266 (3%) | 203 (3%) | 63 (3%) | 139 (3%) | 29 (2%) |
| Abdominal exam: abnormal | 73 (1%) | 62 (1%) | 11 (1%) | 38 (1%) | 6 (1%) |
| Urogenital exam: abnormal | 1,938 (19%) | 1,504 (19%) | 434 (20%) | 1,071 (22%) | 244 (21%) |
| Underweight | 1,655 (16%) | 1,153 (14%) | 502 (23%) | 756 (15%) | 225 (19%) |
| High BP | 4,301 (42%) | 3,585 (45%) | 716 (33%) | 2,187 (45%) | 416 (35%) |
| Low BP | 130 (1%) | 88 (1%) | 42 (2%) | 41 (1%) | 22 (2%) |
| Oxygen saturation: abnormal | 208 (2%) | 153 (2%) | 55 (3%) | 94 (2%) | 24 (2%) |
| Fever | 156 (2%) | 107 (1%) | 49 (2%) | 59 (1%) | 35 (3%) |
| MTRH clinic | 989 (10%) | 742 (9%) | 247 (11%) | 521 (11%) | 104 (9%) |
| Second VL count | |||||
| Median (Q1, Q3) | 0 (0.460) | 0 (0.180) | 730 (0, 21,000) | 0 (0, 83) | 9,900 (2,600, 61,000) |
| Missing | 4,107 (40.3%) | 3,262 (40.8%) | 845 (38.8%) | 0 (0%) | 0 (0%) |
| First VL count | |||||
| Median (Q1, Q3) | 0 (0.580) | 0 (0,60) | 16,000 (3,100, 95,000) | 0 (0.170) | 1,900 (0, 39,000) |
| Missing | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) |
TB, tuberculosis; ARV, antiretroviral; IPT, isoniazid preventive therapy; BP, blood pressure; PHDP, positive health, dignity, and prevention; TX, treatment; MTRH, Moi Teaching and Referral Hospital. The definition of each variable can be found in Appendix Table 3.
Variable description.
| Variable | Type | Description |
|---|---|---|
| Baseline age | Integer | Age at enrollment |
| Baseline arv line | Categorical | Antiretroviral (ARV) therapy line at enrollment |
| Baseline WHO stage | Ordinal | World Health Organization (WHO) staging at enrollment into care |
| Abnormal oxy sat | Binary | Presence of abnormal oxygen saturation: <90 |
| Been hospitalized | Binary | Indicator for patient hospitalized recently: <1 month |
| Fever | Binary | Baseline indicator for fever: temperature > 38 °C |
| High BP | Binary | Systolic > 130 , or diastolic > 80, or isolated systolic hypertension (age > 55) |
| Low BP | Binary | Low blood pressure (BP): >90 mmHg for systolic or >60 mmHg for diastolic |
| Referred to PHDP | Binary | Referral to Positive health, dignity, and prevention (PHDP) |
| Referral ordered | Binary | Indicator for any clinical referral e.g., nutrition clinic |
| STI symptoms status | Binary | Presence of sexually transmitted infection (STI) symptoms |
| TB symptoms | Binary | Presence of tuberculosis (TB) symptoms |
| Abdominal clinical exam | Binary | Abnormal abdominal physical clinical examination |
| Chest X-ray | Binary | Presence of abnormal chest X-ray radiograph |
| General clinical exam | Binary | Baseline indicator for abnormal general physical examination |
| Lymph nodes clinical exam | Binary | Physical assessment for positive swollen lymph nodes |
| Male | Binary | Binary indicator for male gender: 1 is male else 0 |
| vl2 failure | Binary | Binary indicator for second virologic failure |
| On contraceptive | Binary | Patient is on contraceptive (condom, pills, etc.) |
| On health cover | Binary | Patient has health cover/insurance |
| On TB IPT regimen | Binary | Patient is on/started TB isoniazid preventive therapy (IPT) |
| Respiratory clinical exam | Binary | Baseline physical examination/reported abnormal respiratory organs |
| Skin clinical exam | Binary | Baseline physical examination/reported abnormal skin lesion |
| HIV status disclosed | Binary | HIV disclosure status at baseline |
| Underweight | Binary | Body mass index (BMI) < 18.5 |
| Urogenital clinical exam | Binary | Baseline physical examination for abnormal urinary and genital organs |
| Needs family TX support | Binary | Patients needs family treatment (TX) support |
| Num encounters | Numeric | Number of HIV clinical encounters before first virologic measurements |
| BMI | Numeric | Body mass index – measure of body fat using height and weight |
| Prop poor adherence | Numeric | Proportion of clinical encounters with reported poor adherence |
| Prop days on arvs | Numeric | Proportion of days on ARVs medication |
| Days since first VL | Numeric | Number of days since first VL |
| VL count 2 log | Numeric | Logarithmic transformation of second VL count: log(1 + vl count) |
| Prop defaulted visits | Numeric | Proportion of missed appointments |
| VL count 1 log | Numeric | Logarithmic transformation of first VL count: log(1 + vl count) |
| Cig smoker | Binary | Patient reported cigarette smoking |
| Alcohol consumer | Binary | Patient reported alcohol consumption |
| MTRH clinic | Binary | Binary indicator whether patient’s primary clinic is Moi teaching and referral Hospital |
| VL1 failure | Binary | Binary indicator for first virologic failure |
References
Aggarwal, C. C. 2014. Data Classification: Algorithms and Applications, 1st ed. Chapman & Hall/CRC.10.1201/b17320Search in Google Scholar
Akanbi, M. O., A. N. Ocheke, P. A. Agaba, C. A. Daniyam, I. A. Emmanuel, E. N. Okeke, and O. U. Christiana. 2012. “Use of Electronic Health Records in Sub-Saharan Africa: Progress and Challenges.” Journal of Medicine in the Tropics 14: 1–6. Also available at: https://www.bmj.com/content/338/bmj.b375.Search in Google Scholar
Balzer, L. B., V. H. Diane, M. R. Kamya, C. Gabriel, E. D. Charlebois, T. D. Clark, C. A. Koss, D. Kwarisiima, A. James, N. Sang, J. Kabami, M. Atukunda, V. Jain, S. C. Carol, R. C. Craig, A. B. Elizabeth, M. van der Laan, and M. L. Petersen. 2019. “Machine Learning to Identify Persons at High-Risk of Human Immunodeficiency Virus Acquisition in Rural Kenya and Uganda.” Clinical Infectious Diseases, https://doi.org/10.1093/cid/ciz1096.Search in Google Scholar
Breiman, L. 1996. “Bagging Predictors.” Machine Learning 24 (2): 123–40, https://doi.org/10.1023/A:1018054314350.10.1007/BF00058655Search in Google Scholar
Chen, T., and C. Guestrin. 2016. “XGBoost: A Scalable Tree Boosting System.” In KDD ’16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–94.10.1145/2939672.2939785Search in Google Scholar
Chipman, H. A., E. I. George, and R. E. McCulloch. 2010. “BART: Bayesian Additive Regression Trees.” Annals of Appled Statistics 4 (1): 266–98, https://doi.org/10.1214/09-AOAS285.Search in Google Scholar
Feller, D. J., J. Zucker, M. T. Yin, P. Gordon, and N. Elhadad. 2018. “Using Clinical Notes and Natural Language Processing for Automated HIV Risk Assessment.” Journal of Acquired Immune Deficiency Syndromes 77 (2): 160–6https://doi.org/10.1097/qai.0000000000001580.10.1097/QAI.0000000000001580Search in Google Scholar
Friedman, J. H. 2001. “Greedy Function Approximation: A Gradient Boosting Machine.” Annals of Statistics 29 (5): 1189–232, https://doi.org/10.1214/aos/1013203451.Search in Google Scholar
Hamers, R. L., C. Kityo, M. A. L. Joep, T. F. R. de Wit, and M. Peter. 2012. “Global Threat from Drug Resistant HIV in Sub-Saharan Africa.” British Medical Journal 344, https://doi.org/10.1136/bmj.e4159.Search in Google Scholar
Hastie, T., R. Tibshirani, and J. Friedman. 2009. The Elements of Statistical Learning, 2nd ed. Springer.10.1007/978-0-387-84858-7Search in Google Scholar
Krakower, D. S., S. Gruber, K. Hsu, J. T. Menchaca, J. C. Maro, B. A. Kruskal, B. W. Ira, K. H. Mayer, and M. Klompas. 2019. “Development and Validation of an Automated HIV Prediction Algorithm to Identify Candidates for Pre-exposure Prophylaxis: A Modelling Study.” The Lancet HIV 6 (10): e696-704, https://doi.org/10.1016/s2352-3018(19)30139-0.Search in Google Scholar
Lundberg, S. M., and S.-I. Lee. 2017. “A Unified Approach to Interpreting Model Predictions.” In Advances in Neural Information Processing Systems, Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 4765–74. Curran Associates, Inc. Also available at: http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf.Search in Google Scholar
Marcus, J. L., L. B. Hurley, S. K. Douglas, S. Alexeeff, M. J. Silverberg, and J. E. Volk. 2019. “Use of Electronic Health Record Data and Machine Learning to Identify Candidates for HIV Pre-exposure Prophylaxis: A Modelling Study.” The Lancet HIV 6 (10): e688-95, https://doi.org/10.1016/s2352-3018(19)30137-7.Search in Google Scholar
McMahon, J. H., J. H. Elliott, S. Bertagnolio, R. Kubiak, and M. R. Jordan. 2013. “Viral Suppression after 12 Months of Antiretroviral Therapy in Low- and Middle-Income Countries: A Systematic Review.” Bulletin of the World Health Organization 91 (5): 377E–85E, https://doi.org/10.2471/blt.12.112946.Search in Google Scholar
Miller, W. C., K. A. Powers, M. K. Smith, and M. S. Cohen. 2013. “Community Viral Load as a Measure for Assessment of HIV Treatment as Prevention.” The Lancet Infectious Diseases 13 (5): 459–64, https://doi.org/10.1016/s1473-3099(12)70314-6.Search in Google Scholar
Moons, K. G. M., P. Royston, Y. Vergouwe, D. E. Grobbee, and D. G. Altman. 2009. “Prognosis and Prognostic Research: What, Why, and How?” British Medical Journal 338: b375, https://doi.org/10.1136/bmj.b375.Search in Google Scholar PubMed
Naimi, A. I., and L. B. Balzer. 2018. “Stacked Generalization: An Introduction to Super Learning.” European Journal of Epidemiology 33: 459–64, https://doi.org/10.1007/s10654-018-0390-z.Search in Google Scholar PubMed PubMed Central
Ndembi, N., R. L. Goodall, D. T. Dunn, A. McCormick, A. Burke, F. Lyagoba, P. Munderi, P. Katundu, C. Kityo, V. Robertson, D. L. Yirrell, A. S. Walker, D. M. Gibb, Charles F. Gilks, P. Kaleebu, and D. Pillay, Development of Antiretroviral Treatment. 2010. “In Africa Virology Group and Trial Team. Viral Rebound and Emergence of Drug Resistance in the Absence of Viral Load Testing: A Randomized Comparison between Zidovudine-Lamivudine Plus Nevirapine and Zidovudine-Lamivudine Plus Abacavir.” The Journal of Infectious Diseases 201 (1): 106–13, https://doi.org/10.1086/648590.Search in Google Scholar PubMed
Nguyen, P., T. Tran, N. Wickramasinghe, and S. Venkatesh. 2016. “A Convolutional Net for Medical Records.” IEEE Journal of Biomedical and Health Informatics 21 (1): 22–30, https://doi.org/10.1109/jbhi.2016.2633963.Search in Google Scholar PubMed
Petersen, M. L., E. LeDell, J. Schwab, V. Sarovar, R. Gross, N. Reynolds, E. H. Jessica, K. Goggin, C. Golin, J. Arnsten, M. I. Rosen, R. H. Remien, D. Etoori, B. W. Ira, J. M. Simoni, J. A. Erlen, M. J. van der Laan, H. Liu, and D. R. Bangsberg. 2015. “Super Learner Analysis of Electronic Adherence Data Improves Viral Prediction and May Provide Strategies for Selective HIV RNA Monitoring.” Journal of Acquired Immune Deficiency Syndromes 69 (1): 109–18, https://doi.org/10.1097/qai.0000000000000548.Search in Google Scholar
Raghupathi, W., and V. Raghupathi. 2014. “Big Data Analytics in Healthcare: Promise and Potential.” Health Information Science and Systems 2 (3), https://doi.org/10.1186/2047-2501-2-3.Search in Google Scholar PubMed PubMed Central
Ramesh, A. N., C. Kambhampati, J. R. T. Monson, and P. J. Drew. 2004. “Artificial Intelligence in Medicine.” Annals of the Royal College of Surgeons of England 86 (5): 334–8, https://doi.org/10.1308/147870804290.Search in Google Scholar PubMed PubMed Central
UNAIDS. 2018. Fact Sheet - World AIDS Day 2018: 2017 Global HIV Statistics. Also available at: http://www.unaids.org/sites/default/files/media_asset/UNAIDS_FactSheet_en.pdf.Search in Google Scholar
van der Laan, M. 2007. “Super Learner.” Statistical Applications in Genetics and Molecular Biology 6: 25, https://doi.org/10.2202/1544-6115.1309.Search in Google Scholar PubMed
Xu, Y., T. Liu, M. J. Daniels, R. Kantor, A. Mwangi, and J. W. Hogan. 2019. “Classification Using Ensemble Learning under Weighted Misclassification Loss.” Statistics in Medicine 38: 2002–12, https://doi.org/10.1002/sim.8082.Search in Google Scholar PubMed PubMed Central
Zheng, W., L. Balzer, M. van der Laan, and M. Petersen. 2017. “Constrained Binary Classification Using Ensemble Learning: an Application to Cost-Efficient Targeted PrEP Strategies.” Statistics in Medicine 37 (2): 261–79, https://doi.org/10.1002/sim.7296.Search in Google Scholar PubMed PubMed Central
Zhou, Z.-H. 2015. “Ensemble Learning.” In Encyclopedia of Biometrics, edited by S. Z. Li, and A. K. Jain, 411–16. Boston, MA: Springer US.10.1007/978-1-4899-7488-4_293Search in Google Scholar
© 2020 Walter de Gruyter GmbH, Berlin/Boston
