Study designs.

Any peer-reviewed primary studies which assessed a prediction algorithm that utilizes ML techniques applied to clinical problems in the prediction and diagnosis of chronic kidney disease development and progression, including those for CKD screening, CKD prevention, profiling of biomarkers contributing towards CKD, profiling of risk factors leading to CKD, estimation of glomerular filtration rate (GFR) and creatinine levels, prediction of occurrence of CKD, prediction of CKD stages, CKD diagnosis, CKD prognostication, prediction of CKD progression to ESKD and/or requirement for renal replacement therapy, ESKD diagnosis will be included.

Exclusion criteria are 1) Studies that utilize only image-based inputs as the different model development processes require alternative extraction and appraisal tools; 2) studies assessing prediction models of CKD complications other than its progression, including non-exhaustively anemia, electrolyte disturbances, bone disorders, and cardiovascular events; 3) prediction model of RRT including non-exhaustively the choice of RRT modalities which are hemodialysis, peritoneal dialysis, and renal transplantation, and 4) studies reporting only treatment-related outcomes of CKD such as adverse events, rate of complications, the management of complications; 5) informal publication types such as case studies, commentaries, letters to the editor, editorials, meeting abstracts, proceeding papers, conference abstracts, protocols, guidelines, and recommendations; 6) review articles such as narrative review, overview, systematic review, meta-analysis; 7) studies that include participants < 18 years old; 8) animal studies.

Study participants.

Adult humans whose age was equal to or more than 18 years old.

Types of interventions.

The studies will present prediction models utilizing ML techniques, including non-exhaustively various regression techniques, decision trees, random forests, support vector machines, K-nearest neighbor, and neural networks, as defined by individual studies. The models will be for the prediction and diagnosis of chronic kidney disease development and progression with or without mention of ESKD.

Comparators.

We will include studies that compare the performance of ML-based prediction models with those that utilize conventional techniques, including non-exhaustively those that use logistic regression (including penalized LR), cox regression, Poisson regression, least squares linear separation, generalized additive models, discriminant analysis, generalized estimation equations, risk scores, and expert views. Studies that utilize only ML-based tools will also be included.

Primary outcome.

• Performance comparison of ML-based and non-ML-based prediction tools in CKD development and progression

Secondary outcomes.

• ML-based model use case
• Performance of ML-based prediction tool in CKD development and progression
• Stages of model development (internal or external validation or clinical implementation)
• Model development team specialty and the involvement of model end-user such as clinicians during model development
• Evidence of model reporting quality description
• Characteristics of the dataset (size of training, validation and testing datasets, source of dataset, population group, data period, length of follow-up)
• Prediction model construct including ML-based and non-ML based techniques
• Predictor characteristics and selection
• Model outcome characteristics and selection
• Model performance measures used

Data management.

Following database searching, studies will be populated into Covidence systematic review software [29], which will manage study selection and data extraction.

Selection process.

We will carry out two stages of screening. After study de-duplication through Covidence, two reviewers from a team of eight reviewers will screen the titles and abstracts of potential studies independently. We will eliminate abstracts in the initial screen if they do not report ML-based prediction models in CKD.

Included studies will undergo full-text review against the full eligibility criteria. Reasons for exclusion will be recorded for each study. Disagreement between two reviewers at each article screening and selection stage will be resolved by consensus and a third person if necessary. The PRISMA 2020 flow diagram will be generated to describe the workflow and identification of included studies for the systematic review [30].

All reviewers will receive prior training on systematic review methodology, introduction to machine learning in healthcare and the study protocol including the eligibility criteria from lead reviewers (FC, PK and KP). Each reviewer will screen with the lead reviewer (FC) for the initial 20 articles to ensure consistency. Regular team meetings will be held to resolve conflicts and ensure consistent validity across team members.

Screening Interrater reliability will be calculated in both percentage agreement and Cohen’s Kappa [31].

Data collection and management.

The data extraction form will be designed prospectively before data collection and will be pilot tested and refined. Five members of the reviewer team who will participate in data extraction process will receive dedicated instructions and supplementary coding manual with explanations and examples of variables from the lead reviewer (FC). During the pilot phase, the reviewer team will independently extract a random sample of five studies and review inter-rater reliability. Team training and piloting will continue until a Cohen’s kappa of 0.60 (Moderate) is reached. Two independent reviewers from the reviewer team will then begin formal extraction process of the following data items based on items included in the CHARMS checklist: 1) source of data; 2) participant information; 3) outcome(s) to be predicted; 4) candidate predictors; 5) sample size; 6) Missing data; 7) Model development; 8) Model performance; 9) Model evaluation; 10) Results (final model presented) including model performance; 11) interpretation and discussion. In addition, information will be extracted on the study information (authors, year of publication, study design, journal, contact information, study period, geographical location (area and country), and funding), the assessment of reporting standards using an objective measure if mentioned in the study, and any other relevant information. All relevant text, tables, and figures will be examined for data extraction. Disagreements between the two independent reviewers will be resolved by consensus and a third reviewer if necessary. We will contact the study authors to request incompletely reported data in included studies. We will conduct analyses using available data if no response is received within 14 days.

Extraction interrater reliability will be calculated in both percentage agreement and Cohen’s Kappa [31].

Reporting quality assessment.

We will assess the reporting quality of studies against the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) statement, which aims to improve the transparent reporting of prediction modeling studies in all medical settings [32].

The TRIPOD statement provides recommendations for reporting studies on developing, validating, or updating a prediction model. To assess the completeness of reporting amongst each publication, we will utilize the published "TRIPOD Adherence Extraction Form,"which evaluates 22 main items deemed essential in evaluating the transparency of prediction model studies [33]. Each article will only be assessed for items and sub-items that it applies to (development, external validation, or incremental value reporting of prediction models) based on guidance from the adherence form. Each TRIPOD item is given adherence elements to help evaluate an item. The presence or lack of an adherence element within the article will be marked down either as a "Yes" or a "No." For a TRIPOD item to receive a score, all adherence elements must be present. Overall article’s TRIPOD score can be calculated by summing up adhered TRIPOD items and dividing by the total number of applicable TRIPOD items for that article. Findings on reporting quality from the TRIPOD adherence extraction will be summarised and graphically presented.

Risk of bias assessment.

We will assess the risk of bias (ROB) of studies by applying PROBAST (prediction model risk of bias assessment tool) [34]. PROBAST was developed to assess ROB and applicability concerns of a study that evaluates (e.g., develops or validates) a multivariable diagnostic or prognostic prediction model. Reviewers will assess each study based on the published "PROBAST Assessment Form" [35]. PROBAST is organized into four domains: participants, predictors, outcomes, and analysis. A ROB rating (high, low, or unclear) will be assigned to each domain based on answering signaling questions provided by the PROBAST assessment form. Signaling questions can be answered as yes, probably yes, no, probably no, or no information. Based on ROB results from the four domains, an overall ROB rating and prediction model applicability rating will be given to the prediction model following the recommendations in the PROBAST assessment form. A tabular presentation on PROBAST results for each study will be available. Results will be summarized and graphically presented for each domain.

Qualitative synthesis.

We will summarize and analyze the studies that meet the eligibility criteria by themes following the primary and secondary outcomes. We will report details of prediction model performance, comparing ML-based and non-ML-based models, the prediction model use case, choice of predictors and outcomes, ML model construct, prediction model reporting standard, and risk of bias. In addition, we will analyse the studies and their results following standard 4.2 –Conduct a qualitative synthesis, chapter four of Finding What Works in Health Care: Standards for Systematic Review [38], which involves the description of the clinical and methodological characteristics of individuals studies, including their strength and weaknesses, and their relevance to the particular populations and clinical settings.

Quantitative synthesis.

The studies will likely show significant clinical, methodological and statistical heterogeneity. We will therefore synthesize data quantitatively in appropriate subgroups (prediction of CKD development, CKD diagnosis, and prediction of CKD progression to ESKD), guided by Collins et al 2022 –specific to the conduct of systematic reviews of prediction models, including the methods for quantitative synthesis [39].

Measure of effect size.

Performance measures of the ML algorithms will be recorded, including AUROC (or c statistics), 2x2 confusion matrix, sensitivity and specificity to summarize discrimination, and observed vs. expected ratio (O:E) to summarize calibration, or any other measures utilized by individual studies.

Assessment of heterogeneity.

We will assess the clinical heterogeneity of studies based on the ML algorithm use case, the participant characteristics, the predictor choice, and selection. We will assess the methodological heterogeneity based on the ML algorithm construct regarding the data size, ML technique, and performance measures. We will assess statistical heterogeneity using the χ² test and the I² statistic. We will consider an I² value greater than 50% indicative of substantial heterogeneity.

Quantitative data synthesis.

As recommended by Collins et al 2022 [39], random-effects meta-analysis will be used to summarize estimates of model discrimination and calibration.

In order to compare the performance between ML and non-ML methods, we will utilize methods described in Christodoulou et al. [40] by analyzing pairwise differences in logit AUROCs between ML-based and non-ML based techniques by random effects modeling by DerSimonian and Laird method, either pooled or within subgroups stratified by the risk of bias, study outcomes, ML techniques. The estimate and the 95% CI will describe the pairwise differences in logit AUROCs. The average logit(AUC) difference and 95% CI pooled and stratified by the above subgroups will be calculated as an indicator of how prediction models utilizing ML compare with those utilizing non-ML methods.

The meta-analysis will be performed using Review Manager version 5.4.1 (The Cochrane Collaboration, The Nordic Cochrane Centre, Copenhagen, Denmark) [41] and Stata [42].

Additional analysis.

Further subgroup analyses will be performed to explore possible sources of heterogeneity based on the following: study quality (risk of bias and reporting quality assessment), ML techniques, stages of validation, and dataset (stratified as determined by resulted studies).

We will conduct sensitivity analyses based on study quality, study publication years (stratified by year), study populations, the ML model construct, the ML user case, or any other relevant strata.