Types of studies

We plan to include cross‐sectional studies that assess the diagnostic accuracy of CEUS for the diagnosis of liver metastases in people with newly diagnosed colorectal cancer. We will include both non‐comparative studies (CEUS against the reference standard) and comparative studies comparing the index test (CEUS) and one or more comparators (CECT, MRI, or 18F‐FDG PET/CT) in the same study population, either by giving all participants the index test or by randomly allocating participants to receive the index test or one comparator against the same reference standard.
We will include studies irrespective of language or publication status, or whether data were collected prospectively or retrospectively. We will include studies presenting data on a per‐patient basis and exclude studies presenting data on a per‐lesion basis.

Participants

The study population will include adults aged 18 years or older with newly diagnosed colorectal cancer. We will not include people with known liver metastases and people who have already undergone hepatic resection or local treatment. We will exclude studies which assess the diagnostic accuracy of index tests in the detection of liver metastases from other origins, unless data are presented in a way which allows the population with colorectal cancer to be analysed separately.

Index tests

The index test is CEUS and the comparators are CECT, MRI, and 18F‐FDG PET/CT.

The characteristic feature of a liver metastasis on CEUS is a reduced contrast uptake in the portal venous phase and in the late phase, consequently giving the appearance of a hypoechoic lesion. In the arterial phase, liver metastasis can present as a hypervascular or hypovascular lesion. In the case of a hypervascular metastasis, the lesion presents with increased contrast uptake, giving it a hyperechoic aspect. A hypovascular metastasis will be hypoechoic, usually with a peripheral enhancement (Cantisani 2014; D'Onofrio 2015).

On non‐contrast CT, a liver metastasis is usually a low‐ or isoattenuated mass, when compared with the surrounding normal tissue. In the arterial phase, a hypervascular metastasis shows diffuse enhancement, while a hypovascular metastasis shows peripheral ring enhancement. In portal and venous phases, metastatic lesions show a "washout" phenomenon, a characteristic malignant feature, and a thick ring enhancement (Sica 2000; Lincke 2017).

Regarding MRI, although most metastases are hypo‐ to isointense on T1 and iso‐ to hyperintense on T2‐weighted images, typical hallmarks of liver metastases are peripheral ring enhancement, diffusion restriction, and hypointensity on hepatobiliary phase images (Namasivaya 2007;Karaosmanoglu 2016).

On PET imaging, a liver lesion is considered malignant when it demonstrates an area of focally increased FDG uptake, i.e. greater intensity than surrounding normal hepatic tissue uptake. A diagnosis of metastasis can be made by combining the malignant PET feature with malignant CT features (Rachh 2014).

Target conditions

The target condition is colorectal cancer liver metastases in people with newly diagnosed colorectal cancer.

Reference standards

We will accept a reference standard for the diagnosis of liver metastases as one of the following.

• In case of surgical resection, the pathology of the resected lesion

• In case of unresectable liver lesions, biopsy and histology of suspected liver lesions with follow‐up of at least three months to exclude any synchronous lesions not detected by the index test(s)

• In case of negative results on index test(s), a follow‐up with imaging for at least three months to confirm initial negative results.

None of the above‐mentioned reference standards are perfect. In case the reference standard is the pathology of the resected liver specimens, people with negative results of index test(s) do not undergo surgery and require an appropriate follow‐up. The same applies for people who undergo biopsy: those with initial negative index test(s) are not eligible to undergo biopsy. Therefore, a follow‐up with imaging is necessary. Ideally, the same imaging modality should be used for follow‐up as the initial evaluation. It is not possible that all participants undergo the same reference standard, introducing an unavoidable differential verification bias. However, we find it necessary that all participants in the same category (people with resectable liver metastases, people with non‐resectable liver metastases, and people with negative index test(s)) are evaluated by the same reference standard.

Electronic searches

We will conduct electronic searches in the Cochrane Hepato‐Biliary Group Controlled Trials Register and the Cochrane Hepato‐Biliary Group Diagnostic Test of Accuracy Studies Register (both are maintained and searched internally by the CHBG Information Specialist via the Cochrane Register of Studies Web), the Cochrane Library, MEDLINE Ovid, Embase Ovid, LILACS (Bireme), Science Citation Index Expanded (Web of Science), and Conference Proceedings Citation Index – Science (Web of Science) (Royle 2003; De Vet 2008). We will apply no language or document type restrictions. The time spans will be published for each of the separate databases at review stage. See Appendix 1 for preliminary search strategies.

Searching other resources

We will try to identify additional references by manually searching articles retrieved from digital databases and relevant review articles. We will seek information on unpublished studies by contacting experts in the field. In addition, we will handsearch abstract books from meetings over the past 10 years of the European Society for Medical Oncology (ESMO), The American Society of Colon and Rectal Surgeons and National Comprehensive Cancer Network (NCCN). We will also search for other kinds of grey literature in the System for Information on Grey Literature in Europe “OpenGrey” (www.opengrey.eu/).

Selection of studies

We will use Covidence to manage the selection of studies (Covidence 2017). Two review authors (ML and TAB) will independently scrutinise titles and abstracts identified by electronic literature searching to identify potentially eligible studies. Any citation that is identified by either review author as potentially eligible will be selected for full‐text review. The same review authors will independently assess full‐text papers for study eligibility, using predefined inclusion and exclusion criteria. We will resolve any discrepancies by discussion. We will record all studies after full‐text assessment, and their reasons for exclusion, in the 'Characteristics of excluded studies' table and illustrate the study selection process using a PRISMA diagram (Moher 2009).

Data extraction and management

Two review authors (ML and TAB) will independently complete a data extraction form for all included studies and retrieve the following data.

• General information: title, journal, year, publication status, and study design

• Surveillance programme or clinical cohorts

• Sample size: number of participants meeting the criteria and total number of participants screened

• Baseline characteristics: baseline diagnosis, age, sex, ethnicity

• Index tests with predefined positivity criteria and target condition

• Order of tests

• Time between tests

• Reference standard tests

• Numbers of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) findings. We will extract these data for liver metastasis of any size, number, and resectability.

We will summarise the data from each study in 2 x 2 tables (TP, FP, FN, TN), according to the index tests considered, and we will enter the data into Review Manager 5.3 software (Review Manager 2014).

Assessment of methodological quality

Two review authors (ML and TAB) will use QUADAS‐2 for the assessment of the methodological quality of the studies (Whiting 2011; Appendix 2). The QUADAS‐2 items have been incorporated into Review Manager 5 (Review Manager 2014). Appendix 2 contains definitions on when to answer yes, no, or unclear to the signalling questions within the QUADAS‐2 items, as well as definitions on when the risk of bias should be considered high, low, or unclear (Appendix 2). We will classify a study as having a high risk of bias if at least one of the domains of QUADAS‐2 is judged as being high risk. We will resolve any disagreements between the two review authors concerning the methodological quality of the studies by consensus. In case of disagreement, a third author (the arbiter) will make the final decision.

Statistical analysis and data synthesis

We will present each index test in each study as binary data in a 2 x 2 table. The test results from the studies have to be reported as true positive (TP), true negative (TN), false positive (FP), or false negative (FN). We will tabulate and graphically present these values from the selected studies in coupled forest plots (including 95% confidence intervals (CIs)). We will also estimate the positive and negative predictive values, and the positive and negative likelihood ratios (LR+ and LR‐), with 95% CIs. Furthermore, we will plot the results on a receiver operating characteristic diagram (ROC, sensitivity against 1 ‐ specificity). We will present each of the four index tests in their own ROC space, with the data available from each study. Since we expect a common implicit cut‐off among studies, we will use the bivariate model to pool sensitivities and specificities and to estimate the summary operating point (i.e. mean sensitivity and specificity) for each index test. We will perform direct and indirect comparisons by adding the four index tests as covariates to the bivariate model (Reitsma 2005). We will assess the significance of differences in test accuracy by using the log‐likelihood ratio test for comparison of models with and without the index test covariate term. We will consider P values less than 0.05, two‐sided, as statistically significant. We will conduct all analyses and plots using Review Manager 5 (Review Manager 2014) and SAS software (SAS software 2008).

In case of non‐evaluable index test results (especially relevant for CEUS), we plan to analyse data according to the intention‐to‐diagnose (ITD) principle (Schuetz 2012). We will classify participants with non‐evaluable results as false positive if they had a negative reference standard, or false negative result on a positive reference standard. If data for the ITD analyses are not retrievable from the text, we will contact the study authors. If we receive no response, we will include the study in the analyses with data retrievable from the published manuscript and consider the study to be at high risk of bias.

Investigations of heterogeneity

Based on the variability in test accuracy among the studies, we plan to conduct subgroup analyses by adding covariates to the bivariate model concerning the use of different reference standards, different ways of selecting the study populations (e.g. different inclusion and exclusion criteria), different locations of the study populations (country, state, region), differences between age groups, or differences between men and women. Based on the variability in test accuracy among the studies concerning CEUS, we plan to conduct subgroup analyses by adding covariates to the bivariate model in order to investigate whether the variability is due to differences in clinician skills in the performance of CEUS. We will assess the statistical significance of each covariate term by using the log‐likelihood ratio test for comparison of models with and without the covariate.

We plan to investigate the following potential sources of heterogeneity.

• Study design (prospective compared to retrospective)

• Study date (studies conducted before the year 2000 compared to studies conducted after the year 2000) due to advancements in technology and change in diagnostic criteria

• Participant selection (participants recruited from planned screening programmes compared to clinical setting)

• Proportion of participants with resectable liver metastasis

• Maximum diameter of the largest liver lesion

• Differences in operator skills in CEUS performance, assessed by years of experience

• Different reference standards (studies using pathology of resected liver compared with studies using histology of hepatic lesion)

Sensitivity analyses

We plan to assess the effects of risk of bias of included studies on diagnostic accuracy by performing a sensitivity analysis in which we will exclude studies classified as being at high risk of bias in at least one of the domains of QUADAS‐2 (Appendix 2). In addition, we have defined the following signalling questions as being most relevant, and we plan to conduct a sensitivity analysis in which we will exclude studies at high risk of bias.

• "Did all participants receive the same reference standard?"

• "Were the reference standard results interpreted without knowledge of the results of the index test?"

We also plan to conduct a sensitivity analysis in which we will exclude studies published only in abstract or letter format.

Assessment of reporting bias

We do not plan to test for publication bias due to the lack of validated methods for diagnostic test accuracy reviews.