2.1.

Study Participants

Institutional review board approval was obtained, and the requirement for informed consent was waived.

We modeled the 3D-printed radiomic phantom after diseased tissues seen on CT scans of six unique patients. Four patients had pancreatic adenocarcinoma (PDAC), one patient had nonsmall cell carcinoma (NSCLC), and one patient presented with advanced hepatic cirrhosis. We chose these patient scans because of the heterogeneous appearance of the diseased tissues. The patients with PDAC and advanced hepatic cirrhosis received contrast-enhanced abdominal CT exams using a 64 slice CT scanner (Discovery CT750 HD; GE Medical Systems, Milwaukee, Wisconsin) with the following scan parameters: tube voltage of 120 kVp, noise index of 14, tube current modulation ranging from 220 to 380 mA, 0.7 s rotation time, and pitch of 0.984. The images were reconstructed using a $512\times 512$ matrix, the filtered back projection reconstruction algorithm, and a standard convolution kernel. The reconstructed slice thickness was 2.5 mm, with an interval of 2.5 mm. Intravenous contrast administration included 150 mL of iodinated contrast material at $4\mathrm{mL}/\mathrm{s}$ (Iohexol $300\mathrm{mgI}/\mathrm{mL}$, Omnipaque 300, GE Healthcare, Cork, Ireland), respectively. The tumors were manually outlined by radiologists on the axial scans [window/level: 400/40 Hounsfield unit (HU)] using Volume Viewer on Advantage volume share 7 (GE Medical Systems, Milwaukee, Wisconsin). The CT scan of the patient with NSCLC was from a publicly available dataset hosted by the Cancer Imaging Archive.^20,32,33 The patient was imaged on a 16-slice CT scanner (Lightspeed, General Electric, Madison, Wisconsin). Images were reconstructed using a standard and lung convolution kernel. The scan data from the standard convolution kernel were used in this study. Further details of image acquisition parameters are available in the associated publication.²⁰ Across all patient scans, the in-plane pixel size ranged from 0.695 to 0.977 mm ($\text{mean}=0.851\mathrm{mm}$). The tumors’ maximum diameter ranged from 21 to 61 mm, with a mean of 46 mm. The largest tumor diameters were manually measured on a transverse image plane viewed with Volume Viewer on Advantage volume share 7 (GE Medical Systems, Milwaukee, Wisconsin) by an experienced radiologist. The placement of the tumors within the background cirrhotic liver was arbitrary.

2.2.

Phantom Model Fabrication

Figures 1(a)–1(e) show a graphical overview of the workflow used to 3D print the radiomic phantom. The multimaterial 3D printer used in this study can simultaneously deposit up to three different photopolymer resins.³⁴ The resolution of a single droplet of resin in the $x\text{-}y$ direction and the layer thickness (the $z$-direction) is on the order of $48\times 84\times 30\mu \mathrm{m}$, which is smaller than the resolution of a typical CT scanner ($\sim 0.5\times 0.5\times 0.6\mathrm{mm}$).³⁴ The selection of printing material was determined by scanning several solid samples of available resin materials using the patient abdominal CT protocol described in Sec. 2.1 and measuring the HU values. The two materials that had the highest and lowest HU values were selected.

Fig. 1

Workflow to generate 3D-printed phantom. (a) The tumor was segmented from the patient CT exam. A cross-sectional slice from a single patient’s CT image shows the contoured PDAC. (b) Binary masks of the tumors were generated and then used to replace the background voxel values with the tumor voxel intensity values. (c) The combined volume was then supersampled to the resolution of the 3D printer and stacked into slices. Each slice from (c) was then dithered using the Floyd–Steinberg dithering algorithm into binary raster files. (d) Three sets of raster files were generated, one for each resin material. These files define the spatial location of each resin material. (e) Resultant 3D print of the combined volume. Due to the material used, visualizing the internal structure is not possible with the naked eye.

At first, an experienced radiologist segmented each tumor and a circular portion of the cirrhotic liver from the scans at their original resolution. The segments were then individually normalized to have voxel intensity values between 0 and 1. The normalized ratios were used to determine the proportion of resin material deposited into a single voxel.³⁵ Binary images of the individual tumor volumes were then used to mask an area over the cirrhotic liver images [Fig. 1(b)]. Next, we merged the volumes without modifying the original HU values or the normalized ratios of each scan. The combined slices, as shown in Fig. 1(b), were then supersampled using the Whittaker–Shannon (SINC) interpolation method to the resolution of the 3D printer [Fig. 1(c)]. Finally, each slice was dithered using the Floyd–Steinberg dithering algorithm into binary raster files.³⁶ The raster files encode the spatial location over which each material is deposited [Fig. 1(d)]. The disc shape of the phantom was designed to have a diameter of 18 cm and thickness of 4 cm. These dimensions were selected so that the disc would fit within a tissue equivalent enclosure that was originally used with a commercially available low-contrast helical CT QC phantom (model 061, CIRS).

Since the printer can simultaneously print with three different resin materials, three sets of bitmap files were generated, one set for each resin material. Within the first raster file, a value of 1 indicates the deposition of material A, and a value of 0 indicates that material A will not be deposited. The second set of raster files (material B) was generated by inverting material A files so that a value of 0 now had a value of 1. The third set of bitmaps consisted of all zeros since two materials with opposing densities were enough to generate the desired contrast differences. The resulting 3D print is shown in Fig. 1(e).

2.3.

Computed Tomography Scan Modes

A 64 slice CT scanner (HD750, General Electric, Madison, Wisconsin) was used to acquire 30 repeat scans of the radiomic phantom. The scanning parameters were as follows: 120 kVp, 280 mA, 0.7 s, pitch of 0.984, filtered back projection algorithm with a standard kernel, total collimation of 40 mm, display field of view (DFOV) 250 mm, reconstructed slice thickness, and interval of 1.25 mm. The phantom was centered in the gantry using the system onboard laser alignment lights. The associated volume CT dose index was 15.96 mGy. The average radiomic feature values determined from this protocol were considered the reference in percent deviation (pDV) calculations.³⁷ The deviation from the reference was determined by rescanning the phantom, sequentially, five times, without movement between scans using the scan modes listed in Table 1. All parameters of the reference protocol remained fixed while each scan mode was implemented. Figure 2 shows cross-sectional axial slices of the radiomic phantom scanned using each additional scan mode. In addition to commonplace scan modes, such as different tube potentials and currents, we evaluated QR feature robustness with adaptive statistical iterative reconstruction (ASiR), and the phantom positioned vertically off-center by 30 mm in the inferior direction. The latter³⁸ is a practice commonly observed in the clinic. We evaluate its impact on QR features in this study because the off-center placement of the patient within the CT gantry misplaces the thickest portion of the bow-tie filter relative to the patient’s anatomy, which leads to increased beam hardening artifacts and, consequently, increased noise or variability of CT HU values.³⁹ ASiR is a feature that reduces the pixel noise standard deviation while preserving structural detail and is available in 10 different strengths. As the strength or percentage of ASiR increases, the noise magnitude decreases, noise texture becomes coarser and more uniform, and images generally appear smoother.^40,41 The strengths and combinations that we use in this study are based on what we use in our clinic. The DFOV dictates the pixel size in the $x\text{-}y$ direction. As DFOV increases, the size of the pixel increases, and the resolution in the $x\text{-}y$ direction decreases. The rapid switching dual-energy CT variant used in this study acquires two projections nearly simultaneously while operating at a low and high peak tube potential of 80 and 140 kVp. With the two projections, reconstructing many image types is possible; these include virtual monochromatic images (VMI) that depict anatomy from the viewpoint of a monochromatic x-ray source ranging in energy from 40 to 140 keV or material density images. For this study, we chose to evaluate radiomic feature robustness on DECT scans reconstructed with a VMI of 60 keV. The scan modes were chosen due to their use in the clinic.

Table 1

Additional scan modes included in the study. The options were chosen due to their frequent use in the clinic.

Fig. 2

Cross-sectional CT images of the radiomic phantom from each scan mode evaluated in this study. For each image, a window width of 40 and a level of 100 were applied. The differences between each scan mode are (a) baseline image reconstructed with standard kernel. (b) ASiR of 10%. (c) ASiR 20% and (d) ASiR 30%. (e) Lung kernel. (f) Lung kernel with ASiR 40%. (g) Bone kernel. (h) Reduced tube current of 100 mA. (i) Reduced tube potential of 100 kVp. (j) Dual-energy CT image reconstructed at monochromatic 70 keV. (k) Enlarged DFOV of 350 mm with a pixel size of 0.689 mm. (l) Phantom placed off-center by 30 mm. The off-center image was electronically centered within the field of view.

2.4.

Radiomic Feature Extraction

The computational environment for radiobiological research (CERR)⁴² was used to extract the prognostic QR features listed in Table 2. We chose these features because previous literature reports illustrated their potential prognostic capabilities for NSCLC and PDAC. They were extracted from the original images without any preprocessing, such as image smoothing or interpolation of voxel sizes, and the settings used for feature calculation were as follows: (1) the images were discretized using a fixed bin width of 25. (2) The average value of each texture feature was computed over all 13 directions to obtain rotational invariance. (3) For the prognostic PDAC QR features, images were discretized using a bin width of 25 and a patchwise volume of $2\times 2\times 2\mathrm{mm}$ voxels. Detailed descriptions of the feature definitions can be found in Refs. 4243.–44.

Table 2

Prognostic radiomic features analyzed for repeatability and deviation relative to reference values.

2.5.

Statistical Analysis

The structural similarity index (SSIM)⁴⁵ was used to calculate the similarity between the original cirrhotic liver and the resulting 3D print. Repeatability (i.e., precision) of radiomic features was evaluated using the within-subject coefficient of variation (wCV, %):⁴⁶

The pDV (%) of radiomic feature derived from the additional scan modes was calculated as follows:

The one-sample Wilcoxon signed-rank test was used to determine equality between the reference feature and the median feature value derived from the additional scan modes. A $p$-value of $<0.05$ was considered significant. The effect size was calculated as

3.1.

Hounsfield Unit of Printing Materials and Structural Similarity

Figure 3 shows the resulting 3D-printed phantom, a cross-sectional CT scan, and the physician drawn contours overlaid onto each tumor. The overall time taken to 3D print the phantom was $\sim 8\mathrm{h}$. The final radiomic phantom was circular, with a measured diameter of $176\pm 0.2\mathrm{mm}$ and an axial length of $42\pm 0.2\mathrm{mm}$. The two-resin materials with the lowest ($65\pm 5\mathrm{HU}$) and highest ($125\pm 5\mathrm{HU}$) CT numbers were VeroWhite (material A) and TangoPlus (material B). The SSIM between the cirrhotic liver background [Fig. 4(b)] and resultant 3D print [Fig. 4(c)] was 0.71. An SSIM value closer to 1 suggests more similarity between images.

Fig. 3

(a) The 3D-printed radiomic phantom. (b) Axial slice generated from a CT scan shows the embedded tumors within the background tissue. (c) The tumor contours were generated, and the tumor types were labeled as 1–Non-small cell lung carcinoma; 2 to 5–Pancreatic ductal adenocarcinoma.

Fig. 4

(a) An axial slice from a patient CT showing the region of interest (ROI) around the heterogeneous hepatic tissue. (b) A cropped and expanded view of the portion of the patient’s liver on which the background of the 3D print was modeled. (c) An axial slice of the resulting 3D print CT scan.

3.2.

Repeatability and Percent Deviation

The repeatability of the prognostic NSCLC and PDAC radiomic features is shown in Fig. 5, where wCV (%) $<1.0\%$ across features. Figure 6 shows the pDV for the NSCLC radiomic features. The average pDV of first-order energy was 0.01% (range: $-0.49\%$ to 0.89%, $p=0.290$) across all scan modes. The average pDV of GLRLM gray-level non-uniformity (GLN) was 10.2% (range: $-55.2\%$ to 5.57%, $p=0.108$) [Fig. 6(b)]. The application of ASiR 40% to images reconstructed with the lung kernel resulted in the pDV for GLRLM GLN decreasing by a factor of 2, from $-30\%$ to $-15\%$. The average pDV for HLH GLN pDV was 15.7% (range: $-56.3\%$ to 0.52%, $p=0.007$). Similar to GLRLM GLN, pDV for HLH GLN decreased when ASiR 40% was applied to images reconstructed with the lung kernel. Figure 7 shows the pDV of prognostic PDAC radiomic features across scan modes. With the application of ASiR 10% to 30%, the pDV for contrast, dissimilarity, and entropy increased in the negative direction for all tumors, but overall, the deviation remained below 30%. Across all tumor types, the pDV for GLCM-contrast and GLCM-dissimilarity exceeded 40% and 20% when the phantom was scanned with the reduced dose scan modes and with the application of DECT. In addition, the pDV for GLCM-contrast was $\le 10\%$ when the phantom was scanned with a larger pixel size of 0.689 mm and the different strengths of ASiR.

Fig. 5

The within-subject coefficient of variation (wCV, %) for prognostic non-small cell lung carcinoma radiomic features extracted from each tumor. The wCV was computed from the 30 repeated CT scans acquired with the reference protocol. The 95th percentile confidence intervals are displayed for each feature value.

Fig. 6

The percent deviation (pDV, %) and one-sample Wilcoxon signed-rank test compare the prognostic NSCLC features. Comparisons are being made between the average feature value derived from the 30 repeat scans and the additional scan modes. (a) First-order energy, (b) gray- level non-uniformity (GLN), and HLH wavelet GLN.

Fig. 7

Dot plots of the change in radiomic feature values as a function of each scanning technique and tumor type.