Image level classifications usually aim to determine the presence of disease or a specific pattern. The goal in medicine is to utilize the medical images and reports as inputs and outputs of the deep learning model, respectively, to automatically learn important features and biomarkers. For example, a 3D CNN model to determine the degree of Alzheimer's disease progression from structural brain MR images has been proposed [8]. In the study, they used a transfer learning method where features learned through pre-training a CAE with a small number of source domain images were used to fine tune the target domain data, which was then used to train the actual classification model. Their method achieved a better classification performance than a supervised training method directly using the target domain data.

In another study, a slice-level classification model was proposed to detect interstitial lung diseases (ILD) from chest CT scans [9]. Most of the existing ILD detection models require a patch-level manual annotation to train a patch-based classification model. However, because ILD lesions are inherently a mixture of various patterns and have unclear borders, it is very labor intensive to obtain a large amount of patch-based annotations. Hence, the proposed study used a CNN model pretrained with natural images to create slice level predictions for each CT scan, and because multiple ILD subtypes can exist in a single scan, they used a multi-label classification loss which resulted in a high classification accuracy.

Recently, there have been studies focusing on visualizing the evidence of model predictions to overcome the limitations in explainability or interpretability, which has been pointed out as the weakness of deep learning models. Jamaludin et al. trained the multi-task learning model which detects several diseases simultaneously from spinal MR images and visualized salient regions in the image for corresponding predictions as ‘evidence hotspots’ as seen in Fig.1A [10]. This method visualizes the sensitivity of prediction by the change of input pixel value which can be computed as the partial derivative of model output with respect to every input pixel via back propagation. Although this model can visualize details at the pixel level and is relatively fast as it only requires single backpropagation, the output may not be intuitive and can be hard to understand because the salient pixels tends to be spread over a large area of the input image. To overcome this limitation, a visualization method based on prediction difference analysis which quantifies the difference between the output of the original image and the output when we marginalize out the region of interest, was proposed [11]. In essence, it is similar to the occlusion method in that they both look at the output difference as we apply changes to the region of interest [12]. However, while the naive occlusion method replaces the region of interest with simply zero values, the newly proposed method replaces the region with samples from the area surrounding the region of interest. This method produces a more intuitive and useful visualization of the evidence base of a model that classifies HIV positive cases and healthy cases from brain MRIs as seen in Fig.1B.

Many methods have been proposed to overcome the fact that although we have an enormous repository of medical images, most of them lack expert annotations, which are very expensive and time consuming to produce. The most basic approach is to fine tune with a small set of labeled data after extracting relevant features from unlabeled data via unsupervised learning. Cheng et al. obtained a performance better than the current CADx system by pretraining a stacked denoising autoencoder (SDAE) to determine the malignancy of a lesion from chest CT images [13].

Another way to address the lack of annotations in medical images is to use CNN models pre-trained for natural images. Shin et al. tested several well-known CNN architectures and showed that such transfer learning approaches improve performance in CT patch based thoraco-abdominal lymph node detection and ILD classification [14].

In lung nodule detection, which is a major target for CADe systems, the task is divided into candidate detection and false positive reduction. While it is possible to achieve high sensitivity in the candidate detection step due to the well-known features in chest CT images, false positive reduction tends to be more challenging due to the variations in the nodule morphology and size. Hence, the false positive reduction stage is considered the determining stage of the final CADe performance. Dou et al. achieved a false positive rate lower than that of the conventional method through a multilevel contextual 3D CNN that discriminates nodules by fusing extracted features from multi-scale input patches [15].

Ciompi et al. proposed a single system that goes a step further than a CADe system by classifying nodules based on the morphology (solid, non-solid, part-solid, calcified, perifissural and spiculated) for automatic Lung-RADS reporting and malignancy estimation [16]. As seen in Fig 2, they performed data augmentation by extracting multiple view 2D patches from 3D nodule candidate voxels and used a multi-stream CNN architecture that fuses features extracted from multi-scale patches. The study showed a classifier performance better than that of support vector machines (SVM) based on hand-crafted features or features from unsupervised learning models. The result was within the inter-observer variability range of experts, hence providing a possibility of developing a Lung-RADS based automatic pulmonary nodule management system.

In a study by Ghafoorian et al., a CNN model was trained to detect lacunes from T1 and FLAIR brain MRI patches that are closely related to neurodegenerative disorders [17]. By changing the fully-connected layers of the trained patch-based model to convolution layers, they could effectively create a lacunes probability map of the whole brain MRIs. False positive reduction was then performed by training a multi-scale 3D CNN on the extracted candidate voxels. At this stage, they added the distance between the voxel and brain landmarks as contextual information to further enhance the false positive reduction and confirmed that the proposed CADe system improves diagnostic accuracy through an observer study.

The goal of anatomical structure segmentation is to determine the presence or progression of disease through quantitative analysis of the volume and length data of segmented organs and organ substructures. For example, the size of the infant hippocampus is an important index for early brain development, however it is very difficult to accurately segment infant brains using the conventional predefined features used for adults because of poor tissue contrast on infant brain MR images. Guo et al. proposed a segmentation method where sparse patch matching was implemented based on features learned from the complementary information of T1 and T2 weighted MR images by a stacked auto-encoder [18].

Airway segmentation in thorax CT images can be used to support diagnosis by detection and quantification of bronchial wall thickening and changing lumen diameter. Airway segmentation can also increase the accuracy of the segmentation of other structures within the thorax and lower the false positive rate in lung nodule detection. Though there have been many proposals for airway segmentation, most studies have focused on achieving high sensitivity of detecting airways, which led to a high false positive rate [19]. To tackle this problem, Charbonnier et al. trained a CNN to classify leak candidates into airways and leaks from multiple initial airway segmentation results by changing the parameters of the segmentation algorithm [20]. By combining the classified results at the end, they could propose a method that successfully reduced false positives while maintaining airway detection sensitivity.

Organ segmentation in abdominal CT and MR images is also an active area of research. Hu et al. trained a 3D CNN model to accurately segment livers with large variations in shapes or those that have fuzzy borders with adjacent organs or lesions [21]. They proposed a method where probability map produced by the 3D CNN model is used as a shape prior to a global energy function.

For kidney segmentation in CT angiograms, Thong et al. trained a 2D patch based CNN to determine if the center pixel of the patch is part of the kidney [22]. By creating and interleaving pooling layers of various offsets, they could compensate the low-resolution feature maps of deep pooling layers and create a high-resolution probability map of the kidney.

Roth et al. proposed a two-stage method for the segmentation task of the pancreas [23]. During the detection stage, a 3D bounding box is created by the holistically-nested convolutional network on the axial, sagittal and coronal views of CT volumes. In the second stage, a more accurate final pancreas segmentation result is obtained by integrating the mid-level information of CNNs trained to segment the interior and boundary of the pancreas within the bounding box proposed in the first stage.

Yu et al. proposed a 3D volumetric CNN for prostate segmentation on MR images where they expanded the U-net, widely used for 2D biomedical image segmentation, into 3D and added residual connections to combine multiple-scale information [2425].

Lastly, Poudel et al. proposed using a recurrent full-convolutional network to segment the left ventricle from cardiac MR images [26]. Recurrent architectures model the spatial dependency between adjacent 2D short-axis slices, and this study showed improved segmentation results by sharing the high level global feature, which connects the encoder and decoder in a U-net structure, between these slices.

Lesion segmentation is an important step for planning treatment and predicting prognosis. The major challenge in lesion segmentation rises because multiple lesions of various shapes can be located anywhere within the organ. Furthermore, because most of the organ is a non-lesion space, the inequality of class distribution makes the segmentation task even more challenging. To overcome such challenges, various methods have been proposed.

Brosch et al. proposed a 3D encoder network for multiple sclerosis lesion segmentation in brain MR images [27]. They could accurately segment the lesion by fine tuning a U-net shaped encoder-decoder network based on an encoder pretrained by a stacked restricted Boltzmann machine.

In a study done by Ghafoorian et al., a network architecture based on multi-scale T1 and FLAIR MR patches was proposed to segment and quantify white matter hyperintensities, which are related to various brain disorders [28]. When they compared various structures that combine data from multiple scales, the multi-scale late fusion with weight sharing (MSWS) structure, which shares the CNN feature extraction weights and fuses the features for the final segmentation, showed the best performance. They also used the 3D coordinates and distance to major anatomical landmarks of each patch to provide anatomical prior knowledge. They showed that the use of such explicit spatial location features in addition to the contextual features of the patches can achieve a more accurate lesion segmentation.

Vivanti et al. proposed a method of liver tumor segmentation in follow up CT scans, which is composed of ROI (region of interest) selection based on deformable registration between the baseline and follow-up CT scans and a CNN trained to classify the voxels within the ROI as tumors and non-tumors. Finally, by removing segmentation leaks and holes, they could successfully segment tumors in follow up CT scans from baseline CT scans with high accuracy.

The recently trending deep learning method known as the generative adversarial network (GAN) has also been actively applied in the medical domain. GAN is a form of artificial neural network which consist of two sub-networks such as generator and discriminator. Two sub-networks are trained in adversarial manner such that the fake examples generated by generator are indistinguishable from the real examples while the discriminator tries to maximize its discrimination performance. While GAN is mostly used for the creation of synthetic images in the field of natural image processing, it has shown promising results in segmentation and conversion of medical images. Kohl et al. proposed the GAN based segmentation method for the detection of aggressive prostate cancer in MR images [29]. As seen in Fig 3, a model is trained to distinguish between expert segmentations and model-generated segmentations. The results are then fed back to the training process of the generator. Through this process, the segmentation generator was trained to mimic the segmentations of experts and achieved a significant performance enhancement compared to the conventional segmentation methods.

The quality of medical images is a very important factor in medical image analysis. However, to obtain high quality images, patients have to be exposed to high radiation doses or use extra expenses and time. That is why research in enhancing low dose CT images to the quality of normal dose CT images and low quality MR images to high quality images are being actively pursued.

Chen et al. proposed adding a residual skip connection to a U-net form encoder-decoder network to enhance the quality of low-dose CT scans to that of normal dose CT scans [34]. Here, the encoder reduces noise and artifacts and the decoder restores the structural information within the CT image. Also, the residual skip connection supplements the details lost while passing through multiple convolution and deconvolution layers to eventually generate an enhanced CT image, much better than that of a conventional enhancement method.

In a study on enhancing 3T MR images to 7T MR images, Bahrami et al. proposed a method of using both appearance (intensity) and anatomical (brain tissue label) features [35]. They performed tissue segmentation on 3T MR images, and then they trained a 3D CNN that uses the intensity and segmentation labels from 3T MR images as the input and center voxel intensity of 7T MR image voxels as the output. This method not only produced an output very similar to 7T MRI quality but also worked robustly on images collected from different scanners.

Oktay et al. proposed a super resolution method to reconstruct high quality 3D cardiac images from 2D cardiac MR slices [36]. In the study, they used a CNN structure that uses multiple view stacks from both short axis and long axis as the input. Instead of directly reconstructing high resolution MR images, they allowed the network to efficiently learn the difference between low resolution and high resolution images through residuals, thereby producing a cardiac MR image of much higher quality than that of the conventional method.

With the objective of reducing the time and resource spent on extra exams and increasing accuracy of diagnosis and treatment plans, there have been various approaches to utilize deep learning in conversion between different image modalities.

Li et al. showed improvement in diagnostic accuracy of Alzheimer's using positron emission tomography (PET) images estimated from MR images [37]. They trained a 3D CNN model to estimate the corresponding PET voxel form MRI voxels and observed that the diagnostic accuracy was higher when the estimated PET images were used in addition to the MR images.

GAN can also be used in translation or conversion between images of different modalities, and Nie et al. used GAN to estimate CT images from MR images as seen in Fig. 4 [38]. Because the output CT image is blurry when we use a simple GAN loss to estimate CT images, they added a gradient difference loss function in the training process to maintain the intensity gradient between pixels in MRI. As a result, they obtained an image much sharper and closer to a real CT image than that of the conventional method.