1.1.

Bronchoscopy

The initial step in the identification of pulmonary airway disease is typically a bronchoscopy procedure,¹ which may be performed as a result of a persistent or unexplained cough, the presence of blood in the sputum, an abnormal chest x-ray, or the evaluation of a possible lung infection. A bronchoscope is a long, flexible visualization instrument that is inserted into either the nose or the mouth to gain access to the airways. The physician is able to make an initial visual evaluation based on this inspection. Although the qualitative analysis of the mucosal topography and color is possible through bronchoscopic exploration, many of the bronchoscopes currently used in pulmonary diagnostic centers are monochromatic, CCD chip bronchoscopes. The images from these bronchoscopes are presented in color to the physician, however, this observed color is added falsely through illumination effects. The light source used in conjunction with such bronchoscopes repeatedly flashes red, green, and blue, causing the light to appear to the human observer as white light. To construct the color images, the responses of the CCD chip to each of the three light stimuli are combined to form the red, green, and blue components of a 24-bit true-color image. The observed airway epithelial color is therefore greatly susceptible to interscope and interillumination disparities as well as the obvious interobserver variability due to the inherently subjective nature of visual color interpretation. Research into the color changes accompanying the development of lung disease to date has been minimal due to the limitations of appropriate imaging devices. Developments in bronchoscope technology^{2, 3} have resulted in bronchoscopes with trichromatic, CCD chips. As a result it is now feasible to perform digital color analysis of the pulmonary airway mucosa in an attempt to facilitate a greater understanding of pulmonary airway disease response.

The assessment of the pulmonary airways through traditional bronchoscopy is relatively noninvasive, and provides a visual link to the mucosal topography and color, however, alone is generally not sufficient for accurate disease diagnosis. The subtle changes that indicate early changes in disease development may often be missed as a result of this highly subjective assessment especially in inexperienced bronchoscopists. A bronchoscope is also restricted to the upper airway generations and thus this form of assessment alone may miss changes occurring in the lower generation bronchi.

1.2.

Biopsy

A bronchial airway forceps biopsy is traditionally performed in conjunction with a bronchoscopy procedure to evaluate the airway mucosa further. The biopsy affords epithelium and basement membrane tissue specimens for histopathological analysis, including in some cases special staining for molecular and genetic changes. The diagnostic and investigational efficacy of tissue biopsy techniques is limited due to the invasive nature of the technique and the relatively small sample region size attainable. This is especially problematic if the disease process has significant heterogeneity. An additional limitation of airway forceps biopsy is the identification of a region of interest of which to biopsy. Many pulmonary airway diseases are localized and/or show very little visual differentiation between the healthy mucosa, thus identification through a bronchoscopist’s visual interpretation may lead to abnormal mucosa being overlooked.

1.3.

Multidetector X-Ray-Computed Tomography (MDCT)-Based Imaging Techniques

A further technique for disease detection and classification in the airways is through external imaging modalities, particularly MDCT imaging. High-resolution MDCT imaging can provide both structural and functional information to the characterization of the lung parenchyma,^{4, 5} and increasingly these image data sets are being evaluated for other structural features. The structural analysis associated with the 3-D volumetric data sets obtained from MDCT imaging techniques is widely known and facilitates quantitative anatomical airway feature analysis, including airway tree diameter, surface topography, and wall thickening.⁶ Presently research in functional MDCT imaging, which is often not recognized as an aspect of MDCT imaging, includes regional ventilation and perfusion studies, regional pulmonary vascular transport characteristics, and regional pulmonary ventilation studies.^{4, 5} MDCT benefits further extend to airway stenting procedures, including stent sizing, longitudinal examination of individual subjects, and surgical planning. Although the use of volumetric MDCT imaging in the detection and diagnosis of respiratory disease has clear advantages, including that the analysis is not limited to regions accessible with a traditional bronchoscope, and the valuable 3-D structural measurements, shortcomings still exist. For instance, the radiation dose associated with MDCT can be potentially harmful to the patient, and that MDCT images lack important information regarding airway mucosal color attainable with bronchoscopy procedures.

1.4.

Multimodality Image Fusion

Given the prevalence and, in diseases such as cancer and cystic fibrosis, the high mortality rate of respiratory disease and in particular pulmonary airway disease it is apparent that further advancements in methods for screening, detecting, diagnosing, and tracking are required. Currently diagnosis and detection of the state of a patient’s airways is generally initiated via visualization of the pulmonary airways either by macro-optical techniques such as bronchoscopy, or by external imaging techniques such as MDCT. Both techniques provide valuable information to the health of the airway mucosa such as mucosal color that may be an indicator of airway inflammation, or airway tree structure that may indicate regions of stenosis where the airway branch closes in on itself; however, each technique also has associated limitations. To enhance the abilities of these imaging techniques to aid in the determination of airway structure and function it is proposed that the image information be adapted and fused to provide a synergistic and coherent dataset.

The use of multimodality imaging in the medical field has grown considerably in recent years due in part to the advancement in various imaging techniques required to visualize anatomical and physiological structure and function. A familiar quandary faced in multimodality imaging is the fusion of the two varying imaging techniques to obtain a synergistic data set utilizing all available information. This multimodality image-based registration may be generally straightforward when the conflicting modalities have similar image structure, however, the registration complexity is significantly increased when the opposing imaging modality structure is nonuniform. The evaluation of the pulmonary airways involves both traditional clinical bronchoscopy techniques and more modern approaches involving 3-D volumetric MDCT techniques such as virtual bronchoscopy. We propose to merge these complimentary imaging modalities through a unique technique utilizing both shape from shading and virtual bronchoscopy imaging techniques.

2.1.

Macro-Optical Image Acquisition

In 2001 the Olympus Corporation released the 160 series digital bronchoscopes comprising of distal trichromatic CCD chips. Previously digital bronchoscopes consisted of gray-scale CCD chips, and thus the appearance of color was through a false coloring process. The coloring process is reliant on the use of a rapidly rotating color wheel. The color wheel consists of red, green, and blue (RGB) filters. During a single rotation of the color wheel, three images are captured corresponding to white light illumination being transmitted through each of the filters. The three image signals detected are then combined to form a single-color RGB image. Although this coloring technique does result in realistic pseudocolor images, the images are not an accurate representation of the true color of the airway mucosal surface.

To take advantage of the color detection capabilities of the new color digital bronchoscopes, we developed an airway mucosal color imaging system that will facilitate quantitative analysis of the airway epithelium. Physicians for many years have qualitatively assessed airway mucosal color changes in the digital pseudocolor bronchoscope images, however, due to the subjective nature of this assessment, inter- and intraobserver variation has not been quantitatively assessed. The trichromatic bronchoscope used in the imaging system is connected to an Olympus EVIS Video System Tower. The tower consists of an illumination source, a video processor, a monitor, and video recording and printing facilities. In standard bronchoscope practice, an illumination source is required to visualize the airways. The detection of color is heavily dependent on the illumination source and thus the quality of a source is often described by a color-rendering index, which refers to the quality of the illumination, and hence the ability of the source not to distort the appearance of measured colors. The Olympus EVIS Universal Light Source (CLV-U40) used in this system has a high quality xenon 300-W short arc lamp with a superior color-rendering index. The intensity of the xenon lamp is controlled by an automatic gain control. The automatic brightness control on the illumination source ensures that sufficient light is available to visualize the airways while preventing adverse effects of overillumination, such as CCD blooming effects, when the storage capacity of the CCD wells are exceeded, and eye damage or exposed tissue burns.

The digital color images captured by the color CCD chip bronchoscope are transferred to the video processor unit, where, among other processing steps, they are converted to an analog signal and displayed on a color monitor for immediate visualization by the medical team. The current Olympus digital video processor systems do not allow for the direct transferal of the digital feed from the bronchoscope and thus in order to re-digitize this real time image information a Matrox Meteor II frame grabber is employed. The frame grabber, essentially an image acquisition peripheral component interconnection (PCI) card, is installed on a PC and is connected to the video output channels of the bronchoscope monitor via a DBH44 to 8BNC cable. The $640\times 480$ 8-bit digital images of the monitor output are captured at a rate of 30 frames/s and are consequently stored and displayed on the PC for either concurrent or future color analysis.

2.2.

Mucosal Color Analysis

Many color-encoding methods exist including hue, saturation, intensity (HSI), YIQ (used in television broadcasting), XYZ (used by the CIE Chromaticity diagram), and the popular RGB system. Although color is commonly represented in the RGB color space, the HSI color space was selected for use in our color analysis system. The HSI color space is relatively insensitive to changes in the illumination intensity, assuming that the illumination source is “pure.” An increase or decrease in the illumination intensity results only in a corresponding increase or decrease in the $I$ parameter rather than changing each of the three color components as is the case in the RGB color encoding system. The HSI color space mirrors the perception of color by the human eye thus supporting a clear understanding of the measured values.⁷ Hue is a measure commonly referred to as color ranging from red through yellow, green, and blue and is determined by the dominant light wavelength or photon frequency. Saturation is a measure of the purity of a particular hue, or alternatively a measure of the degree of white light present. The images captured with a bronchoscope are not exposed to uniform scene illumination as tissue regions at a great distance from the bronchoscope tip and hence the light source receive lower intensity illumination than those within close proximity of the bronchoscope tip. Due to the reliance of the $I$ parameter on the scene illumination, it was not considered in our color analysis. $H$ and $S$ values are commonly represented on a color wheel. Hue is measured in degrees ranging from 0 to 360 with 0, 120, and 240 deg corresponding to red, green, and blue, respectively, with intermediate values linearly spaced between. Saturation is measured as a percentage, with 100% saturation depicted on the radius of the color wheel and all other intermediates linearly scaled along the wheel radius.

The use of color analysis has been used for a number of years in dermatology, specifically in the assessment of skin lesions. These color analysis techniques often involve color calibration techniques to ensure that the measured color is independent of the imaging system. Color images from a detection system are dependent on three parameters: the physical content of the scene, the illumination incident on the scene, and the characteristics of the camera. The aim of color constancy is to account, if necessary, for the effects of illumination either by mapping the image to an illuminant invariant representation or by determining the description of the illumination for color correction. Integrated into the Olympus Video System Tower is a white balance control. This white balance is a form of color constancy where each of the trichromatic RGB channels is scaled by their maximum value in the image when the scene contains a pure white color patch. Olympus provides a pure white bronchoscope cap with the bronchoscope system that encapsulates the distal tip of the bronchoscope. With this cap is in place, the white balancing calibration process is performed prior to using the system.

Software to control the image acquisition (by the video frame grabber), display, storage, and analysis was developed in Microsoft Visual Studio software development package. The customized software supports various modes of operation including the capture of healthy patient, the capture of potentially diseased patient, the analysis of stored, as well as the real-time capture and analysis of airway images. During the collection of healthy patient data, either single images or image sequences are acquired and stored for potential future use. Prior to the images being saved to disk, $H$ and $S$ values for each pixel in the image are determined via the following set of equations, and are consequently added to a cumulative database of healthy patient color information.

2.3.

Bronchoscope Image Distortion Correction

Generally, endoscopes used in current clinical practice are designed to include fish-eye lenses that induce barrel distortion in the resulting image output. The reason for the inclusion of these lenses is to increase the field of view of the bronchoscope, providing the physician with a greater view of the surrounding environment, however, as a result the interpretation of the image scene is compromised. To facilitate the fusion of both the MDCT-generated image data and the macro-optical images collected with the color bronchoscope, correction of this barrel distortion is required. Many algorithms exist for lens distortion correction. One such algorithm was introduced by Asari ⁸ in 1999, and was further improved by Helferty ⁹ in 2001. Helferty’s algorithm requires the used of a calibration grid consisting of a rectangular lattice of equally spaced dots. An image of the grid is captured by the endoscope and correction parameters are calculated through an iterative optimization calculation. The desired correction parameters are those that are able to successfully map the grid rows and columns into straight lines with zero curvature. The correction coefficients are calculated in the horizontal and vertical directions independently via the following equation for the $x$ axis.

The calculation of the distortion correction parameters are required to be computed only once for each endoscope and can consequently be applied to all images collected by that individual bronchoscope to correct for the barrel distortion. The resulting calculated correction parameters are then applied to each pixel in the distorted image to map to the corrected space via the following equation, where $\rho$ represents the Euclidean distance from the distortion center to a pixel in the corrected space, and ${\rho }^{\prime }$ represents the Euclidean distance from the distortion center to the same pixel in the barrel distorted space:

2.4.

Virtual Bronchoscopy

The development and clinical use of virtual bronchoscopy software has increased in recent years due in part to the advancements in MDCT scanners. A virtual bronchoscopy is a computer-generated visualization of a traditional bronchoscopy procedure and provides a simulated view of traversing the pulmonary airways.¹⁰ The structure of the pulmonary airways is extracted from MDCT volumetric data through automated airway tree segmentation. The segmentation involves a combination of adaptive region growing and a hybrid method utilizing both region growing and mathematical morphology.¹¹ The segmented airway is then surface rendered producing a 3-D structurally accurate image of the inner pulmonary airway wall. The virtual bronchoscopy endoluminal view is generated through rapid rendering techniques where only the airway branches within the field of view are rendered.^{12, 13} Once the virtual bronchoscopy views have been generated, we are able to match these views with the barrel distortion corrected images captured with the true color bronchoscope imaging system. An initial guess at the match of the virtual and true bronchoscope images is approximated at an easy to locate region of the airways such as the trachea, which is further refined through automated mutual information rigid image registration. The virtual bronchoscopy view is rotated and translated in the $x$ , $y$ , and $z$ directions and the orientation that returns the maximum measure of mutual information between the two images is chosen as the correct match. Once this match is made, the depth information for each pixel in the macro-optical color bronchoscope image can be determined from the $z$ coordinate of the corresponding pixel location within the virtual bronchoscopy view. The virtual bronchoscopy program includes a tracking algorithm to perform this image registration for consequent macro-optical images and thus update virtual bronchoscope view¹⁴ and determine the bronchoscope location within the airway tree. At this stage of development, we are limiting our structure and color fusion to single macro-optical images.

Limitations are associated with the registration of the macro-optical color and virtual bronchoscope views, which affect the accuracy of the correct structure and color alignment. First, the lung is not a rigid structure and thus there is no way to ensure that the airways are in precisely the same 3-D shape during both the MDCT scanning and macro-optical bronchoscopy, possibly due to such factors as differences in lung volume, patient orientation, movement, or coughing. There may also be inaccuracies in the registration process itself. For these reasons the fusion of the structure and color information is not performed through direct mapping following registration but rather an initialization parameter, which does not grossly affect the resulting SFS 3-D structure, is used. This method also ensures exact alignment of the determined mucosal structure and color which is essential in future analysis of localized mucosal changes occurring during the course of an airway diseases.

2.5.

3-D Shape Extraction from 2-D Bronchoscope Images

An alternative to the use of external imaging modalities, such as MDCT, in the production of 3-D bronchial trees is the inference of 3-D structural information from shading present in the 2-D bronchoscope images. Video bronchoscopes, such as the color CCD chip bronchoscope previously mentioned, provide an extremely rich and often superfluous supply of information with a single point being imaged a number of times from a number of positions. Through the exploitation of this information redundancy the development of topographical maps with a higher degree of accuracy than that available from current MDCT imaging techniques is attainable. The 3-D topographical maps of an object can be constructed from 2-D images via SFS techniques.

The reconstruction of 3-D shape information from 2-D images has remained an area of high interest in the field of computer vision for many years since the introduction of the image irradiance equation by Horn¹⁵ in 1975. The image irradiance equation is given as

The goal of SFS techniques is to solve the image irradiance equation and in doing so recover depth information for each pixel in the original image. There are various approaches for the determining a solution to the SFS problem including global minimization, curve propagation, and local algorithms. In traditional SFS techniques, assumptions are made regarding the scene illumination, specifically that the scene is under uniform illumination from a distant light source, that the image projection model is orthographic (see Fig. 2 ) and that the object surface is Lambertian. These assumptions, however, are all clearly violated in the case in bronchoscope images. The illumination source is adjacent to the camera lens on the distal end of the bronchoscope and thus tissue regions close to the bronchoscope tip will experience greater illumination than tissue regions farther down the airway lumen. Due to this close proximity of the bronchoscope tip to the airway lumen, the assumption of orthographic projection is not feasible. Orthographic projection assumes parallel projection of the scene to the image plane, essentially setting the projection center at infinity. A Lambertian surface is a surface of perfectly matte properties that reflects light radiance proportional to the cosine of the angle between the object surface normal and the illumination direction. Bronchoscope images often suffer from regions of specular reflection where the incident or illuminant source is perfectly reflected off of the object surface and hence in bronchoscope images appears as a white spot.

Fig. 2

(a) Orthographic projection of the real-world coordinates to the image coordinates and (b) perspective projection of real-world coordinates to the image coordinates.

Okatani and Deguchi^{16, 17} and Deguchi and Okatani^{18, 19} first introduced an SFS technique applicable to endoscope images in 1996 by assuming a point light source at the projection center of the endoscope image rather than the uniform scene illumination previously presumed. Although this advancement did broaden the applications for SFS techniques, confining limitations still persist in the application to pulmonary airway images, including the uncertainty of the image location within the bronchial tree, and the limited observable and hence reproducible bronchial scene. To address these limitations we propose to incorporate information garnered from MDCT imaging techniques. The Okatani and Deguchi SFS algorithm for endoscope images is based on the Kimmel and Bruckstein²⁰ approach, where perspective projection and a point light source at the projection center are assumed. The endoscope image formation process is model through the following equation:

The Okatani and Deguchi developed algorithm based on the level set approach requires an initial known depth contour, $\varphi (x,y,0)=0$ that is propagated via a time-based differential equation where time is proportional to depth.

4.1.

Mucosal Color Analysis

Significant correlation was found in the intervolunteer mucosal $H$ and $S$ values from the eighteen healthy volunteers. A bar graph (see Fig. 4 ) of the mean $H$ and $S$ values for each individual highlights this parity that is further accentuated by the smoothness and single modal distribution of the cumulative $H$ and $S$ histograms constructed from the entire normative database (see Fig. 5 ).

Fig. 4

Intersubject mean hue and saturation values for the airway mucosa of the 18 healthy volunteers.

Fig. 5

Cumulative hue and saturation histograms constructed from the healthy volunteer database.

Airway mucosal colors collected from the abnormal mucosal population of patients with known pulmonary airway diseases were collated within their respective disease classes and the corresponding mean $H$ and $S$ values were presented in an $H$ versus $S$ scatterplot along with those from the healthy database, and are depicted in Fig. 6 . The scatterplot shows that in most cases the abnormal images have a clear shift in the mean $H$ and $S$ values from that in the normal healthy population that may not always be readily visible to the human eye. From a visual inspection of the subglottic stenosis bronchoscope images, we observe two forms of mucosal response; some of the images appear highly vascular while others appear to have significant reduced vascularity. Based on this visual observation, the two groups were divided and treated as separate classes with consequent analyses confirming this distinction. To view the contribution of the $H$ and $S$ values independently as a means of distinguishing between the individual disease classes, a bar graph of the respective $H$ and $S$ values was constructed with associated standard error bars (see Fig. 7 ). Standard two-sided $t$ tests were calculated for each of the mean $H$ and $S$ values in each pulmonary airway disease class against the mean $H$ and $S$ values in the normative database to further quantify the interdisease mucosal color variation. The results (Table 1 ) indicate that the difference in mean mucosal $H$ and $S$ values between the papillomatosis, granulation, and carcinoma disease classes and the healthy normal class are statistically significant, however, the $p$ values determined for the two subglottic stenoses classes were not so impressive. In future classification of these pulmonary images based on mucosal color analysis, more sophisticated methods such as the use of neural networks²¹ may be required where extracted features may include additional color and texture analysis.

Fig. 6

Mean saturation versus hue scatter plot for each of the known airway disease classes.

Fig. 7

Plots exhibiting mean hue and mean saturation values for the cumulative normal volunteers and the cumulative known airway mucosal abnormalities.

Table 1

Hue and saturation p values for the disease classes against the healthy volunteer values.

Airway mucosal color analysis involved the comparison of the potentially abnormal airway mucosa to that of the developed healthy database. During this comparison, the $H$ and $S$ values for each pixel in the image were determined and individually statistically compared to the healthy mucosal color values. Pixels falling outside two standard deviations were highlighted as abnormal and displayed on the original image to make the physician aware of the potential mucosal abnormality.

4.2.

Multimodality Image Fusion

Adaptation of the 2-D optical image information to 3-D structures has the advantage of greatly increasing the structural composition of the image scene. The initial step in this 3-D feature extraction is to correct the barrel distortion present in the macro-optical images acquired with the color bronchoscope imaging system. An example of this correction is presented in Fig. 8 . Through the utilization of the pixel location look-up table, this correction occurs in processing times close to that of real-time analysis.

Fig. 8

Bronchoscope image of a carcinoma and the corresponding bronchoscope barrel distortion corrected image.

Due to the complexity of extracting the initialization parameters for the SFS algorithm from the virtual bronchoscopy software, this process is performed after image collection. The input to the SFS algorithm in addition to the initialization depth and location, is both the distortion corrected image and the color-analyzed distortion-corrected image, the first of which is required to extract the scene shading information and the second to map the color texture including the highlighted abnormal mucosal colors. The resulting 3-D color image structures are then displayed using an interactive graphical user interface that enables user-defined image rotation, translation, and scaling to enable full exploration of the 3-D image scene (see Fig. 9 ).

Fig. 9

Color analysis of the bronchoscope corrected image of the carcinoma in Fig. 8, involving comparison of image pixel hue and saturation values within two standard deviations of the developed normative database.

In this early stage, we limited our fusion of color and structure information to single macro-optical images. With the inclusion of additional frames we believe that the accuracy of the structural reconstruction through SFS will be greatly refined. This extension to multiframe SFS image reconstruction will result in mucosal surfaces being imaged at a number of angles a number of times. By determining the interframe correspondences, the 3-D surfaces can be combined resulting in the convergence of a potentially more accurate structural solution.