Similarity-based multimodality image fusion with shiftable complex directional pyramid

Abstract

For multimodality images, a novel fusion algorithm based on the shiftable complex directional pyramid transform (SCDPT) is proposed in this paper. As well, with the aid of the structural similarity (SSIM) index, a ‘similarity-based’ idea is employed to distinguish regions with ‘redundant’ or ‘complementary’ information between source imagers before the SCDPT coefficients are merged. A ‘weighted averaging’ scheme for regions with ‘redundant’ information and a ‘selecting’ scheme for regions with ‘complementary’ information are then employed, respectively. When merging the low-pass subband coefficients, the SSIM index in spatial domain (SP-SSIM) is employed as similarity measure, and three types of regions are thus determined. Especially, for regions with similar intensity values but different intensity changing directions between source images, a ‘selecting’ scheme based on gradient and energy is proposed. When merging the directional band-pass subband coefficients, the SSIM index in complex wavelet domain (CW-SSIM) is employed as similarity measure. With the CW-SSIM index, not only the magnitude information but also the phase information of SCDPT coefficients can be employed. Compared to the traditional energy matching (EM) index based fusion methods, the proposed method can better deal with ‘redundant’ and ‘complementary’ information of source images. In addition, because of the shift-invariance of the SCDPT and the CW-SSIM index, the proposed fusion algorithm performs well even if the input images are not well registered. Several sets of experimental results demonstrate the validity and feasibility of the proposed method in terms of both visual quality and objective evaluation.

Introduction

With the development of image sensors, multimodality image fusion has emerged as a new and promising research area. Due to different imaging mechanism of sensors, multimodality images of the same scene captured by different image sensors can provide much ‘redundant’ and ‘complementary’ information. Such information from multiple source images can be integrated to create a new and improved image that contains a ‘better’ description of the scene than any of the individual source image (Lu et al., 2006, Pajares and de la Cruz, 2004). This method is called image fusion. Up to now, image fusion has been successfully applied to many fields, such as military affairs, medical imaging, remote sensing, digital camera, and so on (Jan et al., 2005, Li and Yang, 2008).

The multiscale transform (MST) based fusion methods (Piella, 2003, Zhang and Blum, 1999) are considered as important ones among numerous image fusion algorithms. Multiscale image decomposition and reconstruction tools and fusion rules are the two most important components in such fusion algorithms. The commonly used MST tools include the Laplacian pyramid transform (Petrovic´ and Xydeas, 2004, Toe, 1989) and the discrete wavelet transform (DWT) (Amonlis et al., 2007, Lewis et al., 2007, Li et al., 2006, Loza et al., 2010, Pajares and de la Cruz, 2004). Due to the fact that the DWT has many advantages over the pyramid transform, such as localization and direction, the DWT-based methods are generally superior to the pyramid-based methods (Li et al., 2006).

Although the DWT has been widely applied to image fusion, there are some limitations of the separable DWT in the application. Firstly, the DWT is with limited directions and cannot sparsify images with edges. It needs more coefficients to represent the ‘line’ or ‘curve’ discontinuities (Do and Vetterli, 2005). Some artifacts are easily introduced into the fused images (Zhang and Guo, 2009). Secondly, the DWT is not shift-invariant. Then, the corresponding DWT-based fusion methods are also shift-dependent, which is undesirable since different fusion results are obtained once input images are mis-registered.

To overcome the disadvantages of the wavelets in image analysis, Do and Vetterli (2005) developed a ‘true’ image representation tool, i.e., the contourlet transform (CNT). Compared to the traditional DWT, the CNT is not only with multiscale and localization, but also with multidirection and anisotropy. As a result, the CNT can represent edges and other singularities along curves much better. Some image fusion algorithms based on the CNT have been proposed in recent years (Ibrahim and Wirth, 2009, Yang and Guo, 2008). However, the CNT is also lack of shift-invariance.

In 2008, Nguyen and Oraintara, 2008a, Nguyen and Oraintara, 2008b proposed a shiftable complex directional pyramid transform (SCDPT), which is regarded as a complex and shiftable (in energy sense) version of the CNT and has been used in multisepectral and panchromatic image fusion (Saeedi and Faez, 2011). The SCDPT is shown to have a number of desirable properties for image analysis, such as shiftable subband, arbitrarily high directionality, and phase information. When introduced to image fusion, the SCDPT can provide more information for fusion. As well, the SCDPT is with shift-invariance, and the impacts of mis-registration on the fused results can thus be reduced effectively. Therefore, the SCDPT is more suitable for image fusion.

Besides the image multiscale decomposition and reconstruction tool, how to form the MST coefficients of the fused image from the MST coefficients of source images, i.e., fusion rule, is another key issue in the MST-based image fusion algorithms (Zhang and Blum, 1999). Different fusion rules, from the pixel-based to the window-based and region-based (Piella, 2003, Lewis et al., 2007, Zaveri and Zaveri, 2009a, Zaveri and Zaveri, 2009b), have been discussed thoroughly.

However, for the low-pass subband coefficients, the general fusion rule is an approximate ‘averaging’ or ‘weighted averaging’ scheme. This may work well on images with the same modality. Multimodality images may have different dynamic ranges, and much ‘complementary’ information exists between source images (Lewis et al., 2007). Therefore, the ‘averaging’ or ‘weighted-averaging’ scheme may significantly reduce the contrast of the fused image. For the band-pass subband coefficients, the general fusion rule is an approximate ‘absolute-maximum-selecting’ scheme, which implies that only the coefficients with larger absolute value are selected as the fused ones and the rest are discarded. Therefore, some ‘redundant’ information between source images is easily lost during the fusion process and some ‘artifacts’ or noises are also easily introduced into the fused image.

With the particular research on multimodality images, a novel fusion algorithm based on the SCDPT and the structural similarity (SSIM) indexes (Wang and Bovik, 2002, Wang et al., 2004, Wang and Simoncelli, 2005) is proposed in this paper. Source images are decomposed into different subbands with the SCDPT. When merging the low-pass subband coefficients and the band-pass subband coefficients, the SSIM index in spatial domain (SP-SSIM) and the corresponding version in complex wavelet domain (CW-SSIM) are applied to distinguish regions with ‘redundant’ or ‘complementary’ information between source images, respectively. For regions with ‘redundant’ information, a ‘weighted averaging’ scheme is performed. While for regions with ‘complementary’ information, a ‘selecting’ scheme based on some salience measure is performed. Therefore, the proposed fusion algorithm can properly deal with the ‘redundant’ information and ‘complementary’ information between source images. In addition, owing to the shift-invariance of the SCDPT and the CW-SSIM index, the proposed fusion method is with good robustness on mis-registration. Three sets of multimodality images with perfect registration and one set of images with mis-registration are employed to demonstrate the validity and feasibility of the proposed algorithm.

The rest of the paper is organized as follows. Section 2 reviews the theory of the SCDPT in brief. Section 3 describes the image fusion algorithm using the SCDPT. Experimental results and discussion are given in Section 4. And the concluding remarks are presented in Section 5.