Visible and Near-Infrared Image Synthesis Using PCA Fusion of Multiscale Layers

Abstract

This study proposes a method of blending visible and near-infrared (NIR) images to enhance their edge details and local contrast based on the Laplacian pyramid and principal component analysis (PCA). In the proposed method, both the Laplacian pyramid and PCA are implemented to generate a radiance map. Using the PCA algorithm, the soft-mixing method and the mask-skipping filter were applied when the images were fused. The color compensation method uses the ratio between the radiance map fused by the Laplacian pyramid and the PCA algorithm and the luminance channel of the visible image to preserve the chrominance of the visible image. The results show that the proposed method improves edge details and local contrast effectively.

1. Introduction

Recent improvements in image sensors have enabled cameras to realize better quality pictures. Many types of image-quality enhancement algorithms, such as high dynamic range (HDR), are also available today. In the field of ultrasmall cameras embedded in smartphones, cameras can be built with various functions by miniaturizing and improving performance. Cameras can now acquire both infrared and visible (VIS) images, and this capability is utilized in various ways, for example, for unlocking the smartphone function or capturing good-quality photos using the infrared camera. Infrared cameras are often used to capture a significant amount of information that lies beyond the VIS spectrum. The near-infrared (NIR) spectrum covers the wavelength range of 700–1100 nm. Therefore, infrared light is less scattered than VIS light, and hence, infrared images can capture more information, such as fog and shade. Capturing infrared images is also advantageous for obtaining details and edge information that are not generally acquired in the VIS spectrum. Many types of algorithms for fusing VIS and NIR images have been developed [1,2,3,4,5].

Most of these algorithms are intended for using NIR scenes to reveal hidden areas in VIS scenes. The conventional VIS and infrared images are used for dehazing. To synthesize VIS and NIR images, most existing methods use multiresolution decomposition and multiresolution-based convergence algorithms. Multiresolution is essential for obtaining details of NIR images. Infrared scenes can preserve image dehazing, high contrast, and edge information in dark areas better.

Laplacian–Gaussian pyramids and the local entropy-fusion algorithm are conventional VIS–NIR fusion algorithms [1]. Local entropy is utilized by synthetic algorithms to transmit the maximum information from VIS and NIR images. Conventional methods also use the multiresolution decomposition of the Laplacian–Gaussian pyramid, wherein the decomposed multiple resolution images are gently blended in each layer using a local-entropy synthesis method. In short, this conventional algorithm uses a Laplacian–Gaussian pyramid for multiresolution fusion and generates a weighting map using the local entropy to control local contrast and visibility to generate the fusion result. However, the details of the NIR image are not well preserved, and the color expression is not natural.

VIS and infrared image-fusion algorithm is a latent low-rank representation fusion algorithm (i.e., low-rank fusion) [5]. Low-rank fusion can be decomposed into low-rank parts (i.e., a global structure) and saliency parts (i.e., a local structure). The weighted-average strategy is used to fuse the low-rank parts to preserve edge information. Then, the saliency parts are fused using the sum strategy, and the final fusion image is obtained from the sum of the low-rank and saliency parts. The latent low-rank fusion algorithm decomposes an input VIS and an infrared image into a low-rank part and a saliency part of each other. After decomposition, each low-rank and saliency part of the VIS and infrared images are fused together by the sum strategy. Finally, the resulting image is obtained using the fused low-rank and saliency parts. However, this algorithm fails to capture the details of the NIR image and does not properly indicate the edge information during color compensation.

In this study, image fusion of VIS and NIR images using the Laplacian pyramid and principal component analysis (PCA) is proposed. Fusion with NIR images is adopted for enhancing image details. Hence, an image is separated by a Laplacian pyramid for details and a base layer in VIS and NIR image is used to enhance the sharpness of the image. The proposed method is presented to create a weighting map for a particular detail area and to skip the PCA calculation for unnecessary areas. The values of the weighting map are obtained properly from the PCA results of the VIS and NIR scenes to improve image details. After generating a radiance map, the proposed method is used to achieve the Stevens effect in a color appearance model [6] to enhance local contrast and details. Tone scaling is adopted for color compensation in the proposed method. Thus, with regard to color compensation, the proposed method calculates the brightness gap between the input VIS image and the generated radiance map. Consequently, compared with conventional methods, the proposed method results in less damage to the edge information in the NIR images improved local contrast and details, and better preservation of the color chrominance of the input VIS image. The proposed method is to preserve the detailed information by a higher value of PCA and enhance local contrast and details according to the visual properties of the Stevens effect. Thus, the proposed method preserves more detailed information and enhances the details effectively.

2. Laplacian Pyramid

The Laplacian pyramid is a multiscale processing algorithm that decomposes input images into multiscale images [7]. Laplacian-pyramid images can be separately processed at each pyramid level. Multiscale images of the Laplacian pyramid are obtained using Gaussian blur, up/downsampling, and image subtraction.

The first step in the Laplacian pyramid is based on the Gaussian pyramid, which produces multiple images with Gaussian blurring and downsampling. A filter with two dimensions of Gaussian blurring is shown in Equation (1):

$$w\left(x,y\right)=\frac{1}{2\pi {\sigma }^2}·e^{-\frac{x^2+y^2}{2{\sigma }^2}},$$

(1)

where $w\left(x,y\right)$ is a function of the Gaussian blur whose size depends on the σ value. Generally, we use a 5 × 5 Gaussian blur prior to downsampling the images. The 5 × 5 Gaussian blur is to provide adequate filtering at a low computational cost [7].

Image downsampling is performed after blurring. The Gaussian pyramid aims to create a repetitive image that has been smoothened and downsampled. Gaussian blurring and downsampling are represented in Equation (2):

$$g_{l+1}\left(i,j\right)={{\displaystyle \sum }}_{m=-2}^2{{\displaystyle \sum }}_{n=-2}^{2}w\left(m,n\right)g_l\left(2i+m,2j+n\right),$$

(2)

where $w\left(m,n\right)$ is the Gaussian blur filter; $g_l$ is the input image; and $g_{l+1}\left(i,j\right)$ $\left(i,j\right)$ is the Gaussian pyramid image of the results of blurring and downsampling of the input image. The downsampling method reduces the image size to one-half the row and column of the original image. Thus, the Gaussian pyramid is computed by blurring an input image of a different size. Therefore, the Gaussian pyramid is a sequence of low-pass and downsampled images.

Image blurring is calculated as the average pixel value of the surrounding pixels. Specifically, it is the removal of high-frequency components from an image-processing perspective. Therefore, the Gaussian blur is performed before image downsampling because it removes the high-frequency components that may cause aliasing. Aliasing is a result of a sampling-rate signal that is insufficient for changing other signals. Therefore, image blurring and smoothing are performed before downsampling to eliminate aliasing effects.

The second step in creating the Laplacian pyramid is to upsample the Gaussian pyramid and subtract the previous level image of the Gaussian pyramid to obtain a differential image. The upsampling of the Gaussian pyramid is as expressed in Equation (3):

$$Expand(g_{l+1}\left(i,j\right))=4{{\displaystyle \sum }}_{m=-2}^2{{\displaystyle \sum }}_{n=-2}^{2}w\left(m,n\right)g_{l+1}\left(\frac{i-m}{2},\frac{j-n}{2}\right),$$

(3)

where $Expand(g_{l+1}\left(i,j\right))$ is an upsampled image of the Gaussian pyramid, and $g_{l+1}$ is an image of the Gaussian pyramid obtained using Equation (2).

A method for obtaining the differential images is given in Equation (4); the upsampling image obtained using Equation (3) is subtracted from the previous step image:

$$L_0=g_l-Expand(g_{l+1}),$$

(4)

where $L_0$ is the image of the Laplacian pyramid; $g_l$ is the Gaussian pyramid obtained in the previous step; and $Expand(g_{l+1}\left(i,j\right))$ is the upsampling image obtained using Equation (3).

The reconstruction of the Laplacian pyramid is an iterative process of upsampling from the lowest-level image in the Laplacian pyramid and adding each level of differential images. To reconstruct the image without any error, the Gaussian-blurred images should be upsampled, and this blurred image is subtracted from the previous level of input images; this is shown in Equations (3) and (4). In summary, the Laplacian pyramid stores the differential images acquired by the Gaussian pyramid at each level. Figure 1 shows a block diagram of the Laplacian pyramid. The Laplacian pyramid image is shown in Figure 1 in normalized grayscale because detailed information in the original Laplacian pyramid images is difficult to see visually.

3. Principal Component Analysis

The PCA is a method of linearly combining the optimal weights for a data distribution [8] and employs the covariance matrix and eigenvector of the data. The eigenvector of the covariance matrix shows the direction of data distribution. Thus, the eigenvectors determine the influence of the principal component (PC). The direction of the eigenvector cannot be changed; however, it is possible to change its magnitude. Therefore, the eigenvector, as well as the PC, can be regarded as a reference axis with increasing or decreasing size. This allows one to determine the reference axis, which comprises the PC that is calculated using the covariance matrix and the eigenvector. The number of PC axes depends on the data dimension. The first PC axis is the maximum variance, which is the longest and widest extent of data distribution. The second PC axis is perpendicular to the first one, and it is the second-widest and longest extent of data distribution. In summary, PCA deals with the decomposition of the eigenvalue calculated using the input data covariance. When the PCA algorithm is adapted to image fusion, the column vectors of the input images must be produced as the first step.

$$E\left(X\right)=1/M{\displaystyle \sum }_{k=1}^M{X}_k,$$

(5)

$$cov\left(X,Y\right)=\frac{{{\displaystyle \sum }}_{i=1}^n\left(X_i-E\left(X\right)\right)\left(Y_i-E\left(Y\right)\right)}{\left(n-1\right)},$$

(6)

where E(∙) is the average of the input data. $X$ is an input image, and cov(∙) is the covariance matrix of the data pairs $X$ and $Y$.

$$Ax=\lambda x,det\left(A-\lambda I\right)=0,A=cov\left(x,y\right),$$

(7)

$$P_1=\frac{E{v}_1}{{{\displaystyle \sum }}^{\text{}}Ev},P_2=\frac{E{v}_2}{{\displaystyle \sum }Ev},$$

(8)

where $Ev$ is a unique vector and the sum of the PCA weights $P_1$ and $P_2$ is 1.

PCA synthesis is accomplished using the mean weights of $P_1$ and $P_2$, which are calculated by multiplying the weights of each input image in Equation (9).

$$I_{fused}=P_1\times I_{VIS}+ P_2\times I_{NIR},$$

(9)

where $I_{fused}$ is the result of image synthesis calculated using the PCA algorithm [9], and $I_{VIS}$ and $I_{NIR}$ are the input VIS and NIR images, respectively. Therefore, the proposed method assigns the weight obtained from the PCA algorithm to fuse the VIS and NIR input images, as shown in Figure 2:

4. Radiance Map Generation

The proposed algorithm is used to generate a radiance map and to achieve color compensation. The flowchart of the proposed algorithm is shown in Figure 3, where $L_{VIS}$ is the luminance channel of the VIS image, and $L_{NIR}$ is the luminance channel of the NIR image. The input VIS and NIR images can be taken by a multi-image sensor device, which is a prism-based multi-band imaging device. It uses the same light path for VIS and NIR image capturing and can be synchronized between spatially separated VIS and NIR images, so it can be filmed without a time delay.

The proposed radiance map algorithm is used in the Laplacian pyramid, and PCA is used in the CIELAB luminance channel. The Laplacian pyramid of input images is decomposed into edge images and base layers. Each layer is fused by the PCA fusion algorithm using the soft-mixing method. The proposed PCA fusion algorithm has a PCA-skipping mask filter. It allows skipping PCA calculations for unnecessary areas by transferring the pixel value of such areas to the next pixel area without calculating the PCA, as shown in Figure 4.

The criteria to determine whether to implement the PCA is based on the pixel value in the PCA-skipping mask-filter area. If the pixel value in the area is less than 10% of the total pixel value in the histogram distribution of the input image, the PCA algorithm is not implemented. The pixel value that is not calculated using the PCA algorithm is set to 0.5 PC weight, as shown in Figure 4. The lower 10% of the histogram distribution of the entire pixel value is skipped because most edge-layer images have pixel values close to zero, except for the edge components. The proposed filter mask determines whether the pixels in the mask-filter area should be calculated using the PCA fusion algorithm. The size of the PCA filter mask depends on the size of the Laplacian pyramid.

The soft-mixing method fuses two images in the transition areas, as shown in Figure 5 [10]. The transition area range is a linear fusion of input images between $t{h}_1$ and $t{h}_2$ areas and those between $t{h}_3$ and $t{h}_4$ areas. In Figure 5, $t{h}_1$ represents the value of the lower 10%, and $t{h}_2$ represents the value of the top 10% of the total in the histogram distribution of PCA weights, as shown on the x-axis in Figure 5a. Further, $t{h}_3$ denotes the value of the lower 10%, and $t{h}_4$ denotes the value of the top 10% of the total in the histogram distribution of the normalized VIS luminance histogram, as shown on the x-axis of Figure 5b.

Equation (10) is the PCA fusion formula for the VIS and NIR edge layers adapted to Equation (9):

$$ra{d}_{detail}=P_{detail}\times VI{S}_{detail}+ \left(1 -P_{detail}\right) \times NI{R}_{detail},$$

(10)

where $P_{detail}$ is the detail weighting map of the PCA weights applied by the soft-mixing method, and $VI{S}_{detail}$ and $NI{R}_{detail}$ are the edge-layer images calculated from the VIS and NIR input images, respectively, with the Laplacian pyramid.

Equation (11) is the fused expression of PCA in the base layer:

$$bas{e}_{fused}=P_{base}\times VI{S}_{base}+(1-P_{base})\times NI{R}_{base},$$

(11)

where $P_{base}$ the base weighting map of the PCA weights applied using the soft-mixing method, and $VI{S}_{base}$ and $NI{R}_{base}$ are the base-layer images obtained by calculating the VIS and NIR input images, respectively, using the Laplacian pyramid.

Equation (12) applies the fused image obtained using Equation (11) and the NIR images to enhance the local contrast and detail information in the base-layer images.

$$ra{d}_{base}=P_{ra{d}_{base}}\times NI{R}_{base}+(1 - P_{ra{d}_{base}})\times bas{e}_{fused},$$

(12)

where $P_{ra{d}_{base}}$ is the base weighting map of the PCA weights applied using the soft-mixing method in Figure 5b for improving the local contrast, and $NI{R}_{base}$ is the NIR image obtained from the Laplacian pyramid, and $bas{e}_{fused}$ is the fusion image obtained using Equation (11).

Equation (13) shows a radiance map that applies the PCA image-fusion algorithm and the reconstruction process of the Laplacian pyramid:

$$\begin{gathered} \mathrm{Radiance}\mathrm{map}= {{\displaystyle \sum }}_{n=0}^l\left\{ra{d}_n+ Expand\left(ra{d}_{n+1}\right)\right\} \\ \left(\mathrm{base}\mathrm{layer}\mathrm{for} n=l, \mathrm{detail}\mathrm{layer}\mathrm{for} n=0 ~l-1\right), \end{gathered}$$

(13)

where $ra{d}_n$ denotes the result of the fusion algorithm of the PCs calculated using Equations (10) and (12). $Expand\left(ra{d}_{n+1}\right)$ represents the reconstruction process of the Laplacian pyramid (Equation (3)), and the radiance map denotes the radiance map obtained through the PCA and the Laplacian pyramid reconstruction process.

Figure 5a shows the results of applying the soft-mixing method for the edge layer. If the PCA weight is less than $t{h}_1$, the weight shall be used only for the NIR image pixel value, and then the pixel value should be 1. Conversely, if the PCA weight exceeds $t{h}_2$, the weight should only be used as the pixel value of the VIS image, and then the pixel value should be 1. The PCA weighting map is multiplied by two base-layer images and is adapted to the PCA fusion algorithm given in Equation (12).

Figure 5b shows the results of applying the soft-mixing method for the base layer. The PCA fusion algorithm is run differently from the detail-layer fusion algorithm because the base layer has low contrast. The PCA fusion is lower than the contrast of the VIS image only if the PCA weight of the base layer is used. Therefore, the base layer is considered in two stages of the PCA fusion algorithm.

In the first step, the PCA fusion algorithm is used in a manner similar to edge-layer PCA fusion. The VIS base layer and NIR base-layer images are implemented by the PCA fusion algorithm, as shown in Equation (11).

In the second step, the PCA fusion algorithm used in Equation (12) employs the PCA fusion base obtained from Equation (11) and the NIR base layer. Here, the NIR base layer was used because it contained edge information and higher contrast than the PCA fusion base and the VIS image. Moreover, if it were used with the VIS images, the fused image would have contained shade or reverses light. This would have reduced the tone compression effect.

To generate a radiance map, the Laplacian pyramid algorithm was reconstructed following Equation (13): the lower layer was upsampled and then combined with the previous layer. The lowest layer was an image comprising the base fusion of Equations (11) and (12), which upsampled this composed image and added an edge layer image. The images comprising the base and edge images were also upsampled, and then, the next edge-layer images were added.

However, the Laplacian pyramid-based multiscale image fusion algorithms are difficult for preserving details. It is necessary to use the detailed enhancement component on the Laplacian pyramid [11,12]. Thus, Based on the Stevens effect [6], which is the human visual characteristic of local contrast, a gamma value that changes locally according to regional adaptation luminance was applied to the detail layer to improve local contrast. After generating the radiance map, the proposed algorithm was used to realize the Stevens effect to increase the contrast. The Stevens effect increases local perceptual contrast in the detail layer, and the Stevens effect power function determines the luminance-dependent local contrast enhancement.

$$k=1/\left(5L_A+1\right),$$

(14)

where $L_A$ is a function of 20% of the adaptation luminance

$$F=0.2k^4\left(5L_A\right)+0.1{\left(1-k^4\right)}^2{(5L_A)}^{1/3},$$

(15)

where $F$ is a prediction of a variety of luminance-dependent appearance effects in CIECAM02.

$$detail{s}_{stevens}= {(detai{l}_{Radiance map})}^{{\left(F+0.8\right)}^{1/4}},$$

(16)

where $detail{s}_{stevens}$ is a function that is adapted for the Stevens effect. Further, $detai{l}_{Radiance map}$ is a detail layer of the radiance map expressed in Equation (13).

$$Radiance ma{p}_{fin}=bas{e}_{Radiance map}\ast detail{s}_{stevens},$$

(17)

where $Radiance ma{p}_{fin}$ is the resultant radiance map, which is a combination of the base layer of the radiance map and $detail{s}_{stevens}$. Further, $bas{e}_{Radiance map}$ is a base layer of the radiance map expressed in Equation (13).

In conclusion, the edge and base layers synthesized using PCA were implemented by reconstructing the Laplacian pyramid, and the final image was fused using the Stevens effect to increase local contrast. Thus, the proposed radiance map was obtained.

5. Color Compensation

Tone changing through the image fusion causes color distortions originating from the balance changes of RGB channels in the tone changing. The ratio of RGB signals changes, and color saturation would be reduced [13]. Tone changes should be applied to only the luminance channel, and the process sequences preserve the color information of the visible image. Converting from RGB signals to opponent color signals is necessary for constructing a uniform perceptual color space and correlated of various appearance attributes [14]. To correct this chromatic defect of tone mapping, we adopt a color compensation method that restores a ratio of color to luminance after tone mapping.

After VIS and NIR input images are blended to generate the proposed radiance map, the proposed color compensation uses a ratio between the radiance map and the VIS input image. The color compensation must consider the fact that tone scaling has a brightness gap. Therefore, the color compensation ratio is calculated using the radiance map and the VIS input image in the luminance channel in Equation (18). The ratio represents the variation in the luminance channel between the radiance map and the VIS input image:

$$r=\frac{Radiance map}{L_{VIS}},$$

(18)

where $r$ is the surrounding brightness variation between the VIS input image and the radiance map.

$$L_s={\left(g\left(x,y\right)\ast L_{VIS}\left(x,y\right)\right)}^{\gamma },$$

(19)

where $L_s$ is a surrounding image with gamma correction, and $g\left(x,y\right)$ is a Gaussian blur function. We used a 15 × 15 size; the δ (sigma) value of the Gaussian filter size is 15. $L_{VIS}$ is a VIS luminance channel image. Further, γ is the gamma value. We used the γ value as 0.1.

$$r_{adpt}=r\times L_s+\left(1-L_s\right),$$

(20)

where $r_{adpt}$ is an adapted local brightness variation. Further, $r$ is the surrounding brightness variation between the VIS input image and radiance map given in Equation (18), and $L_s$ represents a surrounding image with gamma correction, as shown in Equation (19).

$$a^{\prime }=\left(a-128.0\right)\times r_{adpt}+128.0,$$

(21)

$$b^{\prime }=\left(b-128.0\right)\times r_{adpt}+128.0,$$

(22)

where $a^{\prime }$ and $b^{\prime }$ are the compensated chrominance channels. The term $r_{adpt}$ is introduced to eliminate color noise in dark areas. This term also increases the color gain in the bright areas and preserves the color, which is obtained from VIS input image, in the dark area. The chrominance channel is compensated by using $r_{adpt}$ as proposed in Equations (21) and (22), where $a$ and $b$ are the chrominance channels in the CIELAB color space. This color compensation preserves the VIS input-image color to allow its natural application to the proposed radiance map.

6. Simulations

The proposed resulting images were evaluated against images obtained using conventional methods. As conventional methods, we considered the Laplacian fusion algorithm [1] and the low-rank fusion algorithm [5]. We used input images (i.e., reference images of the conventional method [1]). Both VIS and NIR input images were captured in the daytime. The conventional method (i.e., the low-rank fusion algorithm [5]) was adapted to the proposed color compensation algorithm to compare the Laplacian fusion algorithm [1] with the proposed method for color images.

The resulting images of the low-rank conventional method (d) contained noise, as shown in Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12. Furthermore, we found that compared to the aforementioned conventional methods, the proposed method better represents the edges of objects such as clouds, trees, leaves, and mountains. If an object is clear in the input VIS image, the output images of the proposed method will preserve the hue of the color information represented in the input VIS image.

The resulting values were calculated from

Table 1. Comparison of image quality metrics.

	Laplacian Pyramid Fusion		Low-Rank Fusion		Proposed Method
	FMI	S3	FMI	S3	FMI	S3
Figure 6	0.2189	0.1541	0.2365	0.1511	0.2821	0.3465
Figure 7	0.2492	0.1731	0.2571	0.1257	0.2668	0.3064
Figure 8	0.2572	0.2524	0.316	0.1776	0.3242	0.382
Figure 9	0.2914	0.1833	0.277	0.164	0.3291	0.3574
Figure 10	0.247	0.2027	0.2622	0.1572	0.3124	0.3582
Figure 11	0.4103	0.2902	0.3465	0.1899	0.3448	0.276
Figure 12	0.2646	0.5073	0.3881	0.3127	0.3993	0.4354
average	0.2769	0.2519	0.2976	0.1826	0.3227	0.3517

to evaluate the image quality and image sharpness for comparing the results obtained using the conventional and proposed methods. Feature mutual information (FMI) [15] is a method of calculating image-fusion quality based on the calculation of information contained in the two input images. S3 [16] is the measurement criteria for image clarity, and it is measured according to the locally recognized sharpness of the image.

In Table 1 and Figure 13, a high two-metric score means that the image is of good quality. The average score of the image evaluation indicator for the proposed method is higher than that of the conventional method. The FMI method of evaluating fusion quality has higher scores than conventional methods. The S3 method of fusion quality evaluation, i.e., sharpness evaluation, has higher scores than the conventional methods. Thus, compared with the results of the conventional methods, the results of the proposed method were evaluated to be much better in terms of picture quality

7. Conclusions

We proposed the synthesis of VIS and NIR images. A radiance map was created using the Laplacian pyramid, and the soft-mixing method was employed based on PCA. Each pyramid level was implemented on the luminance channel in the CIELAB color space. The Laplacian pyramid was decomposed into its edge and base layers. These decomposed layers were calculated by combining the PCA fusion algorithm with the soft-mixing method. In particular, the base-layer soft-mixing method had a low contrast in the base layer. Therefore, the results were calculated differently from the applied PCA fusion algorithm. The base layer was fused with the NIR base layer to enhance the local contrast. Color compensation was used to preserve the VIS input image. To preserve the VIS image color, the proposed color compensation was adjusted by using the ratio of the luminance channels of the radiance map and the VIS input image. When the proposed color compensation process was completed, the fused image was converted from the CIELAB color space to the RGB color space. The advantages of the resulting image obtained by the proposed method are better local contrast and excellent detail and color representation. The possible real-life applications of the proposed method are to increase the visibility for a number of the camera embedded devices, such as a self-driving car, CCTV, drone and car black–box. In future work, we will attempt to fuse VIS and NIR images under nighttime conditions.