Nonuniformity Correction of Single Infrared Images Based on Deep Filter Neural Network

Abstract

The fixed-pattern noise (FPN) caused by nonuniform optoelectronic response limits the sensitivity of an infrared imaging system and severely reduces the image quality. Therefore, nonuniform correction of infrared images is very important. In this paper, we propose a deep filter neural network to solve the problems of network underfitting and complex training with convolutional neural network (CNN) applications in nonuniform correction. Our work is mainly based on the idea of deep learning, where the nonuniform image noise features are fully learned from a large number of simulated training images. The network is designed by introducing the filter and the subtraction structure. The background interference of the image is removed by the filter, so the learning model is gathered in the nonuniform noise. The subtraction structure is used to further reduce the input-to-output mapping range, which effectively simplifies the training process. The results from the test on infrared images shows that our algorithm is superior to the state-of-the-art algorithm in visual effects and quantitative measurements, providing a new method for deep learning in nonuniformity correction of single images.

1. Introduction

Infrared focal plane array detectors (IRFPAs) are the main components of infrared imaging technology. For an ideal infrared focal plane array, there should be a consistent response output with the same incident radiation. However, in real situations, due to factors such as material quality and manufacturing process, each array element has slight differences in impedance, capacitive reactance, thermal area, and resistance temperature, making the actual response output inconsistent. Such a nonuniform response causes fixed-pattern noise (FPN) in images, which seriously restricts the application of IRFPAs [1]. If the nonuniformity of infrared images cannot be effectively corrected, it is difficult to separate the target signal from the background.

To solve this problem, researchers have proposed two kinds of nonuniformity correction (NUC) methods. The first method is based on calibration. The advantage of the calibration method is that the algorithm is simple, but in practical applications, the reference source needs to be repeatedly scaled [2]. In order to avoid the problem of repeated calibration, other types of scene-based nonuniformity correction algorithms, such as neural network [3], constant statistics constraint [4], and interframe registration algorithm [5], have been proposed. However, if the image difference between two sequential frames is too large, the previous frame will leave a residual image on the current frame, which will result in “ghosting” artifacts. Therefore, removing the FPN in a single frame by a priori information is currently the main research direction. Recent state-of-the-art methods for single frames include midway histogram equalization (MHE) [6], total variation [7], guided filter [8], non-local means (NLM) [9], and gradient constraint [10]. However, all of these methods have drawbacks: (1) Infrared images have fewer details than visible images, and some structural information may be lost when FPN is removed. (2) Most methods require manual parameter testing and feature extraction to get good corrections, which is a complicated task.

In recent years, as the amount of data has grown and computing power has increased, many new artificial intelligence methods have emerged, including reinforcement learning, transfer learning, and deep learning. Reinforcement learning can operate with only limited knowledge of the environment and with limited feedback on the quality of the decisions, and it is widely used in smart grid networks [11], communication networks [12], etc. Transfer learning can migrate the model for big data to small data and realize personalized migration, which is the direction of future development. Deep learning exhibits the best performance in image processing, such as image classification [13], target detection [14], face recognition [15], and super-resolution [16]. The reason is that through deep learning, the network automatically learns various information features of the image and saves it in the hidden layer [17]. Considering the huge advantages of deep learning in image processing, this paper attempts to apply deep learning to the correction of nonuniform infrared images. In the network architecture we designed, instead of learning the mapping between noise and noiseless pairs, we first process the image with a bilateral filter to obtain high-frequency components with nonuniform noise. Then, the high-frequency component is used as the input for the deep neural network to obtain nonuniform noise, and the nonuniform noise is subtracted from the original image to generate the NUC result. This network model can reduce the mapping range from input to output and make the learning process easier. Moreover, we remove the pooling layer to reduce the loss of information, and add a batch normalization layer to solve the problem of network overfitting. Compared with other state-of-the-art handcrafted methods, the proposed method has some advantages. First, using the trained network model can achieve fast speed for nonuniform infrared images in real time. Then, our deep learning–based method optimizes the NUC parameters by learning directly, with no need to manually extract features and parameters. Finally, by setting the network structure, we can effectively remove fixed-pattern noise while preserving background details. In general, our contributions are summarized as follows:

• A nonuniformity correction method based on deep learning is proposed. Combined with the nonuniform noise model, large numbers of simulated infrared images are used to train our network structure, which can effectively correct real nonuniform infrared images.
• In our network structure, the use of the deep network structure improves the learning ability of the network and solves the underfitting problem. Taking the method of using high-frequency information as input to the network effectively eliminates background interference, and using the subtraction structure to distinguish between fixed-mode noise and noise-free images reduces the mapping range during training, which makes the learning process of the deep model easier.

Figure 1 shows an image with nonuniform noises and the corrected image.

The remainder of this paper is as follows. Section 2 introduces the related work. Section 3 presents our nonuniform correction method. Experiments in Section 4 demonstrate model parameter settings and model performance. Section 5 is our conclusions.

2. Related Work

The methods based on calibration and scene can correct nonuniform infrared images, but the calibration method is affected by the environment and the scene method will produce ghost phenomena. Therefore, this paper focuses on nonuniformity correction from a single image. The nonuniformity correction based on a single image that has been proposed so far can be divided into the following categories.

The first category is the use of data and noise characteristics observed in infrared images. Tendero and Gilles [6] used the local histogram mean and Midway Histogram Equalization (MHE) to adjust intensities of pixels within a column to remove the fixed-pattern noise on a single-frame image. Jian et al. [18] developed a compressive sensing based on MHE, which greatly reduces the calculation data and improves the correction speed. However, the calculation of midway histogram data is complicated, and the method replaces the histogram of each column with the midway histogram of the adjacent columns, which reduces the spatial resolution of the image to some extent. Chang et al. [19] proposed a low-rank-based single-image decomposition model (LRSID) to separate the original image from the FPN perfectly.

The second category uses appropriate filters to remove FPN. Cao [8] proposed the use of guided filter (GF) to subtract the estimated noise from the original infrared image for nonuniform correction, which can achieve a good balance between noise removal and detail preservation effectively. Münch et al. [20] proposed a method based on wavelet decomposition and Fourier transform (WD-FT), in which FPN is removed in the Fourier domain of the vertical wavelet component. Those methods retain the image details, but some vertical structure will be erroneously deleted during the correction process.

The third category is a method based on an optimization model aimed at estimating a corrected image by optimally ensuring correction of the fringe FPN and retention of image details. Zhao et al. [10] proposed a gradient constraint method in which the gradient in the direction of the fringes is maintained while minimizing the energy of the gradient in the opposite direction. Boutemedjet et al. [21] improved that method and proposed a correction scheme based on the unidirectional variation model. The directional characteristics of FPN were combined with edge perceptual weighting to enhance the correction performance around the strong edges to solve the problem of loss of structure information in the above method.

Kuang et al. [17] proposed a new category called Stripe noise removal convolutional neural network (SNRCNN). Based on the use of deep convolutional networks, they used image denoising and super-resolution to eliminate FPN in the input and reconstruct the image at the output. This method is by far the most relevant to our present research work. SNRCNN uses one convolutional layer for nonuniformity correction, and then uses two convolutional layers for super-resolution reconstruction, which achieves good NUC results, but there are still some problems. First, the number of convolutional neural networks used by SNRCNN is small, so there is a problem of network underfitting. The FPN characteristics cannot be fully studied, and some FPN remains after correction. For this problem, we deepened the number of layers in the network and removed the pooling layer to reduce the loss of information, while using batch normalization to prevent network overfitting. Second, SNRCNN learns mapping relationships between nonuniform infrared images and corrected infrared images directly. This is complicated in the training process. In order to simplify the learning process, in our network structure, a high-frequency component containing FPN is first extracted by using a filter, and then a high-frequency component is used as input to train FPN, and the output is separated by the subtraction structure to get the NUC. The benefit of this operation is that the background interference is removed, while the mapping range is reduced, and even when the number of layers is more than SNRCNN, better convergence can be achieved. Finally, SNRCNN uses a red–green–blue (RGB) image training set; however, in the infrared field, the details of the infrared image are weaker than those of the visible image. Models trained with such a training set may easily lose the structure information in the images. Therefore, we use the nonuniform image generation model to construct a simulated infrared image training set, and after training, it can be used for real nonuniform infrared images.

3. Our Methods

In this paper, a high-frequency component I_H and a low-frequency component I_L are separated by using a bilateral filter for a single image I containing nonuniform noise; the high-frequency component I_H contains image structure details and nonuniform noise, which are used as the input for the network model, and through the network model, nonuniform noise can be fitted. Finally, the subtraction structure of the network model will be used to get the corrected image.

3.1. Nonuniformity Model and Training Dataset

The use of a large number of infrared images to train our network model is essential for nonuniform infrared image correction. However, we lack access to the ground truth for nonuniform infrared images, and it is difficult to obtain pairs of images with FPN and clean infrared images in reality. According to the characteristics of nonuniform infrared images, we designed an image model based on which clean infrared images can be used to synthesize datasets with nonuniform noise images to train the network effectively.

On a single nonuniform infrared image, nonuniform noise can be represented as fixed additive noise [21], so the acquired image can be divided into two parts:

y(i,j) = x(i,j) + n(i,j),

(1)

where n(i,j) is nonuniform noise, x(i,j) is the real infrared original image, and y(i,j) is the nonuniform infrared image.

In this paper, we obtained clean infrared images from the literature [17] and the FIR Sequence Pedestrian Dataset [22]. Based on the above model, for each noise-free image x, we added a simulated column FPN n computed based on Equation (1) to generate its noisy image. We obtained a total of 220 pairs of infrared images with a resolution of 360 × 512. As a training set for this paper, some training samples are shown in Figure 2. In this paper, 200 pairs were used as training sets, 20 pairs were used as test sets, and each image was divided into 64 × 64 image blocks for network input.

3.2. Network Design

As shown in Figure 3, our network consists of 3 parts: high- and low-frequency separation, nonlinear mapping learning, and image reconstruction of the subtraction structure.

3.2.1. High-Frequency Information Extraction

The nonuniform image is processed by a low-pass filter and can be divided into a spatial low-frequency component and a spatial high-frequency component. The background information has continuous correlation and is characterized by the low-frequency component; the nonuniformity is generated independently by each detector array element, and the output nonuniformity difference of each array element is large, which is characterized by high frequency [23]. Using bilateral filters to effectively separate spatial high-frequency and low-frequency information, after removing the low-frequency information from the image, the background interference is removed, leaving only the nonuniform noise and the object structure in the high-frequency information. It can be seen from Figure 4 that the use of a bilateral filter to separate the high and low frequencies of the nonuniform infrared image can effectively remove the background interference and make a large number of pixel values close to zero. This sparsity can also improve the effect of nonuniform correction. The high-frequency information shows a significant range reduction, which indicates that the mapping space has shrunk, so network performance should be improved.

3.2.2. Mapping Learning and Subtraction Structure

Suppose the infrared image with nonuniform noise is Y and the ground truth image is Z, using a bilateral filter for image Y, we get the high-frequency component I_H that removes background interference, the network takes I_H as input and outputs nonuniform noise N, and the network parameters are learned through nonlinear mapping. Finally, in the subtraction structure we subtract the estimated nonuniform noise N from the original Y to obtain the final NUC estimated result X. In order to learn more image features, we increase the number of layers in the network. In this step, a shortcut connection is used to avoid the gradient disappearing due to the excessive number of network layers. The shortcut connection neither introduces external parameters nor increases computational complexity. For input i, the output after passing through the 2-layer network is added to the original i to obtain a new output [24]. To increase the nonlinearity, the rectified linear unit (ReLU) is used as an activation function after each convolutional layer, then the feature maps are aggregated to generate the output of nonuniform noise.

Based on the previous discussion, we define the objective function as follows:

N_i = M(I_Hi),

(2)

X_i = Y_i − N_i,

(3)

$$\mathrm{Loss}=\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n\Vert X_i-Z_i{\Vert }_F^2,$$

(4)

where i is the image of each training pair (there are n pairs), M stands for network mapping, and F is the Frobenius norm. Training is carried out by minimizing the loss using a minibatch gradient descent based on back propagation.

In the design of the network, for the first layer, we use a convolution kernel of size c × f1 × f1 × a to generate a feature map, where f1 represents the size of the convolution and c represents the number of image channels. A convolution kernel of size a × f2 × f2 × a1 is used to generate a1 feature maps for the middle layers. For the last layer, we use a convolution kernel of a1 × f3 × f3 × c to estimate the nonuniform noise. The NUC image is obtained by directly subtracting the estimated nonuniform noise from the nonuniform infrared image.

Our middle layers can be expressed as follows:

X_l = max(0,W_l*X_l−1 + B_l),

(5)

X_l+1 = max(0,W_l+1*X_l + B_l+1) + X_l−1,

(6)

where l is the number of layers of convolution, W is the weight of the convolution, B is the bias, and max(·) indicates the ReLU activation function [25].

3.3. More Analysis

• In this paper, the subtraction structure method is adopted to make the network learn the difference between the nonuniform infrared image and the clear infrared image during the training process, instead of directly learning the noise-free infrared image; this method can effectively reduce the mapping range. This makes the training process easier, and the subtraction structure can also propagate nondestructive information directly throughout the network, which is very useful for estimating the final normal infrared image.
• An important concept of neural networks is the pooling layer, which usually comes after the convolutional layer. Although the pooling layer can reduce the dimension of the output data, the pooling operation will bring information loss. Therefore, in order to avoid losing part of the information during the training process, all the pooling layers in our network are removed, and at the same time, we used a batch normalization layer to solve the problem of network overfitting and alleviate internal covariate shift [26]. The role of batch normalization is to perform a normalization process on the data. This method can prevent the input distribution of the hidden layer from constantly changing, calculate the mean and variance of the batch data, and normalize the batch data to reduce the input distribution of each hidden layer node to (−1, 1), which reduces the input space and the difficulty of tuning.

4. Experimental Results and Analysis

4.1. Parameter Settings

According to the experimental research experience, we set the number of network layers L to 20, the size of the convolution kernel is f1 = f2 = f3 = 5 and the number of convolution kernels is a = a1 = 16, and the number of channels c = 1, both of which are described in Section 3.2.2. According to the best effect of the bilateral filter, we set the radius of the bilateral filter to 15, the spatial similarity parameter to 2, and the range parameter to 0.3.

4.2. Training

In this paper, we use the stochastic gradient descent (SGD) optimization algorithm [27]. The advantage of this algorithm is that the training speed is fast. The specific idea of this algorithm is that the objective function iteratively updates the parameters in the opposite direction of the gradient according to a certain learning rate until convergence. Since the sample parameters are randomly selected to update the model parameters, each learning is very fast. The minibatch is set to 32, the initial learning rate is set to 0.01, and the momentum parameter is set to 0.9. This parameter plays a smoothing role and can make the SGD algorithm more stable and normalize the training quickly. We can see that the model can achieve convergence after 50,000 iterations. The algorithm was trained on the NVIDIA GTX 1080 Ti, a single GPU. We used TensorFlow software to train our deep neural networks. Figure 5 shows a graph of test loss and training loss.

Under the same training settings, we compared the proposed algorithm with the SNRCNN algorithm on the convergence curve, as shown in Figure 5b. The convergence performance in this paper is significantly better than SNRCNN; our network converged much faster and more accurately, because the subtraction structure and high-frequency information were used. The combination effectively increases sparsity and reduces the mapping range, reduces the learning complexity, and improves the convergence efficiency during training.

4.3. Algorithm Comparison and Quality Evaluation

We compared our method with state-of-the-art single image–based NUC solutions: the MHE-based method, the guided filter (GF) method, and the deep SNRCNN. In order to objectively compare the improved image quality with the proposed algorithm and other methods, the experimental results were analyzed by four evaluation indicators: processing time (T), peak signal-to-noise ratio (PSNR) [28], root mean squared error (RMSE) [29], and image roughness (ρ) [30].

PSNR is one of the most commonly used parameters for image quality evaluation and is defined as follows:

PSNR = 10 × log (255/RMSE).

(7)

RMSE is the squared error:

$$\mathrm{RMSE}=\sqrt{\frac{{{\displaystyle \sum }}_{n=1}^N(I^n-P^n)}{N}},$$

(8)

where N is the size of the image, I is the input image, and P is the output image.

The definition of image roughness index ρ, which is used to measure the high-pass content of an image, is:

$$\rho =\frac{\Vert h*P{\Vert }_1+\Vert h^T*P{\Vert }_1}{\Vert P{\Vert }_1},$$

(9)

where P denotes the restored image to evaluate, h = [1,−1] is the horizontal mask, h^T = [1,−1] is a vertical mask, and ‖•‖₁ represents the L1 norm. The smaller the value of ρ, the better the image will be.

Figure 6 shows the qualitative results of nonuniform image correction for each method. At the same time, the results of the algorithm are evaluated according to the above evaluation criteria as

Table 1. Evaluation parameters of Original1. RMSE, root mean squared error; PSNR, peak signal-to-noise ratio; ρ, image roughness; T, processing time.

Algorithms	RMSE	PSNR (dB)	ρ	T (s)
GF	47.56	14.62	0.33	0.38 s
MHE	11.80	26.69	0.47	182.54 s
SNRCNN	5.39	33.50	0.57	1.86 s
Ours	0.33	57.87	0.24	0.73 s

and

Table 2. Evaluation parameters of Original2.

Algorithms	RMSE	PSNR (dB)	ρ	T (s)
GF	31.13	18.30	0.39	0.42 s
MHE	12.05	26.51	0.56	186.86 s
SNRCNN	5.28	33.66	0.65	2.16 s
Ours	0.37	56.83	0.28	0.75 s

.

4.4. Analysis of Experimental Data

Compared with other algorithms, the research algorithm in this paper can effectively correct nonuniform noise and is not affected by background interference. The guided filter (GF) correction method removes the nonuniform noise to a certain extent, but the image resolution is reduced and the edge information is blurred, which greatly affects the correction effect. MHE can effectively perform nonuniform correction on the image, but still has some streak noise remaining, and the correction time is too long. SNRCNN can correct nonuniform images very quickly, but because of the small number of training network layers, there is a problem of network underfitting, which can be seen in Figure 6d,i; under the algorithm, some of the stripes have not been removed. The method we propose is significantly better than the alternative solutions because it can better remove FPN and preserve the original details and accurately correct the nonuniform noise in the visual sense. In the evaluation index, the PSNR parameter reached the highest, 57.87 and 56.83 (the larger the PSNR, the smaller the distortion), the image roughness reached the lowest, 0.24 and 0.28, and real time reached 0.73 s and 0.75 s.

5. Conclusions

In this paper, our main contribution is to propose an end-to-end deep learning method for nonuniformity correction of single infrared images. By simulating nonuniform infrared images, a large number of training images are used to obtain the best correction model. Combining the high-frequency information and structure of the infrared image to remove the background interference and reduce the mapping range makes the training easier. Compared with the existing state-of-the-art algorithms, our algorithm has many advantages, including no need to manually extract image features and specific parameters, more reliable removal of FPN, and no artifacts. Of course, deep learning currently faces problems such as long training time, hardware requirements, and reliance on a large number of datasets. So in future work, we plan to expand the training model to handle more types of noise, further optimize our network for faster training, and apply our approach to hardware implementation for better practical value.