Elsevier

Pattern Recognition

Volume 66, June 2017, Pages 313-327
Pattern Recognition

Illumination invariant single face image recognition under heterogeneous lighting condition

https://doi.org/10.1016/j.patcog.2016.12.029Get rights and content

Highlights

  • Two illumination invariant components, namely logarithm gradient orientation (LGO) and logarithm gradient magnitude (LGM), are extracted.

  • An effective post-processing strategy is proposed to integrate both LGO and LGM, generating the logarithm gradient histogram (LGH).

  • Solid theoretical analysis on the illumination invariant properties of the proposed descriptors is presented.

  • Competitive results are reported, both in homogeneous and heterogeneous lighting conditions.

Abstract

Illumination problem is still a bottleneck of robust face recognition system, which demands extracting illumination invariant features. In this field, existing works only consider the variations caused by lighting direction or magnitude (denoted as homogeneous lighting), but the effect of spectral wavelength is always ignored and thus existing illumination invariant descriptors have its limitation on processing face images under different spectral wavelengths (denoted as heterogeneous lighting). We propose a novel gradient based descriptor, namely Logarithm Gradient Histogram (LGH), which takes the illumination direction, magnitude and the spectral wavelength together into consideration, so that it can handle both homogeneous and heterogeneous lightings. Our proposal contributes in three-folds: (1) we incorporate LMSN-LoG filter to eliminate the lighting effect of each image and extract two illumination invariant components, namely logarithm gradient orientation (LGO) and logarithm gradient magnitude (LGM); (2) we propose an effective post-processing strategy to make our model tolerant to noise and generate a histogram representation to integrate both LGO and LGM; (3) we present solid theoretical analysis on the illumination invariant properties of our proposed descriptors. Extensive experimental results on CMU-PIE, Extended YaleB, FRGC and HFB databases are reported to verify the effectiveness of our proposed model.

Introduction

Face recognition has attracted significant attention in the last decades owing to its wide range of applications, including information security, law enforcement, video surveillance, cooperative user applications, etc. Many research efforts have focused on the face recognition problem under relatively well-controlled conditions using sufficient training data [1], [2], [3], [4]. However, in practical application, the performance of face recognition system is greatly affected by the illumination condition. Face recognition under wide range of lighting variations is still an open issue, especially when there is only one sample available for each person, which is a common scenario in many security system.

This work focuses on single-image-based face recognition model for handling illumination variations. As indicated by early research [5], [6], [7], the accuracy of a recognition system heavily depends on the number of training samples for each person. However, in many large-scale identification applications, such as law enforcement, driver license or passport card identification, there is usually only one sample per person stored in the database. The main challenge is how to guarantee robust performance under this extremely small sample size condition for each person. Solutions to this problem can be roughly divided into two aspects [8], i.e., the holistic methods [9], [10], [11] and the local methods [12], [13], [14], [15]. The former one extracted globally stable facial features and the latter one described each face image as a batch of local features which are assumed to be invariant to lighting variation.

The illumination problem, as one of the main challenges for existing face recognition systems, has become a barrier in many face related applications. The well-known face recognition vendor test (FRVT) 2006 [2] has revealed that large variation in illumination would probably affect the performance of face recognition algorithms. A variety of works have been proposed to address this issue and they mainly fall into three categories [16]: preprocessing and normalization techniques [17], [18], illumination modeling based approaches [19], [20] and invariant feature extraction [14], [21], [22], [23], [24], [25].

More specifically, preprocessing and normalization methods like histogram equalization (HE) [17], gamma correction [26] and homomorphic filtering [26] attempt to take holistic normalization on face images such that the restored images appear to be consistent with those under normal illumination. However, most of these methods are ad hoc and hard to obtain satisfactory results when suffering uneven lighting conditions, although their visual effect appears acceptably in some cases. To further investigate the cause of illumination problem, the modeling based approaches turn to exploring the mechanism of face imaging. Based on the assumption that the surface of face is Lambertian, images of the same face under varying lighting conditions can be approximated by a low dimensional linear subspace [19], [27]. The relationships between 2D illuminated images and corresponding 3D facial surface are investigated in some advanced work, e.g., morphing face [28] and shape from shading (SFS) [29]. In theory, these methods model the illumination variation quite well, but they need a great deal of training samples to learn the variation and easily suffer from the over-fitting problem, which largely restricts their use in real applications, especially when only single image per person is available.

Compared to the above two categories, most invariant feature based methods are more effective and do not demand learning. Classical methods such as Local Binary Pattern (LBP) [14], Gabor [30] and their variations [30], [31], [32] are widely conceived to be robust to slight illumination change, but their performance is no longer guaranteed when the lighting condition becomes severe. To overcome this problem, a variety of state-of-the-art methods have been proposed by extracting the reflectance component [21], [22], [16] or alleviating the illumination component based on the Lambert's reflectance model [25], [24], and great success using these effective methods has been reported.

The illumination problem mentioned above is mainly due to the variations caused by either varying lighting direction or varying lighting magnitude. In those methods, a main assumption they make is that the wavelengths of light are the same, and this case is called the homogeneous lighting in this paper. However, it is not always the case in realistic applications. For example, the lighting wavelengths of indoor and outdoor conditions are always different, so is the case of visible (VIS) and near infrared (NIR) spectral face images [33]. As a result, the reflectance component which is determined by the albedo and the normal direction of facial surface will change, since the albedo is related to the spectral wavelength. Some related works like [33], [34] consider the VIS–NIR face matching as a multi-modality face recognition problem rather than an illumination related task. For convenience, we denote the lighting condition with different spectral wavelengths as heterogeneous lighting. As far as we know, there is still lack of work on solving the heterogeneous lighting problem, and theoretical development on a valid image descriptor in this aspect is also limited.

In this paper, we formulate a new illumination invariant feature extraction model for both homogeneous and heterogeneous lightings. In particular, we propose a novel gradient based descriptor, namely logarithm gradient histogram (LGH), to eliminate the illumination variations. We have provided an in-depth analysis on its illumination invariant property. In addition, we also introduce a multi-scale bank-pass filtering as a preprocessing to constrain the illumination effect and enhance facial information. Experiments on several public face databases have been conducted to verify the effectiveness of our proposed method.

The rest of this paper is organized as follows: Section 2 further discusses the invariant feature extraction approaches. Section 3 introduces our proposed descriptor in detail. After that, the theoretical proof of illumination invariant properties of our proposed method is presented in Section 4. Experiments on CMU-PIE, Extended YaleB, FRGC v2.0 and HFB databases and further analysis will be carried out in Section 5 to evaluate our proposed LGH. Finally, the conclusion of this paper is drawn in Section 6.

Section snippets

Related work

In this section, we mainly review related work on illumination invariant descriptors for face recognition. In addition, some work about face recognition using single training image under illumination will be discussed at the end of this section.

According to the Lambertian Law, the intensity of the illuminated image I can be formulated as I(x,y)=F(x,y)L(x,y), which is a product of the illumination component L(x,y) and the reflectance component F(x,y). As commonly assumed, the L(x,y) in the

The approach

In this section, we aim to extract gradient based illumination insensitive components. Specifically, we will elaborate our proposed LGH in three folds: (i) the multi-scale bank-pass filtering used for constraining the illumination effect and enhancing facial information; (ii) two illumination invariant components, i.e., logarithm gradient orientation (LGO) and logarithm gradient magnitude (LGM), and the theoretical analysis on the illumination invariant property; (iii) post-processing for

Illumination invariant analysis

In the former section we claim that LGO and LGM are two illumination insensitive components. In this section, we will prove the illumination invariant property of our proposed features based on the following theorems.

Experiments

In this section, we conducted a series of experiments to evaluate the proposed illumination invariant descriptor for single-image-based face recognition. Three scenarios will be considered in our experiments: (1) the case when images were approximately captured under controlled homogeneous lighting, i.e., with different lighting directions and magnitudes but with the same spectral wavelength; (2) the case when images were captured in uncontrolled lighting conditions, e.g., outdoor environment;

Conclusions

In this paper, we have proposed a novel illumination invariant descriptor LGH to address the illumination problem in face recognition. Different from the existing models, we consider variations caused by the lighting direction, magnitude and the spectral wavelength together, so that the new proposed descriptor is able to handle face recognition under both homogeneous and heterogeneous lighting conditions. We have proposed two illumination invariant components in the logarithm domain after

Acknowledgments

This work was supported partially by the NSFC(Nos. 61522115, 61472456, 61573387, 61661130157), the GuangDong Program (Grant No. 2015B010105005), Guangdong Program for Support of Top-notch Young Professionals (No. 2014TQ01X779), and the China Postdoctoral Science Funds (No. 2015M582469).

Jun-Yong Zhu received his M.S. and Ph.D. degrees in applied mathematics in the School of Mathematics and Computational Science from Sun Yat-sen University, Guangzhou, PR China, in 2010 and 2014, respectively. Currently, he is working toward the post-doctoral research in the Department of Information Science and Technology, Sun Yat-Sen University. His current research interests include heterogeneous face recognition, visual transfer learning using partial labeled or unlabeled auxiliary data and

References (59)

  • A.K. Jain, B. Chandrasekaran, Dimensionality and sample size considerations in pattern recognition practice, in:...
  • M. Turk et al.

    Eigenfaces for recognition

    J. Cogn. Neurosci.

    (1991)
  • P. Belhumeur et al.

    Eigenfaces vs. Fisherfacesrecognition using class specific linear projection

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1997)
  • J. Yang et al.

    Two-dimensional pcaa new approach to appearance-based face representation and recognition

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2004)
  • D. Beymer, T. Poggio, Face recognition from one example view, in: Proceedings of the 5th IEEE International Conference...
  • A.M. Martínez

    Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class

    Pattern Anal. Mach. Intell.

    (2002)
  • R. Brunelli et al.

    Face recognitionfeatures versus templates

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1993)
  • L. Wiskott et al.

    Face recognition by elastic bunch graph matching

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1997)
  • T. Ahonen et al.

    Face description with local binary patternsapplication to face recognition

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2006)
  • S. Shan, W. Gao, B. Cao, D. Zhao, Illumination normalization for robust face recognition against varying lighting...
  • A. Georghiades et al.

    From few to manyillumination cone models for face recognition under variable lighting and pose

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2001)
  • H. Shim et al.

    A subspace model-based approach to face relighting under unknown lighting and poses

    IEEE Trans. Image Process

    (2008)
  • H. Wang, S. Li, Y. Wang, Generalized quotient image, in: Proceedings of the IEEE Computer Society Conference on...
  • T. Chen et al.

    Total variation models for variable lighting face recognition

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2006)
  • X. Tan, B. Triggs, Enhanced local texture feature sets for face recognition under difficult lighting conditions, in:...
  • T. Zhang et al.

    Face recognition under varying illumination using gradientfaces

    IEEE Trans. Image Process.

    (2009)
  • B. Wang et al.

    Illumination normalization based on Weber's law with application to face recognition

    IEEE Signal Process. Lett.

    (2011)
  • M. Sonka, V. Hlavac, R. Boyle, Image Processing: Analysis and Machine Vision, Chapman & Hall, London,...
  • R. Basri et al.

    Lambertian reflectance and linear subspaces

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2003)
  • Cited by (30)

    • Controlling the initiating time of ventilation facilities by spatial analytic method

      2022, Journal of Building Engineering
      Citation Excerpt :

      By benefiting from the improvement of hardware equipment of pattern recognition technology, there have been considerable research results in various fields, such as agriculture [28], fishery [29], architecture [30,31], and even people indoor-falling recognition are certainly applicable [32]. And among them, the identification method of object detection can identify people [33], animals [34] and cars [35] so fast. Therefore, the combination of the IoT and object detection can take more spatial analysis into consideration.

    • Recent development in face recognition

      2020, Neurocomputing
      Citation Excerpt :

      The method has achieved 81.36 ± 4.11% recognition rate (mean ± standard deviation) when tested on YALE dataset [13] having one image as a training image. A gradient based descriptor to address illumination variations has been proposed in [29]. The method handles magnitude, lighting direction and the spectral wavelength together.

    • Adaptive illumination-invariant face recognition via local nonlinear multi-layer contrast feature

      2019, Journal of Visual Communication and Image Representation
      Citation Excerpt :

      Gradient and edge extraction methods compute gradients of a face image as an illumination-insensitive representation since gradients are more stable than the pixel intensities under different lighting conditions [5]. Representative methods include Local Binary Patterns (LBP) [6], Local Ternary Patterns (LTP) [7], Enhanced LDP (EnLDP) [8], Local Directional Number patterns (LDN) [9], Discriminant Face Descriptor (DFD) [10], Logarithmic Fractal Dimension(LFD) [11], discriminative multi-layer illumination-robust feature extraction(DMI) [12], Logarithm Gradient Histogram (LGH) [13] and Binary Gradient Patterns (BGP) [14]. More specifically, DMI model first takes full advantage of these useful information in large-scale features for face recognition.

    • An intelligent machine vision-based smartphone app for beef quality evaluation

      2019, Journal of Food Engineering
      Citation Excerpt :

      Invariant features extraction is still a key challenge in pattern recognition and image processing. Accordingly, several image processing algorithms have been developed to extract invariant features from images captured under uncontrolled conditions in order to improve the performance of vision systems (Legaz et al., 2018; Zhu et al., 2017), but the developed algorithms have rarely been used in smartphone-based image processing (Casanova et al., 2013). Therefore, this study was conducted for the first time to create a new smartphone app on the basis of fresh beef image captured under non-standardized and uncontrolled conditions for estimating tenderness (WBSF value) of the fresh beef sample.

    • Mixed neighborhood topology cross decoded patterns for image-based face recognition

      2018, Expert Systems with Applications
      Citation Excerpt :

      Furthermore, our system achieved the top accuracy of 99.81% on Subset 4 and ranked second on Subset 5 by achieving a score of 98.52% surpassed by Zhu, Zheng, Lu, and Lai (2017a) (only with only 0.37%) which reached 98.89%. Note that the method presented in Zhu et al. (2017a) was not able to reach 100% on Subset 4 . AR Database: On AR dataset, the MNTCDP based system managed to outperform all reported state-of-the-art systems.

    View all citing articles on Scopus

    Jun-Yong Zhu received his M.S. and Ph.D. degrees in applied mathematics in the School of Mathematics and Computational Science from Sun Yat-sen University, Guangzhou, PR China, in 2010 and 2014, respectively. Currently, he is working toward the post-doctoral research in the Department of Information Science and Technology, Sun Yat-Sen University. His current research interests include heterogeneous face recognition, visual transfer learning using partial labeled or unlabeled auxiliary data and non-linear clustering. He has published several papers in international conferences such as ICIP, AMFG and ICDM. His cooperative ICDM 2010 paper won the Honorable Mention for Best Research Paper Awards and his CCBR 2012 paper won the Best Student Paper Awards.

    Wei-Shi Zheng received the Ph.D. degree in applied mathematics from Sun Yat-Sen University in 2008. He is now a professor at Sun Yat-sen University. He had been a postdoctoral researcher on the EU FP7 SAMURAI Project at Queen Mary University of London and an associate professor at Sun Yat-sen University after that. He has now published more than 80 papers, including more than 40 publications in main journals (TPAMI, TNN, TIP, TSMC-B, PR) and top conferences (ICCV, CVPR, IJCAI, AAAI). He has joined the organization of four tutorial presentations in ACCV 2012, ICPR 2012, ICCV 2013 and CVPR 2015 along with other colleagues. His research interests include person/object association and activity understanding in visual surveillance. He has joined Microsoft Research Asia Young Faculty Visiting Programme. He is a recipient of excellent young scientists fund of the national natural science foundation of China, and a recipient of Royal Society-Newton Advanced Fellowship.

    Feng Lu received his M.S. and Ph.D. degrees in computer application in the School of Mathematics and Computational Science from Sun Yat-sen University, Guangzhou, PR China, in 1999 and 2002. He had been a postdoctoral researcher in the Department of Finance Bureau, Guangzhou, Guangdong during 2002–2004. Currently, he is the office director of Department of Public Security of Guangdong Province. His research interests include scientific and technological information in public security, automatic crime data analysis, intelligent analysis and application technology for surveillance video.

    Jian-Huang Lai received his M.S. degree in applied mathematics in 1989 and his Ph.D. in mathematics in 1999 from Sun Yat-Sen University, China. He joined Sun Yat-sen University in 1989 as an assistant professor, where currently, he is a professor with the Department of Data and Computer Science. His current research interests are in the areas of digital image processing, pattern recognition, multimedia communication, wavelet and its applications. He has published over 80 scientific papers in international journals and conferences on image processing and pattern recognition, e.g. IEEE TPAMI, IEEE TNN, IEEE TIP, IEEE TSMC (Part B), Pattern Recognition, ICCV, CVPR and ICDM. Prof. Lai serves as a standing member of the Image and Graphics Association of China and also serves as a standing director of the Image and Graphics Association of Guangdong.

    View full text