Face sketch-photo synthesis and recognition: Dual-scale Markov Network and multi-information fusion☆
Introduction
IN real criminal cases, the photo image of a suspect is not available, because of lack of cameras or the bad quality of camera. Fortunately, the witnesses who are found in most cases can provide certain assist for the police. Professional forensic artist creates sketch drawing of suspect by assisting the witness to recall the appearance characteristics of the criminal suspect. Then the sketch is disseminated to law enforcement officers and media outlets with the hope that someone will recognize the individual and provide relevant information of the suspect. Due to the great help of sketch in law enforcement, a growing number of researchers have been devoted to explore the sketch face recognition methods. But the sketches and photos are two different modalities from different sources that bring great difficulty for identification study. Traditional manual method to find the corresponding photo of probe sketch is feasible, but it costs too much time and manpower. In order to solve the above problems, a face sketch-photo synthesis and recognition method is proposed which is based on dual-scale Markov Network and Multi-information fusion.
Face sketch-photo synthesis is to transform the photo (sketch) into sketch (photo). A typical method is searching for the neighbor photo patches of the test photo and creating the target sketch by a linear combination of the corresponding neighbor sketch patches. The selection of the patch scale is a significant factor in the synthesis procedure. A large scale can lead to the coarse synthesis image with distortion and mosaic effect, while a small scale neglects the connection of face structures. Most existing methods use single patch scale on the sketch-photo synthesis process, which ignore the influence of patch scale. In view of the above problem, this paper proposes a sketch-photo synthesis method based on dual-scale Markov Network.
After synthesis process, sketches and photos are in the same modality, which are needed to do further recognition. According to the famous “Thatcher illusion” [1], cognitive psychologists believe that two kinds of information (structural information and feature information) are utilized for face recognition. Structural information refers to the spatial relations between facial features, and feature information refers to the feature of single facial components. Structural information and feature information are different from each other, but are essential and complementary in face recognition process. Therefore, this paper proposes a data fusion recognition method based on human cognition, which takes both the structural information and feature information into consideration for face recognition.
The main contributions of this paper are summarized as follows:
- (1)
Dual-scale Markov Network-based sketch photo synthesis method is presented to synthesize sketch and photo, which uses larger and smaller scales Markov Network sequentially.
- (2)
Multi-information fusion-based sketch face recognition approach is presented, which utilizes structural information and feature information for sketch face recognition according to Face Recognition Cognitive Theory.
- (3)
Leading accuracy rates are achieved on multiple sketch face databases, which prove the effectiveness of the proposed method.
The arrangement of the rest of this paper is as follows. Section 2 introduces several representative methods on SFR. The proposed sketch face synthesis and recognition method via dual-scale Markov Network and Multi-information is presented in Section 3. Section 4 shows experimental results and analyses on different sketch face databases.
Section snippets
Related works
At present, the studies of sketch face recognition mainly divided into two categories: sketch/photo transformation and matching sketches and photos directly. For the aspect of sketch/photo transformation, Tang and Wang [2], [3], [4], [5], [6], [7] proposed some methods. One is that synthesized a pseudo sketch by applying Eigen-transformation on entire face photo which used for matching in sketch modality [2], [3] and they improve the synthesis framework by applying Eigen transformation on local
Dual-scale Markov Network synthesis
Given a test photo t, training sets p and s, we evenly divide the images into N overlapped patches based on large scale. For the ith test photo patch , we firstly search for the nearest K neighbor photo patches {} of test photo patch among the training set. And the corresponding neighbor sketch patches {} are obtained. Then, their weight vector is estimated using Markov Network.
The joint probability of the test photo patches and weight
Sketch-photo databases
The experiments are conducted on two different sketch- photo databases (CUHK Database [32], AR database [33]) respectively to verify the effectiveness of the proposed method. The CUHK Database provides sketch-photo pairs of 188 subjects from the Chinese University of Hong Kong (CUHK) student system. All photos are taken under normal lighting condition, in frontal pose and with neutral expression. And the corresponding sketches are drawn by the artist while viewing photos. In our experiments, 88
Conclusions
In this paper, dual-scale Markov Network and Multi-information based face sketch-photo synthesis and recognition is proposed. Our method has two components: (1) transforming the sketch and photo into the same modality using dual scale Markov Network; (2) matching the synthesized image and test image based on structural information and feature information- based recognition method. Unlike previous face sketch-photo synthesis methods which only use single scale to synthesize the image, the
Acknowledgment
This work has been supported by National Natural Science Foundation of China (61203261), China Postdoctoral Science Foundation funded project (2012M521335), Jiangsu Key Laboratory of Big Data Analysis Technology/B-DAT (Nanjing University of Information Science & Technology, Grant No.: KXK1404), Research Fund of Guangxi Key Lab of Multi-source Information Mining & Security (MIMS16-02), Shenzhen science and technology research and development funds (JCYJ20170307093018753) and The Fundamental
Zhenxue Chen was born in Shandong, China, in 1977. He received the B.S. degree in automatic from School of Electrical Engineering and Automation at Shandong Institute of Light Industry, Jinan, China, in 2000, the M.S. degree in computer science from School of Information Science and Engineering at Wuhan University of Science and Technology, Wuhan, China, in 2003, and the Ph.D. degree in pattern recognition and intelligent systems from Institute of Image Recognition and Artificial Intelligence
References (41)
- et al.
A new approach for face recognition by sketches in photos
Signal Process.
(2009) - et al.
Image-based facial sketch-to-photo synthesis via online coupled dictionary learning
Inf. Sci.
(2012) - et al.
Face recognition based on human cognition
Comput. Eng. Des.
(2011) - X. Tang, X. Wang, Face photo recognition using sketch, in: 2002 International Conference on Image Processing, vol. 1,...
- X. Tang, X. Wang, Face sketch synthesis and recognition, in: Ninth IEEE International Conference on Computer Vision,...
- et al.
Face sketch recognition
IEEE Trans. Circuits Syst. Video Technol.
(2004) - Q. Liu, X. Tang, H. Jin, et al., A nonlinear approach for face sketch synthesis and recognition, in: 2005 IEEE Computer...
- et al.
Face photo-sketch synthesis and recognition
IEEE Trans. Pattern Anal. Mach. Intell.
(2009) - W. Zhang, X. Wang, X. Tang, Coupled information theoretic encoding for face photo-sketch recognition, in: International...
- B. Xiao, Face Sketch-photo Synthesis and Recognition, Ph.D. Thesis, Xidian University, Xi’an,...
Face sketch-photo synthesis and retrieval using sparse representation
IEEE Trans. Circuits Syst. Video Technol.
Distinctive image features from scale-invariant keypoints
Int. J. Comput. Vision
Matching forensic sketches to mug shot photos
IEEE Trans. Pattern Anal. Mach. Intell.
Cited by (5)
Unsupervised self-attention lightweight photo-to-sketch synthesis with feature maps
2023, Journal of Visual Communication and Image RepresentationCross-domain representation learning by domain-migration generative adversarial network for sketch based image retrieval
2020, Journal of Visual Communication and Image RepresentationCitation Excerpt :Sketching is an intuitive way to express the humans’ thoughts and with the developments of the touch screen devices, sketches could be got easily in different kinds of portable equipments. So the researches of using the sketches in computer vision attract many attentions, such as sketch recognition [1], sketch analysis [2], sketch generation [3] and sketch-photo synthesis and recognition [4]. Furthermore, using free-hand sketch as the query to retrieval images, namely, sketch based image retrieval (SBIR), is showing tremendous potential applications.
Indicators and Measures for Measuring the Level of Information Intelligence
2022, Iranian Journal of Information Processing and ManagementApplication of breathing method in intelligent physical training system based on face image fusion
2021, Proceedings - 5th International Conference on Intelligent Computing and Control Systems, ICICCS 2021Sketch face recognition: P-HOG multi-features fusion
2019, International Journal of Pattern Recognition and Artificial Intelligence
Zhenxue Chen was born in Shandong, China, in 1977. He received the B.S. degree in automatic from School of Electrical Engineering and Automation at Shandong Institute of Light Industry, Jinan, China, in 2000, the M.S. degree in computer science from School of Information Science and Engineering at Wuhan University of Science and Technology, Wuhan, China, in 2003, and the Ph.D. degree in pattern recognition and intelligent systems from Institute of Image Recognition and Artificial Intelligence at Huazhong University of Science and Technology, Wuhan, China, in 2007. Since 2007, he has been an associate professor with the School of Control Science and Engineering, Shandong University. Now, his main areas of interest include image processing, pattern recognition, and computer vision, with applications to face recognition. He has published more than 80 papers in international journals and conferences.
Saisai Yao was born in Shandong, China, in 1993. She received the B.S. degree in Automatic from Shandong University, Jinan, China. She is currently a Ph.D. candidate in Pattern Recognition and Intelligence Systems at the School of Control Science and Engineering, Shandong University, Jinan, China. Her current research interests include sketch face recognition, computer vision and machine learning.
Yunyi Jia received his Ph.D. in Electrical Engineering from Michigan State University in 2014, M.S. in control Theory and Control Engineering from South China University of Technology in 2008, and B.S. in Automation from National University of Defense Technology in 2005. He is currently an assistant professor in the Department of Automotive Engineering at Clemson University. His research interests mainly include robotics, human-robot interaction, intelligent manufacturing, autonomous driving and advanced sensing systems. He is a member IEEE, ASME and SAE.
Chengyun Liu was born in Henan, China, in 1975. She received the B.S. degree in communication from Huazhong Normal Unversity, Wuhan, China, in 1999, the M.S. degree in pattern recognition and intelligent systems from Wuhan University of Science and Technology, Wuhan, China, in 2005, and the Ph.D. degree in pattern recognition and intelligent systems from Shandong University, Jinan, China, in 2016. Now, she is an associate professor with the School of Control Science and Engineering, Shandong University. Her research interests include automatic target detection and recognition, image processing, and computer vision.
- ☆
This paper has been recommended for acceptance by Zicheng Liu.