recent research on face recognition

Suggestions or feedback?

MIT News | Massachusetts Institute of Technology

• Machine learning
• Sustainability
• Black holes
• Classes and programs

Departments

• Aeronautics and Astronautics
• Brain and Cognitive Sciences
• Architecture
• Political Science
• Mechanical Engineering

Centers, Labs, & Programs

• Abdul Latif Jameel Poverty Action Lab (J-PAL)
• Picower Institute for Learning and Memory
• Lincoln Laboratory
• School of Architecture + Planning
• School of Engineering
• School of Humanities, Arts, and Social Sciences
• Sloan School of Management
• School of Science
• MIT Schwarzman College of Computing

An optimized solution for face recognition

Press contact :.

Previous image Next image

The human brain seems to care a lot about faces. It’s dedicated a specific area to identifying them, and the neurons there are so good at their job that most of us can readily recognize thousands of individuals. With artificial intelligence, computers can now recognize faces with a similar efficiency — and neuroscientists at MIT’s McGovern Institute for Brain Research have found that a computational network trained to identify faces and other objects discovers a surprisingly brain-like strategy to sort them all out.

The finding, reported March 16 in Science Advances , suggests that the millions of years of evolution that have shaped circuits in the human brain have optimized our system for facial recognition.

“The human brain’s solution is to segregate the processing of faces from the processing of objects,” explains Katharina Dobs, who led the study as a postdoc in the lab of McGovern investigator Nancy Kanwisher , the Walter A. Rosenblith Professor of Cognitive Neuroscience at MIT. The artificial network that she trained did the same. “And that’s the same solution that we hypothesize any system that’s trained to recognize faces and to categorize objects would find,” she adds.

“These two completely different systems have figured out what a — if not the — good solution is. And that feels very profound,” says Kanwisher.

Functionally specific brain regions

More than 20 years ago, Kanwisher and her colleagues discovered a small spot in the brain’s temporal lobe that responds specifically to faces. This region, which they named the fusiform face area, is one of many brain regions Kanwisher and others have found that are dedicated to specific tasks, such as the detection of written words, the perception of vocal songs, and understanding language.

Kanwisher says that as she has explored how the human brain is organized, she has always been curious about the reasons for that organization. Does the brain really need special machinery for facial recognition and other functions? “‘Why questions’ are very difficult in science,” she says. But with a sophisticated type of machine learning called a deep neural network, her team could at least find out how a different system would handle a similar task.

Dobs, who is now a research group leader at Justus Liebig University Giessen in Germany, assembled hundreds of thousands of images with which to train a deep neural network in face and object recognition. The collection included the faces of more than 1,700 different people and hundreds of different kinds of objects, from chairs to cheeseburgers. All of these were presented to the network, with no clues about which was which. “We never told the system that some of those are faces, and some of those are objects. So it’s basically just one big task,” Dobs says. “It needs to recognize a face identity, as well as a bike or a pen.”

As the program learned to identify the objects and faces, it organized itself into an information-processing network with that included units specifically dedicated to face recognition. Like the brain, this specialization occurred during the later stages of image processing. In both the brain and the artificial network, early steps in facial recognition involve more general vision processing machinery, and final stages rely on face-dedicated components.

It’s not known how face-processing machinery arises in a developing brain, but based on their findings, Kanwisher and Dobs say networks don’t necessarily require an innate face-processing mechanism to acquire that specialization. “We didn’t build anything face-ish into our network,” Kanwisher says. “The networks managed to segregate themselves without being given a face-specific nudge.”

Kanwisher says it was thrilling seeing the deep neural network segregate itself into separate parts for face and object recognition. “That’s what we’ve been looking at in the brain for 20-some years,” she says. “Why do we have a separate system for face recognition in the brain? This tells me it is because that is what an optimized solution looks like.”

Now, she is eager to use deep neural nets to ask similar questions about why other brain functions are organized the way they are. “We have a new way to ask why the brain is organized the way it is,” she says. “How much of the structure we see in human brains will arise spontaneously by training networks to do comparable tasks?”

Share this news article on:

Related links.

• Nancy Kanwisher
• McGovern Institute for Brain Research
• Department of Brain and Cognitive Sciences

Related Topics

• Brain and cognitive sciences
• McGovern Institute
• Artificial intelligence
• Computer science and technology
• Neuroscience

Related Articles

A key brain region responds to faces similarly in infants and adults

Face-specific brain area responds to faces even in people born blind

What's in a face?

Previous item Next item

More MIT News

When the lights turned on in the universe

Read full story →

3 Questions: How to prove humanity online

Lincoln Laboratory and National Strategic Research Institute launch student research program to tackle biothreats to national security

Christine Ortiz named director of MIT Technology and Policy Program

MIT engineers design tiny batteries for powering cell-sized robots

New open-source tool helps to detangle the brain

• More news on MIT News homepage →

Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA, USA

• Map (opens in new window)
• Events (opens in new window)
• People (opens in new window)
• Careers (opens in new window)
• Accessibility
• Social Media Hub
• MIT on Facebook
• MIT on YouTube
• MIT on Instagram

• A-Z Publications

Annual Review of Vision Science

Volume 7, 2021, review article, face recognition by humans and machines: three fundamental advances from deep learning.

• Alice J. O'Toole 1 , and Carlos D. Castillo 2
• View Affiliations Hide Affiliations Affiliations: 1 School of Behavioral and Brain Sciences, The University of Texas at Dallas, Richardson, Texas 75080, USA; email: [email protected] 2 Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, Maryland 21218, USA; email: [email protected]
• Vol. 7:543-570 (Volume publication date September 2021) https://doi.org/10.1146/annurev-vision-093019-111701
• First published as a Review in Advance on August 04, 2021
• Copyright © 2021 by Annual Reviews. All rights reserved

Deep learning models currently achieve human levels of performance on real-world face recognition tasks. We review scientific progress in understanding human face processing using computational approaches based on deep learning. This review is organized around three fundamental advances. First, deep networks trained for face identification generate a representation that retains structured information about the face (e.g., identity, demographics, appearance, social traits, expression) and the input image (e.g., viewpoint, illumination). This forces us to rethink the universe of possible solutions to the problem of inverse optics in vision. Second, deep learning models indicate that high-level visual representations of faces cannot be understood in terms of interpretable features. This has implications for understanding neural tuning and population coding in the high-level visual cortex. Third, learning in deep networks is a multistep process that forces theoretical consideration of diverse categories of learning that can overlap, accumulate over time, and interact. Diverse learning types are needed to model the development of human face processing skills, cross-race effects, and familiarity with individual faces.

Article metrics loading...

Full text loading...

Literature Cited

• Abadi M , Barham P , Chen J , Chen Z , Davis A et al. 2016 . Tensorflow: a system for large-scale machine learning. Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) 265– 83 Berkeley, CA: USENIX [Google Scholar]
• Abudarham N , Shkiller L , Yovel G. 2019 . Critical features for face recognition. Cognition 182 : 73– 83 [Google Scholar]
• Abudarham N , Yovel G. 2020 . Face recognition depends on specialized mechanisms tuned to view-invariant facial features: insights from deep neural networks optimized for face or object recognition. bioRxiv 2020.01.01.890277. https://doi.org/10.1101/2020.01.01.890277 [Crossref]
• Azevedo FA , Carvalho LR , Grinberg LT , Farfel JM , Ferretti RE et al. 2009 . Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain. J. Comp. Neurol. 513 : 5 532– 41 [Google Scholar]
• Barlow HB. 1972 . Single units and sensation: a neuron doctrine for perceptual psychology?. Perception 1 : 4 371– 94 [Google Scholar]
• Bashivan P , Kar K , DiCarlo JJ. 2019 . Neural population control via deep image synthesis. Science 364 : 6439 eaav9436 [Google Scholar]
• Best-Rowden L , Jain AK. 2018 . Learning face image quality from human assessments. IEEE Trans. Inform. Forensics Secur. 13 : 12 3064– 77 [Google Scholar]
• Blanz V , Vetter T. 1999 . A morphable model for the synthesis of 3D faces. Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques 187– 94 New York: ACM [Google Scholar]
• Blauch NM , Behrmann M , Plaut DC. 2020a . Computational insights into human perceptual expertise for familiar and unfamiliar face recognition. Cognition 208 : 104341 [Google Scholar]
• Blauch NM , Behrmann M , Plaut DC. 2020b . Deep learning of shared perceptual representations for familiar and unfamiliar faces: reply to commentaries. Cognition 208 : 104484 [Google Scholar]
• Box GE. 1976 . Science and statistics. J. Am. Stat. Assoc. 71 : 356 791– 99 [Google Scholar]
• Box GEP 1979 . Robustness in the strategy of scientific model building. Robustness in Statistics RL Launer, GN Wilkinson 201– 36 Cambridge, MA: Academic Press [Google Scholar]
• Bruce V , Young A. 1986 . Understanding face recognition. Br. J. Psychol. 77 : 3 305– 27 [Google Scholar]
• Burton AM , Bruce V , Hancock PJ. 1999 . From pixels to people: a model of familiar face recognition. Cogn. Sci. 23 : 1 1– 31 [Google Scholar]
• Cavazos JG , Noyes E , O'Toole AJ. 2019 . Learning context and the other-race effect: strategies for improving face recognition. Vis. Res. 157 : 169– 83 [Google Scholar]
• Cavazos JG , Phillips PJ , Castillo CD , O'Toole AJ. 2020 . Accuracy comparison across face recognition algorithms: Where are we on measuring race bias?. IEEE Trans. Biom. Behav. Identity Sci. 3 : 1 101– 11 [Google Scholar]
• Chang L , Tsao DY. 2017 . The code for facial identity in the primate brain. Cell 169 : 6 1013– 28 [Google Scholar]
• Chen JC , Patel VM , Chellappa R. 2016 . Unconstrained face verification using deep CNN features. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) 1– 9 Piscataway, NJ: IEEE [Google Scholar]
• Cichy RM , Kaiser D. 2019 . Deep neural networks as scientific models. Trends Cogn. Sci. 23 : 4 305– 17 [Google Scholar]
• Collins E , Behrmann M. 2020 . Exemplar learning reveals the representational origins of expert category perception. PNAS 117 : 20 11167– 77 [Google Scholar]
• Colón YI , Castillo CD , O'Toole AJ. 2021 . Facial expression is retained in deep networks trained for face identification. J. Vis. 21 : 4 4 [Google Scholar]
• Cootes TF , Taylor CJ , Cooper DH , Graham J. 1995 . Active shape models—their training and application. Comput. Vis. Image Underst. 61 : 1 38– 59 [Google Scholar]
• Crosswhite N , Byrne J , Stauffer C , Parkhi O , Cao Q , Zisserman A. 2018 . Template adaptation for face verification and identification. Image Vis. Comput. 79 : 35– 48 [Google Scholar]
• Deng J , Guo J , Xue N , Zafeiriou S. 2019 . Arcface: additive angular margin loss for deep face recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 4690– 99 Piscataway, NJ: IEEE [Google Scholar]
• Dhar P , Bansal A , Castillo CD , Gleason J , Phillips P , Chellappa R. 2020 . How are attributes expressed in face DCNNs?. Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) 61– 68 Piscataway, NJ: IEEE [Google Scholar]
• DiCarlo JJ , Cox DD. 2007 . Untangling invariant object recognition. Trends Cogn. Sci. 11 : 8 333– 41 [Google Scholar]
• Dobs K , Kell AJ , Martinez J , Cohen M , Kanwisher N. 2020 . Using task-optimized neural networks to understand why brains have specialized processing for faces. J. Vis. 20 : 11 660 [Google Scholar]
• Dowsett A , Sandford A , Burton AM. 2016 . Face learning with multiple images leads to fast acquisition of familiarity for specific individuals. Q. J. Exp. Psychol. 69 : 1 1– 10 [Google Scholar]
• El Khiyari H , Wechsler H. 2016 . Face verification subject to varying (age, ethnicity, and gender) demographics using deep learning. J. Biom. Biostat. 7 : 323 [Google Scholar]
• Fausey CM , Jayaraman S , Smith LB. 2016 . From faces to hands: changing visual input in the first two years. Cognition 152 : 101– 7 [Google Scholar]
• Freiwald WA , Tsao DY. 2010 . Functional compartmentalization and viewpoint generalization within the macaque face-processing system. Science 330 : 6005 845– 51 [Google Scholar]
• Freiwald WA , Tsao DY , Livingstone MS. 2009 . A face feature space in the macaque temporal lobe. Nat. Neurosci. 12 : 9 1187– 96 [Google Scholar]
• Fukushima K. 1988 . Neocognitron: a hierarchical neural network capable of visual pattern recognition. Neural Netw 1 : 2 119– 30 [Google Scholar]
• Goodfellow I , Pouget-Abadie J , Mirza M , Xu B , Warde-Farley D et al. 2014 . Generative adversarial nets. NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems 2672– 80 New York: ACM [Google Scholar]
• Goodman CS , Shatz CJ. 1993 . Developmental mechanisms that generate precise patterns of neuronal connectivity. Cell 72 : 77– 98 [Google Scholar]
• Grill-Spector K , Kushnir T , Edelman S , Avidan G , Itzchak Y , Malach R. 1999 . Differential processing of objects under various viewing conditions in the human lateral occipital complex. Neuron 24 : 1 187– 203 [Google Scholar]
• Grill-Spector K , Weiner KS. 2014 . The functional architecture of the ventral temporal cortex and its role in categorization. Nat. Rev. Neurosci. 15 : 8 536– 48 [Google Scholar]
• Grill-Spector K , Weiner KS , Gomez J , Stigliani A , Natu VS. 2018 . The functional neuroanatomy of face perception: from brain measurements to deep neural networks. Interface Focus 8 : 4 20180013 [Google Scholar]
• Gross CG. 2002 . Genealogy of the “grandmother cell. Neuroscientist 8 : 5 512– 18 [Google Scholar]
• Grother P , Ngan M , Hanaoka K 2019 . Face recognition vendor test (FRVT) part 3: demographic effects Rep. Natl. Inst. Stand. Technol., US Dept. Commerce Gaithersburg, MD: [Google Scholar]
• Hancock PJ , Bruce V , Burton AM. 2000 . Recognition of unfamiliar faces. Trends Cogn. Sci. 4 : 9 330– 37 [Google Scholar]
• Hasson U , Nastase SA , Goldstein A. 2020 . Direct fit to nature: an evolutionary perspective on biological and artificial neural networks. Neuron 105 : 3 416– 34 [Google Scholar]
• Hayward WG , Favelle SK , Oxner M , Chu MH , Lam SM. 2017 . The other-race effect in face learning: using naturalistic images to investigate face ethnicity effects in a learning paradigm. Q. J. Exp. Psychol. 70 : 5 890– 96 [Google Scholar]
• Hesse JK , Tsao DY. 2020 . The macaque face patch system: a turtle's underbelly for the brain. Nat. Rev. Neurosci. 21 : 12 695– 716 [Google Scholar]
• Hill MQ , Parde CJ , Castillo CD , Colon YI , Ranjan R et al. 2019 . Deep convolutional neural networks in the face of caricature. Nat. Mach. Intel. 1 : 11 522– 29 [Google Scholar]
• Hong H , Yamins DL , Majaj NJ , DiCarlo JJ. 2016 . Explicit information for category-orthogonal object properties increases along the ventral stream. Nat. Neurosci. 19 : 4 613– 22 [Google Scholar]
• Hornik K , Stinchcombe M , White H. 1989 . Multilayer feedforward networks are universal approximators. Neural Netw 2 : 5 359– 66 [Google Scholar]
• Huang GB , Lee H , Learned-Miller E. 2012 . Learning hierarchical representations for face verification with convolutional deep belief networks. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2518– 25 Piscataway, NJ: IEEE [Google Scholar]
• Huang GB , Mattar M , Berg T , Learned-Miller E. 2008 . Labeled faces in the wild: a database for studying face recognition in unconstrained environments Paper presented at the Workshop on Faces in “Real-Life” Images: Detection, Alignment, and Recognition Marseille, France: [Google Scholar]
• Ilyas A , Santurkar S , Tsipras D , Engstrom L , Tran B , Madry A 2019 . Adversarial examples are not bugs, they are features. arXiv:1905.02175 [stat.ML]
• Issa EB , DiCarlo JJ. 2012 . Precedence of the eye region in neural processing of faces. J. Neurosci. 32 : 47 16666– 82 [Google Scholar]
• Jacquet M , Champod C. 2020 . Automated face recognition in forensic science: review and perspectives. Forensic Sci. Int. 307 : 110124 [Google Scholar]
• Jayaraman S , Fausey CM , Smith LB. 2015 . The faces in infant-perspective scenes change over the first year of life. PLOS ONE 10 : 5 e0123780 [Google Scholar]
• Jayaraman S , Smith LB. 2019 . Faces in early visual environments are persistent not just frequent. Vis. Res. 157 : 213– 21 [Google Scholar]
• Jenkins R , White D , Van Montfort X , Burton AM. 2011 . Variability in photos of the same face. Cognition 121 : 3 313– 23 [Google Scholar]
• Kandel ER , Schwartz JH , Jessell TM , Siegelbaum S , Hudspeth AJ , Mack S 2000 . Principles of Neural Science , Vol. 4 New York: McGraw-Hill [Google Scholar]
• Kay KN , Weiner KS , Grill-Spector K. 2015 . Attention reduces spatial uncertainty in human ventral temporal cortex. Curr. Biol. 25 : 5 595– 600 [Google Scholar]
• Kelly DJ , Quinn PC , Slater AM , Lee K , Ge L , Pascalis O. 2007 . The other-race effect develops during infancy: evidence of perceptual narrowing. Psychol. Sci. 18 : 12 1084– 89 [Google Scholar]
• Kelly DJ , Quinn PC , Slater AM , Lee K , Gibson A et al. 2005 . Three-month-olds, but not newborns, prefer own-race faces. Dev. Sci. 8 : 6 F31– 36 [Google Scholar]
• Kietzmann TC , Swisher JD , König P , Tong F. 2012 . Prevalence of selectivity for mirror-symmetric views of faces in the ventral and dorsal visual pathways. J. Neurosci. 32 : 34 11763– 72 [Google Scholar]
• Krishnapriya KS , Albiero V , Vangara K , King MC , Bowyer KW. 2020 . Issues related to face recognition accuracy varying based on race and skin tone. IEEE Trans. Technol. Soc. 1 : 1 8– 20 [Google Scholar]
• Krishnapriya K , Vangara K , King MC , Albiero V , Bowyer K. 2019 . Characterizing the variability in face recognition accuracy relative to race. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , Vol. 1 2278– 85 Piscataway, NJ: IEEE [Google Scholar]
• Krizhevsky A , Sutskever I , Hinton GE. 2012 . Imagenet classification with deep convolutional neural networks. NIPS'12: Proceedings of the 25th International Conference on Neural Information Processing Systems 1097– 105 New York: ACM [Google Scholar]
• Kumar N , Berg AC , Belhumeur PN , Nayar SK. 2009 . Attribute and simile classifiers for face verification. Proceedings of the 2009 IEEE International Conference on Computer Vision 365– 72 Piscataway, NJ: IEEE [Google Scholar]
• Laurence S , Zhou X , Mondloch CJ. 2016 . The flip side of the other-race coin: They all look different to me. Br. J. Psychol. 107 : 2 374– 88 [Google Scholar]
• LeCun Y , Bengio Y , Hinton G. 2015 . Deep learning. Nature 521 : 7553 436– 44 [Google Scholar]
• Levin DT. 2000 . Race as a visual feature: using visual search and perceptual discrimination tasks to understand face categories and the cross-race recognition deficit. J. Exp. Psychol. Gen. 129 : 4 559– 74 [Google Scholar]
• Lewenberg Y , Bachrach Y , Shankar S , Criminisi A. 2016 . Predicting personal traits from facial images using convolutional neural networks augmented with facial landmark information. arXiv:1605.09062 [cs.CV]
• Li Y , Gao F , Ou Z , Sun J. 2018 . Angular softmax loss for end-to-end speaker verification. Proceedings of the 11th International Symposium on Chinese Spoken Language Processing (ISCSLP) 190– 94 Baixas, France: ISCA [Google Scholar]
• Liu Z , Luo P , Wang X , Tang X 2015 . Deep learning face attributes in the wild. Proceedings of the 2015 IEEE International Conference on Computer Vision 3730– 38 Piscataway, NJ: IEEE [Google Scholar]
• Lundqvist D , Flykt A , Ohman A. 1998 . Karolinska directed emotional faces Database of standardized facial images Psychol. Sect., Dept. Clin. Neurosci. Karolinska Hosp. Solna, Swed: https://www.kdef.se/#:∼:text=The%20Karolinska%20Directed%20Emotional%20Faces,from%20the%20original%20KDEF%20images [Google Scholar]
• Malpass RS , Kravitz J. 1969 . Recognition for faces of own and other race. J. Personal. Soc. Psychol. 13 : 4 330– 34 [Google Scholar]
• Matthews CM , Mondloch CJ. 2018 . Improving identity matching of newly encountered faces: effects of multi-image training. J. Appl. Res. Mem. Cogn. 7 : 2 280– 90 [Google Scholar]
• Maurer D , Le Grand R , Mondloch CJ 2002 . The many faces of configural processing. Trends Cogn. Sci. 6 : 6 255– 60 [Google Scholar]
• Maze B , Adams J , Duncan JA , Kalka N , Miller T et al. 2018 . IARPA Janus Benchmark—C: face dataset and protocol. Proceedings of the 2018 International Conference on Biometrics (ICB) 158– 65 Piscataway, NJ: IEEE [Google Scholar]
• McCurrie M , Beletti F , Parzianello L , Westendorp A , Anthony S , Scheirer WJ. 2017 . Predicting first impressions with deep learning. Proceedings of the 2017 IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) 518– 25 Piscataway, NJ: IEEE [Google Scholar]
• Murphy J , Ipser A , Gaigg SB , Cook R. 2015 . Exemplar variance supports robust learning of facial identity. J. Exp. Psychol. Hum. Percept. Perform. 41 : 3 577– 81 [Google Scholar]
• Natu VS , Barnett MA , Hartley J , Gomez J , Stigliani A , Grill-Spector K. 2016 . Development of neural sensitivity to face identity correlates with perceptual discriminability. J. Neurosci. 36 : 42 10893– 907 [Google Scholar]
• Natu VS , Jiang F , Narvekar A , Keshvari S , Blanz V , O'Toole AJ. 2010 . Dissociable neural patterns of facial identity across changes in viewpoint. J. Cogn. Neurosci. 22 : 7 1570– 82 [Google Scholar]
• Nordt M , Gomez J , Natu V , Jeska B , Barnett M , Grill-Spector K. 2019 . Learning to read increases the informativeness of distributed ventral temporal responses. Cereb. Cortex 29 : 7 3124– 39 [Google Scholar]
• Nordt M , Gomez J , Natu VS , Rezai AA , Finzi D , Grill-Spector K. 2020 . Selectivity to limbs in ventral temporal cortex decreases during childhood as selectivity to faces and words increases. J. Vis. 20 : 11 152 [Google Scholar]
• Noyes E , Jenkins R. 2019 . Deliberate disguise in face identification. J. Exp. Psychol. Appl. 25 : 2 280– 90 [Google Scholar]
• Noyes E , Parde C , Colon Y , Hill M , Castillo C et al. 2021 . Seeing through disguise: getting to know you with a deep convolutional neural network. Cognition 211 : 104611 [Google Scholar]
• Noyes E , Phillips P , O'Toole A 2017 . What is a super-recogniser?. Face Processing: Systems, Disorders and Cultural Differences M Bindemann 173– 201 Hauppage, NY: Nova Sci. Publ. [Google Scholar]
• Oosterhof NN , Todorov A. 2008 . The functional basis of face evaluation. PNAS 105 : 32 11087– 92 [Google Scholar]
• O'Toole AJ , Castillo CD , Parde CJ , Hill MQ , Chellappa R. 2018 . Face space representations in deep convolutional neural networks. Trends Cogn. Sci. 22 : 9 794– 809 [Google Scholar]
• O'Toole AJ , Phillips PJ , Jiang F , Ayyad J , Pénard N , Abdi H. 2007 . Face recognition algorithms surpass humans matching faces over changes in illumination. IEEE Trans. Pattern Anal. Mach. Intel. 9 1642– 46 [Google Scholar]
• Parde CJ , Castillo C , Hill MQ , Colon YI , Sankaranarayanan S et al. 2017 . Face and image representation in deep CNN features. Proceedings of the 2017 IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) 673– 80 Piscataway, NJ: IEEE [Google Scholar]
• Parde CJ , Colón YI , Hill MQ , Castillo CD , Dhar P , O'Toole AJ. 2021 . Face recognition by humans and machines: closing the gap between single-unit and neural population codes—insights from deep learning in face recognition. J. Vis. In press [Google Scholar]
• Parde CJ , Hu Y , Castillo C , Sankaranarayanan S , O'Toole AJ. 2019 . Social trait information in deep convolutional neural networks trained for face identification. Cogn. Sci. 43 : 6 e12729 [Google Scholar]
• Parkhi OM , Vedaldi A , Zisserman A. 2015 . Deep face recognition . Rep., Vis. Geom. Group Dept. Eng. Sci., Univ. Oxford UK: [Google Scholar]
• Paszke A , Gross S , Massa F , Lerer A , Bradbury J et al. 2019 . Pytorch: an imperative style, high-performance deep learning library. NeurIPS 2019: Proceedings of the 32nd International Conference on Neural Information Processing Systems 8024– 35 New York: ACM [Google Scholar]
• Pezdek K , Blandon-Gitlin I , Moore C 2003 . Children's face recognition memory: more evidence for the cross-race effect. J. Appl. Psychol. 88 : 4 760– 63 [Google Scholar]
• Phillips PJ , Beveridge JR , Draper BA , Givens G , O'Toole AJ et al. 2011 . An introduction to the good, the bad, & the ugly face recognition challenge problem. Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG) 346– 53 Piscataway, NJ: IEEE [Google Scholar]
• Phillips PJ , O'Toole AJ. 2014 . Comparison of human and computer performance across face recognition experiments. Image Vis. Comput. 32 : 1 74– 85 [Google Scholar]
• Phillips PJ , Yates AN , Hu Y , Hahn CA , Noyes E et al. 2018 . Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms. PNAS 115 : 24 6171– 76 [Google Scholar]
• Poggio T , Banburski A , Liao Q. 2020 . Theoretical issues in deep networks. PNAS 117 : 48 30039– 45 [Google Scholar]
• Ponce CR , Xiao W , Schade PF , Hartmann TS , Kreiman G , Livingstone MS. 2019 . Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell 177 : 4 999– 1009 [Google Scholar]
• Ranjan R , Bansal A , Zheng J , Xu H , Gleason J et al. 2019 . A fast and accurate system for face detection, identification, and verification. IEEE Trans. Biom. Behav. Identity Sci. 1 : 2 82– 96 [Google Scholar]
• Ranjan R , Castillo CD , Chellappa R. 2017a . L2-constrained softmax loss for discriminative face verification. arXiv:1703.09507 [cs.CV]
• Ranjan R , Sankaranarayanan S , Castillo CD , Chellappa R. 2017b . An all-in-one convolutional neural network for face analysis. Proceedings of the 2017 IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) 17– 24 Piscataway, NJ: IEEE [Google Scholar]
• Richards BA , Lillicrap TP , Beaudoin P , Bengio Y , Bogacz R et al. 2019 . A deep learning framework for neuroscience. Nat. Neurosci. 22 : 11 1761– 70 [Google Scholar]
• Ritchie KL , Burton AM. 2017 . Learning faces from variability. Q. J. Exp. Psychol. 70 : 5 897– 905 [Google Scholar]
• Rosch E , Mervis CB , Gray WD , Johnson DM , Boyes-Braem P. 1976 . Basic objects in natural categories. Cogn. Psychol. 8 : 3 382– 439 [Google Scholar]
• Russakovsky O , Deng J , Su H , Krause J , Satheesh S et al. 2015 . ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 115 : 3 211– 52 [Google Scholar]
• Russell R , Duchaine B , Nakayama K. 2009 . Super-recognizers: people with extraordinary face recognition ability. Psychon. Bull. Rev. 16 : 2 252– 57 [Google Scholar]
• Sangrigoli S , Pallier C , Argenti AM , Ventureyra V , de Schonen S. 2005 . Reversibility of the other-race effect in face recognition during childhood. Psychol. Sci. 16 : 6 440– 44 [Google Scholar]
• Sankaranarayanan S , Alavi A , Castillo C , Chellappa R. 2016 . Triplet probabilistic embedding for face verification and clustering. arXiv:1604.05417 [cs.CV]
• Schrimpf M , Kubilius J , Hong H , Majaj NJ , Rajalingham R et al. 2018 . Brain-Score: Which artificial neural network for object recognition is most brain-like?. bioRxiv 407007 : https://doi.org/10.1101/407007 [Crossref] [Google Scholar]
• Schroff F , Kalenichenko D , Philbin J. 2015 . Facenet: a unified embedding for face recognition and clustering. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition 815– 23 Piscataway, NJ: IEEE [Google Scholar]
• Scott LS , Monesson A. 2010 . Experience-dependent neural specialization during infancy. Neuropsychologia 48 : 6 1857– 61 [Google Scholar]
• Sengupta S , Chen JC , Castillo C , Patel VM , Chellappa R , Jacobs DW. 2016 . Frontal to profile face verification in the wild. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) 1– 9 Piscataway, NJ: IEEE [Google Scholar]
• Sim T , Baker S , Bsat M. 2002 . The CMU pose, illumination, and expression (PIE) database. Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition 53– 58 Piscataway, NJ: IEEE [Google Scholar]
• Simonyan K , Zisserman A. 2014 . Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 [cs.CV]
• Smith LB , Jayaraman S , Clerkin E , Yu C 2018 . The developing infant creates a curriculum for statistical learning. Trends Cogn. Sci. 22 : 4 325– 36 [Google Scholar]
• Smith LB , Slone LK. 2017 . A developmental approach to machine learning?. Front. Psychol. 8 : 2124 [Google Scholar]
• Song A , Linjie L , Atalla C , Gottrell G. 2017 . Learning to see people like people: predicting social impressions of faces. Cogn. Sci. 2017 : 1096– 101 [Google Scholar]
• Storrs KR , Kietzmann TC , Walther A , Mehrer J , Kriegeskorte N. 2020 . Diverse deep neural networks all predict human IT well, after training and fitting bioRxiv 2020.05.07.082743. https://doi.org/10.1101/2020.05.07.082743 [Crossref] [Google Scholar]
• Su H , Maji S , Kalogerakis E , Learned-Miller E. 2015 . Multi-view convolutional neural networks for 3D shape recognition. Proceedings of the 2015 IEEE International Conference on Computer Vision 945– 53 Piscataway, NJ: IEEE [Google Scholar]
• Sugden NA , Moulson MC. 2017 . Hey baby, what's “up”? One- and 3-month-olds experience faces primarily upright but non-upright faces offer the best views. Q. J. Exp. Psychol. 70 : 5 959– 69 [Google Scholar]
• Taigman Y , Yang M , Ranzato M , Wolf L. 2014 . Deepface: closing the gap to human-level performance in face verification. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition 1701– 8 Piscataway, NJ: IEEE [Google Scholar]
• Tanaka JW , Pierce LJ. 2009 . The neural plasticity of other-race face recognition. Cogn. Affect. Behav. Neurosci. 9 : 1 122– 31 [Google Scholar]
• Terhörst P , Fährmann D , Damer N , Kirchbuchner F , Kuijper A. 2020 . Beyond identity: What information is stored in biometric face templates?. arXiv:2009.09918 [cs.CV]
• Thorpe S , Fize D , Marlot C. 1996 . Speed of processing in the human visual system. Nature 381 : 6582 520– 22 [Google Scholar]
• Todorov A. 2017 . Face Value: The Irresistible Influence of First Impressions Princeton, NJ: Princeton Univ. Press [Google Scholar]
• Todorov A , Mandisodza AN , Goren A , Hall CC. 2005 . Inferences of competence from faces predict election outcomes. Science 308 : 5728 1623– 26 [Google Scholar]
• Valentine T. 1991 . A unified account of the effects of distinctiveness, inversion, and race in face recognition. Q. J. Exp. Psychol. A 43 : 2 161– 204 [Google Scholar]
• van der Maaten L , Weinberger K. 2012 . Stochastic triplet embedding. Proceedings of the 2012 IEEE International Workshop on Machine Learning for Signal Processing 1– 6 Piscataway, NJ: IEEE [Google Scholar]
• Walker M , Vetter T. 2009 . Portraits made to measure: manipulating social judgments about individuals with a statistical face model. J. Vis. 9 : 11 12 [Google Scholar]
• Wang F , Liu W , Liu H , Cheng J. 2018 . Additive margin softmax for face verification. IEEE Signal Process. Lett. 25 : 926– 30 [Google Scholar]
• Wang F , Xiang X , Cheng J , Yuille AL. 2017 . Normface: L 2 hypersphere embedding for face verification. MM '17: Proceedings of the 25th ACM International Conference on Multimedia 1041– 49 New York: ACM [Google Scholar]
• Xie C , Tan M , Gong B , Wang J , Yuille AL , Le QV. 2020 . Adversarial examples improve image recognition. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition 819– 28 Piscataway, NJ: IEEE [Google Scholar]
• Yamins DL , Hong H , Cadieu CF , Solomon EA , Seibert D , DiCarlo JJ. 2014 . Performance-optimized hierarchical models predict neural responses in higher visual cortex. PNAS 111 : 23 8619– 24 [Google Scholar]
• Yi D , Lei Z , Liao S , Li SZ. 2014 . Learning face representation from scratch. arXiv:1411.7923 [cs.CV]
• Yoshida H , Smith LB. 2008 . What's in view for toddlers? Using a head camera to study visual experience. Infancy 13 : 3 229– 48 [Google Scholar]
• Young AW , Burton AM. 2020 . Insights from computational models of face recognition: a reply to Blauch, Behrmann and Plaut. Cognition 208 : 104422 [Google Scholar]
• Yovel G , Abudarham N. 2020 . From concepts to percepts in human and machine face recognition: a reply to Blauch, Behrmann & Plaut. Cognition 208 : 104424 [Google Scholar]
• Yovel G , Halsband K , Pelleg M , Farkash N , Gal B , Goshen-Gottstein Y. 2012 . Can massive but passive exposure to faces contribute to face recognition abilities?. J. Exp. Psychol. Hum. Percept. Perform. 38 : 2 285– 89 [Google Scholar]
• Yovel G , O'Toole AJ. 2016 . Recognizing people in motion. Trends Cogn. Sci. 20 : 5 383– 95 [Google Scholar]
• Yuan L , Xiao W , Kreiman G , Tay FE , Feng J , Livingstone MS. 2020 . Adversarial images for the primate brain. arXiv:2011.05623 [q-bio.NC]

Data & Media loading...

• Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, deep neural networks: a new framework for modeling biological vision and brain information processing, a revised neural framework for face processing, capabilities and limitations of peripheral vision, visual adaptation, microglia in the retina: roles in development, maturity, and disease, circuits for action and cognition: a view from the superior colliculus, neuronal mechanisms of visual attention, the functional neuroanatomy of human face perception, scene perception in the human brain, the organization and operation of inferior temporal cortex.

Advertisement

Classical and modern face recognition approaches: a complete review

• Published: 02 October 2020
• Volume 80 , pages 4825–4880, ( 2021 )

Cite this article

• Waqar Ali   ORCID: orcid.org/0000-0003-0846-7281 1 , 2 ,
• Wenhong Tian 3 ,
• Salah Ud Din 4 ,
• Desire Iradukunda 5 &
• Abdullah Aman Khan 1  

4836 Accesses

68 Citations

18 Altmetric

Explore all metrics

Human face recognition have been an active research area for the last few decades. Especially, during the last five years, it has gained significant research attention from multiple domains like computer vision, machine learning and artificial intelligence due to its remarkable progress and broad social applications. The primary goal of any face recognition system is to recognize the human identity from the static images, video data, data-streams and the knowledge of the context in which these data components are being actively used. In this review, we have highlighted major applications, challenges and trends of face recognition systems in social and scientific domains. The prime objective of this research is to sum-up recent face recognition techniques and develop a broad understanding of how these techniques behave on different datasets. Moreover, we discuss some key challenges such as variability in illumination, pose, aging, cosmetics, scale, occlusion, and background. Along with classical face recognition techniques, most recent research directions are deeply investigated, i.e., deep learning, sparse models and fuzzy set theory. Additionally, basic methodologies are briefly discussed, while contemporary research contributions are examined in broader details. Finally, this research presents future aspects of face recognition technologies and its potential significance in the upcoming digital society.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Face Recognition Research and Development

Face Recognition: A Review and Analysis

On the frontiers of pose invariant face recognition: a review

Explore related subjects.

• Artificial Intelligence

Abate AF, Nappi M, Riccio D, Sabatino G (2007) 2d and 3d face recognition: a survey. Pattern Recogn Lett 28(14):1885–1906

Google Scholar  

Abbe E, Sandon C (2018) Provable limitations of deep learning coRR. arXiv: 1812.06369

Adini Y, Moses Y, Ullman S (1997) Face recognition: The problem of compensating for changes in illumination direction. IEEE Trans Pattern Anal Mach Intell 19(7):721–732

Afifi M, Abdelhamed A (2019) Afif4: Deep gender classification based on adaboost-based fusion of isolated facial features and foggy faces. J Vis Commun Image Represent 62:77–86

Aghamaleki JA, Chenarlogh VA (2019) Multi-stream CNN for facial expression recognition in limited training data. Multimed Tools Appl 78 (16):22861–22882

Ahonen T, Hadid A, Pietikȧinen M (2004) Face recognition with local binary patterns. In: Proceedings of 8th European Conference on Computer Vision-ECCV, Prague, Czech Republic, Part I, pp 469–481

Akram MU, Awan HM, Khan AA (2014) Dorsal hand veins based person identification. In: 4Th international conference on image processing theory, tools and applications, IPTA 2014, Paris, pp 289–294

Al-Shannaq AS, Elrefaei LA (2019) Comprehensive analysis of the literature for age estimation from facial images. IEEE Access 7:93229–93249

Al-wajih E, Ahmed M (2020) A new application for gabor filters in face-based gender classification. Int Arab J Inf Technol 17(2):178–187

Ali W, Jie S, Aman KA, Saifullah T (2019) Context-aware recommender systems: Challenges and opportunities 48(5):655

Almudhahka NY, Nixon MS, Hare JS (2018) Comparative Face Soft Biometrics for Human Identification. Springer International Publishing, Cham, pp 25–50

An L, Chen X, Liu S, Lei Y, Yang S (2017) Integrating appearance features and soft biometrics for person re-identification. Multimed Tools Appl 76(9):12117–12131

Andreopoulos A, Tsotsos JK (2013) 50 years of object recognition: Directions forward. Comput Vis Image Underst 117(8):827–891

Annamalai P, Raju K, Ranganayakulu D (2018) Soft biometrics traits for continuous authentication in online exam using ICA based facial recognition. I J Netw Secur 20(3):423–432

Arashloo SR (2016) A comparison of deep multilayer networks and markov random field matching models for face recognition in the wild. IET Comput Vis 10(6):466–474

Arigbabu OA, Ahmad SMS, Adnan WAW, Yussof S (2015) Recent advances in facial soft biometrics. Vis Comput 31(5):513–525

Azeem A, Sharif M, Raza M, Murtaza M (2014) A survey: face recognition techniques under partial occlusion. Int Arab J Inf Technol 11(1):1–10

Bailly-Bailliére E, Bengio S, Bimbot F, Hamouz M, Kittler J, Mariéthoz J, Matas J, Messer K, Popovici V, Porée F et al (2003) The banca database and evaluation protocol. In: International conference on audio-and video-based biometric person authentication. Springer, pp 625–638

Bair S, DelVecchio M, Flowers B, Michaels AJ, Headley WC (2019) On the limitations of targeted adversarial evasion attacks against deep learning enabled modulation recognition. In: Proceedings of the ACM Workshop on Wireless Security and Machine Learning, WiseML@WiSec 2019, Miami, pp 25–30

Baocai Y, Yanfeng S, Chengzhang W, Yun G (2009) Bjut-3d large scale 3d face database and information processing. J Comput Res Dev 6:020

Bartlett MS, Movellan JR, Sejnowski TJ (2002) Face recognition by independent component analysis. IEEE Trans Neural Netw 13(6):1450–1464

Bashbaghi S, Granger E, Sabourin R, Parchami M (2018) Deep learning architectures for face recognition in video surveillance coRR. arXiv: 1802.09990

Becerra-Riera F, Morales-Gonzȧlez A, Mėndez-Vȧzquez H (2019) A survey on facial soft biometrics for video surveillance and forensic applications. Artif Intell Rev 52(2):1155–1187

Belhumeur P, Hespanha JP, Kriegman D (1997) Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

Benamara NK, Zigh E, Stambouli TB, Keche M (2018) Combined and weighted features for robust multispectral face recognition. In: Proceedings of 6th IFIP TC 5 International Conference on Computational Intelligence and Its Applications - CIIA, Oran, Algeria, pp 549–560

Beveridge JR, Phillips PJ, Bolme DS, Draper BA, Givens GH, Lui YM, Teli MN, Zhang H, Scruggs WT, Bowyer KW et al (2013) The challenge of face recognition from digital point-and-shoot cameras. In: IEEE Sixth international conference on biometrics: theory, Applications and Systems (BTAS), pp 1–8

Beveridge JR, Zhang H, Flynn PJ, Lee Y, Liong VE, Lu J, de Assis Angeloni M, de Freitas Pereira T, Li H, Hua G et al (2014) The ijcb 2014 pasc video face and person recognition competition. In: IEEE International joint conference on biometrics (IJCB), pp 1–8

Brunelli R, Poggio T (1993) Face recognition: Features versus templates. IEEE Trans. Pattern Anal Mach Intell 15(10):1042–1052

Calo SB, Ko BJ, Lee K, Salonidis T, Verma DC (2018) Controlling privacy in a face recognition application. In: US Patent app 15/876,307. Google patents

Campomanes-Alvarez C, Ȧlvarez BRC, Guadarrama S, Ibȧṅez Ȯ, Cordȯn O (2017) An experimental study on fuzzy distances for skull-face overlay in craniofacial superimposition. Fuzzy Sets Syst 318:100–119

MathSciNet   Google Scholar  

Chatzis V, Bors AG, Pitas I (1999) Multimodal decision-level fusion for person authentication. IEEE Trans Syst Man Cybern Part A 29(6):674–680

Chen Y, Patel VM, Phillips PJ, Chellappa R (2012) Dictionary-based face recognition from video. In: Proceedings of 12th European Conference on Computer Vision ECCV, Florence, Part VI, pp 766–779

Chen BC, Chen C, Hsu WH (2014) Cross-age reference coding for age-invariant face recognition and retrieval. In: European conference on computer vision. Springer, pp 768–783

Chen J, Liu S, Chen Z (2017) Gender classification in live videos. In: 2017 IEEE International conference on image processing, ICIP 2017, Beijing, pp 1602–1606

Chen J, Ranjan R, Sankaranarayanan S, Kumar A, Chen C, Patel VM, Castillo CD, Chellappa R (2018) Unconstrained still/video-based face verification with deep convolutional neural networks. Int J Comput Vis 126(2-4):272–291

Chen G (2019) An experimental study for the effects of noise on face recognition algorithms under varying illumination. Multim Tools Appl 78(18):26615–26631

Cheng H (2015) Sparse representation, modeling and learning in visual recognition - theory, algorithms and applications. Advances in computer vision and pattern recognition. Springer, Berlin

Chihaoui M, Elkefi A, Bellil W, Amar CB (2016) A survey of 2d face recognition techniques. Computers 5(4):21

Choi S, Choi C, Jeong G, Kwak N (2012) Pixel selection based on discriminant features with application to face recognition. Pattern Recognit Lett 33(9):1083–1092

Chugh T, Singh M, Nagpal S, Singh R, Vatsa M (2017) Transfer learning based evolutionary algorithm for composite face sketch recognition. In: IEEE Conference on computer vision and pattern recognition workshops, CVPR workshops, Honolulu, pp 619–627

Conde C, Serrano Ȧ, Cabello E (2006) Multimodal 2d, 2.5d & 3d face verification. In: Proceedings of the International Conference on Image Processing, ICIP 2006, Atlanta, pp 2061–2064

Cox IJ, Ghosn J, Yianilos PN (1996) Feature-based face recognition using mixture-distance. In: Conference on computer vision and pattern recognition (CVPR ), San Francisco, pp 209–216

Cunningham JP, Ghahramani Z (2015) Linear dimensionality reduction: survey, insights, and generalizations. J Mach Learn Res 16:2859–2900

Dagnes N, Vezzetti E, Marcolin F, Tornincasa S (2018) Occlusion detection and restoration techniques for 3d face recognition: a literature review. Mach Vis Appl 29(5):789–813

Danelakis A, Theoharis T, Pratikakis I (2015) A survey on facial expression recognition in 3d video sequences. Multimed Tools Appl 74(15):5577–5615

Dantcheva A, Velardo C, D’Angelo A, Dugelay J (2011) Bag of soft biometrics for person identification - new trends and challenges. Multimed Tools Appl 51(2):739–777

Dantcheva A, Elia P, Ross A (2016) What else does your biometric data reveal? A survey on soft biometrics. IEEE Trans Inf Forensic Secur 11 (3):441–467

De Carrera PF, Marques I (2010) Face recognition algorithms. Master’s thesis in Computer Science. Universidad Euskal Herriko

de Souza GB, Santos DFS, Pires RG, Marana AN, Papa JP (2017) Efficient transfer learning for robust face spoofing detection. In: Proceedings of 22nd Iberoamerican Congress on Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, CIARP, Valparaíso, pp 643–651

Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: Additive angular margin loss for deep face recognition. In: IEEE Conference on computer vision and pattern recognition, CVPR 2019, Long beach, pp 4690–4699

Diarra M, Gouton P, Jerome AK (2016) A comparative study of descriptors and detectors in multispectral face recognition. In: 12Th international conference on signal-image technology & internet-based systems, SITIS, Naples, pp 209–214

Ding C, Tao D (2016) A comprehensive survey on pose-invariant face recognition. ACM Trans Intell Syst Technol 7:37:1–37:42

Du Y, Lu X, Zeng W, Hu C (2018) A novel fuzzy linear discriminant analysis for face recognition. Intell Data Anal 22(3):675–696

Edwards GJ, Cootes TF, Taylor CJ (1998) Face recognition using active appearance models. In: Proceedings of 5th European Conference on Computer Vision Computer Vision - ECCV, Freiburg, vol II, pp 581–595

Efremova N, Patkin M, Sokolov D (2019) Face and emotion recognition with neural networks on mobile devices: Practical implementation on different platforms. In: 14Th IEEE international conference on automatic face & gesture recognition, FG 2019, Lille, pp 1–5

Elad M (2012) Sparse and redundant representation modeling - what next?. IEEE Signal Process Lett 19(12):922–928

Elag M, Kumar P, Marini L, Myers JD, Hedstrom M, Plale BA (2017) Identification and characterization of information-networks in long-tail data collections. Environ Modell Softw 94:100–111

Etemad K, Chellappa R (1997) Discriminant analysis for recognition of human face images. In: International conference on audio- and video-based biometric person authentication, pp 125–142

Feltwell T, Wood G, Linehan C, Lawson S (2017) An augmented reality game using face recognition technology. In: Proceedings of the ACM Conference Companion Publication on Designing Interactive Systems, pp 44–49

Fianyi I, Zia T (2016) Biometric technology solutions to countering today’s terrorism. Int J Cyber Warf Terror 6(4):28–40

Fu Y, Guo G, Huang TS (2010) Age synthesis and estimation via faces: a survey. IEEE Trans Pattern Anal Mach Intell 32(11):1955–1976

Galea C, Farrugia RA (2018) Matching software-generated sketches to face photographs with a very deep cnn, morphed faces, and transfer learning. IEEE Trans Inf Forensic Secur 13(6):1421–1431

Galiani S, Gȧlvez RH (2019) An empirical approach based on quantile regression for estimating citation ageing. J. Informetrics 13(2):738–750

Gao W, Cao B, Shan S, Chen X, Zhou D, Zhang X, Zhao D (2008) The CAS-PEAL large-scale chinese face database and baseline evaluations. IEEE Trans Syst Man Cybern Part A, 38(1):149–161

Garain J, Kumar RK, Kisku DR, Sanyal G (2019) Addressing facial dynamics using k-medoids cohort selection algorithm for face recognition. Multimed Tools Appl 78(13):18443–18474

Gautam G, Mukhopadhyay S (2019) An adaptive localization of pupil degraded by eyelash occlusion and poor contrast. Multimed Tools Appl 78(6):6655–6677

Geetha A, Sundaram M, Vijayakumari B (2019) Gender classification from face images by mixing the classifier outcome of prime, distinct descriptors. Soft Comput 23(8):2525–2535

Geng X, Zhou Z, Smith-miles K (2007) Automatic age estimation based on facial aging patterns. IEEE Trans Pattern Anal Mach Intell 29(12):2234–2240

Geng X, Yin C, Zhou Z (2013) Facial age estimation by learning from label distributions. IEEE Trans Pattern Anal Mach Intell 35(10):2401–2412

Georghiades A, Belhumeur P, Kriegman D (2001) From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell 23(6): 643–660

Gilbert A, Illingworth J, Bowden R (2008) Scale invariant action recognition using compound features mined from dense spatio-temporal corners. In: Proceedings of 10th European Conference on Computer Vision, Marseille, Part I, pp 222–233

Global opportunity analysis and industry forecast, 2015-2022. Available online: https://www.alliedmarketresearch.com/press-release/facial-recognition-market.html . Accessed: 2019-07-23

Gong D, Li Z, Huang W, Li X, Tao D (2017) Heterogeneous face recognition: A common encoding feature discriminant approach. IEEE Trans Image Process 26(5):2079–2089

Gonzalez-Sosa E, Fiėrrez J, Vera-Rodríguez R, Alonso-Fernandez F (2018) Facial soft biometrics for recognition in the wild: Recent works, annotation, and COTS evaluation. IEEE Trans Inf Forensic Secur 13(8):2001–2014

Goswami G, Ratha NK, Agarwal A, Singh R, Vatsa M (2018) Unravelling robustness of deep learning based face recognition against adversarial attacks. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans

Grgic M, Delac K, Grgic S (2011) Scface — surveillance cameras face database. Multimed Tools Appl 51(3):863–879

Grm K, Struc V, Artiges A, Caron M, Ekenel HK (2018) Strengths and weaknesses of deep learning models for face recognition against image degradations. IET Biometr 7(1):81–89

Gu̇nther M, Shafey LE, Marcel S (2016) Face recognition in challenging environments: an experimental and reproducible research survey. In: Face recognition across the imaging spectrum, pp 247–280

Guo G, Li S, Chan KL (2000) Face recognition by support vector machines. In: 4Th IEEE international conference on automatic face and gesture recognition (FG 2000), Grenoble, pp 196–201

Guo G, Fu Y, Dyer CR, Huang TS (2008) Image-based human age estimation by manifold learning and locally adjusted robust regression. IEEE Trans Image Process 17(7):1178–1188

Guo Y, Zhang L, Hu Y, He X, Gao J (2016) Ms-celeb-1m: a dataset and benchmark for large-scale face recognition. In: European conference on computer vision. Springer, pp 87–102

Guo S, Tan G, Pan H, Chen L, Gao C (2017) Face alignment under occlusion based on local and global feature regression. Multimed Tools Appl 76(6):8677–8694

Guo BH, Nixon MS, Carter JN (2018) Fusion analysis of soft biometrics for recognition at a distance. In: IEEE 4Th international conference on identity, security, and behavior analysis, ISBA 2018, Singapore, pp 1–8

Gutta S, Wechsler H (1997) Face recognition using hybrid classifiers. Pattern Recogn 30(4):539–553

https://patents.google.com . Accessed: 2019-07-23

https://techcrunch.com/2017/12/13/china-cctv-bbc-reporter/?guccounter=1 . Accessed: 2019-07-23

Haghiri S, Rabiee HR, Soltani-farani A, Hosseini SA, Shadloo M (2014) Locality preserving discriminative dictionary learning. In: IEEE International conference on image processing, ICIP, Paris, pp 5242–5246

Han X, Yang H, Xing G, Liu Y (2020) Asymmetric joint gans for normalizing face illumination from a single image. IEEE Trans Multimed 22(6):1619–1633

HasanPour SH, Rouhani M, Fayyaz M, Sabokrou M, Adeli E (2018) Towards principled design of deep convolutional networks: Introducing simpnet. CoRR arXiv: 1802.06205

Hashemi VH, Gharahbagh AA (2015) Article:a novel hybrid method for face recognition based on 2d wavelet and singular value decomposition. Amer J Netw Commun 4(4):90–94

He X, Yan S, Hu Y, Niyogi P, Zhang H (2005) Face recognition using laplacianfaces. IEEE Trans Pattern Anal Mach Intell 27(3):328–340

He L, Li H, Zhang Q, Sun Z (2019) Dynamic feature matching for partial face recognition. IEEE Trans Image Process 28(2):791–802

He Q, He B, Zhang Y, Fang H (2019) Multimedia based fast face recognition algorithm of speed up robust features. Multimed Tools Appl 78(17):24035–24045

Heisele B, Ho P, Wu J, Poggio T (2003) Face recognition: component-based versus global approaches. Comput Vis Image Underst 91(1-2):6–21

Heng W, Jiang T, Gao W (2019) How to assess the quality of compressed surveillance videos using face recognition. . IEEE Trans Circ Syst Video Techn 29(8):2229–2243

Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

Hu P, Ning H, Qiu T, Zhang Y, Luo X (2017) Fog computing based face identification and resolution scheme in internet of things. IEEE Trans Ind Inf 13(4):1910–1920

Hu C, Lu X, Liu P, Jing X, Yue D (2019) Single sample face recognition under varying illumination via QRCP decomposition. IEEE Trans Image Process 28(5):2624–2638

Hu C, Wu F, Yu J, Jing X, Lu X, Liu P (2020) Diagonal symmetric pattern-based illumination invariant measure for severe illumination variation face recognition. IEEE Access 8:63202–63213

Huan E, Wen G (2020) Transfer learning with deep convolutional neural network for constitution classification with face image. Multim Tools Appl 79 (17-18):11905–11919

Huang R, Metaxas DN, Pavlovic V (2004) A hybrid face recognition method using markov random fields. In: 17Th international conference on pattern recognition, ICPR, Cambridge, pp 157–160

Huang GB, Mattar M, Berg T, Learned-Miller E (2008) Labeled faces in the wild: a database forstudying face recognition in unconstrained environments. In: Workshop on faces in ‘real-life’ images: detection, alignment, and recognition

Huang K, Dai D, Ren C, Lai Z (2017) Learning kernel extended dictionary for face recognition. IEEE Trans Neural Netw Learn Syst 28(5):1082–1094

Huang P, Gao G, Qian C, Yang G, Yang Z (2017) Fuzzy linear regression discriminant projection for face recognition. IEEE Access 5:4340–4349

Iliadis M, Wang H, Molina R, Katsaggelos AK (2017) Robust and low-rank representation for fast face identification with occlusions. IEEE Trans Image Process 26(5):2203–2218

Iranmanesh SM, Dabouei A, Kazemi H, Nasrabadi NM (2018) Deep cross polarimetric thermal-to-visible face recognition. In: International conference on biometrics, ICB, Gold Coast, pp 166–173

Jaha ES, Nixon MS (2016) From clothing to identity: Manual and automatic soft biometrics. IEEE Trans Inf Forensic Secur 11(10):2377–2390

Jain AK, Park U (2009) Facial marks: Soft biometric for face recognition. In: Proceedings of the International Conference on Image Processing, ICIP, Cairo, pp 37–40

Jha D, Ward L, Yang Z, Wolverton C, Foster I, Liao W.k, Choudhary A, Agrawal A (2019) Irnet: A general purpose deep residual regression framework for materials discovery. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, pp 2385–2393

Jiang B, Zhang J, Deng B, Guo Y, Liu L (2017) Deep face feature for face alignment and reconstruction. CoRR arXiv: 1708.02721

Jing L, Ng MK, Zeng T (2013) Dictionary learning-based subspace structure identification in spectral clustering. IEEE Trans Neural Netw Learn Syst 24(8):1188–1199

Jing X, Wu F, Zhu X, Dong X, Ma F, Li Z (2016) Multi-spectral low-rank structured dictionary learning for face recognition. Pattern Recogn 59:14–25

Jiu M, Sahbi H (2019) Deep representation design from deep kernel networks. Pattern Recogn 88:447–457

Kanade T (1973) Picture processing system by computer complex and recognition of human faces. Ph.D. thesis, Kyoto University, Japan

Kang W, Lu Y, Li D, Jia W (2019) From noise to feature: Exploiting intensity distribution as a novel soft biometric trait for finger vein recognition. IEEE Trans Inf Forensic Secur 14(4):858–869

Kasapakis V, Gavalas D (2017) Occlusion handling in outdoors augmented reality games. Multimed Tools Appl 76(7):9829–9854

Kasturi R, Goldgof DB, Soundararajan P, Manohar V, Garofolo JS, Bowers R, Boonstra M, Korzhova VN, Zhang J (2009) Framework for performance evaluation of face, text, and vehicle detection and tracking in video: data, metrics, and protocol. IEEE Trans Pattern Anal Mach Intell 31(2): 319–336

Kawulok M, Wu J, Hancock ER (2011) Supervised relevance maps for increasing the distinctiveness of facial images. Pattern Recognit 44(4):929–939

Keinert F, Lazzaro D, Morigi S (2019) A robust group-sparse representation variational method with applications to face recognition. IEEE Trans Image Process 28(6):2785–2798

Kemelmacher-Shlizerman I, Basri B (2011) 3d face reconstruction from a single image using a single reference face shape. IEEE Trans Pattern Anal Mach Intell 33(2):394–405

Kepenekci B (2001) Face recognition using gabor wavelet transform. PhD thesis, The Middle East Technical University

Khan K, Attique M, Syed I, Sarwar G, Irfan MA, Khan R (2019) A unified framework for head pose, age and gender classification through end-to-end face segmentation. Entropy 21(7):647

Khan AA, Shao J, Ali W, Tumrani S (2020) Content-Aware summarization of broadcast sports Videos:An Audio–Visual feature extraction approach. Neural Process Lett:1–24

Khan S, Chen L, Yan H (2020) Co-clustering to reveal salient facial features for expression recognition. IEEE Trans Affect Comput 11(2):348–360

Kim K (2005) Intelligent immigration control system by using passport recognition and face verification. In: International symposium on neural networks. Springer, pp 147–156

Kim D, Hernandez M, Choi J, Medioni G (2017) Deep 3d face identification. In: IEEE International joint conference on biometrics (IJCB), pp 133–142

Kreutz-Delgado K, Murray JF, Rao BD, Engan K, Lee T, Sejnowski TJ (2003) Dictionary learning algorithms for sparse representation. Neural Comput 15(2):349–396

Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

Kumar BKS, Swamy MNS, Ahmad MO (2019) Visual tracking using structural local DCT sparse appearance model with occlusion detection. Multimed Tools Appl 78(6):7243–7266

Kwak KC, Pedrycz W (2005) Face recognition using a fuzzy fisherface classifier. Pattern Recogn 38(10):1717–1732

Kwon B, Lee K (2018) An introduction to face-recognition methods and its implementation in software applications. Int J Inf Technol Manag 17 (1/2):33–43

Lades M, Vorbru̇ggen JC, Buhmann JM, Lange J, von der Malsburg C, Wu̇rtz RP, Konen W (1993) Distortion invariant object recognition in the dynamic link architecture. IEEE Trans Comput 42(3):300–311

Lahasan BM, Lutfi SL, Segundo RS (2019) A survey on techniques to handle face recognition challenges: occlusion, single sample per subject and expression. Artif Intell Rev 52(2):949–979

Lanitis A, Taylor CJ, Cootes TF (2002) Toward automatic simulation of aging effects on face images. IEEE Trans Pattern Anal Mach Intell 24 (4):442–455

Lanitis A, Draganova C, Christodoulou C (2004) Comparing different classifiers for automatic age estimation. IEEE Trans Syst Man Cybern Part B 34(1):621–628

Lavanya B, Inbarani HH (2018) A novel hybrid approach based on principal component analysis and tolerance rough similarity for face identification. Neural Comput Appl 29(8):289–299

Lee W, Kim J (2018) Social relationship development between human and robot through real-time face identification and emotional interaction. In: ACM/IEEE International conference on human-robot interaction, HRI, Chicago, pp 379

Li D, Prasad M, Hsu S, Hong C, Lin C (2012) Face recognition using nonparametric-weighted fisherfaces. EURASIP J Adv Signal Process 2012:92

Li X, Dai DQ, Zhang X, Ren CX (2013) Structured sparse error coding for face recognition with occlusion. IEEE Trans Image Process 22 (5):1889–1900

Li H, Hua G, Shen X, Lin Z, Brandt J (2014) Eigen-pep for video face recognition. In: Asian conference on computer vision. Springer, pp 17–33

Li Y, Wang G, Lin L, Chang H (2015) A deep joint learning approach for age invariant face verification. In: CCF Chinese conference on computer vision. Springer, pp 296–305

Li X (2014) Face recognition method based on fuzzy 2dpca. J Electr Comput Eng 2014:919041:1–919041:7

Li Q, Li T, Xia B, Ni M, Liu X, Zhou Q, Qi Y (2016) FIRST: face identity recognition in smart bank. Int J Seman Comput 10(4):569

Li Z, Lai Z, Xu Y, Yang J, Zhang D (2017) A locality-constrained and label embedding dictionary learning algorithm for image classification. IEEE Trans Neural Netw Learn Syst 28(2):278–293

Li X, Makihara Y, Xu C, Yagi Y, Ren M (2018) Gait-based human age estimation using age group-dependent manifold learning and regression. Multim Tools Appl 77(21):28333–28354

Li X, Song A (2013) Fuzzy MSD based feature extraction method for face recognition. Neurocomputing 122:266–271

Li Y, Wang G, Nie L, Wang Q, Tan W (2018) Distance metric optimization driven convolutional neural network for age invariant face recognition. Pattern Recogn 75:51–62

Liao H (2019) Facial age feature extraction based on deep sparse representation. Multimed Tools Appl 78(2):2181–2197

Liao M, Gu X (2019) Face recognition based on dictionary learning and subspace learning. Digital Signal Process 90:110–124

Lin S, Zhao Z, Su F (2017) Clustering lightened deep representation for large scale face identification. In: Proceedings of the Second International Conference on Internet of things and Cloud Computing, ICC, Cambridge, pp 101:1–101:5

Lin S, Liu F, Liu Y, Shen L (2019) Local feature tensor based deep learning for 3d face recognition. In: 2019 14Th IEEE international conference on automatic face gesture recognition (FG 2019), pp 1–5

Liu C, Wechsler H (2000) Evolutionary pursuit and its application to face recognition. IEEE Trans Pattern Anal Mach Intell 22(6):570–582

Liu C, Wechsler H (2002) Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. IEEE Trans Image Process 11(4):467–476

Liu K, Yan S, Kuo CJ (2015) Age estimation via grouping and decision fusion. IEEE Trans Inf Forensic Secur 10(11):2408–2423

Liu B, Gui L, Wang Y, Wang Y, Shen B, Li X, Wang Y (2017) Class specific centralized dictionary learning for face recognition. Multimed Tools Appl 76(3):4159–4177

Liu W, Wen Y, Yu Z, Li M, Raj B, Song L (2017) Sphereface: Deep hypersphere embedding for face recognition. In: 2017 IEEE Conference on computer vision and pattern recognition, CVPR 2017, Honolulu, pp 6738–6746

Liu X, Lu L, Shen Z, Lu K (2018) A novel face recognition algorithm via weighted kernel sparse representation. Fut Gener Comput Syst 80:653–663

Liu H, Lu J, Feng J, Zhou J (2018) Label-sensitive deep metric learning for facial age estimation. IEEE Trans Inf Forensic Secur 13(2):292–305

Liu J, Liu W, Ma S, Wang M, Li L, Chen G (2019) Image-set based face recognition using k-svd dictionary learning. Int J Mach Learn Cybern 10(5):1051–1064

Liu J, Qiao R, Li Y, Li S (2019) Witness detection in multi-instance regression and its application for age estimation. Multim Tools Appl 78 (23):33703–33722

Liu N, Zhang F, Duan F (2020) Facial age estimation using a multi-task network combining classification and regression. IEEE Access 8:92441–92451

Lu C, Min H, Gui J, Zhu L, Lei Y (2013) Face recognition via weighted sparse representation. J Vis Commun Image Represent 24(2):111–116

Lu C, Tang X (2015) Surpassing human-level face verification performance on LFW with gaussianface. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, pp 3811–3819

Lu X, Wang Y, Jain AK (2003) Combining classifiers for face recognition. In: Proceedings of the IEEE International Conference on Multimedia and Expo, ICME, Baltimore, pp 13–16

Luo J, Ma Y, Takikawa E, Lao S, Kawade M, Lu B (2007) Person-specific SIFT features for face recognition. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP, Honolulu, pp 593–596

Luu K, Zhu C, Bhagavatula C, Le THN, Savvides M (2016) A Deep Learning Approach to Joint Face Detection and Segmentation, pp 1–12

Mahmood A, Uzair M, Al-mȧadeed S (2018) Multi-order statistical descriptors for real-time face recognition and object classification. IEEE Access 6:12993–13004

Malioutov DM, Ċetin M., Willsky AS (2004) Optimal sparse representations in general overcomplete bases. In: IEEE International conference on acoustics, speech, and signal processing, ICASSP, Montreal, pp 793–796

Mandavkar AA, Agawane RV (2015) Mobile based facial recognition using otp verification for voting system. In: 2015 IEEE International advance computing conference (IACC), pp 644–649

Marszalec EA, Martinkauppi JB, Soriano MN, Pietikaeinen M (2000) Physics-based face database for color research. J Electron Imaging 9 (1):32–39

Martinez AM (1998) The ar face database. CVC Technical Report24

Meng F, Tang Z, Wang Z (2017) An improved redundant dictionary based on sparse representation for face recognition. Multimed Tools Appl 76 (1):895–912

Miikkulainen R, Liang J, Meyerson E, Rawal A, Fink D, Francon O, Raju B, Shahrzad H, Navruzyan A, Duffy N et al (2019) Evolving deep neural networks. In: Artificial intelligence in the age of neural networks and brain computing Elsevier, pp 293–312.

Moghaddam B, Jebara T, Pentland A (2000) Bayesian face recognition. Pattern Recogn 33(11):1771–1782

Mohammadi A, Bhattacharjee S, Marcel S (2018) Deeply vulnerable: a study of the robustness of face recognition to presentation attacks. IET Biometr 7(1):15–26

Nakano R, Kobashi S, Alam SB, Morimoto M, Wakata Y, Ando K, Ishikura R, Hirota S, Aikawa S (2015) Neonatal brain age estimation using manifold learning regression analysis. In: 2015 IEEE International conference on systems, man, and cybernetics, Kowloon Tong, pp 2273–2276

Nguyen V, Do T, Nguyen V, Ngo TD, Duong DA (2018) How to choose deep face models for surveillance system?. In: 10Th asian conference on modern approaches for intelligent information and database systems, ACIIDS, Dong Hoi City, Extended Posters, pp 367–376

Nixon MS, Correia P, Nasrollahi K, Moeslund T, Hadid A, Tistarelli M (2015) On soft biometrics. Pattern Recogn Lett 68(2):218–230

Nojavanasghari B, Hughes CE, Baltrusaitis T, Morency LP (2017) Hand2face: Automatic synthesis and recognition of hand over face occlusions. Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp 209–215

Osuna E, Freund R, Girosit F (1997) Training support vector machines: an application to face detection. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 130–136

Oulefki A, Mustapha A, Boutellaa E, Bengherabi M, Tifarine AA (2018) Fuzzy reasoning model to improve face illumination invariance. SIViP 12 (3):421–428

Ouyang D, Zhang Y, Shao J (2019) Video-based person re-identification via spatio-temporal attentional and two-stream fusion convolutional networks. Pattern Recogn Lett 117:153–160

Papernot N, McDaniel PD, Jha S, Fredrikson M, Celik ZB, Swami A (2016) The limitations of deep learning in adversarial settings. In: IEEE European symposium on security and privacy, euros&p, Saarbru̇cken, pp 372–387

Parchami M, Bashbaghi S, Granger E, Sayed S (2017) Using deep autoencoders to learn robust domain-invariant representations for still-to-video face recognition. In: 14Th IEEE international conference on advanced video and signal based surveillance, AVSS, Lecce, pp 1–6

Park U, Jain AK (2010) Face matching and retrieval using soft biometrics. IEEE Trans Inf Forensic Secur 5(3):406–415

Pei W, Dibeklioglu H, Baltrusaitis T, Tax DMJ (2020) Attended end-to-end architecture for age estimation from facial expression videos. IEEE Trans Image Process 29:1972–1984

Peng Y, Li L, Liu S, Lei T (2018) Space-frequency domain based joint dictionary learning and collaborative representation for face recognition. Signal Process 147:101–109

Peng C, Wang N, Li J, Gao X (2019) Re-ranking high-dimensional deep local representation for nir-vis face recognition. IEEE Trans Image Process 28(9):4553–4565

Phillips PJ, Flynn PJ, Scruggs T, Bowyer KW, Worek W (2006) Preliminary face recognition grand challenge results. In: 2006. FGR 2006. 7th international conference on Automatic face and gesture recognition. IEEE, pp 15–24

Phillips PJ, Wechsler H, Huang J, Rauss PJ (1998) The feret database and evaluation procedure for face-recognition algorithms. Image Vis Comput 16(5):295–306

Phillips PJ, Beveridge JR, Draper BA, Givens GH, O’Toole AJ, Bolme DS, Dunlop JP, Lui YM, Sahibzada H, Weimer S (2011) An introduction to the good, the bad, & the ugly face recognition challenge problem. In: Ninth IEEE international conference on automatic face and gesture recognition, Santa Barbara, pp 346–353

Pitas K, Loukas A, Davies M, Vandergheynst P (2019) Some limitations of norm based generalization bounds in deep neural networks. CoRR arXiv: 1905.09677

Plenge E, Klein SS, Niessen WJ, Meijering E (2015) Multiple sparse representations classification. PLOS ONE 10(7):1–23

Poder E (2017) Capacity limitations of visual search in deep convolutional neural network. CoRR arXiv: 1707.09775

Poon G, Kwan KC, Pang W (2019) Occlusion-robust bimanual gesture recognition by fusing multi-views. Multimed Tools Appl 78 (16):23469–23488

Pujol FA, Pujol M, Jimeno-morenilla A, Pujol MJ (2017) Face detection based on skin color segmentation using fuzzy entropy. Entropy 19(1):26

Punyani P, Gupta R, Kumar A (2020) Neural networks for facial age estimation: a survey on recent advances. Artif Intell Rev 53(5):3299–3347

Qian J, Yang J, Zhang F, Lin Z (2014) Robust low-rank regularized regression for face recognition with occlusion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 21–26

Qian J (2018) A survey on sentiment classification in face recognition. J Phys Conf Ser 960:012030

Queirolo CC, Silva L, Bellon ORP, Segundo MP (2010) 3d face recognition using simulated annealing and the surface interpenetration measure. IEEE Trans Pattern Anal Mach Intell 32(2): 206–219

Rejeesh MR (2019) Interest point based face recognition using adaptive neuro fuzzy inference system. Multimed Tools Appl 78(16):22691–22710

Radford A, Metz L, Chintala S (2016) Unsupervised representation learning with deep convolutional generative adversarial networks. In: 4Th international conference on learning representations, ICLR 2016, San Juan, Conference Track Proceedings

Raghavendra R, Raja KB, Venkatesh S, Cheikh FA, Busch C (2017) On the vulnerability of extended multispectral face recognition systems towards presentation attacks. In: IEEE International conference on identity, security and behavior analysis, ISBA, New Delhi, pp 1–8

Rajan S, Chenniappan P, Devaraj S, Madian N (2019) Facial expression recognition techniques: a comprehensive survey. IET Image Process 13 (7):1031–1040

Rakshit RD, Nath SC, Kisku DR (2018) Face identification using some novel local descriptors under the influence of facial complexities. Expert Syst Appl 92:82–94

Ramalingam S (2018) Fuzzy interval-valued multi criteria based decision making for ranking features in multi-modal 3d face recognition. Fuzzy Sets Syst 337:25–51

Ranjan R, Sankaranarayanan S, Bansal A, Bodla N, Chen J, Patel VM, Castillo CD, Chellappa R (2018) Deep learning for understanding faces: Machines may be just as good, or better, than humans. IEEE Signal Process Mag 35(1):66–83

Rassadin A, Gruzdev A, Savchenko A (2017) Group-level emotion recognition using transfer learning from face identification. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI, pp 544–548

Roberts LG (1963) Machine perception of three-dimensional solids. Ph.D. thesis Massachusetts Institute of Technology

Rubinstein R, Bruckstein AM, Elad M (2010) Dictionaries for sparse representation modeling. Proc IEEE 98(6):1045–1057

Saeed U, Khan MM (2018) Combining ear-based traditional and soft biometrics for unconstrained ear recognition. J Electron Imaging 27(05):051220

Sajjad M, Nasir M, Ullah FUM, Muhammad K, Sangaiah AK, Baik SW (2019) Raspberry pi assisted facial expression recognition framework for smart security in law-enforcement services. Inf Sci 479:416–431

Salici A, Ciampini C (2017) Automatic face recognition and identification tools in the forensic science domain. In: International tyrrhenian workshop on digital communication. Springer, pp 8–17

Sargano AB, Wang X, Angelov P, Habib Z (2017) Human action recognition using transfer learning with deep representations. In: International joint conference on neural networks, IJCNN, Anchorage, pp 463–469

Savran A, Alyüz N., Dibeklioġlu H, Ċeliktutan O, Gökberk B, Sankur B, Akarun L (2008) Bosphorus database for 3d face analysis. In: European workshop on biometrics and identity management. Springer, pp 47–56

Sawant MM, Bhurchandi KM (2019) Age invariant face recognition: a survey on facial aging databases, techniques and effect of aging. Artif Intell Rev 52(2):981–1008

Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 815–823

Sepas-Moghaddam A, Pereira F, Correia PL (2019) Face recognition: A novel multi-level taxonomy based survey. CoRR arXiv: 1901.00713

Serre T, Wolf L, Poggio T (2005) Object recognition with features inspired by visual cortex. In: IEEE Computer society conference on computer vision and pattern recognition (CVPR), vol 2, pp 994–1000

Shang K, Huang Z, Liu W, Li Z (2018) A single gallery-based face recognition using extended joint sparse representation. Appl Math Comput 320:99–115

Shao C, Song X, Feng Z, Wu X, Zheng Y (2017) Dynamic dictionary optimization for sparse-representation-based face classification using local difference images. Inf Sci 393:1–14

Shi B, Bai X, Liu W, Wang J (2018) Face alignment with deep regression. IEEE Trans Neural Netw Learn Syst 29(1):183–194

Sim T, Baker S, Bsat M (2001) The cmu pose, illumination, and expression (pie) database of human faces. Tech. Rep. CMU-RI-TR-01-02, Carnegie Mellon University, Pittsburgh

Sim T, Baker S, Bsat M (2002) The cmu pose, illumination, and expression (pie) database. In: Proceedings of Fifth IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, pp 53–58

Sing JK, Dey A, Ghosh M (2019) Confidence factor weighted gaussian function induced parallel fuzzy rank-level fusion for inference and its application to face recognition. Inf Fus 47:60–71

Sirovich L, Kirby M (1987) Low-dimensional procedure for the characterization of human faces. J Opt Soc Am A 4(3):519–524

Skocaj D, Leonardis A, Bischof H (2007) Weighted and robust learning of subspace representations. Pattern Recognit 40(5):1556–1569

Soltanpour S, Boufama B, Wu QMJ (2017) A survey of local feature methods for 3d face recognition. Pattern Recogn 72:391–406

Stonham TJ (1986) Practical Face Recognition and Verification with Wisard. Springer, Netherlands, pp 426–441

Sun Y, Wang X, Tang X (2013) Deep convolutional network cascade for facial point detection. In: IEEE Conference on computer vision and pattern recognition, Portland, pp 3476–3483

Sun Y, Wang X, Tang X (2013) Hybrid deep learning for face verification. In: IEEE International conference on computer vision, ICCV, Sydney, pp 1489–1496

Sun Y, Wang X, Tang X (2014) Deep learning face representation from predicting 10, 000 classes. In: IEEE Conference on computer vision and pattern recognition, CVPR, Columbus, pp 1891–1898

Sun Y, Chen Y, Wang X, Tang X (2014) Deep learning face representation by joint identification-verification. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS, pp 1988–1996

Sun Y, Liang D, Wang X, Tang X (2015) Deepid3: Face recognition with very deep neural networks. CoRR arXiv: 1502.00873

Su Y, Yang Y, Guo Z, Yang W (2015) Face recognition with occlusion. In: 3Rd IAPR asian conference on pattern recognition (ACPR), pp 670–674

Sun Y, Zhang M, Sun Z, Tan T (2018) Demographic analysis from biometric data: achievements, challenges, and new frontiers. IEEE Trans Pattern Anal Mach Intell 40(2):332–351

Sun P, Liu H, Wang X, Yu Z, Suping W (2019) Similarity-aware deep adversarial learning for facial age estimation. In: IEEE International conference on multimedia and expo, ICME 2019, Shanghai, pp 260–265

Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650

Tang J, Li Z, Zhu X (2018) Supervised deep hashing for scalable face image retrieval. Pattern Recogn 75:25–32

Tang Z, Wu X, Fu B, Chen W, Feng H (2018) Fast face recognition based on fractal theory. Appl Math Comput 321:721–730

The fei face image database available online:. https://fei.edu.br/~cet/facedatabase.html . Accessed: 2019-07-23

The texas 3d face database, available online:. http://live.ece.utexas.edu/research/texas3dfr/ . Accessed: 2019-07-23

The university of stirling face database, available online. http://pics.stir.ac.uk/ . Accessed: 2019-06-23

The university of york 3d face database, available online:. https://www-users.cs.york.ac.uk/nep/research/3Dface/tomh/3DFaceDatabase.html . Accessed: 2019-07-23

Tolba AS (2000) A parameter-based combined classifier for invariant face recognition. Cybern Syst 31(8):837–849

Tome-Gonzalez P, Fiėrrez J, Vera-Rodríguez R, Nixon MS (2014) Soft biometrics and their application in person recognition at a distance. IEEE Trans Inf Forensic Secur 9(3):464–475

Tosic I, Frossard P (2011) Dictionary learning. IEEE Signal Proc Mag 28(2):27–38

Tsai C, Shih K (2019) Mining a new biometrics to improve the accuracy of keystroke dynamics-based authentication system on free-text. Appl Soft Comput 80:125–137

Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86

Vezzetti E, Marcolin F, Tornincasa S, Ulrich L, Dagnes N (2018) 3d geometry-based automatic landmark localization in presence of facial occlusions. Multimed Tools Appl 77(11):14177–14205

Viola PA, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

Vishwakarma VP, Goel T (2019) An efficient hybrid dwt-fuzzy filter in DCT domain based illumination normalization for face recognition. Multimed Tools Appl 78(11):15213–15233

Vishwakarma VP, Dalal S (2020) A novel non-linear modifier for adaptive illumination normalization for robust face recognition. Multim Tools Appl 79(17-18):11503–11529

Wagner A, Wright J, Ganesh A, Zhou Z, Ma Y (2009) Towards a practical face recognition system: Robust registration and illumination by sparse representation. In: IEEE Computer society conference on computer vision and pattern recognition (CVPR ), Miami, pp 597–604

Wang Y, Anderson PG, Gaborski RS (2009) Face recognition using a hybrid model. In: IEEE Applied imagery pattern recognition workshop, Washington, pp 1–8

Wang D, Cui P, Zhu W (2016) Structural deep network embedding. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, pp 1225–1234

Wang H, Hu J, Deng W (2018) Face feature extraction: A complete review. IEEE Access 6:6001–6039

Wang L, Wang Y, Liu B (2018) Laplace graph embedding class specific dictionary learning for face recognition. J Electr Comput Eng 2018:2179049:1–2179049:11

Wang K, Peng X, Yang J, Meng D, Qiao Y (2020) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:4057–4069

Wen Y, Li Z, Qiao Y (2016) Latent factor guided convolutional neural networks for age-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4893–4901

Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: Computer vision - ECCV 2016 - 14th european conference, Amsterdam, Proceedings, Part VII, pp 499–515

Wiskott L, Krüger N, Kuiger N, von der Malsburg C (1997) Face recognition by elastic bunch graph matching. IEEE Trans Pattern Anal Mach Intell 19(7):775–779

Wong Y, Chen S, Mau S, Sanderson C, Lovell BC (2011) Patch-based probabilistic image quality assessment for face selection and improved video-based face recognition. In: IEEE Biometrics workshop, computer vision and pattern recognition (CVPR) workshops. IEEE, pp 81–88

Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):210–227

Wright J, Ma Y, Mairal J, Sapiro G, Huang TS, Yan S (2010) Sparse representation for computer vision and pattern recognition. Proc IEEE 98(6):1031–1044

Wu H, Shao J, Xu X, Ji Y, Shen F, Shen HT (2018) Recognition and detection of two-person interactive actions using automatically selected skeleton features. IEEE Trans Hum-Mach Syst 48(3):304–310

Wu F, Jing X, Dong X, Hu R, Yue D, Wang L, Ji Y, Wang R, Chen G (2020) Intraspectrum discrimination and interspectrum correlation analysis deep network for multispectral face recognition. IEEE Trans Cybern 50 (3):1009–1022

Xie J, Pun C (2019) Chronological age estimation under the guidance of age-related facial attributes. IEEE Trans Inf Forensic Secur 14 (9):2500–2511

Xie J, Pun C (2020) Deep and ordinal ensemble learning for human age estimation from facial images. IEEE Trans Inf Forensic Secur 15:2361–2374

Xu J, Ma S, Zhang Y, Wei B, Cai X, Sun X (2017) Transfer deep learning for low-resource chinese word segmentation with a novel neural network. In: Natural language processing and chinese computing - 6th CCF international conference, NLPCC 2017, Dalian, Proceedings, pp 721–730

Xu Y, Zhong Z, Yang J, You J, Zhang D (2017) A new discriminative sparse representation method for robust face recognition via l 2 regularization. IEEE Trans Neural Netw Learn Syst 28(10): 2233–2242

Xue Y Non-negative matrix factorization for face recognition. Ph.D. thesis (2007). AAI3302378

Yang M (2002) Kernel eigenfaces vs. kernel fisherfaces: Face recognition using kernel methods. In: 5Th IEEE international conference on automatic face and gesture recognition (FGR), Washington, pp 215–220

Yang W, Yan H, Wang J, Yang J (2008) Face recognition using complete fuzzy LDA. In: 19Th international conference on pattern recognition (ICPR), Tampa, pp 1–4

Yang M, Zhang L, Yang J, Zhang D (2011) Robust sparse coding for face recognition. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 625–632

Yang M, Zhang L, Yang J, Zhang D (2013) Regularized robust coding for face recognition. IEEE Trans Image Process 22(5):1753–1766

Yin X, Yu X, Sohn K, Liu X, Chandraker M (2018) Feature transfer learning for deep face recognition with long-tail data. CoRR arXiv: 1803.09014

Ying Han P, Jin ATB, Heng Siong L (2011) Eigenvector weighting function in face recognition. Discrete Dynamics in Nature and Society

Yoo B, Kwak Y, Kim Y, Choi C, Kim J (2018) Deep facial age estimation using conditional multitask learning with weak label expansion. IEEE Signal Process Lett 25(6):808–812

Yu D, Wu X (2018) 2dpcanet: a deep leaning network for face recognition. Multimed Tools Appl 77(10):12919–12934

Yuille AL, Hallinan PW, Cohen DS (1992) Feature extraction from faces using deformable templates. Int J Comput Vis 8(2):99–111

Zadeh LA (1965) Fuzzy sets. Inf Control 8(3):338–353

Zafar U, Ghafoor M, Zia T, Ahmed G, Latif A, Malik KR, Sharif AM (2019) Face recognition with bayesian convolutional networks for robust surveillance systems. EURASIP J Image Video Process 2019:10

Zhang X, Gao Y (2009) Face recognition across pose: A review. Pattern Recogn 42(11):2876–2896

Zhang Q, Li B (2010) Discriminative k-SVD for dictionary learning in face recognition. In: The twenty-third IEEE conference on computer vision and pattern recognition, CVPR, San Francisco, pp 2691–2698

Zhang L, Zhou W, Chang P, Liu J, Yan Z, Wang T, Li F (2012) Kernel sparse representation-based classifier. IEEE Trans Signal Process 60(4):1684–1695

Zhang H, Zhang Y, Huang TS (2013) Pose-robust face recognition via sparse representation. Pattern Recognit 46(5):1511–1521

Zhang Z, Xu Y, Yang J, Li X, Zhang D (2015) A survey of sparse representation: Algorithms and applications. IEEE Access 3:490–530

Zhang S, Li X, He H, Miao Y (2018) A next best view method based on self-occlusion information in depth images for moving object. Multimed Tools Appl 77(8):9753–9777

Zhang Y, Shao J, Ouyang D, Shen HT (2018) Person re-identification using two-stage convolutional neural network. In: 24Th international conference on pattern recognition, ICPR 2018, Beijing, pp 3341–3346

Zhang Y, Hu C, Lu X (2019) IL-GAN: illumination-invariant representation learning for single sample face recognition. J Vis Commun Image Represent 59:501–513

Zhang W, Zhao X, Morvan J, Chen L (2019) Improving shadow suppression for illumination robust face recognition. IEEE Trans Pattern Anal Mach Intell 41(3):611–624

Zhang F, Zhang T, Mao Q, Xu C (2020) Geometry guided pose-invariant facial expression recognition. IEEE Trans Image Process 29:4445–4460

Zhang L, Luo F (2020) Review on graph learning for dimensionality reduction of hyperspectral image. Geo spatial Inf Sci 23(1):98–106

Zhang M, Li Y, Wang N, Chi Y, Gao X (2020) Cascaded face sketch synthesis under various illuminations. IEEE Trans Image Process 29:1507–1521

Zhao W, Chellappa R, Phillips PJ, Rosenfeld A (2003) Face recognition: a literature survey. ACM Comput Surv (CSUR) 35(4):399–458

Zhao W, Chellappa R (1999) Robust face recognition using symmetric shape-from-shading. Computer Vision Laboratory, Center for Automation Research. University of Maryland Maryland, Md

Zhao W, Chellappa R, Phillips PJ (1999) Subspace linear discriminant analysis for face recognition. Citeseer

Zheng Y, Yang J, Wang W, Wang Q, Yang J, Wu X (2006) Fuzzy kernel fisher discriminant algorithm with application to face recognition. 6th World Congress Intell Control Autom 2:9669–9672

Zheng Y, Elmaghraby A (2011) A brief survey on multispectral face recognition and multimodal score fusion. In: IEEE International symposium on signal processing and information technology, ISSPIT, pp 543–550

Zhi R, Liu M, Zhang D (2020) A comprehensive survey on automatic facial action unit analysis. Vis Comput 36(5):1067–1093

Zhong Y, Den W (2019) Exploring features and attributes in deep face recognition using visualization techniques. In: 2019 14Th IEEE international conference on automatic face gesture recognition (FG 2019), pp 1–8

Zhu C, Zheng Y, Luu K, Savvides M (2017) CMS-RCNN: Contextual Multi-Scale Region-Based CNN For Unconstrained Face Detection. Springer International Publishing, Cham, pp 57–79

Zhu S, Yu K (2018) Concept transfer learning for adaptive language understanding. In: Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, Melbourne, pp 391–399

Download references

Author information

Authors and affiliations.

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China

Waqar Ali & Abdullah Aman Khan

Faculty of Information Technology, The University of Lahore, Lahore, 54000, Pakistan

School of Information and Software Engineerng, University of Electronic Science and Technology of China, Chengdu, 611731, China

Wenhong Tian

Data Mining Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China

Salah Ud Din

School of Electronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China

Desire Iradukunda

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Waqar Ali .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Ali, W., Tian, W., Din, S.U. et al. Classical and modern face recognition approaches: a complete review. Multimed Tools Appl 80 , 4825–4880 (2021). https://doi.org/10.1007/s11042-020-09850-1

Download citation

Received : 23 September 2019

Revised : 10 July 2020

Accepted : 09 September 2020

Published : 02 October 2020

Issue Date : January 2021

DOI : https://doi.org/10.1007/s11042-020-09850-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Face recognition
• Face identification
• Artificial intelligence
• Computer vision
• Machine learning
• Visual surveillance
• Find a journal
• Publish with us
• Track your research

Subscribe to the PwC Newsletter

Join the community, add a new evaluation result row, face recognition.

606 papers with code • 23 benchmarks • 64 datasets

Facial Recognition is the task of making a positive identification of a face in a photo or video image against a pre-existing database of faces. It begins with detection - distinguishing human faces from other objects in the image - and then works on identification of those detected faces.

The state of the art tables for this task are contained mainly in the consistent parts of the task : the face verification and face identification tasks.

( Image credit: Face Verification )

Benchmarks Add a Result

--> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> -->

Trend	Dataset	Best Model	Paper	Code	Compare
		GhostFaceNetV2-1 (MS1MV3)
		GhostFaceNetV2-1
		MS1MV2, R100, SFace
		Fine-tuned ArcFace
		Fine-tuned ArcFace
		ArcFace+CSFM
		PIC - QMagFace
		Prodpoly
		Prodpoly
		PIC - MagFace
		PIC - ArcFace
		FaceNet+Adaptive Threshold
		FaceNet+Adaptive Threshold
		FaceNet+Adaptive Threshold
		Model with Up Convolution + DoG Filter (Aligned)
		Model with Up Convolution + DoG Filter
		GhostFaceNetV2-1
		Model with Up Convolution + DoG Filter
		GhostFaceNetV2-1
		Multi-task
		FaceTransformer+OctupletLoss
		Partial FC
		MCN

Most implemented papers

Facenet: a unified embedding for face recognition and clustering.

On the widely used Labeled Faces in the Wild (LFW) dataset, our system achieves a new record accuracy of 99. 63%.

ArcFace: Additive Angular Margin Loss for Deep Face Recognition

Recently, a popular line of research in face recognition is adopting margins in the well-established softmax loss function to maximize class separability.

VGGFace2: A dataset for recognising faces across pose and age

The dataset was collected with three goals in mind: (i) to have both a large number of identities and also a large number of images for each identity; (ii) to cover a large range of pose, age and ethnicity; and (iii) to minimize the label noise.

SphereFace: Deep Hypersphere Embedding for Face Recognition

This paper addresses deep face recognition (FR) problem under open-set protocol, where ideal face features are expected to have smaller maximal intra-class distance than minimal inter-class distance under a suitably chosen metric space.

A Light CNN for Deep Face Representation with Noisy Labels

This paper presents a Light CNN framework to learn a compact embedding on the large-scale face data with massive noisy labels.

Learning Face Representation from Scratch

The current situation in the field of face recognition is that data is more important than algorithm.

Circle Loss: A Unified Perspective of Pair Similarity Optimization

This paper provides a pair similarity optimization viewpoint on deep feature learning, aiming to maximize the within-class similarity $s_p$ and minimize the between-class similarity $s_n$.

MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition

In this paper, we design a benchmark task and provide the associated datasets for recognizing face images and link them to corresponding entity keys in a knowledge base.

DeepID3: Face Recognition with Very Deep Neural Networks

Very deep neural networks recently achieved great success on general object recognition because of their superb learning capacity.

Can we still avoid automatic face detection?

Recognito-Vision/Linux-FaceRecognition-FaceLivenessDetection • 14 Feb 2016

In this setting, is it still possible for privacy-conscientious users to avoid automatic face detection and recognition?

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Sensors (Basel)

Face Recognition Systems: A Survey

Yassin kortli.

1 AI-ED Department, Yncrea Ouest, 20 rue du Cuirassé de Bretagne, 29200 Brest, France; [email protected] (M.J.); [email protected] (A.A.F.)

2 Electronic and Micro-electronic Laboratory, Faculty of Sciences of Monastir, University of Monastir, Monastir 5000, Tunisia

Maher Jridi

Ayman al falou, mohamed atri.

3 College of Computer Science, King Khalid University, Abha 61421, Saudi Arabia; as.ude.ukk@irtam

Over the past few decades, interest in theories and algorithms for face recognition has been growing rapidly. Video surveillance, criminal identification, building access control, and unmanned and autonomous vehicles are just a few examples of concrete applications that are gaining attraction among industries. Various techniques are being developed including local, holistic, and hybrid approaches, which provide a face image description using only a few face image features or the whole facial features. The main contribution of this survey is to review some well-known techniques for each approach and to give the taxonomy of their categories. In the paper, a detailed comparison between these techniques is exposed by listing the advantages and the disadvantages of their schemes in terms of robustness, accuracy, complexity, and discrimination. One interesting feature mentioned in the paper is about the database used for face recognition. An overview of the most commonly used databases, including those of supervised and unsupervised learning, is given. Numerical results of the most interesting techniques are given along with the context of experiments and challenges handled by these techniques. Finally, a solid discussion is given in the paper about future directions in terms of techniques to be used for face recognition.

1. Introduction

The objective of developing biometric applications, such as facial recognition, has recently become important in smart cities. In addition, many scientists and engineers around the world have focused on establishing increasingly robust and accurate algorithms and methods for these types of systems and their application in everyday life. All types of security systems must protect all personal data. The most commonly used type for recognition is the password. However, through the development of information technologies and security algorithms, many systems are beginning to use many biometric factors for recognition task [ 1 , 2 , 3 , 4 ]. These biometric factors make it possible to identify people’s identity by their physiological or behavioral characteristics. They also provide several advantages, for example, the presence of a person in front of the sensor is sufficient, and there is no more need to remember several passwords or confidential codes anymore. In this context, many recognition systems based on different biometric factors such as iris, fingerprints [ 5 ], voice [ 6 ], and face have been deployed in recent years.

Systems that identify people based on their biological characteristics are very attractive because they are easy to use. The human face is composed of different structures and characteristics. For this reason, in recent years, it has become one of the most widely used biometric authentication systems, given its potential in many applications and fields (surveillance, home security, border control, and so on) [ 7 , 8 , 9 ]. Facial recognition system as an ID (identity) is already being offered to consumers outside of phones, including at airport check-ins, sports stadiums, and concerts. In addition, this system does not require the intervention of people to operate, which makes it possible to identify people only from images obtained from the camera. In addition, many biometric systems that are developed using different types of search provide good identification accuracy. However, it would be interesting to develop new biometric systems for face recognition in order to reach real-time constraints.

Owing to the huge volume of data generated and rapid advancement in artificial intelligence techniques, traditional computing models have become inadequate to process data, especially for complex applications like those related to feature extraction. Graphics processing units (GPUs) [ 4 ], central processing unit (CPU) [ 3 ], and programmable gate arrays (FPGAs) [ 10 ] are required to efficiently perform complex computing tasks. GPUs have computing cores that are several orders of magnitude larger than traditional CPU and allow greater capacity to perform parallel computing. Unlike GPUs, the FPGAs have a flexible hardware configuration and offer better performance than GPUs in terms of energy efficiency. However, FPGAs present a major drawback related to the programming time, which is higher than that of CPU and GPU.

There are many computer vision approaches proposed to address face detection or recognition tasks with high robustness and discrimination, such as local, subspace, and hybrid approaches [ 10 , 11 , 12 , 13 , 14 , 15 , 16 ]. However, several issues still need to be addressed owing to various challenges, such as head orientation, lighting conditions, and facial expression. The most interesting techniques are developed to face all these challenges, and thus develop reliable face recognition systems. Nevertheless, they require high processing time, high memory consumption, and are relatively complex.

Rapid advances in technologies such as digital cameras, portable devices, and increased demand for security make the face recognition system one of the primary biometric technologies.

To sum up, the contributions of this paper review are as follows:

• We first introduced face recognition as a biometric technique.
• We presented the state of the art of the existing face recognition techniques classified into three approaches: local, holistic, and hybrid.
• The surveyed approaches were summarized and compared under different conditions.
• We presented the most popular face databases used to test these approaches.
• We highlighted some new promising research directions.

2. Face Recognition Systems Survey

2.1. essential steps of face recognition systems.

Before detailing the techniques used, it is necessary to make a brief description of the problems that must be faced and solved in order to perform the face recognition task correctly. For several security applications, as detailed in the works of [ 17 , 18 , 19 , 20 , 21 , 22 ], the characteristics that make a face recognition system useful are the following: its ability to work with both videos and images, to process in real time, to be robust in different lighting conditions, to be independent of the person (regardless of hair, ethnicity, or gender), and to be able to work with faces from different angles. Different types of sensors, including RGB, depth, EEG, thermal, and wearable inertial sensors, are used to obtain data. These sensors may provide extra information and help the face recognition systems to identify face images in both static images and video sequences. Moreover, three categories of sensors that may improve the reliability and the accuracy of a face recognition system by tackling the challenges include illumination variation, head pose, and facial expression in pure image/video processing. The first group is non-visual sensors, such as audio, depth, and EEG sensors, which provide extra information in addition to the visual dimension and improve the recognition reliability, for example, in illumination variation and position shift situation. The second is detailed-face sensors, which detect a small dynamic change of a face component, such as eye-trackers, which may help differentiate the background noise and the face images. The last is target-focused sensors, such as infrared thermal sensors, which can facilitate the face recognition systems to filter useless visual contents and may help resistance illumination variation.

Three basic steps are used to develop a robust face recognition system: (1) face detection, (2) feature extraction, and (3) face recognition (shown in Figure 1 ) [ 3 , 23 ]. The face detection step is used to detect and locate the human face image obtained by the system. The feature extraction step is employed to extract the feature vectors for any human face located in the first step. Finally, the face recognition step includes the features extracted from the human face in order to compare it with all template face databases to decide the human face identity.

• Face Detection : The face recognition system begins first with the localization of the human faces in a particular image. The purpose of this step is to determine if the input image contains human faces or not. The variations of illumination and facial expression can prevent proper face detection. In order to facilitate the design of a further face recognition system and make it more robust, pre-processing steps are performed. Many techniques are used to detect and locate the human face image, for example, Viola–Jones detector [ 24 , 25 ], histogram of oriented gradient (HOG) [ 13 , 26 ], and principal component analysis (PCA) [ 27 , 28 ]. Also, the face detection step can be used for video and image classification, object detection [ 29 ], region-of-interest detection [ 30 ], and so on.
• Feature Extraction : The main function of this step is to extract the features of the face images detected in the detection step. This step represents a face with a set of features vector called a “signature” that describes the prominent features of the face image such as mouth, nose, and eyes with their geometry distribution [ 31 , 32 ]. Each face is characterized by its structure, size, and shape, which allow it to be identified. Several techniques involve extracting the shape of the mouth, eyes, or nose to identify the face using the size and distance [ 3 ]. HOG [ 33 ], Eigenface [ 34 ], independent component analysis (ICA), linear discriminant analysis (LDA) [ 27 , 35 ], scale-invariant feature transform (SIFT) [ 23 ], gabor filter, local phase quantization (LPQ) [ 36 ], Haar wavelets, Fourier transforms [ 31 ], and local binary pattern (LBP) [ 3 , 10 ] techniques are widely used to extract the face features.
• Face Recognition : This step considers the features extracted from the background during the feature extraction step and compares it with known faces stored in a specific database. There are two general applications of face recognition, one is called identification and another one is called verification. During the identification step, a test face is compared with a set of faces aiming to find the most likely match. During the identification step, a test face is compared with a known face in the database in order to make the acceptance or rejection decision [ 7 , 19 ]. Correlation filters (CFs) [ 18 , 37 , 38 ], convolutional neural network (CNN) [ 39 ], and also k-nearest neighbor (K-NN) [ 40 ] are known to effectively address this task.

Face recognition structure [ 3 , 23 ].

2.2. Classification of Face Recognition Systems

Compared with other biometric systems such as the eye, iris, or fingerprint recognition systems, the face recognition system is not the most efficient and reliable [ 5 ]. Moreover, this biometric system has many constraints resulting from many challenges, despite all the above advantages. The recognition under the controlled environments has been saturated. Nevertheless, in uncontrolled environments, the problem remains open owing to large variations in lighting conditions, facial expressions, age, dynamic background, and so on. In this paper survey, we review the most advanced face recognition techniques proposed in controlled/uncontrolled environments using different databases.

Several systems are implemented to identify a human face in 2D or 3D images. In this review paper, we will classify these systems into three approaches based on their detection and recognition method ( Figure 2 ): (1) local, (2) holistic (subspace), and (3) hybrid approaches. The first approach is classified according to certain facial features, not considering the whole face. The second approach employs the entire face as input data and then projects into a small subspace or in correlation plane. The third approach uses local and global features in order to improve face recognition accuracy.

Face recognition methods. SIFT, scale-invariant feature transform; SURF, scale-invariant feature transform; BRIEF, binary robust independent elementary features; LBP, local binary pattern; HOG, histogram of oriented gradients; LPQ, local phase quantization; PCA, principal component analysis; LDA, linear discriminant analysis; KPCA, kernel PCA; CNN, convolutional neural network; SVM, support vector machine.

3. Local Approaches

In the context of face recognition, local approaches treat only some facial features. They are more sensitive to facial expressions, occlusions, and pose [ 1 ]. The main objective of these approaches is to discover distinctive features. Generally, these approaches can be divided into two categories: (1) local appearance-based techniques are used to extract local features, while the face image is divided into small regions (patches) [ 3 , 32 ]. (2) Key-points-based techniques are used to detect the points of interest in the face image, after which the features localized on these points are extracted.

3.1. Local Appearance-Based Techniques

It is a geometrical technique, also called feature or analytic technique. In this case, the face image is represented by a set of distinctive vectors with low dimensions or small regions (patches). Local appearance-based techniques focus on critical points of the face such as the nose, mouth, and eyes to generate more details. Also, it takes into account the particularity of the face as a natural form to identify and use a reduced number of parameters. In addition, these techniques describe the local features through pixel orientations, histograms [ 13 , 26 ], geometric properties, and correlation planes [ 3 , 33 , 41 ].

The local binary pattern (LBP) descriptor [ 19 ].

Khoi et al. [ 20 ] propose a fast face recognition system based on LBP, pyramid of local binary pattern (PLBP), and rotation invariant local binary pattern (RI-LBP). Xi et al. [ 15 ] have introduced a new unsupervised deep learning-based technique, called local binary pattern network (LBPNet), to extract hierarchical representations of data. The LBPNet maintains the same topology as the convolutional neural network (CNN). The experimental results obtained using the public benchmarks (i.e., LFW and FERET) have shown that LBPNet is comparable to other unsupervised techniques. Laure et al. [ 40 ] have implemented a method that helps to solve face recognition issues with large variations of parameters such as expression, illumination, and different poses. This method is based on two techniques: LBP and K-NN techniques. Owing to its invariance to the rotation of the target image, LBP become one of the important techniques used for face recognition. Bonnen et al. [ 42 ] proposed a variant of the LBP technique named “multiscale local binary pattern (MLBP)” for features’ extraction. Another LBP extension is the local ternary pattern (LTP) technique [ 43 ], which is less sensitive to the noise than the original LBP technique. This technique uses three steps to compute the differences between the neighboring ones and the central pixel. Hussain et al. [ 36 ] develop a local quantized pattern (LQP) technique for face representation. LQP is a generalization of local pattern features and is intrinsically robust to illumination conditions. The LQP features use the disk layout to sample pixels from the local neighborhood and obtain a pair of binary codes using ternary split coding. These codes are quantized, with each one using a separately learned codebook.

The magnitude of the gradient and the orientation of each pixel in the cell are voted in nine bins with the tri-linear interpolation. The histograms of each cell are generated pixel based on direction gradients and, finally, the histograms of the whole cells are combined to extract the feature of the face image. Karaaba et al. [ 44 ] proposed a combination of different histograms of oriented gradients (HOG) to perform a robust face recognition system. This technique is named “multi-HOG”.

The authors create a vector of distances between the target and the reference face images for identification. Arigbabu et al. [ 46 ] proposed a novel face recognition system based on the Laplacian filter and the pyramid histogram of gradient (PHOG) descriptor. In addition, to investigate the face recognition problem, support vector machine (SVM) is used with different kernel functions.

All “ 4 f ” optical configuration [ 37 ].

For example, the VLC technique is done by two cascade Fourier transform structures realized by two lenses [ 4 ], as presented in Figure 5 . The VLC technique is presented as follows: firstly, a 2D-FFT is applied to the target image to get a target spectrum S . After that, a multiplication between the target spectrum and the filter obtain with the 2D-FFT of a reference image is affected, and this result is placed in the Fourier plane. Next, it provides the correlation result recorded on the correlation plane, where this multiplication is affected by inverse FF.

Flowchart of the VanderLugt correlator (VLC) technique [ 4 ]. FFT, fast Fourier transform; POF, phase-only filter.

The correlation result, described by the peak intensity, is used to determine the similarity degree between the target and reference images.

where F F T − 1 stands for the inverse fast FT (FFT) operation, * represents the conjugate operation, and ∘ denotes the element-wise array multiplication. To enhance the matching process, Horner and Gianino [ 49 ] proposed a phase-only filter (POF). The POF filter can produce correlation peaks marked with enhanced discrimination capability. The POF is an optimized filter defined as follows:

where S ∗ ( u , v ) is the complex conjugate of the 2D-FFT of the reference image. To evaluate the decision, the peak to correlation energy (PCE) is defined as the energy in the correlation peaks’ intensity normalized to the overall energy of the correlation plane.

where i , j are the coefficient coordinates; M and N are the size of the correlation plane and the size of the peak correlation spot, respectively; E p e a k is the energy in the correlation peaks; and E c o r r e l a t i o n − p l a n e is the overall energy of the correlation plane. Correlation techniques are widely applied in recognition and identification applications [ 4 , 37 , 50 , 51 , 52 , 53 ]. For example, in the work of [ 4 ], the authors presented the efficiency performances of the VLC technique based on the “4f” configuration for identification using GPU Nvidia Geforce 8400 GS. The POF filter is used for the decision. Another important work in this area of research is presented by Leonard et al. [ 50 ], which presented good performance and the simplicity of the correlation filters for the field of face recognition. In addition, many specific filters such as POF, BPOF, Ad, IF, and so on are used to select the best filter based on its sensitivity to the rotation, scale, and noise. Napoléon et al. [ 3 ] introduced a novel system for identification and verification fields based on an optimized 3D modeling under different illumination conditions, which allows reconstructing faces in different poses. In particular, to deform the synthetic model, an active shape model for detecting a set of key points on the face is proposed in Figure 6 . The VanderLugt correlator is proposed to perform the identification and the LBP descriptor is used to optimize the performances of a correlation technique under different illumination conditions. The experiments are performed on the Pointing Head Pose Image Database (PHPID) database with an elevation ranging from −30° to +30°.

( a ) Creation of the 3D face of a person, ( b ) results of the detection of 29 landmarks of a face using the active shape model, ( c ) results of the detection of 26 landmarks of a face [ 3 ].

3.2. Key-Points-Based Techniques

The key-points-based techniques are used to detect specific geometric features, according to some geometric information of the face surface (e.g., the distance between the eyes, the width of the head). These techniques can be defined by two significant steps, key-point detection and feature extraction [ 3 , 30 , 54 , 55 ]. The first step focuses on the performance of the detectors of the key-point features of the face image. The second step focuses on the representation of the information carried with the key-point features of the face image. Although these techniques can solve the missing parts and occlusions, scale invariant feature transform (SIFT), binary robust independent elementary features (BRIEF), and speeded-up robust features (SURF) techniques are widely used to describe the feature of the face image.

• Scale invariant feature transform (SIFT) [ 56 , 57 ]: SIFT is an algorithm used to detect and describe the local features of an image. This algorithm is widely used to link two images by their local descriptors, which contain information to make a match between them. The main idea of the SIFT descriptor is to convert the image into a representation composed of points of interest. These points contain the characteristic information of the face image. SIFT presents invariance to scale and rotation. It is commonly used today and fast, which is essential in real-time applications, but one of its disadvantages is the time of matching of the critical points. The algorithm is realized in four steps: (1) detection of the maximum and minimum points in the space-scale, (2) location of characteristic points, (3) assignment of orientation, and (4) a descriptor of the characteristic point. A framework to detect the key-points based on the SIFT descriptor was proposed by L. Lenc et al. [ 56 ], where they use the SIFT technique in combination with a Kepenekci approach for the face recognition.

Face recognition based on the speeded-up robust features (SURF) descriptor [ 58 ]: recognition using fast library for approximate nearest neighbors (FLANN) distance.

• Binary robust independent elementary features (BRIEF) [ 30 , 57 ]: BRIEF is a binary descriptor that is simple and fast to compute. This descriptor is based on the differences between the pixel intensity that are similar to the family of binary descriptors such as binary robust invariant scalable (BRISK) and fast retina keypoint (FREAK) in terms of evaluation. To reduce noise, the BRIEF descriptor smoothens the image patches. After that, the differences between the pixel intensity are used to represent the descriptor. This descriptor has achieved the best performance and accuracy in pattern recognition.

Fast retina keypoint (FREAK) descriptor used 43 sampling patterns [ 19 ].

3.3. Summary of Local Approaches

Table 1 summarizes the local approaches that we presented in this section. Various techniques are introduced to locate and to identify the human faces based on some regions of the face, geometric features, and facial expressions. These techniques provide robust recognition under different illumination conditions and facial expressions. Furthermore, they are sensitive to noise, and invariant to translations and rotations.

Summary of local approaches. SIFT, scale-invariant feature transform; SURF, scale-invariant feature transform; BRIEF, binary robust independent elementary features; LBP, local binary pattern; HOG, histogram of oriented gradients; LPQ, local phase quantization; PCA, principal component analysis; LDA, linear discriminant analysis; KPCA, kernel PCA; CNN, convolutional neural network; SVM, support vector machine; PLBP, pyramid of LBP; KNN, k-nearest neighbor; MLBP, multiscale LBP; LTP, local ternary pattern.; PHOG, pyramid HOG; VLC, VanderLugt correlator; LFW, Labeled Faces in the Wild; FERET, Face Recognition Technology; PHPID, Pointing Head Pose Image Database; PCE, peak to correlation energy; POF, phase-only filter; PSR, peak-to-sidelobe ratio.

Author/Technique Used		Database	Matching	Limitation	Advantage	Result
Local Appearance-Based Techniques
Khoi et al. [ ]	LBP	TDF	MAP	Skewness in face image	Robust feature in fontal face	5%
		CF1999				13.03%
		LFW				90.95%
Xi et al. [ ]	LBPNet	FERET	Cosine similarity	Complexities of CNN	High recognition accuracy	97.80%
		LFW				94.04%
Khoi et al. [ ]	PLBP	TDF	MAP	Skewness in face image	Robust feature in fontal face	5.50%
		CF				9.70%
		LFW				91.97%
Laure et al. [ ]	LBP and KNN	LFW	KNN	Illumination conditions	Robust	85.71%
		CMU-PIE				99.26%
Bonnen et al. [ ]	MRF and MLBP	AR (Scream)	Cosine similarity	Landmark extraction fails or is not ideal	Robust to changes in facial expression	86.10%
		FERET (Wearing sunglasses)				95%
Ren et al. [ ]	Relaxed LTP	CMU-PIE	Chisquare distance	Noise level	Superior performance compared with LBP, LTP	95.75%
		Yale B				98.71%
Hussain et al. [ ]	LPQ	FERET/	Cosine similarity	Lot of discriminative information	Robust to illumination variations	99.20%
		LFW				75.30%
Karaaba et al. [ ]	HOG and MMD	FERET	MMD/MLPD	Low recognition accuracy	Aligning difficulties	68.59%
		LFW				23.49%
Arigbabu et al. [ ]	PHOG and SVM	LFW	SVM	Complexity and time of computation	Head pose variation	88.50%
Leonard et al. [ ]	VLC correlator	PHPID	ASPOF	The low number of the reference image used	Robustness to noise	92%
Napoléon et al. [ ]	LBP and VLC	YaleB	POF	Illumination	Rotation + Translation	98.40%
		YaleB Extended				95.80%
Heflin et al. [ ]	correlation filter	LFW/PHPID	PSR	Some pre-processing steps	More effort on the eye localization stage	39.48%
Zhu et al. [ ]	PCA–FCF	CMU-PIE	Correlation filter	Use only linear method	Occlusion-insensitive	96.60%
		FRGC2.0				91.92%
Seo et al. [ ]	LARK + PCA	LFW	Cosine similarity	Face detection	Reducing computational complexity	78.90%
Ghorbel et al. [ ]	VLC + DoG	FERET	PCE	Low recognition rate	Robustness	81.51%
Ghorbel et al. [ ]	uLBP + DoG	FERET	chi-square distance	Robustness	Processing time	93.39%
Ouerhani et al. [ ]	VLC	PHPID	PCE	Power	Processing time	77%
Lenc et al. [ ]	SIFT	FERET	a posterior probability	Still far to be perfect	Sufficiently robust on lower quality real data	97.30%
		AR				95.80%
		LFW				98.04%
Du et al. [ ]	SURF	LFW	FLANN distance	Processing time	Robustness and distinctiveness	95.60%
Vinay et al. [ ]	SURF + SIFT	LFW	FLANN	Processing time	Robust in unconstrained scenarios	78.86%
		Face94	distance			96.67%
Calonder et al. [ ]	BRIEF	_	KNN	Low recognition rate	Low processing time	48%

4. Holistic Approach

Holistic or subspace approaches are supposed to process the whole face, that is, they do not require extracting face regions or features points (eyes, mouth, noses, and so on). The main function of these approaches is to represent the face image by a matrix of pixels, and this matrix is often converted into feature vectors to facilitate their treatment. After that, these feature vectors are implemented in low dimensional space. However, holistic or subspace techniques are sensitive to variations (facial expressions, illumination, and poses), and these advantages make these approaches widely used. Moreover, these approaches can be divided into categories, including linear and non-linear techniques, based on the method used to represent the subspace.

4.1. Linear Techniques

The most popular linear techniques used for face recognition systems are Eigenfaces (principal component analysis; PCA) technique, Fisherfaces (linear discriminative analysis; LDA) technique, and independent component analysis (ICA).

Example of dimensional reduction when applying principal component analysis (PCA) [ 62 ].

The first five Eigenfaces built from the ORL database [ 63 ].

An image may also be considering the vector of dimension M × N , so that a typical image of size 4 × 4 becomes a vector of dimension 16. Let the training set of images be { X 1 , X 2 ,   X 3 …   X N } . The average face of the set is defined by the following:

Calculate the estimate covariance matrix to represent the scatter degree of all feature vectors related to the average vector. The covariance matrix Q is defined by the following:

The Eigenvectors and corresponding Eigen-values are computed using

where V is the set of eigenvectors matrix Q associated with its eigenvalue λ . Project all the training images of i t h person to the corresponding Eigen-subspace:

where the y k i are the projections of x and are called the principal components, also known as eigenfaces. The face images are represented as a linear combination of these vectors’ “principal components”. In order to extract facial features, PCA and LDA are two different feature extraction algorithms that are used. Wavelet fusion and neural networks are applied to classify facial features. The ORL database is used for evaluation. Figure 10 shows the first five Eigenfaces constructed from the ORL database [ 63 ].

The first five Fisherfaces obtained from the ORL database [ 63 ].

• Independent component analysis (ICA) [ 35 ]: The ICA technique is used for the calculation of the basic vectors of a given space. The goal of this technique is to perform a linear transformation in order to reduce the statistical dependence between the different basic vectors, which allows the analysis of independent components. It is determined that they are not orthogonal to each other. In addition, the acquisition of images from different sources is sought in uncorrelated variables, which makes it possible to obtain greater efficiency, because ICA acquires images within statistically independent variables.
• Improvements of the PCA, LDA, and ICA techniques: To improve the linear subspace techniques, many types of research are developed. Z. Cui et al. [ 67 ] proposed a new spatial face region descriptor (SFRD) method to extract the face region, and to deal with noise variation. This method is described as follows: divide each face image in many spatial regions, and extract token-frequency (TF) features from each region by sum-pooling the reconstruction coefficients over the patches within each region. Finally, extract the SFRD for face images by applying a variant of the PCA technique called “whitened principal component analysis (WPCA)” to reduce the feature dimension and remove the noise in the leading eigenvectors. Besides, the authors in [ 68 ] proposed a variant of the LDA called probabilistic linear discriminant analysis (PLDA) to seek directions in space that have maximum discriminability, and are hence most suitable for both face recognition and frontal face recognition under varying pose.
• Gabor filters: Gabor filters are spatial sinusoids located by a Gaussian window that allows for extracting the features from images by selecting their frequency, orientation, and scale. To enhance the performance under unconstrained environments for face recognition, Gabor filters are transformed according to the shape and pose to extract the feature vectors of face image combined with the PCA in the work of [ 69 ]. The PCA is applied to the Gabor features to remove the redundancies and to get the best face images description. Finally, the cosine metric is used to evaluate the similarity.
• Frequency domain analysis [ 70 , 71 ]: Finally, the analysis techniques in the frequency domain offer a representation of the human face as a function of low-frequency components that present high energy. The discrete Fourier transform (DFT), discrete cosine transform (DCT), or discrete wavelet transform (DWT) techniques are independent of the data, and thus do not require training.
• Discrete wavelet transform (DWT): Another linear technique used for face recognition. In the work of [ 70 ], the authors used a two-dimensional discrete wavelet transform (2D-DWT) method for face recognition using a new patch strategy. A non-uniform patch strategy for the top-level’s low-frequency sub-band is proposed by using an integral projection technique for two top-level high-frequency sub-bands of 2D-DWT based on the average image of all training samples. This patch strategy is better for retaining the integrity of local information, and is more suitable to reflect the structure feature of the face image. When constructing the patching strategy using the testing and training samples, the decision is performed using the neighbor classifier. Many databases are used to evaluate this method, including Labeled Faces in Wild (LFW), Extended Yale B, Face Recognition Technology (FERET), and AR.
• Discrete cosine transform (DCT) [ 71 ] can be used for global and local face recognition systems. DCT is a transformation that represents a finite sequence of data as the sum of a series of cosine functions oscillating at different frequencies. This technique is widely used in face recognition systems [ 71 ], from audio and image compression to spectral methods for the numerical resolution of differential equations. The required steps to implement the DCT technique are presented as follows.

Owing to their limitations in managing the linearity in face recognition, the subspace or holistic techniques are not appropriate to represent the exact details of geometric varieties of the face images. Linear techniques offer a faithful description of face images when the data structures are linear. However, when the face images data structures are non-linear, many types of research use a function named “kernel” to construct a large space where the problem becomes linear. The required steps to implement the DCT technique are presented as Algorithm 1.

DCT Algorithm

,

4.2. Nonlinear Techniques

Kernel PCA Algorithm

. .

The performance of the KPCA technique depends on the choice of the kernel matrix K. The Gaussian or polynomial kernel are linear typically-used kernels. KPCA has been successfully used for novelty detection [ 72 ] or for speech recognition [ 62 ].

• Kernel linear discriminant analysis (KDA) [ 73 ]: the KLDA technique is a kernel extension of the linear LDA technique, in the same kernel extension of PCA. Arashloo et al. [ 73 ] proposed a nonlinear binary class-specific kernel discriminant analysis classifier (CS-KDA) based on the spectral regression kernel discriminant analysis. Other nonlinear techniques have also been used in the context of facial recognition:
• Gabor-KLDA [ 74 ].
• Evolutionary weighted principal component analysis (EWPCA) [ 75 ].
• Kernelized maximum average margin criterion (KMAMC), SVM, and kernel Fisher discriminant analysis (KFD) [ 76 ].
• Wavelet transform (WT), radon transform (RT), and cellular neural networks (CNN) [ 77 ].
• Joint transform correlator-based two-layer neural network [ 78 ].
• Kernel Fisher discriminant analysis (KFD) and KPCA [ 79 ].
• Locally linear embedding (LLE) and LDA [ 80 ].
• Nonlinear locality preserving with deep networks [ 81 ].
• Nonlinear DCT and kernel discriminative common vector (KDCV) [ 82 ].

4.3. Summary of Holistic Approaches

Table 2 summarizes the different subspace techniques discussed in this section, which are introduced to reduce the dimensionality and the complexity of the detection or recognition steps. Linear and non-linear techniques offer robust recognition under different lighting conditions and facial expressions. Although these techniques (linear and non-linear) allow a better reduction in dimensionality and improve the recognition rate, they are not invariant to translations and rotations compared with local techniques.

Subspace approaches. ICA, independent component analysis; DWT, discrete wavelet transform; FFT, fast Fourier transform; DCT, discrete cosine transform.

Author/Techniques Used		Databases	Matching	Limitation	Advantage	Result
		Linear Techniques
Seo et al. [ ]	LARK and PCA	LFW	L2 distance	Detection accuracy	Reducing computational complexity	85.10%
Annalakshmi et al. [ ]	ICA and LDA	LFW	Bayesian Classifier	Sensitivity	Good accuracy	88%
Annalakshmi et al. [ ]	PCA and LDA	LFW	Bayesian Classifier	Sensitivity	Specificity	59%
Hussain et al. [ ]	LQP and Gabor	FERET	Cosine similarity	Lot of discriminative information	Robust to illumination variations	99.2% 75.3%
		LFW
Gowda et al. [ ]	LPQ and LDA	MEPCO	SVM	Computation time	Good accuracy	99.13%
Z. Cui et al. [ ]	BoW	AR	ASM	Occlusions	Robust	99.43%
		ORL				99.50%
		FERET				82.30%
Khan et al. [ ]	PSO and DWT	CK	Euclidienne distance	Noise	Robust to illumination	98.60%
		MMI				95.50%
		JAFFE				98.80%
Huang et al. [ ]	2D-DWT	FERET	KNN	Pose	Frontal or near-frontal facial images	90.63% 97.10%
		LFW
Perlibakas and Vytautas [ ]	PCA and Gabor filter	FERET	Cosine metric	Precision	Pose	87.77%
Hafez et al. [ ]	Gabor filter and LDA	ORL	2DNCC	Pose	Good recognition performance	98.33%
		C. YaleB				99.33%
Sufyanu et al. [ ]	DCT	ORL	NCC	High memory	Controlled and uncontrolled databases	93.40%
		Yale
Shanbhag et al. [ ]	DWT and BPSO	_ _	_ _	Rotation	Significant reduction in the number of features	88.44%
Ghorbel et al. [ ]	Eigenfaces and DoG filter	FERET	Chi-square distance	Processing time	Reduce the representation	84.26%
Zhang et al. [ ]	PCA and FFT	YALE	SVM	Complexity	Discrimination	93.42%
Zhang et al. [ ]	PCA	YALE	SVM	Recognition rate	Reduce the dimensionality	84.21%
Fan et al. [ ]	RKPCA	MNIST ORL	RBF kernel	Complexity	Robust to sparse noises	_
Vinay et al. [ ]	ORB and KPCA	ORL	FLANN Matching	Processing time	Robust	87.30%
Vinay et al. [ ]	SURF and KPCA	ORL	FLANN Matching	Processing time	Reduce the dimensionality	80.34%
Vinay et al. [ ]	SIFT and KPCA	ORL	FLANN Matching	Low recognition rate	Complexity	69.20%
Lu et al. [ ]	KPCA and GDA	UMIST face	SVM	High error rate	Excellent performance	48%
Yang et al. [ ]	PCA and MSR	HELEN face	ESR	Complexity	Utilizes color, gradient, and regional information	98.00%
Yang et al. [ ]	LDA and MSR	FRGC	ESR	Low performances	Utilizes color, gradient, and regional information	90.75%
Ouanan et al. [ ]	FDDL	AR	CNN	Occlusion	Orientations, expressions	98.00%
Vankayalapati and Kyamakya [ ]	CNN	ORL	_ _	Poses	High recognition rate	95%
Devi et al. [ ]	2FNN	ORL	_ _	Complexity	Low error rate	98.5

5. Hybrid Approach

5.1. technique presentation.

The hybrid approaches are based on local and subspace features in order to use the benefits of both subspace and local techniques, which have the potential to offer better performance for face recognition systems.

• Gabor wavelet and linear discriminant analysis (GW-LDA) [ 91 ]: Fathima et al. [ 91 ] proposed a hybrid approach combining Gabor wavelet and linear discriminant analysis (HGWLDA) for face recognition. The grayscale face image is approximated and reduced in dimension. The authors have convolved the grayscale face image with a bank of Gabor filters with varying orientations and scales. After that, a subspace technique 2D-LDA is used to maximize the inter-class space and reduce the intra-class space. To classify and recognize the test face image, the k-nearest neighbour (k-NN) classifier is used. The recognition task is done by comparing the test face image feature with each of the training set features. The experimental results show the robustness of this approach in different lighting conditions.
• Over-complete LBP (OCLBP), LDA, and within class covariance normalization (WCCN): Barkan et al. [ 92 ] proposed a new representation of face image based over-complete LBP (OCLBP). This representation is a multi-scale modified version of the LBP technique. The LDA technique is performed to reduce the high dimensionality representations. Finally, the within class covariance normalization (WCCN) is the metric learning technique used for face recognition.
• Advanced correlation filters and Walsh LBP (WLBP): Juefei et al. [ 93 ] implemented a single-sample periocular-based alignment-robust face recognition technique based on high-dimensional Walsh LBP (WLBP). This technique utilizes only one sample per subject class and generates new face images under a wide range of 3D rotations using the 3D generic elastic model, which is both accurate and computationally inexpensive. The LFW database is used for evaluation, and the proposed method outperformed the state-of-the-art algorithms under four evaluation protocols with a high accuracy of 89.69%.
• Multi-sub-region-based correlation filter bank (MS-CFB): Yan et al. [ 94 ] propose an effective feature extraction technique for robust face recognition, named multi-sub-region-based correlation filter bank (MS-CFB). MS-CFB extracts the local features independently for each face sub-region. After that, the different face sub-regions are concatenated to give optimal overall correlation outputs. This technique reduces the complexity, achieves higher recognition rates, and provides a better feature representation for recognition compared with several state-of-the-art techniques on various public face databases.
• SIFT features, Fisher vectors, and PCA: Simonyan et al. [ 64 ] have developed a novel method for face recognition based on the SIFT descriptor and Fisher vectors. The authors propose a discriminative dimensionality reduction owing to the high dimensionality of the Fisher vectors. After that, these vectors are projected into a low dimensional subspace with a linear projection. The objective of this methodology is to describe the image based on dense SIFT features and Fisher vectors encoding to achieve high performance on the challenging LFW dataset in both restricted and unrestricted settings.

Flowchart of the proposed multimodal deep face representation (MM-DFR) technique [ 95 ]. CNN, convolutional neural network.

• PCA and ANFIS: Sharma et al. [ 96 ] propose an efficient pose-invariant face recognition system based on PCA technique and ANFIS classifier. The PCA technique is employed to extract the features of an image, and the ANFIS classifier is developed for identification under a variety of pose conditions. The performance of the proposed system based on PCA–ANFIS is better than ICA–ANFIS and LDA–ANFIS for the face recognition task. The ORL database is used for evaluation.
• DCT and PCA: Ojala et al. [ 97 ] develop a fast face recognition system based on DCT and PCA techniques. Genetic algorithm (GA) technique is used to extract facial features, which allows to remove irrelevant features and reduces the number of features. In addition, the DCT–PCA technique is used to extract the features and reduce the dimensionality. The minimum Euclidian distance (ED) as a measurement is used for the decision. Various face databases are used to demonstrate the effectiveness of this system.
• PCA, SIFT, and iterative closest point (ICP): Mian et al. [ 98 ] present a multimodal (2D and 3D) face recognition system based on hybrid matching to achieve efficiency and robustness to facial expressions. The Hotelling transform is performed to automatically correct the pose of a 3D face using its texture. After that, in order to form a rejection classifier, a novel 3D spherical face representation (SFR) in conjunction with the SIFT descriptor is used, which provide efficient recognition in the case of large galleries by eliminating a large number of candidates’ faces. A modified iterative closest point (ICP) algorithm is used for the decision. This system is less sensitive and robust to facial expressions, which achieved a 98.6% verification rate and 96.1% identification rate on the complete FRGC v2 database.
• PCA, local Gabor binary pattern histogram sequence (LGBPHS), and GABOR wavelets: Cho et al. [ 99 ] proposed a computationally efficient hybrid face recognition system that employs both holistic and local features. The PCA technique is used to reduce the dimensionality. After that, the local Gabor binary pattern histogram sequence (LGBPHS) technique is employed to realize the recognition stage, which proposed to reduce the complexity caused by the Gabor filters. The experimental results show a better recognition rate compared with the PCA and Gabor wavelet techniques under illumination variations. The Extended Yale Face Database B is used to demonstrate the effectiveness of this system.
• PCA and Fisher linear discriminant (FLD) [ 100 , 101 ]: Sing et al. [ 101 ] propose a novel hybrid technique for face representation and recognition, which exploits both local and subspace features. In order to extract the local features, the whole image is divided into a sub-regions, while the global features are extracted directly from the whole image. After that, PCA and Fisher linear discriminant (FLD) techniques are introduced on the fused feature vector to reduce the dimensionality. The CMU-PIE, FERET, and AR face databases are used for the evaluation.
• SPCA–KNN [ 102 ]: Kamencay et al. [ 102 ] develop a new face recognition method based on SIFT features, as well as PCA and KNN techniques. The Hessian–Laplace detector along with SPCA descriptor is performed to extract the local features. SPCA is introduced to identify the human face. KNN classifier is introduced to identify the closest human faces from the trained features. The results of the experiment have a recognition rate of 92% for the unsegmented ESSEX database and 96% for the segmented database (700 training images).

The proposed CNN–LSTM–ELM [ 103 ].

5.2. Summary of Hybrid Approaches

Table 3 summarizes the hybrid approaches that we presented in this section. Various techniques are introduced to improve the performance and the accuracy of recognition systems. The combination between the local approaches and the subspace approach provides robust recognition and reduction of dimensionality under different illumination conditions and facial expressions. Furthermore, these technologies are presented to be sensitive to noise, and invariant to translations and rotations.

Hybrid approaches. GW, Gabor wavelet; OCLBP, over-complete LBP; WCCN, within class covariance normalization; WLBP, Walsh LPB; ICP, iterative closest point; LGBPHS, local Gabor binary pattern histogram sequence; FLD, Fisher linear discriminant; SAE, stacked auto-encoder.

Author/Technique Used		Database	Matching	Limitation	Advantage	Result
Fathima et al. [ ]	GW-LDA	AT&T	k-NN	High processing time	Illumination invariant and reduce the dimensionality	88%
		FACES94				94.02%
		MITINDIA				88.12%
Barkan et al., [ ]	OCLBP, LDA, and WCCN	LFW	WCCN	_	Reduce the dimensionality	87.85%
Juefei et al. [ ]	ACF and WLBP	LFW		Complexity	Pose conditions	89.69%
Simonyan et al. [ ]	Fisher + SIFT	LFW	Mahalanobis matrix	Single feature type	Robust	87.47%
Sharma et al. [ ]	PCA–ANFIS	ORL	ANFIS	Sensitivity-specificity		96.66%
	ICA–ANFIS		ANFIS		Pose conditions	71.30%
	LDA–ANFIS		ANFIS			68%
Ojala et al. [ ]	DCT–PCA	ORL	Euclidian distance	Complexity	Reduce the dimensionality	92.62%
		UMIST				99.40%
		YALE				95.50%
Mian et al. [ ]	Hotelling transform, SIFT, and ICP	FRGC	ICP	Processing time	Facial expressions	99.74%
Cho et al. [ ]	PCA–LGBPHS	Extended Yale Face	Bhattacharyya distance	Illumination condition	Complexity	95%
	PCA–GABOR Wavelets
Sing et al. [ ]	PCA–FLD	CMU	SVM	Robustness	Pose, illumination, and expression	71.98%
		FERET				94.73%
		AR				68.65%
Kamencay et al. [ ]	SPCA-KNN	ESSEX	KNN	Processing time	Expression variation	96.80%
Sun et al. [ ]	CNN–LSTM–ELM	OPPORTUNITY	LSTM/ELM	High processing time	Automatically learn feature representations	90.60%
Ding et al. [ ]	CNNs and SAE	LFW	_ _	Complexity	High recognition rate	99%

6. Assessment of Face Recognition Approaches

In the last step of recognition, the face extracted from the background during the face detection step is compared with known faces stored in a specific database. To make the decision, several techniques of comparison are used. This section describes the most common techniques used to make the decision and comparison.

6.1. Measures of Similarity or Distances

• Peak-to-correlation energy (PCE) or peak-to-sidelobe ratio (PSR) [ 18 ]: The PCE was introduced in (8).

In general, the Euclidean distance between two points P = ( 1 ,   p 2 ,   … ,   p n ) and Q = ( q 1 ,   q 2 , …   ,   q n ) in the n-dimensional space would be defined by the following:

• Bhattacharyya distance [ 104 , 105 ]: The Bhattacharyya distance is a statistical measure that quantifies the similarity between two discrete or continuous probability distributions. This distance is particularly known for its low processing time and its low sensitivity to noise. For the probability distributions p and q defined on the same domain, the distance of Bhattacharyya is defined as follows: D B ( p ,   q ) = − l n ( B C ( p ,   q ) ) , (17) B C ( p ,   q ) = ∑ x ∈ X p ( x ) q ( x )   ( a ) ;   B C ( p ,   q ) = ∫ p ( x ) q ( x ) d x   ( b ) , (18) where B C is the Bhattacharyya coefficient, defined as Equation (18a) for discrete probability distributions and as Equation (18b) for continuous probability distributions. In both cases, 0 ≤ BC ≤ 1 and 0 ≤ DB ≤ ∞. In its simplest formulation, the Bhattacharyya distance between two classes that follow a normal distribution can be calculated from a mean ( μ ) and the variance ( σ 2 ): D B ( p ,   q ) = 1 4 l n ( 1 4 ( σ p 2 σ q 2 + σ q 2 σ p 2 + 2 ) ) + 1 4 ( ( μ p − μ q ) σ q 2 + σ p 2 ) . (19)
• Chi-squared distance [ 106 ]: The Chi-squared ( X 2 ) distance was weighted by the value of the samples, which allows knowing the same relevance for sample differences with few occurrences as those with multiple occurrences. To compare two histograms S 1 = ( u 1 , …   …   … . u m ) and S 2 = ( w 1 , …   …   … . w m ) , the Chi-squared ( X 2 ) distance can be defined as follows: ( X 2 ) = D ( S 1 , S 2 ) = 1 2 ∑ i = 1 m ( u i − w i ) 2 u i + w i . (20)

6.2. Classifiers

There are many face classification techniques in the literature that allow to select, from a few examples, the group or class to which the objects belong. Some of them are based on statistics, such as the Bayesian classifier and correlation [ 18 ], and so on, and others based on the regions that generate the different classes in the decision space, such as K-means [ 9 ], CNN [ 103 ], artificial neural networks (ANNs) [ 37 ], support vector machines (SVMs) [ 26 , 107 ], k-nearest neighbors (K-NNs), decision trees (DTs), and so on.

Optimal hyperplane, support vectors, and maximum margin.

There is an infinite number of hyperplanes capable of perfectly separating two classes, which implies to select a hyperplane that maximizes the minimal distance between the learning examples and the learning hyperplane (i.e., the distance between the support vectors and the hyperplane). This distance is called “margin”. The SVM classifier is used to calculate the optimal hyperplane that categorizes a set of labels training data in the correct class. The optimal hyperplane is solved as follows:

Given that x i are the training features vectors and y i are the corresponding set of l (1 or −1) labels. An SVM tries to find a hyperplane to distinguish the samples with the smallest errors. The classification function is obtained by calculating the distance between the input vector and the hyperplane.

where w and b are the parameters of the model. Shen et al. [ 108 ] proposed the Gabor filter to extract the face features and applied the SVM for classification. The proposed FaceNet method achieves a good record accuracy of 99.63% and 95.12% using the LFW YouTube Faces DB datasets, respectively.

• k-nearest neighbor (k-NN) [ 17 , 91 ]: k-NN is an indolent algorithm because, in training, it saves little information, and thus does not build models of difference, for example, decision trees.
• K-means [ 9 , 109 ]: It is called K-means because it represents each of the groups by the average (or weighted average) of its points, called the centroid. In the K-means algorithm, it is necessary to specify a priori the number of clusters k that one wishes to form in order to start the process.

Artificial neural network.

Various variants of neural networks have been developed in the last years, such as convolutional neural networks (CNN) [ 14 , 110 ] and recurrent neural networks (RNN) [ 111 ], which very effective for image detection and recognition tasks. CNNs are a very successful deep model and are used today in many applications [ 112 ]. From a structural point of view, CNNs are made up of three different types of layers: convolution layers, pooling layers, and fully-connected layers.

• Convolutional layer : sometimes called the feature extractor layer because features of the image are extracted within this layer. Convolution preserves the spatial relationship between pixels by learning image features using small squares of the input image. The input image is convoluted by employing a set of learnable neurons. This produces a feature map or activation map in the output image, after which the feature maps are fed as input data to the next convolutional layer. The convolutional layer also contains rectified linear unit (ReLU) activation to convert all negative value to zero. This makes it very computationally efficient, as few neurons are activated each time.
• - Average-pooling takes all the elements of the sub-matrix, calculates their average, and stores the value in the output matrix.
• - Max-pooling searches for the highest value found in the sub-matrix and saves it in the output matrix.
• Fully-connected layer : in this layer, the neurons have a complete connection to all the activations from the previous layers. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories by training.

Wen et al. [ 113 ] introduce a new supervision signal, called center loss, for the face recognition task in order to improve the discriminative power of the deeply learned features. Specifically, the proposed center loss function is trainable and easy to optimize in the CNNs. Several important face recognition benchmarks are used for evaluation including LFW, YTF, and MegaFace Challenge. Passalis and Tefas [ 114 ] propose a supervised codebook learning method for the bag-of-features representation able to learn face retrieval-oriented codebooks. This allows using significantly smaller codebooks enhancing both the retrieval time and storage requirements. Liu et al. [ 115 ] and Amato et al. [ 116 ] propose a deep face recognition technique under open-set protocol based on the CNN technique. A face dataset composed of 39,037 faces images belonging to 42 different identities is used to perform the experiments. Taigman et al. [ 117 ] present a system (DeepFace) able to outperform existing systems with only very minimal adaptation. It is trained on a large dataset of faces acquired from a population vastly different than the one used to construct the evaluation benchmarks. This technique achieves an accuracy of 97.35% on the LFW. Ma et al. [ 118 ] introduce a robust local binary pattern (LBP) guiding pooling (G-RLBP) mechanism to improve the recognition rates of the CNN models, which can successfully lower the noise impact. Koo et al. [ 119 ] propose a multimodal human recognition method that uses both the face and body and is based on a deep CNN. Cho et al. [ 120 ] propose a nighttime face detection method based on CNN technique for visible-light images. Koshy and Mahmood [ 121 ] develop deep architectures for face liveness detection that uses a combination of texture analysis and a CNN technique to classify the captured image as real or fake. Elmahmudi and Ugail [ 122 ] present the performance of machine learning for face recognition using partial faces and other manipulations of the face such as rotation and zooming, which we use as training and recognition cues. The experimental results on the tasks of face verification and face identification show that the model obtained by the proposed DNN training framework achieves 97.3% accuracy on the LFW database with low training complexity. Seibold et al. [ 123 ] proposed a morphing attack detection method based on DNNs. A fully automatic face image morphing pipeline with exchangeable components was used to generate morphing attacks, train neural networks based on these data, and analyze their accuracy. Yim et al. [ 124 ] propose a new deep architecture based on a novel type of multitask learning, which can achieve superior performance in rotating to a target-pose face image from an arbitrary pose and illumination image while preserving identity. Nguyen et al. [ 111 ] propose a new approach for detecting presentation attack face images to enhance the security level of a face recognition system. The objective of this study was the use of a very deep stacked CNN–RNN network to learn the discrimination features from a sequence of face images. Finally, Bajrami et al. [ 125 ] present experiment results with LDA and DNN for face recognition, while their efficiency and performance are tested on the LFW dataset. The experimental results show that the DNN method achieves better recognition accuracy, and the recognition time is much faster than that of the LDA method in large-scale datasets.

6.3. Databases Used

The most commonly used databases for face recognition systems under different conditions are Pointing Head Pose Image Database (PHPID) [ 126 ], Labeled Faces in Wild (LFW) [ 127 ], FERET [ 15 , 16 ], ORL, and Yale. The last are used for face recognition systems under different conditions, which provide information for supervised and unsupervised learning. Supervised learning is based on two training modules: image unrestricted training setting and image restricted training setting. For the first model, only “same” or “not same” binary labels are used in the training splits. For the second model, the identities of the person in each pair are provided in the training splits.

• LFW (Labeled Faces in the Wild) database was created in October 2007. It contains 13,333 images of 5749 subjects, with 1680 subjects with at least two images and the rest with a single image. These face images were taken on the Internet, pre-processed, and localized by the Viola–Jones detector with a resolution of 250 × 250 pixels. Most of them are in color, although there are also some in grayscale and presented in JPG format and organized by folders.
• FERET (Face Recognition Technology) database was created in 15 sessions in a semi-controlled environment between August 1993 and July 1996. It contains 1564 sets of images, with a total of 14,126 images. The duplicate series belong to subjects already present in the series of individual images, which were generally captured one day apart. Some images taken from the same subject vary overtime for a few years and can be used to treat facial changes that appear over time. The images have a depth of 24 bits, RGB, so they are color images, with a resolution of 512 × 768 pixels.
• AR face database was created by Aleix Martínez and Robert Benavente in the computer vision center (CVC) of the Autonomous University of Barcelona in June 1998. It contains more than 4000 images of 126 subjects, including 70 men and 56 women. They were taken at the CVC under a controlled environment. The images were taken frontally to the subjects, with different facial expressions and three different lighting conditions, as well as several accessories: scarves, glasses, or sunglasses. Two imaging sessions were performed with the same subjects, 14 days apart. These images are a resolution of 576 × 768 pixels and a depth of 24 bits, under the RGB RAW format.
• ORL Database of Faces was performed between April 1992 and April 1994 at the AT & T laboratory in Cambridge. It consists of a total of 10 images per subject, out of a total of 40 images. For some subjects, the images were taken at different times, with varying illumination and facial expressions: eyes open/closed, smiling/without a smile, as well as with or without glasses. The images were taken under a black homogeneous background, in a vertical position and frontally to the subject, with some small rotation. These are images with a resolution of 92 × 112 pixels in grayscale.
• Extended Yale Face B database contains 16,128 images of 640 × 480 grayscale of 28 individuals under 9 poses and 64 different lighting conditions. It also includes a set of images made with the face of individuals only.
• Pointing Head Pose Image Database (PHPID) is one of the most widely used for face recognition. It contains 2790 monocular face images of 15 persons with tilt angles from −90° to +90° and variations of pan. Every person has two series of 93 different poses (93 images). The face images were taken under different skin color and with or without glasses.

6.4. Comparison between Holistic, Local, and Hybrid Techniques

In this section, we present some advantages and disadvantages of holistic, local, and hybrid approaches to identifying faces during the last 20 years. DL approaches can be considered as a statistical approach (holistic method), because the training procedure scheme usually searches for statistical structures in the input patterns. Table 4 presents a brief summary of the three approaches.

General performance of face recognition approaches.

Approaches		Databases Used	Advantages	Disadvantages	Performances	Challenges Handled
		TDF, CF1999, LFW, FERET, CMU-PIE, AR, Yale B, PHPID, YaleB Extended, FRGC2.0, Face94.	]. , ].		, ].	], various lighting conditions[ ], facial expressions [ ], and low resolution.
			]. ].	].	]. ].	].
		LFW, FERET, MEPCO, AR, ORL, CK, MMI, JAFFE, C. Yale B, Yale, MNIST, ORL, UMIST face, HELEN face, FRGC.	, ]. , , , ].	].	]. ].	, ], scaling, facial expressions.
			, , ]. , , ].	]. , ].	]. , ].	, ], poses [ ], conditions, scaling, facial expressions.
		AT&T, FACES94, MITINDIA, LFW, ORL, UMIST, YALE, FRGC, Extended Yale, CMU, FERET, AR, ESSEX.	].	, , ].	]. ].	, ].

7. Discussion about Future Directions and Conclusions

7.1. discussion.

In the past decade, the face recognition system has become one of the most important biometric authentication methods. Many techniques are used to develop many face recognition systems based on facial information. Generally, the existing techniques can be classified into three approaches, depending on the type of desired features.

• Local approaches: use features in which the face described partially. For example, some system could consist of extracting local features such as the eyes, mouth, and nose. The features’ values are calculated from the lines or points that can be represented on the face image for the recognition step.
• Holistic approaches: use features that globally describe the complete face as a model, including the background (although it is desirable to occupy the smallest possible surface).
• Hybrid approaches: combine local and holistic approaches.

In particular, recognition methods performed on static images produce good results under different lighting and expression conditions. However, in most cases, only the face images are processed at the same size and scale. Many methods require numerous training images, which limits their use for real-time systems, where the response time is an important aspect.

The main purpose of techniques such as HOG, LBP, Gabor filters, BRIEF, SURF, and SIFT is to discover distinctive features, which can be divided into two parts: (1) local appearance-based techniques, which are used to extract local features when the face image is divided into small regions (including HOG, LBP, Gabor filters, and correlation filters); and (2) key-points-based techniques, which are used to detect the points of interest in the face image, after which features’ extraction is localized based on these points, including BRIEF, SURF, and SIFT. In the context of face recognition, local techniques only treat certain facial features, which make them very sensitive to facial expressions and occlusions [ 4 , 14 , 37 , 50 , 51 , 52 , 53 ]. The relative robustness is the main advantage of these feature-based local techniques. Additionally, they take into account the peculiarity of the face as a natural form to recognize a reduced number of parameters. Another advantage is that they have a high compaction capacity and a high comparison speed. The main disadvantages of these methods are the difficulty of automating the detection of facial features and the fact that the person responsible for the implementation of these systems must make an arbitrary decision on really important points.

Unlike the local approaches, holistic approaches are other methods used for face recognition, which treat the whole face image and do not require extracting face regions or features points (eyes, mouth, noses, and so on). The main function of these approaches is to represent the face image with a matrix of pixels. This matrix is often converted into feature vectors to facilitate their treatment. After that, the feature vectors are applied in a low-dimensional space. In fact, subspace techniques are sensitive to different variations (facial expressions, illumination, and different poses), which make them easy to implement. Many subspace techniques are implemented to represent faces such as Eigenface, Eigenfisher, PCA, and LDA, which can be divided into two categories: linear and non-linear techniques. The main advantage of holistic approaches is that they do not destroy image information by focusing only on regions or points of interest. However, this property represents a disadvantage because it assumes that all the pixels of the image have the same importance. As a result, these techniques are not only computationally expensive, but also require a high degree of correlation between the test and the training images. In addition, these approaches generally ignore local details, which means they are rarely used to identify faces.

Hybrid approaches are based on local and global features to exploit the benefits of both techniques. These approaches combine the two approaches described above into a single system to improve the performance and accuracy of recognition. The choice of the required method to be used must take into account the application in which it was applied. For example, in the face recognition systems that use very small images, methods based on local features are a bad choice. Another consideration in the algorithm selection process is the number of training examples needed. Finally, we can remember that the tendency is to develop hybrid methods that combine the advantages of local and holistic approaches, but these methods are very complex and require more processing time.

A notable limitation that we found in all the publications reviewed is methodological: despite that the 2D facial recognition has reached a significant level of maturity and a high success rate, it is not surprising that it continues to be one of the most active research areas in computer vision. Considering the results published to date, in the opinion of these authors, three particularly promising techniques for further development of this area stand out: (i) the development of 3D face recognition methods; (ii) the use of multimodal fusion methods of complementary data types, in particular those based on visible and infrared images; and (iii) the use of DL methods.

• Three-dimensional face recognition: In 2D image-based techniques, some features are lost owing to the 3D structure of the face. Lighting and pose variations are two major unresolved problems of 2D face recognition. Recently, 3D facial recognition for facial recognition has been widely studied by the scientific community to overcome unresolved problems in 2D facial recognition and to achieve significantly higher accuracy by measuring geometry of rigid features on the face. For this reason, several recent systems based on 3D data have been developed [ 3 , 93 , 95 , 128 , 129 ].
• Multimodal facial recognition: sensors have been developed in recent years with a proven ability to acquire not only two-dimensional texture information, but also facial shape, that is, three-dimensional information. For this reason, some recent studies have merged the two types of 2D and 3D information to take advantage of each of them and obtain a hybrid system that improves the recognition as the only modality [ 98 ].
• Deep learning (DL): a very broad concept, which means that it has no exact definition, but studies [ 14 , 110 , 111 , 112 , 113 , 121 , 130 , 131 ] agree that DL includes a set of algorithms that attempt to model high level abstractions, by modeling multiple processing layers. This field of research began in the 1980s and is a branch of automatic learning where algorithms are used in the formation of deep neural networks (DNN) to achieve greater accuracy than other classical techniques. In recent progress, a point has been reached where DL performs better than people in some tasks, for example, to recognize objects in images.

Finally, researchers have gone further by using multimodal and DL facial recognition systems.

7.2. Conclusions

Face recognition system is a popular study task in the field of image processing and computer vision, owing to its potentially enormous application as well as its theoretical value. This system is widely deployed in many real-world applications such as security, surveillance, homeland security, access control, image search, human-machine, and entertainment. However, these applications pose different challenges such as lighting conditions and facial expressions. This paper highlights the recent research on the 2D or 3D face recognition system, focusing mainly on approaches based on local, holistic (subspace), and hybrid features. A comparative study between these approaches in terms of processing time, complexity, discrimination, and robustness was carried out. We can conclude that local feature techniques are the best choice concerning discrimination, rotation, translation, complexity, and accuracy. We hope that this survey paper will further encourage researchers in this field to participate and pay more attention to the use of local techniques for face recognition systems.

Author Contributions

Y.K. highlights the recent research on the 2D or 3D face recognition system, focusing mainly on approaches based on local, holistic, and hybrid features. M.J., A.A.F. and M.A. supervised the research and helped in the revision processes. All authors have read and agreed to the published version of the manuscript.

The paper is co-financed by L@bISEN of ISEN Yncrea Ouest Brest, France, Dept Ai-DE, Team Vision-AD and by FSM University of Monastir, Tunisia with collaboration of the Ministry of Higher Education and Scientific Research of Tunisia. The context of the paper is the PhD project of Yassin Kortli.

Conflicts of Interest

The authors declare no conflict of interest.

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Rs-xception: a lightweight network for facial expression recognition, 1. introduction.

• Development of a lightweight model: The model integrates deep separable convolution and the SE module, which leads to a reduced number of parameters and computational load, making it suitable for resource-constrained environments while maintaining high performance.
• Model adaptability and scalability: RS-Xception demonstrates strong performance across three standard datasets and exhibits adaptability and generalization capabilities across a more complex dataset (RAF-DB).
• Technical validation: Transfer learning is employed to compare the model with other architectures on the same dataset, showcasing its superior performance. Furthermore, transfer learning is leveraged to enhance the accuracy of the model, highlighting its potential to enhance generalization capabilities.

2. Materials and Methods

2.1. depthwise separable convolution, 2.2. se-resnet, 2.3. rs-xception, 3.1. dataset details, 3.2. experimental results, 3.2.1. rs-xception performance on ck+, 3.2.2. rs-xception performance on fer2013, 3.2.3. rs-xception performance on bigfer2013, 3.3. ablation experiments, 4. discussion and conclusions, author contributions, data availability statement, conflicts of interest.

• Belmonte, R.; Allaert, B.; Tirilly, P.; Bilasco, I.M.; Djeraba, C.; Sebe, N. Impact of facial landmark localization on facial expression recognition. IEEE Trans. Affect. Comput. 2021 , 14 , 1267–1279. [ Google Scholar ] [ CrossRef ]
• Liang, L.; Lang, C.; Li, Y.; Feng, S.; Zhao, J. Fine-grained facial expression recognition in the wild. IEEE Trans. Inf. Forensics Secur. 2020 , 16 , 482–494. [ Google Scholar ] [ CrossRef ]
• Lim, C.; Inagaki, M.; Shinozaki, T.; Fujita, I. Analysis of convolutional neural networks reveals the computational properties essential for subcortical processing of facial expression. Sci. Rep. 2023 , 13 , 10908. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Shao, J.; Cheng, Q. E-FCNN for tiny facial expression recognition. Appl. Intell. 2021 , 51 , 549–559. [ Google Scholar ] [ CrossRef ]
• Nassif, A.B.; Darya, A.M.; Elnagar, A. Empirical evaluation of shallow and deep learning classifiers for Arabic sentiment analysis. Trans. Asian Low-Resour. Lang. Inf. Process. 2021 , 21 , 1–25. [ Google Scholar ]
• Kardakis, S.; Perikos, I.; Grivokostopoulou, F.; Hatzilygeroudis, I. Examining attention mechanisms in deep learning models for sentiment analysis. Appl. Sci. 2021 , 11 , 3883. [ Google Scholar ] [ CrossRef ]
• Saeed, S.; Shah, A.A.; Ehsan, M.K.; Amirzada, M.R.; Mahmood, A.; Mezgebo, T. Automated facial expression recognition framework using deep learning. J. Healthc. Eng. 2022 , 2022 , 5707930. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Talaat, F.M. Real-time facial emotion recognition system among children with autism based on deep learning and IoT. Neural Comput. Appl. 2023 , 35 , 12717–12728. [ Google Scholar ] [ CrossRef ]
• Helaly, R.; Messaoud, S.; Bouaafia, S.; Hajjaji, M.A.; Mtibaa, A. DTL-I-ResNet18: Facial emotion recognition based on deep transfer learning and improved ResNet18. Signal Image Video Process. 2023 , 17 , 2731–2744. [ Google Scholar ] [ CrossRef ]
• Bansal, M.M.; Sachdeva, M.; Mittal, A. Transfer learning for image classification using VGG19: Caltech-101 image data set. J. Ambient. Intell. Humaniz. Comput. 2023 , 14 , 3609–3620. [ Google Scholar ] [ CrossRef ]
• Wen, G.; Hou, Z.; Li, H.; Li, D.; Jiang, L.; Xun, E. Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn. Comput. 2017 , 9 , 597–610. [ Google Scholar ] [ CrossRef ]
• Ge, H.; Zhu, Z.; Dai, Y.; Wang, B.; Wu, X. Facial expression recognition based on deep learning. Comput. Methods Programs Biomed. 2022 , 215 , 106621. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Li, D.; Wen, G. MRMR-based ensemble pruning for facial expression recognition. Multimed. Tools Appl. 2018 , 77 , 15251–15272. [ Google Scholar ] [ CrossRef ]
• Hua, W.; Dai, F.; Huang, L.; Xiong, J.; Gui, G. HERO: Human emotions recognition for realizing intelligent Internet of Things. IEEE Access 2019 , 7 , 24321–24332. [ Google Scholar ] [ CrossRef ]
• Alonazi, M.; Alshahrani, H.J.; Alotaibi, F.A.; Maray, M.; Alghamdi, M.; Sayed, A. Automated Facial Emotion Recognition Using the Pelican Optimization Algorithm with a Deep Convolutional Neural Network. Electronics 2023 , 12 , 4608. [ Google Scholar ] [ CrossRef ]
• Arora, M.; Kumar, M.; Garg, N.K. Facial emotion recognition system based on PCA and gradient features. Natl. Acad. Sci. Lett. 2018 , 41 , 365–368. [ Google Scholar ] [ CrossRef ]
• Connie, T.; Al-Shabi, M.; Cheah, W.P.; Goh, M. Facial expression recognition using a hybrid CNN–SIFT aggregator. In Proceedings of the International Workshop on Multi-Disciplinary Trends in Artificial Intelligence, Gadong, Brunei Darussalam, 20–22 November 2017; pp. 139–149. [ Google Scholar ]
• Kaya, H.; Gürpınar, F.; Salah, A.A. Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis. Comput. 2017 , 65 , 66–75. [ Google Scholar ] [ CrossRef ]
• Zhao, L.; Niu, X.; Wang, L.; Niu, J.; Zhu, X.; Dai, Z. Stress detection via multimodal multi-temporal-scale fusion: A hybrid of deep learning and handcrafted feature approach. IEEE Sens. J. 2023 , 23 , 27817–27827. [ Google Scholar ] [ CrossRef ]
• Fan, X.; Tjahjadi, T. Fusing dynamic deep learned features and handcrafted features for facial expression recognition. J. Vis. Commun. Image Represent. 2019 , 65 , 102659. [ Google Scholar ] [ CrossRef ]
• Mehendale, N. Facial emotion recognition using convolutional neural networks (FERC). SN Appl. Sci. 2020 , 2 , 446. [ Google Scholar ] [ CrossRef ]
• Zeng, J.; Shan, S.; Chen, X. Facial expression recognition with inconsistently annotated datasets. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 222–237. [ Google Scholar ]
• Arora, M.; Kumar, M. AutoFER: PCA and PSO based automatic facial emotion recognition. Multimed. Tools Appl. 2021 , 80 , 3039–3049. [ Google Scholar ] [ CrossRef ]
• Debnath, T.; Reza, M.M.; Rahman, A.; Beheshti, A.; Band, S.S.; Alinejad-Rokny, H. Four-layer ConvNet to facial emotion recognition with minimal epochs and the significance of data diversity. Sci. Rep. 2022 , 12 , 6991. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• He, L.; He, L.; Peng, L. CFormerFaceNet: Efficient lightweight network merging a CNN and transformer for face recognition. Appl. 2023 , 13 , 6506. [ Google Scholar ] [ CrossRef ]
• Helaly, R.; Hajjaji, M.A.; M’Sahli, F.; Mtibaa, A. Deep convolution neural network implementation for emotion recognition system. In Proceedings of the 2020 20th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA), Monastir, Tunisia, 20–22 December 2020; pp. 261–265. [ Google Scholar ]
• Huang, Z.Y.; Chiang, C.C.; Chen, J.H.; Chen, Y.C.; Chung, H.L.; Cai, Y.P.; Hsu, H.C. A study on computer vision for facial emotion recognition. Sci. Rep. 2023 , 13 , 8425. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tan, C.; Sun, F.; Kong, T.; Zhang, W.; Yang, C.; Liu, C. A survey on deep transfer learning. In Artificial Neural Networks and Machine Learning—ICANN 2018: Proceedings of the 27th International Conference on Artificial Neural Networks, Rhodes, Greece, 4–7 October 2018 ; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; pp. 270–279. [ Google Scholar ]
• Sarkar, A.; Behera, P.R.; Shukla, J. Multi-source transfer learning for facial emotion recognition using multivariate correlation analysis. Sci. Rep. 2023 , 13 , 21004. [ Google Scholar ]
• Hoo, S.C.; Ibrahim, H.; Suandi, S.A. Convfacenext: Lightweight networks for face recognition. Mathematics 2022 , 10 , 3592. [ Google Scholar ] [ CrossRef ]
• Deng, Z.Y.; Chiang, H.H.; Kang, L.W.; Li, H.C. A lightweight deep learning model for real-time face recognition. IET Image Process. 2023 , 17 , 3869–3883. [ Google Scholar ] [ CrossRef ]
• Xie, S.; Hu, H.; Chen, Y. Facial expression recognition with two-branch disentangled generative adversarial network. IEEE Trans. Circuits Syst. Video Technol. 2020 , 31 , 2359–2371. [ Google Scholar ] [ CrossRef ]
• Kong, C.; Chen, B.; Li, H.; Wang, S.; Rocha, A.; Kwong, S. Detect and locate: Exposing face manipulation by semantic-and noise-level telltales. IEEE Trans. Inf. Forensics Secur. 2022 , 17 , 1741–1756. [ Google Scholar ] [ CrossRef ]
• Hardjadinata, H.; Oetama, R.S.; Prasetiawan, I. Facial expression recognition using xception and densenet architecture. In Proceedings of the 2021 6th International Conference on New Media Studies (CONMEDIA), Tangerang, Indonesia, 12–13 October 2021; pp. 60–65. [ Google Scholar ]
• Liang, X.; Liang, J.; Yin, T.; Tang, X. A lightweight method for face expression recognition based on improved MobileNetV3. IET Image Process. 2023 , 17 , 2375–2384. [ Google Scholar ] [ CrossRef ]
• Chollet, F. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1251–1258. [ Google Scholar ]
• Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7132–7141. [ Google Scholar ]
• He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [ Google Scholar ]
• Zhu, Q.; Zhuang, H.; Zhao, M.; Xu, S.; Meng, R. A study on expression recognition based on improved mobilenetV2 network. Sci. Rep. 2024 , 14 , 8121. [ Google Scholar ] [ CrossRef ]
• Rabea, M.; Ahmed, H.; Mahmoud, S.; Sayed, N. IdentiFace: A VGG Based Multimodal Facial Biometric System. arXiv 2024 , arXiv:2401.01227. [ Google Scholar ]
• Lucey, P.; Cohn, J.F.; Kanade, T.; Saragih, J.; Ambadar, Z.; Matthews, I. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA, 13–18 June 2010; pp. 94–101. [ Google Scholar ]
• Zhang, X.; Chen, Z.; Wei, Q. Research and application of facial expression recognition based on attention mechanism. In Proceedings of the 2021 IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC), Dalian, China, 14–16 April 2021; pp. 282–285. [ Google Scholar ]
• Zhang, H.; Su, W.; Yu, J.; Wang, Z. Identity–expression dual branch network for facial expression recognition. IEEE Trans. Cogn. Dev. Syst. 2020 , 13 , 898–911. [ Google Scholar ] [ CrossRef ]
• Sidhom, O.; Ghazouani, H.; Barhoumi, W. Three-phases hybrid feature selection for facial expression recognition. J. Supercomput. 2024 , 80 , 8094–8128. [ Google Scholar ] [ CrossRef ]
• Mukhopadhyay, M.; Dey, A.; Kahali, S. A deep-learning-based facial expression recognition method using textural features. Neural Comput. Appl. 2023 , 35 , 6499–6514. [ Google Scholar ] [ CrossRef ]
• Jiang, B.; Li, N.; Cui, X.; Liu, W.; Yu, Z.; Xie, Y. Research on Facial Expression Recognition Algorithm Based on Lightweight Transformer. Information 2024 , 15 , 321. [ Google Scholar ] [ CrossRef ]
• Khan, S.; Chen, L.; Yan, H. Co-clustering to reveal salient facial features for expression recognition. IEEE Trans. Affect. Comput. 2017 , 11 , 348–360. [ Google Scholar ] [ CrossRef ]
• Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv 2017 , arXiv:1704.04861. [ Google Scholar ]
• Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4700–4708. [ Google Scholar ]

Click here to enlarge figure

Model	Parameters	Depth	Flops	Time (ms) per Inference Step (CPU)
Xception	22.9 M	81	8900 M	109.4
VGG16	138.4 M	16	15,517 M	69.5
VGG19	143.7 M	19	19,682 M	84.8
ResNet50	25.6 M	107	4100 M	58.2
ResNet101	44.7 M	209	7900 M	89.6
ResNet152	60.4 M	311	11,000 M	127.4
InceptionV3	23.9 M	189	6000 M	42.2
InceptionResNetV2	55.9 M	449	17,000 M	130.2
MobileNet	4.3 M	55	600 M	22.6
MobileNetV2	3.5 M	105	312.86 M	25.9
DenseNet121	8.1 M	242	5690 M	77.1
Improved MobilenetV2 [ ]	3.26 M	25	\	\

Experiments	Accuracy	Precision	Recall	F1 Score
On CK+	97.13%	96.30%	96.20%	96.06%
On FER2013	69.02%	67.51%	67.55%	67.46%
On Bigfer2013	72.06%	71.86%	71.21%	71.38%
DTL on Bigfer2013	75.38%	75.86%	75.22%	74.88%
On RAF-DB	82.98%	82.06%	81.98%	81.93%

Approach	Dataset	Accuracy (%)
CBAM [ ]	CK+	95.1
IE-DBN [ ]	CK+	96.02
CCFS + SVM [ ]	CK+	96.05
Improved MobilenetV2 [ ]	CK+	95.96
Model by Sidhom O et al. [ ]	Fer2013	66.1
Self-Cure Net [ ]	Fer2013	66.17
Improved MobileViT [ ]	Fer2013	62.2
Improved MobilenetV2 [ ]	Fer2013	68.62
PSR [ ]	RAF-DB	80.78
E-FCNN [ ]	RAF-DB	78.31
TDGAN [ ]	RAF-DB	81.91

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

Liao, L.; Wu, S.; Song, C.; Fu, J. RS-Xception: A Lightweight Network for Facial Expression Recognition. Electronics 2024 , 13 , 3217. https://doi.org/10.3390/electronics13163217

Liao L, Wu S, Song C, Fu J. RS-Xception: A Lightweight Network for Facial Expression Recognition. Electronics . 2024; 13(16):3217. https://doi.org/10.3390/electronics13163217

Liao, Liefa, Shouluan Wu, Chao Song, and Jianglong Fu. 2024. "RS-Xception: A Lightweight Network for Facial Expression Recognition" Electronics 13, no. 16: 3217. https://doi.org/10.3390/electronics13163217

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Grab your spot at the free arXiv Accessibility Forum

Help | Advanced Search

Computer Science > Computer Vision and Pattern Recognition

Title: boosting unconstrained face recognition with targeted style adversary.

Abstract: While deep face recognition models have demonstrated remarkable performance, they often struggle on the inputs from domains beyond their training data. Recent attempts aim to expand the training set by relying on computationally expensive and inherently challenging image-space augmentation of image generation modules. In an orthogonal direction, we present a simple yet effective method to expand the training data by interpolating between instance-level feature statistics across labeled and unlabeled sets. Our method, dubbed Targeted Style Adversary (TSA), is motivated by two observations: (i) the input domain is reflected in feature statistics, and (ii) face recognition model performance is influenced by style information. Shifting towards an unlabeled style implicitly synthesizes challenging training instances. We devise a recognizability metric to constraint our framework to preserve the inherent identity-related information of labeled instances. The efficacy of our method is demonstrated through evaluations on unconstrained benchmarks, outperforming or being on par with its competitors while offering nearly a 70\% improvement in training speed and 40\% less memory consumption.

Subjects:	Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as:	[cs.CV]
	(or [cs.CV] for this version)
	Focus to learn more arXiv-issued DOI via DataCite

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Research on Face Recognition Algorithm of Intelligent Elderly Care Based on Machine Learning

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, index terms.

Computing methodologies

Artificial intelligence

Computer vision

Computer vision representations

Image representations

Recommendations

Age-invariant face recognition.

One of the challenges in automatic face recognition is to achieve temporal invariance. In other words, the goal is to come up with a representation and matching scheme that is robust to changes due to facial aging. Facial aging is a complex process that ...

Automatic face analysis system based on face recognition and facial physiognomy

An automatic face analysis system is proposed which uses face recognition and facial physiognomy. It first detects human's face, extracts its features, and classifies the shape of facial features. It will analyze the person's facial physiognomy and then ...

Intelligent 3D Face Recognition

Face recognition technology has been a focus both in academia and industry for the last couple of years because of its wide potential application and its importance to meet the security needs of today's world. This paper proposes a method to tackle an ...

Information

Published in.

Association for Computing Machinery

New York, NY, United States

Publication History

Permissions, check for updates.

• Research-article
• Refereed limited

Contributors

Other metrics, bibliometrics, article metrics.

• 0 Total Citations
• 1 Total Downloads
• Downloads (Last 12 months) 1
• Downloads (Last 6 weeks) 1

View Options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

View options.

View or Download as a PDF file.

View online with eReader .

HTML Format

View this article in HTML Format.

Share this Publication link

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

MIT Technology Review

• Newsletters

The Download: facial recognition for migrant children, and Japan’s megaquake

Mpox is spreading across African countries

• Rhiannon Williams archive page

This is today's edition of  The Download ,  our weekday newsletter that provides a daily dose of what's going on in the world of technology.

DHS plans to collect biometric data from migrant children “down to the infant”

The US Department of Homeland Security (DHS) plans to collect and analyze photos of the faces of migrant children at the border in a bid to improve facial recognition technology, MIT Technology Review can reveal. The technology has traditionally not been applied to children, largely because training data sets of real children’s faces are few and far between, and consist of either low-quality images drawn from the internet or small sample sizes with little diversity. Such limitations reflect the significant sensitivities regarding privacy and consent when it comes to minors. 

In practice, the new DHS plan could effectively solve that problem. But, beyond concerns about privacy, transparency, and accountability, some experts also worry about testing and developing new technologies using data from a population that has little recourse to provide—or withhold—consent. Read the full story .

—Eileen Guo

What Japan’s “megaquake” warning really tells us

On August 8, at 16:42 local time, a magnitude-7.1 earthquake shook southern Japan. The temblor, originating off the shores of mainland island of Kyūshū, was felt by nearly a million people across the region, and initially, the threat of a tsunami emerged. But only a diminutive wave swept ashore, buildings remained upright, and nobody died. The crisis was over as quickly as it began.

But then, something new happened. The Japan Meteorological Agency, a government organization, issued a ‘megaquake advisory’ for the first time. It was in part issued because it is possible that the magnitude-7.1 quake is a foreshock – a precursory quake – to a far larger one, a tsunami-making monster that could kill a quarter of a million people.

The good news, for now, is that scientists think it is very unlikely that that magnitude-7.1 quake is a prelude to a cataclysm. But the slim possibility remains that it was a foreshock to something considerably worse. Read the full story .

—Robin George Andrews

This story is part of MIT Technology Review Explains: our series helping you understand what's coming next. You can read more here.

The US government is still spending big on climate

Friday marks two years since the US signed the landmark Inflation Reduction Act (IRA) into law. In that time we’ve seen an influx of investment from the federal government and private businesses alike. 

The government has already spent hundreds of billions of dollars, and there’s much more to come. And this money is starting to make a big difference in the climate tech sector. But where is it all going? Read our story to find out .

—Casey Crownhart

This story is from The Spark, our weekly newsletter covering climate and energy technologies. Sign up to receive it in your inbox every Wednesday.

The must-reads

I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology.

1 Mpox is spreading rapidly across African countries The World Health Organization has declared it a global health emergency for the second time in two years. ( NYT $) + Cases and deaths are rising across east and central African countries. ( Vox ) + This type of mpox, known as Clade 1, is far deadlier than the previous version. ( BBC )

2 A brain implant helped a man with ALS to speak again Years after the disease robbed him of that ability. ( Reuters ) + An ALS patient set a record for communicating via a brain implant. ( MIT Technology Review ) 3 X’s AI image generator appears to have few filters It’ll generate pictures of Barack Obama doing cocaine, for example. ( NY Mag $) + It does, however, refuse to generate fully nude images. ( The Guardian ) + Text-to-image AI models can be tricked into generating disturbing images. ( MIT Technology Review )

4 Big Tech’s energy usage is skyrocketing But how huge firms disclose their emissions is a bone of contention. ( FT $) + Google, Amazon and the problem with Big Tech’s climate claims. ( MIT Technology Review )

5 Meta has shut down a major misinformation tracking tool Less than three months before the US election. ( NPR )+ Meta’s justification? CrowdTangle was too difficult to maintain. ( Bloomberg $)

6 Apple has started work on a tabletop robot Its former car team has pivoted to building a smart home command center. ( Bloomberg $)

7 Climate change is a gift to harmful invasive plants Sleeper species can thrive in warmer temperatures. ( Economist $)

8 The problem with slapping logos on prostheses Some wearers say it feels more like a product than a part of their body. ( The Atlantic $) + These prosthetics break the mold with third thumbs, spikes, and superhero skins. ( MIT Technology Review )

9 Mark Zuckerberg has commissioned a giant sculpture of his wife He’s continuing in the Roman tradition, apparently. ( The Guardian )

10 ChatGPT randomly started chatting to English users in Welsh 🏴󠁧󠁢󠁷󠁬󠁳󠁿 O diar! (That’s Welsh for ‘oh dear.’) ( FT $)

Quote of the day

“The world that exists today is the product of monopolistic conduct. That world is changing.”

—Judge James Donato, who is presiding over the Epic v Google legal case, tells Google’s lawyer to expect harsh punishment when he makes his final ruling in the next few weeks, the Verge reports.

The big story

The search for extraterrestrial life is targeting Jupiter’s icy moon Europa

February 2024

Europa, Jupiter’s fourth-largest moon, is nothing like ours. Its surface is a vast saltwater ocean, encased in a blanket of cracked ice, one that seems to occasionally break open and spew watery plumes into the moon’s thin atmosphere.

For these reasons, Europa captivates planetary scientists. All that water and energy—and hints of elements essential for building organic molecules —point to another extraordinary possibility. Jupiter’s big, bright moon could host life.

And they may eventually get some answers. Later this year, NASA plans to launch Europa Clipper, the largest-­ever craft designed to visit another planet. The $5 billion mission, scheduled to reach Jupiter in 2030, will spend four years analyzing this moon to determine whether it could support life. Read the full story .

—Stephen Ornes

We can still have nice things

A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line or tweet 'em at me .)

The Download

The Download: the future of music AI, and climate tech funding

Plus: Lego bricks are making science more accessible

• Charlotte Jee archive page

The Download: how we’re using AI, and Trump’s campaign hack

Plus: the problem with the meat industry

The Download: video-generating AI, and Meta’s voice cloning watermarks

Plus: Nvidia's star is on the rise

The Download: AI video games’ research potential, and US government website redesigns

Plus: rump-Biden debate conspiracies are already all over the internet.

Stay connected

Get the latest updates from mit technology review.

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at [email protected] with a list of newsletters you’d like to receive.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• NEWS FEATURE
• 18 November 2020

The ethical questions that haunt facial-recognition research

• Richard Van Noorden

You can also search for this author in PubMed   Google Scholar

A collage of images from the MegaFace data set , which scraped online photos. Images are obscured to protect people’s privacy. Credit: Adam Harvey/megapixels.cc based on the MegaFace data set by Ira Kemelmacher-Shlizerman et al. based on the Yahoo Flickr Creative Commons 100 Million data set and licensed under Creative Commons Attribution (CC BY) licences

In September 2019, four researchers wrote to the publisher Wiley to “respectfully ask” that it immediately retract a scientific paper. The study, published in 2018, had trained algorithms to distinguish faces of Uyghur people, a predominantly Muslim minority ethnic group in China, from those of Korean and Tibetan ethnicity 1 .

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

Nature 587 , 354-358 (2020)

doi: https://doi.org/10.1038/d41586-020-03187-3

Wang, C., Zhang, Q., Liu, W., Liu, Y. & Miao, L. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 9 , e1278 (2019).

Article   Google Scholar  

Stewart, R., Andriluka, M. & Ng, A. Y. in Proc. 2016 IEEE Conf. on Computer Vision and Pattern Recognition 2325–2333 (IEEE, 2016).

Ristani, E., Solera, F., Zou, R. S., Cucchiara, R. & Tomasi, C. Preprint at https://arxiv.org/abs/1609.01775 (2016).

Nech, A. & Kemelmacher-Shlizerman, I. in Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition 3406–3415 (IEEE, 2017).

Guo, Y., Zhang, L., Hu., Y., He., X. & Gao, J. in Computer Vision — ECCV 2016 (eds Leibe, B., Matas, J., Sebe, N. & Welling, M.) https://doi.org/10.1007/978-3-319-46487-9_6 (Springer, 2016).

Google Scholar  

Jasserand, C. in Data Protection and Privacy: The Internet of Bodie s (eds Leenes, R., van Brakel, R., Gutwirth, S. & de Hert, P.) Ch. 7 (Hart, 2018).

Moreau, Y. Nature 576 , 36–38 (2019).

Article   PubMed   Google Scholar  

Zhang, D. et al. Int. J. Legal Med . https://doi.org/10.1007/s00414-019-02049-6 (2019).

Pan, X. et al. Int. J. Legal Med. 134 , 2079 (2020).

Wu, X. & Xhang, X. Preprint at https://arxiv.org/abs/1611.04135 (2016).

Hashemi, M. & Hall, M. J. Big Data 7 , 2 (2020).

Download references

Reprints and permissions

Supplementary Information

• Spreadsheet of Nature survey summary results

Related Articles

• Machine learning
• Computer science

Chatbots in science: What can ChatGPT do for you?

Career Column 14 AUG 24

Has your paper been used to train an AI model? Almost certainly

News 14 AUG 24

Weather and climate predicted accurately — without using a supercomputer

News & Views 13 AUG 24

How I’m looking to medicine’s past to heal hurt and support peace in the Middle East

World View 15 AUG 24

Maxine Singer obituary: biologist who shaped genetic engineering and fought discrimination

Obituary 02 AUG 24

Guidelines on lab-grown embryo models are strong enough to meet ethical standards — and will build trust in science

World View 30 JUL 24

Effort to ‘Trump-proof’ US science grows, but will it succeed?

News 26 JUL 24

Exclusive: the Trump administration demoted this climate scientist — now she wants reform

News 24 JUL 24

Boost French research by increasing freedom for scientists and universities

World View 23 JUL 24

Postdoctoral Fellow in Epigenetics/RNA Biology in the Lab of Yvonne Fondufe-Mittendorf

Van Andel Institute’s (VAI) Professor Yvonne Fondufe-Mittendorf, Ph.D. is hiring a Postdoctoral Fellow to join the lab and carry out an independent...

Grand Rapids, Michigan

Van Andel Institute

Faculty Positions in Center of Bioelectronic Medicine, School of Life Sciences, Westlake University

SLS invites applications for multiple tenure-track/tenured faculty positions at all academic ranks.

Hangzhou, Zhejiang, China

School of Life Sciences, Westlake University

Faculty Positions, Aging and Neurodegeneration, Westlake Laboratory of Life Sciences and Biomedicine

Applicants with expertise in aging and neurodegeneration and related areas are particularly encouraged to apply.

Westlake Laboratory of Life Sciences and Biomedicine (WLLSB)

Faculty Positions in Chemical Biology, Westlake University

We are seeking outstanding scientists to lead vigorous independent research programs focusing on all aspects of chemical biology including...

Assistant Professor Position in Genomics

The Lewis-Sigler Institute at Princeton University invites applications for a tenure-track faculty position in Genomics.

Princeton University, Princeton, New Jersey, US

The Lewis-Sigler Institute for Integrative Genomics at Princeton University

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies