We are born with brains hard-wired to recognize human faces by interpreting what our eyes see. Machine facial recognition similarly relies on computer analysis of data collected by optical or infrared sensors. But despite half a century of effort, we have yet to learn how to teach computers to recognize faces as well as we can.
Superficially, face recognition may seem a simple matter of pattern recognition, but in practice faces pose particularly complex problems for computer recognition. Faces are three-dimensional, so their appearance in two-dimensional images or detectors depends on factors including viewing angle, illumination angle, and the nature of the illumination. Faces change with people's expressions; you look different when you smile, frown, or stick out your tongue. Your appearance also depends on your hair, eyeglasses, cosmetics, and weight.
Yet the ease of observing faces makes them very attractive for biometric identification. The Federal Bureau of Investigation and the Department of Homeland Security are developing facial recognition systems for law enforcement and security. Private companies are working on facial recognition for private security systems and labeling of photos on social networks. Applications for Android and Apple phones can unlock them when you wake the phone with your face visible to the camera—locking the phone if it's lost or stolen.
Government and civilian applications
Law enforcement agencies long ago recognized that faces were easier to recognize when photos were taken in standard formats. That's why old-fashioned police mug shots included front-on and profile views. Government agencies have developed rigid standards for official identification photos that require removing eyeglasses, assuming a neutral expression, and looking directly at the camera. If you can't focus on the camera without your glasses, you may look like a derelict, but such standard photos are considered the best reference for identification.
The photo repository used for facial recognition in the FBI's billion-dollar Next Generation Identification program consists entirely of standard criminal mug shots. A pilot program began in February 2012 with a repository containing some 13 million face-on criminal mug shots supplied by law enforcement agencies. The system electronically compares submitted photos to those in the repository and produces a computer-ranked list of possible matches. The challenge is to find matches for photos not in standard format collected from varied sources ranging from surveillance cameras to social media, and the rankings are intended for police use in investigations, not as proof of identity.
Goals of the pilot program were "to test the facial recognition processes, resolve policy and processing issues, solidify privacy protection procedures, and address user concerns," FBI official Jerome Pender told a Senate Judiciary subcommittee in 2012.1 Submitted photos were not added to the photo repository because the people in them were not definitely identified, and that practice will continue when the system goes on-line with a larger database at the end of the summer of 2014.
Homeland Security has more ambitious goals for its Biometric Optical Surveillance System, including computerized identification of people shown in video recordings of crowds.2 However, firm identification is a far more difficult task, and crowd photos pose challenges beyond viewing angle and illumination. Cameras generally are so far from the crowd that individual faces are recorded with low resolution, and people may be wearing caps, hoods, or sunglasses.
Civilian face-recognition applications generally have different requirements, and they usually check people rather than pictures. A biometric system for access control must accurately recognize people in a limited data base without granting access to anyone else. Unlocking a smartphone requires reliably identifying one person. The Microsoft (Bellevue, WA) Kinect uses its motion sensors to create a 3D profile of a user's face to identify who is playing a game, so accuracy is not crucial.
Face recognition on social media has to work with photographs, not people, but can draw on non-image data, such as Facebook friend status, to limit the number of people likely to be in a picture. Errors usually are not considered critical, but in some circumstances false positives can be embarrassing.
Varied technologies
Two-dimensional computer face recognition starts with finding the face in the picture. Then the computer must locate and digitize the positions of key features, such as the tip of the nose, the corners of the eyes and mouth, and the tip of the chin, as shown in Figure 1. Analyzing that information tells the computer if the photo is taken face-on. If it isn't, the computer needs to estimate the viewing angle and compensate for that angle in its digital representation of the face. Then it compares the digitized positions of facial features in the picture with those of a library of identified faces taken under controlled conditions, such as mug shots.
The development of inexpensive three-dimensional scanning optics makes it practical to directly collect data on the 3D shape of a face with the person present, either supplementing or replacing conventional 3D imaging.3 The added information can overcome the limits of 2D face recognition for some applications. It also can verify that authorized persons are present in applications such as building security or unlocking electronic devices, where criminals might try showing a photo of an authorized person to gain access. Developers are also looking at 3D stereo imaging to enhance performance of facial recognition.4
Computer processing techniques are crucial to performance. Efforts to develop self-learning computer systems such as neural networks for face recognition date back to the 1990s.5 The hope has been that powerful computers could recognize patterns that are more effective than standard facial features, but success has been elusive. Now Yaniv Taigman of Facebook AI Research (Menlo Park, CA) and colleagues claim their "DeepFace" program has used a set of algorithms called "deep learning" to tackle the facial recognition in a set of four million photos labeled with identities but not taken under standard conditions.6
They used explicit 3D modeling of faces to their alignment in the photos. They combined that with a deep neural net architecture that analyzed photos in the database and developed its own face representation technique that could be applied to other photo collections. With no facial alignment processing, they achieved 87.9% correct recognition in a set of test photos "from the wild"—that is, without intentional alignment. Two-dimensional alignment raised recognition to 94.3%, and 3D alignment further improved it to 97.35%, approaching the 97.5% rate for humans. The system takes about 0.33 second per image and presumably is a step toward identifying people shown on social networks, where accuracy requirements may not be as demanding as in security systems.
New optical technology
New optical technology also can improve facial recognition. Although surveillance cameras have helped crack many crimes, the quality of single frames typically is limited by sensor resolution, the small size of distant faces, and the viewing angle. The fewer pixels in the face image, usually measured by the number of pixels between the eyes, the poorer face recognition becomes. However, combining a series of low-resolution images of a face in the same pose can reconstruct a super-resolution image with higher resolution than any single frame. Tests at the Army Research Laboratory (Adelphi, MD) showed that super-resolution imaging, as visible in Figure 2 for a series of four and eight frames, provided considerable improvement in resolution and also improved facial recognition.7
Infrared observations may offer a way to avoid problems in visual face recognition arising from skin color and problematic or uneven illumination. Long-wave infrared sensors can record thermal radiation in the 10 micrometer band from the body rather than reflected light, early studies found that simple intensity-based images often lack the contrast and spatial resolution needed for accurate face recognition. Recent experiments at the Army Research Laboratory showed that recording polarization of the infrared light enhanced facial resolution.8
A recent review by Reza Ghiass of Laval University (Quebec City, Canada) and colleagues warns that thermal infrared face recognition can be limited by obstructions such as eyeglasses, as well as body-temperature changes. But they say that thermal images also may offer new identification criteria, such as visualizing networks of blood vessels and other data not revealed in the visible.9
Outlook
As face-recognition technology requirements improve, developers need to look carefully at how performance requirements vary among applications. False negatives are a big issue for access control; people don't want systems to lock them out of their workplace or personal electronics if they forget to shave or have puffy eyes from allergies. False positives can be problematic in law enforcement. And systems that can be spoofed or compromised are a peril for high-security and financial systems. Social media may seem innocuous, until they misidentify a class-action lawyer in a compromising photo.
The growth of face recognition also is raising concerns from groups such as the Electronic Frontier Foundation (San Francisco, CA) about the erosion of privacy. Privacy advocates worry that a high-tech security state will erode personal freedom, and enable abuses from stacking and harassment to over-aggressive marketing. Google (Mountain View, CA) heeded those concerns when it chose not to approve face recognition apps for Google Glass, although unauthorized versions are offered for jailbroken Glass.
The issues are complex, and the technology is still evolving. Remember that it was human eyes—not a computer—that identified the alleged 2013 Boston Marathon bombers.
References
1. J.M. Pender, "What facial recognition technology means for privacy and civil liberties," http://www.fbi.gov/news/testimony/what-facial-recognition-technology-means-for-privacy-and-civil-liberties.
2. C. Savage, "Facial scanning is making gains in surveillance," New York Times, Aug. 21, 2013, http://www.nytimes.com/2013/08/21/us/facial-scanning-is-making-gains-in-surveillance.html.
3. L. Zhang et al., "3D face recognition based on multiple keypoint descriptors and sparse representation," PLoS ONE V. 9, e100120, (June 2014); doi:10.1371/journal.pone.0100120.
4. Y. Zheng and E. Blasch, "The advantages of stereo vision in a face recognition system," Proc. SPIE 9091, 90910Y; doi: 10.1117/12.2049985.
5. S. Lawrence et al., "Face recognition: A convolutional neural-network approach," IEEE Trans. Neural Netw. 8, 98 (Jan. 1997).
6. Y. Taigman, "DeepFace: Closing the gap to human-level performance in face verification," https://www.facebook.com/publications/546316888800776/.
7. S. Hu et al., "Face recognition performance with superresolution," Appl. Opt. 51, 4250 (June 20, 2012).
8. K. Gurton et al., "Advanced facial recognition for thermal imagery using polarimetric imaging," Opt. Lett. 39, 3857–3859 (July 1, 2014); doi.org/10.1364/OL.39.003857.
9. R. S. Ghiass et al., "Infrared face recognition: a comprehensive review of methodologies and databases," http://arxiv.org/abs/1401.8261.
