The capabilities of face recognition algorithms have skyrocketed the last five years. With substantial gains comes substantial responsibility – and so now we must reexamine the role face recognition can play in the future of biometrics and identity technology, and closely consider potential limits that these technologies may never overcome.

Unlike fingerprint or iris recognition, facial appearances are public. In turn, facial images have become widely available – trillions of them scattered across the Internet, plus the endless hours of faces embedded in video streaming services. A nearly endless source of facial imagery exists in every corner of the digital world. Even relative to other non-biometric computer vision or machine learning classification tasks, the availability of facial appearance data vastly exceeds data available to train other classification algorithms. In many ways, due to the overwhelming legacy of widely distributed facial imagery, it’s likely that facial recognition will continue down this path to become the most accurate machine learning technology in the world.

In this article, we’ll explore classic considerations for facial biometric traits and discuss how they align with modern advances and use cases. Through this examination, we have identified the following “pros” and “cons” of face recognition technology:




Highly Accurate Challenges with Familial Similarity
Convenient Challenges with Cosmetic Modification
Default Method for Humans Risks of Spoofing


Though historically face recognition has been the least accurate of the “Big Three” biometrics (facial appearance, fingerprint ridge patterns, and iris texture), in recent years face recognition technology capabilities have surged to become the world’s most accurate biometric technology.




Comparing Biometric Accuracy (Face, Fingerprint, Iris)

Comparing biometric accuracy

Face: NIST FRVT Ongoing, Oct 2022 – Visa Dataset
Iris: NIST IREX IX report, April 2018 – Table D1, Single Eye – NOTE: the current NIST IREX Ongoing does not include 1:1 measurements
Fingerprint: NIST PFT, Oct 2022 – MINEX III Dataset, Single Finger


While these error rates are not a perfect apples-to-apples comparison, error rates produced by face recognition algorithms are significantly lower than those measured by any single finger or iris sample. 

Fingerprint has long since been considered the gold-standard for biometric accuracy (outside of invasive methods like DNA), so many are often surprised to hear that face recognition error rates are often substantially lower.

A closer look at the last five years of accuracy testing reveals an exponential decrease in error rates.

Evolution of Face Recognition Accuracy, 2017 – 2023

Evolution of face recognition accuracy

Over the past 5 years, when operating at a False Match Rate of 1 in 1 Million, the False

Non-Match Rate (FNMR) error has decreased by over 50x


How did facial recognition become so accurate? 


Public biometric

The exponential accuracy progression of face recognition technology is largely due to the following two factors: 

  • Facial appearance has been the primary biometric trait throughout civilized human existence; and
  • Facial appearance is not private.

In terms of the privacy of facial imagery, historically speaking, our facial appearance is the single least private piece of information about ourselves. When meeting a stranger, people may exhibit reluctance to provide simple information such as their name. Yet, they will provide their facial appearance immediately to countless strangers in day-to-day interactions. When in a public setting, the right to privacy of facial appearance simply does not and cannot reasonably exist. In fact, in some of the most liberal countries on the planet (e.g,. France, Denmark) it is illegal for a person to conceal their face in public. 

A tangible example of our lack of facial privacy in public settings can be seen when attending professional and/or televised sporting events. People pay hefty sums of money to attend games, knowing that their facial appearance may be broadcast on TV to audiences of millions.



Facial Privacy in Public Settings Examples

sleeping man example

Fan Example 1 (Source)

surprised man example

Fan Example 2 (Source)

In Fan Example 1, a fan was sleeping at a nationally televised baseball game. TV broadcasters noticed the fan sleeping and aired it nationally while discussing him at length. The fan attempted to sue the broadcaster (ESPN), but ultimately couldn’t find valid legal grounds and his suit was dismissed. 

In Fan Example 2, a fan was stunned after his football team lost a game in the final seconds. The fan was briefly aired on TV, but it was enough for Internet memesters to turn the image into a meme that went viral. The fan never consented to his image being widely distributed across the Internet, but at the same time he had no recourse to prevent this distribution. 

In both cases, the fans never consented to be broadcast on national television, or to further be spread online. Nor does any other fan when shown in the background of such televised events. Regardless, the lack of inherent privacy to one’s facial appearance means there is no recourse to prevent sharing facial appearance – aside from not appearing in public. 

Though precedent does not protect privacy of facial appearance, privacy concerns still exist regarding facial appearance and facial recognition. In fact, emerging technologies like highly-accurate automated facial recognition can create significant privacy issues due to the highly public nature of facial appearance. 


Deep Learning and Convolutional Neural Networks 

In the last decade, a technological revolution rapidly advanced the world of computer vision and machine learning. Inspired by the human visual processing system, we call this technology, deep learning via convolutional neural networks. This technique applies highly-tuned kernel convolutions against matrices of image pixels to yield powerful feature representations. 

The number of parameters in these models usually falls in the order of hundreds of thousands to hundreds of millions. To learn the optimal model parameters requires yet another order of magnitude more imagery than the number of parameters, as dictated by the “rule of 10” phenomenon in machine learning.

When provided with sufficient data, knowledge about the classification problem domain, machine learning methods, and GPU/supercomputing hardware, algorithm models can be trained that deliver truly stunning accuracy. In some cases, such as face recognition, these models can significantly surpass human performance.



NIST Face Recognition Algorithm Matching Accuracy

NIST face recognition algorithm matching accuracy
The plot above is from the NIST FRVT vendor scorecard for It shows the matching accuracy of the face recognition algorithm on the same pairs of images that expert facial examiners were given three months to study and determine whether or not they were a match. Expert examiners achieved roughly 95% accuracy on the task, while several automated algorithms can now achieve perfect 100% accuracy in a matter of seconds. Shown above, the automated algorithm achieves perfect accuracy, and a significant separation in facial similarity score between the genuine comparisons of the same persons (the first 12 plotted values with very high facial similarity scores) and the impostor comparisons of different persons (the next 8 plotted values with low facial similarity scores). 

Face recognition has benefited tremendously from the combination of expansive data and incredibly powerful deep learning toolkits, and we expect error rates should continue to decline for years to come. 

However, as discussed previously, face recognition will also struggle to cross some hard boundaries. Eventually the “Moore’s law”-like effect achieved by face recognition algorithms these past 5 years may taper out. 



In addition to accuracy, another chief benefit of face recognition technology is convenience. 

Using a face recognition system typically requires little effort. When paired with continuous authentication and real-time screening, they are often completely frictionless. When used as a method of access control, face recognition often requires less user effort and cooperation than fingerprint or iris recognition. 

Due to such convenience, face recognition is the only primary biometric trait that can be used successfully in a fully unconstrained manner. Indeed, accuracies on highly unconstrained benchmarks, like IARPA Janus, have gone from extremely low a decade ago, to now approaching the accuracy of other biometric traits operating in fully cooperative settings in a relatively short period of time. 

Examples of Unconstrained Face Images in the IARPA Janus Dataset

Examples of Unconstrained Face Images
Image source: B. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen, P. Grother, A. Mah, and A.K. Jain,  “Pushing the Frontiers of Unconstrained Face Detection and Recognition: IARPA Janus Benchmark A.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Convenience and ubiquity do come with a cost. While facial appearance is overwhelmingly public information, faces can also be linked to more sensitive private information. This consideration becomes especially important as face recognition accuracy and speed continue to advance.


Default Method for Humans

The most common method humans use to identify another person in day-to-day life is face recognition. Human face perception is such an important task – we have a large, dedicated region of the brain called the fusiform face area whose sole function is human face identification. 

It is natural for automated systems to prioritize compatibility with our manual, legacy methods. For this reason – inherent human understanding – face recognition is often preferable to other biometric traits.

Perhaps there is no better example of the human dependence of facial recognition than the Thatcher effect:

Example of the Thatcher effect

Example of the Thatcher effect 1
Example of the Thatcher effect 2
The two images above are the same, with the exception that the first image is flipped upside down. We can hardly perceive the manipulations in the inverted image due to the fact that inverted faces are not processed by the fusiform face area of the brain. The second image is immediately recognized as being altered and not an accurate face image.

One of the biggest reasons for society to continue investing in the use of properly developed automated face recognition technology is our inherent reliance on the facial biometric in our day-to-day lives. 


Limits of Face Recognition

While face recognition technology continues to deliver unprecedented convenience and accuracy, we must also address some fundamental limitations and challenges. 

Identical Twins

One of the more fundamental limits on face recognition is the challenge of identical twins and to a lesser extent familial relations. Though only accounting for roughly 0.3% of the population, identical twins in particular create a substantial challenge for automated face recognition algorithms. The reason is fairly obvious and illustrated by the following examples of pairs of identical twins:

two children example
two astronauts example
Over time the challenge of identical twins eases slightly due to differences in environmental factors. But this does not fully remove the challenge.  

There are a few algorithmic approaches that can be applied for identical twins. One option could be a focus on Level III facial features like freckles and moles, which are indeed unique between identical twins. 

A more realistic approach for operating facial recognition technology in the presence of identical twins could entail administrative declaration of twin status. Knowing this information in advance allows for improved differentiation between twins while treating the rest of the population normally.

To a lesser extent, familial relationships also produce higher degrees of facial similarity, due to shared genetics. Even the “gold standard” of identification – DNA – faces similar limitations with twins and familial relationships. 

While facial appearance and DNA are driven by genetics, fingerprint friction ridge patterns and iris texture are not. Instead, they are formed environmentally (e.g., fingerprints are formed in the womb). As a result, fingerprint and iris algorithms do not share the same limitations with twins and relatives. 


Facial Spoofing

One of the biggest strengths for face recognition technology is the amount of publicly available data, which enables development of highly accurate algorithms – but this strength is a double-edged sword. 

The abundance of facial imagery on LinkedIn, Facebook, company websites, yearbook photos, and a wide range of other sources, creates substantial risk for identity fraud. Acquiring copies of face photos is exceptionally easy.

As we learned from Beyonce’s 2016 Super Bowl appearance, it’s essentially impossible to remove a photo from the Internet. Because facial appearance is our single most public piece of information, it means that there will be no solution in the form of removing facial images from everywhere they exist. 

Instead, to protect individual identities, significant effort must be invested in the development of spoof detection algorithms (also referred to as “liveness” checks or “presentation attack detection” algorithms).

Liveness checks are critical when face recognition is used for ID-verification through a mobile app or unattended customer service kiosks. As the accuracy of face recognition algorithms continues to increase at exponential rates, these unattended access control applications grow increasingly common. For other use cases, such as forensic identification or identity validation when a human operator is present (e.g., at a border crossing), liveness checks lack relevance. 

While fingerprint and iris biometric modalities can also be spoofed, acquiring samples to use in an attack is far more difficult, due to their private nature.


Decorative Cosmetics and Cosmetic Surgery

While facial appearance is largely determined by genetics, facial appearance can also transform through the use of makeup, hormones, and cosmetic surgeries.

Instead of cosmetics, early facial recognition research often focused on overt traits like facial hair. Over time, however, it has become increasingly clear that while factors like facial hair present a fairly insignificant challenge to facial recognition accuracy, cosmetics create more difficulty than originally understood. 

The fact that cultural use of cosmetics is predominantly by female populations has in turn resulted in minor, but statistically significant, differences in facial recognition accuracy between males and females. While some may attribute the difference in accuracy between genders to some inherent bias in facial recognition algorithms, it is becoming increasingly clear that the decrease in accuracy is instead due to the latent factor of cosmetics use by females.


Single Trait

While we have 10 fingers and two eyes, we only have one face. Thus, while it is more challenging to collect and manage samples from multiple fingers or irises, we can improve accuracy using this technique. With face recognition, this opportunity does not exist. 


Ground Truth Issues

Another challenge with face recognition technology is that authoritative databases often lack pristine identity labels. Whether due to fraud, human operator error, or other factors, prominent government databases all seem to have identity labels errors. These ground truth errors are not inherent for facial biometrics algorithms themselves, but instead stem from the long-term legacy use of facial imagery in databases prior to automated face recognition. 

Ground truth labeling errors can typically only cause an algorithm to measure higher error rates than actually exist (i.e., perform worse), so the error rates we capture in quantitative evaluations generally represent the upper bound for actual error rates on our algorithms. A portion of what’s measured as algorithm errors actually reflects data labeling errors. In other words, face recognition algorithms are more accurate than most benchmarks indicate.

Historically difficult to identify, these legacy errors in identity databases can reduce effectiveness of facial recognition algorithms. While ground truth errors are not easy to discover, automated face recognition algorithms have become astonishingly good at flagging potential errors for human review. However, this approach requires dedicated de-duplication processes.

Identity labels errors present significant challenges across all biometric traits, though fingerprint and iris present more unique challenges than compared to faces. While it may be easy to find ground truth errors in face recognition databases, finding them in fingerprint or iris databases is much more difficult. For potential facial identity error, the human brain is highly adept at comparing facial appearance making it easy to flag such inconsistencies when encountered. For fingerprint recognition and iris recognition, identifying errors is far more difficult, time-consuming, and expensive, often requiring expert analysis. 

Perhaps no method can reduce the incidence of ground truth errors more than the incorporation of multi-biometric modalities. Though many face systems support use cases where fingerprints are available, the use of both face and fingerprint biometrics significantly reduces the chances for fraud to exist in identity databases.



In the last two decades, the capabilities of automated face recognition technologies have undergone a stunning transformation. Once considered too inaccurate for use as a primary biometric trait, facial recognition is now the single most accurate biometric technology in the world. These exponential improvements will not likely slow anytime soon.

At the same time, every new technology comes with limitations. For facial recognition, that primarily includes identical twins and identity spoofing. 

The other two primary biometric traits –fingerprint and iris– also have advantages and limitations.

All factors considered, given the extreme ubiquity of facial appearance in our daily lives, and the astonishing accuracy of modern face recognition algorithms, minimal drawbacks exist for the technology. 

Most importantly, fusing face recognition with another mature biometric technology like fingerprint recognition can create a multi-modal biometric solution for nearly impervious unattended biometric identification.

Multi-modal biometrics are indeed the holy grail of identification. All biometric traits come with tradeoffs and weaknesses, but in scenarios where two or more disjoint traits can be measured together, the incident of failure or fraudulent access becomes exceptionally low. 

This article is a summary of a presentation provided at the National Institution of Standards (NIST) International Face Performance Conference (IFPC) on November 16th, 2022.

Put ROC multimodal biometrics to work for your business

Discover how our combination of face and fingerprint recognition can provide unparalleled accuracy for your identity verification and access control needs. Reach out now to upgrade your systems and safeguard your communities with only the best in multimodal biometric tech.

Share This