Analysis of Face Recognition Vendor Test Results –
Our latest release of the ROC SDK v2.4 delivers another substantial set of improvements, demonstrated by its exceptional performance in the February 2nd, 2023 National Institute of Standards and Technology (NIST) Face Recognition Vendor Test (FRVT) Ongoing benchmark.
While our organization has maintained a steady cadence of enhancements to our face recognition algorithm every 4 to 6 months since founding in 2015, the last 18 months show a clear surge in the pace of our improvements.
Labeled as “rankone-014” in the latest FRVT Ongoing report, the ROC SDK v2.4 face recognition algorithm outperformed hundreds of global competitors across numerous critical benchmarks, achieving:
- #1 global leader in combined accuracy and efficiency
- #7 of 338 global solution providers in lowest Average Error Rate
- #1 of all key marketplace and U.S. competitors in Average Error Rate
- #6 of 338 global solution providers in lowest Visa Border dataset error rate
- #1 lowest Visa Border dataset error rate of all key marketplace and U.S. competitors
Specific performance metrics from ROC SDK v2.4 in the NIST FRVT Ongoing 2/2/23 report, labeled as “rankone-014” in Tables 8 to 29, are as as follows:
| ROC SDK v2.4 Face Recognition Algorithm Accuracy and Efficiency Metrics |
|---|
 |
These error rates represent a 1.3x decrease as compared to our previous ROC SDK v2.3 (rankone-013 in FRVT). Accuracy rates are now consistently in excess of 99.5% accuracy / True Match Rates, while in many cases operating at False Match Rates of 1 in 1,000,000 (1E-6).
As shown above, the Average Error Rate of ROC v2.4 now measures at 0.0061. This is simply computed from Tables 19 to 29 in the FRVT Ongoing report by measuring the arithmetic mean of all eight error rates liste
When comparing our Average Error Rate to other vendors listed, ROC.ai now ranks 7th of 338 global algorithm developers:
| Face Recognition Average Error Rates: Global Top 50 |
|---|
|
| Bar chart of the 50 global face recognition developers with the lowest Average Error Rate in FRVT Ongoing 2/2/2023. Rank One Computing is highlighted at 7th out of 338 total developers benchmarked. |
When looking at key marketplace competitors, ROC.ai delivers the lowest average error rate:
| Face Recognition Average Error Rates: Market Competitors |
|---|
|
| Rank One Computing shown with the lowest average error rate out of ten market competitors in FRVT Ongoing 2/2/2023. |
In addition to achieving best across-the-board accuracy of all key market competitors in FRVT Ongoing, ROC.ai also placed 6th globally out of 318 vendors in the Border dataset. When examining key competitors on the Visa Border dataset, ROC SDK easily had the lowest error rate:
| Visa Border Face Recognition Error Rates: Market Competitors |
|---|
|
| Bar chart of ten market competitors with the lowest Visa Border error rate in FRVT Ongoing 2/2/2023. Rank One Computing is shown with the lowest error rate. |
Ranking 7th globally in accuracy, the ROC SDK trails the global leader in each category by only only a fractions of a percentage points:
| Comparison of ROC v2.4 vs. Top Global Vendor Per Category |
|---|
 |
While Chinese companies are leading 7 of the 8 categories, ROC SDK is often less than 0.1% away from the global leader in each category.
Efficiency Analysis
While ROC.ai is a global leader in accuracy, we are also truly one-of-a-kind in combined accuracy and algorithmic efficiency.
The four summary statistics for face recognition algorithm efficiency are provided in Tables 8 to 18 in the FRVT Ongoing report are:
- Template size – the amount of bytes need to represent a face image
- Comparison time – the amount of time it takes to compare two templates and generate a threshold.
- Template generation time – the amount of time to process a face and produce a comparable template
- Binary size – the amount of memory / RAM needed by a device to run the algorithm
More information about these efficiency metrics can be found in our previous article, Procuring a Face Recognition Algorithm: Efficiency Considerations, as well as Hardware Considerations when Architecting a Face Recognition System.
For each of these four efficiency metrics, the NIST FRVT Ongoing report ranks each of the algorithms amongst all 478 algorithms (from 338 developers), with the best performing algorithm ranking #1. While the top ranks are often from highly inaccurate algorithms, ROC is unique in both ranking as both one of the most efficient algorithms and one of the most accurate algorithms.
The ranking of each of ROC SDK’s algorithm efficiency metrics is as follows:
| ROC SDK v2.4 Face Recognition Algorithm Efficiency Metrics and Rankings |
|---|
 |
Out of the 478 algorithms, ROC SDK v2.4 has the 13th best average efficiency ranking. Our previous versions were also included in the benchmark test – ROC SDK v2.2 has the 12th best efficiency ranking, and ROC SDK v2.0 has the 7th best efficiency ranking). When examining the Average Error Rate across the top 20 most efficient algorithms, it is fairly stunning how much lower the error rate is with ROC compared to the other highly efficient algorithms:
| Face Recognition Efficiency and Error Rates: Global Top 20 by Efficiency Rank |
|---|
|
ROC SDK provides error rates that are 2x to 50x lower than the other highly efficient algorithms. Only two of the top 20 most efficient vendors perform within 4x of ROC’s error rate, and most resulted in a more than 10x higher error rate.
When examining marketplace competitors, our efficiency ranking is substantially better than the top 10 competitors with lowest error rates:
| Efficiency and Error Rates: Market Competitors by Error Rate |
|---|
|
ROC SDK combines the lowest error rate of all competitors, with unbeatable efficiency compared to marketplace competitors.
Finally, when examining the top 20 globals vendors with the lowest error rate, Rank One Computing emerges as being unparalleled in average efficiency ranking:
| Efficiency and Error Rates: Global Top 20 Vendors by Error Rate |
|---|
|
All together, ROC.ai is the unquestioned global leader in combined accuracy and efficiency.
FRVT 1:N Performance
In addition to ranking 7th globally in the FRVT Ongoing report, the ROC SDK v2.4 also achieved standout performance in the Feb 10th, 2023 NIST FRVT 1:N Identification report. The 1:N report measures the accuracy of searching large databases, which is an operational use-case with a long and historic legacy.
Specific achievements of ROC.ai in the 2/10/2023 FRVT 1:N report include:
- Top-10 globally in Rank-1 hit-rate in all investigative search accuracy results (64K image dataset up to 12M image dataset)
- #8 of 388 algorithms in investigation frontal mugshot ranking
- #6 of 250 algorithms in investigation mugshot webcam ranking
- #9 of 277 algorithms in immigration visa border ranking
- #5 of 222 algorithms in immigration visa kiosk ranking
- Rank-1 hit rate of 99.87% searching a database of 12M images
Demographic Biases in Face Recognition
Another key achievement of the ROC SDK v2.4 was further improvements in normalizing error rates and match thresholds across key demographic groups. While certain demographic plots for ROC SDK v2.4 / ‘rankone_014’ were not yet published in the 2/2/2023 FRVT Ongoing report, our organization still notably ranked 7th out of 406 algorithms in lowest False Match Rate (FMR) ratio between West Africa and Eastern Europe, according to the FRVT Demographic Effects in Face Recognition online leaderboard. We also ranked 7th globally for lowest False Non-Match Rate (FNMR) overall.
Further examining the vendor scorecard for ROC SDK v2.4, the FNMR rates across a wide range of countries was quite stable:
The differences in False Match Rate (FMR) between different countries and age groups was also highly stable, where the following results show the factor of difference in FMR at global threshold that corresponds to a FMR of 0.00003:
These results demonstrate the stability of the ROC SDK across a wide range of demographic cohorts.
Face Recognition Accuracy Improvements Over Time
Finally, we will examine the improvement that ROC SDK has delivered over time to our partners and customers.
From the FRVT scorecard, the following plot summarizes our error rate reductions over the last 5+ years:
Over this time the FNMR has been reduced by over 100x. In just the last 18 months, ROC has released four new FR algorithms (in ROC SDK v1.26, v2.0, v2.2, and v2.4). Each of these releases has delivered an average error rate reduction on the NIST FRVT Ongoing benchmark between 1.3x and 1.4x. In total, this has results in a 3.4x error rate reduction over this period of time with roughly a 5x reduction on the Visa datasets:
These improvements are not slowing down. ROC.ai is currently working on our next face recognition algorithm that will be released in the coming months.
Summary
This comprehensive analysis of latest NIST results demonstrates a clear pattern of superior performance. ROC.ai dominates Western countries in face recognition accuracy and efficiency. We rank first in lowest Average Error Rate among all marketplace competitors, and in efficiency compared to all top-tier algorithms. In addition to being unparalleled in combined accuracy and efficiency, ROC.ai also delivers unparalleled customer support and straightforward business practices. Contracting mechanisms like Evergreen Licensing ensure that our customers and partners always have access to all the cutting-edge advancement being delivered by our R&D.
Our differentiation does not end with accuracy, efficiency, and customer satisfaction. ROC.ai has established a rock-solid reputation as the most trustworthy developer of facial recognition algorithms in the world. We are employee-owned and financially-independent. Our technology is developed fully in-house in the U.S.A. We do not answer to the whims of investors who may have short-term goals that do not align with our customers, partners, employees. Instead, we are building a collaborative long-term roadmap to deliver faster, safer, more reliable computer vision solutions that can dutifully serve the greater needs of the world at-large.
The reasons are endless to begin integrating ROC SDK v2.4 face recognition algorithms into your organization. ROC.ai also offers top-tier algorithms for fingerprint detection, object detection, license plate recognition (LPR), optical character recognition (OCR), and more.
Reach out to team up with the world’s most trusted developer of face recognition solutions.