Step #1: Understand your application
Common face recognition applications include forensic search, real-time screening, identity deduplication, and access control. Each application will involve different types of facial imagery (constrained or unconstrained) and will have different technical requirements.
Step #2: Determine if you need an SDK or a system
An SDK allows engineers to build a system.
If you need a system, such as a turn-key surveillance system or a custom designed visitor management system, then you will be engaging with integrators and product developers who either use a third-party face recognition SDK, or develop their own face recognition algorithm. It is important to ask them which is the case, and, if they use a third-party SDK, who is the SDK vendor.
If instead you are a systems integrator or product developer that needs to license an SDK in order to develop a face recognition system and product, then you will be engaging with an SDK vendor.
Step #3: Determine your accuracy and efficiency requirements
The accuracy and hardware requirements of a face recognition system are critical factors for success.
From an accuracy perspective, while the capabilities of modern algorithms are exceptionally high, all face recognition systems will exhibit some degree of errors. Know ahead of time what error rate you can accept in your application.
In terms of efficiency, top-tier face recognition algorithms can have a substantial difference in efficiency with only a minimal difference in accuracy. And, depending on the efficiency of an algorithm, hardware costs can soar, or wholly prevent a concept from being developed. The following links discuss how different efficiency metrics (enrollment speed, template size, comparison speed, and binary size) pertain to forensic search, access control, and real-time screening.
Step #4: Determine your business requirements
Factors that may influence your decision making process include software licensing costs, hardware requirements, budget, technical support, license enforcement mechanisms, cloud versus on-premises, and vendor geographic location (e.g., most Western organizations do not want to use Russian or Chinese developed software in security infrastructure).
Step #5: Analyze the NIST FRVT Ongoing report
The National Institute of Standards and Technology (NIST) conducts FRVT Ongoing, an up-to-date evaluation of the accuracy and efficiency for all legitimate face recognition vendors. A face recognition algorithm that has not been submitted to FRVT Ongoing should never be considered. It takes orders of magnitude less time for a vendor to submit their algorithm for third-party testing to NIST than it will for you to measure these same metrics.
Table 1 and Table 2 in the FRVT Ongoing report will provide a snapshot of each algorithm’s accuracy and efficiency. These metrics can be referenced against your requirements (Step #3) based on the application and types of imagery you will encounter (Step #1).
While NIST FRVT has many other great reports, FRVT Ongoing is updated multiple times a year and is the only report guaranteed to be current. And, given how fast face recognition algorithms are improving, it is critical to analyze current algorithms, not algorithms from years past.
It is important to note that while the NIST report is the only reliable source for third-party benchmarking results, the accuracy and efficiency reported should only serve as a rough guideline and early filtering resource. A subsequent evaluation on internal facial imagery and hardware that closely replicates your operational setting will be needed to properly assess the fitness of an algorithm or system for your use-case.
Step 6: Perform an initial selection of one or more vendors to evaluate
With consideration to the tips above, choose a vendor, or vendors, that meet your technical and business requirements.
Step 7: Ask for an evaluation
Ask selected vendors, integrators, or suppliers for an evaluation of their technology, whether it is an SDK or a system.
Step 8: Evaluate the vendors technology
You have defined the types of imagery and workflows they will be operating on (Step #1), the accuracy and hardware/efficiency requirements (Step #3), business requirements (Step #4), cross-referenced these against the NIST results (Step #5), and received access to technology that seems to meet all of these requirements (Step #7).
Now you can use the evaluation period to measure accuracy on an internally collected dataset. Along the way you will also learn about the API or GUI, ease of integration, and customer support of the SDK system.
Step 9: Negotiate a licensing agreement
If everything checks out in Step #8 then you will next want to receive a contract from the SDK vendor, system integrator, or product supplier. If you are working with an SDK vendor, you should generally be afforded options to license on a monthly, yearly, or perpetual basis. Perpetual licenses should include evergreen licensing!
You need to understand how long you will have access to software licenses, the cost of those licenses, and whether or not you will receive algorithm and system improvements along the way. Without such information in a binding contract all the time and effort put into the procurement, integration, and/or deployment process could be completely undermined by business and legal disputes down the road.
Step 10: Build or deploy your game changing system!
You have been diligent in following the first nine tips and can now move on with strong peace of mind.