How Does Facial Recognition Work?

There is no single accuracy figure for facial recognition, and asking for one is the wrong question.

The evidence for that is not an opinion. The US National Institute of Standards and Technology runs the standing evaluation that the whole field is measured against, and its published data shows the same top-ranked algorithm, unchanged, failing on 0.14% of comparisons in one setting and 3.34% in another. Roughly twenty-four times worse, because the pictures were different. The two were measured at different thresholds, and the better figure came from the stricter one, so the real gap is not smaller than that.

This article walks through what the system actually does, why there are two kinds of error rather than one, where a human chooses the tradeoff, and what the current government evaluation data shows about accuracy and about who bears the errors.

What the system does

Four steps, and the vocabulary matters because it is where the confusion starts.

Detection. The software first has to find a face in the image. This is a separate problem from identifying whose face it is, and it was largely solved before modern machine learning: the classic approach scans the image, discards background regions quickly, and spends effort only on face-like areas.

Template. The system does not store a photo. It produces a template: a compact numerical representation. NIST's testing interface defines it as a data structure holding an embedding, a mapping from a face image into a numerical space where distance corresponds to similarity. One influential method compressed a face to 128 bytes. A template is specific to the algorithm that produced it. It is not a portable copy of your face.

Comparison. Two templates go in and a single number comes out. NIST specifies that algorithms return "a similarity score, higher is more similar." The system does not say yes or no. It produces a number.

Threshold. Somebody decides how high that number has to be. NIST puts it plainly: comparisons are "subject to noise and presentation variations that require setting an acceptance threshold."

That last step is where a human judgment gets baked into a system that then looks purely technical.

One-to-one and one-to-many are different jobs

This distinction does more work than anything else in the subject, and most confusion about facial recognition comes from collapsing it.

Verification is one-to-one. You claim to be someone, and the system compares your face to one stored template. Unlocking a phone, or matching a traveler to their own passport photo. The EU AI Act defines it as "the automated, one-to-one verification, including authentication, of the identity of natural persons by comparing their biometric data to previously provided biometric data."

Identification is one-to-many. An unknown face is compared against a database. The same regulation defines this as establishing identity "by comparing biometric data of that individual to biometric data of individuals stored in a database."

The consequences of an error are not symmetrical, and NIST's Patrick Grother put it better than anyone: "In a one-to-one search, a false negative might be merely an inconvenience. You can't get into your phone, but the issue can usually be remediated by a second attempt." A false positive in a one-to-many search, by contrast, "puts an incorrect match on a list of candidates that warrant further scrutiny."

NIST also distinguishes two ways one-to-many gets used. In identification mode a threshold is set and the system supports mostly automated decisions. In investigation mode the threshold is zero, the system returns a fixed list of fifty candidates, and it is, in NIST's words, "assumed and necessary that a human will be used to review the candidates."

Two errors, one dial

Every biometric system has two ways to be wrong, and they trade against each other.

A false non-match is failing to match two images of the same person. A false match is matching two images of different people. NIST's authentication guidance defines them in ordinary language: "there is some probability that a comparison will not result in a match, which is referred to as a false non-match rate. Similarly, there is a probability that an impostor comparison will result in a match, referred to as a false match rate."

The threshold sets the balance. Raise it and you get fewer false matches and more false non-matches. NIST describes the usual choice: "the chosen threshold usually emphasizes a low FMR to maximize security since false non-matches can often be mitigated by repeating the measurement."

One detail here is easy to miss and it matters later. NIST observes that "the vast majority of biometric systems are configured with a fixed threshold against which all comparisons are made," not tailored to cameras, conditions or demographics. Its own authentication guidance makes this a requirement rather than an accident. A biometric system "SHALL be configured with a fixed threshold," because "it is not feasible to change the threshold for each demographic."

One dial, one setting, everybody.

How accurate is it, actually

Accuracy improved dramatically and then the question changed shape.

NIST's 2018 evaluation found that searching mugshots in a gallery of 1.6 million people, the most accurate algorithm of June 2018 made "a factor of 20 fewer misses than the most accurate algorithm in 2013." NIST attributed this to deep neural networks and wrote that "face recognition has undergone an industrial revolution."

Current figures, from NIST's evaluation data as of 1 September 2026, show the ceiling and the floor at once.

Read the first and third rows together. That is the same top-ranked algorithm. NIST describes border images as "moderately poor webcam border-crossing photos that exhibit pose variations, poor compression, and low contrast," and kiosk images as having "considerable downward pitch angle." The technology did not change between those rows. The photograph did, and so did the threshold, which is why the table states it: the better figure was measured at the stricter setting.

Every one of these numbers is meaningless without its dataset, its threshold and, for a search, the size of the database.

It is not a password

Two properties separate a face from a passcode, and NIST states both. "Biometric characteristics do not constitute secrets. They can often be obtained online or otherwise without consent. A facial image can be obtained by taking a picture." And they cannot be reissued: schemes that would allow revocation exist in the research literature, but NIST notes "the availability of such solutions is limited."

NIST's conclusion follows: biometrics "SHALL only be used as part of multi-factor authentication with a physical authenticator," an alternative non-biometric option must always be available, and biometric comparison "is probabilistic, whereas the other authentication factors are deterministic."

NIST's twins data is the vivid illustration. On its identical-twins track, the algorithm ranked most accurate on ordinary mugshots accepts identical twins as the same person about 99.5% of the time. Apple concedes the same for twins, similar-looking siblings and children under 13. Google is blunter: your device "can be unlocked by someone who looks a lot like you, like an identical sibling," and "can also be unlocked by someone else if it's held up to your face."

That last point is what liveness detection addresses. ISO/IEC 30107 covers presentation attack detection: telling a live face from a photograph, a replay or a mask. Synthetic faces are a separate subject, covered in Lathic's article on deepfakes. NIST requires detection for federal authentication and has evaluated 82 such algorithms from 45 developers, finding it "varies widely across algorithms, use cases and attack types."

Who bears the errors

This is the part most often reported inaccurately, in both directions.

NIST's landmark 2019 study found the two error types behave differently: "false positive differentials are much larger than those related to false negatives and exist broadly, across many, but not all, algorithms tested. Across demographics, false positives rates often vary by factors of 10 to beyond 100 times. False negatives tend to be more algorithm-specific, and vary often by factors below 3."

The direction depends on the algorithm and the dataset. On one set of photos, false positives were highest for West and East African and East Asian people and lowest for Eastern Europeans, but NIST noted that "with a number of algorithms developed in China this effect is reversed." It also found false positives higher in women than men, consistently across algorithms though smaller than the effect of race, and elevated in the elderly and in children.

The mechanisms differ. NIST attributes false negatives largely to image quality, including lighting that underexposes darker skin, and false positives primarily to "anatomic similarity of the faces," occurring "in images of pristine quality," with a contribution from under-representation in training data.

Current data is sharper, and worth reading exactly. NIST's 1:1 demographic test, updated 1 September 2026, ranks 658 algorithms on one figure: the false match rate for the worst-affected demographic group set against the average across all groups, each algorithm's threshold fixed to give an overall false match rate of 3 in 10,000. The most equitable scores about 2.6. At the far end of the ranking the figure is around 326. The gap between algorithms is much wider than the gap between demographic groups inside any one algorithm, which is NIST's own point. In both the most equitable algorithm and the one at the far end, the worst-affected group was the same: women aged 65 and over born in West Africa.

Three caveats NIST insists on. Results do not generalize: announcing the 2019 study, NIST's Patrick Grother said that "while it is usually incorrect to make statements across algorithms, we found empirical evidence for the existence of demographic differentials in the majority of the face recognition algorithms we studied." More accurate algorithms generally produce smaller differentials. And NIST tests submitted algorithms in a laboratory rather than deployed systems, which is why the 2019 report concluded that "it will usually be informative to specifically measure accuracy of the operational algorithm on the operational image data."

Now connect this to the threshold. One fixed setting serves everyone, so a group with a higher false match rate at that setting absorbs more of the error, by design rather than by malice. Lathic's article on AI bias covers the general mechanism; this is the specific version.

What goes wrong in practice

The documented failures are not algorithms declaring guilt. They are processes.

In 2020 Robert Williams was arrested in Detroit after a facial recognition lead. He sued, and the case settled with an order entered in June 2024. The terms are the useful part, because they name the actual failure. Probable cause "must be established using legally authorized methods other than Facial Recognition," and "the photographic lineup shall not contain an image derived from facial recognition."

That second provision is the mechanism. A lead from a search fed a photo lineup, which then appeared to confirm it. The court retained jurisdiction to enforce the agreement for four years.

Practice varies widely. The US Government Accountability Office found that of seven federal law enforcement agencies using facial recognition between October 2019 and March 2022, all seven initially did so without requiring staff training, only two required it by April 2023, and only three had policies addressing civil rights and civil liberties.

Retail has one federal enforcement action. In December 2023 the Federal Trade Commission said Rite Aid had deployed facial recognition across hundreds of stores without reasonable safeguards, generating thousands of false matches, and that it "was more likely to generate false positives in stores located in plurality-Black and Asian communities than in plurality-White communities." The order imposed a five-year ban.

What the law says, and where

Rules vary enormously by jurisdiction, and general statements are almost always wrong. Four specifics, each tied to a place and a date.

The EU AI Act does not ban facial recognition. Article 5 prohibits real-time remote biometric identification in publicly accessible spaces for law enforcement, with narrow exceptions for searches for victims and missing persons, imminent threats to life, and locating suspects for serious offenses, and even then requires prior judicial or independent authorization. All four qualifiers matter. It separately bans building facial recognition databases by untargeted scraping from the internet or CCTV. Those prohibitions have applied since 2 February 2025, ahead of the Regulation's general application on 2 August 2026. One-to-one verification is carved out of the high-risk category.

Illinois has the strongest US state law. Its Biometric Information Privacy Act covers a "scan of hand or face geometry," requires written notice and a written release before collection, and provides liquidated damages of $1,000 for a negligent violation and $5,000 for an intentional or reckless one. The Illinois Supreme Court held in 2019 that a person need not show injury beyond the violation itself.

There is no single US federal statute. What exists is fragmentary: FTC enforcement, agency rulemaking, and state law. At the US border, a DHS rule effective 26 December 2025 allows photographing all non-citizens on entry and departure, while participation by US citizens is voluntary and CBP states it does not retain citizens' photographs beyond 12 hours.

Where you actually meet it

Phone unlock is the least like the others. Apple states that Face ID data "doesn't leave your device, is never available to iOS or iPadOS, apps, or Apple." Google states the face model "is stored securely on your device and never leaves the device." Both are vendor claims. Neither product appears in NIST's published list of evaluated algorithms, which is not the same as having failed it.

Android grades phones differently from police tools. Its three biometric classes all require the same false accept rate of one in 50,000 and the same 10% false reject rate; what separates them is spoof resistance. Class is about how hard a system is to fool, not how accurate it is.

The one thing to take away

Facial recognition is not one technology with one accuracy. It is a pipeline whose output depends on the photograph, the task, the size of the database, the person in front of the camera, and a threshold somebody set.

When you see a number, ask what was compared to what, at what setting, and against how many people. NIST publishes all three for every figure it reports. A number given without them cannot be checked, and should not be trusted.