Facial Recognition

PDF version of this report
You must have Adobe Acrobat reader to view, save, or print PDF files. The
reader is available for free
download.

Facial Recognition

by James G. Barr

Docid: 00018030

Publication Date: 2207

Publication Type: TUTORIAL

Preview

Facial recognition is a form of biometrics, a field
that encompasses a broad range of technologies to identify and authenticate
an individual by
measuring and analyzing her physiological or behavioral traits.
Facial recognition is employed by police departments and security agencies
to identify, locate, and apprehend criminal suspects. It is also used
extensively for access control. Some complain that facial recognition
contributes to today’s “surveillance state” and should be more rigorously
regulated.

Executive Summary

[return to top of this
report]

Related
Faulkner Reports

Biometrics Market Trends Market

Biometrics in Mobile
Devices Tutorial

Biometrics in Healthcare Tutorial

In addition to facial recognition technology, common types of
physiological biometrics include:

Fingerprint recognition
Hand geometry recognition
Iris recognition

Employed by police departments and security agencies to identify, locate,
and apprehend criminal suspects, facial recognition is highly
controversial, especially as the software is prone to misidentifications
(false positives), leading in extreme examples to the arrest and
incarceration of innocent individuals, particularly people of color.

Facial Recognition Use Cases

In addition to law enforcement, facial recognition is utilized for the
following:

Digital Access – Granting or denying
access to a personal computer, smartphone, or mobile application.

Physical Access – Granting or denying
access to a physical facility, like a data center, laboratory, factory, or
office building.

Physical Surveillance – Identifying and
locating a person of interest within a physical facility or within a
space adjoining a facility (the facility’s perimeter).

Transportation Security – In the US,
confirming the identity of domestic travelers, travelers applying to enter
the country, or non-citizens participating in immigration proceedings.

National Security – Confirming the
identity of a foreign national or “researching” a known or suspected
criminal or terrorist.

The Problem with Facial Recognition

While biometric technologies, including facial recognition, have many
legitimate applications, many citizens have expressed reservations about their
use. For example:

Fingerprint recognition, the world’s first biometric, has its origins
in law enforcement, rendering even law-abiding citizens potentially
nervous.
Iris and retina recognition, which depend on eye measurements, seem
“invasive” to some.
Voice recognition, among other modalities, isn’t particularly precise, at
least when compared to fingerprint or iris recognition.

The problem with facial recognition, however, is a lack of transparency
since a facial template can be assembled without a subject’s consent or
even knowledge. At least with fingerprints, you know when you’re “in the
system.”

Transparency, incidentally, is also an issue with other biometric
technologies, notably DNA, which can be collected surreptitiously.

Facial Recognition Technology

[return to top of this
report]

Facial recognition technology (FRT) and its underlying facial recognition
software mathematically maps an individual’s facial features, storing the
data as a faceprint or facial template. Utilizing machine and deep
learning techniques, the software can then compare a “live capture” or
digital image of an individual against a single facial template or a
library of facial templates to establish the person’s identity.¹

As detailed by the US Government Accountability Office (GAO), the facial
recognition process proceeds as shown in Figure 1.

Figure 1. Facial Recognition Process

Source: GAO analysis | GAO-21-526

Related Technologies

Facial recognition has spawned two related technologies:

Facial Detection – Determining whether
a photo or video contains a face, useful for counting the number of people
in a particular space; and

Facial Analysis (aka, facial
classification or facial characterization) – Determining or estimating a
subject’s age, race, or gender, or a subject’s expression or gaze. In the
latter case, for example, “facial analysis can be part of an eye tracking
system, which can allow researchers to analyze how well pilots use their
eyes or gaze to scan their cockpit instruments.”²

FRT Adoption

Although still maturing, facial recognition technology is finding broad
acceptance, especially among US federal agencies. According to a GAO poll,
18 of the 24 agencies surveyed reported using FRT for one or more
purposes, specifically:

Digital access or cybersecurity – 16 agencies
Domestic law enforcement – 6 agencies
Physical security – 5 agencies

Ten agencies reported FRT-related research and development
initiatives. Furthermore, ten agencies reported plans to expand their use of FRT
through fiscal year 2023.³

Issues in Facial Recognition

[return to top of this
report]

“The expansion of facial recognition
technology (FRT) has become a prominent global issue. The European Union’s
draft Artificial Intelligence Act proposes to restrict public FRT use, and
the European Parliament uncovered its stance by calling for a ban on the
technology.”

– Attorney Taylor Kay Lively⁴

Among the major issues fueling concern are:

Data Security – “Faces are becoming
easier to capture … and cheaper to collect,” increasing the potential
for identity theft and cyber stalking.

Personal Privacy – People can’t easily
hide their face from FRT systems.

Less Anonymity – “When in public, most
people expect their face to be recognized by a few people or businesses,”
not to have their movements tracked and their activities recorded.⁵

FRT Still Exhibits a Racial Bias

While civil liberties organizations continue to cite the historic evidence of racial bias in facial recognition, recent
research reveals that FRT systems incorporating deep learning algorithms
are narrowing traditional black-white discrepancies. According to analysts
James Coe and Mustafa Atay:

“Testing results using the traditional machine learning algorithms
resulted in skewed results. These results ultimately displayed that
not only does bias exist, but it can also alter results depending on the
approach or algorithm. The bias that we observe from our testing indicates
that the machine learning algorithms we tested with were biased towards
White Subjects.

“We also [determined] that when using deep learning algorithms, …
higher accuracy is achieved. We notice that we could not witness any
bias using deep learning algorithms, as these algorithms seem to mitigate
the previously detected bias.”⁶

Facial Analysis Technology (FAT) Isn’t Quite Ready

As analyst Kashmir Hill reports, long-standing complaints that FRT can be
“biased, unreliable, or invasive” are having an impact, causing FRT
providers to reconsider their commitment to facial analysis. “Microsoft [recently revealed] that it planned to remove [facial
analysis] features from its artificial intelligence service for detecting,
analyzing and recognizing faces. They will stop being available to new
users …, and will be phased out for existing users within the year.” Interestingly, “the changes are part of a push by Microsoft for tighter
controls of its artificial intelligence products.”⁷

Commercial Interests Are Recklessly Wielding FRT

As proof of things to come, a website called PimEyes provides for a fee
($29.99 per month) the ability to search for a specific face, finding
obscure photos that would otherwise be lost in the cloud.

Analyst Hill reports that “a search takes mere seconds. You upload a
photo of a face, check a box agreeing to the terms of service and then get
a grid of photos of faces deemed similar, with links to where they appear
on the internet. The New York Times used PimEyes on the faces of a dozen
Times journalists, with their consent, to test its powers.

“PimEyes found photos of every person, some that the journalists had never
seen before, even when they were wearing sunglasses or a mask, or their
face was turned away from the camera, in the image used to conduct the
search.”

Fascinatingly, “PimEyes does not include results from social media sites.
The sometimes surprising images that PimEyes surfaced came instead from
news articles, wedding photography pages, review sites, blogs and
pornography sites. Most of the matches for the dozen journalists’ faces
were correct. For the women, the incorrect photos often came from
pornography sites, which was unsettling in the suggestion that it could be
them. (To be clear, it was not them.)”⁸

Whether it is PimEyes (or an inevitable competitor), an FRT-based picture
mining service can be exploited by:

Hiring managers looking to do a deep dive photo check on job
applicants – photos presented without explanation or proper context.
Cyber extortionists looking to blackmail everyday citizens with
compromising – or apparently compromising – photos.
Political operatives seeking embarrassing or even disqualifying photos
of rival party candidates.

Perhaps even more disturbing, people sensitive about their reputation might feel compelled to pay to research
themselves just to be sure what’s out there. Moreover, they will have to
conduct these self-searches on a regular basis.

The Future of Facial Recognition

[return to top of this
report]

Facial recognition has an uncertain future, with safety and
security interests competing against personal privacy concerns.

Crime Prevention Is a Great Enabler

Crime is up – in some jurisdictions, way up – and police departments and
public officials are advocating for greater use of facial recognition to
identify and capture perpetrators. Reversing a recent trend, analyst Paresh Dave reports that “efforts to get [FRT] bans in place are meeting
resistance in jurisdictions big and small from New York and Colorado to
West Lafayette, Indiana. Even Vermont, the last state left with a near-100
percent ban against police facial recognition use, chipped away at its law
last year to allow for investigating child sex crimes.”⁹

The clear mandate from public officials to FRT developers is to improve
and expand your product because public safety interests demand it.

Facial Recognition Technology Must Evolve – Quickly

The first and most obvious requirement is to improve the recognition of
people possessing black and brown skin. This can be achieved by:

Integrating more sophisticated deep learning algorithms
Expanding the number of black and brown facial templates in a provider’s
sample base, thereby facilitating a more nuanced matching process

Another issue, aggravated by the recent pandemic, is the recognition of
people wearing masks or other face coverings. As analyst Eric Hess points
out, “Occlusion detection – recognizing when a face is covered – has been
a feature of professional face detection platforms for some time, so it’s
a natural evolution to teach an algorithm how to recognize a mask and then
work with the partial face information it can discern to provide a match
to the database of known faces. As preventative masks become more common
around the globe, detection and accurate recognition of partially occluded
faces is more important than ever if facial recognition systems are to
continue providing enhanced security and convenience.”

Effective occlusion detection will be essential for:

Healthcare workers, whose faces are often obscured by layers of
personal protective equipment (PPE)
Police and security officers, whose faces may be partially hidden by
plastic shields and other projectile-resistant facewear¹⁰

Contactless Authentication Will Be a Value-Add

One potential opportunity accentuated by the pandemic is using facial
recognition in lieu of finger scanning to authenticate an individual’s
identity. As analyst George Brostoff observes, “Fingerprint scanners can
quickly become a source of infection, especially in public spaces.” Facial
recognition has the virtue of being contactless. Moreover, it “can be
implemented in almost all scenarios that once used pins, badges, FOBs, or
fingerprints.” In addition to businesses, health-conscious schools could
be early adopters.¹¹

Permissible Use Rules Must Be Established

James Andrew Lewis of the Center for Strategic & International
Studies asserts that “FRT can only be used in a manner consistent with
constitutional protections for civil liberties and civil rights. The
best way to ensure this is through legislation and regulation that
outlines specific FRT uses in a manner consistent with those rights. For
law enforcement, this could draw on the protections developed for other
investigatory techniques, such as communications surveillance. Other
government uses can also build the existing body of law regarding data use
and retention. Rules will vary by use case, such as whether FRT is being
used for authentication of identity, forensics, or surveillance. The use
of FRT for commercial purposes does not create the same risks but points
to the need for Congress to create national privacy legislation.”¹²

In terms of accepted – and acceptable – standards, respected bodies like
the US National Institute of Standards and Technology (NIST) and the
International Organization for Standardization (ISO) should be involved
both as sponsors and contributors.
As an example, the ISO has issued a standard on the use of biometrics,
including facial recognition, in video surveillance systems: ISO/IEC
30137-1:2019: Information technology – Use of biometrics in video
surveillance systems – Part 1: System design and specification. Among
its many elements, the standard:

“Provides guidance on the composition of the gallery (or watchlist)
against which facial images … are compared, including the selection of
appropriate images of sufficient quality, and the size of the gallery in
relation to performance requirements; [and]
“Makes recommendations on data formats for facial images and other
relevant information (including metadata) obtained from video footage,
used in watchlist images, or from observations made by human operators.”

Web Links

[return to top of this
report]

http://www.asisonline.org/

http://www.continuitycentral.com/

http://www.iso.org/

http://www.gao.gov/

http://www.nist.gov/

References

¹ Margaret Rouse. “Facial Recognition.” TechTarget. 2020.

^2-3 GAO-21-526 “Facial Recognition Technology: Current and
Planned Uses by Federal Agencies.” US Government Accountability Office.
August 2021.

^4-5 Taylor Kay Lively. “Facial Recognition in the United
States: Privacy Concerns and Legal Developments.” Security Technology.
December 1, 2021.

⁶ James Coe and Mustafa Atay. “Evaluating Impact of Race in
Facial Recognition Across Machine Learning and Deep Learning Algorithms.”
The Authors. 2021.

⁷ Kashmir Hill. “Microsoft Plans to Eliminate Face Analysis
Tools in Push for ‘Responsible A.I.'” The New York Times. June 21,
2022.

⁸ Kashmir Hill. “A Face Search Engine Anyone Can Use Is
Alarmingly Accurate.” The New York Times. May 26, 2022.

⁹ Paresh Dave. “U.S. Cities Are Backing Off Banning Facial
Recognition As Crime Rises.” Reuters. May 12, 2022.

¹⁰ Eric Hess. “Straight Talk About Face Masks and Face
Recognition.” BNP Media. April 14, 2020.

¹¹ George Brostoff. “COVID-19 and Security: How We’re Moving
to a Touchless Future.” Security | BNP Media. July 14, 2020.

¹² James Andrew Lewis. “Facial Recognition Technology:
Responsible Use Principles and the Legislative Landscape.” Center for
Strategic & International Studies. September 29, 2021.

About the Author

[return to top of this
report]

James G. Barr is a leading business continuity analyst
and business writer with more than 40 years’ IT experience. A member of
“Who’s Who in Finance and Industry,” Mr. Barr has designed, developed, and
deployed business continuity plans for a number of Fortune 500 firms. He
is the author of several books, including How to Succeed in Business
BY Really Trying, a member of Faulkner’s Advisory Panel, and a
senior editor for Faulkner’s Security Management Practices.
Mr. Barr can be reached via e-mail at jgbarr@faulkner.com.

[return to top of this
report]