Nell Barber: Using psychology to teach machines facial recognition

Though Nell Barber wasn’t sure what her future held after graduating with a bachelor’s degree in psychology, she now uses her interest in human behavior to design systems that leverage machine learning algorithms to identify faces in a crowd. Her subsequent degree in computer science paved the way for her current research at the Department of Energy’s Oak Ridge National Laboratory – designing software to accurately identify diverse faces from a distance, in a variety of contexts, and with privacy of each person in mind.

“I'm bridging the modalities of computationally understanding a person and their identity,” said Barber of finding a strange natural confluence between social science and computer science. Barber is an identity science researcher in ORNL’s National Security Sciences Directorate.

There are many covariates of facial recognition that deal with a specific person in particular: what a person is doing, how they are walking, how they dress, how they carry themselves, whether they cooperate in the setting, or even whether they realize they are being observed. What flashes across a person’s face can be codified into data using computer vision and machine learning to help a human analyst make decisions about specific behavior and environmental context.  

Human-centric research about a private matter: A face

Barber’s interest in psychology brings a unique flavor to how she crafts her work. She recognizes the impact to a person and people as a group when machines are used to detect something so individual and personal: identity.

The spectrum of capabilities humans have in identifying faces is varied yet not always reliable. Some people can’t remember the faces of people they love while others can recognize a face from a brief exchange year prior. For situations where crowds could hide a person of interest, it isn’t feasible to scan the faces with speed and accuracy the way a computer can.

Computers have multiple ways of collecting data and making sense of the images. Computer vision and machine learning are two methods for helping computers interpret information as a person would. Computer vision entails programming a computer to interpret images as a person would as through an eye. Machine learning teaches a computer to find patterns quicker and with more accuracy. Both brains and computers look for patterns to confirm what is believed to be correct and to detect anomalies, but how each entity arrives at the decision and what is done with the information is where humans and computers diverge.

Where computers thrive in sorting through massive amounts of data to find a needle in the haystack, not all needles are important. Barber is looking to explain the nuances of how humans and computers process information to better program computers to get to the important pieces of data and help a person make an informed decision.

Computers aren’t capable, yet, of making decisions about how to act upon information they scan in many contexts. In a crowded mall, for instance, an algorithm can detect a person hunched over a bag in the food court and may code the behavior as the person has something to hide. A person, however, may not see an imminent threat. Facial recognition sensors may recognize faces, but a human, at least for now, is required to act on the information provided by the software. Barber distinguish how a person is needed to make decisions based on a person’s behavior in the context of the environment and shouldn’t be delegated to a computer.

Addressing diversity limitations, ethics and privacy

During her time at the University of North Carolina Wilmington, Barber learned about the need for training machine learning algorithms to recognize facial features across ethnic groups. The degree of variation across faces can owe to multiple factors including historic events in an area, influences of immigration and geopolitics. She said training data for non-white faces is limited, though the necessity is high for accurate training data representing as much of a population as possible. In real-life situations where the stakes are high for identifying the right person, accuracy and reducing false positives is paramount.

To build a library full of diverse training data, Barber is working on a project for the Intelligence Advanced Research Projects Activity to accurately identify faces from a distance. The Biometric Recognition and Identification at Altitude and Range project, or BRIAR, is an active research project including hundreds of participants at locations around the world who have agreed to have their biometric data collected, which in turn will increase the quality and quantity of training data of faces, gaits, abilities, ages and ethnicities.

Human computer interaction is where Barber strives to make strides both with technological advancements and responsible use. Faces are considered personally identifiable information, though when faces are turned into ones and zeros, privacy of personal information is not always respected. Barber is addressing this concern by building encryption into the systems for every part of the data process, including collection, storage, transmission and sharing across organizations. Advancing these technologies while also ensuring they are being safely and responsibly managed sets her work apart from other programs looking for consumer-driven uses. She chooses to work in this field to show responsible use in real settings.

For those organizations associated with the federal government looking to use facial recognition software, privacy is a vital component of the final product. Computers can sift through enormous amounts of data faster and with more precision than humans can. To be able to identify people of interest quickly and with affordable systems can save lives and prevent accidents. “It's a balance that is not taken lightly by any means,” said Barber.

UT-Battelle manages ORNL for the Department of Energy’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. The Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit energy.gov/science.