Trustworthy facial analysis with scarce and biased data

Abstract

Face is one of the least invasive biometric modalities, and has been used as physical signature to perform person recognition. Face recognition has widespread applicability in various domains such as surveillance, access control, and social media tagging. While face recognition has achieved very high performance in some settings, newer challenges such as developing trustworthy AI systems have emerged. Trustworthy facial analysis relies on three components: data, algorithm, and deployment. This dissertation focuses on the data centric challenges, specifically developing facial analysis models with scarce and biased data. Limited attention has been given to applications with the availability of scarce data, particularly heterogeneous data, i.e., data belonging to different domains, such as sketch to digital face image matching. Such applications have societal impact, but are often challenging in nature. With the rapid increase in number of automated systems, recently, the biased nature of facial analysis systems has also been highlighted, demonstrating the need for automated systems to be fair. There is growing literature of research being done to understand this challenge along with efforts to ensure that these systems are unbiased and work equally well for all sub-groups of our society, irrespective of gender, ethnicity, age, or demographics. To this effect, this dissertation focuses on two key challenges which mar existing face recognition systems: face recognition in scenarios of scarce data such as sketch to photo matching, and bias in automated facial analysis systems.

We begin by exploring the challenging problem of heterogeneous face recognition with scarce data. One such application is forensic sketch to digital face image matching, where a sketch drawn by a forensic artist is required to be matched against a database of high resolution face images. It is an important challenge with great social impact involving matching data from different domains. Improving sketch-face recognition can help law enforcement agencies perform a first-level filtration of the closest matches for the generated sketches, thus improving efficiency. In order to address this problem, we develop a transform learning based algorithm, DeepTransformer, which is feature agnostic in nature, and can be applied to existing features for enhancing the performance of a system. We extended this thread to other applications with scarce data such as skull to digital image matching and caricature to digital image matching for profile linking scenarios as well. DeepTransformer suffers from the challenge of limited feature-level discriminability and relatively large number of learnable parameters. In order to mitigate the above challenges, an effective and novel framework is proposed, termed as Discriminative Shared Transform Learning for crossdomain matching applications. A shared transform learns features in a common space for data belonging to different domains, and requires lesser number of parameters to be learned, thereby making it a suitable choice for scenarios with scarce data. The shared transform is learned while modeling the class variations, therefore effectively handling the increased inter-class similarity and high intra-class distance for cross-domain applications.

The later part of this dissertation focuses on studying another challenge which is affecting the current state-of-art automated facial analysis systems: biased performance with respect to specific sub-groups. Understanding and mitigating the effect of bias is of utmost importance, given the severe effects it can have in our society with rapid growth and deployment of AI based systems. We first perform in-depth analysis of deep learning based face recognition models for factors such as race and age. We observe the similarity in the behaviour of deep learning systems to human behavior as has been observed in several cognitive studies, in terms of the most discriminative regions learned by the model and used by humans, as well as the presence of the in-group effect. As the next step, this thesis presents mitigation strategies to de-bias existing models as well as learn fair models while training from scratch. A filter drop technique is presented, which is based on identifying filters responsible for learning the biasing/protected variable label. The technique involves dropping the filters and updating the model iteratively in order to perform unbiased classification. In order to eliminate the need for additional labels, a novel unbiased feature learning loss function termed as Detox loss is proposed. The proposed loss learns unbiased deep learning models to mitigate bias from existing networks, even with imbalanced data with respect to the protected attribute. It acts as an additional constraint which is a fairness constraint when training the model with the traditional classification loss. The Detox loss enforces that the learned features are distinguished based on the task label only, and not on the biasing-attribute. The results from the analysis and mitigation strategies can be extended for more generic applications as well, thus creating a positive impact on the society and the scientific community as a whole.