Overcoming biases and injuries : tackling challenges in facial analytic systems

Abstract

The remarkable achievements and the robust performance of deep models have been instrumental in the evolution of facial analysis systems. Employed across a wide array of applications, these systems assist in making crucial decisions and predicting important results. However, the reliability of the outcomes produced by these systems continues to be a subject of uncertainty and debate. For instance, there have been numerous instances where deep models have demonstrated varying performance levels across different demographic groups. Models that exhibit high accuracy with individuals with lighter skin tones have shown diminished performance on those with darker skin tones. In addition to this, deep models have also displayed inconsistency across various settings. Facial recognition models that perform exceptionally well on unaltered faces tend to falter when dealing with manipulated faces. We observe that the anomalous and inconsistent performance of facial analytic systems can be attributed to two major problems: (i) bias and (ii) robustness. Bias in predictions emerges from issues like domain shift and imbalanced data distribution, while challenges in robustness stem from both unintentional and intentional variations. This dissertation takes on the challenge of addressing bias resulting from skewed data distribution and offers solutions to enhance the robustness of deep models against unintentional variations due to injuries. We begin our research with an examination of the bias resulting from traditional methods of model training, specifically when training is conducted on datasets characterized by imbalanced distributions. Typically, the conventional training methods prioritize optimizing the model to attain elevated levels of classification accuracy, yet they tend to neglect the performance variances across less represented demographic subcategories (such as male and female under the broader gender category). This oversight can culminate in biased outcomes. To address this, we introduce a novel loss function, termed as Uniform Misclassification Loss (UML), aiming for equitable results when training deep models. The UML function directs the model’s focus towards the subgroup that is performing the worst during training, striving to minimize and balance the misclassification rate across all subgroups. This approach not only mitigates bias but also enhances the overall performance of the model. The UML function relies on prior knowledge of demographic subgroups (referred to as protected attributes). However, there are instances where information on protected attributes is unavailable due to privacy or legal constraints, making bias mitigation challenging. To address this issue, we propose a unique algorithm, Non-Protected Attribute-based Debiasing (NPAD). This algorithm leverages auxiliary information from non-protected attributes to counteract bias, intelligently selecting non-protected attributes to align the model with fairness objectives. For optimizing the model, we introduce Debiasing via Attribute Cluster Loss (DACL) and Filter Redundancy Loss (FRL) functions. DACL guides the model to assimilate class-specific information for reducing bias, while FRL enhances model performance by encouraging the learning of non-redundant features, resulting in unbiased predictions. Next, we shift our focus to mitigating bias in the predictions made by pre-trained models. A variety of highly efficient pre-trained models, widely applied in numerous tasks, have exhibited biased tendencies towards specific groups. In order to maintain the utility of these models while ensuring equitable results, it is crucial to neutralize the bias in their predictions. To address this challenge, we present a novel algorithm aimed at learning a consistent perturbation, referred to as Subgroup Invariant Perturbation (SIP), tailored for a particular dataset. The addition of the learned SIP to the input dataset results in a transformed dataset, which, when fed into a pre-trained model, yields unbiased results. This algorithm is grounded in adversarial perturbations and negates the need for updating model parameters, rendering it computationally efficient. Beyond addressing bias, we also address the robustness challenges faced by face recognition models due to unintentional variations. Despite significant strides in face recognition technology, challenges persist, particularly when dealing with input images that include facial injuries. Face recognition models, typically trained on images of uninjured faces, exhibit a marked decrease in performance when applied to images of injured faces. Injuries alter facial features and overall appearance, complicating recognition by automated systems. To address this issue, we first compiled an Injured Face (IF) dataset, consisting of 150 subjects, each represented by images in both injured and non-injured states. Following this, we introduced a novel loss function, Subclass Injured Face Identification (SCIFI) loss, specifically designed for recognizing injured faces. This loss function categorizes injured and non-injured images into two separate subclasses, operating in a 2-dimensional score space derived from both injured and non-injured images. The goal is to optimize this subclass space to maximize inter-class separation while maintaining uniform distance between the feature representations of samples from different subjects, as well as a consistent distance between samples from the same subject. The extensive evaluations across multiple scenarios verify the efficacy of the SCIFI loss, consistently outperforming existing algorithms and showcasing enhanced performance.