RGB-D face recognition in surveillance videos

Abstract

Biometric analysis of surveillance videos carries inherent challenges in form of variations in pose, distance, illumination and expression. To address these variations, different methodologies are proposed, including utilizing temporal and 3D information. With the introduction of consumer level depth capturing devices such as Microsoft Kinect, research has been performed in utilizing low cost RGB-D depth data for characterizing and matching faces. Face detection being the foremost task in face biometric pipeline has a cascading effect on the performance of any face recognition system that follows. Face detection algorithms generally work best for frontal face images with good illumination and low standoff distance. Developing a face detection system robust to the variates of a surveillance scenario is a highly challenging task. Recognition of the detected faces in surveillance scenarios is a challenging task owing to high variance in pose, illumination, expression and resolution. Also, the quality of depth data in RGB-D videos deteriorates with increase in standoff distance, thus adding to the challenges of RGB-D face recognition. This research introduces the KaspAROV RGB-D video face database which provides face videos and images from Kinect device for over 100 subjects. The database encompasses challenges such as pose, distance, and illumination. Further, a novel face detection system for RGB-D videos taken in unconstrained scenario is proposed. The proposed system makes use of human body detection in color images and fuses it with the corresponding depth map to provide a robust solution for face detection at a distance in RGB-D videos. For recognizing the detected faces we introduce a RGB-D face recognition algorithm which can also work with only RGB probe images in absence of depth data in probe images. The proposed algorithm generates a shared representation from RGB images which contains discriminative information from both the RGB and depth images. This representation is much more discriminative than the RGB images as it gives substantially higher identification accuracy than a conventional fusion based RGB-D recognition pipeline.