Audio visual consistency detection

Abstract

With recent advancement in deep learning, it has been made possible to create fake images and videos with near perfect precision. This has led to a growing concern about the possible misuse of this technology in context to fake news. We propose a novel architecture that attempts to learn temporal and structural association between the facial images and the audio of a person.In our model, given a sequence of frames and the corresponding audio, we predict the future frame using an ensemble of LSTM and GAN. Further, we compute the distance between the prediction and ground truth frame to determine whether the given video is real or fake. This give us the advantage to use this method on any fake video regardless of its method of creation and classify it as real or fake. This method is completely unsupervised since we do not require the fake video dataset, only original video of the person is needed.