Improving speech to speech conversion with incorporation of speaker tonality

Abstract

We benchmark various Speech-to-Text (STT) and Text-to-Speech (TTS) models and performed an extensive literature review on downstream tasks such as Automatic Speech Recognition (ASR), Speech Emotion Recognition, Speaker Identification and Prosody Transfer. This led us to understanding the paradigms existing in the domain of audio processing and enabled us to work on speech processing and synthesis tasks. We prepared a novel multilingual speech-tospeech system with translation using State-of-the-Art ASR, TTS and Voice Conversion models. This allowed us to experiment with speaker embedding conditioning in TTS systems and explore posterior and prior conditions. We present the results in this report.