Phoneme-based language translation for speech synthesis using sparse matrix representations

Abstract

This thesis introduces a novel framework for language translation, transitioning from conventional text-based mapping to a phoneme-level modeling approach. By employing articulatory phoneme representations and sparse binary matrices, the proposed architecture effectively aligns source and target languages at the phoneme level, leveraging a transformer-based encoder-decoder framework. Data preparation involved aligning multilingual text corpora from sources such as Mozilla Common Voice and CVSS, followed by phoneme extraction using tools like eSpeak NG. A distinctive aspect of this work is the development of a phoneme dictionary, constructed by grouping phoneme rows into word-like segments, resulting in a 10 times more expressive vocabulary than conventional row-level mappings. The proposed pipeline demonstrated a 35% improvement in phoneme alignment accuracy, alongside a substantial enhancement in speech intelligibility, achieved through mel-spectrogram generation from articulatory matrices and synthesis via a GAN-based vocoder. This approach simplifies word boundary modeling and lays the groundwork for speech-to-phoneme translation and multilingual adaptation in low-resource settings. This work establishes a transformative direction in language translation, integrating phonological structure with advanced sequence modeling, offering significant implications for text-to-speech (TTS), cross-lingual speech generation, and direct speech-to-speech translation systems.