http://repository.iiitd.edu.in/xmlui/handle/123456789/1 CSE 2026-08-13T03:42:41Z http://repository.iiitd.edu.in/xmlui/handle/123456789/1993 Accessing trust in mental health counselling conversation Agarwal, Ayush; Akhtar, Md. Shad (Advisor) Mental health has emerged as the most critical global healthcare concern which is ac celerated by recent lifestyle changes, evolving individual aspirations, and social media. The use of AI in mental health has been useful in every aspect and stage of mental health- from monitoring, assistance and evaluation to providing a cost-effective, scalable, and qualitative improvement of mental health therapeutic outcomes. However, there is a serious lack of research and assistive tools in providing real-time patient feedback to measure aspects and overall therapeutic outcomes of psychotherapy. To this end, we propose a novel metric of trust in mental health dialogue systems. Specifically, every patient’s utterance in a counseling dialogue is scored on a scale that signals the degree of trust developed in the patient towards the therapist at any point in the therapy ses sion. The contribution of this work is twofold (a) we provide a concretized definition of the novel metric of trust in mental health dialogue systems and a clear annotation framework that scores the strength of trust developed between a patient and therapist during counseling. (b) We discuss various strategies to model trust correctly and finally model it as a knowledge-guided time-series forecasting problem. We perform exten sive experimentation and benchmark trust on various classical and modern methods of time-series forecasting, with our best model achieving an MSE loss of 0.0017. 2024-07-18T00:00:00Z http://repository.iiitd.edu.in/xmlui/handle/123456789/1992 Proximal interpolative fusion of heterogeneous representations for affective and secure voice intelligence Phukan, Orchid Chetia; Buduru, Arun Balaji (Advisor); Sharma, Rajesh The rapid emergence of large-scale Pre-Trained Models (PTMs) trained on massive and diverse corpora has fundamentally reshaped the landscape of numerous AI domains, in- cluding natural language processing, computer vision and voice intelligence. In the sce- nario of Voice Intelligence Systems (VIS), PTMs have completely shifted the paradigm. From usage of handcrafted features and task-specific architectures towards rich, gener- alizable representations that encode acoustic, linguistic, and paralinguistic cues in a unified feature space. As a result, enabling downstream tasks to be learned efficiently even in low-resource or label-scarce settings. In this thesis, we focus on two critical frontiers of VIS: firstly, the affective dimension, which concerns the system’s ability to interpret users’ emotional and expressive states. Secondly, the secure dimension, which requires robustness against synthetic-speech manipulations such as voice cloning and the capability to attribute a fake to its generating model. Both of these tasks are essen- tial for preserving trust and system integrity. These two capabilities are complementary and increasingly necessary across modern applications. Affective understanding sup- ports conversational agents, healthcare monitoring, and human–computer interaction, whereas security underpins voice authentication, forensic analysis, and safety-critical communication. In emerging personalized and identity-aware services, both affective insight and resilience to synthetic speech are essential for trustworthy, user-centric VIS. Both dimensions have seen substantial progress with the rise of large PTMs, and recent research has increasingly leveraged these models to advance VIS. However, PTMs are highly heterogeneous—differing in architectural design, training objectives, linguistic scope, and modality grounding. Consequently, each PTM encodes a distinct and partial view of the speech signal, emphasizing different acoustic, paralinguistic, or semantic cues. Recognizing this, researchers in application such as automatic speech recognition iv have begun exploring fusion of heterogeneous PTMs representations to exploit comple- mentary strengths for better performance, with similar trends emerging in different NLP and computer vision applications. In this thesis, we explore and deepen this direction by investigating how heterogeneous PTM representations can be systematically fused to advance both the affective and secure frontiers of VIS. To enable effective fusion, we introduce Proximal Interpolative Fusion, a novel fusion paradigm. Proximal refers to bringing diverse PTM representations into task-relevant compatibility across geometric, divergence, and interaction spaces. Interpolative denotes learning an intermediate fused manifold rather than collapsing one PTM’s representational space into another, thereby preserving complementary paralinguistic, acoustic, and semantic cues. Building on this paradigm, we develop a set of novel fusion frameworks that are organized into three core families, each capturing a distinct principle for how heterogeneous representa- tions are combined. (i) Geometry Guided Alignment enables fusion by placing PTM representations into shared geometric spaces prior to integration. This family includes HYFuse, which fuses representations through hyperbolic projection and möbius op- erations; MATA, which employs optimal transport for manifold-to-manifold alignment; and MERLINN, which performs dual-geometry fusion across euclidean and hyperbolic spaces. (ii) Interaction Guided Alignment achieves fusion through explicit interaction operators applied between representations. This includes MiO, which integrates PTM representations through bilinear pooling, followed by SCAR, which employs a cascaded cross-attention mechanism. (iii) Divergence Guided Alignment fuses representations by harmonizing their underlying distributional structure. This includes FINDER, which aligns representational distributions using renyi divergence, and COFFE, which lever- ages chernoff distance to jointly optimize separability. Additionally, PARROT serves as a hybrid framework that combines Interaction Guided with Geometry Guided Align- ment through a hadamard interaction branch followed by optimal transport. Across af- fective and secure VIS tasks—including emotion recognition (MATA, HYFuse, PARROT) and synthetic-speech detection and attribution (MiO, MERLINN, FINDER, SCAR, COFFE), these frameworks consistently demonstrate that carefully structured fusion outperforms individual PTMs, underscoring its importance for developing improved affective and secure VIS in the era of PTMs. 2026-06-14T00:00:00Z http://repository.iiitd.edu.in/xmlui/handle/123456789/1990 Visual voice activity detection using multimodal foundation models Shubham; Buduru, Arun Balaji (Advisor) This project explores the task of Visual Voice Activity Detection (VVAD) using only facial video data without access to audio. We evaluate the effectiveness of pretrained models including VideoMAE, ViViT, TimeSformer, ResNet50, as well as multimodal models like ImageBind, LanguageBind, and Video-LLaVA. Our goal is to classify whether a person is speaking in a given video segment using only visual cues. The models are tested on the VVAD-LRS3 dataset, and the results show strong promise for multimodal models even in vision-only setups. We hypothesize that large vision-language models can be adapted for explainable VVAD using prompt-based querying. 2025-07-01T00:00:00Z http://repository.iiitd.edu.in/xmlui/handle/123456789/1989 AI/ML in healthcare: leveraging embeddings for patient diagnosis and treatment optimization Malhotra, Chehak; Gopal, Mehak; Sethi, Tavpritesh (Advisor) This study encapsulates our progress in the integration of advanced AI models within healthcare contexts. Utilizing state-of-the-art models for new tasks, we explore their efficacy in tasks like cancer classification and shock prediction using data from clinical notes and prescriptions. Our study underscores the potential of AI to revolutionize healthcare practices and improve patient outcomes. 2024-01-01T00:00:00Z