Multi-dataset emotion representation learning

Abstract

This study aims to understand the feasibility of a stress detection model through a smartwatch using the physiological data collected for lab and real-life settings. Wearable devices such as smartwatches have the potential to revolutionize mental health care by providing continuous, unobtrusive monitoring of physiological signals that can be used to detect mental states such as stress. When a person experiences stress, there are notable shifts in various bio-signals. by using such biosignals stress levels can be identified. This project builds upon the foundational work from the past two semesters, extending the scope to address the challenges of cross-dataset generalizability and practical real-world deployment of stress detection models. Earlier phases focused on establishing machine learning pipelines for analyzing stress inducing physiological signals such as Electrodermal Activity (EDA), Blood Volume Pulse (BVP), and Skin Temperature (ST). The research emphasized lab-induced and real-life datasets, such as WESAD and Nurse, to understand the variance in stress patterns across controlled and naturalistic environments. These efforts led to key insights about dataset inconsistencies, model limitations, and the importance of edge device deployment for practical applications. A significant milestone before this semester was the publication of the EEVR (Emotion Elicitation in Virtual Reality) framework, which introduced innovative preprocessing techniques and adaptive benchmarking for physiological datasets. The EEVR framework laid the groundwork for cross-dataset evaluation, emphasizing zero-shot Contrastive Language Signal Pre-training (CLSP). These methods allow models to perform well on unseen datasets without extensive retraining, bridging the gap between lab-controlled datasets and real-world data. This semester’s research focused on benchmarking six diverse datasets—CASE, CLAS, SCIENTISST, PhyMer, PhyAAT, and EmoWear—to evaluate and improve the generalizability of stress detection models. Specifically, the objectives were: 1. Feature Extraction: Develop and refine pipelines for extracting statistical, peak-based, and wavelet-transform features from physiological signals across datasets. 2. CLSP Model Benchmarking: Employ zero-shot CLSP models to compare their performance on arousal and valence predictions. 3. Baseline evaluation : Evaluate the performance of baseline traditional ML models, specifically random forest, linear discriminant analysis (LDA) and Multi-Layer Perceptron (MLP), on arousal and valence. 4. Cross-Dataset Evaluation: Analyze how models trained on one dataset transfer to other datasets with different collection protocols and distributions. 5. Framework Validation: Leverage the EEVR methodology to assess the efficacy of preprocessing and benchmark improvements. Electrodermal Activity (EDA) and Photoplethysmography (PPG) are widely recognized for their reliability in stress detection. EDA measures changes in skin conductivity caused by sweat gland activity, which correlates with sympathetic nervous system arousal, making it a strong indicator of stress. On the other hand, PPG tracks blood volume changes using optical sensors, providing insights into heart rate variability (HRV), a metric often used to assess stress levels. A combination of EDA and PPG data enhances the robustness of stress detection, leveraging complementary features from both modalities to improve prediction accuracy. Recent advancements demonstrate the potential of multimodal approaches, integrating EDA and PPG, to capture a more comprehensive view of physiological responses to stress.