Multimodal stream processing for accident detection

Abstract

This report presents a novel architecture for accident anticipation and localization in autonomous driving, integrating monocular depth-enhanced 3D modeling with large language models (LLMs). The system leverages advanced feature extraction techniques, such as MobileNetv2 and Cascade R-CNN, combined with attention mechanisms to process real-time dashcam video inputs. A dual vision attention mechanism enhances feature representation, while a dynamic object attention mechanism prioritizes high-risk objects. The architecture incorporates a GRU-based accident anticipation module and an attention-driven accident localization module to predict accident probabilities and identify hazardous objects. Finally, the model generates real-time verbal alerts using LLMs, offering contextually relevant warnings to passengers, enhancing safety and aware ness. This hybrid approach aims to improve both accident prediction accuracy and anticipation time, making it suitable for deployment in real-world autonomous driving systems. The model’s effectiveness is evaluated using metrics like average precision (AP) and mean Time to Accident (mTTA).