Multi-modal sensor fusion for geometric and semantic scene understanding with graphs

Abstract

Multimodal sensor fusion is a technique which combines data from multiple sensors for applying the combined data in an effective manner which would provide a better understandning of the environment. In some cases such as autonomous driving, sensor fusion of multiple cameras and Lidar could give us semantic and geometric context for applications like localisation of the ego vehicle. Graph registration and smoothing of Multimodal sensors can also be used for tracking semantic and geometric objects around an ego vehicle which could give us more information of environment and of the ego vehicle, for example: we can deduce the velocity of the ego vehicle relative to stationary planes which are inferred through the fused and registered graphs. In this project, we propose a new approach using graph-based representation of semantic and geometric instances combined with Kalman filtering over time series of graphs.