Development of an interactive wall using computer vision

Abstract

This report presents the development of an Interactive Wall, a gesture-based drawing application that integrates Computer Vision, Machine Learning, and Unity to enable intuitive interaction with a virtual canvas. Using Mediapipe, the system captures real-time hand and body gestures for seamless drawing, erasing, and interaction. Key advancements include robust tracking algorithms, efficient Python-Unity communication via gRPC and UDP, and advanced rendering techniques such as shaders, decals, and render textures to create realistic brush strokes and effects. Unity’s physics engine adds dynamic, lifelike interaction with virtual objects, enhancing immersion. Optimizations such as Kalman filtering for smoother gestures and post-processing for visual enhancements improve accuracy and responsiveness. The system is scalable, can support multi-user interaction, AI-based gesture prediction, and customizable brush textures to enrich the user experience after further developments. This report outlines the design, implementation, and testing phases, addressing challenges like occlusions and real-time performance. The Interactive Wall demonstrates the togetherness of gesture recognition and immersive virtual environments, offering a novel and engaging digital interaction platform.