Exploring XR-based tangible interactions

Abstract

This project presents a novel tangible user interface that merges the physical act of sculpting with the creative power of generative AI. By leveraging an Augmented Reality (AR) Sandbox as a direct, physical input for a real-time diffusion model, the system allows users to shape landscapes in sand and witness them instantly transform into vivid, AI-generated biomes. This work explores the potential of tangible interaction to provide a more intuitive, expressive, and accessible means of controlling complex AI systems, bridging the gap between the digital and physical realms. The prototype successfully demonstrates a functional pipeline for real-time visual synthesis, validating the core concept of using tangible interaction to guide generative processes. This exploration aims to revolutionize how users interact with and visualize data, enabling rapid prototyping, interactive manipulation, and enhanced creative expression.