XaiR: An XR Platform that Integrates Large Language Models with the Physical World

With improvements in Large Language Models (LLM) and Machine Learning (ML), the process of AR/XR experience creation with little or no human guidance is evolving. There are new opportunities to be explored, potentially changing the roles of AR platform providers, project managers, and developers. What We Will Cover In this AREA Research webinar, we will describe an approach to produce AR experiences without manual intervention, specifically in scenarios where the AR objects provide guidance to the user about something in the physical world. We propose a cognitive assistant system that can understand, reason about and respond to the physical world in AR, allowing for in-context assistance, especially for step-by-step tasks. We explore the system architectural needs, application requirements as well as the usability of such a technology, specifically in assisting with instruction-oriented use cases. Though LLMs are new and are still limited by accuracy, results have shown an improvement in response time of the AR assistant and only a small increase in task completion time. The system we will present explores the various challenges in building a platform that integrates LLMs with AR devices and examines the challenges we still have left to solve. Who Should Attend The webinar is tailored to AR project managers, architects, developers, and platform providers. Attendees will gain insights about: - Workplace conditions most likely to produce the most reliable results using the approach, and which conditions could lead to errors. - How AR platform providers would go about integrating the new LLM-assisted approach into commercial authoring platform. - Clear benefits and limitations of one specific Multimodal LLM over others in different circumstances.