XaiR: An XR Platform that Integrates Large Language Models with the Physical World

Logo
Presented by

Sruti Srinidh - PhD Student, Carnegie Mellon University (CMU), Christine Perey - Chair, AREA Research Committee

About this talk

With improvements in Large Language Models (LLM) and Machine Learning (ML), the process of AR/XR experience creation with little or no human guidance is evolving. There are new opportunities to be explored, potentially changing the roles of AR platform providers, project managers, and developers. What We Will Cover In this AREA Research webinar, we will describe an approach to produce AR experiences without manual intervention, specifically in scenarios where the AR objects provide guidance to the user about something in the physical world. We propose a cognitive assistant system that can understand, reason about and respond to the physical world in AR, allowing for in-context assistance, especially for step-by-step tasks. We explore the system architectural needs, application requirements as well as the usability of such a technology, specifically in assisting with instruction-oriented use cases. Though LLMs are new and are still limited by accuracy, results have shown an improvement in response time of the AR assistant and only a small increase in task completion time. The system we will present explores the various challenges in building a platform that integrates LLMs with AR devices and examines the challenges we still have left to solve. Who Should Attend The webinar is tailored to AR project managers, architects, developers, and platform providers. Attendees will gain insights about: - Workplace conditions most likely to produce the most reliable results using the approach, and which conditions could lead to errors. - How AR platform providers would go about integrating the new LLM-assisted approach into commercial authoring platform. - Clear benefits and limitations of one specific Multimodal LLM over others in different circumstances.
Related topics:

More from this channel

Upcoming talks (2)
On-demand talks (55)
Subscribers (1389)
The Augmented Reality for Enterprise Alliance (AREA) is the only global non-profit, member-based organization dedicated to widespread adoption of interoperable AR-enabled enterprise systems. The AREA provides a deep and open exchange of good practices, lessons learned, and technological insights which can help enterprises effectively implement AR technology, boost operational efficiency and create long-term benefits. We currently support members from: • Enterprises (buyers of AR solutions) • Providers (of AR solutions) • Non-commercial entities (Universities, public research institutes and agencies)