AI "Sighted Guide" for Accessible VR

Virtual Reality (VR) is rapidly moving from a niche gaming gadget to a mainstream platform for social interaction. Platforms like VRChat allow people to gather, play, and connect in immersive digital worlds. But as this digital frontier expands, it risks leaving people behind. For blind and low vision users, these visually-driven platforms are often inaccessible, presenting major hurdles in navigation, social interaction, and simply understanding what is happening around them.

A team of researchers from Cornell University and Cornell Tech sought to address this gap [1]. Past attempts to make VR accessible often relied on adding spatial sounds or haptic feedback to objects. While useful in simple environments, this approach can quickly become overwhelming and confusing in complex, unpredictable social spaces. It can also make it difficult for blind and low vision users to participate in conversations. The researchers previously explored using human "sighted guides" in VR, which was promising but had limitations: human guides aren't always available, and some users felt the assistance hindered their sense of independence.

To solve these problems, the team introduced a solution: an AI-powered guide. This guide is designed to serve as a "sighted guide" within the virtual world, offering sophisticated, on-demand assistance. The AI guide's core features include the ability to visually interpret the environment and hold a conversation with the user. A user can ask general questions like, "What does this place look like?" or specific ones like, "What is that yellow thing in front of me?". The guide also provides navigation assistance. A user can request, "Take me to the fountain," and after virtually grabbing the guide, they will be walked or teleported to their destination. The guide can also place audio beacons on objects, empowering users to navigate independently back to those spots later.

Another aspect of this AI guide is its ability to change personas. The researchers understood that a one-size-fits-all assistant isn't appropriate for the diverse needs and social contexts of VR. They developed six unique personas, each with a different appearance, behavior, and communication style. These include a Human persona, simulating a friendly but professional sighted guide ; a Guide Dog and a White Cane , which are familiar accessibility aids and act as disability signifiers ; a futuristic Robot ; and a Bird that sits on the user's shoulder, an idea inspired by participant requests in a previous study. Crucially, there is also an Invisible guide, which allows users to receive assistance privately and without visually cluttering their view, a vital option for those who may not always want to disclose their disability.

This system runs on a Meta Quest 2 headset and is built in Unity. It uses OpenAI's GPT-4V and Audio API to power its "eyes" and voice, allowing it to understand user requests and describe the visual world in real-time. The team even developed a method where the AI sees both the user's first-person view and a "bird's eye view" of the scene, giving it better context to understand the user's location and requests. The team's next step is to conduct formal studies with blind and low vision users to see which personas they prefer and in which contexts.

Takeaways for a VR Shopping App

This research offers powerful takeaways for designing a VR shopping application for people with partial visual impairment. The core finding is that personalization is paramount. A shopping assistant, much like the social guide, should not be a single, fixed tool. A user with partial vision might prefer an "invisible" or minimal guide that doesn't obstruct their remaining sight, or perhaps a "shoulder-perching" guide like the Bird, saving their main field of view for the products. The guide's functionality also translates directly: visual interpretation becomes "Read the price tag" or "Describe the image on this cereal box," while navigation assistance  becomes "Take me to the cereal aisle." Finally, the paper's emphasis on balancing assistance with independence  is critical. A shopping app could use features like audio beacons to let a user "mark" their cart or a specific aisle, enabling them to explore and return independently, which directly addresses the "loss of independence" concern noted in the research.

Reference

1. Collins, J., Nicholson, K.M., Khadir, Y., Stevenson Won, A. and Azenkot, S., 2024, October. An AI Guide to Enhance Accessibility of Social Virtual Reality for Blind People. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 1-5).