Buddy Walk is a multimodal AI-powered web application designed to enhance vision assistance technology for users who are blind or have low vision (BLV) by combining real-time image analysis with contextual information from the Google Maps API. The app aims to help users travel more independently, confidently, and safely by providing detailed, accessible descriptions of their surroundings. Our novel approach integrates computer vision, geolocation services, and a large language model (LLM) to provide a variety of assistive services such as accurately identifying and detecting objects and landmarks, responding to queries ranging anywhere from descriptions of images or videos captured by the user or from navigation-related queries, and returning relevant environmental details through an intuitive user interface compliant with the Web Content Accessibility Guidelines (WCAG). The app is developed through continuous user testing sessions and feedback from multiple BLV users and organizations such as Lighthouse Guild and Visions. By integrating multiple data sources and interactive feedback, Buddy Walk focuses on making independent travel safer and more accessible for BLV users and demonstrates the potential for AI-assisted mobility tools to bridge accessibility gaps in urban navigation, paving the way for broader adoption and future integration with emerging assistive technologies.

Publication:

Tyler Ortiz, Mamuna Chaudhry, Jianing Qi, Richard Perez, William Seiple, Hao Tang, Zhigang Zhu, A Multimodal AI-Powered Travel Assistant Geo-App for BLV Users. Accepted to Journal on Technology & Persons with Disabilities, 2026 (Best Paper Award – Dr. Arthur I. Karshmer Award)