Enhancing Online Shopping Accessibility: Insights from the BrowseWithMe Study

Figure: Overview of BrowseWithMe, which converts multimodal information (text and images) on a product web page into a structured representation that a user can interactively query to learn what (s)he wants to know about a product and the suggested outfit for the product.

For many, online shopping is a seamless experience: just click, browse, and checkout. However, for visually impaired people (VIP), this convenience is often overshadowed by frustration. Screen readers can be overwhelming, image descriptions are frequently missing, and cluttered web layouts make navigation challenging. Aiming to address these barriers, a research article presented at the 2018 ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) introduces BrowseWithMe, an AI-powered online clothes shopping assistant. This tool is designed to enhance accessibility and empower VIP to shop with greater ease and independence.

The Problem: Accessibility Gaps in E-Commerce

VIP often face significant challenges when shopping online. Key issues identified in the study include:

1. Unstructured Content: Screen readers parse web pages linearly, forcing users to listen to lengthy, disorganized text (e.g., navigation menus, ads) before reaching product details.

2. Inadequate Image Descriptions: Alt text is frequently missing or vague, leaving users uncertain about colors, patterns, or styles.

3. Inconsistent Website Designs: Each platform requires users to relearn navigation, adding cognitive strain.

Interviews with eight VIP revealed that while online shopping offers convenience, these barriers often drive users to abandon transactions or rely on brick-and-mortar stores.

Proposed solution

The authors of the paper proposed BrowseWithMe, a system that converts cluttered product pages into structured, queryable formats. Key components include:

1. Backend AI Modules

   1. Natural Language Processing (NLP): Extracts product details (price, material, descriptions) from web page text.

   2. Computer Vision (CV): Uses semantic segmentation to identify clothing items (e.g., shirts, pants) and assigns precise color names from the xkcd color palette (e.g., “dusty rose” instead of generic “pink”).

2. Interactive Query System

Users ask specific questions via voice or text commands, such as:

“What is the price?”

“Describe the outfit in the image.”

“Magnify the pants.” (for low-vision users)

The system responds with targeted information, bypassing irrelevant content.

User Feedback and Study Results

In usability tests with eight VIP, BrowseWithMe demonstrated the following observations:

• Efficiency: Participants appreciated skipping irrelevant content. One user noted, “I think for myself, reading a lot of text can be tiresome for eyes, so leaving those and moving to audio can be helpful, and knowing the details of products and specs can be useful...I like having the option of voice."

• Information Access and Trust: Two users, who recently lost their sight, related the experience of using BrowseWithMe to catalogue shopping. In Jackie’s words, "it is more like looking at an ad than at a specific product; it reminded me of reading the Sunday paper...more like browsing to put together a complete outfit. I didn’t expect that." One user shared the system allowed her to learn what fashions are current, while

  removing the burden of sifting through too much information

• Independence: Users reported increased confidence in navigating sites without sighted assistance.

• Social Connection: Participants saw potential for BrowseWithMe to support social connection by enabling users to share clothing images with friends for feedback, mirroring in-store shopping experiences. They also appreciated its role in fostering self-expression and personalization, helping users discover styles, explore trends, and assemble complete outfits even without knowing exactly what they’re looking for.

BrowseWithMe’s CV module outperformed traditional alt text in tests, achieving a 20% higher accuracy rate in image recognition tasks. For example, it correctly identified “black top” and “light peach skirt” in outfit images, while alt text often lacked such detail. Areas for improvement included expanding commands to cover size availability, fabric blends, and social sharing for feedback.

Broader Implications and Future Work

While focused on clothing, the framework could extend to other shopping domains (e.g., groceries, electronics) by adapting its NLP and CV modules. The authors also emphasized the need for:

Customization: Tailoring magnification levels and speech speeds for diverse user needs.

Scalability: Testing the tool across more websites and user groups.

Conclusion

BrowseWithMe exemplifies how AI can address accessibility gaps by prioritizing user agency. By converting passive browsing into active inquiry, it offers a practical solution to longstanding challenges in online shopping. The study underscores the importance of inclusive design principles and collaboration between researchers, developers, and end-users. The source code of the system can be accessed at https://github.com/kothariesha/BrowseWithMe.

Reference

Stangl, A.J., Kothari, E., Jain, S.D., Yeh, T., Grauman, K. and Gurari, D., 2018, October. Browsewithme: An online clothes shopping assistant for people with visual impairments. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 107-118).