Multimodal AI Impacts Retail From Voice Assistants to Checkout-Free Shopping

Amazon and OpenAI are betting big on multimodal artificial intelligence (AI), a technology that could redefine how consumers shop and interact with businesses.

Multimodal AI, an advanced form of AI capable of processing and integrating information from various sources, such as visual, auditory and textual data, is being harnessed by tech giants to create more intuitive shopping experiences and smarter customer service.

“Multimodal AI is a significant game changer,” Shawn DuBravac, CEO of Avrio Institute, a think tank focused on emerging technologies, told PYMNTS. “It will drive advancements across numerous industries, leading to smarter, more adaptive systems that can understand and respond to the world in a more humanlike manner.”

Amazon’s Checkout-Free Vision

Amazon recently unveiled a new multimodal AI system for its Just Walk Out technology. The system aims to enhance the accuracy and scalability of checkout-free retail environments. Unlike its predecessors, which analyzed shopper actions sequentially, this new AI simultaneously processes data from multiple sources throughout the store.

“We accomplish this by analyzing data from cameras and sensors throughout the store simultaneously, instead of looking at which items shoppers pick up and put back in a linear sequence,” Amazon explains. This approach allows the system to handle complex shopping scenarios more precisely, even in challenging conditions like poor lighting or obstructed camera views.

The system’s ability to learn and adapt is particularly noteworthy. Trained on a 3D map of the store and an image catalog of merchandise, it can understand the placement of fixtures and accurately recognize products. This continuous learning capability suggests a future where retail environments become responsive to shopper behaviors.

Voice AI: New Frontier in Customer Interaction

Meanwhile, OpenAI is advancing customer interaction with its new voice interface for ChatGPT. Capable of engaging in conversations across 40 languages, this technology could change customer service and marketing strategies, particularly in the retail and healthcare sectors.

DuBravac envisions a future where businesses leverage this technology to create more engaging customer service interactions. “This can lead to higher customer satisfaction due to the personalized and empathetic nature of the interactions,” he said.

The marketing world, too, is taking notice. Kris Mullins, CMO of Capital Max, an investment and advisory firm, told PYMNTS there is potential for AI voices to become as recognizable as visual logos. “Companies can develop customized dialects or speech patterns that reflect their unique brand identity,” Mullins said. “This can turn the AI into a distinctive brand voice that consumers recognize instantly … creating a deeper brand association.”

Mullins also foresees a shift in market research methodologies. He suggested that companies could use ChatGPT as an interactive, voice-based researcher, engaging customers in natural conversations to uncover their true motivations. This approach could yield more authentic consumer insights than traditional surveys or focus groups.

However, as these technologies advance, they raise critical ethical questions. The humanlike qualities of these AI systems, particularly in voice interactions, have sparked concerns about users forming emotional attachments to AI. OpenAI’s safety analysis acknowledges this potential risk.

To address these concerns, DuBravac emphasized the importance of transparency. He advised businesses to communicate the AI’s nature, ensuring customers understand they’re interacting with a machine, not a human. This could involve implementing periodic reminders during interactions and maintaining a professional tone in AI responses.

For its part, Amazon stressed that its Just Walk Out technology respects user privacy, tracking only product interactions without collecting biometric information.

As multimodal AI continues to evolve, its integration into business strategies is accelerating. Amazon plans to more than double the number of third-party stores using its Just Walk Out technology in 2024. Meanwhile, voice AI is expected to become more prevalent in customer service and marketing.

Mullins suggested that as brands refine their offerings based on AI-gathered insights, customers may feel better understood, potentially leading to increased engagement and brand loyalty. However, he cautioned that this increased engagement must be balanced against the risk of overreliance on AI interactions.

For all PYMNTS AI coverage, subscribe to the daily AI Newsletter.