Locating the Speaker: Spatial Acoustic Perception for Wearable & Portable Devices
This technology enables wearable or other portable mics to recognize and locate speech. It combines a tiny 3D-printed acoustic sensor with AI for spatially-aware speech recognition. Applications range from smart personal assistants to advanced robotics navigation and auditory surveillance.
Background
Wearable devices like earphones are becoming central to our interaction with digital intelligence, especially with the rise of Large Language Models (LLMs). There is a growing need for these intelligent systems to move beyond purely linguistic understanding to grasp the physical context of human communication. Specifically, providing LLMs with spatial awareness of sound sources would enable more natural, intuitive, and truly context-aware interactions, unlocking advanced applications in augmented reality, accessibility, and personal productivity where understanding “who said what, and from where” is crucial.
Current approaches face significant hurdles in achieving this spatial intelligence within wearable form factors. Traditional spatial audio technologies, relying on multi-microphone arrays, are inherently bulky and power-intensive, rendering them impractical for compact earphones. Consequently, LLMs typically process speech in an acoustically ‘blind’ manner, unable to discern the direction of a speaker or other sound events. This limitation prevents them from providing crucial spatial context, such as identifying who said what in a group conversation or offering directional cues, while also struggling with computational demands and privacy concerns when attempting to transmit or process complex audio data on-device.
Technology
This technology integrates spatial audio context into Large Language Models (LLMs) for wearable earphones and other small, portable mics to enable spatially-aware speech understanding. It uses a compact, passive 3D-printed acoustic microstructure over a monaural microphone to physically encode direction-of-arrival (DoA) and reverberation cues into sound waves. These spatial cues are processed into low-dimensional embeddings, while speech is simultaneously encoded into linguistic embeddings. Both embedding types are combined and aligned with the LLM, fine-tuned for spatially aware reasoning and generating context-rich outputs like multi-speaker localization. On-device preprocessing generates compact embeddings, transmitted to a cloud-hosted LLM for inference, balancing privacy, power, and bandwidth.
This system overcomes the limitations of bulky, power-intensive traditional microphone arrays unsuitable for wearables. And, unlike prior LLM integrations, it provides comprehensive spatial speech understanding within a compact form factor. It employs a miniature (1.5 cm x 1.3 cm) passive microstructure, consuming very little energy while achieving precise DoA accuracy (25.72° MAE) with just two microphones.
A complete description has been published in Proceedings of the 42nd International Conference on Machine Learning, Vancouver, Canada, PMLR 267, 2025. https://proceedings.mlr.press/v267/mishra25a.html
Benefits
- Enables precise spatial audio understanding and direction-of-arrival (DoA) estimation in compact wearable devices using a passive microstructure-assisted acoustic sensor.
- Achieves high accuracy in multi-speaker localization and spatially aware speech recognition with low latency and minimal power consumption.
- Maintains user privacy by processing raw audio on-device and transmitting only low-dimensional, non-invertible embeddings to the cloud for LLM inference.
- Significantly reduces hardware size and energy usage compared to traditional multi-microphone arrays, suitable for earbud integration.
- Supports robust performance under noise, reverberation, and varying acoustic environments with a hybrid edge-cloud architecture.
- Enables advanced applications such as augmented reality, accessibility aids, multi-speaker meeting summarization, and immersive extended reality experiences.
- Utilizes synthetic spatial speech datasets for efficient training and fine-tuning of large language models to perform spatial reasoning and generate context-rich natural language outputs.
Applications
- Spatially aware virtual assistants
- Multi-speaker meeting transcription
- Navigation aid for visually impaired
- Robotic navigation
- Scene-aware acoustics for surveillance
- Immersive AR/XR audio experiences
- Selective voice focus in noise
Additional info
This system integrates spatial audio context into Large Language Models on wearable earphones. It uses a passive 3D-printed microstructure covering a microphone to physically encode sound direction and reverberation. Lightweight neural networks process these cues into embeddings, which are combined with linguistic embeddings. A fine-tuned LLM then performs spatially aware reasoning, with efficient on-device preprocessing and cloud inference.