M3ER: A System and Method for Multimodal Emotion Recognition

M3ER a method for recognizing emotions by analyzing facial expressions, speech, and text. M3ER improves accuracy by about 5% over previous methods by using a novel technique that emphasizes reliable data and compensates for noisy or missing inputs.

Background

With the explosion of data and the widespread availability of supercomputing infrastructure in the modern era, computer scientists have made significant progress in developing intelligent systems that can, among other things, communicate with humans by understanding and responding to their spoken words, voices, and facial expressions. However, these systems are not accurate when perceiving and responding to human emotions. Achieving this ability would enable computers or robots to co-exist with humans in the same shared space and be more socially responsive to humans. The merits of such a system are inevitable in all aspects of a human’s everyday life. Due to this potential, the market for emotion recognition technology is currently valued at $20 billion and is expected to increase rapidly over the next few years.

Technology

Researchers at the University of Maryland have created M3ER, an emotion recognition algorithm that uses three different factors – face, speech, and text – to accurately determine if a person is angry, disgusted, fearful, happy, sad, or surprised. Using three factors makes M3ER more accurate since cues from one factor can complement or support those from another. It also makes the algorithm more resistant to absent or faulty sensor data. M3ER contains a preprocessing step to differentiate between useful (non-faulty) and useless data to further improve accuracy. The useful data is then combined using machine learning, and a result is produced. M3ER has been proven to be at least 5% more accurate than contemporary methods.

Benefits

  • Robust performance in real-world scenarios by dynamically handling noisy or missing modality data.
  • Enhanced accuracy (approximately 5% improvement) through the advanced multiplicative fusion network.
  • Effective proxy feature generation that compensates for compromised or ineffectual modalities.
  • Flexible integration with existing multimodal systems using established feature extraction techniques.

Applications

  • Customer sentiment analysis
  • Driver safety monitoring
  • Call center emotion analysis
  • Emotion-aware virtual assistants

Additional info

A multimodal emotion recognition framework robustly handles noisy and missing data. It extracts facial, audio, and textual features using GloVe, MFCC, and facial embeddings, then evaluates modality effectiveness with Canonical Correlation Analysis and a threshold. Ineffective signals are replaced with proxy features via a learned linear transformation. A deep neural network incorporating LSTMs, attention modules, and a multiplicative fusion layer with a tailored loss function adaptively weights each modality, resulting in enhanced accuracy in practical applications.