NDIM: Neuronal Diversity Inspired Model for Multisensory Emotion Recognition

Abstract

Without cross-sensory interaction, a key aspect of multisensory emotion recognition, traditional deep learning methods exhibit inferior performance in this task. On the contrary, the human brain possesses an inherent and remarkable capacity for multisensory recognition. Its diverse neurons exhibit distinct responses to sensory inputs, thus facilitating cross-sensory interaction. Leveraging this superiority, we propose the Neuronal Diversity Inspired Model (NDIM), which incorporates both unisensory and multisensory neurons, aligning with the human brain. To mirror the diverse response characteristics exhibited by various neurons, we introduce innovative connection constraints to regulate feature transmission between neurons. Drawing inspiration from this novel concept of neuronal diversity, our model exhibits biological plausibility, facilitating more effective emotion recognition of multisensory information. Experiments on the RAVDESS and eNTERFAVE'05 datasets show that the NDIM achieves the best accuracy of 99.63\% and 98.45\%, respectively, demonstrating the potential of neuronal-diversity-inspired approaches in advancing multisensory interaction and emotion recognition.