Bimodal Fusion Network for Basic Taste Sensation Recognition from Electroencephalography and Electromyography

Taste sensation can be objectively measured using electroencephalography (EEG) or electromyography (EMG). How-ever, it is still challenging to effectively utilize the complementary information from EEG and EMG signals in taste sensation recognition. This paper proposes a bimodal fusion network (Bi-FusionNet) for recognizing basic taste sensations (sour, sweet, bitter, salty, umami, and blank). Two convolutional backbones with similar structures are designed to separately extract the single-modal features of EEG and EMG. Then, EEG and EMG features are concatenated for bimodal interaction and complementarity. Finally, three loss functions are adopted: a center loss for aggregating intra-class samples, a mean squared error loss for sequence positions for minimizing the difference between signals during the stimulation, and a softmax loss for minimizing the entropy of prediction and true labels. The results on the taste sensation dataset show that bimodal fusion improves recognition performance, and Bi-FusionNet outperforms single-modal methods and other fusion methods. Bi-FusionNet paves the way for the application of multimodal fusion in taste sensation recognition.