Investigating Physiological and Behavioural Sensing Modalities Towards Drowsiness Detection

Monitoring driver drowsiness is a crucial aspect of ensuring road safety. Many studies have explored a variety of physiological signals and behavioural monitoring of drivers using video, or a combination of these approaches. In this paper, we investigate and optimise the effectiveness of various modalities to monitor drowsiness. We developed a physiological model using Electrocenography, Electromyography, Electrooculography and Electrocardiography recordings. A video-based behavioural model was then developed, utilising a pre-trained ResNet-101, face landmarks and handcrafted features for feature extraction, followed by classification with two Long-Short-Term memory blocks. We also investigated a combination of these models using decision fusion to form a hybrid model. The proposed method was trained and evaluated on the publicly available database (DROZY) and compared to other methods on the same database. Our proposed physiological and behavioural models were separately compared with previous approaches where we demonstrated their superior performance when appropriately validated. We further improved the results by combining the physiological and behavioural models which detected drowsiness in 93.10% of trials through leave-one-subject-out cross-validation.