A novel public dataset for multimodal multiview and multispectral driver distraction analysis: 3MDAD

Highlights

• Introducing a new extensive, multimodal, multiview and multispectral dataset.

• Highlighting the issues observed in driving environments in day and night time.

• Exploring public databases and providing an overview of some private datasets.

• Assessing the relevance of using two cameras simultaneously in multiple viewpoints.

Abstract

Driver distraction and fatigue have become one of the leading causes of severe traffic accidents. Hence, driver inattention monitoring systems are crucial. Even with the growing development of advanced driver assistance systems and the introduction of third-level autonomous vehicles, this task is still trending and complex due to challenges such as the illumination change and the dynamic background. To reliably compare and validate driver inattention monitoring methods, a limited number of public datasets are available. In this paper, we put forward a public, well-structured and complete dataset, named Multiview, Multimodal and Multispectral Driver Action Dataset (3MDAD). The dataset is mainly composed of two sets: the first one recorded in daytime and the second one at nighttime. Each set consists of two synchronized data modalities, both from frontal and side views. More than 60 drivers are asked to execute 16 in-vehicle actions under a wide range of naturalistic driving settings. In contrast to other public datasets, 3MDAD presents multiple modalities, spectrums and views under different time and weather conditions. To highlight the utility of our dataset, we independently analyze the driver action recognition results adapted to each modality and those obtained of several combinations of modalities.

Introduction

Intelligent Transportation Systems (ITS) are becoming an important component of our society. They are meant to enhance transportation safety, efficiency and sustainability as well as comfortable driving experience [1], [2]. One of the main key focus of ITS is the Advanced Driver Assistance System (ADAS) technology, which plays a crucial role in ensuring the vehicle, driver, pedestrian and passenger’s safety and comfort. Properly used ADAS technologies can prevent 40% of all vehicle crashes and about 30% of traffic deaths [3]. In the USA, vehicular collisions, in 2016, caused 37,461 fatalities and more than 2.4 million demoralizing injuries, with an estimated cost of 242 billion dollars. Importantly, more than 37,400 people were killed in traffic crashes (a 5% increase from 2015) [4], [5]. A high amount of fatalities occurred in darkness or twilight when it was very difficult for drivers to see clearly.

Driver inattention is an extremely influential contributing factor of road crashes and incidents. It is defined as a diminished attention to activities that are critical for safe driving in the absence of a competing activity [6]. This factor can be clustered into two main classes: distraction on the one hand, and fatigue and somnolence on the other hand [7]. This was confirmed by an online survey, in which Peter et al. [4] affirmed that the two main leading contributors to severe crashes were distraction by secondary tasks and poor visibility in low light that generally made drivers sleepy.

With the continuous improvement and development of advanced driver assistance systems up to very automated driving functions, drivers are allowed to engage temporarily in non-driving related tasks. However, they need to react appropriately to a taking control request when the automated vehicle reaches its limitations. To support the driver in such situations, driver inattention monitoring systems might permit adaptive take-over concepts [8]. Thus, monitoring driver inattention is still important and is a trending topic that faces several challenges [9] such as illumination variations, cluttered background, etc. Non driving related tasks are numerous and can take many forms. Thus, the National Highway Traffic Safety Administration categorized distraction into four groups [10]: cognitive distraction when the visual field of the driver is blocked where they have to be looking while driving, visual distraction when the driver neglects looking at areas they should be looking to while driving, physical distraction when both driver’s hands (or one hand) are taken off the steering wheel to manipulate an object, and auditory distraction when the driver is prevented by sounds from making the best use of their hearing since their attention is drawn to whatever causes the sound. Most of non driving activities can include more than one of these classes, for example talking to a cell phone which creates cognitive, physical and auditory distraction.

Crash data affirm that poor visibility at night is one of the leading contributors to fatal collisions. In fact, for decades, nighttime fatality rates have been three to four times higher than daytime rates [4]. Multiple factors are involved in the road safety difference between day and night. The main factors are poor visibility coupled with distracted driving and drivers’ fatigue that is a natural reaction to darkness. Fatigue does not have a universal definition [11]. In an attempt to avoid accidents, most fatigued and sleepy drivers will try to avoid sleeping. Thus, certain physical and physiological phenomena that precede the onset of sleep can be observed. When a driver is tired and begins to resist getting sleepy, several symptoms can be noticed such as the increased frequency of touching eyes, heads and faces, repeated yawning, the difficulty of keeping eyes open, slower responses and reactions, etc. Consequently, in our study, we consider fatigue and somnolence as a secondary task activity.

To maintain safe driving, monitoring driver inattention is vital and crucial. As a result, several efforts to detect and recognize driver distraction are made utilizing different acquisition devices. To be effective, such sensors need to be human centric and take into account a lot of system components including: driver monitoring (e.g. looking at the driver to recognize their activity and attention state), vehicle sensors (e.g. looking at the vehicle speed, the steering angle, braking, etc.) and vehicle surroundings (e.g. looking at road and other cars to understand the surrounding situation) [12], [13]. In this paper, we focus mainly on driver monitoring given that it offers a deeper understanding about recognizing common inattention.

Different physiological and physical signals have been captured such as ECG, EEG, EOG, etc [14], [15], [16], [17], [18], [19]. However, their high cost and the complexity of their installation have led to the use of vision sensors. These latter offer the most direct method for detecting the early onset of distraction [7], and there are excellent means of optimization since such sensors can be seen as an excellent platform to be shared with other vision-based driver assistance applications. Multiple types of vision sensors can be used including monocular cameras, depth cameras, etc. Given their clarity, color images prove their robustness, especially in controlled environments. However, in naturalistic driving settings, captured color images are impacted by various weather conditions, which influence the image quality [20], [21]. For this reason, researchers tend to supplement or even replace images provided by monochrome and color cameras in the visible spectrum with practical images from other modalities with the intent of improving the performance of the whole system no matter what the weather conditions are like, but still keeping or even ameliorating the types of features and classifiers. At the same time, infrared cameras can produce clear images under the same conditions (day or night), and they are designed to be used in poor lighting and poor weather conditions.

In real-world driving settings, the accuracy of vision-based driver inattention monitoring methods remain limited because of the big number of challenges present in this domain such as dynamic background, occlusion, and bad visibility [22]. To effectively compare these methods, public datasets are crucial. They allow the direct comparison of numerous methods with the state of the art and they open challenging questions to a wider community. Therefore, several datasets have been collected, whether they are in simulated assisted-driving, naturalistic driving settings or a parked vehicle. However, most of them are not publicly available.

In order to facilitate research activities in this field, we propose in this paper a complete, useful and publicly available dataset, named Multiview, Multimodal and Multispectral Driver Action Dataset (3MDAD), which is designed to overcome the limitations of the aforementioned databases. Our new public, multispectral, multimodal and extensive dataset highlights the issues observed in naturalistic driving settings including multiple users, dynamic and cluttered background, varying viewpoints and lighting conditions employing a Kinect camera during both daytime and nighttime. 3MDAD presents an important number of distracted actions reported by the WHO [23]. In daytime, it provides temporally synchronized RGB frames and depth frames. At nighttime, 3MDAD contains temporally synchronized infrared frames and depth frames. Such a dataset is of a valuable benefit to researchers working in different fields like image processing, computer vision, sensors fusion [24], [25], and human-centered intelligent driver assistance systems.

The main contributions of this paper include:

• Introducing a new extensive, multimodal, multiview and multispectral dataset that highlights the issues observed in naturalistic driving environments, employing two Kinect depth cameras in daytime and at nighttime. The dataset is publicly available.¹

• Exploring public databases to our knowledge while positioning ourselves against these datasets, and providing an overview of some selected private datasets

• Assessing the relevance of using two cameras simultaneously in multiple viewpoints where each of them delivers different modality by applying early fusion at both day and night times

The remaining of this paper is organized as follows: In Section 2, the main publicly available datasets and some private datasets related to ours are briefly reviewed. Our new public dataset is described and the main differences with the existing public dataset are pointed out in Section 3. The main naturalistic driving setting challenges are described in Section 4. Section 5 demonstrates the utility of our dataset in recognizing drivers’ in-vehicle actions by reporting several experiments

based on the features extracted from Spatio Temporal Interest Points (STIPs) and automatically extracted features based on deep learning. The conclusion is finally stated in Section 6.