Abstract
Purpose
Cooperative driving refers to a notion that intelligent system sharing controlling with human driver and completing driving task together. One of the key technologies is that the intelligent system can identify the driver’s driving intention in real time to implement consistent driving decisions. The purpose of this study is to establish a driver intention prediction model.
Design/methodology/approach
The authors used the NIRx device to measure the cerebral cortex activities for identifying the driver’s braking intention. The experiment was carried out in a virtual reality environment. During the experiment, the driving simulator recorded the driving data and the functional near-infrared spectroscopy (fNIRS) device recorded the changes in hemoglobin concentration in the cerebral cortex. After the experiment, the driver’s braking intention identification model was established through the principal component analysis and back propagation neural network.
Findings
The research results showed that the accuracy of the model established in this paper was 80.39 per cent. And, the model could identify the driver’s braking intent prior to his braking operation.
Research limitations/implications
The limitation of this study was that the experimental environment was ideal and did not consider the surrounding traffic. At the same time, other actions of the driver were not taken into account when establishing the braking intention recognition model. Besides, the verification results obtained in this paper could only reflect the results of a few drivers’ identification of braking intention.
Practical implications
This study can be used as a reference for future research on driving intention through fNIRS, and it also has a positive effect on the research of brain-controlled driving. At the same time, it has developed new frontiers for intention recognition of cooperative driving.
Social implications
This study explores new directions for future brain-controlled driving and wheelchairs.
Originality/value
The driver’s driving intention was predicted through the fNIRS device for the first time.
Keywords
Citation
Zhu, L., Li, S., Li, Y., Wang, M., Li, Y. and Yao, J. (2018), "Study on driver’s braking intention identification based on functional near-infrared spectroscopy", Journal of Intelligent and Connected Vehicles, Vol. 1 No. 3, pp. 107-113. https://doi.org/10.1108/JICV-09-2018-0007
Publisher
:Emerald Publishing Limited
Copyright © 2018, Lei Zhu, Shuguang Li, Yaohua Li, Min Wang, Yanyu Li and Jin Yao.
License
Published in Journal of Intelligent and Connected Vehicles. Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode
1. Introduction
Compared with the traditional driver assistance system, the cooperative driving system is required to be more intelligent and to have the capacity to identify drivers’ driving intentions in real time. It can also adjust the condition of the vehicle to adapt to the driver so that it can achieve coordinated control and relieve the stress of the driver. Driver’s intention identification is an extremely challenging problem in the field of cooperative driving and it now leads to a proliferation of studies. There are two methods to identify a driver’s driving intention: one is based on driving operation data (Klingelschmitt et al., 2014; Jin et al., 2012; Dang et al., 2013) and the other on brain activity data (Ikenishi and Kamada, 2014; Ikenishi and Kamada, 2015).
Many researches have studied drivers’ driving intentions based on driving-operation data (Zhang et al., 2011; Xiong et al., 2016; Xin, et al., 2017; Su et al., 2017; Bocklisch, et al., 2017). Li et al. (2016) proposed a novel algorithm that combined the hidden Markov model (HMM) and Bayesian filtering (BF) techniques to recognize a driver’s lane-changing intention. Zhou and Wu (2011) studied a recognition method for driver’s intention based on genetic algorithm and ant colony optimization. Frederik et al. (2015) described the development of a driver intention detection algorithm for automated emergency braking systems . Schmidt et al. (2015) presented a mathematical model of the steering wheel angle, which was supposed to contribute to predicting lane-change maneuver. Although a driver’s driving intention can be accurately identified based on driving data, referring to a posteriori method, the intentions may not be confirmed before the driving operation. Driving is a process that involves perception, judgment and operation and is associated with brain activities. Researchers suggested that different driving operations were related to different activities in the Brodmann areas of the cortex (Oka et al., 2015; Orino et al., 2017). Accordingly, we can identify a driver’s driving intention directly through the application of technologies that are used to measure brain activities (Foy et al., 2016), such as functional magnetic resonance imaging (fMRI), electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS).
EEG is the most traditional measurement method and is frequently used in fatigue driving detection and driving intention identification (Ikenishi et al., 2010; Ikenishi et al., 2013; Vakulin et al., 2015). However, EEG is insufficient in spatial resolution and susceptibility to the electromagnetic environment. With the introduction of fMRI and fNIRS, the researchers studied the relationship between brain function and driving behavior in a detailed way. You et al. (2012) and Maguire (2012) used fMRI to study the effects of brain-memory-related areas on the driving behavior of taxi drivers. Calhoun and Pearlson (2012) adopted simulated-driving paradigms to study both the healthy brain and the effects of acute alcohol administration on functional connectivity during such paradigms. Schweizer et al. (2013) identified the brain areas involved while performing different real-world driving maneuvers and assessed the effects of driving while distracted. Takahashi et al. (2010) studied the driver’s brain regions associated with recognition of signals at intersections, using NIRS. Yoshino et al. (2013a, 2013b) used fNIRS to study the relationship between driving behavior and cerebral cortex activity. However, fMRI has some disadvantages because of the harsh experimental environment in which the participants are made to lie down. In addition, although it has a good spatial resolution, it also has the problem of poor temporal resolution (Kato, 2004). Therefore, fMRI is not suitable for real-time identification of driving intentions. Compared with the fMRI device, the fNIRS device is more flexible as it allows the driver to maintain a normal driving posture during the experiment and complete the driving operation flexibly.
For better cooperative driving, the driver’s driving intention must be perceived with the Advanced Driver Assistance System (ADAS) in advance. This study aims to establish a driver’s driving intention identification model. Therefore, we used the NIRx device to measure the data of cerebral cortex activities to identify driver’s braking intentions. The experiment was carried out in a virtual reality environment. During the experiment, the driving simulator (DS) recorded the driving data and the fNIRS device recorded the data of hemoglobin concentration in the cerebral cortex. After the experiment, the driver’s braking intention identification model was established through principal component analysis (PCA) and back propagation neural network (BPNN). The purpose of this study is to establish a model that can accurately identify a driver’s driving intention in real time, and to provide a new research method for driving intention identification in the field of cooperative driving.
2. Methodology
The flow diagram of experimental data acquisition is shown in Figure 1. Firstly, we designed an experiment for the study of driver’s braking intention, which was implemented on the DS. During the experiment, the DS could collect driving data in real time, such as speed, engine speed, acceleration, lateral acceleration, steering wheel angle, front wheel angle, brake pedal pressure and acceleration pedal pressure. The fNIRS device could record the oxy-hemoglobin concentration changes (ΔHbO) and deoxy-hemoglobin concentration changes (ΔHb) of the driver’s cerebral cortex in real time. After that, we established the driver’s braking intention recognition model through these experimental data.
2.1 Experimental design based on driving simulator and functional near-infrared spectroscopy
The experiment was conducted in a virtual reality (VR) lab (Figure 2). A total of 52 participants were invited to participate in the experiment: 10 female and 42 male. All participants possessed a valid Chinese driving license. Their ages ranged from 19 to 38 years, with an average age of 26.5 years. Before the experiment, each driver needed to close his/her eyes and sit in the cab for about 2 min. The experiment was started only after each driver’s cerebral cortex blood oxygen concentration stabilized. The value of the change in cerebral blood oxygen concentration at that time was used as reference for cortical activity. At the beginning of the experiment, participants sat in a realistic vehicle mock-up and controlled it in the DS. The drivers had to keep quiet and calm during the whole experiment. While the experiment was going on, the DS recorded driving-related data, including the vehicle speed, brake pedal pressure, acceleration pedal pressure and steering wheel angle, and the fNIRS device measured the participants’ brain activities. Each experimental condition was introduced as follows.
The NIRx device (i.e. fNIRS device) used in the experiment is shown in Figure 3 and it was provided by NIRx Medical Technologies, LLC. The NIRx device adopted a unique measurement strategy wherein every possible combination of sources and detectors formed a measurement channel. It relies on the optical determination of changes in hemoglobin concentrations in the cerebral cortex, which results from increased regional cerebral blood flow. Two wavelengths were set at 760 and 850 nm for all the recording channels. The frequency in the sample was set at 7.8152 Hz (record data every 0.128 s). The NIRx device channel distribution and collection channel area numbers are shown in Figure 4. We can see that the NIRS cap contains 16 light source points (S1-S16) and 16 light detector points (D1-D16). The data acquisition channel is located between the light source and light detector points. There are 41 data acquisition channels on the NIRx device, which can record the ΔHbO and ΔHb. With this experiment, we could calculate the total hemoglobin concentration changes (ΔTH) based on ΔHbO and ΔHb. ΔTH is the sum of ΔHbO and ΔHb.
During the experiment, the drivers needed to drive according to the road signs. The experimental roads included sections of constant speed, acceleration and deceleration. As shown in Figure 5, the accelerated section road was located before the 70 speed limit sign, the deceleration section road was between the 70 and 50 speed limit signs and the constant speed section road was behind the 50 speed limit sign. Therefore, if a participant drove on this road section and if his vehicle speed was more than 50 km/h, he/she had to slow down the vehicle. If the vehicle speed was no more than 50 km/h, no deceleration operation was required. During the experiment, each driver passed the deceleration section roads six times. After the experiment, about 300 deceleration section road samples were created.
2.2 Experimental data preprocessing
In order to establish the identification model of driver’s brake intention, the experimental data was divided into two groups: braking group and non-braking group (see Figure 6). The braking group had brake operations, with no steering operations. The non-braking group had no brake operations with no steering operations in a time window, of which, traveling at a constant speed of not less than 5 s. After the experiment, a total of 152 braking group samples and 115 non-braking group samples were created. On analyzing the experimental data recorded by the NIRx device, it was found that the ΔTHR (ΔTH range, i.e. the difference between the maximum and minimum values of ΔTH in a time window) reflected the intensity of cerebral cortex activity. ΔTHR was very small when the driver kept calm. Conversely, ΔTHR would increase when he/she started thinking and judging. Therefore, ΔTHR could reflect the brain’s process from a static state to a state of thinking and judging. Also, we could use ΔTHR as an input for driving intention identification classifier.
2.3 Establishment of braking intention identification model
In this experiment, a total of 266 valid sample data were obtained, including 152 samples of braking group and 115 samples of non-braking group. About 80 per cent (120 brake group samples and 95 non-brake group samples) of the total sample was used as training data and about 20 per cent (31 brake group samples and 20 non-brake group samples) was used as testing data. The NIRx device used in this study contained 41 data acquisition channels, which meant that each sample had 41 features. However, after the model was built, too many features and very few samples resulted in overfitting. Therefore, PCA was used to reduce the dimensionality of sample data and the driver’s braking intention identification model was established through BPNN. PCA is one of the most commonly used dimensionality-reduction methods that aims to transform the multi index into a few comprehensive indexes (i.e. the principal component) by using the idea of dimensionality reduction. Each principal component of PCA can reflect most of the information of the original variable, and the information contained is not repeated. PCA processing is as follows:
Standardization of raw data: All samples of braking and non-braking groups are put into matrix X. Equation (1) is the standardizing process, zij is the value of the standardized matrix after the standardization processing, xij is the value of row i and column j of matrix X and n is the row number of matrix X.
Calculating the correlation coefficient matrix R for the standard matrix Z: equation (5) is the calculation of rij.
Calculation of eigenvalues and eigenvectors of matrix R: The eigenvalues of matrix R can be obtained by using equation (6). The eigenvalues are arranged according to the values. (λ1 ≧ λ2 ≧ […] ≧ λn ≧ 0). Then, the corresponding eigenvectors of each eigenvalue are calculated.
Calculation of principal component contribution rate and cumulative contribution rate: equation (7) is the calculation of a single vector’s contribution. Equation (8) is the calculation of the cumulative contribution. Eigenvalue λ1, λ2,[…] and λm correspond to the 1st, 2nd, […] and m-th principal components. In general, the eigenvectors corresponding to the eigenvalues with a cumulative contribution rate of 90 per cent are selected as the eigenvector matrix E [see equation (9)].
Mapping.
(1)(2)(3)(4)(5)(6)(7)(8)(9)
By mapping the sample matrix X to the selected eigenvector E, we can obtain the reduced dimensional matrix T [equation (10)]. In this study, the dimensionally reduced samples were used as input samples for the classifier BPNN. BPNN is a multi-layer feed-forward neural network. It is mainly characterized by that the signal propagates forward and the error propagates backward. The learning process is mainly divided into two stages. The first stage is the forward propagation of the signal. The signal passes through the hidden layer in the input layer to the last output layer. The second stage is the back propagation of the error. The signal goes from the output layer to the hidden layer and finally reaches the input layer to adjust the weight and bias of each layer. The typical BPNN structure is shown in Figure 7. It contains an input layer, an output layer and a hidden layer. By setting the corresponding network structure and parameters and inputting the training data into BPNN to train this network, the driver’s braking intention identification model can be obtained:
The establishment process of the model is shown in Figure 8. Firstly, the time window selected to 3.84 s and we can calculate the range of raw ΔTH in each NIRx device data acquisition channel. Then, we could get the ΔTHR of all samples, which was added to matrix A. After that, the dimension of matrix A was reduced by PCA, and this reduced dimension matrix was divided into training sample matrix B and testing sample matrix C. Finally, BPNN was trained by matrix B. After the training was completed, matrix C was used to test the generalization ability of the classifier. Adjusting the structure and parameters of BPNN helped achieve a better test accuracy.
2.4. Model validation
Three drivers’ driving data under the deceleration condition road were randomly selected to test and verify the model established in this paper. The process is shown in Figure 9. The input data of the braking intention identification model were the ΔTH vector of the driver’s cerebral cortex at some point, and the output data were the testing result of the model to the driver’s braking intention. The internal processing of the model is as follows: In the first place, the vector of ΔTHR with a time window of 3.84 s was calculated based on the raw ΔTH vector. After that, the calculated vector ΔTHR was added to matrix A to obtain a new matrix D. Then, the PCA algorithm was applied to reduce the dimension of matrix D. The reduced vector E was worked out as the input of BPNN. Finally, the output of the model was whether the driver had a braking intent at the moment.
3. Results
Figure 10 shows the relationship between the number of sample features and the test accuracy of BPNN: the x-axis represents the number of sample features after PCA dimensionality reduction and the y-axis represents the test accuracy of BPNN. It can be seen that the samples have different feature quantities corresponding to the different test accuracy of BPNN. When the sample feature number was 15, the network achieved the best test accuracy of 80.39 per cent.
The verification result of the braking intention identification model is shown in Figure 11, in which the x-axis represents the time and the y-axis represents the verification result of the model. In this figure, the blue dashed line indicates the identification result of the model to the driver’s braking intention. A result of more than 0 means that the driver has a braking intention at the time and less than 0 means that the driver has no braking intention. The solid purple line indicates the actual operation of the brake pedal. A result of −1 indicates the driver has no braking operation at the time and 1 indicates the driver has a braking operation. As shown in Figure 11, the output of the model was greater than 1 before the driver’s braking operation, which indicated that the model could identify the driver’s braking intent prior to his braking operation.
4. Discussion
ADAS has raised expectations for the reduction of human error while driving and released or freed people from the task of driving. We called the pattern that ADAS sharing controlling with human driver and completing driving task together as cooperative driving. To achieve better cooperative driving, the driver’s driving intention must be perceived by ADAS in advance. Driver’s driving intention identification is of great significance to automatic driving, and identifying driver’s driving intention with ADAS in advance is helpful for vehicle safety, stability and comfort. This study aimed to establish a driver’s driving intention identification model. To this end, we designed an experiment based on DS and NIRx devices to study the driver’s braking intention. The simulation experiment was carried out on the deceleration road (Figure 5). During the experiment, the DS collected driving-related data and the NIRx device collected cortical activity data. Then, the machine-learning algorithm was used to establish the identification model of driver’s braking intention.
In cooperative driving, the accuracy and timeliness of driver intention recognition are particularly important for ADAS. Studies (Li et al., 2016) have shown that drivers’ driving intentions can be identified accurately from driver’s driving data. However, this method can identify the driver’s driving intention only after the driver has a corresponding driving operation. If the driver’s driving intention can be identified by ADAS in advance, it will be better to realize cooperative driving. To this end, we explored the method of identifying driver’s driving intention by brain activity measurement. In this study, the NIRx device was used to measure the activity of the driver’s cerebral cortex.
At present, researchers have used fNIRS to study the relationship between driving behavior and cerebral cortex activity, such as the relationship between activity in the prefrontal cortex and vehicle speed (Yoshino et al., 2013a, 2013b). Oka et al. (2015) studied the respective active regions of the brain when turning left and right through curve conditions experiments. Orino et al. (2017) studied the relationship between the activity of the entire brain and the behavior of driving operations during actual road travel. However, a few researchers have used fNIRS to predict driver’s driving intentions in advance. In this study, based on the DS and the fNIRS experiments, the prediction model of driver’s braking intention was established. The test accuracy of the model with the driver’s braking intention was 80.39 per cent. Although the recognition rate in this paper was lower than the recognition rate of the model built by Ikenishi and Kamada (2015) (about 88 per cent), fNIRS was used to predict driving intention for the first time, which opened a new research direction for driving intention identification.
It is worth noting that the driving intention identification model established in this study can recognize the driver’s braking intention only from constant speed driving to deceleration driving. The limitation of this study was that the experimental environment was ideal and did not consider the surrounding traffic. At the same time, other actions of the driver were not taken into account when establishing the braking intention recognition model. Besides, the validation data were gathered from 52 drivers’ experimental data, and the verification results obtained in this paper could only reflect the results of some drivers’ identification of braking intention. Because of the individual differences among the drivers, the identification model of the driver’s braking intention established in this paper cannot accurately identify every driver’s braking intention in real time. The individual differences in the brain activities of the drivers while driving are mainly reflected in the slight differences in the changes of hemoglobin concentrations in the cerebral cortex when the driver has a braking intention. Therefore, in the future research, we will establish a driving intention identification model under complex driving conditions and improve the identification accuracy of the model for driving intention to achieve the effect of real-time identification of driver’s braking intention. In addition, we will consider differences in the brain activities of drivers when they have the same driving intention.
5. Conclusion
In this study, the driver’s braking intention identification model was established through PCA and BPNN. In addition, the data of three drivers driving under deceleration condition were gathered randomly to verify the model. The research results showed that the accuracy of the model established in this paper was 80.39 per cent. This study can be used as a reference for future research on driving intention through fNIRS, and it has a positive effect on the research of brain-controlled driving. At the same time, it has developed new frontiers for intention recognition of cooperative driving.
Figures
References
Bocklisch, F. et al. (2017), “Adaptive fuzzy pattern classification for the online detection of driver lane change intention”, Neurocomputing, Vol. 262, pp. 148-158.
Calhoun, V.D. and Pearlson, G.D. (2012), “A selective review of simulated driving studies: combining naturalistic and hybrid paradigms, analysis approaches, and future directions”, Neuroimage, Vol. 59 No. 1, pp. 25-35.
Dang, R. et al. (2013), “Driver lane change characteristics for various highway driving conditions”, Journal of Tsinghua University, Vol. 53 No. 10, pp. 1481-1485.
Foy, H.J., Runham, P. and Chapman, P. (2016), “Prefrontal cortex activation and young driver behaviour: a fNIRS study”, Plos One, Vol. 11 No. 5, p. e0156522.
Ikenishi, T. and Kamada, T. (2014), “Estimation of driver’s steering direction about lane change maneuver at the preceding car avoidance by brain source current estimation method”, IEEE, International Conference on Systems, Man and Cybernetics IEEE, pp. 2808-2281.
Ikenishi, T. and Kamada, T. (2015), “Estimation of driver’s longitudinal intention for the preceding car by brain current distribution estimation method”, IEEE, International Conference on Intelligent Transportation Systems IEEE, Vol. 83, pp. 1311-1316.
Ikenishi, T., Kamada, T. and Nagai, M. (2010), “Parallel factor analysis of driver’s EEG in lane change maneuver for cooperative driver assistance systems”, IFAC Proceedings, Vol. 43 No. 13, pp. 105-110.
Ikenishi, T., Kamada, T. and Nagai, M. (2013), “Analysis of longitudinal driving behaviors during car following situation by the driver’s EEG using PARAFAC”, IFAC Proceedings Volumes, Vol. 46 No. 15, pp. 415-422.
Jin, L., Hou, H. and Jiang, Y. (2012), “Driver intention recognition based on continuous hidden markov model”, International Conference on Transportation, Mechanical, and Electrical Engineering IEEE, pp. 739-742.
Kato, T. (2004), “Principle and technique of NIRS-imaging for human brain FORCE: fast-oxygen response in capillary event”, International Congress, Vol. 1270, pp. 85-90.
Klingelschmitt, S. et al. (2014), “Combining behavior and situation information for reliably estimating multiple intentions”, Intelligent Vehicles Symposium Proceedings IEEE, Vol. 119, pp. 388-393.
Li, K. et al. (2016), “Lane changing intention recognition based on speech recognition models ⋆”, Transportation Research Part C Emerging Technologies, Vol. 69, pp. 497-524.
Oka, N. et al. (2015), “Greater activity in the frontal cortex on left curves: a Vector-Based fNIRS study of left and right curve driving”, Plos One, Vol. 10 No. 5, p. e0127594.
Orino, Y. et al. (2017), “Relationship between brain activity and real-road driving behavior: a vector-based whole-brain functional near-infrared spectroscopy study”, Driving Assessment Conference, pp. 16-22.
Schmidt, K. et al. (2015), “A mathematical model for predicting lane changes using the steering wheel angle”, Journal of Safety Research, Vol. 49 No. 203, pp. 85-90.
Schweizer, T.A. et al. (2013), “Brain activity during driving with distraction: an immersive fMRI study”, Frontiers in Human Neuroscience, Vol. 7 No. 3.
Su, C. et al. (2017), “Forward collision avoidance systems considering driver’s driving behavior recognized by Gaussian mixture model”, Intelligent Vehicles Symposium IEEE, pp. 535-540.
Takahashi, N. et al. (2010), “Fundamental study for new assistive system based on brain activity during car driving”, IEEE International Conference on Robotics and Biomimetics IEEE, pp. 745-750.
Vakulin, A. et al. (2015), “Quantitative sleep EEG and polysomnographic predictors of driving simulator performance in obstructive sleep apnea”, Clinical Neurophysiology, Vol. 127 No. 2, pp. 1428-1435.
Woollett, K. and Maguire, E.A. (2012), “Exploring anterograde associative memory in London taxi drivers”, Neuroreport, Vol. 23 No. 15, pp. 885-888.
Xin, X. et al. (2017), “Vehicle stability control based on driver’s emergency alignment intention recognition”, International Journal of Automotive Technology, Vol. 18 No. 6, pp. 993-1006.
Xiong, L. et al. (2016), “Novel stability control strategy for distributed drive electric vehicle based on driver operation intention”, International Journal of Automotive Technology, Vol. 17 No. 4, pp. 652-663.
Yoshino, K. et al. (2013a), “Correlation of prefrontal cortical activation with changing vehicle speeds in actual driving: a vector-based functional near-infrared spectroscopy study”, Frontiers in Human Neuroscience, Vol. 7, p. 895.
Yoshino, K. et al. (2013b), “Functional brain imaging using near-infrared spectroscopy during actual driving on an expressway”, Frontiers in Human Neuroscience, Vol. 7, p. 882.
You, M.W. et al. (2012), “Structural and functional changes of hippocampus in long life experienced taxi driver”, Journal of the Korean Society of Magnetic Resonance in Medicine, Vol. 16 No. 2, pp. 124-135.
Zhang, R., Yan, X. and Wu, C. (2011), “A recognition model for acceleration intention of automobile drivers based on fuzzy clustering”, International Conference on Transportation Information and Safety, pp. 1938-1947.
Zhou, S. and Wu, C. (2011), “RETRACTED ARTICLE: a recognition method for driver’s intention based on genetic algorithm and ant colony optimization”, International Conference on Natural Computation, Vol. 2, Icn, Shanghai, pp. 1033-1037.
Further reading
Diederichs, F., Schüttke, T. and Spath, D. (2015), “Driver intention algorithm for pedestrian protection and automated emergency braking systems”, IEEE, International Conference on Intelligent Transportation Systems IEEE, pp. 1049-1054.
Acknowledgements
The authors thank the participants for their time. This article was supported by “Fundamental Research Funds YJ 201621 for the Central Universities” at the Sichuan University and “the National Natural Science Foundation of China U1664263.”