How to identify the driver’s fatigue state in the vehicle fatigue driving identification and early warning system？

Driving fatigue refers to the decline in the driver’s reaction ability caused by lack of sleep or long-term continuous driving. This decline is manifested in the driver’s drowsiness, dozing, wrong driving operation or complete loss of driving ability.
Driving fatigue is reflected in both physiological and psychological aspects, including the function of nervous system, blood and eye changes; Psychological reaction includes prolonged reaction time, distraction and uncoordinated action. The investigation and Research on the causes of traffic accidents by Indiana University in the United States found that 85% of accidents are related to drivers, and only 15% are vehicle and environmental factors.
The driver’s behaviors and faults immediately before the accident directly lead to the accident. These behaviors include perceptual delay, wrong decision-making on the environment, improper handling of dangerous situations and so on. Among all driver errors, the most common are perceptual delay and decision-making errors. These errors will lead to inattention, slow reflection, improper operation and so on. The root cause of these errors is driving fatigue. Due to drivers’ fatigue driving, the level of vigilance decreases, resulting in the increase of traffic accidents, which has become a hot spot of general concern in the society.
If the driver drives tired, his ability of observation, recognition and vehicle control will be significantly reduced, which will seriously threaten his own safety and the lives of others. With the development of transportation industry, traffic accidents have become a serious problem faced by all countries.
Main monitoring methods of driver fatigue
The monitoring and early warning technology of driver fatigue and distraction has been highly valued by various countries because of its development prospect in traffic accident prevention. Researchers have carried out various research according to the physiological and operational characteristics of driver fatigue. Some research results have formed products and began to enter the market.
The detection methods of driver’s fatigue state can be roughly divided into detection methods based on driver’s physiological signal, driver’s physiological response characteristics, driver’s operation behavior and vehicle state information.
1. Detection method based on driver’s physiological signal
The research on fatigue began in physiology. Relevant research shows that the physiological indexes of drivers in fatigue state will deviate from the indexes of normal state. Therefore, whether the driver enters the fatigue state can be judged by the physiological indexes of the driver. At present, the more mature detection methods include the measurement of EEG and ECG of drivers.
Researchers have long found that EEG can directly reflect the activity of the brain. It is found that when entering the fatigue state, the activities of delta wave and theta wave in EEG will increase significantly, while the activities of alpha wave will increase slightly. In another study [6], EEG signals were monitored in simulators and real vehicles. The test results show that EEG is an effective method for monitoring driver fatigue. The researchers also found that EEG signal characteristics have great personal differences, such as gender and personality, and are also closely related to people’s psychological activities.
ECG is mainly used in the physiological measurement of driving burden. Research shows that ECG will decrease obviously and regularly when drivers are tired, and there is a potential relationship between HRV (heart rate change) and the change of fatigue degree during driving.
The detection method based on the driver’s physiological signal has high accuracy in fatigue judgment, but the physiological signal needs to be measured by contact, and it is highly dependent on individuals. It has many limitations when actually used in the driver’s fatigue monitoring. Therefore, it is mainly used in the experimental stage as the control parameter of the experiment.
2. Detection method based on driver’s physiological response characteristics
The detection method based on the driver’s physiological response characteristics refers to inferring the driver’s fatigue state by using the driver’s eye movement characteristics and head movement characteristics.
The driver’s eye movement and blink information are considered to be important characteristics to reflect fatigue. Blink amplitude, blink frequency and average closing time can be directly used to detect fatigue. At present, there are many algorithms to study driving fatigue based on eye movement mechanism. The widely used algorithms include PERCLOS, which takes the percentage of eyelid closure time in a period of time as the measurement index of physiological fatigue.
Using face recognition technology to locate the positions of eyes, nose tip and mouth corner, combine the positions of eyes, nose tip and mouth corner, and then according to the tracking of eyes, we can obtain the driver’s attention direction and judge whether the driver’s attention is distracted.
The head position sensor is used to detect the driver’s nodding action. The position of the driver’s head from each sensor is output through the capacitive sensor array, which can track the position of the head in real time and determine whether the driver is sleepy according to the change law of the head position. This study found that there is a very good correlation between nodding action and sleepiness.
The detection method based on the driver’s physiological response characteristics generally adopts non-contact measurement, which has good recognition accuracy and practicability.
3. Detection method based on driver’s operation behavior
Driver fatigue state recognition technology based on driver’s operation behavior refers to inferring driver’s fatigue state through driver’s operation behavior, such as steering wheel operation.
By processing the monitored driver’s steering wheel operation data, the research results reveal the relationship between driver’s steering wheel operation and fatigue to a certain extent. It is pointed out that the operation of steering wheel is an effective means to judge driving fatigue.
Generally speaking, there are few in-depth research results on fatigue identification using driver’s operation behavior. In addition to the fatigue state, the driver’s operation is also affected by personal habits, driving speed, road environment and operation skills. The driving state of the vehicle is also related to many environmental factors such as vehicle characteristics and roads. Therefore, how to improve the prediction accuracy of the driver’s state is the key problem of this kind of indirect measurement technology.
4. Detection method based on vehicle trajectory
The fatigue state of drivers can also be inferred by using vehicle driving information such as vehicle trajectory change and lane departure. Like the fatigue state recognition technology based on the driver’s operation behavior, this method is based on the existing devices of the vehicle, does not need to add too many hardware equipment, and will not interfere with the driver’s normal driving, so it has high practical value.
Fatigue driving recognition system based on video technology
In April 1999, the Federal Highway Administration first proposed PERCLOS as a feasible method to predict motor vehicle drivers’ driving fatigue. After years of development, PERCLOS method has been recognized as the most effective, on-board and real-time driving fatigue evaluation method. PERCLOS is the abbreviation of percentage of eye closure over the pupil time, which means the percentage of eye closure time in unit time.
The principle of PERCLOS is to count the time proportion of eye closure in a certain period of time. The evaluation standard adopted by our system is perclos80, which means that the eye is considered closed when the area of the eyelid covering the pupil exceeds 80%.
PERCLOS measurement principle

The value of PERCLOS can be calculated by measuring T1-T4:

Where f represents the percentage of eye closure time, i.e. the value of PERCLOS.
System scheme and workflow of fatigue driving recognition system based on video technology
The driver fatigue monitoring system obtains the driver’s real-time image through the video acquisition equipment, automatically analyzes the driver’s head posture, eye movement law and facial features to determine the driver’s mental state, and gives the corresponding early warning prompt. The research shows that compared with the law of face or head movement, the law of eye activity, such as blink frequency, blink speed, eye opening range and eye gaze direction, can better reflect the mental state of the subjects at the current time.
Therefore, if the eye size, position information and motion changes in each frame of image can be obtained, the driver’s eye activity law can be counted for a period of time, and the driver’s fatigue state can be evaluated in combination with the fatigue state analysis index. The system flow is shown in the figure below:
Image preprocessing
In the driving environment, the image collected through the video stream will be affected by many factors, including noise information, such as resolution, system noise, abrupt background, etc., which will interfere with the next image operation. Therefore, we preprocess the source image by histogram equalization to remove noise, enhance image contrast, highlight image details and improve image quality.
Before and after equalization
Histogram before equalization histogram after equalization
Face detection
Face detection is an important step before eye location. The system adopts AdaBoost algorithm and uses the provided sample training and detection methods. Firstly, samples are collected and a classifier is trained from the collected samples. The classifier can distinguish human face and non human face well; In the detection link, load the image frame to be tested into the classifier, scan the image pixels to find the face contained in the image and calibrate the area. The subsequent operations will be carried out in the calibrated face area to reduce the calculation area and eliminate the interference of non face factors, which greatly improves the operation rate of the system.
Eye location
This link includes two stages: rough human eye positioning and accurate human eye positioning. Firstly, according to the prior knowledge of Chinese traditional three courtyards and five eyes, there must be an approximate area of human eyes in rough positioning. This area may contain interference such as eyebrows and hair angles at the same time, but it further reduces the calculation area; Then, the rough area of the human eye is converted into a binary image through a certain threshold, and then the gray projection in the vertical direction is carried out to obtain the histogram. Because there is a large difference between the gray of the human eye and the surrounding skin, the Y coordinates of the upper and lower edges of the human eye can be determined from the peaks and troughs in the histogram, and then the eyes can be accurately located.
Eye state judgment
Through the maximum interclass variance method (Otsu), the accurate area of human eyes is binarized with different thresholds under different light rays to obtain the best eye shape when human eyes are open and closed. Through the comparison of continuous n frames, it can be judged that the driver is currently in the closed state when the area of black pixel value is the smallest, and in other cases, it is in the open or semi open state.
Fatigue analysis
The system selects the currently recognized and effective percols fatigue evaluation index, that is, the fatigue state is analyzed by the time proportion of eye closed frames in consecutive n frames. The eye open frames are recorded as “1” value and the eye closed frames are recorded as “0” value. In this way, after consecutive n frames, the staggered sequence of “1” and “0” can be obtained, The analysis of fatigue state can be described by the proportion of “0” value in the sequence. When the percentage is higher than a certain experimental proportion, it can be considered that the driver may be tired.
Through the operation and processing of the above five steps, the system can analyze whether the current driver is in fatigue state and fatigue degree from the video stream obtained by the acquisition equipment, and give different degrees of reminder and alarm, so as to achieve the system goal.
Comprehensive judgment of fatigue degree
The judgment of driver fatigue will be adversely affected by error inspection. Through the calculation of PERCLOS, eye closing time, eye blinking frequency, mouth opening degree and head movement, the comprehensive judgment of driver fatigue can be carried out accurately and effectively.
PERCLOS
PERCLOS refers to the percentage of eye closure time in a specific time. PERCLOS method has three criteria: p70, p80 and em. The research shows that p80 has the best correlation with fatigue degree.
Mouth opening
There are usually three states of the mouth: closing, speaking and yawning. In the fatigue state, people will frequently yawn, and it will be found that there is a trough in the horizontal gray projection curve of the lower half of the area, that is, the position between the lips. Binarize the lower part of the face, and calculate the pixel value of the connected area (the connected area can prevent nostrils and whiskers from affecting the calculation) from the lips up and down to obtain the opening degree of the mouth.
Eye height and mouth height compensation
When the vertical distance from the upper eyelid to the lower eyelid and the vertical distance from the upper lip to the lower lip, because the driver’s head moves relative to the detection equipment, in order to realize the accurate calculation of the driver’s eye height and mouth height, it is necessary to correct the changes caused by the relative changes in the distance between the eyes, mouth and the detection equipment.
Eye closure time
Eye closure time is generally expressed by the time from eye closure to opening. When a person is in a normal awake state, the closing time of his eyes is very short and he will open his eyes quickly. In case of fatigue, the eye closure time will be significantly longer, so the eye closure time can directly reflect the driver’s mental state. By calculating the maximum number of frames from eye closure to opening, the more frames, the longer the closing time, and the more serious the fatigue.
Eyes blink less frequently. The horizontal gray projection is performed on the lower half of the face, and the horizontal gray projection curves of different single person images are observed,
People blink more frequently when they are tired than when they are awake. This paper also takes it as a parameter as the basis of fatigue judgment. Blink once when the eyes are closed to open. The number of blinks in a period of time is accumulated as a parameter for fatigue judgment.
Fatigue parameters of head movement
When the driver is tired, he will nod frequently and tilt his head forward. The horizontal positions of eye pupil and mouth corner are obtained by horizontal gray integral projection. Suppose D1 is the distance from the horizontal position of the pupil to the upper edge of the collected picture, and D2 is the distance from the horizontal position of the corner of the mouth to the lower edge of the collected picture. When the driver nods due to fatigue, D1 increases and D2 decreases. When the driver is tired and the head tilts forward, D1 increases and D2 increases. Nodding and head leaning forward can be used as an important basis for fatigue judgment.
Current situation of fatigue driving monitoring technology
Driver fatigue monitor launched by attention technologies (dd850) is a driver fatigue monitoring and early warning product based on the driver’s physiological response characteristics. The product collects the driver’s eye information through the infrared camera and uses PERCLOS as the fatigue alarm indicator. It can be directly installed on the instrument panel. The alarm sensitivity and alarm sound can be adjusted. At present, it has been popularized and applied, but it is only effective at night.
The s.a.m. fatigue alarm device developed by digital installations in the United States uses a magnetic strip placed under the steering wheel to detect the steering wheel angle. If the driver does not make any correction to the steering wheel for a period of time, the system infers that the driver enters a fatigue state and triggers an alarm.
Safetrac of assistware technology company of the United States uses the front video head to identify the lane line and give an alarm when the vehicle starts to deviate from the lane. The product can also judge the driver’s fatigue state by combining the lane holding state with the driver’s steering wheel operating characteristics.
The astid device in the UK comprehensively considers various factors such as the driver’s sleep information, the length and type of driving completed, and the driver’s steering wheel operation to judge the driver’s fatigue state. Before the device operates, the driver needs to input his sleep information in the past 24 hours. When the visual alarm reaches a certain level, the audible alarm will be triggered, and the driver is advised to stop and rest. After a period of rest, the built-in alarm clock will wake up the driver and reset the driving time.
In addition to these products, there are also fatigue alarm bracelets that detect fatigue through wrist movement and bracelets that can be hung on eyes and legs

If you want to develop your own fatigue monitor system products, brands or software; please don’t hesitate to contact [email protected] não o vamos desiludir.