Advances in lane marking detection algorithms for all-weather conditions

Received Aug 25, 2020 Revised Dec 18, 2020 Accepted Jan 13, 2021 Driving vehicles in all-weather conditions is challenging as the lane markers tend to be unclear to the drivers for detecting the lanes. Moreover, the vehicles will move slower hence increasing the road traffic congestion which causes difficulties in detecting the lane markers especially for advanced driving assistance systems (ADAS). Therefore, this paper conducts a thorough review on vision-based lane marking detection algorithms developed for all-weather conditions. The review methodology consists of two major areas, which are a review on the general system models employed in the lane marking detection algorithms and a review on the types of weather conditions considered for the algorithms. Throughout the review process, it is observed that the lane marking detection algorithms in literature have mostly considered weather conditions such as fog, rain, haze and snow. A new contour-angle method has also been proposed for lane marker detection. Most of the research work focus on lane detection, but the classification of the types of lane markers remains a significant research gap that is worth to be addressed for ADAS and intelligent transport systems


INTRODUCTION
In different weather conditions like rainy and foggy conditions, the possibility of road crashes increases. The lacking clarity in the driver's visions increases the risk of accidents which are likely to cause injuries and casualties. Based on a study carried out in Penang Malaysia, one of the main causes of accidents is the rain which causes the image of the lane markers to be blur, as well as the road surface that becomes more slippery. The unclear image of the lane markers tend to cause incorrect decisions by the drivers, as well as the automated driving assistance system, to change direction or overtake the front vehicles.
The investigated lane marker classification mechanism will also improve the confusion matrix in detecting different lane marker types and reducing road crashes when implemented in ADAS. Detecting lanes, which are designed to delineate and regulate the traffic low on the roads, is crucial for road safety. The lane markers along the roads must be properly detected and understood. However these lane markers tend to become unclear to the road users who are driving vehicles along the road when the weather changes [1,2] especially for advanced driver assistance systems [3][4][5]. The statistics prepared by Malaysian Institute of  [6]. Foggy, snowy and rainy weather conditions cause difficulties on the road users to clearly detect the lane and correctly perceive the types of the lane markers and have turned out to be among the major factors that cause road crashes [7].
A lane detection mechanism based on lane marker classification for foggy weather condition has been proposed in [8], developed based on heuristic ROI. The fogging effect on the lane is removed by using a dark channel prior method in order to allow the proposed mechanism to work. This method, although applies a unique method to defog the road images, is also embedded with the typical Hough transform technique to detect the lane.
Another lane marker classification mechanism has been proposed in [9]. This work has focused on the steady steering on the highways in different weather conditions including the rainy days. However, this approach is carried out only for tractor semi-trailer excluding other vehicles. A different approach of detecting lanes at different weather conditions is proposed in [10] based on the entropy approach. The detection accuracy reported for this work is good although this work is implemented using the surveillance camera. The proposed mechanism is carried out using the entropy-based method and is independent on the lane markers, which may affect the actual lane detection on the physical roads as they are delineated by the lane markers.
Lane marker classification is an essential part of the lane detection mechanisms [11,12] to assist the drivers making the right decisions as well as to enhance the advanced driver assistance systems. A lane marker classification method using contour analysis has been proposed in [13,14]. This research paper presents a comprehensive review on road marking classification mechanisms applied in rainy and foggy weather conditions with an initial framework of lane marker classification mechanism for all-weather conditions. The system model of a generic lane marker classification mechanism is presented in section 2 for a generally good weather condition. In all-weather conditions, there are challenges that need to be addressed to perform lane marker classification as described in section 3. A thorough review on lane marker classification and the related lane detection mechanisms is presented in section 4 along with the proposed framework of the contour-angle method for classifying and detecting the lane markers in all-weather conditions. Useful recommendations and conclusions for the future work are put forward in section 5.

LANE MARKER CLASSIFICATION METHODS
As to contribute in intelligent transport systems including the auto-assist driving systems (ADS), lane marker classification models must be designed as efficiently and effectively as possible to supply reliable input and support to these systems. Three fundamental steps required to carry out lane marker classification in generals are the region of interest (ROI) selection, image pre-processing and feature extraction for marker classification, all of which are further described as follows:

ROI selection
The first essential step in implementing lane marker classification is the selection of the ROI, which contains the necessary features or information of the video frames from which the lane marker classification is carried out. There are various methods proposed in literature to identify the ROI such as vanishing point method [15] as well as agent-based detection and tracking methods [16]. However, the mechanisms for detecting ROI in all-weather conditions for lane marker classification is still at infancy stage. Most of the proposed ROI identification mechanisms in all-weather conditions are developed for lane detection, as given in [17] for instance, and not dedicated for lane marker classification.

Image pre-processing
After identifying the ROI, the next important step is the image pre-processing. The selected ROI must be pre-processed in order to extract the important features from the image of the lane marker for making the classification decision. Generally there are two widely applied image processing techniques in processing the ROI, namely the filtering and colour adjustment or thresholding. The filtering operations are carried out to improve or enhance the quality of the ROI. This can be realized by various pre-processing techniques such as removing the noise that exists in the image, adjusting the illumination that appears in the pixels of the image, blurry image correction, edge enhancement, statistical and math processing and morphology [18].
Of the different image pre-processing methods in literature, two of the frequently used ones are the noise filtering and illumination adjustment. Noise filtering is mainly implemented to remove the noise that exists in the images to be processed. The noise may come from the defects of the road, the unwanted objects that present on the road and the presence of other elements such as snow and rain droplets which fall along the lanes or cover the wind screen of the vehicle as well as the camera used to capture the image of the lane. One of the generally adopted noise filtering methods is Gaussian filtering [19] Gaussian filtering is applied to improve the quality of the ROI by removing the noise. It is a linear filter, hence requiring relatively shorter computational time as compared to other non-linear noise filters. As for the color adjustment and thresholding, one of the widely applied schemes is Otsu thresholding [20][21][22][23][24][25] which is used for converting the image or ROI to black and white. This thresholding approach works by identifying the threshold value of the variance between the background and the foreground colour of the ROI.

Feature extraction for marker classification
When the ROI is filtered and converted to black and white, the next major step is the lane marker classification. In most of the roads available throughout the world nowadays, there are five commonly used lane marker types as shown in Figure 1. They are single dashed (D), single solid (S), dashed solid (DS), solid dashed (SD) and double solid (SS). In order to classify the lane markers, the features which are used in the decision rule need to be determined. In literature, some of the features that have been used include edges [26], histograms [27] and other geometrical features including lines, curves and contours [13,14]. The selection of these features depends on the number of different types of lane markers targeted to be classified, besides the temporal and computational loads that need to be minimized and controlled. Choosing the simple types of features such as lines and curves may require lower computational time to classify certain types of lane markers. However, if the number of lane marker types increases, more types of features will be required to classify the lane markers rendering longer computational time and load.
Road makers which appear to be close to each other such as the solid dashed (SD) and the dashed solid (DS) tend to require slightly more complicated approach to be differentiated and classified. Without a proper classification method, the accuracy of the lane marker detection will be reduced, besides resulting higher computational complexity and longer execution time. In the next section, a specific discussion on the challenges caused by all-weather conditions in lane marker classification.

ALL-WEATHER CHALLENGES
When implementing the lane marker classification in all-weather conditions, the challenges start in the ROI selection as the desired features of the image are expected to be affected by the weather conditions, such as rainy and foggy days. The selected ROI is the best when the amount of required information or features of the video frame is maximized and the amount of noise due to the weather change is minimized. This trade-off is a real challenge to be addressed in classifying lane markers in all-weather conditions.
As the quality of the resulted filtered ROI is expected to be lower than the quality of the filtered ROI in a good weather condition, the classification mechanism must be designed to perform better and more reliable in all-weather conditions. The probability that the desired features of the lane marker such as the edges and the other geometrical features are affected by the weather conditions is higher in all-weather conditions due to the rain drops, lacking visibility and so forth. These rain and water droplets are not only pouring on the road surface, but they are also found on the screen of the cameras, if the camera is mounted outside the car, or at least the wind screens of the vehicles.

LANE MARKER CLASSIFICATION MODELS IN ALL-WEATHER CONDITIONS
Therefore, in the next section, a thorough review on lane marker classification algorithms studied and proposed in the research community to face some of these challenges will be presented. The lane marker classification models which are designed to work in all-weather conditions depend on the types of different weather conditions considered. Therefore, in this section the lane marker classification models proposed in literature will be thoroughly studied according to the types of weathers considered, as described in the following subsections.

Rainy weather
A bifurcation method is proposed [9] for lane change and control in rainy weather. The objective of the research is significant especially in increasing the safety level of huge vehicles like trucks and lorries when driving on a rainy day. This is based on the Hopf bifurcation theory, which is related to the critical point that determines the stability. In this research subject, stability refers to that of the vehicle when moving on a rainy day. Although the model considers the rainy weather as the constraint, the issue of detecting and classifying the lane markers is not addressed in this paper. The lane markers, which divide the road between lanes especially of different directions, are essential to be detected especially in ADS in order to ensure the correct decision is taken while the vehicle is moving and maneuvered.
Lane detection mechanism on rainy days has been proposed in [28,29] for detecting the lanes based on the video captured from the vehicle. This lane detection scheme applies the Canny detector and edge detection scheme to detect the lane on rainy days. Although the lanes are successfully detected using this lane detection model, the types of lane markers which divide the lanes are not classified and detected. Another method is proposed to perform fast learning technique based on convolutional neutral network approach in detecting the lanes [19]. It is claimed in this paper that the lanes can be successfully detected in extreme conditions including rainy weather. However further work is needed if lane marker classification is to be implemented as this approach only works for detecting lanes, not the lane markers which divide the lanes. The pre-processing techniques implemented in the aforementioned lane changing, control and detection schemes in this section are worth to be considered when a new lane marker classification model is to be developed.

Foggy weather
Another challenging type of weather especially in the hilly and cold areas is foggy weather. A lane detection mechanism is proposed in [8], to detect the lanes in foggy weather. A heuristic approach is applied to identify the ROI and the improved dark channel is also implemented to detect the lanes. Although it is claimed that the proposed approach is able to reach 96% detection accuracy, the lane detection still excludes the lane or lane marker type classification.
An image pre-processing technique in foggy weather is presented in [30] based on segmentation. The proposed approach is developed based on the improved dark channel approach to better defogged the image captured in foggy weather. Although the approach is not addressed for lane marker classification, it is a useful technique to pre-process the fogged images or video frames. A lane marker detection technique is presented in [31,32], which detects the dashed and double solid lane markers. Although the proposed method has been demonstrated to detect the two-lane markers in the dark, this detection technique has not been successfully demonstrated to be used in foggy weather.
In general, most of the lane detection schemes [33,34], which are vision-based and use camera to capture the images of the lane and lane markers, are applied to merely detect the lanes, without classifying the types of the lane markers dividing the lanes. As seen in Figure 2, two main steps in lane detection are ROI identification with filtering and thresholding and secondly feature extraction, as shown in Figure 2 Another recent trend in lane detection is by employing around view monitoring (AVM), which have been proposed in [35,36], to name a few. This technique uses cameras at different angles of the vehicle to capture a better all-round view of the lane. Fisheye cameras are typically used in this approach to capture the lane images. Although this detection approach has been demonstrated at different weather conditions, the focus is mainly to detect the lanes, not the markers.
The AVM-based lane detection presented in [33] for example, has been implemented to detect lanes with single solid and single dashed lane markers. The lane detection approach in [37] has been developed to work in complex road conditions and dynamic environment. The correction mechanism has been implemented to improve the quality of the road images before detecting a number double solid and dashed lanes.

Other weather conditions
The recent trends and developments in this research area have shown encouraging growth in detecting the lanes and lane markers in all-weather conditions [38]. Other than fog and rain, other types of challenging weather conditions are snow and haze model [7,[39][40][41]. In [41], a lane detection mechanism is proposed to work in adverse weather conditions especially in snow when the lane markers are covered by the snow. The proposed scheme operates based on vehicle-to-infrastructure (V2I), which is applied to store the reference image of the road in the cloud before it is compared with the distorted one in the adverse weather to detect the lane. Apart from requiring a lot of configuration steps to ensure a reliable V2I communication, this scheme has not been demonstrated to be able to detect the lane markers. As for the hazy weather condition, a number of lane detection schemes have also been proposed as seen in [40,41]. The dehazing method proposed in [42] applies a weighting factor to sharpen the edge of the road image, hence dehazing it. One of the internal operations has also been removed to speed up the dehazing process.
Apart from snow and haze, there are other types of weather conditions which have also been addressed and considered in literature. These weather conditions include dusk and cloudy, as presented in [37,40], for instance. However, these types of weather conditions are not discussed in details in this paper as they are mainly related to the change in color and illumination of the lane images. Table 1 Table 2.  [19] Gaussian smoothing √ Rain, Dusk [8] Dark channel prior √ Fog [30] Dark channel prior √ Fog, Haze [32] Binarization √ D, S Rain, Fog [37] Mask Operation √ SS, D Cloudy [40] Otsu thresholding √ SS, D Dusk [41] V2I √ Snow [42] Weighting factor √ Haze It can be seen from Table 2 that most of the lane marker classification methods in literature are designed to detect up to three types of road markers, which are S, D and SS. These are the most common types of lane markers and the easiest ones to be classified. A few lane marker classification schemes have been proposed in literature to detect the more challenging types of lane markers, which are DS and SD such as in [12,47,50]. Due to the difficulty in detecting and differentiating DS and SD, the accuracy of classifying these lane markers is typically lower than 90%. Our previous method in detecting five lane markers including DS and SD have been demonstrated to increase the accuracy between 94% to 100% [12], [14] in normal weather. With effective pre-processing techniques, this method can be further improved to be applied in foggy and rainy weather.

Contour-angle method for lane marker classification
The contour analysis method is developed based on the study of the contour lines detected on the ROI of the road images. After the image preprocessing stage, which includes the conversion of the image to a binary black and white format, the contour lines are calculated in the vertical direction. The desired contour line, which represents the boundary of the white-coloured road lane marker, is detected via the following mechanism  The binary ROI is scanned vertically in a number of locations which have an equal interval between each other in -axis. The output of this operation is a binary column vector, , where '0' represents a black pixel and '1' represents a white pixel for = 1, ⋯ , .  The next step is to perform a bit-wise XOR operation between consecutive bits, [ ] and [ + 1], in each column vector for = 1, ⋯ , − 1 where is the length of the column vector . The output of this operation is Based on the five common types of lane markers on the road [12][13][14], the number of contour lines ranges from zero to four. For example, the number of contour lines for marker S is two and D is four.  For detecting SD and DS lane markers, there is an additional step needed as the knowledge of is unable to tell apart between these lane markers. The two edges of the long lane marker are denoted as A and B respectively. As for the short lane marker, or the dash line, point C is denoted as the centroid at the central of the lane marker, as seen in Figure 3. In order to differentiate between SD and DS lane markers, two angles are measured, which are ∡ and ∡ , as seen in Figure 3. If ∡ > ∡ , then it is classified as an SD lane marker. If ∡ < ∡ , then it is classified as a DS lane marker.  [12] S, D, SS, SD, DS Contour analysis 94%-100% [50] S, D, SS, SD, DS, DD Semantic data 85%-90% Figure 3. The angle measurement for differentiating SD and DS lane markers The proposed classification method is implemented on the recorded video clips containing image frames of different lane markers as stated in Table 3. These video clips are recorded at 30 frames per second. The classification accuracy is measured by comparing the classified lane markers with the ground truth and the corresponding results are recorded in Table 3. After training the proposed classification mechanism with three video clips, it is then tested to run the classification operation on three other different video clips containing a mixture of different lane marker types. It can be observed from this table that the recorded accuracy values are all above 93%. An accuracy value of 100% is recorded in two different video clips, which are Clip1 and Clip2Test. The proposed method has also been tested with different datasets including those applied in [12][13][14]47].  Type  Frames  Train  Test  Accuracy  1  Clip1  1280x720  30  SS-DS  1099  x  100%  2  Clip2  1280x720  30  D-SS-SD  1840  x  98.04%  3  Clip3  1280x720  30  SS-S  357  x  93.84%  4  Clip1Test  1280x720  30  SS-DS-D  2019  x  97.17%  5  Clip2Test  1280x720  30  D-SD-SS  1287  x  100%  6 Clip3Test 1280x720 30 SS-D-SD 1547 x 97.54%

FUTURE WORK
Vision-based lane marking detection is expected to be an important future area of research in ADAS and intelligent transport systems [33,34]. Based on the review carried out and presented in this paper, there are a number of areas in which more work can be further continued, improved and addressed in the future, as summarized next:

Lane marker classification
Lane detection is no doubt an essential issue that must be addressed for ADAS. In all-weather conditions, the difficulty in detecting the lanes due to the distorted quality of the road and lane images needs to be addressed, as discussed and observed in the many cited references in this paper. However, as far as ADAS is concerned, classifying the types of lanes [12,14] is paramount to ensure that the important decisions such as lane changing or direction changing are correctly made. Although some lane detection algorithms in literature [32,37,40] have been demonstrated to detect some of the lane markers, the number of different types of lane markers detected is only between one to three types for all-weather conditions. The DS and SD lane markers for instance, have not been classified and addressed in the aforementioned schemes.

Rainy and cloudy weather
It is observed from our review that the research in lane marking detection for rainy days [19,28] receives less attention that that of the foggy days. One of the challenges in detecting the lanes in rainy days is the presence of rain droplets which renders the lane images noisy. Therefore, filtering is essential when implementing lane marking detection in rainy days. This is a challenge that should be addressed in the future. Furthermore, the lane marking detection in rainy days must also be implemented along with that of the cloudy weather. This is because cloud and rain always come together. A lane marking detection algorithm which is able to detect the lane and the markers in cloudy days should also be able to detect them in rainy days.

Lane marker and object identification
Classifying the lane markers is essential for regulating and maneuvering the vehicle. However, detecting the lane and the markers is typically affected by the objects found on the roads. Some of the objects spotted on the road image might be interpreted as part of the noise which may be neglected. But most the objects found on the road must be properly identified as they may pose a danger to the road users. These objects must be detected and identified to ensure the primary objective of lane marker detection, which is to ensure the safety of the road users, is achieved. This situation becomes more obvious and significant when the traffic level is increased where higher volumes of vehicles are found on the road. Therefore, lane marker classification must be integrated with object identification, which is an important problem to be addressed and solved in the future.
The need for a vigorous work of research in lane marking classification is high. Some of the research gaps highlighted here should be addressed to enhance the future ADAS and intelligent transport systems so as to ensure the safety and convenience of the road users and the public at large. The future direction of our research will be in lane marker classification in all-weather conditions.

CONCLUSION
The research in lane marking classification has been carried out for classifying different classes of lane markers including the solid, dashed and the combination of solid and dashed lane markers. Various classification methods have been developed which include, but not limited to, deep learning, temporal integration analysis, gradient & Bayesian classifier and so forth. The achievable accuracy values of detecting the lane markers are still considerably low and should be further improved. The proposed method using geometrical methods and contour analysis has been demonstrated to achieve the accuracy value over 90% for most of the lane marker types detected. This method is a good candidate to be further implemented and further improved for detecting the lane markers at different weather conditions and scenarios.