A computer vision-based lane detection technique using gradient threshold and hue-lightness-saturation value for an autonomous vehicle

Automatic lane detection for driver assistance is a significant component in developing advanced driver assistance systems and high-level application frameworks since it contributes to driver and pedestrian safety on roads and highways. However, due to several limitations that lane detection systems must rectify, such as the uncertainties of lane patterns, perspective consequences, limited visibility of lane lines, dark spots, complex background, illuminance, and light reflections, it remains a challenging task. The proposed method employs vision-based technologies to determine the lane boundary lines. We devised a system for correctly identifying lane lines on a homogeneous road surface. Lane line detection relies heavily on the gradient and hue lightness saturation (HLS) thresholding which detects the lane line in binary images. The lanes are shown, and a sliding window searching method is used to estimate the color lane. The proposed system achieved 96% accuracy in detecting lane lines on the different roads, and its performance was assessed using data from several road image databases under various illumination circumstances. results testing all environmental


INTRODUCTION
Driver error or a delayed reaction time are responsible for more than 70% of documented traffic accidents in Great Britain [1]. In order to prevent road accidents and promote safe driving, a number of researchers are currently creating intelligent vehicles. In recent years along with autonomous vehicles, computer vision has been introduced in several places, such as fraud detection [2], forensic system [3], home security systems [4], [5] accident findings [6]. As a consequence of technological developments, various mechanisms have been created to inform drivers of potential lane departures or accidents, lane layout, and the placement of additional vehicles in ways. Because of the significant signals it displays in the surroundings of intelligent vehicles, lane detection on the roadways has become a crucial challenge. When the input image becomes noisy due to environmental instability, locating the lane lines becomes harder.
Sensors, cameras, and lidar have been used in recent lane detecting experiments. Chen et al. [7] suggested a multi-sensor lane-keeping system that developed road surface lanes using a roadmap and a global positioning system (GPS) receiver. The authors implemented a model for detecting lanes in a real-time environment by combining lidar with multiple cameras. One of the primary advantages of these systems is that the lidar or sensors take data directly from the environment, and the modules are not weather-dependent. However, GPS sensitivity is only ten to fifteen meters, and lidar prices are high, which are essential drawbacks of the techniques.
Such cameras capable of acquiring an image in any direction have become increasingly common as computer vision-based technologies have received considerable attention. On the Caltech lane data, the approaches mentioned in [8] produced the desired results in terms of vision-based analysis. They focused on creating algorithms that could recognize three-lane intersections. Inverse perspective mapping (IPM) eliminates the perspective effect, image filtering reduces noise by constructing candidate lanes, and lane model fitting locates lines on road pictures. Furthermore, this technique prioritizes model fitting above system stability and efficiency. In terms of recognizing lanes that are not entirely visible, the solutions provided thus far have several limitations. Specific solutions rely on edge detection [9], which is hampered by the lane-like noise caused by light reflection. Furthermore, the color-based approaches presented in [10] worked effectively when the light changed fast. While optical flow may address some of the issues raised by the other techniques [11], it is computationally demanding and unlikely to function on the road with no texture.
In addition, using the vanishing point theory, several techniques for lane identification on roadways have been developed. Li et al. [12] used Gabor filters with texture inversion to identify lanes. They employed an adaptive soft voting approach to estimate a vanishing point and represent the texture orientation's confidence rating. Khac et al. [13] proposed a technique for evaluating the efficacy of lane identification on both structured and unstructured roadways based on a vanishing point estimate. One of these methodologies' significant flaws is that when the vanishing point's position changes, the vanishing point cannot be accurately predicted. Many computer vision experiments on lane detection have been proposed in recent years. Several organizations have put much work into developing an effective lane detection and turn tracking system. The research that has been done in this area has been discussed in the following section.
Jiao et al. [14] created a real-time lane identification system based on a region of interest (ROI) that can function in noisy conditions and react quickly. A Kalman filter and a least-squares approximation of linear movement were used to track lines. The technology recognizes and tracks the lanes. Fan and Dahnoun [15] introduced a lane detection and classification system based on the stereo vision idea, which is used in advanced driver assistance systems (ADAS). They developed a model that leveraged the concept of ROI to detect lanes and a convolution neural network (CNN) trained on the KITTI dataset to classify lanes as right or left. However, the algorithm could not discern the lanes due to noise in the disparity output.
To construct a lane detection and warning system, Wu et al. [16] specified an area of interest (ROI) in the neighborhood of the vehicle. Extracting the gradients and angles of ROI image pixels, the ROI is divided into non-overlapping portions, generating two essential masks, and making the computation process easier. The departure system is built based on the lane detection results, and four different driving conditions are created. According to the testing results, the average rate of lane detection is 96.12%, while the average rate of departure warning is 98.60%. On the other hand, the computation of the vertical and lateral gradients takes much time.
Youjin et al. [17] proposed a lane detecting method based on vanishing point estimation. The algorithm then identified the vanishing positions of the lane segments using a statistical voting technique. The actual lane segments were chosen by altering the vanishing point and line direction parameters. A real-time inter-frame similarities technique for estimating the lane detection rate was also designed, significantly reducing the false detection rate. A real-time evaluation technique was proposed due to the absence of expressive fluctuation in the lane geometry parameters. The method, however, cannot be used on streets with unusual forms.
In a large number of curving lanes, Cho et al. [18] created a technique for identifying vanishing points. The method assumes the vanishing point in non-flat roadway conditions by combining disparity data with a lane marking scheme. It is feasible to minimize unnecessary information about obstructions by comparing the actual and fitted dispersion values. Outliers also have an impact on the prediction of vanishing points (PVP) estimate using the least squares method. False detection can sometimes occur due to choosing a plus-minus peak value.
Zhang et al. [19] created a lane marking system based on binary blob analysis. The method uses vanishing point detection and IPM to minimize the roadways' perspective impact; the methodologies of binary blob filtering and blob verification are explored to improve the lane detection strategy's efficacy. According to the platform's results, the multi-lane dataset has an average detection performance of 97.7%. However, in a real-time environment, the method did not work smoothly. Das et al. [20] mentioned a lane-marking system that relies on spatiotemporal images collected during video playback. Pixels mined along a horizontal scan line with a fixed position in each frame and a time axis are combined to create the spatiotemporal view. The Hough transform is applied to the accumulated pictures to identify lanes. The technique is incredibly successful for noise sources that are only present for a small space of time, such as misplaced lanes or vehicle obstructions. The system was more efficient and had a greater detection rate in terms of processing. Ying and Li [21] suggested a lane detection method based on IPM. The input picture is converted to a binary image using the adaptive threshold approach, and the lane marker candidates are chosen using predetermined lane templates. Random sample consensus (RANSAC) was employed to reduce outliers, while the Kalman filter was utilized to track lanes on road surfaces.
Using the ridge feature and IPM, Fritsch et al. [22] established a model for detecting multiple lanes. The restored features contain four local maxima since the same lane is distributed with a few changes across the x-axis in the IPM picture. By aggregating the lane attributes at each indigenous supreme point, clusteringbased techniques are applied to find lanes.
We presented a computer vision-based approach for effectively distinguishing lanes in any surrounding environment in this experiment. For lane detection, we generally employed gradient and hue lightness saturation (HLS) thresholding values. After thresholding, a perspective transform was used to assure correct mapping. The remainder of the essay is divided into the following sections: the method of the suggested lane-detecting approach is shown and explained in section 2. The experimental findings are shown in section 3 using input and output. The paper concludes with its limitations, accuracy rate and future works in section 4.

METHOD
The proposed lane-detection approach is shown and explained in this section. Figure 1 depicts the whole operation of the proposed system. The proposed technique is divided into seven main blocks, they are input image, image pre-processing, cropping, thresholding, perspective transform, sliding window search and the lane line illustration. The proposed system has used 975 images from the various roads as datasets in different road structures, various light illuminations, and weather conditions [23]. Each level is fully described in the section with proper mathematical equations and diagrams.

Preprocessing
For effective lane line marking, preprocessing is necessary. The main goals of preprocessing are to improve contrast, remove noise, and create an edge image for the related input images. A crisp picture restores line straightness, making lane limits easier to perceive. A noticeable separation can be seen between the twisted and undistorted photos. Straight lines have taken the place of curved lines. OpenCV is used to determine the camera matrices and distortion factors for chessboard pictures. It is possible to extract the image's corners and utilize that data to fix the distortion. In the last example, we used 2D coordinates, but in this one, we will use 3D coordinates (with the z-axis or depth equal to zero for our chessboard images). Using these maps, one may be able to learn how to remove distortion more precisely from images. When the

Cropped image
Cropping is a technique for eliminating unwanted areas from an image. We just need to concentrate on the regions where lanes are likely to be detected when selecting lane lines. As a result, the cropping step is completed, and subsequent image processing is limited to the predetermined portions of the picture. Figures 3(a)-3(b) shows a cropped picture with a focus on the lanes. Figure 3(b) shows a cropped picture that highlights a specific region of the lanes from Figure 3(a), the preprocessed input image. The picture has been reduced in size. This enhances the performance of the process.

Thresholding
Thresholding [24] is a typical image segmentation technique. This is a technique of segmenting images that involves converting grayscale images to binary images and then separating image elements. For images with a high contrast stage, image thresholding is especially well suited.
Where T stands for the threshold value, (a, b) for the threshold value's positional coordinates, and p (a, b) and f (a, b) for the grayscale picture pixels (a, b). The Sobel operator uses two 3×3 gradient Karnal which are convolved with the original image to calculate approximations of the derivativesone for horizontal changes (x axis), and one for vertical (y axis) [25]. Because the lane lines are almost certainly vertical, the y-axis slope is given additional weight. Scaling correctly necessitates taking into account both total and normalized slope values. There are a few algorithms along with the Sobel filter, such as the canny edge detector, Robert operator, Prewitt operator, and Laplacian of Gaussian algorithm helps to find out the edge of the input image. The Laplacian of Gaussian is very sensitive to noise and could decrease the accuracy of the proposed system. The canny edge technique has good localization, and it extracts image features without altering the image features, but it is also responsive to image noise due to illumination and false zero crossing [26]. However, using the Sobel method it is very easy to find the smooth edge on the image and using the gradient threshold value enhances the accuracy of its  Figure 4 shows the result of using the gradient threshold method where Figure 4(a) shows the cropped image. When the color of the road is either too bright or too light, the HLS color channel [20], [25] is performed to compensate. The threshold of the L (lightness) channel works by decreasing the number of frame borders produced by shadows. The S (saturation) channel's threshold rises, allowing white and yellow lanes to be included. H is pointing in the direction of the line's color (hue). The result of using gradient thresholding is shown in Figure 4(b) and HLS thresholding is shown in Figure 4(c). For the final binary picture thresholding, we integrated gradient and HLS (color) thresholding in a single phase, which considerably enhances the overall outcome of the lane identification process. Figures 5(a) and 5(b) show the gradient threshold and HLS threshold, and Figure 5(c) illustrates the combined effect of gradient and HLS thresholding were.

Perspective transform
A mathematical process called the perspective transformation [27] turns a three-dimensional picture of the world into a two-dimensional representation. Although undistortion and thresholding assist hide critical information, we may isolate it even further by studying a section of the road surface images. We shift our viewpoint to the appropriate downward perspective for the street to concentrate the image on the road segment. While this step provides no more information, it substantially simplifies the process of dividing lane lines and considering things like curvature. The combined thresholding picture's perspective transform is demonstrated in Figure 6. In Figures 6(a) and 6(b), the combined threshold and perspective transform is shown.

Search image via sliding window
Since the lane lines were initially anticipated in a previous frame, the information is applied to detect and track lane lines from the bottom to the top of the picture in a sliding window centered on the line centers [28]. Figure 7(a) shows the perspective transform of the input image, and Figure 7(b) shows the outcome of the sliding window search. This enables us to conduct a highly focused search while reducing execution time is significant. We utilize their x and y pixel coordinates to fit a second-order polynomials model to identify the pixels along the left and right lane lines: To match the lines, windows re-center themselves at the regular image pixel. The dark lines in the original picture will be relocated to their original places. In Figure 8(a), the sliding window search image and Figure 8(b), the depicted lanes of the sliding window search image.
To conform towards the representation of lane lines, the picture is twisted and cropped. A transformation matrix operation of 3×3 was used to distort the image. After the alteration, straight lines will remain that way. Select four places on the input picture and their corresponding points on the output image to view the transformation matrix. Three of the four requirements should contradict each other. Cropping the photos is crucial because we only need to concentrate on locations where lanes are likely to exist while figuring out how to find them. The lane illustration shown in Figure 9(a). The distorted and cropped image's outcome is shown in Figure 9(b).
Finally, we combine all this information and overlay it on top of the original image. The lane marked region is in the green-colored area. While this pipeline is designed to analyze a single image, it can easily be expanded to scan multiple images to detect the lane lines on the road surface. Cropped and warped picture shown in Figure 10(a). Figure 10

Setup for experiments
The system setup is a vital part of computer vision-based operations. The accuracy rate, processing time and validations depend on the system setup. The proposed system has been designed to run on a generalized computerized system. The proposed method is run on a desktop computer with an Intel Core i5-2430M CPU operating at 2.40 GHz ×4 with Windows 8 operating system and the OpenCV library in Python. PyCharm is a cross-platform integrated programming environment that we employed to accomplish our goals.

Results analysis
This section describes the experimental findings, which were validated in heavy traffic and in the dark using a large number of images from the various road image datasets. The result of lane detection using shape and color cues is shown in Figure 11. The lanes that have been detected are shown by the green rectangle drawn on the output image. As a result of shadows created by the road surface or broken lanes, several unsuccessful efforts have happened. Due to road shadow, Figure 11 shows a little discrepancy in the alignment of recognized lane lines and road lane lines. Furthermore, since the width of the lane swings unexpectedly in the spatiotemporal picture as the pitch angle changes, the proposed technique fails to meet the criteria for temporal consistency. The suggested technique produces pleasing results with sharp curvature, lane shifts, hindrances, and lens flashes since it relies on the temporal constancy of lane width on each scan line. The experimental findings show that the proposed approach correctly detects the lane under a variety of environmental situations, including changeable weather and natural lighting. In order to measure the system's strength, 975 frames from the input dataset are used to determine the correctness of the method. There are 936 right classifications out of 975 frames, with nine erroneous classifications. However, the proposed system shows some false results on rain and fog-based roads, which has decreased the accuracy of the system. Table 1 shows the performance of the suggested system. The procedure is accurate to within 96% and has a false positive rate of 0.64%. In Table 2, we have discussed and compared our work with similar research works. The cooperative analysis shows the proposed method is more efficient than other traditional lane detection methods [25], [28] in Table 2. Besides that, it also worked more accurately than some deep learning-based lane detection algorithms [29]- [31]. The accuracy of the proposed method could be developed in future by increasing computerized features and applying deep learning algorithms along with the proposed method.  The system works efficiently with 96% accuracy and the false positive rate is 0.640%. The processing time of the proposed system is 204.55 (Avg. f/s). In future, the accuracy and the efficiency of the system could be developed by increasing the computer feature and applying deep learning algorithms along with the proposed method. The concept is applicable to any road with well-defined lines and may be included in an embedded system to assist ADAS. In the future, a real-time system with hardware implementation will be developed that will picture a real-time scenario and, using the proposed technique, would detect lanes and provide warnings to drivers about road lane scenarios. By making the automatic discussion making, the proposed system could increase the autonomous vehicle features in level 5.