Real-Time Video Processing using Contour Numbers and Angles for Non-urban Road Marker Classification

Received May 9, 2018 Revised Jun 20, 2018 Accepted Jul 11, 2018 Road users make vital decisions to safely maneuver their vehicles based on the road markers, which need to be correctly classified. The road markers classification is significantly important especially for the autonomous car technology. The current problems of extensive processing time and relatively lower average accuracy when classifying up to five types of road markers are addressed in this paper. Two novel real time video processing methods are proposed by extracting two formulated features namely the contour number, , and angle, θ to classify the road markers. Initially, the camera position is calibrated to obtain the best Field of View (FOV) for identifying a customized Region of Interest (ROI). An adaptive smoothing algorithm is performed on the ROI before the contours of the road markers and the corresponding two features are determined. It is observed that the achievable accuracy of the proposed methods at several non-urban road scenarios is approximately 96% and the processing time per frame is significantly reduced when the video resolution increases as compared to that of the existing approach. Keyword:


INTRODUCTION
The number of road accidents in Malaysia increases at an alarming rate of 9.7% every year over the past thirty years [1] contributed by (road users) errors, faulty road environment, and vehicle defects. The road marker, is very important in guiding road users while driving. Different types of road markers in non-urban roads such as double solid (DD), dashed (D), solid-dashed (SD), dashed-solid and single solid (SS), allow drivers to safely decide either to maintain the course in the middle of the lane or to overtake the front vehicles. In earlier works, many efforts were invested in solving lane detection and tracking problems for Auto-Assist Driving System (ADS) [2], [3]. These methods had been used as a warning system to the driver [4]- [8] and surprisingly, the lane detection also was used as part of the system to analyse the driver behaviour [9]. Symbols, alphabets, and custom markers were also being explored and used as the information to alert drivers [10,11].
However, road marker classification [12], [13] still remains an open question due to the varying types of road markers across the globe with limited effort being put to solve the problem in road marker classification. Collado et al. [14] proposed a frequency analysis method for the urban road marker classification, which uses the inverse perspective mapping (IPM) to generate a bird-eye view of the road and produce a modified Hough Transform (HT) for better lane detection. The method then applies power  [15] studied lane changing based on road marker recognition surrounding traffic situation but the road markers classified were limited to only; dashed and solid lines. In a work by Lindner et al. [16], an edge detector technique was designed to subsequently search for a group of four objects namely the lines, curves, parallel curves and closed objects in detecting road markers. Although four types of road markers were classified, most of the road markers are dashed lines with different sizes. Meanwhile, another research from Suchitra et al. classified three road markers namely; dashed, solid and zigzag using a modular approach [17]. The road markers are classified as either dashed or solid using the Basic Lane Marking (BLM), which is based on continuity properties. This approach however, applies the temporal information in the classification operation, which renders slower detection when the road marker type changes whilst driving on the road. In addition to these methods, Nedevschi et al. [18] uses a periodic histogram to determine the type of road markers. This ego-localization is observed to enable the classification of road markers into four types namely single solid, double solid, dashed, and merged.
In a recent research, Paula et al. [19] presented an automatic classification technique to classify five types of road markers. The approach uses between three to five features extracted from the image, which is later fed to the artificial neural network. A two-stage method is modelled for the full classification, with the first stage applies the Bayesian classifier for dashed, single dashed and double solid while the second stage differentiates between dashed-solid and solid-dashed lines for road marker detection. However, the results of the classification were found to have changed abruptly on each frame, which caused inconsistent results while driving. Mathibela et al. [20] proposed a new approach using a unique set of geometric features which functions within a probabilistic RUSBoost and Conditional Random Field (CRF) network to classify the road markers into seven types, including; single boundary, double boundary, separator, zig-zag, intersection, boxed junction and special lanes. Even though more types of road markers were classified using this method, unfortunately, the classification only used static images in urban roads.
On the basis of the comprehensive literature review, the extensive classification of road markers had been rarely studied. It is interesting to look at the aforementioned different approaches for road marker classification although no standard databases available to allow fair comparison. In addition, the dimension, colour and size of the markers are varied across the world.
This paper proposes a novel approach to classify these road using a customized Region of Interest (ROI) in a video acquired from a camera with its calibrated position. Two features are derived, namely; the contour number, , and the contour angle, , are later fed into a two-layer classifier. The first layer classifies the D and SS marker types based calculated , values, while the second layer classifies the DD, DS or SD marker types, by using values as shown in Figure 1. Temporal information integration is applied to improve the classification accuracy by validating the marker type's transitions on the road. This algorithm has been demonstrated an overall accuracy of approximately 95% and reduced reduction in the processing time by ~50% shorter than the existing method.

RESEARCH METHOD 2.1. Camera setup for region of interest selection
Many existing methods for selecting the ROI such as the vanishing point, area-based detection and area-based tracking methods had been used in the past. A technique of calculating the vanishing point using Hough Transform (HT) is carried out by obtaining the intersection line and the ROI at the lower half of the image [21]. In [22], [23], the ROI is selected over the whole horizontal axis and a limited range along the vertical axis, before the feature extraction and road marker or lane classification take place.
In this project, a new method in selecting the ROI based on its calibrated camera's height and Field of View (FOV) is proposed. For the initial setup, a camera with a resolution of 1280x720 located at the center of the car is calibrated on its position with its FOV adjusted towards the planar road surface captured by the camera as shown in Figure 2 The image captured is then divided into 40 equally sized sections or squares, each of which having a size of 160x144 pixels as shown in Figure 3(a) and Figure 3(b). The ROI selection method is applied to every video frame to choose (x,y) as the ROI, which contains the road marker, as shown in Figure 3(c). It can be observed that (x,y) is the nearest to the car where the effects of the road curve on the road marker is low. In the case of lane departure, the ROI will contain no marker for a longer duration which indicates that the vehicle is departed from the right track. In our approach, r 1 , containing the RGB information, is converted to greyscale and filtered through Gaussian filter [24] before undergoing the thresholding process using Otsu Thresholding method [25] for binary conversion. Then the two features, which are the contour number and the angles, will be extracted to complete the classification as discussed next.

The number of contours, , and the angle in Method A and B
A contour is a line that connects all points along the boundary of an object of the same colour. The contour number, in the ROI is calculated by counting the number of all contours along its horizontal axis. The calculation of is carried out in the ROIs, (x,y), for every frames in a complete cycle, which is defined as the time taken by all consecutive frames containing a complete road marker pattern. This pattern will repeat itself after every complete cycle until the road marker type changes or ends as in Figure 4.
The proposed road marker classification is based on the contour number, , and the contour angle, , implemented using two methods of two-layer classification as depicted in Figure 5. The formulation of and for both of these methods, which are named as Method A and Method B, will be presented next.
In Method A, road marker D is detected when the minimum over the cycle is zero and the maximum is one. If over a complete cycle, then the road marker is SS. Otherwise, if the maximum over the cycle is two, then the second layer classification using the angle of the centroids will be run to classify the marker as SD, DS or DD. The angle θ, which is measured between a line connecting the two centroids and the horizontal axis, is calculated only on the first frame in a set of frames showing a particular road marker pattern that repeats itself after the last frame in the set before the pattern changes. This can be identified from the change in value, which is from 10 or 12. There are three threshold angles and , which are determined from training dataset, applied for the classification. If , then the road marker is DS. If , then the road marker is SD and if or , then the road marker is DD.  In Method B, the significant different is when detecting DD. Instead of detecting DD at the second layer using the angle , DD is detected at the first layer based on , when it is equal to two over a complete cycle. Hence, only one threshold angle, , used to detect DS and SD. In order to find the contour angle, as seen in Figure 6 (d) and formulated as follows: Two centroids, and , are calculated from the moments of the markers, , , surrounded by the contours as shown in Figure 6(a) and Figure 6(b). The red line represents the contours and the red dots denote the centroids. The moments are calculated as ∑ , where represents the region in the contour (the red lines) [26]. In the next section, the threshold angles will be determined to classify the road marker as DD, DS or SD (for Method A), where =1,2,3 for Method A and only is used for Method B.

The threshold angles formulation for the second-layer classification in method A and B
In Method A, the threshold angles are formulated based on the angles between two centroids of each of the 300 frames extracted from four video clips (Vid22_USB, Vid24_USB, Vid26_USB, Vid27_USB) as in Table 2. The angles are then plotted accordingly in Figure 7. As for DD, the values are found to be distributed between the negative angles (denoted as DDlower) and the positive angles (denoted as DDupper) due to the change in the location of the two centroids shifting along a curvy road. The angles for the 1 st frames in every set of frames for DS, SD and DD are found to be separated with each other, as seen in Figure 7. It is clear from this figure that the threshold angles, are the separating angles or lines between four groups of angles where for Method A. The three threshold angles, and which are represented by the blue horizontal lines in Figure 7, are calculated such that the angles or values are separated or classified intro three types; namely DD, SD and DS. Hence, these threshold values the midpoint between the maximum angle of a marker type and the minimum angle of the next neighbouring marker type. The minimum and maximum angles of DDupper, SD, DS and DDlower are stored as the first row (minimum angles) and the the second row (maximum angles) of matrix respectively. Based on the values recorded in Figure 7, the corresponding matrix is given below.
where the threshold angles are calculated using the following formula (2) with °, and , as shown in Table 1 (a) for making the classification decision. As for Method B, only applied for the classification decision, as shown in

Resolving errors in classification
Errors in road marker classification tend to happen during the marker transition period. The frame delay time might also cause errors in classification. Both of these cases can be solved by using the temporal information between the classification results during the transition period. The markers tend to be similarly labelled for a set of adjacent frames. Knowing this tendency, the previous set of classification result will be stored and if there is a transition of marker types, it will wait on the next set of frames classification results for a final validation to avoid unnecessary classification errors.

RESULTS AND ANALYSIS
The proposed algorithm was tested with multiple video sequences captured by two different devices with different car velocities using a lower range USB camera (Logitech c310) and a higher range camera (GoPro Hero Silver). To evaluate the robustness of the new algorithm, other sets of videos acquired with different devices at different resolutions and the velocity of the vehicle is between 60-90 km/h for the test video images, which meets the speed limit in the suburban road area. The videos are also taken when the vehicle was moving on different road conditions including flat, hilly and narrow roads. The videos used to determine the threshold angles were taken with a moderate vehicle speed between 50-60 km. The experiments were carried out using a desktop computer equipped with Intel® core ™ i7-4790 CPU @360GHz and 16 GB RAM. The proposed system was implemented by using the Visual Studio C++ 2010 compiler with the library from OpenCV version 2.4.11.
In the classification process, if , the software will classify it as either D or SS depending on found in the next set of frames. For example, if the next set of frames detected with , the system will classify it as a D types whereas if consistently detected over the complete cycle, the system will classify it as SS types. If the maximum value is two over the complete cycle, the road marker will be classified as either DS, SD or DD, by using the value calculated at the 1 st frame from the set of frames. The classification result will remain and will only be reclassified if , changes from 21. For the case of DD type which expects consistently, a sampling up to its 10th frames will be performed when only two contours are detected and classification result will be updated by using the value on its 11 th frame.
The accuracy of the marker classification is calculated using Equation (3). The total sets of frames classified correctly, Ʃ⍴, will be divided by the total sets of frames in the video image Ʃξ, excluding the total number of 1 st sets of frames when the road marker transition happens, Ʃτ1. The exclusion of Ʃτ1 is due to the road marker transition in which the angle calculation is temporarily paused. The clips and the accuracy results are shown in Table 2.
As shown in Table 1, clips 4 to 7 were trained for identifying . Clips 9 and 10 are recorded when the vehicle velocity is between 60 and 80 km/h. The remaining test clips contain at least two types of road markers with different resolutions and frame rates. It can be observed that the accuracy achieved by Method B is better compared to method A as only one threshold angle is used. The processing time per frame for Method B is slightly lower on average as compared with Method A. It can also be seen in Table 1 that different types of video compression formats like .wmv renders a longer processing time, as shown for clip 6 and 7. Our method is also tested against the existing approach presented by Paula at [19] using a few videos with different resolutions. The resulted average processing time per frame is recorded in Table 3. Our approach has been observed to perform better at larger resolutions. This is contributed mainly by the smaller ROI applied in our approach, besides the two formulated features namely the contour number and angle used to classify the road marker types.

CONCLUSION
A novel approach on the road markers classification process is presented in this paper. A smaller ROI is used to process the video frames and extract the formulated features, which are the contour number and angle, as the inputs for the proposed two-layer classifier algorithm. The average accuracy of ~96% has been achieved using our proposed algorithm, with a relatively lower processing time per frame at large video resolution as compared with the existing approach. The errors found in the road marker classification are caused by the abrupt changes in illumination, vanishing road marker, white elements on the road and markers blocking by the other vehicles. Future work will focus on the illumination and faint marker issues that affect the features extraction for classifying the road marker types.