Illumination-invariant vegetation detection for a vision sensor-based agricultural applications

Received Mar 17, 2020 Revised Jun 10, 2020 Accepted Sep 29, 2020 In this paper, we propose a novel method, illumination-invariant vegetation detection (IVD), to improve many aspects of agriculture for vision-based autonomous machines or robots. The proposed method derives new color feature functions from simultaneously modeling the spectral properties of the color camera and scene illumination. An experiment in which an image sample dataset was acquired under nature illumination, including various intensities, weather conditions, shadows and reflections, was performed. The results show that the proposed method (IVD) yields the highest performance with the lowest error and standard deviation and is superior to six typical methods. Our method has multiple strengths, including computational simplicity and uniformly high-accuracy image segmentation.


INTRODUCTION
In the case of vision-based agricultural application system, a crucial issue is to accurately segment only the vegetation part from its background image. In this process, a major challenge is how to overcome the wide range of natural illumination conditions, which strongly affect the crop images in typical outdoor settings because of specular reflections, shadows over the vegetation and weather conditions. Previous studies have been researched a variety of methods for crop detection. All of these methods performed well in a simple environment with normal illumination and soil conditions. Previous studies are consist of two broad categories [1].
The color index based approach used color spaces (RGB, HSV) to distinguish crops and backgrounds through simple formulas. The learning-based approach used the learning method of crop training samples to demonstrate adaptability to changing illumination. In a color-index based approach, the excess green index (ExG) [2] was a method using green color component extraction more important for green crop extraction. The excess green-excess red index (ExGR) [3] took advantage of the high success rate of green crop extraction by suppressing red components by subtracting the ExR index [4]. Similarly, the normalized difference index (NDI) [5] used both green and red color components and then improved performance through local growth processes. The color indices of vegetation extraction (CIVE) [6] and vegetation indices (VEG) [7] used all three color components, and each RGB color to emphasize the role of the green component to select the optimal channel combination for the space. The result of the combination was then converted into a gray image. And in the ExG, NDI, and CIVE methods, crop segmentation was obtained using the Otsu [8] method with a fixed threshold. For the VEG method, the binarization process is performed by the mean value. All of the above color index methods required a fixed threshold, but this fixed threshold method was difficult to generate an appropriate threshold for crop extraction due to illumination changes. This problem results in less crop detection because the green channel is less pronounced than the other two channels in a single image. These methods can only be applied in limited circumstances.
In the learning-based method, there were two stages: training of the crop color data and classification. In the EASA method [9,10], the training process begins with clustering and ends with a Bayesian classifier, where the classification process is completed through a lookup table. The AP-HI approach [11] was based on the assumption that the histogram of hue under a certain intensity is similar to a Gaussian distribution curve. The training process was to calculate the mean and standard deviation of each hue level and then build a corresponding lookup table (LUT). The classification process is performed by checking the discriminating function through the LUT. The artificial neural network method [12] was used to classify crops and backgrounds through learning using the mean shift algorithm and back propagation artificial neural network (BPNN). Guo [13] used decision trees and image noise reduction filters for crop extraction, and Montalvo [14] applied a support vector machine (SVM) for crop identification. Bai [15] used a clustering method based on vegetation segmentation based on particle swarm optimization (PSO) clustering and morphology modeling in CIELAB color space. In addition, various researches on crop row detection using vision cameras and image processing have been conducted [16][17][18][19][20][21][22][23][24][25][26][27][28][29][30].
All these methods are able to adapt to a certain degree of change in lighting. However, these performances depend on the size of the training data to handle different lighting characteristics. Since lighting changes always occur without rules and regulations, training samples are limited and classification results are not guaranteed, especially when highlight takes place [1]. These methods are not robust to segment vegetation from background when images contain specular reflections, shadowed areas and illumination changes. In this paper, we proposed a novel vegetation detection method for vision-based agricultural applications in typical outdoor settings. Our proposed method is intended to be an object segmentation method that automatically processes images regardless of the nature of the illumination conditions without a threshold adjustment for each image, and the method can be applied in real time.

RESEARCH METHOD 2.1. Constraints of environmental changes
Vision camera-based agricultural applications in paddy field has a lot of constraints because it is operated in an outdoor environment for a long time. Reasons of these constraints classify largely two criteria: illumination and morphological variation. The images of cameras taken in outdoor environments are affected by various illumination conditions due to the weather being clear, cloudy, rainy, or severely climatic. Sivalogewaran [31] wrote that this problem is that shadows and diffused reflection can create a scene with a wide range that can lead to saturation or underexposure or high intensity of parts of a scene. Especially, the paddy field is consist of water and muddy, therefore the spectral composition of the illuminant varies are generated more significantly. This variation can occur between shadow areas and indirect illumination in the same scene. Even larger variations occur in scenes that are indirectly illuminated at different times of the day or at different times of the day. These variations make it very difficult to use color or chromaticity information to segment interesting objects within a scene. Although modern camera devices can perform automatic image correction through automatic image white balance or enhancement, but most times this are lacking in industrial cameras that require dynamic adjustment of most camera settings (exposure time, automatic image white balance, or pre-fixed by the user (focal length, iris aperture) [31]. Figure 1 shows the results of crop detection using Otsu, ExG, ExGR, CIVE and VEG methods on paddy filed image. The results show that a lot of image noise was generated in the crop detection results due to various illumination reflections on the field. As shown in Figure 1, if we try to segment crop parts using typical previous method from different illumination intensity in an image, image segmentation is not good result because it observe area of strong reflection and shadow.

Illumination invariant-based vegetation detection (IVD)
An image is formed when light from an illumination source is reflected from various objects in a scene into an array of photodetectors. As shown in Figure 2, the illumination response of an image sensor, R , in an outdoor scene can typically be represent as follows [31]: where G is a geometry function that represent the direction of the light source and the direction of the surface normal, the intensity of the illumination source is I , the reflection coefficient of the objects is ()  , because it is assumed that the spectral sensitivity function is sufficiently narrow. Therefore, (1) can be simplified as follows: Furthermore, if the photodetector is ideally sensitive to light only at a wavelength i  , it can be assumed that ( ) 1 i E   , which yields the following response function: () Taking the logarithm of both sides of (3) can be separated. Because the photodetector responses of each crop and background is independent, the equation can be rewritten as follows: As shown in (5) is subtracted from (4) to obtain an illuminant-independent feature function: As shown in (6) can be represented according to peak spectral responses each sensor channel as follows: We also want to remove the background influence due to segmenting only crops in the image. The background of tillage is nearly soil or water-soaked soil [32]; therefore, the reflection coefficients of the peak spectral responses for each sensor channel and responses are approximately the same and can be represent 1287 using (7)(8)(9). As shown in (7), (8) and (9)  , respectively, to remove the background terms. Then, (8) is subtracted from (7) and (9) The feature F is constant and a new one-dimensional color space consisting of three responses 1 2 3 , , R R R corresponding to the peak sensitivities at the ordered wavelength 1 2 3 , ,    . This feature can be made independent of background variations such changing illumination, reflection and shadow. The reason for this independence is that the feature is a function of only the RGB color and reflection coefficient of the crop. Consequently, it is possible to obtain a uniformly high-quality segmented image during times of changing illumination. Moreover, this feature decreases the disturbances caused by shadow or reflection on a background. In this study, a new color space image based on feature F is proposed, and the crop image is segmented by applying the Otsu method [8] to the new color space image.
In order to eliminate noise from the acquired image, a second median filter was applied. The binary image of the rice row can be determined by applying the Otsu method, which is one of the active threshold methods commonly used for image segmentation. Furthermore, such as in (11), simple image processing, can eliminate small-magnitude noise generated by the reflection of strong illumination on the water-soaked soil surface. This process determines a binary value of 0 in the image pixels when there is an occupied area of adjacent pixels. (i.e., ( , ) ) with connectivity to each pixel is below an arbitrary threshold (i.e., δ).  Moreover, six other commonly used algorithms were applied for comparison. They are the Otsu, ExG, ExGR, CIVE, ExR and VEG methods. Previous studies have focused on the role of the green component and used a combination of channels from RGB color space, as shown in Table 1. These color feature methods are based on the assumption that vegetation and background pixels can be distinctly separated from each image using a predetermined threshold. These methods are not powerful to segment vegetation in the background image if the image contains specular reflections, shadow areas, and illumination changes. In addition, the ground-truth images were segmented manually using the Photoshop software package (Adobe®). To analyze the quantitative performance of proposed method, a performance index to quantify the misclassification error of the different methods was applied. It is defined as follows:

Error
Ground truth image Test image   (12) As shown in (12) means subtracting one image from the ground-truth image of the same size. Figure 3 and Table 1 show that proposed method (IVD) yields the highest performance of 5.85×10 4 error pixels with the lowest standard deviation of 0.97×10 3 pixels, which is better than the others. In Figures 3 and 4, we can observe that the IVD method is generally more accurate than all of the other methods. Especially in the case of large variation in illumination, the IVD method exhibits superior image segmentation results (i.e., it uniformly segmented pixels of crop leaf from background) compared with the Otsu, ExGR, CIVE and ExG methods.  The Otsu method determines the threshold dynamically according to only the gray level of an image; consequently, the image segmentation results were uniform and low-quality, and the algorithm did not remove partial shadows or reflections. The ExG and VEG methods yielded blurry crop leaf areas because these methods emphasize only the green channel among the RGB channels. Moreover, the ExGR method identified very little crop leaf overall because the equation cannot distinctly segment crop pixels and background. The CIVE and ExR methods were observed to have a relatively normal performance. Nevertheless, the image segmentation results were uniform. The proposed method, the IVD method, exhibits high accuracy with uniform image quality according to varying illumination.

Weather condition: rainy
The experiments performed to verify the performance of the crop row detection under rainy condition. The experiment site was a real paddy in Daejeon, South Korea (36.37N, 127.36E); amount of rainfall was 35.2 mm (The Korea Meteorological Administration announced). The camera was mounted on a bracket approximately 1.5m above the paddy. The growth day of rice was about 90 day. The capturing device used was a low-cost color web-camera (Samsung®, SPC-A800M), and captured the image samples (150 images) during a rainy. And each sample images has been photographed from different angles. Moreover, six other commonly used algorithms were also applied for comparison. They are the Otsu, ExG, ExGR, CIVE, ExR and VEG methods. In addition, the ground-truth images were segmented manually using the Photoshop software package (Adobe®). To analyze the quantitative performance of proposed method as before, a performance index to quantify the misclassification error of the different methods was applied with (12). Table 2 show that proposed method (IVD) yields the highest performance of 1.74×10 4 error pixels with the lowest standard deviation of 4.28×10 3 pixels, which is better than the others. In the plot of Figure 5 and Figure 6, we can observe that the IVD method is generally more accurate than all of the other methods. Since a rainy day is normally cloudy, instead of the illumination change is less, it is low luminous environment. Thus, the CIVE, ExGR, and ExR methods yielded the image segmentation results were lowquality that it cannot distinctly segment crop pixels and background. Whereas, the ExG, Otsu and VEG methods yielded blurry crop leaf areas because these methods emphasize only the green channel among the RGB channels. Especially in case of Otsu method, it was remained the shape of rainwater dripping on background image.

CONCLUSION
In this paper, we proposed a novel vegetation detection method for vision-based agricultural applications in typical outdoor settings and verified the superior performance of method in comparison with six different methods. Our method has some strengths, including low computational cost, computational simplicity and uniformly high-accuracy image segmentation. In the future, it will be necessary to apply various crops for verifying versatile applications of the proposed method.