Digital image enhancement by brightness and contrast manipulation using Verilog hardware description language

ABSTRACT


INTRODUCTION
Environmental pollution has been persistently worsening day by day, and air pollution is the key factor for it. Air pollution does not just affect our health, but it can also limit our visibility. The foggy environment may cause digitally captured images to appear blurry, dim, or low in contrast and this will impact computer vision systems that rely on image information. With the need for real-time image information, such as a plate number recognition system, a simple yet effective image enhancement algorithm using a hardware implementation is very much needed to fulfil the need.
The field of image analysis research has undergone a rapid evolution over the past decade. Image processing nowadays has various applications in medical imaging, weather meteorology, machine learning, computer vision, and even artificial intelligence. The main objective of image processing is to improve the quality of the image for human interpretation and analysis [1]. Image enhancement techniques serve as a pre-processing step in various image processing applications, such as segmentation, object detection, and recognition (e.g., plate number recognition system). These techniques change an image to make it perfect for the human observer or to enhance the performance of an automatic computer algorithm [2]. The main objective is to prominence certain features of interest in an image for advanced analysis and image display. It 1347 is a method used to increase the visual quality of an image due to the nonideal acquisition method (e.g., low exposure or hazy images). The enhanced images are more suitable than the original image for a specific application [2]. For large image sizes with high resolution, running real-time image processing algorithms on serial processors proves to have more disadvantages. However, several efforts have been made to implement it on hardware, where more optimized and parallel algorithms are possible. The use of the field programmable gate array (FPGA) technology presents an appropriate alternative [3]. Recently, the size and features of FPGAs have been known to have great improvements. This has increased the interest in using it as an implementation platform for image processing applications, especially those that require real-time processing [3]. In addition to that, an image that suffers from contrast is usually caused by a haze. It limits the visibility of distant scenes, hence causing the image to look blurry [4]. Therefore, contrast enhancement of an image is important to remove the haze and to increase the visibility of an image. There are a lot of algorithms proposed by researchers to enhance images that suffer from low contrast and low exposure. Horé and Yadid-Pecht [5] proposed a general approximation for 2D filters using only power-of-two terms that produced a very good image.
Motivated by the aforementioned findings, this paper aims to improve image quality by introducing a simple yet effective enhancement method using both brightness and contrast adjustment. This is because low quality images will impact computer vision systems that rely on image information. For instance, in the computer vision system, its recognition method requires the extraction of certain features from an image. This is done at the pre-processing stage. If the image is suffered from low exposure and is hazy, the extracted feature does not represent the actual features of interest. Thus, lowering the recognition accuracy. To address this, many researchers proposed various image enhancement algorithms (software-based) to improve the quality of such images. With the need for a real-time image recognition system (e.g., plate number recognition), a hardware implementation is very much needed to fulfil the need. However, the software-based algorithm to hardware implementation is not an easy task. This is because the state-of-the-art enhancement algorithms are usually complex to make it infeasible to be implemented on hardware. Therefore, a less complex enhancement algorithm for hardware implementation by simply combining brightness and contrast manipulation is required to address this issue. The proposed method attempts to enhance the quality of low-exposure and hazy images. In the proposed image enhancement method, an algorithm was developed to control and manipulate the brightness and contrast of the image using hardware description language (HDL) Verilog code. The functionality of the design was verified using Xilinx ISE Design Suite 14.7 for timing waveform simulation. The proposed method can be used as a pre-processing step in a vehicle license plate recognition system. The aim is to enhance the low-exposure and hazy input images to aid the recognition algorithm. Hence, as a case study to evaluate the effectiveness of the proposed method, the target image for this paper is the vehicle registration number. This paper is organized as: section 2 describes the related work. Section 3 describes the proposed method in detail. Section 4 outlines the experimental setup. Section 5 discusses the simulation results, and finally, a conclusion is provided in section 6.

RELATED WORK
The research on the hardware implementation of image processing algorithms has gained much interest in recent years [6]- [10]. It has been the focus of many research papers. This interest is mainly due to the image processing algorithm that requires vast computational resources, especially when it is implemented in real-time processing [11]. Hardware implementation has become an attractive alternative solution for image processing due to the advancements in very-large-scale integration (VLSI) technologies. The goal is to achieve faster and more efficient processing of image processing [11]. As a result, a large number of hardware implementations for different applications have been proposed. Over the last five years, many studies have been published discussing hardware and software optimizations and implementation methods in this field [12]- [15].
The primary goal of an image enhancement method is to improve the brightness, contrast, and visual quality of an image for human viewing [16], [17]. There are lots of image enhancement applications, such as biomedical applications [18], and Haze Removal [4], [17]. Dong et al. [19] proposed a fast, efficient algorithm for the enhancement of low-light video. This paper achieved the enhancement by inverting the input with low-lighting video and applying an optimized image de-hazed algorithm to the inverted video. Another method by Ren et al. [20] uses low-light image enhancement algorithms that can improve the visual quality of an image by adjusting the illumination to be uniform and with strong intensity. Timarchi et al. [21] proposed a single image haze removal algorithm, which aims to improve the quality of the images and restore clear visibility. The proposed algorithm includes a detail enhancement algorithm in the pre-processing stage to restore the faded details of the input image. Weight maps were first calculated using dark channels to blend haze-free areas into the fused picture accurately and then normalized to avoid the out-of-range issue. Another application of the image enhancement method that uses the de-hazing algorithm was proposed by Soma and Jatoth [4]. Two methods for hardware implementation of the image de-hazing algorithm were discussed in the paper. The authors proposed the pixel-wise and grey image-based de-hazing algorithm. The hazy input image was converted into two different images, which are a grey image and a minimum image. These two images were then converted into an average image, which significantly reduces the halos and artifacts present in the final dehazed images. All the aforementioned algorithms here are very good algorithms, especially for reducing the halo and haze in their sample images. Moreover, those algorithms proposed by the authors have produced a great enhanced image that is very detailed as compared to the original image. However, the aim of this paper is to propose a simple yet effective enhancement method by manipulating the brightness and contrast such that the proposed method can be implemented on hardware.
Kehar et al. [22] implemented an image de-hazing model using a metaheuristics-based brightness channel prior. The method involves the evaluation of image brightness, atmospheric light, and transmission map estimation, which then will be applied to the restoration model. Li et al. [23] proposed an image enhancement based on a dark channel prior algorithm in machine vision. A new approach improves the atmospheric light value and transmittance and converts the recovered image to hue, saturation, value (HSV) space. Second, the adaptive stretching algorithm improves the saturation while the multiscale retinex with color restoration (MSRCR) technique improves the brightness component and bilateral filtering improves the saturation component. To finish image enhancement, the image is changed from HSV space to red, green, blue (RGB) space. Ganguly et al. [24] proposed a single image de-hazing method by cascading two models. Atmospheric light and transmission map is estimated for the first model, and haze map estimation for the second model. The hazy image has then been restored by least-square optimization on the haze map.
As the use of vehicles in our daily lives is growing rapidly, and as more vehicles violate traffic regulations, theft of vehicles, illegally accessing restricted areas, and a significant number of accidents occur, crime rates rise linearly [25]. In this fast-paced environment, license plate detection will be critical for any vehicle to be recognized. Recently, many papers [26]- [28] have been published on the implementation of vehicle plate recognition systems using efficient algorithms. However, if the input image enhancement could be implemented on hardware, the recognition speed would be improved to achieve real-time plate recognition. Therefore, as a case study to evaluate the effectiveness of our proposed method, the target image will be the vehicle plate number.

PROPOSED METHOD
Some of the simplest, yet useful, image processing operations in the spatial domain involve adjusting brightness or contrast in an image. A reason for manipulating these attributes is the need to compensate for difficulties in image acquisition. Image processing can increase the overall brightness of the object of interest and magnify the tiny residual variations in contrast across it. These image processing operations can reveal enough detail to allow proper interpretation. Figure 1 illustrates the internal design architecture of the proposed method. The first step of this method is to convert the input image file in bitmap format into a hexadecimal file. This is because the Verilog HDL cannot read a file that is not in American Standard Code for Information Interchange (ASCII) format. To convert the bitmap image into hexadecimal, MATLAB is used. Each pixel value (red, green, and blue) in the image is converted into a hexadecimal number and stored in the input image memory block as shown in Figure 1. Then, the brightness and contrast adjuster sub-module manipulate each fetched pixel value from the memory block (i.e., based on the pointed address) before storing the result (using the same address) into the output image memory block (that will eventually be converted into a bitmap format). This address is determined by using the input column and row (i.e., col and row in Figure 1, respectively). These values are obtained from the column (Col) and row counter sub-module that is controlled by a finite state machine (FSM). The state diagram for this FSM is shown in Figure 2.

System architecture and protocol
Based on Figure 1, the top-level design (TOP_Module) consists of 8 sub-modules, namely FSM, vertical synchronize (VSYNC) and horizontal synchronize (HSYNC) counter, data counter, Col and row counter, comparator, brightness and contrast adjuster, input image memory, and output image memory. The main clock frequency is set at 50 MHz. This clock is used by the FSM and the counters (i.e., data counter, Col and row counter and VSYNC and HSYNC counter) sub-modules. Initially, after the system is reset, the FSM waits for the input signal RUN to be activated before starting the process. Once started, the FSM waits for some vertical and horizontal delays (i.e., vertical delay is used for start-up time and horizontal delay is used for the delay between HSYNC pulses) before running the brightness and contrast manipulation (refer to  Figure 2). The delay is counted by the VSYNC and HSYNC counters sub-modules. To run the brightness and contrast manipulation, the Col and row counter sub-module is used to provide the current target for each column and row of the image. At the same time, the data counter sub-module is used to count the number of pixels that have been processed to be fed into the comparator sub-module such that the input "data_cnt" can be compared to the total number of pixels in the image. If all the pixels have been processed, an internal signal "done" is activated to tell the system that the task is completed.  Table 1 shows the system specifications for the designed architecture. The architecture is coded using Verilog HDL using Xilinx ISE Design Suite 14.7. The clock period of this system is set to 20 ns. The input file format is in hexadecimal format that is created earlier using MATLAB. The output file format from this system is in bitmap image format with a size of 1.125 megabytes (MB). Since the dimensions of the input image are 768×512 pixels, it will create an input file with 1,179,648 lines of data thus a temporary storage of the data needs to be in a size of 1,179,648 data array. The proposed algorithms (i.e., Algorithm 1 and Algorithm 2) in this proposed method is the combination of brightness and contrast manipulation to enhance the image that suffered from low exposure and haze. All the input data from the memory will be processed by the proposed algorithm that is resides in the brightness and contrast adjuster sub-module. Next, the following sub-sections (i.e., sub-sections 3.2 and 3.3) will discuss the algorithm used in the aforementioned sub-module, which combines brightness and contrast manipulation.

Brightness manipulation
The brightness of an image can be adjusted by adding a constant value ( ) to each of the RGB fields of an image. Saturated mathematical calculations are required for this operation. For that, it is good to make sure no values go below 0 or above the maximum allowed in the image bit-depth (8-bit for 24-bit color), which in this project is 255 (8-bit color).

Contrast manipulation
The technique of increasing the contrast range by assigning the deepest pixel value to black. The brightest value to white. The rest to linearly interpolated shades of grey make good use of the display and improves the visibility of visual components.

Algorithm 2. Contrast adjustment
Step 1. Calculate a contrast correction factor ( ), which is given by the following formula.

Performance evaluation method
In order to quantitatively evaluate the performance of the proposed method, peak signal-to-noise ratio (PSNR) and mean-square error (MSE) are used. The PSNR is an objective criterion used to evaluate the fidelity of an image [23]. The PSNR and the MSE represent overall error content in the entire output image. They are well-known performance metrics for assessing the degree of inaccuracy [4]. In order to assess the quality of the hazy/foggy images, measures like MSE and PSNR are frequently used [29].
The logarithmic ratio of peak signal energy (P) to MSE between output ( ) and original ( ) images is defined as PSNR. It can be stated as (10) and (11): where P is the highest pixel value in an image (often P=255), MSE is the MSE, and and are the number of rows and columns of pixels of the image, respectively. In general, a high PSNR number and a lower MSE value would be desirable. MATLAB will be used to perform the image quality metrics evaluation in this paper. All the input and output images will be evaluated in terms of PSNR and MSE.

EXPERIMENTAL SETUP
The proposed method aims to enhance low exposure and hazy images (in terms of quality) by using Algorithm 1 and Algorithm 2, which manipulate brightness and contrast, respectively. The Xilinx ISE Design Suite software development tool was utilized to build the proposed architecture. Specifically, this tool was used to design and verify the designed architecture by using hardware description languages (i.e., Verilog). Since the target image for this experiment is the vehicle registration number, some 100 images of a vehicle license plate with a 768×618-pixel resolution were collected. To mimic a low exposure and hazy scenario, all the collected images have been manipulated to decrease their brightness and haze is added to the image using Adobe Lightroom. The exposure of the image is set to -4.00, which will decrease the image's brightness and to add the haze effect to the image, the dehaze value is set to -100. These values are used across the original images in the dataset to create the input images. Figure 3 shows one of the original input images from the dataset and its output images for all the combinations of and . Figure 3(a) shows the original image, Figure 3(b) shows the input image that has been manipulated with brightness decrement and added haze white, while Figure 3(c) to (v) are the output images. At the beginning, all the input images are converted into a hexadecimal file (.hex) (i.e., using MATLAB) before being enhanced using the proposed method. The proposed method was tested on all 100 images. For brightness and contrast manipulation, the constant value ( ) in Algorithm 1 and the contrast correction factor ( ) in Algorithm 2 are varied from 70 to 110 with an increment of 10 and from 2 to 5 with an increment of 1, respectively. This will create a total of 20 possible images for each 100 images. It is done to determine which combination of and that will give a higher PSNR and lower MSE value.
All the 100 images are manipulated to produce 20 output images. A total of 2,000 output images were evaluated in terms of PSNR and MSE to observe which combination of the value of and give the highest PSNR and lowest PSNR value. The best combination for most of the 100 tested images will then be compared to the other image enhancement methods proposed by Dong et al. [19] and Ren et al. [20]. MATLAB will be used to execute the Dong and Ren methods. For this comparison, ten random images will be selected, and will be evaluated quantitatively and qualitatively. For quantitative evaluation, PSNR and MSE evaluation metrics will be used. For qualitative evaluation, a Google Forms is handed out to the public consisting of three output images from each method. Table 2 shows the PSNR and MSE values for the input image in Figure 3(b) and each output images from Figure 3(c) to Figure 3(v). According to the table, it is observed that that the combination of = 80 and = 3 (image h) gives the highest value of PSNR and lowest MSE value. Hence, for this particular image, this combination of and produced better image when compared to other combinations. The same step is repeated for the rest of the dataset.

RESULTS AND DISCUSSION
In this section, we evaluated our proposed method on our private dataset that consists of 100 images. The images collected for this dataset are images of vehicle registration numbers. The simulation result is obtained to verify the functionality of the proposed architecture. It is then followed by the quantitative and qualitative evaluation to see the performance of the proposed method.

Simulation result
The functionality of the proposed design was verified using the ISIM simulator from ISE Xilinx as illustrated in Figure 4. The figure shows the timing diagram of the system that processes one row of pixels. This is because the system is designed in such a way that it processes the data row by row, from the bottom row of an image to the top row. The timing diagram shows the DATA state where the data_run is high, and the system starts to process the input data (i.e., iR, iG and iB) and store the processed data (i.e., oR, oG and oB) in the output image memory. Each input pixel value of the RGB color channel is different in values. Hence if the neighboring pixels of the same channel have the same pixel value, the input and output data will look the same on the corresponding channel for more than one clock cycle (e.g., 8a pixel value of iR in the labelled figure).  Table 3 shows the results obtained for quantitative evaluation and the best combination of and across all images in the dataset. The best combination of all the images was summarized in the bar chart as shown in Figure 5. Based on Figure 5(a) and 5(b), majority of the images (i.e., 46) produce a desirable value of PSNR and MSE when the value of was set to 90 and set to 4. However, 27 out of 100 images produced better images when the value of was set to 100 and set to 5. For value = 100 and = 4 combination, only three images produce a better image. This may result from the original image being already dark, hence needing a higher value of to increase the brightness of the image. For value = 80 and = 3 combination, only five images produce a better image as compared to the other combinations. This is due to the original image being already bright, hence needing a lower addition of brightness. The remaining images (i.e., 13) produce a better image when the value of was set to 90 and set to 3 and six images when the combination was set to value = 90 and = 5. Based on these results, the best combination of and is 90 and 4, respectively. Hence, for the comparison with the other methods by Dong et al. [19] and Ren et al. [20], ten random images will be selected with the combination of = 90 and = 4. All output images from both methods will be compared with the proposed method quantitatively and qualitatively. Figure 6 shows the sample output images produced by the proposed method with Figure 6(a) input image with low brightness and added haze, Figure 6(b) result of the proposed method, Figure 6(c) result of Dong, Figure 6(d) result of Ren, and Figure 6(e) original image. Each of the outputs will be compared to the original image quantitatively and qualitatively. Based on Table 4, from ten images that are compared quantitatively to other methods, the proposed method produces five images that have the highest PSNR and the lowest MSE value. In comparison, the method by Dong produces three images that are better in terms of PSNR and MSE value. Lastly, a method by Ren produces only two images that are better than the other methods.

Qualitative evaluation
Despite the fact that the previous evaluation methods focused on the performance of the enhancement and restoration methods, human eyesight still plays a role in determining which method is the most effective. When comparing the output of these methods, the human eye cannot detect the minor discrepancies that can be observed. However, when these disparities increase, it will be easier for the human eye to determine which is superior. For the qualitative evaluation, a total of 70 respondents participated in the survey. In the survey, respondents were asked to choose an image that was closest to the original image. Based on Table 4, for images 19, 75, 82 and 87, the majority of the respondents chose output from Dong [19]. While for images 46 and 85, the majority of the respondents chose images from the proposed method. Lastly, images 45, 53, 60 and 67, most respondents chose image from the Ren method.
Despite that our proposed method produces better images quantitatively, but our proposed method only produces two better images when it is compared to the other methods qualitatively. This may be caused by every human having a different interpretation when comparing the output images to the original image. Human incapability to detect every small difference in terms of pixel values leads to the respondents choosing the image subjectively based on their personal preference.

CONCLUSION
In this paper, a combination of brightness and contrast manipulation is presented to enhance images that suffered from low exposure and haze. Our results have a more natural and realistic appearance, as well as better brightness and contrast. In quantitative comparisons, the proposed method outperforms the other two methods. Since the proposed method is coded using HDL, it is feasible for this algorithm to be implemented on hardware as an image enhancement tool. Possible future work for this project could include hardware prototyping on FPGA. This is because image processing algorithms are excellent choices for FPGA implementation. The FPGA is optimized to exploit hardware parallelism and simplify operations. The programmability of FPGAs allows a faster and cheaper design cycle of DSP systems. Another interesting aspect is optimizing the implementations for lower resource usage. Finally, the prototype can be deployed as an application specific integrated circuit (ASIC) to eventually be used in many hardware applications related to low exposure and hazy images. Obviously, the deployment stage will also require a lot of research works.