A new approachto image classification based on adeep multiclass AdaBoosting ensemble

Received Apr 20, 2019 Revised Mar 11, 2020 Accepted Mar 30, 2020 In recent years, deep learning methods have been developed in order to solve the problems. These methods were effective in solving complex problems. Convolution is one of the learning methods. This method is applied in classifying and processing of images as well. Hybrid methods are another multi-component machine learning method. These methods are categorized into independent and dependent types. Ada-Boosting algorithm is one of these methods. Today, the classification of images has many applications. So far, several algorithms have been presented for binary and multi-class classification. Most of the above-mentioned methods have a high dependence on the data. The present study intends to use a combination of deep learning methods and associated hybrid methods to classify the images. It is presumed that this method is able to reduce the error rate in images classification. The proposed algorithm consists of the Ada-Boosting hybrid method and bi-layer convolutional learning method. The proposed method was analyzed after it was implemented on a multi-class Mnist data set and displayed the result of the error rate reduction. The results of this study indicate that the error rate of the proposed method is less than Ada-Boosting and convolution methods. Also, the network has more stability compared to the other methods.


INTRODUCTION
In recent years, deep learning methods have gained a lot of attention and were used in many types of research. In deep learning, first, the nonlinear features of several layers are extracted [1]. Then, they are transferred to a classifier, and finally, they are sent to a combiner layer, in order to perform the combination and prediction. The more hierarchy of layers is (deeper), the more nonlinear features are obtained and better results are presented. These methods are used to solve various complex problems [2]. One of these networks is the convolution method, which has many applications in image processing [3]. This network was originally designed to process images, and various convolution-based models have been developed to classify the images.
On the other hand, hybrid methods are common machine learning techniques. These methods combine different predictions to provide better and more accurate problem-solving results [4]. These methods are used for solving classification problems and have applied in different classification problems in order to provide various predictions [5].
No algorithms are optimal in all areas. Every learning algorithm is limited to a specific model. In fact, if data assumptions are not true, errors will occur [6]. If [2]. Ensemble classifiers are among the multicomponent classifiers defined to produce better results than a single-component classifier [7]. In such classifications, ensemble classifiers are used to obtain better results. Ensemble methods vary in how they create different classifiers and how they combine basic classifiers with respect to their weights [8].
In fact, there are two ensemble frameworks: dependent (serial) and independent (parallel) [9]. In a dependent framework, the output of a classifier is used in the next classifier. Therefore, the knowledge of previous iterations can be used to direct learning in next iterations [10]. Boosting is an instance of such frameworks. In the second framework (known as independent), every classifier is created independently. Their outputs are combined with those of voting methods [11].
So far, many techniques have been proposed for classifying the images. Some of these classifications are bi-class and some others are multi-class [8]. It should be noted that multi-class classifications have many applications. Creating a robust model in line with network stability, reducing classification error and decreasing data dependency in the model are some of image classification issues [12].
The present study proposes a new model based on deep learning methods and hybrid method to improve and reduce the error rate of multi-class image classification. On the other hand, combining these techniques results in higher stability, compared to previous methods. This hybrid boosting method is a bi-layer convolution related hybrid method. As you know, the goal of boosting is to strengthen and improve poor learning, which attempts to strengthen its learning and make the model stronger. In this study, the bi-layer convolution technique was considered a weak learning method.

BACKGROUND
In the past, deep learning networks were used to solve different problems such as image classification, object recognition, image extraction, etc. [13].These networks employed various methods for optimization. Convolutional neural networks (CNNs) are among the most important deep learning networks possessing several layers and serving as a very powerful method for machine vision optimization. These networks consist of three layers named convolutional, pooling, and fully connected. CNNs have been used for image classification. A deep learning model was introduced by Daniel Frits et al. for image classification. This model combines a deep network with PixelRNN and DCGRAN models for image recognition. These models were implemented in PixelRNN and DCGRAN for handwritten data [14]. Gan Project was carried out for image classification, too. It is used in a k-class categorization [15]. Another classification project included the auxiliary deep generative models (ADGMs). ADGMs consist of various sets of encoders and decoders. This project was executed on the MINIST dataset. According to the results, AtlasRBF algorithm produced 1.5 times more convergence than deep generative models, virtual adversarial, and ladder methods [16][17][18].
A major approach to deep networks is to create different layers for learning the features [19,20]. An instance of such networks is an eight-layered network designed by Krizhevsky et al. [21]. It is a deep network architecture based on the convolution of a deep eight-layered network. Given the success of this network, it is used widely to solve many problems such as video classification [22], facial recognition, and action detection [20]. Other relevant projects include the use of a convolution a weak learner in the ensemble AdaBoosting algorithm [23].Weak learner is a learner that no matter what the distribution over the training data is will always do better than chance, when it tries to label the data. This means that the learner algorithm is always going to learn something, not always completely accurate. The combination of a convolutional network and AdaBoosting method has also been used in numeration. The aim of this paper was to propose a model based on the combination of AdaBoosting method and a two-layered convolutional network to produce better results.

RESEARCH METHOD
In the proposed approach, ensemble methods and deep leaning were employed to improve image classification. For this purpose, a convolutional neural network was combined with the AdaBoosting method. Every ensemble problem consists of four sections: a training set, a basic learner, a driver, and a composer [10]. Likewise, the proposed method includes the following section: a. Training set:A training set contains labelled samples used for training. In this study, the MINIST standard dataset was employed for image classification. b. Basic learner: A basic learner is a learning algorithm used for learning a training set. In this project, the basic learner was used to implement deep learning. For this purpose, a convolutional neural network was used as a learning algorithm. However, the CNN consisted two convolutional functions in every  Figure 1 shows a schematic view of the proposed method for combination. The AdaBoosting network was created and combined with the convolutional neural network to obtain a powerful learner from weak convolutional learners. Accordingly, the convolutional network consists of three layers: convolutional, pooling, and fully connected. The fully connected layer is responsible for computing the classification [23,24]. In the proposed algorithm, two deep convolutional layers of every class are responsible for weak learning. The multiclass boosting network is defined in the following way [13].
In this structure,X is all data in Class M also every xirefers to a class number of a label number of an M-classification. F(X) is trained globally and developed in the following way [9].
On the other hand, the multiclass boosting method divides the learning process into several subclasses to improve learning. Therefore, gi: X→RD shows training functions.
In this function, t and α show the frequency and correlation coefficient in each iteration, respectively. In this method, GD-MCBoost was employed to improve learning [13]. Considering the learning classification, the proposed algorithm used a minimization policy in which a descending gradient was employed [15]. Therefore, the minimum weak learner is selected in the following way: The (6) indicates the function used for minimization [7].

RESULTS AND ANALYSIS
The proposed algorithm was implemented on the MINIST dataset in MATLAB for evaluation [25]. The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems [26]. The database is also widely used for training and testing in the field of machine learning [27,28]. It was created by "re-mixing" the samples from NIST's original datasets. Furthermore, the black and white images from NIST were normalized to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels. The MNIST database contains 60,000 training images and 10,000 testing images [29]. Section 4.1 to 4.4 shows the testing results.In these sections,the CNN, AdaBoosting method, and the proposed ensemble multiclass convolutional AdaBoosting method were evaluated to compare the results. On the other hand, every test was evaluated with different iterations to analyze the results better. Figure 2 indicates the process of learning the AdaBoosting algorithm for different iterations. Accordingly, the error rate decreased when the number of iterations increased. In this algorithm, the sample space was divided into classes. Then the learning process was carried out on those classes in the AdaBoosting method. Naturally, the number of iterations and length of learning increased. In other words, there were 100, 500, and 2000 processes in 10, 50, and 200 iterations.The learning cycle find the set of parameters to optimize the error function. In the end, the model and parameters are with the smallest error.

Single-layered CNN
A CNN consists of convolutional, pooling, and fully connected layers [30]. Therefore, a comparison should be made between this method and the proposed method. The CNN was evaluated on the MINIST dataset. Figure 3 shows the results of the CNN. Figure 2 indicates the CNN learning process in 10, 50, and 200 iterations. Accordingly, the CNN is slower than the AdaBoosting method.

Two-layered CNN
Considering the use of two-layered CNNs in weak learning, a convolutional deep learning network was formed. The classification results of that network were evaluated to produce better results. Figure 4 indicates the results of that network.

Deep multiclass AdaBoosting
The proposed method was anensemble of AdaBoosting and convolutional methods. The sample space was divided into ten units in combination with the convolutional method. Figure 5 shows the results of learning process in different iterations.

Results and discussion
In order to evaluate the proposed method, the image classification was conducted on the Mnist data set with different algorithms. The methods used for implementing these experiments were presented in sections 4.1 to 4.4. The mono-and bi-layer convolution methods were carried out to evaluate the AdaBoosting methods. The error rates were determined based on the number of repetitions. The results of these experiments are shown in Table 1. For example, in 500 repetitions, the error rate of Ada-boosting, bilater convolution, and deep Ada-boosting are 0.008, 0.0089, and 0.0067 respectively. On the other hand, if the number of repetitions increased then a better convergence can be achieved in the proposed method.According to the results, the image classification error rate in the proposed method was 8% better than the previous method and had better stability as well. Table 1 shows the error rates in three different situations. Figure 6 shows a comparison drawn between the proposed algorithm and other methods.

CONCLUSION
Deep learning is now used for solving various complex problems. Convolution is one of the deep learning methods, which is used for classifying images. The hybrid methods are considered as a multi-component learning method that intends to strengthen the learning process by using combined approaches. Boosting is one of these methods. This study seeks to find a method for optimizing and reducing errors when classifying images. So far, various methods and models have been presented for this classification. The proposed method is based on a combination of boosting method and deep convolution technique. Boosting has a set of basic learning that improves and strengthens the poor learning in each step. In the proposed method, the bi-layer convolution network is considered as basic learning. This method was implemented using MATLAB software and evaluation was carried out using handwritten data. In this algorithm, the Mnist data set was applied. After evaluating the results of the proposed method, it was analyzed using convolution and boosting methods. The results indicated that in 500 repetitions, the error rate was decreased by 8%. On the other hand, since the boosting method was used, the proposed model has higher stability.In the future, it is possible to improve the proposed algorithm by strengthening the basic algorithm and creating some connections in the weighting process.