An algorithm for characterizing skin moles using image processing and machine learning

Received Jul 20, 2020 Revised Dec 12, 2020 Accepted Dec 22, 2020 Melanoma, the most serious type of skin cancer, forms in cells (melanocytes) that produce melanin, the pigment that gives color to the skin. There are lowincome regions that lack specialized dermatologists, causing skin cancer to be diagnosed in advanced stages. In Peru, in high Andean communities with low resources, the problem is aggravated by the high incidence of ultraviolet radiation and lack of medical resources to make the diagnosis. Normally, mole images are obtained from dermatoscopes. The present work seeks to use mole images obtained from smartphones to make the classification of them as suspected or not suspected of being melanoma, by means of a feature extraction algorithm. The first step is to make color and lighting corrections. After this, the image is segmented using the K-Means algorithm, and we obtain the areas of the mole and skin. With the segmented mole we proceed to extract the main visual characteristics and then use classification algorithms such as support vector machine (SVM), random forest and naïve bayes, which obtained an accuracy of 0.9473, 0.7368 and 0.6842, respectively. These results show that it is possible to use images obtained from smartphones to develop a classification algorithm with 94.73% accuracy to detect melanoma in skin moles.


INTRODUCTION
Skin diseases are the fourth leading cause of the global disease burden, affecting 30-70% of individuals and prevailing in all geographical regions and age groups [1]. The most common form of cancer in the United States is skin cancer, with about 5 million cases occurring annually [2][3][4]. Melanoma is the most dangerous type of skin cancer, causing more than 9.000 deaths per year [2,3]. Although most melanomas are first discovered by patients [5], the diagnostic accuracy of unaided expert visual inspection is only 60% [6].
Melanoma is a cancer of the melanocytes and can develop as a new mole, or as part of a pre-existing mole [7]. According to the American Cancer Society, melanoma is less common than some other types of skin cancer, although it is more likely to grow and spread [8]. However, if detected in its early stage it can be treated effectively. For this reason, it is important to recognize the warning signs to watch for. ABCDE screening, according to Rigel [9] for early detection of this type of cancer, is a simple skin inspection method that allows the first signs to be determined so that one knows when to go to the dermatologist. This method, according to the American Academy of Dermatology, consists of checking the mole, with A being for asymmetry, B for the observation of borders, C for color, D for diameter and finally E for the evolution of the mole over time.
In this context, new algorithms have been developed in artificial intelligence for the classification of images [10], which have been applied to identify different diseases in the skin. These techniques are applied to dermatoscopic images [11,12], which have high resolution and detail. L. Yu et al. in [13], developed a two-part algorithm; the first part is segmentation of the dermoscopic image and the second is classification. The first part uses a fully convolutional neural network (FCRN) for precise segmentation of skin lesions, and further enhances its capabilities by incorporating a multi-scale contextual information integration scheme and the second part uses a classification network. They processed 900 training and 350 test images, and obtained a 0.799 accuracy and a 0.844 segmentation Jaccard index.
Nasr-Esfahani et al. in [14], expose a classification algorithm using a data pre-processing, enhanced images are fed into a pre-trained convolutional neural network (CNN), which is a member of deep learning models. The CNN classifier, which is trained by a large number of samples, distinguishes between melanoma and benign cases. For 6120 images they obtain a sensitivity of 0.82, a specificity of 0.87 and an accuracy of 0.81.
In addition to the deep learning techniques, it is also possible to extract useful features from the images and use machine learning algorithms to perform the classification, in Mustafa et al. [15] it is based on the ABCDE features of the moles to perform the characterization of the images and in the end it uses an SVM classifier for the identification of the melanoma obtaining an accuracy of 86.67%.
In Murugan et al. [16], they compare classification algorithms for the detection of skin cancer. The algorithms used are SVM, random forest and kNN classifier. For feature extraction they are also based on the ABCD rule, obtaining an accuracy of 89.43%, 76.87% and 69.54% for SVM, random forest and kNN classifiers respectively.
Arasi [17] proposes to use naive bayesian and decision tree techniques for the classification of images obtained from a dermatoscope, and to identify melanoma. This technique is also based on the use of the ABCD rule. Using only the texture characteristics in the analysis of the images, 84.5% accuracy is obtained when using the naive bayes classifier and 77.4% when using decision tree.
In all the works cited above, images obtained from a dermatoscope have been used to make the identification of melanoma in the skin. In this work we propose the possibility of using other sources of images, in particular we will use images obtained from smartphones, which have been captured by nurses in health centers in Peru. With this dataset we propose to extract characteristics of the mole based on the ABCD rule and use machine learning techniques such as SVM, random forest and naive bayes to make the identification of the melanoma. Machine learning was chosen instead of deep learning because in the articles cited it is concluded that its accuracy is similar to the deep learning methods and also has less computational load when processing.

RESEARCH METHOD
In this article we propose an algorithm to obtain the characteristics according to ABCD of a skin mole, in order to determine whether this mole is suspected of being a melanoma or not. As can be seen, the evolution of the mole over time is not taken into account. The block diagram of the proposed algorithm is shown in Figure 1.

Acquisition of the images
The images of the moles were captured with smartphones that were available at the health centers, the models were: Samsung S7 (resolution 12 MP), Samsung S6 (resolution 16 MP), ZTE Blade (resolution 15.9 MP) and LG K10 (resolution 13 MP). In each image a marker is added that has a size of 25x15 mm as shown in

Selection of the area of interest
Because the entire image does not need to be processed, the area of interest in the image obtained initially, including the marker, is selected and then a perspective correction is performed [19]. A bilinear transformation is applied to the image to make the correction. Let A be the original color image and select 4 coordinates ( 1 , 1 ), ( 2 , 2 ), ( 3 , 3 ) , ( 4 , 4 ), as shown in Figure 3 To the selected coordinates, the change of perspective is applied and 4 new coordinates are obtained ( 1 , 1 ), ( 2 , 2 ), ( 3 , 3 ) , ( 4 , 4 ) in the resulting picture R, as shown in Figure 3(b). In order to perform the perspective correction [19], a transformation matrix of 4x2 is defined, which transforms the coordinates of the original image and obtains the new desired coordinates as defined in (1).
The transformation matrix has 8 unknowns to solve that would be 11 , 12 , 13 , 14 , 21 , 22 , 23 , 24 and each pair of points produces 2 equations, so selecting 4 points on the image is enough to solve the equation. To the obtained image that has a dimension of 300x300 pixels, another correction of perspective is made having as reference the squares that have the marker that was added to the image. It is known that each square measures 3 3 mm, Figure 4(a) shows the image with that marker. The equivalence made is 1 mm equal to 20 pixels and using this correspondence the size of the final image is calculated using (2) and (3), obtaining the image that has a dimension of × pixels, shown in Figure 4(b).

Image enhancement
At this stage, an image intensity level correction is performed [20], because when the image is obtained, it may have different levels of illumination, which is not uniform. The marker in the image is used for this correction, as the black square will represent intensity level 0 and the white square in the marker will represent intensity level 255.
where, and , are the intensity of the black marker of the image and the intensity of the white marker of the image respectively. Since when applying (4) the resulting image can have intensities greater than 255 or less than zero, an adjustment is made by applying (5) and the final corrected image is obtained.
In Figure 5(a), the image is shown and in Figure 5(b) the enhanced image is shown. This lighting correction process is important to extract the characteristics of the mole correctly. The image is then cropped to just have the mole, without the initial marker. The resulting image is shown in Figure 6.

Segmentation
To the image, the segmentation is made by means of the iterative K-Means grouping method [21], which consists of grouping each pixel according to the smallest distance d, from its centroid , and then recalculating its centroid, this process is repeated until the difference of the centroids is less than 0.001. The flowchart of the K-means segmentation is shown in Figure 7. For this segmentation 2 classes were defined, one class for the mole and one class for the skin. Therefore, the value of k can be, = 1, 2; which results in the image shown in Figure 8. To have the two regions, you assign the value of 1, to the pixels that belong to the mole group; and you assign 0 to the pixels that belong to the skin group. This is assigned in (6), resulting in the image, shown in Figure 9(a).
Once the segmentation in the image is obtained with the K-Means algorithm, morphological operations are applied to correct, eliminate imperfections and close possible holes. This process is performed in (7), (8), (9) and (10).
where, is the structural element shown in (11). Figure 9(b) shows the obtained image . Obtaining the image, it will serve as a mask to be able to segment the object of interest, which is the mole, this is obtained from the image, this is done by applying (12). The resulting image is shown in Figure 10(c).

Feature extraction
The proposal made for the analysis of the mole is the extraction of characteristics according to the ABCD method used by the dermatologist for the classification of suspected or non-suspected melanoma. The image of the segmented mole is the image used to perform the feature extraction. The asymmetry will be obtained, then the irregularity of the edges, color and finally the diameter.

Asymmetry
To obtain this characteristic, first the shape of the mole is approximated to an ellipse [22] as shown in Figure 11(a). Once the approximation is obtained, the main axes of the ellipse and the quadrilateral that circumscribes the ellipse are located as shown in Figure 11(b). After this, the image is rotated by taking the circumscribed quadrilateral as shown in Figure 11(c) and finally the image of the mole is cut into four equal parts as shown in Figure 11(d).
For the analysis of the asymmetry a similarity comparison is made of the 4 sub-images obtained from the division of the image, having the upper right side image defined as , the upper left side image defined as , the lower right side image defined as and the lower left side image defined as . These images are shown in Figure 11(d). To analyse the similarity, the logical operation of OR-EXCLUSIVE is performed by applying (13) to each sub-image with its side and bottom and top sides as appropriate.
The images resulting from this operation are 1 , 2 , 3 , y 4 ; which are the differences that exist between each of the sub-images as shown in Figure 12. From each of these images its area of difference is obtained by adding the number of pixels and dividing by , which represents the total size of the image,  (14). These differentiation values will also be used for edge analysis. From the asymmetry analysis, 4 descriptors will be obtained.

Border
The second characteristic obtained is the irregularity of the edges that could exist, for this we analyze the edge of the mole. The same procedure as Sancen-Plaza et al. in [23] is followed with the area obtained in the process of application (10) shown in Figure 10(b). From this process, 9 descriptors of the degree of irregularity of the border are obtained, according to Santiago-Montero et al. [24].

Color
We obtain the mean and variance of each of the color components that make up the images , , y in the RGB, HSV, La*b* and YCrCb color models. Each of the mean and variance values obtained in each sub-image is subtracted with those of the other sub-images in each color model, according to (17) and (18). In (15) is shown the way to calculate the mean for each of the components where represents the value of each pixel, e l and m are the rows and columns respectively, the total number of pixels and in (16) is shown the standard deviation for each of the components.
With the mean and variance in each of the color models we have 12 descriptors that when applied to the 4 models makes a total of 48 descriptors. Additionally, Haralick's textural parameters are used as color characteristics in the RGB color model. According to [25], 14 second order statistical variables are calculated, which describe properties such as contrast, energy, entropy, local uniformity, maximum probability, hue, importance, and correlation to the image. Finally, for the color characteristic there are a total of 62 descriptors.

Diameter
From the image , we obtain the longest straight distance [26], for this we obtain the distance between each of the points of the edge according to (17) where and represent any two points of the edge with coordinates ( , ) and ( , ) respectively, then we compare the distances that the edge of the image could have to find the longest distance according to (18), this value represents the diameter of the mole as shown in Figure 13. This distance is obtained in pixels and divided by 20, because 20 pixels is the equivalent of 1mm, which was obtained using the marker as a reference. In addition, the area is obtained by applying (19) in pixels of the object in the image ( , ) and making the equivalence of 1 2 equivalent to 400 pixels.
where and indicate the coordinate of the pixels that make up the image. Likewise, according to [27], the moments invariant to the transformations are used, which are desirable characteristics to recognize the objects more easily. Thus, the 7 Hu moments of the image are obtained invariant to translation, scaling and rotation. Finally, in this section, 9 descriptors are obtained, 2 belonging to the diameter and area and 7 to the moments of Hu.

Metadata
In addition to the characteristics obtained by image processing, data collected by the people in charge of taking the images in the health centers will also be used for the classifier. This information corresponds to the person's age, how long the mole has been there at the time the image was obtained, whether there has been any discomfort, the location of the mole (head, trunk, extremities, palms) and whether there have been any recent changes in the mole. In Table 1, the metadata being used and their respective values corresponding to each of the 10 elements are shown.

Classifiers
According to [16,28], the support vector machine (SVM) are supervised learning models, which means that the sample data must be labeled, and can be applied to almost any type of data. It is basically based on the concept of decision planes that separate classes with a hyperplane.
The size of the SVM input vector is 94 and corresponds to the 94 characteristics obtained. In the training phase, the classifier model was built using the cross-validation procedure to find the optimization parameters of the hyperplane to avoid bias with overfitting.
Another classification method used is random forest [17,29]. In Random Forest, several decision trees are built, instead of just one at the time of training. To classify a new object based on attributes, each decision tree gives a classification and finally the mode of the classes is taken is the output of our classifier.
As in the case of SVM, the size of the input vector for random forest is 94 and corresponds to the 94 characteristics obtained. Finally, Naïve Bayes is a probabilistic classifier [18,30] that uses Bayes' rule together with a strong assumption that the features are conditionally independent. In this way this classifier can be trained in a supervised way very efficiently, because the number of parameters needed are linear with respect to the number of features of the classes.

RESULTS AND ANALYSIS
The dataset consists of 95 photographs of moles taken with a smartphone, by technical personnel in different health centers. In addition, all images have a marker on the side of the mole. In Figures 14 and 15, the results of the mole segmentation process are shown. The result image will be the one that will be analyzed to obtain its characteristics. For the processing and analysis of the images we use the Python programming language, and the OpenCV and scikit-learn libraries. Of the 95 images processed, 80% of the images are used for training and 20% for the test. The results for the training images, 76 images, can be seen in the confusion matrix in Table 2, where an accuracy of 0.9473, sensitivity of 0.8571 and specificity of 1.0 were obtained. In this case, SVM was used to perform the classification. For the test process, 19 images were used, of which 8 were Not Suspect and 11 Suspect. The accuracy for these data was 0.9473, a sensitivity of 0.909 and a specificity of 1.0. This is shown in Table 2.
Then using another classification method, in this case Random Forest, an accuracy of 0.9868, sensitivity of 0.9668 and specificity of 1.0 was obtained for the training data, showing in Table 3 their respective confounding matrix. And using the same method for the test data, 0.7368 precision was obtained, with a sensitivity of 0.5454 and a specificity of 1.0, shown in Table 3. Using the last classification method, in  Table 4 its respective confusion matrix. And using the same method for the test data, 0.6842 precision was obtained, with a sensitivity of 0.6363 and a specificity of 0.75, shown in Table 4.    Tables 5 and 6 show the accuracy, sensitivity, and specificity metrics for the different classifiers. Sensitivity indicates the ability of our estimator to identify positive cases and specificity indicates the ability of the estimator to identify negative cases. We observe that specificity values are higher than sensitivity values, so our classifier is better at ruling out the possibility of having melanoma.

CONCLUSION
With the training images, for each of the methods, shown in Table 5, we can see that SVM has 0.9473 accuracy compared to random forest that obtains 0.9868, however, for the test images shown in Table  6, the one that obtains better accuracy is SVM with 0.9473 unlike random forest that falls to 0.7368. In addition, the sensitivity for the first classifier is 0.909 which is a big difference to random forest which gets 0.5454. In conclusion, SVM can be taken as the best classifier, even though in training it gave a lower result, in the test data its performance was maintained.
Comparing our results with works that use machine learning techniques as in the cases of Mustafa et al., which obtains 0.8667 and in Murugan et al., obtains 0.8943 using in both cases SVM, we can observe that our results using the same technique are 0.9473. These results support our hypothesis that by using images from different sources to a dermatoscope it is also possible to make the identification of melanoma with great inference power. In addition, in one of the latest articles using technology assistance to identify skin diseases, such as Liu et al., show that the maximum accuracy obtained by their deep learning based algorithm with dermatoscopic images was 0.90, and in our case it was 0.94 using images taken by smartphones, the results obtained in both studies are similar so it can be concluded that the proposed method of using feature extraction using digital image processing of skin lesions is an effective approach to identify the presence of malignant skin lesions, besides not requiring as many computer resources as it does a deep learning based one. For future work we intend to make use of telemedicine to be able to use our algorithm in populations that do not have access to medical consultations, thus improving health in vulnerable places. We also want to increase our database to be able to have more diversity of samples and to be able to improve our results.