Robotic hex-nut sorting system with deep learning

Received Sep 7, 2020 Revised Dec 23, 2020 Accepted Jan 13, 2021 This article exposes the design and implementation of an automation system based on a robotic arm for hex-nut classification, using pattern recognition and image processing. The robotic arm work based on three servo motors and an electromagnetic end effector. The pattern recognition implemented allows classifying three different types of hex-nut through deep learning algorithms based on convolutional neural network architectures. The proposed method exposes four phases: the first is the design, implementation, and control of a robotic arm. The second is the capture, classification, and image treatment; the third allows gripping the nut through the robot’s inverse kinematic. The final phase is the re-localization of the hex-nut in the respective container. The automation system successfully classifies all the types of hex-nuts, where the convolutional network used is an efficient and recent pattern recognition method, with an accuracy of 100% in 150 iterations. This development allows for obtaining a novel algorithm for robotic applications in hex-nut sorting.


INTRODUCTION
Process automation is an area that robotics today has revolutionized, so that robots of different types such as anthropomorphic [1] or parallel structure [2] are used in industrial tasks, such as collecting and relocating objects [3]. A fundamental part of robotic agents' use in automatic systems is the need for machine vision systems, employed for navigation tasks [4] or elements of interest detection [5].
One of the widely used machine vision techniques is image processing algorithms [6], which has several applications at pattern recognition in images. In [7], various techniques applied to eye recognition are exposing, allowing them to be used in biometric or clinical systems. In [8], processing techniques to improve the analysis in color images are exposed. In [9], an algorithm is exposed to enhance the images' quality, a highly relevant factor in medical-oriented machine vision applications. In [10], a support system for safe driving based on driver numbness detection is presented. In [11], a processing algorithm is presented that allows the counting of nuclear cells automatically, evidencing the benefits of image processing for identifying element units, an aspect used in the development of this work.
Although more applications of image processing algorithms [12,13] can be mentioned, the development presented in [14] employs image processing techniques for analysis of internal space metrics in a nut, highlighting the use of the same for objects localization like is explored in this work. Researches around the general properties of nuts' use are exposed in state of the art [15][16][17]; however, those oriented to robotic automation tasks for pick and place are not referred. This robotic field task exposes several  [18,19]. As a problem to solve in the classification of a specific hexnut kind, image processing requires an additional pattern recognition stage, like neural networks, increasing the complexity of robotic manipulators' gripping task.
To discriminate objects in images, pattern recognition techniques like convolutional networks [20] are very efficient [21]. Applications in biological image classification [22], real-time face recognition [23], or grasping an object in a robotic environment [24], give probes of its versatility [25]. In this work, CNN's are used to discriminate three class of hex-nut by transfer learning method [26], using an electromagnet-type end effector for pick and place tasks, through a robotic arm a machine vision system, and image processing for the hex-nuts localization. As a deep learning algorithm, CNN is a recent technique in pattern recognition; it is being used in the robotic automatization processes [27,28] like is expose in this work, but in state of the art is not explore this technique in hex nut classification with a robotic manipulator.
The paper is divided into four sections. The first is the previous introduction. The second section explains the research method used, the robotic arm, its mathematical model, the convolutional network architecture used, and its training. The third section exposes the results and discussion of hex-nut classification and its analysis, and finally, section four exposes the conclusions achieved.

PROPOSED METHOD
A three-degree-of-freedom robotic arm was implemented with an electromagnet-based end effector capable of attracting the nut and holding it while it is repositioned in the desired position. The robotic arm operates based on servomotors and electromagnet-type end effector with a plastic shaft that protrudes to hold the nut. Figure 1 (left) illustrates the test environment used, where the black background where the nuts to be relocated would be found is highlighted. The system has an interface that allows human intervention and control, through the Telegram application [29] capable of operating from an Android or IOS cell phone. Figure 1 (right) illustrates the local controller based on a raspberry pi 2 model B, programed in python, the pi-camera V1 of the raspberry was used for image acquisition, and its GPIO module is used as a driver to each servomotor.

Figure 1. Workspace and controller circuit
A phased methodology was used, where each phase's names are capture, classification, and image treatment. Design, implementation, and control of the robotic arm, the nut's capture, and the system interface. Figure 2 showing a flowchart of the phases used in all the processes.
For the first stage, image capture, digitization and treatment, a supervisory camera was installed in the work area to capture the input images of the system. Stage two consists of the design, implementation, and control of a robotic arm, where the implementation was carried out through the use of multipurpose brackets, which allow us to fix the servomotors and shape the arm. Three servo motors were used, one at the base, to control the "XY" axis, one in the middle, with vertical movement to control the "Z" axis, and one at the end, for orientation of the end effector. The finished arm was placed in front of the camera so that the nut can be photographed and taken by the arm so that it can later be located. A plastic container is used as a means of storing the nuts taken from the work area, as can be seen in Figure 3. In Figure 4 can be observed the schematic manipulator representation used for the inverse kinematic used to move the robot, along with the coordinate system for each joint. The angle of the joint 1 and the X' component of the final point P was obtained with the frontal view, as shown in the (1) and (2). The angle of the joints 2 and 3 was obtained with the frontal view, by getting "d" length, and alfa and beta angles like shown in the (3)-(6).

′ = √ 2 + 2
(1) The robotic arm has five calibrated positions, the first is the "rest" position, in which the robot will be stretched upwards, the second state is that of "picking up", which will be positioned as close to the nut to be gripping, and the rest, are positioned on top of the different containers. Depending on the type of nut, it will be placed in a separate container as shown in Figure 5.  In the third stage, the way to move the robot for grip the nut is exposed; the electromagnet for the gripper is employed because it allows us to easily activate and deactivate the attraction for clamping ferromagnetic elements such as the nut, and it does not require high precision for grip through the arm. The electromagnet consists of a 70-turn winding on a metal screw, with 24-gauge copper wire, which when 9 volts are applied, generates a stable magnetic field necessary to support a nut.
Finally, to classify the hex-nuts, a convolutional neural network is used whose architecture can be seen in Table 1. The architecture uses four convolutional layers and ending in a flatten layer with three outs. The kernel column shows the number of filters used and each one's size, based on square filters with side L. Figure 6 presents the database used for the nuts' discrimination, although the default background is black, other backgrounds are used to generalize the learning. The top row corresponds to the type 1 nut, the middle row to the type 2 nut, and the final row to the type 3 nut. Figure 7 shows the network's performance, which allow discriminate the hex-nuts, reaching 100% precision in the task, only 150 iterations was necessary for the network learning was established.  Figure 6. Hex-nut database for CNN training Figure 7. CNN training accuracy Figure 8 shows the network's learning through the activations of the last layer, and it is possible to differentiate between those filters that learn the background from those that learn the nut shape. Once the nut is recognized, color to grayscale transformation is used as algorithm of image processing [30], applying thresholding to the image to detect the geometric center [31] and take this pixel as a reference point to locate the robot's end effector over the hex-nut as shown in Figure 9.  Table 2 compares the number of nuts classified, captured, and correctly located, concerning the nuts located exposed under the work area. It is possible to infer that the robotic arm fulfilled its classification task satisfactorily since the respective hex-nuts, were located in the desired places. Likewise, the electromagnet allowed the nuts' proper grip and released them at the appropriate times, generating a robust and reliable system as shown in Figure 10.  Figure 10. Hex-nut deliver A system interface was used, which agreed to design and implement communication between the Telegram application, downloadable from the App Store, and the robotic system which is connected to the internet as shown in Figure 11. The system has a precision and an accuracy of 100%, obtaining an error of 0%, it is achieved by the robustness of the CNN and the nut grip's simplicity. Besides, the online system (Telegram App), allowed efficient communication, by correctly sending and receiving the requested data. Figure 11. Telegram report

CONCLUSION
It was possible to design and validate an automatic hex-nut classification system, which requires machine vision algorithms for its operation. It allows demonstrate that a robotic arm can fulfill classification assistance tasks based on the integration of algorithms for control and automation operations. A high number of test samples were not required because the hex-nuts do not vary in size or shape for each class, which focus the success of developing a useful classification system through convolutional networks, very efficient for this task. The integration with the Telegram tool allows the user to monitor the system's operation remotely, validating short response times and offering a means of feedback to the user.