A haptic feedback system based on leap motion controller for prosthetic hand application

Leap motion controller (LMC) is a gesture sensor consists of three infrared light emitters and two infrared stereo cameras as tracking sensors. LMC translates hand movements into graphical data that are used in a variety of applications such as virtual/augmented reality and object movements control. In this work, we intend to con-trol the movements of a prosthetic hand via (LMC) in which ﬁngers are ﬂexed or extended in response to hand movements. This will be carried out by passing in the data from the Leap Motion to a processing unit that processes the raw data by an open-source package (Processing i3) in order to control ﬁve servo motors using a micro-controller board. In addition, haptic setup is proposed using force sensors (FSR) and vibro-motors in which the speed of these motors is proportional to the amount of the grasp force exerted by the prosthetic hand. Investigation for optimal placement of the FSRs on a prosthetic hand to obtain convenient haptic feedback has been carried out. The results show the effect of object shape and weight on the obtained response of the FSR and how they inﬂuence the locations of the sensors.


INTRODUCTION
Capturing of human body motion is an increasing research area due to potential applications in robotics and informatics areas. Motion capture of the human hand is complex task because each finger, and even each phalanx, has distinct and independent movements that yield in return higher degrees of freedom [1][2][3][4]. Some techniques have been developed to capture hand motions, see [3][4][5][6]. However, accuracy is a major setback to track hand motion. A leap motion controller is a motion sensor that was developed for hand motion tracking with high detection accuracy that reached 0.01 mm [7]. It consists of three parts, two cameras, LEDs, and a microcontroller in which the cameras capture successive images of the hand and then they are passed to the controller to process the images and extract spatial information of the hand and fingers. Leap Motion has been used in several projects to recognize hand gestures [8][9][10][11][12][13]. Although approaching the dexterity of a human hand to control robotic arm is difficult, LM was used to do this task in [14][15][16], where hand gestures have been translated to joint angles to perform a specific task. A potential application for LM is in robotic surgery, in which a surgeon could control surgical instruments from a console located in the operating room [17][18][19]. Touchless interaction has been investigated in many works. Such studies can improve user performance in different applications such as wheelchair control and maneuvering and browsing of medical images using leap motion [20][21][22][23] implantation of leap motion, see [24][25][26]. However, locations of the sensors that are responsible for generating appropriate control signals for haptic actuators need further investigation. The aim of this work is to investigate the robustness of a modified haptic-prosthetic system for telemedicine applications, where, we suggest using the leap motion controller to control the movements of a prosthetic hand and create haptic sensation for the subjects. The haptic system consists of three FSRs and two vibro-motors which are attached to a glove. A microcontroller is used to synchronize the movement of the actuators with FSRs signals. In addition, we investigate the response of FSRs at different spatial locations and using different objects. The paper is organized as follows, section 2 presents the adopted methodology while section 3 shows the used experimental setup. Experimental results and discussion are described in section 4. Our conclusions are drawn in the final section.

2.
METHODOLOGY For the sake of clarity, the proposed methodology is divided into three parts, leap motion, prosthetic hand, and haptic feedback system. Figure 1 shows a block diagram of the suggested control system. In the following, we will explain each part in detail.  The solid arrows refer to control signals while empty arrows refer to feedback signals. The touchless interaction between haptic glove and prosthetic hand is expressed by two arrows, the filled arrow to control the prosthetic hand while the empty arrow for haptic feedback

Leap motion
Leap Motion Controller is human-hand tracking device equipped with infrared sensors that enable the LMC to respond swiftly to the fingers and hand movements. It intends to revolutionize the way that we are using our computers in favor of utilizing the touchless interaction concept. The leap motion has a compact size which makes it suitable for different control applications. In addition, the height of the interaction space is 60 cm above the LMC where the field of the interaction is similar to an inverted pyramid. This pyramidal shape increases the cameras field of view till 150 degrees with area detection capability ranging from 25 mm to 600 mm above the LMC [1]. The first phase of the work methodology is to acquire the joint angles of the humanfingers' movements from LMC using Processing software which is an open source framework for visual art programming. Then, the angles values are converted to appropriate control signals that are fed to actuators for driving prosthetic hand fingers

Prosthetic hand
We suggest to use five-finger prosthetic hand in which each finger can be controlled individually according to the obtained information from the LMC. There are many open-source 3D printable models of five-finger prosthetic hand available in the internet. A reliable, anthropomorphic, and low-cost design to be produced is InMoov hand [27]. This model has been adopted in this work since it is based on sharing and enhancing policy. Some amendments on the fingers should be made to ensure full integration between the Ì ISSN: 2088-8708 model and the proposed haptic system. These modifications related to the rounded finger geometry where we modify the shape of the fingers to allow the feedback sensors fixed on the finger's tip easily. Figure 2 shows the InMoov model and the modified printed design. All models have been printed using Makerbot replicator

Haptic system
There are different configurations to create a haptic sensation. The one that we proposed in this work is based on the feedback obtained from reaction forces on the grasped object. These reactions will trigger haptic actuators where a vibro-motor is used for this purpose. Adding haptic feedback to the touchless interaction has the potential for many applications such as gaming and rehabilitation.

EXPERIMENTAL SETUP
To demonstrate the proposed methodology, practical part of this work has been divided into two subsections.

LMC and haptic system
LMC has been placed before the user to acquire images of the user's hands which they should be set within 150 degrees from LM and from 25 mm to 600mm directly above the device. The captured information is then transferred to personal computer to extract information related to figures movements and directions. Processing software is used for that purpose in which proper libraries should be added to the Processing to be able to read and interpret the incoming information from LMC. A setback is detected due to large amount of data acquired by LMC that Processing software cannot handle smoothly. This lagging problem has been minimized by reducing the frames to be processed and use a personal computer with higher processing capability. A haptic setup is proposed using three force-sensing-resistive (FSR), type Interlink 402, and two vibro-motors, model 1027. The FSRs are laid out at different locations on the modified InMoov hand. As mentioned earlier, these sensors are used to generate the signals that trigger the haptic actuators in which the mean value of the incoming signals is used to activate the haptic actuators, vibrating motors. To mimic a realistic grasp feeling, the speed of the haptic actuators is proportional to the mean value.

Prosthetic hand and sensor calibrations
The prosthetic hand used in this work was InMoov where the fingers' shape has been modified to be capable of holding the FSR, see Figure 2. The hand was printed using 3D MakerBot Replicator and PLA filament. The movement of each finger is actuated by a servomotor type TowerProMG995. Five servo motors for all fingers were used. Servo horn and fishing line are attached with each servomotor to control the motion direction of the fingers where they can be flexed or relaxed relative to the motor rotational direction. A microcontroller type Arduino MEGA is utilized to control the servomotors, vibrating motors and FSRs. The FSRs have been calibrated using different range of weight values and a best fitting polynomial function between the output voltage from each FSR and the used weights was obtained for the used FSRs. These polynomial functions were used to obtain FSRs responses for selective locations on the InMoov. A schematic diagram of the used components and their wiring is shown in Figure 3.

RESULTS AND DISCUSSION
In this work, LMC is used to control the motion of a modified version of InMoov hand and return haptic feedback. In addition, locations of feedback sensors for the suggested haptic system are investigated to provide a more realistic touch experience. To validate the obtained results, five subjects were volunteered to carry out the prosthetic hand control experiments, in which the tests have been carried out for all participants. Table 1 shows the age and gender of the subjects. Although the angles and directions gathered from leap motion functions are different for each participant, the hand response is perfectly matched for all subjects' hands movements. Rarely, response lagging is occurred due to limitation in Processing software ability for images acquiring and processing.  Max  value  Male  60  -82  22  -87  42  -80  35  -52  20  -60  39  Male  20  -61  23  -80  39  -84  42  -66  35  -98  31  Female  50  -67  15  -82  22  -75  24  -51  19  -88  38  Female  25  -72  32  -81  51  -82  57  -75  20  -68  30  Female  17  -65  23  -82  40  -75  33  -53  30  -80  24 An investigation has been carried out for optimal placement of the FSRs such that these FSRs can return a convenient haptic feedback. Since the cylindrical objects are widely utilized to examine the grasping reactions [28] , three cylindrical objects with different diameters and 12 locations on the prosthetic hand have been selected for this investigation. Figure 4 shows the layout of the FSR sensors in which the 12 locations are disseminate along the prosthetic hand and the output responses from FSRs were recoded and presented in Figure 5. It can be noticed that the grasping reactions at locations 3 till 6 are, in general, tend to decrease compared with other locations. Such performance is linked to the cylindrical shape of the grasped object in which line contact between the grasped object and the upper part of the prosthetic palm is occurred. Interestingly, a wide variance in the value of the reactions at locations 7, 8 and 9 are detected when the objects' diameters are changed. This finding is related to the object and modified fingers geometry where the object is surrounded by the fingers and this encircling let FSRs generate electric voltage proportional to the size of the grasped object. However, increasing the object size up to a critical value, which needs further investigating, can cause a superficial result due to missing the contact between the sensors and the grasped object. A similar response for FSRs is occurred when the sensors are positioned at locations 10 and 11 while location 12 returns a less reaction value compared with previous locations. The reason for this rather contradictory result is still not completely clear, but we refer that to palm shape of InMoov. Our findings would seem to show that the Ì ISSN: 2088-8708 FSRs locate in the index and middle fingers' tips give better reactions than other locations assuming that the grasped objects have a cylindrical shape. However, different shapes for tests objects need to be investigated to obtain a comprehensive understanding of relation between the grasped objects and the modified prosthetic hand. Partially captured objects can also affect the grasping reactions that may decrease the reaction value and also may case slipping of the grasped objects. Thus, the applied grasped force from the real hand should be proportional to the object diameter to prevent object sliding. This requires a further study which is beyond the scope of this work. The weight of the object affects the reaction depending on the prosthetic hand layout.
In the case of a horizontal layout, the FSR is influenced by the object weight while in a vertical layout, the weight does not contribute to the grasping reactions. Instead, it has a slipping effect which should be balanced by increasing the applied grasping force.

CONCLUSION
In this work LMC is utilized to control anthropomorphic hand with haptic feedback to perform cylindrical grasping tasks. The proposed system worked fine and smooth response in general was obtained. Although, the LMC enables decent touchless interaction between the user and the prosthetic hand, geometry of the prosthetic hand requires additional modifications and dynamic analysis to ensure full integration with the haptic system. Such investigations will improve the potential of using LMC in telesurgery and tele examination since accuracy and interaction are utmost key factors. We investigated the locations of the feedback sensors for the proposed haptic system where 12 locations distributed along the prosthetic hand were selected. The tip of the middle finger yields a good grasping reaction despite varying in object's shape. Placing the FSR on the hand palm can give acceptable reaction, however, object size can decrease the contact area and return less effective contact reaction. We recommend a further analysis for optimal sensors placement preventing object slipping.

ISSN: 2088-8708
Ì 5777 This may be achieved by fusion the date obtained from FSRs and a proximity sensor, where the return signal from the last sensor can detect the occurrence of object slipping while the FSRs maintain the haptic feedback.

ACKNOWLEDGMENT
We would like to thank Dr. Ali H. Al-Timemy for providing a printed InMoov hand. Dr. Al-Timemy is a member of staff at Al-Khwarizmi college of Engineering, University of Baghdad