Deep Belief Networks for Recognizing Handwriting Captured by Leap Motion Controller

Leap Motion controller is an input device that can track hands and fingers position quickly and precisely. In some gaming environment, a need may arise to capture letters written in the air by Leap Motion, which cannot be directly done right now. In this paper, we propose an approach to capture and recognize which letter has been drawn by the user with Leap Motion. This approach is based on Deep Belief Networks (DBN) with Resilient Backpropagation (Rprop) fine-tuning. To assess the performance of our proposed approach, we conduct experiments involving 30,000 samples of handwritten capital letters, 8,000 of which are to be recognized. Our experiments indicate that DBN with Rprop achieves an accuracy of 99.71%, which is better than DBN with Backpropagation or Multi-Layer Perceptron (MLP), either with Backpropagation or with Rprop. Our experiments also show that Rprop makes the process of fine-tuning significantly faster and results in a much more accurate recognition compared to ordinary Backpropagation. The time needed to recognize a letter is in the order of 5,000 microseconds, which is excellent even for online gaming experience.


INTRODUCTION
Computer aided handwriting recognition can be performed either on-line or off-line. Typically, on-line handwriting recognition is based on specific input devices, such as real-time capturing digital pen, finger-touch screen, or mouse movement. In the recent decade, such input devices have already been equipped with computer vision technology that enables recognition of natural hand or body gestures as seen in Kinect-based handwriting recognition [1]. Huang et al. [2] introduce digit recognition on Kinect by using Multiple Segment and Scaled Coding with 94.6% accuracy. Other methods are proposed by using Depth-Skin Background Mixture Model for hand segmentation [3] and Dynamic Time Warping combined with Support Vector Machines (SVM) for digit recognition [1].
Kinect has the ability to recognize body shape or hand movements, but it faces difficulties in obtaining precise finger position compared to Leap Motion controller. Leap Motion controller (or simply Leap Motion) is a new computer vision device that can track the position of fingers more precisely. The 3D finger data can be obtained at over 100 frames per second [4], making it possible to efficiently track the position of each finger in detail. However, Leap Motion still faces some limitations: it cannot directly recognize a specific sequence of fingers' strokes (movements) performed when drawing a letter or a digit in air.
Several algorithms have been proposed to recognize these sequences of strokes to obtain a better accuracy. Vikram et al. develop handwriting recognition based on Leap Motion by using Dynamic Time Warping algorithm [4]. Agarwal et al. [5] present segmentation and recognition of 3D handwritten text captured by Leap Motion. Hidden Markov Model (HMM) is used to train segmented words and achieves an accuracy of 77.6%. Another algorithm for rapid recognition of hand graphical gestures is SVM. SVM is used to recognize 100 testing samples with an average recognition rate of 82.4% [6].

PRELIMINARIES 2.1. Leap Motion controller
Leap Motion is an input device that allows users to recognize and track hands, fingers, and finger-like tools. Leap Motion is developed by Michael Buckwald and David Holz at Leap Motion, Inc. Leap Motion is able to track 3D finger data precisely and obtains them at over 100 frames per second [4]. Besides, Leap Motion device is also cheaper compared to other similar computer-vision devices. The dimension of the device is 1.2×3×7.6 cm 3 and it uses optical sensors and infrared light, as shown in Figure 1 [16]. Leap Motion's sensors can detect hands or fingers across the upward y-axis and possess a 150 • -vision with effective range of 25 to 600 millimeters [17].  [16] To develop applications that use Leap Motion as input device, we can make use of an available API that can be accessed via various programming languages. Leap Motion API version 2.0 [17] introduces New Skeletal Tracking Model that provides additional information for arms, fingers, and overall tracking enhancement. In order to use Leap Motion, first, it must be connected with an existing USB port and positioned beside the monitor at a distance of the cable's range. Leap Motion device will detect gestures and the position as well as the movement of hands or fingers as the user positions her hands on its range. To retrieve the desired hand tracking data, Leap Motion API provides an update set or a frame of data. Every single frame object from the API represents a frame that contains a list of tracking entities called motion tracking data. Motion tracking data, such as hands, fingers, and tools, can also be accessed through the API.

Restricted Boltzmann machines
A Boltzmann machine is a stochastic recurrent neural network with binary stochastic units. There are two types of layers in Boltzmann machines: visible layer and hidden layer. Visible units or neurons on the visible layer are connected to other units and receive information from the environment. Visible units also bring the input signal from the environment and send it to other units during training. Hidden units, on the other hand, operate freely and extract features from the input signal. Interconnections of units in Boltzmann machines can be done even in a single layer itself. Unfortunately, learning in Boltzmann machines is often impractical and furthermore they also face Int J Elec & Comp Eng ISSN: 2088-8708 4695 scalability issues. In order to make learning in Boltzmann machines easier, advanced restrictions of connectivity must be enforced. This restriction is called Restricted Boltzmann Machines (RBM) [18]. In RBM, there is no connection between units of the same layer. RBM is composed of a visible layer and a hidden layer of stochastic neurons that are connected by symmetrical weights (i.e., there is no weight toward itself). Hidden units in RBM are considered as feature detector.

Deep belief networks
A deep belief network is a generative model, consisting of multiple stochastic layers. A stochastic layer has several latent variables. Latent variables typically have binary values and are called hidden units or feature detectors [13]. Each DBN layer is an RBM and in order to create many layers, several RBMs are stacked on top of each other to construct a DBN construction. Hinton et al. in [13] propose an idea to learn one layer at a time and restrict the connectivity of binary stochastic units in order to make learning more efficient and simpler [18].
DBN training begins with RBM training: starting from input samples and then sequentially progressing toward the next RBM training. Generated patterns formed by the top RBM can deliver a reversed propagation to the input layer by using conditional probability as in belief network or sigmoid belief network. This training procedure is called DBN's greedy layer-wise training [13]. DBN can work with or without supervision. Although DBN can be used in supervised or discriminative mode, basically, greedy layer-wise training is an unsupervised training for each layer of stacked RBMs. It is called a pre-training procedure [19,20,21]. The purpose of pre-training procedure on each layer's training is to put the parameters of all layers in a region of certain parameter ranges that contain a good generalization of local optimum that can be obtained by local gradient descent [20,21].
In an RBM, the joint distribution p(v, h; w) of visible units v and hidden units h with parameters w is defined by the energy function E(v, h; w). For Bernoulli-visible units that lead to Bernoulli-hidden units, the energy function is defined by: where w ij represents the symmetrical interaction between visible unit v i and hidden unit h j , b i and a j are the bias terms of, and I and J are the numbers of visible and hidden units, respectively. Therefore, the conditional probabilities can be calculated by: where ϕ(x) = 1/(1 + exp(−x)). Greedy procedure on stacked RBMs is used to raise the maximum likelihood learning. Equation (3) is a learning rule for updated weights at each RBM to collect the gradient log-likelihood: where E data (v i , h j ) is the expectation of observation on the training data and E model (v i , h j ) is the expectation under the distribution defined by the model. Calculating E model (v i , h j ) is intractable, so Contrastive Divergence (CD) algorithm is used to solve this problem [13]. CD algorithm performs Gibbs sampling to replace E model (v i , h j ) in one or more steps. CD-1, namely CD algorithm that only performs one step of Gibbs sampling on the data, is used to train the model. The following are the steps of CD-1. First, input sample x and updated hidden units are used to compute p(h j = 1, v; w). Second, units are activated by comparing the sample result from the previous step with random values to obtain the binary state for each unit. Third, those binary units are reconstructed by calculating p(v i = 1, h; w). Fourth, the output probability of the reconstruction is calculated, and the reconstruction result becomes an input. Fifth, after all collections of statistics from the previous step are obtained, the weights on the RBM are updated.
After completing pre-training procedure in DBN, the next phase is to perform discriminative fine-tuning with Backpropagation [13,14]. Fine-tuning with Backpropagation is carried out by adding a final output layer, which is a labeled class excluded from the pre-training procedure. ISSN: 2088-8708 Figure 2 depicts the architecture of our proposed approach for in-air handwriting recognition using Leap Motion. The proposed approach is developed based on the idea laid out in [22]. Data acquisition is a process of collecting every letter sample by using Leap Motion. This captured letter sample can be used for recognition, or else, it can be stored in a dataset. The letter dataset is split into two parts, namely: training and testing datasets. The training dataset will be incorporated into a learning system by using several proposed algorithms; and this basically constitutes the training process. After training process has been completed, several model algorithms will be constructed. These model algorithms are then used for testing (by using the testing dataset) in order to obtain and compare their accuracy. The best model algorithm is then selected for further recognition of letter input data captured by Leap Motion.

Data acquisition
Data acquisition is a mechanism to obtain handwritten sample data by using Leap Motion. A sample data is a sequence of information about the movement of hand (inputs only use one hand and one finger) in the air as captured by Leap Motion. The motion of finger-hand draws a letter stroke, which is then displayed on the screen of the simulation application integrated with Leap Motion API. Figure 3 depicts the flow for creating one sample data using the existing API.  Normalizing stabilized position InteractionBox property of Frame is an object on the API to ensure the user's finger-hand always stays on the device's visibility range. While finger-hand is inside InteractionBox, movements of the fingers will be detected on 3D coordinates, as shown in Figure 4(left). When the application is mapped to 2D coordinates, the InteractionBox is as shown in Figure 4(right). InteractionBox simplifies the mapping of every point of finger's tip position by normalizing the range of captured hands from left to right or top to bottom to values between 0 to 1, relatively to the application screen. If w app and h app are the width and height, respectively, of the 2D field in the application screen, InteractionBox provides the normalized position of the final stroke relative to the application's coordinate (x app , y app ), which is given by: where N P is a normalization function in InteractionBox called normalizePoint.
Touch emulation From the positions of all strokes representing a single letter, a sequence of strokes S = (S 1 , S 2 , S 3 , and m is the number of strokes required to draw the letter. Strokes can be in two states: hovering and touching. Not all strokes are used; those that are in hovering state can simply be ignored, as illustrated in Figure 5(left).
Hovering Touching +1 0 -1 Figure 5. Touch emulation: the difference between hovering and touching [17] The state is determined by a normalized distance in the range of +1 to −1. First, the finger enters the hovering state from normalized distance of +1 and the distance decreases toward 0 where the finger is near the touching surface. When the finger penetrates the surface, the normalized distance decreases to the maximum −1, which indicates that it is right now in the touching state. Stroke points are only recorded when the finger's tip is in the touching state.
Image matrix creation The tracing in the previous process is related to the tile-based editor of the simulation application. Tile-based editors are often developed in game technology, specifically to create the game environment [23,24]. In this research, all tiles are of the same width and height, namely 12 pixels, and the tile-map is a 28×28 tile matrix.
If the tile-map is considered as a representation of an image, each tile will have an intensity value and an index position on the tile-map. Each tile indicates a pixel image drawn by the user in the air. Therefore, in order to select a tile that corresponds to the user's creation, a fixed stroke point position with the hovering or touching state is used. Figure 6 provides an illustration on how the selection of tile-based position for each stroke captured by Leap Motion is done.
In Figure 6, m t is map-top distance, m l is map-left distance, T s is the size of a tile, and (x t , y t ) is the selected index of a tile. The sequence of strokes S will be converted into an indexed (vector) sequence of strokes and n is the vector size of the 28×28 image matrix, namely 784. Each V i has an intensity value of 0 or 1 determined by: where d is the touch distance, which defines whether the finger is in hovering or touching state. A coordinate (x t , y t ) of each stroke point is then defined by:  Image matrix dilation Sometimes when the user writes a letter in the air by using Leap Motion too quickly, the handwritten letter becomes unclear. To clarify the handwritten letter, dilation operation can be applied to the image matrix. Dilation simply takes several specific neighboring points of a point and gives them an intensity value of 1, as depicted in Figure 7.  Figure 7. Image matrix dilation for letter "D"

Data, dataset, and data preparation
Each instance of data acquisition creates a single sample data. A single sample data is represented by a 784size vector, which encodes the 28×28 matrix produced in image matrix creation. The data is used for two purposes: as an addition to dataset collection used as training or testing dataset with labeled data, and to be used in the recognition process, where the label will be searched out based on the learning algorithm trained before.
In this research, 22,000 samples of training and 8,000 samples of testing datasets are collected from 10 respondents. The respondents are undergraduate students in Universitas Dian Nuswantoro, Semarang, Indonesia, aged 20-21 years old. Each of them produces around 3,000 handwritings with Leap Motion, captured within 1 month. The training and testing datasets are restricted for upper-case letters from A to Z. Because the learning task is supervised, the dataset used for the DBN is in the form of [( V 1 , L 1 ), ( V 2 , L 2 ), . . . , ( V k , L k )]. For each letter sample, input data is a vector representing the sequence of strokes V i for 1 ≤ i ≤ k whose label is L i , and k is the number of samples in the dataset. All elements of input vector V i are of binary value.
Encoding the label L i , which has value from "A" to "Z", such that L i ∈ {0, 1, 2, . . . , 25} cannot be included directly into a DBN. In order to map L i as an element of a vector, one-hot encoding mechanism is used. In one-hot encoding, label L i containing the value "A" is encoded by [1, 0, 0, . . . , 0], "B" is encoded by [0, 1, 0, . . . , 0], etc.

Learning system
This research makes use of Deep Belief Networks (DBN) in supervised fashion. DBN basically learns the model of F w : { V } → {L}, which maps an input vector V into a labeled vector L with parameter weights w. Figure 8(a) shows the architecture of a DBN with two hidden layers, h 1 and h 2 . Two main processes exist on learning this DBN model. Firstly, the process of greedy layer-by-layer training for each stacked RBM by using Contrastive Divergence algorithm, which is called pre-training. Secondly, the process of fine-tuning the entire network using Resilient Backpropagation or Rprop.

RBM initialization and construction
As described earlier, a DBN consists of stacked RBMs [13], therefore, each RBM layer first needs to be initialized. Figure 8 Step 0: In the first RBM, visible layer v 1 gets direct input from sample V .
Step 1: Train the RBM's visible v 1 and hidden h 1 layers to adjust weights w 1 .
Step 2: After the training is completed, use the pattern activity of the first RBM's hidden layer h 1 as input to visible layer v 2 to train the second RBM.
Step 3: By copying h 1 as v 2 in the second RBM, train visible layer v 2 and hidden layer h 2 to adjust weights w 2 . Note that in this process, class labels of data are not involved.

Training each RBM layer
Every RBM is trained by using Contrastive Divergence with one step of Gibbs sampling algorithm (or CD-1). CD-1 follows the gradient difference function called Kullback-Leibler divergence. Here are the steps of CD-1: Step 1: While maximum epoch is not reached, do step 2 to 7 repeatedly.
Step 2: Use data sample V as visible layer v.
Step 3: Update hidden layer by calculating p(h j = 1, v; w) (cf. Equation (1)), then activate stochastic binary neurons h j on the hidden layer by taking the probability p(h j = 1, v; w).
Step 4: Reconstruct visible neurons v 1 i by calculating p(v i = 1, h; w) (cf. Equation (2)) using the value of stochastic binary neurons calculated before as input.
Step 5: Calculate the output probability of h 1 j from previous reconstruction step, namely p(h 1 j = 1, v 1 ; w).
Step 6: Calculate weights (cf. Equation (3)) by using the learning rate α as follows: Step 7: Update the new weights by adding momentum µ and weight decay w d as follows: Learning rate is a parameter that influences the learning speed of the algorithm in reaching convergence. Momentum is a parameter used to prevent the system from being trapped in local minima. Weight decay works by adding "additional terms" to the normal gradient. The additional term is a derivative of a function that penalizes an excessive weight [25].
The process of updating weights of RBM refers to the update rules of the gradient descent. In the case of large-scale datasets, using batch gradient descent requires quite an expensive computation when it only needs one step for the entire data training. Therefore, we use a more efficient learning mode that learns the dataset for each predetermined batch. This mode is called mini-batch gradient descent [25].

Fine-tuning Rprop
Because DBN is to be used as supervised (after pre-training process is completed), which involves label classes, weights have to be fine-tuned. Fine-tuning is conducted by training the network with Backpropagation algo-ISSN: 2088-8708 rithm [13,14]. In this research, besides using fine-tuning with Backpropagation, we also perform Resilient Backpropagation (Rprop) algorithm as shown in Figure 8(c).
Theoretically, Rprop algorithm is more efficient in terms of learning speed compared to ordinary Backpropagation. The difference between Rprop and Backpropagation is located on the weight update, where Rprop makes use of the sign that identifies the value of the last gradient update [26]. In the Rprop algorithm, we use weights and biases obtained from pre-training process, namely w ij = w 1 (from input to the first hidden layer) and w jk = w 2 (from the first hidden layer to the second hidden layer). The weights and bias connection between the second hidden layer to output layer uses random initialization in the interval (−1, 1). We initialize the maximum epoch and set the values of ∆ 0 = 0.0125, ∆ max = 50.0, ∆ min = 10 −6 , η + = 1.2, and η − = 0.5.

RESULT AND ANALYSIS
A learning algorithm naturally minimizes the cost function in order to obtain optimal result. When the algorithm is being trained using some training dataset, we are basically creating a model algorithm that closely reflects the available data. Afterwards, the obtained model algorithm can then be applied to testing dataset to assess the performance of the algorithm.

Experimental setup
All methods and algorithms in our proposed approach are implemented in C#. All experiments are conducted using this implementation on a computer with a Core(TM) i7-2600 CPU @ 3.40GHz (8 CPUs), 3.7 GHz. Learning is performed with a parallel computation on a logical core for each CPU. For our experiments, we perform data acquisition that produces 30,000 samples of capital letters, partitioned into two groups: 22,000 samples as training and 8,000 samples as testing datasets. There are about 843 samples for each letter from A to Z in training dataset, whereas in testing dataset, each letter has about 307 samples. Every single data in the training dataset will be used as an input for the learning algorithm.
In order to assess the performance of our proposed approach, we not only conduct experiments on DBN with Rprop fine-tuning, but also on three other approaches for the purpose of comparison. The three approaches are MLP with Backpropagation (Bp), MLP with Rprop, and DBN with Backpropagation. Table 1 shows the experimental configuration we use for each of those approaches.   Table 1 shows that the neural network architecture we use when dealing with the letter dataset is 784-400-100-26. In this architecture, the input is a 784-sized vector, which is a flattening of the 28×28 tile-map matrix for each letter sample. Both MLP and DBN use 400 and 100 neurons in the first and second hidden layers, respectively. The size of the output vector is adjusted to the number of letter labels, namely 26 classes with one-hot-encoding mechanism.
Beside using the letter dataset captured by our own data acquisition mechanism, we also make use of the digit dataset from MNIST database [27]. The MNIST digit dataset consists of 60,000 samples in training dataset and 10,000 samples in testing dataset. Each sample in the MNIST digit dataset is formed by a grayscale image. To use them, it is necessary first to perform preprocessing by transforming the grayscale into binary images. Afterwards, width normalization is carried out by changing the original size of 28×28 pixels to 20×20 pixels. The 20×20 tilemap matrix is then flattened into an input vector of size 400 as shown in the last four rows of Table 1. Similar to the letter configuration, in this digit configuration, both MLP and DBN use 400 neurons and 100 neurons in the first and second hidden layers, respectively. Because the digit labels range from 0 to 9, the size of the output vector is 10 with one-hot-coding mechanism.

Int J Elec & Comp Eng
ISSN: 2088-8708 4701 Both for the letter and MNIST digit datasets, we use the same configuration on the pre-training phase of the DBN. There are two RBM architectures in letter configuration: the first RBM is 784-400 and the second RBM is 400-100. In MNIST digit configuration, the first RBM architecture is 400-400 and the second is 400-100. There is no pre-training phase in the MLP. Each RBM in the letter or MNIST digit configurations is trained with CD-1 algorithm. Based on our experiments, we select the same initialization throughout, namely learning-rate = 0.5, momentum = 0.5, and weight-decay = 0.001. The maximum epoch for each RBM is set to 300. The batch size parameter for each RBM is set to 100, which means that if there are 22,000 training samples in the dataset, they will be processed 100 at a time.
The Backpropagation for fine-tuning on DBN or training on MLP uses learning rate = 0.05, momentum = 0.5, and batch size = 100. Unlike Backpropagation, Rprop does not need learning rate and momentum, but instead needs constant values ∆ 0 , ∆ max , ∆ min , η + , and η − (cf. Section 3.3.3.). The initial epoch for Rprop is set to 300 for letter and 500 for MNIST digit configurations. Table 2 shows accuracy and the computational times required in recognizing each letter (from testing dataset) by the four different methods. Recall that there are about 307 samples for each letter in testing dataset. The first four columns of the first row of Table 2 indicate that out of 307 samples of the letter A, 99.35% (Acc.) of them are recognized and the computational time spent until the point when the letters are successfully or unsuccessfully recognized is on average 4,572.2 microseconds (t), with a standard deviation of 1,081.2 (stdev). The last row of the table provides the overall statistics of accuracy and computational times for the whole letter training dataset produced by the four different methods. Table 3 shows similar results for the recognition of the MNIST digit testing dataset. ISSN: 2088-8708  Table 4 summarizes the results for recognizing both letter and MNIST digit testing datasets. The best accuracy (column Acc.) for both testing datasets is DBN with Rprop fine-tuning with accuracy of 99.71% and 96.30%, respectively. This is better than the accuracy of MLP with Rprop fine-tuning, which achieves 99.59% and 95.75%, respectively. These two approaches evidently are much more superior than both DBN and MLP with Backpropagation training. It seems that fine-tuning with Rprop contributes significantly to the accuracy of both MLP and DBN. The last two columns of Table 4 (t and stdev) show that the recognition times of the four approaches are comparable. It seems that using Rprop results in longer recognition times both for MLP and DBN. Out of the four approaches, the recognition time of DBN fine-tuned with Rprop is the longest, namely on average 4,765.3 microseconds with a standard deviation of 1,412.2 for the letter dataset. For the MNIST digit dataset, the recognition times of the four approaches are much more similar to each other, and are a bit more than a half of those for the letter dataset. This is because the resulting neural network for the digit dataset has smaller input as well as output vectors, and thus is quicker to process. The time needed to recognize a letter or a digit is below ten milliseconds, which is excellent even for online gaming experience [28].

Training and fine-tuning
Prior to recognition, the neural networks, both DBN and MLP, must first be trained or fine-tuned using training dataset. The fourth column of Table 4 provides the computational times spent by the four different methods to complete training and pre-training (if any). Because MLP does not have pre-training phase, its training time is much shorter than DBN. The time spent by DBN to complete pre-training is around 5 hours for the letter dataset and 10 hours for the MNIST digit dataset. It is evident that this pre-training phase takes most of the training computational time of DBN and Rprop is really helpful in reducing the training (but outside of pre-training) time.
In Figure 9 and Figure 10, we show how the accuracy of each method changes as the number of epochs increases in the training process. In order to produce these figures, right after each epoch in the training process (with training datasets), each method is paused, and the resulting neural network is then used to recognize letters or digits in the testing datasets. We observe these changes until 300 and 500 epochs for the letter and digit datasets, respectively. In Figure 9(right), both MLP and DBN fine-tuned with Rprop require 35 epochs (cf. Table 4 column 3) to reach the highest accuracy (cf. Table 4 column 2) in the letter dataset. Out of 8,000 letter samples in the testing dataset, DBN fine-tuned with Rprop misrecognizes only 23 samples at epoch 35. In Figure 10(right), MLP and DBN fine-tuned with Rprop require 70 epochs to reach the best accuracy in the MNIST digit dataset. From 10,000-digit samples, DBN with Rprop misrecognizes 367 samples at epoch 70.