A multilabel classification approach for complex human activities using a combination of emerging patterns and fuzzy sets

ABSTRACT


INTRODUCTION
Human Activity Recognition (HAR) is a field in pervasive computing, that is concerned with recognizing the activities performed by a person using data collected from the monitoring devices. In the beginning of HAR, vision-based HAR used cameras to monitor humans within their surrounding environments [1]. The rapid developments of the monitoring devices from cameras to sensors lead to the release of sensor-based HAR [2]. In sensor-based HAR, humans wear sensors or install them in the surrounding environment to track their movements, and environmental conditions. Sensor-based HAR gains a lot of research interest due to the wide range of its application domains in medical, security, and military [3].
In this paper, we are interested in sensor-based HAR in which human activity is comprised of a set of sensor readings occurring within a specific time interval. Each of these readings forms an action that represents an atomic operation such as opening fridge. Within a specific time interval, a person can perform  Many research attempts have been introduced for simple activity recognition, but little work targets complex activity recognition and stills a challenging issue. Complex activity recognition is a multilabel classification problem, and a test instance is assigned multiple class labels. For instance, when user is drinking coffee while watching TV, the recognition model should identify these two activities. Existing datadriven approaches for complex HAR have some limitations. First, in order to recognize complex activities, they require a training dataset of complex activities. Consider we have m activities, and then we have m (m-1) possible combinations that form complex ones, which is difficult to obtain. Secondly, existing approaches for complex HAR limit the number of overlapping activities, and cannot recognize more than two overlapping (i.e. interleaved or concurrent) activities.
In this paper, we present a novel multilabel classification approach of complex human activities. In our approach, we can recognize multilabeled (i.e. complex) activities from single-labeled (i.e. simple) activities. This is achieved by making a combination of Emerging Patterns and Fuzzy Sets. First, we use a training dataset of simple activities and apply a pattern mining technique to extract discriminative features that are exclusively present in each activity called Strong Jumping Emerging Patterns (SJEPs). Then, our recognition model use a scoring function to identify activity label(s) using the extracted SJEPs with fuzzy membership values of incoming sensor events (actions).
The rest of paper is organized as follows. In Section 2, we review the previous work proposed for complex HAR. Then, we formulate our problem and introduce the required preliminaries in Section 3. Section 4 introduces the proposed approach for complex HAR. In Section 5, we present our empirical results. Finally, we conclude our work and discuss our plans for future work in Section 6.

RELATED WORK
The existing approaches for HAR can be divided into two prominent approaches data-driven and knowledge-driven approaches [4]. In this paper, we are interested in data-driven approaches that require a predefined dataset to build the classification model using machine-learning techniques, and then use the built model to identify unlabeled activities. As a multilabel classification problem, the problem of complex HAR can be handled with three main approaches: data transformation, method adaptation, and ensemble of classifiers [5].
The data transformation approach converts the multilabel classification problem into a number of binary classification problems (e.g. binary relevance, classifier chain), multiclass classification problems (e.g. label power set), or use ensemble methods. For example, in [6] the authors applied a dynamic segmentation approach to split complex activities into multiple simple activities, and then apply the Random Forest technique with a voting mechanism to identify activity labels. On the other hand, method adaptation approach extends the traditional methods for binary classification to support multilabel classification. The existing attempts are probabilistic graphical models, pattern mining-based models, or are based on time series classification. In [7,8] the authors extend the traditional Conditional Random Fields (CRFs) to handle complex activities. They proposed a Factorial CRF (FCRF), and a Skip chain CRF (SCCRF) respectively. In [9], the authors presented a modified Hidden Markov Model (HMM) to deal with interleaved activities called Interleaved HMM (IHMM). Also [10] presented a modified Naïve Bayes (NB) called Interval Temporal Bayesian Network (ITBN). Another attempt presented a pattern mining approach to recognize simple, interleaved, and concurrent activities [11], and time-series based classification in [12].

PRELIMINARIES
Before going in depth to the proposed model, we formulate the task of sensor-based HAR as follows. Consider we have a set S of sensors attached to human or installed in the environment. Using these sensors to monitor human behavior, we obtain a dataset = { 1 , 2 , … . } that consists of a set of activity traces . Each trace is as a sequence of triggered sensors (i.e. events), whereas each event is a tuple of < , , , > where, ts represents time stamp, sn: sensor name, sv: sensor value, l: the activity label from the set A of n activity labels. Our main goal is to build a mapping function (MF) that assign new activity traces for simple or complex activities with the correct label(s). Below we list a set of definition required in our proposed approach [13]. Itemset, in our task an item i corresponds to a pair of sensor name and value, such that a set of items forms an itemset = { 1 , 2 , … . }.
The support, for an itemset X in dataset D, referred as s . The Growth rate, for an itemset X from dataset 1 to dataset 2 referred as ( , 1 , 2 ) is computed as: Emerging Pattern (EP), a pattern X is EP, ( ) = { : ( , 2 ) > , ( , 1 , 2 ) ≥ ρ}, where is the threshold of minimum support, and ρ the threshold of minimum growth rate. With regard to the task of HAR, EPs represent the discriminative features between activity classes. Therefore, EPs make absolute separation. In addition, the number of the extracted EPs will be extremely high. In order to reduce the number of the extracted EPs, we introduce the MinEPs.
Minimal Emerging Pattern (MinEP), a pattern X is MinEP from Strong Jumping Emerging Pattern (SJEP), a pattern X is SJEP from 1 to 2 , ( , 1 , 2 ) = { : ( , 1 , 2 ) = ∞ , ∄ ⊂ : ( , 1 , 2 ) = ∞}. Our main goal is the recognition of complex human activities using a training dataset of simple activities, so, we need discriminative features that are exclusively present in each activity class. This is achieved with the specific type of EPs called SJEPs that combine the discriminative characteristics of JEPs and MinEPs. A detailed discussion about different types of EPs with example will be found in [14].

THE PROPOSED APPROACH
In this paper, we present a novel approach for the recognition of simple and complex human activities using a combination of Emerging Patterns and Fuzzy Sets. This is accomplished in two main phases training and testing as shown in Figure 2. In the first phase, we use a training dataset of simple activities and produce; the discriminative SJEPs for each activity class, besides sensor/activity correlation matrix. Then, in the second phase, we segment the incoming sensor data, and then each segment is assigned with the corresponding label(s) using our constructed scoring function. Below, we discuss these two phases in details.

The training phase
The main goals of this phase are the mining of the SJEPs, computation of the discriminative score of SJEPs, and computation of the sensor/activity correlation matrix required at the testing phase.

Mining the strong jumping emerging patterns
From the characteristics of SJEPs introduced before, they make one-versus-all absolute separation (i.e. separate one activity class from the remaining classes). Given a pre-processed dataset of trace sequences for simple activities (i.e. represented as a pairs of sensor name and value), Algorithm 1 outputs the corresponding SJEPs. For each activity class , = 1, … ,

3.
Consider dataset D i contains observation sequences from class , and D′ i contains observation sequences from − 1classes, D′ i = ⋃ D j k j=1 , i ≠ j 4.
Compute ItemSet_List of all itemsets in dataset using the FP-growth algorithm [15]
Add candidate to the set of SJEPs for class 11.

Computation of correlation matrix
The other feature computed in the training phase is the sensor/activity correlation matrix. For each sensor s from the set of sensors = { 1 , 2 , … } used for monitoring activity a from the set of activities = { 1 , 2 , … }, we compute a correlation matrix that measures the interdependency between each sensor and activity using the support ( , ) defined in Section 3.

The testing phase
The input to this phase is a stream of sensor readings for the performed human activities to be recognized. In order to identify these activities, the input stream should be segmented beforehand. Regarding trace segmentation, there exist two main approaches, fixed-size segmentation, and dynamic segmentation. In this paper, we are interested in fixed-size segmentation that uses a fixed-length sliding window to split the given trace into equal-sized time segments, which add the following challenge. The size used for segmentation is considered a challenge, because activities occur with different durations. As a result, activity can be distributed among a number of segments. To resolve this issue, while computing score at specific segment, we take into account the score in previous segment with parameter α that is the _ refer (2).
For the task of activity recognition, within a specific test segment the existing activities executed sequentially or in parallel may be fully contained or partially contained. From this perspective, we make use of fuzzy set theory; that is an activity exists with a specific degree of membership value. Using a combination of the SJEPs extracted in the training phase and the fuzzy set theory, we propose a SJEP-based algorithm for the recognition of simple and complex human activities that consist of three main phases as presented in Algorithm 2. First, we compute the matching-score that identifies the candidate activities existing in the current segment refer (1). Then, we compute the likelihood score (like_score) for each candidate activity using a scoring function refer (3). Finally, using the computed likelihoods and a predefined threshold, we make a decision on the winning activities from these candidates using the following rule.
For each SJEP in the sets SJEPs 4.
Add Label to 8.
For each candidate activity in with cardinality 11.
For each item e in the current segment with cardinality l 12.

End For
For each activity in 18. If Add to ( ) 20.

21.
End For

Definition 4.1:
The matching score (match_score) of the set 1 of SJEPs with the set 2 of sensor events within a specific test is defined as follows

EXPERIMENTAL RESULTS AND EVALUATION
In this section, we present the experimental evaluation of the proposed approach. First, we introduce the datasets used, followed with our settings for the evaluation, and then we present and discuss our obtained results. Regarding the datasets used, we evaluate our approach using two datasets with different monitoring approaches; CASAS benchmarking dataset [16], and dataset of simple, interleaved and complex activities refereed as SICA from [11]. CASAS dataset was collected from a smart home equipped with different types of environmental sensors (e.g. motion, door, temperature), in which 20 participants collected a set of simple and complex observation sequences for eight different ADL activities presented in Table 1. On the other hand, SICA contains observation sequences of simple and complex execution of 26 different activities collected by four volunteers using two sets of wearable sensors as listed in Table 2. Table 1. Activities Collected in CASAS dataset [16] No. Activity 1 Take Medicine 2 Watch DVD 3 Water Plants 4 Answer phone 5 Write Birthday Card 6 Prepare Meal 7 Clean 8 Select an outfit To evaluate our work, we use 3-fold cross validation with CASAS and 10-fold cross validation with SICA dataset, besides the three standard evaluation metrics: Precision, Recall, and F-Measure.
Using the two different datasets, and fixed-time segmentation approach with different sliding windows moving around the average window length ( ), we perform two experiments. In each experiment, first we compare the performance of our approach for simple activity recognition with three popular classifiers, Support vector Machine (SVM), Hidden Markov Model (HMM), and K-Nearest Neighbor (K-NN) as shown in Figure 3. For the recognition of complex activities, we compute the precision, Recall, and F-measure for our approach and compare it against methods in [6] and [11] as depicted in Figure 4 and Figure 5 respectively, where the X-axis refers to the window size and the Y-axis refers to the evaluation metric. Regarding the parameters α, , we reach the best resuts with, α = 0.3, λ = 0.1 in CASAS dataset, and = 0.3 , = 0.8 in SICA dataset.
From the results shown in Figure 3, there is a close competition between our approach, SVM and K-NN except HMM with the lowest value. However, our approach achieves the highest performance for the recognition of simple activities in both datasets. Regarding Precision, Recall, and F-Measure for complex activities, the results shown in Figure 4 and Figure 5 indicate the superiority of our approach against the others in [6,11] at large windows containing multiple overlapping activities. In Figure 4, using CASAS dataset, our approach achieves the highest F-measure rate at window size 1.5 × L avg which is 92.3%, while the other in [6] reaches 81.1% . The same in SICA dataset, we reach 92% for F-Measure, that is almost 10% more than the other in [11].