Information Retrieval from Emotions and Eye Blinks with help of Sensor Nodes

Received Feb 2, 2018 Revised Apr 30, 2018 Accepted May 17, 2018 In everyday life, there are situations where the only way to communicate are emotions. EMOTICONS are the epitome of the same. This aspect of communication can also be used in emergency situations (terrorist attacks, hijacks) where the only way to communicate is by performing some extraordinary actions or through some emotions. Incorporating technology to the above mentioned circumstances the paper proposes a novel framework of detecting an emergency situation by retrieving information from emotions and eye blinks using sensor nodes. The proposed framework can be deployed in places (hotels, banks, airports etc.) which are more suspected to attacks. The framework takes input from real time parameters: eye blinks, emotions, heart rate. Based on behavioral changes, biological changes and physical changes the proposed framework extracts meticulous information. The proposed framework is further validated through implementation of a facial emotion recognition system that successfully recognizes various human emotions. The facial emotion recognition system of the proposed framework is compared with existing SVM technique in terms of accuracy, training and testing error. Accuracy with the proposed system is increased to 78.40% in comparison with existing SVM that is 75.37% and the training error is decreased to 0.004103 whereas with the existing SVM method training error is 0.008935. Keyword:


INTRODUCTION
In recent times many events of hostages and hijacks have happened and there is an urgent need of an automated system which can make the concerned agencies aware of the situation in time and thus help them to take necessary action.Wireless Sensor Networks can be used in attack prone areas to examine the situation and hence will help the concerned agencies to take appropriate action in time.
The 26/11 Mumbai attacks were a planned series of attacks, when a group of 10-12 terrorists of Lashkar-e-Taiba, carried out a sequence of coordinated bombing and shooting that continued for four days [1], [2].Densely populated areas like railway station and 5 star hotels (Hotel Taj) were their main targets.The attacks, which were heavily censured globally, began on 26 th November and lasted until 29 th November 2008, killing 164 people and wounding at least 300.A lot of damage had already been done even before the security agencies came to know what actually had happened.So if there would have been such automated systems, a lot would have been saved in terms of lives and public property.Another such event of hostage includes hijacking of Indian Airline Flight 814 en route from Tribhuvan International Airport, Nepal to Indira Gandhi Airport, India on Friday, 24 December 1999.The most recent Uri attack was an inhuman attack by heavily armed terrorists in Uri area of Baramulla district, Jammu and Kashmir.It is considered as "the deadliest attack on security forces in last two decades".No terrorist group has claimed responsibility for the same.
In this paper we propose a novel framework for detecting an emergency situation which uses the intelligent information retrieved from pulse rate and eye blinks [3] using wireless sensor nodes as well as behavioral patterns [4], [5] exhibited by humans in response to any changes in the former parameters.To the best of our knowledge, such security framework that confirms emergency or intrusion detection on all given parameters does not exist.
The parameters in response abnormal situation discussed here are: a. Eye blink pattern, b.Human Emotions, c. Heart rate For each parameter we intent to analyze the behavioral (emotion), biological (pulse rate) and physical (eye blink) pattern changes as measured by the systems deployed which can in future be used to install an emergency or intrusion warning system.
Author in [6] have presented a novel approach to detect whether the eyes in an image (still) are closed or not and this approach has wide ranging applications in facial expression recognition, fatigue detection, and so on.A similar analysis in [7] has been discussed to detect eye blinks in a video sequence from a standard camera.Emotion is also one of the parameters which play an important part in our proposed framework.In [8] a regional hidden Markov model (RHMM) is proposed for recognizing facial expressions in video sequences.Facial action units are described for eyebrows, eyes and mouth regions by RHMMs.Proposed technique has outperformed other existing methods when tested with extended Cohn-Kanade database.Author in [9] presents an Automatic Face Analysis (AFA) system to inspect facial expressions taking into consideration both permanent facial features (brows, eyes, mouth) and momentary facial features (deepening of facial furrows).

RESEARCH METHOD
Sensor Nodes as known for the monitoring abilities such as accuracy and sensitivity are used as the baseline for the proposed framework.Figure 1 describes the proposed Framework.
The intelligent information collected from standalone parameters, will act as input to the system which after performing calculations will decide whether it has crossed the threshold or not.Upon crossing the threshold an alarm or buzzer or some message will be transmitted to the concerned authorities.The sensors nodes will be deployed to extract the profound information from the given parameters, thus can be used as forerunner for provoking the alarm messages.a. Eye blink This attribute is of utmost importance as far as our proposed framework is concerned.Average eye blinking rate is around 12-19 per minute.(Intentional Eye blinks > 25 Abnormal).
Proposed framework uses the sensor placed in the line of sight of the eye which continuously studies the blink pattern.Sensors used will draw a curve to indicate a change in eye blink [10], [11] rate. b.Human emotions Emotion is a cognitive state that is generated impulsively rather than through responsive effort and is accompanied by physiological changes which are familiar to the outer world.Some emotions which humans face in everyday life are worry, surprise, confuse, happy etc. Human vision can practice emotion as coupled with personality, temperament, and mood.Computer Vision tries to follow the human vision by analyzing digital image as input.Sensors deployed will continuously study the human emotions.A database of emotions with an instance of time will be maintained with sensor nodes.c.Heart rate Heart rate is the number of times a heart beats in a minute.Heart rate [12] varies from person to person depending on one's age, body size, fitness and heart condition.The same also depends on whether the person is running, standing, sitting, moving or is under influence of some medication.Emotions can also increase the heart rate.A healthy heart beats 60 to 100 times in a minute and can spike up due to stress, overexertion or nervousness.Our proposed framework will continuously monitor the heart rate.
The proposed framework unfolds the entire structure of the process combined together.Smart motes (nodes) will be installed as per the requirements.Three types of sensors (eye blink, emotion detection and heart rate sensors) will be deployed.The sensors will work independently and will transfer the sensed attributes to the smart server via smart gate where the actual feature selection will take place.2435 some observation, calculation and comparisons, the final verdict (whether the threshold is reached or not) will be taken.If threshold is crossed, alarm will be raised.For example, if the eye blink sensor senses an unusual blink pattern or number, in addition to a Fear or Surprise emotion detected by the emotion detection sensor or an abnormal heart rate, the situation is an alarming one.Blink pattern and emotions can be calculated by still or moving images via specialized cctv's.For heart rate concerned persons (bar tenders in hotels, bank managers, flight attendants) will be equipped with wireless heart rate sensors or straps.

Figure 1. Proposed Framework for Exigency detection
The proposed framework works in two phases.The first phase (analysis phase) takes input from eye blinks, emotions and heart rate.The second phase constitutes the classification phase.The reactions could be change in behavior or emotions or physiological over given time.In this paper a Perspicacious Information Retrieval technique which senses serious diversion of parameters from baseline values via sensors spanned across target zones is presented.The data is transmitted to a base station which is further processed at main server.The moment major deflection from normal is observed using the information gathered, sensors are trained to issue alarm signals as in when such abnormal activity is sensed.

Implementing information retrieval technique
Figure 2 shows how methodically Perspicacious Information Retrieval Technique (PIRT) is implemented at servers where messages stating unusual human emotion, pulse rate and blink pattern are received at base station.It is important to know that input values would be gathered from some trained staff (air hostess in airplanes, bartenders in hotels, some officials in banks etc) having the knowledge of how to react in an emergency situation.
The proposed framework should be deployed in areas which are in direct line of sight of the concerned persons and will help generate alarm signals as soon as change in behavioral or blink patterns is sensed.Messages acquired are further refined in order to excerpt intelligent information from it.Figure 2 presents a novel prognostic algorithm which takes as input sensed data from the preceding stages and figure out certain conditions on the data received.Various stages of PIRT are: a. Z1: Eye blink This standalone condition evaluates the Blink pattern (BP) and hence the numbers of Blinks in say 1 minute.Average eye blinking rate is around 12-19 per minute.So an unusual number/pattern will be the indication that something abnormal is there.For a precise decision we are taking it 25 Blinks per minute.The value stated which approximately accounts to 25 is the lowest default value.As no signs of change in blink pattern are observed below value -No alarm will be raised.For getting the BRM (blink rate per minute), frames from the video sequence can be extracted and will be further processed for detecting face and hence the eye portion of the image.From the eye landmarks [7] detected from the image, the eye aspect ratio (EAR) that is used as an approximate of the eye opening/closing state can be calculated.The eye aspect ratio (EAR) 2. For a given pattern say number of blinks be BP

Facet value analysis
The facet evaluation is done on the sensed data to further figure out conditions and generate alert signals whenever condition Z1, Z2 or Z3 evaluates to true/false in certain permutations or combination.Based on above analysis we validate a framework that can hatch alert signals based on messages received by the sensors deployed.Figure 3 shows some cause effect graphs which state that condition Z1 in conjunction with Z2 or Z3 or both is a necessary prerequisite for the framework to issue warning signal.

Emotion detection
The paper presents a detailed analysis of one of the parameters of the proposed framework: Human Emotions [13], [14].A Facial Emotion Recognition System is proposed that would recognize 7 emotions labeled Neutral, Angry, Disgust, Fear, Happy, Sad and Surprise using Matlab.The proposed system will extract emotions on the basis of some unique features.Figure 4 presents a novel algorithm for detecting emotions from an image.The image to be tested can be taken from a smart phone using an app IP Webcam which turns the android phone into wireless camera.

RESULTS AND DISCUSSION
The performance of proposed Facial emotion recognition system is evaluated and Table 1 showcases the comparison between the proposed method and existing SVM during the training and testing phase.The proposed method is compared with existing SVM (support vector machine) technique in terms of accuracy, training and testing error.Accuracy and training error comparison between SVM and the proposed method is shown in Figure 5. Accuracy with the proposed method is increased to 78.40% in comparison with existing SVM that is 75.37%.Training error for the proposed method is decreased to 0.004103 whearas with the existing SVM method training error is 0.008935.6 shows the image which system accepts as input and Figure 7 depicts the resultant emotion (neutral) being detected.Two different resultant emotions (surprise, happy) are also shown in Figure 8 and Figure 9. Training error is shown in Figure 10 which reduces with each iteration.

Figure 2 .
Figure 2. Perspicacious Information Retrieval Technique for exigency detection

Figure 3 .
Figure 3. Cause graph based on condition Evaluated

Figure 5 .
Figure 5. Accuracy and training error comparison between SVM and the Proposed Method

Figure
Figure6shows the image which system accepts as input and Figure7depicts the resultant emotion (neutral) being detected.Two different resultant emotions (surprise, happy) are also shown in Figure8and Figure9.Training error is shown in Figure10which reduces with each iteration.

Table 1 .
Comparison between Proposed Method and SVM Deepali Virmani received the doctoral degree in Computer Science from Netaji Subhas Institute of Technology, New Delhi in 2012.Her research interests are in the areas of Sensor Networks, Data Mining and Security.She has published more than 60 papers in referred International journals and conferences.She is branch counselor IEEE student chapter and CSI student branch and has more than 15 years of teaching experience.She is currently working as Associate Professor (Head of Computer Science Department) in Bhagwan Parshuram Institute of Technology approved by Guru Gobind Singh Indraprastha University, New Delhi, India.
ISSN: 2088-8708  Information Retrieval from Emotions and Eye Blinks with help of Sensor… (Puneet Singh Lamba) 2441 Dr.