Blue Eyes Technology - NaSCoIT

Blue Eyes Technology Mandira Bhattarai

Nishma Adhikari

Manisha Bhattarai

B.E.I.T. Nepal College of Information Technology Lalitpur, Nepal 9779860688155

B.E I.T. Nepal College of Information Technology Lalitpur, Nepal 9779860912398

B.E I.T. Nepal College of Information Technology Lalitpur, Nepal 9779860688199

[email protected]

[email protected]

[email protected]

ABSTRACT

In today’s world, technology has reached to new level unfolding its limitations and improving at a tremendous pace. How would our life be if our computer could interact like our friends, feel our physical presence, appreciate our happiness and cheer us up in our despair? Imagine, a wonderful world where the computers that humans have built can interact with them just like their colleagues. We now have reached the technology known as “BLUE EYES” technology that can sense and control human emotions and feelings through various gadgets by providing computer with human power. We all have some perceptual abilities i.e. we can perceive each other’s emotional state by analyzing facial expressions. The eyes, fingers, speech are the elements which helps to sense the emotion level of human body. Its main objective is to create the machine that doesn’t only compute but can perceive and sense the things as we humans do. By analyzing person’s physical state and emotions, the computational machines can simply understand what user desires and where his eye is focusing at. In this technology, after capturing the image it compares eye portion of the captured image to the stored images in the database and can finally perceive the emotions of a p erson like happy, sad, excited, surprised and so on. This paper is about hardware, software, applications and future prospects of “BLUE EYES” technology.

you. This all can be possible with this technology which has a phenomenal future prospect. With this technology, the computers could use facial recognition and speech recognition system to collect the information from human beings and react accordingly. The main aim of “Blue Eyes” technology is creating powerful computers that have sensory and perceptual abilities like that of a human being. This technology uses a camera and microphone to identify user actions and emotions. Blue Eyes uses a sensing technology to identify a user’s action and gather information. The information thus obtained are analyzed and the emotional state of the user is determined. It consists of both software and hardware systems which facilitates a stress free environment where humans and machine can collaborate and work together .

2. SYSTEM OVERVIEW “BLUE EYES” system monitors the status of operator’s visual attention through the measurement of saccadic activity. Two major system units are used:

Keywords CSU (Central System Unit), DAU (Data Acquisition Unit), Emotion M ouse, M AGIC (M anual and Gaze Input Cascaded), SUITOR (Simple User Interest Tracker)

1. INTRODUCTION At present, technology has become the part and parcel of human life. With each day, it offers new layers to unfold. It is undeniable how technology has simplified our life and multiplied our progress. It is an era of artificial intelligence and machine learning. “Blue eyes” technology is one of such technology that enables a machine to think and feel emotions like a man. A technology conducted by the research team of IBM at it research Centre in California since 1997, “BLUE EYES” technology gives the computer an ability of having cognitive skills and emotional characteristics like that of a human being. [6] This technology incorporates the perceptual ability and emotional gesture in a computer so that those computer could sense the emotional levels that a normal human beings go through. Imagine walking to your home all tired and depressed one day, as you open your room the computer senses your presence, greets you a “Hello!” ,understands how you are feeling and cheers you up as your friends would do to

Figure 1: System Overview [1]

2.1 Data Acquisition Unit (DAU)

“BLUE EYES” technology uses a mobile device known as DAU. This unit collects the physiological information coming from the sensors and for verification and processing purposes it further sends that information to the CSU. The wireless interface between user with sensors and CSU is provided by a Bluetooth module that is incorporated with mobile device. PIN codes and ID cards are assigned to the entire operator’s for authentication purposes. The device uses the hardware such as system-core Bluetooth section, Jazz M ultisensor, Atmel 89C52 microcontroller, EEPROM , Beeper, LCD display (HD44780), LED indicator, voltage level monitors and 6 AA batteries. It runs on batteries that is why it requires less power, is lightweight and easy to use. [2]

2.2 Central System Unit (CSU) This unit is incorporated to a personal computer using USB, serial and parallel cable. [2] It is mainly composed of four components:  Connection M anager -> Its main task is to perform low level Bluetooth communication.  Data Analysis M odule -> For obtaining the information about the physiological state of an operator, this module analyzes the raw sensor data. Some analyzers used are:  Saccade detector: It monitors eye movement to determine level of operator’s visual attention.  Pulse rate analyzer: It uses blood oxygenation signal to compute operator’s pulse rate.  Custom analyzers: It recognizes other behaviours than those which are built in the system. The new modules are created using C4.5 decision tree induction algorithm.

people with personalities that are similar or complement each other collaborate well. In today’s world it is very necessary to develop the computers that can understand our emotions and interact with us.

3.1.1 Theory Based on Paul Ekman’s facial expression work, there is a correlation between human emotions and human psychological measurements. His experiments consist of participants attached to devices that can record certain measurements like temperature, galvanic skin response (GSR), pulse, blood pressure and somatic movement. Then, he allowed the participants to mimic the different facial expressions and measured psychological measurement. He has basically given the six basic emotions to mimic such as anger, fear, sadness, disgust, joy and surprise.

3.1.2 Result The result consists of data scores for four physiological measurement i.e. GSA, GSR, pulse, and skin temperature, for each of the six emotions anger, disgust, fear, happiness, sadness, and surprise within the five minute baseline and test sessions. In a second GSA data was sampled 80 times. GSR and temperature were sampled approximately 3-4 times per second and pulse was sampled approximately 1time per second. In order to measure the difference of psychological measure of human, we find the difference between the baseline and test scores. If the difference is more than one and half standard deviation from mean was considered as missing. This obtained results gives the theory of working of the emotion mouse. [3]

3.2 Classification of Emotion Sensors 

For Hand

3.2.1 Emotion Mouse The main goal of human computer interaction (HCI) is to make the computers smart by analyzing the facial expressions of the human beings. This can analyze the facial expression, speech recognition, eye tracking and gesture recognition.

Table 1: Algorithm steps with its modification  

Data Logger M odule-> This module provides assistance in order to store the monitored data. Visualization M odule -> This module provides supervisors with user interface. Some of its features are connection management, data processing, data recording, access verification and system maintenance. Figure 2: Emotion Mouse [3]

3. TECHNOLOGIES USED 3.1 Emotional Computing In 1997, Rosalind Picard describes why emotions are necessary to computing community. There are basically two aspects of affective computing i.e. the computer are given the ability to detect emotions and the ability to express emotions. So, emotion is an important element which can be used into computing for productivity for computer user. A study (Dryer & Horowitz, 1997) has shown that

In this the people touch the mouse and all the psychological parameters are measured. People use their computers to obtain, store and manipulate data using their computer.” Emotion M ouse” can measure person’s heart rate, temperature, galvanic skin response and body movements and pair them with six emotional states: happiness, surprise, anger, fear, sadness and disgust that are stored in the database. As a result, we obtain the psychological data which can be used to determine the six basic emotion of the user i.e. the task the

user is currently doing on the computer. So in the near future the computer can sense the human emotions perfectly and can help human for various purposes. 

For Eye

3.2.2 Manual and Gaze Input Cascaded (Magic) Pointing In the past, Gaze tracking was considered as the superior comp uter input. But as the time passes the people came to know that there are various limitations with the gaze pointing. So, they developed the alternative approach i.e. M anual and Gaze Input Cascaded pointing (M agic). So, with magic pointing we can perform the manual task in order to manipulate and select the particular target. The manual task cannot be performed in gaze tracking as in the magic pointing. In this magic pointing, the cursor is warped to the eye gaze area i.e. towards the target. There are basically two approaches on the basis of cursor placement and target identification. They are Liberal approach and Conservative approach. Liberal Approach In liberal approach, the user can control the cursor on or near the target by the hand .There is 120 pixel threshold so, the cursor cannot move to particular target until the threshold is met. The liberal is “pro-active,” since the cursor waits readily in the anticipated target or on every potential target. The user can make the manual movement to the cursor by hand where he /she is actually looking at in order to get the actual targeted object. The cursor is overactive when the user looks at the same position for the long time when the user does not intend to look at that position.

Figure 4: Conservative approach [3] Both the conservative and the liberal has the following advantages: 1. They help in the reduction of the fatigue and the mental stress. 2. It increases the accuracy. 3. It is a natural model for the user. 4. It help in increasing the speed as manual modifications can be done. 5. It is simple, so it is easy to use.

3.2.3 Eye Tracker

Figure 5: Bright (left) and dark (right) pupil images resulting from on- and off-axis illumination. The glints, or corneal reflections, from the on- and off- axis light sources can be easily identified as the bright points in the iris. [3]

Figure 3: Liberal Approach [3] Conservative Approach In conservative approach, the manual input device should be used in order to move the cursor to a target. Until the manual input device is not activated, the cursor does not warp to the p articular area or the particular target. In this approach, the cursor is on the boundary of the gaze area and it requires the manual input by the hand to move the cursor to the particular target. When the user looks at a target and moves the input device by the manual input device, the cursor will appear “out of the blue” while moving towards the particular target, on the side of the target opposite to the initial actuation vector.

The light source and the camera optical axis are placed on the axis . After they are placed on the axis, then the light is reflected from the interior portion of the eye and this is detected by the camera. In this condition the image of the pupil appears bright. This effect is often can be compared to the red-eye in flash photographs when the flash is close to the camera lens. So, the Bright (left) and dark (right) pupil images resulting from on- and off-axis illumination is shown in the figure above. The glints, or corneal reflections, from the on- and offaxis light sources can be easily identified as the bright points in the iris. The Almaden system uses two near infrared (IR) time multiplexed light sources, composed of two sets of IR LED’s, which were synchronized with the camera frame rate. One light source is placed very close to the camera’s optical axis and is synchronized with the even frames. Odd frames are synchronized with the second light source, positioned off axis. The two light sources are calibrated to provide approximately equivalent whole-scene illumination. [4] 

For Voice

3.2.4 ARTIFICIAL INTELLIGENCE RECOGNITION (AISR)

SPEECH

For the Artificial Intelligence Speech Recognition system to be implemented effectively, it is necessary to consider the grammar used by the speaker and accepted by the system, noise level, noise type, position of the microphone, and speed and manner of the user’s speech are some factors that may affect the quality of speech recognition. Siri (Apple), Google Now (Android) and Cortana (Windows) which all serves as intelligent personal virtual assistants on different platforms are relevant example of an artificial intelligence. These platforms deploys the action and information when user asks “Where is the nearest station?’ or “Call the person ABC”. [1] Artificial Intelligence is based on two basic features. The first one is it determines the behavior pattern and thinking process of the human beings and implements these process through the machines. Intelligence is more to do with a 'situated', 'interactive' and 'real time' concern. With the use of artificial intelligence, machines become smarter and less expensive than natural intelligence. Natural language processing (NLP) refers to artificial intelligence methods of communicating with a computer in a natural language like English. In NLP, the input words are scanned and matched with internally stored word in the database.

3.3 The Simple User Interest Tracker (SUITOR) SUITOR is an advanced approach which tracks the area of the computer screen where the user’s eye is focusing. It helps to determine the topic of interest of a user through such tracking and delivers the information to the hand held devices. It fills a scrolling ticker on the computer screen with the information related to the user’s task. SUITOR knows where you are looking, what applications are you running and which web pages you are browsing. [1] Suppose if I am reading a web page about Apple, the system will display the stock price and news stories that could affect apple. Similarly, if I am going through the headline of any news, this system will pop up the news story in the browser window. SUITOR is an attentive system which is attentive to user’s actions and supplies the information resources as per the user’s interest.

4. APPLICATIONS

S peech Recognition The user speaks to the computer through a microphone which contains three filters. The more number of filters denotes probability of accurate recognition. At present, due to the custom built in integrated circuit, switched capacitor digital circuits are mostly in use. The filter output is passed to ADC to convert analog signal to digital word. Each second, ADC samples the filter output many times. Each of these sample represents different amplitude of the signal. Each value is then converted to a binary number proportional to the amplitude of the sample. These digital values are stores in a buffer area in a large RAM and central processor unit (CPU) controls the input circuits that are fed by the ADCS. This digital information which represents the spoken word, is now accessed by the CPU to process it further. The binary representation of each of these words are considered as a standard template, against which the future words are compared. These templates reside in the memory. After the completion of the process, the system becomes active and is capable of identifying spoken word the computer starts to search and compares the binary input pattern with the templates. [It is to be noted that even if the same speaker talks the same text, there are always slight variations in amplitude or loudness of the signal, pitch, frequency difference, time gap, etc.] Due to this difference, the template and binary input word never match perfectly. The pattern matching process designs the best fit through statistical technique.

- Security and control systems can be controlled using this technology. T he s urveillance cameras not only cap t ure image but t hey do have a vis ion. They can recognize the emotions of the user and assist s ecurit y t o focus on t he p ers ons w hos e emot ional condit ions are ab normal and s eemingl y dangerous . - Assisting Human Operator by monit oring and recording t he op erat or’s p hy s ical condit ion. - Driving systems can be cont rolled by blu e ey e t echnology w hich in t urn can be help ful in reducing t he number of accident s hap p ening in t he w orld. - M edical supervision is also possible by letting the doctors to go through data of number of patients and make related notes. Detection of physiological situation of patient like blood pressure, oxygenation, pulse rate and so on. - It can be used in lie detectors in smart cameras, processing of emotional speech, analyzing the customer movements to detect their intentions. - In automobile industry, power stations and video games. - M ilitary, flight control centers, operating theatres, oven and refrigerators can also use this technology. - To create “Face Responsive Display” and “Perceptive Environment” Generic Control Rooms. [5]

The values of binary input words are subtracted from the corresponding values in the templates. If both the values are same, the difference is zero and there is perfect match. If not, the subtraction produces some difference or error. The smaller the error, the better the match. In case of best match, the word is identified and displayed on the screen. As the CPU has to make many comparisons before recognition occurs, it may take time. This requires a large processor and large RAM . This process of matching the word and template correctly is known as dynamic time warping. It recognizes that different speakers pronounce the same words at different speeds as well as elongate different parts of the same word. [1]

In future “BLUE EYES” technology can be implemented for leading human life in much more simpler way. For example; one can switch on or off Television, music player, a/c, fan, washing machine, oven, coffee machine etc. by just having one look at them or by instructing them t hrough voice commands . Imagine t hat t echnical w orld w here comp ut ers can dis p lay hap p y or funny p ict ures and mails in t he morning t o cheer y ou up w hen y ou are in des p air . [5]

5. FUTURE PROSPECTS

The hardware used can be further improved by using the smaller and less intrusive units in future such as small CM OS camera to monitor operator’s point of gaze, low voltage ICs, use of data mining algorithms for advanced encryption of database and so on. That day

is not too far when this technology will push its way to the hand held mobile devices making the humans more dependent in this technology, Therefore, despite of increase in comfortability, this technology might have some of its negative sides that can have bad impact on the mankind.

6. CHALLENGES “BLUE EYES” technology can face both technological and implementation challenges:  The implementation of gaze tracking method is not so accurate which might annoy the users.  Tracking and implementation of speech recognition for languages with local dialect variations for the same becomes very tedious as it needs to have grammatically correct language having neutral dialect to convert the recognized words into an action.  Single gesture might have multiple interpretations. So, the information obtained from a gesture might not be the correct one.  Distance might affect the accuracy of the information obtained and results produced.  Climate can have effect on the implementation of “Blue Eyes” technology such as in case of electrodes implantation in order to translate under rain or snow. [5]

7. CONCLUSION The “BLUE EYES” technology makes computation devices more user friendly and also makes human life much more

comfortable. It is developed to fulfill the need of human operator for the system monitored in real time. It is helpful in reducing the gap between humans and machines they use as it makes machine to interact with us like our peers. Further research is going on in order to take this technology to next level overcoming all the challenges to make our world of imagination come true.

8. REFERENCES [1] http://dspace.cusat.ac.in/jspui/bitstream/123456789/2221/1/BL UE%20EYES%20TECHNOLOGY.pdf,BLUEEYES TECHNOLOGY.pdf

[2] https://www.mepits.com/project/184/Embedded-Projects/BlueEyes-Technology---M onitoring-Human-Operator-andIntelligence-sensing-System

[3] http://www.ijarse.com/images/fullpdf/267.pdf, BLUE EYES TECHNOLOGY by Swati

[4] Kenneth Holmqvist, M arcus Nystrom, and Fiona M ulvey. Eye tracker data quality: M eaning and method to measure it. In Proceedings of the Symposium on Eye Tracking Research and Applications, pages 45-52. ACM , 2012. [5] https://www.scribd.com/doc/41742795/Blue-Eye-TechnologyFuture-of-computers [6] Joseph j.carr and john m.brown “Introduction to Blue Eyes Technology” , published in IEEE spectrum magazine.II.A.jajszczyk,”Automatically switched Blue eye networks:Benefits and Requirement,” IEEE blue tooth.feb 2005,vol 3,no. 1, pp.