Vanderbilt University
Robotics and Autonomous Systems Lab (RASL)
A novel interface system for seamlessly integrating human-robot cooperative activities in space Nilanjan Sarkar
Mechanical Engineering Electrical Engineering and Computer Science
Craig A. Smith
Psychology and Human Development
Vanderbilt University Nashville, TN
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Inspiration “Exploration of space and solar system will be most effective if human capabilities are synergistically combined with those of robots. Such a human-robot system, developed correctly, will reduce exploration risks, improve efficiency, and achieve overall mission goals faster and in a better manner.� [Objective of the ICASE/USRA/LaRC Workshop on Revolutionary Aerospace Systems Concepts for Human/Robotic Exploration of the Solar System, Nov. 2001]
Robotics and Autonomous Systems Lab (RASL)
Major Obstacle Vanderbilt University
Natural interaction between human and robot ¾ Is implicit communication possible? - e.g., robot brings the right tool to the astronaut without being explicitly commanded
¾Can the robot understand the psychological states of the human? - e.g., robot rushes to help the human if the robot senses “panic”
Robotics and Autonomous Systems Lab (RASL)
Novel Approach Vanderbilt University
他affect detection and recognition 他brainwave monitoring and characterization 他design of control architecture for implicit communication 他integrates research in signal processing, wearable computing, experimental psychology and control theory
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Hypotheses A robot will: 他implicitly understand a task command from an astronaut 他sense the psychological state of the astronaut and take necessary actions
Robotics and Autonomous Systems Lab (RASL)
Rationale Vanderbilt University
¾
¾
Initially, the research will be:
person specific
context specific
Afterwards, with enough understanding and data, an affect recognizer for a class of people will be attempted
Robotics and Autonomous Systems Lab (RASL)
Phase I Tasks Vanderbilt University
The following tasks were proposed:
A. Develop Affect Recognizer 1. Design human subject experiments to elicit target affective states (e.g., engagement, anxiety, fatigue, etc.) 2. Assess physiological indices that are important for the target affective states 3. Conduct pilot studies 4. Data analysis and signal processing for affect detection
B. Investigate the currently available brainwave monitoring technologies
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Phase I Tasks The following additional tasks were performed: 他
Developed a functional human-robot system that is affect-sensitive, and
他
Conducted preliminary brainwave monitoring experiments with EEG
Vanderbilt University
Robotics and Autonomous Systems Lab (RASL)
Development of Affect Recognizer
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Experimental Tasks Three Problem-Solving Tasks (Each lasting ~1 hour) 他
Anagrams
他
Math Word Problems
他
Sound Discrimination
Two Sequences for Each Task 1.
Easy: Fairly Easy --> Trivially Easy
2.
Difficult: Fairly Easy --> Virtually Impossible
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Sample Anagrams Easy Condition
Difficult Condition
AWADR
IYTED
AWARD
DEITY
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Easy Math Problem An astronaut requires 2 pounds of oxygen per day while in space. How many pounds of oxygen are needed for a team of 3 astronauts who are going to spend 5 days in space?
Answer: 30
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Difficult Math Problem Tammy has $9.70 in nickels, dimes, and quarters. The number of nickels is 4 more than 3 times the number of dimes, and the number of quarters is 5 fewer than 2 times the number of nickels. How many nickels does Tammy have?
Answer: 19
Robotics and Autonomous Systems Lab (RASL)
Sound Task Vanderbilt University
他 他 他
Sequence of three tones Judge whether first and third tones are the same or different Difficulty manipulated by varying tone length and frequency difference between tones Easy Trials
Difficult Trials
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Physiological Data Collection
Robotics and Autonomous Systems Lab (RASL)
Physiological Measures ¾EKG
Vanderbilt University
¾Skin
interbeat interval (IBI) mean variability
sympathetic power * parasympathetic power *
¾Finger
¾Digit
Transit Time (PTT)
Skin Temperature
mean slope of change
Tonic
¾Facial
response rate * average amplitude * maximum amplitude
Muscle Activity (EMG)
Brow (Corrugator)
mean slope of change
Phasic
Pulse Amplitude
mean variability
¾Pulse
Conductance
mean variability *
Jaw (Masseter)
mean * variability
Robotics and Autonomous Systems Lab (RASL)
Self Report Measures ¾Sampled:
Vanderbilt University
Two minutes into each task
¾ Key
Affective Parameters:
Every seven minutes thereafter
¾Assessed:
Task Difficulty
Perceived Ability
Affective States
Anxiety Index
Anxiety
Overload
Engagement Index
Task Importance
Hope
Challenge
Interest
(R) Resignation
(R) Apathy
Robotics and Autonomous Systems Lab (RASL)
Correlations of Anxiety with Physiology Vanderbilt University
0.5 0.3 0.1 Subject 2
-0.1
Subject 4
-0.3 -0.5
Symp
Paras
SCRRate
SCRAmp
CorrVar
Physiological Parameter
Mass
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Correlations with Physiology: Anxiety vs. Engagement (Subject 2) 0.7 0.5 0.3 0.1 Anxiety
-0.1
Engagement
-0.3 -0.5
Symp
Paras
SCRRate
SCRAmp
CorrVar
Physiological Parameter
Mass
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Schematic of Fuzzy Logic Analyzer Sympathetic Power
Parasympathetic Power
Skin Conductance Response Rate
Skin Conductance Response Amplitude
Corrugator Variability Masseter Average
Fuzzy Logic Analysis
Affect Diagnostic Output
Vanderbilt University
Robotics and Autonomous Systems Lab (RASL)
Development of a Functional Human-Robot Interactive System
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Our Robot Operates in three modes: 1. Explore -- Wanders about environment 2.
Survival -- Uses sonar to detect and avoid obstacles in environment
3.
Affective -- Responds to affective signals from human operator ™
If robot senses high anxiety, it uses light detection algorithm to go to operator
Robotics and Autonomous Systems Lab (RASL)
Robot Control Architecture Vanderbilt University
A hybrid subsumption control paradigm Survival Mode
Emergency Stop Reverse Affect Layer Sensor Data
Obstacle avoidance Wall Follow Wandering
S
S
Deliberative Mode
S
Reactive Mode
S
S
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Human-Robot Interaction Scenario
Human operator works at computer task while the robot explores the environment and monitors operator’s affective state
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
The Functioning System
QuickTime™ and a DV/DVCPRO - NTSC decompressor are needed to see this picture.
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Physiological Signals & Fuzzy Logic Output for Moderate Anxiety Trigger
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Timing Diagram from Robot Experiment
Vanderbilt University
Robotics and Autonomous Systems Lab (RASL)
Investigation of Brain Wave Monitoring Technologies
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Technologies to Assess Brain Activity 他
EEG - Measures electric currents generated by the brain at the scalp
他
MEG - Measures magnetic fields generated by the brain
他
PET - Measures emissions from radioactively labeled chemicals, monitors metabolic rates and blood flow
他
fMRI - Measures oxygen concentration of blood, correlates to blood flow
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Modality Comparison EEG
MEG
fMRI
PET
Spatial Resolution
7-16 mm
3-11 mm
1 mm
5 mm
Temporal Resolution
Milliseconds
Milliseconds
1 second to minutes
45 seconds to minutes
Millions
Millions
No
No
Cost
$15-120K
Wearability
Yes
$500k (37 channel) $300-400k Shielding No
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Example EEG and MEG Systems
http://www.xltek.com/xlwebv2/products/eeg/xlamb.htm http://www.vsmmedtech.com/MEG_Technical/MEG_Images_275_151.asp http://www.biomag.helsinki.fi/meg.html http://www.elixa.com/mental/MindSet.htm http://www.emedicine.com/neuro/topic445.htm
Robotics and Autonomous Systems Lab (RASL)
EEG Experiment: Cognitive Load 4000
Alpha Power
Vanderbilt University
4500 3500 3000 2500 2000 1500 1000 500 0
Eyes Closed
Eyes Open
Robotics and Autonomous Systems Lab (RASL)
EEG Experiment: Math Problem-Solving 800
Alpha Power
Vanderbilt University
900
700 600 500 400 300 200 100 0
Eyes Open
Easy
Moderate Problem Difficulty
Difficult
Robotics and Autonomous Systems Lab (RASL)
Vanderbilt University
Summary 9
performed every proposed task for Phase I
9
performed two additional tasks beyond what was proposed
9
were successful in eliciting and detecting the target affective states under controlled situations
9
developed a functional human-robot system to demonstrate the feasibility of the central concept
Robotics and Autonomous Systems Lab (RASL)
Conclusions Vanderbilt University
Phase I work demonstrates that: ¾
implicit human-robot communication is feasible can detect human affect on-line and in the context of realistic tasks control system can be made affect-sensitive
¾
brainwave monitoring will likely supplement peripheral physiology in affect detection and implicit communication
Robotics and Autonomous Systems Lab (RASL)
Future Work Vanderbilt University
¾ ¾
Expand range of tasks and contexts to which framework can be applied Increase the reliability and sophistication of affect recognition
¾ ¾ ¾
Increase range of affects detected and discriminated beyond anxiety and engagement to include frustration, fatigue, boredom, etc. Advance analytical tools for extracting relevant information from physiological signals
Increase degree to which physiological recording is ambulatory Further explore utility of EEG recording to improve affect recognition and implicit communication Formalization of affect-sensitive control system