Humanrobot

Vanderbilt University

Robotics and Autonomous Systems Lab (RASL)

A novel interface system for seamlessly integrating human-robot cooperative activities in space Nilanjan Sarkar

Mechanical Engineering Electrical Engineering and Computer Science

Craig A. Smith

Psychology and Human Development

Vanderbilt University Nashville, TN

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Inspiration “Exploration of space and solar system will be most effective if human capabilities are synergistically combined with those of robots. Such a human-robot system, developed correctly, will reduce exploration risks, improve efficiency, and achieve overall mission goals faster and in a better manner.�

[Objective of the ICASE/USRA/LaRC Workshop on Revolutionary Aerospace Systems Concepts for Human/Robotic Exploration of the Solar System, Nov. 2001]

Robotics and Autonomous Systems Lab (RASL)

Major Obstacle Vanderbilt University

Natural interaction between human and robot ¾ Is implicit communication possible?

- e.g., robot brings the right tool to the astronaut without being explicitly commanded

¾Can the robot understand the psychological states of the human?

- e.g., robot rushes to help the human if the robot senses “panic”

Robotics and Autonomous Systems Lab (RASL)

Novel Approach Vanderbilt University

他affect detection and recognition 他brainwave monitoring and characterization 他design of control architecture for implicit communication 他integrates research in signal processing, wearable computing, experimental psychology and control theory

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Hypotheses A robot will: 他implicitly understand a task command from an astronaut 他sense the psychological state of the astronaut and take necessary actions

Robotics and Autonomous Systems Lab (RASL)

Rationale Vanderbilt University

¾

¾

Initially, the research will be:

person specific

context specific

Afterwards, with enough understanding and data, an affect recognizer for a class of people will be attempted

Robotics and Autonomous Systems Lab (RASL)

Phase I Tasks Vanderbilt University

The following tasks were proposed:

A. Develop Affect Recognizer 1. Design human subject experiments to elicit target affective states (e.g., engagement, anxiety, fatigue, etc.)

2. Assess physiological indices that are important for the target affective states 3. Conduct pilot studies 4. Data analysis and signal processing for affect detection

B. Investigate the currently available brainwave monitoring technologies

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Phase I Tasks The following additional tasks were performed: 他

Developed a functional human-robot system that is affect-sensitive, and

他

Conducted preliminary brainwave monitoring experiments with EEG

Vanderbilt University

Robotics and Autonomous Systems Lab (RASL)

Development of Affect Recognizer

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Experimental Tasks Three Problem-Solving Tasks (Each lasting ~1 hour) 他

Anagrams

他

Math Word Problems

他

Sound Discrimination

Two Sequences for Each Task 1.

Easy: Fairly Easy --> Trivially Easy

2.

Difficult: Fairly Easy --> Virtually Impossible

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Sample Anagrams Easy Condition

Difficult Condition

AWADR

IYTED

AWARD

DEITY

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Easy Math Problem An astronaut requires 2 pounds of oxygen per day while in space. How many pounds of oxygen are needed for a team of 3 astronauts who are going to spend 5 days in space?

Answer: 30

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Difficult Math Problem Tammy has $9.70 in nickels, dimes, and quarters. The number of nickels is 4 more than 3 times the number of dimes, and the number of quarters is 5 fewer than 2 times the number of nickels. How many nickels does Tammy have?

Answer: 19

Robotics and Autonomous Systems Lab (RASL)

Sound Task Vanderbilt University

他 他 他

Sequence of three tones Judge whether first and third tones are the same or different Difficulty manipulated by varying tone length and frequency difference between tones Easy Trials

Difficult Trials

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Physiological Data Collection

Robotics and Autonomous Systems Lab (RASL)

Physiological Measures ¾EKG

Vanderbilt University

¾Skin

interbeat interval (IBI) mean variability

sympathetic power * parasympathetic power *

¾Finger

¾Digit

Transit Time (PTT)

Skin Temperature

mean slope of change

Tonic

¾Facial

response rate * average amplitude * maximum amplitude

Muscle Activity (EMG)

Brow (Corrugator)

mean slope of change

Phasic

Pulse Amplitude

mean variability

¾Pulse

Conductance

mean variability *

Jaw (Masseter)

mean * variability

Robotics and Autonomous Systems Lab (RASL)

Self Report Measures ¾Sampled:

Vanderbilt University

Two minutes into each task

¾ Key

Affective Parameters:

Every seven minutes thereafter

¾Assessed:

Task Difficulty

Perceived Ability

Affective States

Anxiety Index

Anxiety

Overload

Engagement Index

Task Importance

Hope

Challenge

Interest

(R) Resignation

(R) Apathy

Robotics and Autonomous Systems Lab (RASL)

Correlations of Anxiety with Physiology Vanderbilt University

0.5 0.3 0.1 Subject 2

-0.1

Subject 4

-0.3 -0.5

Symp

Paras

SCRRate

SCRAmp

CorrVar

Physiological Parameter

Mass

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Correlations with Physiology: Anxiety vs. Engagement (Subject 2) 0.7 0.5 0.3 0.1 Anxiety

-0.1

Engagement

-0.3 -0.5

Symp

Paras

SCRRate

SCRAmp

CorrVar

Physiological Parameter

Mass

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Schematic of Fuzzy Logic Analyzer Sympathetic Power

Parasympathetic Power

Skin Conductance Response Rate

Skin Conductance Response Amplitude

Corrugator Variability Masseter Average

Fuzzy Logic Analysis

Affect Diagnostic Output

Vanderbilt University

Robotics and Autonomous Systems Lab (RASL)

Development of a Functional Human-Robot Interactive System

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Our Robot Operates in three modes: 1. Explore -- Wanders about environment 2.

Survival -- Uses sonar to detect and avoid obstacles in environment

3.

Affective -- Responds to affective signals from human operator ™

If robot senses high anxiety, it uses light detection algorithm to go to operator

Robotics and Autonomous Systems Lab (RASL)

Robot Control Architecture Vanderbilt University

A hybrid subsumption control paradigm Survival Mode

Emergency Stop Reverse Affect Layer Sensor Data

Obstacle avoidance Wall Follow Wandering

S

S

Deliberative Mode

S

Reactive Mode

S

S

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Human-Robot Interaction Scenario

Human operator works at computer task while the robot explores the environment and monitors operator’s affective state

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

The Functioning System

QuickTime™ and a DV/DVCPRO - NTSC decompressor are needed to see this picture.

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Physiological Signals & Fuzzy Logic Output for Moderate Anxiety Trigger

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Timing Diagram from Robot Experiment

Vanderbilt University

Robotics and Autonomous Systems Lab (RASL)

Investigation of Brain Wave Monitoring Technologies

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Technologies to Assess Brain Activity 他

EEG - Measures electric currents generated by the brain at the scalp

他

MEG - Measures magnetic fields generated by the brain

他

PET - Measures emissions from radioactively labeled chemicals, monitors metabolic rates and blood flow

他

fMRI - Measures oxygen concentration of blood, correlates to blood flow

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Modality Comparison EEG

MEG

fMRI

PET

Spatial Resolution

7-16 mm

3-11 mm

1 mm

5 mm

Temporal Resolution

Milliseconds

Milliseconds

1 second to minutes

45 seconds to minutes

Millions

Millions

No

No

Cost

$15-120K

Wearability

Yes

$500k (37 channel) $300-400k Shielding No

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Example EEG and MEG Systems

http://www.xltek.com/xlwebv2/products/eeg/xlamb.htm http://www.vsmmedtech.com/MEG_Technical/MEG_Images_275_151.asp http://www.biomag.helsinki.fi/meg.html http://www.elixa.com/mental/MindSet.htm http://www.emedicine.com/neuro/topic445.htm

Robotics and Autonomous Systems Lab (RASL)

EEG Experiment: Cognitive Load 4000

Alpha Power

Vanderbilt University

4500 3500 3000 2500 2000 1500 1000 500 0

Eyes Closed

Eyes Open

Robotics and Autonomous Systems Lab (RASL)

EEG Experiment: Math Problem-Solving 800

Alpha Power

Vanderbilt University

900

700 600 500 400 300 200 100 0

Eyes Open

Easy

Moderate Problem Difficulty

Difficult

Robotics and Autonomous Systems Lab (RASL)

Vanderbilt University

Summary 9

performed every proposed task for Phase I

9

performed two additional tasks beyond what was proposed

9

were successful in eliciting and detecting the target affective states under controlled situations

9

developed a functional human-robot system to demonstrate the feasibility of the central concept

Robotics and Autonomous Systems Lab (RASL)

Conclusions Vanderbilt University

Phase I work demonstrates that: ¾

implicit human-robot communication is feasible can detect human affect on-line and in the context of realistic tasks control system can be made affect-sensitive

¾

brainwave monitoring will likely supplement peripheral physiology in affect detection and implicit communication

Robotics and Autonomous Systems Lab (RASL)

Future Work Vanderbilt University

¾ ¾

Expand range of tasks and contexts to which framework can be applied Increase the reliability and sophistication of affect recognition

¾ ¾ ¾

Increase range of affects detected and discriminated beyond anxiety and engagement to include frustration, fatigue, boredom, etc.

Advance analytical tools for extracting relevant information from physiological signals

Increase degree to which physiological recording is ambulatory Further explore utility of EEG recording to improve affect recognition and implicit communication Formalization of affect-sensitive control system