REHABILITATION WITH BRAIN-COMPUTER INTERFACE (BCI) SYSTEMS.
3LBS NEURO COMMUNICATION SYSTEM
Professor David Edwin Meyers IXDS6813 Final Project, Spring-Full 2016 Assistant Professor Chris Quinn Master of Arts Degree in Interactive Design Lindsey Wilson College, Columbia, Kentucky
A final project thesis submitted to the faculty of the Department of Interactive Design in partial fulfillment of the requirements for the degree of Master of Arts – Interactive Design Lindsey Wilson College. By Atif Nagi Lindsey Wilson College, Columbia, KY May 2016
TABLE OF CONTENTS 1.0
INTRODUCTION
4.0
EXECUTIVE SUMMARY 3 1.2 CONCEPT BRIEF 4 1.3 SOLUTION 5 1.4 HYPOTHESIS 6 1.5 PROJECT OVERVIEW 7 1.1
2.0
BACKGROUND
MEDICAL RESEARCH 2.2 TECHNOLOGY RESEARCH 2.1
3.0
REQUIREMENT SPECIFICATION 23 4.2 PROJECT TIMELINE 24 4.3 MOOD BOARD 25 4.4 SKETCHES 26 4.1
5.0
SOLUTION DESIGN
DESIGN PRINCIPLES 40 5.2 USER FLOW 42 5.3 WIREFRAMES 43 5.4 INTERACTIVE PROTOTYPE 53 5.1
9 11
RESEARCH
DEMOGRAPHIC & BEHAVIOURS 3.2 COMPETITIVE ANALYSIS 3.3 SUBJECT MATTER EXPERT INTERVIEWS 3.1
EXPLORATION
17 19 20
6.0
CONCLUSION
USER CENTERED DESIGN PROCESS 64 6.2 RECOMMENDATION 65 RESOURCES 68 6.1
22
1.1
EXECUTIVE SUMMARY
Brain-computer interface research and development generates tremendous excitement in scientists, engineers, clinicians, and the general public. This enthusiasm reflects the rich promise of BCIs. They may eventually be used routinely to replace or restore useful function for people severely disabled by neuromuscular disorders; they might also improve rehabilitation for people with strokes, head trauma, and other disorders. I am proposing to leverage brain-computer interfaces as an alternative to verbal or physical communication that would help otherwise incapacitated patients communicate basic responses with medical personnel.
33
1.2
CONCEPT BRIEF
We have a reason to believe through initial research that there is not a single real-world mass production product available for incapacitated patients to communicate with medical personnel. It is a niche market with the potential to grow the disabled healthcare industry. I see an opportunity to give back to the community and solve a real-world healthcare problem. It will be a ray of hope for the patients who are trapped inside their own bodies.
4
1.3
SOLUTION
There are a variety of medical situations in which patients become incapacitated and lose the ability to communicate even while maintaining consciousness. In most cases, it is very critical for healthcare professionals to be able to elicit responses from these patients. Physicians often need simple “yes” and “no” replies to questions related to pain, abilities, senses, etc. Situations such as paralysis, stroke, shock and certain types of comas can completely disable a patient’s ability to communicate with first responders, medical personnel and physicians. A headset using noninvasive brain-computer interfaces that leverage brainwave mapping and related technologies, can be used to provide simple mental feedback from patients who could not otherwise verbally or physically respond.
5
1.4
HYPOTHESIS
Leveraging a brain-computer interface (BCI) system that acquires brain signals, analyzes them, and translates them into commands that are relayed to an output device to carry out elicit responses from the people severely disabled by a neuromuscular disorder.
6
1.5
PROJECT OVERVIEW
My goal was to investigate and research the hypothesis and then propose a solution by applying user-centered design process, e.g. initial research, competitive analysis, stakeholder’s interviews, persona development, technology investigation, concept sketches, solution design, user flow, wireframes, prototype, usability tests and followup stakeholder interviews. Based on research findings, mentioned in the document later, I am proposing 3LBS, a neuro communication system, it’s features and a digital prototype in the first phase and possible product marketing material, e.g., a website, social media presence, that would increase the inquiries about the product and lead to fundraising for the development of an actual product in 2nd phase.
7
2
BACKGROUND
8
BACKGROUND RESEARCH
BACKGROUND: MEDICAL RESEARCH 2.1
The following background research will help to understand the medical terminologies and history of the persona, which is explained later in this document 2.1.1 Locked-in syndrome Locked-in syndrome (LIS) is a condition in which a patient is aware but cannot move or communicate verbally due to complete paralysis of nearly all voluntary muscles in the body except for the eyes. Total locked-in syndrome is a version of locked-in syndrome wherein the eyes are paralyzed as well. Fred Plum and Jerome Posner coined the term for this disorder in 1966. Locked-in syndrome is also known as cerebromedullospinal disconnection, de-efferented state,
pseudocoma, and ventral pontine syndrome.
The GCS is a reliable and objective way of
(1)
recording the initial and subsequent level of consciousness in a person after a brain
2.1.2 Glasgow Coma Scale
injury. It is used by trained staff at the site of an injury like a car crash or sports injury, for
The Glasgow Coma Scale (GCS) is a
example, and in the emergency department
neurological scale that aims to give a
and intensive care units.
reliable and objective way of recording the conscious state of a person for initial as
The GCS measures the following functions:
well as subsequent assessment. A patient is assessed against the criteria of the scale,
Eye Opening (E)
and the resulting points give a patient score
4 = spontaneous
between 3 (indicating deep unconsciousness)
3 = to voice
and either 14 (original scale) or 15 (the more
2 = to pain
widely used modified or revised scale).
1 = none
GCS was initially used to assess level of
Verbal Response (V)
consciousness after a head injury, and the
5 = normal conversation
scale is now used by first aid, EMS, nurses
4 = disoriented conversation
and doctors as being applicable to all acute
3 = words, but not coherent
medical and trauma patients. In hospitals it
2 = no words, only sounds
is also used in monitoring chronic patients in
1 = none
intensive care. (2)
Motor Response (M) 6 = normal 9
BACKGROUND RESEARCH
5 = localized to pain
damage. Moderate and severe brain injuries
of the body, such as the face or the
4 = withdraws to pain
often result in long-term impairments in
hand. Generalized paralysis is a group of
3 = decorticate posture
cognition (thinking skills), physical skills, and/
conditions that affect multiple body parts.
2 = decerebrate
or emotional/behavioral functioning.
Paralysis may be temporary or permanent.
1 = none
(3)
Bell’s palsy commonly causes temporary paralysis of the face. Strokes may
2.1.3 Paralysis
temporarily paralyze one side of the body.
and the best motor response an individual
Paralysis is a loss of muscle function in part
What causes paralysis?
makes. The final GCS score or grade is the
of the body. It can be partial or complete,
sum of these numbers. Every brain injury
temporary or permanent, and localized or
Some patients are born paralyzed. Others
is different, but generally, brain injury is
generalized. Most patients feel no pain in the
develop paralysis after an accident or
classified as:
paralyzed areas of their bodies. They also
medical condition. Common causes of
have no control over how those muscles
paralysis include:
Clinicians use this scale to rate the best eye opening response, the best verbal response,
Severe: GCS 3-8 (You cannot score lower
move. Paralysis can affect any part of the
than a 3.)
body at any time in a person’s life.
Moderate: GCS 9-12 Mild: GCS 13-15
• Stroke • Spinal cord injury
Types of paralysis
• Multiple sclerosis (MS) • Cerebral palsy
Mild brain injuries can result in temporary
There are many paralysis classifications.
• Post-polio syndrome
or permanent neurological symptoms and
They describe where paralysis is located,
• Traumatic brain injury
a neuro-imaging tests such as CT scan or
how severe it is, and how long it will last.
• Neurofibromatosis
MRI may or may not show evidence of any
Localized paralysis affects only one part
• Birth defects 10
BACKGROUND RESEARCH
difficult to identify, doctors may use X-rays, A study published in 2004 showed that
CT scans, MRI scans, and other procedures.
29 percent of paralysis patients become paralyzed from a stroke. The study also
Treating paralysis
showed that 23 percent of patients become paralyzed from spinal cord injuries
Treatments for paralysis depend on the
(Christopher and Dana Reeve Foundation).
underlying cause. Examples of treatments include:
What are the symptoms of paralysis?
• Mobility aids, such as wheelchairs, braces, mobile scooters, and other devices
The symptoms of paralysis are usually
• Surgery and amputation
easy to identify. A patient will lose feeling
• Physical therapy
in a specific or widespread area. Paralysis
• Medications, such as Botox or muscle
will make a muscle difficult or impossible
relaxers for patients with spastic paralysis
to control. Sometimes tingling or numbing
(4)
sensations occur before total paralysis of a limb or muscle. Diagnosing paralysis Diagnosing paralysis is often easy because the loss of muscle function is obvious. For internal body parts where paralysis is more
BACKGROUND: TECHNOLOGY RESEARCH 2.2
The following background research will help to understand the scope of current technology available and choices I made to find the best possible solution to design 3LBS Neuro Communication System. 2.2.1 Biofeedback Biofeedback is the process of gaining greater awareness of many physiological functions primarily using instruments that provide information on the activity of those same systems, with a goal of being able to manipulate them at will. Some of the processes that can be controlled include 11
BACKGROUND RESEARCH
brainwaves, muscle tone, skin conductance,
in conjunction with changes in thinking,
human brain activity by means of EEG.
heart rate and pain perception.
emotions, and behavior — supports desired
Berger was able to identify oscillatory
physiological changes. Over time, these
activity, such as Berger’s wave or the alpha
The definition of biofeedback was defined
changes can endure without continued use
wave (8–13 Hz), by analyzing EEG traces. (6)
through consensus by three professional
of an instrument” (5)
organizations, the Association for Applied Psychophysiology and Biofeedback (AAPB),
Betts Peters, M.A., CCC-SLP and Melanie 2.2.2 Brain–computer interface
Biofeedback Certification International
Fried-Oken, Ph.D., CCC-SLP answered most frequently asked questions (FAQ)
Alliance (BCIA), and the International
A brain–computer interface (BCI), sometimes
about BCI in the ALS association article.
Society for Neurofeedback and Research
called a mind-machine interface (MMI), direct
I am referring few points from this paper’s
(ISNR), arrived at a consensus definition of
neural interface (DNI), or brain–machine
perspective, helping to understand, current
biofeedback in 2008:
interface (BMI), is a direct communication
BCI technology strengths and limitations,
pathway between an enhanced or wired
and the choices I made to design
Biofeedback “is a process that enables
brain and an external device. BCIs are often
information architecture and user interface
an individual to learn how to change
directed at researching, mapping, assisting,
of 3LBS Neuro Communication system.
physiological activity for the purposes
augmenting, or repairing human cognitive or
Here are few of the points:
of improving health and performance.
sensory-motor functions. What is BCI used for?
Precise instruments measure physiological activity such as brainwaves, heart function,
The history of brain–computer interfaces
breathing, muscle activity, and skin
(BCIs) starts with Hans Berger’s discovery of
BCIs can be used for communication,
temperature. These instruments rapidly and
the electrical activity of the human brain and
computer access, or control of devices
accurately ‘feedback’ information to the user.
the development of electroencephalography
such as a wheelchair or prosthetic arm,
The presentation of this information — often
(EEG). In 1924 Berger was the first to record
among other applications. Virtually anything 12
BACKGROUND RESEARCH
that can be controlled by a computer could,
the size of an external hard drive). The brain
A and count each time it flashed, or think
potentially, be controlled by a BCI. BCI is
signals that are picked up by the electrodes
“Yes!� when you saw it appear on the
being examined as a rehabilitation device to
are sent to the computer, which uses
screen. Recognizing the A would trigger a
help people re-gain motor skills that are lost
sophisticated software to translate the brain
spike in your brain signals, which would be
from stroke, as well as a prosthetic device to
signals into computer commands.
detected by the BCI system. Usually, each
replace or compensate for motor skills that will never return.
letter must be selected multiple times, so How does BCI work?
typing with a BCI is quite slow. Systems designed to control a computer cursor
What does BCI look like?
Many people imagine that BCI will allow
often rely on movement imagery. You would
them to simply think of a word or phrase
imagine squeezing your right hand to move
There are basically two types of BCI systems:
and have it appear on the screen, or control
the cursor to the right, and your left hand to
invasive and noninvasive. Invasive systems
a wheelchair by thinking about where they
move the cursor to the left.
require surgery to implant electrodes on or
want to go. Unfortunately, this is not the
near the surface of the brain. Noninvasive
case with current BCI technology. There
Can BCI technology read or control
systems cause little or no discomfort, since
are a variety of types of BCI systems, and
minds?
most use electrodes placed on the scalp,
each one works a little differently. Most BCI
usually held in place in a cap that looks
spelling systems display a series of letters,
All BCI systems require conscious effort
like a fabric swimming cap. require the use
either one at a time or by highlighting letters
from the user to type a message or move
of conductive gel, which must be wiped
in a grid. When the letter you want lights up,
a computer cursor. BCI technology will not
or washed out of the hair after use. The
your brain wave changes. The computer
read minds or share private thoughts with
electrodes, whether invasive or noninvasive,
looks for that change and interprets it as a
others. In addition, BCI cannot be used for
are connected to a computer (usually through
keystroke. For example, if you wanted to
mind control. The electrodes are simply
an additional hardware component about
type the letter A, you would focus on the
sensors that detect brain activity, and 13
BACKGROUND RESEARCH
2.2.3 EEG (Electroencephalogram)
cannot send thoughts or commands into the
BCI technology is constantly changing. The
brain. The user controls the computer, not
future will bring faster, more reliable BCI
the other way around.
systems that work for more potential users.
As mentioned in previous section that the
Hardware will improve with wireless electrode
history of brain–computer interfaces (BCIs)
connections, and with dry electrodes that
starts with Hans Berger’s discovery of the
eliminate the need for conductive gel to be
electrical activity of the human brain and
Although most people are capable of using
applied to the user’s head. Software will
the development of electroencephalography
BCI technology, it is not for everyone.
provide new options for typing and speaking
(EEG) in 1924. The following section will
Current BCI systems are challenging to use,
messages; using the internet, email, and
discuss electroencephalography (EEG)
and require expensive equipment and time-
social media; creating artwork; safely driving
in depth and affordable commercial EEG
consuming setup. People who already have a
a wheelchair; controlling door locks, light
headsets available in the market.
reliable method of controlling a computer or
switches, entertainment systems, and other
communication device (e.g., using hand, foot,
features of the home; and much more. It
An electroencephalogram (EEG) is a test
head, or eye movement) are likely to find
is hoped that the cost of BCI will decline
used to evaluate the electrical activity in
that BCI is slower and more complicated,
as it becomes more widely used, and that
the brain. Brain cells communicate with
and simply not worth the hassle. BCIs will
insurance carriers will eventually cover
each other through electrical impulses. An
be most beneficial for people who have little
it as an assistive technology. Since this
EEG can be used to help detect potential
or no reliable muscle movement, including
technology is being developed for a much
problems associated with this activity.
some people with advanced ALS.
larger market than those with neurological
Who can use BCI?
What does the future hold for BCI technology?
disease, in the future we will see BCI devices
The test tracks and records brain wave
available for many computer functions. (7)
patterns. Small, flat metal discs called electrodes are attached to the scalp with wires. The electrodes analyze the electrical 14
BACKGROUND RESEARCH
impulses in the brain and send signals to a
• Sleep disorders
these games would not offer much in terms
computer, where the results are recorded.
• Strokes
of game-play. What, after all, can you do
The electrical impulses in an EEG recording
• Dementia
with just one button. And yet, in spite of
look like wavy lines with peaks and valleys.
this severe limitation, one-button games
These lines allow doctors to quickly assess
There are no risks associated with an EEG.
can have surprisingly complex game-play.
whether there are abnormal patterns. Any
The test is painless and safe. When someone
Not only that, some one-button games, e.g.
irregularities may be a sign of seizures or
has epilepsy or another seizure disorder, the
Poto and Cabenga, can offer a very stiff
other brain disorders.
stimuli presented during the test (such as a
challenge to gamers.
flashing light) may cause a seizure. However, An EEG is used to detect problems in the
the technician performing the EEG is trained
This ability to unleash complex and
electrical activity of the brain that may be
to safely manage the situation should this
challenging game-play using the simplest of
associated with certain brain disorders. The
occur. (8)
controls is what makes one-button games
measurements given by an EEG are used to confirm or rule out various conditions,
so magical and fun to play. The top five 2.2.4 One–button game interface
including:
online games are G-Switch, Arctic Blue, One Button Bob, Poto and Cabenga, Flabby
One button games have incredibly simple
Physics. (10)
• Seizure disorders (such as epilepsy)
gameplay, like a puzzle game that can be
• A head injuries
played entirely using a one-button. It is
One-button is the limitation on
• Encephalitis (an inflammation of the brain)
amazing how much fun you can have with
interaction. It involves low-level
• A brain tumors
games that use just one button.
mechanics of basic actions. I explored
• Encephalopathy (a disease that causes
There is something almost magical about
this limitation of interaction in the basic
brain dysfunction)
one-button games. You would have thought,
architecture of design solution.
• Memory problems
given the simplicity of their controls, that 15
3
RESEARCH
16
RESEARCH
3.1
DEMOGRAPHIC & BEHAVIOURS
After initial research of incapacitated patients’ case studies, medical conditions, available technology’s strength, and limitation I started to focus on demographics and behaviors, persona development, competitive analysis, and subject matter expert interviews 3.1.1 Primary target audience
to be able to elicit responses from these
3.1.3 Stakeholders
patients. Physicians often need simple “yes” and “no” replies to questions related to pain,
Stakeholders are healthcare personnel,
abilities, senses, etc. These patients can be
e.g., physicians, nurse, EMS, and caregiver
trained to use the headset and communicate
staff, who are going use 3LBS Neuro
with healthcare personals.
Communication System along with the patients.
3.1.2 Secondary target audience The secondary primary target audiences are male and female short-term incapacitated patients with the following background: Short-term incapacitated patients have lost the ability to communicate somewhat due
The primary target audiences for this product
to a shock of a medical condition. EMS
are male and female long-term incapacitated
personnel often need simple “yes” and “no”
patients with the following background:
replies to questions related to pain, abilities, senses, etc. Due to emergency situations,
Long-term incapacitated patients, who
these patients cannot be trained to use the
lose the ability to communicate even while
headset and communicate with healthcare
maintaining consciousness. In most cases,
personals.
it is very critical for healthcare professionals 17
RESEARCH
3.2 Primary persona Name: Anton Bruce Age: 39 Occupation: Database developer Neurological disorder diagnosis: Total locked-in syndrome Location: Cleveland, OH
Technology IT and Internet Software Mobile Apps Social Networks
Anton is a 39 years old Database Administrator (DBA) of a multinational retail chain in Cleveland. Recently he has been diagnosis with Total Locked-in Syndrome by the panel of neurologists from a leading Neurological Institute in Cleveland, which offers the full range of neurological and neurosurgical services. Anton is a victim of a severe stroke. Recently he came out of a four-week deep coma, unaware of what happened to him, what was going on around him, and where he was. He is in a state of total paralysis, no control over his body, eyes wide open, incapable of any verbal communication, in what is known in the medical community as “Total locked-in syndrome.” The syndrome is typically caused by a lesion in the pons, effectively the part of the brainstem that acts as a bridge between brain and body. The doctors have confirmed that Anton’s cognitive, auditory and visual capabilities are intact with his brain by applying auditory, and visual evoke potential (VEP), electroencephalography (EEG), electromyography (EMG), and clinical tests. It’s like living his worst nightmare. Anton only has his thoughts for company and nothing to punctuate his day, which feels endless. There is the indignity
Motivations • To communicate with the healthcare professional and family. • Fight back and recover from his disability. Frustrations Anton’s brain is aware, alert and functional. He can listen to physicians, family and friends but he cannot tell anyone this. He is locked-in his body, struggling and willing to recover but doesn’t know how to communicate.
of being unable to do anything. He can feel pain and touch, but he cannot communicate or express himself. It is like paralysis with feeling. It is very critical for his recovery and survival to somehow communicate with physicians, even simple “yes” and “no” replies to questions related to pain, abilities, senses, etc. 18
RESEARCH
3.3
COMPETITIVE ANALYSIS
Qualitative research “interviews” and online searches were used to uncover strengths and weaknesses as well opportunities and
3.3.2 Tier 2 competitors
3.3.4 SWOT
There are multiple tier 2 researchers.
A quick SWAT analysis of the 3LBS Neuro
Researchers are using brain-computer
Communication System.
interface technology to help patients, who are unable to perform basic activities such
Strengths
as walking, talking, typing or writing due to
Simple and double product development.
severe brain disorders caused by paralysis,
threats.
stroke, shock and certain types of coma, e.g.
Weaknesses
robotics, wheelchair, communication, and
Patient’s learning curve.
3.3.1 Tier 1 competitor
hearing aid devices. Opportunities
After online search and interviews with
3.3.3 Key dimensions
Arshad Ali, MD, a neurosurgeon, It was
It’s a niche market and an opportunity to enter and grow quickly.
determined that there is no direct rival BCI
Competitors were analyzed based on based
product of 3LBS Neuro Communication
on the following dimensions, which will be
Threats:
System. There is some evidence of early
critical factors to the start-up of the program:
1. To find the funding to research and
research and technology projects, but there
develop the product.
is not a single commercial product available
• Sector
2. To enter the healthcare market.
in the healthcare industry to communicate
• Technology
3. Government compliance.
with total locked-in syndrome patients.
• Status (Research vs. Real-world product) • Availability • Cost 19
RESEARCH
SUBJECT MATTER EXPERT INTERVIEWS
Dr. Ali is a Neurosurgeon and expert in
and AML diseases.
Neurovascular diseases. Currently, he is
• Non-clinical test:
a consultant and Head of Neurosurgery,
- Speech (Sensation, receive, process,
Sibu General Hospital (Ministry of Health-
execution)
Malaysia), Sibu Sarawak, East Malaysia. He
- Hearing (Auditory evoked potential)
3.4.1 Dr. Arshad Ali, Neurosurgeon and Neuro-interventionist, Head of Department, The Ministry of Health Malaysia.
is gazetted as Specialist Neurosurgeon with
- Vision (Visual evoked potential)
core credentials to operate as Neurospinal
- Cognitive tests
Neurosurgeon including all in-patient
• Clinical tests: EEG/MRI
and out-patient privileges. As per latest
• Glasgow Coma (A neurological scale)
hospital statistics about Neurosurgery,
• Brain conscious levels
3.4
total admissions are 1209, outpatient 985, emergency surgeries 302 & Elective 91. I spoke to Dr. Ali multiple times. He helped me understand the basic concepts of the
The above items are described in background research section in detail.
brain, common diseases, and diagnosis.
3.4.2 Duane Cash, Inventor / Innovator of Mind-Controlled Tech.
• Brain structure
Duane Cash is a developer, innovator and
• Vegetative state (Unresponsive wakefulness
inventor. He obtained a Master of Fine
syndrome)
Arts Degree in Interactive Design and
• Locked-in syndrome
Game Development from the Savannah
• Total locked-in syndrome
College of Art and Design in 2013. His
• Pathology of common neurovascular
thesis was on “Mind-controlled Systems for
diseases: Globalpuasiya, Myopathies, Troma, 20
RESEARCH
economical available headset Neurosky
way.
Mindwave and experimenting with the rest of devices at a later stage. Here is the summary
I hope some of the items I mentioned help
of our communication:
you in your decision. In my experience, I started with the MindWave Mobile, then
“Although I am recommending the Neurosky
bought the others later, and eventually
headset for this project, I do also recommend
returned to the MindWave Mobile because it
that if your budget allows later, you should
provided what I needed. You might eventually
also eventually pick up the Emotiv Insight,
find that the Emotiv Insight is a good option.
the Melon headset, and Interaxon’s Muse
If you are wishing to show a demo with
headset. These headsets will also give you
more impact, the Emotiv Insight has an
better insight into how brainwave patterns
elegant look that will win the attention of
propagate and might help you better
the audience. This is the headset that I do
understand the interaction between thoughts,
not yet have; I have the EPOC, but still the
Deeper Cognitive Connections with Virtual
speech and actions. Every thought can be
Neurosky is doing the job fine. The key is
Assistants.” He is an interactive designer
mapped to a verbal equivalent. Some EEG
really not in the headset chosen, but in the
who enjoys creating innovative applications
experts will try to dismiss the idea that one
software and how it is written. After all, your
for technology-driven companies that make a
can find anything in a single RAW EEG and
project is looking for a “yes” and “no” and
difference in the lives of others.
they will even say that having the Delta,
not for multiple commands or phrases like
Alpha, Beta, Gamma, and other patterns is
the ThynkWare Speech. The Neurosky can
I discussed the scope of EEG headset
still not sufficient to translate into a verbal
do it.”
products to proof the concept for the final
intent. They just need to look at the patterns
project. Duane advised starting with most
deeper and think about them in a different
Duane Cash, March 1, 2016. 21
4
EXPLORATION
22
EXPLORATION
4.1
REQUIREMENT SPECIFICATION
4.1.1 Requirement specification I planned to design a software application, 3LBS Neuro Communication System, utilizing a brainwave mapping headset and onebutton game philosophy. Phase 1 - Final Project, Spring-Full 2016
• Dashboard/Home
Phase 3 (Expansion)
• Emergency • Talk to caregiver (Yes, no, and not sure)
Once the original functionality of the software
• Room comfort (Lights, thermostat, bed,
is nailed down for the primary persona,
and TV)
start work on other personas to add more functionality, e.g. power user, patients with
Phase 2 - Product prototype and
some motor control to expand the market.
marketing material
Here are a few ideas for future expansion:
Once the concept and digital prototype
• Emergency: Pinpoint pain area e.g.
are finished, start working on building the
highlight different body parts.
physical prototype in the second phase after
• Talk to caregiver: Highlight a help topic e.g.
graduation.
pain, food, bath, love, etc. • Room comfort (Add more functionality
I applied user centered design process in Phase 1, Final Project, Spring-Full 2016 i.e. initial research, information architecture, conceptual sketches, user flow, medium fidelity wireframes, usability research, high fidelity wireframes, and digital prototype. • Technology introduction • Configuration • Practice
• Develop prototype software and hardware.
e.g. room appliances, books, TV channels,
• Primary and secondary persona usability
movies, music, etc.)
research.
• Instant messaging (IM): Talk to a friend or
• Improve UI and functionality.
family.
• Fundraising for product development.
• Introduce more than one cognitive
• Traction channels: Bulls’ eye framework
command for faster navigation.
• Social media presence e.g. Facebook,
• Introduce muscle controls commands for
Pinterest, Linkedin
patients with some motor control e.g. blink
• Open conferences and seminars
and eyebrow movement.
• Apply for patent and explore legal aspects. 23
EXPLORATION
4.2
PROJECT TIMELINE
Flow and architecture | W6 Site maps | W6 User journey | W6 Sketches | W7 Written draft 2 | W7
Thesis statement | W2 Abstract | W2 Written outline | W3 Task timeline | W3 Visual timeline | W3
WEEK 2
WEEK 3
Recommendation | W14 Written paper 2 | W14
DESIGN EXPLORATION
PLAN WEEK 1
PROJECT TIMELINE
WEEK 4
WEEK 5
WEEK 6
RESEARCH Initial research | W4 Competitive analysis | W4 Persona | W4 Stakeholder interviews | W5 Mood board | W5 Recommendation | W5 Written draft 1 | W5
WEEK 7
WEEK 8
WEEK 9
CONCLUSION WEEK 10
WEEK 11
WEEK 12
DESIGN SOLUTION Wireframes | W8 Prototype | W9 - W12 Usability testing | W13 Written paper 1 | W13
WEEK 13
WEEK 14
WEEK 15
WEEK 16
DOCUMENT Design document | W15 Presentation | W16
24
EXPLORATION
4.3
MOOD BOARD
VIEW ONLINE
25
EXPLORATION
4.4
SKETCHES
left, right, forward and backward) plus one
important that each user masters the use of
additional action that exists only in the realm
the Mental Commands detection one action
of the user’s imagination: disappear.
at a time, only increasing the number of concurrent actions after he has first gained
4.4.1 Initial concepts Mental Commands allows the user to choose
confidence and accuracy with a lower
After researching the medical history
up to 4 actions that can be recognized at any
number of actions. (13)
of persona, available technology, user
given time. The detection reports a single
interfaces, BCI, EEG headsets and output
action or neutral (i.e. no action) at a time,
devices. I sketched initial concepts of BCI,
along with an action power which represents
how raw EEG signals will go to processing
the detection’s certainty that the user
application and output to devices for
has entered the Mental Commands state
meaningful results.
associated with that action.
The training application suite is based Emotiv Increasing the number of concurrent actions Xavieron SDK. The Mental Commands
increases the difficulty in maintaining
detection suite evaluates a user’s real time
conscious control over the Mental Command
brainwave activity to discern the user’s
detection results. Almost all new users
conscious intent to perform distinct physical
readily gain control over a single action quite
actions on a real or virtual object. The
quickly. Learning to control multiple actions
detection is designed to work with up to 13
typically requires practice and becomes
different actions: 6 directional movements
progressively harder as additional actions
(push, pull, left, right, up and down) and 6
are added. Although Emotiv Xavier allows a
rotations (clockwise, counter-clockwise,
user to select up to 4 actions at a time, it is 26
EXPLORATION
4.4
SKETCHES
27
EXPLORATION
4.4
SKETCHES
28
EXPLORATION
4.4
SKETCHES
29
EXPLORATION
4.4
SKETCHES
4.4.2 Second concept iteration During this phase, I finally named the product 3LBS Neuro Communication System. I started exploring training suite. Introduce multiple users i.e. novice, skilled and power user. Novice users have to learn only one cognitive action, two-three for skilled, and 13 for power users.
30
EXPLORATION
4.4
SKETCHES
31
EXPLORATION
4.4
SKETCHES
32
EXPLORATION
4.4
SKETCHES
33
EXPLORATION
4.4
SKETCHES
34
EXPLORATION
4.4
SKETCHES
4.4.3 Third concept iteration I applied 20-80 rule to cut down the unnecessary options and focused on persona needs. The 3rd iteration is based on one-button interface. One button is the limitation on interaction. It involves lowlevel mechanics of basic actions. I explored this limitation of interaction in the basic architecture of design solution. Almost all new users readily gain control over a single action quite quickly. Users can find important information through only one cognitive command, intuitive navigation affordance, which gives consistency and predictability to the users. *15 min learning curve as per Classification of EEG Signals in a Brain- Computer Interface System thesis research by Erik Andreas Larsen, Master of Science in Computer Science, Norwegian University of Science and Technology Department of Computer and Information Science.
35
EXPLORATION
4.4
SKETCHES
36
EXPLORATION
4.4
SKETCHES
37
EXPLORATION
4.4
SKETCHES
38
5
SOLUTION DESIGN
39
SOLUTION DESIGN
5.1
DESIGN PRINCIPLES
I applied multiple design principles during solution design. Here are some of the leading principles:
CLEAR
AFFORDANCE
CONSISTENT
The information of the system is straightforward & understandable with clear one-button navigation, simple language (labels), and icons.
I used explicit affordance signaled action buttons by language, negative affordance in the menu and contextual menu, metaphor affordance by using real-world menu icons as a metaphor.
The consistent use of one-button navigation, labels, language, colors, icons and user interface patterns.
FOCUS
ACCESSIBILITY
WAYFINDING
The user interface is singular and focused. It is intuitive because it focuses on core tasks of persona e.g. emergency, talk to a caregiver, and room comfort.
Appliedd W3C standards, e.g., high contrast, B/W design, Font size readability and visual hierarchy.
Users can find their way by landmark icons, well-structured paths, and fewer choices. They know where they are, what they can do, where they are going and where they’ve been through navigation.
40 40
SOLUTION DESIGN
5.1
DESIGN PRINCIPLES
Visibility of system status
10 Usability heuristics for user interface Aesthetic & minimalist design
design I applied Jakob Nielsen’s 10 heuristic
Match real world
principles for interaction design during design and evaluation of user interface.
Help and documentation
Recover from errors
HEURISTICS EVALUATION
Flexibility and efficiency of use
Recognition rather than recall
User control and freedom
Consistency and standards
Error prevention
41 41
SOLUTION DESIGN
5.2
USER FLOW
VIEW ONLINE
The user flow is based on Nielsen’s heuristics e.g. user control and freedom, consistency and standards, Error prevention, Flexibility and efficiency of use, Help users recognize, diagnose, and recover from errors. This user flow considers user’s happy path and also when things go wrong for the user, how to prevent and recover from errors.
42
SOLUTION DESIGN
5.3
WIREFRAMES
the duration of the training period (currently 8
by supplying consistent training data (i.e. a
seconds).
consistent Mental Commands visualization on the part of the user) across several training sessions for each enabled action.
I started converting low-fidelity sketch
Initially, the cube on the screen will not move, as
concepts to medium-fidelity wireframes
the system has not yet acquired the training data
by adding more screens and details based
necessary to construct a personalized signature
A training session is automatically discarded if
on user flow. I am showcasing high-level
for the current set of actions. After Neutral
the wireless signal strength or EEG signal quality
wireframes in this document. To view every
and each enabled action have been trained at
is poor for a significant portion of the training
interaction screen of wireframe, please click
least once, the Mental Commands detection
period. A notification will be displayed to the user
on the prototype button in the next section.
is activated, and the cube will respond to the
if this has occurred. (13)
Mental Commands detection, and your Mental The Mental Commands training process
Commands control, in real time.
Xavieron SDK cognitive suite.
enables the system to analyze brainwaves and develop a personalized signature which
*The training application suite is based Emotiv
Some users will find it easier to maintain the
corresponds to each particular action, as well necessary Mental Commands focus if the cube is automatically animated to perform the intended as the background state, or “neutral�. As the system learns and refines the signatures
action as a visualization aid during training. It is
for each of the actions, as well as neutral,
also important that the system gives feedback to
detections become more precise and easier
the user for visibility of system status.
to perform. Finally, users are prompted to accept or reject During the training process, it is very
the training recording. Ideal Mental Commands
important to maintain cognitive focus for
detection performance is typically achieved 43
SOLUTION DESIGN
5.3
WIREFRAMES
Intro + configuration mode
44
SOLUTION DESIGN
5.3
WIREFRAMES
Configuration mode
45
SOLUTION DESIGN
5.3
WIREFRAMES
Practice mode
46
SOLUTION DESIGN
5.3
WIREFRAMES
Practice mode + emergency command
47
SOLUTION DESIGN
5.3
WIREFRAMES
Help + talk command
48
SOLUTION DESIGN
5.3
WIREFRAMES
Comfort + light Command
49
SOLUTION DESIGN
5.3
WIREFRAMES
Thermostat command
50
SOLUTION DESIGN
5.3
WIREFRAMES
Bed Command
51
SOLUTION DESIGN
5.3
WIREFRAMES
TV + back to home command
52
SOLUTION DESIGN
INTERACTIVE PROTOTYPE 5.4
LAUNCH PROTOTYPE
The Interactive prototype was made in AxurePR. I added missing screens and interactions to get the interactive prototype ready for usability testing with subject matter expert evaluators. *Click on the image to view in a browser.
53
SOLUTION DESIGN VALIDATION RESEARCH
5.5
USABILITY TESTING
Dr. Arshad Ali
Dr. Fariha Jamshed
Dr. Aisha Malik
Duane Cash
Khalil Laghari
MD (Neurosurgeon and Neuro-interventionist)
MBBS, DMRD (Radiologist)
MD (Neurologist)
Innovator of MindControlled Tech.
Doctor of Philosophy (Ph.D.), HCI
100% 3LBS Neuro Communication System is a practical and doable project. I would encouraged you to take this project to next level.
89%
79%
100%
89%
I like the simplicity and learnability of the interface. 3LBS a promising device for physically challenged patients.
I like the simplicity and The interface is selfexplanatory and intuitive. There was an intial learning curve but at the end, it totally worked.
This project is absolutely doable and has practical value. Carousel navigation and one push command make a less cognitive load on a human brain.
Ease of use, simplicity, the minimalism of interface and cognitive functionality are the strongest features of 3LBS Neuro Communication System.
Usability test results
54
SOLUTION DESIGN VALIDATION RESEARCH
5.5
USABILITY TESTING
spinal cord, and the peripheral nerves and/
5. Add more details about each section
or have experience in brain-computer
to set users expectation.
interaction (BCI).
6. Fix back to home screen.
*A detailed usability procedure is described in
5.5.4 Future research
5.5.1 Usability research goals To determine if there were any significant usability issues. Research should identify usability pain points.
final project reserach paper. 5.5.3 Outcomes
Further usability research is required in future, after product prototype is built in phase 2, with primary persona (Total
5.5.2 Research methods and usability procedures The research method was remote usability
I summarized evaluators’ feedback and
locked-in syndrome long-term patient)
recommend to apply following improvements
and secondary (Situational incapacitated
to visual design phase:
patients).
interviews by hybrid task and observational
1. Create a storyboard for the demo video.
research method.
It should explain the product features through animation and personal demo video
The evaluator selection criteria were proxy
recording.
subject matter experts i.e. Physicians, neuro-
2. Add supporting animation in hi-fidelity
interventionist, psychiatrist, and BCI experts.
wireframes. 3. Add feedback progress bar in practice
Subject matter expert evaluators must have
mode.
previous experience with diseases and
4. Add restart button on restart feedback
conditions affecting the nervous system,
screen to empower the user to restart
which includes the brain, the spine, and
training instead of system dependent. 55
SOLUTION DESIGN
5.6
HI-FI WIREFRAMES
LAUNCH PROTOTYPE
I applied usability research recommendation to visual design high-fidelity wireframes. Introduced colors and animation treatment.
56
SOLUTION DESIGN
5.6
HI-FI WIREFRAMES
Intro screens
57
SOLUTION DESIGN
5.6
HI-FI WIREFRAMES
Configuration screens
58
SOLUTION DESIGN
5.6
HI-FI WIREFRAMES
Practice screens
59
SOLUTION DESIGN
5.6
HI-FI WIREFRAMES
Practice and emergency screens
60
SOLUTION DESIGN
5.6
HI-FI WIREFRAMES
Talk to caregiver and talk screens
61
SOLUTION DESIGN
5.6
HI-FI WIREFRAMES
Comfort screens
62
6
CONCLUSION
63
CONCLUSION
USER CENTERED DESIGN PROCESS 6.1
A full user centered design process was explored in the 3LBS Neuro Communication System, Brain-computer Interface project from research, ideation, exploration, define, 1. Research & ideation
design, prototype and usability validation.
2. Persona development
The three-cycle interactive process with paper and interactive prototype usability validation
refined
the
product.
After
multiple iterations, I achieved effectiveness, efficiency, and satisfaction through user flow interactions, intuitive UI, labels, and icons of the user interface, which was supported and
3.3
COMPETITIVE ANALYSIS
Qualitative research “interviews” and online searches were used to uncover strengths and weaknesses as well opportunities and threats.
validated by subject matter experts’ usability
3.3.1 Tier 1 competitor
evaluations.
After online search and interviews with Arshad Ali, MD, a neurosurgeon, It was determined that there is no direct rival BCI
3. Competitive research and technology projects, but there analysis product of 3LBS Neuro Communication
System. There is some evidence of early
is not a single commercial product available in the healthcare industry to communicate with total locked-in syndrome patients.
4. Reserach methods application 64
CONCLUSION
5. User flow
6. Concept evolution
7. Design evolution
Multiple iterations
Multiple iterations
6.2
RECOMMENDATION
Based on the research data found for the proposed hypothesis through professional insight, in
experience, competitive analysis, and industry expert interviews, I recommend building this pro
All subject matter experts including a neurosurgeon, neurologist, brain-computer interaction
experts has consensus that “3LBS Neuro Communication System is a practical and doable pro They encouraged me to take this project to next level.
Seize this project to next level and finally into the global market, I will need to build on the res
with a formal business analysis to address financial considerations to build the product prot polish final product, legal aspects, and marketing efforts.
8. Concept & design validation
9. Hi-Fi design evolution
Usability testing
Multiple iterations
10. Research recommendations
The research data exhibits that there is a market need for 3LB Neuro Communication System
healthcare industry and It will be a ray of hope for the patients who are trapped inside their own
65
CONCLUSION
6.2
RECOMMENDATION
Based on the research data found for the proposed hypothesis through professional insight, industry experience, competitive analysis, and industry expert interviews, I recommend building this product. All subject matter experts including a neurosurgeon, neurologist, brain-computer interaction (BCI) experts has consensus that “3LBS Neuro Communication System is a practical and doable project�. They encouraged me to take this project to next level. Seize this project to next level and finally into the global market, I will need to build on the research with a formal business analysis to address financial considerations to build the product prototype, polish final product, legal aspects, and marketing efforts. The research data exhibits that there is a market need for 3LB Neuro Communication System in the healthcare industry and It will be a ray of hope for the patients who are trapped inside their own body.
66
7
RESOURCES
67
RESOURCES
1. “Locked-in Syndrome.” Wikipedia. Wikimedia Foundation. Web. 07
7. “Brain-Computer Interface (BCI).” ALSA.org. Web. 14 Feb 2016.
Feb. 2016. <https://en.wikipedia.org/wiki/Locked-in_syndrome>.
<http://www.alsa.org/als-care/resources/publications-videos/ factsheets/brain-computer-interface.html>. Betts Peters, M.A., CCC-
2. “Glasgow Coma Scale.” Wikipedia. Wikimedia Foundation. Web.
SLP and Melanie Fried-Oken, Ph.D., CCC-SLP
07 Feb. 2016. <https://en.wikipedia.org/wiki/Glasgow_Coma_Scale>. 8. “EEG (Electroencephalogram).” Healthline. Web. 14 Feb. 2016. 3. “What Is the Glasgow Coma Scale?” Brainline. Web. 08 Feb. 2016.
<http://www.healthline.com/health/eeg>.
<http://www.brainline.org/content/2010/10/what-is-the-glasgowcoma-scale.html>.
9. “8 Mind-blowing Gadgets You Can Control Just With Your Brain.” Hongkiatcom. Web. 20 Feb. 2016. <http://www.hongkiat.com/blog/
4. “Paralysis.” Healthline. Web. 08 Feb. 2016. <http://www.healthline.
brain-controlled-gadgets/>.
com/health/paralysis>. 10. “Best One Button Games.” Best One Button Games. Web. 07 5. “Biofeedback.” Wikipedia. Wikimedia Foundation. Web. 14 Feb.
Mar. 2016.
2016. <https://en.wikipedia.org/wiki/Biofeedback>. 11. “10 Heuristics for User Interface Design: Article by Jakob 6. “ Brain–computer interface.” Wikipedia. Wikimedia
Nielsen.” 10 Heuristics for User Interface Design: Article by Jakob
Foundation. Web. 14 Feb. 2016. <https://en.wikipedia.org/wiki/
Nielsen. Web. 20 Mar. 2016.
Brain%E2%80%93computer_interface>.
68
RESOURCES
12. Krug, Steve. Rocket Surgery Made Easy: The Do-it-yourself Guide to Finding and Fixing Usability Problems. Print. 13. “Understanding the Mental Commands Detection Suite” Emotiv. Web. 20 Feb. 2016. <https://emotiv.zendesk.com/hc/enus/articles/201483705-Understanding-the-Mental-CommandsDetection-Suite>. 14. “Classification of EEG Signals in a Brain-Computer Interface System.” Classification of EEG Signals in a Brain-Computer Interface System. Web. 10 Apr. 2016. 15. Infofree8. “Emotiv.mp4.” YouTube. YouTube, 2010. Web. 25 Mar. 2016. <https://www.youtube.com/watch?v=r2ip_84XRjA>. 16. Gsmarena07. “Emotiv EPOC Heads-on.” YouTube. YouTube, 2012. Web. 25 Mar. 2016. <https://www.youtube.com/ watch?v=LZrat-VG4Ms>. 17. DigitalOmni. “4.5 Year Old Boy Plays with EPOC Emotive.” YouTube. YouTube, 2010. Web. 26 Mar. 2016.
69