Nicolelis Laboratory Series
20 YEARS OF BRAIN-MACHINE INTERFACE RESEARCH Volume 2
Other Books by the Author
Nicolelis MAL. Beyond Boundaries: The New Neuroscience of Connecting Brains and Machines and How It Will Change Our Lives. Times Books/Henry Holt & Co, New York, NY, 2011. Made in Macaiba: A Historia da Criacao de uma Utopia Cientifico-social no Eximperio dos Tapuias (Portuguese Brazilian). CrĂtica, 2016. Cicurel R, Nicolelis MAL. The Relativistic Brain: How it works and why it cannot be simulated by a Turing machine. Kios Press, Montreaux, Switzerland, 2015. Nicolelis MAL. The True Creator of Everything: How the Human Brain Shaped the Universe as We Know It. Yale University Press, New Haven, CT, 2019.
To John K. Chapin, for all those great years of fun and discovery.
1
Contents
Introduction…….……………………………..…………………………………....5 Time Table of Key Discoveries……………………………………………………8 Clinical Brain-Machine Interface Studies in Parkinsonian and Spinal Cord Injury Patients…………………………………………………………………….10 Patil PG, Carmena JM, Nicolelis MAL, Turner DA. Ensemble recordings of human subcortical neurons as a source of motor control signals for a brain-machine interface. J Neurosurgery 55: 27-35, 2004…………………...………………………11 Hanson TL, Fuller AM, Lebedev MA, Turner DA, Nicolelis MAL. Subcortical Neuronal Ensembles: An Analysis of Motor Task Association, Tremor, Oscillations, and Synchrony in Human Patients. J. Neurosci. 32:8620–8632, 2012……………….23 Donati ARC, Shokur S, Morya E, Campos DSF, Moioli RC, Gitti CM, Augusto PB, Tripodi S, Pires CG, Pereira GA, Brasil FL, Gallo S, Lin A, Takigami AK, Aratanha MA, Joshi S, Bleuler H, Cheng G, Rudolph A, Nicolelis MAL. Long-term training with brain-machine interfaces induces partial neurological recovery in paraplegic patients. Sci. Rep. doi: 10.1038/srep30383, 2016……………………………………………..36 Shokur S, Gallo S, Moioli RC, Donati ARC, Morya E, Bleuler H, Nicolelis MA. Assimilation of virtual legs and perception of floor texture by complete paraplegic patients receiving artificial tactile feedback. Sci. Rep. doi:10.1038/srep32293, 2016………………………………………………………………………………..52 Shokur S, Donati ARC, Campos DSF, Gitti C, Bao G, Fischer D, Almeida S, Braga VAS, Augusto P, Petty C, Alho EJL, Lebediev M, Song AW, Nicolelis MAL. Training with brain-machine interfaces, visuotactile feedback and assisted locomotion improves sensorimotor, visceral, and psychological signs in chronic paraplegic patients. PLoS ONE 13(11): e0206464. https://doi.org/10.1371/journal.pone.0206464, 2018………………………………………………………………………………..66 Selfslagh A, Shokur S, Campos D S, Donati A R, Almeida S, Bouri M, Nicolelis M A. Non-invasive, brain-controlled functional electrical stimulation for locomotion rehabilitation in indivduals with paraplegia. Sci. Rep. 9, Article number: 6782, https://doi.org/10.1038/s41598-019-43041-9, 2019……………………………….99
2
Neuroprostheses for Parkinson’s Disease and Epilepsy………………………116 Fanselow EE, Reid AP, Nicolelis MAL. Reduction of pentylenetetrazoleinduced seizure activity in awake rats by seizure-triggered trigeminal nerve stimulation. J Neurosci 20: 8160-8168, 2000…………………....……………………………...117 Fuentes R, Petersson P, Siesser WB, Caron MG, Nicolelis MAL. Spinal Cord Stimulation Restores Locomotion in Animal Models of Parkinson's disease. Science 323: 1578-82, 2009………………………………………………………………...126 Yadav AP, Fuentes R, Zhang H, Vinholo T, Wang C-H, Nicolelis MAL. Chronic Spinal Cord Stimulation Protects against 6-hydroxydopamine Lesions. Sci. Rep. 4: 3839. doi: 10.1038/srep03839, 2014…………………………...……….………………...131 Santana MB, Halje P, Simplicio H, Richter U, Freire M, Petersson P, Fuentes R, Nicolelis MAL. Spinal Cord Stimulation Alleviates Motor Symptoms in a Primate Model of Parkinson's disease. Neuron 84: 716–722, 2014…………………………141 Pais-Vieira M, Yadav AP, Moreira D, Guggenmos D, Santos A, Lebedev MA, Nicolelis MA. A Closed Loop Brain-machine Interface for Epilepsy Control Using Dorsal Column Electrical Stimulation. Sci. Rep. doi:10.1038/srep32814, 2016…….148 Creating a New Sense: The Infrared Rat Studies Papers……………………...157 O'Doherty JE, Lebedev MA, Li Z, Nicolelis MAL. Virtual Active Touch Using Randomly Patterned Intracortical Microstimulation. IEEE Trans Neur Syst Rehab Eng. 20: 85-93, 2012……………………………………………………………………158 Medina LE, Lebedev MA, O’Doherty JE, Nicolelis MAL. Stochastic Facilitation of Artificial Tactile Sensation in Primates. J. Neurosci. 32: 14271-14275, 2012……167 Thomson EE, Carra R, Nicolelis MAL. Perceiving Invisible Light through a Somatosensory Cortical Prosthesis. Nat. Commun. 10.1038/ncomms2497, 2013….172 Hartmann K, Thomson EE, Yun R, Mullen P, Canarick J, Huh A, Nicolelis MA. Embedding a novel representation of infrared light in the adult rat somatosensory cortex through a sensory neuroprosthesis. J. Neurosci. 36:2406 –2424, 2016………179 Thomson E, Zea I, Windham W, Thenaisie Y, Walker C, Pedowitz P, França W, Graneiro AL, Nicolelis MAL. Cortical Neuroprosthesis Merges Visible and Invisible
3
Light Without Impairing Native Sensory Function. eNeuro DOI: 10.1523/ENEURO.0262-17, 2017…………………………...……………………198 Brain-to-Brain Interfaces and Brainets………………....………………………215 Pais-Vieira M, Lebedev MA, Kunicki C, Wang J, Nicolelis MAL. A brain-to-brain interface for real-time sharing of sensorimotor information. Sci. Rep. 3:1319, doi:10.1038/srep01319, 2013…………………...…………………………………216 Ramakrishnan A, Ifft PJ, Pais-Vieira M, Byun YW, Zhuang KZ, Lebedev MA, Nicolelis MAL. Computing Arm Movements with a Monkey Brainet. Sci. Rep. doi:10.1038/srep10767, 2015……………………………………………………...226 Pais-Vieira M, Chiuffa G, Lebedev MA, Yadav A, Nicolelis MA. Building an organic computing device with multiple interconnected brains. Sci. Rep. doi:10.1038/srep11869, 2015……………………………………………………...241 Tseng P-H, Rajangam S, Lehew G, Lebedev MA, Nicolelis MAL. Interbrain cortical synchronization encodes multiple aspects of social interactions in monkey pairs. Sci. Rep. doi.org/10.1038/s41598-018-22679-x, 2018……………………………..255 Key Opinion and Review Articles……………………………………………….270 Nicolelis MAL, Fanselow E, Ghazanfar AA. Hebb’s dream: the resurgence of cell assemblies. Neuron 19: 219-221, 1997…………………………………………271 Nicolelis MAL. Actions from thoughts. Nature 409: 403-407, 2001………...274 Nicolelis MAL, Chapin JK. Controlling robots with the mind. Scientific American 287: 24-31, October 2002……………………………………………….279 Nicolelis MAL. Brain-machine interfaces to restore motor function and probe neural circuits. Nat Rev Neurosci 4: 417-422, 2003………………………………...288 Lebedev MA, Nicolelis MAL. Brain machine interfaces: Past, present and future. Trends Neurosci 29: 536-546, 2006…………………………...…………………...294 Nicolelis MAL, Lebedev MA. Principles of Neural Ensemble Physiology Underlying the Operation of Brain-Machine Interfaces. Nat. Rev. Neurosci. 10: 530540, 2009…………………………...………………………………………….…..305 Lebedev MA, Tate AJ, Hanson TL, Li Z, O'Doherty JE, Winans JA, Ifft PJ, Zhuang KZ, Fitzsimmons NA, Schwarz DA, Fuller AM, An JH, Nicolelis MA. Future
4
developments in brain-machine interface research. Clinics (Sao Paulo) 66 Suppl 1:2532, 2011…………………………………………………………………………...316 Lebedev, MA, Nicolelis MAL. Toward a whole body neuroprosthetic. Prog. Brain Res. 194: 47-60, 2011……………………………………………………….324 Nicolelis MA. Mind in Motion. Sci. Am. 307: 58-63, 2012…………………..338 Nicolelis, MAL. Are we at risk of becoming biological digital machines? Nat. Hum. Behav. 1: Art. 8, DOI: 10.1038/s41562-016-0008, 2017…………………….344 Yadav A and Nicolelis MAL. Electrical stimulation of the dorsal columns of the spinal cord in Parkinson’s disease. Mov. Disord. doi: 10.1002/mds.27033, 2017…………………………………………………………………………….....346 A Comprehensive Review of the BMI Field……………………………………359 Lebedev M and Nicolelis MAL. Brain-machine interfaces: from basics science to neuroprosthetic devices, and neurorehabilitation. Physiol. Rev. 97: 767-837, 2017……………………………………….…………………………….…...……360
5
Introduction Exactly twenty years ago, in July 1999, Nature Neuroscience, then a recently launched scientific journal, published a manuscript entitled “Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex.” The result of one more collaboration between my laboratory and John Chapin’s, this paper launched the field of brain-machine interfaces (BMIs) in earnest and caused a major sensation in the neuroscience community. Curiously, the name brain-machine interface would only appear a year later (Nicolelis, 2001), coined in a review paper entitled “Actions from thoughts” that I wrote for Nature, following a request by one of its editors, Charles Jennings, who happened to be the first editor-in-chief of Nature Neuroscience. To celebrate the twenty-year anniversary of that landmark paper, I have decided to collate all BMI-related articles published by my lab at the Duke University Center for Neuroengineering, as well as the Edmond and Lily Safra International Institute of Neuroscience and the neurorehabilitation laboratory that I founded in Brazil during the past two decades. The aim of this initiative is to place in two volumes, the first of a series entitled the Nicolelis Lab Series, the entire library of manuscripts that my students and I have produced as a result of our research in the BMI field. To some degree, these volumes are also historic documents that show the evolution of our ideas in the area as well as the many “spin offs” that emerged from our tinkering with the original – and now classic – concept of BMIs. That included our transition from using this approach primarily as a basic science tool to probe the brain, in search of the physiological principles governing the behavior of neuronal circuits – or neural ensembles, as I like to call them – to our current major drive to develop assistive and even therapeutic strategies for neurological and psychiatric disorders that incorporate BMIs, either alone or in conjunction with other technologies (e.g. virtual reality and robotics) and clinical approaches. To help seasoned practitioners and newcomers alike, I have divided the manuscripts that form these two volumes in multiple categories, which loosely cover some of the history of the BMI field, at least the way I see it, while offering a reasonable way to reconstruct how the rational and experimental work performed in my labs evolved until it reached its current focus on developing multiple BMI-based therapies. Following this basic structure, Volume 1 begins with a collection of manuscripts that I deem the “Foundation” papers. This section contains some of the key studies executed, initially when I was a postdoc in John Chapin’s lab, and then by my own lab at Duke University that led to the maturation and optimization of the fundamental experimental methods employed in the invention and dissemination of BMI during its first fifteen years (Nicolelis et al., 1995). I am referring to the neurophysiological approach known as chronic, multi-site, multi-electrode recordings in behaving animals, which John Chapin and I originally developed and implemented in rats, between 1989
6
and 1993, and which I lately adapted and scaled up with the goal of carrying out studies with New and Old World non-human primates. This shift from rodents to primates was pivotal for our lab’s ability to follow up the original Nature Neuroscience study with the first demonstration of a BMI in primates, published a year later, in Nature (Wessberg et al., 2000). The next section in Volume 1, Essential Methods, contains a large list of manuscripts that describe in detail all the key neurophysiological, biomedical engineering, computational, robotics, and behavioral approaches that played a central role in the development of all BMIs implemented in my labs. This section is followed by clusters of papers that summarize our experimental studies in rodents and primates. Volume 2 begins with the collection of all our clinical BMI studies to date. The next two sections of Volume 2 are dedicated to three special BMI spinoffs that were originally proposed and developed at great length by our lab at Duke University. I am referring to the creation of sensory neuroprosthesis in rodents (our “infrared rat project”), and two paradigms at the edge of the field: brain-to-brain interfaces and brainets. In the latter section, I include a recent manuscript that reports on the first demonstration of simultaneous, multi-brain, multi-channel recordings based on wireless technology (Tseng et al., 2018). This is particularly relevant because, in the future, this approach will allow neurophysiologists to perform animal social studies while concomitantly recording the brain activity from large numbers of untethered subjects interacting among themselves. By the same token, this approach will allow the concept of “shared BMIs,” i.e. a BMI operated by the simultaneous contributions of multiple individual brains engaged in the collective performance of a behavioral task. The final two sections of Volume 2 contain a series of opinion and review papers, including a very comprehensive and recent review of the entire field of BMI, including its historic origins, and potential future clinical applications (Lebedev and Nicolelis, 2017). Revising all this material brought me a lot of great memories and rekindled some of the tremendous “academic battles” John Chapin and I had to fight in the beginning to convince some of our more recalcitrant and conservative colleagues in systems neuroscience that BMI was not only a legitimate experimental tool to probe the brain, but that, in due time, it could also become a major approach to assist and treat patients suffering from a variety of brain disorders that still today challenge clinicians, given the lack of suitable therapies to ameliorate them. Fortunately, the vast majority of these colleagues, given the evident success of the field and its clear potential benefits, have graciously changed their minds to become enthusiastic supporters of this quickly expanding field of inquiry. A few, sadly, still resist, calling BMI only an “applied science,” as if this is an offense to those of us who took the field, in twenty years, from its basic science origins, all the way to clinical studies that promise to unleash the full potential of this paradigm to millions of patients.
7
Since our public demonstration of the potential impact of BMIs during the opening ceremony of the 2014 Soccer World Cup in Brazil, when a paraplegic patient was able to use a non-invasive BMI to deliver the opening kickoff of that event for a worldwide TV audience estimated at 1.2 billion people, the public fascination with BMIs, and its potential impact for the improvement of patients suffering from brain disorders, has skyrocketed all over the world. Having witnessed, many times over, the enthusiastic reaction of scientific and lay audiences alike all over the planet, when I show them that those humble rat studies, carried out in John Chapin’s Philadelphia lab in the early 1990s, allowed us, merely twenty years later, to offer a paraplegic young man, paralyzed from the mid-chest down, a way to walk again only by his own thinking, I can only say that it was all worthwhile. By a long shot.
8
Time Table of Key Discoveries 1995 First demonstration of chronic, multi-site, multi-electrode recordings in freely behaving rodents (Nicolelis et al., Science 1995). 1998 First chronic, multi-site, multi-electrode recordings in non-human primates (Nicolelis et al., Nature Neuroscience, 1998). 1999 First BMI study using rodents (Chapin et al. Nature Neuroscience 1999). 2000 First BMI for non-human primates (Wessberg et al., Nature 2000). Introduction of the name “brain-machine interface” (Nicolelis Nature, 2000). 2003 First BMI for reaching and grasping in Rhesus Monkeys (Carmena et al. PLOS Biology 2003). 2004 First invasive multi-channel BMI in humans (Patil et al., J. Neurosurgery 2004). 2007 First BMI for bipedal walking using monkey brain activity to control a humanoid robot (Cheng et al., Soc. Neurosci. Abstract 2007). 2009 A new neuroprosthesis for Parkinson’s disease based on spinal cord electrical stimulation (Fuentes et al., Science 2009). 2011 First brain-machine-brain interface for active touch exploration using a multichannel cortical micro-stimulation (O’Doherty et al., Nature 2009). 2013 First BMI for bimanual control (Ifft et al, Science Translation Medicine, 2013) First brain-to-brain interface (Pais-Vieira et al., Scientific Reports 2013). First sensory neuroprosthesis for infrared sensing (Thomson et al., Nature Communications 2013). 2014 First wireless multi-channel BMI in non-human primates (Schwarz et al., Nature Methods 2014). 2015 First brainet for non-human primates (Ramakrishnan et al., Scientific Reports 2015).
9
2016 First demonstration that chronic BMI use leads to neurological recovery in spinal cord injury patients (Donati et al., Scientific Reports, 2016). 2018 First simultaneous, wireless, multi-subject brain recordings in monkeys during performance of a social motor task (Tseng et al., 2018).
10
Clinical Brain-Machine Interface Studies in Parkinsonian and Spinal Cord Injury Patients
11
CLINICAL STUDIES
ENSEMBLE RECORDINGS OF HUMAN SUBCORTICAL NEURONS AS A SOURCE OF MOTOR CONTROL SIGNALS FOR A BRAIN-MACHINE INTERFACE Parag G. Patil, M.D., Ph.D. Division of Neurosurgery, Department of Neurobiology, Duke University Medical Center, Durham, North Carolina
Jose M. Carmena, Ph.D. Department of Neurobiology and Center for Neuroengineering, Duke University Medical Center, Durham, North Carolina
Miguel A.L. Nicolelis, M.D., Ph.D. Departments of Neurobiology, Psychological and Brain Sciences, and Center for Neuroengineering, Duke University Medical Center, Durham, North Carolina
Dennis A. Turner, M.D. Division of Neurosurgery, Department of Neurobiology, and Center for Neuroengineering, Duke University Medical Center, and Neurosurgery and Research Sections, Durham VA Medical Center, Durham, North Carolina Reprint requests: Dennis A. Turner, M.D., Neurosurgery, Box 3807, Room 4530, Duke South Blue Zone, Duke University Medical Center, Durham, NC 27710. Email: dennis.turner@duke.edu Received, December 18, 2003. Accepted, February 23, 2004.
NEUROSURGERY
OBJECTIVE: Patients with severe neurological injury, such as quadriplegics, might benefit greatly from a brain-machine interface that uses neuronal activity from motor centers to control a neuroprosthetic device. Here, we report an implementation of this strategy in the human intraoperative setting to assess the feasibility of using neurons in subcortical motor areas to drive a human brain-machine interface. METHODS: Acute ensemble recordings from subthalamic nucleus and thalamic motor areas (ventralis oralis posterior [VOP]/ventralis intermediate nucleus [VIM]) were obtained in 11 awake patients during deep brain stimulator surgery by use of a 32-microwire array. During extracellular neuronal recordings, patients simultaneously performed a visual feedback hand-gripping force task. Offline analysis was then used to explore the relationship between neuronal modulation and gripping force. RESULTS: Individual neurons (n ⫽ 28 VOP/VIM, n ⫽ 119 subthalamic nucleus) demonstrated a variety of modulation responses both before and after onset of changes in gripping force of the contralateral hand. Overall, 61% of subthalamic nucleus neurons and 81% of VOP/VIM neurons modulated with gripping force. Remarkably, ensembles of 3 to 55 simultaneously recorded neurons were sufficiently informationrich to predict gripping force during 30-second test periods with considerable accuracy (up to R ⫽ 0.82, R2 ⫽ 0.68) after short training periods. Longer training periods and larger neuronal ensembles were associated with improved predictive accuracy. CONCLUSION: This initial feasibility study bridges the gap between the nonhuman primate laboratory and the human intraoperative setting to suggest that neuronal ensembles from human subcortical motor regions may be able to provide informative control signals to a future brain-machine interface. KEY WORDS: Brain-machine interface, Neuroprosthesis, Single-unit recording, Subthalamic nucleus, Thalamus Neurosurgery 55:27-38, 2004
H
DOI: 10.1227/01.NEU.0000126872.23715.E5
uman neuroprosthetic devices currently depend on control signals from residual nerve or muscle activity to restore motor functions lost because of disease or trauma. It has been proposed that these devices could be significantly improved by directly harnessing brain activity from central motor-related regions to drive artificial actuators (6, 14, 17, 21, 22, 31). Recently, laboratory studies involving nonhuman primates have made considerable advances toward the development of such devices. For example, neuronal ensemble recordings from motor areas of cerebral cortex in nonhuman primates have
www.neurosurgery-online.com
been demonstrated to accurately predict three-dimensional arm movements (26, 27, 29) and to successfully control a robotic arm neuroprosthetic device (5, 29). Despite these interesting advances, primate studies have yet to address the fundamental question of whether current brain-machine interface (BMI) technology and approaches may be successfully applied to human patients (9, 21, 22). Singleunit neuronal recordings from human cerebral cortex have been used to drive cursor movement in a simple neuroprosthetic application, establishing feasibility for this approach (14, 15). However, nonhuman primate BMI studies
VOLUME 55 | NUMBER 1 | JULY 2004 | 27
12
PATIL
ET AL.
suggest that multineuronal recordings are critical for neuroprosthetic applications and may require a minimum of 50 to 100 recorded neurons to drive a real-time neuroprosthesis (5, 22). In addition to cortical motor regions, subcortical regions, such as the motor thalamus and subthalamic nucleus (STN), are also involved in motor planning and execution. Neuronal activity associated with upper-extremity movements has been observed in both the STN (1, 25) and motor thalamus (1, 11, 18, 25). Both the motor thalamus and STN are also targets of deep brain stimulation for the treatment of Parkinson’s disease and tremor disorders (2), thereby providing a valuable opportunity to test this hypothesis directly in humans. Here, we hypothesized that ensemble neuronal activity in human subcortical regions may provide effective motor control signals for task prediction and ultimately for neuroprosthetic control. To test this hypothesis, we designed a novel intraoperative visual feedback-associated hand motor task using gripping force to drive cursor movement to a target. In addition, we designed and used a unique 32-microelectrode array for acute multineuronal ensemble recordings, allowing definition of a quantitative relationship between neuronal activity and applied hand force. Our method used linear and nonlinear decoding algorithms to test whether sufficient motor control signals could be derived from these subcortical regions to predict task activity. Multiple individual neurons modulated firing with task performance, and ensembles of neurons provided excellent predictions of patient gripping force, suggesting that subcortical motor targets may provide robust control signals for BMI neuroprosthetic applications.
PATIENTS AND METHODS Patient Selection and Localization of Subcortical Motor Centers Patients with either tremor (essential tremor or Parkinson’s disease with tremor dominance) or severe, refractory Parkinson’s disease were selected for either thalamic (ventralis intermediate nucleus [VIM]) or STN deep brain stimulator (DBS) (Medtronics, Minneapolis, MN) placement. The study was approved and monitored by the Duke University Institutional Review Board. After providing informed consent, all patients were given the option to withdraw from the study at any time. There were no intraoperative complications during the study. All patients underwent postoperative computed tomographic scanning of the head. We observed no evidence of intraoperative hemorrhage or intracranial injury caused by either the recording sessions or DBS placement in any of the participants. For patients who consented to participate, up to 10 minutes of intraoperative recording time was used for these studies. All surgical procedures were performed through frontal burr holes using the Leksell stereotactic frame while patients were awake, after a localization magnetic resonance imaging scan was performed. Standard coordinates were used as the initial
28 | VOLUME 55 | NUMBER 1 | JULY 2004
targets for VIM (4–6 mm anterior to the posterior commissure, 13–15 mm lateral from the midline, and at the anterior commissure–posterior commissure line) and STN (3 mm behind the AC–PC midpoint, 12 mm lateral to the midline, and 4 mm below the anterior commissure–posterior commissure line). The STN was identified physiologically by the cell density, the cell firing frequency, the characteristic length (5–6 mm) of neurons along the recording track, and the presence of kinesthetic cells (25). The VIM was identified by the presence of tremor cells and a location just anterior to the sensory nucleus of the thalamus. For the STN, the multichannel electrode was placed into the middle of the nucleus after the single-channel electrode was used to determine a track length of at least 5.5 mm within the STN. For the motor thalamus, the first track was aimed posteriorly into the sensory nucleus, and the upper aspect of this track (8–10 mm from the sensory nucleus) was used for multichannel recording. This location used for multichannel recording was either directly in or in close proximity to the ventralis oralis posterior (VOP). This location is therefore identified as VOP/VIM.
Motor Task Performance Patients were placed in a supine, semisitting position in front of a video screen monitor. A squeeze ball (Laboratory for Human and Machine Haptics, Massachusetts Institute of Technology, Cambridge, MA) in the hand contralateral to the recording electrode array measured gripping force. Once positioned, the patient was trained to perform the gripping force task as described below, in Results. The patient was instructed to adjust hand-gripping force to reach a target level as rapidly as possible once the target was presented. Once at a target level, the patient was required to hold that level of force for a short period (0.5–1.0 s) before the target level would change. There was no alerting for the next target force level. The patient’s shoulder and arm were allowed to rest on an armboard, so that the only detectable motion used for the task was gripping force. For many of the Parkinson’s disease patients (who were in the “off” state, not receiving medication), there was a considerable delay in reaction time before the force change to move to the new target location was initiated. Typically, patients could perform the task for 5 minutes continuously before tiring, allowing up to 50 task repetitions for a single session while neural ensemble recordings were obtained. Presentation of video images, generation of the motor task, and recording of patient responses was performed by use of custom software written in C⫹⫹ by the first author (PGP).
Electrode Arrays and Electrophysiological Recording Neuronal activity was simultaneously recorded during motor task performance with either a standard single-channel microelectrode (5 m, tungsten, 0.5 M⍀; FHC, Inc., Bowdoinham, ME) or a custom 32-channel platinum-iridium microelectrode array (Fig. 1). The multichannel array was fashioned after that used for chronic single-unit recordings in hippocampal and mesial temporal lobe structures (10, 16) and was
www.neurosurgery-online.com
13
MOTOR SIGNALS
FOR A
BRAIN-MACHINE INTERFACE
chronic recordings in nonhuman primates (23), but it is commercially available and approved by the United States Food and Drug Administration (FDA) for short-term recordings. The single-channel electrode was used to determine the optimal recording depth within a region containing a high density of neurons. The multichannel electrode array was then passed immediately afterward to the same depth (within 1 mm). Although correlations between single-channel electrode recordings and patient movement were sought as part of the standard surgical procedure, correlations between movement and multichannel recordings with the array were not, because of time constraints. Hence, there was no measurement of a direct or specific relationship between active or passive hand movement and the multichannel recordings at the time of electrode placement. Given this functionally random anatomic localization of the multichannel array, we expected the array to record a mixture of responses to hand movement, with no response to movement in some neurons. Neuronal signals were recorded and stored at 40 kHz with a 32-channel personal computer-based acquisition system (Plexon, Inc., Dallas, TX). Spike detection and spike sorting were performed as described previously (23), using a threshold-based criterion for spike detection and principal component analysis for spike sorting.
Detection of Responsive Neurons
FIGURE 1. Multichannel, multineuronal recordings in human subcortical nuclei. A, preoperative and intraoperative appearance of the multichannel array. Left, the array consists of a bundle of thirty-two 40-�m platinumiridium microwires, individually insulated but within an outer sheath (bar ⍽ 1 cm). The array can span up to 4 to 5 mm when inserted to a depth of 10 mm below its brain-protective 1.0-mm stainless steel guide-tube cannula. Right, intraoperative lateral cranial fluoroscopy of the array in situ. The array is seen as diverging microwires within the concentric circular targets of the Leksell stereotactic frame, which is affixed to the patient’s cranium. In the foreground, a contralateral deep brain stimulator (DBS) electrode (Medtronics, Minneapolis, MN) is observed. Each of the four DBS contact electrodes is 1.5 mm in length, for comparison. The 4-mm divergence of the microwires in the STN target region allows the microwires to sample the majority of the STN. B, simultaneously recorded neuronal activity from VOP/VIM with spike detection and sorting. Left, electrophysiological records from four individual microwires. Right, individual waveforms obtained after spike detection and sorting, illustrated on an expanded time base. Waveforms are color-coded to indicate the source channel and normalized by amplitude. Note that channel 12 records two well-differentiated neurons, whereas channel 19 demonstrates a single dominant signal with increased baseline noise, most likely because of increased neuronal density in the neighborhood of this electrode.
organized as a coaxial but diverging bundle to avoid tissue injury (0.6-mm outer diameter). Intraoperative fluoroscopy (Fig. 1A) suggests that during implantation, the microwires diverged to approximately 4 mm, sufficient to cover most of the STN. This microwire array builds on the arrays used for
NEUROSURGERY
Patient responses were sorted according to the change in target force as either increased force or decreased force. Perievent histograms of neuronal activity were constructed separately for positive and negative force transitions by use of 100-ms bins 0.5 to 1.0 second before and 2 to 3 seconds after the time of target force transition. Throughout our analysis, the reference (zero) time for each transition was aligned to the time at which the new target location was presented to the patient, which was defined more precisely in time than the initiation of patient movement. For each patient, the response time between change in target and change in applied force was similar across trials during the recording session and was therefore typically a simple offset from the reference zero time. Neuronal responsiveness to changes in gripping force was defined as changes in neuronal firing rate to beyond the 95% confidence interval for baseline activity. To qualify as a responsive neuron, firing rates were required to modulate beyond the 95% confidence interval in both positive and negative directions with opposite changes in force.
Prediction of Motor Activity from Neuronal Activity Prediction of motor activity from neuronal recordings with a Wiener filter (4) requires approximately 10 minutes of continuous data to determine the parameters of the model (5, 29). However, the conditions in which the current human recordings were performed do not allow continuous recordings for more than 5 minutes. Hence, other linear and nonlinear methods that require fewer data and less training time to generalize than the Wiener filter, such as a Kalman filter (12), normalized
VOLUME 55 | NUMBER 1 | JULY 2004 | 29
14
PATIL
ET AL.
least mean squares (LMS) (12), and feed-forward artificial neural network (ANN) (3) algorithms, were tested. Each model was typically trained using 3 minutes of data, with predictions made for subsequent 30-second epochs. Neuronal data were binned at 100 milliseconds. The Kalman filter was given an initial condition equal to the real initial condition and an a priori estimate of the error covariance set to 0.1; LMS was trained with 10 lags and a learning rate of 0.2; the ANN had 2 layers (10 units hidden-layer) with tangent sigmoidal and linear transfer functions, respectively, and was trained using conjugate gradient descent with Powell-Beale restarts. In cases in which the output of the models contained high-frequency ripple, the signal was low-pass filtered with a fifth-order Butterworth filter (cutoff frequency, 2 Hz) and multiplied by a constant gain factor of 1.2 (all parameters were found experimentally). Analyses were performed in Matlab (The MathWorks, Natick, MA) using the Neural Networks toolbox (8). To predict motor activity at a given moment in time, only neuronal data from up to 1 second before that moment were used in the calculation. Neuronal firing after the moment of interest was excluded from analysis. This prevented the inadvertent use of neuronal activity generated in response to the movement and limited calculations to neuronal activity associated with motor planning.
RESULTS Ensemble Neuronal Recordings from Human Subcortical Motor Centers We recorded the electrophysiological activity of neuronal ensembles from the STN and thalamic motor regions (VOP/ VIM) in patients with Parkinson’s disease and essential tremor, respectively, during surgical placement of DBS devices (Fig. 1A). All patient selection and intraoperative studies were completed in accordance with the approval, policies, and stipulations of the Duke University Institutional Review Board. Microelectrode recording was performed with either a standard single-channel tungsten microelectrode or a 32channel platinum-iridium microelectrode array. Figure 1A illustrates the 32-channel array, shown photographically before implantation and fluoroscopically within the STN during a recording session, opposite a contralateral DBS electrode that had just been placed. Note that the small microwires are barely visible but clearly show divergence and are close to the target area. The results presented here were obtained from 13 STN recording sessions in 8 patients (n ⫽ 119 neurons) and 8 motor thalamic (VOP/VIM) recording sessions in 3 patients (n ⫽ 26 neurons). Between 3 and 55 neurons were recorded simultaneously, depending on the composition of the signals measured from each of the microelectrodes in the array. As illustrated in simultaneously recorded traces in Figure 1B (from VOP/VIM), spike discharges from individual units varied in both morphology and frequency. Details of the sorted action potentials from single neurons obtained in simultaneously
30 | VOLUME 55 | NUMBER 1 | JULY 2004
recorded channels are shown to the right of the traces. Baseline recordings reflected both the observed noise level and the fact that high-quality single neurons could be acutely isolated. Neuronal activity was resolved by use of a threshold criterion for spike detection, whereas individual units were differentiated in a single microelectrode signal by principal-component analysis (5, 23). By use of these methods, high-quality neuronal units could be sensitively and specifically detected and differentiated (Fig. 1B, right). In rare channels, neuronal discharges could not be reliably differentiated and were therefore analyzed as a single composite unit. These results confirm that a simple microwire recording array functions effectively to capture multiple, simultaneous channels of neuronal data in vivo in human subcortical targets. The channels were independent, with varying spike frequency and spike morphology, because of the separation of the microwires, as confirmed by the intraoperative fluoroscopy (Fig. 1A).
Ensemble Neuronal Activity during Motor Task Performance During each operative recording session, patients performed a novel hand-gripping task synchronously with physiological recording (Fig. 2). A visual representation of gripping force was provided to the patient as the location of a black vertical bar on a video monitor (Fig. 2A, inset). The target area into which the patient had to move the black bar was represented by a green rectangle on the screen, which varied randomly in a time-dependent manner. The motor task required the patient to match the black bar to the green target by adjusting the degree of gripping force exerted on a hand-held pressure bulb. The response of the vertical bar movement to gripping force was scaled so that patients could reliably reach positions over the entire target range. The absolute amount of force required for full-range movement therefore varied among patients but was held constant during each recording session. Release of the pressure bulb caused the vertical bar to move off scale to the left. This constant maintenance of gripping pressure required the patient to remain attentive to the task. Our patients were able to perform the hand motor task for 3 to 5 minutes before tiring. Kinematic records of task performance reflect the delayed reaction time and baseline tremor of patients with Parkinson’s disease (black trace, Fig. 2A). Figure 2B shows a 24-channel STN ensemble of neuronal activity represented by a raster plot, which was recorded simultaneously with the epoch of the behavioral task shown in Figure 2A. Consistent with previous studies (1, 11, 18, 25, 28), some neuronal units in both STN and motor thalamus seemed to respond to upper-extremity movement. By simultaneously recording both neuronal activity and force generation, correlation between neuronal activity and motor task performance could be observed and defined quantitatively. For the recording session shown in Figure 2, note that many of the neurons modulate with patient force production (Fig. 2B).
www.neurosurgery-online.com
15
MOTOR SIGNALS
FIGURE 2. Simultaneous measurement of kinematic parameters and neuronal activity during performance of a motor task. A, patients tracked a randomly time-varying target on a video screen (inset) by applying variable gripping force to a pressure measurement device in the contralateral hand. The kinematic record illustrates target force (green) and patient gripping force (black) over time. Complete release of the device results in movement of the black cursor off-screen to the left. Hence, the patient was required to maintain attentiveness and positive gripping force throughout the session. The sensitivity of the pressure ball was adjusted to allow comfortable movement throughout the target range before each experimental session, and although linear, it was not recalibrated. Positive deflection indicates increased gripping force. B, simultaneously recorded STN neuronal activity, in raster format, measured with a 32-channel multielectrode array. Note the correspondence of neuronal modulation to force production in this 24-unit recording. Time scale is the same as in A.
Quantitative Correlation of Neuronal Activity to Motor Task Performance Modulation of neuronal firing with gripping force was examined in detail by constructing peri-event histograms of neuronal activity aligned to the time of target movement (Fig. 3). Overall, 7 (27%) of 26 VOP/VIM neurons showed increased firing rates with increased force (positive correlation), whereas 14 (54%) of 26 VOP/VIM neurons showed decreased firing rates with increased force (negative correlation), as tested with 95% confidence limits around the baseline firing frequency. By comparison, 46 (39%) of 117 STN neurons showed positive correlation between firing frequency and force, and 24 (21%) of 117 STN neurons demonstrated negative correlation. Hence, 81% of VIM/VOP neurons and 61% of STN neurons showed statistically significant modulation with hand-gripping force (19). Temporal relationships between neuronal modulation and changes in force are apparent through comparison between the timing of changes in neuronal firing, as represented by the histograms at the bottom of each panel in Figure 3, and the timing of changes in force, as represented by curves at the top of each panel. Note that because of relatively short epochs
NEUROSURGERY
FOR A
BRAIN-MACHINE INTERFACE
(0.5–1 s) at each target force level, the average force records may not have sharply defined baseline levels and transition times. However, the polarity of correlation and the phasic relationship between firing frequency and force are quite apparent. Two exemplar VOP/VIM neurons (Fig. 3A, left and middle) illustrate a negative correlation of firing frequency and force in which changes in neuronal firing precede changes in force generation. By comparison, the modulation of the remaining VOP/VIM exemplar neuron (Fig. 3A, right) also precedes changes in force generation but is positively correlated. By contrast, changes in STN firing also demonstrated positive (Fig. 3B, left and right) and negative (Fig. 3B, middle) correlation; however, the time course of STN modulation was slower, making determination of the temporal relationship between neuronal firing and force generation more difficult to discern.
Ensemble Neuronal Analysis and Task Prediction
Studies in nonhuman primates demonstrate that neuronal ensemble recordings from motor areas of cerebral cortex accurately predict three-dimensional arm movements (26, 27, 29) and are able to successfully control a robotic arm neuroprosthetic device (5, 29). To assess the suitability of ensembles of subcortical neurons as a source of motor control signals for a human BMI neuroprosthesis, we examined the ability of subcortical ensemble activity to predict patient gripping force. We estimated predictive ability by training linear (Wiener filter, Kalman filter, normalized LMS [12]) and nonlinear (feed-forward ANN [3]) algorithms with force and neuronal data during “training” periods of up to 300 seconds. These trained models and the neuronal activity during a subsequent, nonoverlapping “test” period were then used to predict gripping force during the test period. Despite severe constraints because of limited time, small neuronal ensembles, and operative and patient considerations, the recorded neuronal signals were sufficiently informationrich that the trained models were able to predict handgripping force during the test period quite accurately, as shown for four sessions (Fig. 4, A–D, R2 ⫽ 0.38–0.68). Overall prediction performance across 20 recording sessions was significant, although modest (R2 ⫽ 0.26 ⫾ 0.04; mean ⫾ standard error of the mean). In general, the ANN, LMS, or Kalman filter decoding algorithms outperformed the simpler Wiener filter algorithm. However, no single decoding algorithm demon-
VOLUME 55 | NUMBER 1 | JULY 2004 | 31
16
PATIL
ET AL.
FIGURE 3. Modulation of individual neurons in VOP/VIM and STN during task performance in six patients. A, VOP/VIM neurons exhibiting modulation with task performance. Each panel illustrates average patient gripping force production (top), neuronal firing in raster format (middle), and a peristimulus time histogram relationship (bottom). The histogram is aligned so that target transitions to a new force level occur at time zero. Panels for increasing target force (above) and panels for decreasing target force (below) are displayed for each modulated neuron. Note that the VOP/VIM neuron represented in the left panel demonstrates an inverse correlation of firing with patient force production. The right neuron demonstrates the opposite. B, STN neurons exhibiting modulation with task performance; same format as in A.
strated superiority over the others (data not shown). Furthermore, similar to findings in nonhuman primate studies (29), prediction accuracy increased with both training period duration (Fig. 4E) and the number of neurons within the ensemble (Fig. 4F), suggesting that the longer training intervals and the larger ensemble sizes possible with chronic implantation of a human BMI would probably result in improved BMI performance. These findings demonstrate feasibility for the hypothesis that chronically recorded neuronal units in subcortical motor centers may provide an effective motor control signal for task prediction. In nonhuman primate studies, model training periods of 10 minutes are common, and then animal training
32 | VOLUME 55 | NUMBER 1 | JULY 2004
FIGURE 4. Prediction of gripping force from recorded neuronal activity by linear and nonlinear decoding algorithms. A–D, four examples of gripping force prediction from VOP/VIM and STN neuronal activity during task performance. A, prediction using ANN (4 VOP/VIM neurons, R ⫽ 0.65 for ANN, R ⫽ 0.14 for Wiener). B, prediction using ANN (8 VOP/ VIM neurons, R ⫽ 0.62 for ANN, R ⫽ 0.48 for Wiener). C, prediction using LMS (26 STN neurons, R ⫽ 0.82 for LMS, R ⫽ 0.14 for Wiener). D, prediction using Kalman filter (4 STN neurons, R ⫽ 0.67 for Kalman, R ⫽ 0.35 for Wiener). E, increasing predictive quality with longer training time (3 VOP/VIM neurons, Wiener method). Data are illustrated as mean correlation coefficient ⫾ standard error of the mean for 20 trials with randomly selected, nonoverlapping training and testing epochs. F, increasing accuracy of gripping-force prediction with larger neuronal ensembles. Explanatory power (R2) increased with the number of neurons when linear or nonlinear methods were used (blue). One data point is shown for each of the 20 recording sessions. In each case, the best prediction for the session is plotted.
periods of several days may be needed for accurate task prediction. Our findings resemble the results of nonhuman primate studies (5, 29), because increasing correlations between hand-gripping force and model predictions were obtained by adding neurons and lengthening the training period. Thus, the multichannel human data are highly promising for development of a chronic human multichannel BMI if the recordings can be extended to larger groups of neurons and the models remain useful for these chronic recordings.
DISCUSSION Overall, our findings are novel in several important respects. This is the first demonstration of simultaneous recording of sizable numbers of human subcortical neurons, indicating that microwire arrays (in some form) are likely to be feasible for use in a future human neuroprosthetic device (Fig. 1). The individual neurons recorded and their response to a behavioral task seem to be similar to those observed by many groups, indicating that the multichannel electrodes are faithfully recording appropriate signals from these subcortical mo-
www.neurosurgery-online.com
17
MOTOR SIGNALS
tor regions (Figs. 2 and 3). The power of ensemble recordings over single-unit recordings is the ability to use the neurons together in a decoding algorithm to predict motor activity. In several sessions (Fig. 4), we found that models based on extremely short training periods could accurately predict handgripping force. Such quantitative predictions are a far more stringent analysis than traditional single-unit correlations between changes in neuronal firing characteristics and task performance. In addition, such predictions, calculated on neural and motor activity in a previous, nonoverlapping period, provide the key to implement real-time brain-machine interfaces for neuroprosthetic and motor enhancement applications. Our study thereby provides translation of this conceptual understanding from chronic, nonhuman primate recordings to the acute human intraoperative setting and suggests that ensembles of subcortical neurons may potentially serve as a source of motor control signals for a neuroprosthetic BMI in humans.
Microwire Recording Electrodes Many different electrode designs have been suggested for the chronic ensemble recordings required of a neuroprosthetic BMI. These designs include silicon-based chip electrodes (20), the neurotrophic electrode (14, 15), and either fixed grid arrays of microwires (23) or loose microwires inserted as a bundle (10, 16). Among these, microwire arrays have been demonstrated to record chronically from multiple neurons for periods up to 2 years (23). In addition, at the present time, none of the electrode arrays are FDA-approved or available, except for the neurotrophic electrode and a loose bundle of microwires. We therefore implemented a 32-channel version of these microwire bundle electrodes. During recording sessions, multiple channels demonstrated excellent performance in either the STN or VOP/VIM (Fig. 1) on a short-term basis, and the results correlated well with traditional single-unit, sharp recording electrodes. As demonstrated by high-resolution fluoroscopy, the microwires diverge a few millimeters when in position. Physical divergence of the microwires and independent firing characteristics among separate channels together suggest that the recordings arise from different neurons. This ability to record from multineuronal ensembles is a critical step in the design of a motor BMI neuroprosthesis, because up to 50 to 100 neurons may be needed simultaneously to provide sufficient predictive accuracy for task performance.
FOR A
BRAIN-MACHINE INTERFACE
of STN neurons showed statistically significant modulation during the hand-gripping force task, consistent with nonhuman primate, mammalian, and human studies of motor activity in these structures (7, 18, 19, 30). Different neurons recorded from the microwire arrays clearly demonstrated a wide variety of task-related responses, indicating that these neurons are functionally independent, even though separated by only a few hundred micrometers within these small, subcortical structures. Although only a subset of neurons demonstrated a clear relationship to the task, this is expected, considering that only a fraction of neurons in either STN or thalamus respond directly to hand tasks, compared with either no response or a response to motion of other body regions. Thus, our multichannel array results are very different from previous singlechannel studies, in which the single-channel electrode could be moved and recordings could be limited to responsive neurons. In this case, the multichannel electrode array was placed in a fixed position and maintained at that position for several minutes to allow the neurons to recover before the recording session was started. All of the recording sessions were performed in patients undergoing movement disorder surgery for Parkinson’s disease or for tremor disorders. Responses to the motor task were similar to those observed in normal, nonhuman primates. It therefore seems that qualitatively similar results would be likely to be obtained in subjects without movement disorders, such as quadriplegics requiring a motor BMI neuroprosthesis. Differences between our subjects and future BMI patients may be particularly pronounced in STN. Because all of the STN neurons were recorded in severe Parkinson’s patients not receiving their medications for more than 12 hours (the offmedication behavioral condition), it is possible that the modulation of neuronal firing rate with hand activity could be different (i.e., more STN neurons responsive to the behavioral task) in patients receiving medication (the on-medication behavioral condition), which was not tested. In addition, the on state may enhance the reactivity of the neurons to the task. Such considerations may have some bearing on the relatively pronounced and rapid modulatory effects observed in VOP/ VIM compared with STN (Fig. 3). Thus, the off state of the Parkinson’s disease could potentially dampen the correlation between neuronal firing and the behavioral task.
Individual Neurons and the Motor Task
Multineuronal Ensembles and Motor Task Prediction
Our results confirm that many neurons in human subcortical motor areas are actively involved in hand-gripping force motor tasks (18, 19). Although subcortical neurons are commonly analyzed as a standard aspect of movement disorder surgery, by assessing a subjective relationship to passive or to active movement, a timed behavioral task to define the specific relationship between neuronal activity and a task is critical (19). From our peri-event histograms, aligned with transitions in target location, 81% of VOP/VIM neurons and 61%
There are several ways to analyze the relationship between multineuronal ensembles and the synchronous behavioral task. We have chosen a predictive model, which is a stringent requirement but one that is critical for BMI performance. To achieve a high predictive accuracy, there must be multiple single neurons within the ensemble, which together provide information on the motor task structure. In addition, real-time predictions are critical to define motor performance for a realistic implementation of a human neuroprosthetic BMI.
NEUROSURGERY
VOLUME 55 | NUMBER 1 | JULY 2004 | 33
18
PATIL
ET AL.
The predictive algorithms used in this study interpret information contained within neuronal activity before and not after the motor activity of interest, suggesting that the neuronal ensembles encode a motor plan rather than merely reflecting a past history of sensory activation. STN and VOP/VIM neurons typically become active before muscle activation and movement initiation rather than afterward (7). The predictions (for four sessions) shown in Figure 4 are also highly robust, in that the neuronal ensembles reliably encode both multiple speeds and ranges of force. Likewise, as shown in Figure 4E, as model training periods are lengthened, predictions may improve. In addition, there may be improvements with increased numbers of neurons. The demonstration that even a few short sessions (up to 5 min) of multineuronal recording activity can show predictive accuracy for the task clearly confirms the feasibility of this method of analysis for the human data and the likelihood that these results might be extended to chronic recording conditions.
sion to a motor neuroprosthetic actuator (21, 22). Because all of the DBS systems are placed in subcortical targets, the direct clinical translation from DBS to a motor neuroprosthesis in the same location may be a shorter path. Potentially, the ability of the approach proposed here could be substantially improved by increasing the model training time to allow a better predictive accuracy, increasing the number of neurons driving the BMI and extending the task training time after the model training has been established (13, 22, 24). Therefore, our results strongly suggest that subcortical motor areas may provide additional sites from which to derive direct brain control signals for human neuroprosthetics applications. Our efforts also support the contention that many of the BMI principles derived from work with nonhuman primates are highly relevant to the human intraoperative setting.
Correlation with Nonhuman Primate Data
1. Abosch A, Hutchison WD, Saint-Cyr JA, Dostrovsky JO, Lozano AM: Movement-related neurons of the subthalamic nucleus in patients with Parkinson disease. J Neurosurg 97:1167–1172, 2002. 2. Benabid AL, Koudsie A, Benazzouz A, Piallat B, Krack P, Limousin-Dowsey P, Lebas JF, Pollak P: Deep brain stimulation for Parkinson’s disease. Adv Neurol 86:405–412, 2001. 3. Bishop CM: Neural Networks for Pattern Recognition. Oxford, Oxford University Press, 1995. 4. Brillinger DR: Time Series: Data Analysis and Theory. San Francisco, HoldenDay, 1981. 5. Carmena JM, Lebedev MA, Crist RE, O’Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MA: Learning to control a brainmachine interface for reaching and grasping by primates. PloS Biol 1:E42, 2003. 6. Chapin JK: Neural prosthetic devices for quadriplegia. Curr Opin Neurol 13:671–675, 2000. 7. Cheruel F, Dormont JF, Farin D: Activity of neurons of the subthalamic nucleus in relation to motor performance in the cat. Exp Brain Res 108:206– 220, 1996. 8. Demuth H, Beale M: Neural Network Toolbox. Natick, The Math Works, 2001. 9. Donoghue JP: Connecting cortex to machines: Recent advances in brain interfaces. Nat Neurosci 5[Suppl]:1085–1088, 2002. 10. Ekstrom AD, Kahana MJ, Caplan JB, Fields TA, Isham EA, Newman EL, Fried I: Cellular networks underlying human spatial navigation. Nature 425:184–188, 2003. 11. Guillery RW, Sherman SM: The thalamus as a monitor of motor outputs. Philos Trans R Soc Lond B Biol Sci 357:1809–1821, 2002. 12. Haykin S: Adaptive Filter Theory. Upper Saddle River, Prentice-Hall, 2002. 13. Helms Tillery SI, Taylor DM, Schwartz AB: Training in cortical control of neuroprosthetic devices improves signal extraction from small neuronal ensembles. Rev Neurosci 14:107–119, 2003. 14. Kennedy PR, Bakay RAE: Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport 9:1707–1711, 1998. 15. Kennedy PR, Bakay RAE, Moore MM, Adams K, Goldwaithe J: Direct control of a computer from the human central nervous system. IEEE Trans Rehabil Eng 8:198–202, 2000. 16. Kreiman G, Koch C, Fried I: Category-specific visual responses of single neurons in the human medial temporal lobe. Nat Neurosci 3:946–953, 2000. 17. Lauer RT, Peckham PH, Kilgore KL, Heetderks WJ: Applications of cortical signals to neuroprosthetic control: A critical review. IEEE Trans Rehabil Eng 8:205–208, 2000. 18. Lenz FA, Kwan HC, Dostrovsky JO, Tasker RR, Murphy JT, Lenz YE: Single unit analysis of the human ventral thalamic nuclear group: Activity correlated with movement. Brain 113:1795–1821, 1990.
Several groups have demonstrated the feasibility of a BMI neuroprosthesis in nonhuman primates, but only from cortical recording sites in known motor areas (primary motor, premotor, and posterior parietal). The general principles demonstrated by these studies include the requirement for a large number of neurons to fully elaborate a motor plan (a minimum of 50–100), the use of efficient models to correlate the neuronal firing characteristics with the behavioral task, and considerable animal training over days to weeks. However, there have been several limiting features in the translation of this concept to humans. First, the only FDA-approved (for chronic implantation) single-unit electrodes for humans have been the neurotrophic cone electrodes (14, 15), and these electrodes are limited for multineuronal ensembles. None of the electrodes used in the nonhuman primate studies are FDAapproved, and many of these designs are not feasible to consider for humans because of their materials (i.e., tungsten or silicon rather than platinum-iridium), methods of placement, lack of long-term ability to record neurons, and manufacture. In addition, none of the electronics are implantable at this point, particularly for a 32-channel multineuronal system. Many of the critical engineering and design questions remain unanswered for a human neuroprosthetic system, awaiting the clear demonstration of feasibility of the principle of correlating multineuronal ensemble outputs with a behavioral task. Our present data start to provide these initial feasibility data for short-term recordings from subcortical structures. Although all of the nonhuman primate neuroprosthetic data are from cortical sites, in humans it may be also feasible to consider subcortical motor sites, particularly thalamus, because of the considerable clinical experience with this location and the enhanced stability of the electrodes within the brain. In addition, a proposed motor neuroprosthetic system may seem to resemble a DBS system in many ways, with a 1-mm recording electrode, a computational box, and radiofrequency transmis-
34 | VOLUME 55 | NUMBER 1 | JULY 2004
REFERENCES
www.neurosurgery-online.com
19
MOTOR SIGNALS
19. MacMillan ML, Dostrovsky JO, Lozano AM, Hutchison WD: Involvement of human thalamic neurons in internally- and externally-generated movements. J Neurophysiol 91:1085–1090, 2004. 20. Maynard EM, Nordhausen CT, Normann RA: The Utah Intracortical Electrode Array: A recording structure for potential brain-computer interfaces. Electroencephalogr Clin Neurophysiol 102:228–239, 1997. 21. Nicolelis MA: Actions from thoughts. Nature 409:403–407, 2001. 22. Nicolelis MA: Brain-machine interfaces to restore motor function and probe neural circuits. Nat Rev Neurosci 4:417–422, 2003. 23. Nicolelis MA, Dimitrov D, Carmena JM, Crist R, Lehew G, Kralik JD, Wise SP: Chronic, multisite, multielectrode recordings in macaque monkeys. Proc Natl Acad Sci U S A 100:11041–11046, 2003. 24. Paninski L, Fellows MR, Hatsopoulos NG, Donoghue JP: Spatiotemporal tuning of motor cortical neurons for hand position and velocity. J Neurophysiol 91:515–532, 2004. 25. Rodriguez-Oroz MC, Rodriguez M, Guridi J, Mewes K, Chockkman V, Vitek J, DeLong MR, Obeso JA: The subthalamic nucleus in Parkinson’s disease: Somatotopic organization and physiological characteristics. Brain 124:1777– 1790, 2001. 26. Serruya MD, Hatsopoulos NG, Paninski L, Fellows MR, Donoghue JP: Instant neural control of a movement signal. Nature 416:141–142, 2002. 27. Taylor DM, Tillery SI, Schwartz AB: Direct cortical control of 3D neuroprosthetic devices. Science 296:1829–1832, 2002. 28. Theodosopoulos PV, Marks WJ Jr, Christine C, Starr PA: Locations of movement-related cells in the human subthalamic nucleus in Parkinson’s disease. Mov Disord 18:791–798, 2003. 29. Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, Kim J, Biggs SJ, Srinivasan MA, Nicolelis MA: Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408:361–365, 2000. 30. Wichmann T, Bergman H, DeLong MR: The primate subthalamic nucleus: Part I—Functional properties in intact animals. J Neurophysiol 72:494–506, 1994. 31. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM: Brain-computer interfaces for communication and control. Clin Neurophysiol 113:767–791, 2002.
Acknowledgments We thank Mikhail Lebedev, Joseph O’Doherty, Craig Henriquez, Hyun Kim, Mandayam Srinivasan, Dragan Dimitrov, Christopher Beaver, Ashutosh Pradhan, and Kent New for contributions to this study. This work was supported by a National Institutes of Health Training Grant to PGP and by a Defense Advanced Research Projects Agency grant to JMC, MALN, and DAT. JMC was also supported by the Christopher Reeve Paralysis Foundation. The authors have no competing financial interests.
COMMENTS
P
atil et al. describe their experience measuring thalamic and subthalamic nucleus neuronal activity during the execution of visual motor tasks. The findings are gleaned from a small number of patients who underwent surgical treatment for Parkinson’s disease. In these patients, the authors observed that ensemble recordings of multiple neurons analyzed through a decoding algorithm correlated well with the task of hand gripping on cue. This study has several important implications for neurosurgery. First, the authors effectively and ethically used the operating room to explore brain physiology. Working through their institutional review board, they were able to perform the additional testing during deep brain stimulator lead implantation without incurring additional morbidity. Although this is not a new paradigm, it is important because the data col-
NEUROSURGERY
FOR A
BRAIN-MACHINE INTERFACE
lected from primate research must be challenged ultimately in the human model. The clinical relevance of this work is apparent. The complex organization and function of the central nervous system precludes biological attempts at neurorestoration from being realized in the near future. However, rapid progress in the fields of microprocessor and microelectrode technology may offer solutions to disabling neurological disorders. The concept of a brain-machine interface (BMI), whereby conceived motor tasks are translated and even executed with the assistance of electrical-mechanical devices, soon may become reality. Michael Y. Wang Los Angeles, California
I
n this article, the authors examine the feasibility of using neurons in the subcortical motor areas to drive a human BMI. During implantation of deep brain stimulators, ensemble recordings were made by use of microwire arrays from neurons in either the subthalamic nucleus or thalamic motor areas, while patients performed a visual feedback/hand grip force task. The recordings then were analyzed using linear and nonlinear decoding algorithms to predict task activity. The authors demonstrate that multiple individual neurons modulate firing with task performance and that excellent predictions of grip force could be obtained from measurements of the ensembles of neurons. This suggests that ensembles of subcortical neurons may be used as a source of motor control signals to drive a neuroprosthetic device. This article represents the first simultaneous recording of sizable numbers of human subcortical neurons, and the recordings from these neurons can be used together to predict motor activity. Furthermore, as the authors state, this work represents an important bridge between nonhuman primate laboratory studies and the human intraoperative setting. Charles Y. Liu Los Angeles, California
T
he authors describe the successful recording of multiunit activity from ventralis oralis posterior/ventralis intermedius and subthalamic nucleus, and they correlate this with contralateral grip force in a brief intraoperative recording session. They have managed not only to record usable units; they also have been able to demonstrate a relationship between the activity of the majority of neurons and grip force. This is impressive work, and it may open the door to thinking about subcortical structures as a probe for BMI, which was heretofore unexplored in humans. The authors’ implicit contention is that subcortical structures encode movement with a degree of specificity analogous to the primary motor cortex. This is somewhat implausible, given what we know of the motor system. There is a vast difference between a gross measure of force and accurate and finely tuned movements mediated by the motor cortex. I have doubts as to whether this approach ever will be able to control useful movement from a BMI.
VOLUME 55 | NUMBER 1 | JULY 2004 | 35
20
PATIL
ET AL.
The paradigm also is an area of concern. It is clear that certain cells in the ventralis oralis posterior/ventralis intermedius and subthalamic nucleus have kinesthetic receptive fields. That movements of the extremity can yield a positive correlation to the firing behavior of neurons within these structures is not surprising. This is almost certainly on the afferent side of what in this case is a task involving closedloop motor control. How this kinesthetic feedback could pertain to a BMI in a quadriplegic patient or amputee is not at all apparent. Kim J. Burchiel Portland, Oregon
P
atil et al. inserted multiple microwires to fan out within the thalamus or the subthalamic nucleus in patients undergoing functional neurosurgery for movement disorders. They demonstrated that these microwires can record for individual neurons, and in so doing, they were able to record multiple units simultaneously. The authors further demonstrated that these units are responsive, as one would expect, to motor inputs. In addition, they demonstrated that the activities from these units could be used to predict and in theory serve as the input station of a device that would analyze and direct the output to a machine. This is pioneering work that represents an important first step toward the development of a BMI. The only point of criticism is the authors’ suggestion that in parkinsonian patients, the number of units or patients’ sensitivity of movements may increase in the on-medication state. This is an error, as it is well known that in the parkinsonian state, there is defocusing of motor units and an excessive number of units respond to movement, which is reduced with use of dopamine agonists such as apomorphine (1, 2). Another practical point is that the excursion of the 32 microwires is 4 to 5 mm, which would take up the majority of the ventralis intermedius or the large part of the subthalamic nucleus. Could this alone produce a microlesion effect? It would interesting to learn whether the authors observed a significant benefit merely from the introduction of these 32 wires, and how they perceive the safety of introducing such wires within these brain targets.
ble extracellular neuronal recordings were obtained from both the subthalamic nucleus and ventralis oralis posterior/ ventralis intermediate nucleus during a visual feedback/hand grip force task. The electrode data were then used to train a model to predict grip force on the basis of a set of recordings. It is notable that the data set used was the set of neuronal activity 1 second before movement, which therefore was representative of the “motor planning” stage rather than activity as a response to movement. Before this study, cortical recordings had been used in monkeys to control a robotic arm using similar predictive model techniques. This article represents the necessary transition from nonhuman primates to humans so that this technology may become a clinical application. That subcortical targets were used in this study, rather than cortical recording, confers several practical advantages. The homunculus representations in subcortical nuclei are more compact than at cortical surfaces, thus minimizing the size of electrode arrays needed for recording. In addition, fixation may be more stable with smaller subcortical implants than with larger surface grids. The implications of this technology extend to creating a BMI that could be used by patients with severe neurological compromise such as spinal cord injury or stroke; however, this study was performed in relatively intact patients (no focal injury). The predictive models used herein, as well as in the nonhuman primate studies, require a model training session to acquire data using real-time recordings on the basis of motor responses. Individuals with neurological compromise will not be able to “train” the predictive models. On the basis of neuronal structure and electrode placement, recordings from individual patients are likely to be unique. Therefore, models cannot be trained by one person and used by another. It also has been demonstrated that after peripheral injury, both cortical and subcortical neurons reorganize and continue to change for some time after the acute event. Will this plasticity affect predictive responses in the absence of trophic factors stabilizing neuronal units? Regardless, this article represents a major step toward developing a BMI for human use. This is the beginning of what will be a very exciting technological advance in neurosurgery. Lee Tessler Patrick J. Kelly New York, New York
Andres M. Lozano Toronto, Ontario, Canada
1. Filion M, Tremblay L, Bedard PJ: Abnormal influences of passive limb movement on the activity of globus pallidus neurons in parkinsonian monkeys. Brain Res 444:165–176, 1988. 2. Levy R, Dostrovsky JO, Lang AE, Sime E, Hutchison WD, Lozano AM: Effects of apomorphine on subthalamic nucleus and globus pallidus internus neurons in patients with Parkinson’s disease. J Neurophysiol 86:249–260, 2001.
T
he authors describe a novel study with potential that far exceeds the conclusions reached in this article. Subcortical recordings were obtained from patients undergoing deep brain stimulation procedures for Parkinson’s disease. Ensem-
36 | VOLUME 55 | NUMBER 1 | JULY 2004
T
he concept of a BMI is the subject of great science fiction writing. Only recently has it come to be considered a potential reality. Within the last decade, BMI has become an area of active research pursuit. Multiple and seminal conceptual breakthroughs were required for the first successful demonstration of the proof of principle for a direct brain-computer interaction (9). This required a stable chronic recording platform, located in an area related to motor activity, which could be directed to change via cognitive activity in the presence of complete paralysis and obtain signals that could be transmitted and processed to provide useful applications. The culmi-
www.neurosurgery-online.com
21
MOTOR SIGNALS
nation was a subsequent demonstration that by thought alone, a patient could move a cursor on a computer screen and type out responses to specific inquiries (10). This resulted in considerable recognition in the press and, a World Technology Award for Health and Medicine. More importantly, however, colleagues were stimulated to pursue the goal of BMI use for more advanced purposes such as controlling neuroprosthesis. This article demonstrates use of the operating room as an exquisite laboratory to investigate the human brain. Using the opportunities available during surgery for deep brain stimulation, the authors demonstrated conclusively that subcortical areas could be used as a source of motor control signals for a BMI. Their use of an electrode array to predict motor activity with a relatively small number of neurons for a specific task represents a significant advancement. That neurons could be functionally independent although separated by “a few hundred microns” is not new, as in these small subcortical structures, microelectrode recordings of individual units can identify cells that have different kinesthetic functions only tenths of microns apart (8, 11, 16). That there could be plasticity changes in only a few trials, however, is much more interesting and potentially more useful. The evidence for neuroprosthesis is based on identification of a large number of neurons, usually ranging from 50 to 100, that would need to be recorded simultaneously to provide sufficient predictive accuracy for specific task performance. Thus, to pick up a glass with a prosthetic device, one must reach out to the glass with the appropriate trajectory and distance, so that the glass is not knocked over, and end close enough to grasp the glass. In the process of grasping, one must then hold the glass tightly enough that it does not fall but not so tightly that it shatters. The return can be equally precarious, requiring the need for a smooth approach to the mouth by a different pathway so as not to undershoot or overshoot while carrying the weight of the glass and maintaining the fluid horizontal. This is an extremely demanding task that currently is beyond our abilities. Multineuronal assembly allows a number of these aspects to be examined individually. Considering only one phase, grasping, which was the task studied in this article, there is a suggestion that a smaller number of cells may be necessary to perform it. Nonetheless, even if we were to identify 100 neurons that allowed accurate prediction of the pathway necessary to reach out to a glass, grasp it appropriately, and it bring it close enough to take a drink, this assembly still would fail in a fundamental way to bring a human neuroprosthesis closer to reality. We will not be able to identify these 100 cells and their functional properties before the catastrophic event that necessitates prosthesis. The damage already will have been done. It is necessary to identify units that will fire when reaching, grasping, and returning to the mouth is simply conceptualized although the patient is physically incapable of executing the movements. This brings a quite different set of needs to the forefront. The motor cortex is not static; it demonstrates plasticity after a variety of injuries (3, 7, 12, 15). Cortical representation changes and results in changes in latency, potential,
NEUROSURGERY
FOR A
BRAIN-MACHINE INTERFACE
conduction, recruitment, and so forth (1, 2, 4, 5, 13, 14). Early intervention with conditioning and modification of the unit activity might be essential to BMI success. It would be useful for the authors to repeat this study but instead of actually gripping the device, having the patients simply envision grasping and adjusting the force cognitively rather than physically to determine whether the same type of success can be achieved. An additional problem with classical neuronal conditioning is the need for sensory feedback. Control of individual motor units can be lost during conditioning in the absence of peripheral sensory feedback (18). Nonetheless, we have demonstrated that through a variety of auditory and visual sensory responses given to a patient, lack of sensory feedback responses can be overcome (9, 10). Properly conditioned, even individual pairs of neurons can be trained to co-vary in their performance, and they can be trained separately (6, 17, 19). The demonstration of behavioral conditioning is essential to render feasible the principle of correlating multiple neuronal ensemble output into a behavioral task that can be used in human neuroprosthetics. The authors tend to favor the subcortical structures as a starting point for such a neuroprosthetic BMI. Their main point is that familiarity with the deep brain stimulation system is an excellent starting point. This argument is fallacious. Safety is a far more important concern, cortical approaches are safer, and a small cortical hemorrhage is easily identified at the time of surgery and usually is less serious compared with a subcortical hemorrhage, which may not be identifiable until too late. Furthermore, much of the predictive nature of cortical structures is already known and can be directly approached along the surface of the brain. The only advantage of the subcortical sites is that they provide an area in which a large number of neurons related to different functions are congregated very closely together. Thus, a simple single array could capture neurons involved in multiple functions of the upper and lower extremities. This certainly is worthy of further exploration. This is groundbreaking work. There is still a tremendous amount that needs to be done to bring BMI into the forefront in neurosurgery. I prefer to call this area of research cognitive engineering, as the name encompasses both the ends and the means of the endeavor. Deciphering how this can be accomplished gives us the stimulation, motivation, and indeed the joy to be functional neurosurgeons. Roy A.E. Bakay Chicago, Illinois
1. Brouwer B, Bugaresti J, Ashby P: Changes in corticospinal facilitation of lower limb spinal motor neurons after spinal cord lesions. J Neurol Neurosurg Psychiatry 55:20–24, 1992. 2. Chang CW, Lien IN: Estimate of motor conduction in human spinal cord: Slowed conduction in spinal cord injury. Muscle Nerve 14:990–996, 1991. 3. Cohen LG, Bandinelli S, Findley TW, Hallett M: Motor reorganization after upper limb amputation in man: A study with focal magnetic stimulation. Brain 114:615–627, 1991.
VOLUME 55 | NUMBER 1 | JULY 2004 | 37
22
PATIL
ET AL.
4. Davey NJ, Romaiguere P, Maskill DW, Ellaway PH: Suppression of voluntary motor activity revealed using transcranial magnetic stimulation of the motor cortex in man. J Physiol 477:223–235, 1994. 5. Davey NJ, Smith HC, Wells E, Maskill DW, Savic G, Ellaway PH, Frankel HL: Responses of thenar muscles to transcranial magnetic stimulation of the motor cortex in patients with incomplete spinal cord injury. J Neurol Neurosurg Psychiatry 65:80–87, 1998. 6. Fetz EE, Baker MA: Operantly conditioned patterns on precentral unit activity and correlated responses in adjacent cells and contralateral muscles. J Neurophysiol 36:179–204, 1973. 7. Fuhr P, Cohen LG, Dang N, Findley TW, Haghighi S, Oro J, Hallett M: Physiological analysis of motor reorganization following lower limb amputation. Electroencephalogr Clin Neurophysiol 85:53–60, 1992. 8. Hutchison WD, Allan RJ, Opitz H, Levy R, Dostrovsky JO, Lang AE, Lozano AM: Neurophysiological identification of the subthalamic nucleus in surgery for Parkinson’s disease. Ann Neurol 44:622–628, 1998. 9. Kennedy PR, Bakay RA: Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport 9:1707–1711, 1998. 10. Kennedy PR, Bakay RA, Moore MM, Adams K, Goldwaithe J: Direct control of a computer from the human central nervous system. IEEE Trans Rehabil Eng 9:198–202, 2000. 11. Lenz FA, Dostrovsky JO, Kwan HC, Tasker RR, Yamashiro K, Murphy JT: Methods for microstimulation and recording of single neurons and evoked potentials in the human central nervous system. J Neurosurg 68:630–634, 1988.
12. Levy WJ Jr, Amassian VE, Traad M, Cadwell J: Focal magnetic coil stimulation reveals motor cortical system reorganized in humans after traumatic quadriplegia. Brain Res 510:130–134, 1990. 13. Machida M, Yamada T, Krain L, Toriyama S, Yarita M: Magnetic stimulation: Examination of motor function in patients with cervical spine or cord lesion. J Spinal Disord 4:123–130, 1991. 14. Smith HC, Davey NJ, Savic G, Maskill DW, Ellaway PH, Jamous MA, Frankel HL: Modulation of single motor unit discharges using magnetic stimulation of the motor cortex in incomplete spinal cord injury. J Neurol Neurosurg Psychiatry 68:516–520, 2000. 15. Topka H, Cohen LG, Cole RA, Hallett M: Reorganization of corticospinal pathways following spinal cord injury. Neurology 41:1276–1283, 1991. 16. Vitek JL, Bakay RA, Hashimoto T, Kaneoke Y, Mewes K, Zhang JY, Rye D, Starr P, Baron M, Turner R, DeLong MR: Microelectrode-guided pallidotomy: Technical approach and its application in medically intractable Parkinson’s disease. J Neurosurg 88:1027–1043, 1998. 17. Wyler AR: Interneuronal synchrony in precentral cortex of monkeys during operant conditioning. Exp Neurol 80:697–707, 1983. 18. Wyler AR, Burchiel KJ, Robbins CA: Operant control of precentral neurons: Evidence against open loop control. Brain Res 171:29–39, 1979. 19. Wyler AR, Lange SC, Neafsey EJ, Robbins CA: Operant control of precentral neurons: Control of modal interspike intervals. Brain Res 190:29– 38, 1980.
Patterned primary neurons with cell bodies in red and synapses in green. Biopatterned substrates of glass and gold allow for the controlled growth of neural nets on a microelectrode array chip. (Courtesy of Dr. Christiane Thielemann and the Max Planck Institute for Polymer Research)
23 8620 • The Journal of Neuroscience, June 20, 2012 • 32(25):8620 – 8632
Behavioral/Systems/Cognitive
Subcortical Neuronal Ensembles: An Analysis of Motor Task Association, Tremor, Oscillations, and Synchrony in Human Patients Timothy L. Hanson,1,2* Andrew M. Fuller,2,3* Mikhail A. Lebedev,1,2 Dennis A. Turner,1,5 and Miguel A. L. Nicolelis1,2,3,4,6 1Department of Neurobiology, 2Center for Neuroengineering, 3Department of Biomedical Engineering, and 4Department of Psychology and Neuroscience, Duke University, Durham, North Carolina 27710, 5Department of Neurosurgery, Duke University Medical Center, Durham, North Carolina 27710, and 6Edmond and Lily Safra International Institute of Neuroscience of Natal, 59066-060 Natal, Brazil
Deep brain stimulation (DBS) has expanded as an effective treatment for motor disorders, providing a valuable opportunity for intraoperative recording of the spiking activity of subcortical neurons. The properties of these neurons and their potential utility in neuroprosthetic applications are not completely understood. During DBS surgeries in 25 human patients with either essential tremor or Parkinson’s disease, we acutely recorded the single-unit activity of 274 ventral intermediate/ventral oralis posterior motor thalamus (Vim/Vop) neurons and 123 subthalamic nucleus (STN) neurons. These subcortical neuronal ensembles (up to 23 neurons sampled simultaneously) were recorded while the patients performed a target-tracking motor task using a cursor controlled by a haptic glove. We observed that modulations in firing rate of a substantial number of neurons in both Vim/Vop and STN represented target onset, movement onset/ direction, and hand tremor. Neurons in both areas exhibited rhythmic oscillations and pairwise synchrony. Notably, all tremorassociated neurons exhibited synchrony within the ensemble. The data further indicate that oscillatory (likely pathological) neurons and behaviorally tuned neurons are not distinct but rather form overlapping sets. Whereas previous studies have reported a linear relationship between power spectra of neuronal oscillations and hand tremor, we report a nonlinear relationship suggestive of complex encoding schemes. Even in the presence of this pathological activity, linear models were able to extract motor parameters from ensemble discharges. Based on these findings, we propose that chronic multielectrode recordings from Vim/Vop and STN could prove useful for further studying, monitoring, and even treating motor disorders.
Introduction Neurosurgical implantation of deep brain stimulation (DBS) electrodes is an efficacious treatment for both Parkinson’s disease (PD) and essential tremor (ET) (Ondo et al., 1998; Koller et al., 2001; Kumar et al., 2003; Rodriguez-Oroz et al., 2005; Deuschl et al., 2006). Common DBS targets include the subthalamic nucleus (STN; for PD patients) and the ventral intermediate nucleus of thalamus (Vim; for ET patients). Both structures are involved in motor control (Parent and Hazrati, 1995; Guillery and Sherman, Received Feb. 16, 2012; revised April 20, 2012; accepted April 27, 2012. Author contributions: T.L.H., A.M.F., M.A.L., D.A.T., and M.A.L.N. designed research; T.L.H., A.M.F., and D.A.T. performed research; T.L.H., A.M.F., and M.A.L. analyzed data; T.L.H., A.M.F., M.A.L., D.A.T., and M.A.L.N. wrote the paper. This work was supported by funding from the National Science Foundation Graduate Research Fellowship award DGE-1106401-004 to A.M.F.; the VA Merit Review Award, and NIH Grants R21NS066115 and RO1AG037599 to D.A.T.; and by Defense Advanced Research Projects Agency Grant N66001-06-C-2019, NIH Grant R01NS073125, and the NIH Director’s Pioneer Award DP1OD006798 to M.A.L.N. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Office of the NIH Director or the NIH. We thank Daniel Clayton for surgical assistance and expertise, Susan Halkiotis for manuscript editing and preparation, and Joseph E. O’Doherty for technical, scientific, and programming help. *T.L.H. and A.M.F. contributed equally to this work. The authors declare no financial conflicts of interest. Correspondence should be addressed to Miguel A. L. Nicolelis, 311 Research Drive, Bryan Research Building, Room 327, Duke University, Durham, NC 27710. E-mail: nicoleli@neuro.duke.edu. DOI:10.1523/JNEUROSCI.0750-12.2012 Copyright © 2012 the authors 0270-6474/12/328620-13$15.00/0
2002). The dorsolateral STN receives afferents from motor cortex, premotor cortex, and supplementary motor areas (Parent and Hazrati, 1995; Hamani et al., 2004). Vim projects to these areas as well as receiving afferents from the ipsilateral cerebellum. Individual human Vim/STN neurons are active during voluntary and passive movement, somatosensation, and motor imagery (Lenz et al., 1990, 1994, 2002; Raeva et al., 1999; Magariños-Ascone et al., 2000; Magnin et al., 2000; Rodriguez-Oroz et al., 2001; Abosch et al., 2002; Benazzouz et al., 2002; Theodosopoulos et al., 2003; Williams et al., 2005). Many reports have examined singleunit activity in these regions with respect to tremor (Lenz et al., 1988, 1994, 2002; Zirh et al., 1998; Magariños-Ascone et al., 2000; Magnin et al., 2000; Rodriguez-Oroz et al., 2001; Brodkey et al., 2004; Hua and Lenz, 2005; Amtage et al., 2008) and pathological synchronous oscillations (Levy et al., 2000, 2002; Amirnovin et al., 2004). However, the number of simultaneously recorded cells in these studies was low (ⱕ2), providing limited information about the pathological activity of larger neuronal ensembles. Ensembles of simultaneously recorded neurons have been used to enable brain-machine interfaces (BMIs) for neuroprosthetic control in animal models (Chapin et al., 1999; Wessberg et al., 2000; Nicolelis, 2001; Serruya et al., 2002; Taylor et al., 2002; Hochberg et al., 2006; Fetz, 2007; Patil and Turner, 2008; Nicolelis and Lebedev, 2009; Lebedev et al., 2011; O’Doherty et al., 2011). However, recordings in humans have rarely been ex-
24 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
J. Neurosci., June 20, 2012 • 32(25):8620 – 8632 • 8621
tracted from neuronal ensembles (Kennedy and Bakay, 1998; Kennedy et al., 2000, 2004; Patil et al., 2004; Quiroga et al., 2005). Human subcortical ensembles have only been used in a sole study (Patil et al., 2004), in which our laboratory demonstrated the feasibility of a subcortical motor BMI using motor control signals from neuronal ensembles recorded in motor thalamus [Vim/ ventral oralis posterior motor thalamus (Vop)] or STN during DBS surgery to decode modulations of hand force during a onedimensional target-tracking task. We now examine neuronal population firing patterns during a voluntary motor task in a new sample of 25 human patients. While patients performed contralateral hand movements to acquire visual targets, up to 23 subcortical neurons were recorded simultaneously and acutely in Vim/Vop and STN to elucidate the relationship between neuronal modulations, rhythmic oscillations, and neuronal synchrony. A linear decoder model was applied to reconstruct cursor position from spiking activity. Neurons were classified by oscillatory firing patterns, tremor association, synchrony, and tuning to target and movement parameters.
Materials and Methods Intraoperative recordings were conducted in 25 human patients undergoing placement of therapeutic DBS implants in either Vim or STN. All studies were approved by the Duke University Institutional Review Board and human ethics committees, and all participating patients understood and signed all required consent forms. Patient characteristics and operative plan. All patients selected for this study underwent either Vim (N ⫽ 14, 10 male, 4 female) or STN (N ⫽ 11, 10 male, 1 female) DBS electrode implantation surgery. Patients whose symptom presentation was dominated primarily by medication-resistant tremor (either essential tremor or severe parkinsonian tremor) were candidates for implantation in Vim, while patients with severe PD (typically akinetic/rigid variant with on/off fluctuations and dyskinesia) were candidates for implantation in STN. Both groups of patients were off their medications before and during surgery. Patients first underwent Leksell frame placement, followed by a MRI scan to localize the implantation target. For Vim patients, the target was typically estimated according to anterior–posterior commissure (AC-PC) criteria, located ⬃5– 6 mm in front of the PC, on the AC-PC line with a lateral measure depending on the width of the third ventricle (typically 12–15 mm). Figure 1 A shows a typical electrode trajectory. The first penetration for approaching Vim was the localization pass from a frontal burr hole, with the upper 5 mm of the recording track near the border of Vop and Vim, and the lower 5 mm in Vim and close to ventralis caudalis at the most posterior extent. Typically, the upper 5 mm was the best for multineuron recordings, reflecting more of the anterior motor thalamus (Vop rather than Vim), as shown in Figure 1 A. For STN patients, the target was calculated by indirect methods, based on the AC-PC and 1 mm axial cuts of spoiled gradient recalled acquisition in the steadystate imaging. The initial target was located at 11–12 mm from the midline, 2–3 mm posterior to the midpoint of the AC-PC line, and 4 mm below the AC-PC line (Deuschl et al., 2006). Single-unit recordings were first performed to define the borders of the STN, according to standard electrophysiological criteria, with the goal of attaining at least 5.5– 6 mm of STN. Typically, 2–3 passes were performed. Localization was performed using single-channel tungsten microelectrodes. For both targets, once single-unit recordings had been performed for localization, a 32-channel Pt/Ir microwire (35 m diameter) array (AdTech Medical Instrument) was passed to the appropriate depth via an outer cannula (Patil et al., 2004), where significant activity was noted with the single-unit electrode. After allowing a few minutes for initial recordings to stabilize, the microwire array was slowly advanced through the cannula. Once the number of clearly distinguishable single units was maximized, the microwire array was left in place. At each electrode depth, the patient was instructed to proceed with the voluntary motor task.
Figure 1. A, Sagittal diagram of the human brain at 15 mm off the midline. The vertical line represents the midpoint of the AC-PC line, which is arbitrarily set here as 24 mm in total length. The distance from the midpoint to the PC is 12 mm. The angled line represents a typical electrode trajectory, with the box emphasizing the best location (in terms of cell density) for the placement of the electrode array. This area overlaps Vim/Vop in the motor thalamus. Note that the DBS electrode would typically be placed deeper, at the border of Vim and ventralis caudatus (Vc, the sensory nucleus). The exact angle of the trajectory varied slightly from patient to patient, depending on the frame orientation and location of the frontal burr hole in front of the coronal suture. IC, Internal capsule; SNr, substantia nigra pars reticulata. B, Visualization of all waveforms from 20 sorted units from Patient M (Vim/Vop). Waveforms are 32 samples long and triggered by threshold crossing at sample 8. Individual traces were smoothed and summed into an accumulation buffer. Lighter colors represent a higher density of voltage traces. Traces are labeled by channel followed by unit number. Voltage amplitudes are normalized for display and are therefore represented in arbitrary units. Following completion of the multichannel recording sessions, the microwire array was removed and the DBS treatment electrodes were implanted. As a clinical routine, a brain computed tomography scan was performed within 12 h of the surgery procedure, and in no instance was a hemorrhage or other complication noted. Hence, the clinical risks of temporary placement of the 32 channel microwire array were demonstrated to be very low, as previously reported (Patil et al., 2004). Furthermore, in our research, we have used this electrode in many (N ⫽ 72) patients over several years with no post-op evidence of hemorrhage. Electrophysiological recording. The 32-channel microwire recordings were performed with a Plexon MAP system. Since this study was performed intraoperatively during electrophysiological mapping of the im-
25 8622 • J. Neurosci., June 20, 2012 • 32(25):8620 – 8632
Figure 2. A, Diagram of the bidirectional hand task. The patient uses graded opening/closing of the hand to actuate a one-dimensional cursor toward randomly appearing targets. B, Example off-line prediction of hand/cursor position. Prediction was obtained using a linear Kalman filter with a 500 s training period, from Patient M (Vim/Vop). Forty-eight units (16 single units, 32 multiunits) were included in the ensemble. Position is given in normalized units. plantation sites, the recordings for each patient consisted of one or more epochs (up to 8, mean ⫽ 3.8), between which the electrode depths were altered. For each recording epoch, single units were sorted offline using custom software developed in-house. Extracted spikes consisted of 32 samples each, sampled at 40 kHz, aligned on the crossing of a linear voltage threshold. Sorting was done by projecting these waveforms into a two-dimensional principal component space. Clusters in principal component space were determined visually and selected by use of a lasso tool to define spatial boundaries. If the electrodes were moved (clearly visible as the microdrive corrupted the recording traces) or threshold was otherwise changed during a recording epoch, the record was broken into separate epochs. Recorded neuronal discharges on a given channel that could not be sorted and isolated as single units were classified as belonging to a multiunit, indicating a collection of individual discharges whose identity could not be firmly established. Figure 1 B shows the sorted unit waveforms from a single recording session. We estimated the signal-tonoise ratio (SNR) by dividing the variance of the extracted spike samples by an estimate of the noise variance (Bankman et al., 1993), yielding a mean SNR of 4.69 for all sorted units. Voluntary motor task. Patients were placed in a supine, semisitting position in front of a computer monitor. A 5DT Systems Data Glove 5 Ultra haptic glove was placed over the hand contralateral to the microelectrode array. This glove was used to measure flexion/extension of the fingers, sampled at 1 kHz. The average flexion/extension signal, which effectively measured opening/closing of the hand, was used to control the one-dimensional position of the cursor on a video screen, placed directly in front of the patient for high visibility. Patients were trained to modulate the opening and closing of their hand to acquire targets by moving the cursor into a box placed randomly along a horizontal line (Fig. 2 A). The required target hold time was 200 ms. Once the target was acquired, the box disappeared for 300 ms before reappearing in a new random position, chosen from a uniform distribution representing the horizontal extent of the screen. Therefore, movements did not strictly alternate between left and right; successive jumps would frequently occur in the same direction. During the preliminary training/calibration phase, the cursor gain, offset, and target box size were systematically calibrated by the experimenters to compensate for variations in physical ability. Specifically, in patients exhibiting limited hand mobility, the gain from hand movement
Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
to cursor motion was increased. In patients with pronounced hand tremor, the size of the target was increased. Finally, the offset was set so that the resting position of the patient’s hand corresponded to the middle of the screen. During the recordings that followed, the length of individual motor task sessions varied depending on electrophysiological recording quality and the level of patient fatigue. Figure 2B shows a representative snapshot of the motor task performed by an ET patient. Note the slight 5 Hz tremor that occurred during target hold periods. Neuronal tuning to target and movement. All subsequent data analyses were performed using Matlab (MathWorks). We use the term “tuning” to refer to modulation of neuronal firing rates that is correlated to an external parameter and “tuning strength” to refer to the extent of those modulations. For all sorted single units and multiunits, perievent time histograms (PETHs) of neuronal activity (Awiszus, 1997) were generated using one of two event triggers: (1) the appearance of a new target or (2) movement time. PETHs triggered on target appearance were constructed using a window beginning 0.5 s before each event trigger and ending 1.5 s after, whereas a symmetric 2 s window was used for PETHs triggered on movement time. Movement time was defined as the moment at which the cursor crossed the midpoint between initial cursor position (at target appearance) and the endpoint target position. We chose this standard as a robust definition of movement execution in light of patient tremor and occasional incorrect movements. Regardless of reaction time, this event trigger was locked to movement, being in close proximity to the point of maximum hand velocity before target acquisition. Trials with anomalous movement times ⬍200 ms (premature movement) or ⬎1000 ms (inattention) were discarded. Only neurons with at least 50 valid target acquisition trials were chosen for further analysis. Neuronal tuning to either target or movement was determined by quantifying the deviation of each PETH from a bootstrap distribution of PETHs generated by uncorrelated triggers. We calculated significance using the one-sample Kuiper’s test (Kuiper, 1962; Batschelet, 1981; Zar, 1999), a nonparametric test related to the Kolmogorov–Smirnov (K-S) test (Zar, 1999) but better suited for nonbiased PETH analysis. Unlike the K-S test, Kuiper’s test is equally sensitive throughout the distribution, a useful property in scenarios in which the locations of the peak modulations are not known a priori. Variations of the K-S test have been used previously in the significance evaluation of neuronal PETHs (Ghazanfar et al., 2001; Wiest et al., 2005; Gutierrez et al., 2006). In this study, we used Kuiper’s test to distinguish an observed distribution of eventtriggered spike times from the null hypothesis (uniform probability distribution). Kuiper’s test requires the calculation of the maximum positive and negative deviations of the observed PETH cumulative distribution function (CDF) from a uniform distribution CDF (ramp function); the sum of these two deviations is the statistic V: V ⫽ max[CDFsample ⫺ CDFuniform] ⫹ max[CDFuniform ⫺ CDFsample]. The Kuiper statistic, K, is a normalized version of V, taking into account the size of the observed sample size N, in this case, the number of binned spikes: (K ⫽ VN1/ 2 ⫹ 0.155 ⫹ 0.24N⫺1/ 2 ). To distinguish the test statistic Kobs from the null hypothesis, we generated a bootstrapped distribution of 1000 simulated Kuiper statistics (Ksim). Preliminary analysis determined shuffling of spike timestamps to be a suboptimal control; the process eliminates spike autocorrelations from the bootstrap distribution, thereby potentially biasing the evaluation of the observed distribution in favor of significance. Instead, each value of Ksim was calculated using a PETH constructed from the original spike timestamps but processed using a distribution of randomized event triggers; the triggers were drawn uniformly from the time span of the recording session. In other words, in each bootstrapped trial, the timestamps of the events (target appearance, movement) are randomly and independently assigned to decorrelate them from the spiking data. For each sorted unit, the resulting bootstrapped distribution of Ksim was used to n produce a p value: p ⫽ (number of trials for which Ksim ⬎ Kobs)/(N ⫹ 1). Units were deemed to be tuned to task events (target or movement onset) using the threshold p ⬍ 0.05. These units exhibited temporal modulations in firing rate relative to newly appearing targets and/or target-directed movements. Tuning strength was defined as the z-score of the observed PETH relative to the bootstrap distribution.
26 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
Directional tuning. The directional tuning of each sorted unit or multiunit was defined as the difference in neuronal response for leftward versus rightward movements. Significance of directional tuning was determined using the two-sample Kuiper’s test. This test applied the calculation of the maximum positive and negative deviations between the spike CDFs for leftward and rightward movements: V ⫽ max[CDFsample 1 ⫺ CDFsample 2] ⫹ max[CDFsample 2 ⫺ CDFsample 1]. The Kuiper statistic K was calculated from V using the same equation as the one-sample Kuiper’s test, but in the case of the two-sample Kuiper’s test, the effective sample size (Neff) replaced N to account for the combined contributions of the individual sample sizes N1 and N2: Neff ⫽ (N1N2)/(N1 ⫹ N2). Two PETHs triggered on movement time were generated, one for leftward movements and one for rightward movements. As with the one-sample Kuiper’s test, each unit’s Kobs statistic was compared with a bootstrapped distribution of Ksim generated from randomized trigger times. Units were deemed to be directionally tuned using the threshold p ⬍ 0.05. Tuning strength was defined as the z-score of the observed PETHs relative to the bootstrap distribution. Tremor sensitivity. Although concurrent surface EMG recordings were not permitted under our approved experimental protocol, haptic/position tracking have been used repeatedly to analyze tremor (Bardorfer et al., 2001; Su et al., 2003; Vinjamuri et al., 2009), as has accelerometry (Ghika et al., 1993; Grimaldi et al., 2007; Birdno et al., 2008). For all sorted single units, perievent phase histograms (PEPHs) of neuronal activity were generated using phase of the patient’s tremor as an event trigger, similar to the approach of Lebedev et al. (1994). Tremor was determined from hand velocity, and tremor periods within the range of 100 –2000 ms (0.5–10 Hz) were analyzed. To exclude the impact of voluntary movements, hand velocity peaks occurring within 250 ms of a movement trigger were excluded. Each tremor period was defined in units of phase, with neuronal spike activity captured into 100 bins of equivalent phase aperture (3.6° each). The zero phase for each cycle was defined by a local maximum in hand velocity. Only sorted units with at least 500 valid tremor periods were chosen for further analysis. Each resulting PEPH was a measurement of neuronal firing rate with respect to tremor phase. The one-sample Kuiper’s test, in addition to possessing uniform sensitivity, is also rotationally invariant, meaning that the arbitrary choice of zero phase has no effect on the assessment of statistical significance. For analysis of tremor tuning, we generated a bootstrapped distribution of 1000 simulated Kuiper statistics (Ksim); each was calculated using a PEPH constructed from trials whose binned spike counts were circularly rotated by uniformly random phase offsets. For each analyzed unit, the bootstrapped distribution was used to produce a p value. Units were deemed to be tremor associated (tuned) using the threshold p ⬍ 0.05. Tuning strength was defined as the z-score of the observed PEPH relative to the bootstrap distribution. Oscillatory neurons. For all sorted units with at least 1000 extracted spikes, we used Welch’s method (Oppenheim and Schafer, 1975) with eight nonoverlapping segments to determine the spike train autopower spectral density. The power spectra were smoothed using a 0.5 Hz rectangular sliding window. For each unit, the peak autopower frequency was determined in the 1–25 Hz range, with frequency content ⬍1 Hz discarded for the remainder of the analysis. For the peak frequency, we determined the SNR by dividing peak power by the mean power (assessed from 1 Hz up to the Nyquist frequency of 500 Hz). For the purpose of comparison, the same spectral analysis was performed on hand acceleration traces for all recorded sessions. Preliminary analysis indicated a functional separation of peak frequencies at ⬃2.5 Hz. Units with a peak power frequency ⬍2.5 Hz tended to be dominated by low-frequency power and were therefore judged not to be sufficiently oscillatory in a physiologically relevant frequency range. As in previous studies (Lenz et al., 1988; Amtage et al., 2008), only units with a peak SNR ⬎2 were classified as oscillatory. Sharpness of the peaks in either the spike train or hand acceleration autopower spectra was determined by calculating the maximum power concentrated in a 1 Hz band within the physiologically relevant 2.5–7.5 Hz window. “Peakedness” was defined as the ratio of the power in this band relative to the total power in the 1–25 Hz band.
J. Neurosci., June 20, 2012 • 32(25):8620 – 8632 • 8623
In addition to established linear methods, we developed a heterodyne method for detecting nonstationary or wide-bandwidth synchronized activity between each neuron’s firing rate and associated hand velocity. This method, inspired by the frequency shifting scheme used in radio frequency transceivers, was applied to all sorted units that fulfilled the selection criteria for both target tuning analysis and oscillatory analysis. The spike train and hand velocity recordings were first bandpass filtered (fourth-order Butterworth, zero phase method) between 2 and 12 Hz, leaving both signals with negligible 0 Hz (DC) energy. The two signals were then multiplied, yielding a third time series from which to extract spectral energy. Because multiplication in the time domain is equivalent to convolution in the frequency domain (and vice versa), any synchronous frequencymodulated components in both neuronal firing rate and hand velocity are transferred to DC. The ratio of spectral energy (Eobs) from 0 to 0.125 Hz (signal) over that from 0.25–2 Hz (baseline) was considered as a metric of heterodyne correlation between neuronal activity and hand movement. To provide a control for this estimate, a bootstrap distribution (Esim) was generated by shuffling spike timestamps 1000 times and repeating the above analysis. Since all low-frequency information is filtered out before multiplication, shuffling timestamps was determined to be a suitable control. For all single units, the resulting bootstrapped distribution was used to produce a p value. Units were deemed to be heterodyne-tuned to tremor if the heterodyne correlation between neuronal activity and movement was significant using the threshold p ⬍ 0.05. Heterodyne tremor tuning strength was defined as the z-score of the observed spectra relative to the bootstrap distribution. Efficacy of neuronal recordings for kinematic predictions. Prediction algorithms from the BMI literature were applied to subcortical neuronal populations to extract behavioral parameters. Several algorithms were tested, including the linear Kalman filter, unscented Kalman filter, and the Wiener filter. Figure 2 B shows an example off-line prediction for a 30 s window of task performance. Since all three algorithms achieved the same approximate fidelity in preliminary testing, we chose the Wiener filter for further analysis due to its computational simplicity and extensive presence in the BMI literature (Wessberg et al., 2000; Carmena et al., 2003; Patil et al., 2004). Individual Wiener filters were fit by binning neuronal data into 100 ms time slices with 10 causal lags and regressing against recorded hand position. Model training was performed by the random selection of 50% of these time slices; predictions were then made on a distinct random 25%. This process was repeated with 100 draws of fit and predict time slices. Correlation coefficient ( R) between predicted hand position and actual hand position was measured for each of the draws; the mean correlation coefficient was reported for each recording session. Offline prediction results are reported for all sessions with at least 50 presented targets. We generated neuron dropping curves for selected sessions (Wessberg et al., 2000) by drawing random subsets from the neuronal ensemble. For each subset ensemble size N, we performed 1000 draws of random ensemble subset and Wiener filter fit and prediction; the R values for these draws were averaged to form a smooth neuron dropping curve. Following Wessberg et al. (2000), the resulting curve was then fit to the following hyperbolic function to extrapolate the performance results to larger ensemble sizes: R 2 ⫽ cN/(1 ⫹ cN ). Neuronal synchrony. The use of simultaneous ensemble recordings allows for the analysis of pairwise synchrony between neurons. To determine the statistical significance of the synchrony between two neurons, we analyzed the cross-correlation peak between pairs of spike trains. Pairs were analyzed if they each contained at least 100 spikes and corresponded to a session with at least 50 targets. The cross-correlation coefficient was first calculated for the observed spike trains of the two neurons, then smoothed using a 5 ms rectangular sliding window. The observed test statistic Cobs was defined as the peak coefficient in the ⫾10 ms time lag range. Bootstrap simulations (n ⫽ 1000) of the two spike trains were generated by convolving the spike trains with a Gaussian kernel ( ⫽ 250 ms) and then generating new spike trains via an inhomogeneous Poisson process. The smoothing filter was used to extinguish correlated high-frequency content in the bootstrap distribution while maintaining low-frequency correlation in mean firing rate. Each of these bootstrap simulations was used to produce a cross-correlation coefficient
27 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
8624 • J. Neurosci., June 20, 2012 • 32(25):8620 – 8632
Table 1. Pairwise classifications for single units Vim/Vop cells Target Movement Direction Tremor Oscillatory Heterodyne STN cells Target Movement Direction Tremor Oscillatory Heterodyne
Target
Movement
Direction
Tremor
Oscillatory
Heterodyne
29.2% (N ⫽ 168) — — — — —
32.0%(⫹)** (N ⫽ 75) 34.7% (N ⫽ 75) — — — —
16.0% (N ⫽ 75) 14.7%(⫹)† (N ⫽ 75) 25.3% (N ⫽ 75) — — —
9.5%(⫹)† (N ⫽ 126) 13.8% (N ⫽ 65) 12.3%(⫹)† (N ⫽ 65) 12.4% (N ⫽ 169) — —
5.1% (N ⫽ 158) 1.4%(⫺)* (N ⫽ 74) 4.1% (N ⫽ 74) 2.4% (N ⫽ 169) 21.5% (N ⫽ 274) —
3.8% (N ⫽ 158) 4.1% (N ⫽ 74) 8.1%(⫹)† (N ⫽ 74) 6.3%(⫹)* (N ⫽ 126) 3.8% (N ⫽ 158) 13.3% (N ⫽ 158)
22.9% (N ⫽ 83) — — — — —
38.5%(⫹)** (N ⫽ 26) 42.3% (N ⫽ 26) — — — —
11.5% (N ⫽ 26) 11.5% (N ⫽ 26) 19.2% (N ⫽ 26) — — —
4.1% (N ⫽ 74) 12.5% (N ⫽ 24) 0.0% (N ⫽ 24) 15.9% (N ⫽ 82) — —
4.9% (N ⫽ 81) 8.3% (N ⫽ 24) 4.2% (N ⫽ 24) 1.2% (N ⫽ 82) 17.9% (N ⫽ 123) —
2.5% (N ⫽ 81) 8.3% (N ⫽ 24) 0.0% (N ⫽ 24) 4.1% (N ⫽ 74) 6.2% (N ⫽ 81) 17.3% (N ⫽ 81)
N, Number of analyzed cells per entry. Off-diagonal table entries have a smaller number of analyzed cells because they must fulfill the criteria of two independent classification tests. (⫹, ⫺), Significantly (high, low) level of pairwise classification (Two-tailed Fisher’s exact test: †p ⬍ 0.05, *p ⬍ 0.01, **p ⬍ 0.001).
Table 2. Behavioral tuning of subcortical neurons Csim. The bootstrapped distribution was used to produce a p value, and neuron pairs were deemed to be significantly synchronous using the threshold p ⬍ 0.05. To visualize the time dependence of pairwise synchrony, we generated joint peristimulus time histograms (JPSTHs) for neuron pairs, as originally proposed by Aertsen et al. (1989). Our JPSTHs were adjusted by subtracting the shift predictor histogram and normalizing (bin-by-bin) by the standard deviation, a procedure referred to by Aertsen et al. (1989) as the “true normalization” of the JPSTH.
Results A total of 25 DBS implantation patients were examined. In these patients, we simultaneously recorded from ensembles of up to 23 well isolated neurons from either Vim/Vop or STN, depending on the site of electrode location. Recording sessions varied substantially in terms of duration and target acquisition rate, as limited by individual patient pathology and motivation. Neurons from these subcortical areas were classified by oscillatory firing patterns and tuning to target, movement, direction, and tremor. Moreover, neuronal ensemble data served as input for an offline linear prediction model to reconstruct cursor position. Finally, neuronal pairs were analyzed for evidence of functional synchrony. STN cells (N ⫽ 168) exhibited a higher ( p ⬍ 0.01, Mann– Whitney U test) mean firing rate than Vim/Vop cells (N ⫽ 83): 15.8 ⫾ 1.95 Hz and 11.7 ⫾ 1.02 Hz, respectively (mean ⫾ 1 SE in both cases). In both subcortical areas, we found substantial populations of oscillatory neurons, as well as neurons strongly tuned to target, movement, direction, and tremor. Furthermore, neurons in both subcortical areas tended to show tuning to multiple parameters (Table 1) rather than belonging to disjoint sets. For example, the number of Vim/Vop cells tuned to both target and tremor was higher than would be expected under statistical independence (two-tailed Fisher’s exact test, p ⬍ 0.05). At the ensemble level, a substantial number of analyzed cell pairs were found to exhibit synchrony. Curiously, all tremor-associated neurons exhibited synchrony within the recorded neuronal ensemble. Neuronal tuning to target and movement Both Vim/Vop and STN neurons represented target appearance and movement onset (Table 2). Of all single units tested, 29.2% of 168 Vim/Vop cells and 22.9% of 83 STN cells were found to be tuned to target appearance. Both of these percentages represent significant populations (binomial test, p ⬍⬍ 0.001 in both cases). Figure 3A shows example PETHs for three highly responsive neurons.
Parameter
Unit type
Area
No. of units
# Tuned
Target Target Target Movement Movement Movement Direction Direction Direction
Single Single Multi Single Single Multi Single Single Multi
Vim/Vop STN Both Vim/Vop STN Both Vim/Vop STN Both
168 83 753 75 26 414 75 26 414
49** (29.2%) 19** (22.9%) 87** (11.6%) 26** (34.7%) 11** (42.3%) 48** (11.6%) 19** (25.3%) 5* (19.2%) 36* (8.7%)
Tuned population tested for significance (Binomial test: †p ⬍ 0.05, *p ⬍ 0.01, **p ⬍ 0.001).
As explained above, trials with reaction times outside the 200 – 1000 ms range were discarded for movement tuning, and only sessions with at least 50 valid trials were subjected to further statistical analysis. Because of the additional reaction-time criterion, fewer single units were analyzed for movement tuning than for target tuning. Of these, 34.7% of 75 Vim/Vop cells and 42.3% of 26 STN cells were found to be tuned to movement. Both of these percentages represent statistically significant populations (binomial test, p ⬍⬍ 0.001 in both cases). Figure 3B shows example PETHs for three highly responsive neurons. We also found a strong positive correlation between the strength of target appearance tuning and that of movement tuning, for both Vim/Vop and STN cells. When controlling for the number of session trials, target tuning strength significantly predicted movement tuning strength ( ⫽ 0.70, p ⬍⬍ 0.001 for Vim/Vop;  ⫽ 0.71, p ⬍⬍ 0.001 for STN). This result is consistent with Table 1, which indicates that a larger-than-expected number of neurons in both subcortical areas were tuned to both target and movement. A portion of the correlation between target tuning and movement tuning may be explained by a tight temporal offset between target appearance and movement time. However, visual inspection of some tuned units indicated a clear decoupling of the neural encoding of target appearance and movement. Sorted raster plots from example neurons are shown in Figure 4, A and C. From these, we derived the color maps in Figure 4, B and D, each showing two clear bands of increased spike density. In both panels, the vertical bands are independent of movement time and are clearly related to target appearance (⬃450 ms postappearance). The diagonal bands have a near-unity slope, indicating a clear time-locked relationship between neuronal activity and movement time. For both units, the second peak in firing rate occurred
28 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
J. Neurosci., June 20, 2012 • 32(25):8620 – 8632 • 8625
Figure 4. Separation between single-unit response to target appearance and movement time. Time along the x-axis is relative to target appearance. A and C show spike raster plots relative to target appearance, with individual trials sorted by movement time (red circles). B and D show smoothed color plots for the same two units as A and C, using the relative timing of all spikes and the movement time of their corresponding trials. The dashed line (unity slope) depicts movement time. For generation of the color plots, the data were smoothed using a two-dimensional Gaussian kernel, with ⫽ 20 ms along the x-axis and ⫽ 40 ms along the y-axis. Both units are Vim/Vop cells from Patient M.
Figure 5. A, B, Mean normalized PETHs for all responsive units ( p ⬍ 0.05) from both subcortical areas using two event triggers: target appearance (A), and movement time (B). Reported time is relative to the event trigger. Before aggregation, individual PETHs were convolved with a Gaussian kernel ( ⫽ 40 ms) and normalized relative to mean firing rate. For both panels, the differences between the mean Vim/Vop and STN responses were statistically significant ( 2 test, p ⬍⬍ 0.001). Figure 3. Example PETHs. A, Strongly tuned units to target appearance. i, Vim/Vop cell, Patient M, 465 trials. ii, Vim/Vop cell, Patient M, 375 trials. iii, STN cell, Patient H, 310 trials. B, Strongly tuned units to movement time. i, Vim/Vop cell, Patient M, 390 trials. ii, Vim/Vop cell, Patient M, 310 trials. iii, STN cell, Patient H, 201 trials. C, Strongly direction tuned units. i, STN cell, Patient H, 201 trials. ii, Vim/Vop cell, Patient M, 416 trials. iii, Vim/Vop cell, Patient M, 394 trials. For A, reported time is relative to target appearance. For B and C, reported time is relative to movement time. All plots were smoothed using a Gaussian kernel, ⫽ 40 ms.
⬃300 ms after the defined movement time. From these data, it can be concluded that these neurons were tuned to both target appearance and movement; they modulated their firing rates in relation to both events. Modest differences were seen in the aggregate response patterns of Vim/Vop and STN cells classified as responsive to either target or movement (Fig. 5). Both cell types exhibited a mean response that peaked following target appearance (Fig. 5A); STN cells peaked later
on average. The mean response of the Vim/Vop cells peaked immediately before movement while that of the STN cells peaked concurrently with movement (Fig. 5B). Differences in the relative lags for Vim/Vop and STN neuronal activation likely reflect the position of thalamic and STN neurons in the network hierarchy of motor control (Marsden et al., 2001; Guillery and Sherman, 2002; Gradinaru et al., 2009). The motor regions of the thalamus are more involved with intention, with signals arriving before motor cortex activation, whereas the collaterals from motor cortex to STN deliver signals at the time of motor activation. For both aggregates, the differences between the mean Vim/Vop and STN responses were statistically significant ( 2 test, p ⬍⬍ 0.001). However, similar proportions of Vim/Vop and STN cells were tuned to target appearance; the same was also true for movement tuning (two-tailed Fisher’s exact test, p ⬎ 0.05 in both cases).
29 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
8626 • J. Neurosci., June 20, 2012 • 32(25):8620 – 8632
Table 3. Tremor tuning of subcortical neurons Area
Unit type
No. of units
No. of tuned
Vim/Vop STN
Single Single
169 82
21** (12.4%) 13** (15.9%)
Tuned population tested for significance (Binomial test: †p ⬍ 0.05, *p ⬍ 0.01, **p ⬍ 0.001).
Directional tuning Another metric of interest for the behavioral responsiveness of subcortical neurons was directional tuning; Table 2 gives the directional tuning results for both single units and multiunits. Of all tested single units, 25.3% of 75 Vim/Vop cells and 19.2% of 26 STN cells were found to exhibit directional tuning. Both of these percentages represent statistically significant populations (binomial test, p ⬍ 0.001 for Vim/Vop, p ⬍ 0.01 for STN). Figure 3C shows example PETHs for three strongly tuned neurons. Similar proportions of Vim/Vop and STN cells were tuned to direction (two-tailed Fisher’s exact test, p ⬎ 0.05). Despite the clear separation in the neuronal response to leftward and rightward movements, note the transient regions of convergence that occurred in Figure 3C. In Figure 3Ciii, for example, the neuronal responses to each direction converged just before movement. For many tuned neurons in both Vim/Vop and STN, the degree of directional modulation varied throughout the temporal window. For both Vim/Vop and STN cells, we found strong positive correlations between directional tuning strength and the strength of both target tuning and movement tuning. When controlling for the number of session trials, target tuning strength significantly predicted directional tuning strength ( ⫽ 0.16, p ⬍ 0.05 for Vim/Vop;  ⫽ 0.38, p ⬍ 0.01 for STN). Similarly, movement tuning strength significantly predicted directional tuning strength ( ⫽ 0.24, p ⬍ 0.01 for Vim/Vop;  ⫽ 0.43, p ⬍ 0.05 for STN). The latter finding is consistent with the Vim/Vop pairwise classification result in Table 1. Properties of multiunits Whereas single units are identifiable as distinct neurons, a multiunit is likely comprised of distant neurons with lower SNR spike profiles. From Table 2, it can be seen that a significant population of analyzed multiunits were tuned to target, movement, and direction (binomial test, p ⬍ 0.01 in all cases). A substantial number of these tuned multiunits were found on the same recorded channel as tuned sorted units. Furthermore, when controlling for the number of session trials, the target tuning strength of single units significantly predicted the target tuning strength of samechannel multiunits ( ⫽ 0.18, p ⬍ 0.01). This confirms the presence of correlated tuning in nearby neurons. Thus, a substantial amount of encoded information was present in subcortical multiunits, arguing for the potential inclusion of these signals in future analyses of ensemble activity. Tremor sensitivity To identify potentially pathological neurons within the recorded subcortical populations, we analyzed the tremor sensitivity of single units using the discussed PEPH approach; the results are given in Table 3. Of all single units tested, 12.4% of 169 Vim/Vop cells and 15.9% of 82 STN cells were found to be correlated to observable hand tremor. Both of these percentages represent statistically significant populations (binomial test, p ⬍ 0.001 in both cases). Figure 6 shows example PEPHs for three strongly tremorsensitive neurons. These results demonstrate that for highly tuned units, the dependence of spike rate on tremor phase re-
Figure 6. Example PEPHs triggered on hand tremor phase for strongly tremor tuned units. A, Vim/Vop cell, Patient M, 4478 tremor periods. B, Vim/Vop cell, Patient M, 1259 tremor periods. C, Vim/Vop cell, Patient M, 4642 tremor periods. Zero phase is aligned to local peaks in hand velocity. All panels show individual PEPHs for each temporal quarter of their respective sessions, overlaid with the aggregate PEPH for the session as a whole (ALL). All plots in this figure are smoothed using a Gaussian kernel, ⫽ 15°.
mained stable throughout the recording session (Fig. 6), even if the mean firing rate varied substantially Figure 6, A and C. Similar proportions of Vim/Vop and STN cells were tuned to tremor (two-tailed Fisher’s exact test, p ⬎ 0.05). For Vim/Vop cells (but not STN cells), we found a positive correlation between the strength of tremor tuning and that of directional tuning. When controlling for the number of session trials, directional tuning strength significantly predicted tremor tuning ( ⫽ 0.34, p ⬍ 0.05). However, we found no relationship between tremor tuning and either undirected target or movement tuning ( p ⬎ 0.05 for all cases). However, these results do not distinguish whether these tremor-tuned neurons are involved in a pathological mechanism that causes tremor or merely reflect somatosensory signals indicative of tremor. Oscillatory behavior To explore neuronal oscillations in Vim/Vop and STN and their relationship to patient pathology, we inspected the autopower spectra of single-unit spike trains for strong frequency peaks, yielding peak frequency and SNR (Fig. 7A). Of all tested single units with SNR ⬎ 2, the distribution of peak frequencies showed a clear bimodal distribution with a border between low- and high-frequency oscillations at ⬃2.5 Hz (Fig. 7B). Spike train spectra from Vim/Vop and STN neurons also tended to possess large amounts of energy at low frequencies, suggestive of 1/f (pink) noise. This power-law distribution has been described for cortical neurons as a stochastic process (Davidsen and Schuster, 2002), but may also be related to slow modulations of patient attention and arousal. The observed distribution may explain the 2.5 Hz trough (Fig. 7) that separates neurons dominated by 1/f noise from those exhibiting strong oscillations in the tremorrelevant frequency range (2.5–7.5 Hz). Only neurons with sufficient power within this frequency range were eligible to be classified as oscillatory. The oscillatory classification results are shown in Table 4; 21.5% of 274 Vim/Vop cells and 17.9% of 123 STN cells were classified as oscillatory. No difference was seen in the proportions of oscillatory Vim/Vop and STN cells (two-tailed Fisher’s exact test, p ⬎ 0.05). Figure 8 shows example interspike interval (ISI) plots for three highly oscillatory cells. Note that all three ISI histograms exhibit some degree of bimodality, indicative of periodic bursting behavior. Figure 9A shows the smoothed autopower spectra of spike trains for all analyzed single units, with each individually normal-
30 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
J. Neurosci., June 20, 2012 • 32(25):8620 – 8632 • 8627
Figure 7. A, Distribution of peak frequencies for the spike train autopower spectra of 397 analyzed single units. Units with peak frequency ⬎10 Hz (17.4% of all units) are not shown in this figure; they are distributed with near uniformity in the 10 –25 Hz range with SNR mostly below the classification threshold of 2. B, Histogram showing the number of oscillatory (SNR ⬎ 2) units at each peak frequency. A clear separation between two subpopulations is clear at ⬃2.5 Hz. Table 4. Oscillatory Units Area
Unit type
No. of units
No.of oscillatory
Vim/Vop STN
Single Single
274 123
59 (21.5%) 22 (17.9%)
Figure 8. Example ISI histograms for highly oscillatory neurons. A, Vim/Vop cell, Patient W. B, STN cell, Patient V. C, Vim/Vop cell, Patient J.
ized horizontal trace corresponding to a distinct unit. From this figure, one can visually identify some of the highly oscillatory units as well as observe the congruity between multiple units from the same patient. The difference between the mean normalized spectra for Vim/Vop and STN cells (Fig. 9B) is statistically
Figure 9. A, Smoothed autopower of spike trains for all sorted units with sufficient spike count. The autopower spectra (determined by Welch’s method) of each horizontal trace, corresponding to a distinct unit, has been individually smoothed (using a 0.5 Hz sliding window) and normalized. The horizontal traces are grouped by patient and recording area. The color bar represents arbitrary units of normalized energy density. B, Mean autopower spectra from all analyzed Vim/Vop and STN cells. Spectra are individually smoothed using a 0.5 Hz sliding window and normalized before aggregation. Each of the dashed traces represents a bootstrap simulation in which only half of all analyzed units are aggregated. The difference between the mean Vim/Vop and STN spectra is statistically significant ( 2 test, p ⬍⬍ 0.001).
significant ( 2 test, p ⬍⬍ 0.001). From Figure 9B, it is clear that STN cells tended to concentrate power at a lower frequency (3 rather than 4 Hz). The pairwise classification results in Table 1 reject the notion that oscillatory neurons and behaviorally tuned neurons form disjoint sets. Furthermore, we found no relationship between spike autopower peakedness and the strength of any of the three (target, movement, direction) behavioral tuning metrics ( p ⬎ 0.05 for all cases, for both Vim/Vop and STN). The lack of a clear anticorrelation suggests that the sets of behavioral neurons and oscillatory neurons are far from disjointed. Instead, they appear to exist as overlapping populations. Our next analysis intended to uncover a relationship between strong oscillatory neuronal patterns and observable hand tremor. However, we did not find any clear relationship. Linear regression analysis revealed no relationship between peak frequency (2.5–7.5 Hz range) of spike train autopower spectra and corresponding hand acceleration autopower spectra ( p ⬎ 0.05 for both Vim/Vop and STN). No relationship was found between the sharpness of the two spectra for Vim/Vop neurons ( p ⬎ 0.05), but we did observe a marginally significant positive relationship for STN neurons ( ⫽ 0.72, p ⫽ 0.038). Figure 10 shows overlaid spectra for the spike train autopower and hand acceleration autopower of three highly oscillatory units. For all three cells (representative of the population as a whole), the peak frequencies do not coincide. However, we did find a marginally significant ( ⫽ 0.24, p ⫽ 0.055) correlation between spike autopower peakedness and tremor tuning strength for Vim/Vop cells ( p ⬎ 0.1 for STN cells). These findings call into question the presumed causal linear relationship between the two, suggesting the possibility of an elusive nonlinear relationship.
31 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
8628 • J. Neurosci., June 20, 2012 • 32(25):8620 – 8632
Figure 10. Comparison of spike train autopower and hand acceleration autopower for three highly oscillatory units. A, Vim/Vop cell, Patient W. B, STN cell, Patient V. C, Vim/Vop cell, Patient J. Note that for all three cells, peak frequency does not coincide. Table 5. Heterodyne tremor tuning Area
Unit type
No. of units
No. of tuned
Vim/Vop STN
Single Single
158 81
21** (13.3%) 14** (17.3%)
Tuned population tested for significance (Binomial test: †p ⬍ 0.05, *p ⬍ 0.01, **p ⬍ 0.001).
Our heterodyne decoding analysis further explored this relationship by applying a nonlinear frequency shifting approach to the autopower spectra. Of 239 analyzed single units, 13.3% of Vim/Vop cells and 17.3% of STN cells were found to be tremor associated via heterodyne decoding (Table 5). Both of these percentages represent statistically significant populations (binomial test, p ⬍⬍ 0.001), but the difference between them is not significant (two-tailed Fisher’s exact test, p ⬎ 0.05). Heterodyne decoding is a more sensitive method for detecting tremor correlations than spectral peak analysis if the tremor signal is wide-bandwidth or prone to phase changes. Indeed, this schema may better serve to explain the relationship between the oscillatory activity of neurons and observed tremor. In fact, this can explain the similarity in the proportions of tuned neurons in Tables 3 and 5. Furthermore, Vim/Vop firing indicated a strong positive correlation between tremor tuning strength, identified using PEPHs, and heterodyne tremor tuning strength ( ⫽ 0.31, p ⬍ 0.001). This relationship was marginally significant in STN cells ( ⫽ 0.28, p ⫽ 0.086). These findings are consistent with the pairwise classification results in Table 1, which indicated a higher than expected joint classification for the two tremor tuning analyses for Vim/Vop cells. Efficacy of neuronal recordings for kinematic predictions We also performed off-line predictions of cursor motion using the recorded ensembles. The correlation coefficient (mean ⫾ 1.98 SE) for each of the sessions is shown in Figure 11. Although the predictions varied greatly across sessions and patients, the results compared favorably with our previous study (Patil et al., 2004). The best session for each subcortical area (Vim/Vop, STN) was chosen for further analysis, and neuron dropping curves were generated for these two sessions and fitted to a hyperbolic function (Wessberg et al., 2000). Extrapolation of the hyperbolic fit produced estimates of the approximate ensemble sizes required to achieve R 2 ⫽ 0.9: 106 Vim/Vop neurons or 397 STN neurons.
Figure 11. Dependence of off-line BMI predictions on neuron ensemble size. Each data point corresponds to a recording session. Correlation coefficient ( R) is indicated as mean ⫾ 1.98 SE. The best session for each subcortical area (Vim/Vop, STN) was chosen for further analysis. Neuron dropping curves are shown for each of these sessions and fitted to a hyperbolic function (Wessberg et al., 2000). Extrapolation calculations are presented in Results. Table 6. Pairwise neuronal synchrony Area
No. of pairs
No. of synchronous
Vim/Vop STN
1648 693
708** (43.0%) 179** (25.8%)
Tuned population tested for significance (Binomial test: †p ⬍ 0.05, *p ⬍ 0.01, **p ⬍ 0.001).
Neuronal synchrony We analyzed neuronal synchrony in pairs of sorted units and investigated how its prevalence varied across subcortical areas; the results are given in Table 6. Using the cross-correlation approach, 43.0% of Vim/Vop pairs and 25.8% of STN pairs were found to be significantly synchronous. Both of these percentages represent significant populations (binomial test, p ⬍⬍ 0.001 for both cases). Figure 12 A shows example cross-correlation plots for three highly synchronous pairs, while Figure 12 B shows the normalized JPSTH for the same three pairs. Whereas Figure 12 B, i and ii, clearly shows temporal synchronization along the diagonal (and off-diagonals), the same result is not visually discernible in Figure 12 Biii. We found highly significant differences between the Vim/ Vop and STN in terms of the proportions of synchronous pairs. A significantly higher proportion of Vim/Vop pairs were synchronous than STN pairs (two-tailed Fisher’s exact test, p ⬍⬍ 0.001). It has been reported that the level of tremor in parkinsonian patients is positively correlated to the degree of pairwise synchrony among STN cells (Levy et al., 2000). To test the relationship between tremor tuning and local synchrony, we compared the subpopulation of both Vim/Vop and STN neurons that were synchronous with at least one other neuron in their respective ensembles to the subpopulation of neurons tuned to hand tremor (PEPH method). Only neurons fulfilling the criteria of both individual analyses were considered. The results are shown in Table 7. The observed proportions are significantly different (twotailed Fisher’s exact test, p ⬍ 0.01), indicating a clear interaction between tremor tuning and local synchrony. Only units synchronized to at least one other unit were tuned to tremor, whereas no unsynchronized units were tuned to tremor.
32 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
J. Neurosci., June 20, 2012 • 32(25):8620 – 8632 • 8629
Subcortical encoding of behavior Neurons in both subcortical areas (Vim/ Vop and STN) were found to encode features of patient motor behavior, both voluntary (target tracking) and involuntary (tremor). During voluntary behavior, a substantial number of neurons were found to be tuned to target appearance, movement onset, and movement direction (Table 2). This finding likely indicates that neurons in both structures are broadly tuned across multiple modalities and muscle groups. This observation is consistent with previous studies in which STN neurons have been reported to have large receptive fields that respond to multiple joints (Abosch et al., 2002), with 40% of STN cells tuned to movement in a simple two-dimensional joystick task (Williams et al., 2005). It has been reported that 42% of STN neurons respond to passive movement of either arm or leg, and of these, 25% responded to multiple joint movements (Theodosopoulos et al., 2003), whereas 51% of Vim/Vop neurons are tuned to sensory stimuli, with an overlapping 10% of these cells tuned to volitional movements (Lenz et al., 1990). Some thalamic and STN neurons exhibited bimodal encoding of target and movement (Fig. 4), demonstrating what appears to be superposition of two independent neural representations. Furthermore, we observed a significant overlap between tuning to target appearance and to movement onset in both structures. Additionally, directional tuning strength was positively correlated to movement tuning strength, Figure 12. A, Example plots of cross-correlation coefficient for three highly synchronous neuron pairs. Bootstrapped simulations of coefficient shown in red. B, Example normalized JPSTHs for the same three neuron pairs. i, Pair of Vim/Vop cells, Patient I. indicating that individual neurons encoded both parameters by exhibiting both ii, Pair of STN cells, Patient H. iii, Pair of Vim/Vop cells, Patient Q. For B, reported time is relative to target appearance. common-mode and differential activity relative to movement onset (Fig. 3). Thus, neuTable 7. Comparison of synchrony (cross-correlation method) and tremor tuning rons from both subcortical areas exhibited rate modulations based (PEPH method) on a broad superposition of task parameters (Table 1). Tremor tuning (PEPH method) Synchronized to ⱖ1 unit Not synchronized Our analysis of involuntary motor activity (tremor) yielded Tremor tuned 28 0 phase histograms demonstrating clear phase-locking between neuNot tremor tuned 137 30 ronal activity and tremor periods (Fig. 6). Overall, we report significant populations of tremor tuned cells (12.4% for Vim/Vop, 15.9% Discussion for STN). The literature reports a large range in the prevalence of In this study, we analyzed ensemble activity of human subcortical tremor-related cells. For STN, researchers have reported 11% neurons (either Vim/Vop or STN) in 25 patients, recorded in (Magariños-Ascone et al., 2000), 19% (Rodriguez-Oroz et al., 2001), patients who performed visually guided hand movements. To and 52% (Amtage et al., 2008). For Vim/Vop, researchers have reour knowledge, this constitutes the largest sample of human subported 34% (Lenz et al., 1988), 35.6% (Zirh et al., 1998), and 51% cortical ensemble recordings to date. We quantitatively evaluated (Hua and Lenz, 2005). These disparities are probably due to differthe representation of motor parameters in these neurons, as well ences in recording parameters and classification methodology. For as activity thought to be related to pathological states—tremor example, whereas many researchers have classified tremor tuning sensitivity, oscillations, and pairwise synchrony. The present using linear coherence between spike train and recorded EMGs, we work also supports our previous proposition that a sufficiently large used kinematic hand position recorded by a haptic glove. ensemble of subcortical neurons could enable a motor BMI (Patil et We found substantial overlap between tremor-tuned neurons al., 2004). We further propose that chronic subcortical microelecand those tuned to parameters of voluntary motor behavior (Tatrode technology could serve as the basis of a new generation of ble 1). Moreover, we also observed a positive correlation between neuroprosthetic devices aimed at both monitoring and actively retremor tuning strength and directional tuning strength (Vim/ ducing oscillatory firing and elevated levels of network synchrony. Vop cells only). These results are consistent with earlier studies
33 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles
8630 • J. Neurosci., June 20, 2012 • 32(25):8620 – 8632
(Magariños-Ascone et al., 2000; Rodriguez-Oroz et al., 2001) that reported a large proportion of tremor-related STN cells to be simultaneously related to voluntary movements either through motor or sensory loops. Elusive relationship between oscillatory activity and tremor Within a tremor-relevant frequency range (2.5–7.5 Hz), we observed a substantial population of Vim/Vop and STN cells exhibiting strong oscillations (Table 4). Furthermore, we found that the mean autopower spectra for recorded Vim/Vop cells had a higher peak frequency than that of recorded STN cells (Fig. 9B). This finding is consistent with reports of higher mean tremor frequency in ET patients (4 –12 Hz) than in PD patients (3– 6 Hz) (Deuschl et al., 1998). Whereas Rodriguez-Oroz et al. (2001) reported that oscillatory cells in STN did not represent movements, we found moderate overlap (Table 1) between Vim/Vop and STN neurons exhibiting oscillations and those tuned to voluntary behavioral parameters (target, movement, direction). This important finding indicates that there is not a strict dichotomy between pathological neurons and those encoding motor signals. Similarly, we found overlap but no significant correlation between the presence of oscillatory patterns and tremor tuning in both subcortical areas. This finding corroborates the claims of earlier studies of ventral thalamus and STN (Magnin et al., 2000; Rodriguez-Oroz et al., 2001), in which the sets of tremor-related neurons and oscillatory neurons show modest intersection. In other words, not all tremor-related neurons exhibited oscillations, and some oscillatory neurons exhibited no clear association with tremor. Furthermore, spectral peak detection methods revealed no relationship between spike train and hand acceleration (Fig. 10), though there may be an elusive nonlinear or nonstationary relationship between neuronal oscillations and hand tremor. We hypothesized that for some cells, spike train and tremor comodulated with relatively large bandwidth. Our heterodyne decoding results suggest that a substantial number of cells in both subcortical areas may be tremor-tuned in this manner (Table 5). Overall, no consensus in the field has been reached regarding the definitive relationship between oscillatory behavior and tremor tuning, and our results support the idea that pathological oscillations are idiopathic across both neurons and patients. Network synchrony Dopamine depletion in PD has been reported to promote neuronal synchrony within the basal ganglia (Heimer et al., 2006). In this study, a high percentage of neuronal pairs from both subcortical areas exhibited synchronous behavior (Table 6). This is generally consistent with reports of pairwise synchrony in the majority of analyzed STN pairs (Levy et al., 2000, 2002). However, we observed high levels of functional synchrony in neuronal pairs much further apart than the submillimeter separation in Levy et al. (2000, 2002). Furthermore, Vim/Vop neuron pairs (from ET patients) exhibited significantly higher synchrony than STN pairs. Remarkably, only cells exhibiting synchrony with another cell in the ensemble were found to be tremor-tuned; no unsynchronized cells were tremor-tuned (Table 7). This finding corroborates reports that the prevalence of STN pairs synchronized at high frequencies is correlated to the degree of parkinsonian tremor (Levy et al., 2000) and that patients without observable tremor do not exhibit high-frequency STN synchrony (Levy et al., 2002). Others have reported correlations between STN syn-
chrony and bradykinesia/rigidity (Weinberger et al., 2009). Our findings have extended Levy et al.’s (2000, 2002) results to combinatorial pairs in neuronal ensembles from both STN and Vim/ Vop. It remains unclear to what degree network synchrony is indicative of tremor pathology, although the effective disruption of unstable network activity by DBS stimulation suggests a connection. Chronic subcortical ensemble recordings Acute intraoperative recordings exhibit instability and other limitations, whereas long-term recordings from subcortical structures will offer greater signal quality following the recovery of the electrode-tissue interface. Thousands of subcortical DBS implantations are performed every year with minimal risk (Bronstein et al., 2011). Furthermore, long electrode tracts have been shown to yield more stable extracellular recordings (Porada et al., 2000; Krüger et al., 2010) by mitigating the problem of electrode micromotion seen in cortical implants. We suggest that chronic subcortical ensemble recordings may bring about the viability of subcortical BMI systems, first discussed in our previous study (Patil et al., 2004). Our reported offline prediction results from our best Vim/Vop and STN sessions are comparable to those reported from selected rhesus macaque cortical regions with similar neuron count (Wessberg et al., 2000; Carmena et al., 2003). However, hyperbolic extrapolation (Wessberg et al., 2000) suggests an ensemble size greater than 100 to achieve prediction fidelity above R 2 ⫽ 0.9. This would of course necessitate the design of microelectrode arrays for cannula-based implantation with more recording sensors and demonstrable long-term safety and recording efficacy. We propose that clinical studies using chronic ensemble recordings in humans will permit both the continued study of the neurophysiological mechanisms involved in motor control as well as long-term monitoring of pathological activity. Chronic recordings will facilitate continuous examination of changes in tuning, oscillations, and synchrony as a function of the patient’s symptomatic state both on and off treatment. Specifically, this wealth of electrophysiological data may well be used to instruct the improvement of closed-loop DBS systems that are currently in initial stages of development (Rosin et al., 2011; Rouse et al., 2011). We have shown that within the STN and Vim/Vop thalamus, there is an idiopathic mixture of pathology and behavior tuning; these sites are worthy of more targeted treatment than the current dominant approach of delivering high-frequency DBS through macroelectrodes. Subcortical ensembles remain an untapped resource, with the potential to advance both neuroscience and neurorehabilitation alike.
References Abosch A, Hutchison WD, Saint-Cyr JA, Dostrovsky JO, Lozano AM (2002) Movement-related neurons of the subthalamic nucleus in patients with Parkinson disease. J Neurosurg 97:1167–1172. Aertsen AM, Gerstein GL, Habib MK, Palm G (1989) Dynamics of neuronal firing correlation: modulation of “effective connectivity.” J Neurophysiol 61:900 –917. Amirnovin R, Williams ZM, Cosgrove GR, Eskandar EN (2004) Visually guided movements suppress subthalamic oscillations in Parkinson’s disease patients. J Neurosci 24:11302–11306. Amtage F, Henschel K, Schelter B, Vesper J, Timmer J, Lücking CH, Hellwig B (2008) Tremor-correlated neuronal activity in the subthalamic nucleus of Parkinsonian patients. Neurosci Lett 442:195–199. Awiszus F (1997) Spike train analysis. J Neurosci Methods 74:155–166. Bankman IN, Johnson KO, Schneider W (1993) Optimal detection, classi-
34 Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles fication, and superposition resolution in neural waveform recordings. IEEE Trans Biomed Eng 40:836 – 841. Bardorfer A, Munih M, Zupan A, Primozic A (2001) Upper limb motion analysis using haptic interface. IEEE/ASME Trans Mechatronics 6:253–260. Batschelet E (1981) Circular statistics in biology. New York: Academic. Benazzouz A, Breit S, Koudsie A, Pollak P, Krack P, Benabid AL (2002) Intraoperative microrecordings of the subthalamic nucleus in Parkinson’s disease. Mov Disord 17 [Suppl 3]:S145–S149. Birdno MJ, Kuncel AM, Dorval AD, Turner DA, Grill WM (2008) Tremor varies as a function of the temporal regularity of deep brain stimulation. Neuroreport 19:599 – 602. Brodkey JA, Tasker RR, Hamani C, McAndrews MP, Dostrovsky JO, Lozano AM (2004) Tremor cells in the human thalamus: differences among neurological disorders. J Neurosurg 101:43– 47. Bronstein JM, Tagliati M, Alterman RL, Lozano AM, Volkmann J, Stefani A, Horak FB, Okun MS, Foote KD, Krack P, Pahwa R, Henderson JM, Hariz MI, Bakay RA, Rezai A, Marks WJ Jr, Moro E, Vitek JL, Weaver FM, Gross RE, DeLong MR (2011) Deep brain stimulation for Parkinson disease: an expert consensus and review of key issues. Arch Neurol 68:165. Carmena JM, Lebedev MA, Crist RE, O’Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MA (2003) Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biology 1:E42. Chapin JK, Moxon KA, Markowitz RS, Nicolelis MA (1999) Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat Neurosci 2:664 – 670. Davidsen J, Schuster HG (2002) Simple model for 1/f(alpha) noise. Phys Rev E Stat Nonlin Soft Matter Phys 65:026120. Deuschl G, Bain P, Brin M (1998) Consensus statement of the movement disorder society on tremor. Ad Hoc Scientific Committee. Mov Disord 13 [Suppl 3]:2–23. Deuschl G, Schade-Brittinger C, Krack P, Volkmann J, Schäfer H, Bötzel K, Daniels C, Deutschländer A, Dillmann U, Eisner W, Gruber D, Hamel W, Herzog J, Hilker R, Klebe S, Kloss M, Koy J, Krause M, Kupsch A, Lorenz D, et al. (2006) A randomized trial of deep-brain stimulation for Parkinson’s disease. N Engl J Med 355:896 –908. Fetz EE (2007) Volitional control of neural activity: implications for braincomputer interfaces. J Physiol 579:571–579. Ghazanfar AA, Krupa DJ, Nicolelis MA (2001) Role of cortical feedback in the receptive field structure and nonlinear response properties of somatosensory thalamic neurons. Exp Brain Res 141:88 –100. Ghika J, Wiegner AW, Fang JJ, Davies L, Young RR, Growdon JH (1993) Portable system for quantifying motor abnormalities in Parkinsonsdisease. IEEE Trans Biomed Eng 40:276 –283. Gradinaru V, Mogri M, Thompson KR, Henderson JM, Deisseroth K (2009) Optical deconstruction of parkinsonian neural circuitry. Science 324:354 –359. Grimaldi G, Piet L, Manto M (2007) Effects of wrist oscillations on contralateral neurological postural tremor using a new myohaptic device (‘wristalyzer’). In: Proceedings of the 4th IEEE-EMBS, pp. 44 – 48. Cambridge: IEEE. Guillery RW, Sherman SM (2002) The thalamus as a monitor of motor outputs. Philos Trans R Soc Lond B Biol Sci 357:1809 –1821. Gutierrez R, Carmena JM, Nicolelis MA, Simon SA (2006) Orbitofrontal ensemble activity monitors licking and distinguishes among natural rewards. J Neurophysiol 95:119 –133. Hamani C, Saint-Cyr JA, Fraser J, Kaplitt M, Lozano AM (2004) The subthalamic nucleus in the context of movement disorders. Brain 127:4 –20. Heimer G, Rivlin M, Israel Z, Bergman H (2006) Synchronizing activity of basal ganglia and pathophysiology of Parkinson’s disease. J Neural Transm Suppl 70:17–20. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, Branner A, Chen D, Penn RD, Donoghue JP (2006) Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442:164 –171. Hua SE, Lenz FA (2005) Posture-related oscillations in human cerebellar thalamus in essential tremor are enabled by voluntary motor circuits. J Neurophysiol 93:117–127. Kennedy PR, Bakay RA (1998) Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport 9:1707–1711. Kennedy PR, Bakay RA, Moore MM, Adams K, Goldwaithe J (2000) Direct
J. Neurosci., June 20, 2012 • 32(25):8620 – 8632 • 8631 control of a computer from the human central nervous system. IEEE Trans Rehabil Eng 8:198 –202. Kennedy PR, Kirby MT, Moore MM, King B, Mallory A (2004) Computer control using human intracortical local field potentials. IEEE Trans Neural Syst Rehabil Eng 12:339 –344. Koller WC, Lyons KE, Wilkinson SB, Troster AI, Pahwa R (2001) Longterm safety and efficacy of unilateral deep brain stimulation of the thalamus in essential tremor. Mov Disord 16:464 – 468. Krüger J, Caruana F, Volta RD, Rizzolatti G (2010) Seven years of recording from monkey cortex with a chronically implanted multiple microelectrode. Front Neuroeng 3:6. Kuiper NH (1962) Tests concerning random points on a circle. Proc Koninkl Neder Akad Wetensch 63:38 – 47. Kumar R, Lozano AM, Sime E, Lang AE (2003) Long-term follow-up of thalamic deep brain stimulation for essential and parkinsonian tremor. Neurology 61:1601–1604. Lebedev MA, Denton JM, Nelson RJ (1994) Vibration-entrained and premovement activity in monkey primary somatosensory cortex. J Neurophysiol 72:1654 –1673. Lebedev MA, Tate AJ, Hanson TL, Li Z, O’Doherty JE, Winans JA, Ifft PJ, Zhuang KZ, Fitzsimmons NA, Schwarz DA, Fuller AM, An JH, Nicolelis MA (2011) Future developments in brain-machine interface research. Clinics (Sao Paulo) 66 [Suppl 1]:25–32. Lenz FA, Tasker RR, Kwan HC, Schnider S, Kwong R, Murayama Y, Dostrovsky JO, Murphy JT (1988) Single unit analysis of the human ventral thalamic nuclear group: correlation of thalamic “tremor cells” with the 3– 6 Hz component of parkinsonian tremor. J Neurosci 8:754 –764. Lenz FA, Kwan HC, Dostrovsky JO, Tasker RR, Murphy JT, Lenz YE (1990) Single unit analysis of the human ventral thalamic nuclear group: activity correlated with movement. Brain 113:1795–1821. Lenz FA, Kwan HC, Martin RL, Tasker RR, Dostrovsky JO, Lenz YE (1994) Single unit analysis of the human ventral thalamic nuclear group: tremorrelated activity in functionally identified cells. Brain 117:531–543. Lenz FA, Jaeger CJ, Seike MS, Lin YC, Reich SG (2002) Single-neuron analysis of human thalamus in patients with intention tremor and other clinical signs of cerebellar disease. J Neurophysiol 87:2084 –2094. Levy R, Hutchison WD, Lozano AM, Dostrovsky JO (2000) High-frequency synchronization of neuronal activity in the subthalamic nucleus of parkinsonian patients with limb tremor. J Neurosci 20:7766 –7775. Levy R, Ashby P, Hutchison WD, Lang AE, Lozano AM, Dostrovsky JO (2002) Dependence of subthalamic nucleus oscillations on movement and dopamine in Parkinson’s disease. Brain 125:1196 –1209. Magariños-Ascone CM, Figueiras-Mendez R, Riva-Meana C, CórdobaFernández A (2000) Subthalamic neuron activity related to tremor and movement in Parkinson’s disease. Eur J Neurosci 12:2597–2607. Magnin M, Morel A, Jeanmonod D (2000) Single-unit analysis of the pallidum, thalamus and subthalamic nucleus in parkinsonian patients. Neuroscience 96:549 –564. Marsden JF, Limousin-Dowsey P, Ashby P, Pollak P, Brown P (2001) Subthalamic nucleus, sensorimotor cortex and muscle interrelationships in Parkinson’s disease. Brain 124:378 –388. Nicolelis MA (2001) Actions from thoughts. Nature 409:403– 407. Nicolelis MA, Lebedev MA (2009) Principles of neural ensemble physiology underlying the operation of brain-machine interfaces. Nat Rev Neurosci 10:530 –540. O’Doherty JE, Lebedev MA, Ifft PJ, Zhuang KZ, Shokur S, Bleuler H, Nicolelis MA (2011) Active tactile exploration using a brain-machine-brain interface. Nature 479:228 –231. Ondo W, Jankovic J, Schwartz K, Almaguer M, Simpson RK (1998) Unilateral thalamic deep brain stimulation for refractory essential tremor and Parkinson’s disease tremor. Neurology 51:1063–1069. Oppenheim AV, Schafer RW (1975) Digital signal processing. Englewood Cliffs, NJ: Prentice-Hall. Parent A, Hazrati LN (1995) Functional anatomy of the basal ganglia. II. The place of subthalamic nucleus and external pallidum in basal ganglia circuitry. Brain Res Brain Res Rev 20:128 –154. Patil PG, Turner DA (2008) The development of brain-machine interface neuroprosthetic devices. Neurotherapeutics 5:137–146. Patil PG, Carmena JM, Nicolelis MA, Turner DA (2004) Ensemble recordings of human subcortical neurons as a source of motor control signals for a brain-machine interface. Neurosurgery 55:27–35; discussion 35–38.
35 8632 • J. Neurosci., June 20, 2012 • 32(25):8620 – 8632 Porada I, Bondar I, Spatz WB, Krüger J (2000) Rabbit and monkey visual cortex: more than a year of recording with up to 64 microelectrodes. J Neurosci Methods 95:13–28. Quiroga RQ, Reddy L, Kreiman G, Koch C, Fried I (2005) Invariant visual representation by single neurons in the human brain. Nature 435:1102–1107. Raeva S, Vainberg N, Tikhonov Y, Tsetlin I (1999) Analysis of evoked activity patterns of human thalamic ventrolateral neurons during verbally ordered voluntary movements. Neuroscience 88:377–392. Rodriguez-Oroz MC, Rodriguez M, Guridi J, Mewes K, Chockkman V, Vitek J, DeLong MR, Obeso JA (2001) The subthalamic nucleus in Parkinson’s disease: somatotopic organization and physiological characteristics. Brain 124:1777–1790. Rodriguez-Oroz MC, Obeso JA, Lang AE, Houeto JL, Pollak P, Rehncrona S, Kulisevsky J, Albanese A, Volkmann J, Hariz MI, Quinn NP, Speelman JD, Guridi J, Zamarbide I, Gironell A, Molet J, Pascual-Sedano B, Pidoux B, Bonnet AM, Agid Y, et al. (2005) Bilateral deep brain stimulation in Parkinson’s disease: a multicentre study with 4 years follow-up. Brain 128:2240 –2249. Rosin B, Slovik M, Mitelman R, Rivlin-Etzion M, Haber SN, Israel Z, Vaadia E, Bergman H (2011) Closed-loop deep brain stimulation is superior in ameliorating parkinsonism. Neuron 72:370 –384. Rouse AG, Stanslaski SR, Cong P, Jensen RM, Afshar P, Ullestad D, Gupta R, Molnar GF, Moran DW, Denison TJ (2011) A chronic generalized bidirectional brain-machine interface. J Neural Eng 8:036018. Serruya MD, Hatsopoulos NG, Paninski L, Fellows MR, Donoghue JP (2002) Instant neural control of a movement signal. Nature 416:141–142. Su Y, Allen CR, Geng D, Burn D, Brechany U, Bell GD, Rowland R (2003)
Hanson, Fuller et al. • Population Analysis of Human Subcortical Ensembles 3-D motion system (“data-gloves”): application for Parkinson’s disease. IEEE Trans Instr Meas 52:662– 674. Taylor DM, Tillery SI, Schwartz AB (2002) Direct cortical control of 3D neuroprosthetic devices. Science 296:1829 –1832. Theodosopoulos PV, Marks WJ Jr, Christine C, Starr PA (2003) Locations of movement-related cells in the human subthalamic nucleus in Parkinson’s disease. Mov Disord 18:791–798. Vinjamuri R, Crammond DJ, Kondziolka D, Lee HN, Mao ZH (2009) Extraction of sources of tremor in hand movements of patients with movement disorders. IEEE Trans Inf Technol Biomed 13:49 –56. Weinberger M, Hutchison WD, Dostrovsky JO (2009) Pathological subthalamic nucleus oscillations in PD: can they be the cause of bradykinesia and akinesia? Exp Neurol 219:58 – 61. Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, Kim J, Biggs SJ, Srinivasan MA, Nicolelis MA (2000) Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408:361–365. Wiest MC, Bentley N, Nicolelis MA (2005) Heterogeneous integration of bilateral whisker signals by neurons in primary somatosensory cortex of awake rats. J Neurophysiol 93:2966 –2973. Williams ZM, Neimat JS, Cosgrove GR, Eskandar EN (2005) Timing and direction selectivity of subthalamic and pallidal neurons in patients with Parkinson disease. Exp Brain Res 162:407– 416. Zar JH (1999) Biostatistical analysis, 4th edition. Upper Saddle River, NJ: Prentice Hall. Zirh TA, Lenz FA, Reich SG, Dougherty PM (1998) Patterns of bursting occurring in thalamic cells during parkinsonian tremor. Neuroscience 83:107–121.
36
www.nature.com/scientificreports
OPEN
received: 21 April 2016 accepted: 04 July 2016 Published: 11 August 2016
Long-Term Training with a Brain-Machine Interface-Based Gait Protocol Induces Partial Neurological Recovery in Paraplegic Patients Ana R. C. Donati1,2, Solaiman Shokur1, Edgard Morya3,4, Debora S. F. Campos1,2, Renan C. Moioli3,4, Claudia M. Gitti1,2, Patricia B. Augusto1,2, Sandra Tripodi1,2, Cristhiane G. Pires1,2, Gislaine A. Pereira1,2, Fabricio L. Brasil3,4, Simone Gallo5, Anthony A. Lin1,6, Angelo K. Takigami1, Maria A. Aratanha3, Sanjay Joshi7, Hannes Bleuler5, Gordon Cheng8, Alan Rudolph6,9 & Miguel A. L. Nicolelis1,3,6,10,11,12 Brain-machine interfaces (BMIs) provide a new assistive strategy aimed at restoring mobility in severely paralyzed patients. Yet, no study in animals or in human subjects has indicated that longterm BMI training could induce any type of clinical recovery. Eight chronic (3–13 years) spinal cord injury (SCI) paraplegics were subjected to long-term training (12 months) with a multi-stage BMIbased gait neurorehabilitation paradigm aimed at restoring locomotion. This paradigm combined intense immersive virtual reality training, enriched visual-tactile feedback, and walking with two EEGcontrolled robotic actuators, including a custom-designed lower limb exoskeleton capable of delivering tactile feedback to subjects. Following 12 months of training with this paradigm, all eight patients experienced neurological improvements in somatic sensation (pain localization, fine/crude touch, and proprioceptive sensing) in multiple dermatomes. Patients also regained voluntary motor control in key muscles below the SCI level, as measured by EMGs, resulting in marked improvement in their walking index. As a result, 50% of these patients were upgraded to an incomplete paraplegia classification. Neurological recovery was paralleled by the reemergence of lower limb motor imagery at cortical level. We hypothesize that this unprecedented neurological recovery results from both cortical and spinal cord plasticity triggered by long-term BMI usage. Spinal Cord Injury (SCI) rehabilitation remains a major clinical challenge, especially in cases involving chronic complete injury. Clinical studies using body weight support systems1,2, robotic assistance1–4, and functional electrostimulation of the leg5,6 have proposed potential solutions for assisting SCI patients in walking7,8. Yet, none of these approaches have generated any consistent clinical improvement in neurological functions, namely somatosensory (tactile, proprioceptive, pain, and temperature) perception and voluntary motor control, below the level of the spinal cord lesion. 1
Neurorehabilitation Laboratory, Associação Alberto Santos Dumont para Apoio à Pesquisa (AASDAP), Sâo Paulo, Brazil. 2Associação de Assistência à Criança Deficiente (AACD), São Paulo, Brazil. 3Edmond and Lily Safra International Institute of Neuroscience, Santos Dumont Institute, Macaiba, Brazil. 4Alberto Santos Dumont Education and Research Institute, Sao Paulo, Brazil. 5STI IMT, Ecole Polytechnique Federal de Lausanne, Lausanne, Switzerland. 6Department of Biomedical Engineering, Duke University, Durham, NC, USA. 7Mechanical and Aerospace Engineering, University of California, Davis, CA, USA. 8Institute for Cognitive Systems, Technical University of Munich (TUM), Munich, Germany, Germany. 9Colorado State University, Fort Collins, CO, USA. 10 Department of Neurobiology, Duke University, Durham, NC, USA. 11Department of Psychology and Neuroscience, Duke University, Durham, NC, USA. 12Center for Neuroengineering, Duke University, Durham, NC, USA. Correspondence and requests for materials should be addressed to M.A.L.N. (email: nicoleli@neuro.duke.edu) Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
1
37
www.nature.com/scientificreports/ Since the first experimental demonstrations in rats9, monkeys10,11, and the subsequent clinical reports in humans12–14, brain-machine interfaces (BMIs) have emerged as potential options to restore mobility in patients who are severely paralyzed as a result of spinal cord injuries (SCIs) or neurodegenerative disorders15. However, to our knowledge, no study has suggested that long-term training associating BMI-based paradigms and physical training could trigger neurological recovery, particularly in patients clinically diagnosed as having a complete SCI. Yet, in 60–80% of these “complete” SCI patients, neurophysiological assessments16,17 and post-mortem anatomical18 studies have indicated the existence of a number of viable axons crossing the level of the SCI. This led some authors to refer to these patients as having a “discomplete” SCI17 and predict that these remaining axons could mediate some degree of neurological recovery. For the past few years, our multidisciplinary team has been engaged in a project to implement a multi-stage neurorehabilitation protocol – the Walk Again Neurorehabilitation (WA-NR) – in chronic SCI patients. This protocol included the intensive employment of immersive virtual-reality environments, combining training on non-invasive brain-control of virtual avatar bodies with rich visual and tactile feedback, and the use of closed-loop BMI platforms in conjunction with lower limb robotic actuators, such as a commercially available robotic walker (Lokomat, Hocoma AG, Volketswil, Switzerland), and a brain-controlled robotic exoskeleton, custom-designed specifically for the execution of this project. Originally, our central goal was to explore how much such a long-term BMI-based protocol could help SCI patients regain their ability to walk autonomously using our brain-controlled exoskeleton. Among other innovations, this device provides tactile feedback to subjects through the combination of multiple force-sensors, applied to key locations of the exoskeleton, such as the plantar surface of the feet, and a multi-channel haptic display, applied to the patient’s forearm skin surface. Unexpectedly, at the end of the first 12 months of training with the WA-NR protocol, a comprehensive neurological examination revealed that all of our eight patients had experienced a significant clinical improvement in their ability to perceive somatic sensations and exert voluntary motor control in dermatomes located below the original SCI. EEG analysis revealed clear signs of cortical functional plasticity, at the level of the primary somatosensory and motor cortical areas, during the same period. These findings suggest, for the first time, that long-term exposure to BMI-based protocols enriched with tactile feedback and combined with robotic gait training may induce cortical and subcortical plasticity capable of triggering partial neurological recovery even in patients originally diagnosed with a chronic complete spinal cord injury.
Methods
Eight paraplegic patients, suffering from chronic (>1 year) spinal cord injury (SCI, seven complete and one incomplete, see Fig. 1A, Supplementary Methods Inclusion/exclusion Criteria), were followed by a multidisciplinary rehabilitation team, comprised of clinical staff, engineers, neuroscientists, and roboticists, during the 12 months of 2014. Our clinical protocol, which we named the Walk Again Neurorehabilitation (WA-NR), was approved by both a local ethics committee (Associação de Assistência à Criança Deficiente, Sao Paulo, Sao Paulo, Brazil #364.027) and the Brazilian federal government ethics committee (CONEP, CAAE: 13165913.1.0000.0085). All research activities were carried out in accordance with the guidelines and regulations of the Associação de Assistência à Criança Deficiente and CONEP. Each participant signed written informed consent before enrolling in the study. The central goal of this study was to investigate the clinical impact of the WA-NR, which consisted of the integration between traditional physical rehabilitation and the use of multiple brain-machine interface paradigms (BMI). This protocol included six components: (1) an immersive virtual reality environment in which a seated patient employed his/her brain activity, recorded via a 16-channel EEG, to control the movements of a human body avatar, while receiving visuo-tactile feedback; (2) identical interaction with the same virtual environment and BMI protocol while patients were upright, supported by a stand-in-table device; (3) training on a robotic body weight support (BWS) gait system on a treadmill (Lokomat, Hocoma AG, Switzerland); (4) training with a BWS gait system fixed on an overground track (ZeroG, Aretech LLC., Ashburn, VA); (5) training with a brain-controlled robotic BWS gait system on a treadmill; and (6) gait training with a brain-controlled, sensorized 12 degrees of freedom robotic exoskeleton (see Supplementary Material). In all cases except components 3 and 4 above, patients received continuous streams of tactile feedback from either the virtual (body avatar) or robotic devices (Lokomat and exoskeleton) via a haptic display (consisting of arrays of coined shaped vibrators) applied to the skin surface of the patient’s forearms. Tactile stimulation on the forearm was given in accordance with the rolling of the ipsilateral virtual or robotic feet on the ground. Two BMI strategies were employed throughout training. Initially, patients were required to imagine movement of the arms to modulate EEG activity so that they could generate high level motor commands such as ‘walk’ or ‘stop’. Once patients mastered this first method, they learned to use EEG signals to control individual avatar/ robotic leg stepping by imagining movements of their own legs (see Supplementary Material for details). For the first paradigm, after the selection of the correct state, patients confirmed their choice by performing an isometric contraction of the triceps muscle. Three lower limb actuators were used for the BMI experiments: a simulated 3D virtual avatar, the Lokomat gait trainer (Hocoma AG, Switzerland); and a custom built exoskeleton. The virtual avatar was simulated in MotionBuilder (Autodesk MotionBuilder 2014) and visualized from first person perspective using an immersive head mounted displayer (Oculus Rift, Oculus VR). The Lokomat is a robotic gait device that uses a body weight support (BWS) system integrated with a treadmill. Finally, a custom built exoskeleton was used with our patients. The exoskeleton had autonomous power, self-stabilization, and full lower limb hydraulic actuation. It was built to accommodate a wide weight range of SCI patients (50–80 kg) while not necessitating the use of crutches (See Supplementary Movie S11). Our exoskeleton was used in conjunction with a ZeroG19 system containing an overground BWS system that rides along an overhead fixed track. In this setup, there were no mechanical barriers between the patient and physical therapist. For this reason, the walking setup offers more challenges to the Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
2
38
www.nature.com/scientificreports/
Figure 1. Methodology. (A) Cumulated number of hours and sessions for all patients over 12 months. We report cumulated hours for the following activities: classic physiotherapy activities (e.g. strengthening/stretching), gait-BMI-based neurorehabilitation, one-to-one consultations with a psychologist, periodic measurements for research purposes and routine medical monitoring (vital signs, etc.). (B) Neurorehabilitation training paradigm and corresponding cumulated number of hours for all patients: 1) Brain controlled 3D avatar with tactile feedback when patient is seated on a wheelchair or 2) in an orthostatic position on a stand-in-table, 3) Gait training using a robotic body weight support (BWS) system on a treadmill (LokomatPro, Hocoma), 4) Gait training using an overground BWS system (ZeroG, Aretech). 5–6) Brain controlled robotic gait training integrated with the sensory support of the tactile feedback at gait devices (BWS system on a treadmill or the exoskeleton). (C) Material used for the clinical sensory assessment of dermatomes in the trunk and lower limbs: to evaluate pain sensitivity, examiner used a pin-prick in random positions of the body segments. Nylon monofilaments applying forces ranging between 300 to 0.2 grams on the skin, were used to evaluate patients’ sensitivity for crude to fine touch. Dry cotton and alcohol swabs were used to assess respectively warm and cold sensation. Vibration test was done using a diapason on patients’ legs bone surface. Deep pressure was assessed with an adapted plicometer in every dermatome. subjects, in comparison to off-the-shelf devices, by requiring patients to be in charge of postural and trunk control, upper limb strength and dynamic balance. Further gait training was performed by having subjects utilize a lower limb orthosis and walking assistive devices (hip-knee-ankle-foot orthosis or ankle-foot orthosis with knee extension splint and wheeled triangular walker). Throughout the application of our protocol, the complexity of activities was increased over time to ensure cardiovascular system stability and better patient postural control; starting with orthostatic training at a stand-in-table and progressing all the way to the different gait training robotic systems20–22. In addition to routine general clinical evaluations (i.e. cardiovascular function, intestinal and urinary emptying, skin inspection, spasticity handling), before and after every activity, and a long-term treatment of osteoporosis, multiple clinical evaluations were periodically performed in order to identify possible changes in the neurological status of the SCI and to assess psychological and physical conditions. Such clinical evaluation started on the first day patients began training (Day 0), and were repeated after 4, 7, 10, and 12 months. Clinical evaluations included: the American Spinal Injury Association (ASIA) Impairment Scale (International Standards for the Neurological Classification of Spinal Cord Injury23), the Semmes-Weinstein Monofilament Test24 (Fig. 1B), the evaluation of temperature, vibration, proprioception and deep pressure sensitivity, a muscle strength test (Lokomat L-force Evaluation)25,26, the Thoracic-Lumbar Scale for trunk control assessment27, Walking Index Spinal Cord Injury II (WISCI)28, Spinal Cord Independence Measurement III (SCIM)29, McGill Pain Questionnaire30 and Visual Analogue Scale (VAS)31,32 for pain evaluation, the range of motion of lower limb joints of the Medical Research Council scale33, Modified Ashworth Scale34 and the Lokomat L-stiff Evaluation for spasticity35, the World Health Organization Quality of Life Assessment Instrument-Bref (WHOQoL-Bref)36, the Rosenberg Self-Esteem Scale37, and the Beck Depression Inventory (BDI)38.
Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
3
39
www.nature.com/scientificreports/
Patient 1
Lesion level
Gen
Age
Lesion grade
Right
Left
Time since the lesion (years)
Etiology
F
32
ASIA A
T11
T10
13
Closed trauma Closed trauma
Patient 2
M
26
ASIA B
T4
T4
6
Patient 3
M
32
ASIA A
T10
T11
5
Open injury
Patient 4
M
38
ASIA A
T8
T8
5
Closed trauma
Patient 5
M
36
ASIA A
T7
T7
3
Closed trauma
Patient 6
M
29
ASIA A
T4
T4
8
Closed trauma
Patient 7
M
27
ASIA A
T7
T5
6
Closed trauma
Patient 8
F
29
ASIA A
T11
T11
11
Closed trauma
Table 1. Patients’ demography.
Throughout training, the potential occurrence of functional cortical plasticity was evaluated through longitudinal analyses of EEG recordings. For this, patients were instructed to imagine movements of their own legs while EEG signals from 11 scalp electrodes were recorded over the leg primary somatosensory and motor cortical areas. Independent Component Analysis (ICA,39) was employed to determine potential cortical sources, represented by individual independent components (ICs), of novel leg representations in the primary motor and somatosensory cortices and to detect functional changes of these representations over time. To evaluate brain dynamics modulation, before and after many months of training, we calculated for each IC the Event Related Spectral Perturbations (ERSPs) with respect to a baseline of 1 second prior to the event and normalized by the average power across trials at each frequency. Event Related Potentials (ERPs), sampled from two EEG electrodes located over the leg representation area, averaged over all patients, before and after training, were also calculated and used for statistical comparison.
Results
Altogether, the eight paraplegic patients enrolled in this protocol (Table 1 for patient’s demography) participated in a total of 2,052 sessions, for a collective total of 1,958 hours, divided into multiple phases of neurorehabilitation training (Fig. 1A). Figure 1B details the distribution of hours of the six stages employed in our WA-NR protocol. As part of the neurological evaluation, we periodically tested all patients’ sensitivity to fine touch, pain, temperature, vibration, pressure, and proprioception (Fig. 1C). Figure 2A describes the individual improvement of each patient, in number of dermatomes, for tactile (Semmes-Weinstein monofilament test) and pain sensitivity (ASIA sensory evaluation) for all eight patients at the end of 10th month of training. The left graph shows the improvement of normal sensation, while the right graph depicts the improvement in altered sensation (hyper or hypoesthesia). In each of the graphs, improvement is described in terms of number of dermatomes, i.e. the body area that receives sensory innervation from a given spinal nerve root. In this analysis, we measured the extent of the Zone of Partial Preservation (ZPP), i.e. the dermatomes and myotomes (the set of muscles innervated by one spinal nerve) caudal to the neurological level of injury that remain partially innervated23. The ZPP applies only for complete SCI patients. Figure 2B displays the patients’ average improvement, in number of dermatomes below the SCI, for both tactile and pain sensation after 10 months of training. Inspection of these two figures reveals significant improvement in every monofilament and pin tested for all patients. Although we observed significant improvements in both the zone of normal sensation and in the ZPP, the largest gains (5.1 +/− 0.9 dermatomes) were consistently observed in the latter in every patient. In terms of tactile sensitivity, the largest improvements in number of dermatomes (1 normal/2 altered both pink and orange) were observed for the pink (300 g) and orange monofilaments (10 g). For the orange monofilament, the improvement was 1 dermatome for normal sensory area and 2 dermatomes for the ZPP area. Figure 2C depicts two of the patients who exhibited the best improvement in both fine tactile and nociceptive perception. Notice that the improvement was larger for the latter and includes many dermatomes below the SCI level. Throughout the study, all patients reported some type of pain sensation below the SCI level, as measured by the McGill’s questionnaire. Yet, patients experienced some difficulty in reporting the exact location of such pain. This finding corroborates the results described by Demirel et al.40 who showed that 68% of their SCI patients reported pain in the lower limbs. As the training progressed, the patients’ self-reported pain intensity in daily life as measured by the visual analogue scale (VAS,32) - decreased on average from 2.0 to 1.3 (maximum 10, n = 8) while perception of pain at the moment of the evaluation (Present Pain Intensity evaluation integrates McGill Questionnaire30) stayed very low (sum over all patients = 3, with a maximum possible score = 40) throughout the year. An improvement in patients’ ability to perceive and relate more effectively the occurrence of lower limb pain as well the pain location in the body was observed at the end of the study. Supplementary Movie S1 shows the mean improvement for pain sensation over the 1 year training period. Figure 2D–I describes the mean evolution of every somatosensory parameter (pain, tactile, temperature, pressure, vibration, and proprioception) measured over the time course of our study. These graphs revealed that we did not observe any significant improvement in temperature sensitivity (Fig. 2F). Conversely, pressure sensitivity improved by 1 (normal) and 2 (altered) dermatomes, between the 4th and 10th month (Fig. 2G). Figure 2H indicates that the average sensitivity to vibration also improved, mainly for the hip joint (anterior superior iliac spine),
Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
4
40
www.nature.com/scientificreports/
Figure 2. Sensory improvement after neurorehabilitation training. (A) Left table reports the improvement for altered sensation (hyper or hypoesthesia) and the right table reports normal sensation. Number of dermatomes recovered after 10 months training compared to baseline (recorded at day 1 of training) for pain and tactile sensory modalities for all patients and body sides. Sensory modalities are reported by pain, crude touch (applied with a 300 gr. monofilament), and gradually more selective touch (applied with 10 gr. to 0.2 gr. monofilaments). The brightness of each colored square represents the magnitude of improvement, considering each monofilament and pin employed (lightest represents highest improvement). (B) Average sensory improvement (mean +/− SEM over all patients) after 10 months training. (C) Example of improvement in the Zone of Partial Preservation for sensory evaluation for two patients. (D–G) Mean +/− SEM of lowest dermatome with normal (red) or altered (blue) sensation for (D) Pain (E) Tactile – crude touch, pink monofilament (F) Temperature and (G) Pressure on the body calculated over all patients. From (D–G), y-axis exhibits dermatomes in a cranio-caudal order, following the anatomic sequence. Baseline was recorded during the year following the injury, time 0 represents the starting day of our training. For each modality we show the average over raw (top graph) and z-scored data (lower graph). P-values for Wilcoxon rank sum test are reported on z-scored data (*p < 0.05, **p < 0.01, ***p < 0.001). (H) Mean score for perception of vibration on eight leg bones presented (most proximal to most distal order). Score convention was the following: 0 for no sensation, 1 for altered sensation and 2 for normal sensation. (I) Mean score for proprioception (0: absent, 1: present) over lower limb joints. Note that measurements for temperature, vibration and proprioception were introduced 4 months after the beginning of the training.
but also for the knee and the ankle. In all patients, improvement went from almost total absence to mid-level vibration sensation (see Figure S1 for details per patient). The average proprioception sensation across all patients also improved significantly during the period between the 4th–12th months (Fig. 2I). This improvement was mainly observed at the hip level, since patients became capable of distinguishing hip flexion and extension for both the right and left sides. Secondary improvement was detected at the knee and ankle, from the 7th month to the 12th month (see Figure S2 for details per patient). Improvement in motor function was also extensively documented in our eight patients. Figure 3A depicts the evolution of the ASIA motor protocol assessment for all patients, considering five key lower limb muscles: hip flexors (rectus femoris proximal portion), knee extensors (rectus femoris distal portion), ankle dorsiflexors (tibialis anterior), ankle plantar flexors (gastrocnemius), and long toe extensors. This analysis revealed that every patient exhibited some degree of improvement in voluntary muscle contraction below the SCI level. Indeed, seven patients experienced an improvement, ranging from two to multiple key muscles below the SCI level. The bottom shelf of Figure 3A depicts the average improvement for all eight patients for 11 muscles (five key muscles and six secondary muscles: gluteus medius, gluteus maximus, hip adductor, medial and lateral hamstrings, and long toe flexor) ranged in a proximal to distal order. Overall, the strongest motor improvement was observed in the six most proximal muscles. Motor recovery was also documented through multi-channel surface EMG recordings (Fig. 3B,C). For this evaluation, patients were instructed verbally by a physiotherapist to alternate movements of their left leg with
Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
5
41
www.nature.com/scientificreports/
Figure 3. Clinical and neurophysiological assessments of lower limb motor recovery. (A) Detail of key muscle improvement per patient, according to clinical evaluation (ASIA) for all patients and the average over all 8 patients for key and secondary lower limb muscles listed in a proximal to distal order (secondary muscles are in italic). Patients that changed classification to ASIA C after 12 months of training have 2 lines rings around their names. (B) Details of the EMG recording procedure in SCI patients. (B1) Raw EMG for the right gluteus maximus muscle for patient P1 is shown at the top of the topmost graph. The lower part of this graph depicts the envelope of the raw EMG, after the signal was rectified and low pass filtered at 3 HZ. Gray shaded areas represent periods where the patient was instructed to move the right leg, while the blue shaded areas indicate periods of left leg movement. Red areas indicate periods where patients were instructed to relax both legs. (B2) All trials over one session were averaged (mean +/− standard deviation envelopes are shown) and plotted as a function of instruction type (gray envelope = contract right leg; blue = contract left leg; red = relax both legs). (B3) Below the averaged EMG record, light green bars indicate instances in which the voluntary muscle contraction (right leg) was significantly different (t-test, p < 0.01) than the baseline (periods where she/he was instructed to relax both legs). Dark green bars depict periods in which there was a significant difference (p < 0.01) between muscle contraction in the right versus the left leg. (C) EMG envelops and t-tests for all recording sessions, involving 4 muscles, for all 8 patients: left and right gluteus maximus (GMx) and reto femoral proximal (RFP) muscles. Color convention and figure organization follows the one of panel B. Data was collected after 7 months of training for all patients and for all but patients P2 and P8 after 12 months.
movements of their right leg, and periods in which neither leg should be moved (Fig. 3B). The first EMG session was recorded after 7 months of training (all patients), then a second session was obtained at 12 months (all patients except P2 and P8). Since all patients were completely paralyzed below the level of injury, none of them exhibited any motor activity below the level of their SCI at the onset of the training. However, 7 months into training, EMG recordings revealed that all patients started to show signs of motor recovery, indicated by their ability to voluntarily control at least one muscle below the level of the SCI (Fig. 3C). By the 12th month of training, this motor recovery had stabilized and, in most cases, improved significantly. Figure 3B,C shows that patient P1 exhibited the strongest and most consistent voluntary contractions (Fig. 3B,C, Supplementary Movies S3, S4, S8, S9) in both left and right gluteus maximus (GMx) and reto femoral proximal (RFP) muscles (contraction significance, t-test, p < 0.01, is indicated in light green under each graph). This patient’s motor control was clearly selective, i.e., a stronger contraction of the right GMx and RFP was observed when the patient was instructed to contract Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
6
42
www.nature.com/scientificreports/ Right Leg 7 months of training
Right Leg 12 months of training
Left Leg 7 months of training
Left Leg 12 months of training
P1
17.7
21.14
16.85
16.12
P2
1.11
1.17
0.95
1.31
P3
2.76
4.42
4.34
4.6
P4
0.47
1.48
0.9
1.07
P5
0.93
4.3
1.22
2.13
P6
0.25
0.6
0.3
0.8
P7
0.45
0.89
0.58
1.2
P8
5.01
Patient
9.36
Table 2. Torque (N·m) measurement 7 and 12 months after beginning of training. Recordings were performed while patient was standing in the Lokomat (L-Force measurement). Patient was instructed by the physiotherapist to contract left or right leg (hip flexion). Peak of torque over six repetitions is reported. the right leg, while left GMx and RFP contractions were produced following the command to contract the left leg (see selectivity significance reported in dark green under each figure, t-test, p < 0.01). Patient P3 exhibited strong significant contractions in both left and right GMx muscles (Fig. 3C, Supplementary Movie S4). However, reliable RFP contractions, absent at the 7th month, appeared and became consistent at the end of the 12th month of training. The same evolution trend in motor control was observed in three other patients, P4, P5, and P7: while at the 7th month these patients exhibited weak or no voluntary GMx muscle contractions, after 12 month of training such contractions became evident in the EMG recordings (bilateral for P4 and P5, left side in P7, Fig. 3C). Finally, patients P2 and P8 also displayed significant voluntary muscle contractions in their GMx muscles during the 7th month of training. P8 also exhibited selective contractions in both her RPF muscles. Altogether, these longitudinal EMG recordings suggest that a sustained and long-term training protocol may be required to trigger motor improvement in ASIA A patients. Motor contractions were also documented through the employment of the Lokomat torque sensors during the same type of task used for EMG measurements (Table 2). Patient P1 showed the best recovery at 21.14 N⋅m (12th month, right leg) for hip flexion (thus RFP contraction), followed by patient P8 at 9.36 N⋅m (7th month), and P3 and P5 at 4.6 and 4.3 N⋅m. Patients P2, P4, P6, P7 produced torques in the 0.8–1.5 N⋅m range. Figure 4A and Supplementary Movie S2 illustrate the temporal progression of the average motor recovery observed in our patients, from month 0 to month 12, considering both key ASIA muscles and secondary lower limb muscles. Notice that this motor recovery clearly progressed from proximal to distal muscles, being more pronounced at the level of the hip joint. Thus, the muscles that exhibited the best recovery were: the gluteus maximum (maximum score 3; average score over all patients 1.56 and 1.5 for right and left side respectively), and gluteus medius (maximum score 2; average score 1.25 and 1.06), the proximal portion of the rectus femoris (max 3; average 1.56 and 1.5), and the hip adductors (max 2; average 1.18 and 1.06). At the level of the knee joint, the greatest improvements were observed in the medial and lateral hamstring (max 1; average 0.38 and 0.31), and the distal portion of the rectus femoris (max 2; average 1.06 and 0.88). At the ankle level, the greatest motor improvement was located in the sural triceps (max 1.5; average 0.31 and 0.25) and the anterior tibialis (max score 1; average 0.5 and 0.38). Overall, the global patterns of both sensory and motor recovery indicated a proximal to distal progression, below the level of the patients’ SCI that evolved to include sacral roots. Supplementary Movies S3–S10 shows patients’ clear lower limb contractions while in a hanging or lying positions after 1 year of training with the WA-NR protocol. As a result of this sensory and motor recovery, Fig. 4B depicts the progression over time of the ASIA classification showing that 50% of our patients (n = 4) changed their ASIA classification in 12 months of training: three of these patients moved from ASIA A to ASIA C and one patient moved from ASIA B to ASIA C. To further document this recovery, Fig. 4C displays the individual patient improvement in thoracic-lumbar strength and stability measured in different positions – seated and laying down – and static and dynamic balance. Between the 7th and 10th months, five of our patients improved significantly in this type of motor control. To illustrate how this motor recovery functionally impacted the patients, Fig. 4D depicts the progression of their Walking Index for SCI over the last 5 months of training. As one can see, all patients showed significant improvement in assisted walking skills; two patients increased their performance by six levels, four patients by five levels, and two more patients by three levels. For example, while Patient 1 was initially not even able to stand using braces when placed in an orthostatic posture (score 0), after 10 months of training the same patient became capable of walking using a walker, braces and the assistance of one therapist (score 6). At this stage, this patient became capable of producing voluntary leg movements mimicking walking, while suspended overground in the Lokomat (Supplementary Movie S8). The same patient produced close to 20N*m of hip flexion force while in the Lokomat (Table 2). In another example, Patient 7, started at score 6 and progressed all the way to score 12, which means that he/she was capable of walking with two crutches and lower limb orthoses (hip-knee-ankle-foot orthoses), while requiring no assistance by a therapist. In addition to the partial recovery in neurological functions, our patients also exhibited improvements in gastrointestinal function and their overall skin condition. Figure 4E plots the monthly evolution of both the mean number of standing/walking hours and the z-scored mean frequency of bowel functioning. The top right graph in Fig. 4E shows that these latter two variables are highly correlated (r = 0.72, n = 10 measurements). Notice that peak bowel function was reached 3 months after the training starting. During the patient’s vacation period Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
7
43
www.nature.com/scientificreports/
Figure 4.â&#x20AC;&#x201A; Clinical and functional improvements. (A) Graphical representation of clinical assessment of motor strength, calculated with the ASIA protocol, for key muscles and other muscles in the lower limb (mean over all patients). The scale for a given muscle goes from complete transparency for no muscle activation (muscle strength score 0) to contraction against gravity (score 3). Time 0 of x-axis means day 1 of the neurorehabilitation training. (B) Patients with ASIA classification improvements: four patients changed ASIA classification over the course of the neurorehabilitation training, three moved from ASIA A to C and one moved from ASIA B to C. ASIA A is characterized by absence of both motor and sensory functions in the Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
8
44
www.nature.com/scientificreports/ lowest sacral area, ASIA B by the presence of sensory functions below the neurological level of injury, including sacral segments S4-S5 and no motor function is preserved more than three levels below the motor level on either side of the body, ASIA C by the presence of voluntary anal sphincter contraction, or sacral sensory sparing with sparing of motor function more than three levels below the motor level, majority of key muscles have muscle grade less than 323. (C) Thoracic-lumbar control scale evaluates quantitatively motor skill of the thoracolumbar region. Score ranges between 0 and 65. It has 10 items that considers supine, prone, sitting and standing postures. In the present study, the last item (orthostatic position) was scored 0 due to the limitations of the pathology. (D) Functional assessment of autonomy in walking given by the Walking Index for Spinal Cord Injury scale. The scale ranges between 0, for a patient who is unable to stand and/or to participate in assisted walking, to 20 for a patient who ambulates 10 meters with no walking devices, no braces and no physical assistance. (E) Correlation between average time spent in a standing position in orthostatic or gait training (mean +/− SEM, values are average hours per month) and mean frequency for bowel function (values calculated per month and z-scored per patient).
(months 4–5), lack of standing/walking was correlated with a significant reduction in bowel function. Upon restarting of the standing/walking training, bowel function increased again. EEG measurements were employed to investigate potential cortical functional reorganization that could correlate with the type of sensorimotor improvement observed during training. An Independent Component Analysis (ICA39) was applied to 11 EEG channels both at the beginning, and after 8–10 months of training. Briefly, the ICA algorithm isolates maximally independent sources from multi-channel EEG signals to both discard non-brain signals (such as muscle artifact or noise) and identify the spatial location of distinct brain-derived signals contained in the overall EEG recording. At the onset of training, in three out of seven patients instructed to imagine moving their own legs, we could only identify a total of four significant independent components (sum for all patients) in the putative leg representation area of the primary somatosensory (S1) and motor (M1) cortices. The top shelf of Fig. 5A depicts the projection of these four components in the S1/M1 region (in a top and coronal slice) and the corresponding event related spectrogram perturbation (ERSP) found for each component is shown in Fig. 5B. The bottom shelf of Fig. 5A (respectively 5B for the ERSP) displays the same information, obtained after 8–10 months of training. Notice that at this point, a total of 12 significant components (sum for all patients) could be isolated in the leg area of the primary sensorimotor cortex of all seven patients who were asked to imagine locomotion movements. Further analysis of the ERSP revealed the electrophysiological origins of the four independent components identified early in training in three patients (Fig. 5B, top shelf). They corresponded to the presence of desynchronization of beta waves (16–20 Hz) for Patients 4 and 5, a clear power reduction in mu rhythm (7–12 Hz, first panel left) in Patient 5, and a smaller desynchronization of mu in Patient 6. In healthy subjects, mu rhythm desynchronization is observed during motor imagery41,42. Desynchronization of beta waves (16.5–20 Hz) is also observed during the preparation period, prior to the execution of voluntary limb movements43–45, as well as during motor imagery46. The bottom shelf of Fig. 5B illustrates the changes that occurred in the ERSP after 8–10 months of training. At that point, a total of 12 Independent Components (IC) could be isolated in the leg representation area of the S1/M1 of all seven patients analyzed. In 4 out of 12 (P1 second IC, P2 first IC, P3 first IC and P7), we detected a reduction in mu power. Interestingly, the remaining eight ICs depicted a significant desynchronization in both mu and beta waves. Altogether, these findings indicate that, after prolonged BMI-based neurorehabilitation training, the leg representation area of S1/M1 cortices of all patients exhibited the type of desynchronization of beta wave which has been associated with motor imagery in healthy subjects46. Further evidence for functional plasticity taking place over the training period was obtained through an event-related potential analysis, considering two central EEG electrodes, located on the leg representation area of the primary S1/M1 cortices, for all seven patients analyzed. Figure 5C shows that, at the onset of training, there was no significant desynchronization of the EEG (red line) when patients were asked to imagine walking. Eight to ten months later, however, a significant EEG desynchronization (green line) was observed when the average for all seven patients was considered. According to our inclusion and exclusion criteria, all our patients exhibited a complete range of motion (ROM) of the joints and a maximal grade of lower limb spasticity of 2 on the Ashworth scale. As our training protocol progressed, we observed that all subjects maintained the original complete range of motion and did not develop any muscles contractures. Moreover, their level of lower limb spasticity did not increase their performance during the orthostatic or gait training. By using a Lokomat L-stiff test to quantify the level of spasticity of hip and knee muscles for flexors and extensors, we observed that, on average, all patients exhibited a reduced spasticity level by the end of 12 months. In addition, half of our patients maintained a stable bone mineral density index while the other half demonstrated a slight improvement. There was no correlation between bone mineral density and neurological improvement. The SCIM questionnaire was applied to assess our patients’ level of functional independence to perform daily activities and mobility at home and in a community environment. Although all included patients exhibited a good level of independence in daily activities at the onset of training (ranging from 64 to 74 where scores ranged between 0 (dependent) to 100 (independent), some patients managed to improve their level of functional independence by the end of the training. For instance, two patients experienced an improvement in the frequency of bowel function with no need of using auxiliary devices (laxatives); two became more independent in the bathroom environment (perineal hygiene, setting towels/napkins and diapers); and two improved their level of independence in the process of transferring from the wheelchair to the toilet and one from the wheelchair to the car. Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
9
45
www.nature.com/scientificreports/
Figure 5. EEG recording. (A) Functional Cortical dynamics over the leg sensory motor area over time. Projection of the position of dipoles found by Independent Component Analysis during leg motor imagery. Analysis revealed that the number of dipoles observed in the S1/M1 cortex evolved from four at onset of the protocol to 12 at the end of training. Two sets of sessions are shown for patients 1 to 7: one recorded in the first 2 months of training (Onset) and one recorded between the 7th and 9th month of training (End). (B) For the onset and end of training, the Event Related Spectral Perturbation (ERSP) is shown for each of the Independent Components (IC) depicted in panel A. At the onset of training, one IC was found for P4 and P5, and two for patient P6. At the end of training, two ICs were found for each of the following patients: P1, P2, P3, P4 and P5. A single IC was identified for patient P6, and one for patient P7; and none for P8. Decrease in power in Beta waves (16·5–20 Hz) is associated with muscle contraction; suppression mu wave (7·5–12·5 Hz) are related to motor actions46. (C) Mean event related potential over all patients for two central electrodes (Cz and CPz) for Onset and End of training period. Significant desynchronization or synchronization is marked with an ‘*’. Overall, all patients ranked very high on emotional stability and obtained good scores on the quality of life, depression and self-esteem assessment, with minimal fluctuations throughout the study. According to individual demand, psychological support was increased, but no use of psychiatric medication was required.
Discussion
As far as we can tell, this is the first clinical study to report the occurrence of consistent, reproducible, and significant partial neurological recovery in multiple chronic SCI patients. This partial recovery was manifested by improvements in both somatic sensations and voluntary motor control, below the level of the spinal cord lesions. This sensorimotor improvement was also paralleled by autonomic improvements, such as bowel function. Moreover, this is also the first report demonstrating partial clinical neurological improvements in SCI patients subjected to long-term training with a BMI-based gait protocol. Up to now, all previous clinical reports involving BMIs focused on decoding and control strategies of artificial prosthetic devices using the subject’s own electrical brain activity alone12–14,47,48. In these studies, patients were able to control the movements of artificial devices using their brain activity. Yet, none of these studies described any type of neurological recovery as a consequence of BMI training. A total of eight chronic SCI patients were trained over the course of 12 months in a multi-stage, progressive neurorehabilitation protocol – the Walk Again Neurorehabilitation (WA-NR) protocol - that employed a non-invasive, EEG-based, closed-loop BMI approach. This protocol required that patients brain-control both Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
10
46
www.nature.com/scientificreports/ virtual and mechanical actuators while receiving rich visuo-tactile feedback, aimed at restoring autonomous locomotion. Common to all periods of the WA-NR was the employment of both: (1) an EEG-based BMI, which required patients to produce motor imagery related to walking, was responsible for controlling the initiation of a series of lower limb motor behaviors (standing, walking and kicking a soccer ball), and (2) a multi-channel sensory substitution (remapping)49,50 strategy that utilized a haptic display applied to the skin surface of the patients’ forearms to deliver both tactile and proprioceptive-like feedback. Such real-time tactile/proprioceptive feedback of autonomous bipedal walking was combined with visual feedback (3) during physical training using a robotic Body Weight Support (BWS) system on a treadmill (LokomatPro), an overground BWS system (ZeroG), and a robotic exoskeleton. For all our patients, clinical diagnosis of total (ASIA A) or partial (ASIA B) paralysis was confirmed, over multiple years, by routine clinical neurological examination, performed by different neurologists belonging to the clinical staff of the hospital in which these patients were followed. Previously, these patients were enrolled in a traditional physical rehabilitation program that mainly aimed at increasing independence in daily living activities, while seated in a wheelchair. Two patients (P2 and P6) had routine training in a standing orthostatic position (stand in table device). Six patients (P1, P3, P4, P5, P6, P7) had walking training using parallel bars or using a walker. None of these subjects exhibited any level of sensory or motor improvement or recovery in the many years they were followed prior to enrollment in our study. At the onset of our protocol, the ASIA status of all eight patients was confirmed by our own initial neurological evaluation. That further supports our contention that the neurological improvement observed here resulted only from the new WA-NR introduced in the present study. Overall, all eight patients involved in the study experienced a significant improvement in tactile, proprioceptive, vibration, and nociceptive (but not temperature) perception. Such improvement was already noticeable after 7 months, but reached its peak at the 10th month of training. On average, such a sensory recovery spanned multiple dermatomes below the SCI level, being more vigorous and consistent for altered nociceptive perception (more than five dermatomes on average) than for tactile, vibration or proprioception (between one-two dermatomes). Thus, as a rule, the pattern of sensory recovery documented in all eight patients indicated a larger effect mediated by small myelinated or non-myelinated fibers, which normally convey nociceptive and high-threshold tactile information, than through the large myelinated fibers that normally mediate fine tactile discrimination and proprioception. This suggests that axons running through the spinothalamic tract were the main mediators of this somatosensory recovery. As such, this observation may imply that the spinothalamic tract may be more resistant to the initial SCI and/or remain more amenable than dorsal column-medial lemniscal fibers to underlie plastic recovery, even many years after a spinal cord lesion. Interestingly, this is consistent with previous studies in which somatosensory plasticity was documented in animals51,52. It is important to mention that, although we have not documented any significant recovery in thermo sensation, this negative result may reflect primarily the lack of specificity of the clinical method employed to evaluate temperature sensing. In the future, we intend to repeat this analysis using a more sensitive technique. In addition to significant sensory recovery, we also observed widespread improvement in voluntary muscle control below the level of SCI, even in patients clinically classified as having a complete SCI. Such a recovery in motor function, which progressed from proximal to distal muscles over time (Fig. 4A) and was more intense at the level of anti-gravitation (extensor) and flexor muscles involved in hip movements – despite the fact that some motor recovery was seen at the level of knee and even ankle joints - was corroborated by clinical examination, EMG recordings (Fig. 3B,C), and direct measurements of L-force generated by patients (Table 2). Such a pattern of motor recovery suggests mediation by intact fibers of the vestibulo-spinal tract (extensor muscles) that run in the ventrolateral portion of the spinal cord, next to the spino-thalamic tract. Motor recovery at flexor muscles suggests that some fibers of the rubro-spinal tract may have also remained intact in some of our patients. Altogether, the partial neurological improvement observed meant that 50% of our patients could be reclassified (three from ASIA A to C and one from ASIA B to C) in less than a year of training with our neurorehabilitation protocol. Prior to the present study, the literature contains only a single case report indicating that a patient with tetraplegia was reclassified from ASIA A to ASIA C after 3 years of being subjected to functional electrical stimulation bicycle therapy53. As far as we can tell, no independent study has reproduced this result so far. Heretofore, partial neurological recovery after an SCI has been mainly reported in subacute incomplete SCI patients. For instance, repetitive transcranial magnetic stimulation (rTMS), applied over the arm and leg representations of the primary somatosensory cortex of incomplete SCI patients, led to limited and variable improvements in sensory and motor functions54–57 primarily when high rTMS intensities were employed. Recently, the use of epidural stimulation at the lumbosacral level, combined with standing and stepping training, has allowed chronic ASIA A and B SCI patients to voluntary control paralyzed leg muscles. However, such motor control could only occur in the presence of the epidural stimulation58. In other words, these patients did not recover the ability to control their muscles without the assistive device. As such, none of the four subjects described by Angeli et al.58 changed their original ASIA classification. Interestingly, the study methodology also included a pre-implantation phase with extensive locomotor training (80 sessions) using a body weight support system on a treadmill. Neurological evaluations, neurophysiological measurements and ASIA exams, were performed before and after assisted gait training and implantation phases and no significant neurological recovery was observed suggesting that an isolated gait trainer with a BWS system on a treadmill does not produce meaningful neurological recovery in complete SCI patients. Moreover, no neurological recovery was described in a recent case report of a single SCI patient who was able to walk again using a BMI gait protocol that employed functional electrical stimulation of the lower limbs6. Since our patients suffered their spinal cord lesions many years before enrolling in our protocol, the likelihood that the sensorimotor improvements observed here were due to spontaneous recovery can be basically ruled out. Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
11
47
www.nature.com/scientificreports/
Figure 6. Hypothesis for mechanism of neurological improvement in SCI patients. (A) Example of spinal cord lesion in the thoracic area. Cross section is shown in the three parts of the spinal cord: at the lesion level, on top and under the lesion. Example of two efferent and one afferent pathways and their corresponding spinal tract are shown (rubrospinal and vestibulospinal tracts; spinothalamic tract respectively). At the lesion level we hypothesize preservation of 2–25% of white matter18 which might include the spinothalamic tract (sensory: pain, temperature, crude touch and pressure), vestibulospinal tract (motor: extensors muscles), rubrospinal tract (motor: flexors muscles) and dorsal columns (sensory: proprioception and fine touch). Under the lesion, Central Pattern Generators (CPGs) and its interaction with descending pathways (reticulospinal tract) and sensory afferents, modulating the gait pattern50. (B) Proposed components for the rehabilitation mechanism: direct brain control of virtual or robotic legs, continuous stream of tactile stimulation representing the missing haptic feedback from the legs and robotic actuators to train patients to walk. Cortical and spinal plasticity are hypothesized to change and to modulate neurological circuits in the preserved area around the lesion through motor (red) and sensory (blue) connections.
Indeed, a review by the International Campaign for Cures of Spinal Cord Injury Paralysis59 about spontaneous recovery after SCI, based on pharmaceutical clinical trials that focused on acute neuroprotection in SCI60–62, reported that the majority of spontaneous recovery occurs during the first 3 months after the SCI. Small residual clinical improvements can persist for up to 18 months, but only minor changes occur afterwards. Thus, 1 year after an SCI, 80% of the initial ASIA A cases remain A, about 10% convert to ASIA B and about 10% to ASIA C. A survey of 987 SCI patients showed that, between 1 and 5 years after the lesion, a conversion from ASIA A to a higher grade occurs in only 6.5% of patients (3.5% to B, 1.05% to C and 1.05% to D)63. Since some of the patients that moved from ASIA A to ASIA C in the present study had suffered their SCI more than a decade ago, it is highly unlikely that spontaneous recovery accounts for our findings. In complete motor lesions (ASIA A and B), the majority of functional recovery occurs within the ZPP, following a craniocaudal sequence. Recovery within the ZPP appears to be due to both CNS and peripheral plasticity, while recovery beyond the ZPP would probably demand some CNS repair, likely involving axon regeneration59. Concerning partial motor recovery, the same review indicates that it likely occurs in myotomes with sensory preservation. Overall, the chances of a recovery of more than two spinal levels below the initial ASIA level are very small. Our findings revealed that motor recovery was indeed more significant within the ZPP. However, we also observed patients’ partial recovery in voluntary motor activity located in more than two dermatomes below the ZPP. A previous study with ASIA B patients64 suggested that preservation of pinprick sensation can be useful in predicting motor recovery: presence of sacral pinprick sensation 4 weeks after the SCI significantly predicted ambulation 1 year later. Although these findings are not directly comparable to ours, by the differences in patient samples, the fact that we were able to document a 50% improvement in ASIA classification after many years of SCI raises the hypothesis that further motor clinical improvement could be seen with longer training. The clear functional significance of the sensorimotor recovery observed here was further demonstrated by both the major overall improvement in the patients’ Walking Index. In other words, the observed partial sensorimotor recovery was translated into a meaningful improvement in the patients’ daily routine. But what mechanism could account for this partial neurological recovery? Kakulas et al.18 showed that about 60% of SCI patients diagnosed clinically as having a complete spinal cord injury (ASIA A) still have 2–27% of the total area of spinal cord white matter preserved18. This finding is further supported by the observation that some of these surviving axons can exhibit functionality17 in more than 80% of such ASIA A patients. Sherwood and colleagues defined these cases as having a “discomplete” SCI and suggested that the residual axons could be Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
12
48
www.nature.com/scientificreports/ functionally rescued to mediate some level of clinical recovery17. We propose that such a mechanism may have accounted for the partial neurological recovery observed in our study (Fig. 6A). Assuming that residual spinal cord connectivity mediated the clinical recovery, what are the potential physiological mechanisms involved? Concomitantly to the partial neurological recovery, we documented the occurrence of cortical functional plasticity through longitudinal analysis of EEG recordings. This plasticity manifested itself by the emergence of consistent activation in the leg representation area of S1/M1, as measured by ICA, event-related spectrogram perturbation, and event-related potential analysis. The most prevalent EEG feature identified over training was the desynchronization of mu rhythm (7.5–12.5 Hz)41,42,65. Secondarily, we have also observed an increase in beta wave desynchronization. Altogether, these results are consistent with our prior hypothesis66,67 that long-term BMI use is capable of changing the cortical body representation by including either new artificial actuators (robotic limbs or avatar bodies) or even by inducing a reactivation of the representation for paralyzed limbs, in the present case, legs. Therefore, based on these EEG findings, we propose that our long-term BMI-based training triggered a significant process of functional plasticity in S1/M1. Such functional cortical plasticity may have accounted for the re-emergence of lower limb representations in these cortical areas, as documented by the EEG analysis described above, and in another study with the same patients50. Such a functional cortical plasticity likely led to the reactivation of upper motor cortical neurons that normally project to the spinal cord, via the corticospinal tract. Given that a small fraction of spinothalamic, vestibulospinal and rubrospinal tract axons may have survived the initial SCI event and remained silent for many years, even in our ASIA A patients, the peculiar motor recovery observed in our study, involving primarily hip extensor and flexor muscles, could be explained by the functional reactivation of these residual axons as a byproduct of plasticity induced by long-term, intensive BMI training (Fig. 5). But what are the key components of BMI training that triggered such a massive cortical plasticity? In our view, the driving force behind this plasticity includes: the direct brain control of robotic actuators by attentive and motivated patients, the reliance on patient’s motor imagery of walking to operate a BMI, a continuous stream of rich tactile/proprioceptive feedback, and the use of robotic actuators that allowed patients to routinely walk upright for long periods of time. This last feature may also have accounted for the generation of complex interactions between the supraspinal centers (cortical and subcortical structures) and the spinal cord. Particularly, in the case of locomotion, the use of robotic gait training may have triggered the engagement of central pattern generators (CP), both at the supraspinal and spinal levels, and contributed to the generation of tactile and proprioceptive feedback from the patients’ own legs. These two components may have also induced functional plasticity at the spinal level and contributed to the type of sensory recovery observed below the level of the SCI, mediated mainly by the spinothalamic tract, and the somatosensory cortical plasticity described above. In both animals68,69 and humans70, central pattern generators (CPGs) have been shown to generate bilateral rhythmic patterns, alternating motor activity between the flexor and extensor motoneurons in the absence of descending inputs71. Animal studies have shown that the gait pattern itself is generated in the spinal cord by CPGs, which are modulated by a peripheral sensory feedback. Conversely, once gait is re-established, proprioceptive and cutaneous afferents signals derived from receptors located in muscles, joints, and skin, can once again drive intraspinal circuits that interact with motor neurons, interneurons and CPGs in order to assist movement adaptations, such as postural corrections69. The level of this CPG activity would be determined by the brainstem locomotor command systems through the reticulospinal pathways68 (Fig. 5). The existence of CPGs in humans has been suggested, but its exact location is still unclear71. Another possible source for sensory improvement observed in our patients could be the long-term use of visuo-tactile stimulation in virtual reality. Sensory modalities are not independent from each other; experiments have shown mechanisms of cross modal interaction72,73, and cross modal integration to create a robust perception74. In particular, vision of a body part was found to influence tactile perception75–77. For example, one experiment has shown that observation of a body part increases tactile acuity during passive touch in healthy subjects78. The effect was present when subjects were touched on their forearm while observing a different part of their own arm; and their accuracy was even increased when the body part was magnified79. No effect was observed if a neutral object was used in place of the arm. Patients with sensory deficiency (stroke patients) were found to improve their sensory resolution by observing their own body. In another experiment, magnifying the arm of a patient with chronic pain increased his/her pain, while minifying decreased it80. Similarly, we hypothesize that long-term training observing a human avatar mimicking the position and orientation of the patients’ body could have induced a positive effect on our patients’ sensory acuity. Based on our clinical findings, we propose that long-term gait training with a BWS that employs BMI-based robotic actuators, combined with rich tactile feedback, could recruit the activation of CPGs in SCI patients81. Likely, BMI-based training and tactile feedback in a virtual reality environment could also enhance CPG activity by recruiting cortical afferents to influence locomotion control. If some corticospinal or vestibulospinal axons are still intact in a fraction of SCI patients, these locomotion-related signals could reach lower alpha-motor neurons below the level of the SCI. Moreover, by making patients walk routinely upright and against load, peripheral tactile and proprioceptive feedback would be generated and transmitted back to the spinal cord, contributing to the process of spinal cord functional reorganization (Fig. 6B). In addition to partial neurological recovery, we observed a clear linear correlation between the number of hours spent upright walking/standing with the amount of bowel function (Fig. 4E). These and other autonomic improvements will be further explored in future studies using the same methodology. We observed that significant clinical recovery was closely related to long-term and frequent use of a BMI paradigm that attempts to recreate lower limb movements in a realistic way (either in a virtual reality environment or through the use of brain-controlled robotic walkers). Further support for this contention can be found in the observation that after the two 30-day periods of vacation afforded to patients during the 12 month duration of
Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
13
49
www.nature.com/scientificreports/ the protocol, we noticed a reduction in sensory and motor capabilities below the SCI level (Figs 2 and 3). Such a temporary reduction was quickly reversed, however, by the restart of the BMI-based training protocol. Introduction of rich tactile feedback signals, delivered via a haptic display, also seemed to have played a key role in the patient’s recovery. Experiments reported elsewhere have shown that patients are capable of incorporating an avatar body and extending their body schema toward the avatar legs employed in our training50. This suggests that the introduction of tactile feedback likely enhanced the ability of patients to exhibit cortical and/ or subcortical functional plasticity during training with our BMI protocol, as we had previously documented in monkeys67,82. Overall, the results obtained in our study suggest that BMI applications should be upgraded from merely a new type of assistive technology to help patients regain mobility, through the use of brain-controlled prosthetic devices, to a potentially new neurorehabilitation therapy, capable of inducing partial recovery of key neurological functions. Such a clinical potential was not anticipated by original BMI studies. Therefore, the present findings raise the relevance of BMI-based paradigms, regarding their impact on SCI patient rehabilitation. In this context, it would be very interesting to repeat the present study using a population of patients who suffered a SCI just a few months prior to the initiation of BMI training. We intend to pursue this line of inquiry next. Based on our findings, we anticipate that this population may exhibit even better levels of partial neurological recovery through the employment of our BMI protocol.
References
1. Alexeeva, N. et al. Comparison of training methods to improve walking in persons with chronic spinal cord injury: a randomized clinical trial. J. Spinal Cord Med. 34, 362–379, doi: 10.1179/2045772311Y.0000000018 (2011). 2. Field-Fote, E. C. & Roach, K. E. Influence of a locomotor training approach on walking speed and distance in people with chronic spinal cord injury: a randomized clinical trial. Phys. Ther. 91, 48–60, doi: 10.2522/ptj.20090359 (2011). 3. Hartigan, C. et al. Mobility Outcomes Following Five Training Sessions with a Powered Exoskeleton. Top Spinal Cord Inj Rehabil 21, 93–99, doi: 10.1310/sci2102-93 (2015). 4. Zeilig, G. et al. Safety and tolerance of the ReWalk exoskeleton suit for ambulation by people with complete spinal cord injury: a pilot study. J. Spinal Cord Med. 35, 96–101, doi: 10.1179/2045772312Y.0000000003 (2012). 5. King, C. E. et al. Brain-computer interface driven functional electrical stimulation system for overground walking in spinal cord injury participant. In Conf Proc IEEE Eng Med Biol Soc.2015/01/09 edn 1238–1242. Conf Proc IEEE Eng Med Biol Soc, Chicago, IL, August 26–30, 2014. 6. King, C. E. et al. The feasibility of a brain-computer interface functional electrical stimulation system for the restoration of overground walking after paraplegia. J Neuroeng Rehabil 12, 80, doi: 10.1186/s12984-015-0068-7 (2015). 7. Louie, D. R., Eng, J. J. & Lam, T. Gait speed using powered robotic exoskeletons after spinal cord injury: a systematic review and correlational study. J Neuroeng Rehabil 12, 82, doi: 10.1186/s12984-015-0074-9 (2015). 8. Mehrholz, J., Kugler, J. & Pohl, M. Locomotor training for walking after spinal cord injury. Cochrane Database Syst Rev, CD006676, doi: 10.1002/14651858.CD006676.pub2 (2008). 9. Chapin, J. K., Moxon, K. A., Markowitz, R. S. & Nicolelis, M. A. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat. Neurosci. 2, 664–670, doi: 10.1038/10223 (1999). 10. Carmena, J. M. et al. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS biology 1, E42, doi: 10.1371/journal.pbio.0000042 (2003). 11. Wessberg, J. et al. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408, 361–365, doi: 10.1038/35042582 (2000). 12. Collinger, J. L. et al. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381, 557–564, doi: 10.1016/ S0140-6736(12)61816-9 (2013). 13. Hochberg, L. R. et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171, doi: 10.1038/nature04970 (2006). 14. Patil, P. G., Carmena, J. M., Nicolelis, M. A. & Turner, D. A. Ensemble recordings of human subcortical neurons as a source of motor control signals for a brain-machine interface. Neurosurgery 55, 27–35 discussion 35–28 (2004). 15. Nicolelis, M. A. Brain-machine interfaces to restore motor function and probe neural circuits. Nat Rev Neurosci 4, 417–422, doi: 10.1038/nrn1105 (2003). 16. McKay, W. B., Lim, H. K., Priebe, M. M., Stokic, D. S. & Sherwood, A. M. Clinical neurophysiological assessment of residual motor control in post-spinal cord injury paralysis. Neurorehabil Neural Repair 18, 144–153, doi: 10.1177/0888439004267674 (2004). 17. Sherwood, A. M., Dimitrijevic, M. R. & McKay, W. B. Evidence of subclinical brain influence in clinically complete spinal cord injury: discomplete SCI. J. Neurol. Sci. 110, 90–98 (1992). 18. Kakulas, B. A., Lorimer, R. L. & Gubbay, A. D. In Spinal Cord Monitoring. Basic Principles, Regeneration, Pathophysiology and Clinical Aspects (eds Stalberg, E., Sharma, H. R. & Olsson, Y.) Ch. 395–407, (Springer Vienna, 1998). 19. Hidler, J. et al. ZeroG: overground gait and balance training system. J. Rehabil. Res. Dev. 48, 287–298 (2011). 20. Benito-Penalva, J. et al. Gait training in human spinal cord injury using electromechanical systems: effect of device type and patient characteristics. Arch. Phys. Med. Rehabil. 93, 404–412, doi: 10.1016/j.apmr.2011.08.028 (2012). 21. Eng, J. J. et al. Use of prolonged standing for individuals with spinal cord injuries. Phys. Ther. 81, 1392–1399 (2001). 22. Hoekstra, F. et al. Effect of robotic gait training on cardiorespiratory system in incomplete spinal cord injury. J. Rehabil. Res. Dev. 50, 1411–1422, doi: 10.1682/JRRD.2012.10.0186 (2013). 23. Ditunno, J. F. Jr., Young, W., Donovan, W. H. & Creasey, G. The international standards booklet for neurological and functional classification of spinal cord injury. American Spinal Injury Association. Paraplegia 32, 70–80, doi: 10.1038/sc.1994.13 (1994). 24. Bell-Krotoski, J. “Pocket filaments” and specifications for the semmes-weinstein monofilaments. J. Hand Ther. 3, 26–31 (1990). 25. Bolliger, M., Banz, R., Dietz, V. & Lunenburger, L. Standardized voluntary force measurement in a lower extremity rehabilitation robot. J Neuroeng Rehabil 5, 23, doi: 10.1186/1743-0003-5-23 (2008). 26. Lünenburger, L., Colombo, G., Riener, R. & Volker, D. Clinical Assessments Performed during Robotic Rehabilitation by the Gait Training Robot Lokomat In IEEE 9th International Conference on Rehabilitation Robotics. 345–348. Chicago, IL, June 28 -July 1, 2005. 27. Pastre, C. B. et al. Validation of the Brazilian version in Portuguese of the Thoracic-Lumbar Control Scale for spinal cord injury. Spinal Cord 49, 1198–1202, doi: 10.1038/sc.2011.86 (2011). 28. Morganti, B., Scivoletto, G., Ditunno, P., Ditunno, J. F. & Molinari, M. Walking index for spinal cord injury (WISCI): criterion validation. Spinal Cord 43, 27–33, doi: 10.1038/sj.sc.3101658 (2005). 29. Catz, A. et al. A multicenter international study on the Spinal Cord Independence Measure, version III: Rasch psychometric validation. Spinal Cord 45, 275–291, doi: 10.1038/sj.sc.3101960 (2007). 30. Melzack, R. The McGill pain questionnaire: from description to measurement. Anesthesiology 103, 199–202 (2005).
Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
14
50
www.nature.com/scientificreports/ 31. Carlsson, A. M. Assessment of chronic pain. I. Aspects of the reliability and validity of the visual analogue scale. Pain 16, 87–101 (1983). 32. Wewers, M. E. & Lowe, N. K. A critical review of visual analogue scales in the measurement of clinical phenomena. Res. Nurs. Health 13, 227–236 (1990). 33. Paternostro-Sluga, T. et al. Reliability and validity of the Medical Research Council (MRC) scale and a modified scale for testing muscle strength in patients with radial palsy. J Rehabil Med 40, 665–671, doi: 10.2340/16501977-0235 (2008). 34. Bohannon, R. W. & Smith, M. B. Interrater reliability of a modified Ashworth scale of muscle spasticity. Phys. Ther. 67, 206–207 (1987). 35. Riener, R., Brunschweiler, A., Lünenburger, L. & Colombo, G. In 9th Annual Conference of the International FES Society. p. 287–289. 36. Jang, Y., Hsieh, C. L., Wang, Y. H. & Wu, Y. H. A validity study of the WHOQOL-BREF assessment in persons with traumatic spinal cord injury. Arch. Phys. Med. Rehabil. 85, 1890–1895 (2004). 37. Rosenberg, M. Society and the adolescent self-image. (Princeton University Press, 1965). 38. Beck, A. T., Ward, C. H., Mendelson, M., Mock, J. & Erbaugh, J. An inventory for measuring depression. Arch. Gen. Psychiatry 4, 561–571 (1961). 39. Cardoso, J. F. High-order contrasts for independent component analysis. Neural Comput. 11, 157–192 (1999). 40. Demirel, G., Yllmaz, H., Gencosmanoglu, B. & Kesiktas, N. Pain following spinal cord injury. Spinal Cord 36, 25–28 (1998). 41. Pfurtscheller, G., Brunner, C., Schlogl, A. & Lopes da Silva, F. H. Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks. Neuroimage 31, 153–159, doi: 10.1016/j.neuroimage.2005.12.003 (2006). 42. Pfurtscheller, G., Neuper, C., Andrew, C. & Edlinger, G. Foot and hand area mu rhythms. Int. J. Psychophysiol. 26, 121–135 (1997). 43. Jasper, H. & Penfield, W. Electrocorticograms in man: Effect of voluntary movement upon the electrical activity of the precentral gyrus. Arch. Psychiatr. Nervenkr. 183, 163–174 (1949). 44. Pfurtscheller, G. & Lopes da Silva, F. H. Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin. Neurophysiol. 110, 1842–1857 (1999). 45. Pfurtscheller, G., Neuper, C., Flotzinger, D. & Pregenzer, M. EEG-based discrimination between imagination of right and left hand movement. Electroencephalogr. Clin. Neurophysiol. 103, 642–651 (1997). 46. Neuper, C. & Pfurtscheller, G. Event-related dynamics of cortical rhythms: frequency-specific features and functional correlates. Int. J. Psychophysiol. 43, 41–58 (2001). 47. Aflalo, T. et al. Neurophysiology. Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Science 348, 906–910, doi: 10.1126/science.aaa5417 (2015). 48. Hochberg, L. R. et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372–375, doi: 10.1038/nature11076 (2012). 49. Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B. & Scadden, L. Vision substitution by tactile image projection. Nature 221, 963–964 (1969). 50. Shokur, S., Gallo, S., Moioli, R. C., Donati, A. R. C., Morya, E., Bleuler, H. & Nicolelis, M. A. Assimilation of virtual legs and perception of floor texture by complete paraplegic patients receiving artificial tactile feedback. Sci. Rep. In press. (2016). 51. Calford, M. B. & Tweedale, R. Acute changes in cutaneous receptive fields in primary somatosensory cortex after digit denervation in adult flying fox. J. Neurophysiol. 65, 178–187 (1991). 52. Nicolelis, M. A., Lin, R. C., Woodward, D. J. & Chapin, J. K. Induction of immediate spatiotemporal changes in thalamic networks by peripheral block of ascending cutaneous information. Nature 361, 533–536, doi: 10.1038/361533a0 (1993). 53. McDonald, J. W. et al. Late recovery following spinal cord injury. Case report and review of the literature. J. Neurosurg. 97, 252–265 (2002). 54. Belci, M., Catley, M., Husain, M., Frankel, H. L. & Davey, N. J. Magnetic brain stimulation can improve clinical outcome in incomplete spinal cord injured patients. Spinal Cord 42, 417–419, doi: 10.1038/sj.sc.3101613 (2004). 55. Benito, J. et al. Motor and gait improvement in patients with incomplete spinal cord injury induced by high-frequency repetitive transcranial magnetic stimulation. Top Spinal Cord Inj Rehabil 18, 106–112, doi: 10.1310/sci1802-106 (2012). 56. Kuppuswamy, A. et al. Action of 5 Hz repetitive transcranial magnetic stimulation on sensory, motor and autonomic function in human spinal cord injury. Clin. Neurophysiol. 122, 2452–2461, doi: 10.1016/j.clinph.2011.04.022 (2011). 57. Tazoe, T. & Perez, M. A. Effects of repetitive transcranial magnetic stimulation on recovery of function after spinal cord injury. Arch. Phys. Med. Rehabil. 96, S145–155, doi: 10.1016/j.apmr.2014.07.418 (2015). 58. Angeli, C. A., Edgerton, V. R., Gerasimenko, Y. P. & Harkema, S. J. Altering spinal cord excitability enables voluntary movements after chronic complete paralysis in humans. Brain 137, 1394–1409, doi: 10.1093/brain/awu038 (2014). 59. Fawcett, J. W. et al. Guidelines for the conduct of clinical trials for spinal cord injury as developed by the ICCP panel: spontaneous recovery after spinal cord injury and statistical power needed for therapeutic clinical trials. Spinal Cord 45, 190–205, doi: 10.1038/ sj.sc.3102007 (2007). 60. Bracken, M. B. et al. Methylprednisolone or tirilazad mesylate administration after acute spinal cord injury: 1-year follow up. Results of the third National Acute Spinal Cord Injury randomized controlled trial. J. Neurosurg. 89, 699–706, doi: 10.3171/ jns.1998.89.5.0699 (1998). 61. Geisler, F. H., Coleman, W. P., Grieco, G. & Poonian, D. The Sygen multicenter acute spinal cord injury study. Spine (Phila Pa 1976) 26, S87–98 (2001). 62. Tadie, M. et al. Early care and treatment with a neuroprotective drug, gacyclidine, in patients with acute spinal cord injury. Rachis 15, 363–376 (2003). 63. Kirshblum, S., Millis, S., McKinley, W. & Tulsky, D. Late neurologic recovery after traumatic spinal cord injury. Arch. Phys. Med. Rehabil. 85, 1811–1817 (2004). 64. Oleson, C. V., Burns, A. S., Ditunno, J. F., Geisler, F. H. & Coleman, W. P. Prognostic value of pinprick preservation in motor complete, sensory incomplete spinal cord injury. Arch. Phys. Med. Rehabil. 86, 988–992, doi: 10.1016/j.apmr.2004.09.031 (2005). 65. Wolpaw, J. R., McFarland, D. J., Neat, G. W. & Forneris, C. A. An EEG-based brain-computer interface for cursor control. Electroencephalogr. Clin. Neurophysiol. 78, 252–259 (1991). 66. Ifft, P. J., Shokur, S., Li, Z., Lebedev, M. A. & Nicolelis, M. A. A brain-machine interface enables bimanual arm movements in monkeys. Science translational medicine 5, 210ra154, doi: 10.1126/scitranslmed.3006159 (2013). 67. O’Doherty, J. E. et al. Active tactile exploration using a brain-machine-brain interface. Nature 479, 228–231, doi: 10.1038/ nature10489 (2011). 68. Grillner, S., Wallen, P., Saitoh, K., Kozlov, A. & Robertson, B. Neural bases of goal-directed locomotion in vertebrates–an overview. Brain Res Rev 57, 2–12, doi: 10.1016/j.brainresrev.2007.06.027 (2008). 69. Rossignol, S. & Frigon, A. Recovery of locomotion after spinal cord injury: some facts and mechanisms. Annu. Rev. Neurosci. 34, 413–440, doi: 10.1146/annurev-neuro-061010-113746 (2011). 70. Dietz, V. Spinal cord pattern generators for locomotion. Clin. Neurophysiol. 114, 1379–1389 (2003). 71. Scivoletto, G. et al. Plasticity of spinal centers in spinal cord injury patients: new concepts for gait evaluation and training. Neurorehabil Neural Repair 21, 358–365, doi: 10.1177/1545968306295561 (2007). 72. Maravita, A., Spence, C. & Driver, J. Multisensory integration and the body schema: close to hand and within reach. Curr. Biol. 13, R531–539 (2003).
Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
15
51
www.nature.com/scientificreports/ 73. Shimojo, S. & Shams, L. Sensory modalities are not separate modalities: plasticity and interactions. Curr. Opin. Neurobiol. 11, 505–509 (2001). 74. Ernst, M. O. & Bulthoff, H. H. Merging the senses into a robust percept. Trends Cog. Sci. 8, 162–169, doi: 10.1016/j.tics.2004.02.002 (2004). 75. Fiorio, M. & Haggard, P. Viewing the body prepares the brain for touch: effects of TMS over somatosensory cortex. Eur. J. Neurosci. 22, 773–777, doi: 10.1111/j.1460-9568.2005.04267.x (2005). 76. Taylor-Clarke, M., Kennett, S. & Haggard, P. Vision modulates somatosensory cortical processing. Curr. Biol. 12, 233–236 (2002). 77. Tipper, S. P. et al. Vision influences tactile perception without proprioceptive orienting. Neuroreport 9, 1741–1744 (1998). 78. Kennett, S., Taylor-Clarke, M. & Haggard, P. Noninformative vision improves the spatial resolution of touch in humans. Curr. Biol. 11, 1188–1191 (2001). 79. Taylor-Clarke, M., Kennett, S. & Haggard, P. Persistence of visual-tactile enhancement in humans. Neurosci. Lett. 354, 22–25 (2004). 80. Moseley, G. L., Parsons, T. J. & Spence, C. Visual distortion of a limb modulates the pain and swelling evoked by movement. Curr. Biol. 18, R1047–1048, doi: 10.1016/j.cub.2008.09.031 (2008). 81. Jackson, A. & Zimmermann, J. B. Neural interfaces for the brain and spinal cord–restoring motor function. Nat Rev Neurol 8, 690–699, doi: 10.1038/nrneurol.2012.219 (2012). 82. Shokur, S. et al. Expanding the primate body schema in sensorimotor cortex by virtual touches of an avatar. Proc. Natl. Acad. Sci. USA. 110, 15121–15126, doi: 10.1073/pnas.1308459110 (2013).
Acknowledgements
First we would like to thank the patients for their contribution to this research. We also want to thank Neiva Paraschiva, Andrea Arashiro, Maria Cristina Boscaratto (AASDAP, Associação Alberto Santos Dumont para Apoio à Pesquisa) Susan Halkiotis (Duke University), Lumy Sawaki Adams (Department Physical Medicine and Rehabilitation, University of Kentucky HealthCare, Cardinal Hill Rehabilitation Hospital), Dora Fischer, Guilhaume Bao, Nicole Peretti, Kyle Fast for their work, commitment and support during this study. We acknowledge the clinical support of Marcelo de Jesus Justino Ares, Alice Conceição Rosa Ramos, Adriana Rosa Lovisotto Cristante, Isolda Araújo, Juliana Campos, Denise Yoshihara, Márcia Bellas Santos and Luana Ferreira (AACD, Associação de Assistência à Criança Deficiente, São Paulo, Brazil). We thank Ramzi Sellaouti, Roberto Dinis, Elmira Amrollah, Nadhir Dhaouadi for the technical support (BIA). This study was funded by grants from the Brazilian Financing Agency for Studies and Projects (FINEP 01·12·0514·00), Brazilian Ministry of Science, Technology and Innovation (MCTI), and the Itaú Unibanco S.A. We also acknowledge the support from the National Institute of Science and Technology (INCT) Brain Machine-Interface (INCEMAQ) CNPq 573966/2008-7 of Brazilian Ministry of Science, Technology and Innovation (MCTI/CNPq/FNDCT/CAPES/ FAPERN).
Author Contributions
A.R.C.D. and S.S.H. equally contributed to study design, data interpretation, data collection, data analysis, literature search, figure, tables, writing and editing. E.M., D.S.F.C., R.C.M., C.G.P. and C.M.G. contributed to study design, data collection, data interpretation and data analysis. P.B.A., S.T. and G.A.P. contributed to literature search, data collection and data analysis. F.L.B. and S.G. contributed to study design and data collection. A.L.L., A.K.T. and M.A.A. contributed to data collection and data analysis. H.B., S.J., G.C. and A.R. contributed to study design, technical developments, and setup testing. M.A.L.N. proposed the key neurorehabilitation concept tested in this study, defined the clinical scope of the study, and contributed to study design, data interpretation, data collection, data analysis, writing the manuscript, designing figures, and manuscript editing.
Additional Information
Supplementary information accompanies this paper at http://www.nature.com/srep Competing financial interests: The authors declare no competing financial interests. How to cite this article: Donati, A. R. C. et al. Long-Term Training with a Brain-Machine Interface-Based Gait Protocol Induces Partial Neurological Recovery in Paraplegic Patients. Sci. Rep. 6, 30383; doi: 10.1038/ srep30383 (2016). This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ © The Author(s) 2016
Scientific Reports | 6:30383 | DOI: 10.1038/srep30383
16
52
www.nature.com/scientificreports
OPEN
received: 10 May 2016 accepted: 04 August 2016 Published: 19 September 2016
Assimilation of virtual legs and perception of floor texture by complete paraplegic patients receiving artificial tactile feedback Solaiman Shokur1, Simone Gallo2, Renan C. Moioli3,4, Ana Rita C. Donati1,5, Edgard Morya3,4, Hannes Bleuler2 & Miguel A.L. Nicolelis1,3,4,6,7,8,9 Spinal cord injuries disrupt bidirectional communication between the patient’s brain and body. Here, we demonstrate a new approach for reproducing lower limb somatosensory feedback in paraplegics by remapping missing leg/foot tactile sensations onto the skin of patients’ forearms. A portable haptic display was tested in eight patients in a setup where the lower limbs were simulated using immersive virtual reality (VR). For six out of eight patients, the haptic display induced the realistic illusion of walking on three different types of floor surfaces: beach sand, a paved street or grass. Additionally, patients experienced the movements of the virtual legs during the swing phase or the sensation of the foot rolling on the floor while walking. Relying solely on this tactile feedback, patients reported the position of the avatar leg during virtual walking. Crossmodal interference between vision of the virtual legs and tactile feedback revealed that patients assimilated the virtual lower limbs as if they were their own legs. We propose that the addition of tactile feedback to neuroprosthetic devices is essential to restore a full lower limb perceptual experience in spinal cord injury (SCI) patients, and will ultimately, lead to a higher rate of prosthetic acceptance/use and a better level of motor proficiency. Spinal cord injuries can induce significant bidirectional loss of communication between the subject’s brain and his/her body, i.e. the patient can neither generate body movements as a result of the loss of cortical communication with the spinal cord, nor can their brain receive somatosensory feedback originating in the body’s periphery1. For the past 15 years, studies in animals2,3 and patients4 have suggested that brain-machine interfaces (BMI) may provide a novel therapeutic alternative to restore mobility in severely paralyzed patients5. Yet, despite the high prevalence of concurrent somatosensory and motor deficits in such patients, few BMI studies have aimed at restoring both motor and somatosensory feedback simultaneously in paralyzed subjects3,6. In the absence of tactile feedback, BMI users have to rely mainly on vision to enact their direct brain control of an artificial actuator. Since in these cases the user does not receive any tactile or proprioceptive feedback related to the actuator performance, we and others have argued that such BMI setups are unlikely to have a significant clinical impact on severely impaired patients4. This is easy to understand when one realizes that a patient would have to look at his prosthetic hand every time he moves it to reach and hold an object, or look at the floor every time his lower limb exoskeleton touches the ground during autonomous locomotion. As part of a project to develop and test a non-invasive, brain-controlled lower body exoskeleton for SCI patients, we implemented a new rehabilitation paradigm in which patients interacted with a rich immersive virtual reality environment7 (VRE). In this VRE, patients interacted with the movements of 3D virtual avatar legs on different surfaces and environments, through a virtual reality head mounted display (Fig. 1a). During this task, 1
Neurorehabilitation Laboratory, Associação Alberto Santos Dumont para Apoio à Pesquisa (AASDAP), São Paulo, Brazil. 2STI IMT, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland. 3Edmond and Lily Safra International Institute of Neuroscience, Santos Dumont Institute, Macaiba, Brazil. 4Alberto Santos Dumont Education and Research Institute, São Paulo, Brazil. 5Associação de Assistência à Criança Deficiente (AACD), São Paulo, Brazil. 6Department of Neurobiology, Duke University, Durham, NC, USA. 7Department of Biomedical Engineering, Duke University, Durham, NC, USA. 8Department of Psychology and Neuroscience, Duke University, Durham, NC, USA. 9Center for Neuroengineering, Duke University, Durham, NC, USA. Correspondence and requests for materials should be addressed to M.A.L.N. (email: nicoleli@neuro.duke.edu) Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
1
53
www.nature.com/scientificreports/
Figure 1. Tactile shirt and pseudo proprioception results. (a) Subject wearing the tactile shirt. Three vibrators are aligned on ulna bone. Detachable vibrators are connected to a PCB converted by cloth and Velcro bands. Details of the vibrator modules are shown: the clipping systems are used both as mechanical clipping and electrical connectors (power + data). (b) Two different tactile paradigms are proposed: feedback on stance (tactile feedback rolls on the forearm of the patient in synchrony with the foot contact) or swing (feedback roll during the swing and all three vibrators at once on foot contact). (c) Results given for the experiment evaluating the pseudo proprioception test with four different tactile feedbacks (left to right): 1) feedback on stance moving from wrist to elbow (Distal to Proximal, DtP), 2) feedback on stance moving from elbow to wrist (Proximal to Distal, PtD), 3) on swing DtP and 4) on swing PtD. Patients used their arms to show the position of the walking avatar’s legs, relying on tactile feedback only. Scores based on ability to reproduce the avatar walk for different speeds are reported including the number of patients for whom a certain paradigm was the best or the worst choice and the rank of the four paradigms.
patients also wore a haptic display device integrated in the long sleeves of a shirt (Fig. 1a). Each of the sleeves contained three small coin-shaped vibrators lined up along the forearm’s distal-proximal axis. By varying the magnitude and temporal sequence of activation of these vibrators, we were able to deliver somatosensory information, originating from the movements of virtual limbs observed by the eight SCI patients, onto the skin surface of their forearms.. Our central goal was to allow these SCI patients to use this haptic display in order to sense the position of the virtual legs in space, the contact of the virtual foot with the floor, and the type of surface with which they were in contact. In this context, the tactile paradigm presented here is at the cross-roads of two major concepts: sensory substitution and sensory remapping. Sensory substitution consists of using one sensory modality to carry information from another one8. This can be seen in the classic studies of Bach-y-Rita in which visually impaired patients learned to use a matrix of vibrators to translate visual information into tactile feedback delivered on the skin of the back8 or on the tongue’s surface9. Haptic displays have also been used to mitigate audition10 or vestibular deficits11,12 among others. On the other hand, sensory remapping techniques (for example those used with amputee patients) aim at providing somatotopically matching haptic stimulation onto a different body part13. Here, we implemented a paradigm that includes: (a) a non-invasive tactile remapping, obtained by providing vibrotactile14 stimulation on the patient’s forearms to inform about lower limb tactile and proprioceptive
Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
2
54
www.nature.com/scientificreports/ information during a virtual gait/walking task; and (b) a tactile substitution procedure in which complex haptic cues (tactile and proprioceptive information from lower limbs) were replaced with simpler encoded tactile stimuli. Using this apparatus, SCI patients were able to receive both the tactile feedback providing temporal and physical cues about virtual feet interacting with different floor surfaces, and the proprioceptive feedback describing the position in space of virtual lower limbs. Analysis of the patients’ performance in a crossmodal congruency task15 revealed that subjects experienced an assimilation of the virtual legs as a projection of their own body.
Results
All experiments were performed with naïve SCI patients (Table S1) using an immersive virtual reality system where the subjects’ lower limbs were simulated by a human-like 3D avatar seen from the first person’s perspective through a head mounted display (Fig. 1a). Tactile and proprioceptive sensations generated by the avatar’s virtual legs were mapped on the patients’ forearms by means of arrays of vibrators that defined a haptic display (Fig. 1a). Initially, we were interested in identifying the most intuitive tactile feedback paradigm for SCI patients to perceive the position of their legs when relying on tactile feedback only. Next, we analyzed how the avatar legs were assimilated into the patients’ brain body representation using the same apparatus. Lastly, we investigated the parameters of tactile stimulation that allowed our patients to experience the vivid sensation of walking on three different types of ground surface: sand, grass or paved street.
An Intuitive Representation of Leg Position Using a Forearm Haptic Display. We ran experiments with seven (P1-P7) patients. Here, we considered different tactile feedback modalities and searched for the one that patients felt was more intuitive for perceiving the position of the avatar leg when relying on the tactile feedback solely. Two main questions were asked: (a) Should the artificial feedback represent the rolling of the foot on the floor (feedback on stance, Fig. 1b) or the swinging of the leg when the body is balancing; and (b) Should the tactile stimulation be delivered from the wrist to the elbow (Distal to Proximal, DtP) or from the elbow to the wrist (Proximal to Distal, PtD). This was important as the artificial tactile feedback proposed here is an abstract substitution of the sensory feedback from the legs. Thus, the direction of the movement on the forearm can be perceived as representing different phases of the gait cycle. For example, during the stance phase, a PtD sequence of forearm vibration reproduces the same direction of movement produced by the foot rolling on the floor (i.e. perception of the foot going forward). Meanwhile, a DtP sequence moves in the same direction as the foot’s movement in relation to the subject’s body (perception of the body going forward compared to the foot during stance). To induce a movement sensation on the forearm while using an array of eccentric rotating mass (ERM) vibrators, we employed a well-documented illusion called the tactile apparent movement16,17. This illusion appears when two vibrating actuators are sequentially triggered on the subject’s skin with a specific stimulation length and specific delay between the onset of the two vibrations. As a result, the subject perceives one continuous touch going from the first to second vibrator instead of two discrete stimuli. The parameters of stimulation length and onset delay inducing the apparent movement illusion were calculated for each patient (see Supplementary Methods, Fig. S1). During the experiment, patients were asked to report the position of the avatar leg relying on tactile feedback only. The avatar’s walk was randomly changed between three speeds (Fig. S2a). A score, ranging from 0 to 3, was given to evaluate a patient’s performance for each speed: 0 - if the patient could not report the position of the avatar leg; 1 - if patients could report the position at constant speed; 2 - if they could report the position when the speed was changed, but confused the left and right leg; and 3 - for a perfect execution of the task. Final score was the sum of scores for the three speeds and thus ranged between 0 and 9. Three out of seven patients accurately reported the leg position at every speed when the tactile feedback was of type Stance-Distal to Proximal (DtP), three had good performance (score range 5–8), and one was unable to report the leg position (Fig. 1c, Table S2 for details). For the Stance-Proximal to Distal (PtD) paradigm, three patients had good or perfect performances whereas the other four had average to bad performance (score range 1–4). Three patients were perfect with the Swing-Distal to Proximal (DtP) tactile paradigm, two were good and the last two were average or bad. Finally, for Swing-Proximal to Distal (PtD) paradigm, four patients exhibited average performance, one was good and two were perfect. We observed that four patients had their highest score with Stance-DtP, one reached maximal performance with Stance-PtD, three with Swing-DtP and two with Swing-PtD. Not every patient had his/her best performance with the same tactile paradigm. However, we wanted to use only one type of tactile feedback for the whole group for all tests with the tactile shirt. Therefore, we identified the type of feedback that elicited the best performance in a majority of our subjects. Using a simple metric ranking (see Methods), we classified the four paradigms as follows (from best to worst): Stance-DtP, Swing-DtP, Swing-PtD and Stance-PtD (Fig. 1c). Overall, tactile stimulation going from Distal to Proximal was more intuitive for feedback delivered during both the avatar leg Stance and Swing phases. At the end of the experiment, patients were also asked to report their subjective experience (Fig. S2b). All patients answered that they ‘agreed’ or ‘totally agreed’ that ‘the tactile feedback [matched] the walk’ when the tactile feedback was on Stance-DtP, Swing-DtP, and Swing-PtD. For Stance-PtD, one patient ‘disagreed’ that feedback matched the walk. No patient reported that ‘The tactile feedback was disturbing’ for any of the four feedback modalities. All patients ‘agreed’ or ‘totally agreed’ they ‘could imagine/feel [themselves] walking well with [their] eyes closed, and [they] knew where the avatar legs were’ for Stance-DtP, Swing-DtP, and Swing-PtD, while one patient disagreed with that statement during the Stance-PtD modality (Fig. S2b). Considering both the behavioral assessment and the responses to the questionnaire, it appeared that feedback on Stance, with a sequence of vibrations going from wrist to elbow (DtP) was the best feedback at the group level Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
3
55
www.nature.com/scientificreports/
Figure 2. Crossmodal congruency task. (a) Experimental setup, subject wore the tactile shirt and the head mounted display. Vibrators were placed on patients’ forearms and visual distractors on the virtual avatar’s toe and heel. (b) A cross congruent task (CCT) block is run after a Vision only block (VO: 1 minute observation of the avatar walking) or a VT block (same as VO with tactile feedback on the stance). After a random sequence of 2 VO and 2 VT blocks, we ended the session with another sequence of CCT block following a VO or a VT block (c) Mean and standard error for the Cross Congruent Effect (CCE). Yellow bar shows average response time for trials where tactile stimulation and visual distractors were on the same arm and pink bar shows those where they were contralateral. P values for T-test are shown.
to deliver tactile feedback to our patients using our forearm haptic display. Note that during the experiments patients were never told what the vibrotactile pattern represented (if it was on Stance or on Swing, etc.). This allowed them to make bias-free connections between the vibrotactile stimulation and the avatar’s walk.
Incorporation of Virtual Legs with Visuo-Tactile Stimulation. After we identified the most intuitive protocol for delivering tactile feedback, we investigated the change in the patients’ perception of the avatar legs in relation to their own bodies while using our haptic display. We used a psychophysical measurement while tactile feedback (of type Stance-DtP) was synchronized with the avatar walk (Vision + Tactile, VT). We compared this condition to the case where tactile feedback was absent and only visual feedback was provided (Vision only, VO). We employed the Crossmodal Congruency Task (CCT)15,18,19, a well-documented indirect measurement of tool incorporation by human subjects, and the corresponding Cross Congruent Effect (CCE), which calculates the interaction between a vibration on the body and a visual distractor on the tip of the tool (in our case the avatar feet), to measure how the virtual avatar legs were assimilated by our SCI patients (high CCE correlates with integration of the tool; see Experimental Procedures). All our patients were able to discriminate the position of the vibrators used in this experiment (Fig. S3). For all patients, at the beginning of the session, we ran a CCT block succeeding a first Vision Only block (VO1) and a CCT block succeeding a Vision + Tactile block (VT1) (Fig. 2a,b). CCE for the same side and CCE Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
4
56
www.nature.com/scientificreports/
VO1 Beginning of the session VT1 VO4 End of the session VT4
Congruent
Incongruent
Mean CCE
Diff side
787.1 ± 18.6
828.1 ± 19.9
41.0 ± 8.0
Same side
755.4 ± 20.8
789.3 ± 19.7
33.9 ± 8.3
Diff side
807.8 ± 18.4
804.4 ± 20.6
−3.4 ± 8.1
Same side
748.1 ± 17.2
855.8 ± 24.4
107.6 ± 8.4
Diff side
820.7 ± 22.2
855.1 ± 22.0
34.4 ± 9.1
Same side
762.4 ± 15.5
829.7 ± 21.2
67.4 ± 7.3
Diff side
815.9 ± 24.5
837.8 ± 21.9
21.9 ± 9.0
Same side
792.0 ± 19.8
861.5 ± 23.4
69.5 ± 8.3
Table 1. Mean Response Cross Congruent Effect over all patients. Mean ± standard error of reaction time (in ms) over all eight patients for blocks succeeding a Vision Only (VO) block or blocks succeeding a Vision + Tactile (VT) block. Mean CCE corresponds to the mean RT for congruent – mean RT for incongruent visuo-tactile simulation is show in the right- most columns.
for different sides refer to the cases where the visual distractors and the vibration are on the same arm or on the contralateral arm. CCE(Mn) refers to the Cross Congruent effect recorded right after the nth block of a modality M. The mean reaction times (RTs), for all conditions over all patients, are shown in Table 1. Mean CCE(VO1), calculated as the RT difference between trials where the tactile stimulation on the forearm and the visual stimulation on the avatar foot had incongruent elevations (e.g. vibration on the proximal vibrator and visual stimuli on the tip of the foot) and trials where they were congruent, was equal to 41 ± 8 ms (±SEM) when the vibrator and distractor were on a different side (incongruent: 828 ± 20[ms], congruent:787 ± 19 [ms]) and 33 ± 8 ms for the same side (incon: 755 ± 21 [ms], cong: 787 ± 18 [ms]). We found that CCE(VT1) was −3 ± 8 ms for different sides (incon: 804 ± 21 [ms], cong: 807 ± 18 [ms]) versus 107 ± 8 ms for same side (incon: 855 ± 24 [ms], cong: 748 ± 17 [ms]). Three-way analysis of variance (ANOVA) with the factor congruency (congruent/incongruent), side (same side/different side) and modality (VT/VO) on RT, showed a significant effect of congruency (P < 0.01), and a significant effect of the interaction between side and congruency (P < 0.05). CCE was higher for same side than for different side delivery of distractor and tactile stimulation. We also found a three-way interaction between congruency, side and modality (P < 0.05). A post-hoc analysis with Bonferroni correction was run to determine the difference between CCE(VT1) and CCE(VO1). CCE was significantly higher for same side compared to different sides (P < 0.01) for VT modality. No difference was found for CCE(VO1) between the same and different sides (P > 0.5) (Fig. 2c). In other words, we found a significant CCE effect (used as the indirect measurement of the avatar leg assimilation) after the first VT block and no such effect was observed right after the first VO block. Interestingly, when we repeated two more blocks of VO and VT (VO2, VO3 and VT2, VT3 randomly sequenced, Fig. 2b), and then ran CCE(VO4) and CCE(VT4), we observed different dynamics (see RTs in Table 1). The mean RT difference of CCE(VO4) was 34.4 ± 9.1 ms for different sides and 67.4 ± 7.3 ms for the same side; CCE(VT4) was 21.9 ± 9.0 ms for different sides and 69.5 ± 8.301 ms for the same side. Significant differences between the same side and different sides were found both in CCE(VT4) (P < 0.001) and CCE(VO4) (P < 0.05). Simply stated, there was a significant increase in RT during the CCE applied after a VO block at the end of the session; something that did not happen at the beginning of the session. Altogether, these results suggest that patients extended their body representation to integrate the avatar legs after using the visuo-tactile stimulation.
Simulation of Floor Texture. Patients explored and selected 120 times per session the set of tactile param-
eters that best represented for each of them the sensation of walking on three virtual ground textures labeled as: sand (SAT), grass (GRT) and paved street (PST) (exploratory phase). In other words, while seeing through the head mounted display, a black control surface or the images of three distinct ground surfaces (sand, grass, and pavement) upon which the avatar was walking, patients received a variety of tactile feedback patterns on their forearms. Four factors of the tactile feedback were varied to create a catalog of perceived tactile sensations: amplitude of the distal, middle and proximal vibrators (DV, MV and PV) and the stimulation timing (ST) (Fig. 3a–d). See Floor Texture Simulation Test section in Methods for a description of the procedure to explore the full catalog of sensations presented to the patient. This exploratory session lasted for 40–45 minutes. At the end of the session, a complete catalog correlating tactile stimulation patterns and ground surfaces was obtained for each patient. After a 20-minute break, we removed the floor from the virtual reality simulation and begin to replay the set of chosen tactile stimulation parameters in a random order, using the same haptic display applied to the skin of the patients’ forearm. Now we asked the same patients to report the floor type to which each set of tactile stimulation parameters corresponded (inverse task phase). Figure 4a shows that in 11 out of 15 sessions, the patients’ accuracy rate was significantly above chance (mean + 2σ = 0.42). In other words, most patients were able to correctly describe the ground texture originally associated with a particular set of tactile parameters used to stimulate their forearm skin. Overall, 6 out of 8 patients performed this task above chance: patients P2, P3, P4, P6, and P7 were significantly above chance in both of their sessions; patient P5 was significantly better than chance in one of two sessions. Details of the patients’ performance per floor type and session are shown in Fig. S4a. Mean accuracy was significantly higher than chance for all three surfaces (Fig. 4b). However, patients were better at finding SAT than GRT (multiple comparisons, Bonferroni correction, N = 14, P < 0.01) and PST (P < 0.05).
Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
5
57
www.nature.com/scientificreports/
Figure 3.â&#x20AC;&#x201A; Floor texture simulation, setup. (a) Subjects wore the tactile shirt and the head mounted display; a 3D avatar was walking on either an empty floor or one of the three ground textures: Sand (SAT), Grass (GRT), and Paved Street (PST). (b) A tracking system detected the coordinates of the hand position and translated to a tactile modality defined by the amplitude of Proximal, Middle and Distal Vibrators (PV,MV,DV) and the Stimulation Timing (ST). (c) The axis of the four parameters describing the tactile feedback was randomized at each trial. Example of axis configuration and corresponding choices for SAT given by a subject. (d) Varying the Stimulation Timing (1 to 10) changes the Inter-Stimulus Onset Interval (ISOI) and Duration of Stimulation (DoS) parameters of the tactile feedback. It is possible to simulate the following sensation of touch: one single vibration on the whole forearm, a continuous feedback going from distal to proximal vibrator positions or three distinct vibrations on distal then middle and proximal forearm.
Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
6
58
www.nature.com/scientificreports/
Figure 4. Floor texture simulation, results. (a) During the inverse task the parameters chosen during the exploratory phase were replayed in a random order; subject had to report the floor type relying on the tactile feedback only (chance level = 0.42). (b) Patients’ accuracy per floor type (multiple comparisons, N = 14) and for trials with or without floor. (c) Mean answer over all patients and responses to Q1 (asked prior to the experiment): ‘I remember the sensation of my feet on X’ (X: SAT,GRT or PST, ‘I strongly disagree’ −2 to ‘I strongly agree’ +2). (d) Detail of patients responses for Q1. (e) Subjects’ responses to question ‘I had during the experiment the sensation of walking on X’ after a V only session (patients observed the avatar walking) and a Vision + tactile session (tactile feedback on stance) (*for P < 0.05, Wilcoxon test). (f) Detail of patients’ answers to question Q2. (g) Principal tactile factors of the experiment, ground types are classified (using knn classification) using each one of the four factors. Mean classification accuracy over all sessions for four factors (multiple-comparison test, N = 14, *: P < 0.05). (h) Mean and standard deviation for principal factors: Amplitude of proximal Vibration (PV) and Stimulation Timing (ST) for all sessions. Color convention as follows: red for SAT, green for GRT and blue for PST. (i) Values from panel h are joined following structure similarities (e.g. sessions where SAT was on top left corner and PST on the lower right). During the exploratory phase, in half of the trials, patients could see the avatar walking on a virtual ground that mimicked the one they were asked by the experimenter to identify (Fig. 3a), while in the other half of the trials no ground surface was visible under the avatar’s feet (no floor). No difference was found between trials in which the ground was shown and trials where the ground was absent from the virtual environment (T-test, P > 0.5) (Fig. 4b). While all patients suffered a spinal cord lesion at least 3 years (and some as much as 10 years) prior to the onset of the experiment (Table S1), all patients except P8 reported still having vivid memories of the sensation produced by the interaction of their feet with sand and grass (Fig. 4c,d). For the sensation of walking on a paved street, all patients - except P5 and P8 – also reported such memories. After the experiment, six out of eight patients reported that they experienced again the vivid sensation of walking on the three chosen ground surfaces (Fig. 4e,f ‘Vision + Tactile’). P5 and P8 were the only patients that did not report this sensation. Patients’ responses were translated into numerical grades (Fig. 4e, −2 for ‘I fully disagree’ to 2 for ‘I fully agree’). The group average (N = 8) was positive for all three conditions: 0.88 for SAT and GRT, 0.75 for PST. The patients’ responses were different if the same question was asked after a session where no tactile feedback was given (Fig. 4e,f Vision Only). In this control experiment, patients only observed the avatar walking, but did not receive tactile feedback. Patients’ answers were close to 0 for all three conditions when asked if they had the feeling of walking on the ground surfaces: paved street (0.25), grass (−0.13) and sand (−0.13). Differences between “Vision Only” and “Vision + Tactile” were significant for SAT and GRT (Fig. 4e, Wilcoxon test, P < 0.05, N = 8), but not for the PST. At the end of “Vision Only” sessions, patients were also asked
Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
7
59
www.nature.com/scientificreports/ Patient
Session
SAT vs. GRT
SAT vs. PST
1
1
1
2
2
1
***
2
2
***
3
1
3
2
4
1
4
2
5
1
5
2
6
1
6
2
7
1
7
2
8
1
GRT vs. PST
*** ***
***
***
*
*** *
*** ***
**
***
*** *
Table 2. ANOVA on Euclidian distance between chosen parameters for floor textures. For each pair of texture i and j we compared the distribution of the Euclidian distances of all combinations of trials for both textures to the distribution of Euclidian distance of trials of type i and trials of type j. We reported stars in the cell only when the distance between the textures was significantly bigger than the distance between trials for each texture taken individually (*for P < 0.05, **for P < 0.01, ***for P < 0.001).
to report on the visual realism of the different ground surfaces. No significant difference was found among the three surfaces (P = 0.26, ANOVA). The ‘sensation of walking on X’ after a vision only session was not correlated with the patients’ report of the visual realism of the ground surface (Fig. S4b) (R2 < 0.01), suggesting that the vision of the avatar and the interaction of the avatar with the ground are more important than the visual realism of the ground itself. Next, we analyzed the principal factors that patients relied on to differentiate the ground type (see Supplementary Methods). The stimulation timing (ST) was found to be the primary factor (Fig. 4g and S4c for details per patients) while the amplitude of the proximal vibrator (PV) came in second. Both were better than the middle (MD) and distal vibrator (DV, multiple comparisons, N = 14, P < 0.01). This suggests that patients relied on the tactile vibration level produced at the end of the avatar leg stance phase or the moment in which the foot pushed off the floor to move to a swing movement. Surprisingly, the patients seemed not to consider the vibration amplitude at the beginning of the stance (when the heel strikes the floor) as an important feature for discriminating the ground texture. We used these two main factors (PV and ST) to characterize the way patients chose each ground type. Figure 4h shows the parameters’ mean and standard deviation for all three surfaces. Similarities can be seen in patients 3, 5, 6, 7 and 8 (shown in Fig. 4i). For SAT, these patients expect three distinct stimuli during stance with a light feedback when the foot pushes off the ground. On the contrary, for paved street PST they were looking for one long stimulation with a strong proximal vibration. GRT was somehow in between sand and paved street, with a middle range vibration amplitude for PV. The probability of these similarities being due to chance was found to be very low (P < 0.001, see Supplementary Methods). Parameters chosen by the patients to describe the ground types were significantly different from each other for at least two textures in 10 out of 15 sessions (Table 2 and Supplementary Methods). The most common distinguishable textures were sand and paved street. No difference was found between trials where the floor was shown in the VR and those where it was not. Thus, the parameters selected by the patients to identify a given ground surface were not influenced by the presence of that ground in the virtual reality environment. We also noticed that sessions where patients chose the same factors for all textures during the exploratory phase, namely Patient 1, Sessions 1 and 2; Patient 5, Session 2; and Patient 8, Session 1, correspond to the sessions where the subjects were not able to differentiate the floor textures during the inverse task. P1 was the patient with the lowest score in the sensory discrimination task (Fig. S3), and also the one with the lowest score in the Stance-DtP paradigm during the pseudo-proprioceptive test.
Discussion
A novel solution to overcome sensory deficiency in the lower limbs for patients with Spinal Cord Injury (SCI) is proposed. In this study, missing haptic sensation from the lower limbs was replaced by rich tactile stimulation on the skin of SCI patients’ forearms. This feedback was integrated with an immersive virtual reality environment where a 3D human avatar was simulated. Three major effects of integration of the tactile feedback with the simulated 3D avatar were observed: (a) SCI patients could rely on this feedback to perceive the position of the virtual leg during locomotion; (b) patients incorporated the avatar legs as an extension of their body schema; and (c) patients experienced a realistic sensation of walking on different ground surfaces relying on the tactile feedback only. Interestingly, patients were never instructed on what the tactile feedback displayed or represented. As totally naïve subjects, they acquired vivid tactile/proprioceptive sensations after exposure to 1 minute of synchronous Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
8
60
www.nature.com/scientificreports/ visuo-tactile stimulation (patients observed the 3D human avatar walking and received the tactile stimulation on their forearm). This suggests that such tactile feedback became intuitive to the subjects rather quickly. As such, we propose that our haptic display paradigm was capable of inducing patients to experience a proprioceptive illusion that allowed them to deduce the position of the virtual avatar legs relying solely on the tactile feedback. Human perception is multimodal20 and different sensory inputs are merged and weighted in a statistically optimal fashion in order to reduce the variance of the final estimate21. Each sensory input has proficiencies: vision can provide precise spatial encoding, audition is more likely to accurately convert temporal features, while the haptic sense can perceive both spatial and temporal information with high precision21. For this reason and because it is applied directly on the body, haptic feedback is believed to be the most intuitive augmented sensory feedback to describe limb movements, as it provides intuitive spatiotemporal cues directly on the limb thus reducing the level of abstraction required from the user to understand the movement22. Here, a haptic display proved to be capable of mapping the tactile/proprioceptive cues, describing bipedal walking, onto the skin of the patients’ forearm. Patients learned very quickly to link the haptic feedback to the avatar walking movement. Multi-sensory training protocols have been reported to be more effective for learning23. Moreover, the benefits of a multi-sensory training phase are maintained further than when a single modality is used24. Similarly our protocol for integrating the tactile/proprioceptive feedback with an immersive virtual environment likely helped patients to internalize the relationship between a visually observed leg movement and the corresponding tactile stimulation in order to recall it once the visualization was removed. In this context, the emergence of this intuitive pseudo proprioception has an important implication for future use of brain controlled exoskeletons and other prosthetic devices, as the proprioceptive information is essential for controlling locomotion and other autonomous movements25,26. Walking generates large amounts of parallel multisensory information streams that are used in the fine control of the gait cycle. Among these inputs, haptic information is essential as proven by our ability to walk with our eyes closed and inability to walk when tactile feedback from the legs is disrupted27. The effects of haptic perception during walking are complex as illustrated by the change in walking patterns resulting from a reduction of the tactile sensitivity in the foot sole28. The importance of tactile feedback can also be appreciated when one examines the SCI patients’ perception of their own body. Sitting in a wheelchair after complete loss of sensory motor functions for several years changes the perception of one’s own body29. Experiments showed distortion of perceived body parts in SCI patients (they perceived a reduction in hip size29), as well as a decrease of body ownership of the legs and an increase in symptoms of depersonalization30. SCI patients were also found to consider their wheelchair as a ‘substitution’ of themselves or a part of their body31. These observations are not surprising as numerous experiments have shown that body representation in the human brain15,32 or primates33 modulates to incorporate additional limbs34, prosthetic limbs35,36, tools37, or virtual limbs18,38–40. The so called ‘rubber hand’ illusion41–43, which has been extensively studied, illustrates the plasticity of the brain representation of the body or body schema38,41. In the present work, we attempted to answer the following questions: do SCI patients incorporate virtual legs? Is tactile feedback necessary for this effect to appear, or is the vision of an avatar resembling a human body enough? In response, we found that the incorporation effect was very quick and robust. The assimilation effect, recorded through well documented psychophysical measurement (CCE15,18), was observed after 1 minute of a congruent visuo-tactile stimulation. Additionally we observed that once the incorporation effect appeared, it was maintained for at least a few minutes (as shown by the maintenance of the effect after the last VO block). Overall, our results are in concordance with previous findings on incorporation of upper limbs15,42,44 and lower limbs19 in healthy subjects or amputee patients36,45. We also extended these previous observations to SCI patients. Moreover, our findings create a bridge between limb incorporation and sensory remapping since our subjects integrated the avatar’s legs while receiving tactile feedback on their forearm. Finally, reacquiring haptic feedback from the legs recreates the interaction with the external world, or in the case of walking, with the ground surface. Spinal cord injured (SCI) patients participating in this study found specific vibrotactile patterns that corresponded to the complex sensation of walking on sand, grass or paved street. The experimental paradigm allowed the patients to select among all possible discernable combinations of tactile parameters (10’000 combinations). The patients selected parameters which are easily clustered, showing that they had a clear idea of the sensation they were seeking. As a result, six out of eight patients performed at above chance identification rates. One important question regarding these results is whether the patients associated a specific vibrotactile pattern to a given surface, or whether the vibrotactile pattern elicited a genuine sensation of walking on a specific floor. Several elements point towards the second hypothesis. The patients received no training, nor did they receive any feedback regarding their selections during the exploratory phase or the inverse task. Second, there was a break of at least 20 min between the exploratory phase and the inverse task; a delay larger than the one reported for haptic working memory (around 10 seconds)46. Finally, strong similarities in the selected parameters were found among patients, making it highly unlikely for their choices to be a mere repetition of a casual initial association. Another interesting discovery of our experiment was the parameter that patients relied on the most to identify the ground surfaces. When the parameter corresponding to the structure of stimulation was somehow expected to be important (patients expected three separated stimulations when walking on a granular surface such as sand, a single impact when walking on a paved street and smooth stimulation for grass), it was surprising to see the second and only other important parameter was the sensation at the end of the stance. Patients considered the sensation of the foot pushing on the floor before starting a swing as a signature for describing the different ground types: strong feedback for PST, softer for GRT, lightest for SAT. Our experiments highlight the importance of tactile feedback over vision for perception of floor texture. Presence or absence of the virtual floor in the virtual simulation did not influence patients’ choices of what they Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
9
61
www.nature.com/scientificreports/ perceived as a certain ground type, although they all judged that the 3D simulation of the virtual grounds were realistic. The subjective and conscious feeling, or the ‘kind of ‘ sensation often referred to as qualia47, is an important aspect of our proposed approach to restore tactile perception. A more direct implementation of floor texture display could have been to sample material features from real-world surfaces and remap them through the tactile shirt48,49. We have opted for a different approach where naïve subjects search for their preferred sensation. Lately, the nature of the subject’s perception during the use of sensory substitution devices has been questioned50: if someone uses an apparatus to perceive, through vibrators, a stream of visual data, can one conclude that the user is ‘seeing with the skin’? While our work does not pretend to permanently solve this debate, we believe that our experimental paradigm avoided some of the issues that make interpretation of sensory substitution devices tedious. Our patients did not learn to discriminate among different stimuli; they reported the one they considered closest to their perception of walking on different surfaces. As a result, the majority of patients described the sensations as realistic and similar to the one of walking on the corresponding floor type. The patients that did not find the sensation realistic were also the ones that reported not remembering walking on some of the floor types, suggesting that the sensation found during the experiment was linked to their personal experience of walking on these floors prior to the SCI. One patient once spontaneously reported ‘I was walking in a happy mood, because I was walking on the beach’. Altogether our experiments show important emergent positive effects when chronic SCI patients take advantage of our tactile shirt and highlight the importance of this feedback for BMIs and neuroprosthetics to be clinically relevant. In addition to the assistive benefits during locomotion, allowing patients to rely less on visual feedback, by relying on tactile feedback, patients can look forward instead of looking down at their feet to know where they are in their walking cycle. Continuous use of the haptic interface seems to change the patients’ body schema by altering their cortical representation of lower limbs through the process of incorporation of the virtual avatar legs. Interaction with our haptic interface also induced in the SCI patients a more vivid sensation of walking and the return of interactions with part of their peri-personal world that had been lost many years prior, i.e. the space immediately under their feet. To some degree, by reacquiring the ability to experience contact with the ground, and to perceive different types of ground surfaces, our haptic interface provided the patients with a much richer lower limb “phantom” sensation. Such an enriched illusion likely contributed to making these patients much more amenable to the idea of walking with the assistance of a custom-designed robotic exoskeleton7, since their walking experience with this orthosis generated a much more realistic sense of bipedal locomotion that they had experienced since their SCI.
Methods
This study’s protocol was approved by the ethics committee of AACD (Associação de Assistência à Criança Deficiente, São Paulo, Brazil) and carried out in accordance with its guidelines. All participants provided written informed consent before enrolling in the study. All our patients were initially evaluated using the American Spinal Injury Association (ASIA) Impairment Scale (modified from the Frankel classification) to quantify the severity level of their spinal cord injury. This scale grades SCI from ASIA A, for a complete lesion with no sensory or motor function below the neurological level of the injury, to E, for normal sensory and motor functioning1. Seven ASIA A patients and one ASIA B patient, all in the chronic phase of the SCI (at least one year after the SCI), were selected as subjects for all experiments reported here (see Table S1 for patients’ demography). All our patients had lesions below or equal to the thoracic dermatome T4. Accordingly, they all exhibited normal sensory motor functioning in the upper limbs. Three different psychophysical experiments were performed in the course of 6 months: a pseudo-proprioception test, a cross congruent task and a task involving the simulation of floor textures. Globally, these experiments were designed to: (a) provide the patients with an immersive visuo-tactile experience of walking; and (b) assess the impact of an augmented somatosensory feedback on the patients’ perception of their own body. During all experiments patients were seated in their wheelchair while wearing a tactile shirt (see Tactile Shirt description). In all the experiments, patients also wore a head mounted display on which a 3D human avatar was projected. The avatar could stand and walk and, as it performed these movements, tactile feedback, reproducing the touch of the avatar feet on the ground, was delivered on the skin of the patients’ forearm through the employment of a haptic display (e.g. the tactile shirt: see Integration of the virtual body avatars with the tactile shirt). In the pseudo-proprioception task, we delivered four distinct tactile feedback paradigms emulating different features of the avatar’s walk and determined which paradigm was the most intuitive for the patients. The Cross Congruent task (CCT)15,37 was used to explore the brain representation boundaries of the body schema in our patients after they used the tactile shirt. Finally, by simulating floor textures, we investigated whether spatiotemporal changes in the patterns of forearm tactile stimulation could give patients the perception of walking on different types of surfaces, like grass, sand or a paved street. Because there is no a priori answer for how to stimulate someone’s forearm in order to render the complex sensation of walking on different floor surfaces, we proposed a novel approach to search the best vibro-tactile parameters to render the floor types without making any assumption on the user’s perception.
Virtual Reality Environment and Setup. Three virtual human avatars (one woman and two men) were
modified based on free online stock models from Maximo (Maximo Inc. 2015). The virtual avatars were animated to walk and stop; animation blending was accomplished using MotionBuilder (Autodesk Inc. 2015). Custom written C++code controlled the triggering of avatar movements, the type of surface where the avatar walked and
Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
10
62
www.nature.com/scientificreports/ calculated the interaction of the avatar with the surface. In addition, we integrated the Oculus rift (Oculus VR) head mounted display with MotionBuilder using a technique called OpenGl intercept51.
Tactile Shirt as a Haptic Display. To deliver artificial tactile and proprioceptive feedback signals, origi-
nating from the movements of the human 3D avatar, we created a haptic display embedded in the long sleeves of a shirt. This haptic display was named “tactile shirt” and employed eccentric mass (ERM) vibrators to deliver somatosensory feedback to the skin of the patient’s forearm. The ERM vibrator consisted of a DC motor rotating an eccentric mass at different angular velocities, allowing the generations of various amplitudes and frequencies of vibration. ERM frequency and amplitude were coupled, and the maximum stimulation amplitude was reached at about 150–250 Hz, which corresponds to the peak response frequency of Pacinian Corpuscles (a type of rapidly adapting mechanoreceptor which is sensitive to mechanical transients and rapid vibrations in the range of (~40–400 Hz)) in the human hairy skin52. Our tactile shirt used three coin-shaped, 2 cm diameter, ERM vibrators to deliver sensory feedback, reproducing ipsilateral lower limb tactile or proprioceptive signals, to each of the patients’ forearms (Fig. 1a) The three vibrators were placed 6 cm apart from each other along patients’ forearms, following the longitudinal axis of the ulna bone (Fig. 1a). Since all our patients had thoracic lesions (T4-T11), in theory none of them should exhibit sensory deficits in the forearms. Preliminary tests showed that 6 out of 8 patients could discriminate vibrator position with 50 ms-long vibrations on their forearms (Fig. S3). We decided to move the shirt to the ventral part of the forearm for Patient P6, who had difficulties feeling stimuli on the ulna. Patient P1 was found to have difficulties discriminating vibration pulses under 70 ms, so longer pulse trains were employed with this patient.
Integration of the Virtual Body Avatars with the Tactile Shirt. During the experiments, patients sat
in a wheelchair, wearing an Oculus Rift head-mounted display (HMD) (Oculus, VR) in which virtual legs were projected, mimicking the position and orientation of the patients’ own bodies (Fig. 1a). Patients also wore headphones playing white noise to avoid biases due to noise from the vibrators. Prior to the first experiment, we ran two tests with the head-mounted display to evaluate if all patients could perceive correctly the 3D Virtual Reality (VR). We also evaluated whether any of them experienced any sign of motion sickness (Fig. S5). One subject experienced strong motion sickness during the second test while quickly moving the head. Thus, to have a single setup suited for all patients, we kept a fixed camera point of view for all patients during all the experiments.
Pseudo Proprioception Test. We tested two different feedback modalities related to different stages of the
avatar body locomotion and assessed which one was more intuitive for patients to perceive the position of the avatar leg when relying on tactile feedback only: (a) feedback given during the stance phase of the virtual avatar legs; (b) feedback given during the swing phase; as well as two directions of tactile stimulation on the forearm: proximal to distal (PtD) or distal to proximal (DtP). The four combinations given by the two modalities x two directions were tested in separate experiments (randomized order), with each one divided in four blocks (Fig. S2a). The first block lasted 30 seconds during which the patients were looking at the virtual avatar legs through the HMD. At this stage, the avatar walked at medium speed of 66 steps per minute; patients did not receive any tactile feedback on their forearms. They were asked to look at the avatar legs and imagine that they were their own. The second block lasted 1 minute and included the delivery of tactile stimulation on the forearm skin surface, in synchrony with the avatar walk and according to one of the four tactile conditions. For the third block, the HMD was turned off while the avatar continued walking at the same speed and tactile feedback was delivered accordingly. Patients were asked to rely on the tactile stimulation while imagining their own legs moving. This block lasted 30 seconds. After the third block, the avatar stopped walking, and after a 20 second break the last block started. Here, subjects were instructed to match the upcoming avatar walk, provided through tactile feedback, with a corresponding movement of their arms. The avatar resumed walking, first at 66 steps per minute. The walking speed was randomly reduced to 50 steps per minute or increased to 100 steps per minute without the user’s knowledge and without specific time patterns (Fig. S2a). A camera filmed the arms of the patient during this exercise and compared these to the joint positions of the avatar legs. A score, ranging from 0 to 3, was given to evaluate a patient’s performance for each speed. A 0 score was given if the patient’s arm movements were not synchronized with the avatar walk at any phase for a certain speed. A score of 1 was given if a patient managed to follow the walk at constant speed, but did not manage to follow the walk when the avatar’s walking speed changed (for example from medium to high speed). A score of 2 or 3 was given if the patient managed to follow the avatar walk synchronously during constant and changing speeds. Scores differed when patients were synchronous with the avatar walk but exhibited a contralateral inversion (for example showing the stance with the left arm during right leg stance and vice versa, score 2), and when patients were synchronous and used the correct arm to show the correct leg (score 3). We ranked the four tactile paradigms Pi based on the following scoring: Score(Pi) = ∑patients who had their best score with Pi – Σpatients who had their worst score with Pi. At the end of each test, patients answered a questionnaire (Fig. S2b) and restarted a new test with a different tactile paradigm.
Cross Congruent Task (CCT) test. To measure whether the virtual avatar leg was “incorporated” by the patients, we ran an adapted version of the cross congruent task (CCT)15,18,53. This task is based on the observation that human subjects are slower in detecting a tactile stimulus on the index finger if a visual distractor appears close to the thumb (and vice versa)54. This effect, named crossmodal interference, is stronger if the distractor is placed on the same hand as the tactile stimulation than when the visual distractor and tactile stimulation are Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
11
63
www.nature.com/scientificreports/ contralateral. Similarly after active use of a tool, the multimodal interaction between visual stimuli on the tip of the tool and tactile feedback on the user’s hand changes55: visual distractors on the tip of the tool led to an increase in response time (RT) when the vibration and the distractor were incongruent compared to the congruent case. These findings were interpreted as revealing the incorporation of the tool as an extension of the subject’s arm, due to the projection of cortical visual receptive fields to the distal edge of the tool. Patients wore the tactile shirt while observing the 3D avatar through the head-mounted display (Fig. 2a). A trial started with a visual fixation cross placed between the avatar’s feet for 1000 ms. Next, a 50 ms vibration was randomly triggered in one of four locations: proximal or distal location of the left or right forearms. Light distractors (3 cm radius 3D blue spheres) appeared 100 ms before the mechanical vibration in one of the four locations: left/right toe/heel of the avatar leg (100 ms offset was found to maximize the crossmodal interaction56). Patients were asked to indicate, using two keys on a keyboard, whether a vibration was delivered on the front part (distal) or in the back (proximal) of the forearm while ignoring the visual distractor. Each session started with 5 minutes of training. The experiment started only if patients had >85% accuracy in detecting the correct position of vibration during this training. The experience consisted of 5 minute long CCT blocks run immediately after subjects experienced 1 minute of observation of the avatar walking (Visual Only, VO), either with simultaneous tactile feedback moving from wrist to elbow of the patient’s forearm or when the ipsilateral avatar foot was in contact with the floor (Vision + Tactile feedback, VT) (Fig. 2b). For each CCT block we tested all 16 configurations of tactile feedback (four positions) and visual distractor (four positions). On each block CCT we repeated each configuration four times. We repeated the same experiment for all patients. Response time (RT) was measured and trials with RT longer than 1800 ms or faster than 200 ms were discarded as well as trials with RT beyond the range of mean ± 3 × std per (8.5% of the overall trials of all sessions).
Floor Texture Simulation Test. We investigated the set of tactile parameters that could be used with our
tactile display in order to induce the illusion of walking on three different ground surfaces: sand (SAT), grass (GRT), and paved street (PST). We then compared the parameters obtained for all patients. Patients were seated in their wheelchairs wearing the tactile shirt (see Fig. 3a) and a head mounted display. A 3D human avatar was shown in first person view. The avatar and the virtual environment were rendered using MotionBuilder (Autodesk) software. The avatar’s walking was based on motion capture of a healthy subject walking at 45 steps per minute. We presented a catalog of textures to our subjects. Patients were asked to choose those that best represented the sensation of walking on SAT, GTR, and PST. The catalog was created by varying four parameters describing the tactile stimulation: the amplitude of the distal vibrator (DV), the amplitude of the middle vibrator (MD), the amplitude of the proximal vibrator (PV) and the stimulation timing (ST) (Fig. 3a,b). Each one of these four factors had 10 possible levels. The number of possible combinations was thus 104 = 10’000. Vibrator amplitude 1 represented the lowest perceived sensation, and 10 the strongest before sensory saturation (both found empirically). The stimulation timing was defined by two factors: Duration of the Stimulus (DoS) and the Inter-Stimuli Onset Interval (ISOI). ISOI represents the time between onset of one vibrator and the next. The DoS was chosen to satisfy the following relation: DoS = Stance Duration – 2xISOI; where Stance Duration was fixed to 2000 ms throughout the entire session. ISOI varied between 100 ms and 820 ms. An ST level 1 referred to the shortest ISOI and longest DoS (Fig. 3d) and corresponded to three long vibrations delivered almost simultaneously. For an ST of level 10, the onset asynchrony was longer than the stimulation duration, resulting in three short and distinct vibrations (not overlapping). Between these two extremes some levels of ST produced a continuous moving touch illusion known in haptics as apparent movement16. Each experiment started with patients seated in front of a table wearing the head-mounted display to observe the 3D avatar and with their arms placed on a table (Fig. 3a). A thick tape was used to delineate two square areas of 50 × 50 cm2 on the table. Patients were asked to keep the left hand inside the left square and right hand in the right square. A tracking system recorded the position of both hands in the two referential areas defined by the two squares. As patients moved their hands over the two square areas, they received a particular pattern of tactile stimulation, defined by the four factors described above (Fig. 3b). More specifically, planar, Cartesian coordinates of the left and right hand were mapped onto four tactile parameters. Hence, each spatial position inside the two squares was assigned with a particular pattern of tactile stimulation to the patient’s forearms. Axes were randomized at each trial. The session started with a 15-minute training phase. The experimenter asked the patient to find the tactile feedback that represented best for her/him walking on sand or grass or paved street. Patients freely explored the 2D spaces defined by the two squares. As they explored this virtual “tactile space”, patients were asked to observe the avatar walking on a black floor. Tactile feedback was delivered via the tactile display during the stance phases of the avatar on the ipsilateral arm. Next, an exploratory phase started where the same procedure was repeated 40 times per surface (a total of 120 trials), in a randomized order. To avoid patients learning the position of a certain texture on the table, the four axes on the table were randomized for every trial (four possible axis configurations for left side, four for right side and all possible permutations of the four parameters = 4 × 4 × 4! = 384 possible configurations, Fig. 3c). Here, instead of the experimenter announcing the floor type, a cube appeared for 5 seconds in the Virtual environment (VE) at the beginning of each trial: red for SAT, green for GRT and blue for PST. For half of the trials (randomized order), immediately after the cube disappeared from view, the corresponding floor – sand, grass, or paved street - was displayed in the VE (Fig. 3a). For the other half of trials, the floor stayed black. Patients had 2 minutes to explore the table (and thus the catalog of textures) and confirm that the correct sensation was identified by raising their right hand by 10 cm. The task was the same in the visual presence or absence of the corresponding floor in the VE. After confirmation, the avatar was stopped. Following a 2–5 second inter-trial time, a new colored cube appeared Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
12
64
www.nature.com/scientificreports/ and a new trial started. The trials were divided as follows: first block of 36 trials, a break of 5 minutes; a second block of 36 trials, a break of 45 minutes; and a third block of 48 trials. Inside a block there were always the same number of trials with and without floor and the same distribution of surface types. After the exploring phase was concluded and patients had a 20 minute break, the next step of the experiment started. The experimenter played back in the haptic display applied to the patients’ forearms the 120 textures chosen during the searching phase. Patients had to say which floor type the avatar was walking on or say ‘I don’t know’. The patient held their arm on the table and, using the head-mounted display, observed the avatar walking on an empty black floor. This phase was named the inverse task. At the end of the first session a questionnaire was administered: Q1) During the experiment I had the impression of walking on SAT, GRT, PST. Q2) I remember the sensation of my feet on SAT,GRT, PST. Two months later we ran a control experiment with all patients. The patient observed the avatar walking on the three different floors for 15 minutes. No tactile feedback was used. The session was followed by a questionnaire containing question Q1 and the following question: Q3) The floor of type SAT/GRT/PST I saw in the head-mounted display was visually realistic.
References
1. Ditunno, J. F., Young, W., Donovan, W. H. & Creasey, G. The International Standards Booklet for Neurological and Functional Classification of Spinal Cord Injury. J. Orthopsychiatry 32, 70–80 (1994). 2. Carmena, J. M. et al. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol. 1, E42 (2003). 3. O’Doherty, J. E. et al. Active tactile exploration enabled by a brain-machine-brain interface. Nature 479, 228–231 (2011). 4. Lebedev, M. A. & Nicolelis, M. A. L. Brain-machine interfaces: past, present and future. Trends Neurosci. 29, 536–546 (2006). 5. Hochberg, L. R. et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171 (2006). 6. Chatterjee, A., Aggarwal, V., Ramos, A., Acharya, S. & Thakor, N. V. A brain-computer interface with vibrotactile biofeedback for haptic information. J. Neuroeng. Rehabil. 4, 40 (2007). 7. Donati, A. R. C., Shokur, S., Morya, E., Campos, D. S. F., Moioli, R. C., Gitti, C. M., Augusto, P. B., Tripodi, S., Pires, C. G., Pereira, G. A., Brasil, F. L., Gallo, S., Lin, A., Takigami, A. K., Aratanha, M. A., Joshi, S., Bleuler, H., Cheng, G., Rudolph, A. & Nicolelis, M. A. L. Long-term training with brain-machine interfaces induces partial neurological recovery in paraplegic patients. Sci. Rep. 6, 30383, doi: 10.1038/srep30383 (2011). 8. Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B. & Scadden, L. Vision substitution by tactile image projection. Nature 221, 963–964 (1969). 9. Bach-y-Rita, P., Kaczmarek, K. A., Tyler, M. E. & Garcia-Lara, J. Form perception with a 49-point electrotactile stimulus array on the tongue: a technical note. J. Rehabil. Res. Dev. 35, 427–430 (1998). 10. Saunders, F. A., Hill, W. A. & Franklin, B. A wearable tactile sensory aid for profoundly deaf children. J. Med. Syst. 5, 265–270 (1981). 11. Rupert, A. H. An instrumentation solution for reducing spatial disorientation mishaps. IEEE Eng. Med. Biol. Mag. 19, 71–80 (2000). 12. Danilov, Y. P., Tyler, M. E., Skinner, K. L., Hogle, R. A. & Bach-y-Rita, P. Efficacy of electrotactile vestibular substitution in patients with peripheral and central vestibular loss. J. Vestib. Res. 17, 119–130 (2007). 13. Kim, K., Colgate, J. E., Santos-Munne, J. J., Makhlin, A. & Peshkin, M. A. On the Design of Miniature Haptic Devices for Upper Extremity Prosthetics. IEEE/ASME Trans. Mechatronics 15, 27–39 (2010). 14. Kapur, P., Jensen, M., Buxbaum, L. J., Jax, S. A. & Kuchenbecker, K. J. Spatially distributed tactile feedback for kinesthetic motion guidance. 2010 IEEE Haptics Symposium p 519–526, March 25–26, 2010, Waltham, MA, doi: 10.1109/HAPTIC.2010.5444606 (2010). 15. Maravita, A., Spence, C. & Driver, J. Multisensory integration and the body schema: close to hand and within reach. Curr. Biol. 13, R531–R539 (2003). 16. Sherrick, C. E. & Rogers, R. Apparent haptic movement. Percept. Psychophys. 1, 175–180 (1966). 17. Kirman, J. H. Tactile apparent movement: The effects of interstimulus onset interval and stimulus duration. Percept. Psychophys. 15, 1–6 (1974). 18. Sengul, A. et al. Extending the Body to Virtual Tools Using a Robotic Surgical Interface: Evidence from the Crossmodal Congruency Task. PLOS One 7, e49473 http://dx., doi:org/10.1371/journal.pone.0049473 (2012). 19. Van Elk, M., Forget, J. & Blanke, O. The effect of limb crossing and limb congruency on multisensory integration in peripersonal space for the upper and lower extremities. Conscious. Cogn. 22, 545–555 (2013). 20. Stein, B. E. & Meredith, M. A. The merging of the senses. MIT Press, Cambridge, MA (1993). 21. Ernst, M. O. & Bülthoff, H. H. Merging the senses into a robust percept. Trends Cogn. Sci. 8, 162–169 (2004). 22. Lieberman, J. & Breazeal, C. IEEE Xplore Abstract - Development of a Wearable Vibrotactile Feedback Suit for Accelerated Human Motor Learning. Robot. Autom. 23, 919–926 (2007). 23. Shams, L. & Seitz, A. R. Benefits of multisensory learning. Trends Cogn. Sci. 12, 411–417 (2008). 24. Kim, R. S., Seitz, A. R. & Shams, L. Benefits of Stimulus Congruency for Multisensory Facilitation of Visual Learning. PLoS One 3, e1532 (2008). 25. Dietz, V. Proprioception and locomotor disorders. Nat. Rev. Neurosci. 3, 781–790 (2002). 26. Conway, B. A., Hultborn, H. & Kiehn, O. Proprioceptive input resets central locomotor rhythm in the spinal cat. Exp. Brain Res. 68, 643–656 (1987). 27. Giuliani, C. A. & Smith, J. L. Stepping behaviors in chronic spinal cats with one hindlimb deafferented. J. Neurosci. 7, 2537–2546 (1987). 28. Eils, E. et al. Modified pressure distribution patterns in walking following reduction of plantar sensation. J. Biomech. 35, 1307–1313 (2002). 29. Fuentes, C. T., Pazzaglia, M., MR, L., Scivoletto, G. & Haggard, P. Body image distortions following spinal cord injury. J. Neurol. Neurosurg. Psychiatry 82, 201–207 (2013). 30. lenggenhager, B., Pazzaglia, M., Scivoletto, G., Molinari, M. & Aglioti, S. M. The sense of the body in individuals with spinal cord injury. PLoS One 7, e50757 (2012). 31. Pazzaglia, M., Galli, G., Scivoletto, G. & Molinari, M. A Functionally Relevant Tool for the Body following Spinal Cord Injury. PLoS One 8, e58312 (2013). 32. Schaefer, M., Xu, B., Flor, H. & Cohen, L. G. Effects of different viewing perspectives on somatosensory activations during observation of touch. Hum. Brain Mapp. 30, 2722–2730 (2009). 33. Iriki, A., Tanaka, M. & Iwamura, Y. Coding of modified body schema during tool use by macaque postcentral neurones. Neuroreport 7, 2325–2330 (1996). 34. Guterstam, A., Petkova, V. I. & Ehrsson, H. H. The Illusion of Owning a Third Arm. PLoS One 6, e17208 (2011).
Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
13
65
www.nature.com/scientificreports/ 35. Carmena, J. M. et al. Learning to Control a Brain\textendashMachine Interface for Reaching and Grasping by Primates. PLoS Biol. 1, e42 (2003). 36. Canzoneri, E., Marzolla, M., Amoresano, A., Verni, G. & Serino, A. Amputation and prosthesis implantation shape body and peripersonal space representations. Sci. Rep. 3, 1–8 (2013). 37. Maravita, A. & Iriki, A. Tools for the body (schema). Trends Cogn. Sci. 8, 79–86 (2004). 38. Shokur, S. et al. Expanding the primate body schema in sensorimotor cortex by virtual touches of an avatar. Proc. Natl. Acad. Sci. USA 110, 15121–15126 (2013). 39. Ifft, P. J., Shokur, S., Li, Z., Lebedev, M. A. & Nicolelis, M. A. L. A Brain-Machine Interface Enables Bimanual Arm Movements in Monkeys. Sci. Transl. Med. 5, 210ra154 (2013). 40. Slater, M., Perez-Marcos, D., Ehrsson, H. H. & Sanchez-Vives, M. V. Towards a Digital Body: The Virtual Arm Illusion. Front. Hum. Neurosci. 2 (2008). 41. Botvinick, M. & Cohen, J. Rubber hands ‘feel’ touch that eyes see. Nature 391, 756 (1998). 42. Ehrsson, H. H., Holmes, N. P. & Passingham, R. E. Touching a rubber hand: feeling of body ownership is associated with activity in multisensory brain areas. J. Neurosci. 25, 10564–10573 (2005). 43. Tsakiris, M. & Haggard, P. The rubber hand illusion revisited: visuotactile integration and self-attribution. J. Exp. Psychol. Hum. Percept. Perform. 31, 80–91 (2005). 44. Ehrsson, H. H., Spence, C. & Passingham, R. E. That’s my hand! Activity in premotor cortex reflects feeling of ownership of a limb. Science 305, 875–877 (2004). 45. Ehrsson, H. H. et al. Upper limb amputees can be induced to experience a rubber hand as their own. Brain 131, 3443–3452 (2008). 46. Grunwald, M. (ed). Human Haptic Perception: Basics and Applications (Birkhäuser Basel, 2008). 47. Ramachandran, V. S. & Hirstein, W. Three laws of qualia: what neurology tells us about the biological functions of consciousness. J. Conscious. Stud. 4, 429–457(29) (1997). 48. Pai, D. K. & Rizun, P. The WHaT: A wireless haptic texture sensor. Proc. - 11th Symp. Haptic Interfaces Virtual Environ. Teleoperator Syst. Haptics 3–9, doi: 10.1109/HAPTIC.2003.1191210 (2003). 49. Maclean, K. E. The ‘Haptic Camera’: A Technique for Characterizing and Playing Back Haptic Properties of Real Environments. Proc. Haptics Symp. 459–467 (1996). 50. Deroy, O. & Auvray, M. Reading the world through the skin and ears: A new perspective on sensory substitution. Front. Psychol. 3, (2012). 51. Zielinski, D. J., McMahan, R. P., Shokur, S., Morya, E. & Kopper, R. Enabling Closed-Source Applications for Virtual Reality via OpenGL Intercept-based Techniques. IEEE 7th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS). Minneapolis, MN, doi: 10.1109/SEARIS.2014.7152802 (2014). 52. Bolanowski, S. J., Gescheider, G. A. & Verrillo, R. T. Hairy skin: psychophysical channels and their physiological substrates. Somat. Mot. Res. 11, 279–290 (1994). 53. Maravita, A., Spence, C., Sergent, C. & Driver, J. Seeing your own touched hands in a mirror modulates cross-modal interactions. Psychol. Sci. 13, 350–355 (2002). 54. Spence, C., Pavani, F. & Driver, J. Crossmodal links between vision and touch in covert endogenous spatial attention. J. Exp. Psychol. Hum. Percept. Perform 26, 1298–1319 (2000). 55. Longo, M. R., Sadibolova, R. & Tamè, L. Embodying prostheses - how to let the body welcome assistive devices: Comment on:The embodiment of assistive devices-from wheelchair to exoskeleton by M. Pazzaglia and M. Molinari. Phys. Life Rev. doi: 10.1016/j. plrev.2016.01.012 (2016). 56. Shore, D. I., Barnes, M. E. & Spence, C. Temporal aspects of the visuotactile congruency effect. Neurosci. Lett. 392, 96–100 (2006).
Acknowledgements
Authors thank the patients for their contribution to this research. We also want to thank Neiva Paraschiva, Andrea Arashiro, Maria Cristina Boscaratto (AASDAP, Associação Alberto Santos Dumont para Apoio à Pesquisa) Susan Halkiotis (Duke University), for their work, commitment and support during this study. We thank Jeremy Olivier and Mohamed Bouri (EPFL, Ecole Polytechnique Fédérale de Lausanne), Nicole Peretti, Kyle Fast, Anthony Lin, Angelo Takigami, Maria Adelia Aratanha, Fabricio L. Brasil, and Lara Azevedo Godoy (AASDAP) for the technical support; Debora Campos, Patricia Augusto, Claudia Gitti and Cristhiane Pires for the clinical support. This study was funded by FINEP (Brazilian Funding Authority for Studies and Projects, project number 01.12.0514.00) and Itaú Bank.
Author Contributions
M.A.L.N., S.S.H. and S.G. contributed to study design, data interpretation, data collection, data analysis, literature search, figure, tables, writing and editing. R.C.M. contributed to data analysis and literature search. A.R.C.D. and E.M. contributed to data collection. H.B. contributed to study design, data interpretation, literature search, writing and editing.
Additional Information
Supplementary information accompanies this paper at http://www.nature.com/srep Competing financial interests: The authors declare no competing financial interests. How to cite this article: Shokur, S. et al. Assimilation of virtual legs and perception of floor texture by complete paraplegic patients receiving artificial tactile feedback. Sci. Rep. 6, 32293; doi: 10.1038/srep32293 (2016). This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ © The Author(s) 2016
Scientific Reports | 6:32293 | DOI: 10.1038/srep32293
14
66
RESEARCH ARTICLE
Training with brain-machine interfaces, visuotactile feedback and assisted locomotion improves sensorimotor, visceral, and psychological signs in chronic paraplegic patients a1111111111 a1111111111 a1111111111 a1111111111 a1111111111
OPEN ACCESS Citation: Shokur S, Donati ARC, Campos DSF, Gitti C, Bao G, Fischer D, et al. (2018) Training with brain-machine interfaces, visuo-tactile feedback and assisted locomotion improves sensorimotor, visceral, and psychological signs in chronic paraplegic patients. PLoS ONE 13(11): e0206464. https://doi.org/10.1371/journal.pone.0206464 Editor: Mariella Pazzaglia, University of Rome, ITALY Received: June 4, 2018 Accepted: October 12, 2018 Published: November 29, 2018 Copyright: © 2018 Shokur et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: All relevant data are within the manuscript and the supporting information files. We have uploaded an anonymized dataset with our supporting information. Funding: This study was funded by grants from the Brazilian Financing Agency for Studies and Projects (FINEP 01�12�0514�00), Brazilian Ministry of Science, Technology, Innovation and Communication (MCTIC), and the Itaú Unibanco S.
Solaiman Shokur1, Ana R. C. Donati1,2, Debora S. F. Campos1, Claudia Gitti2, Guillaume Bao1, Dora Fischer1,2, Sabrina Almeida1,2, Vania A. S. Braga1, Patricia Augusto1, Chris Petty3, Eduardo J. L. Alho1,4, Mikhail Lebedev5,6, Allen W. Song3, Miguel A. L. Nicolelis ID1,5,6,7,8,9,10,11* 1 Neurorehabilitation Laboratory, Associação Alberto Santos Dumont para Apoio à Pesquisa (AASDAP), São Paulo, Brazil, 2 Associação de Assistência à Criança Deficiente (AACD), São Paulo, Brazil, 3 Brain Imaging and Analysis Center, Duke Univ Medical Center, Durham, NC, United States of America, 4 Department of Neurosurgery, University of Sao Paulo Medical School, Sao Paulo, Brazil, 5 Department of Neurobiology, Duke University Medical Center, Durham, NC, United States of America, 6 Duke Center for Neuroengineering, Duke University, Durham, NC, United States of America, 7 Department of Biomedical Engineering, Duke University, Durham, NC, United States of America, 8 Department of Neurology, Duke University, Durham, NC, United States of America, 9 Department of Neurosurgery, Duke University, Durham, NC, United States of America, 10 Department of Psychology and Neuroscience, Duke University, Durham, NC, United States of America, 11 Edmond and Lily Safra International Institute of Neuroscience, Macaı́ba, Brazil * nicoleli@neuro.duke.edu
Abstract Spinal cord injury (SCI) induces severe deficiencies in sensory-motor and autonomic functions and has a significant negative impact on patients’ quality of life. There is currently no systematic rehabilitation technique assuring recovery of the neurological impairments caused by a complete SCI. Here, we report significant clinical improvement in a group of seven chronic SCI patients (six AIS A, one AIS B) following a 28-month, multi-step protocol that combined training with non-invasive brain-machine interfaces, visuo-tactile feedback and assisted locomotion. All patients recovered significant levels of nociceptive sensation below their original SCI (up to 16 dermatomes, average 11 dermatomes), voluntary motor functions (lower-limbs muscle contractions plus multi-joint movements) and partial sensory function for several modalities (proprioception, tactile, pressure, vibration). Patients also recovered partial intestinal, urinary and sexual functions. By the end of the protocol, all patients had their AIS classification upgraded (six from AIS A to C, one from B to C). These improvements translated into significant changes in the patients’ quality of life as measured by standardized psychological instruments. Reexamination of one patient that discontinued the protocol after 12 months of training showed that the 16-month break resulted in neurological stagnation and no reclassification. We suggest that our neurorehabilitation protocol, based uniquely on non-invasive technology (therefore necessitating no surgical operation),
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
1 / 33
67 Sensory-motor, visceral and psychological improvement in paraplegics
A. We also acknowledge the support from the National Institute of Science and Technology Program (INCT INCEMAQ 610009/2009-5) of the National Council of Scientific and Technological Development (CNPq/MCTIC). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
can become a promising therapy for patients diagnosed with severe paraplegia (AIS A, B), even at the chronic phase of their lesion.
Competing interests: The authors have declared that no competing interests exist.
Spinal Cord Injuries (SCI) cause a wide array of disabilities with devastating motor, sensory, and autonomic deficits that impair the functional capacity of patients to perform routine living and working activities. SCI also leads to significant impairments in the patient’s quality of life (QoL) [1], his/her body image [2] and sexuality [3]. As such, SCI rehabilitation must consider the patient’s physical, emotional, social and affective life aspects [4], aiming at promoting the patient’s physical independence and autonomy, while promoting the reintegration of the individual into society. Epidemiological studies on SCI [5] reveal a global incidence, considering traumatic or non-traumatic etiology, around 250,000 to 500,000 new cases added every year. The severity of clinical impairment and recovery prognosis depends on the SCI level [6], the extension and mechanism of injury, presence/absence of residual spinal cord fibers and pre-existing clinical comorbidities. In the US, the most common rank observed one year after the original SCI is AIS A (34%), which includes patients with complete neurological loss SCI (but not necessary anatomically complete lesions [7–9]) [10]. The majority of such AIS A cases involves lesions at the thoracic level (67%) [10]. Studies show a considerable amount of spontaneous improvement during the first year following the lesion [11,12], and stagnation at the chronic phase. For example, following 26 weeks after the original lesion, the rates of motor score recovery for AIS A patients drops to less than 1 point (on a scale where complete paraplegia is 0, and normal function is 50). Between 1 and 5 years after the SCI injury, only about 3.5% of AIS A patients are upgraded to AIS B, 1.05% to C and 1.05% to D [13,14]. These statistics indicate that the most commonly observed cases of chronic paraplegic SCI are represented by AIS A patients who have very little chance of spontaneous neurological recovery at the chronic phase of the injury. The rehabilitation process with SCI patients involves both learning of tasks as in the use of a wheelchair, and a variety of compensation mechanisms to recover from lost motor functions [15]. The use of stem cell therapy for complete SCI has also been investigated in recent years, both experimentally and clinically, but despite some interesting outcomes, it has not yet been established as a standard effective treatment [16]. Potential new treatments for controlling neuropathic pain, spasticity, bladder, and intestinal functions have also been extensively studied. Those include electrical spinal cord stimulation, chemical neuromodulation [17], drug delivery pumps with catheters inserted in the subarachnoid space [18] and sacral stimulators [19–21]. Despite important recent improvements in rehabilitation techniques, the chances of neurological recovery for motor complete SCIs (AIS A/B) remain low. Indeed, whereas a number of neurorehabilitation protocols have induced some level of neurological recovery in motor incomplete SCI patients (AIS C/D) (including stepping training [22,23], operant conditioning [24,25] or functional electrical stimulation [26]), improvement in motor complete SCI has been principally observed through compensatory mechanisms [15]. Only recently, a few studies in rats [27] and humans [28] have shown partial motor recovery (neurological and functional) in severe cases of SCI, following training with invasive epidural stimulation (see [29] for a review) or invasive pelvic nerve stimulation [30]. Notably, clinical improvement was noticed when such invasive stimulation was paired with direct patient control of the stimulating system, via a brain-machine interface [31] (see [32] for a review).
Introduction
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
2 / 33
68 Sensory-motor, visceral and psychological improvement in paraplegics
In a previous study, we have reported significant levels of neurological recovery in eight chronic motor complete (3–13 years post-lesion) SCI patients, after 12 months of training with a multi-step, non-invasive neurorehabilitation protocol [33]. Named the Walk Again Neuro-Rehabilitation (WA-NR), this protocol combines locomotion training, brain-machine interfaces (BMIs [34]) and visuo-tactile feedback. In the WA-NR protocol, SCI patients learn to use their brain activity, recorded via EEG, to control the locomotion of virtual human avatars and robotic gait devices. To close the control loop, continuous streams of tactile feedback are delivered to the skin of patients’ forearms, via a haptic display referred to as the tactile shirt [35], in synchrony with regular visual feedback. In the present study, we report a detailed analysis of a subgroup of patients from our original study, including six chronic (3–13 years post-lesion) AIS A patients and one AIS B (6 years post-lesion) SCI patient, who continued to train under the WA-NR protocol for a period of 28 months. As a result of this training, we observed significant levels of neurological recovery which included improvements in nociceptive, tactile, and proprioceptive function, a marked improvement in multiple visceral functions (bladder control, bowel function and sexual functions in some patients), and significant gains in voluntary motor control of the lower-limbs (confirmed by both clinical evaluation [6], and neurophysiological measurement (EMGs)). The patients’ anatomical lesion level was revealed by MRI analysis of their SCI. Overall, we observed that the observed sensory and motor improvements occurred in the areas of the body innervated by portions of the spinal cord below the original anatomical SC lesion. As a consequence of such major sensory recovery, we identified a complex reorganization in the patients’ perception of their bodies. Finally, we observed that this partial neurological recovery induced significant improvement in both the psychological and physical aspects of the patients’ self-report on their quality of life [36]. By the end of the training, all seven patients had improved their AIS grade, representing, as far as we can tell, the largest cohort of chronic complete paraplegic patients reported in the literature to exhibit a consistent partial neurological recovery, following a purely non-invasive neurorehabilitation approach.
Materials and methods Participants Eight paraplegic patients (Table 1), 27–38 years old, with traumatic and chronic SCI (lesion 3–13 years before onset of the training) at thoracic level (T4-T11), participated in the current study: seven patients, from hereon designated as Group 1 (GR1), including six AIS A patients (P1, P3, P4, P5, P6, P8) and one AIS B (P2), followed the WA-NR protocol [37] for a total of 28 months of training. One subject (P7, AIS A) discontinued the training after 12 months and is therefore discussed separately. The participants in the current study are the same that participated in a previous protocol reported by our group [37]. For clarity, the convention used for subject names is the same in both studies. Initially, the level and the grade of each SCI was estimated, using the ASIA standard assessment [6] (Table 1). This measurement was done within the first and the third year post-lesion and is referred to in this paper as the baseline. Patients’ original AIS classification and neurological status were confirmed by our medical team at the onset of training (referred to as time 0). Details of patients ASIA score is reported in S1 Fig. To complete our neurological investigation, we examined the patients’ spinal cord using MRI. This measurement was performed once at the end of the protocol. Patient P1 was excluded from the MRI examination for safety reasons (presence of undefined material used for arthrodesis). Data from patient P7 were not recorded due to protocol discontinuation. For three out of the six tested patients, namely P2,
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
3 / 33
69 Sensory-motor, visceral and psychological improvement in paraplegics
Table 1. Patientsâ&#x20AC;&#x2122; demography. Subject
Training period (months)
Sex
Age
AIS1
Lesion Level Clinic2 Right Left
Lesion Level MRI3
Time Since lesion (years)
Etiol4
Time since baseline ASIA months5
P1
28
F
32
A
T10
T11
No data
13
C
142
P2
28
M
26
B
T4
T4
T1-T3
6
C
54
P3
28
M
32
A
T11
T10
T10-L1
5
O
30
P4
28
M
38
A
T8
T8
T7-T9
5
C
40
P5
28
M
36
A
T7
T7
T7-T10
3
C
22
P6
28
M
29
A
T4
T4
T3-T5
8
C
54
P7
12
M
27
A
T5
T7
No data
6
C
62
P8
28
F
29
A
T11
T11
T4-L4
9
C
110
1) ASIA neurological standards evaluation 2) Clinical lesion level (AIS) 3) Anatomic lesion level (MRI guided). 4) Traumatic Etiology. C: Closed Trauma; O: Opened Injury. 5) The first measurement is done by the clinical institution that followed them before their enrollment in our protocol done at n months before the onset of our training. https://doi.org/10.1371/journal.pone.0206464.t001
P3, and P4, MRI images revealed the existence of some degree of spinal cord continuity at the lesion level (S2 Fig and S1 Video). MRI analysis was partially compromised by artifacts due to the presence of metallic implants in two cases (P5, P6) and by the complexity of the injury in one case (P8), making it difficult to distinguish between neurological and pathological tissue. To assess the effect of our intervention, we clinically evaluated GR1 patients eight times: at the protocol onset, and after 4, 7, 10, 12, 16, 22 and 28 months of training. Patient P7 was evaluated during the first year and then once at the end of 28 months, after 16 months of training discontinuation.
Inclusion criteria Subjects were adult paraplegics, grade AIS A (complete), B (motor complete) (3) with traumatic SCI at the thoracic level, at least 6 months before the onset of the study, with the absence or offset comorbidities, and emotionally stable. We excluded patients with non-traumatic SCI, decompensated comorbidities, a degree of spasticity exceeding a score of 2 (on the Ashworth scale), degree of osteoporosis (T- score) < -4, and presence of joint deformities, fractures, pressure ulcers grade > 3, peripheral neuropathy of the upper limbs, brain injury, degenerative neuromuscular injury amputation of upper or lower limbs, pacemakers (cardiac/neural), cephalic (cranial/brain) implants and with emotional instability.
Design of experiment The WA-NR protocol [33] consisted of two main classes of training: active locomotion training (TR-LOC) and BMI-based neurorehabilitation exercises (TR-BMI). Progressively more complex stages, as part of a multistep strategy, were employed during application of the WA-NR protocol [33], assuring that patients had enough time to acquire upper limb and trunk strengthening, cardiovascular stability, as well as emotional adaption to the experience of orthostatic posture. The TR-LOC included training with both a robotic gait training device (Lokomat) and a body weight support system (ZeroG). The steps for the TR-BMI training included BMI control of a virtual avatar while the patient was in a seated position, BMI control of an avatar while patient was in an orthostatic position,
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
4 / 33
70 Sensory-motor, visceral and psychological improvement in paraplegics
Table 2. Average monthly training. Periods (Months range) TR-BMI
TR-LOC
0–4
4–7
7–10
10–12
12–16
16–22
22–28
P1
4
3
1
0
1
1
0
P2
5
4
2
0
0
2
0
P3
5
5
2
0
2
2
0
P4
5
5
1
0
1
1
0
P5
6
4
1
0
1
2
0
P6
5
4
1
0
0
1
0
P7
3
5
1
0
0
0
0
P8
3
2
1
0
1
1
0
P1
3
0
2
0
2
1
1
P2
2
0
2
0
2
2
1
P3
2
0
3
0
2
2
1
P4
3
0
3
0
2
2
2
P5
4
0
2
0
2
2
1
P6
2
0
3
0
1
2
2
P7
3
0
2
0
0
0
0
P8
2
0
2
0
1
1
1
Values are average numbers of sessions per month for BMI-based neurorehabilitation exercises (TR-BMI) and active locomotion training (TR-LOC). https://doi.org/10.1371/journal.pone.0206464.t002
BMI control of the Lokomat, and BMI control of an exoskeleton (all steps of this training are detailed in [33]). Table 2 shows the average number of training sessions for each patient. On average, during the first four months, patients had two interventions per week and once per week for the rest of the protocol. The period 10–12 corresponded to a break period for all the patients, with no training involved. Active locomotion training. TR-LOC activities included locomotion training with a body weight support (BWS) system, robotic gait therapy device on a treadmill (Lokomat, Hocoma), and training with an over-ground fixed track BWS system (ZeroG, Aretech LLC). During gait training, subjects were guided to attempt to execute lower-limb motor tasks actively, despite the SCI. For the Lokomat training, the physiotherapist verbally instructed the patients to try to actively perform lower limb movements, in conjunction with the computergenerated orthosis movements, which included flexion and extension of hip, knee, and ankle. Each session was distributed in blocks, during which specific movements were trained; for instance, hip flexion during swing phase. The device provided real-time biofeedback regarding the patient’s joints torque. BWS was limited by the maximum knee extension, without joint collapse during the stance phase of gait (up to 80% of BWS, avoiding further reductions in this device). Guidance force was fixed at 100%, and treadmill speed was set between 1–1.5 km/h, to promote the safest possible training environment. During ZeroG training, subjects wore leg orthosis (for joint stabilization: hip, knee, ankle), and employed a wheeled triangular walker, while being assisted by the physiotherapist. BWS was progressively decreased from 75% to 30% throughout the sessions (bone densitometry guided the lower BWS level). The training promoted postural control, dynamic balance control, cardiovascular conditioning, upper limb and trunk strengthening, lower limb voluntary activation during stance and swing gait subphases. BMI-based neurorehabilitation exercises. A 16-channel EEG cap was used for all the TR-BMI sessions. Overall, we implemented and tested two BMI control strategies. The hybrid state machine (Fig 1A) strategy employed 16 channel EEG recordings over the arm motor
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
5 / 33
71 Sensory-motor, visceral and psychological improvement in paraplegics
cortex area (Fig 1B) and two channel EMG recordings of the subject’s upper arms. Subjects used left and right arm motor imagery to select specific actions in a state machine (Fig 1C) and two stage EMG activation to confirm the selection and the state transition to trigger the action. This two-step confirmation strategy ensured that the movement of the avatar or the robotic legs was not triggered by mistake in case a false positive was detected at the EEG decoding phase. Patients could use this strategy to control sit/stand-up and walk/stop states. Alternatively, the same approach was used for patients to trigger walk/stop and kick states. The single leg control (SLC) protocol employed 16 channel EEG recordings, clustered around the medial longitudinal fissure, meaning that it was more densely focused over the putative leg motor representation area of the primary sensorimotor cortex (Fig 1B). Subjects used left or right leg motor imagery to trigger the stepping of the corresponding limb. Both BMI strategies were tested to control a 3D virtual avatar and a robotic leg actuator (Lokomat or a custom-built exoskeleton [38]). An array of vibrotactile actuators placed on the patients’ forearms provided online artificial proprioceptor/tactile feedback regarding the position of the legs (virtual or robotic) during the locomotion and the contact of the robotic actuator feet with the floor. This feedback was delivered to the patients’ forearms using a portable haptic display developed by our team and called the “tactile shirt” [35]. An LED display, integrated into the Lokomat and the exoskeleton, informed the patient about the status of the experiment and the outcome of the EEG classifier. For both BMI strategies, we used linear discriminant analysis (LDA), using features extracted by a six-dimensional common spatial pattern (CSP) to construct an EEG classifier. The first BMI strategy was tested during the first 6 months of training and the second one for the rest of the protocol.
Statistics Given the small samples of subjects, when possible, the details per subject are shown rather than a group average. Graphs with group scores report group mean and SEM. Correlation analysis. To study the main factors that influenced the observed neurological improvement in our patients, we performed a correlation analysis between the improvement rate and training related factors (TR) and external factors (EX). The improvement rate was calculated as the difference of the AIS score (motor or sensory) between two consecutive AIS assessments. We considered the eight measurements done by our team for patients in GR1 (at the onset of training and after 4,7,10, 12, 16, 22 and 28 months) and the six measurements for patient P7. The training-related factors were: the number of training hours of locomotion training (TR-LOC), and the number of hours with the BMI-based training (TR-BMI) (Table 2). The external factors were the SCI height (EX-SCH, listed from cervical to sacral, i.e., first cervical C2 = 1 and last sacral dermatome S4-S5 = 28), the time since the lesion (EX-TME), and the patients’ age (EX-AGE). For the correlation analysis between the improvement rates and the experiment factors, the Pearson correlation coefficient and p-values of the correlation are calculated with the standard corrcoef Matlab function.
Clinical assessments Magnetic resonance imaging (MRI). Spinal cord MRI scans, using a 1.5T GE-Genesis equipment, with gadolinium-based intravenous contrast, were collected 28 months after the training onset. Images were obtained in axial, sagittal and coronal planes, and in T1, T2 and FIESTA (Fast Imaging Employing Steady-state Acquisition) sequences. Submillimeter cuts were evaluated by a radiologist blinded to the experimental paradigm to detect possible
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
6 / 33
72 Sensory-motor, visceral and psychological improvement in paraplegics
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
7 / 33
73 Sensory-motor, visceral and psychological improvement in paraplegics
Fig 1. Brain-machine interfaces control strategy. (A) We have developed and validated two BMI strategies: the hybrid EEG and EMG state machine control (HSM) and the EEG single leg control (SLC). Both strategies were used to control the actuation of a virtual avatar and a robotic gait device (Lokomat or exoskeleton). A portable haptic device was used to inform the user of the position of the virtual or robotic leg and the contact of these actuator’s feet with the floor in real time [35]. (B) For both BMI strategies, a 16 channel EEG cap was used. Electrodes were clustered over the arm area of the sensorimotor cortex for HSM and leg area for the SLC strategy. The ground and reference electrodes are reported in gray and light blue respectively. (C) The control strategy for the HSM (left panel) is based on navigation of a state machine (middle panel), using motor imagery, and a two-step EMG confirmation, using isometric muscle contraction of the biceps (IMC). For example, when the subject is in a standing position (Stop/Safe sate) and wants to start walking, (s)he will imagine moving the left arm and confirms the choice with a left bicep IMC. The subject has then to produce a right bicep ICM to trigger walking. The SLC strategy uses the decoding of leg motor imagery through EEG signals. If left motor imagery (LMI) is detected the left step is triggered. Once in this position, if right motor imagery is detected, the right step is triggered, and if no state is detected for 5 seconds, the actuator (the avatar or the robotic gait device) returns to the idle position. https://doi.org/10.1371/journal.pone.0206464.g001
residual neural fibers at the level of the SCI. An additional 3D reconstruction of the spinal cord was obtained using the following method: (1) a spinal cord toolbox [39] was used to align the slices of the spinal cord; (2) centerline image was set manually on each axial slice, and (3) segmentation was obtained with sct_propseg function within the cerebrospinal fluid and confirmed by an experienced doctor. The three-dimensional mesh was rendered with ITK-SNAP software. Neurological evaluation: Sensory and motor scores. A somatosensory score was calculated by summing left and right dermatomes with normal (coefficient 2) and altered (coefficient 1) sensation for each patient [6]. Then, the total somatosensory improvement was calculated as the difference between the score after n months of training and the initial score. The motor evaluation followed the standard AIS motor assessment methodology [6]. Motor evaluation was conducted through a functional examination of 12 muscles, among them five key muscles (rectus femoris proximal portion, rectus femoris distal portion, tibialis anterior, extensor hallucis longus, gastrocnemius) and seven non-key muscles (hip adductors, gluteus maximus, gluteus medius, medial and lateral hamstring,flexor hallucis longus and extensor digitorum longus). Note that the presence of voluntary anal contraction and/or presence of any motor function (grade 1 or above) more than three levels below the motor level on a given side will determine if a patient is AIS B or AIS C (6). The motor score described the level of voluntary strength below the SCI level for each patient, ranging from 0 (absence of contraction) to 5 (for normal contraction, produced against gravity and strong opposing force). The lower extremity motor score (LEMS) was obtained by summing all key muscles (score for a healthy subject is 50, i.e., five key muscles, with a maximal score of 5, bilaterally). Proprioception measurements. For the proprioception evaluation, the experimenter performed individualized joint mobilizations at the lower limbs: hip flexion and extension (F/E), knee F/E, ankle dorsiflexion/plantarflexion, hallux F/E and toes (2nd to fifth) F/E. The patient was blindfolded and in a horizontal supine position, while the examiner performed manual joint mobilization at an approximately angular speed of 60˚/second. For the hip flexion, we lifted the leg up maintaining the knee extended and the lower limb aligned, to exclusively move the hip towards flexion, up to 60˚ (initial position = 0˚). For the hip extension, the initial position is 60˚ of flexion, lowering the leg towards the clinical table. To perform knee mobilizations, we kept the femur lifted with a hip flexion of 30˚. For knee flexion, we moved the tibia from extension position 0˚ towards the clinical table, performing 30˚ of knee flexion. For knee extension, we moved the tibia from 30˚ of flexion, back to complete extension 0˚. For ankle dorsiflexion, we moved the foot up from neutral position (0˚) to 10˚ of dorsiflexion. For ankle plantarflexion, we moved the foot down from the neutral position (0˚) to 40˚ of plantarflexion. For hallux flexion, we moved it down from the neutral position to 45˚ of flexion. For hallux extension, we moved it up from the neutral position to 45˚ of extension. For toes flexion, we
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
8 / 33
74 Sensory-motor, visceral and psychological improvement in paraplegics
moved them down from the neutral position to 30˚ of flexion and finally for toes extension; we moved it up from the neutral position to 40˚ of extension. Patients were instructed to describe when (s)he could perceive the stimulus. We performed 10 different mobilizations for each leg. Then, we calculated the proprioception score as the number of joints in which mobilization was perceived, meaning that a maximum score of 20 corresponded to a subject with proprioceptive function present at the entire lower limb area. Vibration measurement. For this assessment, a vibrating diapason was placed on different bony prominences, including: the ribs; anterior superior iliac spine (hip); patella (knee); medial and lateral malleolus (ankle); hallux, calcaneus bone and the sole of the foot (foot), while patients remained blindfolded in a supine position. Patients described the vibration sensations they experienced after the stimuli were delivered to the trunk and later to the lower limbs, following a random sequence that included bilateral areas like the anterior superior iliac spine, patella, medial and lateral malleolus, calcaneus, hallux, and the sole. Patients were asked whether they could feel the vibration and to describe the stimulus, considering two parameters: location (which area) and side of the body (right or left). The vibration score for each patient was obtained by summing the regions (rib, hip, knee, ankle, and foot) where the patient confirmed perceiving the stimulation (independently from their ability to place it on the correct body part or not). Note that the vibration and the confusion map should be considered as a whole to describe each patient’s vibration perception, the first one indicating if the subject can detect the presence of the stimulus, and the second describing the ability to localize it properly. To avoid a mechanism of learning by association, no feedback on the outcome was given to the patient; the experimenter did not tell the patient if his/her answers were correct or not. Also, throughout the assessment, the patients stayed blindfolded and in a supine position. Autonomic Visceral function evaluation. Visceral recovery was measured using questionnaires that were based on ISCOS datasets for urinary, intestinal and sexual functions [40– 43], in addition to direct clinical measurements included in the AIS sacral exam to evaluate anal sphincter function [6]. EMG recording and analysis. EMG activity was recorded using bipolar surface electrodes, amplified with actiCHamp amplifier (Brain Vision LLC, Morrisville, NC), and digitized at 2000HZ. Openvibe, an open-source software, was used for data collection [44]. A linear EMG envelope was obtained by rectifying and low-pass filtering the EMG signal, using the 2nd order anticausal Butterworth with 1Hz cutoff frequency. During the test, patients were suspended by the body weight support (Hocoma AG, Switzerland) while their legs were attached to the Lokomat working in passive mode (motors off and backdrivable). The physiotherapist verbally instructed patients to flex and extend the left or right hip for 4 seconds and to relax in between tasks. Quality of life assessment WHOQOL-BREF. The WHOQOL-BREF [36] is a cross-culturally valid self-assessment questionnaire with four Quality of life (QoL) domains: physical, psychological, social relationships and environment (see S1 Table). The score ranges from 1 to 5, on a Likert scale, according to the graduation of agreement or disagreement of the participant. The scores pointing 0 indicate poor QoL and 100 indicate good QoL. Note that higher values in WHOQOL-BREF always mean a positive effect; as such, a high score for the question ‘How often do you have negative feelings such as blue mood, despair, anxiety, depression?’ means low occurrences of negative feelings in the patient.
Study approval Our protocol was approved by both the local ethics committee (Associação de Assistência à Criança Deficiente, Sao Paulo, Sao Paulo, Brazil #364.027) and the Brazilian Federal
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
9 / 33
75 Sensory-motor, visceral and psychological improvement in paraplegics
Government Ethics Committee (CONEP, CAAE: 13165913.1.0000.0085). All participants signed written informed consent before enrolling in the study.
Results Sensory improvement Fig 2A depicts the normal tactile (dark pink) and altered (light pink) sensitivity (hyper or hypo-sensitivity)–which defines the zone of partial preservation (ZPP)–exhibited by each patient at the onset of the training. In this figure, each patient’s gains regarding normal (light pink) and altered (light blue) somatic sensitivity are also plotted for 12, 22 and 28 months of training. Overall, five patients (P1, P3, P4, P6, P8), all diagnosed with complete paraplegia (AIS A) at the onset of the training, recovered nociception in their trunk and lower limbs. The most dramatic improvement was observed in patient P6, whose T4 SCI occurred 8 years prior to our study. At the onset of training, this patient’s ZPP extended to T5-T6. However, 22 months after training onset, P6’s ZPP increased by 15 dermatomes on the right side, and 16 dermatomes on the left side. Patient P2 (T4, AIS B, motor complete) recovered normal nociceptive sensation in three to six dermatomes below the original SCI. Patient P5 regained altered sensations in the sacral dermatomes (S4-S5) 28 months after the training onset, even though he did not exhibit sensitivity below T7 on both right and left sides at the onset of training. The average increase of ZPP for the GR1 patients (excluding P2 and therefore only AIS A patients) is shown in Fig 2B. On average, patients recovered partial nociceptive sensation in areas situated 11 dermatomes below their original lesion (11.17 on the right side and 11.33 on the left side). Overall, the patients’s sensory score improvement rate for nociception during the 28-month WA-NR period was significantly higher than the spontaneous recovery rate registered before the onset of the training (calculated as the difference between the score registered at onset of the training and the baseline score) (Fig 2C, Wilcoxon rank sum test, n = 7, p<0.001). Fig 2D displays the temporal evolution of the sensory score improvement for GR1 patients (black line), compared to the score at the onset of the training for nociception (NC), crude touch (CT), pressure (PR) and temperature (TE). The GR1 patients exhibited a consistent and significant increase between the 4th and 28th month of training in NC (12.15 ± 2.42; mean± SE, Wilcoxon test, p<0.001), CT (10.14±1.16, p<0.001), PR (14.28±2.13, p<0.001), but no significant improvement in TE (0.71±0.44, p>0.1). Periods with reduced training (between the 10th and 12th months) also coincided with the absence of changes (NC) or reduction (CT, PR) in the group averages. Consistent with this observation, patient P7’s scores (continuous red line) stagnated between the 12th and 28th months. Note that during the first year, this patient had exhibited an improvement rate above the GR1 average in NC, CT and PR sensations.
Proprioception, vibration sensitivity recovery, and body schema All patients exhibited a significant recovery in lower-limb proprioceptive sensitivity (Fig 3A). This assessment was first introduced 4 months into the training protocol. The mean proprioception score at the end of the 28 months of training was 17.14±1.89 (mean ± SE) over 20, in complete contrast with the measurement at the 4th month of the training, where no patient was able to report proprioceptive sensation in any of the tested joints (average group score = 0.0). After 28 months of training, six out of seven patients recovered proprioception up to the ankle and five patients up to the toes (see S3 Table). Patient P7’s proprioception improved up to the ankle after 10 months of training and 18 months later, despite no further training, this patient still preserved this improvement (Fig 3A).
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
10 / 33
76 Sensory-motor, visceral and psychological improvement in paraplegics
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
11 / 33
77 Sensory-motor, visceral and psychological improvement in paraplegics
Fig 2. Sensory improvement following the WA-NR training protocol. (A) Patients’ normal (dark pink) and altered nociceptive sensations (light pink) areas measured with the standard ASIA assessment [6] at the onset of the training. Dermatomes in dark blue represent body areas where patients recovered normal sensations after our training; areas in the light blue are those where patients gained altered sensation (reported after 12, 22 and 28 months of training). (B) Mean ± standard error of the mean (SEM) for gained dermatomes with normal (dark pink), altered (light pink) and zone of partial preservation (ZPP) with nociceptive sensation for GR1 patients except P2. (C) Mean±SEM of the nociception improvement rate during the period of the WA-NR (difference of score between the onset and the end of the training, normalized by the number of months between the two evaluations) compared to the mean improvement rate before the training (difference of score between baseline and onset measurement normalized by the number of months between the two measurements). (D) Means ± SEM for the gained sensory score for each assessment of GR1 patients for nociceptive sensation, crude touch, sensitivity to pressure and temperature compared to the onset of training. Patient P7 stopped the training after 12 months and was therefore reported separately from the group average (in red). The score obtained after 4 months of training is compared to the one registered after 12 and 28 months of training (t-test, � P<0.05, �� P<0.01, ��� P<0.001). https://doi.org/10.1371/journal.pone.0206464.g002
In parallel, the patients’ vibration sensitivity was enhanced markedly between the first measurement done after 4 months of training (4.0±0.94, mean ±SE) and the 28th month of training (7.71±0.97) (p<0.05, Wilcoxon test) (Fig 2B). Different from the other sensory modalities, we observed a notable decrease in patient P7’s vibration score after protocol discontinuation. As part of our evaluation protocol, patients were asked to report either the presence or the absence of sensation, and to identify the location on the body where they felt the vibration (Fig 3C and 3D). As training progressed, patients started to recover vibration sensations in more distal parts of the body. However, the newly recovered sensory areas were disorganized in terms of the patients’ body representation. Because of that effect, patients tended to perceive the vibration stimuli on more distal body areas than the point upon which the stimuli were delivered. In Fig 2D we use a new graphical representation, which we refer to as the vibration map, to depict the evolution of patient P6’s recovery on vibration sensation. During the first evaluation (4th month) this patient reported no sensation in both left and right legs. Two months later, the patient started perceiving two regions on the right side (the ribs and the hip) and three on the left side (ribs to the knee). However, when asked to report the stimuli location, the patient reported feeling that the vibration applied to his hips felt like it originated in the knees, whereas the vibration on his left knee felt as if it was delivered to his left ankle. In other words, the patient perceived the stimulation more distal than the actual location to which it was applied. Several other instances of spatial localization errors were observed with this patient throughout the assessments. However, with training, the overall vibration map became more organized, and by the end of the protocol, this patient was able to correctly perceive vibration stimulation up to the knee on the right side and up to the foot on the left side. Other patients in GR1 also experienced similar errors related to the correct spatial localization of a vibratory stimulus. They included, in addition to locating the stimulus in a more distal joint, locating it in a more proximal joint or in a contralateral joint. In all cases, these errors were triggered by the expansion of the sensory recovery (Fig 3E) below the original SCI. As a rule, patients often located the vibratory stimulus to more distal than proximal joints (39 cases vs. 5 respectively). The instances of misallocating the stimulus to a contralateral joint stayed constant over the training period (overall 36 cases). The highest incidence of these spatial localization errors took place between the 12th and the 16th month of training. After 22 months of training, and following a period in which the vibration score plateaued, the number of errors decreased. Consequently, the resulting vibration map became more organized.
Motor function improvements Motor functions were tested following the standard AIS evaluation of five key muscles [6] (Fig 4A). To have a complete picture, we assessed seven non-key lower-limb muscles (Fig 4B), three abdominal muscles and the anal sphincter (Fig 4C).
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
12 / 33
78 Sensory-motor, visceral and psychological improvement in paraplegics
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
13 / 33
79 Sensory-motor, visceral and psychological improvement in paraplegics
Fig 3. Proprioception and sensitivity to vibration. (A) Mean±SEM for proprioception (B) for vibration score for GR1 (black) and patient P7 (red). (C) Eight stimulation areas for the vibration test. Patients were blindfolded during the exam and had to report if they could feel the vibration and report the location of the stimulation (rib, hip, knee, ankle or foot). (D) Vibration map for patient P6. A solid line connects the actual vibration position (left circle (R)ib, (H)ip, (K)nee, (A)nkle and (F)oot) and the felt vibration position (right circle) by the patient. A dashed line means that the patient indicated the contralateral leg. (E) Occurrences of confusion toward a proximal joint (e.g., stimulation was done in the knee and patient-reported he felt in the hip), distal joint or with a contralateral joint. https://doi.org/10.1371/journal.pone.0206464.g003
Except for residual contraction in one muscle for patient P3 (motor score of 1 for extensor digitorium longus), all patient received a motor score of 0 for the all tested lower-limb muscles (12 per leg) at their admission (baseline measurement). This initial evaluation was confirmed by our medical team at the onset of the training, meaning that, before our training, none of our patients showed any voluntary motor activity in their lower limbs. Throughout the 28 months of training (Fig 4A), we observed continuous motor improvement in the key muscles; with first visible contractions (motor score = 1) and first active movements (without gravity action, motor score = 2) appearing after 7 months of training. After 22 months of training patients, P1 reached a motor score of 3 (active movement against gravity) for the hip flexion (S2 Video). After 28 months of training, we found active movements (score of 2 and above) in 10 out of 24 tested lower-limb muscles of patient P1, eight muscles for patient P3 and P8, four muscles for patient P4 and P5, and one for patient P6. We also observed motor function recovery in the abdominal muscles, in the three patients who had the highest lesions, namely P4, P5 and P6 (Fig 4C). Finally, in five patients we measured the presence of sphincter control, a muscle whose spinal motor roots originate in the most caudal part of the spinal cord (S4-S5). The lower extremity motor score (LEMS) was calculated (Fig 4D) by summing the scores for the lower-limb key muscles. The most prominent improvement was observed in the patients with more distal SCI locations (S2, S3 and S4 Videos), namely patients P1, P3 and P8 (lesion range T10-T11, AIS A) (final score respectively 10, 12 and 11). Patients with a more proximal SCI began to show motor improvements after a longer training period: on the 10th month for P2 (T4, AIS B), P4 and P5 (T7-T8, AIS A) and the 12th month for P6 (T4, AIS A). This observed difference in improvement rate among the patients was somewhat expected; the lower-limb muscles are innervated by lumbar and sacral roots. Therefore a patient with a T11 lesion is closer to this level than a patient with a T8 lesion. As in the case of the sensory recovery, patient P7 displayed an above-average improvement in the motor score during the first year (Fig 4E, red line). However, after protocol discontinuation, he underwent a partial motor regression over the next 16 months, ending up at a level comparable to what he had achieved at the 7th month of training. Overall, the motor improvement observed in each one of our patients (ranging from 4 to 12 points and, on average, 7.2 points by the end of the protocol) was highly significant when compared to the onset of training (0 for all), and with the spontaneous improvement rates reported in the literature (scores <1 point when the rehabilitation started a year after the injury [12]). In some cases, the lesion extended over several spinal levels (for example patient P8’s lesion is between T4 and L4). This raised an important question: are all the muscles that recovered activity innervated at the lesion level rather than below? In other words, could our observations be explained by a mechanism of spontaneous recovery at the lesion level of the lower motor neurons (a mechanism known as root recovery [45,46])? To answer this, we calculated, for each patient, the distance between the muscle’s innervation root and the lowest part of the anatomical lesion. For example, patient P4’s MRI scan revealed a lesion at the T7-T9 level. As expected, at the onset of the training all myotomes under the lesion level were silent (Fig 5A). In clear contrast, by the end of the protocol, we observed that recovery extended up to seven
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
14 / 33
80 Sensory-motor, visceral and psychological improvement in paraplegics
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
15 / 33
81 Sensory-motor, visceral and psychological improvement in paraplegics
Fig 4. Motor improvement. Clinical evidence. (A) Five key lower-limb muscles (proximal and distal rectus femoris, tibialis anterior, extensor hallucis longus and gastrocnemius) were evaluated eight times throughout the training [6] (0 to 28 months) and before starting the training (B: baseline). The motor score describes the amplitude of a contraction from 0 (absence contraction) to 5 (normal contraction). (B) Motor score for seven non-key lower-limb muscles measured at the onset of the training (0) and the end of the end of the training (28 months) for all patients. (C) Number of patients (over eight) with present muscle contraction for three abdominal muscles and the anal sphincter muscle (myotome S4-S5). (D) The LEMS is obtained for each patient by summing the score of all key muscles reported in panel A bilaterally. Missing data periods are in black. (E) Mean± SEM of the motor score for GR1 patients is reported in black and motor score for patient P7 is in red. https://doi.org/10.1371/journal.pone.0206464.g004
spinal levels below the original anatomical lesion (e.g., a level 2 contraction found in the gluteus maximus muscle which is rooted in L4). We performed the same analysis for patients P2, P3, P5, P6 and P8 (Fig 5B). The anatomical lesion level (AL) for all these patients was obtained through analysis of MRIs (Table 1, S2 Fig). At the onset of the training, we detected very few cases of active myotomes below the AL. A very different result was observed 28 months later (at the protocol’s end), we observed multiple instances of voluntary contractions in muscles that are innervated by spinal nerves that originate below the original AL. For example, for patient P2 we observed contraction in muscles that originated up to 13 spinal segments below the AL; similarly, for patient P3, we identified voluntary muscle contractions five segments below the AL, six for P5, 11 for P6 and two for P8 (Fig 5B). Therefore, we found that the motor improvement in our patients was well below the original SC lesion level and could not be simply explained by a spontaneous root recovery mechanism. The occurrence of significant motor recovery in all patients was further corroborated through surface EMG recordings of voluntary contractions in multiple lower-limb muscles (Fig 5C). For example, in the case of the gluteus maximus (GMx, hip extensor, L5-S2), we observed a significant increase in contraction force in all patients. The first EMG measurements were obtained 7 months after the training onset. At this point, we did not observe significant contractions in patients P2, P4, P5, P6, and P8. However, after 2 years of training, the same five patients regained the capacity to control GMx voluntarily: P2 and P8 activated GMx bilaterally; P4 and P5 consistently contracted their left GMx. Patient P6 performed less frequent contractions; they were nevertheless aligned with therapist instructions suggesting that the motor activations were voluntary. Patients P1 and P3 experienced the earliest signs of motor recovery; they began producing contractions of their previously paralyzed muscles 7 months after the protocol onset, and contraction force strengthened with continued training. Patient P7 went from 0 to consistent left gluteus contractions between the 7th and 13th month of training.
Visceral functions improvements Our patients also exhibited an expressive autonomic function recovery, which was represented by consistent improvements in intestinal, urinary and sexual function, as described in Tables 3–5. Overall, five patients, whose condition changed from AIS A/B to C 28 months after training onset, (P1, P2, P3, P4, P8) regained voluntary anal sphincter motor control (Table 3). Moreover, all seven patients recovered pain and deep pressure sensitivity at the last sacral dermatome, assessed using a pinprick test and the deep anal pressure evaluation, after 2 years of WA-NR training. Consequently, all of them significantly improved their ability to inhibit defecation voluntarily (Table 4). Four patients recovered the ability to experience anal sensation during defecation and, even more importantly for the patients’ quality of life, five regained awareness of the need for defecation.
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
16 / 33
82 Sensory-motor, visceral and psychological improvement in paraplegics
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
17 / 33
83 Sensory-motor, visceral and psychological improvement in paraplegics
Fig 5. Motor improvement. Neurological and neurophysiological evidence. (A) Example of active myotomes compared to the Anatomical Lesion (AL) shown for patient P4. For each muscle, the graphic shows the corresponding myotome level (considering the principal nerve root, S2 Table), and the clinical score at the onset (0) and the end (+28 months) of the WARN training. The MRI of this patientâ&#x20AC;&#x2122;s spinal cord revealed an AL extending between T7 and T9 segments. Accordingly, at the onset of the training, the patient had preserved motor functions in the upper-limbs and the upper abdomen muscle (spinal nerve roots are located at T7-T8 level), but could not contract the middle and lower abdomen muscles (T9-T12 segment) nor any of the lower limb muscles. After 28 months of the WANR training, this patient had recovered partial motor functions in the middle and lower abdomen as well as in multiple lower limb muscle innervated under the AL, namely, rectus femoris proximal (L2) and distal (L3), hip adductor (L2), gluteus medius/maximus, tibialis anterior (L4), and gastrocnemius (S1). B) For each patient, we considered the muscles where motor functions were clinically observed (ASIA motor score = >1) and calculated the distance to the AL. The distance was calculated as the number of myotomes between the spinal nerve root of the muscle and the lowest segment of the AL. A positive value in the graph, corresponds to a muscle that is rooted below the anatomical lesion; and negative values refer to muscles rooted above the lesion. We report results for the onset (0) and the end of the training (28). Trunk muscles are reported with an open circle, lower limb muscles with an open triangle and upper limb muscles are not considered. (C) EMG envelops for the gluteus maximums muscle for all patients. Patients were instructed to contract their legs for periods of 5 seconds over a 3-minute period. Verbal instructions were given to the patient by the PT; and instruction periods are shown in gray in the graph. A dark gray area highlights the trials where the patient had a significant GMx contraction (> mean + 3xSD of the baseline), and light gray indicates those where the contraction did not reach significance. Muscle responses are shown for all patients (P1 to P8) at an early stage and later in training. https://doi.org/10.1371/journal.pone.0206464.g005
Four patients became capable of voluntarily inhibiting urination (Table 5), while six subjects recovered their ability to perceive the need for bladder emptying and five to perceive the catheter during bladder emptying. Due to these improvements, these patients no longer had to rely on indirect clinical signs, such as sweating, tachycardia and increased spasticity to sense the need to empty their bladders. Such an improvement in urinary functions possibly contributed to the reduction of urinary infections. Indeed, one patient (P6), who experienced repeated cases of lower tract urinary infections before the training onset, exhibited a marked decrease in these events over time: considering the 28 month time range of our protocol, this patient experienced four episodes of lower tract urinary infection during the first year, two during the second year and none during the last 6 months of training. We also observed improvements in sexual and genital function in both female and male patients (Table 5). Both female patients recovered sensitivity during sexual intercourse, as well as awareness and sensitivity to menstruation flow and cramps. While the reflexive erection (controlled by parasympathetic centers S2-S4) was preserved in all male subjects, three of them started experiencing psychogenic erections (physiologically controlled by supraspinal sympathetic centers and thoracolumbar sympathetic spine center at T11-L2), sensitivity during intercourse and ejaculations (conveyed by synergism between supraspinal and spinal centers T11-L2 and S2-S4) functions. Overall, improvement in bowel, bladder, and sexual function suggested that patients experienced some degree of neurological recovery at the level of sacral segments (S2-S5), which was significantly below the patientsâ&#x20AC;&#x2122; original SCI. Table 3. ASIA sacral evaluations. Sacral (ASIA)
0
4
7
10
12
16
22
28
0
0
0
1
2
3
4
5
Deep anal pressure evaluation (S4-S5 dermatomes)
1
1
1
3
4
4
6
7
Pinprick pain sensitivity (S4-S5 dermatomes)
1
1
1
3
4
4
5
7
6
6
6
6
6
6
6
7
Motor Voluntary anal contraction (S4-S5 myotomes) Sensitivity
Reflex Pinprick reflex evaluation (S4-S5)
One motor, two sensory and one reflex tests about the last sacral dermatome/myotome function. Tests measured directly by a physician [6]. Results are shown for eight measurements done throughout the training period (at training onset and after 0, 4, 7, 10, 16, 22, and 28 months) for GR1 patients (n = 7). https://doi.org/10.1371/journal.pone.0206464.t003
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
18 / 33
84 Sensory-motor, visceral and psychological improvement in paraplegics
Table 4. Intestinal function evaluation. Intestinal
0
4
7
10
12
16
22
28
Ability to voluntary inhibit defecation
0
0
0
0
1
1
2
5
Fecal incontinence
1
1
1
1
0
0
0
0
During feces elimination (defecation)
1
2
2
2
2
2
4
4
Awareness of the need for bowel emptying
1
1
1
1
2
2
5
5
Motor
Sensitivity
Two motor and two sensory tests about the bowel/intestinal functioning for GR1 (n = 7). Tests were collected through a self-questionnaire based on ISCOS data sets [40,41]. https://doi.org/10.1371/journal.pone.0206464.t004
Principal factors for sensory and motor recovery Having documented a marked sensory-motor recovery in our patients, we next investigated which aspects of the WA-NR training protocol (TR) and which external factors (EX) could best account for these partial clinical improvements. Improvement in nociception in our patients was significantly correlated with BMI training hours (Fig 6A, R = 0.29, P = 0.03, Pearson coefficient of correlation), whereas correlation with TR-LOC hours (R = 0.23), did not reach significancy (P = 0.08). Thus, the improvement rate in nociception was most marked during a period with increased BMI-based training. The same was true for the improvement rates for crude touch sensation (R = 0.44, P = 0.002 for TR-BMI, R = 0.17 for TR-LOC, n.s), for pressure improvement (R = 0.43, P = 0.006, versus R = 0.04, n.s.), for proprioception (R = 0.33, P = 0.04 versus R = 0.07, n.s.) as well as for motor improvements (R = 0.34, P = 0.01, for TR-BMI, and 0.17, n.s., for TR-LOC). Factors like the patientsâ&#x20AC;&#x2122; age and time since their SCI had no influence on the improvement rate in any of the measured sensory-motor metrics. None of the external factors were correlated with the improvements. Table 5. Genitourinary evaluations. Urinary
0
4
7
10
12
16
22
28
Ability to voluntary avoid urination
0
0
0
0
0
0
1
4
Involuntary urine leakage
7
5
6
6
6
6
6
3
Awareness of the need for bladder emptying
1
2
1
1
1
1
5
6
During bladder emptying with catheter
1
1
1
1
1
1
4
5
0
4
7
10
12
16
22
28
Sensitivity: during sexual intercourse
0
0
0
0
1
1
2
2
Sensitivity: Menstruation awareness
0
0
0
0
0
0
2
2
Sensitivity: During sexual intercourse
1
1
1
1
1
1
1
2
Motor: psychogenic erection
0
0
0
0
0
0
0
2
Motor: Reflex
5
5
5
5
5
5
5
5
Motor: Ejaculation
0
0
0
0
0
0
0
2
Motor
Sensitivity
Sexual, Genital Female
Male
Responses to the questionnaire [40,41] related to urinary functions are divided into motor and sensitivity aspects for all patients (n = 7). Questions related to sexual aspects are divided between female (two participants) and male (five participants) patients. https://doi.org/10.1371/journal.pone.0206464.t005
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
19 / 33
85 Sensory-motor, visceral and psychological improvement in paraplegics
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
20 / 33
86 Sensory-motor, visceral and psychological improvement in paraplegics
Fig 6. Neurological improvement and correlation with training the protocol. (A) The coefficient of correlation between sensory and motor neurological improvements and the number of training hours of active locomotion, BMI-based training, patients SCI height, time since lesion and patients ‘age. (B) AIS grade improvement at 28 months. Patient P7 stopped the training after 12 months. A follow-up measurement was done with this patient, 16 months after he stopped the training. (C) WHOQOL-BREF [36] score for four subdomains. https://doi.org/10.1371/journal.pone.0206464.g006
Altogether, all seven patients who remained enrolled in the WA-NR protocol for 28 months changed their initial AIS classification (Fig 6B, see details in S1 Fig). Patients P1, P3, and P8, who had the lowest lesions (T10 –T11), and P2 (AIS B at admission), changed their AIS grade during the first year. Patients P4, P5 and P6, who had higher lesions (T4-T8), changed AIS grade during the second year of training. P7 (T5/T7) was the only patient that stagnated; possibly because he discontinued the training early.
Quality of life and neuropathic pain To measure the patients’ quality of life (QoL), the WHOQOL-BREF (World Health Organization Quality of Life Assessment Instrument-Bref) questionnaire was applied six times (first time after 7 months of training) with GR1 patients. Total and partial scores, divided into four domains (physical, psychological, social and environmental) are exhibited in Fig 6C. We observed a significant increase in the physical domain between the 7th (76.0±4.5, mean±SEM) and 22nd months (82.1±3.9) of training (t-test, P<0.05). Similar to the clinical motor score, the WHOQOL physical domain score decreased during the break period between the 10th and 12th months of the protocol (Table 2). In the psychology domain, we found a significant improvement throughout the training (from 80.9±3.38 to 88.13±3.7, considering the seventh and 28th month, t-test, P<0.05). We, however, did not see any significant changes for the social (80.9±3.9 to 84.5±4.6, P>0.1) and environment (68.3±3.9 to 68.7±7.3, P>0.1) domains, the two aspects on which our protocol did not focus. The details for the physical domain are shown in Table 6. Following training, patients reported better ability in performing their activities of daily living; as well as their capacity for working. This suggests that the training with the WA-NR helped patients reacquire confidence in their ability to work and to achieve higher goals in their lives. The score for pain and discomfort started and remained positive over the 2 years of the training period; patients did not report ‘that physical pain prevents [them] from doing what [they] need to do.’ For a more detailed measurement of this aspect, the McGill standard evaluation was also applied (Table 7). We considered only the cases of neuropathic (or more precisely, myelopathic) pain like burning/tingling, shocks perceived in areas at the level and under the lesion) and discarded those that were related to external factors (headaches, a postural pain above the lesion level unrelated to the training, etc.). Patient P4, P5, P6, and P7 only presented a few cases of light neuropathic pain (McGill score of 1). Patients P1 and P8 reported a moderate and severe case of pain, respectively, at the onset of the training. The pain reported by P8 was especially uncomfortable, describing it as a sensation of burning in the feet; P1 described moderate pain similar to the sensation of shocks in the leg. The perceived pain attenuated for both patients after following the WANR training, downgrading to 0 for P1 and 1 for P8 (light pain). In one case an increase of neuropathic pain was observed after the onset of the training. Following partial neurological recovery, patient P3 reported a moderate pain in the right thigh. We found that the patients’ level of fatigue and energy stayed constant during all periods of training and only decreased during the break period (between the 10th and the 12th months); possibly because the WA-NR promoted routine physical training for our patients. Our training did not positively or negatively influence the patients’ sleep. The mobility item measured
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
21 / 33
87 Sensory-motor, visceral and psychological improvement in paraplegics
Table 6. Mean ± SEM score for all physical domains of the WHOQOL-BREF [36] questionnaire. 7
10
12
16
22
28
Pain and discomfort
WHOQOL physical domains
92.9±4.6
96.4±3.6
89.3±7.4
89.3±7.4
96.4±3.6
92.9±7.1
DIFF (28–7) 0.0
Energy and fatigue
85.7±5.1
82.1±7.1
64.3±7.4
82.1±7.1
82.1±7.1
89.3±5.1
3.6
Sleep and rest
75.0±7.7
71.4±6.5
78.6±6.5
71.4±8.5
78.6±6.5
71.4±10.1
-3.6
Mobility
82.1±4.6
75.0±0.0
71.4±3.6
71.4±3.6
78.6±3.6
78.6±6.5
-3.6
Activities of daily living
71.4±6.5
78.6±3.6
71.4±8.5
75.0±7.7
82.1±4.6
85.7±5.1
14.3
Dependence on medication
64.3±14.3
82.1±10.5
78.6±6.5
78.6±10.1
78.6±11.5
67.9±11.8
3.6
Work capacity
71.4±8.5
75.0±5.5
67.9±7.1
67.9±7.1
78.6±3.6
78.6±3.6
7.1
The questionnaire was done respectively after 7,10, 12, 16, 22 and 28 months of training. In the last column, we report the difference between the mean score after 28 months of training and the mean score of the first assessment. https://doi.org/10.1371/journal.pone.0206464.t006
patients’ autonomy and accessibility to perform their daily life activities (e.g., indoor accessibility.); the item ‘dependence on medication’ measured patients general use of medication (considering both chronic and acute pain). As expected, our training did not influence these two items. Concerning the psychological domain, we documented an improvement in five out of the six sub-items (Table 8). Patients’ reported enjoying their lives more (an increase of 7.1 for positive feelings), corroborated with a decrease in the occurrence of negative feelings, such as blue mood, despair, anxiety, and depression (improvement of 10.7 points). Patients reported being more satisfied with themselves (+7.1 point in self-esteem) reaching a very high score by the end of the training (96.4%). We believe that the patients’ improvement in the thinking/ Table 7. McGill score. P1
P2
P3
P4
P5
P6
P7
P8
Months of training
0
28
0
28
0
28
0
28
0
28
0
28
0
12
0
Throbbing
0
1
0
0
1
1
0
2
0
0
1
0
1
1
3
0
Shooting
2
0
0
1
0
0
0
0
0
0
0
1
1
1
0
0
Stabbing
0
0
0
0
0
0
0
0
0
0
0
0
1
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2
1
0
0
0
0
0
0
1
0
Sharp Cramping
28
Gnawing
0
0
1
0
0
0
0
0
0
0
1
0
0
0
0
0
Hot-burning
0
0
0
0
0
0
0
1
1
0
0
1
0
1
3
1
Aching
0
0
0
0
1
2
1
0
0
0
0
0
0
0
1
2
Heavy
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Tender
0
0
0
0
0
2
0
1
0
0
0
0
0
0
1
2 0
Splitting
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Tiring-exhausting
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
0
Sickening
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3
0 0
Fearful
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Punishing-cruel
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Sum
2
1
1
1
2
5
3
5
1
0
2
2
3
4
14
5
Sum neuropathic
2
1
1
1
1
5
0
1
1
0
2
1
2
2
3
1
Detail for the McGill score considering the descriptors for two evaluations, at the onset (0) and at the end of the training (after 12 months for P7 and after 28 months for the other patients). Cases where the descriptors of the pain are below the lesion are considered as neuropathic pain and underlined, those above the lesion (as postural pain, headache, etc.) are not considered to be due to the spinal lesion and therefore not considered for the current analysis. https://doi.org/10.1371/journal.pone.0206464.t007
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
22 / 33
88 Sensory-motor, visceral and psychological improvement in paraplegics
concentration/memory item reflected the fact that training with the BMI specifically required subjects to focus on their rehabilitation tasks. Several studies have shown that BMI training is beneficial for improving focus and concentration [47]. Also, we observed that body-image plateaued at a high score throughout the training (~90%). Finally, one of the most interesting results came from asking whether patients ‘[..]feel [their] life to be meaningful’ (spiritual/personal beliefs). We observed a continuous improvement over the training period, suggesting that the participation in the WA-NR protocol had a positive impact on the patients’ selfesteem.
Discussion The present study reports a systematic and unprecedented partial neurological recovery in patients diagnosed with chronic complete (AIS A) and motor complete (AIS B) paraplegia, following long-term non-invasive neurorehabilitation [33]. Based on a study that gathered clinical data from a group of eight SCI patients over a 28-month period, we found that the longer the patients trained under the protocol that combined a BMI, visuo-tactile feedback and active locomotion, the larger was the sensory-motor and visceral recovery observed below the SCI level. This was true for both somatosensory (tactile, nociceptive, proprioceptive, pressure and vibration), motor (voluntary contraction observed for multiple myotomes below the original lesion), and autonomic functions (sexual, intestinal and urinary). At the core of the WA-NR protocol, concurrent BMI-based control of virtual and mechanical actuators combined brain activation with continuous visuo-tactile feedback and physical training, assisted by a body weight support system and robotic gait therapy devices. Thus, the principal difference between the WA-NR protocol and other existing neurorehabilitation paradigms is that it focuses neither on the physical, nor on the BMI approach per se, but instead it creates the conditions to simultaneously engage both cortical and peripheral signals that converge towards the level of the SCI. As such, the intended goal is to reinforce the potential physiological role played by spinal tracts that have survived the original injury. Depending on the cause, a traumatic SCI can lead to a variety of lesions, generated by contusion, compression or penetration, which is followed by massive necrosis of affected neural circuits during the first 18 hours post-injury [48]. During the subsequent weeks, the neuroimmunological system deflagrates a cascade of mechanisms that may expand the lesion above and below its original epicenter. Cavity or cyst formation and demyelination may also occur, damaging both ascending and descending pathways [48]. Previous studies have shown that 84% of clinically complete paraplegics (AIS A) exhibit some neurophysiological activity below the level of the injury [7,49] and are therefore referred to as “discomplete” SCI [8]; post-mortem analysis by Kakulas et al. [9] confirmed the presence of fibers in patients diagnosed as having a clinically complete injury. Confirming this assessment, MRI scans of our patients showed the residual spinal cord continuity in three out of the six patients (two AIS A, one AIS B). Based on these new observations, we suggest that even a small portion of surviving spinal cord axons can contribute to a meaningful clinical and functional recovery, provided that they can be properly re-engaged by a long-term rehabilitation protocol like ours. In support of this point of view, animal models of SCI have shown that sparing of about 10–15% of the spinal cord is sufficient to support a partial recovery of locomotion [50]. In human subjects, the exact amount of spared SC white matter needed to observe similar levels of functional neurological recovery remains unknown. However, in a study in which a cordotomy was performed in 44 SCI patients [51] to alleviate secondary cancer-induced pain, Nathan et al. showed that even a complete bilateral section of the anterior portion of the spinal cord, containing motor tracts,
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
23 / 33
89 Sensory-motor, visceral and psychological improvement in paraplegics
Table 8. Mean ± SEM score for all psychological domains of the WHOQOL-BREF [36] questionnaire. 7
10
12
16
22
28
Positive feelings
WHOQOL psychology domains
71.4±3.6
78.6±3.6
75.0±5.5
75.0±5.5
82.1±4.6
78.6±6.5
DIFF (28–7) 7.1
Self-esteem
89.3±5.1
85.7±5.1
89.3±5.1
89.3±5.1
96.4±3.6
96.4±3.6
7.1
Thinking, memory and concentration
82.1±7.1
82.1±4.6
82.1±4.6
78.6±6.5
89.3±5.1
89.3±5.1
7.1
Bodily image and appearance
89.3±5.1
89.3±5.1
85.7±5.1
92.9±4.6
92.9±4.6
89.3±5.1
0.0
Negative feelings�
78.6±3.6
85.7±5.1
85.7±5.1
89.3±5.1
92.9±4.6
89.3±5.1
10.7
Spirituality/personal beliefs
75.0±5.5
78.6±3.6
78.6±6.5
78.6±8.5
82.1±4.6
85.7±5.1
10.7
The questionnaire was done respectively after 7,10, 12, 16, 22 and 28 months of training. The difference of mean score after 28 months as compared to the mean score of the first assessment. �
High scores mean fewer negative feelings.
https://doi.org/10.1371/journal.pone.0206464.t008
did not significantly affect patients’ motor functions. They suggested that the motor tracts in the posterior half of the spinal cord could compensate for the functions originally mediated by the anterior portion. Our studies confirm the role played by the employment of BMI-based training to induce both partial sensory and motor recoveries. We found that periods of increased hours of BMIbased training yielded the most pronounced clinical improvements. Importantly, we also observed that the improvement rate was dependent neither on the period since the patients’ original SCI, nor the patients’ age. But what mechanism could account for this recovery? We hypothesize that by triggering an extensive process of cortical and spinal cord functional plasticity, our BMI paradigm created the conditions for our patients to recover sensory-motor and visceral functions. These results are in accordance with recent observations in a rat model [27,31]. Moreover, our results indicate that both neurological and local muscular factors contribute to the final motor outcome following the WA-NR protocol. Indications that recovery is happening due to neurological factors include a proximal-to-distal order of recovery. The L2 -innervated muscles had voluntary contractions in all eight patients, while more distal levels like L5 were seen in two patients and S1 in one patient only. If there were only neurological factors involved in the final outcome, it should be expected that all muscles innervated by a nerve root would respond equally, which is not the case. The analysis of L4-innervated muscles allows us to compare the tibialis anterior (key-muscle) with five other muscles (gluteus maximus, gluteus medius, medial hamstring, lateral hamstring, and extensor digitus longus) which differ in size and location. The glutei have shown better scores among the majority of patients, while hamstrings and extensor digitus longus had poor responses. Tibialis anterior, which is a key-muscle for L4 evaluation had lower scores than the glutei, showing that non-key muscle testing is indeed important to rule out inhomogeneities of recovery due to local muscular factors. Previously, BMI-evoked cortical plasticity has been reported in the rehabilitation of stroke patients (36). These studies have shown a considerable clinical effect of BMI training even in patients with severe neurological impairment. Similar to the classical physical therapy protocols for incomplete SCI [52], repeated active motor tasks, which promote activity-dependent rehabilitation, have been recognized for inducing partial motor recovery in stroke patients [53,54]. Constraint-induced movement therapy, for example, has been successfully used for stroke rehabilitation, even at the chronic phase [55]. However, this approach relies on the utilization of the patient’s residual motor functions, a condition that is absent in 30–50% of stroke victims [56]. Therefore, BMI-based therapy was adopted for these most severe stroke cases, in which no residual motor function was present. This method aimed at rehearsing a lost motor
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
24 / 33
90 Sensory-motor, visceral and psychological improvement in paraplegics
function, through real-time decoding of the patient’s motor imagery and online feedback (visual or proprioceptive [57,58]). In this approach, the BMI serves as a cortical operant conditioning (as shown in pioneer work of Fetz and collaborators [59]). We believe that a similar neurophysiological mechanism was activated in our SCI patients when they were subjected to the WA-NR protocol. At the onset of the training, our patients, who were diagnosed with complete paraplegia, could not follow classical training with the active motor task. Instead, our BMI-based neurorehabilitation, which integrated decoding of cortical motor imagery with visuo-tactile feedback, provided the driving force for triggering the kind of cortical plastic process that mediated their recovery. Concomitant to a partial somatosensory and motor recovery, we also demonstrated that training with the WA-NR protocol induced a significant change in our patients’ perception of their own bodies, as evident from the partial recovery of tactile, proprioceptive and vibratory sensitivity. Thus, our protocol affected the patients’ body schema, which involves the multisensory integration of visual, kinesthetic and proprioceptive information to provide humans with a sense of being and existing in space [60,61]. Previous studies have shown that the body schema is plastic and, hence, can become distorted after SCI [62,63] or a limb amputation [64]. At training onset, patients did not detect the presence of vibration stimulation in their lower limbs. When they first began perceiving tactile sensations from the parts of their body that had remained numb since their original SCI (i.e., for many years), their perception of the tactile stimuli location was distorted. As a rule, after the patients began to regain sensations in their legs, they tended to perceive vibration in more distal parts of the leg, compared to the actual stimulation site. One explanation of this effect is the sensory stimulation activated not only the cortical areas representing the stimulated body part, but it also invaded adjacent representations of the more distal parts that had remained silent for a long time. Such “filling-in” of sensory deprived areas of the primary somatosensory cortex has been previously reported in animal models [65] and amputee patients [66,67]. But, the observation of a constant number of misplacement errors toward the opposite leg, an area not adjacent in the cortical somatotopic representation, suggests additional mechanisms to the classic ‘invading’ representation explanation. A possible explanation is that the peripheral information entered the somatotopic representation of the stimulated body part but was interpreted incorrectly because of the body schema distortion caused by many years of sensory deprivation. Following the sensory recovery in the lower-limbs, we observed that at first the patients could detect the presence of stimulation but their spatial representation was distorted, and then, the sensory map became more organized. We believe that the extensive training using congruent visual, tactile and proprioceptive signals, was essential for patients to recover a more structured body representation. Indeed, we have demonstrated in a previous study that the congruent use of the virtual reality and tactile feedback setup used in the WA-NR induced a significant change of body schema representation in SCI patients [35], in coherence with numerous studies showing this effect with healthy subjects [68,69]. Overall, we propose that the temporal evolution of our patients’ perception of their bodies emerged from a complex process of activity-dependent plasticity, occurring in the body representations that exist at the cortical level, which altogether define the patient’s body schema [60]. Likely, this cortical plasticity was paralleled by a similar process taking place at subcortical levels, both of which were influenced by peripheral contributions to this body representation. In addition to lower limb sensory-motor dysfunction, the most devastating effects of SCI are genitourinary, gastrointestinal and sexual dysfunction [70]. Surveys have shown that paraplegic patients ranked as their first priority the recovery of sexual functions (26%), followed by bladder/bowel functions (18%), before movement and sensation recovery (16% and 7.5%) [71]. In our study, five out of
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
25 / 33
91 Sensory-motor, visceral and psychological improvement in paraplegics
the seven patients became aware of the need for bowel emptying and recovered the ability to inhibit voluntary defecation 22–28 months after the training onset. In addition, four patients regained the ability to inhibit urination and six recovered awareness of the need for bladder emptying. These changes had a very positive impact on the patients’ health, as they reduced the risk of patients acquiring a urinary infection and exposure to autonomic dysreflexia that could lead to uncontrolled hypertension. Significant gains in the patients’ ability to inhibit urination also gave them more flexibility in their daily activities and improved the social interactions. Another fundamental observation of this long-term study was that four patients (two males and two females) partially recovered their sexual functions. This observation suggests that the partial neurological recovery obtained was not constrained to the lower-limb somatosensory and motor functions, which were the primary focus of our BMI and the physical training, but, instead, resulted from a generalized neurological recovery, which also manifested itself as an improvement in major visceral functions. Although the rate of clinical improvement of the patients was higher during the first year of training compared to the second, the patients’ recovery continued to improve for the 28 months of training. Yet, it is not entirely clear at this point to what extent the effects of WA-NR are preserved after a patient stops this training. The results for one patient showed that, after displaying promising clinical improvements, above the group average, during the first 12 months of training, this patient maintained the majority of his sensory gains for the remaining 18 months, even after he discontinued training. This was true for the rate of improvement for tactile, nociceptive, proprioception and pressure sensitivity modalities, but not for vibration. Moreover, upon discontinuation, the same patient exhibited a clear regression of motor functions gained during the first year of training but did not return to his starting point (which was equal to zero at the protocol onset). These observations indicate the importance of continued engagement with the WA-NR protocol. Longer follow-up periods with more patients will be needed to evaluate the long-term effects of discontinued training on the partial gains in sensory, motor, and visceral functions obtained with the WA-NR protocol. During our research, the patients had integral psychological support with the purpose of guiding expectations adjustments, regarding the neurological improvements that the patients could potentially exhibit. Our major concern was the impact of the walking training on patients, as subjects were previously prepared to independently perform activities of daily living (ADL) in a wheelchair and were already adapted to this condition. An important question that was raised in this study was how the training would impact the patients’ own subjectivity and change (if it does) their quality of life (QoL) perception. QoL is not merely the absence of disease, but the state of complete physical, psychological and social wellbeing. In recent years, QoL improvement for SCI has become a rehabilitation goal [72,73], and its assessment is considered beneficial for evaluating the outcome of a multidisciplinary rehabilitation team approach. The patient’s self-perception of the QoL is considered an efficacy measurement of treatment and can also be used to evaluate the cost-effectiveness of interventions, and research and rehabilitation programs [74]. In our study, we observed increases in total, physical and psychological subdomains of QoL following our protocol. The physical domain exhibited fast improvement at the onset of the training, and a decrease during the break period, stressing the importance of maintaining continuous rehabilitation activities in these patients. Unlike changes observed in the physical domain, the psychological domain exhibited a gradual improvement and a long-term effect on the course of training with the WA-NR protocol. While partial neurological recovery can contribute to an improvement in QoL in spinal cord injury patients [75], we suggest that another important factor can explain the positive effect observed during the execution of the WA-NR protocol: patients were fully engaged in
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
26 / 33
92 Sensory-motor, visceral and psychological improvement in paraplegics
their training as they were encouraged to imagine themselves performing movements while they received rich visual, vibrotactile and proprioceptive feedback; in other words, patients were required to be active protagonists of their physical therapy and neurorehabilitation process. As part of our follow-up assessing the patients’ quality of life, we also controlled for instances of neuropathic pain. Below-lesion neuropathic pain (NP) is present in 34% of SCI patients [76] and can significantly reduce subjects’ quality of life and compromise the neurorehabilitation process [77]. In our case, as confirmed by WHOQOL results, we did not observe cases of NP that prevented patients from performing their training normally. However, in one case a severe NP reported at the onset of the protocol, and described as a sensation of burning in the feet, was later alleviated. We propose here a hypothesis for this mechanism of reduction of NP. Motor cortex (M1) is thought to play an important role in modulation of pain [78–80]; indeed stimulation of M1 is a well-studied treatment for chronic NP especially after deafferentation (post-stroke pain, brachial plexus avulsion, phantom limb pain and also post spinal cord injury NP). By promoting higher activation in the sensory-motor cortical areas, the BMI may have played an important role in the reduction of NP. Indeed, in a case study with one chronic SCI patient, 4 months of training with a non-invasive BMI was shown to promote a reduction in neuropathic pain [81]). Further studies with a group of patients with higher levels of neuropathic pain are necessary to investigate this hypothesis in a more comprehensive way. Overall, our patients’ training with the WA-NR promoted a sense of empowerment [82] and acted by strengthening the sense of competence, self-worth, and self-esteem of our patients. It allowed the individual to overcome the situation of helplessness and develop control over their own lives. The concept of patients’ empowerment has been gathering interest in various health care domains [83–88] since it promotes the concept of self-determination of patients as agents of their health and healthcare [89]. We suggest that the empowerment component may have influenced the perception of our patients’ capacity to contribute to their own recovery and, consequently, help improve their QoL. This process generates a sense of autonomy for patients, as they realized that they were actively involved in seeking some degree of clinical improvement. This improvement resonated in patients’ daily habits, as, for example, their dressing habit; for the first time since the lesion, two patients started using shorter cloth (skirts/shorts) revealing parts of their body they had been hiding. Overall, our long-term clinical results open new therapeutic perspectives for the rehabilitation of the most severe cases of SCI (AIS A and B), even at the chronic phase, while using a purely non-invasive BMI-based approach. Therefore, our findings suggest that existing or future technologies, created for incomplete AIS C patients, may also be used for patients originally classified as AIS A/B. For example, a large number of existing orthoses and exoskeletons, which require lower limb EMGs for their actuation (see [90] for a review), could now be potentially considered for use with AIS A patients, following a period of BMI-based training, using a protocol like the WA-NR. Our findings also indicate that, given a small fraction of spared spinal cord white matter, a much larger than expected population of AIS A/B patients might benefit from neurorehabilitation protocols that actively engage the patients’ mental and physical activity, while providing them with rich visual and tactile feedback.
Supporting information S1 Fig. ASIA score sheet for all patients at onset and at the end of the training. (PDF) S2 Fig. MRI of patients’ spinal cord. (A) MRI cuts of sagittal and (B) axial planes (T2 sequence) at SCI level for patients P2, P3, P4. Myelomalacia (hyperintense signal, 7mm length)
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
27 / 33
93 Sensory-motor, visceral and psychological improvement in paraplegics
is visible for patient P2 at the level of thoracic vertebra T2, and continuity of neural fibers are visible at the lesion level. For patient P3, we observed the spinal cord injury extending between thoracic vertebras T10 and L1 with the presence of remaining fibers (dark gray). T12 axial plane for the same patient reveals injury arachnoiditis (inflammation of spinal meninges) and fiber continuity (better visualization). For patient P4, close to the vertebral body fracture at T8-T9 level, we also observe neural fiber continuity; axial plane reveals dural sac septations at injury level and fiber continuity. (C) Example of 3D segmentation and projection on sagittal and coronal plans for patient P4, confirms spinal cord continuity at lesion level. (PDF) S1 Video. 3D MRI reconstruction. Three-dimensional reconstruction based on FIESTA sequence images for patient P4. The rendering was done with OsiriX Lite software. Published with permission of Associação Alberto Santos Dumont para Apoio à Pesquisa (AASDAP), Sao Paulo, Brazil. (MP4) S2 Video. Motor examination in a suspended position for patient P1. The motor exam was done 9 and 22 months after the onset of the training. The patient is instructed to flex the right hip. Published with permission of Associação Alberto Santos Dumont para Apoio à Pesquisa (AASDAP), Sao Paulo, Brazil. (MP4) S3 Video. ASIA motor examination of patient P3. The motor exam is done 32 months after the onset of the training. The patient is asked to align the lower limbs, performing hip adduction and knee extension, for the left side and later for the right side. Published with permission of Associação Alberto Santos Dumont para Apoio à Pesquisa (AASDAP), Sao Paulo, Brazil. (MP4) S4 Video. Motor examination in a suspended position for patient P8. The motor exam was done 29 months after the training onset. The patient was instructed to move both legs alternatively backward (right and later left side) and forward (right and later left side); with 75–80% of body weight support during forward movement and 65–70% during backward movement. Is possible to see EMGs electrodes placed at the lower limbs, for neurophysiology analysis, and the hands of a therapist are stabilizing the pelvis, avoiding body rotations. Published with permission of Associação Alberto Santos Dumont para Apoio à Pesquisa (AASDAP), Sao Paulo, Brazil. (MP4) S1 Table. WHOQOL-BREF set of questions per domain. (DOCX) S2 Table. For each muscle, the corresponding nerve, nerve root range. The reported principal nerve is the one reported in ASIA assessment (except extensor digitorum longus which is not part of the ASIA assessment). (DOCX) S3 Table. Patient’s proprioception score per movement after 28 months of training. (DOCX)
Acknowledgments We want to thank Neiva Paraschiva, Adriana Ragoni, Andrea Arashiro, Maria Cristina Boscaratto, Fabio Asnis, Nathan Rios, Seidi Yonamine (AASDAP, Associação Alberto Santos
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
28 / 33
94 Sensory-motor, visceral and psychological improvement in paraplegics
Dumont para Apoio à Pesquisa), Hougelle Simplı́cio Gomes Pereira (ISD, Instituto Santos Dumont) and Susan Halkiotis (Duke University) for their work, help and support for this study. We finally want to thank the patients for their long-term commitment, and their trust in this research.
Author Contributions Conceptualization: Solaiman Shokur, Ana R. C. Donati, Debora S. F. Campos, Miguel A. L. Nicolelis. Data curation: Eduardo J. L. Alho. Formal analysis: Solaiman Shokur, Ana R. C. Donati, Guillaume Bao, Chris Petty, Eduardo J. L. Alho, Mikhail Lebedev, Allen W. Song, Miguel A. L. Nicolelis. Funding acquisition: Miguel A. L. Nicolelis. Investigation: Solaiman Shokur, Ana R. C. Donati, Debora S. F. Campos, Claudia Gitti, Guillaume Bao, Dora Fischer, Sabrina Almeida, Vania A. S. Braga, Patricia Augusto, Miguel A. L. Nicolelis. Methodology: Solaiman Shokur, Ana R. C. Donati, Miguel A. L. Nicolelis. Project administration: Miguel A. L. Nicolelis. Supervision: Miguel A. L. Nicolelis. Validation: Ana R. C. Donati, Miguel A. L. Nicolelis. Writing – original draft: Solaiman Shokur, Ana R. C. Donati, Miguel A. L. Nicolelis. Writing – review & editing: Solaiman Shokur, Guillaume Bao, Eduardo J. L. Alho, Mikhail Lebedev, Miguel A. L. Nicolelis.
References 1.
Craig a, Tran Y, Middleton J. Psychological morbidity and spinal cord injury: a systematic review. Spinal Cord. 2009; 47: 108–114. https://doi.org/10.1038/sc.2008.115 PMID: 18779835
2.
Pazzaglia M, Galli G, Scivoletto G, Molinari M. A Functionally Relevant Tool for the Body following Spinal Cord Injury. Tsakiris M, editor. PLoS One. Public Library of Science; 2013; 8: e58312. https://doi. org/10.1371/journal.pone.0058312 PMID: 23484015
3.
Hess MJ, Hough S. Impact of spinal cord injury on sexuality: broad-based clinical practice intervention and practical application. J Spinal Cord Med. 2012; 35: 211–8. https://doi.org/10.1179/2045772312Y. 0000000025 PMID: 22925747
4.
Lude P, Kennedy P., Elfström M.L. and CSB. Quality of Life in and After Spinal Cord Injury Rehabilitation: A Longitudinal Multicenter Study P.
5.
World Health Organization, The International Spinal Cord Society. International Perspectives on Spinal Cord Injury. 2013.
6.
Ditunno JF, Young W, Donovan WH, Creasey G. The International Standards Booklet for Neurological and Functional Classification of Spinal Cord Injury. J Orthopsychiatry. 1994; 32: 70–80. https://doi.org/ 10.1038/sc.1994.13
7.
Sherwood AM, Dimitrijevic MR, Barry McKay W. Evidence of subclinical brain influence in clinically complete spinal cord injury: discomplete SCI. J Neurol Sci. 1992; 110: 90–98. https://doi.org/10.1016/ 0022-510X(92)90014-C PMID: 1506875
8.
Dimitrijevic MR, Faganel J, Lehmkuhl D, Sherwood A. Motor control in man after partial or complete spinal cord injury. Adv Neurol. 1983; 39: 915–26. Available: http://ovidsp.ovid.com/ovidweb.cgi?T= JS&PAGE=reference&D=emed1ab&NEWS=N&AN=6660129%5Cnhttp://www.ncbi.nlm.nih.gov/ pubmed/6660129 PMID: 6660129
9.
Kakulas BA, Gubbay DA, Lorimer RL. White matter changes in human spinal cord injury. Spinal Cord Monitoring. Springer International Publishing; 1998. pp. 395–407.
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
29 / 33
95 Sensory-motor, visceral and psychological improvement in paraplegics
10.
National Spinal Cord Injury Statistical Center and others. Facts and Figures at a Glance. Birmingham, AL: University of Alabama at Birmingham. 2015.
11.
Harrop JS, Naroji S, Maltenfort MG, Ratliff JK, Tjoumakaris SI, Frank B, et al. Neurologic improvement after thoracic, thoracolumbar, and lumbar spinal cord (conus medullaris) injuries. Spine (Phila Pa 1976). 2011; 36: 21–25. https://doi.org/10.1097/BRS.0b013e3181fd6b36 PMID: 21192220
12.
Steeves JD, Lammertse D, Curt A, Fawcett JW, Tuszynski MH, Ditunno JF, et al. Guidelines for the conduct of clinical trials for spinal cord injury (SCI) as developed by the ICCP panel: clinical trial outcome measures. Spinal Cord. 2007; 45: 206–221. https://doi.org/10.1038/sj.sc.3102008 PMID: 17179972
13.
Kirshblum S, Millis S, McKinley W, Tulsky D. Late neurologic recovery after traumatic spinal cord injury. Arch Phys Med Rehabil. 2004; 85: 1811–1817. https://doi.org/10.1016/j.apmr.2004.03.015 PMID: 15520976
14.
Steeves JD, Lammertse D, Curt A, Fawcett JW, Tuszynski MH, Ditunno JF, et al. Guidelines for the conduct of clinical trials for spinal cord injury (SCI) as developed by the ICCP panel: clinical trial outcome measures. J Orthopsychiatry. 2006; 45: 206–21. https://doi.org/10.1038/sj.sc.3102008 PMID: 17179972
15.
Curt A, Van Hedel HJA, Klaus D, Dietz V. Recovery from a Spinal Cord Injury: Significance of Compensation, Neural Plasticity, and Repair. J Neurotrauma. 2008; 25: 677–685. https://doi.org/10.1089/neu. 2007.0468 PMID: 18578636
16.
Oh SK, Jeon SR. Current Concept of Stem Cell Therapy for Spinal Cord Injury: A Review. Korean J Neurotrauma. 2016; 12: 40. https://doi.org/10.13004/kjnt.2016.12.2.40 PMID: 27857906
17.
Petraglia FW, Farber SH, Gramer R, Verla T, Wang F, Thomas S, et al. The Incidence of Spinal Cord Injury in Implantation of Percutaneous and Paddle Electrodes for Spinal Cord Stimulation. Neuromodulation J Int Neuromodulation Soc. 2016; 19: 85–90. https://doi.org/10.1111/ner.12370 PMID: 26644210
18.
Kawano O, Masuda M, Takao T, Sakai H, Morishita Y, Hayashi T, et al. The dosage and administration of long-term intrathecal baclofen therapy for severe spasticity of spinal origin. Spinal Cord. 2018; https:// doi.org/10.1038/s41393-018-0153-4 PMID: 29895878
19.
Barboglio Romo PG, Gupta P. Peripheral and Sacral Neuromodulation in the Treatment of Neurogenic Lower Urinary Tract Dysfunction. Urol Clin North Am. 2017; 44: 453–461. https://doi.org/10.1016/j.ucl. 2017.04.011 PMID: 28716325
20.
Gstaltner K, Rosen H, Hufgard J, Märk R, Schrei K. Sacral nerve stimulation as an option for the treatment of faecal incontinence in patients suffering from cauda equina syndrome. Spinal Cord. 2008; 46: 644–647. https://doi.org/10.1038/sc.2008.6 PMID: 18317481
21.
Redshaw JD, Lenherr SM, Elliott SP, Stoffel JT, Rosenbluth JP, Presson AP, et al. Protocol for a randomized clinical trial investigating early sacral nerve stimulation as an adjunct to standard neurogenic bladder management following acute spinal cord injury. BMC Urol. 2018; 18: 72. https://doi.org/10. 1186/s12894-018-0383-y PMID: 30157824
22.
Field-Fote EC, Roach KE. Influence of a locomotor training approach on walking speed and distance in people with chronic spinal cord injury: a randomized clinical trial. Phys Ther. American Physical Therapy Association; 2011; 91: 48–60. https://doi.org/10.2522/ptj.20090359 PMID: 21051593
23.
Harkema SJ, Hillyer J, Schmidt-Read M, Ardolino E, Sisto SA, Behrman AL. Locomotor Training: As a treatment of spinal cord injury and in the progression of neurologic rehabilitation. Arch Phys Med Rehabil. Elsevier Inc.; 2012; 93: 1588–1597. https://doi.org/10.1016/j.apmr.2012.04.032 PMID: 22920456
24.
Manella KJ, Roach KE, Field-Fote EC. Operant conditioning to increase ankle control or decrease reflex excitability improves reflex modulation and walking function in chronic spinal cord injury. JNeurophysiol. 2013; 109: 2666–2679. https://doi.org/10.1152/jn.01039.2011 PMID: 23468393
25.
Thompson a. K, Pomerantz FR, Wolpaw JR. Operant conditioning of a spinal reflex can improve locomotion after spinal cord injury in humans. J Neurosci. 2013; 33: 2365–75. https://doi.org/10.1523/ JNEUROSCI.3968-12.2013 PMID: 23392666
26.
Murillo N, Kumru H, Opisso E, Padullés JM, Medina J, Vidal J, et al. Recovery of assisted overground stepping in a patient with chronic motor complete spinal cord injury: A case report. NeuroRehabilitation. 2012; 31: 401–407. https://doi.org/10.3233/NRE-2012-00810 PMID: 23232164
27.
van den Brand R, Heutschi J, Barraud Q, DiGiovanna J, Bartholdi K, Huerlimann M, et al. Restoring voluntary control of locomotion after paralyzing spinal cord injury. Science. 2012; 336: 1182–5. https://doi. org/10.1126/science.1217416 PMID: 22654062
28.
Rejc E, Angeli CA, Atkinson D, Harkema SJ. Motor recovery after activity-based training with spinal cord epidural stimulation in a chronic motor complete paraplegic. Sci Rep. 2017; 7: 13476. https://doi. org/10.1038/s41598-017-14003-w PMID: 29074997
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
30 / 33
96 Sensory-motor, visceral and psychological improvement in paraplegics
29.
James ND, McMahon SB, Field-Fote EC, Bradbury EJ. Neuromodulation in the restoration of function after spinal cord injury. Lancet Neurol. Elsevier Ltd; 2018; 17: 905–917. https://doi.org/10.1016/S14744422(18)30287-4 PMID: 30264729
30.
Possover M, Forman A. Recovery of supraspinal control of leg movement in a chronic complete flaccid paraplegic man after continuous low-frequency pelvic nerve stimulation and FES-assisted training. Spinal Cord Ser Cases. 2017; 3: 16034. https://doi.org/10.1038/scsandc.2016.34 PMID: 28503316
31.
Bonizzato M, Pidpruzhnykova G, DiGiovanna J, Shkorbatova P, Pavlova N, Micera S, et al. Brain-controlled modulation of spinal circuits improves recovery from spinal cord injury. Nat Commun. Springer US; 2018; 9: 3015. https://doi.org/10.1038/s41467-018-05282-6 PMID: 30068906
32.
Lebedev MA, Nicolelis MAL. BRAIN-MACHINE INTERFACES: FROM BASIC SCIENCE TO NEUROPROSTHESES AND BMI S WITH ARTIFICIAL SENSATIONS. 2017; 767–837. https://doi.org/10. 1152/physrev.00027.2016 PMID: 28275048
33.
Donati ARC, Shokur S, Morya E, Campos DSF, Moioli RC, Gitti CM, et al. Long-term training with brainmachine interfaces induces partial neurological recovery in paraplegic patients. Sci Rep. Nature Publishing Group; 2016; 6: 30383. https://doi.org/10.1038/srep30383 PMID: 27513629
34.
Lebedev MA, Nicolelis MAL. Brain–machine interfaces: past, present and future. 2006; 29. https://doi. org/10.1016/j.tins.2006.07.004 PMID: 16859758
35.
Shokur S, Gallo S, Moioli RC, Donati ARC, Morya E, Bleuler H, et al. Assimilation of virtual legs and perception of floor texture by complete paraplegic patients receiving artificial tactile feedback. Sci Rep. Nature Publishing Group; 2016; 6. https://doi.org/10.1038/srep32293 PMID: 27640345
36.
Jang Y, Hsieh C-L, Wang Y-H, Wu Y-H. A validity study of the WHOQOL-BREF assessment in persons with traumatic spinal cord injury. Arch Phys Med Rehabil. Elsevier; 2004; 85: 1890–1895. https://doi. org/10.1016/j.apmr.2004.02.032 PMID: 15520987
37.
Donati ARC, Shokur S, Morya E, Campos DSF, Moioli RC, Gitti CM, et al. Long-Term Training with a Brain-Machine Interface-Based Gait Protocol Induces Partial Neurological Recovery in Paraplegic Patients. Sci Rep. Nature Publishing Group; 2016; 6: 30383. https://doi.org/10.1038/srep30383 PMID: 27513629
38.
Nicolelis MAL, Shokur S, Lin A, Moioli RC, Brasil FL, Nicole P, et al. The Walk Again Project: Using a Brain-Machine Interface for establishing a bi-directional Interaction between paraplegic subjects and a lower limb exoskeleton. 44th Society for Neuroscience Meeting. Washington, DC; 2014.
39.
De Leener B, Lévy S, Dupont SM, Fonov VS, Stikov N, Louis Collins D, et al. SCT: Spinal Cord Toolbox, an open-source software for processing spinal cord MRI data. Neuroimage. 2017; 145: 24–43. https:// doi.org/10.1016/j.neuroimage.2016.10.009 PMID: 27720818
40.
Biering-Sørensen F, Craggs M, Kennelly M, Schick E, Wyndaele J-J. International lower urinary tract function basic spinal cord injury data set. Spinal Cord. 2008; 46: 325–330. https://doi.org/10.1038/sj.sc. 3102145 PMID: 18040278
41.
Krogh K, Perkash I, Stiens SA, Biering-Sørensen F. International bowel function basic spinal cord injury data set. Spinal Cord. 2009; 47: 230–234. https://doi.org/10.1038/sc.2008.102 PMID: 18725887
42.
Alexander MS, Biering-Sørensen F, Elliott S, Kreuter M, Sønksen J. International Spinal Cord Injury Male Sexual Function Basic Data Set. Spinal Cord. 2011; 49: 795–798. https://doi.org/10.1038/sc. 2010.192 PMID: 21283085
43.
Alexander MS, Biering-Sørensen F, Elliott S, Kreuter M, Sønksen J. International Spinal Cord Injury Female Sexual and Reproductive Function Basic Data Set. Spinal cord Off J Int Med Soc Paraplegia. 2011; 49: 787–790. https://doi.org/10.1038/sc.2011.7 PMID: 21383760
44.
Renard Y, Lotte F, Gibert G, Congedo M, Maby E, Delannoy V, et al. OpenViBE: An Open-Source Software Platform to Design, Test, and Use Brain–Computer Interfaces in Real and Virtual Environments. Neural Comput. The MIT Press; 2010; 19: 35–53. https://doi.org/10.1162/pres.19.1.35
45.
Wu L, RJ M, GJ H, Jr. DJF. Recovery of zero-grade muscles in the zone of partial preservation in motor complete quadriplegia. Arch Phys Med Rehabil. 1992; 73: 40–43 4p. Available: http://search. ebscohost.com/login.aspx?direct=true&db=cin20&AN=107482075&site=ehost-live PMID: 1729972
46.
Waters RL, Yakura JS, Adkins RH, Sie I. Recovery following complete paraplegia. Arch Phys Med Rehabil. 1992; 73: 784–9. Available: http://www.ncbi.nlm.nih.gov/pubmed/1514883 PMID: 1514883
47.
Mahmoudi B, Erfanian ÆA. Electro-encephalogram based brain–computer interface: improved performance by mental practice and concentration skills. 2006; 959–969. https://doi.org/10.1007/s11517006-0111-8 PMID: 17028907
48.
Basso DM. Neuroanatomical substrates of functional recovery after experimental spinal cord injury: implications of basic science research for human spinal cord injury. Phys Ther. 2000; 80: 808–17. https://doi.org/10.1097/00002060-198804000-00007 PMID: 10911417
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
31 / 33
97 Sensory-motor, visceral and psychological improvement in paraplegics
49.
Dimitrijevic MR. Residual motor functions in spinal cord injury. Adv Neurol. 1988; 47: 138–155. Available: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=emed88&NEWS=N&AN= 3278516 PMID: 3278516
50.
Fouad K, Krajacic A, Tetzlaff W. Spinal cord injury and plasticity: Opportunities and challenges. Brain Research Bulletin. 2011. pp. 337–342. https://doi.org/10.1016/j.brainresbull.2010.04.017 PMID: 20471456
51.
Nathan PW. Effects on movement of surgical incisions into the human spinal cord. Brain. 1994; 117: 337–346. https://doi.org/10.1093/brain/117.2.337 PMID: 7514479
52.
Dobkin B, Apple D, Barbeau H, Basso M, Behrman A, Deforge D, et al. Weight-supported treadmill vs over-ground training for walking after acute incomplete SCI. Neurology. 2006; 66: 484–492. https://doi. org/10.1212/01.wnl.0000202600.72018.39 PMID: 16505299
53.
Barbeau H. Locomotor training in neurorehabilitation: emerging rehabilitation concepts. Neurorehabil Neural Repair. 2003; 17: 3–11. https://doi.org/10.1177/0888439002250442 PMID: 12645440
54.
Van Peppen R, Kwakkel G, Wood-Dauphinee S, Hendriks H, Van der Wees P, Dekker J. The impact of physical therapy on functional outcomes after stroke: what’s the evidence? Clin Rehabil. 2004; 18: 833– 862. https://doi.org/10.1191/0269215504cr843oa PMID: 15609840
55.
Taub E, Uswatte G, Pidikiti R. Constraint-Induced Movement Therapy: a new family of techniques with broad application to physical rehabilitation—a clinical review. J Rehabil Res Dev. 1999; 36: 237–251. PMID: 10659807
56.
Soekadar SR, Birbaumer N, Slutzky MW, Cohen LG. Brain–machine interfaces in neurorehabilitation of stroke. Neurobiol Dis. 2015; 83: 172–179. https://doi.org/10.1016/j.nbd.2014.11.025 PMID: 25489973
57.
Birbaumer N, Cohen LG. Brain-computer interfaces: communication and restoration of movement in paralysis. J Physiol. Blackwell Publishing Ltd; 2007; 579: 621–636. https://doi.org/10.1113/jphysiol. 2006.125633 PMID: 17234696
58.
Ramos-Murguialday A, Broetz D, Rea M, Läer L, Yilmaz O, Brasil FL, et al. Brain-machine-interface in chronic stroke rehabilitation: A controlled study. Ann Neurol. 2013; 74: 100–108. https://doi.org/10. 1002/ana.23879 PMID: 23494615
59.
Fetz EE. Operant conditioning of cortical unit activity. Science (80-). 1969; 163: 955–8.
60.
Holmes NP, Spence C. The body schema and the multisensory representation(s) of peripersonal space. Cogn Process. 2004; 5: 94–105. https://doi.org/10.1007/s10339-004-0013-3 PMID: 16467906
61.
Lenggenhager B. Video Ergo Sum: Manipulating Bodily. 2007; 1096: 1095–1099. https://doi.org/10. 1126/science.1143439
62.
Fuentes CT, Pazzaglia M, Longo MR, Scivoletto G, Haggard P, MR L, et al. Body image distortions following spinal cord injury. J Neurol Neurosurg Psychiatry. 2013; 82: 201–207. https://doi.org/10.1136/ jnnp-2012-304001
63.
Ionta S, Villiger M, Jutzeler CR, Freund P, Curt A, Gassert R. Spinal cord injury affects the interplay between visual and sensorimotor representations of the body. Sci Rep. 2016; 6. https://doi.org/10. 1038/srep20144 PMID: 26842303
64.
Canzoneri E, Marzolla M, Amoresano A, Verni G, Serino A. Amputation and prosthesis implantation shape body and peripersonal space representations. Sci Rep. 2013; 3: 1–8. https://doi.org/10.1038/ srep02844 PMID: 24088746
65.
Lebedev M a, Mirabella G, Erchova I, Diamond ME. Experience-dependent plasticity of rat barrel cortex: redistribution of activity across barrel-columns. Cereb Cortex. 2000; 10: 23–31. https://doi.org/10.1093/ CERCOR/10.1.23 PMID: 10639392
66.
Ramachandran VS, Stewart M, Rogers-Ramachandran DC. Perceptual correlates of massive cortical reorganization. Neuroreport. 1992; 3: 583–586. PMID: 1421112
67.
Merzenich MM, Nelson RJ, Stryker MP, Cynader MS, Schoppmann A, Zook JM. Somatosensory cortical map changes following digit amputation in adult monkeys. J Comp Neurol. 1984; 224: 591–605. https://doi.org/10.1002/cne.902240408 PMID: 6725633
68.
Botvinick M, Cohen J. Rubber hands’ feel’touch that eyes see. Nature. Nature Publishing Group; 1998; 391: 756. https://doi.org/10.1038/35784 PMID: 9486643
69.
Tsakiris M, Haggard P. The rubber hand illusion revisited: visuotactile integration and self-attribution. J Exp Psychol Hum Percept Perform. 2005; 31: 80–91. https://doi.org/10.1037/0096-1523.31.1.80 PMID: 15709864
70.
Benevento BT, Sipski ML. Neurogenic bladder, neurogenic bowel, and sexual dysfunction in people with spinal cord injury. Phys Ther. American Physical Therapy Association; 2002; 82: 601–12. Available: http://www.ncbi.nlm.nih.gov/pubmed/12036401 PMID: 12036401
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
32 / 33
98 Sensory-motor, visceral and psychological improvement in paraplegics
71.
Anderson KD. Targeting recovery: priorities of the spinal cord-injured population. J Neurotrauma. 2004; 21: 1371–1383. https://doi.org/10.1089/neu.2004.21.1371 PMID: 15672628
72.
Hill M, Noonan V, Sakakibara B, Miller W. Quality of life instruments and definitions in individuals with spinal cord injury: A systematic review. Spinal Cord. 2010; 48: 438–450. https://doi.org/10.1038/sc. 2009.164 PMID: 20029393
73.
Whalley Hammell K. Exploring quality of life following high spinal cord injury: a review and critique. Spinal Cord. 2004; 42: 491–502. https://doi.org/10.1038/sj.sc.3101636 PMID: 15263890
74.
Hammell K. Quality of life after spinal cord injury: a meta-synthesis of qualitative findings. Spinal cord Off J Int Med Soc Paraplegia. 2007; 45: 124–139. https://doi.org/10.1038/sj.sc.3101992 PMID: 17091119
75.
Hospital TW. Editorial Assessing quality of life in traumatic spinal cord injury: an evolving landscape. 2012; 27–28. https://doi.org/10.3171/2012.6.AOSPINE12568.28
76.
Siddall PJ, McClelland JM, Rutkowski SB, Cousins MJ. A longitudinal study of the prevalence and characteristics of pain in the first 5 years following spinal cord injury. Pain. 2003; 103: 249–257. https://doi. org/10.1016/S0304-3959(02)00452-9 PMID: 12791431
77.
Cairns DM, Adkins RH, Scott MD. Pain and depression in acute traumatic spinal cord injury: origins of chronic problematic pain? Arch Phys Med Rehabil. 1996; 77: 329–35. https://doi.org/10.1016/S00039993(96)90079-9 PMID: 8607754
78.
Garcia-Larrea L, Peyron R. Motor cortex stimulation for neuropathic pain: From phenomenology to mechanisms. Neuroimage. 2007; 37. https://doi.org/10.1016/j.neuroimage.2007.05.062 PMID: 17644413
79.
Fontaine D, Hamani C, Lozano A. Efficacy and safety of motor cortex stimulation for chronic neuropathic pain: critical review of the literature. J Neurosurg. 2009; 110: 251–256. https://doi.org/10.3171/ 2008.6.17602 PMID: 18991496
80.
Tsubokawa T, Katayama Y, Yamamoto T, Hirayama T, Koyama S. Chronic motor cortex stimulation for the treatment of central pain. Acta Neurochir Suppl (Wien). 1991; 52: 137–9. Available: http://www.ncbi. nlm.nih.gov/pubmed/1792954
81.
Yoshida N, Hashimoto Y, Shikota M, Ota T. Relief of neuropathic pain after spinal cord injury by braincomputer interface training. Spinal Cord Ser Cases. 2016; 2: 16021. https://doi.org/10.1038/scsandc. 2016.21 PMID: 28053764
82.
Pulvirenti M, Mcmillan J, Lawn S. Empowerment, patient centred care and self-management. Heal Expect. 2014; 17: 303–310. https://doi.org/10.1111/j.1369-7625.2011.00757.x PMID: 22212306
83.
Aujoulat I, Marcolongo R, Bonadiman L, Deccache A. Reconsidering patient empowerment in chronic illness: A critique of models of self-efficacy and bodily control. 2008; 66: 1228–1239. https://doi.org/10. 1016/j.socscimed.2007.11.034 PMID: 18155338
84.
Lorig KR, Holman HR, Med AB. Self-Management Education: History, Definition, Outcomes, and Mechanisms. https://doi.org/10.1207/S15324796ABM2601_01 PMID: 12867348
85.
Funnel M, Anderson R. Patient empowerment: a look back, a look ahead. 2003. https://doi.org/10.1177/ 014572170302900310 PMID: 12854337
86.
Swendeman D, Ingram BL, Rotheram-borus MJ. AIDS Care: Psychological and Socio-medical Aspects of AIDS / HIV Common elements in self-management of HIV and other chronic illnesses: an integrative framework.: 37–41. https://doi.org/10.1080/09540120902803158
87.
Liu LCILC. A Study of the Empowerment Process for Cancer Patients Using Freire ‘ s Dialogical Interviewing. 2004; 12: 41–50.
88.
Aujoulat I, Deccache A. Patient empowerment in theory and practice: Polysemy or cacophony? 2007; 66: 13–20. https://doi.org/10.1016/j.pec.2006.09.008 PMID: 17084059
89.
McAllister M, Dunn G, Payne K, Davies L, Todd C. Patient empowerment: the need to consider it as a measurable patient-reported outcome for chronic conditions. BMC Health Serv Res. 2012; 12: 157. https://doi.org/10.1186/1472-6963-12-157 PMID: 22694747
90.
Tucker MR, Olivier J, Pagel A, Bleuler H, Bouri M, Lambercy O, et al. Control strategies for active lower extremity prosthetics and orthotics: a review. J Neuroeng Rehabil. 2015; 12: 1. https://doi.org/10.1186/ 1743-0003-12-1 PMID: 25557982
PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018
33 / 33
99
www.nature.com/scientificreports
OPEN
Received: 17 September 2018 Accepted: 10 April 2019 Published: xx xx xxxx
Non-invasive, Brain-controlled Functional Electrical Stimulation for Locomotion Rehabilitation in Individuals with Paraplegia Aurelie Selfslagh1,2, Solaiman Shokur1, Debora S. F. Campos1, Ana R. C. Donati1,3, Sabrina Almeida1,3, Seidi Y. Yamauti1, Daniel B. Coelho 4, Mohamed Bouri2 & Miguel A. L. Nicolelis1,5,6,7,8,9,10,11 Spinal cord injury (SCI) impairs the flow of sensory and motor signals between the brain and the areas of the body located below the lesion level. Here, we describe a neurorehabilitation setup combining several approaches that were shown to have a positive effect in patients with SCI: gait training by means of non-invasive, surface functional electrical stimulation (sFES) of the lower-limbs, proprioceptive and tactile feedback, balance control through overground walking and cue-based decoding of cortical motor commands using a brain-machine interface (BMI). The central component of this new approach was the development of a novel muscle stimulation paradigm for step generation using 16 sFES channels taking all sub-phases of physiological gait into account. We also developed a new BMI protocol to identify left and right leg motor imagery that was used to trigger an sFES-generated step movement. Our system was tested and validated with two patients with chronic paraplegia. These patients were able to walk safely with 65–70% body weight support, accumulating a total of 4,580 steps with this setup. We observed cardiovascular improvements and less dependency on walking assistance, but also partial neurological recovery in both patients, with substantial rates of motor improvement for one of them. Every year, about 250,000–500,000 people worldwide suffer a spinal cord injury (SCI) as a result of traffic accidents, falls, other traumatic accidents or violence1. SCI leads to severe impairments of the sensory, motor and autonomic functions below the lesion level, as well as secondary clinical conditions such as pressure ulcers, urinary tract infections, and osteoporosis. In addition to the gravity of these clinical effects, SCI incurs substantial financial costs, both for the individual and society. For patients diagnosed with the most severe cases of SCI (patients presenting no motor functions below the lesion (AIS B) and patients presenting neither motor nor sensory functions (AIS A)), at the chronic phase of the lesion (a year after the injury), the chances for a spontaneous recovery are negligible2,3. Up to now, there is no systematic approach to restoring neurological functions in such patients. Over the past decades, a variety of new approaches for SCI rehabilitation have been introduced4,5. These efforts aimed at inducing activity-dependent plasticity5 by the utilization of robotic trainers6–8, epidural electrical stimulation9–11, body-weight support trainers12, brain-machine interfaces11,13–15, transcranial magnetic stimulation16,17 and surface functional electrical stimulation (sFES)18,19. Recently, our group has shown that a training protocol (the Walk Again NeuroRehabilitation protocol (WANR))14,20 ̶ induced a significant level of neurological 1
Neurorehabilitation Laboratory, Associação Alberto Santos Dumont para Apoio à Pesquisa (AASDAP), São Paulo, 05440-000, Brazil. 2STI IMT, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland. 3Associação de Assistência à Criança Deficiente (AACD), São Paulo, 04027-000, Brazil. 4Biomedical Engineering, Federal University of ABC, São Bernardo do Campo, SP, 09606-045, Brazil. 5Department of Neurobiology, Duke University Medical Center, Durham, NC, 27710, USA. 6Duke Center for Neuroengineering, Duke University, Durham, NC, 27710, USA. 7 Department of Biomedical Engineering, Duke University, Durham, NC, 27708, USA. 8Department of Neurology, Duke University, Durham, NC, 27710, USA. 9Department of Neurosurgery, Duke University, Durham, NC, 27710, USA. 10 Department of Psychology and Neuroscience, Duke University, Durham, NC, 27708, USA. 11Edmond and Lily Safra International Institute of Neuroscience, Macaíba, Brazil. Aurelie Selfslagh and Solaiman Shokur contributed equally. Correspondence and requests for materials should be addressed to M.A.L.N. (email: nicoleli@neuro.duke.edu) Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
1
www.nature.com/scientificreports/
100
www.nature.com/scientificreports
recovery in a group of eight subjects with chronic complete paraplegia (seven AIS A, one AIS B), by combining assisted walk training with robotic walkers, electroencephalography (EEG)-based brain-machine interface (BMI), and continuous visuotactile feedback. Extending our previous findings, the present study describes our group’s effort to further enhance clinical neurological recovery in severe cases of SCI by providing more selective lower-limb musculoskeletal recruitment through sFES, while maintaining our fundamental philosophy of activating both ascending and descending neural pathways during the neurorehabilitation process. As such, the BFNR (BMI, sFES, NeuroRehabilitation) protocol described here integrates four key elements to potentiate recovery in patients with SCI: muscle activation through sFES21–23 (see24 for a review), balance control through body weight support25, real-time decoding of motor command (BMI)14,20 and sensory feedback through a portable haptic device26. In recent years, BMI-FES (systems that use BMI to detect patients’ voluntary motor commands to trigger their muscle contractions through an FES) have been extensively studied as rehabilitation tools (some researchers call it therapy or partial restoration) for patients with severe cases of stroke27–29, to potentiate recovery of upper-limb28,30,31 or lower-limb motor function32 (see33 for a review). On the other hand, several studies with SCI patients have shown remarkable results when using BMI-sFES technology as an assistive device, by bypassing the spinal cord lesion and permitting the patient to perform motor functions34,35. However, to date, little is known about the potential of the BMI-sFES approach as a true rehabilitative tool/therapy for patients with SCI. To our knowledge, only one study with rats has rigorously tested this hypothesis: Bonizzato and colleagues15 recently showed that a training paradigm integrating BMI with epidural stimulation accelerated and enhanced the animal’s neurological recovery when compared to training that used only the epidural stimulation. Here, we studied both the assistive and the restorative aspects of a novel BMI-sFES system with two patients with chronic SCI. First, we demonstrated the validity of each aspect of our approach: a custom 16 channel sFES system, a closed-loop proportional- integral (PI) controller to cope with muscle fatigue, the integration of a portable haptic device to cope with the lack of lower-limb sensory functions and a novel BMI protocol based on the detection of left and right leg motor imagery. Next, we tested our neurorehabilitation protocol with two patients with SCI (originally AIS A) who had been previously trained with the Walk Again Neurorehabilitation (WANR) protocol and, as a consequence, converted to AIS C20 (Supplementary Tables S1, S2). At the onset of the current protocol both patients presented motor functions more than three segments below the motor level (presence of knee extension (L3) for both patients); however, their overall motor score at that point in time was low (lower extremity motor score of 4 for P1 and 2 for P2 out of a maximum possible score of 50). To identify the effect of the BFNR protocol, we ran a series of neurological36, neurophysiological and functional (WISCI II)37 assessments at the onset and at the end of training. Overall, we observed a significant improvement in muscle responses and reduction of muscle fatigue in our two patients in response to the sFES. Patients also experienced an increase in muscle volume and functional improvement for walking (measured as the ability to walk 10 meters in full load, using only passive assistive devices, such as walkers, orthoses, and crutches). More remarkably, we also documented clinical neurological improvement in both patients. Indeed, one of these patients reached a substantial increase of 9 points for the lower extremity motor score (LEMS) after 22 sessions with the BFNR training.
Results
Two individuals with chronic paraplegia due to SCI, designated as P1 and P2 (40 and 32 years old; time since lesion: 4.5 and 10 years) with thoracic level injuries (patient P1 lesion at T7, and P2 at T4) were enrolled in our protocol (Supplementary Table S3). Before the sFES training, these patients had followed the WANR training for 28 and 34 months14,20 respectively (Fig. 1A). Both patients had experienced sensory and motor improvements during the WANR and had been converted from AIS A to AIS C by the time of onset of the current protocol. First, we validated our setup for 6 months, for a total of 25 sessions, with patient P1. During this period, we tested our conditioning protocol (12 sessions), validated the closed-loop sFES stimulation (one session), validated the sFES-generated locomotion (10 sessions), and performed the integration of the tactile feedback with our training protocol (two sessions). Next, we tested the BFNR (BMI, sFES, NeuroRehabilitation) protocol with both patients. To isolate the effect of our training protocol, a 2-month washout period was given before the onset of the BFNR (Fig. 1A). The subjects did not receive any locomotion training, nor training in an upright position during this period. Clinical assessments, including the lower-limb circumference perimeter measurement38, evaluation of walking function with assistive devices37 and neurological status exam36 were performed before and after the training protocol for both patients (measurements M4 to M5). We also compared both patients’ neurological improvement rate during the BFNR protocol with their improvement during the WANR protocol (measurements M1 to M2 for P1, M1 to M3 for P2), as well as their spontaneous improvement measured before they started the WANR protocol (measurements M0 to M1). For patient P1, we also controlled for whether our pilot validation tests induced any changes (measurements M2 to M3). The BFNR protocol was comprised of three basic phases: (1) lower-limb sFES conditioning (FC); (2) sFES-generated locomotion training (FL); and (3) brain-controlled sFES-generated locomotion (B + FL) (Fig. 1B). Patient P2 performed nine FC sessions, followed by four FL sessions, and then 10 B + FL sessions. One extra FL session was recorded after the fifth B + FL session for analyses of patients’ walking aptitude using the sFES. Patient P1, who had already followed a conditioning phase during the pilot validation phase, had two FC sessions followed by the same sequence of FL and B + FL sessions as patient P2. During all BFNR phases we used a custom 16-channel sFES system39 targeting the following muscles in both legs (Fig. 1C): gluteus maximus (GMx), gluteus medius (Gmd), rectus femoris proximal (RFP), hamstrings (Hs), vastus lateralis (VL), tibialis anterior (TA), gastrocnemius (Gs) and soleus (Sl). During the FL and B + FL phases, patients were partially suspended (65–70%) with a body weight support system (Fig. 1D, Supplementary Fig. S1) (Zero-G, Aretech LLC., Ashburn, VA). During their walking over a Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
2
www.nature.com/scientificreports/
101
www.nature.com/scientificreports
Figure 1. Methodology. (A) The timeline for experiment protocol, and six clinical measurements, ranged from M0 to M5. The training frequency (per month) is reported in each cell. WANR: Walk Again Neurorehabilitation Protocol14,20; sFES: surface functional electrical stimulation; BFNR: BMI and sFES neurorehabilitation. (B) Number of sessions per training phase for the BFNR protocol. (C) Eight lower-limb muscles are bilaterally stimulated with the sFES: three muscles for hip mobilization (flexion, extension, and abduction), two for the knee (flexion and extension) and three for the ankle (one dorsiflexor and two plantar flexors). (D) Patients were secured by a body-weight support system (Zero-G, Aretech LLC., Ashburn, VA) and used a walker for stability. A stimulation model, containing a reference kinematic profile and gait phases (detailed in subfigure G), was used as a base for the feedforward current stimulation. Patients’ hip and knee angles were measured in real-time. The command of the Proportional Integral (PI) controller was computed based on the error (eθ) between the current joint angle (θm) and the reference kinematics (θd). We used a sigmoid function to convert the PI command into feedback currents for the flexors (Iflex) and extensors muscles (Iext). The feedback and feedforward currents were summed to produce the actual values applied to the electrodes. (E) EEG electrodes were placed around the medial longitudinal fissure. Channel Fz was used as the reference and the Fpz as Gnd. Arm and leg sensorimotor areas are schematized on the figure on the right; Primary Motor (M1) and Sensory (S1) cortices, Pre-motor Cortex (PMC), Supplementary Motor Area (SMA) and posterior parietal cortex are shown. (F) The tactile shirt: six vibrators aligned on the subject’s ulna bone. The subject felt a continuous tactile stimulation going from the wrist to the elbow by the swing phase of the ipsilateral leg and a simultaneous stimulation on all three vibrators at the onset of the stance. (G) sFES stimulation pattern for the lower-limb muscles to reproduce the eight sub-phases of a normal gait. The expected range of motion and the proposed stimulation current are shown for hip F/E, knee F/E, ankle dorsiflexion/plantar flexion and hip adduction/abduction. The pulse width and frequency were fixed. Three types of ramps were used to reproduce the gait cycle realistically. four-meter-long linear track, patients assisted themselves with a low-cost walker. The session was supervised by a physiotherapist or a physician. The patients’ hip and knee joint angles were measured every 7 ms. A PI controller adapted the stimulation amplitude in real-time to the muscle conditions determined by the lower-limb joint angles measurements. Two additional elements completed the setup: (a) a 16-channel EEG electrode (Fig. 1E) placed around the medial longitudinal fissure, densely clustered over the putative leg sensorimotor cortical area to detect the cortical activation for leg motor imagery; and (b) a portable haptic device26 that permitted the patients to perceive the position of the stimulated leg on their upper-arm (Fig. 1F). Continuous waves of tactile vibrations delivered to the patients’ forearms coincided with the stimulation of the lower-limb muscles to indicate the beginning and end of the swing phase of a stride. The preparation, followed by the training session, required approximatively 95 minutes, divided as follows: the sessions started with 15 minutes stretching, targeting both the upper-limbs, trunk, and lower-limb muscles. Next, approximately 35 minutes were used during the session preparation (placement of the EEG and sFES electrodes, placement of the position sensors and portable tactile feedback, placement of body weight support harness and moving patient to an upright position). The training lasted 45 minutes. We developed a novel sFES stimulation strategy that used time-based muscle contractions considering the eight sub-phases40 of a physiological gait pattern41,42 (Fig. 1G): (1) initial foot contact; (2) load response; (3) mid-stance; (4) terminal stance; (5) pre-swing; (6) initial swing; (7) mid-swing; and (8) terminal swing. To reproduce adequate gait during each sub-phase, we fine-tuned the stimulation time, the stimulation ramps up/down, the amplitude and the frequency for each muscle.
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
3
www.nature.com/scientificreports/
102
www.nature.com/scientificreports
Figure 2. Setup validation. (A) Validation test of the PI controller; range of motion of the knee joint with and without the PI controller during a knee extension task with P1. We compare the joint angle produced by the controller in both cases (mean ± SD, n = 56) to the expected target. (B) Root mean square error between the desired and measured knee angle (mean ± SD) for both patients. Two-sided two-sampled t-test is reported. (C) Current applied without PI controller and with PI controller for the first and second half of the attempts (mean ± SD, n = 56). (D) The amplitude of current for the vastus lateralis (VL) muscle during the conditioning sessions for patient P1. The duration of the session was set to 10 minutes and increased by 5 minutes every session (max. 40 min). When the therapist observed a clear sign of fatigue, the current was increased by five mA. The initial amplitude of a session was always set to 60 mA for this patient. (E) Mean ± SEM of the time between two changes of current amplitude, considering all muscles, during the first 20 minutes of conditioning sessions three to nine for patient P1. Red line shows the linear regression of the means and corresponding linear function. (F) 3D reconstruction of the gait based on motion capture data measured during an FL session with
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
4
www.nature.com/scientificreports/
103
www.nature.com/scientificreports
P2, and (G) angles for hip, knee ankle in the sagittal plane (mean + − SD over 54 full gait cycles). The lower part of the graph reports if the movement was correct (green) or not (orange or red). We report in orange the period where the movement was correct but the amplitude of the movement smaller (arrows pointing down) or higher (arrows up) than expected (small or big arrows describe if the difference was respectively in the 0–10° or 10–20° range). (H) Self-reported sets of comfortable walking speeds for patient P1. Walking speed varied on each run and the patient was asked to report if the speed was comfortable or not and if it was his favorite speed of the current session. (I) An experimented physiotherapist, blinded to the experimental conditions used an 11-item visual evaluation scale on qualitative and kinematics’ aspects of patients’ P1 and P2 sFES-generated walk when the tactile shirt was turned off (blue dot) or turned on (orange triangle).
PI controller validation. We introduced a PI controller to adapt the electrical stimulation pattern of the sFES system, based on muscle responses, to generate a reliable gait and cope with variance regarding muscle responsiveness and fatigue. To illustrate this effect, Fig. 2A compares the knee angle for patient P1 during a sFES-generated knee extension task in a seated position, when the PI controller was turned on (blue line) or off (grey line). To guarantee well-controlled conditions, only the knee joints were controlled for this test, while the other joints remained immobilized. In both conditions, the same desired target angle (black line) is used for comparison. The test was repeated with both patients. We found that the root mean square error (RMS) for the PI-on condition (left knee 8.2°, right knee 7.8°, for P1, 11.2° and 9.3° for P2) was significantly smaller compared to PI-off (left knee 14.5°, right knee 14.4° for P1, 15.5° and 17.6° for P2) (two-sided two-sample t-test attests the significant difference (t(110) = 15.13, p = 0 for P1; t(110) = 15.20, p = 0 for P2) (Fig. 2B). These findings confirm that the PI controller produced more reproducible responses to the patients even in the presence of muscular fatigue. To cope with the increase in fatigue, the PI controller increased the amplitude of the stimulation for the last part of the sessions (Fig. 2C). The mean current employed was only 6% higher than in the PI-off condition for P1 and 7.8% for P2. Considering the entire session, the mean currents for both conditions were comparable for both P1 (mean current of 49 mA for PI-on and 47 mA for PI-off), and P2 (42.51 mA PI-on, 39.96 PI-off), suggesting that the use of the PI did not notably increase the necessary current for stimulation. sFES-Conditioning validation. The FC phase was introduced to delay the onset of muscle fatigue while
increasing muscle responses. Three factors were gradually adjusted during this phase: the stimulation time, the amplitude and the number of muscles stimulated simultaneously. The first training session was set to 10 minutes whereas the following sessions were gradually increased up to 40 minutes. At the onset of the sessions, the stimulation amplitude was set to the minimum current that produced a clear muscle contraction. Then, if the therapist observed a notable decrease in a given muscle response, the corresponding stimulation current was increased by 5 mA (maximum: 100 mA). An example of stimulation amplitude is shown for nine sessions of VL muscle for patient P1 (Fig. 2D). For session three, during the first 20 minutes of training, it was necessary to increase the current five times (from 60 mA to 80 mA) to observe a muscle activation that led to an overt movement. On the following day, at the same moment throughout the session, only 70 mA were needed to generate the same response. This positive trend continued throughout the training. Thus, after the sixth session of conditioning, the patients’ VL muscle response did not decay after 40 minutes of stimulation at 60 mA. We calculated the mean time before fatigue for all muscles during the first 20 minutes of the session. Considering only sessions that lasted 20 minutes or more (to avoid bias due to the session length), we observed a significant linear increase of the average time before fatigue (Fig. 2E, y = 0.62x + 11.03, P = 0.01), confirming that the patients’ muscle responses improved following FC sessions.
sFES stimulation strategy validation. Gait-analysis based on 3D construction of motion capture data of patient P2 revealed that the majority of the markers of the physiological walk were present (Fig. 2F, detailed analysis in Supplementary Fig. S2, Supplementary Video S1). Indeed, on initial contact, we observed the accurate hip flexion, and the expected neutral position for the knee and the ankle. Through the mid-stance phase, we observed an accurate, neutral position for both the hip, knee and ankle joints. For the pre-swing, as expected, we observed the flexion of both the hip and the knee and the ankle plantar-flexion. We documented the flexion of the hip and the knee during the initial swing, and a neutral position of the ankle during the mid-swing. On the terminal swing, the hip flexion was correctly maintained, the ankle was in neutral position, and the knee extended. In three cases the range of motion was smaller than the normal gait: hip extension during the terminal-stance, the second peak of knee flexion during the swing phase, and the ankle dorsiflexion during mid and terminal stance. The hip extension and the ankle dorsiflexion limitations reduced the overall step length, whereas the limitation on the knee flexion reduced the foot clearance during the swing. Nevertheless, none of these limitations was severe enough to compromise the patient’s gait. We also wanted to be sure to use the fastest walking pace, to maximize the number of steps per session, while guaranteeing a comfortable and safe training environment for the patients. We performed a systematic evaluation of the maximum comfortable speed during six sessions with patient P1. For each run of 12 steps, we varied the speed and asked the patient whether the run was comfortable or not and whether the run was the patient’s favorite among the ones achieved in the current session. The patient was told before the session that the speed could change from one run to another, but no further information was given during the session. Over time, the patient reported feeling comfortable with speeds that were gradually increased (Fig. 2H); while the threshold for comfortable speed was between 6 and 8 seconds per step (s/step) during session seven, it reached a maximum of 3 s/step on the 10th session. By then, the preferred speed improved from 8 s/step to 3 s/step.
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
5
www.nature.com/scientificreports/
104
www.nature.com/scientificreports
Haptic feedback validation. SCI induces a loss of tactile and proprioceptive feedback information that is needed to generate a correct walking pattern43,44. To overcome this limitation, we integrated a portable haptic device, developed by our group (named the tactile shirt)26, composed of arrays of vibrators to display relevant events using a tactile illusion called the apparent movement45. We ran two sessions with patient P1 to evaluate the difference of behavior in the presence or absence of the tactile feedback (Fig. 2I). An expert physiotherapist, blinded to the conditions of the experiment used an 11-item visual assessment developed by our team to evaluate differences in terms of the patient’s adaptation to our locomotion training setup (questions 1–5) and the patient’s gait kinematics during the sFES-generated locomotion (questions 6–11) (Supplementary Table S4 for details). The presence of the tactile shirt did not positively or negatively influence the patient’s trunk alignment, his need for compensatory movements, or his coordination with the walker. As expected, neither did we observe any substantial difference in any of the kinematic-related questions: the presence of the tactile feedback did not change the patient’s muscle responses during the stance (questions 6–8), or the swing (questions 9–11) phases. On the other hand, we observed that the patient developed a tendency to look less at his own body during the walk when the tactile feedback was present (question 4). This phenomenon allowed patients to express a better understanding of the position of the leg in space, as well as the movement of the lower limbs. During the sessions, the therapist instructed the patient about correct trunk posture, upper and lower-limb participation, as well as the best way to coordinate the walker. We noticed a reduction in the need for these verbal instructions when tactile feedback was employed (question 5). Overall, we found that tactile feedback was useful for the patient to perform the task more independently. BFNR validation. After completing the pilot validation tests, we applied our BFNR protocol to both patients. Here, we present a quantitative and qualitative analysis of the patients’ walking with our sFES system, their performance with the BMI and their performances with the setup integrating the BMI and the sFES. Overall, both patients managed to walk correctly using the sFES and the stimulation paradigm proposed in this project (see Supplementary Movies S2 and S3 for runs with both patients). To have a complete view of the behavior of the two patients with our setup, we carried out a qualitative analysis of the patients’ sFES-generated walking patterns during their last FL sessions. Figure 3A shows the hip and knee range of motion for these sessions and Figure 3B the detailed study carried out by a trained physiotherapist blinded to the experiment. Considering the metrics for high-level coordination, we did not find notable differences between the patients; both patients had good trunk alignment during the tasks; they, nevertheless, used compensatory movements to maintain their upper body. As patients used the tactile shirt in both sessions, we found results consistent with the pilot tests for P1. Basically, both patients did not need verbal instructions, nor did they need to constantly look at their bodies during the sFES-generated locomotion. During the stance phase, both patients had good knee position but poor plantigrade position during single support. Patient P1 had expected responses during the swing phase, with his foot rarely dragging on the floor. Patient P2 had difficulties in this aspect of the swing phase, due to a smaller response of the tibialis anterior muscle. We hypothesize that this lack of muscle response was due to an observed tension in this patient’s gastrocnemius (Gs) muscle (which is the antagonistic muscle of this movement). Likely, this resulted from his positioning in his wheelchair since his knees and feet usually stayed flexed in his wheelchair, which could lead to increased tension in the Gs muscle. Both patients in this protocol had been previously trained with the BMI protocol using an arm or leg motor imagery as described in the WANR protocol14,20 for over 2 years. However, the specific BMI protocol proposed in the current study was new to both patients. Each B + FL session started with an 8-minute open-loop training block where patients were randomly instructed to imagine moving their left or right leg. The recorded EEG data was used to train the EEG classifier. The classification accuracy for both patients was above chance (73.9% ± 3.6% for P1 and 83.2% ± 8.2% for P2) (Fig. 3C). Patient P2’s classifier accuracy was significantly higher than for P1 (t-test, P = 0.002). To confirm that patients effectively used leg motor imagery during the B + FL sessions, we analyzed the principal components calculated with the common spatial pattern filter (CSP)46 (Fig. 3D). The CSP algorithm detects the EEG electrodes that maximize the variance for the two classes (here, left leg versus right leg motor imagery). For left leg motor imagery, we found the main activation over the C2 electrode for both patients (see Fig. 1E for electrode placement), while for right leg motor imagery, the CSP filter detected the highest variance over electrodes CCP1h, confirming cortical activation in the leg motor area. Next, patients learned to perform leg motor imagery to trigger the stimulation of the corresponding leg through an EEG-based brain-machine interface (Fig. 3E). An array of LEDs informed the patients of the current state of the trial and the brain activity classification. A trial was defined as follows: after a relax period of 4 seconds where the patients could adjust themselves, and 1-second focus time, the patients had a 4-second time window to perform the motor imagery (Fig. 3F). If the patient managed to maintain the correct motor imagery for 2 seconds, the trial was considered as successful, and the corresponding step was triggered. If the timeout was reached, the trial was registered as a failure, and an automatic step started. The time to trigger a step, therefore, ranged between 2 and 4 seconds. During each session, patients performed 72 steps divided into six runs separated by a 1-minute break. The number of correct steps per run is reported in Fig. 3G. Patient P2 outperformed patient P1 regarding the number of successful trials per session by performing 88% of the runs above chance and more than one-fifth of the runs with a perfect score (Fig. 3G). Throughout the training, we observed a clear improvement in terms of control accuracy for patient P2. Indeed, when considering the second half of the training, 25 out of 30 runs were significantly better than chance (mean + 2*SD).
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
6
www.nature.com/scientificreports/
105
www.nature.com/scientificreports
Figure 3. Protocol validation. (A) The ranges of motion for hip and knee joints for both patients. The eight gait subphases are shown inside the gray boxes. Dashed line shows the onset of the swing phase. (B) Qualitative and kinematics’ aspects of patients’ sFES-generated walk. (C) Patients’ classifier accuracy for the EEG training phase (mean ± SD, on the 10 B + FL sessions and ttest). (D) The coefficient of the principal component found by the CSP algorithm for a B + FL training session for left and right leg motor imagery. Electrode placement is the same as in Fig. 1E (three electrode names C3, C4 and Pz are shown as references). (E) For each step, the following sequence was reproduced: the patient relaxed and adjusted himself for 4 seconds, focused during 1 second and had 4 seconds to produce the motor command. During this window, if the patient maintained the correct motor imagery for 2 seconds, the step was triggered and considered as successful. Otherwise, if the timeout was reached, an automatic step was triggered, and the trial was considered as failed. (F) Example of successful and failed step. RMI/LMI: right/left motor imagery, NS: no state, tLMI: cumulated time of left motor imagery (G) Number of correctly performed steps for all patients for all B + FL runs over 12 steps. The chance
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
7
www.nature.com/scientificreports/
106
www.nature.com/scientificreports
level is reported with black dashed lines, 95% interval of confidence is shown in gray. (H) Mean ± SE time to control a step with the B + FL protocol. Each dot represents 12 steps of a run. Fastest possible time to perform the task was 2 seconds, and the chance level is shown with a black dashed line, 95% interval of confidence in gray (3.23–3.37 [s]). Linear regression is represented by dashed lines for each patient.
Patient P1 had just 50% of the runs above chance level. This shows that the BMI protocol and the need to maintain the leg motor imagery for 2 seconds was particularly difficult for patient P1 who, despite having a good classifier score (on average >70%), did not manage to perform the BMI control above chance level. We also analyzed the mean time to perform a step for each session (Fig. 3H). We found a small but statistically significant trend towards reducing the time to perform the task in patient P2 (y = −0.11x + 3.44, p = 0) and patient P1 (y = −0.03x + 3.7, p = 0.03) with training. Altogether, we observed significant control performances in patient P2. Patient P1, who started with lower performance rates, improved (although at a slower rate) with the training and could have potentially reached better control with more extended BMI training.
Clinical improvements. Following the BFNR protocol, we observed notable physical improvements in both
patients. This was first verified by an increase in muscle volume through the perimeter assessment38. We found, as expected, an increase of the thigh volume of 276 cm3 in patient P1 and the lower-leg volume of 636 cm3 (Table 1, Supplementary Fig. S3). Patient P2 also exhibited this effect, with an increase of 838 cm3 of thigh volume and 280 cm3 for the lower-leg volume. We next analyzed the patients’ walking functions using assistive devices. For this, we employed the WISCI II evaluation37, which assesses the amount of physical assistance needed, as well as the required devices (orthosis and gait assistive devices), for a patient with SCI to walk 10 meters (Table 2). The exam was performed with no body weight support and no sFES. At the onset of the BFNR (M5), patient P1 was able to walk 10 meters using a lower-limb orthosis (HKAFO: hip-knee-ankle-foot orthosis), a triangular walker and required additional physical assistance for hip and trunk stabilization (WISCI score of 6). Following the BFNR protocol (M6) this patient’s functional score improved to 9, as he required no physical assistance to perform the task. He also managed to perform the same test with crutches and one assistant (score of 7). Additionally, P1 was able to accomplish the task 19 seconds faster than at the onset of the BFNR. No notable changes were observed during the pilot validation tests (M3 to M4). Considering the necessary level of assistance to perform the 10 m task, we did not register changes for patient P2 (score of 6 at the onset and the end of the protocol), he nevertheless managed to perform the test 14 seconds faster. Both patients experienced a small decrease in the resting heart rate (HR) throughout the protocol (81 bpm to 74 for P1, 79 bpm to 76 for P2, Table 2). Similarly, the HR increase due to the effort was smaller at the end of training for P2 (HR increase of 41 bpm at the onset of the training (M5 and 23 bpm at the M6). Patient P1 improved during the pilot tests (an increase of HR of 30 bpm was observed at M2, and an increase of 1 at M3). During the measurement following the BFNR, this patient managed to perform the task with less assistance which translated into a higher increase in HR (36 bpm). The Borg fatigue assessment is a subjective self-reported level of exertion during exercise ranging between 0 (no fatigue) to 10. Patients were asked to report the level of fatigue in their upper-limbs. We observed a small decrease in the fatigue assessment following the BFNR for patient P1. Blood pressure and oxygen saturation were also measured before and after each WISCI assessment (Table 2). All recorded values were in the normal range. No clear neurological improvement was observed in the patients’ sensory domain (light touch, pin-prick sensation) during the BFNR (Fig. 4A). Patient P2 had an improvement rate comparable to the one he had during the first 28 months of WANR, while patient P1 stagnated. On the other hand, we documented significant motor improvement in both patients. The improvement was assessed through the lower extremity motor score measurement (LEMS) of five lower-limb muscle groups (rectus femoris proximal and distal, tibialis anterior, extensor hallucis longus and gastrocnemius). For a complete view, we recorded seven additional lower-limb muscles (hip adductors, gluteus maximus, gluteus medius, medial and lateral hamstring,flexor hallucis longus and extensor digitorum longus). The scoring method followed the standard ASIA motor exam conventions, ranging from 0 (no contraction) to 5 (normal contraction). We show the LEMS score for both patients in the lower panel of Fig. 4A, and the details per muscle in Table 3. For patient P1, we observed, a 1 point LEMS increase during the sFES pilot phase and a gain of 3 points during the BFNR phase. This was an encouraging result, as it demonstrated that even at the chronic phase of an SCI, it is still possible to observe neurological improvement. Notably, patient P2 exhibited a much higher motor improvement (Supplementary Movie S4); starting with an LEMS of 2 measured at the onset of the BFNR training. This patient reached a final score of 11 points. Considering all 12 muscles, this patient went from a score of 6 to 21 (Table 3). By the end of the training, patient P2 could contract the Rectus Femoris Distal (RFD) muscles with a score of 2. This was confirmed both clinically and thorough EMG recordings (Fig. 4B) showing the presence of significant contractions at the end of the BFRN. We next analyzed the proportion of muscles that improved with the training, considering the muscles that were stimulated with the sFES separately from those that were not (lower part of Table 3). For patient P1, no clear improvement was observed during the pilot tests. During the BFRN for this patient, none of the non-stimulated muscles improved, but 5 out of the 16 stimulated muscles improved. For patient P2, we observed an improvement in one out of the eight non-stimulated muscles and 14 out the 16 stimulated ones. We observed that the best motor improvement occurred in muscles that were stimulated with sFES, with notably better responses for patient P2.
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
8
P1
P2
Perimeter diff (cm)
Volume diff (cm3)
Perimeter diff (cm) Volume diff (cm3)
−0.7
2.1
1.0
1.4
1.1
Hip
2.6
276
0.6
3.7
0.2
2.05
2.5
0
0.8
1.6
0.4
1.8
2.5
2 636
3.9
280
1.7
4.3
1.15
2.5 Ankle
838
3.5
0.1 Knee
107
www.nature.com/scientificreports
www.nature.com/scientificreports/
0
−3.1
−2.75
Table 1. Difference between the post-BFNR (M5) and Pre BFRN (M4) measurements for the lower-limb perimeter.
P1 M3
P2 M4
M5
M6
M5
M6
WISCI Score
6
6
6
9
6
6
Time to walk 10 m [s]
99
98
108
89
66
52
Vital signs (pre) PA (mmHg)
130 × 90
120 × 80
130 × 90
120 × 80
120 × 70
120 × 80
Heart rate (bpm)
94
92
81
74
79
76
Sat O2 (%)
99%
97%
96%
97%
96%
96%
Borga (Upper limbs)b
0
0
0
0
0
0
Borg (Respiratory)c
0
0
0
0
0
0
Vital signs (post) PA (mmHg)
130 × 85
120 × 80
130 × 90
130 × 80
130 × 80
130 × 70
HR (bpm)
124
93
96
110
120
99
Sat O2 (%)
98%
92%
97%
99%
99%
98%
Borg (Upper limbs)
3
4
5
3
2
2
Borg (Respiratory)
2
2
5
3
2
1
Table 2. Walking functions assessment with assistive devices with the Walking Index for SCI (WISCI II) evaluation37. We evaluate the time to perform the 10 m task and vital signs recorded before and after the test. a The Borg fatigue assessment is a subjective self-reported level of exertion during exercise; bLevel of arms fatigue; cMeasurement the general physical fatigue (heat, sweat or dyspnea).
Overall, the motor improvement in both our patients could not be compared to the type of spontaneous improvement rates reported in the literature (scores <1 point when the training starts 26 weeks after the injury2) or those recorded for our patients prior to the start of training in our laboratory (0 for both patients, M1-M0). For patient P2, the improvement observed following the BFNR (9 points after just 5 months) was even higher than the one he experienced during the WANR protocol (i.e., a 4 point improvement during the first 28 months).
Discussion
This paper introduces a novel, non-invasive neurorehabilitation protocol for locomotion training for patients with severe chronic paraplegia resulting from spinal cord injuries, which targets both musculoskeletal training and corticospinal plasticity. Our setup integrated several technical novelties: a new sFES paradigm targeting muscle activation to produce locomotion, a BMI paradigm based on detection of left and right leg motor imagery to trigger the sFES, and sensory substitution to provide patients with tactile feedback from the lower limbs. We also introduced a step-by-step conditioning/training protocol and validated it with two individuals with paraplegia. Using this novel approach, we successfully tested a stimulation protocol to reproduce a smooth gait pattern; patients were able to use our setup to walk with partial suspension and little or no external help. This result was possible thanks to muscle conditioning preceding the sFES-generated walk. Following an SCI, muscles with myotomes originating below the lesion level undergo severe disuse atrophy47,48. Several years after the Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
9
www.nature.com/scientificreports/
www.nature.com/scientificreports
Figure 4. Neurological improvements. (A) Six neurological evaluations (M0 to M5) for both patients. Sensory score improvement is calculated using standard ISNCSCI assessment, compared to the score at the onset of the training (for both patients, the score at time = 0 was subtracted from all evaluations). The lower extremity motor score (LEMS) is based on clinical evaluation of five key lower-limb muscles (0 for no contraction, 5 for normal), the maximum score is 50. (B) Rectus muscle EMGs for patient P2 before and after the BFNR protocol. Signal was filtered between 5 Hz and 250 Hz with a Butterworth 4th-order band-pass, and a 60 Hz notch filter. The envelope was obtained after rectification of the filtered signal and a 1 Hz Butterworth high-pass filter. Thick black lines represent the physiotherapist’s instruction time to contract the muscles. The black horizontal line is the mean + 3 x SD of the activation as calculated during the baseline periods.
lesion, the volume of muscle fibers is reduced, becoming more and more prone to the process of fat infiltration. Concurrently, slow fatigable fibers tend to transform into fast fatigable fibers47. As a result of our training protocol, our patients experienced an increase in the volume of their leg muscle fibers and a parallel increase in their resistance to muscle fatigue21,49–51.
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
10
108
www.nature.com/scientificreports
www.nature.com/scientificreports/
P1 Muscles
P2
M3
M4
M5
Pilot tests
Key muscles
Auxiliary muscles
M6
M5
BFNR
M6
BFNR
Function
Main muscles
R
L
R
L
R
L
R
L
R
L
R
L
Hip flexor
Iliopsoas, rectus femoris proximal
1
1
1
1
1
1
1
1
0
0
1
1
Knee extensor
Vastus, rectus femoris distal
1
1
1
1
1
1
1
2
1
1
2
2
Ankle dorsiflexor
Tibialis anterior
0
0
0
0
0
0
0
0
0
0
1
1
Hallux extensor
Extensor hallucis longus
0
0
0
0
0
0
0
0
0
0
0
1
Ankle plantar flexor
Gastrocnemius, soleus
0
0
1
0
0
0
1
1
0
0
1
1
Hip adductor
Adductors
2
2
2
2
1
2
1
1
1
1
1
1
Knee flexor
Medial hamstring
0
0
0
0
0
0
0
0
0
0
1
1
Knee flexor
Lateral hamstring
0
0
0
1
0
1
0
1
0
0
1
1
Hip abductor
Gluteus medius
1
1
1
1
1
1
1
1
0
0
1
1
Hip extensor
Gluteus maximus
2
2
1
1
1
1
2
2
1
1
1
1
Toes extensor/flexor
Extensor/flexor digitorum longus
0
0
0
0
0
0
0
0
0
0
0
0
Hallux flexor
Flexor hallucis longus
0
0
0
0
0
0
0
0
0
0
0
0
LEMS (sum key muscles)
4
5
4
7
Sum all muscles
14
14
12
16
sFES muscles (16 muscles) Non-sFES muscles (8 muscles)
Regression
2
0
2
11
6
21
0
Stagnated
14/16
11/16
2/16
Improved
2/16
5/16
14/16
Regression
0
1/8
0
Stagnated
8/8
7/8
7/8
Improved
0
0
1/8
Table 3. Muscle score for five key muscles36 and seven auxiliary muscles. The test is done without any sFES stimulation. The muscles that were targeted during the BFNR training are in underlined. The lower part of the table shows the number of muscles among the ones that were targeted (sFES muscles) and the ones that were not targeted with the sFES (Non-sFES muscles) that have regressed, stagnated or improved during the pilot test and the BFRN.
Another important aspect of our strategy was the stimulation of a large number of muscles (16 lower-limb muscles) with precise timing. This was possible thanks to a custom-built programmable sFES stimulator39 to target each muscle individually. The conventional sFES technique to reproduce locomotion uses stimulation of the peroneal nerve to elicit the triple-flexion reflex during the swing and stimulation of the gluteus and the quadriceps during the stance phase of the gait35,52,53. This technique is limited by the high variability of the muscle response and the rapid habituation to the stimulation24. Instead, in our protocol, we decided to directly stimulate the lower-limb muscles to produce a consistent gait pattern. A major technical difficulty of our approach came from the inherent non-linearity of muscle responses54 (sigmoidal shape response type) and the effect of muscular fatigue due to the recruitment of muscles fibers in a non-selective, spatially fixed and temporally synchronous pattern55,56. In SCI subjects, following the lesion, there is also a trend towards the substitution of muscle fibers into rapidly fatigable fibers47. To cope with these difficulties, we proposed and successfully tested a closed-loop sFES controller using the joint angles of the lower-limbs to adapt the stimulation in real time. We also introduced online tactile feedback to help patients perceive gait events. Previously, our group has reported that this sort of feedback promotes cortical plasticity26 and provides patients with the type of lower-limb sensory information needed to improve their gait. Here, we observed that the use of tactile feedback increased the patients’ confidence and independence during the walk. Overall, using our setup and following our protocol, our patients acquired the ability to produce a smooth sFES-generated gait pattern. Indeed, for both patients, the main clinical features of human gait were respected: absence of crouching at stance, neutral position of the ankle at initial contact and during stance phase, as well as sufficient foot clearance during the swing. Our BFNR protocol was specifically designed considering the use of BMI for neurorehabilitation purposes (rather than assistance)35,57 for patients with SCI with minimal or no motor function. Therefore, the focus was not placed at achieving the highest level of control accuracy or speed, but rather to create a condition in which both afferent and efferent signals, converging at the level of the lesion, are engaged simultaneously. Thus, to trigger a step, patients had to perform and maintain leg motor imagery of the corresponding leg for two seconds. This protocol was particularly complicated for one patient who experienced more difficulties in control performance. Future work with a larger cohort of patients is necessary to address this issue; for example, a shorter trial-time could potentially facilitate the task. As the final outcome of this study, we observed that patients exposed to our protocol exhibited important clinical improvements, including, as expected, increases in muscle volume58. Moreover, we observed better outcomes in walking functions using assistive devices. As a result, our patients’ step frequency increased, and one patient was able to walk using less assistance at the end of the protocol. This was probably a consequence of better Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
11
109
www.nature.com/scientificreports/
www.nature.com/scientificreports
overall physical condition, which was indirectly inferred by the decrease in resting heart rate, smaller heart rate increase after the effort, improved muscular resistance, better coordination with walking devices due to the use of a walker and body weight support in the protocol, and the occurrence of a significant partial neurological recovery in the motor domain. Considering this latter component, partial motor neurological recovery was observed in different degrees: it was moderate in patient P1 and substantial in patient P2. Importantly, P2 was the patient who had good BMI control and good sFES responses. Further clinical trials are necessary to disambiguate if achieving good BMI-control accuracy is a necessary condition or even a determinant for patients to benefit from this training. Yet, there is one major question that remains open: what is the mechanism that induces this improvement? We hypothesize that, like in the case of our previous results with the WANR, the concurrent occurrence of cortical plasticity, induced by BMI use11,13,14 and spinal plasticity, promoted by sFES through ascending signals59–61 was an essential agent for the observed neurological motor improvement in our study. Recent work with human subjects showed that the use of invasive epidural stimulation has the potential to induce partial neurological restoration in chronic complete SCI patients9. Importantly, a study with rats15 showed that the motor recovery was faster and more important when the epidural stimulation was paired with brain-machine interfaces compared to conditions where epidural stimulation was used alone. These findings provide clear support for our hypothesis. For both patients, the cortical activation during leg motor imagery was found close to the cortical leg sensory-motor area. The presence of activation in posterior areas was previously reported in patients with chronic complete SCI due to cortical reorganization towards the primary somatosensory cortex62–64. Given the improvements obtained in this pilot study in terms of muscular volume and performance, better walking functions and neurological recovery, we propose that our BFNR protocol has the potential to become a valuable neurorehabilitation therapy for patients with SCI in the future.
Methods
Patients inclusion/exclusion criteria. Subjects were adults diagnosed with paraplegia, with traumatic SCI
thoracic injury at the chronic phase of the lesion, emotionally stable and with the absence of offset comorbidities. The inclusion criteria for the present study considered patients that had previously participated in the WANR protocol and who exhibited evidence of upper motor neuron injury, clinically manifested by the presence of spasticity and positive response to spinal reflexes. The screening was performed to test their muscles’ responsivity to sFES before inclusion. Exclusion criteria included lower motor neuron injury, the absence of muscle response during sFES responsivity screening, a degree of spasticity exceeding a score of 2 (Ashworth scale), and a degree of osteoporosis (T-score) greater than −4. Two patients were included in this study. Prior to the intervention, patient P1 and P2 followed a neurorehabilitation protocol introduced by our team (Walk Again Neurorehabilitation Protocol, WANR)20 which integrated the use of body weight support (BWS, Zero-G, Aretech LLC., Ashburn, VA), Lokomat6, brain-machine interface (both arms and legs motor imagery), and tactile feedback26 for 28 months and 34 months respectively. The two patients, referred to here as P1 and P2, were previously referred to as P5 and P6 by Donati et al.14 and Shokur et al.20,26.
Study approval. The study was approved by both the local ethics committee (Associação de Assistência à Criança Deficiente, São Paulo, São Paulo, Brazil #364.027) and the Brazilian federal government ethics committee (CONEP, CAAE: 13165913.1.0000.0085). The experiments were carried out as a feasibility study that did not meet the clinicaltrials.gov definition of an Applicable Clinical Trial. All research activities were carried out in accordance with the guidelines and regulations of the Associação de Assistência à Criança Deficiente and CONEP. The participants signed a written informed consent before enrolling in the study, and also signed a written informed consent for open access publication (print and digital) of their images. The experiments were carried out at the Associação Alberto Santos Dumont para Apoio à Pesquisa (AASDAP), São Paulo. sFES conditioning. An initial screening, which tested all muscles individually, helped set the stimulation amplitude and frequency parameters for the conditioning. The pulse width was fixed at 300 μs for the whole program65. The conditioning phase aimed at preparing the muscles progressively to produce a muscle contraction that allowed lower-limb movements of flexion, extension, and abduction, with the final goal to produce a sFES-generated gait. The stimulation time for each muscle increased gradually, up to 40 minutes maximum. The ramp up/down stimulation time decreased from one session to the next one. During the first eight sessions, the patient was maintained in the supine position, and the legs were positioned on rollers to allow free movement. The muscles were first stimulated individually, then in pairs and finally alternating between flexors and extensors (leg-by-leg pattern). The leg-by-leg pattern was applied in two phases: extensors of one leg were stimulated, while the flexors of the opposite leg contracted for 10–15 seconds. The muscles then rested for 2 seconds before the opposite pattern was applied. During the session, a physiatrist supervising the experiment evaluated the muscle response (visual evaluation of the range of motion of the joints and presence or absence of fatigue), the tonus behavior, the sensory perception of the stimulation and the ability to voluntarily improve the muscle contraction together with the stimulation. For the last session, the patient was supported in an upright position in a body weight support system (Zero-G, Aretech LLC., Ashburn, VA). The stimulation sequence, called stationary gait, differed from the leg-by-leg sequence due to the continuity of the stimulation to extend both lower-limbs. The following criteria were expected for a patient to continue with the FC phase in standing position: adequate muscle responses to allow the hip and knee extension, stable blood pressure, no signs of fatigue during the first 25 minutes, and no spasticity disturbing the stimulation pattern.
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
12
110
www.nature.com/scientificreports/
www.nature.com/scientificreports
Body-weight support system. A robotic body weight support system (Zero-G, Aretech LLC., Ashburn, VA) was used to support 65–70% of the weight of the patient during gait. This percentage was selected to provide the minimum suspension required to avoid a crouching position of the lower-limbs, ensuring good contact between the foot and the ground and reducing muscle effort to help avoid premature fatigue. Real-time tracking system. During all experiments, hip and knee joint angles were measured in real-time at 148 Hz. The system used four inertial measurement units (IMU, Trigno TM Inertial Measurement Sensors, Delsys, USA). One sensor was placed on each femur (thigh) and tibia (lower leg) of the patient. The joint angle was computed based on the difference in orientation of the sensors on the adjacent segments66. The patient was asked to keep the trunk position straight throughout the training. At the first session, the sensors were positioned on the body relative to anatomical references (patella, tibial tuberosity, and malleolus) and we calculated the parameters needed for the algorithm. Description of the gait pattern and rationale. To produce the gait movement, we designed an activation sequence based on 16 lower-limb muscles stimulated non-invasively. The stimulation pattern, shown in Fig. 1G was determined based on eight gait sub-phases, namely (1) initial contact, (2) load response, (3) mid-stance, (4) terminal stance, (5) pre-swing, (6) initial swing, (7) mid-swing and (8) terminal swing. The stride duration could vary from 4 to 20 seconds. The highest joint range of motion of the human gait takes place in the sagittal plane where flexion and extension of the hip, knee, and ankle occur. We selected extensors and flexors for each joint. We describe here each muscle activation of the stimulation pattern, considering the start of the stance as the start of the gait cycle. In this position, during initial contact, the vertical reaction force is acting in front of the hip joint promoting an internal flexor moment that will induce hip flexion. The GMx is therefore activated to challenge this internal moment to extend the hip and to align the trunk and pelvis in the orthostatic posture. During the single support phase (that includes mid-stance and terminal stance sub-phases), the patients’ lack of lateral pelvic stabilization could potentially compromise hip joint integrity. To avoid this issue, the Gmd muscle was stimulated to stabilize the pelvis segment in the frontal plane. The Sl muscle was recruited to control the advancement of the tibia in mid-stance, together with the knee extensor, VL, to maintain dynamic stability and extension of the knee at stance. The pre-swing phase started when the opposite foot touched the ground. At this instant, the Gs was stimulated to assist the knee flexion, and the Hs stimulation reinforced this movement. This condition promoted lifting the heel off the ground. After this, during the initial swing, the RFP was stimulated to flex the hip to lift the foot from the ground. Then the RFP acted to counteract the external lower-limb extension moment promoted by gravity. The RFP stimulation during terminal swing also allowed better control of the downward movement of the limb towards the ground, preparing the limb for the next cycle (next initial contact). The Hs stimulation was progressively replaced by the VL to extend the knee gradually in the mid- and terminal swing. The TA stimulation promoted ankle dorsiflexion to keep it in a neutral position during the swing phase and to avoid dragging the foot on the ground. The TA stimulation decreased progressively, slightly before the foot touched the ground, to let the foot set down gently on the floor in a flat position, minimizing joint injuries. Three types of different speeds ramps were used (slow, middle, rapid). sFES Stimulator. We used a custom 20-channel stimulator for this project 39,67. The stimulator sends charge-balanced, rectangular, biphasic pulses. Surface electrodes were used to stimulate 16 selected muscles as shown in Fig. 1C. Two sizes of electrodes were used: 5 × 10 cm (for GMx, Gmd, VL, Hs, Gs) and 5 × 5 cm to target important muscles more locally (RFP) or smaller muscles (TA, Sl). The stimulator allowed the modulation of pulse width, frequency and current intensity at a refresh rate of 10 ms. The pulse width was maintained at 300 μs as this was previously found as an optimal value65. By default, a frequency of 30 Hz was used68. This electrostimulator and the risk analysis associated report has been approved by the SwissMedic Ethical Committee for Clinical Investigations (SwissMedic Ethical approval, Ref MD-2007-MD-0031). PI controller. The PI controller ran at 50 Hz and adapted the stimulation amplitude to reach the predefined target angles (see Description of the gait pattern and rationale). For safety, the maximum current modulation was saturated within a range of 20% above or below the command. The parameters for the PI controllers were found using the pure PI control (i.e., without pre-established stimulation pattern) on healthy subjects to achieve typical joint trajectories (ramp target, sinusoidal target and kinematics curves from a normal gait cycle). Proportional and integral gain, Kp and Ki were respectively set to 0.5 and 0.001. PI controller characterization experiment. To evaluate the effect of the PI controller, we designed a specific experiment that isolated only the knee extension. The patient was sitting in his wheelchair with his feet hanging. The vastus of each leg was stimulated alternately to induce left and right knee extension. We designed the desired angle trajectory to be composed of ramps and plateaus to study the characteristics of the controller with constant and changing targets. We pseudo-randomized the trials where the PI controller was on (PI-on) and trials where it was off (PI-off) (three PI-off and three PI-on were shuffled every six repetitions). Only the vastus muscle was stimulated to act as knee extensor, and the knee flexion was performed passively by gravity. The whole experiment was composed of 100 repetitions (50 open-loop, PI-off, and 50 closed-loop, PI-on). The current amplitude model for the open-loop was composed of ramps and plateaus between two amplitudes. Those two amplitudes were selected during preliminary tests on the same day to obtain two levels of extension close to the desired target.
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
13
111
www.nature.com/scientificreports/
www.nature.com/scientificreports
Custom visualization system and LEDs. During the B + FL phase, two arrays of four LEDs were placed on each side of the walker to indicate the output of the EEG classifier and the experiment’s protocol. The left and rightmost LEDs informed the patients of the experiment with a simple color convention: black (turned off) for idle time, pink for focus, and left or right blue LED for corresponding leg motor imagery. At the end of each trial, the patient could see if he succeeded or failed the trial (green or red LED). Patients’ EEG classifier value was discretized in the [−1.5; +1.5] range and shown using the six remaining LEDs. EEG and BMI. We used the opensource software OpenViBE 0.16.0, for the acquisition and processing of EEG signals69. EEG signals were acquired at 2000 Hz with the V-amp amplifier (Brain Products, GmbH). Sixteen channels were recorded and clustered around the leg representation of the primary sensorimotor cortex (Fig. 1E). We used actiCAP active electrodes, connected to the wireless MOVE (both Brain Products, GmbH), carried with the patient. The wireless receiver was directly connected to the amplifier via USB to a computer. EEG signals were band-pass filtered at 8–30 Hz (mu and beta), corresponding to the well-documented frequency band for motor-related EEG activity70. We then used two algorithms for the detection of motor imagery: the Common Spatial Patterns (CSP) algorithm46 as a spatial filter and Linear Discriminant Analysis (LDA) for classification. The CSP aims at learning spatial filters, such that the variance of the signals is maximized for one class (e.g., one mental imagery task) and minimized for the other class. The Linear Discriminant Analysis (LDA) algorithm was used to estimate a linear hyperplane to separate feature vectors from two classes)71. In our case, if the LDA output was positive, the feature vector was assigned to the right motor imagery class. Otherwise, it was assigned to the left leg motor imagery class. Every BMI session started with 8 minutes of training (40 trials, 20 left and right leg motor imageries, randomized order). Each trial consisted of 1-second preparation, followed by 1.25 s of cue presentation and a 3.75 s motor imagery period. The intertrial time was randomly chosen between 1.5 and 3.5 seconds. Patients were instructed to produce motor imagery involving only one leg at the time as, e.g., ‘imagine kicking a ball with your right leg,’ ‘imagine making a circular movement with your left ankle,’ and to avoid motor imageries that involved the movement of both legs such as ‘cycling’ or ‘walking.’ We used a 5-fold, cross-validation for calculation of the classifier accuracy. During this acquisition, the patient already had the sFES electrodes placed and connected to the stimulator (with the stimulator turned off); he was standing, in the BWS system, but without suspension and was supported in an upright position. Before the first BMI run, the patient remained for 30 seconds without motor imagery in a standing position to record the baseline classifier. The mean of this signal was subtracted from the classifier for the run to avoid any offset. The baseline could be calculated one more time before the fourth run if needed to compensate for any new offset. During the online control, we used the parameters for the CSP and the LDA that were calculated during the training phase. The only difference with the training phase was a threshold of 0.1 on the classifier value to reduce chances of false positives: we considered detection of left or right motor imagery only when the classifier was, respectively, smaller than −0.1 or above 0.1. Tactile shirt. The tactile shirt was composed of the three vibrators aligned on the ulna as described in26. Patients received continuous tactile feedback going from the wrist to the elbow (apparent movement as described in45) coinciding with the stimulation of the lower-limb muscles to indicate the beginning the swing (start of phase six, Fig. 1G). At the end of phase eight, they received stimulation of all three vibrators together for 600 ms. Calculation of the chance level for the BMI. The baseline data was gathered during resting time with the same patients, collected at the onset of each BMI session. All experimental conditions were the same as those for the current experimental paradigm. Patients were using the same EEG cap, and setup (standing position). EEG signals were recorded for 30 seconds prior to the session. We gathered all the baseline datasets and cut them into 4 s time windows corresponding to step trials. Then we applied the exact same algorithm and simulated a sequence of runs based on these datasets. We report the statistics for chance level, obtained by simulating 600 steps (50 runs of 12 steps): 5.72 + − 1.82 (mean + −SD) correct steps performed over 12 steps run, and the time to perform a step is on average 3.31 + −0.84[s]. Session interruption and data exclusion. A clinical check was done before and during the sessions. If the patient presented abnormal low blood pressure or severe cases of spasticity before or during the session, the session was, respectively, not initiated or interrupted. Qualitative visual gait evaluation. Each evaluation session had six FL runs; each run had 12 steps. Step time was pseudo-randomized per run (8 s/step, 6 s/step or 4 s/step). Video clips of the patients’ runs were cut, shuffled and given to an expert physiotherapist blinded to the conditions of the experience, with a questionnaire developed by our group (referred to here as visual gait score for sFES, see Supplementary Table S4 for details). The questionnaire included three control, five high-level and seven kinematics-oriented questions. The answers were Likert-type, ranging from 1 to 5. The control questions assessed respectively whether the patient exhibited spasticity, clear signs of fatigue or reduction of muscle response to the stimulation during the run. If any of the three control questions had a score superior or equal to 4, the run was discarded. No runs were removed for any of the two sessions used for the tactile-on versus tactile-off experiment with patient P1. One of the runs of patient P2 and none of P1 were discarded from the qualitative analysis of sFES-generated walk. 3D Gait analysis. Standard gait-analysis procedures were employed to collect data using a motion-capture system comprised of 12 cameras and a 4 Mb resolution (Raptor-4) and the Cortex 6.0 software (Motion Analysis, Santa Rosa, CA, USA). The kinematic data were acquired at 150 Hz. The marker-set protocol adopted was
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
14
112
www.nature.com/scientificreports/
www.nature.com/scientificreports
comprised of 28 anatomical reflective markers (Helen Hayes marker set). Markers were placed directly onto the skin in the patient’s pelvic and lower-extremity segments. Offline, raw marker-trajectory data were filtered using a 4th-order, low-pass Butterworth filter with a cut-off frequency of 10 Hz. Data processing was performed using the Cortex software version 6.0. Visual 3D software (C-motion Inc., Germantown, MD, USA) was used to perform all kinematics calculations. The foot strike and toe-off were determined when the velocity of the heel and toe markers respectively crossed a 20 mm/s threshold level. The definition of the anatomical-segment coordinate system was used to determine the 3D position and orientation of the lower extremity and pelvis segments through a combination of anatomical-frame conventions proposed previously72. Patient P2 performed 54 full gait cycles, with 101 time-normalized points, which corresponded to the time-normalized angles (pelvis, hip, knee, ankle, and foot).
Data Availability
The custom code used for the experiments and the data that support the findings of this study are available from the corresponding author upon reasonable request.
References
1. World Health Organization. Spinal cord injury, Fact sheet No. 384. 2. Fawcett, J. W. et al. Guidelines for the conduct of clinical trials for spinal cord injury as developed by the ICCP panel: Spontaneous recovery after spinal cord injury and statistical power needed for therapeutic clinical trials. Spinal Cord 45, 190–205 (2007). 3. Kirshblum, S., Millis, S., McKinley, W. & Tulsky, D. Late neurologic recovery after traumatic spinal cord injury. Arch. Phys. Med. Rehabil. 85, 1811–1817 (2004). 4. Lam, T., Wolfe, D. L., Domingo, A., Eng, J. J. & Sproule, S. Lower Limb Rehabilitation Following Spinal Cord Injury. spinal Cord Inj. Rehabil. Evid. Version 5, 1–73 (2014). 5. Behrman, A. L., Bowden, M. G. & Nair, P. M. Neuroplasticity After Spinal Cord Injury and Training: An Emerging Paradigm Shift in Rehabilitation and Walking Recovery. Phys. Ther. 86, 1406–1425 (2006). 6. Jezernik, S., Colombo, G., Keller, T., Frueh, H. & Morari, M. Robotic Orthosis Lokomat: A Rehabilitation and Research Tool. Neuromodulation 6, 108–115 (2003). 7. Esquenazi, A., Talaty, M., Packel, A. & Saulino, M. The ReWalk Powered Exoskeleton to Restore Ambulatory Function to Individuals with Thoracic-Level Motor-Complete Spinal Cord Injury. Am. J. Phys. Med. Rehabil. 91, 911–921 (2012). 8. Swinnen, E., Duerinck, S., Baeyens, J.-P., Meeusen, R. & Kerckhofs, E. Effectiveness of robot-assisted gait training in persons with spinal cord injury: a systematic review. J. Rehabil. Med. 42, 520–526 (2010). 9. Rejc, E., Angeli, C. A., Atkinson, D. & Harkema, S. J. Motor recovery after activity-based training with spinal cord epidural stimulation in a chronic motor complete paraplegic. Sci. Rep. 7, 13476 (2017). 10. Gerasimenko, Y. P. et al. Noninvasive Reactivation of Motor Descending Control after Paralysis. J. Neurotrauma 32, 1968–1980 (2015). 11. Capogrosso, M. et al. A brain-spine interface alleviating gait deficits after spinal cord injury in primates. Nature 539, 284–288 (2016). 12. Hidler, J. et al. ZeroG: Overground gait and balance training system. J. Rehabil. Res. Dev. 48, 287 (2011). 13. Lebedev, M. A. & Nicolelis, M. A. L. Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation. Physiol. Rev. 97, 767–837 (2017). 14. Donati, A. R. C. et al. Long-Term Training with a Brain-Machine Interface-Based Gait Protocol Induces Partial Neurological Recovery in Paraplegic Patients. Sci. Rep. 6, 30383 (2016). 15. Bonizzato, M. et al. Brain-controlled modulation of spinal circuits improves recovery from spinal cord injury. Nat. Commun. 9, 3015 (2018). 16. Tazoe, T. & Perez, M. A. Effects of repetitive transcranial magnetic stimulation on recovery of function after spinal cord injury. Arch. Phys. Med. Rehabil. 96, S145–55 (2015). 17. Raithatha, R. et al. Non-invasive brain stimulation and robot-assisted gait training after incomplete spinal cord injury: A randomized pilot study. NeuroRehabilitation 38, 15–25 (2016). 18. McDonald, J. W. et al. Late recovery following spinal cord injury. Case report and review of the literature. J. Neurosurg. 97, 252–65 (2002). 19. Kapadia, N. et al. A randomized trial of functional electrical stimulation for walking in incomplete spinal cord injury: Effects on walking competency. J. Spinal Cord Med. 37, 511–524 (2014). 20. Shokur, S. et al. Training with brain-machine interfaces, visuo- tactile feedback and assisted locomotion improves sensorimotor, visceral, and psychological signs in chronic paraplegic patients. PLoS One 13, 1–33 (2018). 21. Greve, J. M. D. et al. Functional electrical stimulation (FES): muscle histochemical analysis. Spinal Cord 31, 764–770 (1993). 22. Ragnarsson, K. T. Functional electrical stimulation after spinal cord injury: current use, therapeutic effects and future directions. Spinal cord Off. J. Int. Med. Soc. Paraplegia 46, 255–274 (2008). 23. Sadowsky, C. L. & McDonald, J. W. Activity-based restorative therapies: Concepts and applications in spinal cord injury-related neurorehabilitation. Dev. Disabil. Res. Rev. 15, 112–116 (2009). 24. Popovic, M. R., Masani, K. & Micera, S. Functional Electrical Stimulation Therapy: Recovery of Function Following Spinal Cord Injury and Stroke. In Neurorehabilitation Technology 105–121, https://doi.org/10.1007/978-1-4471-2277-7_7 (Springer London, 2012). 25. Hornby, T. G., Zemon, D. H. & Campbell, D. Robotic-assisted, body-weight-supported treadmill training in individuals following motor incomplete spinal cord injury. Phys. Ther. 85, 52–66 15p (2005). 26. Shokur, S. et al. Assimilation of virtual legs and perception of floor texture by complete paraplegic patients receiving artificial tactile feedback. Sci. Rep. 6 (2016). 27. Soekadar, S. R., Birbaumer, N. & Cohen, L. G. Brain–Computer Interfaces in the Rehabilitation of Stroke and Neurotrauma. In Systems Neuroscience and Rehabilitation 54, 3–18 (Springer Japan, 2011). 28. Ramos-Murguialday, A. et al. Brain-machine interface in chronic stroke rehabilitation: A controlled study. Ann. Neurol. 74, 100–108 (2013). 29. Silvoni, S. et al. Brain-Computer Interface in Stroke: A Review of Progress. Clin. EEG Neurosci. 42, 245–252 (2011). 30. Shindo, K. et al. Effects of neurofeedback training with an electroencephalogram-based brain-computer interface for hand paralysis in patients with chronic stroke: A preliminary case series study. J. Rehabil. Med. 43, 951–957 (2011). 31. Biasiucci, A. et al. Brain-actuated functional electrical stimulation elicits lasting arm motor recovery after stroke. Nat. Commun. 1–13, https://doi.org/10.1038/s41467-018-04673-z 32. Takahashi, M. et al. Event related desynchronization-modulated functional electrical stimulation system for stroke rehabilitation: A feasibility study. J. Neuroeng. Rehabil. 9, 56 (2012). 33. Soekadar, S. R., Birbaumer, N., Slutzky, M. W. & Cohen, L. G. Brain–machine interfaces in neurorehabilitation of stroke. Neurobiol. Dis. 83, 172–179 (2015).
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
15
113
www.nature.com/scientificreports/
www.nature.com/scientificreports
34. Ajiboye, A. B. et al. Restoration of reaching and grasping movements through brain-controlled muscle stimulation in a person with tetraplegia: a proof-of-concept demonstration. Lancet 389, 1821–1830 (2017). 35. King, C. E. et al. The feasibility of a brain-computer interface functional electrical stimulation system for the restoration of overground walking after paraplegia. J. Neuroeng. Rehabil. 12, 80 (2015). 36. Kirshblum, S. C. et al. International standards for neurological classification of spinal cord injury (Revised 2011). J. Spinal Cord Med. 34, 547–554 (2011). 37. Morganti, B., Scivoletto, G., Ditunno, P., Ditunno, J. F. & Molinari, M. Walking index for spinal cord injury (WISCI): Criterion validation. Spinal Cord 43, 27–33 (2005). 38. Nicholas, J. J., Taylor, F. H., Buckingham, R. B. & Ottonello, D. Measurement of circumference of the knee with ordinary tape measure. Ann. Rheum. Dis. 35, 282–4 (1976). 39. Stauffer, Y., Bouri, M., Clavel, R., Brodard, R. & Allemand, Y. A novel verticalized reeducation device for spinal cord injuries: the WalkTrainer, from design to clinical trials. In Robotics 2010: Current and Future Challenges 194–209 (2010). 40. Mijailović, N., Gavrilović, M., Rafajlović, S., urić-Jovičić, M. & Popović, D. Gait Phases Recognition from Accelerations and Ground Reaction Forces: Application of Neural Networks. Telfor J. 1, 34–36 (2006). 41. Winter, D. A. & Yack, H. J. EMG profiles during normal human walking: stride-to-stride and inter-subject variability. Electroencephalogr. Clin. Neurophysiol. 67, 402–411 (1987). 42. Kadaba, M. P. et al. Repeatability of kinematic, kinetic, and electromyographic data in normal adult gait. J. Orthop. Res. 7, 849–860 (1989). 43. Dietz, V. Proprioception and locomotor disorders. Nat. Rev. Neurosci. 3, 781–790 (2002). 44. Conway, B. A., Hultborn, H. & Kiehn, O. Proprioceptive input resets central locomotor rhythm in the spinal cat. Exp. Brain Res. 68, 643–656 (1987). 45. Sherrick, C. E. & Rogers, R. Apparent haptic movement. Percept. Psychophys. 1, 175–180 (1966). 46. Ramoser, H., Müller-Gerking, J. & Pfurtscheller, G. Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans. Rehabil. Eng. 8, 441–446 (2000). 47. Burnham, R. et al. Skeletal muscle fibre type transformation following spinal cord injury. Spinal Cord 35, 86–91 (1997). 48. Shields, R. K. Muscular, Skeletal, and Neural Adaptations Following Spinal Cord Injury. J. Orthop. Sport. Phys. Ther. 32, 65–74 (2002). 49. Peckham, P. H., Mortimer, J. T. & Marsolais, E. B. Alteration in the force and fatigability of skeletal muscle in quadriplegic humans following exercise induced by chronic electrical stimulation. Clin. Orthop. Relat. Res. 326–334 (1976). 50. Martin, R., Sadowsky, C., Obst, K., Meyer, B. & McDonald, J. Functional Electrical Stimulation in Spinal Cord Injury: From Theory to Practice. Top. Spinal Cord Inj. Rehabil. 18, 28–33 (2012). 51. Ragnarsson, K. T. Physiologic effects of functional electrical stimulation-induced exercises in spinal cord-injured individuals. Clin. Orthop. Relat. Res. 53–63 (1988). 52. Graupe, D. & Kohn, K. H. Functional neuromuscular stimulator for short-distance ambulation by certain thoracic-level spinal-cordinjured paraplegics. Surg. Neurol. 50, 202–207 (1998). 53. Gallien, P. et al. Restoration of gait by functional electrical stimulation for spinal cord injured patients. Paraplegia 33, 660–664 (1995). 54. Ferrarin, M., Palazzo, F., Riener, R. & Quintern, J. Model-based control of FES-induced single joint movements. IEEE Trans. Neural Syst. Rehabil. Eng. 9, 245–257 (2001). 55. Gregory, C. M. & Bickel, C. S. Recruitment patterns in human skeletal muscle during electrical stimulation. Phys. Ther. 85, 358–364 (2005). 56. Bigland-Ritchie, B., Jones, D. A. & Woods, J. J. Excitation frequency and muscle fatigue: Electrical responses during human voluntary and stimulated contractions. Exp. Neurol. 64, 414–427 (1979). 57. Kilicarslan, A., Prasad, S., Grossman, R. G. & Contreras-Vidal, J. L. High accuracy decoding of user intentions using EEG to control a lower-body exoskeleton. In 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 75, 5606–5609 (IEEE, 2013). 58. Sloan, K. E., Bremner, L. A., Byrne, J., Day, R. E. & Scull, E. R. Musculoskeletal effects of an electrical stimulation induced cycling programme in the spinal injured. Paraplegia 32, 407–415 (1994). 59. Rushton, D. Functional Electrical Stimulation and rehabilitation—an hypothesis. Med. Eng. Phys. 25, 75–78 (2003). 60. Beaumont, E. et al. Functional electrical stimulation post-spinal cord injury improves locomotion and increases afferent input into the central nervous system in rats. J. Spinal Cord Med. 37, 93–100 (2014). 61. Ko, C.-Y. et al. Evaluation of physical and emotional responses to vibrotactile stimulation of the forearm in young adults, the elderly, and transradial amputees. Physiol. &amp; Behav. 138, 87–93 (2015). 62. Kokotilo, K. J., Eng, J. J. & Curt, A. Reorganization and Preservation of Motor Control of the Brain in Spinal Cord Injury: A Systematic Review. J. Neurotrauma 26, 2113–2126 (2009). 63. Lotze, M., Laubis-Herrmann, U., Topka, H., Erb, M. & Grodd, W. Reorganization in the primary motor cortex after spinal cord injury - A functional Magnetic Resonance (fMRI) study. Restor. Neurol. Neurosci. 14, 183–187 (1999). 64. Green, J. B., Sora, E., Bialy, Y., Ricamato, A. & Thatcher, R. W. Cortical sensorimotor reorganization after spinal cord injury: An electroencephalographic study. Neurology 50, 1115–1121 (1998). 65. Métrailler, P. Système robotique pour la mobilisation des membres inférieurs d’une personne paraplégique. 3191, (EPFL, 2005). 66. Dejnabadi, H., Jolles, B. M. & Aminian, K. A New Approach to Accurate Measurement of Uniaxial Joint Angles Based on a Combination of Accelerometers and Gyroscopes. IEEE Trans. Biomed. Eng. 52, 1478–1484 (2005). 67. Schmitt, C. et al. A Study of a Knee Extension Controlled by a Closed Loop Functional Electrical Stimulation. In 9th Annual Conference of the International FES Society 3–5 (2004). 68. Malesevic, N. M., Popovic, L. Z., Schwirtlich, L. & Popovic, D. B. Distributed low-frequency functional electrical stimulation delays muscle fatigue compared to conventional stimulation. Muscle and Nerve 42, 556–562 (2010). 69. Renard, Y. et al. OpenViBE: An Open-Source Software Platform to Design, Test, and Use Brain–Computer Interfaces in Real and Virtual Environments. Neural Comput. 19, 35–53 (2010). 70. Pfurtscheller, G. & Neuper, C. Motor imagery and direct brain-computer communication. Proc. IEEE 89, 1123–1134 (2001). 71. Lotte, F. & Congedo, M. A review of classification algorithms for EEG-based brain – computer interfaces. J Neural Eng 4, R1–R13 (2007). 72. Cappozzo, a, Catani, F., Della Croce, U. & Leardini, A. Position and orietnation in space of bones during movement: anatomical frame definition and determination. Clin. Biomech. 10, 171–178 (1995).
Acknowledgements
This study was funded by the Brazilian Financing Agency for Studies and Projects (FINEP 01·12·0514·00), Brazilian Ministry of Science, Technology, and Innovation (MCTI). We acknowledge the National Institute of Science and Technology (INCT) Brain Machine-Interface (INCEMAQ) CNPq 573966/2008-7 of Brazilian Ministry of Science, Technology, and Innovation (MCTI/CNPq/FNDCT/CAPES/FAPERN). We want to thank
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
16
114
www.nature.com/scientificreports/
www.nature.com/scientificreports
Neiva Paraschiva, Adriana Ragoni, Andrea Arashiro, Maria Cristina Boscarato, Fabio Asnis, Nathan Rios, Dr. Tiago Kunrath, and Susan Halkiotis (Duke University) for their work, help, and support for this study. We finally want to thank the patients for their long-term commitment, and their trust in this research and our team.
Author Contributions
A.S. and S.S. contributed equally to this work. A.S., S.S. and M.A.L.N. contributed to the study design, data interpretation, data collection, data analysis, literature search, figures, tables, writing, and editing. D.S.F.C. contributed to the study design, data collection, analysis, figures, and editing. A.R.C.D. and S.A. contributed to the study design and data collection and analysis. S.Y. contributed to data analysis and figures. D.B.C. contributed to data collection and analysis. M.B. contributed to the study design and data interpretation.
Additional Information
Supplementary information accompanies this paper at https://doi.org/10.1038/s41598-019-43041-9. Competing Interests: The authors declare no competing interests. Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. © The Author(s) 2019
Scientific Reports |
(2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9
17
115
116
Neuroprostheses for Parkinsonâ&#x20AC;&#x2122;s Disease and Epilepsy
117 The Journal of Neuroscience, November 1, 2000, 20(21):8160–8168
Reduction of Pentylenetetrazole-Induced Seizure Activity in Awake Rats by Seizure-Triggered Trigeminal Nerve Stimulation Erika E. Fanselow,1 Ashlan P. Reid,2 and Miguel A. L. Nicolelis1,2 Departments of 1Neurobiology and 2Biomedical Engineering, Duke University Medical Center, Durham, North Carolina 27710
Stimulation of the vagus nerve has become an effective method for desynchronizing the highly coherent neural activity typically associated with epileptic seizures. This technique has been used in several animal models of seizures as well as in humans suffering from epilepsy. However, application of this technique has been limited to unilateral stimulation of the vagus nerve, typically delivered according to a fixed duty cycle, independently of whether ongoing seizure activity is present. Here, we report that stimulation of another cranial nerve, the trigeminal nerve, can also cause cortical and thalamic desynchronization, resulting in a reduction of seizure activity in awake rats. Furthermore, we demonstrate that providing this stimulation only when seizure activity begins results in more effective and safer seizure reduction per second of stimulation than with previous methods. Seizure activity induced by intraperitoneal injection of pentylenetetrazole was recorded from microwire electrodes in the thalamus and cortex of awake rats while the infraorbital branch of the trigeminal nerve was stimulated via a chronically implanted nerve cuff electrode. Continuous unilateral stimulation of the trigeminal
nerve reduced electrographic seizure activity by up to 78%, and bilateral trigeminal stimulation was even more effective. Using a device that automatically detects seizure activity in real time on the basis of multichannel field potential signals, we demonstrated that seizure-triggered stimulation was more effective than the stimulation protocol involving a fixed duty cycle, in terms of the percent seizure reduction per second of stimulation. In contrast to vagus nerve stimulation studies, no substantial cardiovascular side effects were observed by unilateral or bilateral stimulation of the trigeminal nerve. These findings suggest that trigeminal nerve stimulation is safe in awake rats and should be evaluated as a therapy for human seizures. Furthermore, the results demonstrate that seizure-triggered trigeminal nerve stimulation is technically feasible and could be further developed, in conjunction with real-time seizure-predicting paradigms, to prevent seizures and reduce exposure to nerve stimulation.
Seminal neurophysiological studies performed several decades ago demonstrated that stimulation of either cranial nerves or areas of the brainstem can cause desynchronization of the cortical EEG (Moruzzi and Magoun, 1949; Zanchetti et al., 1952; Magnes et al., 1961; Chase et al., 1967). Such desynchronization typically reflects a state of arousal and full vigilance in mammals and is opposite to the high degree of EEG synchronization observed during seizure activity. Building on these classical findings, several researchers showed that stimulation of the vagus nerve can lead to EEG desynchronization (Zanchetti et al., 1952; Chase et al., 1966, 1967; Chase and Nakamura, 1968). More recently, several studies have demonstrated that the desynchronization induced by vagus nerve stimulation (VNS) in dogs can be used to reduce strychnine- or pentylenetetrazole (PTZ)-induced seizure activity (Zabara, 1985, 1992). This paradigm was demonstrated subsequently to be effective in other animals, with other seizure models (Lockard et al., 1990; Woodbury and Woodbury, 1990; McLachlan, 1993), and has been used with moderate success in treating humans who have otherwise intractable epileptic seizures (Penry and Dean, 1990; Uthman et al., 1990, 1993; Ben-Menachem et al., 1994; Vagus Nerve Stimulation Study Group, 1995; McLachlan, 1997; Schachter and Saper, 1998). Because 0.5–2% of the population has epilepsy (Schachter and Saper, 1998; McNamara, 1999) and 10 –50% of these patients do not respond well to antiepileptic medications and/or may not be candidates for resective epilepsy surgery (McLachlan, 1997; Schachter and Saper, 1998), there is a substan-
tial need for potential alternative therapies for chronic seizures. Indeed, the VNS technique has recently received FDA approval and is currently being used in patients. There are, however, several limiting factors of the VNS technique, which, if addressed, could greatly increase the efficacy and applicability of cranial nerve stimulation for seizure reduction in patients. First, the standard implementation of VNS in humans typically involves stimulating the vagus nerve on a fixed, intermittent duty cycle (e.g., 30 sec on; 5 min off; 24 hr a day), independently of whether any seizure activity is ongoing or imminent (although the use of manual patient- or caregiver-triggered stimulation via a handheld magnet has also been used) (Terry et al., 1990; Uthman et al., 1993). This type of protocol has been used in previous studies for two main reasons. First, although continuous stimulation may have a greater therapeutic effect than intermittent stimulation (Takaya et al., 1996), continuous stimulation can cause nerve damage, whereas intermittent stimulation does not (Agnew et al., 1989; Agnew and McCreery, 1990; Ramsay et al., 1994). Second, the side effects associated with VNS are typically experienced during the stimulation (Uthman et al., 1993; Ramsay et al., 1994; McLachlan, 1997), so giving intermittent stimulation reduces their occurrence. However, because stimulation is delivered regardless of whether seizure activity is present or is likely to occur, this fixed stimulation protocol has the disadvantage that the patient may receive excess stimulation. The second main problem is that the vagus nerve is involved in, among other things, cardiovascular and abdominal visceral functions. Indeed, because of the pattern of vagus innervation of the heart, the vagus nerve can only safely be stimulated unilaterally (i.e., on the left side only). This is a potential limitation in the efficacy of cranial nerve stimulation because the effects of the stimulation may be bilateral (Chase et al., 1966; Zabara, 1992; Henry et al., 1998, 1999) and may, therefore, be aided by adding more stimulation sites. For these reasons, use of a nerve without
Received May 19, 2000; revised Aug. 7, 2000; accepted Aug. 11, 2000. This work was funded by a grant from the Klingenstein Foundation to M.A.L.N. and by National Institute of Dental Research Grant DE-11121-01 to M.A.L.N. We thank Dr. James O. McNamara for helpful comments on this manuscript. Correspondence should be addressed to Erika E. Fanselow, Department of Neurobiology, Duke University Medical Center, Durham, NC 27710. E-mail: efanse@neuro.duke.edu. Copyright © 2000 Society for Neuroscience 0270-6474/00/208160-09$15.00/0
Key words: epilepsy; trigeminal nerve; seizure detection; seizure control; pentylenetetrazole; bilateral stimulation
118 Fanselow et al. • Seizure Reduction by Trigeminal Nerve Stimulation
Figure 1. Schematic drawing of nerve cuff electrode and ASD device. Field potential signals from chronically implanted microwires in the VPM thalamus and/or SI cortex were sent to an amplifier and recording unit for collection, as well as to the ASD device. When the ASD device detected seizure activity, it sent a signal to the stimulator, which delivered a current pulse to the implanted nerve cuff electrode. Scale bar: inset, 1 mm.
the types of visceral fibers that are found in the vagus nerve could potentially be more effective for seizure reduction. Here, we demonstrate that trigeminal nerve stimulation reduced PTZ-induced seizures in awake rats. In addition, we show that such stimulation was more effective when it was bilateral. Finally, we describe a real-time interface for automatically detecting seizure activity and eliminating it by providing stimulation only when seizure activity is present. These results suggest that if such techniques were implemented in human patients, they could greatly decrease the amount of stimulation necessary for seizure control while increasing the efficacy of cranial nerve stimulation as a therapy for intractable epilepsy.
MATERIALS AND METHODS Subjects. Eight adult female L ong–Evans hooded rats weighing between 230 and 375 gm served as subjects in this study. All procedures and experiments were conducted in compliance with Duke University Medical C enter animal use policies and were approved by the Duke University Institutional Animal C are and Use Committee. Induction of seizures. Seizures were induced by intraperitoneal injection of P TZ (40 mg / kg). This dose of P TZ induced generalized seizure activity for 1–2 hr. This seizure activity was manifested in two ways: (1) highly synchronous, large-amplitude activity in the thalamic and cortical field potential traces (see Figs. 2, 3, 5–7) and (2) clonic jerking of the body and forelimbs. These two indicators of seizure activity were highly correlated at all times, as assessed by concurrent visual inspection of the animal and the real-time field potential traces. Occasionally, a supplemental dose of P TZ (7–10 mg / kg) was given if seizures ceased in ⬍1 hr. Nerve cuff electrodes. The infraorbital nerve(s) was stimulated unilaterally or bilaterally via chronically implanted nerve cuff electrodes. These electrodes were constructed in-house and consisted of two bands of platinum (0.5 mm wide and 0.025 mm thick; ⬃0.8 mm separation between bands) that ran circumferentially around the infraorbital (IO) nerve (see Fig. 1, inset). The platinum bands were held in place and electrically insulated by a thin Sylgard coating. Each band was connected to a piece of flexible, three-stranded Teflon-coated wire that was used to pass current between the bands (Fanselow and Nicolelis, 1999).
J. Neurosci., November 1, 2000, 20(21):8160–8168 8161
Chronic implantation of microwire electrodes. Microwire electrodes (N BLabs, Denison, TX) were chronically implanted into the ventral posterior medial thalamus (V PM) and /or primary somatosensory cortices (SI) for use in recording field potentials in these areas (Fig. 1). Three rats had arrays of 16 microwires implanted in layer V of the SI cortex and bundles of 16 electrodes implanted into the V PM thalamus, both contralateral to the stimulated nerve. Five rats had two arrays of 16 electrodes implanted, one each into layer V of the left and right SI cortices so that recordings could be made both ipsilateral and contralateral to the nerve being stimulated. These implants were performed under pentobarbital anesthesia (50 mg / kg). Small craniotomies were performed over the areas into which electrodes were to be implanted [coordinates from Paxinos and Watson (1986)]. Electrodes were slowly lowered into these areas, and recordings were made throughout the implantation process to assess electrode location. After electrodes were in the correct position, they were cemented to skull screws by the use of dental acrylic (Nicolelis et al., 1997). Chronic implantation of nerve cuff electrodes. After implantation of the microwires, nerve cuff electrodes were implanted either unilaterally or bilaterally. A dorsoventral incision was made on the face several millimeters caudal to the caudal edge of the whiskerpad. Tissue was dissected until the infraorbital nerve was exposed, and the cuff electrode was positioned around the nerve such that the nerve lay inside the cuff. The cuff was then tied around the nerve to hold it in place, and the wound was sutured. The Teflon-coated leads from the platinum bands were run subcutaneously to the top of the head where they were attached to connector pins and affixed to the skull. Recording procedures. Field potential recordings from V PM thalamus and SI cortex were made using chronically implanted microwires (Nicolelis et al., 1997). Field potentials were collected using a Grass Model 15 amplifier and stored on a personal computer. Signals were collected at a sampling rate of 512 Hz and bandpass filtered during collection at 1–100 Hz. During each recording session, 16 field potential channels were recorded, 8 from each area from which recordings were made in a given rat (either V PM and SI, or SI left and SI right). In addition, one channel was recorded for each nerve cuff being stimulated (unilateral or bilateral stimulation) to indicate when stimulation occurred. During experiments, animals were awake and allowed to move freely in a 30 cm ⫻ 30 cm recording chamber. Stimulation parameters. Stimulation of the IO nerve cuff electrodes was provided by the use of a Grass S8800 stimulator in conjunction with a Grass SI U6 stimulus isolation unit. Unimodal square current pulses with a duration of 500 sec were given at a range of currents and frequencies. Current values were varied from 3 to 11 mA (2 mA intervals), and frequency values were varied from 1 to 333 Hz (1, 5, 10, 20, 50, 100, 125, 200, and 333 Hz). Animals tolerated stimulation at these levels without indication of pain, although in some animals there appeared to be a sensation of pressure on the face at the highest current and frequency settings. This was evidenced by a tendency for the animals to back up when the stimulus began, in the direction away from the stimulated side if unilateral stimulation was provided or straight back in the case of bilateral stimulation. In addition, at lower stimulus intensities animals would occasionally scratch at the whiskerpad on the side of the face being stimulated during the first few seconds of stimulation. However, the scratching was neither intense nor prolonged. Automatic seizure detection device. A device was designed and built in-house to automatically detect seizure activity in real time and immediately trigger a stimulator when a seizure was detected (Fig. 1). The automatic seizure detection (ASD) device first low-pass filtered the raw field potentials obtained from the microwire arrays at 30 Hz. C ircuitry then determined whether the field potential activity surpassed a threshold voltage value, indicative that seizure activity was present. When the field potential voltage crossed the threshold, a TTL pulse was sent to the Grass S8800 stimulator, which delivered a 0.5 sec train of 500 sec pulses at 333 Hz. The current level was dictated by the stimulation protocol for a given trial. Trains of stimuli were presented as long as the field potential activity remained above the threshold value (i.e., as long as seizure activity was ongoing). The train duration for the seizure-triggered stimulation (0.5 sec) was chosen because it was the shortest duration that we found to be effective for stopping the seizure activity, and we wanted to keep the stimulation as short as possible to reduce the total amount of stimulation given. The voltage threshold was set manually for each experiment. Generally, the seizure activity was three to five times that of the background activity, and the threshold was set high enough to identif y seizure activity only. After the threshold was set for a given experiment, it was not moved. The seizure activity recorded on the field potential traces was directly correlated with behavioral manifestation of the seizures (clonic jerking of the body and forelimbs). When setting the seizure detection threshold, we always verified that the seizure activity identified by the ASD device was directly correlated with this behavioral component of the seizures. E xperimental protocols. The first part of this study was performed to determine whether stimulation of the IO branch of the trigeminal nerve was capable of eliminating P TZ-induced seizure activity in awake rats. To do this, we delivered continuous stimulation to the IO nerve during episodes of P TZ-induced seizure activity via the nerve cuff electrode (Fig. 1) for 1 min stimulus-on periods, separated by 1 min stimulus-off periods. This protocol was performed with both unilateral and bilateral stimulation
119 8162 J. Neurosci., November 1, 2000, 20(21):8160–8168
Fanselow et al. • Seizure Reduction by Trigeminal Nerve Stimulation
of the IO nerve. Stimulus parameters were varied between the stimulus-on periods as described above. In the second part of this study, we assessed the effectiveness of stimulating the IO nerve only when seizure activity was present by using the ASD. For this protocol, the ASD device was turned on for 1 min stimulus-on periods, separated by 1 min stimulus-off periods, as in the first protocol, but stimulation was only provided during the stimulus-on periods when seizure activity was detected by the ASD device. Data anal ysis. We measured seizure activity in the field potential recordings in three ways: seizure frequency, seizure duration, and integrated seizure activity. These parameters were quantified by the use of a custommade analysis program developed using Matlab. The field potential traces were first bandpass filtered at 5–30 Hz. A sliding window (1 sec window with 0.5 sec overlap) was used to quantif y the activity of the absolute values of the field potential traces. Within each window, the amplitude (i.e., voltage) range of the absolute value of the field potential activity in each trace was divided into 10 equal parts, and within each sliding window the number of voltage values falling into each of the 10 divisions of the amplitude range was calculated. Then, a threshold of 50% of the amplitude range was used to identif y seizure activity. If activity within three consecutive windows was above this threshold, the activity was considered to be part of a seizure. From these data, the number of seizures and their durations could be calculated by counting the number of windows during which activity was above the threshold. In addition, a measure we call the “integrated seizure activity” was calculated by summing all of the values for all of the amplitude range intervals for a given on or off period of stimulation. This algorithm was applied in a uniform, blinded manner to all of our data, allowing for objective quantification of the three measures of seizure activity. Statistical anal yses. Using the values for seizure number, seizure duration, and integrated seizure activity, we assessed the efficacy of IO nerve stimulation and ASD by comparing each stimulation-on period with the stimulation-off period directly preceding it. Thus, results are presented as ratios of seizure activity during stimulus-on periods to seizure activity during stimulus-off periods. We used multivariate ANOVAs (M ANOVAs) to assess whether there were statistically significant changes in seizure duration, seizure frequency, or integrated seizure activity between periods of no stimulation and periods of stimulation for each stimulus parameter setting. In addition, repeated measure ANOVAs were used when comparing one measure with another (e.g., number of seizures compared with seizure duration). When significant differences were indicated by M ANOVA or ANOVA analyses, T ukey’s honestly significant difference post hoc tests were used to identif y which effects were significant ( p ⬍ 0.05).
RESULTS Control experiments In control experiments in which PTZ was administered, but no IO nerve stimulation was provided, the average number of seizures per minute was 5.98 ⫾ 0.45, and the average seizure duration was 3.94 ⫾ 0.23 sec. In contrast to studies of VNS in rats (Woodbury and Woodbury, 1990) and dogs (Zabara, 1992), we did not observe any substantial cardiovascular side effects during IO nerve stimulation (Fig. 2). We recorded electrocardiogram (EKG) signals in anesthetized rats while stimulating the IO nerve and did not observe any substantial change in heart rate during stimulation.
Stimulation of the infraorbital nerve reduces seizure activity Stimulation of the infraorbital nerve by the use of the periodic stimulation paradigm substantially reduced PTZ-induced seizure activity compared with that of control periods (Figs. 3, 4, 5). This effect was dependent on both the current and the frequency of the stimulation. There were no significant differences between the cortex and thalamus on any of the measures [Rao R(3,134) ⫽ 0.33; p ⬎ 0.8]. As expected from previous studies, the seizure reduction effect of IO nerve stimulation was greater with increasing current levels (Fig. 3b–d). For these experiments, pulse duration and frequency were held constant at 0.5 msec and 333 Hz, respectively, while current was varied between 3 and 11 mA, in 2 mA increments. At currents of 3 and 5 mA, there were no differences between periods of IO nerve stimulation and periods of no stimulation. However, at 7, 9, and 11 mA, nerve stimulation caused a significant decrease in overall seizure activity (Fig. 3b, 7 mA, 43.2 ⫾ 7.0%; 9 mA, 65.5 ⫾ 4.7%; 11 mA, 77.5 ⫾ 4.3%; p ⬍ 0.001) and in the number of seizures initiated (Fig. 3c, 7 mA, 36.4 ⫾ 5.8%; 9 mA, 50.5 ⫾ 4.6%;
Figure 2. EKG activity is not significantly altered during IO nerve stimulation. a, b, Two examples of EKG activity during IO nerve stimulation (stim; horizontal bars) in an anesthetized rat. Calibration: vertical, 100 V; horizontal, 1 sec. c, The EKG traces and instantaneous heart rate (Inst. rate) over a 15 min period during which stimulation was twice provided continuously for 1 min, as well as five times for shorter bursts. Small changes in the EKG can be seen when stimulation is provided, but they are minor and rapidly stabilize, even during ongoing stimulation. i, ii, The traces that are shown at a faster time scale in a and b, respectively. Calibration: vertical, 100 V for the EKG traces, 200 beats/min for the instantaneous heart rate; horizontal, 10 sec. The stimulus parameters in these traces were 50 Hz, 11 mA, and 0.5 msec pulse duration.
11 mA, 58.7 ⫾ 6%; p ⬍ 0.0001). There was also a significant decrease in the seizure duration at 9 mA (Fig. 3d, 52.5 ⫾ 3.7%; p ⬍ 0.0001). Different stimulus frequencies had different effects on the seizure activity (Fig. 4). For these experiments, pulse duration and current were held constant at 0.5 msec and 9 mA, respectively. Stimulation at high frequencies (100, 125, 200, and 333 Hz) caused a significantly smaller number of seizures than did periods of no stimulation (Fig. 4a; p ⬍ 0.05), as described above. Stimulation frequencies of 50 Hz and lower did not cause any significant changes in the number of seizures initiated (Fig. 4a; p ⫽ 1.0), but seizures did tend
120 Fanselow et al. • Seizure Reduction by Trigeminal Nerve Stimulation
J. Neurosci., November 1, 2000, 20(21):8160–8168 8163
Figure 3. Stimulation of the IO nerve reduces seizure activity in a current-dependent manner. a1–a3, Filtered field potential traces showing seizure activity during three sequential 1 min periods (a1, no stimulus; a2, stimulus on; a3, no stimulus). The stimulus parameters for this figure were 11 mA, 333 Hz, and 0.5 msec pulse. b—d, The amount of seizure activity during 1 min periods of stimulation at different current levels compared with the period of no stimulation directly preceding each stimulus-on period. Values are presented as a percent of the average stimulus-off period measurements. b, Integrated seizure activity. c, Number of seizures. d, Seizure duration. Error bars represent ⫾SEM. A solid line connects stimulation-off values; a dashed line connects stimulation-on values. Stimulation-on values significantly different from stimulation-off values are designated by an asterisk. Thick, black horizontal lines at 100% denote the level of no change in seizure activity. Calibration: vertical, 200 V; horizontal, 10 sec.
current was high enough, stimulating unilaterally was as effective as stimulating bilaterally. However in the middle range of stimulation intensities, bilateral stimulation allowed us to use less current per nerve while still maintaining a high degree of seizure reduction.
Automatic detection of seizure activity and termination of seizures
Figure 4. Effect of varying stimulus frequency using the periodic stimulation paradigm. a, Number of seizures. b, Seizure duration. Labeling conventions are described in Figure 3 (note the change in the scale of the y-axis in b).
to be longer than those during control periods at these frequencies (Fig. 4b; 10 Hz; p ⬍ 0.02).
Bilateral versus unilateral stimulation Bilateral stimulation was significantly more effective at reducing seizures than was unilateral stimulation either contralateral or ipsilateral to the recording site (Fig. 5). This effect was significant for the integrated seizure activity measure (Fig. 5b) at a current level of 7 mA (75.7 ⫾ 5.7%; p ⬍ 0.002), as well as for the number of seizures (Fig. 5c) at 7 and 9 mA (7 mA, 63.7 ⫾ 5.3; 9 mA, 78.1 ⫾ 3.7%; p ⬍ 0.01). It is important to point out that the superior effect of bilateral stimulation was only evident for the middle range of stimulation intensities used in this study. That is, if the current was too low, presumably below the threshold for seizure reduction, there was no advantage in stimulating both nerves, and if the
Use of the ASD device to stimulate the IO nerve only when seizure activity was detected successfully reduced the amount of seizure activity relative to control periods. Figure 6 shows that when the seizure detector identified seizure activity in the field potential traces and triggered the stimulator, the seizure stopped. As in the experiments described above, the degree of seizure reduction was dependent on the current level (Fig. 7). For this set of experiments, we held the pulse duration constant at 0.5 msec, and the frequency at 333 Hz. Current was varied from 3 to 11 mA in 2 mA increments. Figure 7b shows that the integrated seizure activity level was significantly reduced at 9 and 11 mA (9 mA, 55.2 ⫾ 7.2%; p ⬍ 0.03; 11 mA, 56.6 ⫾ 8.0%; p ⬍ 0.01). The number of seizures was significantly decreased at 7 and 9 mA (Fig. 7c, 7 mA, 19.3 ⫾ 5.8%; p ⬍ 0.05; 9 mA, 22.5 ⫾ 6.1%; p ⬍ 0.0001). In addition, the seizure duration was decreased at 7, 9, and 11 mA (Fig. 7d, 7 mA, 40.2 ⫾ 3.3%; 9 mA, 45.2 ⫾ 3.6%; 11 mA, 49.4 ⫾ 4.0%; p ⬍ 0.0001 for all). To compare the efficacy of the ASD device with that of the periodic stimulation paradigm, we calculated the ratio of the percent of seizure reduction to stimulus-on time (Fig. 8). By comparing these ratios between ASD stimulation and periodic stimulation protocols, we observed that, at least in the acute seizure model (PTZ) used in this study, delivering stimulation only when seizure activity was detected was up to 39.8 times more effective at seizure reduction per second of stimulation than was periodic stimulation not related in any way to seizure activity. There was an important difference between the nature of the seizure reduction effect using the ASD device and that observed using the periodic stimulation paradigm described above. With the periodic stimulation paradigm, the number of seizures and the seizure durations were reduced by approximately the same amount at each current level (Fig. 3, compare c, d). However, when the ASD device was used, the seizure durations were reduced significantly more than the number of seizures (Fig. 7, compare c, d; p ⬍ 0.000001). In addition, analysis of the data revealed that in control experi-
121 8164 J. Neurosci., November 1, 2000, 20(21):8160–8168
Fanselow et al. • Seizure Reduction by Trigeminal Nerve Stimulation
Figure 5. Effects of bilateral stimulation versus unilateral stimulation. a1—a3, Filtered field potential traces showing seizure activity during three sequential 1 min periods (a1, no stimulus; a2, bilateral stimulation; a3, no stimulus). The stimulus parameters were 9 mA, 333 Hz, and 0.5 msec pulse duration. b—d, Values presented as ratios of stimulus-on/stimulus-off measurements. b, Integrated seizure activity. c, Number of seizures. d, Seizure duration. A solid line connects responses contralateral to the stimulation site; a line with long dashes connects responses ipsilateral to the stimulation site; a line with short dashes connects responses to bilateral stimulation. Responses to bilateral stimulation that are significantly different from those to ipsilateral and contralateral stimulation are represented by an asterisk. Other labeling conventions are described in Figure 3.
of a stimulus and the next spontaneous seizure (i.e., in the epoch after a stimulus-on period), which was 7.59 ⫾ 1.29 sec. Thus, the average delay between the end of a period of stimulation and the next spontaneously occurring seizure is an average of 24% longer than the interseizure interval during control experiments where no stimulation was present.
DISCUSSION
Figure 6. Seizure-specific stimulation stops synchronous activity. When the field potential amplitude reached a threshold value, the seizure detector (ASD device) triggered IO nerve stimulation. a—c, Traces correspond to the roman numerals i–iii, respectively (see Fig. 7). Note that the stimulation outlasts seizure activity because pulses were provided in 500 msec trains. Also note that the traces indicating seizure detection and stimulation are only indicators and are not indicative of stimulation current or frequency. Calibration: vertical, 200 V; horizontal, 500 msec.
ments where PTZ was administered but no stimulation was provided, the average time between the end of one spontaneously occurring seizure and the beginning of the next was 6.1 sec (calculated from the average number of seizures and the average seizure duration). We also measured the latency between the end
The results of this study demonstrate three substantial advances in the use of cranial nerve stimulation for the treatment of seizures. First, we showed that stimulation of the trigeminal nerve can reduce PTZ-induced seizure activity in rats. This indicates that the seizure reduction effect of cranial nerve stimulation is not limited to stimulation of the vagus nerve but instead may be mediated by a more nonspecific arousal mechanism that can be recruited by stimulation of a number of cranial nerves. Second, we showed that bilateral trigeminal nerve stimulation could have the same seizure reduction effect as unilateral stimulation but required much less current to do so. This finding is therapeutically relevant, because it suggests that multisite stimulation could help maximize the seizure reduction effect of any technique using cranial nerve stimulation, while using the lowest current levels possible. Finally, we showed that in the acute seizure model used in this study (PTZ), automatic, real-time seizure-triggered stimulation reduces seizures more effectively per second of stimulation than does periodic stimulation that is unrelated to seizure onset. This means that the use of a real-time brain–device interface that would automatically detect seizure activity and trigger a nerve stimulator only when such activity was present could provide a high degree of seizure control while potentially reducing the overall amount of stimulation presented to a patient. We propose that these findings may significantly improve the efficacy of cranial nerve stimulation as a therapy for patients with intractable epileptic seizures.
Mechanism of seizure reduction by cranial nerve stimulation The mechanism by which cranial nerve stimulation causes desynchronization of thalamic and cortical activity and reduces seizure activity is unknown. However, one theory is that such stimulation activates the midbrain reticular formation and that this activation results in generalized arousal via the reticular-activating system. In support of this view, Gellhorn (1960) showed that stimulation of the midbrain reticular formation suppresses focal strychnine spikes in cats. In addition, several methods of eliminating seizure-related
122 Fanselow et al. • Seizure Reduction by Trigeminal Nerve Stimulation
J. Neurosci., November 1, 2000, 20(21):8160–8168 8165
Figure 7. Seizure reduction using the ASD device. a1–a3, Filtered field potential traces showing seizure activity during three sequential 1 min periods (a1, no stimulus; a2, stimulus on; a3, no stimulus). The stimulus parameters were 9 mA, 333 Hz, and 0.5 msec pulse duration. Within each segment, the trace labeled seizure detector indicates where the ASD device detected seizure activity; the trace labeled stimulus on indicates where the ASD device sent a TTL pulse to trigger the IO nerve stimulator when it detected such activity. The roman numerals i–iii and arrows indicate parts of the traces that were enlarged to create Figure 6. b, Integrated seizure activity. c, Number of seizures. d, Seizure duration. Labeling conventions are described in Figure 3.
Figure 8. Comparison of the amount of seizure reduction versus the amount of stimulation provided. Stimulation was provided by the use of the periodic stimulation paradigm (dashed line) or the ASD device (solid line). The y-axis represents the ratio of seizure activity reduction to seconds of stimulation in a given stimulus-on period. Asterisks designate the ratios of ASD seizure reduction to seconds of stimulation that were significantly higher than those obtained by the use of the periodic stimulation protocol. a, Integrated seizure activity. b, Number of seizures. c, Seizure duration.
activity by activating multiple sensory modalities have been demonstrated. These include the reduction of absence seizures by acoustic stimuli (Rajna and Lona, 1989) and the reduction of interictal focal activity or absence seizures by motor or mental activity (Jung, 1962; Ricci et al., 1972) or by thermal stimulation (McLachlan, 1993). Because such a wide range of manipulations can reduce seizure-related activity, it is reasonable to suggest that seizure reduction in these cases is caused by a generalized effect on arousal mediated by the brainstem reticular formation. This is supported by the classical work of Moruzzi and Magoun (1949)
demonstrating that stimulation of the midbrain reticular formation causes EEG desynchronization. This hypothesis is consistent with our finding that seizure reduction effects are not specific to the vagus nerve but can instead be achieved by stimulation of multiple cranial nerves that convey information to the reticular formation. One important factor to consider with regard to both the mechanism of seizure reduction by trigeminal nerve stimulation and its applicability to long-term use in humans is the nature of the fiber types that must be activated to cause the seizure reduction effect. Multiple studies of the VNS technique have shown that the level of stimulation, in terms of stimulus frequency and intensity, must be high enough to activate slowly conducting c-fibers (Chase et al., 1967; Woodbury and Woodbury, 1990). The frequency range we found to be therapeutic in the present study was somewhat different from that typically used in animal and human VNS studies. In animal studies the usual therapeutic range was generally 10 –30 Hz (Woodbury and Woodbury, 1990; Zabara, 1992; Takaya et al., 1996), although Lockard et al. (1990) used higher stimulation frequencies (50 –250 Hz) in monkeys. In human studies the range used for stimulation was typically 20 –30 Hz (McLachlan, 1997). This difference between VNS studies and ours may be caused by the difference in the relative numbers of fiber types between the vagus nerve and the infraorbital nerve. In cat, the vagus nerve is composed of 65–90% unmyelinated fibers (Foley and DuBois, 1937; Agostoni et al., 1957), whereas the rat IO nerve contains ⬃33% slowly conducting, unmyelinated fibers (Klein et al., 1988). However, it is not clear what the relationship is between fiber composition and the stimulus frequency/intensity required for seizure reduction, so interpreting these differences is difficult. A complicating factor is that although it has been shown that for seizure reduction the level of stimulation must be sufficient to activate c-fibers, it has not been demonstrated that these fibers are necessary for the seizure reduction effect. Finally, it is important to note that according to studies by Torebjork and colleagues (Torebjork, 1974; Torebjork and Hallin, 1974), c-fibers do not conduct if electrical stimuli are presented at frequencies above ⬃10 Hz. This means that although high stimulation frequencies were required for the seizure reduction effect observed here, it is likely that at such frequencies the c-fibers were not activated or were activated to a lesser degree than other fibers in the nerve. Furthermore, it is possible that cells in the trigeminal nucleus were not able to follow with sustained responses at the high rates of stimulation we provided. For example, Andresen and Yang (1995)
123 8166 J. Neurosci., November 1, 2000, 20(21):8160–8168
demonstrated using a slice preparation of the rat medulla that neurons in the nucleus of the solitary tract (NTS) responded with lower EPSP amplitudes as the frequency of solitary tract stimulation was increased. These results may also be relevant to trigeminal nerve stimulation. If this is the case, it is unclear why our results show that higher frequency stimulation is more effective for seizure elimination than are lower stimulation rates. However, the study by Andresen and Yang (1995) also demonstrated that bursts of highfrequency stimulation resulted in less EPSP attenuation than did continuous high-frequency stimulation, suggesting that an optimal stimulation protocol could involve short bursts of high-frequency stimulation rather than continuous trains. The delay between the onset of seizure-triggered stimulation and the end of the seizure activity might shed some light on the mechanism by which trigeminal stimulation reduces seizure activity. The average time between the onset of the seizure-triggered stimulus and the end of the seizure was 529.9 ⫾ 40.3 msec (note that Fig. 6 demonstrates some of the shortest delays). It is interesting that this value is similar to the minimum effective stimulus train duration (500 msec) that we determined empirically. However, it should be noted that there was a wide range of delays, and this may be caused by at least two factors. First, the phase of the synchronous oscillations during which the stimuli occur may have a profound impact on the efficacy of the stimulation. Second, it is possible that the ability to abort a seizure varies depending on the amount of time the seizure has been ongoing before a sufficient stimulus arrives. Thus, differences in the phase of the oscillatory seizure activity at which the stimuli occur or the threshold used for seizure detection may affect the efficacy of the stimulation. These mechanisms could explain the variation in the amount of time required to abort a seizure. Another important mechanism-related issue is whether the trigeminal stimulation was merely able to stop seizure activity during the stimulation itself or whether it also had an effect on the number of seizures initiated. In control files where PTZ was administered but no stimulation was provided, the average time between the end of one spontaneously occurring seizure and the beginning of the next was 6.1 sec (calculated from the average number of seizures and the average seizure duration, as reported in Results). We also measured the latency between the end of a period of stimulation and the next spontaneous seizure (i.e., in the epoch after a stimulus-on period), which was 7.59 ⫾ 1.29 sec. Thus, the average delay between the end of a period of stimulation and the first spontaneous seizure after the stimulus ends is actually, on average, 24% longer than the interseizure interval during control files with no stimulation present. These results are supported by results from other laboratories (Zabara, 1992; Takaya et al., 1996) showing that the seizure reduction effect of vagus nerve stimulation can outlast the stimulus duration.
Bilateral versus unilateral IO nerve stimulation The fact that bilateral stimulation can be more effective than unilateral stimulation in the middle of the therapeutic-current range has implications for how such stimulation could be used to most effectively reduce seizure activity. Specifically, because bilateral stimulation at 7 mA was just as effective as unilateral stimulation at 11 mA (Fig. 5), the use of bilateral nerve cuff electrodes would reduce the amount of current delivered to each nerve, while still maintaining the same seizure reduction effect as higher stimulation current at a single site. This would be beneficial because it would reduce the potential for damage to nerve fibers at the stimulation site (Agnew et al., 1989; Agnew and McCreery, 1990), and it would reduce the intensity of any possible side effects associated with the stimulation. Bilateral stimulation is a further improvement over VNS, because the vagus nerve cannot be safely stimulated bilaterally without substantial risk of cardiovascular side effects (Schachter and Saper, 1998). It is important to point out that our finding that bilateral stimulation of the IO nerve was more effective than unilateral stimulation is in contrast to two previous studies reporting that bilateral
Fanselow et al. • Seizure Reduction by Trigeminal Nerve Stimulation
stimulation of the vagus nerve was no more effective than unilateral stimulation (Chase et al., 1966; Zabara, 1992). This discrepancy is likely either caused by differences in fiber composition between the vagus nerve and the IO nerve or caused by the fact that the stimulus parameters used in those studies were beyond those for which bilateral stimulation is superior to unilateral stimulation. Details about the stimulus parameters used for assessing the efficacy of bilateral stimulation in those two studies were not provided. Another important point to consider is that we have tested the effect of bilateral stimulation with the PTZ seizure model, which involves generalized, tonic-clonic seizures (Fisher, 1989). Further testing with focal seizure models such as localized application of alumina gel (Lockard et al., 1990) or penicillin (McLachlan, 1993) to the cortex will be necessary to determine whether there is an advantage to bilateral stimulation in eliminating these types of seizures as well. Evidence to support an advantage in using bilateral stimulation to treat focal seizures is that, in our study, unilateral stimulation eliminated seizure activity in both hemispheres at the same time, suggesting that the effect of the stimulation is not restricted to one hemisphere. Such results have also been found for VNS in cats (Chase et al., 1966), dogs (Zabara, 1992), and humans (Henry et al., 1998, 1999). These results suggest that because each nerve being stimulated can reduce seizures bilaterally, the effect of stimulating both nerves could be additive within a given hemisphere.
A brain–device interface for automatic, real-time detection and reduction of seizure activity This study showed that triggering trigeminal nerve stimulation only when a seizure began is a much more effective method for reducing seizure activity than is providing stimulation on a fixed duty cycle, as has been used in past studies. This finding is an important advancement in the use of cranial nerve stimulation therapies in epilepsy for several reasons. First, stimulating only when seizure activity occurs would, for many patients, reduce the overall amount of stimulation required for maintaining seizure control. Thus, the amount of potentially unnecessary stimulation usually occurring between seizure periods would be reduced, decreasing the possibility of damage to the nerve (Agnew et al., 1989; Agnew and McCreery, 1990; Ramsay et al., 1994). It is important to note, however, that several researchers have demonstrated a prophylactic effect of vagus nerve stimulation such that after stimulation, seizures are less likely for a period of time related to the duration of the preceding stimulation (Zabara, 1992; Takaya et al., 1996). This implies that perhaps the best overall treatment stimulation protocol might involve the use of seizuretriggered stimulation combined with intermittent prophylactic nonseizure-triggered stimulation. The second advantage of this technique is that it would reduce the side effects experienced by patients when the stimulus is on. For example, patients undergoing VNS treatment report hoarseness, coughing, and throat pain as the most common side effects of the stimulation (Ramsay et al., 1994; McLachlan, 1997; Schachter and Saper, 1998). These side effects are generally only experienced when the stimulation is on. However, if stimulation were only presented in response to the detection of seizure activity (or occasionally prophylactically, as described above), these side effects would be experienced as infrequently as possible. In the future, the type of real-time, automatic seizure detector described here could be implemented in humans by building on and combining a number of existing technologies. First, seizure detection could be performed by a computer microchip programmed with a seizure detection algorithm and carried by the patient, similar to the Holter monitors used for continuous EKG monitoring. Input would be delivered to this microchip from multiple scalp EEG electrodes that would be able to pick up and amplify extracranial EEG signals. Finally, when the microchip detected seizure activity in the EEG signals, it would trigger an implanted stimulator similar to those used in the VNS technique (Terry et al., 1990), which would stimulate one or more trigeminal nerve cuff elec-
124 Fanselow et al. • Seizure Reduction by Trigeminal Nerve Stimulation
trodes. This device would function in a manner analogous to cardiac pacemakers, commonly used to treat heart arrhythmia, and would require a minimum of invasive procedures. In essence, this device would constitute a “brain pacemaker” for seizure monitoring and control. The application of nonlinear computational methods for detection of seizure activity (Gabor et al., 1996; Webber et al., 1996) could be extremely beneficial if incorporated into the seizure detector described here. Such seizure detection algorithms would allow for more accurate identification of seizure activity than the rather simple amplitude-based algorithm we used in this study. Another substantial advance could be in the implementation of seizure prediction algorithms that can identify seizures seconds or minutes before the behavioral onset (Martinerie et al., 1998; Le Van Quyen et al., 1999). There is evidence that the sooner stimulation is provided after a seizure begins, the more effectively the seizure can be stopped (Uthman et al., 1993); also stimulation is more likely to prevent seizure activity if it is presented before rather than after a seizure has begun (Woodbury and Woodbury, 1990). Therefore, it is possible that providing stimulation before the clinically defined onset of a seizure may prevent seizures before they begin or become behaviorally relevant to the patient. Such a technique could dramatically improve the efficacy of the cranial nerve stimulation therapy.
Application to human patients Further studies in other animals and with other seizure models will be needed to determine whether it is appropriate to apply the techniques described here to human patients and what the best methods for doing so would be. The results presented here apply to seizures caused by systemic administration of PTZ, which is a model of acute, generalized, tonic-clonic seizures. It will be important to determine whether the results also apply to chronic seizure models, as well as to other seizure types, such as focal seizures (e.g., temporal lobe seizures) and absence seizures, if the described techniques are to be applicable to humans. Because the cellular mechanisms involved in different seizure types may not necessarily be similar, it is vital to ask whether the seizure reduction effects from trigeminal nerve stimulation would apply to different seizure types. If the effects of trigeminal nerve stimulation are spatially restricted or only affect certain types of cellular excitation or inhibition, then this technique may be of limited use in treating multiple seizure types. If, however, as proposed in the current paper, the mechanism by which trigeminal nerve stimulation reduces seizure activity is, indeed, a generalized, widespread effect on cortical arousal level, perhaps mediated by the brainstem reticular-activating system, it is possible that this technique would be useful in treating a wide range of seizures. In support of this view, the VNS technique has proven to be effective in multiple seizure models including intraperitoneal injection of PTZ (Zabara, 1985, 1992; Woodbury and Woodbury, 1990), intraperitoneal injection of 3-mercaptopropionate (Woodbury and Woodbury, 1990), maximal electroshock (Woodbury and Woodbury, 1990), topical application of penicillin to the cortex (McLachlan, 1993), and chronic local application of alumina gel (Lockard et al., 1990). Another issue that will need to be addressed before this technique is applied to humans is that because the trigeminal nerve is involved in transmitting both somatosensory and pain information from the head, it is vital that the level of stimulation be below that that might cause discomfort such as facial pain or headaches. It is not known what stimulus parameters would be required to achieve seizure reduction without resulting in painful sensations such as these. However, such side effects could be substantially reduced by using the lowest effective stimulus parameters, which would be aided by the use of bilateral stimulation. Finally, it would also be possible to develop an effective therapy by combining the VNS technique, which is currently in use in
J. Neurosci., November 1, 2000, 20(21):8160–8168 8167
human patients, with the automatic seizure detection technique described in this paper.
Conclusions The results described in this study could serve to substantially increase the efficacy of cranial nerve stimulation as a technique for reducing or eliminating seizures in patients who suffer from intractable epilepsy. Further development and testing of trigeminal nerve stimulation for patients with epilepsy is justified on the basis of the results presented here. In addition, our findings suggest that in the future, it will be feasible to develop a completely implantable and relatively noninvasive brain–device interface capable of automatically detecting seizure activity and triggering stimulation of cranial nerves to safely and efficiently reduce seizure activity.
REFERENCES Agnew WF, McCreery DB (1990) Considerations for safety with chronically implanted nerve electrodes. Epilepsia 31[Suppl 2]:S27–S32. Agnew WF, McCreery DB, Yuen TG, Bullara LA (1989) Histologic and physiologic evaluation of electrically stimulated peripheral nerve: considerations for the selection of parameters. Ann Biomed Eng 17:39 – 60. Agostoni E, Chinnock JE, De Burgh Daly M, Murray JG (1957) Functional and histological studies of the vagus nerve and its branches to the heart, lungs and abdominal viscera in the cat. J Physiol (Lond) 135:182–205. Andresen MC, Yang M (1995) Dynamics of sensory afferent synaptic transmission in aortic baroreceptor regions of nucleus tractus solitarius. J Neurophysiol 74:1518 –1528. Ben-Menachem E, Manon-Espaillat R, Ristanovic R, Wilder BJ, Stefan H, Mirza W, Tarver WB, Wernicke JF (1994) Vagus nerve stimulation for treatment of partial seizures: 1. A controlled study of effect on seizures. First International Vagus Nerve Stimulation Study Group. Epilepsia 35:616 – 626. Chase MH, Nakamura Y (1968) EEG response to afferent abdominal vagal stimulation. Electroencephalogr Clin Neurophysiol 24:396. Chase MH, Sterman MB, Clemente CD (1966) Cortical and subcortical patterns of response to afferent vagal stimulation. Exp Neurol 16:36 – 49. Chase MH, Nakamura Y, Clemente CD, Sterman MB (1967) Afferent vagal stimulation: neurographic correlates of induced EEG synchronization and desynchronization. Brain Res 5:236 –249. Fanselow EE, Nicolelis MA (1999) Behavioral modulation of tactile responses in the rat somatosensory system. J Neurosci 19:7603–7616. Fisher RS (1989) Animal models of the epilepsies. Brain Res Rev 14:245–278. Foley JO, DuBois F (1937) Quantitative studies of the vagus nerve in the cat. J Comp Neurol 67:49 – 67. Gabor AJ, Leach RR, Dowla FU (1996) Automated seizure detection using a self-organizing neural network. Electroencephalogr Clin Neurophysiol 99:257–266. Gellhorn E (1960) Further experiments on the influence of afferent stimulation on cortical strychnine discharges. Electroencephalogr Clin Neurophysiol 12:613– 619. Henry TR, Bakay RA, Votaw JR, Pennel PB, Epstein CM, Faber TL, Grafton ST, Hoffman JM (1998) Brain blood flow alterations induced by therapeutic vagus nerve stimulation in partial epilepsy: I. Acute effects at high and low levels of stimulation. Epilepsia 39:983–990. Henry TR, Votaw JR, Pennel PB, Epstein CM, Bakay RA, Faber TL, Grafton ST, Hoffman JM (1999) Acute blood flow changes and efficacy of vagus nerve stimulation in partial epilepsy. Neurology 52:1166 –1173. Jung R (1962) Blocking of petit-mal attacks by sensory arousal and inhibition of attacks by an active change in attention during the epileptic aura. Epilepsia 3:435– 437. Klein BG, Renehan WE, Jacquin MF, Rhoades RW (1988) Anatomical consequences of neonatal infraorbital nerve transection upon the trigeminal ganglion and vibrissa follicle nerves in the adult rat. J Comp Neurol 268:469 – 488. Le Van Quyen M, Martinerie J, Baulac M, Varela F (1999) Anticipating epileptic seizures in real time by a non-linear analysis of similarity between EEG recordings. NeuroReport 10:2149 –2155. Lockard JS, Congdon WC, DuCharme LL (1990) Feasibility and safety of vagal stimulation in monkey model. Epilepsia 31[Suppl 2]:S20 —S26. Magnes J, Moruzzi G, Pompeiano O (1961) Synchronization of the EEG produced by low-frequency electrical stimulation of the region of the solitary tract. Arch Ital Biol 99:33– 67. Martinerie J, Adam C, Le Van Quyen M, Baulac M, Clemenceau S, Renault B, Varela FJ (1998) Epileptic seizures can be anticipated by non-linear analysis [see comments]. Nat Med 4:1173–1176. McLachlan RS (1993) Suppression of interictal spikes and seizures by stimulation of the vagus nerve. Epilepsia 34:918 –923. McLachlan RS (1997) Vagus nerve stimulation for intractable epilepsy: a review. J Clin Neurophysiol 14:358 –368. McNamara JO (1999) Emerging insights into the genesis of epilepsy. Nature 399[Suppl 6738]:A15–A22.
125 8168 J. Neurosci., November 1, 2000, 20(21):8160–8168
Moruzzi G, Magoun HW (1949) Brainstem reticular formation and activation of the EEG. Electroencephalogr Clin Neurophysiol 1:455– 473. Nicolelis MA, Ghazanfar AA, Faggin BM, Votaw S, Oliveira LM (1997) Reconstructing the engram: simultaneous, multisite, many single neuron recordings. Neuron 18:529 –537. Paxinos G, Watson C (1986) The rat brain, Ed 2. New York: Academic, Harcourt, Brace, and Jovanovich. Penry JK, Dean JC (1990) Prevention of intractable partial seizures by intermittent vagal stimulation in humans: preliminary results. Epilepsia 31[Suppl 2]:S40 —S43. Rajna P, Lona C (1989) Sensory stimulation for inhibition of epileptic seizures. Epilepsia 30:168 –174. Ramsay RE, Uthman BM, Augustinsson LE, Upton AR, Naritoku D, Willis J, Treig T, Barolat G, Wernicke JF (1994) Vagus nerve stimulation for treatment of partial seizures: 2. Safety, side effects, and tolerability. First International Vagus Nerve Stimulation Study Group. Epilepsia 35:627– 636. Ricci G, Berti G, Cherubini E (1972) Changes in interictal focal activity and spike-wave paroxysms during motor and mental activity. Epilepsia 13:785–794. Schachter SC, Saper CB (1998) Vagus nerve stimulation. Epilepsia 39: 677– 686. Takaya M, Terry WJ, Naritoku DK (1996) Vagus nerve stimulation induces a sustained anticonvulsant effect. Epilepsia 37:1111–1116. Terry R, Tarver WB, Zabara J (1990) An implantable neurocybernetic prosthesis system. Epilepsia 31[Suppl 2]:S33—S37. Torebjork HE (1974) Afferent C units responding to mechanical, thermal
Fanselow et al. • Seizure Reduction by Trigeminal Nerve Stimulation
and chemical stimuli in human non-glabrous skin. Acta Physiol Scand 92:374 –390. Torebjork HE, Hallin RG (1974) Responses in human A and C fibres to repeated electrical intradermal stimulation. J Neurol Neurosurg Psychiatry 37:653– 664. Uthman BM, Wilder BJ, Hammond EJ, Reid SA (1990) Efficacy and safety of vagus nerve stimulation in patients with complex partial seizures. Epilepsia 31[Suppl 2]:S44 –S50. Uthman BM, Wilder BJ, Penry JK, Dean C, Ramsay RE, Reid SA, Hammond EJ, Tarver WB, Wernicke JF (1993) Treatment of epilepsy by stimulation of the vagus nerve. Neurology 43:1338 –1345. Vagus Nerve Stimulation Study Group (1995) A randomized controlled trial of chronic vagus nerve stimulation for treatment of medically intractable seizures. Neurology 45:224 –230. Webber WR, Lesser RP, Richardson RT, Wilson K (1996) An approach to seizure detection using an artificial neural network (ANN). Electroencephalogr Clin Neurophysiol 98:250 –272. Woodbury DM, Woodbury JW (1990) Effects of vagal stimulation on experimentally induced seizures in rats. Epilepsia 31[Suppl 2]:S7–S19. Zabara J (1985) Time course of seizure control to brief, repetitive stimuli. Epilepsia 26:518. Zabara J (1992) Inhibition of experimental seizures in canines by repetitive vagal stimulation. Epilepsia 33:1005–1012. Zanchetti A, Wang SC, Moruzzi G (1952) The effect of vagal afferent stimulation on the EEG pattern of the cat. Electroencephalogr Clin Neurophysiol 4:357–361.
126
RESEARCH ARTICLES
References and Notes 1. P. Kim, C. M. Lieber, Science 286, 2148 (1999). 2. T. Rueckes et al., Science 289, 94 (2000).
3. V. V. Deshpande et al., Nano Lett. 6, 1092 (2006). 4. R. H. Baughman et al., Science 284, 1340 (1999). 5. U. Vohrer, I. Kolaric, M. H. Haque, S. Roth, U. DetlaffWeglikowska, Carbon 42, 1159 (2004). 6. S. Gupta, M. Hughes, A. H. Windle, J. Robertson, J. Appl. Phys. 95, 2038 (2004). 7. V. H. Ebron et al., Science 311, 1580 (2006). 8. G. M. Spinks et al., Adv. Mater. 14, 1728 (2002). 9. S. V. Ahir, E. M. Terentjev, Nat. Mater. 4, 491 (2005). 10. H. Koerner, G. Price, N. A. Pearce, M. Alexander, R. A. Vaia, Nat. Mater. 3, 115 (2004). 11. P. Miaudet et al., Science 318, 1294 (2007). 12. S. Courty, J. Mine, A. R. Tajbakhsh, E. M. Terentjev, Europhys. Lett. 64, 654 (2003). 13. M. Zhang et al., Science 309, 1215 (2005). 14. See supporting material on Science Online. 15. S. Sapmaz, Ya. M. Blanter, L. Gurevich, H. S. J. van der Zant, Phys. Rev. B 67, 235414 (2003). 16. Specific strength (strength normalized to density), and corresponding specific Young’s modulus and work capacity, are used because of their fundamental and practical importance, as well as ease of reliable determination from force and weight-per-length measurements. 17. J. D. W. Madden et al., IEEE J. Oceanic Eng. 29, 706 (2004). 18. R. Pelrine, R. Kornbluh, Q. Pei, J. Joseph, Science 287, 836 (2000).
Spinal Cord Stimulation Restores Locomotion in Animal Models of Parkinson’s Disease Romulo Fuentes,1*† Per Petersson,1,2* William B. Siesser,3 Marc G. Caron,1,3 Miguel A. L. Nicolelis1,4,5,6,7,8 Dopamine replacement therapy is useful for treating motor symptoms in the early phase of Parkinson’s disease, but it is less effective in the long term. Electrical deep-brain stimulation is a valuable complement to pharmacological treatment but involves a highly invasive surgical procedure. We found that epidural electrical stimulation of the dorsal columns in the spinal cord restores locomotion in both acute pharmacologically induced dopamine-depleted mice and in chronic 6-hydroxydopamine–lesioned rats. The functional recovery was paralleled by a disruption of aberrant low-frequency synchronous corticostriatal oscillations, leading to the emergence of neuronal activity patterns that resemble the state normally preceding spontaneous initiation of locomotion. We propose that dorsal column stimulation might become an efficient and less invasive alternative for treatment of Parkinson’s disease in the future. atients suffering from Parkinson’s disease (PD) experience chronic and progressive motor impairment (1). The main cause of PD is basal ganglia dysfunction, resulting from
P 1
Department of Neurobiology, Duke University Medical Center, Durham, NC 27710, USA. 2Department of Experimental Medical Science, Neuronano Research Center, Lund University, BMC F10, 221 84 Lund, Sweden. 3Department of Cell Biology, Duke University, Durham, NC 27710, USA. 4Center for Neuroengineering, Duke University, Durham, NC 27710, USA. 5Department of Biomedical Engineering, Duke University, Durham, NC 27710, USA. 6Department of Psychology and Neuroscience, Duke University, Durham, NC 27710, USA. 7 Edmond and Lily Safra International Institute of Neuroscience of Natal (ELS-IINN), Natal RN 59066060, Brazil. 8Ecole Polytechnique Fédérale de Lausanne, Lausanne 1015, Switzerland. *These authors contributed equally to this work. †To whom correspondence should be addressed. E-mail: fuentes@neuro.duke.edu
1578
degeneration of neurons in the dopaminergic nigrostriatal pathway (2). Dopamine replacement therapy, through administration of the dopamine precursor 3,4-dihydroxy-L-phenylalanine (L-dopa), effectively ameliorates symptoms associated with PD and remains the treatment of choice to date (3). Unfortunately, L-dopa pharmacotherapy has proven less efficient in the long term and is associated with several complications (4). Additional therapeutic strategies, employed in conjunction with pharmacological treatment, have thus attracted considerable attention. In particular, improved techniques for electrical stimulation of the basal ganglia—referred to as deep-brain stimulation (DBS)—are effective for treatment of motor symptoms in PD (5). Furthermore, DBS permits a reduction of L-dopa dosage in PD patients (6). However, a disadvantage of DBS is the require-
20 MARCH 2009
VOL 323
SCIENCE
19. B. I. Yakobson, L. S. Couchman, J. Nanopart. Res. 8, 105 (2006). 20. K. E. Evans, A. Alderson, Adv. Mater. 12, 617 (2000). 21. R. H. Baughman, Nature 425, 667 (2003). 22. R. H. Baughman, S. Stafström, C. Cui, S. O. Dantas, Science 279, 1522 (1998). 23. We thank T. Mirfakhrai, J. D. W. Madden, and V. M. Agranovich for their contributions. Supported by Air Force Office of Scientific Research grant FA9550-05-C0088, NSF grant DMI-0609115, Office of Naval Research MURI grant N00014-08-1-0654, Robert A. Welch Foundation Grant AT-0029, Honda Corporation, Lintec Corporation, and the Brazilian agency CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico). A provisional patent for nanofiber actuators and strain amplifiers has been filed by the authors (Patent Office Provisional Filing No. 61089275).
Supporting Online Materials www.sciencemag.org/cgi/content/full/323/5921/1575/DC1 Materials and Methods SOM Text Figs. S1 to S11 References Movie S1 10 November 2008; accepted 30 January 2009 10.1126/science.1168312
ment of a highly invasive surgical procedure, as well as the crucial dependence on accurate targeting of very small brain structures (7). Hence, it is desirable to identify a less invasive method to electrically stimulate neuronal circuits to obtain beneficial effects similar to those of DBS. Some clues for new PD therapies may come from epilepsy studies. In both animal models and in epilepsy patients, stimulation of peripheral nerve afferents is effective in desynchronizing aberrant low-frequency neural oscillatory activity, thereby reducing the frequency and duration of seizure episodes (8–10). Aberrant low-frequency neural oscillations are well documented in patients (11, 12) and in animal models of PD (13). These findings led us to hypothesize that stimulation of afferent somatic pathways could alleviate motor symptoms of PD by disrupting aberrant lowfrequency oscillations. Dopamine, akinesia, and synchrony. The first set of experiments was carried out using an inducible mouse model of PD, first in wild-type animals and then in dopamine-transporter knockout (DAT-KO) mice (14). Through pharmacological inhibition of dopamine synthesis, we induced acute dopamine depletion in both types of animals (2, 13, 14). In patients, the cardinal symptoms of idiopathic PD have been reported to be clinically apparent after degeneration of 60 to 70% of the dopaminergic neurons of the substantia nigra pars compacta, which results in a 30 to 50% reduction of striatal dopamine levels (15, 16). By means of two intraperitoneal injections (250 mg/kg) of the tyrosine hydroxylase inhibitor alpha-methylpara-tyrosine (AMPT) during a 6-hour period (15, 16), we achieved acute pharmacological dopamine depletion slightly below the levels observed in PD patients in wild-type C57/BL6J mice (69% reduction of striatal dopamine levels; mean T SD = 4.5 T 2.0 ng dopamine per mg
www.sciencemag.org
Downloaded from www.sciencemag.org on March 19, 2009
and reversal of this negative hydrostatic pressure to a positive value would result in a corresponding pressure-induced expansion in the length direction for positive hydrostatic pressure. Application possibilities. In addition to extending the capabilities of artificial muscles to giant strokes and strain rates at extreme temperatures, the present actuator mechanism provides other application possibilities that relate to the structural change of the nanotube sheets during large-stroke actuation. The nanotubes diffract light perpendicular to the alignment direction, which can be dynamically modulated at over kilohertz frequencies for optical applications (14) (movie S1). The ability to tune the density of nanotube sheets and then freeze this actuation is being used for optimizing nanotube electrodes for organic light-emitting displays, solar cells, charge stripping from ion beams, and cold electron field emission.
127
tissue in depleted animals compared with 14.4 T 3.3 in controls; P < 0.005 Mann-Whitney, n = 6/6) (fig. S1). Equivalent symptoms of main clinical motor manifestations in PD patients were found in AMPT-injected mice (figs. S2 and S3). In par-
ticular, locomotive activity was significantly reduced [average locomotion scores in nondepleted and depleted animals were (mean T SEM) 3.7 T 0.1 and 0.4 T 0.02 mm/s, n = 11 and 14, respectively], and a preferential reduction of faster move-
Fig. 1. Acute inhibition of dopamine synthesis produces a Parkinsonian state. (A) Examples of LFP spectrograms and firing-rate plots recorded in MI during two 5-min periods before and after dopamine depletion. Top row: Locomotion during recording periods. Second row: LFP power. Third row: LFP power standardized to the nondepleted 5-min period. Fourth row: Close-up of low-frequency range shown in third row. Note the increased power in low frequencies in the depleted state (black arrows) and the relative normalization of spectral power upon locomotion (red arrow). Bottom row: Average firing per second for 6 MI units. (B) Set-up for electrical stimulation of dorsal columns. The stimulation electrode (red) is implanted above the spinal cord, and connection wires are passed subcutaneously to a connector attached to the skull. Two stimulus-isolator units provide biphasic constantcurrent pulses at desired frequency and intensity. (C) Schematic dorsal (left) and sagittal (right) views of the implanted electrode. r, rostral; c, caudal.
Control
Locomotion (cm/500 ms)
1
−8 −10 −12 −14
50 40 30 20 10
2 0 −2
15 2
10
0
5
Neurons n=6
30
0
60 s
60 s
B
r stimulation electrode spinal cord
r
c
50
40 20
Air
Stimulation type
Air+100 n.V
30 20
5
40
Slow
Medium
Fast
30
61-120
121-180
20
0
10
Locomotion speed (mm/s)
-5 8 4 0 2 0 −2
SCIENCE
2 0
Standardized firing rate (s.d.)
www.sciencemag.org
-14
50
21-60
Fig. 2. DCS restores locomotion and desynchronizes corticostriatal activity. (A) Relative change in amount of locomotion in depleted and nondepleted mice. DCS frequencies specified on x axis. n.V, trigeminal nerve stimulation. Mean and SD shown; means for all conditions before and after depletion are significantly different, a = 0.005. (B) DCS preferentially increases the fraction of faster movement components in dopamine-depleted animals but not in controls. (C) Average spectrograms of striatal LFPs and firing rates recorded around 300-Hz stimulation events (yellow bar). Top row: LFP power (n = 21 events; black trace denotes spectral index). Second row: LFP power standardized to first 240 s. Rows 4 and 5, respectively: Firing rate for 98 striatal and 96 cortical units standardized to firing rates during first 240 s and ordered by responsiveness after DCS (19); n = 36 events. Neurons exhibiting significant changes during the 30-s period after stimulation (black line) are indicated with red and blue rectangles (excitatory and inhibitory responses). Middle row: Average locomotion; n = 36 events.
-12
10
Locomotion (mm/s)
Train
-10
Striatal neurons n = 98
300
-8
40
60
MI neurons n = 96
100
Depleted
C
Depleted Non-depleted
0 10
c
spinal cord thoracic vertabrae 1 and 2
Frequency (Hz)
Relative change
Relative change
-10
stimulation electrode
Standardized power (s.d.)
0
connection wires spinal processes
Log (power)
10
C
stimulus-isolator units
30 20
Firing rate (Hz)
−2
Standardized spectral power (s.d.)
Frequency (Hz)
50 40 30 20 10
B
Depleted Non-depleted
Depleted
2
Log (power)
A
A
ments indicated bradykinesia in the depleted state (fig. S2) (17, 18). Neuronal activity patterns of dorsolateral striatum and primary motor cortex (MI) were also significantly altered. Differences were found both on the population level, through inspection of local field potentials (LFP) and in the firing patterns of single cortical and striatal neurons (13). Figure 1A shows an example of LFP spectrograms recorded in MI during two 5-min periods before and after dopamine depletion (second row, left and right, respectively). Spectral analysis revealed particularly powerful oscillations ~1.5 to 4 Hz and in the lower beta range (10 to 15 Hz), whereas the power of oscillations >25 Hz was decreased in relation to baseline conditions (standardized spectrograms, Fig. 1A, third and fourth row, and fig. S6). Important differences in single- and multi-unit activity were also found. The firing rates of a majority of 52 striatal and cortical neurons, which were positively identified after a 6-hour depletion period, showed significant differences (70.0% in motor cortex and 75.0% in striatum, a = 0.001) when we compared the more active nondepleted state and the immobile depleted condition (see activity raster plots shown for a few units in Fig. 1A, bottom row). During dopamine depletion, a higher proportion of neurons tended to discharge phase-locked to LFP oscillation, in effect resulting in increased synchronicity [52.7% (64/129) in depleted versus 37.0% (44/127) in nondepleted state; a = 0.001] (see fig. S6 for details). DCS alleviates akinesia and synchrony. The effect of dorsal column stimulation (DCS) was next evaluated in mice before and after acute pharmacological dopamine depletion. DCS was achieved by chronic implantation of custom-
Downloaded from www.sciencemag.org on March 19, 2009
RESEARCH ARTICLES
−2
-200 -150 -100 -50
VOL 323
0 50 Time (s)
20 MARCH 2009
100
150
200
1579
128
RESEARCH ARTICLES
Frequency (Hz) Frequency (Hz)
A
B
Control
10
−12
50 40 30 20 10
5 0 −5
2 0 −2
0.6 0.4 0.2 0 −12 −8 −4
0
4
8 12
−12 −8 −4
Time (s)
0
4
8 12
Time (s)
B 6-OHDA Rats 4
5
6
7
8
9
40 10
8
6
4
Locomotion (mm/s)
3
10
Sham non-DCS 6-OHDA non-DCS Sham DCS 6-OHDA DCS
20
0
2
−200
−100
0
100
200
Time (s) 0
60 120 180 240 300 360 420 480 540 600
Time (min)
C 8
6-OHDA
Relative change (%)
0
0 −2
n = 35
MI neurons n = 34 Locomotion (mm/50 ms)
D
2
n = 56
Striatal neurons n = 74
C Standardized firing rate (s.d.)
Cumulative distance (m)
−10
20
L-DOPA injections (mg/kg) 2
−8
40 30
A DAT-KO Mice 1
Depleted
50
20 MARCH 2009
Locomotion (mm/s)
300 Sham Fig. 4. DCS restores locomotion in severely 6 200 dopamine-depleted mice and in chronically lesioned rats. (A) The cumulative amount of locomotion scored 4 100 in animals receiving DCS in combination with suc2 0 cessive L-dopa injections (black) was significantly 0 -100 Slow Medium Fast higher at all time points than that observed for 21-166 167-333 334-500 the group only receiving L-dopa (gray). (B) DCS Sham 6-OHDA Locomotion speed (mm/s) (yellow shaded area) induced a prominent increase in locomotion in 6-OHDA–lesioned rats (shaded area around trace is SEM) compared with the preceding non-DCS sessions. In the sham group, in contrast, DCS caused a moderate response comparable to non-DCS sessions (mean T SEM, n = 64 stimulation and 64 control sessions for both sham-treated and lesioned rats). (C) (Left) DCS specifically increases locomotion in 6-OHDA–lesioned rats (mean and SEM shown; all means are significantly different from the others, P < 0.001, Kruskal-Wallis and Dunn’s multiple comparison test; flashes indicate DCS sessions). (Right) A preferential relative increase of faster movement components locomotion was found in the 6-OHDA–lesioned group reflecting alleviation of bradykinetic symptoms. Relative changes in amount of locomotion in three speed intervals (DCS/non-DCS sessions) are shown.
VOL 323
SCIENCE
www.sciencemag.org
Downloaded from www.sciencemag.org on March 19, 2009
Fig. 3. Activity patterns during spontaneous locomotion. (A) Average spectrogram of striatal LFP aligned to the onset of spontaneous locomotion in control (n = 115 events) and dopamine-depleted condition (n = 51 events). The gradual shift from lower to higher frequencies indicated by the average spectral index (black trace) starts before locomotion onset (dashed white line). (B) Standardization of spectrogram relative to directly preceding nonlocomotion periods (collected between 20 and 10 s before locomotion onset from 112 stationary 10-s periods). (C) Firing rate (binned at 0.5 s) of striatal and MI units around the onset of spontaneous locomotion. Significant changes in firing rate (as compared to stationary period) are indicated with magenta (excitatory response) and blue (inhibitory response) crosses. (D) Average locomotion during recorded events.
in oscillatory LFP activity during spontaneous locomotion events in nondepleted (115 events in 10 animals) and depleted (51 events in 5 animals) mice (Fig. 3, A and B). In both states, significant spectral shifts from lower to higher frequencies, assessed by spectral index changes (P < 0.01) (19), were detected a number of seconds before the initiation of locomotion (nondepleted: mean T SD =
Standardized power (s.d.)
1580
A brain state permissive of locomotion. During the relatively rare instances when the depleted animals displayed locomotion, low-frequency oscillations were diminished (Fig. 1A). This situation bears an obvious resemblance to the DCS-induced state. Thus, a certain decrease of low-frequency oscillations may be required to initiate locomotion. We analyzed the detailed temporal patterns of shifts
Log (power)
made flat bipolar platinum electrodes positioned epidurally above the dorsal columns of the spinal cord at the upper thoracic level (Fig. 1, B and C). DCS had a dramatic effect on the amount of locomotion displayed during stimulation periods in the dopamine-depleted animals. This effect was strongest for 300-Hz stimulation; on average, the amount of locomotion during stimulation periods was more than 26 times as high as during the 5-min period before stimulation (Fig. 2A and Movie S1). DCS had a smaller, albeit clear, effect, using lower stimulation frequencies. In contrast, control experiments using air puffs or trigeminal nerve stimulation were not effective (Fig. 2A and fig. S5). DCS caused increased locomotion also during nondepleted conditions, but this increase was moderate (4.9 times prestimulus values at 300 Hz) in comparison to that in depleted animals (Fig. 2A. Locomotion was normally initiated a few seconds after the onset of DCS, with a slightly longer delay in depleted animals (median = 3.35/1.35 s, interquartile range = 2.22/1.22 s, P = 0.023, Mann-Whitey, in depleted/nondepleted animals at 300 Hz). In addition, a small residual effect was found after high-frequency stimulation in depleted, but not in nondepleted, animals (3.4 and 0.95 times prestimulus values, respectively for the 30 s after 300-Hz DCS). DCS also proved efficient for alleviation of bradykinesia as indicated by the relatively larger increase in the amount of fast-movement components in depleted animals (Fig. 2B). Analysis of LFP recordings during DCS in both MI and in striatum showed a shift in spectral power from lower to higher frequencies (average spectrograms from a total of 21 events of DCS at 300 Hz obtained from nine animals are shown in Fig. 2C). The spectral shift was maintained throughout the stimulation period and lasted for ~50 s after the end of stimulation. To condense the spectral shift into a single measure, a spectral index was computed by dividing the spectral range analyzed into an upper and lower half and calculating the ratio of the summed power of the frequencies in the two intervals [(25 to 55 Hz)/(1.5 to 25 Hz)]. The spectral index (black trace in Fig. 2C) illustrates the rapid spectral shift induced by DCS and the prolonged effect after DCS had ceased. DCS also affected the firing patterns of individual neurons. To avoid interference from stimulation artifacts, the 30-s stimulation periods were excluded from the analysis of spike data. Even during the period following stimulation, though, numerous neurons showed significantly altered firing rates (47.9% in MI and 41.8% in striatum, a = 0.01; Fig. 2C, rows 4 and 5, respectively). The fraction of units entrained to LFP dropped notably (from 42.7/38.8% in MI/striatum the 30 s before DCS to 24.5/24.0% the 30 s after DCS, a = 0.01). Although the onset of locomotion was delayed a few seconds, changes in the neural activity were detected almost immediately after DCS onset (mean T SD evoked potential latency = 44 T 5 ms), perhaps indicating that the electrophysiological changes have a permissive rather than a directly instructive role for the initiation of locomotion.
129
2.9 T 1.7 s, range 0.1 to 5.5 s, n = 88, MI and striatal LFP; depleted: 3.0 T 1.7, range 0.2 to 5.5 s, n = 48, MI and striatal LFP). Yet, there were also important differences, most notably below 25 Hz. A more differentiated decrease in power of oscillations below 8 Hz and an increase above 17 Hz was observed in nondepleted animals, whereas the spectral power in a broader range between 5 and 25 Hz was decreased in depleted animals. Because these different patterns occurred before the onset of locomotion, it is unlikely that they were due to differences in locomotion between the two groups. Instead, they could be part of the explanation of why depleted animals moved slower and for shorter time periods. On the single-neuron level, the same type of firing rate changes after DCS also occurred in conjunction with spontaneous locomotion events. From a total pool of 193 neurons (from nine control and five dopamine-depleted recording sessions in 11 animals), 111 modulated their firing rate during locomotion and, unexpectedly, 59 of these neurons showed a pattern of early activation, 2.9 T 1.4 s (mean T SD) before actual locomotion onset (range = 0.5 to 4.5 s, n = 59 striatal and MI units from depleted and nondepleted conditions) (Fig. 3C). DCS in combination with L-dopa treatment. To find the minimum dose of L-dopa (alone or combined with 300-Hz DCS) required to restore locomotion, DAT-KO mice were used. These mice have <5% of normal striatal content of dopamine (14). Dopamine can be further decreased to virtually undetectable levels by injecting AMPT (250 mg/kg intraperitoneally), resulting in a completely akinetic animal model (14). By gradually increasing dopamine levels through repeated L-dopa injections every hour, we tested the locomotion thresholds. In the group receiving only L-dopa injections (n = 6 sessions from 4 mice), locomotion typically first occurred after the fifth injection (5 mg/kg dose, corresponding to a total dose of 15 mg during the first 5 hours). When L-dopa treatment was combined with DCS, the same amount of locomotion was displayed after the second injection (2 mg/kg dose, corresponding to a total dose of 3 mg in the first 2 hours) (n = 10 experiments from seven mice (Fig. 4A). That means that in combination with DCS, one-fifth of the L-dopa total dose was enough to produce equivalent locomotion to L-dopa alone. There was also a general increase in the amount of locomotion displayed in the L-dopa+DCS group over the entire range studied. Thus, L-dopa+DCS seems to be superior to L-dopa alone in terms of the ability to rescue locomotive capability after severe dopamine depletion. Finally, animals in the L-dopa+DCS group consistently showed higher values of spectral index than the L-dopa only group. This suggests that DCS facilitates locomotion, even in severely depleted animals, through similar mechanisms (fig. S7). DCS is effective after chronic lesions. Although the acute dopamine depletion model employed in the first set of experiments could reproduce all the main symptoms of PD, it was important to confirm
the effectiveness of DCS in an animal model that also involves loss of nigrostriatal dopaminergic connections. Chronic dopaminergic denervation of the striatum was achieved using bilateral 6-OHDA lesions in rats (n = 4), resulting in progressive deterioration of motor function and sustained weight loss, both cardinal signs of successful lesioning (20, 21). When placed in the open field, lesioned rats displayed reduced locomotion compared with controls (n = 4), which received vehicle injections in identical sites in the striatum (mean T SEM = 2.85 T 0.068 and 7.78 T 0.144 mm/s on average, respectively). Quantification of immunohistochemical staining of the dopamine-synthesizing enzyme tyrosine hydroxylase indicated that lesioned rats had only ~20% of the immune signal found in sham-lesioned animals (fig. S8). Rats were tested during two 1-hour sessions in the open field, the first hour without stimulation and the second with DCS applied for 30 s every tenth minute. In the lesioned group, DCS resulted in remarkably increased amounts of locomotion compared to the first hour, whereas sham animals actually moved less during DCS sessions than during non-DCS sessions (Fig. 4C). Hence, there were specific improvements of motor function in the Parkinsonian state compared with controls. In lesioned rats, DCS not only alleviated hypokinesia during stimulation but also caused an increase in locomotion after the stimulation period. This residual effect lasted ~100 s (Fig. 4B). The effect of DCS on bradykinesia in 6OHDAâ&#x20AC;&#x201C;lesioned rats was also evaluated. Lesioned animals showed a relative increase in the number of scored locomotion events for all movement speeds, but this effect was more pronounced for faster movements, indicating a specific effect on bradykinetic symptoms in addition to the general improvement in the overall amount of locomotion (Fig. 4D). Discussion. We demonstrated that stimulation of the dorsal column pathways using epidural implanted bipolar electrodesâ&#x20AC;&#x201D;a simple, easy-toperform, semi-invasive methodâ&#x20AC;&#x201D;can restore locomotive capability in two animal models of PD symptoms: acutely dopamine-depleted mice and rats with dopaminergic neuronal loss. In parallel with the behavioral improvements, DCS shifted activity patterns in the primary motor cortex and in the dorsolateral striatum into a state closely resembling that found before and during spontaneous initiation of locomotion in normal and depleted animals. This suggests that DCS helps motor-related brain areas shift into a state permissive of the initiation of movements. What could be the mechanisms through which DCS allows a shift into a locomotion-permissive state? The first possible explanation could be that DCS, in addition to stimulating specific somatosensory pathways, may also recruit brainstem arousal systems, leading to sufficient cortical and striatal desynchronization required for voluntary initiation of movements (8). Such a possibility can be raised when the phenomenon of paradoxical kinesia is considered, that is, rare events in which PD pa-
www.sciencemag.org
SCIENCE
VOL 323
tients, aroused by frightening situations, exhibit sudden and transient improvement in motor function (22, 23). Here, the increase by a factor of 4.9 in locomotion produced by DCS in control animals, albeit much less than that observed in dopamine-depleted animals, could in theory support such an arousal hypothesis. Yet, a variety of observations suggest that this may not be the main mechanism accounting for DCS-induced locomotion. First, neither air puffs alone nor stimulation of trigeminal nerve afferents, both potent somatosensory arousal stimuli, induced locomotion in either control or dopamine-depleted animals. Secondly, in control experiments carried out in both awake and lightly anesthetized animals, DCS produced only a minimal arousal response when compared with other tactile, proprioceptive, and nociceptive stimuli (fig. S9, A and B). This is in line with a previous study that demonstrated that dorsal column recruitment produces no significant arousal effect (24). Overall, these data suggest that DCS may increase locomotion behavior primarily through direct modulation of lemniscal/ thalamic pathways. However, more experiments will be required to settle this issue. Our electrophysiological data suggest possible mechanisms for the success of DCS in restoration of locomotion, based on existing theories of basal ganglia pathology in PD and specifically considering the circuitry known to be involved in initiating voluntary locomotion (25). The command to the spinal cord to initiate locomotion, via reticulospinal pathways, is issued by the diencephalic and mesencephalic locomotor regions. For these midbrain structures to become active and trigger locomotion, they must be relieved from the tonic inhibition exerted by the output nuclei of the basal ganglia. This is accomplished by activation of striatal medium spiny neurons projecting to the output nuclei of the basal ganglia (26, 27). Under normal circumstances, the cortex has a powerful excitatory influence on the striatum. In contrast, with reduced striatal dopamine levels, the activation threshold of the projection neurons from the striatum is significantly increased (25), making it less likely that cortical input to the striatum will be conveyed through this pathway. As a consequence, brainstem motor regions remain under tonic inhibition, and the initiation of goal-directed locomotion and other types of volitional motor activity become impaired. DCS may exert its effect by activating large cortical areas, increasing the cortical and thalamic input to the striatum. This may, in turn, promote the depolarization and, consequently, facilitate the activation of striatal projection neurons. Another consequence of the reduced cortical control of striatum at low dopamine levels is that both thalamic and internally driven striatal low-frequency oscillations become more prominent (28, 29). These oscillations may lead to increased synchronicity because the generation of action potentials tends to occur at more distinct phases of the LFP oscillation (13, 30). This was confirmed in our experiments in which both motor cortex and striatum showed excessive low-frequency synchronized oscillatory activity
20 MARCH 2009
Downloaded from www.sciencemag.org on March 19, 2009
RESEARCH ARTICLES
1581
in dopamine-depleted animals and an increased entrainment of spikes to low-frequency components of the LFPs. Such synchronous activity interferes with normal information processing in these circuits and should likely be considered pathogenic in PD (12). Our data show that DCS effectively abolishes aberrant synchronous lowfrequency oscillations. It is, therefore, tempting to speculate that the suppression of low-frequency oscillations is particularly important for amelioration of motor symptoms in PD (31). Finally, the combined effect of L-dopa and DCS allowed for recovery of motor function at significantly lower doses of L-dopa in severely dopamine-depleted animals. The considerably less invasive nature of the epidural DCS electrode compared with DBS electrodes suggests that DCS could be a complement for treatment of symptoms of PD in earlier stages of the disease. We therefore propose that DCS should be investigated further in extensive experiments employing primate models of PD, preferably over longer time periods, to evaluate the potential viability of this new procedure as a treatment for Parkinsonian patients. References and Notes 1. 2. 3. 4.
S. Fahn, Ann. N.Y. Acad. Sci. 991, 1 (2003). A. Carlsson, Acta Neurol. Scand. Suppl. 51, 11 (1972). O. Hornykiewicz, Amino Acids 23, 65 (2002). K. M. Shaw, A. J. Lees, G. M. Stern, Q. J. Med. 49, 283 (1980).
5. A. L. Benabid, Curr. Opin. Neurobiol. 13, 696 (2003). 6. J. S. Perlmutter, J. W. Mink, Annu. Rev. Neurosci. 29, 229 (2006). 7. P. Plaha, Y. Ben-Shlomo, N. K. Patel, S. S. Gill, Brain 129, 1732 (2006). 8. E. E. Fanselow, A. P. Reid, M. A. Nicolelis, J. Neurosci. 20, 8160 (2000). 9. C. M. DeGiorgio, A. Shewmon, D. Murray, T. Whitehurst, Epilepsia 47, 1213 (2006). 10. M. S. George et al., Biol. Psychiatry 47, 287 (2000). 11. P. Brown et al., J. Neurosci. 21, 1033 (2001). 12. C. Hammond, H. Bergman, P. Brown, Trends Neurosci. 30, 357 (2007). 13. R. M. Costa et al., Neuron 52, 359 (2006). 14. T. D. Sotnikova et al., PLoS Biol. 3, e271 (2005). 15. D. J. Brooks, P. Piccini, Biol. Psychiatry 59, 908 (2006). 16. K. G. Lloyd, L. Davidson, O. Hornykiewicz, J. Pharmacol. Exp. Ther. 195, 453 (1975). 17. X. Drouot et al., Neuron 44, 769 (2004). 18. K. Sakai, D. M. Gash, Brain Res. 633, 144 (1994). 19. Materials and methods are available as supporting material on Science Online. 20. M. A. Cenci, I. Q. Whishaw, T. Schallert, Nat. Rev. Neurosci. 3, 574 (2002). 21. C. Winkler, D. Kirik, A. Bjorklund, M. A. Cenci, Neurobiol. Dis. 10, 165 (2002). 22. I. Schlesinger, I. Erikh, D. Yarnitsky, Mov. Disord. 22, 2394 (2007). 23. M. Glickstein, J. Stein, Trends Neurosci. 14, 480 (1991). 24. P. D. Wall, Brain 93, 505 (1970). 25. S. Grillner, P. Wallen, K. Saitoh, A. Kozlov, B. Robertson, Brain Res. Brain Res. Rev. 57, 2 (2008). 26. S. M. Brudzynski, M. Wu, G. J. Mogenson, Can. J. Physiol. Pharmacol. 71, 394 (1993).
27. M. R. DeLong, Trends Neurosci. 13, 281 (1990). 28. Y. Smith, D. V. Raju, J. F. Pare, M. Sidibe, Trends Neurosci. 27, 520 (2004). 29. C. J. Wilson, Neuron 45, 575 (2005). 30. J. D. Berke, M. Okatan, J. Skurski, H. B. Eichenbaum, Neuron 43, 883 (2004). 31. A. A. Kuhn et al., Brain 127, 735 (2004). 32. We thank W. M. Chan, G. Lehew, and J. Meloy for outstanding technical assistance; R. Gainetdinov, S.-C. Lin, H. Zhang, and K. Dzirasa for valuable comments; and S. Halkiotis for proofreading the manuscript. This work was supported by the National Institute of Neurological Disorders and Stroke (NINDS) R33NS049534 and the International Neuroscience Network Foundation to M.A.L.N., R01NS019576 and R01MH073853 to M.G.C., Ruth K. Broad Postdoctoral Award to R.F., and NRC and Knut and Alice Wallenberg Foundation to P.P. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NINDS or the National Institutes of Health. M.A.L.N. dedicates this paper to Lily Safra for her continuing support and to the memory of his grandfather, Angelo Nicolelis, who suffered from Parkinson’s disease. M.A.L.N. acknowledges a visiting professorship, Chaire Blaise Pascal, from the Région Ile de France at the Ecole Supérieure de Physique et de Chimie Industrielles, Paris.
Supporting Online Material www.sciencemag.org/cgi/content/full/323/5921/1578/DC1 Materials and Methods Figs. S1 to S9 Movie S1 References 20 August 2008; accepted 10 December 2008 10.1126/science.1164901
REPORTS Alfvén Waves in the Lower Solar Atmosphere David B. Jess,1,2* Mihalis Mathioudakis,1 Robert Erdélyi,3 Philip J. Crockett,1 Francis P. Keenan,1 Damian J. Christian4 The flow of energy through the solar atmosphere and the heating of the Sun’s outer regions are still not understood. Here, we report the detection of oscillatory phenomena associated with a large bright-point group that is 430,000 square kilometers in area and located near the solar disk center. Wavelet analysis reveals full-width half-maximum oscillations with periodicities ranging from 126 to 700 seconds originating above the bright point and significance levels exceeding 99%. These oscillations, 2.6 kilometers per second in amplitude, are coupled with chromospheric line-of-sight Doppler velocities with an average blue shift of 23 kilometers per second. A lack of cospatial intensity oscillations and transversal displacements rules out the presence of magneto-acoustic wave modes. The oscillations are a signature of Alfvén waves produced by a torsional twist of T22 degrees. A phase shift of 180 degrees across the diameter of the bright point suggests that these torsional Alfvén oscillations are induced globally throughout the entire brightening. The energy flux associated with this wave mode is sufficient to heat the solar corona. olar observations from both ground-based and spaceborne facilities show that a wide range of magneto-acoustic waves (1, 2) propagate throughout the solar atmosphere. However, the energy they carry to the outer solar atmosphere is not sufficient to heat it (3). Alfvén waves (pure magnetic waves), which are incompressible and can penetrate through the stratified solar atmosphere without being reflected (4), are the most promising
S
1582
wave mechanism to explain the heating of the Sun’s outer regions. However, it has been suggested that their previous detection in the solar corona (5) and upper chromosphere (6) is inconsistent with magnetohydrodynamic (MHD) wave theory (7, 8). These observations are best interpreted as a guided-kink magneto-acoustic mode, whereby the observational signatures are usually swaying, transversal,
20 MARCH 2009
VOL 323
SCIENCE
periodic motions of the magnetic flux tubes (7, 9). Numerical simulations (10) show that subsurface acoustic drivers and fast magneto-sonic kink waves (11, 12) can convert energy into upwardly propagating Alfvén waves, which are emitted from the solar surface. These numerical simulations are also in agreement with current analytical studies. In particular, it has been shown that footpoint motions in an axially symmetric system can excite torsional Alfvén waves (13). Other Alfvén wave modes may exist, although these are normally coupled to magneto-sonic MHD waves (14). In the solar atmosphere, magnetic field lines clump into tight bundles, forming flux tubes. Alfvén waves in flux tubes could manifest as torsional oscillations (7) that create simultaneous blue and red shifts, leading to the nonthermal broadening of any isolated line profile, and should thus be observed as full-width halfmaximum (FWHM) oscillations (15). A promising location for the detection of Alfvén waves is 1 Astrophysics Research Centre, School of Mathematics and Physics, Queen’s University, Belfast, BT7 1NN, Northern Ireland, UK. 2Solar Physics Laboratory, NASA Goddard Space Flight Center, Code 671, Greenbelt, MD 20771, USA. 3Solar Physics and Space Plasma Research Centre, University of Sheffield, Hicks Building, Hounsfield Road, Sheffield, S3 7RH, England, UK. 4Department of Physics and Astronomy, California State University Northridge, 18111 Nordhoff Street, Northridge, CA 91330, USA.
*To whom correspondence should be addressed. E-mail: d.jess@qub.ac.uk
www.sciencemag.org
Downloaded from www.sciencemag.org on March 19, 2009
130
131
OPEN SUBJECT AREAS: BIOMEDICAL ENGINEERING PARKINSON’S DISEASE
Received 11 June 2013 Accepted 6 January 2014 Published 23 January 2014
Correspondence and requests for materials should be addressed to M.A.L.N. (nicoleli@ neuro.duke.edu)
Chronic Spinal Cord Electrical Stimulation Protects Against 6-hydroxydopamine Lesions Amol P. Yadav1, Romulo Fuentes2, Hao Zhang3, Thais Vinholo3, Chi-Han Wang3, Marco Aurelio M. Freire2 & Miguel A. L. Nicolelis1,2,3,4,5 1
Department of Biomedical Engineering, Duke University, Durham, NC, 27780, 2Edmond and Lily Safra Institute of Neuroscience of Natal, Natal, Brazil, 59066-060, 3Department of Neurobiology, Duke University, Durham, NC, 27710, 4Duke Center for Neuroengineering, Duke University, Durham, NC, 27710, 5Department of Psychology and Neuroscience, Duke University, Durham, NC, 27708.
Although L-dopa continues to be the gold standard for treating motor symptoms of Parkinson’s disease (PD), it presents long-term complications. Deep brain stimulation is effective, but only a small percentage of idiopathic PD patients are eligible. Based on results in animal models and a handful of patients, dorsal column stimulation (DCS) has been proposed as a potential therapy for PD. To date, the long-term effects of DCS in animal models have not been quantified. Here, we report that DCS applied twice a week in rats treated with bilateral 6-OHDA striatal infusions led to a significant improvement in symptoms. DCS-treated rats exhibited a higher density of dopaminergic innervation in the striatum and higher neuronal cell count in the substantia nigra pars compacta compared to a control group. These results suggest that DCS has a chronic therapeutical and neuroprotective effect, increasing its potential as a new clinical option for treating PD patients.
P
arkinson’s disease (PD) is a debilitating neurodegenerative disorder caused by progressive loss of the dopaminergic neurons of the nigrostriatal pathway1. A variety of pharmacological approaches, of which L-dopa administration is the most effective, have been used to alleviate PD motor symptoms by supplementing the dopamine (DA) deficiency observed in the striatum2. Despite its initial efficacy, long-term administration of L-dopa results in fluctuations in the clinical response and in L-dopa-induced dyskinesia (LID), a condition which is difficult to treat2. The therapeutic approach involving electrical stimulation of subcortical nuclei, known as deep brain stimulation (DBS), is also very effective in alleviating the motor symptoms of PD3. The best candidates for this treatment are idiopathic PD patients with motor fluctuations and L-dopa-induced dyskinesia3. Depending on the selection criteria, 1.6 to 4.5% of PD patients are eligible for subthalamic nucleus DBS4, making DBS available only to a small percentage of the overall population of PD patients. Based on previous results obtained in three different rodent models of PD5, we have proposed that a new neuromodulation procedure that does not invade the brain tissue, known as dorsal column stimulation (DCS), has the potential to emerge as an additional therapeutic option for PD patients. In our hands, the most effective DCS effects in rodent PD models were obtained when continuous high frequency electrical stimulation was delivered to the large superficial fibers, running through the dorsal columns of the spinal cord, by a transversally oriented stimulating electrode, implanted at the high thoracic segments of the cord. Following the publication of our animal study, a hasty testing of DCS in two patients with advanced PD was carried out. This study did not produce any improvement in motor function6. However, these initial negative results were easily explained by the significant methodological differences between the stimulation protocols employed in the animal and clinical studies. These included: the localization of the electrodes (high cervical in the human study vs. high thoracic in the rodent study), and the orientation of the electrode poles relative to the spinal cord (longitudinal in the clinical study vs. transversal in the animal experiments). These differences caused the effective surface of contact between the electrodes and the spinal cord to be around seven times larger in the rodent protocol when compared to the human study. Such a marked difference could account for the activation of substantially more ascending dorsal column fibers in the rodents, explaining why such a potent therapeutic effect was observed in the animal study and absent in the clinical one7. SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
1
www.nature.com/scientificreports Soon after, however, support for our hypothesis that DCS can be effective as a PD therapy was subsequently obtained with the publication of four other clinical reports. These studies clearly indicated that DCS, originally intended for treating chronic pain, led to significant alleviation of motor symptoms in a total of 18 PD patients. For example, a PD patient, who had been previously implanted with a quadripolar spinal cord stimulator at low thoracic level to treat low back pain, experienced significant improvement of his motor symptoms during high frequency (130 Hz) DCS8. Furthermore, DCS also produced significant improvement of gait, posture, stability, and bradykinesia in 15 PD patients who received a spinal cord implant designed to treat low back and leg pain9. Next, a patient with advanced PD and other sensory symptoms also experienced improvement in gait and posture with quadripolar DCS applied at thoracic level10. Lastly, a female patient with PD and chronic neuropathic pain, experienced increasing improvement of motor PD symptoms over a two year period after initiating the DCS treatment. This improvement included alleviation of tremor and rigidity, and an improvement in gait and posture11. In our view, the pro-kinetic effect of DCS described in animal models and in several PD patients could be explained by the activation of somatosensory fibers, running through the dorsal column of the spinal cord, which led subsequently to the modulation of the ongoing activity of somatosensory and motor supraspinal structures5,12,13. Accordingly, in our original study, we observed that the motor effect of DCS is almost instantaneous and lasts as long as the electrical stimulation is maintained (i.e., see the supplementary video from previous studies5,8). Up to now, however, the results reported with spinal cord stimulation in animal models of PD have been limited to its acute effect. Nonetheless, DCS is also known to cause changes in gene expression at supraspinal structures, which in turn may lead to long-term sustained effects14,15. In the present study, therefore, we explored, for the first time, the potential effects of chronic delivery of DCS in rats with bilateral intrastriatal 6-hydroxydopamine (6-OHDA) lesions. While the lesioned rats exhibited sustained weight loss, postural and gait abnormalities, paralleled by destruction of dopaminergic striatal projections, a group of DCS-treated animals showed a dramatic and consistent reversal of these signs, including significant gain of body weight gain, marked improvement in motor functions, and less dopaminergic neuronal loss throughout the nigrostriatal system. These results are compatible with a neuroprotective effect of DCS against chemically induced dopaminergic lesions, and suggest that DCS could have long-term benefits as a potential new therapy for PD.
Results Chronic DCS prevents severe body weight loss in 6-OHDA lesioned rats. Rats received a bilateral 6-OHDA striatal or a sham lesion, and their weight and motor symptoms were evaluated for a period of 6 weeks (See Fig. 1a for time course of the experiment). While the sham lesion surgery, conducted on a group of control rats, caused minor weight loss that was recovered almost immediately, bilateral intrastriatal 6-OHDA lesions resulted in sustained weight loss (Fig. 1b). Twice a week DCS treatment (30 minutes per session) caused a dramatic recovery of body weight in 6-OHDA lesioned rats. Indeed, DCS treated animals not only recovered significantly faster than non-treated 6-OHDA rats; their weights were significantly higher from the 11th day post lesion to the end of the experiment (Day 11, p , 0.05; days 12–14, p , 0.01; days 15–42, p , 0.001, Bonferroni multiple comparisons, DCS treated compared to non-treated Fig. 1b). Even though the body weight of both 6-OHDA and 6-OHDA 1 DCS rats was significantly lower than that of the sham control animals throughout the experiment (ANOVA, two factor experiment with repeated measures, groups 3 days interaction: p , 0.0001), by SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
the 6th week, the weight of DCS treated rats approached that of the control rats much more than non-treated animals. In addition to faster weight recovery, DCS also prevented severe weight loss in lesioned rats during the early period after lesioning. 6OHDA 1 DCS treated rats had a maximum weight loss ([30.94 6 1.96]% ,9 days post lesion) that was significantly lower than nontreated 6-OHDA rats ([40.35 6 1.68]% ,15 days post lesion, p , 0.05, Mann-Whitney test, Fig. 1c). Weight recovery of 6-OHDA 1 DCS rats began in week 2 and continued to improve until week 6, while non-treated 6-OHDA rats started weight recovery only in the 6th week, implying that chronic DCS almost immediately reversed the trend of weight loss. Fig. 1d shows that after initiation of DCS treatment, weight change (normalized to week 1) was significantly higher at every time point for treated rats as compared to non-treated subjects: week 2 (0.68 6 3.37, 210.38 6 2.08, p , 0.05), week 3 (7.92 6 3.95, 28.87 6 3.26, p , 0.05), week 4 (12.26 6 4.52, 28.46 6 2.57, p , 0.01), week 5 (21.19 6 3.52, 23.42 6 2.3, p , 0.001) and week 6 (28.45 6 3.23, 6.32 6 1.66, p , 0.001; all measurements obtained with Mann-Whitney test). The slope of body weight recovery, calculated from a line connecting the minimum weight value with the final day value, was also higher for DCS treated rats than non-treated rats, (3.15 6 0.21 vs. 2.2 6 0.33, p , 0.05, Mann-Whitney test, Fig. 1e) confirming that chronic DCS not only prevented sustained weight loss by initiating weight recovery earlier, but it also accelerated the process of weight recovery in the 6-OHDA lesioned rats. Long term DCS restores motor function in PD rats. Bilateral intrastriatal lesioning with 6-OHDA resulted in a significant loss of motor function. Lesioned rats developed a characteristic crouched posture, which we quantified from video images by measuring the length of the major axis of an ellipse fitting the animal’s body. The crouched posture of the 6-OHDA rats resulted in an axis length shorter than in control rats. Quantitative analysis revealed that there was a significant interaction between groups and weeks (ANOVA, two factor experiment with repeated measures, groups 3 weeks: p , 0.05). Thus, starting at week 1.5, a gradual reversal of posture abnormalities was observed in the chronic DCS treated rats. Fig. 2a shows that axis length was significantly higher for 6-OHDA 1 DCS rats (n 5 6) for week 3.5 (63.59 6 0.74 versus 56.54 6 2.10, p , 0.05) and 4.5 (64 6 1.79 versus 56.37 6 1.79, p , 0.05) as compared to non-treated rats, n 5 8 (Bonferroni multiple comparisons). Overall, 30 min continuous DCS twice a week was sufficient to restore normal posture in 6-OHDA lesioned rats. Lesioned rats in this study showed no significant differences in the distance traveled during a 30 min open field session between the control and the DCS treated animals (Fig. 2b). Nonetheless, control rats showed a progressive decrease in traveled distance over the weeks, while 6-OHDA rats, both treated and non-treated, showed an irregular pattern (ANOVA, two factor experiment with repeated measures, groups 3 weeks interaction: p , 0.0001, but no differences were seen on post-hoc analysis). In the same way, the average speed displayed in the open field did not show significant differences between the groups (Fig. 2c, ANOVA, two factor experiment with repeated measures, groups 3 week interaction: p , 0.01, no differences on post-hoc analysis). Yet, the untreated lesioned rats displayed a striking symptom related to locomotion: a rigid gait resulting in a non-smooth locomotion (see Supplementary Video). This symptom could be quantified by calculating the spectral power at the frequency range of 0.5–4.75 Hz of the instant acceleration vector of the rat displacement: high power values represent a jerky, non-smooth gait (see Supplementary Figure 1 for an example), compatible with rigidity and crouched posture. DCS treated rats, on the other hand, showed a much lower spectral power in the same frequency (0.5–4.75 Hz) of instant acceleration vector. This allowed DCS-treated rats to exhibit a smoother gait, suggesting that DCS 2
132
www.nature.com/scientificreports
Figure 1 | DCS improves weight recovery of 6-OHDA lesioned rats. (a) Time course of the experiment. Numbers indicate experimental weeks from onset at time 0 (6-OHDA lesion). Rats in 6-OHDA 1 DCS group sustained epidural DCS electrode implantation (week 21) 1 week before the 6-OHDA lesion (week 0). Rats in 6-OHDA and sham control groups underwent only bilateral lesion procedure; 6-OHDA group (total 52.5 mg 6-OHDA), sham control group (only vehicle solution). Timing of motor assessment and 30 minute DCS sessions is illustrated by grey and black upward arrows respectively. Six weeks post lesion (week 6), brains were collected and processed for immunohistochemistry (IHC). (b) Changes in body weight after bilateral intrastriatal 6-OHDA lesion with or without DCS treatment. Lesioned, non-treated rats (n 5 8) suffered sustained weight loss with little-to-none recovery. Lesioned rats with DCS treatment (n 5 7, 30 min, 333 Hz continuous DCS during 30 min twice a week, starting 7th day, black arrow) recovered body weight significantly faster than non-treated rats (p , 0.0001, two-way repeated measure ANOVA). *:p , 0.05(day 11), **:p , 0.01(days 12â&#x20AC;&#x201C;14), ***:p , 0.001(days 15â&#x20AC;&#x201C;42), Bonferroni multiple comparisons. c) Maximum weight loss is significantly larger for non-treated rats compared to DCS treated rats. *:p , 0.05, Mann-Whitney test. d) DCS treatment reverses the trend of weight loss almost immediately, while non-treated rats continue to lose weight till week 5 as shown by weight change relative to week 1, which is significantly higher for treated rats as compared to non-treated. *:p , 0.05, **:p , 0.01, ***:p , 0.001, Mann-Whitney test. (e) DCS treatment results in accelerated weight recovery. Rate of weight gain is significantly higher in treated rats compared to non-treated. *:p , 0.05, Mann-Whitney test, a.u. (arbitrary unit). All error bars are s.e.m.
had an effect in improving animal locomotion. Overall, there was a significant interaction between groups and weeks in the spectral power of the acceleration vector (Fig. 2d, ANOVA, two factor experiment with repeated measures, groups 3 weeks: p , 0.05). At week 5, lesioned rats (2.46 6 0.66) had higher spectral power than both DCS treated (0.91 6 0.17) and control animals (0.96 6 0.13), p , 0.05, SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
Bonferroni multiple comparisons. Again, this finding suggests that the DCS treatment had a significant effect in improving motor behavior in lesioned animals. Further, the length of the stride was measured at week 4 for animals in the three groups. Untreated lesioned rats (58.74 6 2.59) had shorter strides as compared to both DCS treated (69.12 6 2.77) and 3
133
www.nature.com/scientificreports
Figure 2 | DCS restores motor functions in PD rats. (a) Changes in rat posture (measured as major axis length, greater length implying better posture) with or without DCS treatment. Lesioned rats develop crouched posture resulting in shorter major axis length. DCS treatment restores posture significantly faster than non-treated rats [groups 3 weeks interaction: p , 0.05, two-way repeated measure ANOVA, *:p , 0.05 at week 3.5 and 4.5 between DCS treated (n 5 6) and non-treated rats (n 5 8), n.s.: DCS treated rats were not significantly different from controls (n 5 4) from week 2, Bonferroni multiple comparisons]. (b) Distance travelled and (c) average speed during a 30 min open field session was not significantly different between the groups (groups 3 weeks interaction: p , 0.0001 for distance, groups 3 weeks interaction: p , 0.01 for speed, but no differences on post-hoc analysis for both). (d) Spectral power of the acceleration vector in frequency range 0.5–4.75 Hz (indicating jerky non-smooth locomotion) was significantly higher in lesioned rats than DCS treated and controls towards the end [p , 0.05, two-way repeated measure ANOVA, *:p , 0.05 (6-OHDA 1 DCS compared to 6-OHDA), 1:p , 0.05 (6-OHDA compared to controls), Bonferroni multiple comparisons]. (e) Stride length measured at week 4 was significantly higher for DCS treated rats as compared to non-treated (p , 0.01, one-way ANOVA, *:p , 0.05, **:p , 0.01-Tukey’s multiple comparison test). All error bars are s.e.m.
control animals (79.05 6 4.02), however the stride length of treated rats was not significantly different from that of controls (One-way ANOVA, p , 0.01, Tukey’s Multiple Comparison test, Fig. 2e). Long-term DCS protects nigrostriatal dopaminergic system. 6OHDA lesions resulted in severe damage to nigrostriatal dopaminergic striatal projections as compared to sham controls, as evidenced by the loss of TH immunoreactivity in these areas (Fig. 3a, bottom and SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
middle panels). Quantification of the striatal TH immunoreactivity by a contrast index (CI) showed significant differences when the three groups were compared (one-way ANOVA, p , 0.05, Fig. 3c, top panel). 6-OHDA lesioning caused a 67% decrease of striatal TH levels when compared to the sham control group (6-OHDA CI 0.114 6 0.022, sham lesion 0.34 6 0.024; Bonferroni’s Multiple Comparison, p , 0.05). However, DCS treatment of the 6-OHDA lesioned rats resulted in a 35% decrease of striatal TH levels. The difference 4
134
www.nature.com/scientificreports
Figure 3 | DCS protects nigrostriatal dopaminergic system. (a) Representative immunostaining for tyrosine hydroxylase (TH) in striatum (DAB stain). Note the higher dopaminergic innervation in the striatum (CPu, Acb) of DCS treated rat as compared to non-treated, scale bar 5 1 mm, CC 5 Corpus Callosum, LV 5 Lateral Ventricle, CPu 5 Caudate Putamen, Acb 5 Nucleus Accumbens. 6-OHDA lesion caused a 67% decrease of striatal TH levels (measured by contrast index), with respect to the sham control group, while treatment of the 6-OHDA lesioned rats with DCS resulted only in a 35% decrease, (top panel, 3c). The difference between the TH levels of 6-OHDA (n 5 6) and 6-OHDA 1 DCS (n 5 6) groups was significant (*:p , 0.05, Bonferroni’s Multiple Comparison, (b) Representative immunostaining for tyrosine hydroxylase in substantia nigra pars compacta (SNc), scale bar 5 500 um, SNc 5 substantia nigra pars compacta, VTA 5 ventral tegmental area. 6-OHDA lesion resulted in a severe loss of TH immunoreactivity (measured by contrast index) in the SNc. 6-OHDA rats showed 74% loss in TH CI as compared to sham controls while DCS treated rats showed only 44%. There was significant difference between the TH levels of 6-OHDA (n 5 6) and 6-OHDA 1 DCS (n 5 6) groups in the SNc (*:p , 0.05, Bonferroni’s Multiple Comparison, middle panel, 3c. Dopaminergic neuronal cell loss in SNc (expressed as % of neuronal count in sham controls) was significantly higher in 6-OHDA (93.32 6 1.62, n 5 8) rats as compared to 6-OHDA 1 DCS (87.43 6 1.95, n 5 6) rats (*:p , 0.05, t-test, 2 tailed). All error bars are s.e.m.
between the TH levels of the 6-OHDA and 6-OHDA 1 DCS groups (around 32% of the TH control levels) was significant (6-OHDA 1 DCS CI 0.222 6 0.018, 6-OHDA CI 0.114 6 0.022, Bonferroni’s Multiple Comparison, p , 0.05). Similarly, 6-OHDA lesioning resulted in a severe loss of TH immunoreactivity, measured by the contrast index in the subtantia nigra pars compacta (SNc), as shown in the example in Fig. 3b. Six weeks after lesioning, 6-OHDA rats showed a significant decrease in TH CI (0.106 6 0.014) as compared to sham lesion levels (0.410 6 SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
0.038; one-way ANOVA followed by Bonferroni’s Multiple Comparison, p , 0.05), which represents a 74% loss. Meanwhile, 6-OHDA rats treated with DCS exhibited only a 44% loss, which was significantly different from non-treated rats (6-OHDA 1 DCS CI 0.230 6 0.061, 6-OHDA CI 0.106 6 0.014, p , 0.05, Bonferroni’s Multiple Comparison, Fig. 3c, middle panel). 6-OHDA lesioning also resulted in severe loss of dopaminergic neurons from the SNc. Neuronal cell count depletion (expressed as % of sham control cell count) was significantly higher in non-treated rats [(93.32 6 1.62)%, n 5 8] as 5
135
www.nature.com/scientificreports
Figure 4 | Neuroprotective effect of DCS on weight and PD symptoms. (a) There was significant correlation between weight and major axis length (measure of posture) throughout the experiment (Spearman test, p , 0.0001). (b) Body weight had a significant negative correlation with spectral power of acceleration vector (indicating jerky non-smooth locomotion) â&#x20AC;&#x201C; Spearman test, p , 0.0001. (c) Spectral power of acceleration vector and speed were significantly correlated throughout the experimental period (Spearman test, p , 0.0001). Stride length measured at week 4 had significant correlation with major axis length ((d), Spearman test, p , 0.01) and body weight ((e), Spearman test, p , 0.01) indicating that recovery of body weight was related to an overall improvement in motor symptoms.
compared to DCS treated [(87.43 6 1.95)%, n 5 6] animals (p , 0.05, t-test, Fig. 3c, bottom panel). Global effects of neuroprotection. We also investigated the potential relation between body weight and multiple motor symptoms by calculating the linear correlations between them. This analysis revealed a positive correlation between weight and major body axis length (Spearman test: r 5 0.6639, p , 0.0001, Fig. 4a). Likewise, a significant negative correlation was found between weight and SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
non-smooth gait (Spearman test: r 5 0.346, p , 0.0001, Fig. 4b), indicating that rats with greater weight loss had severe motor symptoms. Body weight also exhibited a significant correlation with stride length measured at week 4 (Spearman test: r 5 0.7186, p , 0.01, Fig. 4e), indicating that an improvement in body weight was correlated with an improvement in posture and gait. There was a strong correlation between non-smooth gait and speed (Spearman test: r 5 0.4987, p , 0.0001, Fig. 4c), confirming our claim that the non-smooth gait was predominantly observed during high speed 6
136
www.nature.com/scientificreports
Figure 5 | DCS reverts weight, behavioral, and cellular parameters back to normality. (a) Principal component analysis (PCA) was performed using 9 rats having all the data variables at a single time point at the end of the experiment [weight (day 42); axis length, acceleration power, distance and speed at week 5.5; stride (week 4); TH immunoreactivity of striatum and SNc and SNc cell count). Representation of the individual rat data points, using the first three PCs on a 3D space, shows that the rats tend to cluster according to their experimental group. Agglomerative cluster analysis of the data (dotted ellipsoids) correctly identified the 6-OHDA rats (red dots), but failed to separate the control (blue squares) and the 6-OHDA 1 DCS (green triangles) rats. (b) PCA was performed with four motor parameters [speed, distance, non-smooth gait (acceleration power), and posture (axis length)] for all time points and the first two PC scores of all groups were averaged for each time point and plotted against time post lesion. The resulting graph shows the progression of the groups throughout the 6 weeks of the experiment. While the three experimental groups start close to each other (week 0.5–1.5), the 6OHDA and 6-OHDA 1 DCS groups split from control after week 2. By the end of the experimental time, the 6-OHDA 1 DCS group joins the control group (c) Euclidean distances of the average PCs of each group shows that the motor parameters of the 6-OHDA 1 DCS group started to drift from the untreated 6-OHDA group after week 1.5 and became close to control parameters after week 4.
displacement. A significant correlation between major body axis length and stride length (Spearman test: r 5 0.6275, p , 0.01 Fig. 4d) was also observed, indicating that rats with better posture had improved gait at the end of the experimental period. Overall, all these linear correlations supported our contention that chronic DCS treatment improved the clinical effects of 6-OHDA lesions in rats. Finally, we performed a cluster analysis using the principal component scores derived from a set of motor variables (posture, nonsmooth gait, distance and speed at week 5.5, stride at week 4), histological variables (TH immunoreactivity of striatum and SNc, and SNc cells/slide) and weight at day 42. The cluster analysis correctly grouped and isolated the data points from the 6-OHDA rats but failed to separate the control and the 6-OHDA 1 DCS treated rats (Fig. 5a). Next, using only the motor variables (speed, distance, posture, non-smooth gait), we analyzed how the three groups of animals behaved over time, by plotting the progression of the first two principal components which account for 79% of the total variance. This analysis revealed that, immediately after the lesion, the DCS treated group initially clustered together with the 6-OHDA lesioned group. Yet, 1.5 weeks afterwards, the DCS treated group started to separate from the lesioned group and move towards the control group. By the end of the experimental period (week 6), the 6OHDA 1 DCS group joined the control group’s space and became indistinguishable from it (Fig. 5b). This effect can be clearly documented by plotting the Euclidean distances of 6-OHDA and 6OHDA 1 DCS groups from the control group against time (Fig. 5.c). At week 1.5 the 6-OHDA 1 DCS group separates from 6-OHDA and starts approaching the control’s trajectory, reaching it by week 4 (Fig. 5c).
Discussion In this study we quantified for the first time the long-term effects of electrical stimulation of the dorsal column of the spinal cord (DCS) on body weight, motor symptoms and survival of nigrostriatal dopaminergic neurons in a chronic rat model of Parkinson’s disease. We found that chronic DCS applied on a regular basis was associated with progressive improvement in characteristic PD motor symptoms and accelerated recovery of lost weight. This improvement in clinical signs was paralleled with the maintenance of a higher density of dopaminergic innervation in the striatum and neuronal cell count SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
in the SNc of DCS-treated rats when compared to a group of untreated 6-OHDA lesioned animals. These results show that long-term DCS is associated with functional and structural recovery in a classic animal model of PD, suggesting that this method may be considered in the future as a potential therapy for PD patients. Comparison between the three groups of animals utilized in this study - sham control, 6-OHDA, and 6-OHDA 1 DCS - revealed a progressive weight increase in the control and the chronic DCS groups after the initial procedures (sham lesion and 6-OHDA lesion respectively). The control group reached its initial weight within 2 weeks post-surgery, as expected16–18. Afterwards, this group presented a normal weight increase of approximately 3% per week. Animals treated with chronic DCS showed a weight increase of 6% per week, following the 6-OHDA lesion, which was more prominent after 2 weeks of treatment. It is not clear if the increase was only an effect of improved motor function (i.e. restoring the ability of the animal to feed itself) or increased appetite (due to overall effects of treatment) or both. For example, dysphagia caused by oropharyngeal dysfunction and hyposmia, which could be responsible for weight loss, are common findings in advanced PD patients19,20. It is also suggested that neuroendocrinological dysregulation or lower concentrations of orexins could play an important role in the feeding behavior of PD patients21. Consistent with our present findings, data from advanced PD patients subjected to subthalamic (STN) DBS shows increased appetite and an average weight gain of 13% within ,16 months of treatment22, weight gain of 9.7 6 7 kg within 12.7 6 7.8 months with 60% improvement in UPDRS-III motor scores23 and a positive correlation between motor symptom improvement and weight gain24. Our findings indicate that this latter correlation was also obtained when DCS was applied to our PD animals. Future studies in our laboratory will address how the activity of neural ensembles controlling motor and feeding functions is affected in Parkinsonian states. The 6-OHDA lesion did not cause quantitative decrease in the animal’s average speed and traveled distance. This could be explained by the phenomenon known as starvation-induced hyperactivity25, which may have masked or compensated for the expected hypokinetic symptoms. Yet the lesioned animals exhibited clear motor symptoms, such as crouched posture, short strides, and non-smooth displacement across the open field, all of which were significantly 7
137
www.nature.com/scientificreports reduced by the DCS treatment delivered only twice a week. Using the current protocol for deep brain stimulation (DBS) as a benchmark, one can postulate that a more frequent treatment or even continuous DCS could very likely lead to even larger motor effects. At this point it is important to mention what mechanisms could account for the motor effects of DCS in PD. Both experimental evidence and modeling, obtained to explain the mechanism of DCS to treat chronic pain conditions, indicate that electrical stimulation delivered in the dorsal epidural space, as we did in the current work, activates mainly the superficial fibers of the dorsal columns and the dorsal roots of the corresponding spinal segment26. Thus, the consensus reached by this literature proposes that the mechanisms underlying the DCS effects reported here should emerge from the exclusive activation of ascending somatosensory fibers running through the dorsal portion of the spinal cord. This activation could, in turn, modulate the activity of multiple supraspinal structures, including thalamic, striatal and cortical areas5,12. We should, therefore, emphasize that even though higher DCS intensities can additionally recruit deeper structures of the dorsal columns, such as the corticospinal tract (CST) (which in rodents is located in the ventral part of the dorsal columns) we never observed any sign, such as muscle twitching of proximal or distal myotomes, that could support the thesis that the CST was recruited by our experimental protocol to stimulate the spinal cord. Even in humans, where CST are located more laterally, DCS at intensities above therapeutic levels can recruit fibers of the CST or local spinal motorneurons or interneurons27,28. However, since our experiments were conducted at a stimulation intensity that did not cause any muscle twitching, it is highly unlikely that activation of CST has contributed in any way to the effects described hereby. Yet, given the anatomical differences between rodent and primate nervous system (i.e. position of the CST within the spinal cord29) it is crucial to confirm these results in non-human primate models. To address that very issue, we have recently concluded a series of studies showing that DCS produces the same beneficial effects, in terms of improvement of motor symptoms, in a primate model of PD. Such results are currently being prepared for publication (personal observation). Degeneration of the nigrostriatal dopaminergic system is a common finding in postmortem studies of PD patients, and also a good indicator of the stage of the disease30. After long term treatment with DCS twice a week, we found that 6-OHDA rats exhibited a moderate yet significant reduction in the depletion of striatal TH-staining and TH-IR neuronal cells in SNc, as compared to non-treated lesioned animals. Although the mechanisms underlying such a reduction in dopaminergic degeneration have not been determined, we think they might be mediated by increased production or delivery of neurotrophic factors. Previous studies have shown that intrastriatal injections of brain derived neurotrophic factor (BDNF) can attenuate the effect of 6-OHDA lesions31,32. A recent study involving STN-DBS has shown proof of a neuroprotective effect on the SNc neurons in a rodent 6-OHDA model33, while subsequent experiments from the same group showed an increase in levels of nigrostriatal BDNF following STN DBS34. This could be the case with our DCS treated animals. Although clinical studies conducted with advanced PD patients to measure disease progression using 18F-fluorodopa PET failed to confirm a neuroprotective effect of clinically effective STN-DBS35, it would be interesting to investigate in the future whether neuroprotective effects of chronic DCS can be observed in a clinical population. Considering the increased longevity of the population worldwide, the introduction of novel treatments that address both the symptoms and progressive nature of PD constitute a major priority for the management of Parkinsonian patients. Based on recently described preliminary evidence showing efficacy of DCS in a series of PD patients worldwide8–11 and our own results in both acute and chronic PD animal models, we propose that chronic epidural DCS, a procedure that does not invade the brain, does not have serious side effects SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
and can be carried out at much lower costs and risks for patients, could be employed at the early clinical stages of PD to manage some of its cardinal motor symptoms. This conclusion is further supported by our preliminary data showing that DCS can also alleviate motor symptoms in a genetic model of PD in rats and in a primate model of PD (R.F. and M.A.L.N. personal observations). At this point, there are very few reports of DCS in PD patients. Thus, it is difficult to predict if the eligibility of DCS will be substantially better than for DBS. However, the lack of major side effects, the relative ease with which the surgical procedure can be performed, and the fact that there is no need to penetrate into brain tissue suggest that DCS could become an early stage therapy in the future management of PD patients.
Methods Animals. A total of 41 male Long-Evans rats (body weight ranging from 310 to 450 g) were housed individually with ad-libitum food and water in a temperature controlled room on a 12 h light/12 h dark cycle. Animal procedures were performed according to prior approved protocols by Duke University Institutional Animal Care and Use Committee and in accordance with the National Institute of Health Guide for the Care and Use of Laboratory Animals (NIH Publications No. 80–23). Rats were divided into 3 groups (sham control, 6-OHDA lesion and 6-OHDA lesion 1 DCS). 6-OHDA lesion and stimulation electrode implant procedures. Rats in the 6OHDA 1 DCS group underwent two separate surgical procedures. First, they were implanted with spinal stimulation electrodes under anesthesia induced with 5% halothane, ketamine (100 mg/kg), xylazine (10 mg/kg) and atropine (0.05 ml). Postoperative weight was monitored daily. The implantation procedure was adapted from previous studies5,36. The electrodes were inserted in the epidural space under T2 (thoracic vertebra) and tied to it with surgical suture. This prevented electrode migration and facilitated stimulation over a long period. One week later, after recovery of initial weight, rats were anesthetized for a second surgery, with 5% halothane, followed by intramuscular injections of ketamine (100 mg/kg), xylazine(10 mg/kg) and atropine (0.05 ml). A total of 52.5 ug 6-OHDA hydrobromide (Sigma Company, USA - 3.5 mg/ml in 0.05% ascorbate saline) was injected bilaterally into the striatum, at 3 locations on each side, using a needle, driven by a syringe pump (Sage, Model 361, Firstenberg Machinery Co Inc., USA) via 10 uL Hamilton syringe, at 1 uL/min. The needle was left in situ for 5 minutes and withdrawn slowly, to prevent backtracking of the drug. Anteroposterior, mediolateral and dorsoventral coordinates for the injections were: 11.0, 1/23.0, 25.0; 20.1, 1/ 23.7, 25.0 and 21.2, 1/24.5, 25.037 from bregma. Destruction of noradrenergic fibers and terminals was prevented by 1,3-Dimethyl-2-imidazolidinone (DMI, Sigma Company, 25 mg/kg), administered IP, 30 minutes prior to 6-OHDA treatment38. Rats belonging to the 6-OHDA lesion and sham control groups underwent only the surgical procedure required to perform the lesion; animals in the 6-OHDA lesion group received bilateral injections of 6-OHDA, while rats in the sham control group received only vehicle solution (0.05% ascorbate saline). Time course of the entire experiment is shown in Fig. 1a. The lesion procedures were performed by the same individuals throughout all the experiment and all groups were run in parallel. Extreme care was taken to consistently maintain the timing of the various methods and conditions during the lesion procedure. Weight and motor behavioral assessment. In the post-lesion period, rats had access to water soaked food pellets and fruit loop cereals and their body weight was recorded daily. 15 rats that did not display severe motor symptoms and lost less than 20% of initial weight at day 7 post-lesion were discarded from the experiment. Rats with severe weight loss (.20% body weight) were retained in their original groups and subjected to the entire experiment (N 5 18, 10 in 6-OHDA lesion 1 DCS group, and 8 in 6-OHDA lesion group). These were labeled as ‘‘strongly lesioned rats’’. Rats that lost more than 25% of initial weight were additionally hand-fed with peanut butter daily, until their weight reached 75%. A few days after lesioning, strongly lesioned rats exhibited several motor impairments, including postural and gait instability, and reduced forelimb dexterity39. These symptoms manifested into an inability to grasp or chew food, which was clearly observed while the rats tried to eat. Motor behavior was assessed twice a week, on days 4, 7, 11, 14, 18, 21, 25, 28, 32, 35, and 39 post lesion. Rats were placed in an elliptical open field (85 cm 3 70 cm axes, 60 cm tall) for 30 min and motor behavior was recorded from a bottom view camera. The posture and position of the rats in the open-field was extracted from digitized video recordings with custom designed algorithms implemented in Matlab (MathWorks, USA). Image processing of single frames from the video was used to extract the rat shape. An ellipsoid was fitted to the rat shape, and the length of the major axis of the ellipse (expressed in pixels) was used as a measure of posture. Digital videos of the ellipse superimposed on rat shape were saved and later used to confirm the accuracy of the algorithm. We observed that rats with crouched posture had shorter major axis length, while normal rats had longer lengths. For posture analysis only those frames during which the rats were mobile were used. Instantaneous speed and instantaneous acceleration vectors were calculated from the distance between the
8
138
www.nature.com/scientificreports X,Y location of the rat every 1/30 seconds in the open field. Lesioned rats showed jerky movement that lacked smoothness and fluidity. To quantify this abnormal behavior, we calculated the derivative of the instantaneous speed, thus obtaining a measurement of instantaneous acceleration. A spectrogram with a window of 4 seconds, sliding every 0.033 seconds was constructed with the ’mtspecgramc’ function (Chronux toolbox) from the acceleration signal. The power spectra between 0.5– 4.75 Hz of the time bins where locomotion was greater than 5 cm/s were averaged. For measuring stride length of rats, videos from top view camera were used. The front most part of the camera view was selected for better visualization of hind legs. A locomotion bout with at least 3 strides in the selected part of the open field was randomly selected from entire session and stride length was measured using ‘implay’ function in Matlab and averaged for the 3 strides. Stride was defined as the pixel distance between hind leg (right/left) take off point and subsequent landing point of same leg. Chronic electrical stimulation of the dorsal column of the spinal cord. After the 30 min behavioral session, rats of the 6-OHDA 1 DCS group were allowed to rest in their home cage for 30 min. Following this period, they were reintroduced in the open field and continuous DCS was applied for 30 min. Before each stimulation session, stimulation current intensities were determined for each rat, as described in previous study5. DCS consisted of biphasic square pulses of 1 ms duration, delivered at 333 Hz at a current intensity ,1.2 times the sensory threshold (mean 6 sd, intensity at 333 Hz was 167 6 52 uA). These intensities ensured that DCS did not cause an arousal effect5. Tyrosine hydroxylase staining and quantification. Forty-two days after lesion, animals were perfused with 4% paraformaldehyde and the brains were kept in 30% sucrose until sectioning. During tissue sectioning, 30ı̀m free-floating sections were obtained from the striatum (AP: 2 to 21) and substantia nigra (SN) (AP: 24.2 to 26.6), defined according to the rat brain Atlas40. Tyrosine hydroxylase (TH) immunohistochemistry was used to study the extent and position of striatal lesions and quantify the depletion of dopaminergic neurons in the nigrostriatal pathway as described elsewhere41. For quantification of TH-staining, the tissue samples were mounted and pictures were taken using a microscope with the same camera configuration and light intensity for each slice. TH-reactivity in both striatum and substantia nigra pars compacta (SNc) in all groups (sham control, 6-OHDA lesion and 6-OHDA lesion 1 DCS) was assessed by computer densitometry using digital images captured with the camera attached to the microscope. Average densitometric values were obtained by using the ImageJ software (http://rsb.info.nih.gov/ij/) from 2 images where both structures could be unequivocally defined (see Figure 3). The measurements were obtained inside a 0.04 mm2 square window positioned across the structures of interest. In order to evaluate the general TH-reactivity throughout the striatum and SNc we obtained three samples per structure. To minimize the effects of within-group variability, we adopted a normalized scale based on the non-reactive cortical white matter (averaged over measurements of 3 different sites using the same window). For every animal, the average optical density (OD) for the striatum or SNc was designated S, for the cortical white matter W and a contrast index C was calculated according to the equation: C 5 (S 2 W)/(S 1 W)42. Using a high-magnification microscope (NIS-element, Nikon, Japan) equipped with a software package, a total of 6 SNc slices between AP: 5.8–6.33 mm of bregma were used to bilaterally count immunoreactive neurons and an average cell count was calculated for every animal. To obtain an unbiased estimate of cell numbers, we applied the Abercrombie’s correction factor43, which compensates for the over counting of sectioned profiles, using the equation P 5 A(M/M 1 L), where P is the corrected value, A is the raw density measure, M is the section’s thickness (in micrometers) and L is the average diameter of cell bodies (n 5 40 by group) along the axis perpendicular to the plane of section. The mean value of TH-IR nigral neurons for each animal was expressed as a percentage of cell loss, compared to the mean cell count for sham control rats. Statistical analysis. All results are expressed as mean 6 sem. Statistical analysis was performed using a computer program (Graphpad Prism 5.0, Graphpad Software, USA). Weight and motor behavior data was subjected to two-way analysis of variance (ANOVA) with repeated measures, followed by post-hoc multiple comparison tests. Whenever distributions failed the normality test, non-parametric tests such as MannWhitney (t-test) were used. Spearman’s Rank Correlation test was used to study the correlation between different parameters. 1. Carlsson, A. Biochemical and pharmacological aspects of Parkinsonism. Acta Neurol. Scand. Suppl. 51, 11–42 (1972). 2. Nagatsua, T. & Sawadab, M. L-dopa therapy for Parkinson’s disease: past, present, and future. Parkinsonism Relat Disord 15 Suppl 1, S3–8, doi:10.1016/S13538020(09)70004-5 (2009). 3. Benabid, A. L. Deep brain stimulation for Parkinson’s disease. Curr. Opin. Neurobiol. 13, 696–706 (2003). 4. Morgante, L. et al. How many parkinsonian patients are suitable candidates for deep brain stimulation of subthalamic nucleus? Results of a questionnaire. Parkinsonism Relat Disord 13, 528–531, doi:10.1016/j.parkreldis.2006.12.013 (2007).
SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
5. Fuentes, R., Petersson, P., Siesser, W. B., Caron, M. G. & Nicolelis, M. A. Spinal cord stimulation restores locomotion in animal models of Parkinson’s disease. Science 323, 1578–1582, doi:10.1126/science.1164901 (2009). 6. Thevathasan, W. et al. Spinal cord stimulation failed to relieve akinesia or restore locomotion in Parkinson disease. Neurology 74, 1325–1327, doi:10.1212/ WNL.0b013e3181d9ed58 (2010). 7. Nicolelis, M. A., Fuentes, R., Petersson, P., Thevathasan, W. & Brown, P. Spinal cord stimulation failed to relieve akinesia or restore locomotion in Parkinson disease. Neurology 75, 1484; author reply 1484–1485, doi:10.1212/ WNL.0b013e3181f46f10 (2010). 8. Fenelon, G. et al. Spinal cord stimulation for chronic pain improved motor function in a patient with Parkinson’s disease. Parkinsonism Relat Disord 18, 213–214, doi:10.1016/j.parkreldis.2011.07.015 (2012). 9. Agari, T. & Date, I. Spinal cord stimulation for the treatment of abnormal posture and gait disorder in patients with Parkinson’s disease. Neurol. Med. Chir. (Tokyo). 52, 470–474 (2012). 10. Landi, A. et al. Spinal Cord Stimulation for the Treatment of Sensory Symptoms in Advanced Parkinson’s Disease. Neuromodulation: journal of the International Neuromodulation Society, doi:10.1111/ner.12005 (2012). 11. Hassan, S., Amer, S., Alwaki, A. & Elborno, A. A patient with Parkinson’s disease benefits from spinal cord stimulation. Journal of clinical neuroscience: official journal of the Neurosurgical Society of Australasia, doi:10.1016/j.jocn.2012.08.018 (2013). 12. Aguilar, J. et al. Spinal direct current stimulation modulates the activity of gracile nucleus and primary somatosensory cortex in anaesthetized rats. The Journal of physiology 589, 4981–4996, doi:10.1113/jphysiol.2011.214189 (2011). 13. Stancak, A. et al. Functional magnetic resonance imaging of cerebral activation during spinal cord stimulation in failed back surgery syndrome patients. Eur J Pain 12, 137–148, doi:10.1016/j.ejpain.2007.03.003 (2008). 14. Dejongste, M. J., Hautvast, R. W., Ruiters, M. H. & Ter Horst, G. J. Spinal Cord Stimulation and the Induction of c-fos and Heat Shock Protein 72 in the Central Nervous System of Rats. Neuromodulation: journal of the International Neuromodulation Society 1, 73–84, doi:10.1111/j.1525-1403.1998.tb00020.x (1998). 15. Maeda, Y., Ikeuchi, M., Wacnik, P. & Sluka, K. A. Increased c-fos immunoreactivity in the spinal cord and brain following spinal cord stimulation is frequency-dependent. Brain Res. 1259, 40–50, doi:10.1016/j.brainres.2008.12.060 (2009). 16. Lenard, L. et al. Feeding and body weight regulation after 6-OHDA application into the preoptic area. Brain Res. Bull. 27, 359–365 (1991). 17. Ferro, M. M. et al. Comparison of bilaterally 6-OHDA- and MPTP-lesioned rats as models of the early phase of Parkinson’s disease: histological, neurochemical, motor and memory alterations. J. Neurosci. Methods 148, 78–87, doi:10.1016/ j.jneumeth.2005.04.005 (2005). 18. Roedter, A. et al. Comparison of unilateral and bilateral intrastriatal 6hydroxydopamine-induced axon terminal lesions: evidence for interhemispheric functional coupling of the two nigrostriatal pathways. J. Comp. Neurol. 432, 217–229 (2001). 19. Muller, J. et al. Progression of dysarthria and dysphagia in postmortem-confirmed parkinsonian disorders. Arch. Neurol. 58, 259–264 (2001). 20. Ondo, W. G. et al. Weight gain following unilateral pallidotomy in Parkinson’s disease. Acta Neurol. Scand. 101, 79–84 (2000). 21. Bachmann, C. G. & Trenkwalder, C. Body weight in patients with Parkinson’s disease. Mov. Disord. 21, 1824–1830, doi:10.1002/mds.21068 (2006). 22. Moro, E. et al. Chronic subthalamic nucleus stimulation reduces medication requirements in Parkinson’s disease. Neurology 53, 85–90 (1999). 23. Macia, F. et al. Parkinson’s disease patients with bilateral subthalamic deep brain stimulation gain weight. Mov. Disord. 19, 206–212, doi:10.1002/mds.10630 (2004). 24. Gironell, A., Pascual-Sedano, B., Otermin, P. & Kulisevsky, J. [Weight gain after functional surgery for Parkinsons disease]. Neurologia 17, 310–316 (2002). 25. Pirke, K. M., Broocks, A., Wilckens, T., Marquard, R. & Schweiger, U. Starvationinduced hyperactivity in the rat: the role of endocrine and neurotransmitter changes. Neurosci. Biobehav. Rev. 17, 287–294 (1993). 26. Holsheimer, J. Which Neuronal Elements are Activated Directly by Spinal Cord Stimulation. Neuromodulation: journal of the International Neuromodulation Society 5, 25–31, doi:10.1046/j.1525-1403.2002._2005.x (2002). 27. Dimitrijevic, M. R., Faganel, J., Sharkey, P. C. & Sherwood, A. M. Study of sensation and muscle twitch responses to spinal cord stimulation. Int. Rehabil. Med. 2, 76–81 (1980). 28. Nashold, B. S. Jr., Somjen, G. & Friedman, H. The effects of stimulating the dorsal columns of man. Med. Prog. Technol. 1, 89–91 (1972). 29. Kaas, J. H. et al. Cortical and subcortical plasticity in the brains of humans, primates, and rats after damage to sensory afferents in the dorsal columns of the spinal cord. Exp. Neurol. 209, 407–416, doi:10.1016/j.expneurol.2007.06.014 (2008). 30. Kish, S. J., Shannak, K. & Hornykiewicz, O. Uneven pattern of dopamine loss in the striatum of patients with idiopathic Parkinson’s disease. Pathophysiologic and clinical implications. N. Engl. J. Med. 318, 876–880, doi:10.1056/ NEJM198804073181402 (1988). 31. Shults, C. W., Kimber, T. & Altar, C. A. BDNF attenuates the effects of intrastriatal injection of 6-hydroxydopamine. Neuroreport 6, 1109–1112 (1995).
9
139
www.nature.com/scientificreports 32. Klein, R. L., Lewis, M. H., Muzyczka, N. & Meyer, E. M. Prevention of 6hydroxydopamine-induced rotational behavior by BDNF somatic gene transfer. Brain Res. 847, 314–320 (1999). 33. Spieles-Engemann, A. L., Collier, T. J. & Sortwell, C. E. A functionally relevant and long-term model of deep brain stimulation of the rat subthalamic nucleus: advantages and considerations. Eur. J. Neurosci. 32, 1092–1099, doi:10.1111/ j.1460-9568.2010.07416.x (2010). 34. Spieles-Engemann, A. L. et al. Subthalamic Nucleus Stimulation Increases Brain Derived Neurotrophic Factor in the Nigrostriatal System and Primary Motor Cortex. J Parkinsons Dis 1, 123–136 (2011). 35. Hilker, R. et al. Disease progression continues in patients with advanced Parkinson’s disease and effective subthalamic nucleus stimulation. J. Neurol. Neurosurg. Psychiatry 76, 1217–1221, doi:10.1136/jnnp.2004.057893 (2005). 36. Fuentes, R., Petersson, P. & Nicolelis, M. A. Restoration of locomotive function in Parkinson’s disease by spinal cord stimulation: mechanistic approach. Eur. J. Neurosci. 32, 1100–1108, doi:10.1111/j.1460-9568.2010.07417.x (2010). 37. Winkler, C., Kirik, D., Bjorklund, A. & Cenci, M. A. L-DOPA-induced dyskinesia in the intrastriatal 6-hydroxydopamine model of parkinson’s disease: relation to motor and cellular parameters of nigrostriatal function. Neurobiol. Dis. 10, 165–186 (2002). 38. Hrdina, P. D. & Dubas, T. C. Brain distribution and kinetics of desipramine in the rat. Can. J. Physiol. Pharmacol. 59, 163–167 (1981). 39. Deumens, R., Blokland, A. & Prickaerts, J. Modeling Parkinson’s disease in rats: an evaluation of 6-OHDA lesions of the nigrostriatal pathway. Exp. Neurol. 175, 303–317, doi:10.1006/exnr.2002.7891 (2002). 40. Paxinos, G. & Watson, C. The rat brain, in stereotaxic coordinates. Compact 3rd edn, (Academic Press, 1997). 41. Tischler, A. S. Triple immunohistochemical staining for bromodeoxyuridine and catecholamine biosynthetic enzymes using microwave antigen retrieval. J. Histochem. Cytochem. 43, 1–4 (1995). 42. Freire, M. A., Oliveira, R. B., Picanco-Diniz, C. W. & Pereira, A. Jr. Differential effects of methylmercury intoxication in the rat’s barrel field as evidenced by NADPH diaphorase histochemistry. Neurotoxicology 28, 175–181, doi:10.1016/ j.neuro.2006.06.007 (2007).
SCIENTIFIC REPORTS | 4 : 3839 | DOI: 10.1038/srep03839
43. Abercrombie, M. Estimation of nuclear population from microtome sections. Anat. Rec. 94, 239–247 (1946).
Acknowledgments We thank J. Meloy for help with building stimulation electrodes; L. Oliveira and S. Halkiotis for technical support and Miguel Pais-Vieira for thoughtful comments on the manuscript. This research was supported by NIH Transformative award (R01-NS073125-03), NIH Director’s Pioneer award (DP1-OD006798) and the grant ‘‘Plano de Acao Brasil Suica CNPq 590006/2010-0’’ awarded to M.A.L.N. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Office of the NIH Director or the NIH.
Author contributions A.P.Y., R.F., H.Z. and M.A.L.N. designed experiments; A.P.Y., R.F. and M.A.L.N. wrote the paper; A.P.Y., H.Z., R.F., M.A.M.F. and M.A.L.N. analyzed the data; A.P.Y., H.Z. and C.W. performed the surgeries; A.P.Y. and T.V. conducted experiments; T.V. performed immunohistochemistry.
Additional information Supplementary information accompanies this paper at http://www.nature.com/ scientificreports Competing financial interests: The authors declare no competing financial interests. How to cite this article: Yadav, A.P. et al. Chronic Spinal Cord Electrical Stimulation Protects Against 6-hydroxydopamine Lesions. Sci. Rep. 4, 3839; DOI:10.1038/srep03839 (2014). This work is licensed under a Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported license. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0
10
140
141
Neuron
Report Spinal Cord Stimulation Alleviates Motor Deficits in a Primate Model of Parkinson Disease Maxwell B. Santana,1,7,9 Pär Halje,2,9 Hougelle Simplı́cio,1,8 Ulrike Richter,2 Marco Aurelio M. Freire,1 Per Petersson,2,10 Romulo Fuentes,1,10 and Miguel A.L. Nicolelis1,3,4,5,6,10,* 1Edmond
and Lily Safra International Institute of Neuroscience of Natal, 590660 Natal, Brazil Neurophysiology and Neurotechnology, Neuronano Research Center, Department of Experimental Medical Sciences, Lund University, BMC F10, 221 84 Lund, Sweden 3Biomedical Engineering 4Center for Neuroengineering 5Department of Neurobiology 6Department of Psychology and Neuroscience Duke University, Durham, NC 27708, USA 7Psychobiology, Federal University of Rio Grande do Norte, 59072 Natal, Brazil 8State University of Rio Grande do Norte, RN 59607-360 Mossoro, Brazil 9Co-first author 10Co-senior author *Correspondence: nicoleli@neuro.duke.edu http://dx.doi.org/10.1016/j.neuron.2014.08.061 2Integrative
SUMMARY
Although deep brain electrical stimulation can alleviate the motor symptoms of Parkinson disease (PD), just a small fraction of patients with PD can take advantage of this procedure due to its invasive nature. A significantly less invasive method—epidural spinal cord stimulation (SCS)—has been suggested as an alternative approach for symptomatic treatment of PD. However, the mechanisms underlying motor improvements through SCS are unknown. Here, we show that SCS reproducibly alleviates motor deficits in a primate model of PD. Simultaneous neuronal recordings from multiple structures of the cortico-basal ganglia-thalamic loop in parkinsonian monkeys revealed abnormal highly synchronized neuronal activity within each of these structures and excessive functional coupling among them. SCS disrupted this pathological circuit behavior in a manner that mimics the effects caused by pharmacological dopamine replacement therapy or deep brain stimulation. These results suggest that SCS should be considered as an additional treatment option for patients with PD. INTRODUCTION Chronic electrical stimulation of subcortical brain structures, a procedure known as deep-brain stimulation (DBS), has become an important complement to dopamine replacement therapy in the symptomatic treatment of Parkinson disease (PD) (Benabid et al., 1987). However, partially because of the highly invasive nature of this surgical procedure (Morgante et al., 2007) and its 716 Neuron 84, 716–722, November 19, 2014 ª2014 Elsevier Inc.
need for additional complex and costly technologies, only a small fraction of all patients with PD who could possibly benefit from this therapy are actually eligible for implantation. In this context, the recent demonstration that electrical epidural spinal cord stimulation (SCS) alleviates akinesia in rodent models of PD (Fuentes et al., 2009) and reduces motor symptoms in patients (Agari and Date, 2012; Fénelon et al., 2012; Hassan et al., 2013; Landi et al., 2012) is a significant finding because SCS, unlike DBS, is minimally invasive. Following the initial report on rodent PD models, SCS has been evaluated for treatment of PD in a few clinical case studies. Results of these studies range from no measurable improvements in two patients (Thevathasan et al., 2010) to significant symptomatic relief (Agari and Date, 2012; Fénelon et al., 2012; Hassan et al., 2013; Landi et al., 2012) in 19 patients. More importantly, in some cases SCS achieved PD symptom relief equivalent to the best effects obtained with pharmacological treatment (Fénelon et al., 2012). At present, the underlying causes for the different outcomes in these preliminary clinical studies are not known, but variations in electrode design, spinal cord implantation location, and choice of stimulation parameters have been suggested as possible contributing factors (Fuentes et al., 2010; Nicolelis et al., 2010). One way to optimize the application of this potential therapy for PD is to establish what neurophysiological changes are associated with the relief of symptoms, and to evaluate how these changes can be most effectively induced. Here, we have addressed these questions by characterizing the behavioral and neurophysiologic SCS effects in 6-OHDA (6-hydroxydopamine) lesioned marmoset monkeys (Callithrix jacchus), a primate model of PD. RESULTS Using previously described procedures (Annett et al., 1992; Mitchell et al., 1995), five adult male marmosets were caused to be parkinsonian through 6-OHDA stereotactic micro-injections in the medial forebrain bundle of either one (n = 2) or both
142
Neuron Spinal Cord Stimulation Alleviates Parkinsonism
hemispheres (n = 3). Injections resulted in neurodegeneration of the midbrain dopaminergic neurons projecting to the forebrain in the lesioned hemispheres, as assessed postmortem through immunohistochemical quantification of tyrosine hydroxylase (Figure S1 available online). In lesioned animals, the number of dopaminergic neurons in the midbrain was reduced to 42% ± 23% of the values observed in control animals. In addition, axonal terminal staining density in the caudate-putamen decreased to 44% ± 10% of the levels seen in normal subjects. The severity of parkinsonism was regularly examined in all marmosets using manual scoring of a wide range of clinical signs/deficits (see Supplemental Experimental Procedures and Movie S1 for details) and automated quantification of spontaneous motor behavior. On average, spontaneous locomotion was reduced to approximately one-fourth of prelesion activity, and PD signs approached the maximum score (mean ± SD: 75% ± 29%) in all the eight categories assessed (Bankiewicz et al., 2001; Verhave et al., 2009). Once PD clinical signs had been confirmed, animals underwent implantation with bipolar epidural SCS electrodes positioned symmetrically over the dorsal midline of the spinal cord at a high thoracic level (T3–T4). Four of the five animals (two with bilateral and two with unilateral medial forebrain bundle lesions) also underwent chronic implantation with microelectrode arrays/bundles in both hemispheres for subsequent recording of neuronal ensemble activity (both single units and local field potentials [LFPs]; Figure S2). These implants targeted multiple structures in each animal, including the primary motor cortex, putamen, the subthalamic nucleus, globus pallidus pars externa (GPe) and interna, and the ventrolateral and ventral posterolateral thalamic nuclei. In the two unilateral lesion animals, instead of the globus pallidus, parts of primary somatosensory cortex were implanted. Prior to SCS, all animals were also subjected to acute pharmacological inhibition of dopamine synthesis (subcutaneous injections of alpha-methyl-p-tyrosine [AMPT] 2 3 240 mg/kg) to further aggravate the PD signs. To avoid SCS current intensities that could be experienced as uncomfortable, before each stimulation session the intensity of each stimulation frequency was adjusted and set to 1.7 times the minimum intensity at which any behavioral response could be consistently detected (small postural changes, head or neck movements). Overall, as previously reported in rodents (Fuentes et al., 2009), we observed that SCS induced a clear alleviation of motor impairment in monkeys who exhibited severe clinical PD signs. Because the stimulation frequencies used in this study (range, 4–300 Hz) proved equally effective, the analysis of the SCS effects, both behavioral and electrophysiological, were performed by pooling all the frequencies, unless otherwise stated. All recordings/stimulation sessions were performed in freely behaving animals in a transparent acrylic box. Based on our automated image analysis of digital videos obtained from multiple cameras during neuronal recording sessions, SCS induced a 221% increase in general motility of trunk, head, limbs, and tail (p < 0.05, Wilcoxon signed rank test, Figure 1A), a 192% increase in the frequency of bouts of spontaneous locomotion (p < 0.001, two-proportional z test, Figure 1B), and a 144% increase in the duration of locomotion periods (p < 0.05, Wilcoxon rank sum test, Figure 1C). These improvements resulted in a 243% increase in the total distance covered (p < 0.05, Wilcoxon
rank sum test, Figure 1D) by the monkeys. Remarkably, SCS induced a preferential increase in the fraction of faster locomotion components, indicating a specific reduction of bradykinesia, (p < 0.05, Kolmogorov-Smirnov test of difference in speed histogram distributions, Figure 1E). Overall, the resulting distance covered in locomotive behavior was practically normalized by SCS (on average 91% of intact values, but locomotion differed somewhat in that stimulated animals tended to extend bout duration [628%] whereas reducing the frequency [11%] compared to the intact state). The improvements in motor disability were also assessed by an observer blinded to stimulation conditions that rated specific clinical signs, such as freezing, hypokinesia, bradykinesia, coordination, gait, posture, and gross and fine motor skills, during the OFF and ON periods. The motor deficits that exhibited the highest reduction during SCS were freezing (31%), hypokinesia (23%), posture (23%), and bradykinesia (21%). Overall, the PD score showed, on average, a significant reduction of 18.4% ± 13.9% (p < 0.001, Wilcoxon signed rank sum test [including all five subjects], Figure 1F; Figure S1). In addition to the general clinical improvements observed in all monkeys, in a few instances SCS resulted in an extraordinary functional recovery. An example of this is shown in Figure 1G, where a severely parkinsonian animal, who reached the maximum PD score on all clinical signs rated, showed a dramatic improvement during SCS. This allowed the animal to find and retrieve a food item with no difficulty whatsoever (see Movie S2). Chronic, multisite neuronal extracellular activity was analyzed both in terms of changes in single neuron’s firing patterns and at the level of LFPs. We observed that SCS induced changes in neural populations throughout the cortico-basal gangliathalamic loop in parkinsonian animals (Figure 2A). As shown in Figures 2B and 2C, the most noticeable effect was the suppression of LFP power in a frequency interval roughly spanning the beta-band (8–20 Hz), which was abnormally strong in all PD monkeys (respective peak power frequencies for the four subjects were 10, 11, 12, and 15 Hz; Figure S3A; Stein and BarGad, 2013). This suppression of LFP oscillations was observed in all animals and, notably, in all parts of the cortico-basal ganglia-thalamic loop (although it did not reach significance level, p < 0.05, in GPe when comparing averaged power spectra; Figures 2B and 2C, p < 0.05, Wilcoxon rank sum test on band power 8–20 Hz). In agreement with the behavioral effects, beta suppression could be obtained using both low- and high-frequency SCS paradigms with approximate equal efficacy (average LFP power spectra ON/OFF stimulation for all animals and frequencies are presented in Figure 2B). Next, we examined the effects of SCS on single unit activity. Overall, approximately one-third of the neurons recorded in the lesioned hemispheres displayed significant changes in firing rates associated with SCS. In contrast to the effect on LFP and motor behavior, SCS modulation of neuronal firing rate differed markedly according to the stimulation frequency (Figure 3A; see also Figure S3C). Whereas stimulation at low frequencies (4–20 Hz) caused mostly excitatory neuronal responses, inhibitory firing modulation predominated during high-frequency stimulation (80–300 Hz; Pearson correlation between the ratio of inhibitory/excitatory responses and stimulation frequency was: primary motor cortex, r = 0.96, p < 0.01; putamen, r = 0.74, Neuron 84, 716–722, November 19, 2014 ª2014 Elsevier Inc. 717
143
Neuron Spinal Cord Stimulation Alleviates Parkinsonism
Figure 1. Spinal Cord Stimulation Alleviates Motor Symptoms in Parkinsonian Primates (A) Average effect on general motility in response to SCS. Each color line represents one recorded animal over all trials. (B–D) Average recovery of locomotion: bout distance, bout frequency and duration, respectively (colors represent the four different subjects and asterisks denote significant group differences). (E) Reduction in bradykinesia reflected by the preferential recovery of faster movement components in locomotion. (F) Average improvements in PD score in all testing sessions divided by symptom category (mean and SEM shown). (G) Example of functional motor improvement from a state of severe parkinsonism enabling an animal to retrieve food reward using skilled reaching and grasping movements.
p < 0.05; ventral posterolateral thalamic nuclei, r = 0.94, p < 0.01; ventrolateral thalamic nuclei, r = 0.76, p < 0.05; subthalamic nucleus, r = 0.92, p < 0.01; GPe and globus pallidus pars interna not significant, Figure 3B). Consistent with the LFP oscillatory activity, we observed a large fraction of neurons with beta oscillatory firing in the OFF period (Figure 3C) that was partially suppressed during the SCS ON period (Figures 3D, 3E, and S2). In total, 152 of 273 (56%) neurons from the lesioned hemispheres displayed significant beta range (8–16 Hz) rhythmic firing patterns during the OFF periods (p < 0.01, as compared to spectra computed from equivalent random spike trains). Of these 152 neurons, 39 (26%) significantly decreased their beta rhythmicity during the SCS ON period. The change in beta power for all units with significant rhythmic activity in the beta range is summarized in Figure 3F. Taken altogether, these data suggest that SCS-induced motor deficit relief is primarily associated with the disruption of synchronized oscillatory activity rather than with specific changes in firing rate. However, because cortico-basal ganglia activity is known to be strongly influenced by behavioral state, neurophysiologic changes could also result, to some extent, from sec718 Neuron 84, 716–722, November 19, 2014 ª2014 Elsevier Inc.
ondary changes in animal motor behavior. Therefore, to further clarify how SCS induces neurophysiologic changes that may cause symptomatic relief, we compared the neuronal activity patterns from the lesioned hemisphere during SCS, in the two hemilesioned animals, to either the patterns of the intact hemisphere or to the lesioned hemisphere following L-DOPA treatment (subcutaneous 15 mg/kg with benserazide 6.25 mg/kg). Recordings were split into 4 s epochs that were classified as either active or inactive states based on automatically quantified motor activity. To facilitate comparisons between states, two separate indices were constructed. First, two vectors were created summarizing the mean differences (parkinsonian versus intact state) and (parkinsonian versus L-DOPA-treated state), respectively, in two multidimensional parameter spaces—spectral LFP power and firing rate per structure. Each recorded epoch could then be represented as a point in the parameter spaces and be quantitatively compared to the intact or L-DOPA treated state by geometrical projection onto these vectors. Using this metric, it was evident that SCS treatment caused a shift toward healthy brain activity patterns resembling the effect of L-DOPA treatment. This effect was observed only for the analysis of LFP spectral power and not for neuronal firing rates (Figure S3B).
144
Neuron Spinal Cord Stimulation Alleviates Parkinsonism
Figure 2. Spinal Cord Stimulation Alters Neuronal Activity Patterns in Basal Ganglia Circuits (A) Example of parallel changes in LFP power in multiple structures of the cortico-basal ganglia-thalamic loop. For each brain structure, right depicts pooled LFP spectrograms (brain slice figures reproduced with permission from Palazzi and Bordier, The Marmoset Brain in Stereotaxic Coordinates, Springer Science+Business Media). Note the immediate reduction of low-frequency oscillations (beta band) in response to SCS (red bar, stimulation frequency: 4 Hz; color codes denote decibels above pink noise background for LFPs). (B) Average LFP spectra for all recording sessions normalized to pink noise showing a significant SCS-induced reduction in LFP beta-power in all structures except GPe. Shaded area denotes 95% CI with 100 bootstraps. (C) Changes in normalized firing rates of individual neurons were diverse but, on average, they decreased in response to SCS in globus pallidus pars interna and ventrolateral thalamic nuclei.
Importantly, this shift could not be explained as a secondary effect due to an active or inactive behavioral state because a twoway ANOVA, used to estimate the relative effect size of SCS compared to that of behavioral state, showed that only 2.9% and 0.8% of the total variance (eta-squared; using the metric for the intact and L-DOPA treated state, respectively) could be attributed to behavioral state change, whereas the effect of SCS treatment explained 13.4% and 10.8% of the total variance. Consequently, the main effect shared by both SCS and LDOPA treatment appears to be the suppression of the excessive neuronal population synchronization associated with the parkinsonian state. Although it is not clear how coordinated low-frequency activity patterns arise in PD, it is possible that an altered functional coupling between the circuit elements of the cortico-
basal ganglia-thalamic loop may be a key underlying factor (Williams et al., 2002). To test this idea, we computed the coherence of the LFP signals between pairs of different neural structures as an indirect measure of their functional connectivity. In the 6OHDA lesioned dopamine-depleted hemispheres, we found strong coherence between pairs of structures, but only in the parkinsonian low-frequency range (8–15 Hz) (Figure 4A, black traces). Importantly, like the L-DOPA treatment, SCS reduced the beta coherence (Figure 4A red trace), leading to a significant functional decoupling between the different structures. This suggests that SCS brings the functional connectivity of the corticobasal ganglia-thalamic circuit closer to the normal state of the intact brain. Indeed, this decoupling of parkinsonian LFP oscillations in the beta band was observed between all the recorded Neuron 84, 716–722, November 19, 2014 ª2014 Elsevier Inc. 719
145
Neuron Spinal Cord Stimulation Alleviates Parkinsonism
Figure 3. Spinal Cord Stimulation Alters the Firing Rate and Rhythmicity of Neuronal Units in Basal Ganglia Circuits (A) Standardized neuronal firing rate response to different SCS frequencies in multiple structures of the basal ganglia circuits (neurons rank ordered according to responses). (B) The fraction of inhibitory responses increased with higher SCS frequencies. (C) Autocorrelograms of two single units in primary motor cortex exemplifying beta-range rhythmic firing pattern in a parkinsonian animal (SCS OFF). (D) Autocorrelograms of the same two units showing that the rhythmic spiking is effectively interrupted by SCS. (E) The respective power spectra OFF/ON (black/red) for the units shown in (C) and (D). Note the peak (arrow) in the beta-range during the OFF period, which disappears during the ON period. (F) Changes in power of rhythmic beta-firing plotted for all 183 units that presented significant beta oscillations either in the OFF or ON period. Colored circles represent the units with significant suppression in beta power during the ON period. Black line denotes equal power in ON and OFF conditions, thus units located to the right of the line display beta suppression.
structures and was found to be very similar following L-DOPA and SCS (Figure 4B). To further explore the underlying mechanisms whereby SCS alters network activity, we recorded neural activity while delivering single or pairs of SCS pulses. These recordings showed that primary somatosensory pathways (ventral posterolateral thalamic nuclei and primary somatosensory cortex) are activated early by SCS and that a disruption of beta oscillations through a phase-reset mechanism appears to cause the observed widespread desynchronization in the beta band (Figure S4; Fuentes et al., 2010; Popovych and Tass, 2012). DISCUSSION In conclusion, we observed that SCS caused clear clinical improvements in a primate model of PD (comparable to, for example, the reported long-term reduction in UPDRS III score by DBS, 28%, Follett et al., 2010) and, in particular, for motor signs known to be difficult to treat with DBS. These include deficits in posture, gait, and speed of locomotion (Krack et al., 720 Neuron 84, 716–722, November 19, 2014 ª2014 Elsevier Inc.
2003). Concurrent multisite neuronal recordings showed that significant behavioral improvements induced by SCS were strongly associated with desynchronization of neuronal activity within the cortico-basal ganglia circuitry and reduction in betafrequency coherence between structure pairs. We therefore propose that SCS should be further tested in clinical studies aimed at measuring its long-term efficacy as a less invasive, long-term therapy for patients with PD. EXPERIMENTAL PROCEDURES Five adult male common marmosets (Callithrix jacchus) 300–550 g were used in the study. The animals were housed in pairs in cages (1.0 3 1.0 3 2.3 m) in a vivarium with a natural light cycle (12/12 hr) and outdoor temperature. All animal procedures were carried out according to approved protocols by AASDAP Ethics Committee and strictly in accordance with the NIH Guide for the Care and Use of Laboratory Animals (NIH Publications no. 80-23). This project was approved by SISBIO/Brazilian Institute of Environment and Natural Resources under no. 20795-2. Neurotoxic lesions were inflicted with the animals under deep anesthesia. Two microliters of 6-OHDA solution (4 mg/ml, 0.05% ascorbic acid, saline) were injected into the medial forebrain bundle in (AP/ML/DV): 6.5/1.2/6.0;
146
Neuron Spinal Cord Stimulation Alleviates Parkinsonism
Figure 4. Spinal Cord Stimulation and L-DOPA Treatment Suppresses Multistructure LFP Coherence (A) Example of LFP coherence spectra from one of the hemilesioned animals (black trace) showing coherent oscillations restricted to the beta-band in the parkinsonian condition (arrows) that are suppressed by SCS (red trace; bold line and shaded area denote median and interquartile range, stimulation artifacts around 20 and 40 Hz have been removed). (B) Connectivity diagram representing the pooled LFP coherence in the 8–15 Hz range in relation to the 30–40 Hz band between all pairs of electrodes in the different structures (values represent averages from all five recordings in the two hemilesioned animals; all changes in beta-to-gamma coherence for SCS ON/OFF are significant p < 0.05, Wilcoxon rank sum test). Note that the excessive beta-band coherence, represented by the warm colors in the parkinsonian state, is effectively reduced by SCS in the same way as for L-DOPA treatment.
6.5/1.2/7.0; 6.5/2.2/6.5; 6.5/2.2/7.5; and 6.5/3.2/8.0 (Annett et al., 1992). AP coordinates were scaled according to the dimensions of the skull of each animal (Stephan et al., 1980). The following parkinsonian symptoms were assessed in the transparent acrylic box: episodes of freezing, uncoordinated gait, difficulty using fine motor skills, episodes of bradykinesia, hypokinesia, balance impairment, and posture. The assessment methods were based on previously described procedures (Bankiewicz et al., 2001; Campos-Romo et al., 2009; Fahn and Elton, 1987; Verhave et al., 2009) and are thoroughly described in the Supplemental Experimental Procedures. Automatic motion tracking was performed using custom developed software in MATLAB. LFPs and action potentials were recorded using a multi-channel recording system (Plexon). Analyses of recorded signals were performed according to previously described methods (Fuentes et al., 2009; Halje et al., 2012). The position of the recording electrode positions and the extent of dopaminergic lesions were verified through quantitative tyrosine hydroxylase staining in all animals.
Swedish Society for Medical Research; the Olle Engkvist, Parkinson Research, Crafoord, Åke Wiberg, Magnus Bergvall, Kockska and Segerfalk Foundation; Hjärnfonden, and an NIH Transformative award (R01-NS073125-03).
SUPPLEMENTAL INFORMATION
Benabid, A.L., Pollak, P., Louveau, A., Henry, S., and de Rougemont, J. (1987). Combined (thalamotomy and stimulation) stereotactic surgery of the VIM thalamic nucleus for bilateral Parkinson disease. Appl. Neurophysiol. 50, 344–346.
Supplemental Information includes Supplemental Experimental Procedures, four figures, and two movies and can be found with this article online at http://dx.doi.org/10.1016/j.neuron.2014.08.061. AUTHOR CONTRIBUTIONS M.S, H.S, P.H., R.F, P.P, and M.A.L.N. designed the experiments; M.S, H.S, R.F., and P.P performed the surgeries; M.S. conducted experiments; M.S. and M.F. performed immunohistochemistry; M.S., P.H., U.R., P.P., R.F., and M.A.L.N. analyzed the data, and M.S., P.H, U.R, P.P., R.F., and M.A.L.N. wrote the paper. ACKNOWLEDGMENTS We thank Tobias Palmér for developing the video tracking software used for automatic quantification of locomotor behavior, Jim Meloy and Gary Lehew for building recording electrodes, Ivani Brys for statistical discussion, Carlos Eduardo Idalino for support with data analysis, Pedro Calvacanti for support in IHC, and Marcelo Carvalho for technical support. This research was supported by The Michael J. Fox Foundation for Parkinson’s Research; FINEP 01.06.1092.00; INCEMAQ—Program of National Institutes of Science and Technology of CNPq/MCT; the Swedish Research Council (325-2011-6441);
Accepted: August 28, 2014 Published: October 30, 2014 REFERENCES Agari, T., and Date, I. (2012). Spinal cord stimulation for the treatment of abnormal posture and gait disorder in patients with Parkinson’s disease. Neurol. Med. Chir. (Tokyo) 52, 470–474. Annett, L.E., Rogers, D.C., Hernandez, T.D., and Dunnett, S.B. (1992). Behavioural analysis of unilateral monoamine depletion in the marmoset. Brain 115 (Pt 3), 825–856. Bankiewicz, K.S., Sanchez-Pernaute, R., Oiwa, Y., Kohutnicka, M., Cummins, A., and Eberling, J. (2001). Preclinical models of Parkinson’s disease. In Current Protocols in Neuroscience (John Wiley & Sons), pp. 9.4.1–9.4.32.
Campos-Romo, A., Ojeda-Flores, R., Moreno-Briseño, P., and FernandezRuiz, J. (2009). Quantitative evaluation of MPTP-treated nonhuman parkinsonian primates in the HALLWAY task. J. Neurosci. Methods 177, 361–368. Fahn, S., and Elton, R.L. (1987). Unified Parkinson’s disease rating scale. Recent Dev. Park. Dis. 2, 153–163. Fénelon, G., Goujon, C., Gurruchaga, J.-M., Cesaro, P., Jarraya, B., Palfi, S., and Lefaucheur, J.-P. (2012). Spinal cord stimulation for chronic pain improved motor function in a patient with Parkinson’s disease. Parkinsonism Relat. Disord. 18, 213–214. Follett, K.A., Weaver, F.M., Stern, M., Hur, K., Harris, C.L., Luo, P., Marks, W.J., Jr., Rothlind, J., Sagher, O., Moy, C., et al.; CSP 468 Study Group (2010). Pallidal versus subthalamic deep-brain stimulation for Parkinson’s disease. N. Engl. J. Med. 362, 2077–2091. Fuentes, R., Petersson, P., Siesser, W.B., Caron, M.G., and Nicolelis, M.A.L. (2009). Spinal cord stimulation restores locomotion in animal models of Parkinson’s disease. Science 323, 1578–1582. Fuentes, R., Petersson, P., and Nicolelis, M.A. (2010). Restoration of locomotive function in Parkinson’s disease by spinal cord stimulation: mechanistic approach. Eur. J. Neurosci. 32, 1100–1108.
Neuron 84, 716–722, November 19, 2014 ª2014 Elsevier Inc. 721
147
Neuron Spinal Cord Stimulation Alleviates Parkinsonism
Halje, P., Tamtè, M., Richter, U., Mohammed, M., Cenci, M.A., and Petersson, P. (2012). Levodopa-induced dyskinesia is strongly associated with resonant cortical oscillations. J. Neurosci. 32, 16541–16551.
Nicolelis, M.A., Fuentes, R., Petersson, P., Thevathasan, W., and Brown, P. (2010). Spinal cord stimulation failed to relieve akinesia or restore locomotion in Parkinson disease. Neurology 75, 1484, author reply 1484–1485.
Hassan, S., Amer, S., Alwaki, A., and Elborno, A. (2013). A patient with Parkinson’s disease benefits from spinal cord stimulation. J. Clin. Neurosci. 20, 1155–1156.
Popovych, O.V., and Tass, P.A. (2012). Desynchronizing electrical and sensory coordinated reset neuromodulation. Front. Hum. Neurosci. 6, 58.
Krack, P., Batir, A., Van Blercom, N., Chabardes, S., Fraix, V., Ardouin, C., Koudsie, A., Limousin, P.D., Benazzouz, A., LeBas, J.F., et al. (2003). Fiveyear follow-up of bilateral stimulation of the subthalamic nucleus in advanced Parkinson’s disease. N. Engl. J. Med. 349, 1925–1934. Landi, A., Trezza, A., Pirillo, D., Vimercati, A., Antonini, A., and Sganzerla, E.P. (2012). Spinal cord stimulation for the treatment of sensory symptoms in advanced Parkinson’s disease. Neuromodulation 16, 276–279. Mitchell, I.J., Hughes, N., Carroll, C.B., and Brotchie, J.M. (1995). Reversal of parkinsonian symptoms by intrastriatal and systemic manipulations of excitatory amino acid and dopamine transmission in the bilateral 6-OHDA lesioned marmoset. Behav. Pharmacol. 6, 492–507. Morgante, L., Morgante, F., Moro, E., Epifanio, A., Girlanda, P., Ragonese, P., Antonini, A., Barone, P., Bonuccelli, U., Contarino, M.F., et al. (2007). How many parkinsonian patients are suitable candidates for deep brain stimulation of subthalamic nucleus? Results of a questionnaire. Parkinsonism Relat. Disord. 13, 528–531.
722 Neuron 84, 716–722, November 19, 2014 ª2014 Elsevier Inc.
Stein, E., and Bar-Gad, I. (2013). b oscillations in the cortico-basal ganglia loop during parkinsonism. Exp. Neurol. 245, 52–59. Stephan, H., Baron, G., and Schwerdtfeger, W.K. (1980). The Brain of the Common Marmoset (Callithrix jacchus) : A Stereotaxic Atlas. (New York: Springer). Thevathasan, W., Mazzone, P., Jha, A., Djamshidian, A., Dileone, M., Di Lazzaro, V., and Brown, P. (2010). Spinal cord stimulation failed to relieve akinesia or restore locomotion in Parkinson disease. Neurology 74, 1325–1327. Verhave, P.S., Vanwersch, R.A., van Helden, H.P.M., Smit, A.B., and Philippens, I.H.C.H.M. (2009). Two new test methods to quantify motor deficits in a marmoset model for Parkinson’s disease. Behav. Brain Res. 200, 214–219. Williams, D., Tijssen, M., Van Bruggen, G., Bosch, A., Insola, A., Di Lazzaro, V., Mazzone, P., Oliviero, A., Quartarone, A., Speelman, H., and Brown, P. (2002). Dopamine-dependent changes in the functional connectivity between basal ganglia and cerebral cortex in humans. Brain 125, 1558–1569.
148
www.nature.com/scientificreports
OPEN
received: 18 April 2016 accepted: 11 August 2016 Published: 08 September 2016
A Closed Loop Brain-machine Interface for Epilepsy Control Using Dorsal Column Electrical Stimulation Miguel Pais-Vieira1,2,3,*, Amol P. Yadav1,4,*, Derek Moreira1, David Guggenmos1, Amílcar Santos1, Mikhail Lebedev4,5 & Miguel A. L. Nicolelis1,4,5,6,7 Although electrical neurostimulation has been proposed as an alternative treatment for drug-resistant cases of epilepsy, current procedures such as deep brain stimulation, vagus, and trigeminal nerve stimulation are effective only in a fraction of the patients. Here we demonstrate a closed loop brainmachine interface that delivers electrical stimulation to the dorsal column (DCS) of the spinal cord to suppress epileptic seizures. Rats were implanted with cortical recording microelectrodes and spinal cord stimulating electrodes, and then injected with pentylenetetrazole to induce seizures. Seizures were detected in real time from cortical local field potentials, after which DCS was applied. This method decreased seizure episode frequency by 44% and seizure duration by 38%. We argue that the therapeutic effect of DCS is related to modulation of cortical theta waves, and propose that this closedloop interface has the potential to become an effective and semi-invasive treatment for refractory epilepsy and other neurological disorders. Drug-resistant epilepsy constitutes about 22.1% of the total cases of epileptic patients1. Historically, these cases have been treated with surgery2, but more recently electrical neurostimulation has emerged as a potential alternative therapeutic approach3. Deep brain4, vagus5, and trigeminal6,7 nerve stimulation, a procedure pioneered in our laboratory, have been proposed over the past decade as new alternatives to treat refractory epilepsy. However, each of these three alternative therapies has its advantages and disadvantages. For example, deep brain stimulation (DBS) has a success rate of 60% in patients with refractory epilepsy8, but requires extremely invasive brain surgery. Therefore, a smaller number of patients will be eligible for DBS when compared to the other alternative therapies9. Trigeminal nerve stimulation (TNS) is far less invasive than DBS, but has a success rate of only 30.2%6. Lastly, vagus nerve stimulation (VNS) is also less invasive than DBS, but its success rate is the lowest among all three therapies at 24–28% in randomized clinical trials10,11. Electrical stimulation of the posterior funiculus, also known as the dorsal column, of the spinal cord is a semi-invasive method12 which we have demonstrated to be effective for Parkinson’s disease (PD) treatment in rodents13,14 and primates15, and others have shown to be effective in Parkinsonian patients16,17. Remarkably, the neurophysiological hallmark of Parkinson’s disease in animal models is defined by hypersynchronized neuronal activity in the beta band of local field potentials (LFPs)13,15. The LFP patterns observed during these periods of hypersynchronized neuronal activity in Parkinson’s disease resembled some of the patterns of hypersynchronized neuronal activity previously reported in pentylenetetrazol (PTZ) injected rats18. This latter similarity and the fact that this neuronal hypersynchronization can be specifically disrupted by DCS13–15 led us to hypothesize that DCS could be used as an alternative treatment for chronic refractory epilepsy. Although a recent study has demonstrated that DCS improved seizure related activity in anesthetized rats injected with PTZ19, the full clinical 1
Department of Neurobiology Duke University, Durham, NC 27710, USA. 2Centro de Investigação Interdisciplinar em Saúde, Instituto de Ciências da Saúde, Universidade Católica Portuguesa, Porto, Portugal. 3Instituto de Ciências da Vida e da Saúde, Universidade do Minho, Braga, Portugal. 4Department of Biomedical Engineering Duke University, Durham, NC 27710, USA. 5Duke Center for Neuroengineering Duke University, Durham, NC 27710, USA. 6Department of Psychology and Neuroscience Duke University, Durham, NC 27710, USA. 7Edmond and Lily Safra International Institute of Neuroscience of Natal, Natal, Brazil. *These authors contributed equally to this work. Correspondence and requests for materials should be addressed to M.A.L.N. (email: nicoleli@neuro.duke.edu) Scientific Reports | 6:32814 | DOI: 10.1038/srep32814
1
149
www.nature.com/scientificreports/
Figure 1. Closed loop brain-machine interface setup. (A) Local Field Potentials recorded from primary somatosensory cortex are analyzed in real time. High amplitude signals trigger the microstimulator (Master8) which will deliver an electrical pattern to the dorsal columns (DCS). (B) Recording electrodes placement44. (C) Stimulating electrodes placement (resting in the epidural space between the vertebrae and the spinal cord). (D) Raw LFP recording with multiple crossings of pre-established threshold (red dashed lines). The yellow bars indicate DCS delivered whenever the threshold was crossed. Bottom: Spectrogram depicting a seizure episode.
potential of DCS can only be truly addressed in awake animals with DCS being applied in a closed loop mode (i.e. triggered only when a seizure is detected by an alternative measurement, such as cortical neuronal recordings). While PTZ injection may not be the best model to represent the subset of patients with refractory epilepsy20, it has provided the most promising results of DCS as an alternative to current neurostimulation techniques19. Here we developed a closed-loop brain-machine interface (BMI) that utilized chronic cortical implants to detect seizure activity in awake, freely moving PTZ-treated rats (Fig. 1A,B). This BMI also allowed DCS to be delivered using the method we previously developed to suppress Parkinson’s symptoms in rodents13. Overall, we observed that this closed-loop BMI substantially reduced the frequency and duration of seizure episodes.
Results
A total of 10 rats (six male and four female) were implanted with stimulation and recording electrodes. Several days after the animals recovered from this implantation surgery, they were injected with PTZ and the efficacy of our closed-loop BMI in suppressing seizure episodes and reducing their duration was examined in 30 experimental sessions. Cortical microelectrode implants were placed in the primary somatosensory cortex (S1) and used for local field potential recordings (LFPs). Dorsal column stimulation electrodes were placed at the level of vertebral T1-T2 segments)13,14 (Fig. 1C). Two types of experiments were conducted in these 10 animals.
Experiment 1: BMI-On versus BMI-Off. In the first experiment (6 male and 3 female rats; 23 experimental sessions), seizure parameters were measured in PTZ-treated rats either with or without DCS driven by the closed loop BMI (BMI-On and BMI-Off sessions, respectively). In BMI-On sessions, each time a seizure detection threshold was crossed (Fig. 1D), five trains of 200 electrical biphasic pulses (100–200 uAmp) were delivered at the frequency of 500 Hz to the dorsal column. In BMI-Off sessions, the recording and stimulation equipment were connected the same way, but no DCS was delivered to the animals. Injection of PTZ induced characteristic spike and wave discharges (SWDs)21 that were very evident in cortical LFP recordings (Fig. 1D) and triggered body twitches as their main behavioral manifestation. SWD frequency Scientific Reports | 6:32814 | DOI: 10.1038/srep32814
2
150
www.nature.com/scientificreports/
Figure 2. DCS improves seizure related activity. (A) DCS reduced the frequency of seizure episodes. The only case where the frequency of seizures was not reduced (red line, Fig. 2A), corresponds to a session that ended earlier due to technical problems. Symbols X and +correspond each one to a rat with a single BMI-Off or -On session. (B) DCS reduced seizure duration. ‘Partial’ indicates seizures where the BMI was activated only during a fraction of the episode. (C,D) Examples of raw LFP signals and corresponding spectrogram for a BMI-Off and a BMI-On session. During BMI-Off sessions, pre-ictal activity (approximately 1600 seconds) presented a characteristic signature pattern (see text for details). (E) Detail of BMI-Off session presented in C (color code as above). (F,G) In BMI-Off sessions, the pre-ictal theta frequency signal was a good predictor of seizure duration, however during BMI-On sessions, DCS specifically disrupted this signal. Also, note that long seizures (≥60 secs) were mostly absent during BMI-On sessions. typically increased until a seizure episode occurred (Fig. 1D). Once a seizure was detected, our BMI delivered the DCS after each SWD with a 50 ms delay. Comparison of the BMI-On and BMI-Off sessions showed that closed loop DCS affected multiple physiological parameters (Fig. 2A–G). In particular, DCS reduced the overall number of seizure episodes by 44% (BMI-On: 0.05 ± 0.01 episodes/min; BMI-Off: 0.09 ± 0.02 episodes/min; Paired Samples test t = 2.816, df = 5; P = 0.0373; Fig. 2A) as well as the number of SWDs by 72% (BMI-On: 1.8 ± 0.3 SWD/min; Off: 6.5 ± 2.6 SWD/ min; Wilcoxon signed-rank test = 21; P = 0.0313). Additionally, DCS reduced seizure duration by 34.86% (BMI-On: 31.39 ± 2.4 secs; Off: 48.19 ± 3.5 secs; Min: 9 secs; Max: 136 secs; Mann-Whitney U = 551.5; P = 0.0012, Fig. 2B; Partial indicates episodes where the BMI failed to deliver DCS). No differences were found in seizure episode characteristics when rats, tested in the same conditions, were compared across consecutive sessions (paired Scientific Reports | 6:32814 | DOI: 10.1038/srep32814
3
151
www.nature.com/scientificreports/ samples t-test; Duration: T = 0.3314, df = 4; P = 0.7570, n.s.; Frequency: T = 0.48, df = 4; P = 0.67, n.s,), suggesting that the differences between BMI-On and –Off sessions were not due to repeated PTZ administration. Further analysis of the distribution of seizure episode durations showed that DCS negatively skewed this distribution, meaning that long seizure episodes (longer than 60 s) became much less frequent (BMI-On: 1/30 = 3.3% episodes; BMI-Off: 11/62 = 17.74% episodes; Fisher’s exact test: P = 0.048; compare Y axis values in Fig. 2 between F,G). Frequency spectral analysis indicated that DCS specifically disrupted the LFP spectral pattern that preceded the onset of each seizure episode22 in PTZ treated rats. This LFP pattern consisted of an elevated theta band (~4 Hz to 8–10 Hz), which often appeared as a parabola22. These PTZ-related theta episodes, which usually lasted approximately 5–10 s (compare Fig. 2 panels C,D), occurred in a very narrow range of frequencies and occasionally appeared in higher harmonic frequencies (see arrow in Fig. 2C, also 2E). Thus, although pre-ictal activity very often included other bands, spectrogram changes associated with the period occurring immediately before the seizure episode most reliably appeared in the theta frequency. During BMI-On sessions, this PTZ-induced elevated theta band pattern was disrupted. This means that, after the delivery of DCS, the specific parabola pattern was no longer present even when this frequency band still presented a high potency signal. The main effect observed was an increase of LFP power in a wide theta range (4.5–8 Hz) (BMI-On: −27.57 ± 1.4 dB; BMI-Off: −33.01 ± 1.3 dB; t = 2.64, df = 90; P = 0.0098; also see right shift in X axis values in Fig. 2F,G). Lastly, DCS also induced longer periods with reduced pre-ictal theta band power (BMI-On: 2.81 ± 1.81 secs; BMI-Off: 1.51 ± 0.13 secs; Mann-Whitney U = 306; P = 0.0038). Thus, DCS induced a reduction in the proportion of long seizure episodes, an increase in theta power and range (compare Long and Regular in Fig. 2F,G), and allowed for longer periods with low power in the theta band. These findings suggest to us that the elimination of the theta pattern by DCS may have accounted for the mechanism that led to seizure reduction. In support of this theory, we observed that theta band power during the pre-ictal period was a good predictor of longer seizure duration in BMI-Off sessions (F1,55 = 17.09; R2 = 0.23;P < 0.0001: see Fig. 2E,F). By contrast, during BMI-On sessions, theta band power was no longer correlated to seizure duration (BMI-On: F1,28 = 0.32; R2 = 0.01;P = 0.579, n.s.; Fig. 2G).
Experiment 2: Mixed BMI on and off episodes within a session. To test how fast our BMI became effective in reducing PTZ induced seizures, we turned the BMI on and off periodically within the same experimental session. We called these experiments the mixed sessions (N = 7 rats, 4 male and 3 female in seven sessions; see Fig. 3A). Seizure episode durations now varied between 9 and 76 seconds. Under these conditions, we found that our closed-loop BMI still drastically reduced episode duration by 42.15% (BMI-On: 26.5 ± 2.1 secs; BMI-Off: 45.81 ± 3.2 secs; Mann-Whitney U = 74; P < 0.0001; see Fig. 3B; Partial indicates episodes where the BMI failed to deliver DCS). As in the case of the first experiment, pre-ictal theta band power was a good predictor of seizure duration in the mixed sessions when the BMI was off (F1,15 = 5.80; R2 = 0.28; P = 0.0293; Fig. 3C). Once again, when the BMI was on, the theta band power no longer correlated with seizure duration (BMI-On: F1,28 = 0.38; R2 = 0.01; P = 0.54, n.s.). Conspicuously, analysis of long seizure episodes (i.e. ≥60 seconds) now revealed that turning the BMI on in a fraction of the seizure episodes significantly reduced the number of these long seizures even when the BMI was off (BMI-On: 0/30 = 0% episodes; BMI-Off: 2/17 = 11.8%; Fisher’s exact test: P = 0.145, n.s.; also see Fig. 3C, compare Regular to Long). This finding suggested that, during the course of a PTZ session, the delivery of the DCS pattern during one seizure episode could, to some extent, affect the characteristics of the following episode13–15, even if no DCS was delivered at that particular episode. In other words, we found evidence for a long-lasting effect of DCS, similar to what we had reported before when we used DCS to treat rat and monkey models of Parkinson’s disease13–15. To further test this possibility, we looked at the characteristics of the pre-ictal theta band signal, which in Experiment 1 was very different between BMI-On and BMI-Off sessions, during the mixed sessions. In this latter case we found that, not only was the pre-ictal theta band signal potency now similar between BMI-On and BMI-Off seizure episodes (BMI-On: 30.50 ± 1.5 dB; BMI-Off:27.25 ± 2.2 dB; paired samples t test t = 1.452,df = 6; P = 0.1968, n.s.), but that the theta band amplitude signals obtained during the BMI-Off episodes were now closer to those measured during BMI-On episodes (also compare values in X axis in Fig. 2F to values in Fig. 3C). Lastly, analysis of low power theta band durations (which in experiment 1 were smaller for BMI Off episodes), also revealed that these were now similar between BMI-On and BMI-Off episodes (BMI-On: 2.31 ± 0.34 secs; BMI-Off: 2.44 ± 0.38 secs; Man-Whitney U = 163.5, P = 0.7084, n.s.). Therefore, these results suggest that, in this experiment, BMI-Off episodes where, to some extent, affected by DCS delivered during the BMI-On episodes. DCS is effective in both male and female rats. To identify possible gender specific differences in our results23, we further pooled male or female rats from both experiments and compared the main findings of this study according to animal gender. The use of DCS reduced the overall duration of seizure episodes in both male (t-test with Welch’s correction, t = 4.665, df = 59, P < 0.0001) and female rats (t-test with Welch’s correction, t = 3.563, df = 59, P = 0.0007). Additionally, the pre-ictal theta band signal was predictive of seizure episodes in both male (BMI-Off: F1,42 = 13.37; R2 = 0.24;P = 0.0007) and female rats (BMI-Off: F1,33 = 4.357; R2 = 0.12; P = 0.0447) when DCS was Off, but not when it was On (BMI-On Male: F1,29 = 0.42; R2 = 0.01;P = 0.52, n.s.; BMI-On Female: F1,28 = 0.30; R2 = 0.01;P = 0.59, n.s.).
Discussion
We demonstrated here the efficacy of a closed-loop BMI that triggered DCS in response to pre-seizure and seizure patterns in LFP activity in PTZ-treated rats. Overall, we observed that our BMI quite effectively reduced the number of seizure episodes, and their duration, while also changing the overall pattern of LFP activity associated Scientific Reports | 6:32814 | DOI: 10.1038/srep32814
4
152
www.nature.com/scientificreports/
Figure 3. Intermittent delivery of DCS improves seizure related activity. (A) Example of session where the BMI was turned On or Off in successive seizure episodes. (B) DCS reduced seizure duration. Partial indicates seizure episodes where DCS was delivered only in a fraction of the episode. (C) In BMI-Off episodes, the preictal theta frequency signal was a good predictor of seizure duration. During BMI-On episodes, DCS specifically disrupted this signal. Also, note that long seizures (≥60 secs) were absent during BMI-On episodes and during BMI-Off episodes as well (see text for details).
with the pre-ictal phase of PTZ-triggered seizures. Therefore, we propose that the main anti-seizure effect of DCS is obtained via the reduction in the pre-ictal theta band activity, a good predictor of seizure duration. Lastly, we found that our BMI was effective in both male and female rats, even though our experiments were not controlled for the estrous cycle23. Previous studies have shown increases as well as decreases in epileptic related activity after treatment with DCS19,24. Here, we have specifically used DCS in response to a change in LFPs signal and consistently observed improvement in multiple physiological parameters. We attribute the differences between our findings and previous results to the fact that we delivered DCS only in response to LFP changes instead of stimulating indiscriminately24. Another important factor is that we only employed high frequency DCS in the present study, since we and others have observed (D.G.: personal observation) increased seizure activity when low frequency DCS was delivered24. Using transcranial electrical stimulation in a different model, Berenyi et al. have developed a closed loop BMI for epilepsy21. While that study was able to achieve reductions in seizure related activity somewhat higher than the ones achieved here, it is important to note that they used a different chemical agent. Future studies comparing different BMI approaches and epilepsy models will help identifying pros and cons, as well as efficacy, of each technique. At this point it is important to recall that the PTZ model – as used here - may not be the best animal model to represent the subset of patients with refractory epilepsy20. Therefore, the effects of our closed loop BMI will have to be further tested in other animal models of epilepsy. It could be argued that the differences found between BMI-On and -Off seizure episodes reported here could be the result of differences originating from repeated PTZ administration. Although repeated administration of PTZ is often used as a model for chronic seizures (see Erkeç and Arihan 2015 for a review)25 our results cannot be explained by such effect alone. First, in experiment 1, not only BMI-On and -Off sessions were typically alternated, but some rats started with BMI-Off sessions while others started with BMI-On sessions. Second, there was no difference in seizure episodes (duration and frequency) in rats tested in the same conditions in consecutive sessions. Third, most PTZ kindling protocols involve more than 10 doses of PTZ25–27 or intervals of more than 20 days between the series of PTZ injections28. Lastly, results from experiment 2 (where the BMI was turned On and Off within the same session) further controlled for the possibility of differences in BMI-On and -Off sessions being the result of PTZ-induced kindling alone. This study also reports, for the first time, that the pre-ictal theta band signal can be used as good predictor for seizure episode duration in PTZ-treated rats. It is not yet clear why this signal is related to the duration of the Scientific Reports | 6:32814 | DOI: 10.1038/srep32814
5
153
www.nature.com/scientificreports/ seizure episodes, but a possible explanation involves a mechanism where a state of seizure derives from an abnormal transition between brain states resulting from an imbalance between corticofugal inhibition and thalamocortical excitation. In healthy rodents, strong increases in theta band signal, with partial increases in other frequencies, are also present during whisker twitching29,30, a non-pathological seizure-like state controlled by S1 that is characterized by general immobility coupled with improved ability to detect incoming tactile stimuli. From this cortically controlled state, neural activity does not usually evolve to seizure episodes, but rather transitions to a state of quiet waking where the animal is either immobile or engaged in stereotyped behaviors31. In a rat model of cortical injury generated epilepsy, an initial stimulus from the injured area to the thalamus will make the thalamocortical loop transition to a hypersynchronization state characterized by seizures32. This state is maintained by the thalamus and can be reversed by optogenetic thalamic stimulation32. Together, these findings suggest that many of the differences found in theta power in this and previous studies could be the result of this critical balance in the thalamocortical loop where S1 corticofugal inhibition33 maintains theta oscillations within a normal range (i.e. whisker twitching), but that otherwise, if theta oscillations become mostly dependent on a hyper excitable thalamus32,34, this state will then transition to a state of hypersynchronized seizure activity. Note that such a mechanism could additionally explain the differences found in previous DCS studies. Thus, if theta band activity critically reflects a balance between thalamic and cortical activity, electrical stimulation to the lemniscal pathway could result in thalamic increased excitability or, if sufficiently strong, it could in addition stimulate S1 and increase cortico-thalamic inhibition33. The differences in seizure related activity found when DCS was applied with low or high frequencies can be partially explained within this framework. Low frequency DCS would increase seizure activity24 because it should affect mostly thalamocortical synapses, while increasing stimulation frequency should be able to increase both the thalamus and S1 (Supplementary Figure S1), therefore activating the corticofugal synapses and improving seizure related activity (ref. 19 and here). Lastly, this critical balance between S1 inhibition and thalamic excitation could also explain the predictive power of the pre-ictal signal. The observation that S1 controls the state of whisker twitching (which is characterized by high potency theta oscillations) suggests that the bimodal distribution of the predictive pre-ictal theta signal found here may actually correspond to two different brain states resulting from the initial conditions imposed by the pre-ictal theta signal (one prone to long seizures and another one prone to short seizures). In this scenario, a theta signal smaller than −45 dB in S1 (Fig. 2F) after the injection of PTZ would constitute the critical potency required to disrupt the balance in the thalamocortical loop, transitioning to a state of long seizures, while a theta signal larger than −45 dB, while still disrupting the balance in the thalamocortical loop, would promote transition to a state where short seizures occur. It is important to note however that DCS activates multiple regions, making it unlikely that the proposed mechanism would be the only source for the brain state transitions described. Future studies involving recordings and stimulation across the thalamocortical loop will allow dissecting to what extent DCS affects this theta signal in each structure as well as its significance in different thalamocortical states. Electrical neural stimulation has been used as an alternative to surgery for intractable epilepsy cases, primarily through deep brain stimulation, vagal nerve stimulation, and trigeminal nerve stimulation. Deep brain stimulation has presented an efficacy of up to 68% responders after 5 years35 but is extremely invasive and cannot be performed in many patients. Meanwhile, vagal and trigeminal nerve stimulation procedures, have demonstrated relatively low efficacy, with 49%36 and 50%37 respectively, and may have more side effects38 than deep brain stimulation. Thus, on one side, deep brain stimulation has achieved very good seizure reduction, but is extremely invasive and expensive, making it a solution for only a small fraction of the patients in need. On the other side, more peripheral stimulation procedures, which are much less expensive and invasive, have a much lower efficacy rate and are associated with increased side effects. Finally, DCS seems to rest in the middle ground between these other electrical neurostimulation alternatives, since it’s less invasive than deep brain stimulation and few side effects have been reported when DCS is used for other diseases39. Translating our findings into human patients will allow comparison of the efficacy rate between DCS and these other alternatives. Previously we have shown that DCS ameliorates symptoms of Parkinson’s disease by desynchronizing pathological low frequency corticostriatal oscillations, therefore creating a brain state permissible for the initiation of locomotion in severely dopamine depleted rodents and non-human primates13,15. More specifically, high frequency DCS inhibited oscillatory neuronal activity synchronized at the beta frequency in the basal ganglia through activation of various structures along the dorsal column medial lemniscal pathway15. Our present PTZ results are in line with these previous Parkinson’s disease studies, by suggesting that DCS is responsible for the desynchronization of pathological synchronous activity characteristic of the PTZ model of epilepsy. In fact, this observation can be generalized to other disorders, since we and others have demonstrated that pathological synchronous activity seems to be the hallmark of pathological brain states recorded from multiple models of neurological and neuropsychiatric disorders such as Parkinson’s disease13–15, epilepsy (here and ref. 19), bipolar disorder40, and schizophrenia41. Therefore, based on this cumulative body of evidence showing abnormal timing in brain circuitry, we propose that the aforementioned diseases can all be classified within a broad spectrum of pathological timing brain states (i.e. hyper- or hyposynchronized), that resemble the basic neurophysiological hallmarks of epilepsy. While these diseases share excessive synchronization as a common feature, they differ in the type of neural circuits involved in each case42. A testable prediction of this hypothesis would be that any type of nerve stimulation capable of significantly altering the balance between regions responsible for these synchronizations, should also be able to induce at least a partial relief of symptoms in these disorders42,43. In conclusion, we propose that DCS should be tested in other rodent and primate models of chronic epilepsy to measure its efficacy in controlling these pathological brain states over longer periods of time. These studies would be essential to determine the true potential of DCS as a non-pharmacologic alternative therapy for use in humans suffering from chronic, untreatable epilepsy.
Scientific Reports | 6:32814 | DOI: 10.1038/srep32814
6
154
www.nature.com/scientificreports/
Methods
All animal procedures were performed in accordance with the National Research Council’s Guide for the Care and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use Committee. Long Evans male and female rats weighing between 250–400 g were used in all experiments.
Surgery for microelectrode array implantation. Animals went through two different surgeries: one to
implant recording electrodes and the other to implant the stimulation electrodes. Recording electrodes: Fixed or movable microelectrode bundles or arrays of electrodes were implanted in the S1 of rats and additional regions (for the present study we did not evaluate the activity in other regions). Anesthesia was induced with 5% halothane, and maintained with ketamine (100 mg/kg), xylazine (10 mg/kg) and atropine (0.05 ml). Craniotomies for S1 recordings were made and arrays lowered at the following stereotaxic coordinates: [(AP) −3.5 mm, (ML), −5.5 mm (DV) −1.5 mm]44. Stimulation electrodes for spinal stimulation were also implanted under anesthesia as described above. Postoperative weight was monitored daily. The implantation procedure was performed as previously described14. Specifically, stimulation electrodes were inserted in the epidural space under thoracic vertebra T2 and, to prevent electrode migration, were tied to it with surgical suture.
Electrophysiological recordings. A Multineuronal Acquisition Processor (64 channels, Plexon Inc, Dallas, TX) was used to record neuronal spikes, as previously described45. Briefly, neural signals were recorded differentially, amplified (20,000–32,000X), filtered (filtering band between 400 Hz and 5 kHz), and digitized at 40 kHz. Local field potentials (LFPs) were acquired by band-pass filtering the raw signal (0.3–400.0 Hz), preamplified (1,000), and digitized at 1,000 Hz using a digital acquisition card (National Instruments, Austin, TX) and a multineuronal acquisition processor (Plexon). Pentylenetetrazole administration. Each recording session, independently of the experiment, was per-
formed on a different day. Both male and female rats were tested under the exact same conditions. PTZ (SIGMA Aldrich) administration was prepared by dilution of 100 mg/kg of PTZ in 1 ml saline. This was then administered IP under isofluorane anesthesia. As BMI sessions were preceded by an initial baseline recording period, rats injected with the PTZ could be immediately brought to the recording room with a delay of no more than 5 minutes. BMI sessions started approximately 5–10 minutes after the administration of PTZ. The recording sessions (in both experiments) lasted 60 minutes. In preliminary experiments we observed that PTZ effects were less variable within the first 60–90 minutes.
Data analysis. Neuronal data obtained from a total of 30 recording sessions was processed and analyzed
using NeuroExplorer (version 3.266; NEX Technologies, Madison, AL) and custom scripts written in Matlab (12.0; Mathworks, Natick, MA). A seizure episode was defined as a period where observable muscle spasms and high amplitude oscillations in raw LFP trace, were accompanied by increased power across multiple frequency bands. Seizure episodes were initially identified during the session using both behavior and raw LFP traces as indicators, and later confirmed through detailed reanalysis of raw LFP traces and spectrograms. Comparison of seizure and SWD frequencies (calculated in seizure episodes or SWD events per minute) was made using a paired samples t test or Wilcoxon signed ranks test. When an animal had more than one BMI-On or BMI-Off session, a single value resulting from the mean of the sessions was used for comparison. Seizure duration was compared using the Mann-Whitney test. Analysis of the overall distribution of seizure durations indicated a bimodal distribution. Accordingly, seizure episodes were analyzed as Long (≥60 secs) or Regular (<6 0 secs). Then the proportion of Regular and Long seizure episodes was calculated for BMI-On and BMI-Off episodes. Lastly, the proportion of Regular and Long seizure episodes was compared using Fisher’s exact test. These calculations were performed separately for each experiment. For comparison of pre-ictal theta band spectrogram power we used values from 4.5–8 Hz frequencies in the 5 seconds before the timestamp that was identified as the start of the seizure episode. Theta power was calculated from the original signal processed in Neuroexplorer, followed by processing with custom scripts written in Matlab. Values were normalized with the Log of power spectral density (dB) and initially analyzed in bins of 100 ms. Calculation of theta power for correlation was made using a single 5 second bin (the 5 seconds immediately before seizure onset) for theta frequency that was then correlated to seizure episode duration. For ease of presentation, spectrograms are presented in bins of 100 ms and smoothed with a Gaussian filter of 300 ms. Statistical comparison of pre-ictal theta band power was made using an independent (Experiment 1) or paired (Experiment 2) samples t test. Pearson correlation was calculated using seizure episode duration and the pre-ictal theta power. The duration of theta band potency decrease was compared using data from the spectrogram of the whole session initially processed in 1 second bins in Neuroexplorer. A Zscore was calculated for theta band frequency for each bin across the session, and then periods of 5 seconds before the onset of each seizure episode were analyzed. As increased theta band Zscores were present almost exclusively during seizure episodes or spike-and-wave discharges, we analyzed instead periods where Zscores decreased (i.e. indicating a low potency theta band signal). Decreases in theta potency were considered here as a Zscore equal or below 1.0 standard deviation. The duration of each response was then considered as the number of consecutive bins where the potency of the signal corresponded to this criterion. Lastly, duration of low theta power responses was compared between BMI-On and BMI-Off episodes in each separate experiment, using the Mann-Whitney U test.
Brain-machine interface based on Dorsal Column Stimulation. Our brain-machine interface used Dorsal Column Stimulation (DCS) cues that were generated by an electrical microstimulator (Master 8. AMPI, Jerusalem, Israel) controlled by a custom Matlab script (Natick, USA) receiving information from a Plexon system over the internet. This real-time neural analysis and stimulation system has been previously described for a different purpose46,47. Here, we have pre-determined for each rat, a threshold in raw LFP traces that was typically crossed only in the presence of LFP epileptic activity (i.e. spike and wave discharges or seizure episodes). Upon Scientific Reports | 6:32814 | DOI: 10.1038/srep32814
7
155
www.nature.com/scientificreports/ detection of such threshold crossing, a pattern of 200 (bipolar, biphasic, charge balanced; 200 μsec) pulses at 500 Hz was delivered to the dorsal column of the spinal cord at the level of T1-T2 segments. Current intensity varied from 100–200 μA. Seizure episodes where DCS failed to stimulate for at least 75% of the episode duration were considered as ‘Partial’ stimulation and were excluded from final analysis. These included a total of 6/53 = 11.32% episodes in Experiments 1 and 2. In Experiment 1 rats were typically tested in BMI-On and BMI-Off sessions on alternate days. Similarly, in Experiment 2, rats seizure episodes with BMI-On were alternated with BMI-Off episodes. Changes to these pre-established conditions were made when technical problems occurred (e.g. incomplete session, cable disconnecting, noise, inadequate threshold etc.).
References
1. Picot, M. C., Baldy-Moulinier, M., Daures, J. P., Dujols, P. & Crespel, A. The prevalence of epilepsy and pharmacoresistant epilepsy in adults: a population-based study in a Western European country. Epilepsia 49, 1230–1238, doi: 10.1111/j.1528-1167.2008.01579.x (2008). 2. Horsley, V. B-surgery. Br. Med. J. 2, 670–675 (1886). 3. Mogul, D. J. & van Drongelen, W. Electrical control of epilepsy. Annu. Rev. Biomed. Eng. 16, 483–504, doi: 10.1146/annurevbioeng-071813-104720 (2014). 4. Fisher, R. et al. Electrical stimulation of the anterior nucleus of thalamus for treatment of refractory epilepsy. Epilepsia 51, 899–908, doi: 10.1111/j.1528-1167.2010.02536.x (2010). 5. De Herdt, V. et al. Vagus nerve stimulation for refractory epilepsy: a Belgian multicenter study. Eur. J. Paediatr. Neurol. 11, 261–269, doi: 10.1016/j.ejpn.2007.01.008 (2007). 6. DeGiorgio, C. M. et al. Randomized controlled trial of trigeminal nerve stimulation for drug-resistant epilepsy. Neurology 80, 786–791, doi: 10.1212/WNL.0b013e318285c11a (2013). 7. Fanselow, E. E., Reid, A. P. & Nicolelis, M. A. Reduction of pentylenetetrazole-induced seizure activity in awake rats by seizuretriggered trigeminal nerve stimulation. J. Neurosci. 20, 8160–8168 (2000). 8. Bergey, G. K. et al. Long-term treatment with responsive brain stimulation in adults with refractory partial seizures. Neurology 84, 810–817, doi: 10.1212/WNL.0000000000001280 (2015). 9. Morgante, L. et al. How many parkinsonian patients are suitable candidates for deep brain stimulation of subthalamic nucleus? Results of a questionnaire. Parkinsonism Relat. Disord. 13, 528–531, doi: 10.1016/j.parkreldis.2006.12.013 (2007). 10. Morrell, M. J. & Group, R. N. S. S. i. E. S. Responsive cortical stimulation for the treatment of medically intractable partial epilepsy. Neurology 77, 1295–1304, doi: 10.1212/WNL.0b013e3182302056 (2011). 11. Vagus Nerve Stimulation Study Group. A randomized controlled trial of chronic vagus nerve stimulation for treatment of medically intractable seizures. Neurology 45, 224–230 (1995). 12. Shealy, C. N., Mortimer, J. T. & Reswick, J. B. Electrical inhibition of pain by stimulation of the dorsal columns: preliminary clinical report. Anesth. Analg. 46, 489–491 (1967). 13. Fuentes, R., Petersson, P., Siesser, W. B., Caron, M. G. & Nicolelis, M. A. Spinal cord stimulation restores locomotion in animal models of Parkinson’s disease. Science 323, 1578–1582, doi: 10.1126/science.1164901 (2009). 14. Yadav, A. P. et al. Chronic spinal cord electrical stimulation protects against 6-hydroxydopamine lesions. Sci. Rep. 4, 3839, doi: 10.1038/srep03839 (2014). 15. Santana, M. B. et al. Spinal cord stimulation alleviates motor deficits in a primate model of Parkinson disease. Neuron 84, 716–722, doi: 10.1016/j.neuron.2014.08.061 (2014). 16. Agari, T. & Date, I. Spinal cord stimulation for the treatment of abnormal posture and gait disorder in patients with Parkinson’s disease. Neurol. Med. Chir. (Tokyo). 52, 470–474 (2012). 17. Fenelon, G. et al. Spinal cord stimulation for chronic pain improved motor function in a patient with Parkinson’s disease. Parkinsonism Relat. Disord. 18, 213–214, doi: 10.1016/j.parkreldis.2011.07.015 (2012). 18. Zhang, T. et al. Pre-seizure state identified by diffuse optical tomography. Sci. Rep. 4, 3798, doi: 10.1038/srep03798 (2014). 19. Jiao, J., Jensen, W., Harreby, K. R. & Sevcencu, C. The Effect of Spinal Cord Stimulation on Epileptic Seizures. Neuromodulation 19, 154–160, doi: 10.1111/ner.12362 (2016). 20. Loscher, W. Critical review of current animal models of seizures and epilepsy used in the discovery and development of new antiepileptic drugs. Seizure 20, 359–368, doi: 10.1016/j.seizure.2011.01.003 (2011). 21. Berenyi, A., Belluscio, M., Mao, D. & Buzsaki, G. Closed-loop control of epilepsy by transcranial electrical stimulation. Science 337, 735–737, doi: 10.1126/science.1223154 (2012). 22. Schevon, C. A. et al. Evidence of an inhibitory restraint of seizure activity in humans. Nat.Commun. 3, 1060, doi: 10.1038/ ncomms2056 (2012). 23. Christensen, J., Kjeldsen, M. J., Andersen, H., Friis, M. L. & Sidenius, P. Gender differences in epilepsy. Epilepsia 46, 956–960, doi: 10.1111/j.1528-1167.2005.51204.x (2005). 24. Harreby, K. R., Sevcencu, C. & Struijk, J. J. The effect of spinal cord stimulation on seizure susceptibility in rats. Neuromodulation 14, 111–116; discussion 116, doi: 10.1111/j.1525-1403.2010.00320.x (2011). 25. Erkec, O. E. & Arihan, O. Pentylenetetrazole Kindling Epilepsy Model. Epilepsi 21, 6–12, doi: 10.5505/epilepsi.2015.08108 (2015). 26. Corda, M. G. et al. Pentylenetetrazol-induced kindling in rats: effect of GABA function inhibitors. Pharmacol. Biochem. Behav. 40, 329–333 (1991). 27. Ilhan, A., Iraz, M., Kamisli, S. & Yigitoglu, R. Pentylenetetrazol-induced kindling seizure attenuated by Ginkgo biloba extract (EGb 761) in mice. Prog Neuropsychopharmacol Biol. Psychiatry 30, 1504–1510, doi: 10.1016/j.pnpbp.2006.05.013 (2006). 28. Davoudi, M., Shojaei, A., Palizvan, M. R., Javan, M. & Mirnajafi-Zadeh, J. Comparison between standard protocol and a novel window protocol for induction of pentylenetetrazol kindled seizures in the rat. Epilepsy Res 106, 54–63, doi: 10.1016/j. eplepsyres.2013.03.016 (2013). 29. Fanselow, E. E., Sameshima, K., Baccala, L. A. & Nicolelis, M. A. Thalamic bursting in rats during different awake behavioral states. Proc. Natl. Acad. Sci. USA 98, 15330–15335, doi: 10.1073/pnas.261273898 (2001). 30. Nicolelis, M. A., Baccala, L. A., Lin, R. C. & Chapin, J. K. Sensorimotor encoding by synchronous neural ensemble activity at multiple levels of the somatosensory system. Science 268, 1353–1358 (1995). 31. Gervasoni, D. et al. Global forebrain dynamics predict rat behavioral states and their transitions. J. Neurosci. 24, 11137–11147, doi: 10.1523/JNEUROSCI.3524-04.2004 (2004). 32. Paz, J. T. et al. Closed-loop optogenetic control of thalamus as a tool for interrupting seizures after cortical injury. Nat. Neurosci. 16, 64–70, doi: 10.1038/nn.3269 (2013). 33. Schlaier, J. R. et al. Effects of spinal cord stimulation on cortical excitability in patients with chronic neuropathic pain: a pilot study. Eur. J. Pain 11, 863–868, doi: 10.1016/j.ejpain.2007.01.004 (2007). 34. Fanselow, E. E. & Nicolelis, M. A. Behavioral modulation of tactile responses in the rat somatosensory system. J. Neurosci. 19, 7603–7616 (1999).
Scientific Reports | 6:32814 | DOI: 10.1038/srep32814
8
156
www.nature.com/scientificreports/ 35. Salanova, V. et al. Long-term efficacy and safety of thalamic stimulation for drug-resistant partial epilepsy. Neurology 84, 1017–1025, doi: 10.1212/WNL.0000000000001334 (2015). 36. Englot, D. J., Rolston, J. D., Wright, C. W., Hassnain, K. H. & Chang, E. F. Rates and Predictors of Seizure Freedom With Vagus Nerve Stimulation for Intractable Epilepsy. Neurosurgery, doi: 10.1227/NEU.0000000000001165 (2015). 37. Zare, M. et al. Trigeminal nerve stimulation: A new way of treatment of refractory seizures. Adv. Biomed. Res. 3, 81, doi: 10.4103/2277-9175.127994 (2014). 38. Kahlow, H. & Olivecrona, M. Complications of vagal nerve stimulation for drug-resistant epilepsy: a single center longitudinal study of 143 patients. Seizure 22, 827–833, doi: 10.1016/j.seizure.2013.06.011 (2013). 39. Hayek, S. M., Veizi, E. & Hanes, M. Treatment-Limiting Complications of Percutaneous Spinal Cord Stimulator Implants: A Review of Eight Years of Experience From an Academic Center Database. Neuromodulation 18, 603–608; discussion 608-609, doi: 10.1111/ ner.12312 (2015). 40. Dzirasa, K. et al. Impaired limbic gamma oscillatory synchrony during anxiety-related behavior in a genetic mouse model of bipolar mania. J. Neurosci. 31, 6449–6456, doi: 10.1523/JNEUROSCI.6144-10.2011 (2011). 41. Dzirasa, K. et al. Dopaminergic control of sleep-wake states. J. Neurosci. 26, 10577–10589, doi: 10.1523/JNEUROSCI.1767-06.2006 (2006). 42. Cicurel, R. & Nicolelis, M. A. L. The relativistic brain: how it works and why it cannot by simulated by a Turing machine 1.1 edn, ISBN: 1511617020 (Kios Press, 2015). 43. Nicolelis, M. A. L. Beyond boundaries: the new neuroscience of connecting brains with machines-and how it will change our lives 1st edn, (Times Books/Henry Holt and Co., 2011). 44. Paxinos, G. & Watson, C. The rat brain in stereotaxic coordinates. 4th edn, (Academic Press, 1998). 45. Nicolelis, M. A. L. Methods for neural ensemble recordings. 2nd edn, (CRC Press, 2008). 46. Pais-Vieira, M., Chiuffa, G., Lebedev, M., Yadav, A. & Nicolelis, M. A. Building an organic computing device with multiple interconnected brains. Sci. Rep. 5, 11869, doi: 10.1038/srep11869 (2015). 47. Pais-Vieira, M., Lebedev, M., Kunicki, C., Wang, J. & Nicolelis, M. A. A brain-to-brain interface for real-time sharing of sensorimotor information. Sci. Rep. 3, 1319, doi: 10.1038/srep01319 (2013).
Acknowledgements
We are grateful for the assistance from Jim Meloy for the design and production of the multielectrode arrays as well as setup development and maintenance, Laura Oliveira, Terry Jones, and Susan Halkiotis for administrative assistance and preparation of the manuscript. This work was funded by a grant from The Hartwell Foundation.
Author Contributions
M.P.-V., A.P.Y., M.L. and M.A.L.N. designed the experiments; M.P-V., A.P.Y., D.M., A.S. and D.G. performed the experiments; M.P-V., A.P.Y. and M.A.L.N. analyzed the data; M.P-V., A.P.Y., M.L. and M.A.L.N. wrote the manuscript.
Additional Information
Supplementary information accompanies this paper at http://www.nature.com/srep Competing financial interests: The authors declare no competing financial interests. How to cite this article: Pais-Vieira, M. et al. A Closed Loop Brain-Machine Interface For Epilepsy Control Using Dorsal Column Electrical Stimulation. Sci. Rep. 6, 32814; doi: 10.1038/srep32814 (2016). This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ © The Author(s) 2016
Scientific Reports | 6:32814 | DOI: 10.1038/srep32814
9
157
Creating a New Sense: The Infrared Rat Studies
158 IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 1, JANUARY 2012
85
Virtual Active Touch Using Randomly Patterned Intracortical Microstimulation Joseph E. O’Doherty, Mikhail A. Lebedev, Zheng Li, and Miguel A. L. Nicolelis
Abstract—Intracortical microstimulation (ICMS) has promise as a means for delivering somatosensory feedback in neuroprosthetic systems. Various tactile sensations could be encoded by temporal, spatial, or spatiotemporal patterns of ICMS. However, the applicability of temporal patterns of ICMS to artificial tactile sensation during active exploration is unknown, as is the minimum discriminable difference between temporally modulated ICMS patterns. We trained rhesus monkeys in an active exploration task in which they discriminated periodic pulse-trains of ICMS (200 Hz bursts at a 10 Hz secondary frequency) from pulse trains with the same average pulse rate, but distorted periodicity (200 Hz bursts at a variable instantaneous secondary frequency). The statistics of the aperiodic pulse trains were drawn from a gamma distribution with mean inter-burst intervals equal to those of the periodic pulse trains. The monkeys distinguished periodic pulse trains from aperiodic pulse trains with coefficients of variation 0.25 or greater. Reconstruction of movement kinematics, extracted from the activity of neuronal populations recorded in the sensorimotor cortex concurrent with the delivery of ICMS feedback, improved when the recording intervals affected by ICMS artifacts were removed from analysis. These results add to the growing evidence that temporally patterned ICMS can be used to simulate a tactile sense for neuroprosthetic devices. Index Terms—Bidirectional interface, brain–machine interface, intracortical microstimulation, neural prosthesis.
I. INTRODUCTION
S
ENSORY neuroprostheses and sensory substitution systems for the restoration of hearing [1], [2] and vision [3]–[8] have been investigated for several decades. Interest in neuroprosthetic devices that combine both motor and sensory components has developed more recently [9]–[14]. One example of a bidirectional neuroprosthesis is a robotic limb Manuscript received February 01, 2011; revised April 30, 2011, June 16, 2011; accepted July 03, 2011. Date of publication December 27, 2011; date of current version January 25, 2012. This work was supported in part by the Defense Advanced Research Projects Agency (DARPA) under Grant N66001-06-C-2019, in part by the Telemedicine and Advanced Technology Research Center (TATRC) under Grant W81XWH-08-2-0119, and in part by the National Institutes of Health through NICHD/OD under Grant RC1HD063390. The work of M. A. L. Nicolelis was supported by the NIH Director’s Pioneer Award Program under Grant DP1OD006798. J. E. O’Doherty is with the Department of Physiology and the W. M. Keck Foundation Center for Integrative Neuroscience, University of California, San Francisco, CA 94143 USA (e-mail: joeyo@phy.ucsf.edu). M. A Lebedev and Z. Li are with the Department of Neurobiology and the Center for Neuroengineering, Duke University, Durham, NC 27710 USA (e-mail: lebedev@neuro.duke.edu; zheng@cs.duke.edu). M. A. L. Nicolelis is with the Departments of Neurobiology, Biomedical Engineering, Psychology, and the Center for Neuroengineering, Duke University, Durham, NC 27710 USA (e-mail: nicoleli@neuro.duke.edu). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TNSRE.2011.2166807
controlled by brain activity while sensory information from prosthetic sensors is delivered to somatosensory areas of the brain [15], [16]. Other possible implementations include sensorized neuroprostheses for the restoration of bipedal walking [17] and putative systems combining both speech production [18], [19] and hearing [1], [2], [20]. In recent years, we have been studying intracortical microstimulation (ICMS) delivered through microelectrode arrays chronically implanted in the primary somatosensory cortex (S1) as a means of adding a somatosensory feedback loop to a brain–machine interface (BMI) [9], [10], [21]. Taken together with previous work showing that primates [22]–[24] and rodents [25]–[28] can discriminate ICMS patterns, there is growing evidence that ICMS of S1 could equip neuroprosthetic limbs with the sense of touch. One of the neuroprosthetic devices that we envision in the future is a BMI-operated robotic arm that is equipped with touch sensors [9], [15], [16]. In such a sensorized neuroprosthesis, the touch sensors would detect instances when the arm interacts with external objects sending signals to the brain in the form of ICMS. We have suggested that long-term operation of such a system, which we call a brain–machine–brain interface (BMBI), could result in the incorporation of the prosthesis into the brain’s representation of the body, so that the artificial limb starts to act and feel as belonging to the subject [15]. Notwithstanding initial encouraging results [9], [10], it is unclear whether ICMS would be sufficient to reproduce the rich sensory information of the world of touch [29]. In particular, it is not well understood which kinds of ICMS patterns are most useful for virtual active touch. Previously, we have shown that both New World [21] and Old World monkeys [10] can discriminate temporal ICMS patterns applied to S1 that consist of short (50–300 ms) high-frequency (100–400 Hz) pulse-trains presented at a lower secondary frequency (2–10 Hz). In these experiments, ICMS served as a cue that instructed the direction of reach. These patterns of ICMS could, in principle, mimic a wide variety of tactile inputs, especially when combined with spatial encoding [21]. Modulations of sensory inputs in this frequency range correspond to the sensation of flutter [30]–[32]. These timescales are also similar to neuronal modulations involved in texture encoding in the somatosensory system [33]–[36], which makes such ICMS patterns worthy candidates for exploration. In this study, we examined the ability of rhesus monkeys to discriminate a range of temporal ICMS patterns applied to S1 in the context of an active exploration task in which ICMS mimicked the tactile properties of virtual objects. We manipulated the ICMS patterns in a graded fashion, modulating the degree of periodicity of the pulse-trains while maintaining a constant
1534-4320/$26.00 © 2011 IEEE
159 86
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 1, JANUARY 2012
average pulse rate. We sought to determine the minimal perturbation of the periodic pattern that the monkeys could discriminate. The degree of randomness (as quantified by the coefficient of variation, CV) was varied from trial to trial, which allowed us to quantify the monkeys’ sensitivity to ICMS frequency modulations. Concurrently with ICMS delivery, we recorded from large populations of cortical neurons using multielectrode implants. Kinematics of reach movements were extracted from this large-scale activity offline to estimate the accuracy of a BMBI with a somatosensory feedback loop that transmits aperiodic ICMS patterns. This investigation of sensitivity to ICMS periodicity in S1 was motivated by a possible application in neuroprosthetic limbs. We expect that the patterns of ICMS triggered by the interaction of an upper-limb neuroprosthesis with objects in the environment could be highly irregular. The precise temporal structure of such patterns would depend on the interaction of touch sensors in the robotic prosthesis with the specific surface structure of the manipulated objects and on the specific exploratory movements used by the individual to interact with the objects. Therefore, by knowing the limits of the nervous system in discriminating aperiodic ICMS patterns, we can infer a principled upper bound on the maximum fidelity touch sensor that could be used in a neuroprosthesis, beyond which no additional function would be restored. This study builds on the results obtained by Romo et al. about ICMS of S1 [22], [23], [31] and our own previous work [9], [21]. One notable difference between the temporal ICMS patterns implemented here and those used by Romo et al. is that the aperiodic patterns of ICMS that we used had the same mean pulse interpulse intervals as the periodic comparison ICMS pulse trains. Thus the average number of pulses in a pulse train was the same for both periodic and aperiodic patterns. This allowed us to probe S1 sensitivity to the temporal structure of ICMS without the confound of average stimulus intensity. Romo et al. used periodic pulse trains with different frequencies [22], which left open the possibility that some of their results could be explained by differences in average ICMS intensity. Another major difference between this study and previous studies of ICMS-evoked S1 sensations in primates is that the ICMS patterns employed here were used in an active-exploration paradigm in which ICMS was used to simulate the tactile properties of virtual objects. Our monkeys explored the virtual objects, making self-paced exploratory movements, and decided which objects to explore, in what order, and for how long. This is a more realistic model of a clinical somatosensory neuroprosthesis than previous designs. II. METHODS A. Implants The experiments were conducted in two rhesus monkeys (M and N) chronically implanted with multielectrode arrays in several cortical areas following our implantation methods [37]. We used this same electrode array design for both large-scale neural recordings and ICMS delivery [9], [10]. Each monkey received four 96-channel microelectrode arrays placed in the arm and leg
Fig. 1. Implants and task paradigm. (a) The monkeys were implanted with microwire arrays targeting M1 and S1 of the upper and lower limbs. (b) Channels used for stimulation with monkey M are accented in red. (c) Objects on the screen consisted of a central response zone surrounded by a peripheral feedback zone. Movement of the avatar though the feedback or response zone triggered the delivery of ICMS pulse trains. (d). Monkeys initiated a trial by holding the avatar in the center of the screen until two peripheral objects appeared (500–1000 ms, random per trial; left sub-panel). Next, the monkeys freely explored the objects (middle sub-panel). Finally, an object was selected by holding the avatar within the response zone for 2000 ms (right sub-panel).
representation areas in sensorimotor cortex [Fig. 1(A)]. Each hemisphere was implanted with two arrays: one in the arm representation and one in the leg representation. Within each array, electrodes were grouped in two 4 4 uniformly spaced grids of electrode triplets. The electrodes within each triplet had different lengths, staggered at 300 m intervals. One grid was aligned over primary motor cortex (M1) and the other over S1. The monkeys were implanted for a series of studies beyond those described here. For the purpose of this study, we recorded neuronal activity from the right hemisphere arm arrays while the monkeys performed a manual task with their left hands. Stimulation was applied to the right hemisphere arm subdivision of S1 in monkey M and the right hemisphere leg subdivision of S1 in monkey N. All animal procedures were performed in accordance with the National Research Council’s Guide for the Care and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use Committee. B. Behavioral Task The monkeys were trained in a reaching task in which they manipulated a hand-held joystick to move a virtual reality arm (avatar) displayed on a computer screen [Fig. 1(C) and (D)]. The monkeys reached with the avatar arm towards screen objects and searched for an object with a particular artificial texture indicated by ICMS of S1. The objects were circular in shape and visually identical. Each monkey was previously trained in other variants of this task. In this study, the monkeys were shown two objects, one of which was associated with a periodic ICMS pattern and the other with an aperiodic pattern [Fig. 2(A)]. The monkeys were rewarded for selecting the object paired with periodic ICMS. The objects appeared at different locations on the screen, with the constraint that the distance from the screen center to each object was fixed, and the angle between the objects was 180 (i.e., centrally symmetric). If the correct object
160 O’DOHERTY et al.: VIRTUAL ACTIVE TOUCH USING RANDOMLY PATTERNED INTRACORTICAL MICROSTIMULATION
87
The temporal pattern of ICMS consisted of 200 Hz pulse trains delivered for 50 ms and presented at a lower secondary frequency. The secondary frequency for the rewarded artificial texture was a constant 10 Hz. For the unrewarded textures, the timing of the ICMS bursts was aperiodic. The interval between each aperiodic burst was a random variable drawn from a gamma distribution of instantaneous inter-burst intervals [39] (1) where is the probability density function, is the inter-burst interval, is the shape parameter, is the scale parameter, and is the Gamma function. We computed the shape and scale parameters as a function of the mean inter-pulse interval, , and the coefficient of variation, CV, the ratio of the standard deviation to the mean
Fig. 2. Example aperiodic ICMS pulse trains. (a) Raster indicates the range of variability of inter-burst intervals (CV = 0:8, cyan; CV = 0:5, blue; CV = 0:25, green; CV = 0:05, red; CV = 0, black). Note that each vertical line indicates a short burst of ICMS, not a single pulse. (a) Distributions of inter-burst intervals corresponding with the rasters shown in (a).
was selected, the monkeys were rewarded with a drop of fruit juice. Each trial commenced when the monkey grasped the joystick with its left hand. At this point, a circular target appeared in the center of the screen. The monkey placed the avatar arm on that center target for 0.5–1.0 s. Then, the central target disappeared and two peripheral objects appeared. Each consisted of a central response zone and a peripheral feedback zone [Fig. 1(C)]. When the avatar hand entered the feedback or response zones, ICMS pulse trains were delivered to S1. A trial was concluded with a reward when the monkey placed the avatar hand within the response zone of the correct object for 2 s; no reward was delivered if the incorrect object was selected. The monkeys were permitted to explore the virtual objects in any sequence, but the trial ended when they stayed over an object’s response zone longer than the hold period of 2 s. Then a 0.5 s delay was issued before the next trial began. C. ICMS Patterns ICMS trains consisted of symmetric [38], biphasic, chargebalanced pulses of ICMS delivered in a bipolar fashion through adjacent pairs of microwires [9], [21]. For monkey M, the anodic and cathodic phases of stimulation each had amplitudes of 150 A and pulse widths of 105 s; for monkey N, 150 A and 200 s, respectively. The anodic and cathodic phases were separated by 25 s. ICMS was delivered to different subdivisions of S1 for each monkey. For monkey M, the hand representation area of S1 was used as the target for ICMS, so that the monkey experienced putative sensations in its hand [Fig. 1(B)]. For monkey N, ICMS was applied to the thigh representation area of S1. Two electrode pairs were used for each animal.
(2) This allowed the construction of aperiodic pulse trains with inter-burst intervals equal in expectation to the periodic pulse trains while giving control over the degree of aperiodicity: the higher the CV, the more aperiodic the pulse-train. A pulse train with a CV of zero was equivalent to the periodic, rewarded pattern. The average number of ICMS pulses per unit time was the same for the periodic and aperiodic patterns. Examples of pulse trains with different CVs are shown in Fig. 2. D. Artifact Suppression An important question arising from the use of ICMS for sensory feedback is whether the stimulation causes artifacts in cortical neural ensemble recordings and how these artifacts can be dealt with to minimize their impact on BMI operations. To address this question, we processed the neural recordings by removing (blanking) a window of neural activity immediately subsequent to each pulse of ICMS and then performed decoding with the processed data. By systematically varying the length of the blanking intervals we could determine the amount of artifact removal that produced the most accurate movement reconstructions. Artifact removal was implemented as follows. The stimulation artifacts had stereotypical shapes when recorded by the spike acquisition system, and we could reliably detect artifacts by using spike-sorting templates that matched the artifact shapes, allowing us to determine the precise time of each stimulation pulse. We then ignored all spiking on every channel for milliseconds after each stimulation pulse, where varied in value from 0 to 10, in integer steps. This was done by first counting spikes in 1 ms nonoverlapping time windows (i.e., binning at 1 ms resolution). We then zeroed the spike counts in bins that were equal to or less than ms after the stimulation pulse. For example, for , we zeroed the bins at , and , where is the 1 ms bin of the stimulation pulse [see Fig. 6(A)]. For , we zeroed the bin at the stimulation pulse only. Then, we summed adjacent bins to produce spike counts in 100 ms nonoverlapping bins for our decoders. For this last step, we adjusted the spike counts
161 88
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 1, JANUARY 2012
Fig. 3. Summary of the behavioral performance of monkeys M (circles) and N (diamonds) for 10 sessions as they learned the task. Each symbol shows the mean performance for the session. Filled symbols depict sessions with perfor0 05, mance significantly different from chance (chance level of 50%; P one-sided binomial test). Curves are the sigmoidal lines of best fit.
< :
by multiplying by the quantity , where is the number of zeroed 1 ms bins in the 100 ms bin. This operation preserves, in expectation, the number of spikes in each 100 ms bin, by performing extrapolation. E. Kinematics Extraction The X and Y position of the avatar was extracted from cortical activity using a fifth-order unscented Kalman filter [40] and Wiener filter [37]. For both algorithms, we evaluated the decoding accuracy after artifact blanking. We performed two-fold cross-validation with each algorithm on 26 sessions, 13 from each monkey. For the unscented Kalman filter, we used a tuning model with linear weights for position, velocity, distance from center of workspace, and magnitude of velocity. The unscented Kalman filter had three future taps, two past taps, and one tap in the movement model (see [40] for details). The tuning model weights were fit with adaptive ridge regression [41], with the ridge parameter found by cross-validation on the training data. For the Wiener filter, we used 10 taps of spiking history and predicted the position only. The Wiener coefficients were fit using ridge regression with the ridge parameter found by cross-validation on the training data. III. RESULTS A. Learning Initially, both monkeys were required to discriminate between the periodic (rewarded) ICMS pattern and an aperiodic pattern with a CV of 0.8. Each monkey learned this discrimination task in approximately eight daily sessions (Fig. 3). Monkey N stabilized at a performance level of approximately 90% correctly executed trials, monkey M at an 85% level. These learning curves are consistent with our previous results on rhesus monkey learning with ICMS-instructed tasks [9].
Fig. 4. Psychometric curves for different coefficients of variation on the aperiodic pulse trains. (a) Mean performance at differentiating periodic versus aperiodic ICMS pulse trains as a function of CV for monkey N. Each symbol represents the mean performance across sessions; error bars indicate 95% confidence intervals. Curves are the sigmoidal lines of best fit. Symbols are as in Fig. 3. (b) Same as (a), but for monkey M.
B. Psychometrics After both monkeys learned to discriminate periodic ICMS from aperiodic with a CV of 0.8, we began to vary the CV of the aperiodic ICMS pattern on every trial. In these sessions, the distribution of CVs was picked so that for half of the trials the CV of the unrewarded object was greater or equal to 0.6. These sessions continued for two weeks, yielding a database for psychometric analysis. Psychometric curves (i.e., graphs showing the proportion of correctly performed trials as the function of CV, Fig. 4) indicated a clear dependency of discrimination accuracy on the degree of randomness of the comparison ICMS pattern. Performance stabilized for CVs higher than 0.8 and gradually decreased for CVs lower than that value. The threshold CV for discrimination for both monkeys was 0.25. Below this value, the monkeys performed at chance levels. C. Active Exploration Discrimination of ICMS patterns was performed through active exploration: a monkey would probe the feedback zone of an object with the avatar to acquire an ICMS pattern and then either select that object if it perceived the ICMS pattern as periodic or explore the other object if the pattern was judged as different from periodic. This active exploration was evident from an analysis of object exploration intervals [Fig. 5(A) and (B)]. We designated intervals during which the avatar hand continuously stayed over a given object as â&#x20AC;&#x153;visits.â&#x20AC;? For very low CVs, the statistics of visit durations were the same for periodic and aperiodic patterns [Fig. 5(A)]. The distribution of these intervals indicated short (less than 2 s) exploratory visits and a prominent peak at 2 s that corre-
162 Oâ&#x20AC;&#x2122;DOHERTY et al.: VIRTUAL ACTIVE TOUCH USING RANDOMLY PATTERNED INTRACORTICAL MICROSTIMULATION
89
Fig. 5. Active exploration of the virtual objects as a function of CV. (a) Histograms of visit durations to the rewarded object (CV = 0, black trace) versus the unrewarded object (CV = 0:05, red trace) for trials with that CV combination. Visits are quantified as intervals when the avatar was continuously over the feedback zone or the response zone. Vertical line indicates the hold interval (2 s). (b) Same as (a), but for trials with unrewarded objects with aperiodic ICMS CVs of 0.8 (cyan trace). (c) Fraction of the short visits (less than 2 s) for the rewarded object (square symbols) versus the unrewarded object (triangular symbols) expressed as a function of CV. Gray shaded zones correspond to the data in (a) and (b).
sponded to selecting an object. This is because the monkey was required to hold the avatar hand over the response zone of the object for 2 s to obtain a reward (or to get a trial cancellation if the object was selected incorrectly). Accordingly, the peak at 2 s corresponded to visits for which the monkey selected a given object. Intervals longer than 2s were possible because visits comprised the portion spent over the feedback zone (but outside of the response zone) as well as the time spent over the response zone. We called visits with durations less than 2 s short visits, and those with durations of 2 s or longer long visits. Both the short-visit portion of the distribution and the long-visit part were preserved for periodic ICMS ( , Fig. 5(A), black line) versus weakly aperiodic ICMS e.g., CV of 0.05 (Fig. 5(A), red line). The distributions of visit-durations were markedly different for higher CVs [Fig. 5(B)]. The distribution of visit-durations for aperiodic ICMS with a CV of 0.8 (cyan line) revealed a predominance of short visits with an average duration of 0.8 s and a small proportion of long visits. For the periodic pattern (black line), the distribution showed the predominance of long visits. These data indicate that it took the monkey on average 0.8 s to recognize the unrewarded aperiodic ICMS pattern and to switch to the correct object (periodic pattern) when sufficiently aperiodic ICMS patterns were used. The change in monkey exploratory behavior for different degrees of ICMS-pattern aperiodicity is clear from the statistics of visits, expressed as the proportion of short visits normalized by the total number of visits [Fig. 5(C)]. When a monkey touched
Fig. 6. Exploration of blanking intervals. (a) Schematic of the blanking procedure. Spikes and ICMS pulses were categorized into 1 ms bins. For each bin containing an ICMS pulse, that bin and a variable number of bins (three shown here) were blanked subsequently. (b). Mean movement reconstruction accuracy as a function of blanking intervals for both monkeys (symbols as in Fig. 3) and both algorithms (Wiener filter, WF, dashed lines; unscented Kalman filter, UKF, solid lines). Bars indicate standard error. Shaded region corresponds to no blanking.
an object associated with an aperiodic pattern (Fig. 5(C), triangles), it tended to make more short visits than when the monkey touched an object associated with a periodic pattern (squares). For high CVs (CV greater than 0.7), the proportion of short visits constituted approximately 80% of the total number of visits. For lower CVs, this value decreased, indicating that the monkey made a decision to stay on the unrewarded object more often. D. Kinematics Extraction Fig. 6 shows the average accuracy for the extraction of avatar position from cortical ensemble activity for different lengths of artifact blanking intervals. Consistent with our previous results [40], the unscented Kalman filter consistently outperformed the Wiener filter. For monkey M, the peak accuracy was dB (mean standard error) for the unscented Kalman filter and dB for the Wiener filter. For monkey N, these values were dB and dB, respectively. These accuracy values are within the range that we typically observe for BMI predictions [9], [17], [41].
163 90
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 1, JANUARY 2012
Both algorithms benefited somewhat from artifact blanking, more so for monkey M. For monkey M, maximum accuracy was achieved with 5 ms of artifact blanking for both the Weiner filter and the unscented Kalman filter. For monkey N, maximum accuracy was achieved with 2 ms of artifact blanking for both decoders. These values reflect the difference in artifact duration and amplitude in two monkeys. The artifacts were more prominent and of longer duration in monkey M because of the close proximity of the stimulation site (hand representation of S1) to the area where neuronal activity was collected (arm representation of M1 and S1). The artifacts were smaller and of shorter duration for monkey N, which received stimulation in the leg representation area of S1 with recordings performed in the arm representation area. Curiously, the performance of the unscented Kalman filter was slightly better for the no blanking condition than for 0 ms of blanking. This was because the recording channels that detected ICMS artifacts occasionally recorded additional mechanical artifacts related to monkey head movements. Apparently, the filter could utilize these mechanical artifacts that influenced the spike recording channels (blanked them or introduced erroneous spikes) to improve predictions, and its performance was very slightly reduced when these artifacts were removed. This underscores the importance of registering the artifacts and removing them to minimize their influence on the filter performance. To quantify the decrease in predictions caused by the presence of ICMS artifacts, we recorded from both monkeys as they performed a center-out task without any ICMS. For this task, they had to move the avatar from the center of the screen to a single peripheral object and hold for 2 s. For monkey M, accuracy in this task was 23% higher than during the ICMS sessions with the unscented Kalman filter and 32% higher with the Wiener filter. For monkey N, these values were 2.7% and 27%, respectively. Thus, the artifacts worsened the predictions, even after optimal blanking, but still within a tolerable range. IV. DISCUSSION This study continued our work on the development of an artificial somatosensory channel for BMIs [9], [10], [21]. Monkeys scanned virtual objects with an avatar hand and discriminated their artificial textures as represented by temporal patterns of ICMS. This paradigm models the requirements of a clinically relevant neuroprosthetic arm sensorized with an artificial tactile channel. Such a neuroprosthetic arm could be used to touch external objects and estimate their tactile properties (roughness/ smoothness, hardness/softness, wetness/dryness, temperature) using sensors on the prosthetic hand. The transmission of this information to the nervous system is a difficult problem because of the artificial nature of the stimulation methods. We explored the capability of temporally patterned ICMS as a way to deliver somatosensory feedback to the brain by parametrically varying the degree of randomness of ICMS trains. Monkeys learned to distinguish regular ICMS patterns from irregular ones, a result which suggests that they were able to discriminate the fine temporal structure of ICMS trains. Irregular bursts of sensory discharges are expected to occur in practical neuroprostheses, when the prosthesis interacts with realistically textured objects.
A neuroprosthetic hand used to scan a ridged surface, for example, would generate an ICMS burst each time a ridge interacts with the prosthetic sensor. In this setting, the degree of periodicity of ICMS pulse trains could inform the prosthesis user about the regularities or irregularities of an object’s material or shape. Our results complement previous work on ICMS frequency discrimination conducted by Romo et al. who trained their monkeys to discriminate periodic ICMS pulse trains [22]–[24] and to discriminate the mean rate of aperiodic pulse trains [22], [42]. Our study expanded the range of temporal patterns that could be represented by ICMS of S1 by changing the regularity of the secondary frequency. Moreover, ICMS in our experiments served as somatosensory feedback during virtual active touch, rather than merely a cue in a forced choice task as in the majority of previous studies. The animals actively explored virtual objects with an avatar hand, spending similar times over these objects as would be needed for normal interaction with the environment. Additionally, chronically implanted electrodes were used for ICMS delivery, which allowed us to monitor longterm learning to utilize ICMS as sensory feedback. In previous studies, stimulating electrodes were often inserted in the brain anew during each daily session. Long-term usage of ICMS in the present experiments (as well as in previous experiments with the same monkeys) did not result in deterioration of performance, which indicates that the charge-balanced ICMS used here did not damage the electrodes or brain tissue, or that any such damage was below a threshold where it would begin to impact task performance. Our results show that monkeys detect distortions in the 10 Hz ICMS secondary frequency after random variations of that frequency exceeded 25%, that is, instantaneous frequency fluctuated from 7.5 to 12.5 Hz. This estimate of the detection threshold can be used in future neuroprosthetic designs as a characteristic sensitivity value. Future studies should probe the sensitivity of discrimination to different primary and secondary frequencies. Additionally, spatiotemporal ICMS [21] and ICMS of different durations should be explored as ways to encode information in BMBI sensory channels. The interaction of a neuroprosthesis with realistically textured objects in the natural world will inevitably result in a stream of temporally patterned sensory information. Either the user or the neuroprosthesis (or some combination thereof) will therefore need to deal with these signals. Texture analysis could be delegated, in part, to a shared control algorithm [43]. In this mode of operation, a sensation processor would analyze raw signals from sensors on the prosthesis and interpret them in the context of how the neuroprosthetic device “skin” moved against the surface of textured objects. Simplified ICMS patterns—representing different classes of textures—could then be sent to the brain. Alternatively, ICMS could directly encode a signal representing both the spatiotemporal movements of the prosthetic limb and the intrinsic microstructure of the material being touched. In this case, the temporal patterns of ICMS would have to be interpreted by the user in the context of the particular exploration pattern used [44]. The choice of the encoding scheme will likely be dictated by the requirements of the specific neuroprosthetic application.
164 O’DOHERTY et al.: VIRTUAL ACTIVE TOUCH USING RANDOMLY PATTERNED INTRACORTICAL MICROSTIMULATION
It would be of interest for future studies to explore the optimal temporal properties of ICMS modulations at different S1 sites. Romo et al. reported best results when they applied ICMS to rapidly adapting neurons in area 3b [22]. We stimulated in area 1, where the distinction between rapidly adapting and slowly adapting categories of neurons is less clear. Additionally, we used multi-session training periods with chronically implanted electrodes, in contrast to Romo et al., who used independent stimulation sessions with acute electrodes. It is possible that that our longer training interval facilitated the discrimination capacity of the monkeys. The distinction between different S1 locations and the role of learning will need to be studied in more detail in future studies. BMBIs equipped with afferent ICMS feedback loops need to compensate for electrical artifacts produced by ICMS pulses that may interfere with neuronal recordings. In our previous BMBI designs, we either discounted the entire period of ICMS application [9] or used interleaved recording and ICMS delivery intervals [10]. These previous approaches limited the flexibility of ICMS delivery. In this study, we did not impose limitations on the timing of ICMS delivery and treated ICMS artifacts as they occurred. We found that blanking the periods after ICMS delivery by short intervals (2–5 ms) improved the accuracy of extraction of limb kinematics from neuronal activity. Overall accuracy of predictions was 20%–30% less as compared to sessions in which ICMS was not used. Nonetheless, the predictions were still acceptable and within range of previously reported accuracy of BMI decoding. This result suggests that artifact blanking is practical for bidirectional neuroprostheses using irregular ICMS pulse trains. The precise character of perceptions evoked by periodic versus aperiodic patterns of ICMS will have be evaluated in human subjects [45]. There is a suggestion by Fridman et al. that ICMS amplitude, pulse-width, and frequency all interact to contribute to a unitary perception of “perceived intensity” [46]. Therefore, one might argue that our monkeys discriminated the periodic and aperiodic pulse trains on the basis of their instantaneous peak intensities rather than their temporal patterns. However, this simple explanation is unlikely because the peak instantaneous frequency of ICMS was 200 Hz for both the periodic as well as the aperiodic artificial textures. Therefore, our results indicate that the monkeys must have been using a strategy beyond simply detecting the maximum instantaneous frequency. One possible neural implementation could employ a leaky integrator mechanism that detected variability of the ICMS secondary frequency by integrating neural responses to ICMS within an optimal time window, thus detecting transient increases in ICMS frequency. This and other alternative mechanisms will need to be elucidated with future studies. Our current and previous [9], [10], [21] results suggest that new perceptions may evolve as subjects practice with ICMS. We observed that it took monkeys 1–2 weeks to start to understand ICMS, even if they were previously overtrained with a vibrotactile variant in the same task. However, once they learned the first ICMS task, learning subsequent tasks took much less time. A virtual active touch setting where subjects evoke ICMS and associated sensations through their own actions [47]–[49] may contribute to shaping the artificial perception and lead to
91
the development of anticipatory cortical modulations similar to corollary discharge [50]. The problem of artifacts will be compounded as multiple stimulation channels are employed with asynchronously delivered pulses. Excessive masking of the recordings by ICMS artifacts should be avoided in the design of such systems. In the future, the problem of artifacts [51], as well as the unreliable spatial extent of ICMS [52] could be mitigated by optogenetic stimulation [53]–[55]. ACKNOWLEDGMENT The authors would like to thank D. Dimitrov for assistance with the animal surgeries, S. Shokur for design and programming of the monkey avatar, and G. Lehew, J. Meloy, T. Phillips, L. Oliveira, and S. Halkiotis for invaluable technical support. REFERENCES [1] M. M. Merzenich, D. N. Schindler, and M. W. White, “Feasibility of multichannel scala tympani stimulation,” Laryngoscope, vol. 84, pp. 1887–1893, Nov. 1974. [2] J. B. Fallon, D. R. F. Irvine, and R. K. Shepherd, “Cochlear implants and brain plasticity,” Hear. Res., vol. 238, pp. 110–117, Apr. 2008. [3] P. Bach-y-Rita, C. C. Colins, F. A. Saunders, B. White, and L. Scadden, “Vision substitution by tactile image projection,” Nature, vol. 221, pp. 963–964, Mar. 1969. [4] P. Bach-y-Rita, K. A. Kaczmarek, M. E. Tyler, and J. Garcia-Lara, “Form perception with a 49-point electrotactile stimulus array on the tongue: A technical note,” J. Rehabil. Res. Dev., vol. 35, pp. 427–430, Oct. 1998. [5] P. Bach-y-Rita and S. W. Kercel, “Sensory substitution and the humanmachine interface,” Trends Cogn. Sci., vol. 7, pp. 541–546, Dec. 2003. [6] W. H. Dobelle, M. G. Mladejovsky, and J. P. Girvin, “Artificial vision for the blind: Electrical stimulation of visual cortex offers hope for a functional prosthesis,” Science, vol. 183, pp. 440–444, Feb. 1974. [7] G. Dagnelie, “Psychophysical evaluation for visual prosthesis,” Annu. Rev. Biomed. Eng., vol. 10, pp. 339–368, 2008. [8] E. D. Cohen, “Prosthetic interfaces with the visual system: Biological issues,” J. Neural Eng., vol. 4, pp. R14–R31, Jun. 2007. [9] J. E. O’Doherty, M. A. Lebedev, T. L. Hanson, N. A. Fitzsimmons, and M. A. L. Nicolelis, “A brain-machine interface instructed by direct intracortical microstimulation,” Front. Integr. Neurosci., vol. 3, p. 20, 2009. [10] J. E. O’Doherty et al., “Active tactile exploration enabled by a brainmachine-brain interface,” Nature, vol. 479, pp. 228–231, Nov. 2011. [11] F. A. Mussa-Ivaldi et al., “New perspectives on the dialogue between brains and machines,” Front. Neurosci., vol. 4, p. 44, 2010. [12] S. Stanslaski et al., “An implantable bi-directional brain-machine interface system for chronic neuroprosthesis research,” in Proc. IEEE Eng. Med. Biol. Soc. Conf., 2009, vol. 2009, pp. 5494–5497. [13] A. H. Fagg et al., “Toward a biomimetic, bidirectional, brain machine interface,” in Proc. IEEE Eng. Med. Biol. Soc. Conf., 2009, vol. 2009, pp. 3376–3380. [14] T. C. Marzullo, M. J. Lehmkuhle, G. J. Gage, and D. R. Kipke, “Development of closed-loop neural interface technology in a rat model: Combining motor cortex operant conditioning with visual cortex microstimulation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 18, no. 4, pp. 117–126, Apr. 2010. [15] M. A. Lebedev and M. A. L. Nicolelis, “Brain-machine interfaces: Past, present and future,” Trends Neurosci., vol. 29, pp. 536–546, Sep. 2006. [16] M. A. L. Nicolelis and M. A. Lebedev, “Principles of neural ensemble physiology underlying the operation of brain-machine interfaces,” Nat. Rev. Neurosci., vol. 10, pp. 530–540, Jul. 2009. [17] N. A. Fitzsimmons, M. A. Lebedev, I. D. Peikon, and M. A. L. Nicolelis, “Extracting kinematic parameters for monkey bipedal walking from cortical neuronal ensemble activity,” Front. Integr. Neurosci., vol. 3, p. 3, 2009. [18] J. S. Brumberg, A. Nieto-Castanon, P. R. Kennedy, and F. H. Guenther, “Brain-computer interfaces for speech communication,” Speech Commun., vol. 52, pp. 367–379, Apr. 2010.
165 92
IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 1, JANUARY 2012
[19] F. H. Guenther et al., “A wireless brain-machine interface for real-time speech synthesis,” PLoS One, vol. 4, p. e8218, 2009. [20] N. R. Peterson, D. B. Pisoni, and R. T. Miyamoto, “Cochlear implants and spoken language processing abilities: Review and assessment of the literature,” Restor. Neurol. Neurosci., vol. 28, pp. 237–250, 2010. [21] N. A. Fitzsimmons, W. Drake, T. L. Hanson, M. A. Lebedev, and M. A. L. Nicolelis, “Primate reaching cued by multichannel spatiotemporal cortical microstimulation,” J. Neurosci., vol. 27, pp. 5593–5602, May 2007. [22] R. Romo, A. Hernández, A. Zainos, and E. Salinas, “Somatosensory discrimination based on cortical microstimulation,” Nature, vol. 392, pp. 387–390, Mar. 1998. [23] R. Romo, A. Hernández, A. Zainos, C. D. Brody, and L. Lemus, “Sensing without touching: Psychophysical performance based on cortical microstimulation,” Neuron, vol. 26, pp. 273–278, Apr. 2000. [24] V. de Lafuente and R. Romo, “Neuronal correlates of subjective sensory experience,” Nat. Neurosci., vol. 8, pp. 1698–1703, Dec. 2005. [25] S. Butovas and C. Schwarz, “Detection psychophysics of intracortical microstimulation in rat primary somatosensory cortex,” Eur. J. Neurosci., vol. 25, pp. 2161–2169, Apr. 2007. [26] A. R. Houweling and M. Brecht, “Behavioural report of single neuron stimulation in somatosensory cortex,” Nature, vol. 451, pp. 65–68, Jan. 2008. [27] S. K. Talwar et al., “Rat navigation guided by remote control,” Nature, vol. 417, pp. 37–38, May 2002. [28] S. Venkatraman and J. M. Carmena, “Active sensing of target location encoded by cortical microstimulation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 19, no. 3, pp. 317–324, Jun. 2011. [29] D. Katz and L. E. Krueger, The World of Touch. Hillsdale, N.J.: Erlbaum, 1989. [30] V. B. Mountcastle, M. A. Steinmetz, and R. Romo, “Frequency discrimination in the sense of flutter: Psychophysical measurements correlated with postcentral events in behaving monkeys,” J. Neurosci., vol. 10, pp. 3032–3044, Sep. 1990. [31] E. Salinas, A. Hernández, A. Zainos, and R. Romo, “Periodicity and firing rate as candidate neural codes for the frequency of vibrotactile stimuli,” J. Neurosci., vol. 20, pp. 5503–5515, Jul. 2000. [32] M. A. Lebedev, J. M. Denton, and R. J. Nelson, “Vibration-entrained and premovement activity in monkey primary somatosensory cortex,” J. Neurophysiol., vol. 72, pp. 1654–1673, Oct. 1994. [33] R. J. Sinclair and H. Burton, “Tactile discrimination of gratings: Psychophysical and neural correlates in human and monkey,” Somatosens. Mot. Res., vol. 8, pp. 241–248, 1991. [34] R. J. Sinclair, J. R. Pruett, and H. Burton, “Responses in primary somatosensory cortex of rhesus monkey to controlled application of embossed grating and bar patterns,” Somatosens. Mot. Res., vol. 13, pp. 287–306, 1996. [35] E. Gamzu and E. Ahissar, “Importance of temporal cues for tactile spatial- frequency discrimination,” J. Neurosci., vol. 21, pp. 7416–7427, Sept. 2001. [36] J. R. Phillips and K. O. Johnson, “Tactile spatial resolution. II. Neural representation of bars, edges, and gratings in monkey primary afferents,” J. Neurophysiol., vol. 46, pp. 1192–1203, Dec. 1981. [37] J. M. Carmena et al., “Learning to control a brain-machine interface for reaching and grasping by primates,” PLoS Biol., vol. 1, p. E42, Nov. 2003. [38] A. Koivuniemi and K. Otto, “Asymmetric vs. symmetric electric pulses for intracortical microstimulation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 19, no. 5, pp. 468–476, Oct. 2011. [39] A. D. Dorval, A. M. Kuncel, M. J. Birdno, D. A. Turner, and W. M. Grill, “Deep brain stimulation alleviates parkinsonian bradykinesia by regularizing pallidal activity,” J. Neurophysiol., vol. 104, pp. 911–921, Aug. 2010. [40] Z. Li et al., “Unscented Kalman filter for brain-machine interfaces,” PLoS One, vol. 4, p. e6243, 2009. [41] Y. Grandvalet, , L. Niklasson, Ed. et al., “Perspectives in Neural Computing,” in Least Absolute Shrinkage is Equivalent to Quadratic Penalization. New York: Springer Verlag, 1998, pp. 201–206. [42] A. Hernández, A. Zainos, and R. Romo, “Neuronal correlates of sensory discrimination in the somatosensory cortex,” in Proc. Nat. Acad. Sci. USA, May 2000, vol. 97, pp. 6191–6196. [43] H. K. Kim et al., “Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces,” IEEE Trans. Biomed. Eng., vol. 53, no. 6, pp. 1164–1173, Jun. 2006. [44] S. J. Lederman and R. L. Klatzky, “Hand movements: A window into haptic object recognition,” Cogn. Psychol., vol. 19, pp. 342–368, Jul. 1987.
[45] E. Heming, R. Choo, J. Davies, and Z. Kiss, “Designing a thalamic somatosensory neural prosthesis: Consistency and persistence of percepts evoked by electrical stimulation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 19, no. 5, pp. 477–482, Oct. 2011. [46] G. Y. Fridman, H. T. Blair, A. P. Blaisdell, and J. W. Judy, “Perceived intensity of somatosensory cortical electrical stimulation,” Exp. Brain Res., vol. 203, pp. 499–515, June 2010. [47] R. Sinclair and H. Burton, “Responses from area 3b of somatosensory cortex to textured surfaces during active touch in primate,” Somatosens. Res., vol. 5, pp. 283–310, 1988. [48] C. Simões-Franklin, T. A. Whitaker, and F. N. Newell, “Active and passive touch differentially activate somatosensory cortex in texture perception,” Hum. Brain Mapp., vol. 32, pp. 1067–1080, 2011. [49] S. J. Bolanowski, R. T. Verrillo, and F. McGlone, “Passive, active and intra-active (self) touch,” Behav. Brain. Res., vol. 148, pp. 41–45, Jan. 5, 2004. [50] T. B. Crapse and M. A. Sommer, “Corollary discharge across the animal kingdom,” Nat. Rev. Neurosci., vol. 9, pp. 587–600, Aug. 2008. [51] J. D. Rolston, R. E. Gross, and S. M. Potter, “A low-cost multielectrode system for data acquisition enabling real-time closed-loop processing with rapid recovery from stimulation artifacts,” Front. Neuroeng., vol. 2, p. 12, 2009. [52] M. H. Histed, V. Bonin, and R. C. Reid, “Direct activation of sparse, distributed populations of cortical neurons by electrical microstimulation,” Neuron, vol. 63, pp. 508–522, Aug. 2009. [53] F. Zhang, A. M. Aravanis, A. Adamantidis, L. de Lecea, and K. Deisseroth, “Circuit-breakers: Optical technologies for probing neural signals and systems,” Nat. Rev. Neurosci., vol. 8, pp. 577–581, Aug. 2007. [54] E. S. Boyden, F. Zhang, E. Bamberg, G. Nagel, and K. Deisseroth, “Millisecond-timescale, genetically targeted optical control of neural activity,” Nat. Neurosci., vol. 8, pp. 1263–1268, Sep. 2005. [55] F. Zhang, L. P. Wang, E. S. Boyden, and K. Deisseroth, “Channelrhodopsin-2 and optical control of excitable cells,” Nat. Methods, vol. 3, pp. 785–792, Oct. 2006. Joseph E. O’Doherty received the B.S. degree in physics from East Carolina University, Greenville, NC, in 2001 and the Ph.D. degree in biomedical engineering from Duke University, Durham, NC, in 2011. He is currently a Postdoctoral Scholar at the W. M. Keck Foundation Center for Integrative Neuroscience, in San Francisco, CA. He has held a previous research appointment at the Duke University Center for Neuroengineering, Durham, NC (2011). His research interests include methods for providing artificial somatic sensation and proprioception for neural prostheses.
Mikhail A. Lebedev received the M.S. degree in physics from the Moscow Institute of Physics and Technology, Moscow, Russia, in 1986 and the Ph.D. degree in neurobiology from the University of Tennessee, Memphis, in 1995. He is a Senior Research Scientist at the Duke University Center for Neuroengineering, in Durham, NC. He has held research appointments at the Institute for the Problems of Information Transmission, Moscow (1986–1991), the International School for Advanced Studies, Trieste, Italy (1995–1997), and the U.S. National Institute of Mental Health (1997–2002). His research interests include primate neurophysiology and brain–machine interfaces.
Zheng Li received the B.S. degree in computer science and mathematics from Purdue University, West Lafayette, IN, in 2004, and the Ph.D. degree in computer science from Duke University, Durham, NC, in 2010. He is a Postdoctoral Associate at the Duke University Center for Neuroengineering, Durham, North Carolina. His research interests are in the computational aspects of brain–machine interfaces.
166 O’DOHERTY et al.: VIRTUAL ACTIVE TOUCH USING RANDOMLY PATTERNED INTRACORTICAL MICROSTIMULATION
Miguel A. L. Nicolelis received the M.D. and Ph.D. degrees from the University of Sao Paulo, Sao Paulo, Brazil, in 1984 and 1988, respectively. He is the Anne W. Deane Professor of Neuroscience with the departments of Neurobiology, Biomedical Engineering and Psychology at Duke University, Durham, NC. He is the Co-Director of Duke’s Center for Neuroengineering. He is also Founder and President of the Edmond and Lily Safra International Institute for Neuroscience of Natal, Brazil and a Fellow of the Brain and Mind Institute at the École Polytechnique Fédérale de Lausanne, Switzerland. He has authored
93
over 170 manuscripts, edited numerous books and special journal issues, and holds three U.S. patents. Dr. Nicolelis’ research was highlighted in MIT Review’s Top Emerging Technologies, and he was named one of Scientific American’s Top 50 Technology Leaders in America. Other honors include the Whitehead Scholar Award; Whitehall Foundation Award; McDonnell-Pew Foundation Award; the Ramon y Cajal Chair at the University of Mexico and the Santiago Grisolia Chair at Catedra Santiago Grisolia. He was awarded the International Blaise Pascal Research Chair from the Fondation de l’Ecole Normale Supérieure and the 2009 Fondation IPSEN Neuronal Plasticity Prize. He is a member of the French and Brazilian Academies of Science.
167 The Journal of Neuroscience, October 10, 2012 • 32(41):14271–14275 • 14271
Brief Communications
Stochastic Facilitation of Artificial Tactile Sensation in Primates Leonel E. Medina,1,2 Mikhail A. Lebedev,2,3 Joseph E. O’Doherty,1,2 and Miguel A. L. Nicolelis1,2,3,4,5 1Department of Biomedical Engineering, 2Center for Neuroengineering, 3Department of Neurobiology, and 4Department of Psychology and Neuroscience, Duke University, Durham, North Carolina 27710, and 5Edmond and Lily Safra International Institute of Neuroscience of Natal, 59066-060 Natal, Brazil
Artificial sensation via electrical or optical stimulation of brain sensory areas offers a promising treatment for sensory deficits. For a brain–machine– brain interface, such artificial sensation conveys feedback signals from a sensorized prosthetic limb. The ways neural tissue can be stimulated to evoke artificial sensation and the parameter space of such stimulation, however, remain largely unexplored. Here we investigated whether stochastic facilitation (SF) could enhance an artificial tactile sensation produced by intracortical microstimulation (ICMS). Two rhesus monkeys learned to use a virtual hand, which they moved with a joystick, to explore virtual objects on a computer screen. They sought an object associated with a particular artificial texture (AT) signaled by a periodic ICMS pattern delivered to the primary somatosensory cortex (S1) through a pair of implanted electrodes. During each behavioral trial, aperiodic ICMS (i.e., noise) of randomly chosen amplitude was delivered to S1 through another electrode pair implanted 1 mm away from the site of AT delivery. Whereas high-amplitude noise worsened AT detection, moderate noise clearly improved the detection of weak signals, significantly raising the proportion of correct trials. These findings suggest that SF could be used to enhance prosthetic sensation.
Introduction Perhaps somewhat counterintuitively, combining informative signals with noise can enhance signal processing in biological systems (Moss et al., 2004). This phenomenon is referred to as stochastic facilitation (SF) or stochastic resonance (McDonnell and Ward, 2011). SF occurs when moderate levels of noise increase the number of neural threshold crossings, but also keep those crossings in synchrony with the signal (Moss et al., 2004). More intense noise masks the signal and worsens its detection. Examples of SF in human sensory processing include improved detection of visual stimuli (Kitajo et al., 2003), as well as mechanical stimuli applied to the fingers (Collins et al., 1996) and the foot sole (Wells et al., 2005). Similarly, SF has been reported for the cercal sensory system of crickets (Levin and Miller, 1996), CA1 neurons in hippocampal slices from rats (Stacey and Durand, 2001), and the spinal circuits of the cat (Martínez et al., 2007). Received July 2, 2012; revised Aug. 10, 2012; accepted Aug. 24, 2012. Author contributions: L.E.M., M.A.L., J.E.O., and M.A.L.N. designed research; L.E.M. and M.A.L. performed research; L.E.M., M.A.L., and M.A.L.N. analyzed data; L.E.M., M.A.L., and M.A.L.N. wrote the paper. This work was supported by NIH Grants DP1OD006798 and RC1HD063390 to M.A.L.N. The document is solely the responsibility of the authors and does not necessarily represent the views of the Office of the Director or NIH. We are grateful to Gary Lehew and Jim Meloy for engineering the experimental setup and the multielectrode arrays, Dragan Dimitrov and Laura Oliveira for conducting neural surgery, Solaiman Shokur for avatar design, Zheng Li for assistance in the decoding algorithm, Tamara Phillips for experimental support, and Susan Halkiotis for administrative assistance and editing the manuscript. The authors declare no financial conflicts of interest. Correspondence should be addressed to Miguel A. L. Nicolelis, Department of Neurobiology, Box 3209, Room 327E, Bryan Research Building Duke University Medical Center, 101 Research Drive, Durham, NC 27100. E-mail: nicoleli@neuro.duke.edu. J.E. O’Doherty’s present address: Department of Physiology and Center for Integrative Neuroscience, University of California, San Francisco, San Francisco, CA. DOI:10.1523/JNEUROSCI.3115-12.2012 Copyright © 2012 the authors 0270-6474/12/3214271-05$15.00/0
However, there is an ongoing debate as to whether SF serves any positive function in the neural circuitry (McDonnell and Abbott, 2009; McDonnell and Ward, 2011). Additionally, it is unclear whether SF could be used in assistive devices for people with sensory disabilities, for example, systems for sensory substitution (Bach-y-Rita and Kercel, 2003) and neural prosthetics with artificial sensation (Romo et al., 1998; O’Doherty et al., 2011). Recently we described a sensorized neural prosthetic, called BMBI (brain–machine– brain interface), where neuronal activity recorded in the primary motor cortex (M1) was used to control the movements of a virtual arm while artificial tactile feedback from such an actuator was generated by intracortical microstimulation (ICMS) patterns delivered directly to somatosensory cortex (S1) (O’Doherty et al., 2009, 2011, 2012). This paradigm opens the possibility of aiding future human users of neuroprosthetic limbs with sensory signals arising from a variety of limb sensors. However, to develop sensorized neural prostheses further, detailed exploration is needed into ICMS as the means to reproduce the exteroception and proprioception normally provided by somatic sensation. Of particular interest is an inquiry into the processing of sensations evoked by ICMS when they are mixed with noisy inputs. For natural sensation, this problem is routinely solved by biological organisms in real-life environments (Moss et al., 2004). For artificial sensation, however, the research is only starting (O’Doherty et al., 2012). In the present study, we hypothesized that SF could enhance detection of weak ICMS patterns under an active exploration paradigm (O’Doherty et al., 2011; 2012). Accordingly, we delivered mixtures of stimulus and noise to S1 in different proportions and tested how their different combinations affected the subject’s active exploration of artificial textures (ATs).
168 14272 • J. Neurosci., October 10, 2012 • 32(41):14271–14275
Medina et al. • SF of Artificial Tactile Sensation in Primates
Figure 1. Experimental setup and task schematics. A, Arrays of microelectrodes were implanted in S1 and M1. Stimulating electrodes for monkey M (inset) are highlighted in red (AT) and yellow (noise). B, Monkeys manipulated a joystick to move a virtual arm and reach toward objects on a computer screen. C, Experimental task sequence. In the illustrated case, the selection of the right-hand object resulted in reward. The activity of both AT and noise ICMS channels are shown below each box.
Figure 2. Detection of signal in noise and M1 modulations. Example traces of horizontal movement of the virtual hand for three trials with different noise levels: low (A), optimal (B), and high (C). The colored horizontal bars represent the position of the targets (green, RAT; brown, NAT). In the trials shown in A and C, the monkey incorrectly selected the NAT, whereas in B it correctly picked the RAT and received a reward (indicated by a black arrow). The pulses delivered through both stimulating channels are shown. The RAT is subthreshold. The corresponding neuronal ensemble activity (monkey M, 120 units from M1) is shown in the bottom plots. Note that neuronal modulations reflect the speed and direction of arm movements. The color scale shows normalized firing rate (in Hz).
Materials and Methods Two rhesus monkeys, M (male) and N (female), were chronically implanted with cortical stainless steel microelectrode arrays in M1 and S1 in both hemispheres (Fig. 1 A) (Nicolelis et al., 2003). Neuronal activity was sampled in the arm and hand areas of M1 and S1 in the hemisphere contralateral to the working arm. We used four electrodes of the S1 implants for ICMS delivery. Stimulation was applied to the right hemisphere arm area of S1 in monkey M and the right hemisphere leg area of S1 in monkey N using a custom-built, four-channel, current-controlled stimulator (Hanson et al., 2012). The monkeys were trained to move an
image of the forearm and the hand (animated using MOTIONBUILDER; Autodesk) on a computer screen with a joystick (Fig. 1 B). Using this virtual hand, they explored virtual objects presented on the screen, which were visually identical, yet differed in their ATs. Each trial started with the monkeys placing the virtual hand over a circular target that appeared at the center of the screen. At this time, ICMS noise of randomly chosen intensity started and continued until the end of the trial. Following the initial central hold, two visual objects appeared on the left and on the right of the screen, equidistant from the center. Then, the monkeys freely explored each of these objects to find the one associated with a periodic
169 Medina et al. • SF of Artificial Tactile Sensation in Primates
J. Neurosci., October 10, 2012 • 32(41):14271–14275 • 14273
that point. We then measured the psychometric threshold for RAT detection in the absence of noise for each monkey. Current amplitudes in the range 5–150 A were tested. The detection threshold (Gescheider, 1997) was evaluated as the current level at which the proportion of correct responses was 0.75. The psychometric curve (see Fig. 3C) for monkey M was slightly shifted to the left compared with that of monkey N and therefore its threshold was lower (31 vs 45 A for monkeys M and N, respectively). In subsequent sessions, ICMS noise was presented throughout each trial. Noise amplitude for each trial was randomly selected in multiples of 15 A, up to 120 A. Data with noise were collected for eight and nine daily sessions for monkeys M and N, respectively (monkey M: 370 ⫾ 60; monkey N: 418 ⫾ 87 trials per session). For these sessions, the RAT amplitude was either suprathreshold (50 A) or subthreshold (20 A). Representative trials for a subthreshold stimulus combined with different noise intensities are shown in Figure 2. In these examples, the monkey correctly detected the subthreshold RAT (20 A) in the presence of an optimal noise (30 A) (Fig. 2 B), while failing to do so when the noise amplitude was too low (15 A; Fig. 2 A) or too high (90 A; Fig. 2C). Figure 2 also depicts the neuronal ensemble activity recorded in M1. The firing rates of M1 neurons were Figure 3. Stochastic facilitation via ICMS. A, B, Fraction of trials correct per session in terms of the noise amplitude level for clearly modulated in association with the monkey M (blue diamonds) and N (red squares) for suprathreshold (A) and subthreshold (B) RAT. *Significantly different from arm movements and reflected movement average performance ( p ⬍ 0.05 in all cases, randomization test). C, Psychometric curves (in terms of ICMS amplitude) for monkeys speed and direction. M (blue) and N (red). Lines, Best sigmoidal fits; open circles, performance at chance level ( p ⬎ 0.05, Wilcoxon rank sum test). Figure 3A shows the fraction of correct detection (averaged by session) as a function of noise amplitude for suprathreshICMS pattern, which we call the rewarded artificial texture (RAT). The old RATs (50 A). Note that in the absence of noise, monkey M other object lacked such ICMS feedback [null artificial texture (NAT)]. performed better (89.8% correct) than monkey N (77.9% corThe RAT consisted of pulse pairs (with 10 ms interpulse interval for rect), which is consistent with a more rapid saturation of the monkey M and 5 ms for monkey N) delivered at a frequency of 10 Hz. psychometric curve for monkey M (Fig. 3C). For monkey M, the The monkey selected one of the objects by holding the virtual hand over performance decayed from this high level after noise amplitude it for 2 s, and a juice reward was given if they correctly selected the RAT exceeded 30 A [significantly lower success rate compared with object (Fig. 1C). Each ICMS pulse was a symmetric, biphasic, chargethe average for each noise level ⬎30 A, p ⬍ 0.05, randomization balanced current waveform delivered through a pair of electrodes imtest (Edgington and Onghena, 2007)]. It is possible that we only planted 3 mm from each other (i.e., bipolar stimulation). For noise delivery, a different electrode pair was used, implanted 1 mm away from observed decay simply because the performance was close to satthe first pair of electrodes (Fig. 1 A). The interpulse interval of such an uration from the start and could not improve in principle. Howaperiodic signal was drawn from a gamma distribution with the coeffiever, for monkey N, whose performance level had room for cient of variation (ratio of the standard deviation to the mean) equal to 1 improvement, we observed a similar effect: the performance was (O’Doherty et al., 2012). resistant to weak noise and started to deteriorate when noise All animal procedures were performed in accordance with the Naamplitude exceeded 120 A ( p ⬍ 0.001, randomization test). tional Research Council’s Guide for the Care and Use of Laboratory Changes in the monkey’s detection performance with noise Animals and were approved by the Duke University Institutional Animal addition were very different when subthreshold RATs (20 A) Care and Use Committee. were used. This dependency was characterized by a nonmonoResults tonic curve with peaks at noise amplitude of 60 and 45 A for Monkeys M and N were previously overtrained in a similar active monkeys M and N, respectively. Thus, for monkey M, elevated exploration task (O’Doherty et al., 2011), so it took them just one performance of ⬃0.7 was observed for three noise amplitudes: and three training sessions, respectively, to achieve ⬃90% accu30, 45 and 60 A ( p ⬍ 0.05 for all points, randomization test). For monkey N, the curve had a single peak of 0.76 at 45 A ( p ⬍ racy in the two-target task of this study. No noise was presented at
170 14274 • J. Neurosci., October 10, 2012 • 32(41):14271–14275
Medina et al. • SF of Artificial Tactile Sensation in Primates
0.01, randomization test). Therefore, in both monkeys, the addition of a moderate level of ICMS noise to a weak AT stimulus improved the animal’s detection performance. Furthermore, neither the presence of ATs nor the addition of noise substantially affected movement-related modulations observed in M1 (Fig. 2), nor did it degrade the performance of a decoding algorithm that extracted the position of the virtual arm from the population activity of M1 neurons [unscented Kalman filter (Li et al., 2009); monkey M: 125 units, monkey N: 65 units; Fig. 4].
Discussion The present findings demonstrate for the first time SF effects for an artificial tactile sensation. The ability of monkeys to detect weak ICMS signals was clearly improved by the addition of moderate levels of ICMS noise. This result is consistent with previous model and experimental demonstrations of SF in physiological systems (Moss et al., 2004). Interest in SF for neural systems has dramatically increased over the last 20 years, driven by theories claiming that noise may play an important role in normal neural processing (McDonnell and Abbott, 2009; McDonnell and Ward, 2011). Since the presence of moderate amounts of internal noise may be normal and even advantageous for neural systems, several publications proposed implementing noise and SF in artificial systems designed to restore normal functions to people with disabilities, for Figure 4. Movement predictions from M1 activity. A, Position of the avatar and off-line predictions from M1 activity for six example in cochlear implants (Morse et consecutive trials with different noise amplitudes. B, C, Performance of the decoding algorithm measured as the correlation al., 2007), assistive devices for postural coefficient in terms of noise level for monkey M (B) and monkey N (C). n.s., No significant difference (one-way ANOVA test, n ⫽ balance (Harry et al., 2005), and hand sen- 1724, p ⬎ 0.57 for monkey M; n ⫽ 3421, p ⬎ 0.07 for monkey N). sorimotor tasks (Kurita et al., 2011). Here conclusion, we suggest that the incorporation of SF effects in we demonstrated the practicality of this idea for prosthetic sensensorized neural prosthetics may enhance a variety of functional sation evoked by ICMS. By adding background noise, we effeccapabilities. tively gained control over the detection threshold for an artificial tactile sensation. It has yet to be determined how SF would work in the case of prosthetic sensations evoked by spatiotemporal References input patterns delivered through multielectrode implants Bach-y-Rita P, Kercel SW (2003) Sensory substitution and the human(Fitzsimmons et al., 2007). For such multichannel systems machine interface. Trends Cogn Sci 7:541–546. CrossRef Medline Collins JJ, Imhoff TT, Grigg P (1996) Noise-enhanced tactile sensation. Nawith stimulation sites distributed over the cortical surface, the ture 383:770. CrossRef Medline superposition of spatiotemporal noise patterns could enhance Edgington ES, Onghena P (2007) Randomization tests, 4th ed. Boca Raton, signal detection in both the temporal and spatial dimensions. FL: CRC. Furthermore, SF patterns distributed across corticothalamic Fitzsimmons NA, Drake W, Hanson TL, Lebedev MA, Nicolelis MA (2007) loops could be obtained in sensorized neural prosthetics that use Primate reaching cued by multichannel spatiotemporal cortical microthalamic stimulation as an additional channel for artificial sostimulation. J Neurosci 27:5593–5602. CrossRef Medline matic sensation (Heming et al., 2010). Although in our present Gescheider GA (1997) Psychophysics: the fundamentals, 3rd ed. Mahwah, NJ: Lawrence Erlbaum Associates. study we did not manipulate the statistics of the noise channel, it Hanson TL, Ómarsson B, O’Doherty JE, Peikon ID, Lebedev MA, Nicolelis would be of great interest in the future to examine a variety of MA (2012) High-side digitally current controlled biphasic bipolar minoise-frequency characteristics. If noise contains particular frecrostimulator. IEEE Trans Neural Syst Rehabil Eng 20:331–340. CrossRef quencies, SF may accordingly filter frequency components of the Medline input signal. Importantly for practical neuroprosthetic systems, Harry JD, Niemi JB, Priplata AA, Collins JJ (2005) Balancing act [noise the potential risks of neural damage associated with ICMS may be based sensory enhancement technology]. IEEE Spectr 42:36 – 41. lower for a SF-based neural prosthetic because of the lower curCrossRef Heming E, Sanden A, Kiss ZH (2010) Designing a somatosensory neural rents injected into the neural tissue (McCreery et al., 2010). In
171 Medina et al. • SF of Artificial Tactile Sensation in Primates prosthesis: percepts evoked by different patterns of thalamic stimulation. J Neural Eng 7:064001. CrossRef Medline Kitajo K, Nozaki D, Ward LM, Yamamoto Y (2003) Behavioral stochastic resonance within the human brain. Phys Rev Lett 90:218103. CrossRef Medline Kurita Y, Shinohara M, Ueda J (2011) Wearable sensorimotor enhancer for a fingertip based on stochastic resonance. Paper presented at 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, May 9 –13. CrossRef Levin JE, Miller JP (1996) Broadband neural encoding in the cricket cercal sensory system enhanced by stochastic resonance. Nature 380:165–168. CrossRef Medline Li Z, O’Doherty JE, Hanson TL, Lebedev MA, Henriquez CS, Nicolelis MA (2009) Unscented Kalman filter for brain-machine interfaces. PLoS One 4:e6243. CrossRef Medline Martínez L, Pérez T, Mirasso CR, Manjarrez E (2007) Stochastic resonance in the motor system: effects of noise on the monosynaptic reflex pathway of the cat spinal cord. J Neurophysiol 97:4007– 4016. CrossRef Medline McCreery D, Pikov V, Troyk PR (2010) Neuronal loss due to prolonged controlled-current stimulation with chronically implanted microelectrodes in the cat cerebral cortex. J Neural Eng 7:036005. CrossRef Medline McDonnell MD, Abbott D (2009) What is stochastic resonance? Definitions, misconceptions, debates, and its relevance to biology. PLoS Comput Biol 5:e1000348. CrossRef Medline McDonnell MD, Ward LM (2011) The benefits of noise in neural systems: bridging theory and experiment. Nat Rev Neurosci 12:415– 426. CrossRef Medline Morse RP, Morse PF, Nunn TB, Archer KA, Boyle P (2007) The effect of
J. Neurosci., October 10, 2012 • 32(41):14271–14275 • 14275 Gaussian noise on the threshold, dynamic range, and loudness of analogue cochlear implant stimuli. J Assoc Res Otolaryngol 8:42–53. CrossRef Medline Moss F, Ward LM, Sannita WG (2004) Stochastic resonance and sensory information processing: a tutorial and review of application. Clin Neurophysiol 115:267–281. CrossRef Medline Nicolelis MA, Dimitrov D, Carmena JM, Crist R, Lehew G, Kralik JD, Wise SP (2003) Chronic, multisite, multielectrode recordings in macaque monkeys. Proc Natl Acad Sci U S A 100:11041–11046. CrossRef Medline O’Doherty JE, Lebedev MA, Hanson TL, Fitzsimmons NA, Nicolelis MA (2009) A brain-machine interface instructed by direct intracortical microstimulation. Front Integr Neurosci 3:20. Medline O’Doherty JE, Lebedev MA, Ifft PJ, Zhuang KZ, Shokur S, Bleuler H, Nicolelis MA (2011) Active tactile exploration using a brain-machine-brain interface. Nature 479:228 –231. CrossRef Medline O’Doherty JE, Lebedev MA, Li Z, Nicolelis MA (2012) Virtual active touch using randomly patterned intracortical microstimulation. IEEE Trans Neural Syst Rehabil Eng 20:85–93. CrossRef Medline Romo R, Hernández A, Zainos A, Salinas E (1998) Somatosensory discrimination based on cortical microstimulation. Nature 392:387–390. CrossRef Medline Stacey WC, Durand DM (2001) Synaptic noise improves detection of subthreshold signals in hippocampal CA1 neurons. J Neurophysiol 86:1104 –1112. Medline Wells C, Ward LM, Chua R, Timothy Inglis J (2005) Touch noise increases vibrotactile sensitivity in old and young. Psychol Sci 16:313–320. CrossRef Medline
172
ARTICLE Received 24 Aug 2012 | Accepted 15 Jan 2013 | Published 12 Feb 2013
DOI: 10.1038/ncomms2497
Perceiving invisible light through a somatosensory cortical prosthesis Eric E. Thomson1,2, Rafael Carra1,w & Miguel A.L. Nicolelis1,2,3,4,5
Sensory neuroprostheses show great potential for alleviating major sensory deficits. It is not known, however, whether such devices can augment the subject’s normal perceptual range. Here we show that adult rats can learn to perceive otherwise invisible infrared light through a neuroprosthesis that couples the output of a head-mounted infrared sensor to their somatosensory cortex (S1) via intracortical microstimulation. Rats readily learn to use this new information source, and generate active exploratory strategies to discriminate among infrared signals in their environment. S1 neurons in these infrared-perceiving rats respond to both whisker deflection and intracortical microstimulation, suggesting that the infrared representation does not displace the original tactile representation. Hence, sensory cortical prostheses, in addition to restoring normal neurological functions, may serve to expand natural perceptual capabilities in mammals.
1 Department
of Neurobiology, Duke University, Box 3209, 311 Research Drive, Bryan Research, Durham, North Carolina 27710, USA. 2 Edmond and Lily Safra International Institute for Neuroscience of Natal (ELS-IINN), Natal 01257050, Brazil. 3 Department of Biomedical Engineering, Duke University, Durham, North Carolina 27710, USA. 4 Department of Psychology and Neuroscience, Duke University, Durham, North Carolina 27710, USA. 5 Center for Neuroengineering, Duke University, Durham, North Carolina 27710, USA. w Present address: University of Sao Paulo School of Medicine, Sao Paulo 01246-000, Brazil. Correspondence and requests for materials should be addressed to M.A.L.N. (e-mail: nicoleli@neuro.duke.edu). NATURE COMMUNICATIONS | 4:1482 | DOI: 10.1038/ncomms2497 | www.nature.com/naturecommunications
& 2013 Macmillan Publishers Limited. All rights reserved.
1
173
ARTICLE
NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2497
M
ost sensory neuroprosthetic systems aim to restore function within the same modality as a pre-existing sensory deficit. For instance, cochlear implants restore auditory function1, and stimulation along the visual pathway can restore visual function2,3. In some cases, however, it might be necessary to augment, rather than simply restore, the usual function of a sensory area4,5. For instance, in people that suffer permanent damage to the visual cortex, we might ask the somatosensory cortex to take over some of the roles of the visual system. Such cross-modal plasticity has been demonstrated in juvenile animals6–8. For example, rewiring experiments in newborn ferrets have shown that when visual inputs are rerouted to the auditory cortex, the auditory cortex acquires many anatomical and functional properties of the visual cortex7,8, and even mediates visually guided behaviour8. Such results suggest that the function of a primary sensory area can dramatically change depending on the type of input it receives from the environment9,10. To date, however, the literature lacks a clear demonstration of such functional plasticity in normal adult mammals. This would require that the adult brain be plastic enough to extract novel information embedded within pre-existing representations, and use this information to generate appropriate behaviours. In the present study, we test whether adult rats can incorporate a novel sensory modality into their perceptual repertoire. Specifically, we examined whether adult rats could learn to discriminate among infrared (IR) sources after we coupled the output of a head-mounted IR detector to electrical microstimulators in the whisker representation of S1. We discovered that, after training with this sensory prosthetic device,
adult rats learned to navigate their IR world as if they had acquired a novel distal sensory modality. Results Behavioural performance. We initially trained six rats on a simple visual discrimination task. Rats were placed in a circular chamber that included three reward ports (Figs 1a,b). On each trial, a visible LED was activated in one of the ports, and rats were rewarded with water for poking their nose in that port. Once they reached criterion on this task (70% correct, after 25±5 days (mean±s.e.m.)), we surgically affixed an IR detector to the rat’s head, and implanted stimulating microelectrodes in the whisker region of S1 cortex (Figs 1c,d). After this surgical procedure, we returned the animals to the same chamber, where they had to learn to perform the same task using IR light, which is invisible to rats (see Methods for details on the task and training)11. To allow the rats to perceive IR light levels, the value of the IR detector output was converted into a pattern of electrical stimulation in S1, with stimulation frequency updated every 50 ms depending on the IR intensity detected (Fig. 1d). Electrical stimulation frequency increased as rats approached, or oriented their heads toward, an IR source (see Supplementary Movie 1). Note that such electrical stimulation in S1 is known to induce some type of tactile sensation in humans12 and monkeys13. It took 26±6 days for all six implanted rats to learn to discriminate among the IR sources at or above the criterion used in the initial visual task (470% correct). While training with their new IR gear, rats underwent clear changes in behavioural strategy.
b
a
c
Port 1 IR LED
Port 2
IR detector
Water port
θ
Visible LED
θ Port 3 Stimulating electrodes
Online processing (5 ms)
IR level (v)
* * * *
Frequency (Hz)
e
400
Current (μA)
d
0 IR level
Time (s)
Time (s)
Figure 1 | Methods. (a) Set-up of the IR behaviour chamber. Three reward ports line the walls of the circular chamber. The proximity of the ports to one another is indicated by the angle y. The rat has an IR detector (red) affixed to its head, and the cone emanating from the detector represents the area within which it will respond to IR stimuli. The red lines emanating from port 3 represent the IR signal emitted from that IR source. (b) Arrangement of each reward port, which includes a recess with a water spout, an IR LED above, and a broad-spectrum visible LED below. (c) Design of stimulating electrodes (see Methods for details). The inset shows how each stimulating electrode pair is configured in the array (scale bar, 300 mm). (d) Example of electrode placement. Cytochrome oxidase-stained barrel field shows the location of four stimulating electrodes. Each penetration is indicated with a red asterisk. Reference line: 500 mm. (e) Coupling IR levels with ICMS. On each trial, the IR light turns on, which activates the IR detector that is mounted on the rat’s head. Processing converts the detected IR level into a stimulation frequency. This value is sent to the microstimulator, which produces the desired current pulses. The inset on the right-hand-side illustrates the structure of each biphasic waveform in the pulse train. 2
NATURE COMMUNICATIONS | 4:1482 | DOI: 10.1038/ncomms2497 | www.nature.com/naturecommunications
& 2013 Macmillan Publishers Limited. All rights reserved.
ARTICLE
NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2497
At first, they did not associate intracortical microstimulation (ICMS) with the task, and would poke randomly in the reward ports, occasionally scratching their faces in response to microstimulation. Eventually they learned to actively forage through the behaviour chamber, sweeping the IR sensor on their heads back and forth to sample their IR world (see Supplementary Movies 2–4). Quantitatively, the rats’ performance increased from 41±6 to 93±2% correct during learning (Fig. 2a), and their best singlesession performance averaged 95±3% correct. Their behavioural latency on correct trials (the time between stimulation onset and poking in the correct reward port) decreased significantly as they improved at the task (latency dropped from 2.3±0.01 to 1.3±0.03 s; Fig. 2b; r ¼ 0.71; P ¼ 1.9017 10 11 (t-test)). We performed a series of additional psychophysical tests on the new modality. First, we varied task difficulty by changing the angle between the ports. Moving the ports closer together increased uncertainty about the stimulus source, as the light from each IR source had a broad wavefront (Fig. 1a), and the IR detector had a relatively wide ‘receptive field’ (see Methods). While animals consistently performed above 90% of the trials correctly when the ports were 90° apart, this performance dropped off quickly as the angles between the IR sources were reduced below 60° (Fig. 2c; Supplementary Movie 3). Behavioural performance was significantly dependent upon task difficulty (P ¼ 1.1 10 19; analysis of variance). Because the rats swept their heads back and forth to sample the IR signals, we hypothesized that the rats were sensitive to the intensity of IR light, not just its presence or absence. To test this, in some sessions we randomly interspersed trials in which stimulation frequency was held constant, so the animal only
100
received binary information about IR presence (Fig. 2d, left). Performance was significantly degraded in such constantfrequency trials (P ¼ 0.0001; two-way analysis of variance), demonstrating that the rats were sensitive to graded IR signals (Fig. 2d, right). Rats are typically considered blind to the IR spectrum11: their cone spectral sensitivity is negligible above 650 nm14, which is well outside the spectral emission of our IR source (940 nm peak emittance, with a range of non-zero emission between 825 and 1000 nm; see Methods). However, in the sensory prosthetics literature there has been some concern about the true range of spectral visual sensitivity of the rat15, so we also experimentally checked whether our animals succeeded by using their visual system to discriminate the IR cues. In two animals we added random ‘no stimulation’ trials in which the IR light turned on, but there was no stimulation delivered to the cortex. Performance on the task was abolished in all 11 sessions, dropping from 69±3.2 to 8±5.2% correct, a significant change (P ¼ 7.6 10 6; paired t-test) (Fig. 2e). Percent correct was below 33% on the trials without stimulation because in the majority of trials the animals did not poke in any of the ports, and simply let the trial time out (see Online Supplementary Movie 4). Analysis of activity in S1. Next, we investigated the effects, within S1, of acquiring the new sensory modality. For instance, are some neurons ‘hijacked’ by IR inputs16, such that they no longer respond to whisker stimulation? To address this question, we recorded from multi-electrode arrays chronically implanted in S1 infragranular cortex in two well-trained rats (N ¼ 76 single units over five sessions). We presented three different stimuli,
3
75
50
Percent correct
Latency (s)
Percent correct
90 2
1
80
70 25
0
10
20 30 Session number
0
40
0
10
20 30 Session number
30
40
40
50 60 70 80 Difficulty (degrees)
90
Constant
80
Percent correct
IR level Frequency
100
90 PC variable
Frequency
100 Variable
70 60 50
50
0 IR level
50
60
70
80
90
100
No stim
Stim
PC constant
Figure 2 | Rats discriminate among IR sources using graded stimulation frequencies. (a) Learning curve for IR-only trials. Graph shows percentage of correct trials as a function of session number (130 sessions in four rats). Black circles/lines indicate mean/s.e.m. for blocks of three sessions. (b) Latency decreases as rats learn the task. Scatter plot of latency on IR-only trials, using same conventions as panel a (data are from 66 sessions in two rats). (c) Discrimination performance varies with angle between ports. Plot shows percentage of correct trials versus task difficulty (angle between ports). Data are from 100 sessions in two rats. (d) Performance degrades when stimulation frequency is constant rather than variable. Plot on the right shows performance on trials in which the stimulation frequency is variable (top left inset) versus held constant (bottom left inset). Data are from 34 sessions in two rats. (e) When stimulation is turned off, performance is abolished. Left bar shows performance on trials with IR light turned on, but no stimulation. The right bar shows the performance from the same sessions for the trials in which we stimulated as usual. Error bars are s.e.m. Data are from 11 sessions in two rats. NATURE COMMUNICATIONS | 4:1482 | DOI: 10.1038/ncomms2497 | www.nature.com/naturecommunications
& 2013 Macmillan Publishers Limited. All rights reserved.
3
174
175
ARTICLE
NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2497
each lasting 200 ms: unilateral whisker deflection (via air puffs), ICMS (using current magnitudes that were applied during the task) and both stimuli delivered simultaneously. S1 neurons were clearly not hijacked by ICMS, even after many months of microstimulation. In fact, most (83%) S1 neurons showed quite robust responses to whisker deflection (Fig. 3a,b). The breakdown of all response types is detailed in Fig. 3c. Briefly, of 76 units, 84% (64/76) responded to one or the other stimulus, while 16% (12/76) showed no significant response to either stimulus. The majority (83% (63/76)) showed significant responses to whisker deflection. Of these, 83% (52/63) were multimodal neurons, exhibiting significant responses to ICMS as well. Only one neuron in our sample showed a response to ICMS but not whisker deflection. Among the multimodal S1 neurons, the response to both stimuli delivered simultaneously was highly sublinear (Fig. 3d; P ¼ 4.6 10 17; two-tailed paired t-test). Population responses in S1 increased according to a saturating function of whisker deflection magnitude (Fig. 3e). Response magnitudes also increased with ICMS frequency up to 350 Hz, but dropped off at 400 Hz, likely due to adaptation (Fig. 3f). This neuronal response-profile is akin to an IR receptive-field for the range of IR stimulation frequencies employed in the task. Discussion In 1969, Bach-y-Rita performed the classic ‘sensory substitution’ experiments, in which visual stimuli were projected onto the skin via mechanical actuators, allowing congenitally blind patients to experience a visual world for the first time4. Here we have applied the logic of sensory substitution directly to the somatosensory cortex, bypassing the body’s periphery, with the goal of building a cortical sensory prosthesis capable of augmenting the subject’s perceptual capability5. Instead of giving a binary, top–down signal instructing the rats where to move17, we connected S1 to a new graded sensory cue available in their environment, and they spontaneously adopted novel foraging behaviours in response. Note that, in principle, this experimental paradigm could use any novel stimulus (for example, magnetic or radio waves) to be represented in S1. We observed that neurons in the stimulated regions of S1 maintained their ability to respond to whisker deflection. This suggests that two different cortical representations (one tactile and one IR) became superimposed on the animal’s S1 cortex, creating a novel bimodal processing region. However, it is important to emphasize that behavioural studies will be needed to determine the consequences, for whisker-based tactile discrimination, of adding this new information to S1. The mechanisms by which animals learn to use this new information source will also be an interesting avenue for future research, as such research should suggest how to accelerate sensory prosthetic acquisition. To investigate the mechanisms of plasticity, including those theories implicating glial cells18, it will be helpful to examine functional and structural changes in cortical and subcortical tactile processing centres as rats learn to discriminate among IR sources. Overall, our behavioural results suggest that animals initially treated S1 electrical stimulation as an unexpected whisker deflection, and eventually treated it as a stimulus originating away from the body in the surrounding environment. However, we are unable, using the methods in this paper, to determine whether the fully trained rats consciously experienced microstimulation as a new distal sensory modality, or simply learned to associate a tactile sensation with an otherwise imperceptible distal sensory cue. This is a question that could presently be addressed with sensory substitution experiments in humans. Indeed, one such study suggests that some subjects experienced tactile stimuli 4
as visual in nature after training with a visual-to-tactile peripheral substitution device19. The next generation of sensory neuroprosthetics research in humans will likely continue to rely on ICMS rather than optogenetics20,21. Indeed, our experimental strategy has been quite different from optogenetic approaches22–25: instead of stimulating a specific cell-type population, we indiscriminately stimulated all S1 neurons in the vicinity of the stimulating electrodes’ tips. Despite the fact that this unusual signal was delivered at extremely high frequencies, and likely spread through most of S126–28, these animals readily learned to exploit this new sensory channel. One potential application of the technology used in this study is in the design of a new generation of motor neuroprostheses that, instead of simply sending a brain-derived motor control signal to move a prosthetic limb, would provide continuous sensory feedback to the user’s brain from the prosthesis29–31. Such closed-loop bidirectional brain–machine–brain interaction could significantly improve reaction time, behavioural accuracy and likely aid the integration of the limb into the user’s internal body image9,30,32,33. However, there is another aspect of the present work that has rarely been explored in the neuroprosthetics literature: the potential to expand or augment a species’ normal perceptual range. In that pursuit, we have implemented, as far as we can tell, the first cortical neuroprosthesis capable of expanding a species’ perceptual repertoire to include the near IR electromagnetic spectrum, which is well outside the rat photoreceptors’ spectral sensitivity. Thus, by taking advantage of this novel paradigm, our rats were able to transcend the limitation of perceiving only those stimuli that can activate their bodies’ native sensory transducers. Methods Behavioural task and training. All experiments were performed on female Long Evans rats B14-weeks-old (B250 g; Harlan Laboratories). Rats were trained in a chamber with three reward ports that were situated 90° apart (Fig. 1a). Each port was fit with a visible broad-spectrum LED, and an infrared LED (Opto Semiconductors Inc., 940 nm peak emittance (range of non-zero emissions was between 825 and 1000 nm), and IR intensity dropped to half-max at 120°) (Fig. 1a). We first trained water-deprived rats to poke in the port whose visible LED was activated. Each trial began when a visible light in one randomly selected port was activated. Rats received water when they broke the photobeam in that port (correct trials). Trials counted as incorrect when the rats poked in a different port, or did not poke at all and let the trial terminate (this was set to occur between 15 and 20 seconds after the onset of the light). On incorrect trials, they received no water, an error tone, and a longer delay to the next trial. For some rats, we delivered air puffs to the face on incorrect trials. Once animals performed above a criterion value (470% correct) 44 days in a row, we implanted an array of stimulating electrodes into S1 (Fig. 1d). The array includes an IR detector attached to the connector (see Fig. 1c and Surgery below). After the animals recovered from surgery, we determined the minimal currents required to evoke a behavioural response in at least two electrode pairs (thresholds were between 1 and 200 mA, see Stimulating Electrodes and Stimulator below for details). We then trained them in the same behavioural chamber, but incrementally replacing visible light with ICMS linked to IR levels from their detector (Fig. 1e). Initially, each trial had the same structure as trials in the initial behavioral task described above, except that the onset of the visible light was preceded by IR-level dependent stimulation (up to 400 Hz) that lasted 0.6–1.5 s. In four of six rats, we began with brief durations of stimulation (600–700 ms) to acclimate them to stimulation. In two of the rats, we started with longer stimulation durations to get estimates of behavioural latency as they learned the task (Fig. 2b). Thus the animals learned that stimulation indicated the presence of the visible light. The stimulation frequency was exponentially proportional to the IR levels (Fig. 1e). We used an exponential function because IR intensity dropped logarithmically from the IR source. We stimulated at high frequencies for two reasons: one, in preliminary experiments we found that using a lower frequency range (that is, 0–100 Hz) yielded less reliable performance; two, we wished to avoid evoking kindling seizures34. Stimulation frequencies were updated every 50 ms based on IR levels, and it took B5 ms for the stimulation frequency to actually change in the rat once the command was sent. Pulse frequency was the only variable that tracked IR levels: current amplitudes were kept the same for each pulse. Once it became clear, based on visual inspection, that the animals were comfortable with stimulation, we added ‘IR-only’ trials in which the IR-dependent
NATURE COMMUNICATIONS | 4:1482 | DOI: 10.1038/ncomms2497 | www.nature.com/naturecommunications
& 2013 Macmillan Publishers Limited. All rights reserved.
ARTICLE
NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2497
a
b
Whisker only
Whisker only
ICMS only
Whisker+ICMS
50 50
100
100
50
50
Time (s)
No pref Whisker ICMS Whisker ICMS No resp
30
20
6 0.
4
2
0.
0.
0
Time (s)
*
*
*
1.5 Normalized spikes/s
40
.2
Time (s)
d 50
–0
0. 6 –0 .4
4 0.
2 0.
0
.2 –0
6
.4
0.
Time (s)
c
Number of neurons
–0
4 0.
–0
0.
.4
6 0.
4 0.
0.
–0
2
500
0
500
.2
20
.4
20
2
50
0
50
.2
100
10
–0
Firing rate (spikes/s)
50
–0
Firing rate (spikes/s)
20 100
1.0
0.5
10
0
0 Both
One
Neither
Whisker
e
f
ICMS
Both
Linear(E)
1.0
Normalized spikes/s
Normalized spikes/s
1.0
0.75
0.5
0.75
0.5
0.25
0
0.25 5
10
15
20
25
30
35
40
50
100
Air pressure (psi)
150
200
250
300
350
400
ICMS frequency (hz)
Figure 3 | Neurons in S1 respond to both whisker deflection and ICMS. (a) Sample peristimulus time histograms from neurons (rows) recorded simultaneously in an anesthetized animal during whisker deflections. The grey strip indicates time of whisker deflection (0–200 ms). Blue/red lines indicate mean± three s.d. values from the mean firing rate during the baseline period before stimulation. Rows 1–5 are single-unit responses, while row 6 is a representative multiunit response. (b) Sample peristimulus time histograms from the same neurons in panel a under three stimulus conditions. Left column: response to whisker deflection (38 psi air puff), but with the response during stimulation clipped out to enable comparison with the other two columns. Middle column: response to ICMS only (250 Hz). Right panel: response to simultaneous whisker deflection and ICMS. The light yellow strip indicates the time, after stimulation, we used to compare response magnitudes in the three cases. Note there are no responses during electrical stimulation because of electrical artifacts, so the firing rates drop to zero during those epochs. (c) Overall distribution of response types. Columns are segregated by whether neurons responded significantly to both stimuli, only one stimulus, or neither. (d) Mean ( þ s.e.m./ standard deviation) response of multimodal neurons to each of the three stimuli. The observed response to both stimuli is significantly sublinear (fourth bar indicates expected response under assumption of linearity). Asterisks indicate significance with Po1.0 10 6 (paired, two-tailed t-test). (e) Multiunit response magnitude (mean firing rate±s.e.m.) as a function of whisker deflection magnitude (intensity of air pressure applied to whiskers) for 16 channels in two animals. Zero is baseline (mean rate before stimulus onset), and responses for each channel were normalized to the maximum mean response over all three stimulus conditions. (f) Multiunit response magnitude versus ICMS frequency, with the same conventions as in panel e, with 13 channels in two animals. NATURE COMMUNICATIONS | 4:1482 | DOI: 10.1038/ncomms2497 | www.nature.com/naturecommunications
& 2013 Macmillan Publishers Limited. All rights reserved.
5
176
177
ARTICLE
NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2497
stimulation would appear without any accompanying visible light. That is, on these IR-only trials they could only use the signals from ICMS to get to the correct port. They trained on this until their performance on the IR-only trials reached a criterion value of 70% correct. For some rats that stayed above criterion for four sessions in a row, we then varied task difficulty in which across sessions, we pseudo-randomly chose a new angle between the ports (either 90, 60, 45 or 30 degrees; Fig. 2c). On some sessions, we added a small percentage of trials (B15%) in which the stimulation frequency was held constant, regardless of the IR intensity. Such constant-frequency trials allowed us to compare performance when the stimulation frequency depended on IR levels with those trials when stimulation frequency was a constant function of IR intensity (Fig. 2d, left hand side). All behaviour control was run in custom Matlab scripts using the data acquisition toolbox (run with NIDAQ PCI-7742 card, National Instruments). Stimulating electrodes and stimulator. In preliminary studies, we found that the thresholds for evoking behavioural responses were lower with electrode pairs within the same penetration of the cortex. Hence, for each biphasic stimulating electrode, we joined pairs of 30-mm stainless steel microwires to one another, each pair separated by 300 mm (Fig. 1c inset). We attached a single infrared detector (Lite-On Inc) to the connector (Omnetics), and powered the IR detector through the two extra reference pins on the connector. The phototransistor in the detector had a peak spectral sensitivity at a wavelength of 940 nm. The range of sensitivity was from 860 to 1020 nm, and its ‘receptive field’ (that is, the angular range within which a 940 nm test stimuli would evoke a response from the detector) was 20° in diameter at half-max (Fig. 1a). We used bipolar stimulation with charge-balanced, biphasic pulse trains, using a custom-controlled stimulator, as described elsewhere35. Pulses were 100 ms in duration, with 50 ms between the cathodic and anodic phases of the pulses (Fig. 1e). Current magnitudes varied between 1 and 300 mA. Before training rats on the IRversion of the task, we determined current thresholds by placing them in the empty behavioural chamber (Fig. 1a), and stimulating with 1 mA at 200 Hz for 500 ms. If an animal noticeably moved in response to stimulation (this typically involved locomoting, moving their heads to the side or scratching at their faces), that current level was taken as the threshold current. If not, we increased the current amplitude by 50%, keeping the frequency and duration the same, until we noted such a response. We used electrical microstimulation in lieu of optogenetics in this study, as one goal is to integrate this technology into human studies in the near future (see Discussion). Surgery. Detailed surgical procedures are described elsewhere36. Briefly, we implanted the stimulating electrodes into S1 ( 2.5 mm posterior and 5.5 mm relative to bregma, 1.5 mm deep) under pentobarbital anaesthesia (0.065 mg g 1), and the rats were given at least a week to recover from the surgery before being deprived again. The Duke University Institutional Animal Use Committee approved all surgical and behavioural methods. Neural recording and sorting. The basic recording set-up is described in detail elsewhere36. Namely, on those channels that we were not using for stimulation, we recorded neural activity using the (Multichannel Acquisition Processor (Plexon, Inc., Dallas, Texas)). To stimulate and record simultaneously without damaging the head stage, we peeled off the wires connected to the stimulator from the connector to bypass the head stage, while the other channels went directly to the head stage. Sorting of neural data is also described elsewhere36. Briefly, in addition to template-based online sorting, all voltage traces around a thresholdcrossing event were saved for offline sorting. For offline sorting, we used clustering in principal component space, signal to noise ratio and autocorrelation functions that showed a clear absolute and relative refractory period to determine whether the data came from single units or multiple units. In our recordings, we lightly anesthetized head-fixed animals under isofluorane anaesthesia (0.8–3% isofluorane mixed with pure oxygen). We controlled the duration and magnitude (psi) of air via a modified fluid dispenser (Oki DX-250, Garden Grove, CA), which was controlled with TTL pulses sent from Matlab. Before recording, we positioned the air dispenser to stimulate the majority of the large whiskers, B10 mm from the whisker pad. We delivered three different stimuli, in pseudo-random order: air puff to whiskers, electrical stimulation and both stimuli delivered simultaneously (see Fig. 3). Statistical methods. We calculated significant responses in peristimulus time histograms using a bootstrap cumulative sum algorithm described in more detail elsewhere37. This involved taking bootstrap samples of the cumulative response during a baseline period (period before time zero in Fig. 3a), and using these to generate a cumulative sum 95% confidence interval (with Bonferonni correction for multiple comparisons), and comparing this to the actual cumulative sum during the period of stimulation. A significant response was when the actual cumulative sum left the 95% confidence interval, and ended when the response dropped below a 95% confidence interval estimated from bootstrapping the original data. During electrical stimulation, recordings were saturated by stimulus artifact, so to directly compare responses between whisker deflection and ICMS, we compared 6
mean firing rates during the 130 ms period following the offset of both stimuli. To compare relative response magnitudes among multiple neurons, we normalized the mean response to each stimulus by the maximum mean response over all three stimulus conditions, so the maximum response was assigned a value of one. To generate the predicted responses to both stimuli delivered simultaneously (Fig. 3c), we used the normalized responses described in the previous paragraph. For a linear system, the predicted response to both stimuli would simply be the sum of the responses to each stimulus taken individually. That is, R(E þ W) ¼ R(E) þ R(W), where E is electrical stimulation, W is whisker deflection, and R(x) signifies the response to input x.
References 1. Wilson, B. S. et al. Better speech recognition with cochlear implants. Nature 352, 236–238 (1991). 2. Dobelle, W. H. Artificial vision for the blind by connecting a television camera to the visual cortex. ASAIO J. 46, 3–9 (2000). 3. Zrenner, E. et al. Subretinal electronic chips allow blind patients to read letters and combine them to words. Proc.. Biol. Sci./The Roy. Soc. 278, 1489–1497 (2011). 4. Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B. & Scadden, L. Vision substitution by tactile image projection. Nature 221, 963–964 (1969). 5. Nagel, S. K., Carl, C., Kringe, T., Martin, R. & Konig, P. Beyond sensory substitution--learning the sixth sense. J. Neural. Eng. 2, R13–R26 (2005). 6. Frost, D. O. & Metin, C. Induction of functional retinal projections to the somatosensory system. Nature 317, 162–164 (1985). 7. Sur, M., Garraghty, P. E. & Roe, A. W. Experimentally induced visual projections into auditory thalamus and cortex. Science 242, 1437–1441 (1988). 8. von Melchner, L., Pallas, S. L. & Sur, M. Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature 404, 871–876 (2000). 9. Nicolelis, M. Beyond Boundaries: The New Neuroscience of Connecting Brains With Machines--And How It Will Change Our Lives. 1st edn, (Times Books/ Henry Holt and Co.) (2011). 10. Stevenson, I. H., Cronin, B., Sur, M. & Kording, K. P. Sensory adaptation and short term plasticity as Bayesian correction for a changing brain. PloS one 5, e12436 (2010). 11. Muntz, W. R. A behavioural study on photopic and scotopic vision in the hooded rat. Vision Res. 7, 371–376 (1967). 12. Penfield, W. & Boldrey, E. Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain 60, 389–443 (1937). 13. Romo, R., Hernandez, A., Zainos, A. & Salinas, E. Somatosensory discrimination based on cortical microstimulation. Nature 392, 387–390 (1998). 14. Jacobs, G. H., Fenwick, J. A. & Williams, G. A. Cone-based vision of rats for ultraviolet and visible lights. J. Exp. Biol. 204, 2439–2446 (2001). 15. Pardue, M. T. et al. Visual evoked potentials to infrared stimulation in normal cats and rats. Doc. Ophthalmol. 103, 155–162 (2001). 16. Griffin, D. M., Hudson, H. M., Belhaj-Saif, A. & Cheney, P. D. Hijacking cortical motor output with repetitive microstimulation. J. Neurosci. 31, 13088–13096 (2011). 17. Talwar, S. K. et al. Rat navigation guided by remote control. Nature 417, 37–38 (2002). 18. Chen, N. et al. Nucleus basalis-enabled stimulus-specific plasticity in the visual cortex is mediated by astrocytes. Proc. Natl Acad. Sci. USA 109, E2832–E2841 (2012). 19. Ortiz, T. et al. Recruitment of occipital cortex during sensory substitution training linked to subjective experience of seeing in people with blindness. PloS one 6, e23264 (2011). 20. Ethier, C., Oby, E. R., Bauman, M. J. & Miller, L. E. Restoration of grasp following paralysis through brain-controlled stimulation of muscles. Nature 485, 368–371 (2012). 21. Witten, I. B. et al. Recombinase-driver rat lines: tools, techniques, and optogenetic application to dopamine-mediated reinforcement. Neuron 72, 721–733 (2011). 22. Choi, G. B. et al. Driving opposing behaviors with ensembles of piriform neurons. Cell 146, 1004–1015 (2011). 23. Gerits, A. et al. Optogenetically induced behavioral and functional network changes in primates. Curr. Biol. 22, 1722–1726 (2012). 24. Huber, D. et al. Sparse optical microstimulation in barrel cortex drives learned behaviour in freely moving mice. Nature 451, 61–64 (2008). 25. Lee, S. H. et al. Activation of specific interneurons improves V1 feature selectivity and visual perception. Nature 488, 379–383 (2012). 26. Ferezou, I. et al. Spatiotemporal dynamics of cortical sensorimotor integration in behaving mice. Neuron 56, 907–923 (2007). 27. Frostig, R. D., Xiong, Y., Chen-Bee, C. H., Kvasnak, E. & Stehberg, J. Large-scale organization of rat sensorimotor cortex based on a motif of large activation spreads. J. Neurosci. 28, 13274–13284 (2008).
NATURE COMMUNICATIONS | 4:1482 | DOI: 10.1038/ncomms2497 | www.nature.com/naturecommunications
& 2013 Macmillan Publishers Limited. All rights reserved.
ARTICLE
NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2497
28. Ghazanfar, A. A. & Nicolelis, M. A. Feature article: the structure and function of dynamic cortical and thalamic receptive fields. Cereb. Cortex 11, 183–193 (2001). 29. Nicolelis, M. A. Brain-machine interfaces to restore motor function and probe neural circuits. Nat. Rev. Neurosci. 4, 417–422 (2003). 30. O’Doherty, J. E. et al. Active tactile exploration using a brain-machine-brain interface. Nature 479, 228–231 (2011). 31. Dhillon, G. S. & Horch, K. W. Direct neural sensory feedback and control of a prosthetic arm. IEEE Trans Neural Syst Rehabil Eng 13, 468–472 (2005). 32. Koralek, A. C., Jin, X., Long, 2nd J. D., Costa, R. M. & Carmena, J. M. Corticostriatal plasticity is necessary for learning intentional neuroprosthetic skills. Nature 483, 331–335 (2012). 33. Marasco, P. D., Kim, K., Colgate, J. E., Peshkin, M. A. & Kuiken, T. A. Robotic touch shifts perception of embodiment to a prosthesis in targeted reinnervation amputees. Brain 134, 747–758 (2011). 34. Goddard, G. V. Development of epileptic seizures through brain stimulation at low intensity. Nature 214, 1020–1021 (1967). 35. Hanson, T. et al. High-side digitally current controlled biphasic bipolar microstimulator. IEEE Trans Neural Syst Rehabil Eng 20, 331–340 (2012). 36. Wiest, M., Thomson, E. & Meloy, J. In: Nicolelis, M. A. L. (ed). Methods for Neural Ensemble Recordings Frontiers in Neuroscience (2008). 37. Wiest, M. C., Thomson, E., Pantoja, J. & Nicolelis, M. A. Changes in S1 neural responses during tactile discrimination learning. J. Neurophysiol. 104, 300–312 (2010).
Acknowledgements We thank J. O’Doherty who helped with experimental design, hardware and software set-up, and provided manuscript comments; A. McDonough, J. Lou, S. Tica, A. Huh,
E. Lehew, trained animals; P. Lee trained animals and performed histology; G. Lehew and J. Meloy helped build the experimental set-up and recording electrodes; T. Hanson and A. Fuller built the stimulators; M. Lebedev provided manuscript comments; M. Coleman and K. Sylvester helped with neurophysiology and behavioural training. S. Halkiotis helped edit and prepare the manuscript for publication; M. Viera helped in experimental design. This work was funded by NIH (NIMH) DP1MH099903, and NIH R01DE011451 to MALN. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Author contributions E.E.T. and M.A.L.N. designed the experiments and wrote the paper; E.E.T. and R.C. carried out behavioural experiments; E.E.T. performed surgeries, analysed and collected neural recordings.
Additional information Supplementary Information accompanies this paper at http://www.nature.com/ naturecommunications Competing financial interests: The authors declare no competing financial interests. Reprints and permission information is available online at http://npg.nature.com/ reprintsandpermissions/ How to cite this article: Thomson, E.E. et al. Perceiving Invisible Light through a Somatosensory Cortical Prosthesis. Nat. Commun. 4:1482 doi: 10.1038/ncomms2497 (2013).
NATURE COMMUNICATIONS | 4:1482 | DOI: 10.1038/ncomms2497 | www.nature.com/naturecommunications
& 2013 Macmillan Publishers Limited. All rights reserved.
7
178
179 2406 • The Journal of Neuroscience, February 24, 2016 • 36(8):2406 –2424
Systems/Circuits
Embedding a Panoramic Representation of Infrared Light in the Adult Rat Somatosensory Cortex through a Sensory Neuroprosthesis X Konstantin Hartmann,1,6* Eric E. Thomson,1* Ivan Zea,1 X Richy Yun,1 X Peter Mullen,1 X Jay Canarick,1 X Albert Huh,7 and Miguel A. L. Nicolelis1,2,3,4,5 Departments of 1Neurobiology, 2Biomedical Engineering, and 3Psychology and Neuroscience, and 4Center for Neuroengineering, Duke University Medical Center, Durham, North Carolina 27710, 5Edmond and Lily Safra International Institute of Neuroscience of Natal (ELS-IINN), 59066-060, Natal, Brazil, 6Bernstein Center for Computational Neuroscience, Humboldt University of Berlin, 10115 Berlin, Germany, and 7Georgia Regents University-Medical College of Georgia, Augusta, Georgia 30912
Can the adult brain assimilate a novel, topographically organized, sensory modality into its perceptual repertoire? To test this, we implemented a microstimulation-based neuroprosthesis that rats used to discriminate among infrared (IR) light sources. This system continuously relayed information from four IR sensors that were distributed to provide a panoramic view of IR sources, into primary somatosensory cortex (S1). Rats learned to discriminate the location of IR sources in ⬍4 d. Animals in which IR information was delivered in spatial register with whisker topography learned the task more quickly. Further, in animals that had learned to use the prosthesis, altering the topographic mapping from IR sensor to stimulating electrode had immediate deleterious effects on discrimination performance. Multielectrode recordings revealed that S1 neurons had multimodal (tactile/IR) receptive fields, with clear preferences for those stimuli most likely to be delivered during the task. Neuronal populations predicted, with high accuracy, which stimulation pattern was present in small (75 ms) time windows. Surprisingly, when identical microstimulation patterns were delivered during an unrelated task, cortical activity in S1 was strongly suppressed. Overall, these results show that the adult mammalian neocortex can readily absorb completely new information sources into its representational repertoire, and use this information in the production of adaptive behaviors. Key words: barrel cortex; rat; sensory prosthetic; whisker system
Significance Statement Understanding the potential for plasticity in the adult brain is a key goal for basic neuroscience and modern rehabilitative medicine. Our study examines one dimension of this challenge: how malleable is sensory processing in adult mammals? We implemented a panoramic infrared (IR) sensory prosthetic system in rats; it consisted of four IR sensors equally spaced around the circumference of the head of the rat. Each sensor was coupled to a microstimulating electrode placed in the somatosensory cortex of the rat. Within days, rats learned to use the prosthesis to track down items associated with IR light in their environment. This is quite promising clinically, as the largest demand for sensory prosthetic devices is in adults whose brains are already fully developed.
Introduction In recent decades, sensory prosthetic systems such as cochlear implants (Wilson et al., 1991) have helped tens of thousands of
Received Aug. 28, 2015; revised Jan. 8, 2016; accepted Jan. 13, 2016. Author contributions: K.H., E.E.T., R.Y., A.H., and M.A.L.N. designed research; K.H., E.E.T., I.Z., R.Y., P.M., J.C., and A.H. performed research; K.H. and E.E.T. analyzed data; K.H., E.E.T., and M.A.L.N. wrote the paper. This work was funded by the National Institutes of Health (NIH) Director’s Pioneer Award DP1-MH-099903, and NIH Grant R01-DE-011451 to M.A.L.N. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Office of the NIH Director or the NIH. We thank G. Lehew and J. Meloy for help building the experimental setup and electrodes; T. Hanson, A. Fuller, and J. O’Doherty for help setting up and
patients regain functional levels of sensory processing (Macherey and Carlyon, 2014). In the future, these peripheral prosthetic devices will likely be complemented by systems with higher throughput that directly activate neurons in the CNS, in particular the cerebral cortex (Dobelle and Mladejovsky, 1974; Dobelle, troubleshooting stimulators; S. Halkiotis for editing and preparing the manuscript; M. Viera for advice at multiple stages of the experiments; Manu Raghavan for statistics advice; Derek Silva Moreira for experimental help; A. Graneiro for help troubleshooting the setup and running experiments; D. Guggenmos for advice with the histology; anad J. Pedowitz for help in training animals. *K.H. and E.E.T. contributed equally to this work.
180 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
2000; Coulombe et al., 2007; Srivastava et al., 2009; Davis et al., 2012; O’Doherty et al., 2012; Andersen et al., 2014; Lewis et al., 2015). Such central neuroprosthetic systems could eventually become as routine in the treatment of sensory deficits as deep brain stimulation has become in the treatment of Parkinson’s disease (Deuschl et al., 2006; Fuentes et al., 2009; Santana et al., 2014; Yadav et al., 2014) and even some psychiatric disorders such as depression (Mayberg et al., 2005). To work properly, cortical sensory neuroprosthetics will require that the adult brain be plastic enough to continuously process real-time streams of synthetic information sources, and then use the extracted information to guide appropriate behavioral responses. Classically, the highest levels of cortical plasticity have been observed during early development (Frost and Metin, 1985). For instance, in a set of rewiring experiments in newborn ferrets, primary auditory cortex was induced to process visual information when visual information was routed to A1 (Sur et al., 1988). Following this procedure, individual neurons in rewired A1 exhibited visual response properties typically associated with neurons in V1, such as center-surround receptive fields (RFs; Frost and Metin, 1985; Sur et al., 1988; Horng and Sur, 2006), and rewired A1-mediated visual behaviors in these transformed animals (von Melchner et al., 2000). Recently, we reported similar findings in adult rats in which information from a single infrared (IR) sensor was delivered, via cortical microstimulation, to primary somatosensory cortex (S1; Thomson et al., 2013). After a month of training with this hardware, rats readily discriminated the location of IR sources in the environment in exchange for a water reward. This cortical prosthesis completely bypassed the native sensory transducers of the animals, delivering sensory information never experienced by these animals into a region of cortex already devoted to processing somatosensory information. This type of sensory enrichment system was recently extended when the output of a geomagnetic compass was coupled to stimulating electrodes in S1 of rats placed in a T-maze (Norimoto and Ikegaya, 2015). Overall, these experiments show that the adult mammalian brain can adaptively absorb completely new sources of sensory information delivered directly to cortical tissue. They also demonstrate the viability of using rodent models in the development of cortical sensory prostheses. However, several questions left open by these initial studies will have to be addressed for the development of more realistic neuroprosthetic systems. For instance, how does the brain respond to these qualitatively new information sources? Will animal performance improve with the addition of more sensors, or would this additional information be a source of confusion? Does the spatial distribution of the stimulating electrodes matter? If different combinations of sensors were inactivated, would behavioral performance degrade gracefully, the way natural sensory systems do? Many of these questions would be impossible to ask using a neuroprosthesis containing only a single transducer. Thus, in the present study we have extended the IR discrimination paradigm to include four IR sensors distributed to provide a full panoramic (360°) representation of the surrounding IR environment (Fig. 1A). The rats were not confused by the additional sensors and stimulating electrodes, but learned the task an order of magniThe authors declare no competing financial interests. Correspondence should be addressed to Dr. Miguel A. Nicolelis, Box 3209, Department of Neurobiology, Duke University, Durham, NC 27710. E-mail: nicoleli@neuro.duke.edu. DOI:10.1523/JNEUROSCI.3285-15.2016 Copyright © 2016 the authors 0270-6474/16/362407-19$15.00/0
J. Neurosci., February 24, 2016 • 36(8):2406 –2424 • 2407
tude faster than when equipped with just a single IR sensor. In animals trained to discriminate IR sources, individual S1 neurons exhibited both IR and robust tactile evoked responses. These and many other neurophysiological adaptations described below underlie the emergence of a multimodal IR tactile representation in the adult rat S1, allowing animals to perform an IR discrimination task at nearly 100% accuracy.
Materials and Methods Training and behavioral task. All experiments were conducted on female Long–Evans rats ⬃10 weeks old (200 –224 g; Harlan Sprague Dawley). For behavioral training, rats were trained in a circular chamber (50.8 cm diameter) with four reward ports that were situated 90° apart. The reward was given through a poke hole that was fit with a photo beam. Each port was additionally fit with a visible broad-spectrum LED, and an infrared LED (Opto Semiconductors; 940 nm peak emittance; range of nonzero emissions, 825–1000 nm). The IR sources had an angular width at halfmaximum of 120°). A custom-made pushbutton was affixed to the floor in the center of the chamber, which rats pressed to initiate each trial (27 mm diameter; Fig. 1A). Initially, 17 rats were pretrained in a simple visual discrimination task, in which they were placed in a circular chamber with four water ports that contained, in a vertical arrangement, a visible-light LED, an infrared LED, and a water spout (Fig. 1A). In the initial pretraining, rats learned a very simple visual discrimination task in which one of the four visible lights turned on, and animals were rewarded when they selected that port. Selection was indicated by breaking the photobeam in front of the water spot. Once the performance of the animals crossed 85% correct on the visible light (see Materials and Methods) version of the task (which took 9 ⫾ 3 d), we then prepared them for the IR discrimination training. Once animals learned to use the center button to start a trial, they normally performed well above 95% correct. First, four groups of eight microelectrodes were chronically implanted in the whisker representation— barrel cortex— of both S1s of each rat (see Stimulating electrode implantation surgery). Each microelectrode group was associated with a different IR detector (Fig. 1B). We chose this approach as the most natural way to associate each IR detector with a topographic homolog in the rat S1. So, for instance, the front-right IR detector was intended to stimulate the region of S1 corresponding to the right-front whiskers, corresponding to the left anterior barrel field in S1. Next, we retrained rats on the IR version of the original discrimination task, replacing visible light with IR light. Thus, for rats to be rewarded in a trial they now had to track the IR beam all the way to the port from which the IR beam was being emitted. For each session, the four IR detectors were snapped into place on the head of each animal via a magnetic seal. The output of each IR detector was independently connected to the activity in the stimulation channel in the appropriate quadrant of S1, so, for instance, if the head of the animal was oriented so that the right-front IR detector faced the active IR source, then the left-front region of S1 would be highly activated (Figs. 1C–E). The frequency of stimulation in each stimulating channel varied approximately exponentially with IR levels in its corresponding IR detector (Thomson et al., 2013). We excluded two animals from subsequent analysis: one developed an infection at the site of implantation, and the acrylic head cap came off the other after surgery. Initial current levels for microstimulation were found by stimulating in the lightly anesthetized animal a week after surgery. For each of the four locations, we determined the minimal threshold for at least one pair of stimulating electrodes, defined as the smallest current we could deliver (20 pulses at 200 Hz) and evoke a clear response in S1 (as determined by listening to responses in an audio monitor). We then trained animals in the same behavioral chamber, but replaced the visible light with intracortical microstimulation (ICMS) proportional to the level of IR light detected by a sensor. A trial was counted correct when the rat poked the port with the activated IR light. After the animals were able to perform the task solely with IR light and met the criterion of reaching 85% correct, we sometimes conducted a series of
181 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
2408 • J. Neurosci., February 24, 2016 • 36(8):2406 –2424
Port 2
C
Port 1
Button
2
1
3
4
Current [μA]
B
N1
N2
N4
N3
Port 3
IR Intensity [V]
A
Time [s]
Port 4 90 120 150 Sensor 1 Sensor 2 Sensor 3 Sensor 4
E
60
180
30
0
180
330
210 300
240 270
FWHM (degrees)
D
135 90 45 0 0
0
Figure 1. Methods for rat IR discrimination training. A, Schematic of the behavioral chamber used for the task. Four ports are arranged symmetrically around the inner surface of a large (24 inch) cylinder. Each port contains a nose poke, an IR light, and a visible light. In the middle of the floor is a button the rat pushes to initiate a trial. B, Topographical organization of information from four IR detectors in rat S1. The IR sensors were placed 90° apart from each other, with each sensor coupled to a different stimulating electrode pair in S1. Note that the representation of the left whiskers is found in the right hemisphere, but the front-facing whiskers are still represented on the anterior part of the barrel field in S1 (N1 through N4 represent the four locations of neuronal stimulation). C, Stimulation frequency depended on IR intensity in each sensor. The intensity of each IR light was translated into different stimulation frequencies, in real time, in its corresponding stimulation channel. D, Polar plot showing the response of each IR sensor as a function of angle in the chamber, when the sensor array is at a fixed position in the chamber, relative to a single activated IR source. The red point on the edge indicates the relative location of the IR source. E, Full-width at half-maximum (FWHM) of the response profiles as a function of position in the chamber. The red point is the position of the active IR source, while the FWHM is the mean FWHM of all four sensors at the given position (see D). As you move further way, or oblique, to the source, the response profiles become narrower. The black point indicates the position of the data represented in D.
additional experiments, which are described below (e.g., remapping experiments). All behavioral and microstimulator controls were performed using custom MATLAB scripts using the data acquisition toolbox (run with NIDAQ PCI-7742 card, National Instruments). Stimulating electrode implantation surgery. We implanted the electrodes bilaterally in S1 under full anesthesia. To initiate the anesthesia, rats were first put to sleep with isoflurane (Isothesia, Henry Schein Animal Health). For final anesthesia, we used a ketamine (Ketaset, Fort Dodge)-xylazine (AnaSed, Akorn Animal Health) anesthetic with 100 mg/kg ketamine and 0.06 mg/kg xylazine. During surgery, supplemental doses (33% of the initial dose, administered intramuscularly) of ketamine were provided when necessary. Throughout surgery, we delivered at least 3 ml of saline (Hospira) every 2–3 h, injected subcutaneously. Further, 0.02 mg/kg atropine (Atropine, West-Ward Pharmaceuticals) was given within the first hour and then every 2–3 h until the animal regained consciousness. Surgeries took between 3.5 and 6 h, excluding recovery from anesthesia. Coordinates for the craniotomy were taken with the stereotactic tools relative to bregma at 2.5 mm posterior and 5.5 mm bilateral. Electrodes were lowered 1.5 mm into the brain at an angle of 10°. Electrode pairs in one penetration were 300 m apart, so the electrode tips of each pair were positioned at 1.5 and 1.2 mm below the surface of the cortex (Thomson et al., 2013). The craniotomy was covered with eye ointment before the electrodes were fixed in place, and the craniotomy was sealed using dental cement (Perm Reline, Coltene/Hobbylinc).
After surgery, the rats were given at least 7 d to recover. During this time, we provided them with Tylenol for pain relief and monitored their weight. The Duke University Institutional Animal Care and Use Committee approved all surgical and behavioral methods. Four IR sensor neuroprosthetic. The electrodes, implanted in the somatosensory cortex, formed the first part of the neuroprosthetic. Electrodes were implanted in each hemisphere (Fig. 2B). In both hemispheres, the layout was identical and consisted of 16 electrodes and one ground electrode. Preliminary studies revealed that the thresholds for evoking behavioral responses were lower when two electrodes were paired in the same penetration of the cortex (Thomson et al., 2013). For our stimulating and recording electrodes, we therefore joined two 42 m stainless steel polyimide insulated microwires. The second part of the neuroprostheses consisted of an attachable cap that included four IR sensors. The phototransistor in the IR sensors (Lite-On) had a peak spectral sensitivity at a wavelength of 940 m with a range of sensitivity of 860 –1020 m. The sensor had a 20° width at half of its maximum sensitivity. The four infrared sensors were placed 90° apart from each other on a horizontal plane (Fig. 1B). The head cap could be plugged into the connector (Omnetics Connector Corporation) on the head of the rat and was sealed in place with a small magnet. To more fully characterize the response profile of the system, the IR sensor response as a function of IR sensor angle and position in the chamber was measured off-line in the behavioral chamber. This was done by affixing the panoramic IR sensor array onto a stepper motor (Portescap) that was rotated in 15° increments, with the response of the
182 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
B
60 40 20 0
1
2
D
3 4 5 Session #
80
4 3 2 1
6 180°
E
**
5 Percent Correct [%]
80
100
C
Response Latency [s]
100
Percent Correct [%]
A
J. Neurosci., February 24, 2016 • 36(8):2406 –2424 • 2409
1
2
3 4 5 Session #
60 40 20 0
6
4 IR Sensors 1 IR Sensor 0
5
180°
F
10
15 20 25 Session #
30
35
Arena Layout
G
15
0.05
0.05
0.1
0.1
5
0.15
0.15 1 IR
0° After Learning
0° Before Learning
0
1 2 3 4 Number Ablated
90
20
2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
3
4
3
4
3
4
3
4
3
4
3
4
3
4
3
4
3
4
60 45 Angle [°]
1 2 1 2 4 3 3 4
2
1
3
4
30
Ch4 on
Ch3 on
Ch2 on
Ch1 on
40
20 0
4 IR Sensors 1 IR Sensor
Left off
***
70
Right off
40
Back off
60
*** **
60 Front off
***
80
60
Ch4 off
80
***
**
**
80
Ch1 off
*
Percent Correct [%]
100
90
100
I
All on
H
Ch3 off
4 IR
Percent Correct [%]
Percent Correct [%]
100
Ch2 off
# Sessions
25
2
1
2
1
2
1
3
4
3
4
3
4
Condition Figure 2. Learning to use distributed IR information to discriminate IR sources. A, The mean percentage of correct responses for the first sessions from eight animals (mean ⫾ SEM). The dotted line marks the 85% threshold. Also, see Movie 1. B, The latency of the response (time between stimulation onset and animal breaking a photobeam in one of the ports) over different sessions, from the same rats as in A. C, Comparison of rats with one IR sensor (gray) and rats with four IR sensors (black) implanted on the head. The black line is the same data shown in A. The one sensor group consists of three animals and is averaged with a sliding window over 3 d. D, The number of sessions to reach 85% correct was significantly different for one vs four IR detectors. Rats with four IR sensors needed 3.75 ⫾ 0.45 sessions to reach 85% correct, while rats with only one IR sensor needed 22.3 ⫾ 7.62 sessions. A two-sided t test shows a significant difference between both groups ( p ⫽ 0.0019). E, F, Circular histogram of head direction at trial onset relative to the last visited port. E, The head direction before they learned to use the IR implant. F, The head direction after they learned to use the IR implant. Ports are located at 0°, 90°, 180°, and 270°. Before they learned, they faced 178° away of the last visited port, and after they learned they faced 155° away from that port. G, Performance vs task difficulty. The schematics above show the layout of the task chamber for the four-IR sensor implant. Rats with four IR detectors got ⬎85% correct even when the ports were 30° apart. The figure compares the performance of rats with four IR sensors (N ⫽ 6; red line) vs one IR sensor (N ⫽ 3; green line). The graph shows the mean percentage of correct responses, and the error bars show the SEM. Rats with four IR sensors showed a trend of decrease in performance in IR-only trials for smaller angles ( p ⫽ 0.132, ANOVA). The performance of rats with only one sensor dropped significantly for 45° and 30° ( p ⫽ 0.0008 and p ⬍ 0.0001, respectively, multiple-comparison test). For 45° and 30°, rats with four IR sensors performed significantly better than rats with only one IR sensor ( p ⬍ 0.0001 and p ⫽ 0.0029 respectively, multiple-comparison test). H, The performance decreased significantly as more sensors were deactivated ( p ⬍ 0.0001, one-way ANOVA). The graph shows the mean/SEM of behavioral performance, the dotted line is chance performance (25%). Data are pooled from six animals, disregarding which channels were deactivated. The performance with only one of the four sensors was at 52.8 ⫾ 2.07% (percentage correct ⫾SEM) and therefore was still above the chance level of 25%, regardless of which sensor was still active. When all sensors were deactivated, the performance dropped to 14.5 ⫾ 3.77%. A post hoc test showed that, with the deactivation of each sensor, the performance decreased significantly. I, Behavioral performance when specific channel combinations were inactivated shows that certain channels show a stronger influence on the performance than others. Inactivating of single sensors showed a trend of reduced performance but was not significant. Surprisingly, when both front-facing sensors were deactivated, the drop in performance was not significant ( p ⫽ 0.363, multiple-comparison test). In contrast, a significant decrease occurred when both back, both left, or both right sensors were deactivated (all p ⬍ 0.0001 when compared with all sensors active condition, multiple-comparison test). No difference was found between the left and right sensors ( p ⬎ 0.99, multiple-comparison test). When three of four sensors have been deactivated, the performance dropped significantly (all p ⬍ 0.001). For comparison of p values across all ablation conditions, see Table 1.
183 2410 • J. Neurosci., February 24, 2016 • 36(8):2406 –2424
Movie 1. Panoramic IR discrimination, related to Figure 2. A rat locates the source of the IR light with the neuroprosthetic. The movie is captured with an IR-sensitive camera. Each trial is initiated when the rat presses the center button and a choice is made by poking the nose into the correct port. Audio signal is a direct feed from the microstimulator, and is not available to the rat. IR sensors measured at each angle to yield a full IR response profile at a given position in the behavioral chamber (Fig. 1D). Such measurements were made at each position in the behavioral chamber, at 1 inch spatial resolution. The main differences in responses at each position were the angular preference, and width at half maximum of the response. The latter is shown as a function of position in the chamber (Fig. 1E). The in-house-made stimulator used a bipolar stimulation with charge-balanced, biphasic pulse trains (Hanson et al., 2012). The frequency of stimulation depended on the intensity of the detected IR light. We used seven different frequencies, ranging from 0 to 425 Hz (Thomson et al., 2013). Neural recording. The basic setup for recording neuronal activity has been described in detail previously (Wiest et al., 2008; Thomson et al., 2013). Briefly, in the channels we did not use for stimulation, we recorded neural activity using the Multichannel Acquisition Processor (MAP; Plexon). To stimulate and record simultaneously, the wires connected to the stimulator bypassed the head stage, while the other channels went directly to the head stage. Sorting of neural data is also described previously (Wiest et al., 2008). Briefly, in addition to templatebased on-line sorting, all voltage traces around a threshold-crossing event were saved for off-line sorting. For off-line sorting, we used clustering in principal component space and refractory periods to assign data either as single units or multiunits. Sensor ablation. To confirm that the animals used all four IR sensors and not only a subset of them, in six rats we ran whole sessions during which we deactivated the ICMS in one or multiple locations. The number of ICMS locations that were deactivated ranged from zero (all active) to four (all deactivated). We used different combinations of deactivated channels to test for preferences of a side or an orientation. The deactivation of all four ICMS sites was a necessary control to test whether the rats relied solely on the stimulation itself or whether they used other cues to guide their behavior. We used three to five different ablations in one session, with at least 40 trials per ablation; the order in which the ablations were presented was randomly determined. A session in this experiment consisted of 140 –220 trials, depending on the number of different ablations used in the session, 20% of which were trials with visible light. Varying angles between reward ports. As a common measurement of a sense (modality), we were interested in the spatial resolution of the ability of the rats to discriminate IR sources. We moved the reward ports closer together so that the angle between them within a session was 90°, 60°, 45°, or 30°. A session consisted of 60 trials at two or three angles, with 20% of trials with visible light. The order in which the angles were presented was randomly determined. This was done with six rats. IR sensor and S1 remapping. To test the ability to adapt to a new input structure, and determine whether the rats were using the spatially distributed information provided by the prosthesis, we changed the connection between the four sensors and the four ICMS sites. We used three different permutations: front-back flip, where the front sensors controlled the
Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
stimulation in the posterior ICMS sites and vice versa; left-right flip, where the left sensors stimulated the left hemisphere and vice versa; and a diagonal flip, a combination of the left-right flip and the front-back flip. In the diagonal flip, the front right sensor controlled the ICMS in the posterior barrel cortex in the right hemisphere, and all other channels were connected respectively. One remapping was applied consistently until the rats learned to use the new mapping. The threshold of learning was a performance of ⬎85% correct in IR-only trials. In these experiments, to start, 100% of the trials had only the IR light as a cue. If the rats did not reach a performance of 25% after three sessions, we included trials with visible light for a few sessions. This was necessary only when the very first remapping was a diagonal remapping. Task-irrelevant microstimulation control animals. To control for stimulation-induced plasticity, we trained and implanted two other animals that received stimulation that was not informative for any task. They were treated in the exact same way as the animals in the IR discrimination task, except they performed only the visible version of the task. At random times during this task, they were presented with stimulation patterns collected from rats that performed the actual task. Histology. At the end of the experiments, seven rats were killed to confirm the electrode positions. The rats were anesthetized with 1 ml of Euthasol and perfused with saline, followed by a 3% paraformaldehyde in 0.1 M phosphate buffer. The brain was removed from the skull, and the subcortical structures were carved out. The cortex was carefully flattened for 5 h in a 3% paraformaldehyde solution. After the cortex was flattened, it was fully fixed in a 4% paraformaldehyde solution overnight and then dehydrated in a 30% sucrose solution for another 24 h. Cryostat sections of 40 m were stained with cytochrome oxidase to show the barrel fields, as described previously (Thomson et al., 2013). Analysis. Data acquisition and analysis was performed in MATLAB using custom-written code, the Statistics Toolbox, and the Circular Statistics Toolbox (Berens, 2009). Here we discuss specific techniques we used. The stimulus population vector P in response to the microstimulation pattern S ⫽ ⬍f1, f2, f3, f4⬎ is as follows:
P共S兲 ⫽
冘 4
fi vi ,
i⫽1
where vi is a vector pointing in the direction of IR detector i (e.g., the top right detector location is represented by ⬍1, 1⬎), and fi is the frequency of stimulation in stimulating channel i. We used a bootstrap analysis to generate the 95% confidence ellipses around the mean of two-dimensional (2D) quantities. Namely, we took 10,000 bootstrap samples from the set of all values of the quantity, and calculated the mean of each bootstrap sample. We then fit the distribution of such bootstraps with a 2D Gaussian with the covariance matrix of the bootstrap samples, oriented in the direction of the eigenvectors, with the major and minor axes that determine the size of the ellipse determined by the covariance matrix. Multiple-comparison tests were performed in MATLAB to determine individual differences when a general ANOVA was significant. The test consisted of multiple t tests with Bonferroni correction for multiple comparisons. In calculating the IR-RFs, we took the mean response for each stimulus delivered, used linear interpolation to fill in gaps for where there were no stimuli presented, and then smoothed the resulting map using a 2D Gaussian filter with a width ⫽ 30 and ⫽ 10. To extract head direction and initial movement at trial onset, manual video analysis was used. Head bearing was determined by the nose direction with a graduator on the monitor screen. For the initial movement, only trials where the IR light came from the left or right side of the animal head were used (45–135° or 225–315° relative to head bearing of 0°). A trial was counted as having head movement if the animal swiveled its head ⬎10° after IR source onset; trials without such angular head movements (16% of trials) were discarded.
184 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
J. Neurosci., February 24, 2016 • 36(8):2406 –2424 • 2411
Table 1. The p values for comparison of performance in ablation experiments Ch 1 off Ch 2 off Ch 3 off Ch 4 off Front off Back off Right off Left off Ch 1 on Ch 2 on Ch 3 on Ch 4 on
All on
Ch1 off
Ch2 off
Ch3 off
Ch4 off
Front off
Back off
Right off
Left off
Ch1 on
Ch2 on
Ch3 on
0.999 0.895 0.137 0.051 0.692 * * * * * * *
0.999 0.803 0.609 0.995 * 0.097 * * * * *
0.998 0.985 1 0.007 * 0.014 * * * *
1 0.999 0.050 0.011 * * * * *
0.999 * 0.026 * * * * *
0.022 * 0.064 * * * *
0.998 0.994 0.796 0.027 0.038 0.119
1 0.136 * * 0.003
0.040 0.132 0.241 *
0.931 0.956 0.997
1 1
1
The p values are calculated using a multiple-comparison test. Ch, Channel. *Values ⬍0.001.
Results Behavioral task We trained 15 animals to discriminate the location of an IR source (see Experimental procedures). Briefly, each animal was trained in a location-discrimination task in a cylindrical arena that contained four water ports affixed to its border; in addition to a water spout, each port contained a visible and an IR light (Fig. 1A; Thomson et al., 2013). In the task, a light in a single port turned on, and, to receive a water reward, the animal had to break a photobeam in that port. Initially each rat performed the task with visible light only. Once they reached 85% correct, they were implanted with four groups of stimulating electrodes in four quadrants of S1 (Fig. 1; Experimental procedures). In subsequent sessions, we used the output of four IR sensors, which were evenly distributed around the head of the rat to control the stimulation frequency of stimulating electrodes in four quadrants of S1 (Fig. 1B; Experimental procedures). When one of the IR sensors was closer to, or oriented toward, an IR source, it evoked higherfrequency microstimulation in its corresponding S1 stimulating channel (Fig. 1C–E). Performance in IR discrimination task Thirteen animals quickly learned to discriminate IR sources using the synthetic panoramic IR signals delivered to S1. Figure 2A shows the learning curve of these animals, in which the percentage correct is plotted as a function of the number of days using the IR neuroprosthesis. On average, the animals surpassed the 85% correct mark within 3.75 ⫾ 0.45 sessions (mean ⫾ SEM). The mean best performance (the mean of their best five sessions) of the rats was 98.0 ⫾ 0.51% correct. As rats learned the task, the latency, or time between ICMS onset and selection of the port, decreased significantly from 5 to ⬍2 s (Fig. 2B). Figure 2, C and D, compares the learning rates of rats when trained with four IR sensors, with the data from a previous study in which animals were equipped only with a single IR sensor (Thomson et al., 2013). The rats wearing the new panoramic IR neuroprosthesis crossed the 85% correct threshold significantly faster than animals implanted with a single IR detector (3.75 ⫾ 0.45 sessions with four IR sensors vs 22.3 ⫾ 7.62 sessions for rats with one IR sensor). Unlike rats implanted with a single IR detector, animals implanted with our panoramic IR neuroprosthesis did not sweep their heads laterally in an attempt to sample IR signals from the surrounding environment (Thomson et al., 2013). Instead, once they received an ICMS with IR light onset, these rats often turned directly toward the correct port, likely because the four IR sensors
provided them with a more complete perspective on the IR sources (see Discussion). To investigate the animals’ behavior when using the new implant, we recorded the head direction when the animals initiated each trial. Specifically, we calculated the head bearing relative to the last poked port. Here we found that before the animals used the IR implant, they typically faced directly away from the previous port poked (178° away). Interestingly, after they learned to use the implant, they faced 155° away (Fig. 2 E, F ). This change with learning was significant ( p ⬍ 0.0001, nonparametric ANOVA), and both distributions were significantly different from a uniform distribution ( p ⬍ 0.0001 and p ⬍ 0.0001, respectively, Raleigh test). In the next set of experiments, we varied the task difficulty by moving the IR sources closer together in the chamber, randomly setting the angles between ports to 90°, 60°, 45°, or 30° (Fig. 2G, inset, arena layout icons). Surprisingly, while we observed a slight decrease in performance as the ports were closer together, this was not statistically significant in rats wearing a four-IR detector neuroprosthesis (Fig. 2G; p ⫽ 0.132, ANOVA). This is in contrast to the results in our previous study with a single sensor (Fig. 2G; Thomson et al., 2013, their Figs. 1, 2). Ablating IR sensors How would the animals respond if we shut off different combinations of IR sensors, so that rats did not have the full panoramic view of the IR environment? We examined this in six animals, in which different combinations of IR sensors were inactivated on randomly selected trials (see Experimental procedures). We call such inactivations sensor ablations. On average, with each additional sensor ablated, the performance of the rats dropped significantly (p ⬍ 0.0001, one-way ANOVA; Fig. 2H). The p values for the comparisons of performance in all ablation conditions are indicated in Table 1. With the inactivation of one channel, performance dropped significantly when compared with performance when all IR sensors were active (91.4 ⫾ 0.80% correct with all sensors vs 86.1 ⫾ 0.95% with only three sensors active; p ⫽ 0.0209, multiple-comparison test). With each additional channel deactivated, the performance decreased significantly (all p ⬍ 0.0001, multiple-comparison test). The performance reached 74.7 ⫾ 1.7% with two sensors deactivated, and 52.8 ⫾ 2.07% with three channels deactivated. The rats were still able to perform the task above chance level (25%), with only one of four IR sensors activated. When all four sensors were deactivated, so that the animal was receiving no information from the prosthetic, the performance dropped drastically, down to 14.5 ⫾ 3.8%, which is below chance (Fig. 2H). Note that
185 2412 • J. Neurosci., February 24, 2016 • 36(8):2406 –2424
performance can be lower than chance when rats do not make a behavioral selection, and allow the trial to time out. Figure 2I summarizes the performance for every particular sensor ablation experiment, showing that some IR sensor combinations were more important than others for task performance. Surprisingly, we observed that ablating individual back-facing IR sensors caused a larger decrement in performance than the small but insignificant decrease observed when a single front-facing sensor was inactivated (Fig. 2I ). Similarly, ablating both rearfacing channels led to a decrease in performance to 70.0 ⫾ 2.1% ( p ⬍ 0.0001, multiple-comparison test), which was significantly larger than the decrement observed after ablating both frontfacing channels ( p ⫽ 0.002, multiple-comparison test; Table 1). With just a single sensor active, even if it was a front sensor, performance was still significantly above chance ( p ⬍ 0.05; see Discussion). Finally, when the two sensors on one side of the animal were ablated, the performance dropped significantly (to 70.1 ⫾ 3.6% when the right side was ablated and to 65.5 ⫾ 5.3% when the left side was ablated; p ⬍ 0.0001, multiple-comparison test). There was no significant difference between the left-only and right-only conditions ( p ⬎ 0.999, multiple-comparison test). Topography remapping Our sensory neuroprosthetic system gives us complete control over the mapping between the IR sensor and the stimulating electrode. This allowed us to examine the rat’s sensitivity to the spatial distribution of IR signals within S1. More specifically, once animals had learned the task with the normal configuration, how did they react when that mapping between the IR sensor and stimulating electrode was changed (Fig. 3A)? We examined the following three topographic remappings in nine animals: left-right remapping, in which we connected IR sensors to stimulating channels in the ipsilateral cortical hemispheres instead of the usual contralateral loci (Fig. 3A); frontback remappings, in which the front IR sensors project to the barrels that represent the rear whiskers and vice versa (Fig. 3A); and third, diagonal remappings combine the first two remappings, completely reversing the usual topography in both dimensions. For convenience, hereafter we will call the first two examples—left-right and front-back— one-dimensional (1D) remappings, whereas the diagonal remapping is classified as a 2D remapping. The impact of all remappings was immediate and profound. For instance, in left-right remappings, when an IR source appeared on the right side of the animal, the rat would promptly move to the left and select the incorrect port. Despite such understandable miscalculations, rats quickly started to recalibrate their performance during their first sessions with 1D remappings. Quantitatively, Figure 3B shows the behavioral performance for a series of remappings in one animal. It started with a frontback remapping (its performance the day before remapping is indicated by the green dot). The performance of the animal dropped during the first session, but it soon recovered as it learned the new relationship between IR sensors and stimulating electrodes. After animals successfully relearned the task following an initial remapping, we then tested their ability to adapt to subsequent remappings. In this case, after the front-back shuffle, we then remapped the left and right IR sources, resulting in a 2D diagonal remapping. Again, the animal learned the correct association within a few sessions (Fig. 3B). The performance of the animal on subsequent remappings is indicated in Figure 3B.
Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
Curiously, none of the three animals that started with a diagonal remapping learned the task unaided: we used visual cues to help them properly recalibrate their maps. Figure 3C shows a representative animal that started on the diagonal remapping. The rat struggled on the first three sessions with below-chance performance. Indeed, all animals stayed at below chance level (25%) when first introduced to the 2D remapping, and ultimately lost motivation in the task. To keep them motivated, we supplemented ICMS with visible light to guide them to the correct port (visible trials: sessions in which we used visible trials are indicated in Fig. 3C). Over the next few sessions, we included a decreasing proportion of visible trials. Such visually guided recalibration of the mapping was never necessary for 1D remappings, but it was needed for all three animals to learn the 2D remapping. Figure 3, D and E, shows the pooled data for each remapping group. No difference was found between the two 1D remappings ( p ⫽ 0.74, multiple-comparison test): on average, it took them 3.83 ⫾ 0.7 sessions to learn the first 1D remappings (Fig. 3D, blue and green lines). On the other hand, rats that started with the diagonal remapping took 11.7 ⫾ 2.3 sessions to relearn the task, even though they had the help of the visible light (Fig. 3D, orange). This was a significant difference in learning rate ( p ⬍ 0.0001, multiple-comparison test). These results show that the rats clearly used the spatial information available in the distributed IR prosthesis. We also examined, in more quantitative terms, the pattern of errors after left-right remappings. Specifically, we measured whether the initial head movement was toward or away from the IR source in the first 20 trials of the first remapping session (Fig. 3F ). This measure was also made for the session preceding the first remapping, and on the third day of the remapping. As expected, the animals turned in the opposite direction directly after remapping significantly more often than before ( p ⫽ 0.0001, ANOVA). After learning the remapping, the animals turned directly toward the IR source, indicating that they had fully adapted ( p ⫽ 0.0003, ANOVA). Interestingly, once a rat experienced its first remapping, it learned subsequent remappings significantly faster, suggesting the existence of a central metaplasticity mechanism (see Discussion). Figure 3E shows the learning curve for all secondary remappings in animals after they reached criterion (85% correct) on their first remapping. Interestingly, the speed of learning for secondary remappings did not vary for the 1D versus 2D diagonal cases (p ⫽ 0.79, ANOVA). Rats reached the 85% correct threshold in 2.17 ⫾ 0.2 sessions. Not only were secondary remappings learned faster than initial diagonal remapping (p ⬍ 0.0001, multiple-comparison test), but they were also learned faster than the first 1D remappings (Fig. 3F; p ⫽ 0.048, multiple-comparison test). The previous results address the effect of remapping in an already trained animal, but what if we change how they are trained initially? That is, is it important that they start with the most “natural” mapping in which (for example) the front-right detector activated stimulates electrodes in the region of S1 representing the front-right whiskers (Fig. 1)? To answer this, we trained five rats with the diagonal mapping condition from day 1. The overall percentage correct in the first five sessions for rats initially trained on the normal mapping was 64 ⫾ 7%, while in the diagonal-mapped animals it was 40.0 ⫾ 10% correct, showing a significant difference (Fig. 3H; p ⫽ 0.0007, two-sided t test). While the diagonally mapped animals took longer to learn the task (Fig. 3I ), they did ultimately reach a best five-session performance of 98.4 ⫾ 0.43% correct, which did not differ significantly from the best five-session performance of the 11 normally
186 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
A
Normal
J. Neurosci., February 24, 2016 • 36(8):2406 –2424 • 2413
Front-Back
Left-Right
100
B
C
60 40
0
5
40
0
5
10 15 Sessions
2
4
***
6
*
8 10 Sessions
12
***
14
H
F ***
80
80 60 40 20 0
before day 1 day 3 Remapping
I
*
6 3 0
experienced Remapping Condition
Normal Left -Rright Front - Back Diagonal
20 0
1
2
3 4 Sessions
5
6
80
60 40 20
60 40 Normal Diagonal
20 0
1
2
3 4 Sessions
5
6
7
Di
ag
on
al
0 No rm al
40
100
12 9
60
0
16
80
100
***
Percent Correct [%]
Needed Sessions
Left - Right Front - Back Diagonal
20
15
0
100
Normal Left - Right Front - Back Diagonal +VIS Trials 20 25
20
Percent Correct [%]
60
Percent Correct [%]
Percent Correct [%]
80
G
40
E
100
0 0
60
10 15 20 25 30 35 40 45 Sessions
Move Toward [%]
D
Normal Left - Right Front - Back Diagonal
20 0
100 80
Percent Correct [%]
Percent Correct [%]
80
Diagonal
Figure 3. Disrupting the topographic IR map temporarily impairs task performance. A, Once animals had learned the task to criterion (at least 85% correct), we performed a series of remapping experiments in which we changed the mapping from IR sensor to stimulating electrode. This panel illustrates the four different mapping conditions we used, such as switching hemispheres, front-back switches, or both. B, C, Examples from two rats run through multiple remappings. After the rats were proficient with one mapping of the stimulation, a new mapping was introduced. Each remapping type is depicted in a different color. The performance on the day before the first remapping is depicted by the green dot at session 1. Interestingly, all animals that started with the diagonal remapping stayed at chance performance, and in those animals we added visual cues as an additional aid (this is indicated by the black x). Also, see Movie 2. D, A summary slide of all learning curves for the first remapping and all subsequent remappings. Data are collected from nine rats in three groups. Each group started with a different type of remapping (left-right, front-back, or diagonal). Subsequent remappings were chosen pseudo-randomly for all rats in all groups. One-dimensional remappings (left-right and front-back) were learned significantly faster ( p ⬍ 0.0001, multiplecomparison test) and without the guidance of visible light. No difference in learning speed was found between the left-right and front-back remappings ( p ⫽ 0.74, multiple-comparison test). It took 3.83 ⫾ 0.7 sessions to learn the first one-dimensional remapping. The first diagonal remapping took 11.7 ⫾ 2.3 sessions. All rats needed visible light as guidance when confronted with a diagonal remapping initially. We started visible light when the rats performed worse than 10% on their fourth day of remapping. E, Analysis of error patterns in the first 20 trials to during left-right remapping. For those trials in which IR lights were activated to the left or right of the animal, the plot shows the percentage of trials in which the animal turned toward the (Figure legend continues.)
187 2414 • J. Neurosci., February 24, 2016 • 36(8):2406 –2424
Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
(Georgopoulos et al., 1986). The stimulus population vector is the weighted sum of the location of each IR sensor (Fig. 4D), where the location of each sensor is weighted by the stimulation frequency in its corresponding stimulating electrode. Quantitatively, the stimulus population vector P in response to S ⫽ ⬍f1, f2, f3, f4⬎ is as follows:
P共S兲 ⫽
Movie 2. Left-right remapping example, related to Figure 3. The connection between the sensors and the stimulation channels was switched between left and right (Fig. 3A). During the first trials, the rat was expecting the normal mapping and turns in the wrong direction when an IR light is activated to the left or right. After training (in this case, ⬃18 min), the rat has begun to adapt to the new mapping. (The lights on the left side of the arena are used for off-line image processing and are not relevant for the task.)
mapped rats ( p ⫽ 0.718, two-sided t test). Interestingly, the average response latencies of the diagonally mapped rats (2.56 ⫾ 0.168 s) never matched those of the rats with the normal topography (1.73 ⫾ 0.07 s; p ⫽ 0.0011, two-sided t test). Note that, while there was a weak but significant correlation in learning rates between the learning time of the visual task and the IR task (p ⫽ 0.0494, Spearman correlation), we found no further correlations between learning rates in subsequent subexperiments. Stimulus population vectors In the next series of experiments, we recorded neuronal activity in S1 while rats performed the IR discrimination task. As shown in Figure 4A, to work around the stimulus artifact that occurs during electrical microstimulation, we modified the task to stimulate S1 intermittently. Specifically, after sampling the IR level in a sensor, we stimulated for 75 ms at the corresponding frequency, followed by a 100 ms quiet period. During this period of quiescence, we were able to record S1 neuronal activity. This was done in all four stimulation locations, as depicted in Figure 4B. Quantitatively, every 175 ms there was a four-element vector of frequencies S ⫽ ⬍f1, f2, f3, f4⬎ delivered to four regions of S1 (Fig. 4 B, C). Figure 4B represents five such stimuli, depicted at times T1 . . . T5. To compactly depict the electrical stimulation delivered to the brain in one of these distributed bursts, we represented the full 4D representation of the stimulus into a more easily visualizable 2D space using a stimulus population vector. This population vector is defined exactly like those popularized in the motor control literature 4 (Figure legend continued.) correct light (see Materials and Methods). Performance was significantly worse during the first 20 trials on the first day of remapping compared with the day before the first remapping and the third day of remapping (p ⫽ 0.0001 and p ⫽ 0.0003, respectively, ANOVA). F, After learning the first remapping, all subsequent remappings were learned in 2.17 ⫾ 0.2 sessions, regardless of the type of remapping ( p ⫽ 0.79, ANOVA). G, Number of sessions the rats needed to reach the 85% threshold for different remapping conditions. After rats learned one remapping, they learned subsequent remappings significantly faster ( p ⬍ 0.0001, multiple-comparison test). This suggests that a metaplasticity mechanism is in play. H, Mean ⫾ SEM performance on the first five sessions for animals trained with normal mapping from the start (N ⫽ 8) and those trained with diagonal mapping (N ⫽ 5 animals). The animals trained with diagonal mapping performed significantly worse ( p ⫽ 0.007, two-sided t test). I, Learning curve for normally mapped vs diagonally mapped animals, using the same groups of animals as in G. The dotted line marks the 85% threshold.
冘f s , 4
i⫽1
i i
(1)
where si is a vector pointing in the direction of IR detector i (e.g., the rear-left detector location is represented by ⬍⫺1, ⫺1⬎. For instance, when the front two IR detectors are strongly activated, the population vector is at the top of the graph (Fig. 4A–D, time point T5). Quantitatively, the rear-left IR sensor (sensor 3) is represented by the vector ⬍⫺1, ⫺1⬎, and if we stimulate at 200 Hz in its corresponding stimulating channel, then the contribution from sensor 3 to the population vector will be ⬍⫺200, ⫺200⬎. The contribution from all four sensors is summed to yield the full population vector. Figure 4, B and C, works through an artificial example of how to visualize five such vectors. This is meant for illustrative purposes to help explain the population vector concept. Figure 4E shows, in red, the set of all actual population vectors observed in all 20 sessions in three rats. The set of all possible population vectors in our stimulus set is shown as gray points in Figure 4E. The upper bound in each direction is indicated with the black diamond. We stimulated between 0 and 425 Hz in an individual channel, and these limits determined the upper bounds of 850 Hz for any individual population vector (Eq. 1). Note the relative paucity of such vectors in the posteromedial region, which exists because the animal rarely backs up into an IR source. That is, the two rear sensors are rarely activated strongly together: the animal does not approach the target moving backward, but typically orients toward the target and then runs forward. To show how the stimulus population vectors change over time within a trial, Figure 4F shows the set of population vectors that occurred in one session, with contiguous substimuli from a trial connected by lines (the red points are error trials, green are correct trials). The lighter points/lines represent the stimuli that occurred earlier in the trial, and darker rendering represents the later stimuli. Note the characteristic shape of the curves: stimuli tended to start around the origin (⬍0, 0⬎) in the space of population vectors, and as the animal approached the correct IR port, the population vector shifted to the top part of the graph, indicating that the front two IR sensors were strongly activated. The mean of the first and last three stimuli in the session are shown in black. On average, 14 ⫾ 0.53 stimuli occurred on each trial (Fig. 4G). Neuronal peristimulus time histogram during the execution of the IR detecting task We used multiple measures to quantify the neuronal responses in S1 during a session. Figure 4H shows one measure: the mean neuronal population peristimulus time histogram (PSTH) in response to the first three and last three stimuli in a session. The times of actual stimulation are numbered 1– 6 in the PSTH, where the first 3 are aligned to the onset of the cortical stimulation and the last 3 to the offset. During the 75 ms stimulation periods, the PSTH is zero because of stimulus artifact, and then there is a 100 ms period of neuronal activity recording after the stimulus ends (Fig. 4A). The set of all such neuronal population PSTHs obtained in a given recording session is shown in Figure 5, where the location
188 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
J. Neurosci., February 24, 2016 • 36(8):2406 –2424 • 2415
A Stim Only
T1
B
Stims + Record
T2
T3
T4
T5
75ms Current [μA]
Current [μA]
Current [μA]
Channel 1 S R
100ms
Time [s]
Channel 2 Channel 3 Channel 4
Time [s]
Time [s]
E
D 200
< 0, 0, 50, 50 >
T2
< 0, 0, 50, 0 >
< -50,-50 >
4-Vector
T3 < 10, 50, 10, 0 >
< -50, 50 >
T4
< 0, 100 >
< 50, 50, 0, 0 >
T5 < 100, 100, 0, 0 >
Sensor 1 T4
100 T3 0 T2 -100
T1 Sensor 4
Sensor 3
< 0, 200 >
850
T5 Sensor 2
Anteriorposterior
T1
Population Vector < 0, -100 >
Anteriorposterior [Hz]
C
510
170 0 −170
−510
-200 -200
100 0 Mediolateral
850
200
−850 −850
G Frequency
F
-100
510
850
0.04 0
5
10 15 20 Number Of Stimuli
25
30
3.5 3
−170
Spike Count
Anteriorposterior [Hz]
510
0.08
170
H
−170 0 170 Mediolateral[Hz]
0.12
0
0
−510
−510
2 1.5 1 0.5
H(S) = 6 bits −850 −850
2.5
1 2 3
0 −510
−170 0 170 Mediolateral [Hz]
510
850
-0.5 -0.25
0
// 4
5 6
0.25 0.5 0.75 1 Time [s]
1.25 1.5
Figure 4. Quantifying the stimulus projected into S1. A, Illustration of change in protocol used when simultaneously stimulating and recording. When just stimulating, we updated the stimulation frequency in each channel every 50 ms based on the IR levels in its corresponding detector (Fig. 1F). In animals in which we stimulated and recorded, we sampled the IR levels approximately every 175 ms, and delivered stimulation for 75 ms in each stimulating channel, and then turned off the stimulators for 100 ms to facilitate recording. B, Each stimulation pattern can be represented by a sequence of four-element vectors S of microstimulation frequencies in four locations in S1. The panel shows five simplified patterns of microstimulation. C, Conversion from 4D vector representation of stimulus to the 2D population vector, where each time point (T1–T5) corresponds to one of the five time points in B. The actual numbers (0 –100) are arbitrary and are used only to clarify the procedure. D, The same four population vectors from C are depicted as locations in the 2D population vector space. Again, this is for illustration only, to facilitate understanding of simple stimulus population vectors, and does not include real data. All four sensor location vectors are indicated for reference, as are five population vectors (labeled T1 . . . T5). The black square is the outline of the bound of all possible stimulus population vectors, which are discussed more in D. E, The set of all possible population vectors (gray points), with the set of all actual population vectors over all recording sessions overlaid (red points; 19 sessions in three animals, with 171 ⫾ 53 trials per session and 14 ⫾ 0.53 stimuli per trial). Note that the actual vectors (red) tend to cluster in the medial anterior portion of the space, with relatively few population vectors on the extreme lateral and caudal regions of the space of population vectors. The range from 0 to 850 Hz results from the maximum stimulation frequency of 425 Hz per channel, which can sum up to 850 Hz in the two-dimensional transformation. F, The set of actual population vectors (Figure legend continues.)
189 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
2416 • J. Neurosci., February 24, 2016 • 36(8):2406 –2424 0.2 sig019i
sig009i
sig020i
sig010i
0.2
0.2 2
//
0 0.2 sig023i
//
0
//
0
0
sig013i
sig024i
-0.5
0
// 0.5
1
1.5
Time (s)
0.2
0
-0.5
0
// 0.5
1
1.5
0 -0.5
0
// 0.5
1
1.5
0
-0.5
//
//
0
2 0
0
1
2
3
4
# Channels Activated
0.4 sig002i
//
0
//
0
sig005i
0.1
4
N3
sig032i
sig031i
1.5
0.2 sig001i
0.2 sig028i
0
1
N2
N4 sig027i
// 0.5
Time (s)
N1
0.1
0
Time (s)
Time (s)
Spike Count (z)
0.5
sig006i
0.2
2
0.1
0
-0.5
0
// 0.5
Time (s)
1
1.5
0
-0.5
0
// 0.5
1
1.5
0
Time (s)
-0.5
0
// 0.5
Time (s)
1
1.5
0
-0.5
0
// 0.5
1
1.5
Time (s)
Figure 5. S1 PSTH in response to stimulation sequences. The mean response to the first three, and last three, stimulus vectors, as depicted in Figure 4F. The arrows from the brain show the location of the recording electrodes. Each PSTH follows the conventions shown in Figure 4H. The inset shows the spike count z-score as a function of the number of stimulating channels activated. This is a typical profile, with maximum response occurring when two channels are coactivated. The z-score was calculated by comparing the spike count after stimulation, and we used the mean/SD of spike counts before trial onset (calculated over all trials; data are from N ⫽ 3 animals, 20 recording sessions, 299 multiunit recordings). Even when no stimulating channels are active, the response is typically higher than in the baseline period before the trial begins.
of each recording microelectrode is also indicated. We consistently found the IR-evoked neuronal responses to be broadly distributed across the recording microelectrodes in all four quadrants of S1. The inset on the right side of Figure 5 depicts the mean neuronal response as a function of the total number of stimulating channels activated. This graph shows that the larger numbers of spikes were produced by S1 neurons until two IR channels were activated. Beyond that point, when three or four IR 4 (Figure legend continued.) delivered in a single session, with those stimuli in the same trial connected by lines. Red indicates error trials, while green shows correct trials. Lighter circles represent earlier stimuli in the trial, and more saturated colors indicate later stimuli. The black spots represent the mean value of the first three and last three stimuli in the session. The early stimuli hovered around the origin, while the final three stimuli tended to be closer to the rostral pole, indicating the two front IR channels (and stimulating electrodes) were highly activated. In this session. There were 6.0 bits of entropy contained in the stimulus set. The light gray circles indicate the set of all possible population vectors as before, and are included for reference. G, Count histogram of the number of stimuli shown on each trial, over all sessions. The distribution is approximately bimodal, with the peak centered at ⬃25 indicating those trials in which the animal let the trial time out without poking, and reached the upper limit of net stimulation. H, PSTH in response to the first three and last three stimuli in the same session, as indicated in D. The stimulation periods are indicated by the numbers 1– 6, and time is relative to the onset of the first stimulus. Note that this PSTH does not represent the response to the exact same stimulus each time: while the first three stimuli tend to cluster around the origin and the last three tend to cluster in the mediorostral region, but there is a good deal of variability (D). This is simply a way to get a quick overview of the types of responses we see. We indicate the potential break in time between the first three and last three stimuli on the x-axis with the slashes.
channels were activated, S1 neuronal firing tended to decrease, though this change was not significant (ANOVA, p ⬎ 0.05). Infrared neuronal receptive fields in the rat S1 cortex While the PSTH in response to the first and last three stimuli gives a convenient visual representation of neuronal activity during the task, it is also a relatively incomplete sample of the full range of neuronal firing patterns. To give a full representation of how S1 neurons responded to IR stimuli, we introduced a new measurement: the IR-RF. The IR-RF is the mean S1 neuronal firing response magnitude (spike count), as a function of the stimulus population vector, for all of the IR stimuli shown in a session. Figure 6A shows all of the IR-RFs for the same S1 neuronal units depicted in Figure 5. The mean value for each stimulus is color coded in a contour plot defined over the set of stimuli, where red lines depict peak response values. For instance, unit 019i has a peak at the most anterior/medial portion of the stimulus space, indicating that it responds maximally when the two front IR sensors are strongly activated (Fig. 4), while unit 031i has its IR-RF center located in the posterior-right region of the space. The contour plots suggest that the locations of the IR-RF peaks are quite variable from neuron to neuron. However, they are not randomly distributed. For the 179 S1 units recorded under the normal mapping, the distribution of IR-RFs peaks tend to cluster near where the IR stimuli clustered, in the anterior/medial space of the IR stimulus population vectors (Fig. 6B, red ellipse,
190 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
A
J. Neurosci., February 24, 2016 • 36(8):2406 –2424 • 2417
sig019i
sig020i
sig009i
sig023i
sig024i
sig013i
sig010i
Rmax
N1
N2
N4
N3
Rmin
sig028i
sig032i
sig001i
sig002i
sig031i
sig027i
sig005i
sig006i
C
B
850
510 Anteriorposterior [Hz]
510
170 0 −170
0
−170
−510
−510
-510
−170 0 170 Mediolateral [Hz]
510
E
120
D 125
−850 −850
850 90
F
60
150
100
* 0
180
25
20
1
850
G
30
300
20
10
0
40
2
Preference (# Stimulators Active)
510
200
*
100
330
210
0
−170 0 170 Mediolateral [Hz]
30
75 50
−510
RF Diameter
−850 -850
Count (# Neurons)
170
Peak Z Score
Anteriorposterior [Hz]
850
Overall N1 N2 N3 N4
300
240 270
<40
>90
Session Performance [%]
0 <40
>90
Session Performance [%]
Figure 6. IR receptive fields. A, Mean number of spikes in response to different stimuli. The stimuli are represented as population vectors (Fig. 6), and the mean spike count in the 60 ms time window is represented by isocontour lines (low rates in blue, high in red: see color bar). Same session and units as used in Figure 5. The small red point on each contour plot shows the point where the receptive field is maximized. B, Distribution of locations of RF peaks in all units with the normal mapping (179, multiunits total). They tend to be located along (Figure legend continues.)
191 2418 • J. Neurosci., February 24, 2016 • 36(8):2406 –2424
shows the 95% confidence ellipse for the mean of the distribution of IR-RF centers). For comparison, Figure 6C plots the distribution of all stimulus population vectors presented to all animals that were recorded. Quantitatively, the 2D correlation between the IR-RF centers and the actual distribution of IR stimuli was 0.44, a significant correlation ( p ⬍ 0.001, F test). Altogether, these data suggest that the overall distribution of neuronal IRRFs tended to reproduce the spatial distribution of stimulus population vectors used, though with a slight preference for larger stimuli (i.e., closer to the borders of the diamond) on the mediolateral axis. To test the hypothesis that response preferences in S1 neurons tend to match those of the stimuli delivered, we then compared the IR-RFs for three animals in which the mapping between front and rear channels was reversed (104 units). If neurons tend to naturally prefer activation of the anterior channels (Fig. 1B), then we would expect the IR-RF centers to be localized in the posterior portion of the population vector space when the front and back channels are reversed. However, this is not what we observed. Instead, the IR-RFs continued to favor the front stimuli, with the mean IR-RF center located at ⬍⫺126.3, 205.4⬎, and a significant 2D correlation between IR-RF centers and stimulus distribution (2D correlation coefficient, 0.20; p ⫽ 0.003, F test). This confirms the hypothesis that response profiles, as summarized in the IRRFs, tend to match the distribution of stimuli actually delivered during the task. There was not a simple relationship between stimulating electrode location (or even hemisphere) and RF center. Neurons did not simply respond maximally to localized stimulation. Neurons tended to respond most strongly to stimulation delivered in two locations (Fig. 5, inset), often between hemispheres. Of 179 S1 units recorded with the normal mapping, 65% (117 of 179 S1 units) had receptive field centers located in a region in which at least two microstimulators were active (Fig. 6 D, E). Of those with a preference for stimulation at two or more locations, 58% (68 of 117 S1 units) had a preference for stimulation across hemispheres. We next examined the effects of learning on RF magnitude and diameter. Figure 6F displays the IR-RF peak magnitude when the behavioral performance of the animals was fairly poor (⬍40% correct) versus when rats reached excellent behavioral performance (⬎90% correct). This did not change with learning ( p ⫽ 0.91, t test), suggesting that S1 responds with robust responses very early in training, which is what we observed informally even on the first day that animals were training on the IR task. Figure 6G depicts the IR-RF diameter as a function of be4 (Figure legend continued.) the midline, and rostral. That is, they tend to favor the activation of the two front IR channels. The histograms on the sides show the marginal frequencies. The red spot shows the mean, and the circle around it shows the 95% CI for the mean (see Materials and Methods). The mean RF centers are also indicated for the four different recording locations. C, Distribution of locations of actual stimulus vectors presented in all experiments (see Fig. 4E). They tend to be located along the midline, and rostral, just like the receptive field centers. The histograms on the sides show the marginal frequencies. D, Bar plot showing the relative number of neurons with preference for stimulation from pairs of stimulating electrodes vs individual stimulating electrodes. E, Proportion with preference for pairs is significantly greater than chance ( p ⫽ 0.003, 2 test). Angular preference of all neurons are from B. Angular histogram shows the relative number of neurons with IR-RF centers at different angular locations. F, Peak response magnitude (z-score, with mean/SD spike count calculated from a 60 ms baseline period over all trials in a given session) does not depend on training ( p ⫽ 0.49, two-sided t test). G, RF size changes with learning. Mean ⫾ SEM RF diameter for early- and late-training animals are significantly different ( p ⫽ 0.009, two-sided t test).
Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
havioral performance. The IR-RF diameter is a measure of the area enclosed by the contour line at 75% of the maximum value of the IR-RF (see Experimental procedures). This analysis revealed that the IR-RF width actually increased with learning ( p ⫽ 0.009, t test). Population discrimination of 75 ms stimulus pulses Next, we addressed the question of how well S1 neuronal firing patterns could predict the stimulus presented to the animal in a given 75 ms epoch. To do this, we applied naive Bayesian classifiers. Because many stimulus vectors do not occur often enough to build a reliable classifier, we narrowed our classification problem to the eight most frequent IR stimulation patterns in each session (a 3 bit classification task). We then ran the classifier on different numbers of S1 neurons, from 1 to 14 units at a time, depending on how many we were able to record simultaneously. Figure 7A displays the mean performance of the classifier as a function of the number of neurons used to train and test the classifier: a neuron-dropping curve (Wessberg et al., 2000; Lebedev, 2014). The red line depicts chance level. As one can see, the percentage of trials correctly classified by the Bayesian algorithm rose from 20% (using one unit) up to 52% when the spike counts from 14 units were combined into a single vector. When we broke our analysis to look into high- and lowperforming sessions, as described in Figure 6, we observed that neural ensemble discrimination of single-trial IR stimuli was significantly better in highly trained animals (Fig. 7B; p ⫽ 0.048, two-sample Kolmogorov–Smirnov test). This suggests an improvement in neuronal discrimination power as a function of learning. S1 responses to uninformative microstimulation patterns Electrical stimulation of the brain, outside of any task context, clearly produces significant responses, and can even induce drastic plasticity (Nudo et al., 1990; Recanzone et al., 1992; Maldonado and Gerstein, 1996). To examine whether the responses we observed depended on the microstimulation patterns providing information to be used to perform the IR discrimination task, we delivered identical microstimulation patterns in two control animals, but in a way that was unrelated to any task (see Experimental methods). Briefly, we pulled stimulus sequences from rats that actually performed the task, and delivered them at random times to animals that performed a normal visual version of the task (no IR light), such that microstimulation provided no information about the stimulus in each port. This allowed us to reproduce the general context of stimulation, but remove its task relevance. In both animals, after 2 weeks of training with taskirrelevant microstimulation, S1 sensory-evoked responses were drastically suppressed. Figure 8A shows an example from one animal in which we recorded 14 S1 units simultaneously. The mean response to the first three and last three stimuli is shown in Figure 8B, showing the gross suppression effect (Fig. 8B, top) compared with the normal animals (Fig. 8B, bottom). Overall, the mean ⫾ SEM of the maximum response to all stimuli was 1.4 ⫾ 0.13 when the stimulation was uninformative, and 17.4 ⫾ 1.6 when informative, showing a significant difference ( p ⬍ 0.001; Fig. 8C). Similarly, the mean response magnitude as a function of the number of stimulating channels activated was effectively a mirror image of the case with informative stimulation, with values reflected across the x-axis (Fig. 8D).
192 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
Percent Correct [%]
A
J. Neurosci., February 24, 2016 • 36(8):2406 –2424 • 2419
above baseline was 728 ⫾ 91 total spikes for multiunit activity and 77 ⫾ 21 total spikes from 15 single units in the same animals. The mean response to all whisker deflections, over all 36 multiunit recordings, is shown in Figure 9C; clear onset and offset responses can be observed. These results suggest that S1 has not been hijacked by the IR neuroprosthetic (Griffin et al., 2011). Instead, S1 seemed to have incorporated an ability to represent both stimuli simultaneously into a multimodal map of the tactile and IR environment.
60 50 40 30
Discussion 20 10
Percent Correct [%]
B
All (n=19) >90 (n=6) <40 (n=5)
60 50 40 30 20 10
2
4
6 8 10 Number Of Units
12
14
Figure 7. Discrimination of microstimulation patterns in S1. A, We selected eight of the most frequent population vectors for each session and used a Naive Bayes’ classifier to predict the stimulus based on neuronal activity in S1. Plot shows the percentage of population vectors classified correctly as a function of the number of units used to predict the stimulus (mean ⫾ SEM, N ⫽ 20 sessions, 299 multiunits total). The red line is chance (12.5%). Note that this represents a differential response to a single 75 ms stimulation bout (e.g., stimulus T3 in Fig. 4B). On average, there are 11.3 ⫾ 5.1 stimuli/trial (mean ⫾ SD). B, Same data as in A, but with discrimination separated by behavioral performance, as in Figure 6.
We also examined the ability of the neuronal responses to discriminate microstimulation patterns, expecting it to be significantly diminished compared with the case when such patterns were informative about events in the environment of the animal. Surprisingly, these inhibitory responses carried almost as much information about microstimulation patterns (Fig. 8E; see Discussion). S1 neuronal firing responses to whisker deflections in rats implanted with IR neuroprosthesis We observed strong rapid S1 neuronal sensory-evoked responses to whisker deflections in animals trained on the IR discrimination task. Figure 9A shows the carbon monoxide-stained section through both hemispheres for one animal that was highly trained in the task (performance consistently above 95% correct) to show the location of the recording electrodes. Figure 9B displays the responses of 15 S1 units (multiunit activity), in the same animal, to deflections of the right whiskers using an air puff. We observed significant responses in all S1 units in the three rats (N ⫽ 36 units; Griffin et al., 2011). The S1 neuronal response latency was 10.8 ⫾ 0.38 ms, and the mean response duration was 92 ⫾ 12.5 ms. The response magnitudes were quite large: the mean number of spikes
The adult mammalian cortex is able to overcome the biological constraints imposed by the native peripheral sensory transducers of an organism, absorbing a spatially distributed representation of a completely novel sensory modality. Adult rats quickly adapted their behavior to use a panoramic IR neuroprosthesis, attaining near perfect performance in a sensory discrimination task involving the new sensory modality (Fig. 2). The cortical neuroprosthesis used in this study is, as far as we can tell, the first device to project a panoramic topographically distributed projection of the subject’s surround to both hemispheres of a primary cortical area. Based on the ablation experiments (Fig. 2H ), it is clear that rats used all four information channels, though none are indispensable. Instead, performance dropped gradually with each IR sensor inactivated, but even when only one sensor was active, performance was still significantly above chance (Fig. 2H ). Note that the use of IR sources was largely arbitrary. We could have used any information source that bypassed the natural transduction pathways of the rats, as long as a portable sensor was available (e.g., magnetic fields; Nagel et al., 2005; Norimoto and Ikegaya, 2015). In the 1960s, Bach-y-Rita et al. (1969) pioneered sensory substitution experiments in which continuous streams of visual information were delivered mechanically to the skin on the back of blind human subjects, using actuators laid out to preserve the topographic organization of the original visual stimulus. While our rats needed just a few days to show excellent performance with the prosthesis (Fig. 2), in humans there is a great deal of variance in the learning curves for peripheral sensory substitution devices. The time taken to get used to different peripheral devices ranges from 30 h to months (Nagel et al., 2005; Amedi et al., 2007; Kim and Zatorre, 2008; Kärcher, 2012; Kaspar et al., 2014). Even compared with the studies with the shortest learning times (30 – 40 h), our rats performed as well or better after 3 h (Amedi et al., 2007; Kim and Zatorre, 2008). Hence, we hypothesize that people who one day take advantage of sensory neuroprosthetic systems that deliver information directly to the CNS— particularly primary sensory areas—will learn to use the new information faster than those using feedback delivered to the periphery. Current motor neuroprostheses typically record a motor control signal in the brain and send this to a prosthetic limb (Chapin et al., 1999; Carmena et al., 2003; Nicolelis, 2011; Ifft et al., 2013). Using the technology described here, the brain could be placed in a continuous two-way communication with information from sensors in prosthetic limbs (O’Doherty et al., 2012). Our data suggest that when designing prosthetic systems for primary sensory areas, it would be prudent to exploit preexisting topographic maps, as rats learned faster when stimulating channels were aligned with the somatotopic map of the barrel field (Fig. 3I ). One explanation of this might be that the brain already contains the downstream connectivity pattern required to extract spatial information from the topographic map in S1, and this
193 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
2420 • J. Neurosci., February 24, 2016 • 36(8):2406 –2424
A
sig019i
sig020i 0.5
1 //
0 0.4
sig024i
0 -0.5 0
0.2 // 0.5 1 Time [s]
1.5
-0.5 0
// 0.5 1 Time [s]
sig028i 0.5
//
0
N4
N3
sig032i
0 -0.5 0
// 1 0.5 Time [s]
1.5
-0.5 0
// 0.5 1 Time [s]
//
0
-0.5 0
C
40 20 2
D
// 0.5 1 Time [s]
0
Percent Correct [%] 0
1 2 3 Number Channels Active
4
1.5
0
-0.5 0
// 0.5 1 Time [s]
1.5
16 12 8 4 Irrelevant Relevant Condition Relevant Irrelevant Shuffled
50
2
sig006i
60
E
4
//
20
6
0 -0.15 -0.3 -0.45
1.5
sig002i
0
0 1
// 0.5 1 Time [s]
0.2
−4
// 3 4 5 Stimulus Number
-0.5 0
sig005i
−2
0
0
1
Max Response [z]
Response Magnitude [z]
1.5
sig001i
2
0
1.5 //
0
Response Magnitude [z]
// 0.5 1 Time [s]
0.5
0.1
B
-0.5 0
1
0.2
sig014i 0.2
N2
sig031i
//
0 sig013i
N1
0.1
0
//
0
0
1.5
sig010i 0.2
1
0.2 0
sig009i
40 30 20 10
2
4 6 8 10 Number Of Units
12
14
Figure 8. Uninformative stimuli produce inhibitory responses. A, PSTH in all neurons recorded in an animal receiving noninformative microstimulation sequences. Specifically, we delivered microstimulation sequences pulled from animals performing the actual IR discrimination task, but delivered at random times, to an animal doing an unrelated visual task. B, Mean response magnitude to the first three and last three stimuli in two animals, six sessions, 78 units. Red bars, Response when ICMS is not task relevant; black bars, mean response when ICMS is used for IR discrimination. Note the inhibitory response to uninformative ICMS. C, Mean peak response magnitude in all neurons in which ICMS was not task relevant vs when it was task relevant. D, Mean ⫾ SEM response magnitude (z-score) as a function of number of units active for the task-irrelevant (red) and task-relevant (black) ICMS. Again, note the difference in magnitude and the sign of the response. E, Surprisingly, the inhibitory responses carried just as much information about stimulus identity. The discrimination performance as a function of the number of units is shown for animals stimulated during the IR task (black), outside the task context (red), and with stimulus labels shuffled (gray). This suggests that information about microstimulation patterns is present in latent form, ready to be exploited by the brain when it becomes task relevant.
194 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
J. Neurosci., February 24, 2016 • 36(8):2406 –2424 • 2421
Left Hemisphere
A
Right Hemisphere
2
1
rost
rost lat
3
4
lat
B
120 sig020i
sig019i
150 sig009i
sig010i 150
40
80
100
20
40
50
100
0
0 120
0 150 sig014i
0 sig013i
sig024i
sig023i
50
100
200
50
100
100
80 40 0−0.2
0
0.2
0.4
0−0. 2
0
0.2
N4 120
40 0−0.2
sig006i
0
0.2
0.4
40
Response [z]
C
0
0.2
0
0.2
0.4
0
0.2
0.4
200 sig028i 150
100
100
50
50 0 200 sig032i
0 sig031i
150
100
100
50
50
0 −0.2
0.4
0−0.2
150
150
80
0−0.2
0.4
sig027i
80
0
0.2
N3
sig002i
80 40
0
N2
N1
sig001i
50
0
0
0.2
0.4
0−0.2
40 30 20 10 0 −0.2 −0.1
0
0.1 0.2 Time [s]
0.3
0.4
Figure 9. S1 neurons that respond to ICMS are still responsive to whisker deflections. A, Flattened cortical sections through both S1 hemispheres in one animal show the location of electrodes. The asterisks mark the electrode locations. B, Multiunit response to whisker deflections is robust for all units. Figure shows PSTH in units from the same rat depicted in A, after training on the IR discrimination task, upon deflection of the right whiskers with a 200 ms air puff delivered as the rat sat in the behavioral chamber. C, Mean multiunit response to whisker deflection, in all 36 units recorded in three animals. Bin width, 1 ms.
makes the natural mapping easier to acquire (Diamond et al., 1999; Harris et al., 1999). Previous work has shown that the barrel cortex integrates information across hemispheres to perform simple spatial discrim-
ination tasks (Krupa et al., 2001; Shuler et al., 2001). Our results show that interhemispheric stimulation patterns can be readily discriminated using a topographically distributed multisite ICMS in a very short time.
195 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
2422 • J. Neurosci., February 24, 2016 • 36(8):2406 –2424
0 Sensor 1 Sensor 2 Sensor 3 Sensor 4
Sensor Output [V]
2 4 6 8 10 -3
-2.5 -2 -1.5 -1 Time From End Of Trial [s]
-0.5
0
Figure 10. Voltage traces from four IR sensors. Representative mean ⫾ SEM sensor output as a function of time during trial. Time 0 is the time at which the rat selected a port. Note that ⬃1.5 s before the end of the trial, the two front sensors rapidly saturate as the animal runs toward the port, while the signals from the rear sensors drop. However, when the animal gets very close to the port, the IR source is broad enough to actually stimulate the rear IR sensors, as you can see with the final peak close the time of the poke. See Discussion.
Neuronal coding of IR stimuli After training with the IR neuroprosthesis, S1 neurons simultaneously encoded both tactile and IR stimuli in a synthetic multimodal map (Fig. 9). Future experiments will be needed to determine the effects of this new IR modality on the ability of the animals to perform tactile discrimination tasks using their whiskers (Ni and Maunsell, 2010). One surprising finding was that inactivation of rear-facing channels had a more deleterious effect on IR discrimination than inactivating the forward-facing IR sensors. The front channels were the dominant source of inputs to the system for the duration of most trials (Fig. 4F), so why did animals seem to preferentially use the rear channels? One possibility is that the rats use heightened activity in the back channels at the end of a trial as an indicator that they are getting very close to the correct port. Indeed, Figure 10 shows that the mean output of the rear voltage sensors actually increases when they are very close to the correct port. Neurobehavioral plasticity There were multiple types of plasticity exhibited here. Clearly, animals had to learn to use the new information source projected to S1. This demonstrates that, throughout the life of the animal, the brain continues to monitor the statistical structure of its inputs. Even at a behavioral level, animals made subtle adjustments to their behavior to more closely align the sensors on their head with the IR sources in their environment (Fig. 2 E, F ). It will be crucial, in a future study, to track behavior at finer spatiotemporal scales, and fully quantify the relationships among stimulation, behavior, and the neuronal response in S1. At a finer-grained level, our data suggest that responses in the adult cortex continuously adapt so that they will match the structure of those inputs. This was suggested by our original recording data, in which RF centers matched the general distribution of stimuli (Fig. 6). This was confirmed in an independent set of experiments in which IR-RF centers continued to match the input distribution, even when the front and rear channels were swapped. These results are similar to what is observed in development (Sengpiel et al., 1999).
Future experiments should probe just how far this plasticity can be pushed. For instance, it would be helpful to deliver predefined test pulses in awake and anesthetized animals to determine the degree to which responses to those test pulses indeed change over time. While the brains of rats were quite capable of adopting new behavioral strategies to harvest information from their IR environment, the remapping experiments examined the ability of animals to change that strategy when pushed into unusual regimes it would not normally experience. Thereby, we follow experiments like those using the retinal inversion lenses by Stratton (1896), the rewiring studies of Sur et al. (1988), frogs with rotated eyes by Udin (1983), and the work on the effects of adding a third eye to the frog by Katz and Constantine-Paton (1988). We are not aware of such inversion/remapping experiments being performed in previous sensory substitution systems in humans. While rats initially performed poorly after a remapping was applied (Fig. 3A), they quickly adapted when the remapping was relatively simple (front-back or left-right switches). However, when the topography was more complex (the diagonal remapping), they needed the help of visual cues to relearn the new mapping (Fig. 3 B, D). This illustrates the sensitivity of the brain to pre-existing topographic maps, even artificial maps, suggesting that that it cannot learn arbitrary input/output mappings. Interestingly, once animals learned one remapping, they learned subsequent remappings significantly faster (Fig. 3G). This suggests that the remapping experiments are triggering a central metaplasticity mechanism, such that animals literally learned to learn the new mappings. Such a boost in learning rate after being exposed to demanding or enriched environments and tasks was suggested by Hebb (1949). While metaplasticity may involve the modulation of molecular-level synaptic plasticity mechanisms (Abraham and Bear, 1996; Rebola et al., 2011), there are likely also higher-level cognitive mechanisms at work, such as changes in the expectation of the animal about the task. With the present rodent model, we can more thoroughly explore these options, for instance by blocking NMDA receptors during remapping experiments. One surprising effect was the influence of the task relevance of microstimulation on responses in S1 (Fig. 8). We delivered uninformative microstimulation patterns in rats performing an unrelated visual task. We expected similar response properties in both groups of animals, as it seems that passive stimulation is sufficient to induce significant response plasticity (Nudo et al., 1990; Recanzone et al., 1992; Maldonado and Gerstein, 1996). Instead, we saw drastically suppressed responses in the majority of units examined (Fig. 8). This complements previous observations that responses in primary sensory areas are strongly dependent on task context (Fanselow and Nicolelis, 1999; Fanselow et al., 2001; Nicolelis and Fanselow, 2002; Niell and Stryker, 2010; Fu et al., 2014). Our results suggest that responses depend not just on the statistical structure of the inputs, but also on their relevance. The causes of the suppressed activity could be a combination of attentional effects, differences in movement patterns when the information is not relevant, or other top-down corticocortical effects on S1 processing (Pais-Vieira et al., 2013). Curiously, these suppressed responses also carried significant information about the stimulation patterns delivered to the brain (Fig. 8E). Hence, while the S1 neuronal responses to uninformative stimuli were suppressed, the information remained latent in patterns of inhibitory activity. We hypothesize that the cortical response profile would become excitatory when the patterns of
196 Hartmann, Thomson et al. • Embedding an Infrared Representation in S1
stimulation become informative about events significant to the animal. Overall, this study highlights the remarkable plasticity of the adult mammalian brain. The brain seems to remain exquisitely tuned to different information sources well past any classic critical period in development. With the addition of more information channels, the ability of adult rats to exploit the information only improved (Fig. 2). Further, S1 neurons showed clear preferences for those stimuli statistically most likely to be delivered during the task, suggesting that the cortex continuously tunes its responses to the statistical structure of its inputs. However, the brain did not indiscriminately respond to the most likely stimuli. Rather, S1 responses were actually suppressed when statistically probable stimuli were not informative about task-relevant features in the environment of the animal. That is, in addition to activity-dependent plasticity mechanisms, powerful relevancesensitive mechanisms strongly sculpt cortical activity at a given time.
References Abraham WC, Bear MF (1996) Metaplasticity: the plasticity of synaptic plasticity. Trends Neurosci 19:126 –130. CrossRef Medline Amedi A, Stern WM, Camprodon JA, Bermpohl F, Merabet L, Rotman S, Hemond C, Meijer P, Pascual-Leone A (2007) Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nat Neurosci 10:687– 689. CrossRef Medline Andersen RA, Kellis S, Klaes C, Aflalo T (2014) Toward more versatile and intuitive cortical brain-machine interfaces. Curr Biol 24:R885–R897. CrossRef Medline Bach-y-Rita P, Collins CC, Saunders FA, White B, Scadden L (1969) Vision substitution by tactile image projection. Nature 221:963–964. CrossRef Medline Berens P (2009) CircStat: a MATLAB toolbox for circular statistics. J Stat Softw 31:1–21. Carmena JM, Lebedev MA, Crist RE, O’doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquz CS, Nicolelis MA (2003) Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol 1:E42. CrossRef Medline Chapin JK, Moxon KA, Markowitz RS, Nicolelis MA (1999) Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat Neurosci 2:664 – 670. CrossRef Medline Coulombe J, Sawan M, Gervais JF (2007) A highly flexible system for microstimulation of the visual cortex: design and implementation. IEEE Trans Biomed Circuits Syst 1:258 –269. CrossRef Medline Davis TS, Parker RA, House PA, Bagley E, Wendelken S, Normann RA, Greger B (2012) Spatial and temporal characteristics of V1 microstimulation during chronic implantation of a microelectrode array in a behaving macaque. J Neural Eng 9:065003. CrossRef Medline Deuschl G, Schade-Brittinger C, Krack P, Volkmann J, Schäfer H, Bötzel K, Daniels C, Deutschländer A, Dillmann U, Eisner W, Gruber D, Hamel W, Herzog J, Hilker R, Klebe S, Kloss M, Koy J, Krause M, Kupsch A, Lorenz D, et al. (2006) A randomized trial of deep-brain stimulation for Parkinson’s disease. N Engl J Med 355:896 –908. CrossRef Medline Diamond ME, Petersen RS, Harris JA (1999) Learning through maps: functional significance of topographic organization in primary sensory cortex. J Neurobiol 41:64 – 68. CrossRef Medline Dobelle WH (2000) Artificial vision for the blind by connecting a television camera to the visual cortex. ASAIO J 46:3–9. CrossRef Medline Dobelle WH, Mladejovsky MG (1974) Phosphenes produced by electrical stimulation of human occipital cortex, and their application to the development of a prosthesis for the blind. J Physiol 243:553–576. CrossRef Medline Fanselow EE, Nicolelis MA (1999) Behavioral modulation of tactile responses in the rat somatosensory system. J Neurosci 19:7603–7616. Medline Fanselow EE, Sameshima K, Baccala LA, Nicolelis MA (2001) Thalamic bursting in rats during different awake behavioral states. Proc Natl Acad Sci U S A 98:15330 –15335. CrossRef Medline Frost DO, Metin C (1985) Induction of functional retinal projections to the somatosensory system. Nature 317:162–164. CrossRef Medline
J. Neurosci., February 24, 2016 • 36(8):2406 –2424 • 2423 Fu Y, Tucciarone JM, Espinosa JS, Sheng N, Darcy DP, Nicoll RA, Huang ZJ, Stryker MP (2014) A cortical circuit for gain control by behavioral state. Cell 156:1139 –1152. CrossRef Medline Fuentes R, Petersson P, Siesser WB, Caron MG, Nicolelis MA (2009) Spinal cord stimulation restores locomotion in animal models of Parkinson’s disease. Science 323:1578 –1582. CrossRef Medline Georgopoulos AP, Schwartz AB, Kettner RE (1986) Neuronal population coding of movement direction. Science 233:1416 –1419. CrossRef Medline Griffin DM, Hudson HM, Belhaj-Saïf A, Cheney PD (2011) Hijacking cortical motor output with repetitive microstimulation. J Neurosci 31: 13088 –13096. CrossRef Medline Hanson TL, Ómarsson B, O’Doherty JE, Peikon ID, Lebedev M, Nicolelis MA (2012) High-side digitally current controlled biphasic bipolar microstimulator. IEEE Trans Neural Syst Rehabil Eng 20:331–340. CrossRef Medline Harris JA, Petersen RS, Diamond ME (1999) Distribution of tactile learning and its neural basis. Proc Natl Acad Sci U S A 96:7587–7591. CrossRef Medline Hebb DO (1949) The organization of behavior; a neuropsychological theory. New York: Wiley. Horng SH, Sur M (2006) Visual activity and cortical rewiring: activitydependent plasticity of cortical networks. Prog Brain Res 157:3–11. CrossRef Medline Ifft PJ, Shokur S, Li Z, Lebedev MA, Nicolelis MA (2013) A brain-machine interface enables bimanual arm movements in monkeys. Sci Transl Med 5:210ra154 –210ra154. CrossRef Medline Kärcher SM, Fenzlaff S, Hartmann D, Nagel SK, König P (2012) Sensory augmentation for the blind. Front Hum Neurosci 6:37. CrossRef Medline Kaspar K, König S, Schwandt J, König P (2014) The experience of new sensorimotor contingencies by sensory augmentation. Conscious Cogn 28: 47– 63. CrossRef Medline Katz LC, Constantine-Paton M (1988) Relationships between segregated afferents and postsynaptic neurones in the optic tectum of three-eyed frogs. J Neurosci 8:3160 –3180. Medline Kim JK, Zatorre RJ (2008) Generalized learning of visual-to-auditory substitution in sighted individuals. Brain Res 1242:263–275. CrossRef Medline Krupa DJ, Matell MS, Brisben AJ, Oliveira LM, Nicolelis MA (2001) Behavioral properties of the trigeminal somatosensory system in rats performing whisker-dependent tactile discriminations. J Neurosci 21:5752–5763. Medline Lebedev MA (2014) How to read neuron-dropping curves? Front Syst Neurosci 8:102. CrossRef Medline Lewis PM, Ackland HM, Lowery AJ, Rosenfeld JV (2015) Restoration of vision in blind individuals using bionic devices: a review with a focus on cortical visual prostheses. Brain Res 1595:51–73. CrossRef Medline Macherey O, Carlyon RP (2014) Cochlear implants. Curr Biol 24:R878 – R884. CrossRef Medline Maldonado PE, Gerstein GL (1996) Reorganization in the auditory cortex of the rat induced by intracortical microstimulation: a multiple singleunit study. Exp Brain Res 112:420 – 430. Medline Mayberg HS, Lozano AM, Voon V, McNeely HE, Seminowicz D, Hamani C, Schwalb JM, Kennedy SH (2005) Deep brain stimulation for treatmentresistant depression. Neuron 45:651– 660. CrossRef Medline Nagel SK, Carl C, Kringe T, Märtin R, König P (2005) Beyond sensory substitution—learning the sixth sense. J Neural Eng 2:R13–R26. CrossRef Medline Ni AM, Maunsell JH (2010) Microstimulation reveals limits in detecting different signals from a local cortical region. Curr Biol 20:824 – 828. CrossRef Medline Nicolelis MA (2011) Limbs that move by thought control. New Scientist 210:26 –27. Nicolelis MA, Fanselow EE (2002) Dynamic shifting in thalamocortical processing during different behavioural states. Philos Trans R Soc Lond B Biol Sci 357:1753–1758. CrossRef Medline Niell CM, Stryker MP (2010) Modulation of visual responses by behavioral state in mouse visual cortex. Neuron 65:472– 479. CrossRef Medline Norimoto H, Ikegaya Y (2015) Visual cortical prosthesis with a geomagnetic compass restores spatial navigation in blind rats. Curr Biol 25:1091–1095. CrossRef Medline Nudo RJ, Jenkins WM, Merzenich MM (1990) Repetitive microstimulation
197 2424 • J. Neurosci., February 24, 2016 • 36(8):2406 –2424 alters the cortical representation of movements in adult rats. Somatosens Mot Res 7:463– 483. CrossRef Medline O’Doherty JE, Lebedev MA, Li Z, Nicolelis MA (2012) Virtual active touch using randomly patterned intracortical microstimulation. IEEE Trans Neural Syst Rehabil Eng 20:85–93. CrossRef Medline Pais-Vieira M, Lebedev MA, Wiest MC, Nicolelis MA (2013) Simultaneous top-down modulation of the primary somatosensory cortex and thalamic nuclei during active tactile discrimination. J Neurosci 33:4076 – 4093. CrossRef Medline Rebola N, Carta M, Lanore F, Blanchet C, Mulle C (2011) NMDA receptordependent metaplasticity at hippocampal mossy fiber synapses. Nat Neurosci 14:691– 693. CrossRef Medline Recanzone GH, Merzenich MM, Dinse HR (1992) Expansion of the cortical representation of a specific skin field in primary somatosensory cortex by intracortical microstimulation. Cereb Cortex 2:181–196. CrossRef Medline Santana MB, Halje P, Simplício H, Richter U, Freire MA, Petersson P, Fuentes R, Nicolelis MA (2014) Spinal cord stimulation alleviates motor deficits in a primate model of Parkinson disease. Neuron 84:716 –722. CrossRef Medline Sengpiel F, Stawinski P, Bonhoeffer T (1999) Influence of experience on orientation maps in cat visual cortex. Nat Neurosci 2:727–732. CrossRef Medline Shuler MG, Krupa DJ, Nicolelis MA (2001) Bilateral integration of whisker information in the primary somatosensory cortex of rats. J Neurosci 21: 5251–5261. Medline Srivastava NR, Troyk PR, Dagnelie G (2009) Detection, eye-hand coordina-
Hartmann, Thomson et al. • Embedding an Infrared Representation in S1 tion and virtual mobility performance in simulated vision for a cortical visual prosthesis device. J Neural Eng 6:035008. CrossRef Medline Stratton G (1896) Some preliminary experiments on vision without inversion of the retinal image. Psychol Rev 3:611– 617. Sur M, Garraghty PE, Roe AW (1988) Experimentally induced visual projections into auditory thalamus and cortex. Science 242:1437–1441. CrossRef Medline Thomson EE, Carra R, Nicolelis MA (2013) Perceiving invisible light through a somatosensory cortical prosthesis. Nat Commun 4:1482. CrossRef Medline Udin SB (1983) Abnormal visual input leads to development of abnormal axon trajectories in frogs. Nature 301:336 –338. CrossRef Medline von Melchner L, Pallas SL, Sur M (2000) Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature 404:871– 876. CrossRef Medline Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, Kim J, Biggs SJ, Srinivasan MA, Nicolelis MA (2000) Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408:361–365. CrossRef Medline Wiest M, Thomson E, Meloy J (2008) Multielectrode recordings in the somatosensory system. In: Methods for neural ensemble recordings (Nicolelis MAL, ed), pp 97–124. Boca Raton, FL: CRC. Wilson BS, Finley CC, Lawson DT, Wolford RD, Eddington DK, Rabinowitz WM (1991) Better speech recognition with cochlear implants. Nature 352:236 –238. CrossRef Medline Yadav AP, Fuentes R, Zhang H, Vinholo T, Wang CH, Freire MA, Nicolelis MA (2014) Chronic spinal cord electrical stimulation protects against 6-hydroxydopamine lesions. Sci Rep 4:3839. CrossRef Medline
198 New Research
Sensory and Motor Systems
Cortical Neuroprosthesis Merges Visible and Invisible Light Without Impairing Native Sensory Function Eric E. Thomson,1,4 Ivan Zea,1 William Windham,1 Pedowitz,7 Wendy França,1 Ana L. Graneiro,8 and
Yohann Thenaisie,6 Cameron Walker,1 Miguel A.L. Nicolelis1,2,3,4,5
Jason
DOI:http://dx.doi.org/10.1523/ENEURO.0262-17.2017 1
Department of Neurobiology, Duke University, Durham, NC 27710, 2Biomedical Engineering, Duke University, Durham, NC 27710, 3Psychology and Neuroscience, Duke University, Durham, NC 27710, 4Duke Center for Neuroengineering, Duke University, Durham, NC 27710, 5Edmond and Lily Safra International Institute of Neuroscience of Natal, Natal, 59066-060, Brazil, 6Department of Biology, École Normale Supérieure De Lyon, Lyon, 69342, France, 7School of Medicine, University of California San Diego, La Jolla, CA 92093, and 8Florida International University, Miami, FL 33199
Abstract Adult rats equipped with a sensory prosthesis, which transduced infrared (IR) signals into electrical signals delivered to somatosensory cortex (S1), took approximately 4 d to learn a four-choice IR discrimination task. Here, we show that when such IR signals are projected to the primary visual cortex (V1), rats that are pretrained in a visual-discrimination task typically learn the same IR discrimination task on their first day of training. However, without prior training on a visual discrimination task, the learning rates for S1- and V1-implanted animals converged, suggesting there is no intrinsic difference in learning rate between the two areas. We also discovered that animals were able to integrate IR information into the ongoing visual processing stream in V1, performing a visual-IR integration task in which they had to combine IR and visual information. Furthermore, when the IR prosthesis was implanted in S1, rats showed no impairment in their ability to use their whiskers to perform a tactile discrimination task. Instead, in some rats, this ability was actually enhanced. Cumulatively, these findings suggest that cortical sensory neuroprostheses can rapidly augment the representational scope of primary sensory areas, integrating novel sources of information into ongoing processing while incurring minimal loss of native function. Key words: Behavior; multisensory integration; neuroprosthesis; plasticity
Significance Statement Using a sensory neuroprosthesis that projects information from the IR environment to primary sensory areas, we show that adult rats can rapidly integrate completely novel sensory information into preexisting cortical maps. When the prosthesis is implanted in V1, animals can learn to perform a multimodal integration task, fusing IR and visual information that is simultaneously superimposed on the same cortical area. When the prosthesis is implanted in S1, the tactile function of S1 is left undisturbed, and often enhanced. Hence, it is possible to merge multiple streams of information onto the same primary cortical area without compromising its original function. This is auspicious for the development of sensory prosthetic systems for adult victims of brain injury.
Received July 17, 2017; accepted November 20, 2017; First published December 07, 2017. The authors declare no competing financial interests.
November/December 2017, 4(6) e0262-17.2017 1–17
Author contributions: E.E.T., I.Z., and M.A.N. designed research; E.E.T., I.Z., W.W., Y.T., C.W., J.P., W.F., and A.G. performed research; E.E.T. and I.Z. analyzed data; E.E.T., I.Z., and M.A.N. wrote the paper.
199 New Research
Introduction A fundamental goal in neuroscience is to delineate the mechanisms and limits of adult neurobehavioral plasticity. This task is important for both basic neuroscience and modern rehabilitative medicine (Lledo et al. 2006). This question was previously addressed using a real-time closed-loop sensory prosthetic system that allowed researchers to investigate how adult mammals responded when information from a completely new source, infrared light (IR), was projected to primary sensory regions (Thomson et al. 2013; Hartmann et al. 2016). In this cortical neuroprosthesis, the output of four head-mounted IR detectors was coupled to topographically distributed stimulating electrodes chronically implanted in the somatosensory cortex (S1) of adult rats (Thomson et al. 2013; Hartmann et al. 2016). As with ablation-induced sensory-cortex rewiring in newborns (Frost and Metin, 1985; Sur et al. 1988), adult rats readily learned to use this new source of information, ultimately performing as well as in corresponding visual discrimination tasks. Similar results were also observed when information from the geomagnetic environment was projected to V1 (Norimoto and Ikegaya, 2015). Such initial results with cortical prosthetic systems raised several key questions that we presently address. One, because of the facility with which S1 absorbed information about IR sources, we postulated that different primary sensory areas should display similar levels of sensory plasticity. We advance this thesis based on previous results, but also on anatomic grounds: similar canonical microcircuits are iterated across the neocortical mantle (Haeusler and Maass, 2007; Kouh and Poggio, 2008; Habenschuss et al. 2013; Harris and Shepherd, 2015). To test this equipotentiality of plasticity hypothesis, we implanted stimulating electrodes in primary visual cortex (V1) of adult rats to measure their ability to use the IR neuroprosthesis and compare this with the performance of animals that used an S1-based prosthesis. In the original rewiring experiments, performed in newborns, the transformed areas received inputs from only one sensory modality–for instance, rewired A1 received only visual inputs (Sur et al. 1988; von Melchner et al. 2000). In contrast, in animals using the IR prosthesis, both native and IR information was simultaneously available to Research reported in this publication was supported by the NINDS of NIH under Award Number R01DE011451 to MALN. E. Thomson and I. Zea are co-authors. Acknowledgments: The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. We thank J. Meloy for help building the experimental setup and electrodes; J. O’Doherty and G. Lehew for helpful discussions; S. Halkiotis for editing this manuscript; and Joshua Khani, Tom Nayer, Brittany Klein, Kevin Oh, Andrew Pollizzi, and Chris Penny for help with experiments. Correspondence should be addressed to Miguel A.L. Nicolelis, MD, PhD, Box 103905, Dept. of Neurobiology, Duke University, 210 Research Drive, GSRBII Room 4028, Durham, NC 27710. E-mail: nicoleli@neuro.duke.edu. DOI:http://dx.doi.org/10.1523/ENEURO.0262-17.2017 Copyright © 2017 Thomson et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International license, which permits unrestricted use, distribution and reproduction in any medium provided that the original work is properly attributed.
November/December 2017, 4(6) e0262-17.2017
2 of 17
the primary sensory area implanted with the prosthesis, and this allowed us to ask additional questions about how the brain handles the simultaneous superposition of multiple sources of information. For instance, can the brain integrate both information sources simultaneously to perform a multisensory integration task? (Deneve and Pouget, 2004; Ghazanfar and Schroeder, 2006; Macaluso, 2006; Stein and Stanford, 2008; Klemen and Chambers, 2012). To examine this question, we developed a visual-IR multisensory integration task, in which animals were presented with multiple visible and IR cues simultaneously, and rats were rewarded only if they ignored distractor visual and IR cues, and selected the stimulus that combined both visible and IR lights. As far as we can tell, no previous sensory prosthetic system has been tested with a behavioral task with such a high degree of difficulty, requiring subjects to integrate information from two qualitatively different sources simultaneously superimposed onto the same cortical area. Further, when a new source of information is projected to a cortical area, does this have unintended deleterious side effects? There are multiple lines of evidence that intracortical microstimulation (ICMS), by itself, is a powerful inducer of cortical plasticity (Recanzone et al. 1992; Maldonado and Gerstein, 1996; Godde et al. 2002). However, there is also evidence from monkeys that repeated microstimulation in V1 can impair performance on visual detection tasks (Ni and Maunsell, 2010). Also, there is evidence from the motor system of monkeys that ICMS can even highjack the cortex, displacing its original function (Griffin et al. 2011). To directly address the question of how a prosthetic system used in a discrimination task affects the original function of the implanted area, we examined how an S1-implanted prosthesis influenced performance on a whisker-based tactile discrimination task. The results will have potential clinical implications for neuroprosthetic design, as well as more basic implications about the limits of plasticity of a given region of primary sensory cortex.
Materials and Methods General All experiments were performed on adult female LongEvans rats (Harlan Sprague Dawley Laboratories), 250 – 300 g. All animals were treated humanely to minimize stress, and the Duke University Institutional Animal Care & Use Committee (IACUC) approved all surgical and behavioral methods. Behavioral methods Behavior chamber The majority of behavioral experiments were performed in a cylindrical behavioral chamber (50.8 cm or 20 inches; Fig. 1A). A button was placed in the center of the chamber that allowed the rats to initiate a trial. In some sessions, the chamber included a mechanical button; because the mechanical button was difficult to learn to use, it was later replaced with an infrared photobeam recessed into the floor. Each chamber contained four ports around its circumference, initially evenly spaced 90˚ apart (Fig. 1A). On eNeuro.org
200 New Research
A
B
Port 2
C
IR light
2
1
3
4
S1BF V1
Water
Port 1
3 of 17
Visible light
θ Button
N1
Port 3
N2 N3
N4
Port 4
V1
D
E Optimized
200
Rostral Medial
350 µm
300
Original
2 mm
ground
100 0 Min
Low
Mid
High
Max
300 µm
Frequency (Hz)
400
F
350 µm
IR Level
Figure 1. Methods. A, Setup of the IR behavior chamber. Four reward ports line the walls of the circular chamber. The angle indicates how far apart adjacent ports are. Inset shows design of each individual port. B, Illustration of the mapping from four IR detectors (1– 4) to four microstimulating electrode locations (N1–N4) in V1. C, Dorsal view of rat skull showing the placement of the S1 and V1 stimulating electrode arrays. The green and blue zones correspond to the S1 barrel field (S1BF) and V1, respectively, and these areas were reconstructed from (Paxinos and Watson, 2007). The dark horizontal and vertical reference lines intersect at bregma, and each small square in the grid is 1 mm2. D, Transforms from IR levels detected in individual IR detectors to microstimulation frequency in individual stimulating channels. The black line is the original transform used in previous articles, and some animals in the present article. The red line shows the optimized transform (see Methods). E, Design of electrode arrays consisting of two bundles of eight stimulating electrodes placed 2 mm apart (see Methods for more details). F, Flattened cortical slice from animal implanted in V1, stained with cytochrome-oxidase. Inset shows detail of macrovibrissae barrels. Oval at bottom (caudal) shows the location of tissue damage from the V1 implant. Reference line: 1 mm.
the inner surface of each port there was a visible LED and an infrared LED (Opto Semiconductors; 940-nm peak emittance with a range of non-zero emission between 825 and 1000 nm). The IR sources had an angular width at half-maximum of 120°. Each port contained a water spout inside a conical recess that also included an infrared photobeam, allowing us to detect when the rat selected that spout (Fig. 1A, inset). Visual discrimination task In the visual discrimination task, the animal initiated each trial by pressing a button in the center of the behavioral chamber, and then a single visible light was turned on in a single port. The animal received a small amount of water if it selected that port. Otherwise, an error tone sounded and there was a timeout delay (4 –10 s) before the next trial could be initiated. We considered threshold performance to be when an animal reached 85% correct and performed at least 150 trials in a session, at which point the animal was ready to be implanted with stimulating electrodes and learn the IR discrimination task. A total of 35 rats were trained on the visual discrimination task (21 to compare S1 and V1 learning rates on the basic IR discrimination task, 6 to examine visual-IR inteNovember/December 2017, 4(6) e0262-17.2017
gration—3 in S1 and 3 in V1—and 8 to examine how whisker discrimination is influenced by microstimulation in S1). IR discrimination Once an animal reached threshold performance on the initial visual discrimination task, we trained them to discriminate single IR light sources in the same behavioral chamber. We first implanted an electrode array into the appropriate area (either S1 or V1, as described under Surgical procedures). After at least a week of healing from surgery, and after determining current thresholds for the four stimulating channels (see Microstimulation), we then trained them on a task that was structurally identical to the initial visual discrimination task. The only difference was that we replaced the visual light with IR light. Each rat was equipped with four IR sensors distributed evenly around the circumference of the animal’s head on the horizontal plane (Fig. 1B). The IR sensors (Lite-On) had a peak spectral sensitivity at a wavelength of 940 m and a 20° width at half of its maximum sensitivity. After processing (see IR¡Stim transform), information from each sensor was projected to a stimulating channel at a different cortical location. The information was proeNeuro.org
201 New Research
jected in a topographically natural way: for instance, information from the left sensors was projected to the right cortical hemispheres, and information from the anterior sensor was projected to the anterior region of S1, which represents the rostral whiskers (Hartmann et al. 2016). A similar pattern was replicated in V1 (Fig. 1B, C). Note also that surgeries to implant S1 and V1 were randomly interleaved, so as not to introduce bias or trends in the procedures. During training on the IR-discrimination task, the visual lights were progressively replaced with IR lights. Specifically, they began with a mixture of two types of trials: IR only trials, in which only IR lights were turned on, and IR ⫹ vis trials, in which IR light (and ICMS) was shown, followed by visible light, to help animals associate ICMS with reinforced visual light. Specifically, on IR ⫹ vis trials, we would turn on the IR light, and then 400 –500 ms later, turn on the visible light, and then 500 ms later, turn off the IR light for the rest of the trial. On their first day of training, the percentage of IR-only trials was 20%, and this increased to 100% by the fifth day of training. All percentage correct values on IR discrimination were based on IR-only trials. A total of 41 rats were trained on the IR discrimination task (21 used to compare learning rates on the basic IR discrimination task, 6 to look at visual-IR integration, 8 to look at whisker discrimination, and 6 trained in the naive task; these 6 were not pretrained on the visual task). We did not perform a priori power calculations to determine the number of animals to use in the study. Integrated IR ⫹ visual discrimination In the integrated IR ⫹ visual discrimination task, we trained six animals to ignore visual and IR distractors and select the port in which both the IR and visual lights were activated simultaneously. On each trial, each of the four ports was randomly assigned one of four stimulus conditions: none, IR only, visual only, or both IR and visual (see Results). The IR- and visual-only ports were distractors, and the animal received a reward only when it approached the port with both IR and visual lights on. Naive training One group of six animals was taught to perform the IR discrimination task without being pretrained in the visual discrimination task. We first placed these naive rats in the behavioral chamber with only a single port placed along its circumference, to familiarize them with the behavioral chamber. Using the single-port setup, the rat was taught to poke a center button to initialize trials and subsequently poke the port to receive a reward. To avoid generating a directional bias in the rats, we randomly selected a position in the chamber’s circumference where the port would be placed every session. Once they were familiarized with the chamber (performing 100 trials per session for 3 d), we proceeded to implant them with stimulating electrodes in either S1 or V1. We then trained the rats using only infrared trials (no mixed visual ⫹ IR trials as in the standard IR discrimination protocol). The trainer was blind to the implant location: an assistant was in charge of setting a rat in the chamber while the trainer would make adjustNovember/December 2017, 4(6) e0262-17.2017
4 of 17
ments to the stimulation currents during the session. Every day, the order of the animals was randomized, and the software was written so that the trainer could only adjust currents relative to the initial current. IR¡Stim transform The IR to microstimulation frequency transform is a step function from IR sensor voltage (which we sometimes refer to as “IR level” or “IR intensity”) to microstimulation frequency as shown in Fig. 1D. The original transform, described in Thomson et al. (2013), was roughly exponential in shape, with 17 step intervals. In the previous article, researchers used this transform to stimulate rats when neuronal activity was not simultaneously recorded. When online recording was introduced during the task to limit bandwidth, the transform was restricted to only seven intervals (Hartmann et al. 2016). We used this same transform, but also developed a new optimized transform such that each stimulation frequency would occur with equal probability over the course of a session. To this end, we measured the probability of different voltage values occurring across multiple sessions. We then chose a new transform that would guarantee that each voltage step would occur with equal frequency, thereby increasing the effective information available to the animal on each trial. The voltage steps required to guarantee equal likelihood of occurrence were roughly logarithmically spaced, and we increased the frequency constantly for each step, so the transform from IR level to stimulation frequency is logarithmic, as seen in Fig. 1C. It is important to mention that we used all frequencies, and we observed no harmful effects [we previously have avoided frequencies 20 –100 Hz (Thomson et al. 2013; Hartmann et al. 2016) for fear of inducing kindling (Goddard, 1967)]. Optimizing the sensitivity of IR sensors IR sensors played a crucial role in the IR prosthetic system. The IR photodetector circuit included a simple voltage divider that let us adjust the sensitivity of the sensor. In practice, changing the resistance by 100 k⍀ decreased the measured output of the IR sensor by ⬃20% at a given IR level. This setup allowed us to scale the sensitivity in each IR sensor independently. In the IR-visual integration task, rats performed the task with their sensitivity set at a high value (resistance set to 475 k⍀) for several days until performance plateaued, usually 75% correct for 5 consecutive days. We then decreased sensitivity in steps, allowing the animal to reach a behavioral plateau at each new sensitivity level. For each rat, the optimum IR sensitivity was selected based on the best performance plateau, calculated over all sensitivity levels tested. Aperture-width discrimination We trained eight rats to discriminate the width of a tunnel using only their facial whiskers, a task discussed extensively previously (Krupa et al. 2001, 2004; Wiest et al. 2010). More specifically, we trained them to use their whiskers to discriminate the size of a variable-width aperture. Rats were trained to sample the width of a variable-width aperture using their whiskers (Fig. 4) and were rewarded for moving to the left reward port when the eNeuro.org
202 New Research
aperture was narrow and to the right reward port when the aperture was wide (see Results). Once they reached criterion performance (80% correct) with widths of 54 and 78 mm, we then switched some rats to a multiple width version (with either 12 or 14 widths). For the multiple width task, widths less than 66 mm counted as narrow. Surgical procedures Surgeries were performed under ketamine (Ketaset) and xylazine (AnaSed, Akorn Animal Health) anesthetic with 100 mg/kg ketamine and 0.06 mg/kg xylazine. After cleaning the surface of the skull, we placed three to six titanium screws (Antrin Miniature Specialties) into the skull, sealed them in place with Metabond (Parkell), and coated the rest of the cleaned skull surface with Metabond before continuing with the surgery. We then performed the craniotomy, removed the dura, and implanted electrodes that were built in-house (see Electrode design). The coordinates for S1 were –2.5 mm posterior and 5.5 mm lateral to bregma. The V1 coordinates were –7.45 posterior and 3.25 lateral to bregma, oriented 30.5° relative to the midline (with the anterior portion more lateral; Fig. 1C). After the electrodes were lowered to the proper depth (1.5 mm in S1, and either 0.8 or 1.3 mm in V1; Results), we sealed the craniotomy with Quick Set dental acrylic (Coltene) and cyanoacrylic (Hobbylinc.com). Locations of electrodes were verified histologically (see below) and physiologically by examining manual responses to whisker deflections (S1) and LED flashes (V1). Electrophysiology Electrode design The stimulating/recording arrays consisted of 32 42-m stainless steel microwires arranged into four groups of eight (see Fig. 1E; Hartmann et al. 2016). Microwires were paired, such that each cortical penetration contained two microwires, with the potential for each couple to be used as an anodic/cathodic dyad for microstimulation (see Microstimulation). The pairs were 300 m apart, and this was also the distance between different adjacent wires in the same group. Clusters were 2 mm apart within a hemisphere (Fig. 1E). All electrodes were designed and built in-house. Histology For histologic verification, we used cytochrome oxidase (CO) staining of flattened cortical sections, as described extensively elsewhere (Wong-Riley, 1979; Hartmann et al. 2016). Briefly, after perfusing the animal with 0.1 M phosphate buffer, we removed subcortical tissue and flattened the cortex overnight in 30% sucrose. After freezing, we sliced the brain at 40 – 60 m and collected free-sections in 0.1 M phosphate buffer (pH 7.4) and placed them in standard CO reaction solution (Wong-Riley, 1979). We gently agitated the slices at room temperature for 2– 6 h until the sections appeared golden-brown to the eye (Fig. 1F). Recording Single- and multiunit extracellular neurophysiological data were recorded using the Multichannel Acquisition Processor (MAP; Plexon). After collecting data, during November/December 2017, 4(6) e0262-17.2017
5 of 17
offline sorting we selected spike wave form features (principal components, peak amplitude, etc.) that provided optimal separation using Offline Sorter software (Plexon; typically we used the principal components, but sometimes we used other features such as wave form peak or wave form energy). Single-unit quality was determined by separation of the unit from the noise in the 2- or 3-D feature space and the presence of a clear absolute and relative refractory period in the autocorrelogram for the putative unit. Microstimulation We stimulated using a four-channel biphasic, chargebalanced microstimulator (Hanson et al. 2012). Each stimulation pulse consisted of a 100-s pulse from the anodic wire, followed by a 50-s pause, followed by a 100-s pulse in the cathodic wire. After the animals recovered from surgery, we determined the minimal current threshold for four stimulating channels, one in each location. In awake relaxed animals, we would deliver a 250-Hz train of stimulation for 75 ms, starting with a low current (5 A), and incrementally increased the current amplitude until we visually saw a reaction from the animal (a movement of the head, locomotion, etc.). We also measured neuronal thresholds: we applied 250-Hz trains of stimulation for 75 ms in isoflurane-anesthetized animals and noted the current amplitude at which we observed a response above baseline (the threshold current), and also the amplitude at which the neuronal responses saturated (the saturating current, which is the current above which we did not observe an increase in response). In practice, we have found that animals learn the task faster when we apply currents well above the threshold for evoking a response, as long as the currents are not aversive to the animal (Hartmann et al. 2016). Hence, for our initial currents for the prosthesis, we typically started well above the neuronal threshold, at twice the saturating current (typically 50 –90 A). We took it as a sign that a current was aversive if the animal began scratching its face or stopped participating in the task in response to ICMS. Whisker receptive field mapping To determine the principal whiskers that evoked activity in an S1 unit, we recorded in lightly anesthetized (isoflurane) animals while stimulating individual whiskers with a hand-held probe or air puff and logged the whiskers that evoked a noticeable change in the baseline response on an audio monitor. Note that the goal was to quickly determine which whiskers needed to be trimmed during the whisker discrimination task, not to construct detailed quantitative receptive field maps, so we used this relatively coarse method to minimize the animals’ time under general anesthesia. Analysis Stimulus population vector To compactly represent the set of four microstimulation bursts delivered to the brain at a given time, we employed a stimulus population vector, which is based on population vector representations of movement from the motor eNeuro.org
203 New Research
6 of 17
Table 1. Statistical analysis Figure 2B 2E 2F 2F 2F 2F 2G 2G 3E 4C 4D 5E 5E 5E 5E 5E 5F 5F 5F 5F 5F 5G 5G 5G 5G 5G 6E 6E
Test Two-tailed t test Two-tailed t test ANOVA Two-way ANOVA Two-way ANOVA Two-way ANOVA Two-tailed t test Two-tailed t test F test Two-tailed t test Two-tailed t test Two-way ANOVA Two-way ANOVA Two-way ANOVA Two-tailed t test Two-tailed t test Two-way ANOVA Two-way ANOVA Two-way ANOVA Two-tailed t test Two-tailed t test Two-way ANOVA Two-way ANOVA Two-way ANOVA Two-tailed t test Two-tailed t test Chi-squared test Chi-squared test
Quantities compared S1 vs. V1: num sessions to learn S1 vs. V1: num trials to learn V1: percent correct (PC) vs. difficulty Factor 1 (angle): mean PC Factor 2 (implant location): mean PC Interaction (angle ⫻ location): mean PC Naive V1 vs. pretrained S1: num sessions to learn Naive S1 vs. V1: num sessions to learn Stimulus location vs. receptive field peak correlation S1 vs. V1: num sessions to learn integrated task S1 vs. V1: num trials to learn integrated task Factor 1 (animal group): PC for two-width case (54/78) Factor 2 (treatment): PC for two-width case Interaction (group ⫻ treatment): PC two widths Two-width PC change versus zero: after stimulation Two-width PC change versus zero: after clipping Factor 1 (animal group): PC multiwidth (12/14 widths) Factor 2 (treatment): PC multiwidth Interaction (group ⫻ treatment): PC multiwidth Multiwidth PC change versus zero: after stimulation Multiwidth PC change versus zero: after clipping Factor 1 (animal group): sensitivity Factor 2 (treatment): sensitivity Interaction (group ⫻ treatment): sensitivity Sensitivity change versus zero: group after stim Sensitivity change versus zero: group after clip Proportion w/anticipatory response in stim vs. control Proportion w/response to stim in stim vs. control
control literature (Georgopoulos et al. 1986). Briefly, it provides an intuitive geometric 2-D representation of the full stimulus being delivered to the four cortical locations at a given time. For instance, if the two anterior stimulating channels are maximally activated, then the population vector points to the front. If the two left channels are active, it will point to the left, etc. Mathematically the stimulus population vector at time t, s(t) is defined as 4
s共t兲 ⫽
兺 f (t)v , i
i
i⫽1
where fi(t) is the microstimulation frequency in channel i, and vi is a vector that points in the direction of IR channel i on the rat’s head (for instance, v1 ⫽ ⬍1,1⬎: see Fig. 1B). For more discussion of the stimulus population vector, see Results and Fig. 3. For a more extensive introduction, see (Hartmann et al. 2016). Psychometric curve fitting For the aperture-width discrimination task in which 12 or 14 widths were presented, we fit the behavioral data to a Weibull function: a saturating exponential function with four free parameters. We used Matlab to fit our behavioral data, in the least squares sense, to the following: min ⫹ (max ⫺min) 1 ⫺ e⫺冉 冊 ,
关
x k
兴
where min is the minimum value, max is the maximum value, k is the shape parameter, and is the scale paNovember/December 2017, 4(6) e0262-17.2017
P-value 0.0032 0.0047 0.0015 0.0003 0.6186 0.5024 0.8518 0.6433 0.7 1 0.6258 0.5485 0.0004 0.8597 0.0136 0.0014 0.9187 0.0179 0.6384 0.153 0.009 0.7111 0.0105 0.0868 0.0107 0.3449 0.009 0.053
Effect size 1.6259 1.6048 0.9904 0.4261 0.043 0.1336 0.1201 0.4082 0.0259 0 0.4306 0.0833 1.115 0.4738 1.1567 2.0877 0.026 1.1543 0.1217 0.6874 2.1161 0.0833 1.115 0.4738 1.6174 0.4786 0.18 0.13
rameter. This is the standard Weibull curve modified to be constrained to have a minimum and maximum value (it typically varies between zero and one). Statistics The different effect size calculations for Table 1 depend on the statistic used as described in detail in Cohen (1988) (Thompson, 2006; Faul et al. 2007; Olivier and Bell, 2013). For t tests, we used Cohen’s d (Cohen, 1988; Faul et al. 2007). For ANOVA measures, we used Cohen’s f, first calculating 2 as the ratio of variance explained SSeffect/ SStotal (Thompson, 2006) and then using the standard conversion to f (Cohen, 1988; Thompson, 2006): f⫽
冪
2 . 1 ⫺ 2
Similarly, for regression analysis (which used an F test for significance), we defined the effect using the same measure, but with the regression coefficient r taking the place of (Cohen, 1992; Faul et al. 2007). For chi-squared tests, we used Cohen’s statistic (Cohen, 1992; Faul et al. 2007; Olivier and Bell, 2013): K
⫽
兺 共p i⫽1
oi
⫺ pei兲2 , pei
where pei is the expected frequency of observations, under the null hypothesis, to fall in group i, and poi is the eNeuro.org
204 New Research
80 75 2
3
4 5 6 Sessions
7
8
E
80 60 40
S1 V1
20 0
1
2 3
C
4 5 6 Sessions
7
8
Percent Correct
60 4IR-S1 4IR-V1 1IR-S1
40 20
5
10 15 20 25 Sessions
30 35
80 75 100
200 300 400 IR Only Trials
500
60 Naïve-S1 Naïve-V1 VisCue-S1
40 20 0
1
2
3 4 Sessions
5
80 60 40
S1 V1
20 0
F
80
85
80
100
9
100
95 90
70
9
Percent Correct
1
100 Percent Correct
85
100
Percent Correct
B
Percent Correct
95 90
70
G
D 100
100
300 500 700 IR Only Trials
900
100
Percent Correct
Percent Correct
A 100
7 of 17
90 80
4IR-S1 4IR-V1 1IR-S1
70
90
60
45 Angle [°]
30
Arena Layout
Figure 2. Discriminating IR light with visual cortex. A, Learning curve for IR-discrimination task using V1 (n ⫽ 6 animals). Note animals typically surpassed criterion (85% correct) within the first day. B, Direct comparison of learning curve for the same task with V1 implants (blue) and S1 implants (green; n ⫽ 15 for S1 implants). C, Same information as in B, but with learning curve superimposed when an animal has just a single IR sensor on its head (single-sensor learning data are from a previous study [Thomson et al. 2013]; n ⫽ 4 animals). D, Trial-based moving average analysis of performance in IR discrimination task. The curve shows percentage correct as a function of trial number (moving average of 20 trials) from same data shown in A. E, Same analysis as in C, but with S1-implanted animals shown in green for direct comparison. F, Performance in IR-discrimination task as a function of angle between the ports (see Fig. 1A). Includes data from V1 and S1 implanted animals with four IR sensors, as well as data from S1-implanted animals with a single IR sensor for comparison (blue, green, and red lines respectively with mean ⫾ SEM percentage correct). G, Performance of naive V1- and S1-implanted animals (mean ⫾ SEM percentage correct), that were not pretrained on a visual discrimination task (solid lines). Data from S1-implanted animals that were pretrained on a visual discrimination task are included for comparison. There is no significant difference, among any of the three groups, in the number of sessions it takes to reach 85% correct in the task (p ⬎ 0.05, two-tailed t test).
observed frequency in group i. All calculations of effect size were conducted in Matlab.
Results To examine the ability of primary visual cortex (V1) to absorb a new and otherwise invisible source of electromagnetic signals, we used ICMS to deliver information generated by IR sources directly to V1. Rats first learned to discriminate visual cues in a circular behavioral chamber with four light sources (see Methods; Fig. 1A). We then implanted them with an IR prosthesis and trained them on an IR discrimination task in which they had to correctly identify which of four ports in the behavioral chamber had an active IR source (Fig. 1A). Infrared information was coupled to V1 via four IR sensors that were arranged around the circumference of the rat’s head, so they were able to perceive a full 360° view of the circular arena (see Methods; Fig. 1B). We implanted November/December 2017, 4(6) e0262-17.2017
stimulating electrodes bilaterally in V1 and projected information from IR sensors to the cortex in a topographically natural manner, such that the information from the left IR environment was projected to the V1 of the right hemisphere (see Methods; Fig. 1B; Hartmann et al. 2016). Surprisingly, when IR information was delivered to V1, rats learned to discriminate IR sources extremely quickly, often within their first day of training (Fig. 2A). This was the case in four of six rats. Quantitatively, the V1-implanted animals surpassed 85% correct in the IR discrimination task in 1.3 ⫾ 0.2 d of training (n ⫽ 6 animals). This is significantly faster than S1-implanted animals, who took 3.8 ⫾ 0.4 d to surpass 85% correct (n ⫽ 15 rats; p ⫽ 0.003; two-tailed t test; Fig. 2B). Note that we have significantly more S1-implanted animals because we were able to include data from eight animals from a previous study (Hartmann et al. 2016). For comparison, Fig. 2C shows eNeuro.org
205 New Research
the learning curve when information from a single IR sensor was projected to S1 (data are from a previous study, Thomson et al. 2013). In this case, animals took on average 22.3 ⫾ 7.6 d to reach 85% correct on the task. Because the V1-implanted animals were clearly learning to perform the IR-discrimination task within the first session, we examined how many trials it took them to learn the task, using a 20-trial moving average (Fig. 2D). When IR information was delivered to V1, it took them on average 26.0 ⫾ 4.0 IR-only trials to reach 85% correct, versus 138.6 ⫾ 21.1 IR-only trials when projected to S1 (Fig. 2E), a significant difference (p ⫽ 0.0047, two-tailed t test). During training, such IR-only trials were interleaved with mixed trials in which IR light was followed by visible light, to facilitate learning the association between IR light and reward (see Methods). The total number of trials (both IR-only and mixed) required to learn the task was 174.33 ⫾ 22 for V1-implanted animals and 600.2 ⫾ 67 for S1implanted animals, a significant difference (p ⬍ 0.001; two-tailed t test). To examine the spatial acuity of the animals’ ability to discriminate IR lights, once they were above criterion at the initial task, we varied the angle between the ports (Fig. 1A). When the ports are placed closer together (i.e., is decreased), task difficulty is increased because multiple IR lights can activate the same IR sensor, thereby generating ambiguity in the stimulus. We found that performance in the task decreased as ports were moved together from 90˚ down to 30˚, which is the closest we could physically move the ports together in our setup (Fig. 2F). The decrease was small but statistically significant (p ⫽ 0.0015; ANOVA). Their performance as a function of angle was not significantly different from when implants were in S1 (p ⫽ 0.619; two-way ANOVA). As a control, we then tested whether V1-stimulated animals learned to discriminate IR sources faster because they were pretrained in a visual discrimination task that was structurally identical to the subsequent IR discrimination task. That is, did the V1 animals learn faster because they were already using V1 for a visual task, so their learning generalized more quickly when microstimulation was applied to V1 (perhaps because they were already attending to V1)? To test this, we trained six animals on the IR discrimination task that had no pretraining on a visual discrimination task, three implanted in V1 and three in S1. The trainer was blind to the location of implant (see Methods). The hypothesis was that if the V1-implanted animals learned faster because of visual pretraining, then when they did not undergo such training, their performance should converge on that observed in the S1stimulated animals. As can be seen in Fig. 2G, this is exactly what we observed. The naive V1-implanted animals took 4.0 ⫾ 0.57 d to learn the IR discrimination task, which was not significantly different from the S1implanted animals pretrained on a visual task, who took 3.8 ⫾ 0.4 d (p ⫽ 0.85; two-tailed t test). The naive S1implanted animals took 4.33 ⫾ 0.33 d to learn the task, which was also not significantly different from the naive V1 animals (p ⫽ 0.64, two-tailed t test). November/December 2017, 4(6) e0262-17.2017
8 of 17
Response to microstimulation in V1 To help understand how the brain is processing this new source of sensory information, we recorded from V1 neurons in animals that performed well above criterion at the task (16 sessions in four animals), using methods discussed extensively elsewhere (Hartmann et al. 2016). Recording during the task required that we change the stimulation protocol. Namely, instead of continuously updating the stimulating frequency every 50 ms, which produces continuous stimulus artifact, during recording we would stimulate the brain intermittently. That is, every 140 ms we would sample the IR levels, and stimulate for just 75 ms based on those levels, and then for the rest of that 140-ms period we would turn off the microstimulators, to allow for an artifact-free epoch for recording the neuronal response during that stimulus cycle. To visualize the neuronal response to the distributed IR stimulus, we compactly represented the IR-based microstimulation patterns using a stimulus population vector [see Methods; Fig. 3A, B shows how such vectors are constructed; see also Hartmann et al. (2016)]. This vector provides a compact geometric 2-D representation of the full set of four 75-ms ICMS bursts delivered to the cortex at a given time. For instance, if the two anterior stimulating channels are maximally activated, then the population vector points to the front. If the two left channels are active, it will point to the left, and so on. Fig. 3C shows the set of all population vectors delivered during one session in a V1-implanted rat. To show how the stimulus changed over time, contiguous stimuli within a trial are connected by a line, and later stimuli are indicated by darker circles in the figure. In a typical trial, the stimulating population vector started near the origin at the center of the graph, meaning that the microstimulating electrodes have very low levels of activation, and moved to the top (meaning that the two anterior microstimulating electrodes were activated). The mean of the first and last three stimulus population vectors are overlaid in red, showing that this trend of starting near the origin and moving anterior is representative. We used peristimulus time histograms (PSTHs) to quantify the mean responses of V1 neurons to ICMS during the IR-discrimination task. Four representative PSTHs depicting the response of V1 neurons to microstimulation during the IR task are shown in Fig. 3D. These PSTHs display the mean neuronal response to the first and last three microstimulation bursts in a trial, averaged over all trials in a session. Note that during the time of microstimulation, the PSTH is zero because of ICMS-induced stimulus artifact, so each PSTH shows responses to microstimulation after this brief artifact period. There is a gap between the first and last three ICMS bursts to indicate that there were a variable number of substimuli on each trial. Although they are useful portraits, such PSTHs provide a limited description of the full range of neuronal responses to ICMS as animals perform the IR discrimination task: the first and last three stimuli are incomplete samples of the full space of stimuli delivered to the brain during the task. To give a more complete picture of neuronal response, we used the IR receptive field (IR-RF), eNeuro.org
206 New Research
Population vector
T1
< 0, 0, 50, 50 >
< 0, -100 >
T2
< 0, 0, 50, 0 >
< -50, 50 >
T3
< 10, 50, 10, 0 >
< -50, 50 >
T4
< 50, 50, 0, 0 >
< 0, 100 >
T5 < 100, 100, 0, 0 >
B
200
< 0, 200 >
C
T5
2fmax
T4
Anteroposterior
4-vector
Anteroposterior
A
9 of 17
T3
0 T2 T1
fmax
0
−fmax
−200 −200
0
200
Mediolateral −2fmax −2fmax
−fmax
0
2fmax
-0.5
0
0.5
Time (s)
1
0
1.5
-0.5
0
0.5
Time (s)
1
1.5
Anteroposterior
spike count
spike count
Anteroposterior
2
0
sig009
0.5
4 sig024
N2
N4
N3
5 sig006
spike count
spike count
N1
0.05
0 -0.5
Mediolateral
E
0
0.5
Time (s)
1
1.5
0
-0.5
F
850
Anteroposterior
sig028
0. 1
0
0.5
Time (s)
1
90
120
Anteroposterior
Mediolateral
Anteroposterior
Mediolateral
V1
fmax
Mediolateral
D
60
150
425
0
Mediolateral
1.5
30
180
0 20
−425
−850 −850
40
210
80
240 −425
0
425
330
60 270
850
300
Mediolateral
S1
Anteroposterior
G
H
850
90
120
60
150
425
0
30
180
0 25
−425
−850 −850
210
240 −425
0
425
850
330
45 65 270
300
Mediolateral
November/December 2017, 4(6) e0262-17.2017
eNeuro.org
207 New Research
10 of 17
Figure 3. Neuronal responses to intracortical microstimulation in V1. A, Example of reduction of four-element full representation of electrical stimulation vector (four-element vector where each number represents frequency of microstimulation at a different location) down to two-dimensional population vector. There are five examples from five different time points T1–T5. B, Geometric representation of the population vectors from A. C, Stimulus population vectors from all trials in a session, with mean of first and last three substimuli overlaid in red. The black diamond outline is the convex hull of stimulus population vectors, showing the perimeter of the set of all possible vectors. Note that only a fraction of this set is actually presented in a given behavioral session. See text for more details. D, Neuronal data from four representative units. PSTHs show mean response to first three and last three microstimulation pulses, averaged over all trials in the session (note there are different axis limits for the different PSTHs). Associated with each PSTH is an IR-RF, a contour plot that shows the mean number of spikes in response to each stimulus population vector, with hot colors representing high firing rates. E, Distribution of all 240 IR-RF centers from all units recorded: note they tend to concentrate at the two medial corners of the stimulus space, although these are fairly rare events (C). F, Polar count histogram illustrates the medial, rather than anterior, concentration of IR-RF angular distribution. G, H, Same as E and F, but from data from S1 from a previous study (Hartmann et al. 2016). Note the anterior distribution of the IR-RF centers, much closer to the distribution of the stimuli.
which measures the mean spike count as a function of stimulus population vector, with the count calculated for all population vectors delivered during a session. Fig. 3D shows the IR-RFs for four neurons, adjacent to their PSTHs. For instance, unit 28 preferred IR stimuli that were presented to the left of the animal. This explains why its firing responses to the first and last three stimuli, shown in the PSTH, were relatively small. This lateralization of IR-RF centers was typical in V1, as can be seen in Fig. 3E, which shows a contour plot depicting the distribution of IR-RF peaks for all 240 V1 units recorded in this study. There was a pronounced trend for these V1 IR-RF peaks to be lateralized to the left and right sides. This is more clearly demonstrated by the polar count histogram of receptive field peak angular location in Fig. 3F. This is in sharp contrast to what was previously observed in S1, in which the IR-RF centers tended to closely match the stimulus statistics: that is, the S1 neurons had a strong preference for stimuli in the anteromedial sector of the stimulus space, as shown in Figs. 3G and 3H, which are reproduced from previous data (Hartmann et al. 2016). Although previous research demonstrated the existence of a highly significant 2D correlation between IR-RF peak location and stimulus location in S1 (Hartmann et al. 2016), in V1 there was no such match between IR-RF peak and stimulus location: the 2-D correlation between IR stimulus distribution (the set of all stimulus population vectors) and IR-RF center was weak and not significant ( ⫽ – 0.03; p ⫽ 0.7; F test). Integration of visual and IR information in V1 When using the IR prosthesis, both native (visual) and novel (IR) information streams were projected to the same primary sensory area. To determine whether rats could learn to simultaneously integrate visual and IR information within V1, we trained them on a visual-IR integration task. During this task, we presented multiple visual and IR lights in the behavior chamber simultaneously. To receive a reward, rats had to select the single port in which both visible and IR lights were active (see Methods). On each trial, four stimuli were presented in the standard cylindrical behavioral chamber (Fig. 4A): in the target port, both the visual and IR lights were activated simultaneously. In another, a visual distractor light was turned on, while a third port contained an IR distractor light. The fourth port showed neither light. The location of each of these ports was selected randomly on each trial. November/December 2017, 4(6) e0262-17.2017
In preliminary testing, one cohort of three animals had trouble learning this task with high accuracy, often going to the first port that they noticed with an IR light activated. This seemed to be partly because they were being overstimulated in the relatively small chamber in which they were initially trained. To test this hypothesis, we trained a second cohort of three animals in a larger-diameter behavior chamber (20-inch or ⬃0.5-m diameter), and significantly decreased the sensitivity of the IR sensors, so when a trial was initiated in the middle of the chamber, the IR sensors were not as likely to trigger ICMS at the start of the trial. Further, we optimized the transform from IR sensor output to ICMS frequency so that no frequencies were oversampled (see Methods; Fig. 1D). After such optimizations were in place, animals readily learned to perform the visual-IR integration task, reaching 85% correct within 3.67 ⫾ 0.67 sessions (n ⫽ 3 animals; Fig. 4B). Despite this performance, we noticed that some of the animals tended to make more errors toward the IR or visual distractor and hypothesized that they might be biased by microstimulation frequency. Hence, we systematically varied the maximum frequency of stimulation until we found the frequency that optimized percentage correct in the task (see Methods). Before frequency optimization, the best performance over five sessions was 82.7 ⫾ 1.0%, and after optimizing frequency, the best performance over five sessions was 93.5 ⫾ 1.4% (Fig. 4C). During initial training, we stimulated at a maximum frequency of 300 Hz and found that the optimal frequency, that minimized errors, was 133 ⫾ 33 Hz. We next tested the hypothesis that processing both IR and visual inputs in V1 would interfere with performance of the visual-IR integration task. In particular, we implanted a group of three animals with stimulating electrodes in S1 and trained them in the visual-IR integration task. In these animals, the IR information would be processed in S1, while the visual information would go to V1, so direct sensory interference between the two types of information would not occur. Yet, animals would still have to directly compare both sources of sensory information—IR light coming through S1 and visible light through V1—to solve the task properly. Hence, if the S1-implanted animals performed better on the visual-IR integration task, this would provide an indirect measure of sensory interference within V1 during the sensory integration task, while any residual errors could be interpreted as a measure of intrinsic task difficulty. eNeuro.org
208 New Research
A
B 100 IR distractor
95 Percent Correct [%]
Visual distractor
No light
90 85 80 75 70
Target: IR+Visible light
65 1
3
5 n-3 Session Number
n
D 100
C 100
Percent Correct [%]
90 Percent Correct [%]
11 of 17
80 70 V1 S1
60
90
80
70
V1 S1
60 1
2
3 4 5 Session Number
6
100
300
500 Trial Number
700
Figure 4. Visual-IR integration task. A, Schematic of the task. In a cylindrical chamber with the same ports shown in Fig. 1A, four stimuli are randomly assigned to the ports. The target stimulus is the one with both visual and IR lights on. The IR distractor has only an IR light on, the visual distractor only a visual light, and there is one port with no light activated. B, Learning curve from three animals trained on the visual-IR integration task (implants in V1). The discontinuous right portion of the curve shows the last four sessions using an optimized frequency (see text). C, Comparison of learning curves between V1- and S1-implanted rats in the visual-IR integration task (n ⫽ 3 animals for each group). D, Same data as in C, but with trial-based moving average (moving window average taken with 20 trials).
Surprisingly, the S1-implanted animals reached 85% correct within 3.67 ⫾ 0.88 sessions (Fig. 4C), which was not significantly different from the V1-implanted animals (p ⬎ 0.5, two-tailed t test). When we analyzed the results by trial, we saw similar results (Fig. 4D). The S1-implanted animals reached 85% correct within 139 ⫾ 51 trials, and the V1 implanted animals reached 85% correct within 103 ⫾ 45 trials, and these learning rates were not statistically different (p ⫽ 0.626; two-tailed t test). These results suggest that ICMS in V1 does not appreciably impair the ability of V1 to process incoming visual stimuli while simultaneously receiving native visual information. Moreover, it demonstrates that rats can combine invisible (IR) and visible light, delivered to two different primary cortical areas (S1 and V1), as efficiently as when both sensory signals converge in their V1, to solve this difficult sensory integration task. Effect of prosthesis use on native tactile processing In a final set of experiments, we further explored the consequences, for the native sensory modality, of projecting new information to a primary sensory area. Previous research has suggested that sensory prosthetic systems November/December 2017, 4(6) e0262-17.2017
that employ ICMS might compromise the original function of the cortical region to which information is projected (Ni and Maunsell, 2010). Here, we measured whether rats in which IR information was projected to the whisker representation in S1 showed any decline in their ability to perform a whisker-dependent tactile discrimination task. Toward this end, we trained animals in two behavioral tasks each day: the stimulation-based IR discrimination task and a whisker-dependent aperture-width discrimination task (see Methods; Fig. 5A; Krupa et al. 2001, 2004; Thomson et al. 2014). Briefly, in the aperture-width discrimination task, a variable-width aperture is moved to one of two widths (54 or 78 mm), and after sampling the aperture with its large facial whiskers, the rat must select the correct associated reward port to receive reward (Fig. 5A). We trained eight rats on this two-width discrimination task, and trained six of these rats on a multiwidth version of the task in which we randomly selected from more than just two widths (either 12 or 14). Widths ⬎66 mm were considered wide, and those narrower than 66 mm were considered narrow. This allowed us to measure the bias (preference for one width over another), sensitivity (maximum slope of the eNeuro.org
209 New Research
B
Central Nose Poke (CNP)
Movable bars
C
1
0.8
Slope = Sensitivity
0.8 0.6 P(Left)
P(Left)
0.6
Stimulus chamber
0.4 0.2
Reward chamber
D
Baseline
Stimulation
0 78 76 74 72 70 67 65 62 60 58 56 54 Aperture Width [mm]
Whisker Clip
Percent Correct [%]
100
60 40 Stimulated Control
20 0
300
500
700
900 Trials
end-700
end-300
end
Stimulated
8
Control 6 4 2 0 -2
G 6
0.15
4
0.1
2 0 -2
-4 -6
100
0 78 76 74 72 70 67 65 62 60 58 56 54 Aperture Width [mm]
F
E
80
0.4 0.2
Shift = Bias
Percent Correct Change
Right reward
Percent Correct Change
Left reward
1
Sensitivity Change
A
12 of 17
ΔStim
ΔClip
0.05 0 -0.05 -0.1
-4
ΔStim
ΔClip
-0.15
ΔStim
ΔClip
Figure 5. Effects of S1 prosthesis on whisker discrimination. A, Aperture-width discrimination task. After the movable bars making up aperture move to the desired position, the rat enters the stimulus chamber, sweeps its whiskers across the aperture (movable bars controlled by the computer), and activates the central nose poke (CNP). The animal then retreats into the stimulus chamber and pokes in the right or left reward port depending on the aperture width. B, Psychometric curve from a single session in the 14-width aperture-width discrimination task. The points are the mean proportion of left responses at each width; the red line is the best minimum least-squares fit to the points, fitted using a Weibull function. C, As in B, but in an animal with worse performance (lower maximum slope of the curve and lower peak performance at each end of the spectrum). D, Performance of stimulated and matched control animal in the whisker discrimination task during different stages of the experiment. Before stimulation (Baseline), while learning the IR discrimination task via ICMS (Stimulation), and after whiskers were clipped (Clipped). See text for details. E, Mean ⫾ SEM change in performance after the two transitions illustrated in D. ⌬Stim, transition from baseline to stimulation; ⌬Clip, transition from normal stimulation to stimulation with whiskers clipped. This analysis is restricted only to 54/78-mm widths, and includes eight animals (four stimulated/four control for the ⌬stim transition) and seven animals (four stimulated/three control) for the ⌬clip transition (one control animal stopped performing the task after whisker clipping). F, Same as E, but for the full multiple-aperture with discrimination task (either 12 or 14 widths). This includes six animals for the first transition and five animals for the second (as before, one control animal stopped behaving after whisker clipping). Percentage correct is calculated as the integral under the best Weibull fit to the psychometric curve (see Methods). G, Change in behavioral sensitivity, or the measure of the maximum slope of the best fit curves to the psychometric data, with each transition in the task. Lower slope means less behavioral sensitivity.
psychometric curve), and standard overall percentage correct on the task over the full range of widths presented (Fig. 5B, C). Once rats were trained on the two tasks, we implanted them with stimulating electrodes in S1 as before. We trained four rats to perform stimulation-based IR discrimination (the stimulated group) and four to perform the equivalent visual discrimination task as described above (the control group). This allowed us to track tactile discrimination performance in the two groups, each performing two similar tasks a day, with only one group receiving microstimulation in S1. Fig. 5D shows the raw performance (percentage correct) in the multiwidth discrimination task in three key phases, in both the stimulated and control groups. The first, baseline, phase shows the percentage correct in both groups on the initial four-choice visual discrimination November/December 2017, 4(6) e0262-17.2017
task, before any S1 microstimulation. The second, stimulation, phase shows performance on the aperture-width discrimination task after the stimulated group began training on the IR discrimination task. The third and final whisker clip phase was when all whiskers, except those corresponding to the barrels being stimulated, were clipped on each side of the face. It is important to note that at each phase, each rat was performing two tasks each day: the whisker discrimination task and either the IR discrimination task (stimulated group) or visual discrimination task (control group). Fig. 5D plots their performance only in the aperture-width discrimination task. The third, whisker clipping, phase was crucial because it is known that rats can perform the aperture-width discrimination task above chance with just one whisker remaining on each side of the face (Krupa et al. 2001). Hence, to control for the possibility that the stimulated eNeuro.org
210 New Research
group could perform the task using the whiskers that were outside of the region being stimulated in S1, we clipped all of the facial whiskers that did not correspond to barrels that were stimulated by the prosthesis (stimulated group) or simply whiskers outside of the region of the implant (control group). After determining the whiskers that evoked activity at the site of S1 stimulation, we clipped the remaining whiskers on the face of the rat. This meant we clipped ⬃20 whiskers, leaving only eight whiskers on each side of the face. To determine whether ICMS, or trimming whiskers, altered the performance of the stimulated relative to control animals in the aperture-width discrimination task, we performed a 2 ⫻ 2 ANOVA [factor 1: animal group (control/ stimulated); factor 2: treatment (stimulation/whisker clipping)]. Crucially, there was no significant difference in performance change between the two groups of animals using any of the measures, but there was a significant effect of treatment using each measure (␣ ⫽ 0.05: Figs. 5E–G; Table 1). For instance, Fig. 5E shows the mean response change for all animals in the 54 vs. 78 whisker discrimination task, both when they were switched from the baseline to the stimulation phase (⌬Stim) and when they were switched from the stimulation phase to the whisker clip phase (⌬Clip). Fig. 5F shows a similar plot, but for percentage correct calculated using data from the multiwidth version of the task. Fig. 5G examines changes in sensitivity (the slope of the psychometric curve). Using all of these measures, there was no significant difference between groups. This suggests that microstimulation did not appreciably impair the ability of the stimulated rats to use their whiskers in the aperture-width discrimination task. However, there were significant effects of treatment in each case (Table 1). Because the changes were similar in the two groups between phases of the task, to examine the sources of the differences we lumped the data from the stimulated and control groups together and looked for potential effects of each change in condition in a post hoc analysis. Surprisingly, percentage correct actually increased between baseline and stimulation phases, suggesting an effect of continued learning (Fig. 5E, F). This was significant in the eight animals trained on the twowidth task (p ⫽ 0.01; two-tailed t test), but not significant in the multiwidth task (p ⫽ 0.153). Not surprisingly, the performance drop was significant after whisker clipping (p ⫽ 0.001 for the 54/78 task; p ⫽ 0.009 for the multiwidth task; two-tailed t test). The animals’ sensitivity increased during the stimulation period (a significant change: p ⫽ 0.01; two-tailed t test), and dropped after whisker clipping, although this drop was not significant (p ⫽ 0.34). Surprisingly, using all of these measures, the performance drop after whisker clipping was actually less for the stimulated group (Fig. 5E–G), but this trend was not significant (␣ ⫽ 0.05). We next examined neuronal activity in S1 during performance of the aperture-width discrimination task in the stimulated and control groups. It is known that S1 neurons in animals trained to use S1 in an ICMS-based IR discrimination task maintain their ability to respond to November/December 2017, 4(6) e0262-17.2017
13 of 17
whisker deflection outside of a task context (Thomson et al. 2013; Hartmann et al. 2016). Here, we recorded from populations of S1 neurons as two stimulated and two control animals performed the two-width aperture-width discrimination task. Fig. 6A, B shows sample PSTHs and the set of all PSTHs from all 104 S1 single units from the control animals, respectively. Fig. 6C, D shows the same for the 117 S1 units in the stimulated animals. Overall 87% of neurons in the control group and 86% of those in the stimulated group displayed a significant response at some point in the trial. However, there were some quantitative differences in the two groups of animals. As has been reported extensively before, S1 neuronal activity during the aperture-width discrimination task can be analyzed in different time epochs around the instant of whisker contact with the port edges (Krupa et al. 2004; Pais-Vieira et al. 2013; Thomson et al. 2014). For instance, anticipatory activity in S1 has been commonly observed during the 400 ms that precedes the contact between the whiskers and the aperture (Krupa et al. 2004; Wiest et al. 2010; Pais-Vieira et al. 2013). We observed such anticipatory activity here as well, and the proportion of neurons showing such responses was larger in the stimulated group (0.71) than the control group (0.51; p ⫽ 0.009; chi-squared test for proportions). Further, the proportion of neurons responding to whisker deflections in the stimulus epoch (the period 350 ms after whisker contact with the aperture) was larger in the stimulated group than the control group (0.84 vs. 0.73), although this difference was not statistically significant (p ⫽ 0.053; chisquared test).
Discussion We demonstrated that primary sensory areas in adult rats exhibit a noteworthy ability to absorb novel sources of information without compromising preexisting sensory function. It seems that primary cortical sensory areas in mammals can rapidly integrate two independent sensory streams while exhibiting very little large-scale interference. As was pointed out previously (Hartmann et al. 2016), the fact that the novel information source was IR light was arbitrary; other information sources, such as X-rays or microwave radiation, could be used if an appropriate portable sensor were available. Comparative analysis: accelerated learning with area-specific pretraining This study is a first step in a comparative neuronal, behavioral, and functional analysis of the absorption of novel sources of sensory information in multiple primary sensory areas (Yang and Zador, 2012). Based on the anatomic and functional similarities in different sensory areas (Miller, 2016), and the fact that diverse areas can so readily acquire new functions in juveniles (Sur et al. 1988), we expected both V1 and S1 to exhibit the same properties when presented with IR information. This is the result we observed, as rats implanted with S1 and V1 prosthetic devices both learned at the same rate (Fig. 2F). However, when pretrained on a structurally similar visual discrimination task, significant differences emerged between the two populations: the V1-implanted eNeuro.org
211 New Research
B
0.4
40
0 0.6 0.3 0 0.8
60 0.2
100
D 20
Cell #
#spikes
0.4
80
0.4 0 0.8 0.4 0 0.3
0
Stimulated
40
- 0.2
60 80
0 0.2
- 0.4
100
0.1 0
t
20
0 0.4 0.2 0 0.25
Cell #
#spikes
0.2
C
s
d
Control
# spikes above mean
A
14 of 17
−0.8 −0.4 0
0.4
0.8
-0.4
Time (s)
0 Time (s)
E
0.4
Mean Response (spikes)
0.02
0
−0.02
−0.04
control stim
−0.06
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
Time (s)
Figure 6. Neuronal responses during the aperture-discrimination task. A, Four example PSTHs during the aperture-width discrimination task described in Fig. 5A, from control animals that were not stimulated. Bin widths are 50 ms. Reference lines show mean ⫾ SD during the baseline period (the period before – 400 ms). Time 0 is the time of stimulus onset, when the whiskers contact the aperture. B, Set of all 104 PSTHs from two control animals. d, door opens to start trial; s, stimulus onset (whiskers contact aperture); t, tone sounds indicating rat has reached end of stimulus chamber. Spike counts in B are calculated by subtracting the mean spike count during the baseline period. PSTHs below the gray line are those for which there was no significant response. C, D, Same as A and B but for the stimulated animals (n ⫽ 117 neurons). E, Mean ⫾ SEM PSTH for all neurons for control (red) and stimulated (black) groups.
animals learned the task significantly faster than those implanted in S1 (Fig. 2). Typically, after pretraining on a visual discrimination task, V1-implanted rats learned the corresponding IR discrimination task during their first day of training, several days and hundreds of trials ahead of their S1-implanted counterparts. When this visual pretraining was removed, this difference in learning rates between S1 and V1 learning rates disappeared (Fig. 2F), suggesting that there is no intrinsic difference in learning rates between the two populations. Cumulatively, these data support our initial hypothesis that there is an equipotentiality of plasticity across adult primary sensory areas: they are equally capable of absorbing a new source of distal sensory information and using it to perform a sensory discrimination task. November/December 2017, 4(6) e0262-17.2017
However, when pretrained on a structurally similar task that relies on a particular primary cortical area, there was a much more rapid transfer of learning when the new information was projected to that same primary cortical area (Fig. 2A, E). This suggests there exists a form of metaplasticity, in which the V1-implanted animals have learned to learn the task, transferring their earlier training to the new task. Such phenomena have been observed in the whisker system (Harris et al. 1999) and in cross-modal learning transfer (Over and Mackintosh, 1969). We do not yet know the underlying reason for this rapid transfer of learning, but speculate that during the visual task attention is already directed to V1, so the process of extracting and using the information from the prosthesis is facilitated in the V1-implanted animals. One way to test this hypothesis would be to pretrain the animals on a structurally eNeuro.org
212 New Research
similar tactile discrimination task and see whether the S1-implanted animals learn the IR-discrimination task faster than the V1-implanted animals. Originally, it seemed a natural hypothesis that the V1implanted animals learned faster because V1 is simply a more natural home for IR information: IR light is a distal electromagnetic cue, which V1 already processes, so there is no need to adapt new behavioral strategies for processing information as there is in S1. However, the fact that S1 and V1 learning rates converge when the animals are not pretrained on a visual discrimination task suggests there is no such intrinsic preference for IR information in V1 compared with S1. Multimodal integration within and across primary sensory areas With the visual-IR integration task used in this study, information from both the visual and IR modalities had to be integrated and compared for rats to successfully complete a trial. If the animal solely focused on one type of information, they would incorrectly approach a distractor port (Fig. 3A). This is typically what happened at first, but within four sessions, subjects learned to approach the correct port (Fig. 3B). In the future, we will merge such multimodal stimuli in different combinations and intensities to more thoroughly quantify how animals integrate IR and visual information, in particular to measure how closely they approach the theoretical Bayesian optimum, as when variability is added to one of the cues, making it less reliable than the other (Deneve and Pouget, 2004; Dadarlat et al. 2015). Surprisingly, there was no significant difference in performance in the visual-IR integration task when the IR information was projected to S1 or V1 (Fig. 3C). We had expected the S1-implanted animals to learn faster because there would be no intra-area sensory interference between the visual and IR modalities. Instead, our findings suggest that it may not matter whether the two information sources are projected to the same or different cortical areas. The limits of such multimodal sensory augmentation are unknown. How many different types of information can be projected to one sensory area before the animal becomes confused? This question could have direct clinical relevance: in designing closed-loop feedback systems for artificial limbs, researchers aim to provide subjects with multiple types of sensory information, such as temperature, pressure, and proprioception (Donati et al. 2016). Hence, it will be important to determine the practical limits of augmented brain function. Conservation of native sensory function It has been shown that when IR information is projected to the whisker region of S1, S1 neurons still show robust response to whisker deflections (Thomson et al. 2013; Hartmann et al. 2016). However, it was not known whether this cortical sensory neuroprosthesis would impair the native sensory function of the target cortical area. Would S1 become effectively hijacked by ICMS, and less capable of processing tactile information (Griffin et al. 2011)? There is evidence from monkeys that learning to perform ICMSNovember/December 2017, 4(6) e0262-17.2017
15 of 17
based threshold detection tasks in V1 impairs performance on visual detection tasks (Ni and Maunsell, 2010). Contrary to such findings, when we trained rats with the ICMS-based prosthesis on a whisker-based tactile discrimination task, we found no decline in the animalsâ&#x20AC;&#x2122; tactile discrimination (Fig. 5). Therefore, our results suggest that projecting novel sensory information to a primary sensory cortical area does not necessarily compromise the ability of the animal to use that area for sensory discrimination. There are multiple differences between our paradigm and those in the previous study that could explain the different results. The previous research used a detection task in fixated, head-fixed animals and purposely limited the cue to a small region of visual space (Ni and Maunsell, 2010). By comparison, we used a discrimination task in which freely moving animals swept multiple whiskers across the target stimulus: this stimulus is known to activate a broadly distributed population of neurons across the trigeminal system (Krupa et al. 2004; Ferezou et al. 2006; Pais-Vieira et al. 2013, 2015). Similarly, the IR prosthesis was used by freely moving animals, with information distributed to four locations, in animals trained to forage for IR information in their environment over extended time scales, rather than a single 250-ms microstimulation train localized to a single location, as in the previous study (Ni and Maunsell, 2010). Finally, detection and discrimination tasks are quite different tasks, likely recruiting different underlying brain states optimized to their particular goal (Sherman, 2001). For instance, when performing a detection task, neurons in the cortex and thalamus may tend to fire in bursts, which is optimal for indicating that an event has occurred, but not for tracking its fine-grained features. When in discrimination mode, thalamic neurons exhibit more tonic background firing rates, and may track features of the environment at a finer grain and respond with lower magnitudes than in detection mode (Fanselow and Nicolelis, 1999; Adibi and Arabzadeh, 2011; Ollerenshaw et al. 2014). Overall, although ICMS use may selectively increase localized detection thresholds (Ni and Maunsell, 2010), this does not mean that it harms performance in discrimination tasks in freely moving animals actively extracting sensory information from their environment. In conclusion, our findings demonstrate that the mammalian brain remains extremely sensitive to the statistical structure of the inputs it receives throughout adulthood. This has long been clear from studies of newborns (Frost and Metin, 1985; Sur et al. 1988) and deafferented adults (Kaas et al. 1983; Kossut et al. 1988; Nicolelis et al. 1993). The current study shows that it is possible to superimpose, without any deafferentation, multiple streams of information simultaneously onto the same primary cortical area, whether it be S1 or V1. In this context, the present study strongly suggests that the upper limits of plasticity of a primary sensory cortical area, as well as its ability to adaptively absorb and use multiple sources of information, are much greater than previously anticipated. This is auspicious for the development of sensory cortical prosthetic systems, in particular those that might require the use of sensory eNeuro.org
213 New Research
substitution systems, such as those in which visual information is projected to the somatosensory system (Bach-yRita et al. 1969).
References Adibi M, Arabzadeh E (2011) A comparison of neuronal and behavioral detection and discrimination performances in rat whisker system. J Neurophysiol 105:356 –365. CrossRef Bach-y-Rita P, Collins CC, Saunders FA, White B, Scadden L (1969) Vision substitution by tactile image projection. Nature 221:963– 964. CrossRef Cohen J (1988) Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, N.J.: L. Erlbaum Associates. Cohen J (1992) A power primer. Psychol Bull 112:155–159. Medline Dadarlat MC, O’Doherty JE, Sabes PN (2015) A learning-based approach to artificial sensory feedback leads to optimal integration. Nat Neurosci 18:138 –144. CrossRef Deneve S, Pouget A (2004) Bayesian multisensory integration and cross-modal spatial links. J Physiol Paris 98:249 –258. CrossRef Donati AR, Shokur S, Morya E, Campos DS, Moioli RC, Gitti CM, Augusto PB, Tripodi S, Pires CG, Pereira GA, Brasil FL, Gallo S, Lin AA, Takigami AK, Aratanha MA, Joshi S, Bleuler H, Cheng G, Rudolph A, Nicolelis MA (2016) Long-term training with a brainmachine interface-based gait protocol induces partial neurological recovery in paraplegic patients. Sci Rep 6:30383. CrossRef Fanselow EE, Nicolelis MA (1999) Behavioral modulation of tactile responses in the rat somatosensory system. J Neurosci 19:7603– 7616. Medline Faul F, Erdfelder E, Lang AG, Buchner A (2007) GⴱPower 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 39:175–191. Medline Ferezou I, Bolea S, Petersen CC (2006) Visualizing the cortical representation of whisker touch: voltage-sensitive dye imaging in freely moving mice. Neuron 50:617–629. CrossRef Frost DO, Metin C (1985) Induction of functional retinal projections to the somatosensory system. Nature 317:162–164. CrossRef Georgopoulos AP, Schwartz AB, Kettner RE (1986) Neuronal population coding of movement direction. Science 233:1416 –1419. CrossRef Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory? Trends Cogn Sci 10:278 –285. CrossRef Medline Goddard GV (1967) Development of epileptic seizures through brain stimulation at low intensity. Nature 214:1020 –1201. CrossRef Godde B, Leonhardt R, Cords SM, Dinse HR (2002) Plasticity of orientation preference maps in the visual cortex of adult cats. Proc Natl Acad Sci U S A 99:6352–6357. CrossRef Griffin DM, Hudson HM, Belhaj-Saif A, Cheney PD (2011) Hijacking cortical motor output with repetitive microstimulation. J Neurosci 31:13088 –13096. CrossRef Habenschuss S, Jonke Z, Maass W (2013) Stochastic computations in cortical microcircuit models. PLoS Comput Biol 9:e1003311. CrossRef Medline Haeusler S, Maass W (2007) A statistical analysis of informationprocessing properties of lamina-specific cortical microcircuit models. Cereb Cortex 17:149 –162. CrossRef Hanson TL, Omarsson B, O’Doherty JE, Peikon ID, Lebedev MA, Nicolelis MA (2012) High-side digitally current controlled biphasic bipolar microstimulator. IEEE Trans Neural Syst Rehabil Eng 20: 331–340. CrossRef Harris JA, Petersen RS, Diamond ME (1999) Distribution of tactile learning and its neural basis. Proc Natl Acad Sci U S A 96:7587– 7591. Medline Harris KD, Shepherd GM (2015) The neocortical circuit: themes and variations. Nat Neurosci 18:170 –181. doi:10.1038/nn.3917 CrossRef Medline Hartmann K, Thomson EE, Zea I, Yun R, Mullen P, Canarick J, Huh A, Nicolelis MA (2016) Embedding a panoramic representation of infrared light in the adult rat somatosensory cortex through a sensory neuroprosthesis. J Neurosci 36:2406 –2424. CrossRef November/December 2017, 4(6) e0262-17.2017
16 of 17
Kaas JH, Merzenich MM, Killackey HP (1983) The reorganization of somatosensory cortex following peripheral nerve damage in adult and developing mammals. Annu Rev Neurosci 6:325–356. CrossRef Klemen J, Chambers CD (2012) Current perspectives and methods in studying neural mechanisms of multisensory interactions. Neurosci Biobehav Rev 36:111–133. CrossRef Kossut M, Hand PJ, Greenberg J, Hand CL (1988) Single vibrissal cortical column in SI cortex of rat and its alterations in neonatal and adult vibrissa-deafferented animals: a quantitative 2DG study. J Neurophysiol 60:829 –852. Kouh M, Poggio T (2008) A canonical neural circuit for cortical nonlinear operations. Neural Comput 20:1427–1451. CrossRef Krupa DJ, Matell MS, Brisben AJ, Oliveira LM, Nicolelis MA (2001) Behavioral properties of the trigeminal somatosensory system in rats performing whisker-dependent tactile discriminations. J Neurosci 21:5752–5763. Krupa DJ, Wiest MC, Shuler MG, Laubach M, Nicolelis MA (2004) Layer-specific somatosensory cortical activation during active tactile discrimination. Science 304:1989 –1992. CrossRef Lledo PM, Alonso M, Grubb MS (2006) Adult neurogenesis and functional plasticity in neuronal circuits. Nat Rev Neurosci 7:179 – 193. CrossRef Medline Macaluso E (2006) Multisensory processing in sensory-specific cortical areas. Neuroscientist 12:327–338. CrossRef Medline Maldonado PE, Gerstein GL (1996) Reorganization in the auditory cortex of the rat induced by intracortical microstimulation: a multiple single-unit study. Exp Brain Res 112:420 –430. CrossRef Miller KD (2016) Canonical computations of cerebral cortex. Curr Opin Neurobiol 37:75–84. CrossRef Medline Ni AM, Maunsell JH (2010) Microstimulation reveals limits in detecting different signals from a local cortical region. Curr Biol 20:824 – 828. CrossRef Nicolelis MA, Lin RC, Woodward DJ, Chapin JK (1993) Induction of immediate spatiotemporal changes in thalamic networks by peripheral block of ascending cutaneous information. Nature 361: 533–536. CrossRef Norimoto H, Ikegaya Y (2015) Visual cortical prosthesis with a geomagnetic compass restores spatial navigation in blind rats. Curr Biol CrossRef Olivier J, Bell ML (2013) Effect sizes for 2x2 contingency tables. PLoS One 8:e58777. CrossRef Medline Ollerenshaw DR, Zheng HJ, Millard DC, Wang Q, Stanley GB (2014) The adaptive trade-off between detection and discrimination in cortical representations and behavior. Neuron 81:1152–1164. CrossRef Over R, Mackintosh NJ (1969) Cross-modal transfer of intensity discrimination by rats. Nature 224:918 –919. CrossRef Pais-Vieira M, Kunicki C, Tseng PH, Martin J, Lebedev M, Nicolelis MA (2015) Cortical and thalamic contributions to response dynamics across layers of the primary somatosensory cortex during tactile discrimination. J Neurophysiol 114:1652–1676. CrossRef Pais-Vieira M, Lebedev MA, Wiest MC, Nicolelis MA (2013) Simultaneous top-down modulation of the primary somatosensory cortex and thalamic nuclei during active tactile discrimination. J Neurosci 33:4076 –4093. CrossRef Paxinos G, Watson C (2007) The Rat Brain in Stereotaxic Coordinates (6th ed.). Amsterdam; Boston: Academic Press/Elsevier. Recanzone GH, Merzenich MM, Dinse HR (1992) Expansion of the cortical representation of a specific skin field in primary somatosensory cortex by intracortical microstimulation. Cereb Cortex 2:181–196. CrossRef Sherman SM (2001) Tonic and burst firing: dual modes of thalamocortical relay. Trends Neurosci 24:122–126. CrossRef Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9:255–266. CrossRef Sur M, Garraghty PE, Roe AW (1988) Experimentally induced visual projections into auditory thalamus and cortex. Science 242:1437– 1441. CrossRef Thompson B (2006) Foundations of Behavioral Statistics: An Insightbased Approach. New York: Guilford Press. eNeuro.org
214 New Research Thomson E, Lou J, Sylvester K, McDonough A, Tica S, Nicolelis MA (2014) Basal forebrain dynamics during a tactile discrimination task. J Neurophysiol 112:1179 –1191. CrossRef Thomson EE, Carra R, Nicolelis MA (2013) Perceiving invisible light through a somatosensory cortical prosthesis. Nat Commun 4:1482. CrossRef von Melchner L, Pallas SL, Sur M (2000) Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature 404:871–876. CrossRef
November/December 2017, 4(6) e0262-17.2017
17 of 17
Wiest MC, Thomson E, Pantoja J, Nicolelis MA (2010) Changes in S1 neural responses during tactile discrimination learning. J Neurophysiol 104:300 –312. CrossRef Wong-Riley M (1979) Changes in the visual system of monocularly sutured or enucleated cats demonstrable with cytochrome oxidase histochemistry. Brain Res 171:11–28. Medline Yang Y, Zador AM (2012) Differences in sensitivity to neural timing among cortical areas. J Neurosci 32:15142–15147. CrossRef Medline
eNeuro.org
215
Brain-to-Brain Interfaces and Brainets
216
A Brain-to-Brain Interface for Real-Time Sharing of Sensorimotor Information SUBJECT AREAS: SOMATOSENSORY SYSTEM MOTOR CONTROL NEUROSCIENCE WHISKER SYSTEM
Received 20 December 2012 Accepted 8 February 2013 Published 28 February 2013
Correspondence and requests for materials should be addressed to M.A.L.N. (nicoleli@ neuro.duke.edu)
* Current address: Neuroscience Research Institute, Peking University, Beijing, 100191.
Miguel Pais-Vieira1, Mikhail Lebedev1,4, Carolina Kunicki5, Jing Wang1* & Miguel A. L. Nicolelis1,2,3,4,5 1 Department of Neurobiology, Duke University, Durham, NC 27710, USA, 2Department of Biomedical Engineering, Duke University, Durham, NC 27710, USA, 3Department of Psychology and Neuroscience, Duke University, Durham, NC 27710, USA, 4 Duke Center for Neuroengineering, Duke University, Durham, NC 27710, USA, 5Edmond and Lily Safra International Institute for Neuroscience of Natal (ELS-IINN), RN 59066-060, Natal, Brazil.
A brain-to-brain interface (BTBI) enabled a real-time transfer of behaviorally meaningful sensorimotor information between the brains of two rats. In this BTBI, an ‘‘encoder’’ rat performed sensorimotor tasks that required it to select from two choices of tactile or visual stimuli. While the encoder rat performed the task, samples of its cortical activity were transmitted to matching cortical areas of a ‘‘decoder’’ rat using intracortical microstimulation (ICMS). The decoder rat learned to make similar behavioral selections, guided solely by the information provided by the encoder rat’s brain. These results demonstrated that a complex system was formed by coupling the animals’ brains, suggesting that BTBIs can enable dyads or networks of animal’s brains to exchange, process, and store information and, hence, serve as the basis for studies of novel types of social interaction and for biological computing devices.
I
n his seminal study on information transfer between biological organisms, Ralph Hartley wrote that ‘‘in any given communication the sender mentally selects a particular symbol and by some bodily motion, as his vocal mechanism, causes the receiver to be directed to that particular symbol’’1. Brain-machine interfaces (BMIs) have emerged as a new paradigm that allows brain-derived information to control artificial actuators2 and communicate the subject’s motor intention to the outside world without the interference of the subject’s body. For the past decade and a half, numerous studies have shown how brain-derived motor signals can be utilized to control the movements of a variety of mechanical, electronic and even virtual external devices3–6. Recently, intracortical microstimulation (ICMS) has been added to the classical BMI paradigm to allow artificial sensory feedback signals7,8, generated by these brain-controlled actuators, to be delivered back to the subject’s brain simultaneously with the extraction of cortical motor commands9,10. In the present study, we took the BMI approach to a new direction altogether and tested whether it could be employed to establish a new artificial communication channel between animals; one capable of transmitting behaviorally relevant sensorimotor information in real-time between two brains that, for all purposes, would from now on act together towards the fulfillment of a particular behavioral task. Previously, we have reported that specific motor11,12 and sensory parameters13,14 can be extracted from populations of cortical neurons using linear or nonlinear decoders in real-time. Here, we tested the hypothesis that a similar decoding performed by a ‘‘recipient brain’’ was sufficient to guide behavioral responses in sensorimotor tasks, therefore constituting a Brain-to-Brain Interface (BTBI)15 (Figure 1). To test this hypothesis, we conducted three experiments in which different patterns of cortical sensorimotor signals, coding a particular behavioral response, were recorded in one rat (heretofore named the ‘‘encoder’’ rat) and then transmitted directly to the brain of another animal (i.e. the ‘‘decoder’’ rat), via intra-cortical microstimulation (ICMS). All BTBI experiments described below were conducted in awake, behaving rats chronically implanted with cortical microelectrode arrays capable of both neuronal ensemble recordings and intracortical microstimulation16. We demonstrated that pairs of rats could cooperate through a BTBI to achieve a common behavioral goal.
Results In our training paradigm, animals learned basic elements of the tasks prior to participating in any BTBI experiments. First, prospective encoder rats were trained to respond to either tactile or visual stimuli until they reached 95% correct trials accuracy. Meanwhile, decoder rats were trained to become proficient while receiving ICMS as a stimulus. A train of ICMS pulses instructed the animal to select one of the levers/nose pokes, whereas a single ICMS pulse instructed a response to the other option. Decoder rats reached a 78.77% 6 2.1 correct trials SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
1
www.nature.com/scientificreports
Figure 1 | Experimental apparatus scheme of a BTBI for transferring cortical motor signals. Arrows represent the flow of information from the encoder to the decoder rat. In the motor task, the encoder rat has to identify a visual stimulus, signaled by an LED (red circle), and then press one of two levers to receive a small water reward. Meanwhile, M1 neural activity is recorded from the encoder rat and transmitted to the decoder animal, by comparing the pattern of the encoder’s M1 to a template trial (previously built with the firing rate average of a trial sample). The difference between the number of spikes in a given trial and the template trial is used to calculate a Zscore. The Zscore is then converted, through a sigmoid function centered on the mean of the template trial, into an ICMS pattern. Thus, the microstimulation patterns varied in real time, according to the number of spikes recorded from the encoder rat’s M1, on a trial by trial basis. Once microstimulation is delivered to the M1 cortex of the decoder rat, this animal has to select the same lever pressed by the encoder. Notice that the correct lever to press is cued only by the pattern of the decoder’s M1 microstimulation. If the decoder rat pressed the correct lever, both rats were rewarded. Thus, when the information transfer between the brains of the two rats was successful, the encoder rat received an additional reward that served as positive reinforcement.
performance level. After this preliminary training was completed, the animals were run in pairs, each one in a separate operant box. The next phase of training began with the encoder rat performing ,10 trials of the motor or tactile task, which were used to construct a cortical ensemble template, i.e. the mean cortical neuronal activity for one of the responses. The increased firing rate associated with the right lever press was selected as the parameter extracted from the neuronal ensemble in the motor task. The increased firing rate associated with the ‘‘Narrow’’ trials was selected as the parameter extracted from the neuronal ensemble in the tactile task. A BTBI session followed in which ICMS trains applied to the cortex of the decoder rat reflected the difference between the template and single-trial neuronal ensemble rates produced by a sample of the encoder rat’s M1 or S1 activity. ICMS duration (i.e. number of pulses delivered) was proportional to the difference between the sampled neuronal ensemble firing rate recorded during a given trial and the template normalized by the standard deviation. The time window for the analysis of neuronal activity and ICMS parameters was adjusted in each recording session to maximize the directional signal. The decoder rats reacted to ICMS patterns and not any other cues, as was evident from control experiments in which the performance of those rats dropped to chance level after the ICMS cable was disconnected from the stimulator. Furthermore, the encoder rat received feedback information describing the single trial performance of the decoder rat; each time the decoder rat responded correctly to the ICMS cue, the encoder rat received an additional reward (i.e. water). In experiment 1 (Figure 1), encoder rats (N 5 3) pressed one of two levers after an LED on top of the lever was turned on. While the rats did so, M1 neuronal activity was recorded, compared to the template and transformed into ICMS trains applied to M1 of the decoder rats (N 5 4) who performed the same lever press task. As would be expected, the encoder rats performed better (95.87% 6 1.83 SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
correct trials) (Figure 2 A) than the decoder rats (64.32 6 1.1%; range: 60 – 72% correct trials; Binomial test: P , 0.05 in all sessions) (Figure 2 A and B). Yet, the performance of the decoder animals was above chance and highly significant. Indeed, in some experiments the decoder rat’s performance using the BTBI was very close to the maximum performance obtained when ICMS was used alone in these animals (72% BTBI vs 78% ICMS alone, see above). The primary factor that influenced the decoder rat’s performance was the quality of spatial information extracted from the encoder rat’s M1. The performance was high if the chosen neuronal ensemble accurately encoded left versus right presses (Figure 2 C and Figure 3 A–D). The higher the deviation from the template and hence the larger the duration of the ICMS (i.e. number of pulses delivered), the better was the decoder rat’s performance (Figure 2 C and 3 B–D). For this first experiment, a total of 538 units and 110 multiunits were recorded from encoder rats. Sessions were comprised of 48.04 6 1.5 trials. The response latency of the encoder rats was 20.06 6 1.0 seconds, while decoder animals responded at 13.59 6 0.5 seconds. Note that this difference reflects only the effect of both rats working as a dyad (see below for comparison of latencies during training and testing). In addition to the neuronal transfer from the encoder to the decoder rats, feedback information, related to the decoder rats’ performance, was sent back to the encoder animal. This feedback provided an additional reward to the encoder rat every time the decoder rat performed a trial correctly. Under these conditions, the encoder rats’ response latency decreased after the decoder rat made an error (after correct response: 20.67 6 1.665 seconds and after an incorrect response 5 15.26 6 2.031 seconds; Mann Whitney U 5 13570; P , 0.0001). Furthermore, an analysis of the variation in Z-scores, demonstrated that the signal to noise ratio of the neural activity extracted from the encoder rat’s M1 increased after the decoder rat 2
217
www.nature.com/scientificreports
Figure 2 | Behavioral performance using a BTBI for transferring cortical motor signals. A) Performance of encoder and decoder animals during transfer of motor information via a BTBI. The performance of the encoder animals was above 90% in all but one session. The BTBI allowed the decoder animals to repeatedly perform significantly above chance. This performance immediately dropped to chance levels when the cable was disconnected but the system remained fully functional. B) The performance of the decoder animals across a session is presented with a moving average of 10 trials. C) The panel depicts the fraction of right lever presses after different microstimulation patterns were delivered to the decoder’s M1. As the number of microstimulation pulses increased, a higher fraction of right lever presses occurred. The microstimulation threshold for response in most animals was situated between 41 and 60 pulses.
committed an error (Chi Square 5 4.08, df51; P 5 0.0434). Thus, both the behavior and neuronal modulations of the encoder rat became dependent on the trial by trial behavioral performance of its dyad partner, the decoder rat. In experiment 2, we tested whether a BTBI could enable a realtime transfer of tactile information between a pair of rats’ brains SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
(Figure 4). Encoder rats (N 5 2) were trained to discriminate the diameter of an aperture width with their whiskers17. If the aperture was narrow, rats were required to nose poke on the left side of the chamber, otherwise they had to poke on the right side of the chamber. Decoder rats (N 5 5) were trained to poke on the left water port (narrow aperture) in the presence of ICMS and on the right water port (wide aperture) in the absence of ICMS. Similar to experiment 1, the difference between the S1 neuronal ensemble activity, recorded while the encoder rat examined the aperture with its whiskers in each trial, and an average template obtained previously, was utilized to create ICMS patterns applied to the decoder rat’s S1. We named these ICMS patterns virtual narrow and virtual wide. A total of 120 units and 223 multiunits were recorded in experiment 2. The BTBI accuracy for tactile information transfer was similar to that observed in experiment 1 (Figure 5 A–B). While encoder rats performed at 96.06 6 1.14% correct, decoder animals performed somewhat worse but significantly above chance (Percent correct: 62.34 6 0.59%, range 60 – 64.58%; Binomial test: P , 0.05 in all sessions) (Figure 5 A–B and Figure 6 A–D). In this second experiment, the response latency of encoder rats was 2.66 6 0.1 seconds, while in decoders the latency was 2.68 6 0.09 seconds. To further demonstrate that the accuracy of the decoder rats’ performance was based on the ICMS patterns, which in turn were triggered by larger number of spikes produced by S1 neuronal ensembles, we compared the fraction of Virtual Narrow choices with the number of ICMS pulses delivered to the decoder’s S1 cortex. Increases in the number of ICMS pulses delivered to the decoder’s S1 were associated with a higher fraction of Virtual Narrow choices (# 25 pulses: 0.3966 6 0.04476 correct; .25 pulses: 0.5433 6 0.02991 correct; Paired samples t-test 5 2.321, df 5 16, P 5 0.0338) (see Figure 5 C and Figure 6 A–D ). Since ICMS patterns were directly derived, through a transfer function, from the neural ensemble activity recorded from the encoder animal’s S1 cortex in each trial, this result demonstrates that the decoder rat’s correct choices relied on the accuracy of the ICMS pattern in reproducing the number of action potentials generated by the real tactile stimulus information presented to the encoder rat. Feedback information, providing an additional reward to the encoder rat every time the decoder rat performed a trial correctly, also induced changes in the neural activity of the encoder rat. The encoder’s latency of response was similar after correct and incorrect trials (After correct 5 2.6 6 0.1 secs; After incorrect 5 2.7 6 0.2 secs; Mann Whitney U 5 19790, P 5 0.49). However, similarly to the effects observed in experiment 1, the signal to noise ratio of neural activity in S1 also increased after an incorrect trial (Chi Square 5 4.2, df51; P 5 0.0404). It could be argued that the results reported here could have been obtained if prerecorded signals from encoder rats had been used to guide the behavior of the decoder rats. Qualitative and quantitative observation of the behavior of the animals reveals that this is not at all the case. In both motor and tactile BTBI sessions we observed drastic changes in the behavior of encoder and decoder rats as soon as they started to work as part of a dyad. Both encoder and decoder animals either made quick attempts to respond earlier or, conversely, they reduced their response rate or even stopped performing according to the dyad behavior. Thus, response latencies during motor BTBI sessions were largely increased for encoder animals (encoder training: 14.77 6 0.9 seconds; encoder BTBI session: 20.06 6 1.0 seconds; t 5 3.975, df 5 1170, P , 0.0001) and decreased in decoder rats (decoder training: 16.29 6 0.6 seconds; decoder BTBI sessions: 13.59 6 0.5 seconds; t 5 3.559, df 5 1636, P 5 0.0004). During the tactile BTBI sessions the responses latency was reduced in both encoder (encoder training: 5.40 6 0.6 seconds; encoder BTBI sessions: 2.66 6 0.1 seconds; Mann-Whitney U 5 13960, P , 0.0001) and decoder animals (decoder training: 4.632 6 0.6 seconds; decoder BTBI sessions: 2.68 6 0.09 seconds; t 5 4.638, df 5 12, P 5 0.0006) as they 3
218
www.nature.com/scientificreports
Figure 3 | Trial examples of a BTBI for transferring cortical motor signals. A) Examples of M1 neurons recorded while the encoder rat performed the task. Time 5 0 corresponds to the lever press. Very different patterns of increased and decreased activity were observed before and after the lever press, suggesting that multiple task parameters were encoded by this M1 ensemble. B) Sample of trial by trial choices of the rat dyad (encoder and decoder) during execution of the motor task. The encoder’s performance is depicted by a blue line, while a red line indicates the decoder’s choices in the same trials. In trials 4,7,11 and 13 the behavioral response of the decoder rat did not match the one of the encoder. The overall performance of the decoder rat in this session was 69% correct. C) The bars represent the number of encoder’s M1 neuronal spikes recorded during each trial. The neuronal ensemble used in this session encoded very accurately each of the behavioral responses. D) Number of ICMS pulses delivered to the decoder’s M1 that resulted from the comparison of each trial in C to the template.
started to work as a dyad. Therefore, the dyad performance depended on the nature of the task performed jointly by the animal pair. Likely the increased latencies observed in the motor task reflect the fact that pressing a lever is a learned artificial behavior, while the exploratory nose poking necessary for the tactile task is part of the rats’ natural behavioral repertoire. These overall changes in the dyad behavior, irrespective of their direction (e.g. increased or decreased latency), are a clear indicator that a fundamentally more complex system emerged from the operation of the BTBI; one which required considerable adaptation from the participant animals so that they could jointly perform the sensorimotor tasks. As the ICMS cues were delivered to primary cortical areas that are commonly involved in processing motor and somatosensory information in intact animals, we further asked how the decoder rat’s S1 cortex represented both real tactile stimuli, generated by mechanical stimulation of its own facial whiskers, and ICMS signals representing the encoder rat’s whisker stimulation, during operation of a BTBI. To measure this, we tested pairs of encoder and decoder rats during passive transmission of tactile information via a BTBI, while SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
the whiskers of the encoder and decoder rats were mechanically stimulated. This experiment consisted of two parts: first, the encoder animal was lightly anesthetized and head fixed to an automated whisker stimulator that accurately reproduces the movement and speed at which the whiskers interact with the bars in the active tactile discrimination task (see Methods). The animal’s S1 neural activity following each movement of the bars was analyzed in real time and delivered, as an ICMS pattern, to the decoder rat’s S1. Meanwhile, the decoder rat remained in an open field in a different room while its S1 neural activity was recorded. After this phase was completed, the decoder animal was also lightly anesthetized and placed in the automated whisker stimulator. This allowed us to determine how the decoder rat’s S1 neuronal sample, that responded via the BTBI to the tactile stimuli delivered to the encoder’s whiskers, responded to tactile stimuli elicited by passive whisker stimulation of their own vibrissae. Passive whisker stimulation, in either the encoder or decoder rats, induced significant firing modulations in the decoder rat’s S1. These were characterized by clear increases of firing activity occurring 4
219
www.nature.com/scientificreports
Figure 4 | Experimental apparatus scheme of a BTBI for transferring cortical tactile information. A) In the tactile discrimination task, the encoder animal was required to sample a variable width aperture using its facial whiskers. The width could be ‘‘Narrow’’ as shown in the left photograph, or ‘‘Wide’’. After sampling, the encoder animal had to report whether the aperture was narrow or wide by nose poking on a left or right reward port respectively. If correct, the animal received a small water reward. As the encoder explored the aperture, a sample of its S1 activity was recorded, compared with a template trial and then transferred to the decoders’ S1 via ICMS. The pattern of microstimulation constantly varied according to the number of spikes recorded from the encoder rat’s S1 in each trial. The decoder rat was required to make a response in the reward port corresponding to the width sampled by the encoder, guided only by the microstimulation pattern. If the decoder rat accurately responded in the correct reward port, both rats received a small water reward. Thus, the encoder rat received an additional reward in case both animals of the dyad performed a trial successfully.
immediately after the moving bars touched the whiskers of each animal (see Figure 7 A and B). These significant S1 neuronal responses occurred in 70.91% (39/55 multiunits) of the microwires implanted in the decoder rat’s S1, which were used to deliver ICMS patterns through the BTBI, and in 93.06% (67/72 multiunits) of the microwires from which S1 neuronal activity was recorded from decoder rats (see Figure 7 B and C). The magnitude of the S1 tactile responses elicited by mechanical stimulation of the decoder rat’s facial whiskers was 4.82 6 0.4 spikes/trial and the duration was 111.4 6 11 ms. During the BTBI transmission, ICMS of the decoder’s S1 induced a significant increase of S1 neurons firing activity lasting for 119.7 6 20 ms. Due to the microstimulation artifact in the recordings, we focused our analysis on the firing activity increases occurring after the last pulse of microstimulation was transmitted (see red traces Figure 7 C). Analysis of the data obtained during passive BTBI communication further demonstrated that S1 neurons in the decoder’s brain responded differently for each of the virtual tactile stimuli. More than half of the S1 multiunits recorded presented differential firing rates for Virtual Wide and Virtual Narrow stimuli (28/44 5 63.64% multiunits). Also, the Virtual Narrow stimulus was characterized by higher neuronal response magnitudes (Virtual Narrow: 3.861 6 0.6229 spikes/trial; Virtual Wide: 2.200 6 1.079 spikes/trial; Wilcoxon sum of ranks 5 197; P 5 0.0182) and durations (Virtual Narrow: 102.7 6 16.28 ms; Virtual Wide: 31.54 6 14.85 ms; Wilcoxon sum of ranks 5 200; P 5 0.0074). To measure whether the differences in firing rates were due to discrimination or due to an ‘upstate’ related to the repeated microstimulation, we also compared which S1 multiunits exhibited different firing rates for Virtual Narrow and Virtual Wide. From the total of S1 multiunits that displayed differences in firing rates for the discrimination period, we found that 35.7% (10/28 multiunits) had no significant differences in the baseline firing rate. Thus, more than one third of the S1 multiunits recorded showed no signs of an ‘upstate’ in their baseline due to repeated microstimulation. This supports the hypothesis that after SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
the decoder rats learned to use the BTBI, via ICMS cues, their S1 became capable of accurately representing, processing, storing and recalling information about both the tactile stimuli delivered to its own whiskers, as well as mechanical displacements of the encoders’ facial vibrissae. Finally, to further demonstrate the range of potential operation of our BTBI preparation, we tested whether a long-distance communication of a rat dyad, with the encoder rat performing the tactile discrimination task at the ELS-IINN (Natal, Brazil) and the decoder rat receiving patterns of microstimulation and responding at Duke University (Durham, USA), would be capable of performing the same task. For this, neural activity recorded from S1 of the encoder rat performing the tactile discrimination task was sent via an internet connection and delivered, as an ICMS pattern, to the decoder rat S1 (Figure 8). Even under these extreme conditions, the BTBI was also able to transfer in real-time behaviorally meaningful neuronal information. Although the mean time of data transmission observed in this long-distance BTBI was increased from 20 ms (during transmission in our Duke lab) to 232 6 217.5 ms, a similar number of correct responses was found (short distance transmission: 62.34 6 0.59%; long distance transmission: 62.25% 6 0.71) in 26.5 6 0.5 trials in the decoder animals.
Discussion The present study demonstrates for the first time that tactile and motor information, extracted in real time from simultaneously recorded populations of cortical neurons from a rat’s brain, can be transmitted directly into another subject’s cortex through the utilization of a real-time BTBI. Operation of a BTBI by an encoderdecoder rat dyad allowed decoders to rely exclusively on neural patterns donated by encoders in order to reproduce the encoder’s behavioral choice. ICMS patterns reflecting the number of action potentials recorded from either the encoder rat’s M1 or S1 during a single trial were sufficient for decoder rats to repeatedly perform two different tasks, significantly above chance levels, in real-time. 5
220
www.nature.com/scientificreports
Figure 5 | Behavioral performance using a brain-to-brain interface to transfer cortical tactile information. A) Performance of encoder and decoder animals during operation of a BTBI for tactile information sharing. Notice that the performance of the encoder animals was above 85% in all sessions. The performance of the decoder animals was above 60% in all sessions presented and immediately dropped to chance levels when the cable was disconnected but the system remained fully functional. B) Performance of all decoder animals analyzed with a moving average of 10 trials. C) The panel depicts the fraction of the decoder’s responses in the Narrow reward port after different patterns of microstimulation were delivered. As the number of microstimulation pulses increased a higher fraction of responses was observed in the Narrow reward port (Virtual Narrow choice), suggesting that the microstimulation threshold of response for decoder animals was situated between 26-40 pulses.
Interestingly, half of the number of pulses used to stimulate the decoder’s M1 were sufficient to successfully deliver a message to the decoder’s S1 cortex, suggesting that primary sensory cortical areas may have a lower threshold to operate a BTBI. SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
We also demonstrated that operation of a BTBI induced differential patterns of activation in the decoder rat’s S1. Thus, the same S1 neurons that responded to the mechanical stimulation of the decoder rat’s own whiskers were capable of representing information derived from stimulation of the encoder rat’s whiskers via the BTBI. Additionally, continuous operation of the BTBI also affected the behavior and neural activity of the encoder rat, which was able to reduce its response latency and increase the signal/noise ratio of its S1/M1 neuronal activity in response to an error by the decoder rat. As far as we can tell, these findings demonstrate for the first time that a direct channel for behavioral information exchange can be established between two animal’s brains without the use of the animal’s regular forms of communication. Essentially, our results indicate that animal brain dyads or even brain networks could allow animal groups to synchronize their behaviors following neuronal-based cues. Successful BTBI operation required four simultaneous conditions to be present: first, the encoder animals had to achieve a very high level of performance in both tasks. As proof, only one successful BTBI session was obtained when the encoder rat’s performance was below 80%. Second, recordings from the encoder’s cortex had to yield stable neural ensemble activity which was highly correlated to the behavior that needs to be encoded by the BTBI. Note that successful BTBI operation was achieved using information collected from random ensembles of neurons dispersed within each cortical area. This finding indicates that information was not anatomically segregated either in S1 or in M1. Third, the midpoint of the sigmoid transfer function (which was set at the beginning of the session) had to closely match the midpoint of the neural function that represented the two stimuli/actions. We found that such a midpoint tended to be the same for each cortical region, suggesting that groups of neurons with similar physiological profiles were recorded in most cases. Fourth, our results showed that both encoder and decoder rats changed their behavior according to the dyad performance. This observation suggests that operation of a BTBI induces the establishment of a highly complex system, formed by a pair of interconnected brains. As such, this brain dyad behaved in a way that could not be predicted if only pre-recorded neural signals had been used for encoding purposes. We speculate that the description of the complex system generated by the dyad transferring information and collaborating in real time, will reveal fundamental properties about the neural basis of communication and social interactions18,19. Although we have shown accurate transfer of brain-derived motor and sensory information through a BTBI, it remains to be explained how the brain simultaneously integrates information generated by direct ICMS and by natural stimuli (e.g. real whisker stimulation). Previous studies in rhesus monkeys have shown that the brain is able to decode highly complex ICMS patterns in a single trial7. Specifically, it has been shown that a brain-machine-brain control loop allows for continuous update of information in the S1 cortex, while a monkey explores a virtual tactile stimulus. The effects of a neuroprosthetic’s operation on cortical neuronal responses have also been studied in the representation of the rat forelimb sensorimotor cortex, where it was shown that information flow can be altered by S1 microstimulation20–22. Lastly, a recent study has shown that the ability to use a BMI is mediated by the striatum in mice4. Altogether, this body of evidence supports the notion that continuous use of ICMS to deliver information to the brain is associated with plastic changes in neuronal ensemble responses in cortical and subcortical regions. The data obtained here during passive BTBI operation supports this conclusion by showing that as animals learned to use the microstimulation cues, differential patterns of S1 neuronal responses emerged for each of the virtual tactile stimuli. This finding is consistent with our previous observation that S1 neurons undergo significant functional plasticity during the period in which rats learn a tactile discrimination task23. Accordingly, our results further suggest that successful 6
221
www.nature.com/scientificreports
Figure 6 | Trial examples of a BTBI for transferring cortical tactile signals. A) Examples of S1 neurons recorded while an encoder rat performed the aperture discrimination task. Time 5 0 corresponds to the moment the animal breaks the photo beam in front of the discrimination bars. B) Blue lines represent the choices of the encoder rat and red line represents the choices of the decoder rat. In trials 8, 15 and 17 the decoder rat selected the incorrect reward port. C) Number of action potentials recorded from 3 S1 neurons in each trial after the whiskers sampled the discriminanda. Typically, a higher spike count was found for narrow trials, when compared to wide trials. D) Number of pulses delivered to the S1 cortex of the decoder rat in each trial. The number of pulses delivered to the S1 cortex of the decoder rat was directly derived from the number of spikes present in the encoder animal in each trial. The overall performance achieved by the rat dyad in this session was 64% correct trials.
BTBI operation is fundamentally linked to the ability of S1 ensembles to undergo plastic reorganization in response to microstimulation patterns24. Altogether, the results described here indicate that the channel capacity (amount and precision of information, bandwidth) and the dynamic properties of cortical neuronal ensembles are the two major determinants of the amount and quality of information that can be transferred between animal brains via a BTBI. Thus, beyond the neurobiological challenge of understanding how the brain integrates natural and virtual stimuli, a second class of problems directly related to the characteristics of the BTBI as a channel for information transfer must be addressed. In general terms, the BTBI can be described as a discrete noisy channel, meaning ‘‘a system whereby a sequence of choices from a finite set of elementary symbols S1; : : : ;Sn can be transmitted from one point to another ’’25. The limit for the amount of information that can be transferred by unit of time (i.e. capacity)25 is currently unknown for a BTBI channel. In the tasks used here, the minimum and maximum inputs depended on the range of the firing rate in the neurons used, while the output depended mostly on the electrical microstimulation threshold of cortical ensembles in the decoder’s brain. However, channel capacity SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
can be increased while still using the rationale described in Figure 1 (neural data - transfer function – stimulation delivery). For example, it will be important to test in the future the effect of other types of inputs (e.g. larger neuronal ensembles; Local Field Potentials), transfer functions (e.g. exponential, linear, step functions) and outputs (e.g. one versus several pairs of microelectrodes used for ICMS, disposed in 2D or 3D cortical space, delivering photostimulation instead of electrical current) on the overall dyad performance. In this context, we expect that the use of newly introduced microelectrode cubes, created in our laboratory, that spread across 3D cortical space, to deliver spatiotemporal patterns of information from the encoder’s brain to the decoder’s will provide a significant increase in BTBI bandwidth, likely leading to a substantial improvement in the overall animal dyad performance. Lastly, it is important to stress that the topology of BTBI does not need to be restricted to one encoder and one decoder subjects. Instead, we have already proposed that, in theory, channel accuracy can be increased if instead of a dyad a whole grid of multiple reciprocally interconnected brains are employed. Such a computing structure could define the first example of an organic computer capable of solving heuristic problems that would be deemed non-computable 7
222
www.nature.com/scientificreports
Figure 7 | Neural activity in the decoder brain discriminates stimuli applied to the encoder’s whiskers. PSTHs on the left panels show S1 neuronal responses during the wide tactile stimulus whereas PSTHs on the right panels depict narrow tactile stimulus. The top and middle panels show S1 activity recorded in anesthetized encoder and decoder rats while their facial whiskers were passively stimulated by a set of moving bars. The moving bars generate a tactile stimulus exactly like the one produced during the tactile discrimination task. The lower panels represent the decoder rat’s S1 activity while receiving ICMS (red traces) via a BTBI that transmitted tactile information from an anesthetized encoder rat which was having its whiskers passively stimulated. Time zero in all panels corresponds either to the tactile stimulus or the last microstimulation pulse. A) A clear peak of S1 activity can be observed immediately after the encoder’s whiskers contacted the bars (other peaks occurred due to rebounding of the moving bars). Increased counts of action potentials were typically associated with the narrow stimulus (compare peaks in left versus right panels). B) Like encoder rats, when the decoder rats’ whiskers were passively stimulated by the moving bars, clear peaks of S1 activity with different heights can be observed (see left versus right panels). C) When the encoder rats’ whiskers were passively stimulated (shown in A) and the BTBI was used to transfer tactile information in real time (shown in C), clear increases in activity were observed in the decoder’s S1 cortex after time 0. These S1 firing modulations were larger when the narrow stimulus was applied to the encoders’ whiskers when compared to the wide stimulus (see left versus right panels) and were observed in the same S1 neuronal ensembles that responded to natural whisker stimuli (shown in B). Thus, the S1 neuronal responses observed in the decoder rat demonstrate that it learned to use the BTBI and that a representation of the tactile stimuli applied to the encoders’ whiskers could be superimposed on the preexisting representation depicting tactile stimuli applied to its own facial whiskers.
SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
8
223
www.nature.com/scientificreports was that in the rat-to-brain mode the patterns of microstimulation depended on the behavior of the encoder animal, while during the brain-to-brain interface mode the patterns of microstimulation depended solely on the neural activity of the encoder rat. Neural activity was first studied in encoder animals performing the behavioral task to identify units that accurately encoded for the motor activity associated with each lever press. During the rat-to-brain mode the operant chambers remained as in the training phase, however the presence of a correct choice in the encoder operant chamber activated the pattern of electrical microstimulation cue corresponding to the same lever on the decoder animal operant chamber. A correct choice by the decoder rat was signaled by a tone and both encoder and decoder animals were allowed a brief period of access to water. The goal of this phase was to create a template trial based on the neural activity of the encoder rat pressing one of the levers in several trials. During the brain-to-brain interface mode the number of action potentials in each trial was compared to the template trial and the Zscore of the difference between them was used to determine the number of pulses present in the pattern of microstimulation. A sigmoid function was used to transfer the Zscore value to the number of pulses present in the microstimulation pattern. A higher Zscore was associated with an increased number of pulses in the microstimulation cue (i.e. right lever) while a lower Zscore was associated with a decreased number of pulses in the microstimulation cue (e.g. left lever).
Figure 8 | Intercontinental brain-to-brain interface to transfer cortical tactile information. To test the full potentialities of the BTBI, a brain-to-brain interface to transfer cortical tactile information was established between our laboratory at the IINN-ELS in Brazil and our laboratory at Duke University in the USA. An encoder rat performed a tactile discrimination task at the IINN-ELS. Meanwhile its neuronal activity in S1 was recorded and sent over the internet to our laboratory at Duke University. The sigmoid transformation algorithm was used to transfer the number of action potentials into microstimulation patterns that there were then delivered to the decoder rat’s S1 cortex. As the decoder rat made a behavioral response, feedback was sent over the internet to the encoders’ chamber back at the IINN-ELS.
by a general Turing-machine. Future works will elucidate in detail the characteristics of this multi-brain system, its computational capabilities, and how it compares to other non-Turing computational architectures26.
Methods All animal procedures were performed in accordance with the National Research Council’s Guide for the Care and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use Committee. Long Evans rats weighing between 250–350 g were used in all experiments. Motor brain-to-brain interface. The behavioral motor task consisted of a dark operant chamber equipped with two levers, one LED (Light Emitting Diode) above each lever and, on the opposite wall, a water reward port. Animals were trained to press one of two levers, cued by an LED turned on at the beginning of each trial. A correct choice opened the reward port and allowed brief access to water (300 ms). When animals reached stable performances above 80% correct choices they were assigned either to an encoder or decoder group. The operant chamber configuration remained similar in both the encoder and the decoder groups. Animals assigned to the encoder group were implanted with recording arrays of 32 microelectrodes in the primary motor cortex and after recovery resumed the initial training scheme. Animals assigned to the decoder group were implanted with arrays of 4 to 6 microstimulation electrodes in the primary motor cortex and were further trained to associate the presence of electrical microstimulation pulses with the correct lever press. Extra training followed, with a sequence of 60 to 100 pulses indicating a correct choice in the right lever while the absence of microstimulation pulses (1 pulse) indicated a correct left lever choice. During the electrical microstimulation training phase a trial started with a brief period of white noise, followed by the electrical microstimulation cue. Immediately after this cue both LEDs were turned on. If a correct choice was made the reward port would open and the animal was allowed a brief period of access to water (300 ms), otherwise both LEDs were turned off and the intertrial interval started. When decoder animals reached stable performances above chance, brain-to-brain interface (BTBI) sessions with motor activity were performed. These sessions were composed of three different phases: 1) identification of cells accurately encoding the motor action performed, 2) data collection for template trials during a rat-to-brain mode and, 3) brain-to-brain interface using real-time analysis followed by electrical microstimulation. There were no setup differences between the rat-to-brain mode and the brain-to-brain interface. The only difference between these two conditions
SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
Active tactile brain-to-brain interface. Mildly water deprived animals were trained to perform a behavioral discrimination task as previously described17. Briefly, this task required animals to discriminate between a wide or narrow aperture in order to receive a water reward. The animal was placed in the behavioral box compartment where it waited for the central door to open and allow access to the second compartment, the inner chamber. After the animal entered the inner chamber, it had to pass through the variable width discrimination bars and then nose poke the center of the front wall. The nose poke in the inner chamber opened two water reward pokes located in the outer chamber from which the animal had to select one. The reward poke on the right corresponded to the wide aperture, whereas the poke on the left corresponded to the narrow aperture. As the animal chose from one of the reward pokes, the door separating the inner and outer chambers closed. Correct responses were rewarded by 50 ml water rewards. Incorrect responses were followed by immediate closing of the reward pokes. The percent of trials performed correctly was used as a measure of tactile discrimination. Animals were then evenly assigned to encoder or decoder groups. Encoder animals (n 5 2) were implanted with recording arrays of 32 microelectrodes in the right S1 and after recovery resumed the initial training scheme. Animals assigned to the decoder group were implanted with arrays of six microstimulation electrodes in the right S1 (n 5 7). In six decoder animals, recording electrode arrays were also implanted either in the right (N 5 1) or in the left S1 (N 5 5). After recovering from surgery decoder animals were further trained to associate the presence of electrical microstimulation pulses with the correct lever press. Extensive training followed, with a sequence of 50 pulses indicating a correct choice in the left reward poke while the absence of microstimulation pulses (1 pulse) indicated a correct choice in the right reward poke. Decoder animals were required to identify the microstimulation cue and associate it with a behavioral response in one of the reward pokes. A brief tone indicated the beginning of the trial immediately followed by the microstimulation cue. After a period of 500 ms both reward pokes would open and the rat was required to make a response in one of the photo beams. A correct choice was followed by a brief tone and access to water. When decoder animals reached stable performances of . 65% correct trials for 3 consecutive sessions, tactile BTBI sessions began. Neural activity was first studied in encoder animals performing the behavioral task to identify units that accurately encoded for the tactile stimuli associated with sampling the width between bars. During the rat-to-brain mode the operant chambers remained as in the training phase, however the microstimulation cue presented to the decoder animal always matched the stimulus presented to the encoder animal. After a correct response by the encoder rat, a brief tone followed by a microstimulation cue of 1 or 50 pulses was sent to the decoder animal and both reward ports in the second chamber would open. If the decoder rat accurately discriminated the microstimulation cue both rats were rewarded. During the brain-to-brain interface mode the neural activity of the encoder rat was analyzed from the moment that the rat broke the discrimination bars photo beam to the moment that the rat broke the photo beam in the center poke. The number of action potentials found in this interval was then counted and compared to the distribution of the Zscores relative to the spikes present in all the previous Wide trials. A Zscore was determined and transferred using a sigmoid function, into the number of pulses present in the pattern of microstimulation. Passive tactile brain-to-brain interface. The encoder animal was anesthetized and remained head fixed in one room, while the decoder rat was in an open field in a different location with the neural activity also being recorded. The encoder rats’ whiskers were then stimulated by a set of moving bars that accurately reproduce the dynamics observed in the active tactile width discrimination task14. The bars were set up for Wide or Narrow widths and, for each width, the head fixed animal was stimulated for approximately 6 minutes at 0.3 Hz. Neural activity was first analyzed in real time and units with clear whisker related activity were used for the session. To establish a baseline distribution; an initial group of 100 wide stimulus trials was recorded. Then the passive brain-to-brain interface mode started. The encoders’ whiskers were stimulated by the moving aperture corresponding to a Wide stimulus. Meanwhile the number of action potentials recorded from the encoder animal was
9
224
www.nature.com/scientificreports counted. The number of action potentials was compared to the distribution of action potentials found at the baseline at the beginning of the session and a Zscore was calculated. This Zscore was transferred into the number of pulses to be used in the microstimulation using a sigmoid function. The decoder rat then received the pattern of microstimulation derived from the sigmoid function. Immediately after the encoder animal had been passively stimulated with both Wide and Narrow widths, the decoder animal was anesthetized, head fixed, and its whiskers were passively stimulated with the same Wide and Narrow stimuli as the encoder animal. Surgery for microelectrode array implantation. Fixed or movable microelectrode bundles or arrays of electrodes were implanted in the M1 and S1 of rats. Craniotomies were made and arrays lowered at the following stereotaxic coordinates for each area: S1 [(AP) 23.0 mm, (ML), 15.5 mm (DV) 20.7 mm], M1 [(AP) 12.0 mm, (ML) 12.0 mm, (DV) 21.5 mm]. Electrophysiological recordings. A Multineuronal Acquisition Processor (64 channels, Plexon Inc, Dallas, TX) was used to record neuronal spikes, as previously described27. Briefly, differentiated neural signals were amplified (20000–32,0003) and digitized at 40 kHz. Up to four single neurons per recording channel were sorted online (Sort client 2002, Plexon inc, Dallas, TX ). Online sorting was validated offline using Offline Sorter 2.8.8 (Plexon Inc, Dallas, TX). Intracortical electrical microstimulation. Intracortical electrical microstimulation cues were generated by an electrical microstimulator (Master 8, AMPI, Jerusalem, Israel) controlled by custom Matlab script (Nattick, USA) receiving information from a Plexon system over the internet. Patterns of 1–100 (bipolar, biphasic, charge balanced; 200 msec) pulses at 400 Hz (motor BTBI) or 250 Hz (tactile BTBI) were delivered to the cortical structures of interest (M1 and S1 respectively). Current intensity varied from 38–200 mA (motor BTBI) and 30–240 mA (tactile BTBI). Data analysis. For both behavioral tasks the number of correct responses was used as a measure of behavioral performance. We also analyzed the animals’ response latency as a measure of independency between the performance of each animal alone or in a dyad. Neuronal data were processed and analyzed using Neuroexplorer (version 3.266, NEX Technologies) and custom scripts written in Matlab (7.9.0, Mathworks, Natick, MA). Statistical significance of neural responses was evaluated using a method based on cumulative-summed spike counts28,29. Comparisons of characteristics of neural responses for different conditions were performed using non-parametric tests (Mann-Whitney-Wilcoxon or Kruskal-Wallis). Signal-to-noise ratio of neural responses was calculated as the proportion of responses, occurring after a correct or incorrect decoder response, that presented Zscore absolute values above 0.3 standard deviations. This specific value was used because it corresponded to the midpoint of the sigmoid curve). Statistical significance was determined using a chi square test for proportions. 1. Hartley, R. V. L. Transmission of Information. Bell Technical Journal, 535–564 (1928). 2. Jackson, A. & Zimmermann, J. B. Neural interfaces for the brain and spinal cordrestoring motor function. Nat Rev Neurol 8, 690–699 (2012). 3. Ethier, C., Oby, E. R., Bauman, M. J. & Miller, L. E. Restoration of grasp following paralysis through brain-controlled stimulation of muscles. Nature 485, 368–371 (2012). 4. Koralek, A. C., Jin, X., Long, J. D., 2nd, Costa, R. M., Carmena, J. M. Corticostriatal plasticity is necessary for learning intentional neuroprosthetic skills. Nature 483, 331–335 (2012). 5. Lebedev, M. A. et al. Future developments in brain-machine interface research. Clinics (Sao Paulo) 66 Suppl 1, 25–32 (2011). 6. Moritz, C. T., Perlmutter, S. I. & Fetz, E. E. Direct control of paralysed muscles by cortical neurons. Nature 456, 639–642 (2008). 7. O’Doherty, J. E., Lebedev, M. A., Li, Z. & Nicolelis, M. A. Virtual active touch using randomly patterned intracortical microstimulation. IEEE Trans Neural Syst Rehabil Eng 20, 85–93 (2012). 8. Venkatraman, S. & Carmena, J. M. Active sensing of target location encoded by cortical microstimulation. IEEE Trans Neural Syst Rehabil Eng 19, 317–324 (2011). 9. O’Doherty, J. E. et al. Active tactile exploration using a brain-machine-brain interface. Nature 479, 228–231 (2011). 10. Vato, A. et al. Shaping the dynamics of a bidirectional neural interface. PLoS Comput Biol 8, e1002578 (2012). 11. Chapin, J. K., Moxon, K. A., Markowitz, R. S. & Nicolelis, M. A. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat. Neurosci. 2, 664–670 (1999). 12. Laubach, M., Wessberg, J. & Nicolelis, M. A. Cortical ensemble activity increasingly predicts behaviour outcomes during learning of a motor task. Nature 405, 567–571 (2000).
SCIENTIFIC REPORTS | 3 : 1319 | DOI: 10.1038/srep01319
13. Ghazanfar, A. A., Stambaugh, C. R. & Nicolelis, M. A. Encoding of tactile stimulus location by somatosensory thalamocortical ensembles. J. Neurosci. 20, 3761–3775 (2000). 14. Krupa, D. J., Wiest, M. C., Shuler, M. G., Laubach, M. & Nicolelis, M. A. Layerspecific somatosensory cortical activation during active tactile discrimination. Science 304, 1989–1992 (2004). 15. Nicolelis, M. A. Beyond Boundaries: The New Neuroscience of Connecting Brains with Machines and How It Will Change Our Lives. (Times Books, 2010). 16. Fitzsimmons, N. A., Drake, W., Hanson, T. L., Lebedev, M. A. & Nicolelis, M. A. Primate reaching cued by multichannel spatiotemporal cortical microstimulation. J. Neurosci. 27, 5593–5602 (2007). 17. Krupa, D. J., Matell, M. S., Brisben, A. J., Oliveira, L. M. & Nicolelis, M. A. Behavioral properties of the trigeminal somatosensory system in rats performing whisker-dependent tactile discriminations. J. Neurosci. 21, 5752–5763 (2001). 18. Hasson, U., Ghazanfar, A. A., Galantucci, B., Garrod, S. & Keysers, C. Brain-tobrain coupling: a mechanism for creating and sharing a social world. Trends Cogn Sci 16, 114–121 (2012). 19. Mattout, J. Brain-computer interfaces: a neuroscience paradigm of social interaction? A matter of perspective. Front Hum Neurosci 6, 114 (2012). 20. Rebesco, J. M. & Miller, L. E. Stimulus-driven changes in sensorimotor behavior and neuronal functional connectivity application to brain-machine interfaces and neurorehabilitation. Prog. Brain Res. 192, 83–102 (2011). 21. Rebesco, J. M., Stevenson, I. H., Kording, K. P., Solla, S. A. & Miller, L. E. Rewiring neural interactions by micro-stimulation. Front Syst Neurosci 4 (2010). 22. Talwar, S. K. et al. Rat navigation guided by remote control. Nature 417, 37–38 (2002). 23. Wiest, M. C., Thomson, E., Pantoja, J. & Nicolelis, M. A. Changes in S1 neural responses during tactile discrimination learning. J. Neurophysiol. 104, 300–312 (2010). 24. Recanzone, G. H., Merzenich, M. M. & Dinse, H. R. Expansion of the cortical representation of a specific skin field in primary somatosensory cortex by intracortical microstimulation. Cereb. Cortex 2, 181–196 (1992). 25. Shannon, C. E. The Mathematical Theory of Communication. (University of Illinois Press, 1949). 26. Siegelmann, H. T. Computation beyond the turing limit. Science 268, 545–548 (1995). 27. Nicolelis, M., Stambaugh, C., Brisben, A. & Laubach, M. in Methods for Neural Ensemble Recordings (ed M. A. Nicolelis) 121–156 (CRC Press, 1999). 28. Gutierrez, R., Carmena, J. M., Nicolelis, M. A. & Simon, S. A. Orbitofrontal ensemble activity monitors licking and distinguishes among natural rewards. J. Neurophysiol. 95, 119–133 (2006). 29. Wiest, M. C., Bentley, N. & Nicolelis, M. A. Heterogeneous integration of bilateral whisker signals by neurons in primary somatosensory cortex of awake rats. J. Neurophysiol. 93, 2966–2973 (2005).
Acknowledgements The authors would like to thank Jim Meloy for outstanding electrode manufacturing; Laura Oliveira, Susan Halkiotis and Edgard Moria for miscellaneous assistance, and Eric Thomson and Hao Zhang for thoughtful comments on experiments and manuscript. This work was supported by NIH R01DE011451 and by a National Institute of Mental Health award DP1MH099903 to MALN; Fundação BIAL 199/12 and also by the Program for National Institutes of Science and Technology and the National Council for Scientific and Technological Development CNPq/MCTI/INCT-INCEMAQ 610009/2009-5; Financiadora de Estudos e Projetos FINEP 01.06.1092.00, Fundação de Amparo a Pesquisa do Rio Grande do Norte FAPERN 01/2011. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Author contributions M.P.V., M.A.L. and M.A.L.N. designed experiments, analyzed data and wrote the paper; M.P.V., C.K. and J.W. conducted experiments.
Additional information Supplementary information accompanies this paper at http://www.nature.com/ scientificreports Competing financial interests: The authors declare no competing financial interests. License: This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/ How to cite this article: Pais-Vieira, M., Lebedev, M., Kunicki, C., Wang, J. & Nicolelis, M.A.L. A Brain-to-Brain Interface for Real-Time Sharing of Sensorimotor Information. Sci. Rep. 3, 1319; DOI:10.1038/srep01319 (2013).
10
225
226
www.nature.com/scientificreports
OPEN
received: 11 March 2015 accepted: 30 March 2015 Published: 9 July 2015
Computing Arm Movements with a Monkey Brainet Arjun Ramakrishnan1,2,*, Peter J. Ifft2,3,*, Miguel Pais-Vieira1,2, Yoon Woo Byun2,3, Katie Z. Zhuang2,3, Mikhail A. Lebedev1,2 & Miguel A.L. Nicolelis1,2,3,4,5 Traditionally, brain-machine interfaces (BMIs) extract motor commands from a single brain to control the movements of artificial devices. Here, we introduce a Brainet that utilizes very-large-scale brain activity (VLSBA) from two (B2) or three (B3) nonhuman primates to engage in a common motor behaviour. A B2 generated 2D movements of an avatar arm where each monkey contributed equally to X and Y coordinates; or one monkey fully controlled the X-coordinate and the other controlled the Y-coordinate. A B3 produced arm movements in 3D space, while each monkey generated movements in 2D subspaces (X-Y, Y-Z, or X-Z). With long-term training we observed increased coordination of behavior, increased correlations in neuronal activity between different brains, and modifications to neuronal representation of the motor plan. Overall, performance of the Brainet improved owing to collective monkey behaviour. These results suggest that primate brains can be integrated into a Brainet, which self-adapts to achieve a common motor goal.
BMIs are computational systems that link biological circuits to external devices, such as computer cursors, robotic prostheses and communication aids1. Heretofore, BMIs have been utilized either: (i) to extract motor signals from neural activity and convert them to the movements of external devices2–5, (ii) to deliver sensory signals from the environment to the brain6–8, or (iii) to combine both operations and enable bidirectional communications between the brain and machine9. In each of these implementations, a BMI serves as an accessory to a single brain. Recently an entirely new direction was proposed for BMI research – a brain to brain interface (BtBI)10. BtBI allows animal brains to exchange and share sensory and motor information to achieve a behavioural goal11,12–17. BtBI is a hybrid computational system since it incorporates both biological components (the primate brains) and digital parts (the BMI system). In the present study, we have designed and tested a more elaborate computational architecture which we refer to as a Brainet10. Our Brainets involved groups formed by 2-3 monkeys in a shared BMI that enacted conjoint motor behaviours. Previously, human psychophysics studies have shown that two or more individuals who are performing movements simultaneously often entrain to each other’s behavior, even if they are not explicitly instructed to do so18–22. However, the neurophysiological mechanisms of such joint actions are not well understood. In particular, we were interested in investigating the possibility that neuronal ensemble could directly control conjoint behaviors enabled by multiple interconnected BMIs. Our study adds to previous attempts to overcome limitations of one individual confronted with a high processing load by mixing contributions of multiple individuals23–29. Particularly relevant to our present work, several EEG studies13,30–34 have combined brain derived signals from multiple subjects to enhance visual discrimination, motor performance, and decision making. A recent EEG study30 has implemented shared control that involved dynamic collaboration of multiple individuals in real time to achieve a common goal. However, in none of these EEG experiments participants interacted with each other over a long term. Moreover, no large-scale intracranial cortical recordings were obtained in order to investigate 1
Department of Neurobiology, Duke University, Durham, NC, USA. 2Duke University Center for Neuroengineering, Duke University, Durham, NC, USA. 3Department of Biomedical Engineering, Duke University, Durham, NC, USA. 4 Department of Psychology and Neuroscience, Duke University, Durham, NC, USA. 5Edmund and Lily Safra International Institute of Neurosciences of Natal, Natal, Brazil. *These authors contributed equally to this work. Correspondence and requests for materials should be addressed to M.A.L.N. (email: nicoleli@neuro.duke.edu) Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
1
227
www.nature.com/scientificreports/
Figure 1. Experimental setups for B2 and B3 experiments. (A) Monkeys were seated in separate rooms, each facing a computer monitor showing the virtual avatar arm (inset in C) from a 1st person perspective. (B) Shows the shared control task, (X,Y) position of the virtual arm was decoded during centre-out movements from the two monkeys’ brains with each given 50% control of the arm. Electrode array location shown on brains. (C) Shows the partitioned control task. X position of the arm was decoded from one monkey and Y position from the other during centre-out movements toward targets. (D) Shows the 3-monkey task. Each monkey observed and had 50% control over 2 of the 3 dimensions (X, Y, or Z). Together, the three monkeys must accurately perform a 3-D centre-out movement to achieve reward. Drawings by Miguel A.L. Nicolelis.
the neurophysiological mechanism underlying conjoint motor behaviour. As such, the present study is the first to implement Brainets based on the chronic extracellular recordings of up to 783 individual cortical neurons distributed across multiple rhesus monkey brains. To this end, we have implemented Brainet architectures consisting of two (B2) (Fig. 1B,C) or three (B3) (Fig. 1A,D) monkey brains. These Brainets (i) controlled 2D/3D movements of an avatar arm by sharing signals derived from multiple brains (Fig. 1B,D), (ii) partitioned control by delegating subtasks to different subjects (Fig. 1C,D) and (iii) enabled a super-task that was composed of individual BMI tasks (Fig. 1A,D).
Results
Four monkeys participated in the experiments. They were chronically implanted with multielectrode arrays in motor (M1) and somatosensory cortices (S1)1,35. Extracellular electrical activity from 570-783 neurons was sampled chronically from monkeys M (410-501 neurons), C (196-214), O (156-229), and K (140-156) up to five years after implantation36. B2s operated using at least 550 neurons, while the B3s operated with at least 775 neurons. During experimental sessions, monkeys sat in separate rooms where each of them viewed a realistic monkey avatar arm on a computer screen (Fig. 1A). Movements Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
2
228
www.nature.com/scientificreports/ Group name
Acquisition time
Precision
Hit rate
HC Dyad MO
0.7s
2.04
82.02%
B2: Dyad MO
1.86s
2.04
86%
B2: Dyad MC
1.75s
2.04
80%
B3: Triad MCK
3.31s
0.37
78%
Table 1.
of the avatar arm were generated using either a hand-held joystick (hand control) or from modulations in VLSBA in a mode called brain control4,5. In brain control, a neural decoding algorithm called the unscented Kalman filter (UKF)37 extracted reach kinematics from neuronal ensemble activity in real-time.
Experiment 1: Shared control. In the first experiment (shared control task, Fig. 1B), two monkeys
worked together to move the avatar arm in 2D space. Each monkey contributed equally (50%) to both the X and Y coordinates. The monkeys moved the avatar arm from the screen centre towards a circular target (Movies S1, S2). These movements were enacted either using the joystick (dyad M&O) or the Brainet (monkey dyads M&O and M&C). To obtain training data for brain control, we started each session with a passive observation epoch during which monkeys watched the computer-generated avatar arm movements along centre-out trajectories38. One UKF was fit per monkey, and the UKF outputs (X and Y coordinates) were combined across monkeys to produce the conjoint brain-controlled movements of the avatar arm. The behaviour of the dyad MO improved significantly with training (Fig. 2A–D). Over several weeks of training (4 weeks of hand control followed by 3 weeks of brain control), we observed a significant reduction in target acquisition time between early and late sessions (Figs. 2A,B) (p < 0.001 and p < 0.01; KS test for hand control and brain control, respectively), trial duration (p < 0.001 and p < 0.02) and inter-reward interval (p < 0.004 and p < 0.02). The mean trial duration across weeks of training significantly decreased for both hand control (∆t = 0.13 s, 1-way ANOVA: p < 0.05) (Fig. 2C) and brain control training (∆t = 0.7 s, 1-way ANOVA: p < 0.05) (Fig. 2D). To assess the performance of the B2 in manual control and brain control modes, we also calculated performance metrics reported in the previous literature39,40. Table 1 shows the overall mean target acquisition time, target hit rate and the movement precision required, which was quantified as the ratio of target size to the size of the workspace40. As monkeys M and O learned the shared control task, their coordination improved in both hand control and brain control modes (Fig. 2E–J). In the initial session, with monkey M initiating movements earlier than monkey O (Fig. 2E,I; hand control), the lag between the monkeys’ reaction times was 200 ± 12 ms (mean ± SEM). With conjoint training, reaction time lag between the two monkeys steadily decreased to 10 ± 27 ms over 21 sessions (13 sessions of hand control and 8 sessions of brain control, Fig. 2I). This gradual reduction in lag was statistically significant (hand control: linear regression: r2 = 0.57; p < 0.01, brain control: linear regression: r2 = 0.53; P < 0.05). As the monkey dyad became more coordinated, the cortical neural activity lag between the two brains decreased and stabilized near zero (Fig. 2G,H,J). More specifically, cortical activity lag decreased during hand control (linear regression: r2 = 0.41; P < 0.05) and remained approximately zero (4.2 ± 5.1 ms) throughout brain control (purple data in Fig. 2J, linear regression: R2 = 0.01; P > 0.05). For the other dyad (M&C), the behavioural responses were synchronous and lag remained close to zero (3 ± 35 ms; linear regression: P = 0.44) throughout all sessions (8 sessions) (Fig. S1, Movie S2). Dyad M&C performed brain control after passive observations of the avatar arm movements and without a requirement to coordinate their behaviours in the manual task. This training sequence probably aided better coordination between the subjects. Table 1 shows the mean target acquisition time, target hit rate and the movement precision required for the dyad M&C. Concurrent changes in neural activity during Brainet operation could be related to multiple factors: common visual feedback of the avatar arm movements, common representation of the target (spatial location and temporal onset), and common BMI outputs (movements to the same location). To understand if the interplay between the common sensory input and common motor behaviour could account for the genesis of concurrent neural activity we carried out further analyses. First, we investigated the relationship between the behavioural outcomes in each trial and concurrent neuronal modulations between the monkeys in the same trials. We found correct task outcomes were associated with greater neural correlations between monkeys, both in hand control (Fig. 3A; P < 0.01; unpaired t-test) and brain control (Fig. 3B; P < 0.01). To clarify the role of concurrent neuronal activity further, we conducted an analysis where a k-NN classifier predicted the trial outcome, based on the neural correlations between monkeys, both before (700 ms before target appearance until target appearance) and after target onset (from 100 to 800 ms after target appearance). The neuronal correlations were significant predictors of trial outcome (P < 0.05, 1-proportion z-test) not only after target appearance but also before (Fig. 3C,D). Notably, the Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
3
229
www.nature.com/scientificreports/
Figure 2. B2 shared control task. (A,B) Boxplots comparing target acquisition time (left panel), trial duration (centre panel), inter-reward interval (right panel) in hand control (A) and in brain control (B). (C,D) Changes in trial duration with conjoint training across weeks. Mean ± SEM trial duration for each of the four weeks of hand control experiments (C) and three weeks of brain control (D) for dyad M&O. Dashed line shows linear regression fit. Trial duration reduced significantly with improvement in coordination between monkeys. (E,F) Trial-averaged movement profiles from an early session (dashed) and a late session (solid) for monkey M (red) and O (blue) aligned to the time of target onset (time = 0 ms.). Plots are shown up to 1.5 s for hand control (with mean target acquisition time of 0.8 s) and 3 s for brain control (with mean target acquisition time of 2.2 s). The target was located 9 cm from the centre (y-axis). As the dyad trained together over a span of 7 weeks the reaction time lag, the time of movement onset of monkey O relative to monkey M, decreased during both hand control (E) and brain control (F) (G,H) Neural activity PETH from an early session and a late session for hand control (G) and brain control (H) Each row is a single neuron, colour denotes normalized firing amplitude. Monkey M and O neurons marked by black (M) and grey (O) vertical bands. Notice that neuronal modulations are more intense and synchronized across the monkey pair in the later session as compared to the early session. (I) Changes in reaction time lag between monkey M and O across experiments. Lag was derived from peak of crosscorrelation of two monkeys’ behavioural traces (E,F) Trends fit with linear regression. As the dyad trained together the reaction time lag reduced to 10 ± 27 ms. (J) Lag in neural activity between two monkeys over the same experiments as in (E) computed again by finding the peak of cross correlogram on each session. As the dyad trained together the reaction time lag reduced to 4.2 ± 5 ms. Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
4
230
www.nature.com/scientificreports/
Figure 3. (A-B) Cross correlogram of Monkey M vs. Monkey O neural activity on correct trials (upper left panel) and error trials (lower left panel) for both a representative hand control (A) and brain control session (B) Mean cross correlogram for the correct and error trial group shown in panel on right of each panel. The correlation is higher in correct trials indicative of increased synchronous neuronal modulations between monkeys. (C,D) k-NN prediction of trial outcome (reward or error) using the mean neuronal cross correlogram on a single trial (i.e. left panels in A-B) between the two monkeys either prior to target appearance (“Pre-target”) or after target appearance (“Post-target”). Chance level prediction (95% confidence interval) shown in yellow band. * denotes P < 0.01, unpaired t-test. Neuronal synchrony between monkeys before and after target onset was predictive of trial outcome. (E,F) Extra correlation analysis. Velocity profile in a trial for dyad M&O were cross-correlated. The average cross correlation (trial specific correlation, red trace) was estimated for all trials: hand control (E) and brain control (F) Extra correlation was the excess correlation (in the red trace) that cannot be accounted for by the distribution of across-trial correlation (enclosed by the grey dashed lines). The vertical line shows the time lag at peak correlation. (G-H) Partitioned control task: (G) Panels show the average trajectory to the 4 target locations, denoted by colour, from the first (left column) and the last (right column) session. Trajectories derived from the monkeys’ controlled axes predictions (controlled) are shown in the upper panels and non-controlled axes predictions (non-controlled) are shown in the lower panels. Non-controlled traces were initially more similar to the controlled traces (left panels) but over time became shrunk and convoluted (right panels). (H) The fraction of rewarded trials was computed based on trials when the avatar arm is moving according to the predictions of the controlled axis (blue) or the non-controlled axes (red). Mean fraction correct trials shown for each of the three weeks of experiments. Shown separately is fraction correct amongst all trials (upper panel) and only among trials where a reward was achieved (lower panel). The percentage of trials in which the complementary trace reached the target (red bars) decreased significantly over training.
Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
5
231
www.nature.com/scientificreports/ effect of neural synchrony on the trial outcome was stronger in brain control mode (Fig. 3D). We suggest that the presence of neural correlation prior to target onset indicated that the monkeys attended to the avatar arm since the common visual input from the avatar arm was the only source of neural correlation during that period. This interpretation in terms of an increased attention agrees well with the better performance on the trials with high neural correlation: the more the monkeys attend to the avatar arm and the other screen events, the better they perform. Next, we examined whether the increase in concurrent neuronal activity between the two monkeys could emerge due to the fact that the two animals move simultaneously, without necessarily paying attention to each other, following visual stimulus onset. If this scenario were true, there would not be any trial specific neuronal correlation beyond the level that could be observed if the behavioural trials of one of the monkeys were randomly shuffled (provided that target locations still matched for two monkeys). To investigate this possibility, we used movement velocities as a proxy to test the correlation between monkeys. To determine the additional component to the coordination between the monkeys, over and above the coordination that comes about as a result of jointly performing reaching movements to a specific target location, we calculated a parameter called “extra-correlation”. Extra-correlation is the correlation between the two monkeys’ movement velocities in a trial that cannot be explained by the across-trial correlation. Across-trial correlation was computed by correlating velocity of movement of a monkey from one trial with that of the partner monkey on another randomly chosen trial in which the monkeys moved to the same target location. This was performed for several such combinations of trials (40-50 combinations) to obtain a distribution of the across-trial correlation between the velocities of the dyad. We observed that extra correlation in movement velocities, which was significant in 3/13 sessions during hand control (Fig. 3E), was however, significant during all brain control sessions for both dyad MO (Fig. 3F) and MC (Fig. S2). These results suggest that increases in concurrent activity is not only due to motor outputs triggered by the same visual stimulus, but also due to a significant trial specific coordination in movement velocities, especially during brain control.
Experiment 2: Partitioned control. In the next experiment, a B2 (dyad M&C) performed a parti-
tioned control task where each monkey performed a subtask of a 2D movement (Fig. 1C). One monkey controlled only the X position of the avatar arm (X-monkey) and the other monkey controlled only the Y position (Y-monkey). The targets appeared on the diagonals at 45°, 135°, 225°, or 315°, which meant that both the X-monkey and Y-monkey had to simultaneously and accurately enact movements along their individual axes to achieve a correct trial. Each session started with a period of passive observations of the avatar arm movements. Based on the data recorded during this period, one UKF was fit per monkey. During brain control, the UKF outputs (X for monkey M, and Y for monkey C) were combined. The unused outputs (Y for monkey M, and X for monkey C) were computed, but not shown to the monkeys as any kind of feedback. We compared the used and unused X and Y outputs in an offline analysis to estimate cortical adaptation to preferentially represent the coordinate being controlled through the BMI. During brain control mode, despite being given control of only one axis, a monkey could in principle generate motor plans at the neural level that encoded movements in both the X and Y dimensions. Alternatively, each monkey could parse the 2D trajectory into a controlled dimension (which it had to attend to) and non-controlled dimension (which it could disregard). For example, if monkey M controlled movements along X-axis, we asked how this animal’s cortical neurons changed their contribution to movements along Y-axis, which they did not control. That provided us with a way to quantify how much functional plasticity was occurring in the monkeys’ cortex as a result of partitioned Brainet operation. To measure that, we first plotted the actual avatar movement traces generated by the B2 (Fig. 3G; upper panels) which corresponds to the axes the monkeys controlled (X for monkey M, and Y for monkey C). We then compared this plot with the traces for the complementary pair of dimensions (Y for monkey M, and X for monkey C; Fig. 3G: lower panels). We found that the members of the B2 contributed more along the dimension they controlled and less along the dimension they did not control. This result is evident from the comparison of the actual avatar movement traces generated by the B2 (Fig. 3G; upper panels) to the complementary traces (Fig. 3G; lower panels). These traces were similar in the early sessions (Fig. 3G; left panels) but diverged over time (Fig. 3G, right panels). By the 3rd week of training, average complementary traces were shrunk and convoluted compared to the average actual traces. As a result of this cortical adaptation, the complementary trajectories seldom reached the target. The percentage of trials in which the complementary trace reached the target (red bars in Fig. 3H) decreased from 26% to 12% over training (regression, r2 = 0.42 upper panel of Fig. 3H, P < 0.05). Similar reduction was observed when only the rewarded trials were included (r2 = 0.38, P < 0.05, lower panel Fig. 3H).
Experiment 3: Brainet triad control of an avatar arm in 3-D space. In the third experiment, we built a 3D super-task out of individual 2D BMI tasks. Each of the three monkeys forming a B3 viewed the 2D projection of a spherical target in 3D space, from an X-Y, Y-Z, or X-Z reference frame (Fig. 1D). The monkeys moved the avatar arms on their displays in their 2D space to the projected target location. As they reached the projected target, a 3D avatar arm that received inputs from all the three UKF decoders reached the true target location in 3D space (Movie S3). This design reduced task difficulty for any one individual monkey by breaking down the 3D task into simpler 2D subtasks. Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
6
232
www.nature.com/scientificreports/
Figure 4.â&#x20AC;&#x201A; Triad (B3) control of 3D movements. (A) Boxplots comparing target acquisition time (left panel), trial duration (centre panel), inter-reward interval (right panel). (B) Reduction in trial duration (left panel) and concurrent increase in the reward rate (right panel) with conjoint training across weeks. (C) Normalized contribution of each of the three monkeys across a representative subset of 30 trials. The relative contribution of each monkey varied from trial to trial. (D) Fraction of trials that were correctly performed by a dyad (black) as a triad (green), or incorrectly (purple) shifted across the 11 triad experiments. The fraction of total trials with a rewarded outcome in which all three monkeys contributed (green), or those in which two monkeys contributed (black) increased significantly within each week and across sessions whereas the fraction of erroneous/unattempted trials reduced significantly. (E,F) Decoded trajectories and neural data from the triad experiment. (E) Mean X,Y,Z traces produced by individual monkeys shown separately by colour among trials where all monkeys contributed (left column) or when only monkeys M and C contributed (right column). Mean X,Y,Z (the value used to move avatar) shown in black. Distance to target in each axis is 5 screen-cm. When one monkey (monkey K) opted out, the working dyad generated higheramplitude trajectories (Right column, X axis and Z axis) as opposed to when all the members contributed (left column). (F) PETHs aligned on target onset for same trial subsets as in (E) Rows represent individual neurons and colour indicates normalized firing rate (z-score). Neurons from different monkeys marked by colour along right edge (same colours as in (E)). Increased effort by the working dyad also resulted in stronger cortical modulations between the members (right panel) as compared to when all the members contributed (left panel).
Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
7
233
www.nature.com/scientificreports/ Each session started with a period of passive observation during which each monkey watched the computer-generated avatar arm perform the corresponding subtask. A UKF model was fit per monkey and its subtask, and the individual monkey outputs (X, Y for monkey M; Y, Z for monkey C; and X, Z for monkey K) were combined to produce 3D movement of the avatar arm. Put another way, during brain control, pairs of monkeys shared equal control over one of the axes: Monkeys M and K controlled the X-axis, monkeys M and C controlled the Y-axis, and monkeys C and K controlled the Z-axis. Clear behavioural improvements occurred during the brain control epochs over a span of three weeks of training. We observed a significant reduction in target acquisition time (p < 0.02; KS test), trial duration (p < 0.03; KS test) and inter-reward interval (p < 0.002; KS test) between early and late sessions (Fig. 4A). The mean trial duration significantly decreased (4.25 to 3.65 s, 1-way ANOVA: p < 0.05, Fig. 4B panel on the left) over the span of three weeks and the mean reward rate increased from 6 to 10 trials per minute (Fig. 4B, panel on the right). Across 11 sessions, the B3 significantly improved its performance from 20% correct trials in the first session to 78% correct trials in the last session (P < 0.01, 1-way ANOVA with bootstrapping, green+ black bars in Fig. 4D). Table 1 shows the overall mean target acquisition time, target hit rate and the movement precision required for the B3. Due to the addition of the third dimension, movement precision required of B3 was scaled by a factor of 5.5 as compared to B2. However, the target acquisition times increased only by a factor of 1.9 (Dyad MC vs. Triad MCK). Target hit rate remained nearly the same for both paradigms. The increase in target acquisition times indicates the cost of synchronizing an additional monkey into a B3. However, as a result of the design, the B3 performed reach movements in conditions that required greater movement precision, without compromising the hit rate. Even though the relative contribution of each monkey varied from trial to trial (Fig. 4C), the highest rate of success was attained when all three monkeys contributed (Fig. 4D). Furthermore, the percent of total trials with a rewarded outcome in which all three monkeys contributed to the final movement outcome (green bars in Fig. 3D) grew from 7% to 50%, the largest increase of any possible dyad or triad tested in this study. The increase was significant within each week of training (post-hoc Tukey test: P < 0.05) and across weeks (1-way ANOVA with bootstrapping: P < 0.001). The B3 design was resilient to any one individual monkey underperforming in a given trial because any monkey dyad could successfully move the avatar arm to the true target location in 3D space. We observed that the number of rewarded trials for the dyad M&C was lower as compared to the triad but improved with sessions as well (from 12% to 27%, P < 0.01, 1-way ANOVA with bootstrapping). This improvement suggests that the B3 could benefit from the redundancy that was built into the design in the form of shared control. However, when one monkey opted out, the working dyad would need to produce higher-amplitude trajectories (Fig. 4E, right column) as opposed to when all the members contributed (Fig. 4E, left column). Increased effort by the working dyad also resulted in stronger cortical modulations between the members (Fig. 4F, right panel) as compared to when all the members contributed (Fig. 4F, left panel).
Neuron-dropping curves: decoding improves with number of neurons. During B2 and B3
operations, cortical ensembles in each monkey exhibited clear task-related activity during both passive observation (Fig. 5A,C) and brain control modes (Fig. 5B,D). Analysis of these neural signals confirmed that the accuracy of arm movement decoding would improve when VLSBA was recorded and combined from multiple brains (Fig. 5C). This finding extends our previous results, using neuron dropping curve (NDC) analyses, where we showed that decoding accuracy consistently improved when larger neuronal samples were recorded from a single brain4,5. Here, this analysis has been extended to visualize the effect of different relative quantities of two and three brains performing together as part of a Brainet. NDCs were constructed for the decoding of avatar arm position during passive observations and brain control mode for the B2 (Fig. 5E,F) and B3 (Fig. 5G-I). NDCs were plotted as families of curves, where each curve represented decoding accuracy for a neuronal sample composed of a variable-size ensemble from one monkey (Mk1) and a fraction of the full ensemble from the other monkey (Mk2). The NDCs indicated that decoding accuracy benefited from mixing the contributions from different brains as well as the overall neuronal mass. The best accuracy was typically achieved when all neurons from all monkeys were combined (Fig. 5E-I), with the only exception being the Z-axis prediction in B3 (Fig. 5I). When only a small number of Mk1 cells (fewer than 10) were added to Mk2 ensembles during passive observation, decoding accuracy stayed the same or slightly decreased (Fig. 5E-I). However, by scaling to 102 or 103 neurons, this trend shifted such that additional Mk1 neurons added to Mk2 ensembles yielded more accurate predictions. The NDC analysis indicated that Brainet decoding accuracy improved with an increase in the number of neurons. However, a larger total number of neurons was not the sole factor that contributed to the improvement, particularly, the improvement in performance over time. The total number of recorded neurons per monkey did not change considerably throughout the study (Monkey M: 410-501; Monkey C: 196-214; Monkey O: 156-229; Monkey K: 140-156). Yet, we observed a significant improvement in performance (Fig. 2A,B, Fig. 4A,B), as well as improvement in coordinated behaviour (Fig. 2I) and occurrence of concurrent neuronal activity (Fig. 2J). Coordinated behavior was observed even during hand control when the recorded neurons did not influence behavior. Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
8
234
www.nature.com/scientificreports/
Figure 5.â&#x20AC;&#x201A; Neuronal representations during B2 and B3 shared control experiments. Neuronal modulations from monkey M (orange) and C (green) during a 10 second window of passive observation. (A) and brain control (B) Centre and peripheral target onset times are denoted by grey vertical lines. Cortical ensembles in each monkey exhibited clear task-related activity during both passive observation and brain control modes. (C) Passively observed trajectory (black) compared with predicted trajectory using only Monkey M (green), only Monkey C (red), or both (blue) neuronal ensembles. Grey vertical lines from (A) again denote relevant task events. Accuracy of arm movement decoding improved when VLSBA was recorded and combined from multiple brains. (D) Decoded X and Y trajectories from monkey M (black, solid) and monkey C (red), as well as the average of the two (dotted) during 8 second window of brain control experiment. (E,F) Neuron dropping curves (NDC) showing effect of ensemble size of each of the two monkeys on prediction accuracy during B2 passive observation. The number of neurons used from Mk1 marked by x-axis. The percent of Mk2 population used for predictions denoted by colour (see Legend). Accuracy of predictions measured as correlation coefficient r. The decoding accuracy benefited from mixing the contributions from different brains as well as the overall neuronal mass. (G-I) Same NDC analysis as (E,F) except for prediction of X, Y, and Z position during B3 passive observation. Prediction of X (G), Y (H), and Z (I) shown separately. Again (as seen in E-F), the decoding accuracy benefited from mixing the contributions from different brains as well as the overall neuronal mass.
Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
9
235
www.nature.com/scientificreports/ Additionally, improvements in Brainet performance were related to conjoint behavior of the monkeys (as in team work) rather than improvements in individual skills or individual decoding accuracy. All monkeys involved in these experiments had been previously overtrained on the single-BMI task, which makes individual improvements in BMI control an unlikely explanation for the overall improvement of the Brainet performance. Furthermore, we assessed the decoding accuracy in individual monkeys across sessions of manual control (Fig. 6A) and passive observations (Fig. 6B–D). The decoding accuracy showed some fluctuations, which is typical as the motivation levels vary between days of the week. However, there was no systematic trend overall, which contrasts to a steady improvement in Brainet performance over time. These observations suggest that coordinated behavior rather than individual improvements the major factor, which led to better performance.
Discussion
We have successfully developed a shared monkey BMI that utilized chronic, simultaneous VLSBA recordings from pairs or triplets of primate brains to enact conjoint 2D and 3D movements of a monkey avatar arm. This shared BMI, or Brainet, has scaled the conventional definition of BMI from a technology that derives neuronal signals from a single brain to one that can functionally link multiple brains to coordinate movements in space and time in an unsupervised way. Previously, joint motor behaviour was studied extensively in human psychophysics studies 18,19,41. Two individuals who are performing movements can become entrained when they mutually affect each other’s behaviour42, even if they are not explicitly instructed to do so22. In our study, we have shown evidence for both behavioural and neurophysiological entrainment in nonhuman primates that are interacting, indirectly, via a common avatar limb in a virtual environment. The avatar arm provided the same individual action opportunity for all participants involved – also known as simultaneous affordance21,43. Optimal performance has been proposed to occur in conjoint action when participants can monitor others’ movements20 which was facilitated in our experiment through common visual feedback. The various Brainet configurations tested in this study revealed different types of behavioural and neurophysiological adaptations. Individual monkeys in a B2 could optimize the components of the arm movement they controlled directly over the non-controlled components. At the same time, the dyad adapted spatial components (Fig. 3A,B) while maintaining temporal behaviour/brain coordination. These results suggest that neurophysiological adaptations may be context-dependent. Shared BMI control introduced redundancy and computational complexity into the Brainet’s operation by allowing the final motor goal to be achieved despite occasional suboptimal behaviour by an individual monkey. Furthermore, as observed in some recent EEG studies33,34, the NDCs have demonstrated that merging neuronal populations from two/three brains are optimal from the perspective of decoding accuracy. Another important aspect of our Brainet design is the ability to compensate for performance fluctuations in individual members. For example, in our B3 experiments, we often noticed that monkey C compensated for monkey K’s lack of contribution to movements along the y-axis (Fig. 5D). These observations highlight the potential of a Brainet in assisting individual members to overcome their limitations with the help of the partners. In conclusion, this study has successfully established a general paradigm for establishing a closed loop shared BMI, formed by two or more brains, through visual feedback. Shared BMI allowed multiple monkey brains to adapt in an unsupervised manner. Based on these evidences, we propose that primate brains can be integrated into a self-adapting computation architecture capable of achieving a common behavioural goal.
Methods
Subjects and implants. All studies were conducted with approved protocols from Duke University Institutional Animal Care and Use Committee and were in accordance with NIH Guidelines for the Use of Laboratory Animals. Four adult rhesus macaque monkeys (Macaca mulatta) participated in this study. Monkey M and K were chronically implanted with four and six 96 channel arrays, respectively, in bilateral arm and leg areas of M1 and S1 cortex. Monkey C was implanted with eight 96 channel multielectrode arrays in bilateral M1, S1, dorsal premotor (PMd), supplementary motor area (SMA), and posterior parietal cortex (PPC)36. In Monkeys M, C, and K, the multielectrode array consisted of a 4 × 8 (M) or 4 × 10 (C,K) grid of cannulae containing microwire bundles which enable different depths of cortical tissue to be sampled. Microelectrode array design and surgical procedure has been previously described 35,36,44. Monkey O was implanted with four 448-channel multielectrode arrays, one each in left and right hemisphere M1 and S136. In each of these arrays, shafts that contained the microwires were arranged in a 10 × 10 layout. Each shaft consisted of 3-4 microwires with 500 μ m tip spacing or 6 -7 microwires with 1000 μ m tip spacing. For experiments in this study, the recordings in were drawn from M1 (all monkeys) and S1 (only monkey M) cortical areas. Experimental Setup. Prior to experiments of this study, each monkey was trained to perform
centre-out reaching movements using their left arm to move a spring-loaded joystick. For hand control versions of shared control, both monkeys moved the joystick. For all brain control experiments, the upper limbs of each participating monkey were gently restrained using an established technique38.
Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
10
236
www.nature.com/scientificreports/
Figure 6.â&#x20AC;&#x201A; Decoding accuracy in individual monkeys. A UKF decoder trained post hoc predicted monkeyâ&#x20AC;&#x2122;s own movements in every session. (A-C) Upper panels show the performance of the decoder for the x-component of the trajectories and lower panels for y-component of the trajectories. Each data point represents the average performance for the week. Correlation coefficient r was utilized to measure the accuracy of predictions. The performance of the decoder was compared across weeks to monitor the changes in the contribution of the neuronal population during the course of training in the shared control task (A,B): manual control in (A) and passive observation epoch preceding brain control in (B) and the partitioned control task (C). (D) shows the decoder performance for the x-, y- and z- components of the trajectories (top, middle and bottom panels, respectively) in the triad task. Overall, the performance of the decoder remained stable and no consistent trends were observed.
Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
11
237
www.nature.com/scientificreports/ Approximately 50 cm in front of each monkey, at eye level, a computer monitor (26 cm × 30 cm.) displayed the virtual avatar limb9,38 and target objects from a first-person perspective. The virtual environment was created using Motionbuilder software (Autodesk, Inc.). In all configurations of the task, each participating monkey was placed into a different experiment room with no visual or auditory contact with the other monkeys. During experiments, neural activity was recorded from each monkey using commercially available Plexon systems (Plexon, Dallas, TX). Spike timestamp information from each of the four (for dyad experiments) or five (for triad experiments) recording systems was sent over the local network to a master computer for neural decoding steps. The master computer contained the BMI software suite, which controlled task flow, decoder input/output, and reward delivery. Once kinematic predictions were generated by the BMI, the avatar movement commands were sent over UDP packets to Motionbuilder in order to update the position of the avatar limb being displayed on the screens. In all dyad experiments, the two monkeys were provided identical visual feedback. In the triad experiment, the visual feedback was different for each monkey. Dyad Task Paradigms. Shared control task. A circular target (5 cm diameter) appeared in the centre of the screen. The monkeys must hold the avatar arm within the centre target for a random hold time uniformly drawn from the interval 500-1200 ms. Next, the central target disappeared and was replaced by a circular disc (5 cm. in diameter) in one of 4 peripheral locations (left, right, above, or below the centre-point on the screen). The centre of the target disc was 9 cm from the centre. When the monkeys placed the arm within the target such that the cursor (0.5 cm. diameter) at the base of the middle finger (Fig. 1C) was completely within the target, for a hold interval (500 ms), the trial was considered correct and a small juice reward was dispensed. Note that the cursor is shown for illustration purposes; it was not visible to the monkey. Trials where the target was not reached within 10 seconds (hand control) or 15 seconds (brain control) were considered error trials, resulting in no juice and a 1 second timeout period. The position of the avatar arm was exactly the average of the position derived from each of the two monkeys. In hand control, this was the mean (X,Y) joystick position. For brain control experiments, the neural decoder was fit using 5-6 minutes of data collected during passive observation. The neural decoding algorithm used throughout this study was the unscented Kalman filter (UKF)37. During brain control, the avatar hand (X,Y) position was decoded from each monkey’s brain individually, then averaged. Next, the averaged position was used to update the avatar hand position, thus providing the visual feedback to both monkeys. Passive observation. During this task mode, each of the two monkeys in the dyad passively observed identical, pre-programmed reach trajectories on the screen38. The hand moved along the ideal trajectories between centre and peripheral target starting ~500 ms after target onset. The observed hand accelerated and decelerated in a realistic manner when starting and stopping a reach. Juice rewards were dispensed when the hand reached and held inside the displayed target. This mode was used for 6-7 minutes at the beginning of all brain control experiments. Partitioned control task. All sessions began with 5-6 minutes of passive observation trials. In this task, all targets appeared at the diagonal locations: 45°, 135°, 225°, and 315° in Cartesian space. Both the monkeys viewed the hand move along the ideal trajectories between centre and peripheral target. Centre and peripheral target sizes, the distance of the target from the centre were the same as in the previous task. Following passive observation, one UKF model was fit for each monkey. The UKF models for monkeys M and C were used to decode only X or only Y, respectively. Next, the task was switched to brain control. Monkey M was given full control of the X position of the avatar hand, and monkey C was given full control of Y position. Together the dyad had to reach toward the target to complete the trial successfully. The pair had to work together to move the hand to the centre, hold briefly, and then move out to the peripheral target, then hold. The hold time and reward contingencies were retained from the previous task. A total of 8 such experiments were completed over a span of three weeks. Triad Task Paradigm. Monkeys M, C, and K formed a Brainet triad to generate 3D reaching movements of the virtual avatar arm. Once again, centre-out movements were performed. Here the peripheral target was located at one of the 8 equidistant corners of a 3D cube. The projected target was 5cm. in diameter and its centre was located 7 cm from the centre of the screen. Each monkey observed only a 2D projection of this movement on their display screen. Monkeys M, C and K were shown the target from an X-Y, Y-Z, or X-Z reference frame respectively. The monkey viewed the avatar arm always from a first-person perspective. Sessions began with 6-7 minutes of passive observation trials viewed from each monkey’s designated reference frame. Three UKF decoders, one for each monkey, were then initialized and fit on the passive observation data. Each UKF was trained to decode the two dimensions that could be viewed on that monkey’s screen (e.g. X and Y for monkey M). Thus, two UKF models represented X, two represented Y, and two represented Z. Pairs of monkeys with a common dimension were given 50% control of that axis. More specifically, Monkeys M and K each had 50% control of the X axis, Monkeys M and C each Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
12
238
www.nature.com/scientificreports/ had 50% control over the Y axis, and Monkeys C and K each had 50% control of the Z axis. The decoded and averaged X, Y, and Z positions derived from the Brainet triad were then used as a control signal for resultant 3D movements. 2D movement projections were shown as visual feedback to each monkey. For analysis purposes, we considered a monkey to have contributed in a trial if the movements generated by that monkey covered at least 25% of the total distance.
Analyses. All the analyses were performed in Matlab (Mathworks Inc.) using built-in functions,
open-source code, or custom-designed analysis tools developed within the Duke University Center for Neuroengineering. Analysis of neuronal activity. Neurons that were not active for greater than 50% of the session duration were considered quiet neurons and were removed. For the rest, recorded action potential events were counted in bins of 50 ms width and aligned on target onset to obtain PETHs. PETHs were typically calculated for each trial and then were averaged across trials. The average modulation profile for each neuron was normalized by subtracting the mean bin count and dividing by the standard deviation of the cell’s bin count; both values were calculated for raw spike trains prior to any PETH calculations. With this normalization, PETHs express the event-related modulations as a fraction of the overall modulations, or statistically, the z-score. Estimating reaction time lag. The joystick positions (in hand control) or the UKF outputs (in brain control) of the two monkeys were cross-correlated after subtracting the data mean and normalizing to a -1 to 1 scale. The position of the peak of the cross-correlation relative to the ordinate provided an estimate of the time lag between the dyad members. Lag was calculated for every rewarded trial and then averaged across all trials to get an estimate of the average lag between the dyad members in a session. Estimating neural activity lag. Temporal relationships between the neural activity of monkey dyads was determined using a cross-correlation with the mean subtracted and normalized to set the auto-covariance terms equal to 1. This allowed the cross-correlations between different trial conditions to be comparable. To compute the between-monkey lag, the single trial PETHs (spike counts within 50 ms bins during a fixed − 0.5 to 1 s window relative to target appearance) were collected for each neuron on each trial. All pair wise combinations between Monkey 1 cells and Monkey 2 cells were cross-correlated. This yielded a mean cross-correlogram for each trial, as shown in the left panels in Fig. 2G,H. The time of the peak of the cross-correlogram on each trial was identified and defined to be the single-trial lag between the two monkeys. Mean lag for a session was computed by averaging across trials (Fig. 2F). Trial duration. Trial durations were calculated from the target onset time to when the reward was delivered. To avoid potential confounds due to satiation effects, mean/ average trial duration (Fig. 2C,D, Fig. 4B) was estimated during the initial 15 minutes of every session when the motivation was highest. However, trial durations shown in Fig. 2A,B and Fig. 4A include all trials in the session.
k-NN prediction of trial outcome. To predict trial outcome based on level of between-monkey correlation, single trial PETHs were computed for each neuron on each trial. The “pre-target” window was defined as from 700 ms before target appearance until target appearance. The “post-target” window was defined as from 100 to 800 ms after target appearance. The activity profile of each neuron during the specified epoch was normalized and then cross-correlated with the activity profile from each neuron from the other monkey. The cross correlogram for each pairwise neuronal combination was then averaged to produce a single between-monkey cross correlogram for a given trial. The set of single-trial cross correlograms were then used to predict the outcome of their corresponding trials or that of a randomly chosen trial (control). 80% of trials were designated to be training data and 20% of trials were test data. A k-nearest neighbour classifying algorithm (k-NN with k = 5) was fit and then tested by predicting trial outcome on the designated test trials. Performance was quantified in terms of fraction correct prediction. Training/test data was redrawn 10 times. We reported the distribution of the fraction correct values after redrawing the training/test data (Fig. 2I,J). Neuron dropping analysis. Neural activity was recorded and binned into 100 ms bins during passive observation experiments. An unscented Kalman filter (UKF) with three 100 ms past taps of neural activity and two 100 ms future taps of neural activity was used to decode the X and Y position of the observed avatar hand on the screen. UKF prediction of X and Y were computed for one of the monkeys for a wide range of neurons, ranging from 1 to the full ensemble, in increments of 3 (n < 20), 7 (at 20 < n < 100), or 15 (n > 100). The predicted X and Y traces were compared with the actual X and Y traces, the correlation coefficient r was computed for each, and then averaged. In B2 configurations, a second monkey was then added to the analysis. X and Y UKF predictions using 25%, 50%, 75% and 100% of the 2nd monkey’s neurons were then computed. The predicted X and Y were averaged with the X and Y from Monkey 1’s prediction, as was done during on-line experiments. The accuracy of these averaged predictions was quantified using correlation coefficient as well. Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
13
239
www.nature.com/scientificreports/ In B3 configurations, the predictions were generated in a similar way as for B2. The two coordinates controlled by each monkey (X-Y, Y-Z, or X-Z) were likewise decoded from the ensembles of varying sizes using a UKF decoder. For X, Y, and Z, predictions were averaged across the two monkeys who shared each axis. For example, the X predictions from Monkey M and K were computed and averaged to produce one r value for each of the different quantities of Monkey M and K neurons. Fixed percent quantities of the second monkey were added to the first monkey’s ensemble to show the effect of adding neurons from different subjects in a Brainet. Normalized contribution. In a trial, the fraction of the total distance moved by each monkey in the B3 was computed along each axis. A monkey’s normalized contribution (Fig. 4C) was calculated by summing its contributions (in the two directions) and normalizing it by dividing by 3. Negative values (capped at -0.2) indicate movements generated in the direction away from the target. ANOVA on bootstrapped samples. All the trials from a session were resampled several times with replacement and the fraction of error trials, rewarded trials contributed by all the members of the triad, or by a dyad was computed for each sample. A one-way ANOVA was performed on these samples across sessions. F value of the original sample was compared with those of the bootstrapped samples.
References
1. Lebedev, M. A., Nicolelis, M. A. Brain-machine interfaces: past, present and future. Trends Neurosci. 29, 536–546 (2006). 2. Chapin, J. K., Moxon, K. A., Markowitz, R. S., Nicolelis, M. A. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat. Neurosci. 2, 664–670 (1999). 3. Wessberg, J. et al. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408, 361–365 (2000). 4. Carmena, J. M., et al. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol. 1, DOI : 10.1371/journal.pbio.0000042 (2003). 5. Lebedev, M. A., et al. Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface. J. Neurosci. 25, 4681–4693 (2005). 6. Fitzsimmons, N. A., Lebedev, M. A., Peikon, I. D., Nicolelis, M. A. Extracting kinematic parameters for monkey bipedal walking from cortical neuronal ensemble activity. Front. Integr. Neurosci. 3, DOI : 10.3389/neuro.07.003 (2009). 7. House, W. F. Cochlear implants. Ann. Otol. Rhinol. Laryngol. 85 (suppl 27), , 1–93 (1976). 8. Thomson, E. E., Carra, R., Nicolelis, M. A. Perceiving invisible light through a somatosensory cortical prosthesis. Nat. Commun. 4, 1482, DOI : 10.1038/ncomms2497 (2013). 9. O’Doherty, J. E., et al. Active tactile exploration using a brain-machine-brain interface. Nature 479, 228–231 (2011). 10. Nicolelis, M. A. Beyond Boundaries: The New Neuroscience of Connecting Brains with Machines---and How It Will Change Our Lives. Macmillan (2011). 11. Pais-Vieira, M., Lebedev, M. A., Kunicki, C., Wang, J., Nicolelis, M. A. A brain-to-brain interface for real-time sharing of sensorimotor information. Sci. Rep. 3, DOI: 10.1038/srep01319 (2013). 12. Wander, J. D., Rao, R. P. Brain–computer interfaces: a powerful tool for scientific inquiry. Curr. Opin. Neurobiol. 25, 70–75 (2014). 13. Yoo, S. S., Kim, H., Filandrianos, E., Taghados, S. J., Park, S. Non-invasive brain-to-brain interface (BBI): establishing functional links between two brains. PLoS One 8, e60410, DOI: 10.1371/journal.pone.0060410 (2013). 14. Deadwyler, S. A., et al. Donor/recipient enhancement of memory in rat hippocampus. Front. Syst. Neurosci.7, DOI: 10.3389/ fnsys.2013.00120 (2013). 15. Grau, C., et al. Conscious Brain-to-Brain Communication in Humans Using Non-Invasive Technologies. PloS One 9, DOI: 10.1371/journal.pone.0105225 (2014). 16. Shanechi, M. M., Hu, R. C., Williams, Z. M. A cortical–spinal prosthesis for targeted limb movement in paralysed primate avatars. Nat. Commun. 5, DOI: 10.1038/ncomms4237 (2014). 17. Trimper, J. B., Wolpe, P. R., Rommelfanger, K. S. When “I” becomes “We”: ethical implications of emerging brain-to-brain interfacing technologies. Front. Neuroeng. 7, DOI: 10.3389/fneng.2014.00004 (2014). 18. Knoblich, G., Butterfill, S., Sebanz, N. Psychological Research on Joint Action: Theory and Data. In Psychology of Learning and Motivation-Advances in Research and Theory (ed. Ross, B.) 54, 59–101 (Academic Press, 2011). 19. Sebanz, N., Bekkering, H., Knoblich, G. Joint action: bodies and minds moving together. Trends Cogn. Sci. 10, 70–76 (2006). 20. Vesper, C., Butterfill,S., Knoblich, G., Sebanz, N. A minimal architecture for joint action. Neural Netw. 23, 998–1003 (2010). 21. Richardson, M. J., Marsh, K. L., Baron, R. M. Judging and actualizing intrapersonal and interpersonal affordances. J. Exp. Psychol. Hum. Percept. Perform. 33, 845–859 (2007). 22. Richardson, M. J., Marsh, K. L., Isenhower, R. W., Goodman, J. R., Schmidt, R. C. Rocking together: Dynamics of intentional and unintentional interpersonal coordination. Hum. Mov. Sci. 26, 867–891 (2007). 23. Bahrami, B., Olsen, K., Latham, P. E., Roepstorff, A., Rees, G., Frith, C. D. Optimally interacting minds. Science 329, 1081–1085 (2010). 24. Gürkök, H., Nijholt, A., Poel, M., Obbink, M. Evaluating a multi-player brain–computer interface game: Challenge versus coexperience. Entertainment Computing 4, 195–203 (2013). 25. Hasson, U., Landesman, O., Knappmeyer, B., Vallines, I., Rubin, N., Heeger, D. J. Neurocinematics: The neuroscience of film. Projections 2, 1–26 (2008). 26. Kerr, N. L., MacCoun, R. J., Kramer, G. P. Bias in judgment: comparing individuals and groups. Psychol. Rev. 103, 687–719 (1996). 27. Kerr, N. L., Tindale, R. S. Group performance and decision making. Annu. Rev. Psychol. 55, 623–655 (2004). 28. Laughlin, P. R., Bonner, B. L., Miner, A. G. Groups perform better than the best individuals on letters-to-numbers problems. Organizational Behavior and Human Decision Processes 88, 605–620 (Academic Press, 2002). 29. Le Groux, S., et al. Disembodied and collaborative musical interaction in the multimodal brain orchestra. In: Proceedings of the 2010 Conference on New Interfaces for Musical Expression. NIME, Sydney, Australia (2010). 30. Poli, R., Cinel , C., Matran-Fernandez, A., Sepulveda, F., Stoica, A. Towards cooperative brain-computer interfaces for space navigation. In: Proceedings of the 2013 International Conference on Intelligent User Interfaces. ACM (2013).
Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
14
240
www.nature.com/scientificreports/ 31. Poli, R., Valeriani, D., Cinel, C. Collaborative Brain-Computer Interface for Aiding Decision-Making. PloS One 9, e102693 DOI: 10.1371/journal.pone.0102693 (2014). 32. Wang, Y., Jung, T. P. A collaborative brain-computer interface for improving human performance. PLoS One 6, e20422 DOI: 10.1371/journal.pone.0020422 (2011). 33. Yuan, P., Wang, Y., Gao, X., Jung, T-P., Gao, S. A collaborative brain-computer interface for accelerating human decision making. In: Proceedings of the 7th International Conference on Universal Access in Human-Computer Interaction: Design Methods, Tools, and Interaction Techniques for eInclusion, 672–681 (Springer, 2013). 34. Eckstein, M. P., et al. Neural decoding of collective wisdom with multi-brain computing. NeuroImage 59, 94–108 (2012). 35. Nicolelis, M. A., et al. Chronic, multisite, multielectrode recordings in macaque monkeys. Proc. Natl. Acad. Sci. 100, 11041– 11046 (2003). 36. Schwarz, D. A., et al. Chronic, Wireless Recordings of Large Scale Brain Activity in Freely Moving Rhesus Monkeys. Nat. Methods, 11, 670–676 (2014). 37. Li, Z., O’Doherty, J. E., Hanson, T. L., Lebedev, M. A., Henriquez, C. S., Nicolelis, M. A. Unscented Kalman filter for brainmachine interfaces. PLoS One 4, e6243 DOI: 10.1371/journal.pone.0006243 (2009). 38. Ifft, P. J., Shokur, S., Li, Z., Lebedev, M. A., Nicolelis, M. A. A brain-machine interface enables bimanual arm movements in monkeys. Sci. Transl. Med. 5, DOI: 10.1126/scitranslmed.3006159 (2013). 39. Hochberg, L. R., et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171 (2006). 40. Wolpaw, J. R., McFarland, D. J. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. USA. 101, 17849–17854 (2004). 41. Knoblich, G., Sebanz, N. Evolving intentions for social interaction: from entrainment to joint action. Philos. Trans. R. Soc. Lond. B. Biol. Sci. 363, 2021–2031 (2008). 42. Marsh, K. L., Richardson, M. J., Baron, R. M., Schmidt, R. Contrasting approaches to perceiving and acting with others. Ecol. Psychol. 18, 1–38 (2006). 43. Gibson, J. J. The theory of affordances in Perceiving, Acting and Knowing (eds. Shaw, R. & Bransford, J.) 67–82 (Wiley, 1977). 44. Lehew, G., & Nicolelis, M. A. L. State-of-the-Art Microwire Array Design for Chronic Neural Recordings in Behaving Animals in Methods for Neural Ensemble Recording (ed. Nicolelis, M.) Ch. 1 (CRC Press, 2008).
Acknowledgements
We are grateful for the assistance from Gary Lehew and Jim Meloy for the design and production of the multielectrode arrays, from Dragan Dimitrov and Laura Oliveira for performing the electrode implantation surgery, from Zheng Li for expertise with the BMI software, from Tamara Phillips for animal care and logistical matters, and from Terry Jones and Susan Halkiotis for administrative assistance and preparation of the manuscript. The work in this project is funded by NIH DP1MH099903 and the BIAL Foundation Grant 199/12.
Author Contributions
M.A.L.N conceived the idea of a Brainet. A.R., P.J.I., M.A.L., and M.A.L.N designed the experiments. K.Z.Z. assisted A.R., P.J.I. in training the animals. K.Z.Z., M.P.V., Y.W.B. assisted A.R., P.J.I. in performing the experiments, and collecting data. P.J.I assisted A.R. in analysing the data. A.R., P.J.I., M.A.L., and M.A.L.N. wrote the manuscript.
Additional Information
Supplementary information accompanies this paper at http://www.nature.com/srep Competing financial interests: The authors declare no competing financial interests. How to cite this article: Ramakrishnan, A. et al. Computing Arm Movements with a Monkey Brainet. Sci. Rep. 5, 10767; doi: 10.1038/srep10767 (2015). This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/
Scientific Reports | 5:10767 | DOI: 10.1038/srep10767
15
241
www.nature.com/scientificreports
OPEN
Building an organic computing device with multiple interconnected brains
received: 03 March 2015 accepted: 09 June 2015 Published: 09 July 2015
Miguel Pais-Vieira1, Gabriela Chiuffa1, Mikhail Lebedev1,4, Amol Yadav2 & Miguel A. L. Nicolelis1,2,3,4,5 Recently, we proposed that Brainets, i.e. networks formed by multiple animal brains, cooperating and exchanging information in real time through direct brain-to-brain interfaces, could provide the core of a new type of computing device: an organic computer. Here, we describe the first experimental demonstration of such a Brainet, built by interconnecting four adult rat brains. Brainets worked by concurrently recording the extracellular electrical activity generated by populations of cortical neurons distributed across multiple rats chronically implanted with multi-electrode arrays. Cortical neuronal activity was recorded and analyzed in real time, and then delivered to the somatosensory cortices of other animals that participated in the Brainet using intracortical microstimulation (ICMS). Using this approach, different Brainet architectures solved a number of useful computational problems, such as discrete classification, image processing, storage and retrieval of tactile information, and even weather forecasting. Brainets consistently performed at the same or higher levels than single rats in these tasks. Based on these findings, we propose that Brainets could be used to investigate animal social behaviors as well as a test bed for exploring the properties and potential applications of organic computers.
After introducing the concept of brain-to-brain interfaces (BtBIs)1, our laboratory demonstrated experimentally that BtBIs could be utilized to directly transfer tactile or visuomotor information between pairs of rat brains in real time2. Since our original report, other studies have highlighted several properties of BtBIs1,3, such as transmission of hippocampus representations between rodents4, transmission of visual information between a human and a rodent5, and transmission of motor information between two humans6,7. Our lab has also shown that Brainets could allow monkey pairs or triads to perform cooperative motor tasks mentally by inducing, accurate synchronization of neural ensemble activity across individual brains8. In addition to the concept of BtBIs, we have also suggested that networks of multiple interconnected animal brains, which we dubbed Brainet1, could provide the core for a new type of computing device: an organic computer. Here, we tested the hypothesis that such a Brainet could potentially exceed the performance of individual brains, due to a distributed and parallel computing architecture1,8. This hypothesis was tested by constructing a Brainet formed by four interconnected rat brains and then investigating how it could solve fundamental computational problems (Fig. 1A–C). In our Brainet, all four rats were chronically implanted with multielectrode arrays, placed bilaterally in the primary somatosensory cortex (S1). These implants were used to both record neural ensemble electrical activity and transmit virtual tactile information via intracortical electrical microstimulation (ICMS). Once animals recovered from the implantation surgery, the resulting 4-rat Brainets (Fig. 1) were tested in a variety of ways. Our central 1
Department of Neurobiology, Duke University, Durham, North Carolina 27710. 2Department of Biomedical Engineering, Duke University, Durham, North Carolina 27710. 3Department of Psychology and Neuroscience, Duke University, Durham, North Carolina 27710. 4Duke Center for Neuroengineering, Duke University, Durham, North Carolina 27710. 5Edmond and Lily Safra International Institute for Neuroscience of Natal, Natal, Brazil. Correspondence and requests for materials should be addressed to M.A.L.N. (email: nicoleli@neuro.duke.edu) Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
1
242
www.nature.com/scientificreports/
Figure 1.â&#x20AC;&#x201A; Experimental apparatus scheme for a Brainet computing device. A) A Brainet of four interconnected brains is shown. The arrows represent the flow of information through the Brainet. Inputs were delivered as simultaneous ICMS patterns to the S1 cortex of each rat. Neural activity was then recorded and analyzed in real time. Rats were required to synchronize their neural activity with the remaining of the Brainet to receive water B) Inputs to the Brainet were delivered as ICMS patterns to the left S1, while outputs were calculated using the neural responses recorded from the right S1. C) Brainet architectures were set to mimic hidden layers of an artificial neural network. D) Examples of perievent histograms of neurons after the delivery of ICMS.
goal was to investigate how well different Brainet architectures could be employed by the four rats to collaborate in order to solve a particular computational task. Different Brainet designs were implemented to address three fundamental computational problems: discrete classification, sequential and parallel
Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
2
243
www.nature.com/scientificreports/
Figure 2. The Brainet can synchronize neural activity. A) The different colors indicate the different manipulations used to study synchronization across the network. During the pre-session, rats were tested for periods of spurious neural synchronization. No ICMS or rewards were delivered here. During sessions, rats were tested for increased neural synchronization due to detection of the ICMS stimulus (red period). Successful synchronization was rewarded with water. During the post session, rats were tested for periods of neural synchronization due to the effects of reward (e.g. continuous whisking/licking). Successful synchronization was rewarded with water, but no ICMS stimulus was delivered. B) Example of neuronal activity across the Brainet. After the ICMS there was a general tendency for neural activity to increase. Periods of maximum firing rate are represented in red. C) The performance of the Brainet during sessions was above the pre-sessions and post-sessions. Also, delivery of ICMS alone or during anesthetized states also resulted in poor performances. ** and *** indicate P < 0.01 and P < 0.0001 respectively. D) Overall changes in R values in early and late sessions show that improvements in performances were accompanied by specific changes in the periods of synchronized activity. E) Example of a synchronization trial. The lower panels show, in red, the neural activity of each rat and, in blue, the average of neural activity for the remaining of the Brainet. The upper panels depict the R value for the correlation coefficient between each rat and the remaining of the Brainet. There was an overall tendency for the Brainet to correlate in the beginning of the test period.
computations, and memory storage/retrieval1. As predicted, we observed that Brainets consistently outperformed individual rats in each of these tasks.
Results
All experiments with 4-rat Brainets were pooled from a sample of 16 animals that received cortical implants from which we could simultaneously record the extracellular activity from 15–66 S1 neurons per Brainet (total of 2,738 neurons recorded across 71 sessions).
Brainet for neural synchronization. Rats were water deprived and trained on a task that required
them to synchronize their neural activity after an ICMS stimulus. A total of six rats were used in 12 sessions to run this first experiment. As depicted in Fig. 1A–C, the processing chain in these experiments started with the simultaneous delivery of an ICMS pattern to one of the S1 cortices of all subjects, then processing of tactile information with a single-layer Brainet, followed by generation of the system output by the contralateral S1 cortex of each animal. Each trial was comprised of four epochs: waiting (baseline), ICMS delivery, test, and reward. ICMS patterns (20 pulses at 22–26 Hz) were unilaterally delivered to the S1 of each rat. Neuronal responses to the ICMS were evaluated during the test period when S1 neuronal ensemble activity was sampled from the hemisphere contralateral to the stimulation site (Figs. 1D and 2A–E) (Fig. 2A–E). Rats were rewarded if their cortical activity became synchronized during the test period. The correlation coefficient R was used as the measure of global Brainet synchrony. Thus, R measured the linear correlation between the normalized firing rate of all neurons in a given rat and the average normalized firing rate for all neurons recorded in the remaining three rats (see Methods for details). If at least three rats presented R values greater or equal to 0.2, a trial was considered successful, and all four rats were rewarded. Otherwise no reward was given to any rat. Two conditions served as
Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
3
244
www.nature.com/scientificreports/ controls: the pre-session, where no ICMS or water reward were delivered, and the post-session, where no ICMS was delivered but rats were still rewarded if they satisfied the correlation criterion (Fig. 2A). Behaviorally, rats remained mostly calm or immobile during the baseline period. After the ICMS pattern was delivered simultaneously to all animals, rats typically displayed periods of whisking and licking movements. A sample of S1 neuronal population activity during this period is shown in Fig. 2B (also see Fig. 1D for examples of individual neurons perievent histograms). Typically, after the delivery of ICMS, there was a sharp decrease in the neuronal firing rate of the neurons (~20 ms), followed by a sudden firing rate increase (~100 ms). While the main measure of accuracy for this task was the degree in which cortical neuronal populations fired synchronously, it is important to emphasize that the build up of these ensemble firing patterns depended highly on how single S1 neurons modulated their firing rate as a result of electrical microstimulation. Thus, ICMS served as a reset signal that allowed rats to synchronize their neural activity to the remaining network (Fig. 2D,E). Note that, in this task, rats were not exchanging neural information through the BtBI. Instead the timing of the ICMS stimulus, the partial contact allowed through the Plexiglas panels, and the reward were the only sources of information available for rats to succeed in the task. As the Brainet consistently exhibited the best performance during the first trials, we focused our subsequent analysis on the first 30-trial block of each session. Overall, this 4-rat Brainet was able to synchronize the neural activity of the constituent rats significantly above Pre-Session (Brainet: 57.95 ± 2%; Pre-Sessions: 45.95 ± 2%; F2,24 = 10.99; P = 0.0004; Dunnett’s test: P < 0.001) and Post-Session levels (46.41 ± 2%; Dunnett’s test: P < 0.01; Fig. 2C). Over approximately 1.5 weeks (total of 12 sessions), this Brainet gradually improved its performance, from 54.76 ± 3.16% (mean ± standard error; the first 6 days) to 61.67 ± 3.01% correct trials (the last 6 days; F1,2 = 5.770, P = 0.0175 for interaction; Bonferroni post hoc comparisons: pre vs session initial start P > 0.05; pre vs session end P < 0.01; session vs post start P > 0.05; session vs post end P < 0.001). The high fidelity of information transfer in this Brainet configuration was further confirmed by the observation that the performance of individual rats reached 65.28 ± 1.70%. In other words, a 4-rat Brainet was capable of maintaining a level of global neuronal synchrony across multiple brains that was virtually identical to that observed in the cortex of a single rat (Brainet level = 61.67 ± 3.07%; Man-Whitney U = 58.0; P = 0.4818, n.s.). A comparison of correlation values between sessions from the first (n = 6) and the last days (n = 6) further demonstrated that daily training on this first task resulted in a statistically significant increase in correlated cortical activity across rats, centered between 700 ms and 1000 ms of the testing period (F = 1.622; df = 1.49; P = 0.0043, Fig. 2D). The lower panel of Fig. 2E shows the normalized firing rate for each rat (in red) and for the remaining Brainet (in blue) in one trial. The upper panels show R value changes for the correlation between neuronal activity in each rat and the remaining Brainet. Notice the overall tendency for most rats to increase the R values soon after the delivery of the ICMS pattern (T = 0 seconds). To determine if reward was mandatory for the correlation to emerge in the Brainet, we performed three control sessions with awake animals receiving ICMS (but no reward). The performances dropped to levels below chance (performance: 30.67 ± 3.0%; see Fig. 2C). Further, in another three sessions where ICMS was applied to anesthetized animals, the Brainet performed close to chance levels again (performance: 38.89 ± 4.8%; see Fig. 2C). These results demonstrated that the Brainet could only operate above chance in awake behaving rats in which there was an expectation for reward. After determining that the Brainet could learn to respond to an ICMS input by synchronizing its output across multiple brains, we tested whether such a collective neuronal response could be utilized for multiple computational purposes. These included discrete stimulus classification, storage of a tactile memory, and, by combining the two former tasks, processing of multiple tactile stimuli.
Brainet for stimulus classification. Initially, we trained our 4-rat Brainet to discriminate between two ICMS patterns (Fig. 3A,B, 8 sessions in 4 rats). The first pattern (Stimulus 1) was the same as in the previous experiment (20 pulses at 22–26 Hz), while the second (Stimulus 2) consisted of two separate bursts of four pulses (22–26 Hz). The Brainet was required to report either the presence of Stimulus 1 with an increase in neuronal synchrony across the four rat brains (i.e. R ≥ 0.2 in at least three rats), or Stimulus 2 by a decrease in synchrony (i.e., R < 0.2 in at least three rats). By requiring that the delivery of Stimulus 2 be indicated through a reduction in neuronal synchronization, we further ensured that the Brainet performance was not based on a simple neural response to the ICMS pattern. As in the previous experiment, Stimulus 1 served as a reset signal that allowed rats to synchronize their neural activity to the remaining network. Meanwhile, because Stimulus 2 was much shorter than Stimulus 1, it still induced neural responses in several S1 neurons (Fig. 3B), but its effects were less pronounced and not as likely to induce an overall neural synchronization across the Brainet (see Supplementary Figure 1). Following training, the Brainet reached an average performance of 61.24 ± 0.5% correct discrimination between Stimuli 1 and 2, which was significantly above No-ICMS sessions (52.97 ± 1.1%, n = 8 sessions; Brainet vs No-ICMS: Dunn’s test: P < 0.01). Moreover, using this more complex task design, the Brainet outperformed individual rats (55.86 ± 1.2%) (Kruskal-Wallis statistic = 10.87, P = 0.0044; Brainet vs Individual Rats; Dunn’s test: P < 0.05; also see Fig. 3C). To improve the overall performance of this 4-rat Brainet, we implemented an adaptive decoding algorithm that analyzed the activity of each neuron in each specific bin separately, and then readjusted Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
4
245
www.nature.com/scientificreports/
Figure 3. The Brainet can both synchronize and desynchronize neural activity. A) Architecture of a Brainet that can synchronize and desynchronize its neural activity to perform virtual tactile stimuli classification. Different patterns of ICMS were simultaneously delivered to each rat in the Brainet. Neural signals from all neurons from each brain were analyzed and compared to the remaining rats in the Brainet. The Brainet was required to synchronize its neural activity to indicate the delivery of a Stimulus 1 and to desynchronize its neural activity to indicate the delivery of a Stimulus 2. B) Example of perievent histograms of neurons for ICMS Stimulus 1 and 2. C) The Brainet performance was above No-ICMS sessions, and above individual rats’ performances. * indicates P < 0.05; ** indicates P < 0.01; n.s. indicates non significant.
the neuronal weights following each trial (see Methods for details). Figure 4A depicts this Brainet architecture. Notice the different weights for each of the individual neurons (represented by different shades of grey), reflecting the individual accuracy in decoding the ICMS pattern. Figure 4B illustrates a session in which all four rats contributed to the overall decoding of the ICMS stimuli (the red color indicates periods of maximum decoding). Using this approach, we increased both the overall Brainet performance (74.18 ± 2.2% correct trials; n = 7 rats in 12 sessions) and the number of trials performed (64.17 ± 6.2 trials) in each session. The neuronal ensembles of this Brainet included an average of 50 ± 43 neurons (mean ± standard error). Figure 4C depicts the improved performance of the Brainet compared to that of the No-ICMS sessions (54.34 ± 2.2% correct trials, n = 11 sessions) and the performance of individual rats (61.28 ± 1.1% correct trials, F = 26.34; df = 2, 56; P < 0.0001; Bonferroni post hoc comparisons; Brainet vs No-ICMS: P < 0.0001; Brainet vs Individual rats P < 0.0001). When rats were anesthetized (2 sessions in five rats) or trial duration was reduced to 10 s (i.e. almost only comprising the ICMS and the test period – 2 sessions in four rats), the Brainet’s performance dropped sharply (anesthetized: 60.61 ± 2.8% correct; short time trials: 62.57 ± 3.14%). Once again, this control experiment indicated that the Brainet operation was not solely dependent on an automatic response to the delivery of an ICMS. Next, we investigated the dependence of the Brainet’s performance on the number of S1 neurons recorded simultaneously. Figure 4D depicts a neuron dropping curve illustrating this effect. According to this analysis, Brainets formed by larger cortical neuronal ensembles performed better than those containing just a few neurons9. The difference between the Brainet classification of the two stimuli during regular sessions and during those in which no-ICMS was delivered is shown in Fig. 4E. During the regular sessions stimulus classification remained mostly in the quadrants corresponding to the stimuli delivered (lower left and upper right quadrants), while during the No-ICMS sessions the 4-rat Brainet trial classification was evenly distributed across all quadrants. As different rats were introduced to the Brainet, we also compared how neuronal ensemble encoding in each animal changed during initial and late sessions (the first three versus the remaining days).
Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
5
246
www.nature.com/scientificreports/
Figure 4. Brainet for discrete classification. A) Architecture of a Brainet for stimulus classification. Two different patterns of ICMS were simultaneously delivered to each rat in the Brainet. Neural signals from each individual neuron were analyzed separately and used to determine an overall classification vote for the Brainet. B) Example of a session where a total of 62 neurons were recorded from four different animals. Deep blue indicates poor encoding, while dark red indicates good encoding. Although Rat 3 presented the best encoding neurons, all rats contributed to the network’s final classification. C) Performance of Brainet during sessions was significantly higher when compared to the No-ICMS sessions. Additionally, because the neural activity is redundant across multiple brains, the overall performance of the Brainet was also higher than in individual brains. *** indicates P < 0.0001. D) Neuron dropping curve of Brainet for discrete classification. The effect of redundancy in encoding can be observed in the Brainet as the best encoding cells from each session are removed. E) The panels depict the dynamics of the stimulus presented (X axis: 1 or 2) and the Brainet classifications (Y axis: 1 to 2) during sessions and No-ICMS sessions. During regular sessions, the Brainet classifications mostly matched the stimulus presented (lower left and upper right quadrants). Meanwhile, during No ICMS sessions the Brainet classifications were evenly distributed across all four quadrants. The percentages indicate the fraction of trials in each quadrant (Stimulus 1, vote 1 not shown). F) Example of an image processed by the Brainet for discrete classification. An original image was pixilated and each blue or white pixel was delivered as a different ICMS pattern to the Brainet during a series of trials (Stimulus 1 - white; Stimulus 2 - blue). The left panel shows the original input image and the right panel shows the output of the Brainet.
Overall, there was a significant increase in ICMS encoding (initial: 59.67 ± 1.4%, late: 65.08 ± 1.2%, Mann-Whitney U = 281.0, P = 0.0344) and, to a smaller extent, in the correlation coefficients between neural activity of the different animals (initial: 0.1831 ± 0.007, late: 0.2028 ± 0.005, Mann-Whitney U = 275.0, P = 0.0153) suggesting that improvements in Brainet performances were accompanied by cortical plasticity in the S1 of each animal. To demonstrate a potential application for this stimulus discrimination task, we tested whether our Brainet could read out a pixilated image (N = 4 rats in n = 4 sessions) using the same principles demonstrated in the previous two experiments. Blue and white pixels were converted into binary codes (white - Stimulus 1 or blue - Stimulus 2) and then delivered to the Brainet over a series of trials. The right panel of Fig. 4F shows that a 4-rat Brainet was able to capture the original image with good accuracy (overall 87% correct trials) across a period of four sessions.
Brainet for storage and retrieval of tactile memories. To test whether a 3-rat Brainet could store and retrieve a tactile memory, we sent an ICMS stimulus to the S1 of one rat and then successively transferred the information decoded from that rat’s brain to other animals, via a BtBI, over a block of four trials. To retrieve the tactile memory, the information traveling across different rat brains was delivered, at the end of the chain, back to the S1 cortex of the first rat for decoding (Fig. 5A). Opaque panels were placed between the animals, and cortical neural activity was analyzed for each rat separately. The architecture of inputs and outputs of the 3-rat Brainet’s is shown in Fig. 5A, starting from the bottom shelf and Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
6
247
www.nature.com/scientificreports/
Figure 5. A Brainet for storage and retrieval of tactile memories. A) Tactile memories encoded as two different ICMS stimuli were stored in the Brainet by keeping information flowing between different nodes (i.e. rats). Tactile information sent to the first rat in Trial 1 (‘Stimulus Decoding’), was successively decoded and transferred between Rats 2 and 3, and again transferred to Rat 1, across a period of four trials (memory trace in red). The use of the brain-to-brain interface between the nodes of the network allowed accurate transfer of information. B) The overall performance of the Brainet was significantly better than the performance in the No-ICMS sessions and better than individual rats performing 4 consecutive correct trials. In this panel, * indicates P < 0.05 and *** indicates P < 0.001. C) Neuron dropping curve of Brainet for storage and retrieval of memories. D) Example of session with multiple memories (each column) processed in blocks of four trials (each row). Information flows from the bottom (Stimulus delivered) towards the top (Trials 1–4). Blue and red indicate Stimulus 1 or 2 respectively. Correct tactile memory traces are columns which have a full sequence of trials with the same color (see blocks: 3, 5, 7 and 9). In this panel, * indicates an incorrect trial.
progressing to the top one. The experiment started by delivering one of two different ICMS stimuli to the S1 of the input rat (from now on referred to as Rat 1) during the first trial (Trial 1). Neuronal ensemble activity sampled from Rat 1 was then used to decode the identity of the stimulus (either Stimulus 1 or 2). Once the stimulus identity was determined, a new trial started and a BtBI was employed to deliver a correspondent ICMS pattern to Rat 2, defining Trial 2 of the task. In this arrangement, the BtB link between Rat 1 and Rat 2 served to store the pattern (Pattern Storage I). Next, neuronal ensemble activity was recorded from the S1 of Rat 2. In the third trial, it was Rat 3’s turn to receive the tactile message
Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
7
248
www.nature.com/scientificreports/ (Pattern Storage II) decoded from the neural ensemble activity of Rat 2, via an ICMS mediated BtB link. During the fourth and final trial, Rat 1 received the message decoded from the neural activity of Rat 3. Using this Brainet architecture, the memory of a tactile stimulus could only be recovered if the individual BtB communication links worked correctly in all four consecutive trials. The chance level for this operation was 6.25%. Under these conditions, this Brainet was able to retrieve a total of 35.37 ± 2.2% (9 sessions in 9 rats) of the tactile stimuli presented to it (Kruskall Wallis statistic = 14.89; P = 0.0006, Fig. 5B), contrasting with 7.91 ± 6.5% in No-ICMS sessions (n = 5 sessions; Dunn’s test: P < 0.001). For comparison purposes, individual rats performed the same four-trial task correctly in only 15.63 ± 2.1% of the trials. This outcome was significantly lower than a 3-rat Brainet (Dunn’s test: P < 0.001). As in the previous experiments, larger neuronal ensembles yielded better encoding (Fig. 5C). As an additional control, rats that were not processing memory related information in a specific trial (e.g. Rats 2 and 3 during the Stimulus Decoding Stage in Rat 1) received Stimulus 1 or Stimulus 2, randomly chosen. Thus, in every single trial all rats received some form of ICMS, but only the information gathered from a specific rat was used for the overall tactile trace. The colored matrix in Fig. 5D illustrates a session in which a tactile trace developed along the 3-rat Brainet. A successful example of information transfer and recovery is shown in the third block of trials (blue column on the left). The figure shows that the original stimulus (Stimulus 1 – bottom blue square) was delivered to the S1 of Rat 1 in the first trial. This stimulus was successfully decoded from Rat 1’s neural activity, as shown by the presence of the blue square immediately above it (Trial 1 – Stimulus Decoding). In Trial 2 (Pattern Storage I), Stimulus 2 was delivered, via ICMS to the S1 of Rat 2, and again successfully decoded (as shown by the blue square in the center). Then, in Trial 3 (Pattern Storage II), the ICMS pattern delivered to Rat 3 corresponded to Stimulus 1, and the decoding of S1 neural activity obtained from this animal still corresponded to Stimulus 1, as shown by the blue square. Lastly, in Trial 4 (Stimulus Recovery), Rat 1 received an ICMS pattern corresponding to Stimulus 1 and its S1 neural activity still encoded Stimulus 1 (blue square). Thus, in this specific block of trials, the original tactile stimulus was fully recovered since all rats were able to accurately encode and decode the ICMS pattern received. Similarly, columns 5, 7, and 9 also show blocks of trials where the original tactile stimulus (in these cases Stimulus 2, red square) was accurately encoded and decoded by the Brainet. Alternatively, columns with an asterisk on top (e.g. 1 and 8) indicate incorrect blocks of trials. In these incorrect blocks, the stimulus delivered was not accurately encoded in the brain of at least one rat belonging to the Brainet (e.g. rat 3 in block 1).
Brainet for sequential and parallel processing. Lastly, we combined all the processing abilities demonstrated in the previous experiments (discrete tactile stimulus classification, BtB interface, and tactile memory storage) to investigate whether Brainets would be able to use sequential and parallel processing to perform a tactile discrimination task (N = 5 rats in N = 10 sessions). For this we used blocks of two trials where tactile stimuli were processed according to Boolean logic10 (Fig.6A–B). This means that in each trial there was a binary decision tree (i.e. two options encoded as Stimulus 1 or 2). In the first trial, two different tactile inputs were independently sent to two dyads of rats (Dyad 1: Rat 1-Rat 2; Dyad 2: Rat 3-Rat 4; bottom of Fig. 6A). In the next trial, the tactile stimuli decoded by the two dyads were combined and transmitted, as a new tactile input, to a 4-rat Brainet. Upon receiving this new stimulus, the Brainet was in charge of encoding a final solution (i.e. identifying Stimulus 3 or 4, see Supplementary Figure 2). As shown at the bottom of Fig. 6A, odd trials were used for parallel processing, i.e. each of two rat dyads independently received ICMS patterns, while neural activity was analyzed and the original tactile stimulus decoded (i.e. Stimulus 1 or 2). Then, during even trials (Fig. 6A, top), ICMS was used to encode a second layer of patterns, defined as Stimulus 3 and Stimulus 4. Note that ICMS Stimuli 3 and 4 were physically identical to Stimuli 2 and 1 respectively; however, because the stimuli delivered in the even trials were contingent on the results of the odd trials, we employed a different nomenclature to identify them. The decision tree (i.e. truth table) used to calculate the stimuli for the even trials is shown in the colored matrix at the center of Fig. 6A. The matrix shows that, if both dyads encoded the same tactile stimulus in the odd trial (Stimulus 1-Stimulus 1, or Stimulus 2-Stimulus 2; combinations with blue encasing), the ICMS delivered to the entire Brainet in the even trial corresponded to Stimulus 4. Otherwise, if the tactile stimulus decoded from each rat dyad in the odd trial was different (Stimulus 1-Stimulus 2, or Stimulus 2-Stimulus 1; combinations with red encasing), the ICMS delivered to the entire Brainet in the even trial corresponded to Stimulus 3. As such, the ICMS pattern delivered in even trials was the same for the whole Brainet (i.e. all four rats). At the end of each even trial, the stimulus decoded from the combined neuronal activity of the four brain ensemble (top of Fig. 6A) defined the final output of the Brainet. Chance level was set at 12.5%. Overall, this Brainet performance was much higher than chance level or No-ICMS sessions (Brainet: 45.22 ± 3.4%, n = 10 sessions) significantly above No-ICMS sessions (n = 5 sessions) (No-ICMS: 22.79 ± 5.4%; Kruskal-Wallis statistic = 7.565, P = 0.0228; Dunn’s test: P < 0.05 Fig. 6C). Additionally, the Brainet also outperformed each individual rat (groups of three consecutive trials: 30.25 ± 3.0%; Dunn’s test: P < 0.05). As our last experiment, we tested whether a 3-rat Brainet could be used to classify meteorological data (see Methods for details). Again, the decision tree included two independent variables in the odd Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
8
249
www.nature.com/scientificreports/
Figure 6. A Brainet for parallel and sequential processing. A) Architecture of a network for Parallel and Sequential processing. Information flows from the bottom to the top during the course of two trials. In first trial, odd trial for parallel processing, Dyad 1 (Rat 1-Rat 2) received one of two ICMS patterns, and Dyad 2 (Rat 3-Rat 4) received independently one of two ICMS patterns. During Trial 2, even trial for sequential processing, the whole Brainet received again one of two ICMS patterns. However, the pattern delivered in the even trial was dependent on the results of the first trial and was calculated according to the colored matrix presented. As depicted by the different encasing of the matrix (blue or red), if both dyads encoded the same stimulus in the odd trial (Stimulus 1-Stimulus1 or Stimulus 2-Stimulus 2), then the stimulus delivered in the even trial corresponded to Stimulus 3. Otherwise, if each dyad encoded a different stimulus in the odd trial (Stimulus1-Stimulus 2 or Stimulus 2-Stimulus 1), then the stimulus delivered in even trial was Stimulus 4. Each correct block of information required three accurate estimates of the stimulus delivered (i.e. encoding by both dyads in the even trial, as well as the whole Brainet in the odd trial). B) Example of session with sequential and parallel processing. The bottom and center panel show the dyads processing the stimuli during the odd trials (parallel processing), while the top panel shows the performance of the whole Brainet during the even trials. In this panel, * indicates an incorrect classification. C) The performance of the Brainet was significantly better than the performance during the No-ICMS sessions and above the performance of individual rats performing blocks of 3 correct trials. In this panel, * indicates P < 0.05.
trials and a dependent variable in the even trials (see Supplementary Figure 3). Figure 7A illustrates how Boolean logic was applied to convert data from an original weather forecast model . In the bottom panel, the yellow line depicts continuous changes in temperature occurring during a period of 10 hours. Periods where the temperature increased were transferred to the Brainet as Stimulus 1 (see arrows in periods between 0 and 4 hours), whereas periods where the temperature decreased were transferred as Stimulus 2 (see arrows in periods between 6 and 10 hours). The middle panel of Fig. 7A illustrates changes in barometric pressure (green line). Again, periods where the barometric pressure increased were translated as Stimulus 1 (e.g. between 1-2 hours), while periods where the barometric pressure decreased were translated as Stimulus 2 (e.g. 3–5 hours). Both Stimulus 1 and 2 were delivered to a Brainet during odd trials; changes in temperature were delivered to Rat 1 alone, while changes in barometric pressure were delivered to Rats 2 and 3. As in the previous experiment, Stimuli 3 and 4 were physically similar to Stimuli 1 and 2. In even trials, increases and decreases in the probability of precipitation (top panel Fig. 7A) were calculated as follows: an increase in temperature (Stimulus 1; Rat 1) combined with a decrease in barometric pressure (Stimulus 2; Rats 2 and 3) was transferred to even trials as an increase in the probability of precipitation Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
9
250
www.nature.com/scientificreports/
Figure 7. Parallel and sequential processing for weather forecast A) Each panel represents examples of the original data, reflecting changes in temperature (lower panel), barometric pressure (center panel), and probability of precipitation (upper panel). The arrows represent general changes in each variable, indicating an increase or a decrease. On the top of each panel is represented the ICMS pattern that resulted from each arrow presented. B) Lower and center panels show trials where different rats of the Brainet (Rat 1 lower panel, and Rats 2-3 center panel) processed the original data in parallel. Specifically, Rat 1 processed temperature changes and Rats 2-3 processed barometric pressure changes. The upper panel shows the Brainet processing changes in the probability of precipitation (Rats 1–3) during the even trials. * indicates trials where processing was incorrect.
(i.e. a Stimulus 4), whereas any other combination was transferred as Stimulus 3, and associated with a decrease in precipitation probability. This specific combination of inputs was used because it reflects a common set of conditions associated with early evening spring thunderstorms in North Carolina. Overall, our 3-rat Brainet predicted changes in the probability of precipitation with 41.02 ± 5.1% accuracy which was much higher than chance (No-ICMS: 16.67 ± 8.82%; n = 3 sessions; t = 2.388, df = 4; P = 0.0377) (also see Fig. 7B).
Discussion
In this study we described different Brainet architectures capable of extracting information from multiple (3-4) rat brains. Our Brainets employed ICMS based BtBs combined with neuronal ensemble recordings to simultaneously deliver and retrieve information to and from multiple brains. Multiple BtBIs were used to construct some of our Brainet designs. Our experiments demonstrated that several Brainet architectures can be employed to solve basic computational problems. Moreover, in all cases analyzed the Brainet performance was equal or superior to that of an individual brain. These results provide a proof of concept for the possibility of creating computational engines composed of multiple interconnected animal brains. Previously, Brainets have incorporated only up to two subjects exchanging motor or sensory information2, or up to three monkeys that collectively controlled the 3D movements of a virtual arm8. These studies provided two major building blocks for Brainet design: (1) information transfer between individual brains, and (2) collaborative performance among multiple animal brains. Here, we took advantage of these building blocks to demonstrate more advanced Brainet processing by solving multiple Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
10
251
www.nature.com/scientificreports/ computational problems, which included discrete classification, image processing, storage and retrieval of memories, and a simplified form of weather forecasting1,2,8. All these computations were dependent on the collective work of cortical neuronal ensembles recorded simultaneously from multiple animal brains working towards a common goal. One could argue that the Brainet operations demonstrated here could result from local responses of S1 neurons to ICMS. Several lines of evidence suggest that this was not the case. First, we have demonstrated that animals needed several sessions of training before they learned to synchronize their S1 activity with other rats. Second, the decoding for individual neurons in untrained rats was close to chance levels. Third, attempts to make the Brainet work in anesthetized animals resulted in poor performance. Fourth, network synchronization and individual neuron decoding failed when animals did not attend to the task requirements and engaged in grooming instead. Fifth, removing the reward contingency drastically reduced the Brainet performance. Sixth, after we reduced trial duration, the decoding from individual neurons dropped to levels close to chance. Altogether, these findings indicate that optimal Brainet processing was only attainable in fully awake, actively engaged animals, with an expectation to be rewarded for correct performance. These features are of utmost importance since they allowed Brainets to retain the computational aptitudes of the awake brain11 and, in addition, to benefit from emergent properties resulting from the interactions between multiple individuals2. It is also noteworthy to state that the Brainets implemented here only allowed partial social interactions between subjects (through the Plexiglas panels). As such, it is not clear from our current study, to what extent social interactions played (or not) a pivotal role in the Brainet performance. Therefore, it will be interesting to repeat and expand these experiments by allowing full social contact between multiple animals engaged in a Brainet operation. In this context, Brainets may become a very useful tool to investigate the neurophysiological basis of animal social interactions and group behavior. We have previously proposed that the accuracy of the BtBI could be improved by increasing the number of nodes in the network and the size of neuronal ensembles utilized to process and transfer information2. The novel Brainet architectures tested in the present study support these suggestions, as we have demonstrated an overall improvement in BtBI performances compared to our previous study (maximum of 72% correct in the previous study versus maximum of 87% correct here)2. Since neuron dropping curves did not reach a plateau, it is likely that the performance of our Brainet architectures can be significantly improved by the utilization of larger cortical neuronal samples. In addition, switching between sequential and parallel processing modes, as was done in the last experiment, allowed the same Brainet to process more than two bits of information. It is important to emphasize, however, that the computational tasks examined in this study were implemented through Boolean logic10,12. In future studies we propose to address a new range of computational problems by using simultaneous analog and digital processing. By doing so, we intend to identify computational problems that are more suitable for Brainets to solve. Our hypothesis is that, instead of typical computational problems addressed by digital machines, Brainets will be much more amenable to solving the kind of problems faced by animals in their natural environments. The present study has also shown that the use of multiple interconnected brains improved Brainet performance by introducing redundancy in the overall processing of the inputs and allowing groups of animals to share the attentional load during the task, as previously reported for monkey Brainets8. Therefore, our findings extended the concept of BtBIs by showing that these interfaces can allow networks of brains to alternate between sequential and parallel processing13 and to store information. In conclusion, we propose that animal Brainets have significant potential both as a new experimental tool to further investigate system neurophysiological mechanisms of social interactions and group behavior, as well as provide a test bed for building organic computing devices that can take advantage of a hybrid digital-analogue architecture.
Methods
All animal procedures were performed in accordance with the National Research Council’s Guide for the Care and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use Committee. Long Evans rats weighing between 250–350 g were used in all experiments.
Tasks of synchronization and desynchronization. Groups of four rats, divided in two pairs (dyads), were placed in two behavioral chambers (one dyad in each chamber). Rats belonging to the same dyad (i.e. inside the same chamber) could see each other through a Plexiglas panel, but not the animals in the other dyad. Each trial in a session consisted of four different periods: baseline (from 0–9 seconds), ICMS (9–11 seconds), test (11–12 seconds), and reward (13–25 seconds). During the baseline period no action was required from rats. During the ICMS period a pattern of ICMS (20 pulses, at 22–26 Hz, 10–100 uA) was delivered to all rats simultaneously. During the Test period, neural activity from all neurons recorded in each rat was analyzed and compared to the neural activity of all other animals as a population. Spikes from individual channels were summed to generate a population vector representing the overall activity which generally constitutes a good indicator of whisking and/or licking activity14. The population vectors for each of the four rats were then normalized. Lastly, we calculated the Pearson correlation between the normalized population vector of each rat and the general population of rats (the Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
11
252
www.nature.com/scientificreports/ average of the neural population vectors from three remaining rats). During Pre-Sessions neural activity was analyzed in each trial, but no ICMS or water reward was delivered. During Sessions, neural activity was analyzed after the delivery of an ICMS stimulus and if the threshold for a correct trial was reached (at least three rats with R> = 0.2) then a water reward was delivered. During the Post-Sessions, neural activity was recorded and a water reward was delivered if animals reached the threshold for a correct trial, however no ICMS stimuli were delivered. Additionally, we also tested the effect of ICMS alone and in anesthetized animals (Ketamine/Xylazine 100 mg/kg). During the synchronization/desynchronization task two different ICMS patterns were delivered: Stimulus 1 consisted of the same pattern that was used for the synchronization task and the threshold for a correct trial remained the same. Stimulus 2 consisted of two short bursts of ICMS (2 × 4 pulses, 22–26 Hz separated by 250 ms interval) and the threshold for a correct response was less than three rats reaching an R value of 0.2 during the testing period.
Adaptive decoding algorithm. During the experiments where the adaptive decoding algorithm was used (discrete classification, tactile memory storage, sequential and parallel processing), the ICMS patterns remained as previously. Neural activity was separately analyzed for each neuron in each rat and 25 ms distributions were built and filtered with a moving average of 250 ms. The overall structure of the sessions included an initial period of 16–30 trials where Stimuli 1 and 2 were delivered to rats in order to build the distributions for each stimulus. The overall firing rate for each bin in the test period was then analyzed and, according to the probability distributions, a vote for Stimulus 1 or for Stimulus 2 was calculated. Bins with similar spike distributions for both stimuli were not analyzed. A final vote for each cell was then calculated, using the votes from all the bins that presented differences in the firing rate for the two stimuli. Lastly, the final votes for each cell in the population were filtered with a sigmoid curve. This filtering allowed the best encoding cells in the ensembles to contribute significantly more than other cells to the overall decision made by the Brainet made in each trial. Additionally, the weight of the cell population could be automatically adjusted at different intervals (e.g. every 10 or 15 trials). For the image processing experiment, groups of four rats were tested. An original image was pixilated and converted into multiple trials. Each trial corresponded to a white (Stimulus 1) or blue (Stimulus 2) pixel in the original image. In each trial one of two different ICMS stimuli was delivered to the Brainet. After the neural activity from the Brainet was decoded, a new image corresponding to the overall processing by the Brainet was recreated. Memory storage experiment. For this specific experiment only three rats were used in each session and ICMS frequency patterns varied between 20–100 Hz. The number of pulses remained the same as in the previous experiments. Each memory was processed across a period of four trials which represented four different stages of a memory being processed: Stimulus delivery (Trial 1), Pattern Storage I (Trial 2), Pattern Storage II (Trial 3), and lastly, Stimulus Recovery (Trial 4). Information was initially delivered to the S1 cortex of the first rat (Rat 1) in the first trial – Stimulus Delivery. In Trial 2, information decoded from the cortex of Rat 1 was delivered as an ICMS pattern to the second rat (Rat 2) - Pattern Storage I. In Trial 3, information decoded from the S1 of Rat 2 was delivered to Rat 3 - Pattern Storage II. In Trial 4, neural activity decoded from the cortex of Rat 3 was decoded and delivered to the cortex of Rat 1 as a pattern of ICMS. Lastly, if the stimulus encoding and decoding was correct across all four trials (chance level of 6.25%) a memory was considered to be recovered. The overall number of memories decoded, the percent of stimuli decoded and the accuracy of the brain-to-brain interface information transfer were measured. As a control measure the Plexiglas panels separating the dyads were made opaque for this experiment. Additionally, as the tactile pattern was delivered to each rat in the specific memory stage (delivery, storage or recovery), a random Stimulus 1 or 2 was delivered to the remaining rats. This random stimulation of the remaining individuals ensured that, in each trial, rats could not identify whether or not they were participating in the tactile trace. Sequential and parallel processing experiment. Each block of information processing consisted
of two trials: the first trial corresponded to parallel processing and the second trial corresponded to sequential processing. Two dyads of rats were formed: Dyad 1 (Rat 1-Rat 2) and Dyad 2 (Rat 3-Rat 4). During the first trial each dyad processed one of two ICMS stimuli independently of the other dyad. After the delivery of the ICMS stimuli to each dyad, neural activity was decoded and the stimulus for Trial 2 was computed from the results. If both dyads encoded a similar stimulus (Stimulus 1 - Stimulus 1, or Stimulus 2 - Stimulus 2), then the ICMS stimulus in Trial 2 was Stimulus 3. Otherwise, if the dyads encoded different ICMS stimuli (Stimulus 1 - Stimulus 2, or Stimulus 2 - Stimulus 1), then the ICMS stimulus in Trial 2 would be Stimulus 4. Stimuli 1 and 3 and Stimuli 2 and 4 had the exact same physical characteristics (number of pulses). During the second trial the same stimulus was delivered simultaneously to all four rats, and the Brainet encoded an overall response. A block of information was considered to be correct only if both Trials 1 and 2 were correct in both the dyads and in the Brainet. For the weather forecasting experiment groups of three animals were tested. Sessions were run as described above for sequential and parallel processing. However, Trial one (parallel processing) was processed only by one rat (temperature) and one dyad of rats (barometric pressure), while Trial two (sequential processing: probability of precipitation) was processed by the whole Brainet (three rats).
Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
12
253
www.nature.com/scientificreports/ To establish a simple weather forecast model we used original data from Raleigh/Durham Airport (KRDU), at WWW.Wunderground.com. Estimates were collected on August 2, 2014. We used periods characterized by increases and decreases in temperature and barometric pressure as independent variables, and increases in the probability of precipitation as the dependent variable. A total of 13 periods were collected. These included a total of 26 independent inputs for even trials (13 variations in temperatures and 13 variations in barometric pressure), as well as 13 additional changes in the probability of precipitation, to be compared with the Brainet outputs (i.e. the actual forecast). Specifically, for this experiment, increases in temperature (Stimulus 1 for the first rat) with decreases in barometric pressure (Stimulus 2 in Rats 2-3), during the odd trials, were computed as an increase in the probability of precipitation (Stimulus 4 to the Brainet in the even trial). Otherwise, increases or decreases in temperature (Stimulus 1 or 2 in the odd trial) combined with an increase in barometric pressure (Stimulus 1 for Rats 2 and 3), were computed as a decrease in the probability of precipitation (Stimulus 3 for the Brainet) in the even trial. Stimuli 1 and 3, and Stimuli 2 and 4 had the exact same physical characteristics (number of pulses).
Surgery for microelectrode array implantation. Fixed or movable microelectrode bundles or arrays of electrodes were implanted bilaterally in the S1 of rats. Craniotomies were made and arrays lowered at the following stereotaxic coordinates: [(AP) − 3.5 mm, (ML), ± 5.5 mm (DV) − 1.5 mm].
Electrophysiological recordings. A Multineuronal Acquisition Processor (64 channels, Plexon Inc,
Dallas, TX) was used to record neuronal spikes, as previously described15. Briefly, differentiated neural signals were amplified (20000–32,000× ) and digitized at 40 kHz. Up to four single neurons per recording channel were sorted online (Sort client 2002, Plexon inc, Dallas, TX).
Intracortical electrical microstimulation. Intracortical electrical microstimulation cues were generated by an electrical microstimulator (Master 8 , AMPI, Jerusalem, Israel) controlled by custom Matlab script (Nattick, USA) receiving information from a Plexon system over the internet. Patterns of 8–20 (bipolar, biphasic, charge balanced; 200 μ sec) pulses at 20–120 Hz were delivered to S1. Current intensity varied from 10–100 μ A.
References
1. Nicolelis, M. Beyond boundaries: the new neuroscience of connecting brains with machines--and how it will change our lives. 1st edn, (Times Books/Henry Holt and Co., 2011). 2. Pais-Vieira, M., Lebedev, M., Kunicki, C., Wang, J. & Nicolelis, M. A. A brain-to-brain interface for real-time sharing of sensorimotor information. Sci Rep 3, 1319, doi: 10.1038/srep01319 (2013). 3. West, B. J., Turalska, M. & Grigolini, P. Networks of Echoes Imitation, Innovation and Invisible Leaders (Springer 2014). 4. Deadwyler, S. A. et al. Donor/recipient enhancement of memory in rat hippocampus. Front Syst Neurosci 7, 120, doi: 10.3389/ fnsys.2013.00120 (2013). 5. Yoo, S. S., Kim, H., Filandrianos, E., Taghados, S. J. & Park, S. Non-invasive brain-to-brain interface (BBI): establishing functional links between two brains. PLoS One 8, e60410, doi: 10.1371/journal.pone.0060410 PONE-D-12-31631 (2013). 6. Rao, R. P. et al. A Direct Brain-to-Brain Interface in Humans. PLoS One 9, e111332, doi: 10.1371/journal.pone.0111332 PONE-D-14-32416 (2014). 7. Grau, C. et al. Conscious brain-to-brain communication in humans using non-invasive technologies. PLoS One 9, e105225, doi: 10.1371/journal.pone.0105225 PONE-D-14-17198 (2014). 8. Ramakrishnan, A. et al. Computing arm movements with a monkey brainet. Sci Rep In Press (2015). 9. Carmena, J. M. et al. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol 1, E42, doi: 10.1371/journal.pbio.0000042 (2003). 10. Boole, G. in The Mathematical Analysis of Logic, being an essay towards a calculus of deductive reasoning. Cambridge: MacMillan, Barclay & MacMillan (1847). 11. Krupa, D. J., Wiest, M. C., Shuler, M. G., Laubach, M. & Nicolelis, M. A. Layer-specific somatosensory cortical activation during active tactile discrimination. Science 304, 1989–1992, doi: 10.1126/science.1093318 304/5679/1989 (2004). 12. Harris, J. M., Hirst, J. L. & Mossinghoff, M. J. Combinatorics and graph theory. 2nd edn, (Springer, 2008). 13. Grama, A. Introduction to parallel computing. 2nd edn, (Addison-Wesley, 2003). 14. Pais-Vieira, M., Lebedev, M. A., Wiest, M. C. & Nicolelis, M. A. Simultaneous Top-down Modulation of the Primary Somatosensory Cortex and Thalamic Nuclei during Active Tactile Discrimination. J Neurosci 33, 4076–4093, doi: 10.1523/ JNEUROSCI.1659-12.2013 33/9/4076 (2013). 15. Nicolelis, M. A. L. Methods for neural ensemble recordings. 2nd edn, (CRC Press, 2008).
Acknowledgements
The authors would like to thank James Meloy for microelectrode array manufacturing and setup development, Po-He Tseng and Eric Thomson for comments on the manuscript, Laura Oliveira, Susan Halkiotis, and Terry Jones for miscellaneous assistance. This work was supported by NIH R01DE011451, R01NS073125, RC1HD063390, National Institute of Mental Health award DP1MH099903, and by Fundacao BIAL 199/12 to MALN. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Author Contributions
M.P.V. and G.S. performed the experiments; M.P.V. and M.A.N. conceptualized the experiments; M.P.V., A.Y., M.L. and M.A.N. analyzed the data. M.P.V., M.L. and M.A.N. wrote the manuscript. M.P.V. prepared Figures 1–7 and SF1–3. G.S. also prepared Figure 4. All authors reviewed the manuscript. Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
13
254
www.nature.com/scientificreports/
Additional Information
Supplementary information accompanies this paper at http://www.nature.com/srep Competing financial interests: The authors declare no competing financial interests. How to cite this article: Pais-Vieira, M. et al. Building an organic computing device with multiple interconnected brains. Sci. Rep. 5, 11869; doi: 10.1038/srep11869 (2015). This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the articleâ&#x20AC;&#x2122;s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/
Scientific Reports | 5:11869 | DOI: 10.1038/srep11869
14
255
www.nature.com/scientificreports
OPEN
Received: 18 October 2017 Accepted: 27 February 2018 Published: xx xx xxxx
Interbrain cortical synchronization encodes multiple aspects of social interactions in monkey pairs Po-He Tseng1,2, Sankaranarayani Rajangam1,2, Gary Lehew1,2, Mikhail A. Lebedev1,2 & Miguel A. L. Nicolelis1,2,3,4,5,6 While it is well known that the primate brain evolved to cope with complex social contingencies, the neurophysiological manifestation of social interactions in primates is not well understood. Here, concurrent wireless neuronal ensemble recordings from pairs of monkeys were conducted to measure interbrain cortical synchronization (ICS) during a whole-body navigation task that involved continuous social interaction of two monkeys. One monkey, the passenger, was carried in a robotic wheelchair to a food dispenser, while a second monkey, the observer, remained stationary, watching the passenger. The two monkeys alternated the passenger and the observer roles. Concurrent neuronal ensemble recordings from the monkeys’ motor cortex and the premotor dorsal area revealed episodic occurrence of ICS with probability that depended on the wheelchair kinematics, the passenger-observer distance, and the passenger-food distance – the social-interaction factors previously described in behavioral studies. These results suggest that ICS represents specific aspects of primate social interactions. Observing the behaviors of others is essential for primates, including humans, to be able to handle the complex dynamics of their social groups. Such observations may allow individuals to learn social ranks, recognize threats and potential allies, as well as learn new motor skills1,2. Experiments in monkeys have demonstrated that, while an animal observes the actions performed by a different subject, frontal and parietal neurons of the observer respond as if the observer performed the same action by itself3–14. Such cortical neurons, which represent the actions of others, are classically known as “mirror neurons”15. Studies of mirror neurons have provided insights on how cortical neuronal ensembles mediate social interactions through imitation and motor cooperation16–22. Several of these studies employed interactive tasks where monkey pairs cooperated or competed14,23,24. Such experiments usually employed single-unit recordings from one of the subjects’ brains, but not large-scale neuronal ensemble recordings obtained simultaneously from both brains of the interacting animals. This shortcoming resulted mostly from the major technical challenges involved in obtaining concurrent neuronal recordings from multiple subjects. Because of this limitation, the neuronal correlates of social interaction among multiple subjects have not been fully investigated in studies dealing with mirror neuron activity. In the present study, we overcame this technical barrier by introducing a new paradigm that allowed multichannel wireless recording from the brains of a monkey pair engaged in a whole-body navigation task. Using this new neurophysiological approach, we asked how cortical activity recorded simultaneously from two monkeys could reflect such parameters as the animals’ relative position in space and their closeness to food – crucial factors determining primate social interaction25–28. In the whole-body navigation paradigm, pairs of monkeys alternatively played one of two distinct roles from day to day: while one monkey, the passenger, was carried by a motorized wheelchair, another, the observer, was seated in a stationary chair, watching the passenger’s whole-body movement. Both monkeys were motivated to attend to the wheelchair movements because they were both rewarded upon the passenger reaching the target: the passenger collected grapes, while the observer received juice. Our analysis showed that ICS between the two monkeys reflected the passenger’s whole-body movements 1
Department of Neurobiology, Duke University, Durham, NC, 27710, USA. 2Duke University Center for Neuroengineering, Duke University, Durham, NC, 27710, USA. 3Department of Biomedical Engineering, Duke University, Durham, NC, 27708, USA. 4Department of Psychology and Neuroscience, Duke University, Durham, NC, 27708, USA. 5Department of Neurology, Duke University, Durham, NC, 27710, USA. 6Edmund and Lily Safra International Institute of Neurosciences, Natal, 59066060, Brazil. Correspondence and requests for materials should be addressed to M.A.L.N. (email: nicoleli@neuro.duke.edu) SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
1
256
www.nature.com/scientificreports/
Figure 1. Interbrain cortical synchronization (ICS) during the navigation task. (A) Locations of cortical implants in three monkeys (C,J, and K). Neuronal-ensemble recordings were conducted in M1 (red dots) and PMd (blue dots), in both hemispheres. (B) The experimental setup. Two monkeys (passenger and observer) were placed in a 5.0-by-3.9 m room. The passenger sat in an electrically actuated wheelchair. The observer sat in a stationary chair placed in the corner of the room. During each trial, the passenger moved from a starting location (shown on the left) to a stationary grape dispenser. Five representative routes of the wheelchair are plotted in different colors. These routes were randomly generated by a computer program. (C) Color plots of neuronal-ensemble activity for two representative trials. Each horizontal line corresponds to a unit. Color represents normalized (z-scored) firing rate of 69 units were recorded in monkey C (observer in this experiment) and 47 in monkey K (passenger). Episodes of ICS are marked by red horizontal lines. (D) Continuous evaluation of ICS for the trials shown in (C). Instantaneous values of the distance correlation were computed with a sliding window, of the same 3-s width as the red bars in (C). Correlation peaks are marked by arrows. (E) Wheelchair routes for the same trials as in (C) and (D). The routes are color-coded to indicate ICS.
and spatial location, the distance between the passenger and observer, and the distance between the passenger and food reward.
Results
Experiments were conducted in three female monkeys, monkeys C, J and K. Of these, monkey C was the most dominant, as judged by the feeding priority in conflict over food29–31, followed by monkey K, and then by monkey J (see Methods: Social ranking). Monkeys were chronically implanted with multiple cortical multielectrode arrays. A monkey pair participated in each experimental session; monkey pairs C-K and C-J were tested. Neuronal ensemble activity was recorded in both monkeys simultaneously, using a 256-channel wireless recording system32,33. In monkey J, we recorded bilaterally from the primary motor cortex (M1), and in monkeys C and K, bilaterally from M1 and dorsal premotor cortex (PMd) (Fig. 1A). The number of recorded units ranged, depending on the session, 66–68 in monkey J, 70–90 in monkey C (M1, 43–54; PMd, 27–36), and 43–48 in monkey K (M1, 30–33; PMd, 43–48). During the experiments, animals were seated in their monkey chairs and placed inside a 5.0m-by-3.9 m room (Fig. 1B). The chair of the monkey, called observer, remained stationary. The chair of the other monkey, called passenger, was mounted on an electrically actuated cart that traveled freely in the room. The task consisted of the passenger navigating toward a grape dispenser to collect a grape. At the time the passenger obtained the grape, the observer also received a juice reward. The grape dispenser was situated in the corner of the room; the observer stayed in another corner. The passenger started navigation from a location near the wall opposing the grape dispenser and observer. For both monkey pairs (C-K and C-J), the passenger and observer roles were alternated among monkeys in different experiments. The wheelchair was moved by a computer program along a randomly generated trajectory.
Interbrain cortical synchronization between the passenger and observer. We observed that cortical activity concurrently recorded in both the passenger and observer exhibited episodic ICS, where the
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
2
257
www.nature.com/scientificreports/
Figure 2. The dependence of ICS on wheelchair position and velocity, expressed as conditional probability of ICS episodes. (A–D) Probability of ICS episodes as a function of room x,y coordinates. The values of probability at each pixel are averaged with immediate neighbors and are color coded. The one-dimensional bars below each two-dimensional plot show the dependence of probability on the distance between the passenger and food reward (top), and between the passenger and the observer (bottom). The results for monkey pair C-K are shown in (A,C) and for monkey pair C–J in (B,D). Monkey C was either passenger (panels on the left) or observer (panels on the right). In (A,B) the observer was in the right corner of the room (relative to the passenger at the starting location), and the grape dispenser was in the left corner. In (C,D), the locations of the observer and grape dispenser were swapped. (E,F) Color-coded probability of ICS episodes as a function of wheelchair velocity for monkey pairs C–K (E) and C–J (F). Horizontal axis corresponds to rotational velocity, and vertical axis corresponds to translational velocity. (See Fig. S4 for the number of samples contributed to each pixel).
correlation of neuronal firing patterns between the two monkeys was significantly higher than the one obtained for permuted data (i.e., null distribution). While the cause of this ICS was not apparent in individual trials, statistical analysis of the ICS episodes (see Methods: Episodic ICS) showed that their probability of occurrence depended on wheelchair position and velocity, the distances between the passenger and grape dispensers, and the distance between the passenger and observer. Figure 1C–E shows two representative trials containing several ICS episodes, with correlation coefficients as high as 0.5. ICS episodes constituted 19.7% ± 2.1% (mean ± standard error) of the total session time for monkey pair C-K, and 35.7% ± 4.7% for C-J (Supplementary Fig. S1). To determine the factors influencing such an ICS, we calculated the probability of ICS episodes as a function of different parameters of the wheelchair movements: translational and rotational velocity, room coordinates of the wheelchair, distance from the wheelchair to the grape dispenser, and distance from the wheelchair to the observer (see Methods: ICS probability). We found that ICS probability depended on each of these parameters, and was also influenced by monkey pair composition and task roles (passenger vs. observer) assigned to each monkey. Figure 2 illustrates how ICS probability depended on the wheelchair position (Fig. 2A–D) and velocity (Fig. 2E,F). When monkey C was paired with monkey K, and C was the passenger, episodes of high ICS became more frequent when the distance between the monkeys decreased (Fig. 2A,C, left panels, where the two distributions look like mirrored images as the observer and the reward swapped locations). This relationship reversed when monkey C was the observer (Fig. 2A,C, right panels): synchrony episodes occurred less frequently when the monkeys were close to each other. For the same monkey pair, the probability of ICS episodes also depended on the wheelchair velocity. This dependence also reversed after the monkeys’ roles changed (compare left and right panels in Fig. 2E). Similar effects were observed for monkey pair C-J. Wheelchair position, velocity and SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
3
258
www.nature.com/scientificreports/ monkey roles influenced the probability of ICS (Fig. 2B,D,F). Notably for this monkey pair, higher levels of ICS were frequent when monkey C, as the passenger, was far away from monkey J (Fig. 2D, left), the observer. When C approached J, ICS episodes became less frequent. The dependence of ICS on the passenger’s position changed dramatically after C became the observer (Fig. 2D, right), as did the dependence of ICS on wheelchair velocity (Fig. 2F). In addition to the effect of the distance between the monkeys, ICS probability was affected by the distance between the passenger and the grape dispenser (Fig. 2A–D). These ICS patterns depended on the monkey pair composition. For example, in monkey pair C-K, the probability of ICS episodes increased when the passenger (C or K) approached the grape dispenser (Fig. 2A,C; Spearman correlation = −0.28 ± 0.18). Conversely, for pair C-J, the ICS decreased when the passenger (C or J) was close to the grape dispenser (Fig. 2B,D; Spearman correlation = 0.47 ± 0.17). While ICS probability clearly depended on where the passenger was positioned relative to the observer and the grape dispenser, these neural patterns could also reflect room landmarks, for example, left versus right corner. We tested for this possibility by swapping the locations of the grape dispenser and observer on different sessions. This manipulation produced a reversal of the ICS patterns (e.g., compare Fig. 2A and C), indicating that room landmarks were not as important as the positions of the observer and the grape dispenser. ICS probability also depended on the wheelchair velocity. In the example of Fig. 2E, the dominant monkey C was the passenger and monkey K was the observer. In these settings, ICS probability increased when the wheelchair had high rotational velocity. However, when the monkeys’ roles were swapped, the ICS probability did not vary with increases of the wheelchair rotational velocity. ICS patterns depended on the wheelchair velocity in the monkey pair C-J, as well (Fig. 2F). Since in the monkey pair C-K we recorded from M1 and PMd in both animals, we could compare the engagement of each of these cortical areas in ICS. For every ICS episode, determined by the analysis for the entire neuronal sample, we measured whether a particular pair of cortical areas exhibited ICS (permutation test, p < 0.05). When monkey K was the observer, M1-to-M1 synchronization was found to be most frequent (60.5% ± 1.3% of the episodes), followed by M1-to-PMd (51.5% ± 1.0%) and PMd-to-PMd (42.5% ± 1.4%). The same trend was found when monkey C was the observer: 53.0% ± 1.3% for M1-to-M1, 47.0% ± 1.0% for M1-to-PMd, and 43.6% ± 1.4% for PMd-to-PMd.
Neuronal modulation to wheelchair velocity and acceleration. Having established that ICS depended on the passenger’s position and velocity, we examined the relationship between the presence of ICS and the modulation of neuronal firing rates to position and velocity, in both the passenger’s and observer’s cortex. For that, we first evaluated neuronal modulation patterns to wheelchair kinematics, i.e. velocity and acceleration (Fig. 3). This analysis revealed that the passenger had more velocity and acceleration modulated units in both M1 and PMd (χ2(1) > 9.35, p < 0.05). In the passenger’s PMd, 40.6% and 19.5% units (data from all monkey pairs combined) were modulated to velocity and acceleration, respectively, whereas in the observer’s PMd these values dropped to 0.8% and 0%. Considering M1, the corresponding values were 43.8% and 15.5% for the passenger and 0.5% and 0% for the observer. The dependence of neuronal firing rate on different parameters of navigation was quantified using modulation depth as a metric (see Methods: Modulation depth). Modulation depth for both velocity and acceleration was on average higher in the passenger compared to the observer (ANOVA, F(1,1434) > 270.2 for monkey role, p < 0.05; See Table S1, S2 and Methods: ANOVA for modulation depth). Additionally, significant differences were found when M1 modulation depth was compared to that of PMd (permutation test, p < 0.05). Monkey pair-dependent effects were found: in monkey C, modulation was stronger in PMd than M1 when C was paired with K but weaker when C was paired with J (permutation test, p < 0.05) (Table S9). To assess whether there was any correspondence between the modulation depth values during navigating and observing, we calculated the correlation coefficient between the neuronal population features recorded in different areas (Fig. S2). We found significant correlations for both M1 (Spearman correlation rho = 0.24–0.64, p < 0.05) and PMd (rho = 0.48–0.81, p < 0.05). The presence or absence of ICS clearly correlated with changes in modulation of neuronal rates to wheelchair kinematics (Fig. 3A,B, top vs. bottom). Figure 3A,B shows two representative units. The first was recorded in monkey C’s PMd when C was paired with K (Fig. 3A). When monkey C was the passenger, this PMd unit was non-directionally modulated to rotational velocity: its firing rate increased for both rightward and leftward rotations (see X-axis). Yet, during ICS episodes in which monkey C became the observer, this PMd unit’s velocity modulation pattern became more prominent since its neuronal firing rate increased for rightward rotations and decreased for leftward rotations (Fig. 3A, right, see X-axis). The second example illustrated is an M1 unit recorded in monkey J when it was paired with monkey C (Fig. 3B). This M1 unit responded to both translational and rotational velocity and its firing increased during the ICS episodes. As before, this M1 unit also changed its velocity modulation properties when the monkeys’ roles were reassigned. When monkey J was the passenger, the unit increased firing rate in response to wheelchair rightward rotation combined with forward movement. When monkey J was the observer, the unit decreased firing rate to this pattern of wheelchair movement. For the entire neuronal sample, more units were modulated to the wheelchair velocity in the presence of synchrony episodes (39.0% ± 5.0%, data from all monkeys combined) than in their absence (23.9% ± 3.2%) (Fig. 3C, left). This difference was statistically significant for all monkey combinations (χ2(1) = 70.1, p < 0.05), and was not related to changes in absolute firing rate between the presence and absence of ICS (see Supplementary Materials: Controlling firing rate in ICS). The analysis of population-averaged modulation depth also showed a clear main effect of the presence of synchronization (ANOVA, F(1,1434) = 7.6, p < 0.05) (Table S1; Fig. 3D, left). Thus, we observed both a significantly larger number of modulated units and an increase in modulation depth in the individual units during the ICS episodes. Additionally, ANOVA showed the main effect of monkey dominance rank. SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
4
259
www.nature.com/scientificreports/
Figure 3. Modulation of passenger’s and observer’s units to wheelchair kinematics. (A) Modulation patterns to rotational (horizontal axis) and translational (vertical axis) velocity in a representative PMd unit recorded in monkey C. Color represents normalized (z-scored) firing rate. In these experiments, monkey C was paired with monkey K, and acted as passenger (left panels) or observer (right panels). ICS episodes were detected, and neuronal modulation was assessed separately when these episodes were present (ICS+, top panels) and absent (ICS−, bottom panels). (B) Modulation patterns in an M1 unit recorded in monkey J. Monkey J was paired with monkey C. Conventions as in (B). (C) Bar plots representing average proportion of units modulated to the wheelchair velocity (left panel) and acceleration (right panel) for different monkey pairs, monkeys, and monkey roles (passenger or observer). Values are shown separately for the presence and absence of ICS episodes. Error bars represent 95% confidence interval obtained by 1,000 bootstrap replicates. (D) Averaged neural modulation depth to the wheelchair velocity (left panel) and acceleration (right panel). Conventions as in (C). In this analysis, monkey C had a higher dominance rank than the others. We found weaker neuronal modulations in the more dominant monkeys (F(1,1434) = 13.6, p < 0.05), and the modulation depth also depended on the monkey pairs (C-K or C-J pair, F(1,1434) = 9.3, p < 0.05). The analysis of neuronal modulations to acceleration showed that more units were modulated to acceleration in the presence of ICS episodes (19.3% ± 1.2%; data from all monkeys combined) than in their absence (10.3% ± 1.3%) (χ2(1) = 38.8, p < 0.05) (Fig. 3C, right). However, population-average modulation depth to wheelchair acceleration did not increase during the synchronous
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
5
260
www.nature.com/scientificreports/
Figure 4. Modulation of passenger’s and observer’s units to the wheelchair room coordinates. (A) Modulation of a PMd unit recorded in monkey C. Color represents normalized (z-scored) firing rate. In these experiments, monkey C was paired with monkey K, and acted as passenger (left panels) or observer (right panels). The observer was seated in the left corner (left panels) or right corner (right panels) (relative to the passenger at the starting location). Neuronal modulation was assessed separately when ICS episodes were present (ICS+, top panels) or absent (ICS−, bottom panels). (B) Modulation of a PMd unit recorded in monkey K. Monkey K was paired with monkey C. Conventions as in (A). (C) Bar plots representing average proportion of units modulated to wheelchair position for different monkey pairs, monkeys, and monkey roles (passenger or observer). Proportions are shown separately for the presence and absence of ICS episodes. Error bars represent 95% confidence interval obtained by 1,000 bootstrap replicates. (D) Averaged modulation depth. Conventions as in (C).
episodes (ANOVA, F(1,1434) = 0.03, p = 0.86), and did not depend on the dominance rank (F(1,1434) = 0.06, p = 0.81) or monkey pair (F(1,1434) = 0.39, p = 0.53) (Table S2; Fig. 3D, right).
Spatial modulation. M1 and PMd units were modulated to wheelchair room position in both the passenger’s and observer’s brains (Fig. 4). The color plots of Fig. 4A,B show two representative PMd units whose firing rate changed when the passenger traveled to different room locations. These spatial modulation patterns were affected by the presence or absence of ICS episodes and monkey roles. Figure 4A shows a PMd unit recorded in monkey C when it was paired with monkey K. The neuronal rate increased with decreasing distance between the monkeys when monkey C was the passenger or observer. In both cases, the unit’s firing response to the distance between the monkeys increased during the ICS episodes (Fig. 4A, top vs bottom). The second illustrated unit was recorded in monkey K’s PMd. This unit’s rate decreased when monkey K was the passenger and approached monkey C (Fig. 4B, left). Conversely, the firing rate increased when monkey K approached the grape dispenser. The response pattern of the same unit was very different when monkey K was the observer (Fig. 4B, right). In this case, the neuronal rate increased when monkey K approached monkey C, but decreased when it approached the grape dispenser. For both the passenger and observer roles of monkey K, spatial modulation was stronger during the synchrony episodes (Fig. 4B, top vs bottom). Several analyses were conducted to assess different features of spatially-dependent modulations of neuronal rates. We first assessed neuronal rate as a function of room x,y coordinates. Next, we analyzed the dependence of firing rate on the distance between the monkeys. Lastly, we analyzed the role of the distance from the passenger to grape dispenser. In each analysis, we used the same ANOVA as in the previous section (see Methods: ANOVA for modulation depth). The analysis of modulation to x,y room coordinates showed that more units (χ2(1) = 153.2, p < 0.05) were modulated to wheelchair position in the passenger (55.5% ± 6.9%, data from all monkeys combined) compared to the observer (30.9% ± 14.1%) (Fig. 4C; Table S9). Consistent with this result, average spatial modulation depth was higher in the passenger than observer in all cases (ANOVA, F(1,1434) = 88.1, p < 0.05; Methods:
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
6
261
www.nature.com/scientificreports/ ANOVA for modulation depth; Table S3). The presence of ICS episodes increased spatial modulation strength (F(1,1434) = 31.8, p < 0.05). When expressed as the percentage of spatially modulated units, this effect was significant for all monkey pairs (chi χ2(1) = 179.0, p < 0.05): 66.7% ± 2.5% of units (all monkeys combined) were significantly modulated to wheelchair location in the presence of ICS episodes and 43.8% ± 5.2% in their absence. Additionally, modulation depth was lower for more dominant monkeys (F(1,1434) = 8.6, p < 0.05) (Fig. 4D; Table S3). In terms of the responses to the distance between the monkeys, 47.6% ± 4.3% of units (data from all monkeys combined) were modulated in the passenger and 29.5% ± 4.1% in the observer. The difference between these proportions was statistically significant (χ2(1) = 102.0, p < 0.05). Modulation strength in the passenger was stronger than in the observer (ANOVA, F(1,1434) = 7.2, p < 0.05; Methods: ANOVA for modulation strength) (Table S4). The presence of ICS resulted in stronger modulation (F(1,1434) = 11.6, p < 0.05) (Fig. 5A). In addition, another two ANOVA factors, social rank and role, also showed significant main effects: in a monkey pair, stronger modulations occurred in the monkey with a higher dominance rank (F(1,1434) = 12.1, p < 0.05), and in the monkey that acted as observer (F(1,1434) = 7.2, p < 0.05) (Table S4). Since it has been shown that mirror neurons are modulated differently to actions occurring in the extrapersonal and peripersonal space4, we conducted a separate examination of neuronal activity recorded when the inter-monkey distance was less than 1 m. For this distance range, neuronal rates significantly increased during the ICS episodes in both the passenger and observer (ANOVA, F = (1,1434) = 358.7, p < 0.05; see Methods: ANOVA for firing rates) (Fig. 5C; Table S5). The rates were lower for the dominant monkey in the pair (F = (1,1434) = 7.2, p < 0.05), and units in the observer had higher rates compared to the passenger (F = (1,1434) = 43.6, p < 0.05). Modulation to distance between the passenger and the grape dispenser was found in 52.0% ± 5.4% of the passenger’s units and in a small proportion (5.3% ± 3.4%, χ2(1) = 418.0, p < 0.05) of the observer’s units. Individual units were more strongly modulated in the passenger than the observer (ANOVA, F(1,1434) = 270.6, p < 0.05; Methods: ANOVA for modulation strength) (Fig. 5B; Table S6). Additionally, modulation strength increased during ICS episodes (F(1,1434) = 5.8, p < 0.05) (Fig. 5B; Table S6). For the 1-meter zone around the grape dispenser, neuronal rates were higher during the ICS episodes (ANOVA, F(1,1434) = 610.9, p < 0.05; see Methods: ANOVA for firing rates) and when the monkey was the subordinate monkey (F(1,1434) = 54.3, p < 0.05) or the observer monkey (F(1,1434) = 4.5, p < 0.05) (Fig. 5D; Table S7). Finally, we analyzed the changes in neuronal spatial modulation after the locations of the observer and grape dispenser were swapped. We tested whether the swapping symmetrically reflected the spatial modulation pattern relative to the longitudinal axis of the room. Our analysis showed that a symmetric reflection of the spatial modulation patterns occurred only for monkey pair C-K (average flip-index = 0.176 ± 0.018, bootstrapping test, p < 0.05; see Methods: flip-index), but not C-J (flip-index = −0.039 ± 0.015, bootstrapping test, p < 0.05). The reflection was clearer for units recorded in monkey C, the most dominant monkey, (flip-index of 0.22 ± 0.03) than monkey K (flip-index of 0.11 ± 0.03) (bootstrapping test, p < 0.05) (Fig. 5E). The ANOVA for the flip index did not show differences in units’ flip-index for cortical area, monkey role, nor the presence of ICS (ANOVA, F(1,218) < 1.7, p > 0.19; see Methods: ANOVA for flip-index; Table S8). In monkey pair C-J, the pattern change could not be described as a symmetric reflection (flip-index = −0.039 ± 0.015, bootstrapping test, p < 0.05), suggesting that cortical neuronal modulation was affected by the arrangement of the observer and dispenser positions in room-centered coordinates.
Discussion
In this study, a whole-body navigation/observation task, combined with simultaneous wireless recordings of cortical neuronal ensemble activity from pairs of monkeys, was employed to investigate the neuronal correlates of spatial social interactions in primates. One monkey (the observer) remained stationary while another monkey (the passenger) navigated using a robotic wheelchair. This paradigm allowed us to examine how the social interaction between the monkey pair was affected by whole-body movements of either the dominant or the subordinate animals. Simultaneous recordings of cortical ensemble activity from both the passenger and observer revealed that their M1 and PMd units modulated their firing rate in accordance to the roles of the monkey (passenger or observer), as well as the wheelchair position and velocity. Moreover, we found that cortical units located in the two monkey brains exhibited episodes of transient synchronized firing. The probability and magnitude of such ICS depended on the wheelchair kinematics, the distance between the monkeys, and the distance between the passenger and the food reward. Based on these findings, we propose that high ICS defines a fundamental neurophysiological manifestation underlying social interactions in primates, and likely, other animals. Using concurrent wireless interbrain recordings, we identified the occurrence of episodes of social interaction even without considering behavioral measurements of such an interaction. In our approach, ICS was analyzed as a stochastic process, where the occurrence of synchronous episodes was described by conditional probability that depended on multiple parameters, including wheelchair position and velocity, inter-subject distance, and the distance between the passenger and food reward. The color maps of the conditional probability of such ICS episodes were similar to classical neuronal tuning curves, with the key difference that our metric represented the combined activity of multiple brains rather than the activity of an individual brain. Such ICS maps allowed us to assess the role of experimentally controlled parameters in social interaction. Evidently, it is possible – and likely - that other uncontrolled factors contributed to the ICS observed here. The contribution from factors like eye/head movements, lip smacking, facial expressions, eye contact, other whole-body signals34–37, and auditory responses (e.g., caused by subtle noise from the wheelchair motor) will have to be investigated in greater detail in future studies. These uncontrolled factors would only have a significant effect on neural activities if both monkeys attended to them simultaneously, and joint attention has been proposed as a basic mechanism for social interaction38.
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
7
262
www.nature.com/scientificreports/
Figure 5. Neuronal activity patterns across monkeys and ICS conditions. (A) Bar plots represent average modulation depth to the distance between the passenger and observer. Results are presented separately for different monkey pairs, monkeys, monkey roles (passenger or observer), and the presence/absence of ICS episodes. Error bars represent 95% confidence interval obtained by 1,000 bootstrap replicates. Panels (B–E) use the same conventions as in (A). (B) Average modulation depth to the distance between the passenger and grape dispenser. (C) Average normalized firing rates for the close (<1 m) distance between the passenger and observer. (D) Average normalized firing rates for the close (<1 m) distance between the passenger and grape dispenser. (E) Average flip-index for M1 (left) and PMd units (right).
Mathematically, ICS can be described as an occurrence of specific neuronal firing patterns, like zero-lag synchrony39, in a high-dimensional neuronal space composed of the activity of multiple brains. Here, we only considered neuronal firing patterns generated jointly by two monkey brains, but in the future, the same analytical approach employed in the present study could be expanded to characterize meaningful and effective social interactions, from a systems neurophysiological point of view, that occur in large groups of primates, and likely, in other animal species, like rats and mice. Concurrent EEG and brain imaging can also be employed to study the same type of ICS in human subjects. In fact, several recent publications employed functional magnetic resonance imaging (fMRI) to assess brain processing in human subjects engaged in interactive and social behaviors40–52. The term “hyperscanning” was coined for such fMRI studies43. In these studies, participants interacted simultaneously or sequentially through interfaces, such as videos43–48,51 and/or audios49,52. For example, one fMRI study49 found a high level of synchronization between the subjects that viewed the same segment of a popular movie. Voxel-to-voxel synchronization was observed in the visual and auditory cortical areas, including primary, secondary and association areas. Another study investigated neural correlates of verbal communications49. fMRI recordings were first conducted in a speaker and then the audio was replayed while fMRI was obtained from a listener. Correlation analysis of the obtained brain scans suggested that successful verbal communication requires brain-to-brain coupling. Such coupling occurred when the speaker and listener shared the same language, and it diminished when the listener was told a story in a foreign language. Similar brain-to-brain coupling was demonstrated for non-verbal communications in humans, including communication with gestures47 and facial expressions44. Interestingly, examination of the activity of each communicating brain alone failed to elucidate the extent of the neural circuitry involved in these interbrain interactions46. Recently, a dyadic imaging approach was developed53,54, where two participants laying side-by-side were scanned simultaneously in the same MRI scanner. Such a setting allowed face-to-face social interaction to be studied. In one study50, brain activity obtained while two subjects placed in the same scanner established eye contact was compared with fMRI data obtained while a subject visualized pictures of opened/closed eyes of another person. This comparison revealed that establishing
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
8
263
www.nature.com/scientificreports/ eye contact with another person generates a very different pattern of brain activation than the one obtained when subjects just looked at pictures of human eyes. At this point, it is important to highlight that our study, as far as we can tell, is the first to employ concurrent wireless neuronal ensemble recordings in monkey dyads to demonstrate that ICS of cortical motor areas can be involved in representing multiple aspects of primate social interactions. Additional knowledge on brain-to-brain coupling has been provided by brain-machine interface (BMI) studies where neural activity was simultaneously decoded from the brains of a group of subjects trained to control various external devices jointly. The term “Brainet” was recently coined by our laboratory to describe BMI implementations of this kind55. Historically, such multi-brain systems were first implemented in the 1970s using electroencephalographic (EEG) recordings in humans41,56. These early applications enabled a variety of cooperative tasks, including EEG-controlled music performance by multiple subjects56,57 and drawing of Lissajous curves by two participants synchronizing their alpha EEG rhythms58. In our lab, intracranial recordings from ensembles of monkey cortical neurons were utilized for a similar purpose: pairs or trios of monkeys learned to cooperatively control the reaching movements performed by a virtual arm using their combined cortical ensemble activity55. Several factors contributed to the occurrence of ICS observed in our study. First, wheelchair movements triggered episodes of ICS because they simultaneously engaged cortical processing in both monkeys. When the wheelchair moved, the passenger’s M1 and PMd units responded to the animal’s whole-body displacement. Conversely, in the observer, equivalent cortical neuronal populations responded as a way to represent the passenger’s movements in a mirror fashion. In addition, the distance between the monkeys was an important social-interaction factor that affected the probability of the two monkey motor cortices being synchronized. Furthermore, cortical activity in both monkeys’ brains was related to the expectation of simultaneously acquiring rewards, when the passenger monkey reached the region where it could collect grapes and the observer received a fruit juice reward. As such, this reward expectation defined another important way in which the monkey pairs interacted socially. Consequently, the distance between the passenger and reward was found to be correlated with the probability of ICS episodes. Although previous studies have considered behavioral manifestations of similar social-interaction factors, including joint displacement and relative position of group members59,60 and food seeking and consumption29,61,62, our study is the first to document the occurrence of ICS in at least two cortical motor areas, M1 and PMD, as the potential neurophysiological manifestations underlying such social interactions. In this context, our present finding extends previous results obtained in individual monkeys that showed that M1 and PMd neuronal ensembles also encode reward magnitude and expectation63–65. Here we showed that potential rewards are also encoded by ICS. Since responses to reward in the acting monkey likely represented dopaminergic inputs to M1 and PMd64, it is reasonable to suggest that similar dopaminergic effects could be present in the observer’s brain in our task. Taking this argument further, one could raise the hypothesis that dopaminergic effects during the observation of rewarded actions may explain the phenomenon of learning by observation66,67. In the context of the present study, ICS could be interpreted as a potential neuronal manifestation of social learning and even a mechanism to facilitate knowledge transfer from one subject to the other. Interestingly, ICS also reflected high-order variables involved in social interactions, such as the composition of monkey pairs and their assigned roles in the task. Indeed, the magnitude of the ICS in a monkey pair changed when a passenger and an observer flipped their roles. For example, strong ICS was observed when monkey C and monkey K were close to each other and C was the passenger. Conversely, this ICS decreased when monkey C became the observer. These results could be related to the social hierarchy of the monkeys in our colony. In support of this suggestion, it was observed in previous studies that dominant monkeys freely roam in their surrounding space, while submissive monkeys suppress their behaviors in that space to avoid conflict7,62,68,69. In our experiment, monkey C’s dominance rank was higher than monkey K and J. This difference in the monkey’s social rank could translate into the effects we observed here. Irrespective of the social ranking of our monkey colony, it is reasonable to suggest that the one-meter space around the observer could be considered as “the zone of potential conflict” for the monkey pairs. This could explain the occurrence of prominent ICS when the passenger entered this zone. Although ICS obviously was not related to direct connectivity between the monkeys’ brains, it is possible that the Hebbian rule “fire together, wire together” is a valid metaphor for describing the plastic neurophysiological changes resulting from continuous social interaction. Using this analogy, the enhancement of “social connectivity” between multiple animal brains, instead of synaptic connectivity, may be strengthened or weakened, through changes in ICS resulting from social contact. For example, if the passenger’s whole-body movement, associated with bursts of activity in the passenger’s M1 and PMd neurons, is consistently coupled with modulations in firing rate of the observer’s equivalent neurons, the two monkey brains get “effectively synchronized”, although they are not interconnected directly. In real-life situations, such social connectivity is likely to result in causal relationships. For example, cortical activity of a monkey engaged in a behavior would translate into the cortical activity of a nearby monkey, and so on, until the repetition of this social interaction could effectively reshape the interbrain functional connectivity of an entire social group. Our lab has called the resulting chain of synchronized brains a Brainet55. According to our view, the occurrence of strong episodes of ICS would be the major determinant leading to the creation of such Brainets through animal social interactions occurring in their natural environments. In this context, such Brainets could include a large number of individual brains. Given this new view on how social interactions are represented by the combined activity of multiple brains and how ICS could underlie social learning, it would be of interest to compare our findings with the previous theoretical framework built around the notion of mirror neurons. Several observations in our study are reminiscent of previous findings related to mirror neuron activity. For example, the observation that a significant fraction of M1 and PMd units in the observer’s brain are modulated to the wheelchair’s rotation and translational velocity is consistent with the literature on mirror neurons representing observation of actions in premotor4,8,15,16 and motor6,12 cortical areas. Mirror neurons are described as cortical neurons that respond the same way when a subject performs or observes an action. This definition is only marginally applicable to our findings because, SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
9
264
www.nature.com/scientificreports/ generally, a given PMd or M1 neuron had different tuning patterns during the monkey’s navigating or observing. Yet, the modulation depth for navigation was correlated with the modulation depth for observation, suggesting the existence of typical mirror neuron activity. Additionally, average modulation depth was higher in the passenger than the observer. While it is tempting to attribute this result to a stronger representation of self-motion in M1 and PMd, as opposed to motion observation, it is also possible that the observer was not attentive enough to the passenger’s movements in our experiments. Indeed, paying attention to the passenger was not required by the task. It would be of interest in the future to employ a behavioral task where the observer must attend to the passenger to obtain the reward. In one previous study70, monkeys were required to attend to the movements performed by a robot, and a significant portion of PMd neurons (~20%) became modulated to spatial attention orientation. The other result that somewhat resembles previous findings on mirror neurons is our demonstration of neuronal tuning to the distance between the passenger and observer. A previous study has already demonstrated that the distance between an observer (a monkey) and an actor (a human performing a motor task in front of the monkey) influences mirror activity in premotor cortex4. In our experiments, M1 and PMd neurons in both the passenger and observer were modulated to the distance between the monkeys, especially when the passenger entered the 1-m zone surrounding the observer. Interestingly, swapping the passenger and observer often changed the pattern of this distance tuning, which indicated that the neuronal representation of “someone entering my space” was different from the representation of “me entering someone’s space”. Our finding of distance-tuning to reward also bears resemblance to previous reports of mirror-like activity related to observation of ingestive actions61,63–65,71–73. Here, we demonstrated that both the passenger’s and observer’s M1 and PMd units were modulated to the distance from the passenger to reward. As in the case of tuning to inter-monkey distance, reward-related tuning patterns depended on the monkey pair composition and the assigned monkey roles. This finding suggests that monkey dominance ranks, defined by preferential access to food and aggression62, played a role in the cortical encoding of reward location. Despite being generally consistent with the mirror-neuron framework, our findings clearly go beyond this classical paradigm by describing, for the first time, a potential neuronal mechanism underlying social interaction in the form of episodic ICS involving multiple motor cortical areas. Previously, classic sensorimotor physiology described cortical motor control and associated sensory processing as activities confined to distinct cortical areas located in a single brain. For example, in an individual animal, cortical motor areas would be involved with the planning of an arm movement and then contribute to its execution, while adjusting the movement based on sensory feedback. The finding of mirror neurons adds to this view the notion that the same cortical areas that control movements also respond to the observation of movements performed by others. These descriptions apply to two types of behavior: (1) production of actions, and (2) observation of actions; but do not integrate them. Social interaction is a type of behavior where actions are amalgamated with observations. Accordingly, our discovery that social interaction between pairs of monkeys can be represented by widespread ICS merges both action and observation as part of a common neurophysiological interactive process, taking place simultaneously in the motor cortical areas of multiple primate brains interacting as part of a social group. Therefore, we further propose that studying the patterns of such ICS will increase our knowledge of movements that are planned and executed in the context of the rich social interactions that characterize the lives of primate and other animal groups, including humans. This new view indicates the need to reconsider the role of motor cortical areas to include their involvement in animal social interactions and how the latter influence the moment to moment operation of such cortical motor circuits. Overall, the present demonstration of neural correlates of social interaction in the form of ICS has implications for future clinical application as well, especially for disorders that include social interaction deficits, such as autism spectrum disorders, which may involve difficulties in representing/understanding the actions of others, while generating appropriate social behaviors74,75. Previously, these conditions have been linked to disorders of the brain mirror system76–79. Our current findings are relevant to this framework and add to it by showing that social interactions may be encoded by the episodes of ICS. Accordingly, we suggest that our approach could be used as a diagnostic tool for detecting abnormal interbrain activities during human social interactions. In addition to being a potential biomarker for quantifying the severity of different forms of autism, measurement of ICS could be used as a tool for monitoring the effects of autism treatment, like behavior therapy, and also as part of a neurofeedback system for improving social motor skills, like in high-performance collective sports.
Methods
All animal procedures were performed in accordance with the National Research Council’s Guide for the Care and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use Committee.
Study Design. Three adult rhesus macaques (monkey C, K, and J) were used in this study. They were chronically implanted with microwire arrays in multiple cortical areas (monkeys C and K in January 2012, and monkey J in January 2017). In monkey K, we recorded from M1 and PMd; in monkey C, from M1 and PMd; and in monkey J, from M1. Neuronal spiking activity was recorded using a wireless recording system developed in-house that samples from 128 channels simultaneously in each monkey. A pair of monkeys (C and K, or C and J) was placed in the experimental room. One monkey (the observer) sat in a stationary chair while the other monkey (passenger) sat in the robotic wheelchair that navigated in the room. The room size was 5.0-by-3.9 m. The wheelchair navigated in the 3.5-by-2.6 m part of that room. The maximum translational and rotational velocities of the wheelchair were 0.3 m/s and 0.5 rad/s, respectively, to ensure the comfort of the passenger. When placed in the initial position, the passenger faced the observer in one corner of the room and a food dispenser in another corner. The positions of the observer and the food dispenser were swapped in different experiments. After the SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
10
265
www.nature.com/scientificreports/ passenger arrived at the dispenser’s location, the dispenser dropped a piece of fruit (grape, apple, blueberry, or carrot) on a plate mounted on a robotic arm, the arm extended towards the observer, and the observer grasped the food.
Behavioral task. The robotic wheelchair carried the passenger from a randomly generated starting position toward the food dispenser. The navigation trajectory passed through two randomly generated checkpoints before arriving at the food dispenser location. After the passenger retrieved the food, the wheelchair autonomously navigated back to a new randomly generated starting position. During the wheelchair navigation toward the dispenser, the observer monkey received juice rewards: small drops of juice were delivered every 10–30 s, but only if the observer’s head pointed in the direction the room center. A large juice reward was delivered when the wheelchair arrived at the food dispenser. Social ranking. We assessed the social hierarchy in monkey pair C-K and C-J using a paradigm where they competed for food29–31. The two monkeys sat in chairs and faced each other at the distance where they could reach for a small piece of fruit (grape, apple, or strawberry) placed on a tray. We conducted 50 trials, where the food was placed in between the monkeys, and we counted how often each of the monkeys reached out first. For the pair C-K, monkey C and K obtained the fruit 32 and 18 times each (χ2(1) = 7.8, p < 0.05); for monkey pair C-J, monkey C and J obtained the fruit 36 and 14 times each (χ2(1) = 19.4, p < 0.05). These results suggested that monkey C was the dominant monkey in each pair. System integration. The experiment setup included three components: (1) the experiment control sys-
tem, (2) the wheelchair navigation system, and (3) the wireless recording system. The experiment control system supervised the sequence of task events. The autonomous wheelchair navigation system controlled the wheelchair and reported the position of the wheelchair to the experiment control system. The wireless recording system recorded neuronal ensemble activity from two monkey brains simultaneously and sent the spike timestamps to the experiment control system. The three systems communicated using a local network.
Experiment Control System. The experiment control system controlled the task sequence, including starting a trial, setting target locations for the wheelchair, determining whether the wheelchair has reached the target, delivering food and liquid rewards, and ending a trial. This system received the wheelchair coordinates from the wheelchair navigation system at 10 Hz and sent target locations to the wheelchair navigation system. The experimental control system also received multichannel neuronal data from the wireless recording system. Wheelchair Navigation System. To move the wheelchair from one location to another, the robust autonomous navigation was implemented. The wheelchair was equipped with a Roboteq VDC2450 dual channel motor controller and wheel encoders to provide closed-loop control and odometry. A lidar (RPLidar 360 Laser Scanner by Robopeak) was installed at the front side of the wheelchair to sense the distance to its surroundings. The motor controller and the lidar were interfaced via a local wired/wireless network with a Raspberry Pi (RP), which communicated with the computer that ran the experiment control system. We used Robotic Operating System (ROS) software to provide the wheelchair functionality, including autonomous navigation, obstacle avoidance, simultaneous localization and mapping (SLAM). ROS ran on two computers: the RP and a dedicated desktop computer for navigation. The two computers communicated through ROS topics, on which one computer could publish messages and the other could subscribe. To localize the wheelchair, a map of the experiment room (Fig. S3) was first generated by Hector SLAM80 before the very first session, and this map was used for all the sessions. Then, combining sensor data published by the RP and wheelchair velocity commands published by the navigation computer, a particle filter approach was used to localize the wheelchair at 10 Hz. Given the position of the wheelchair and the navigation destinations from the experiment control computer, the navigation computer computed and published the wheelchair velocity commands within the predefined ranges (0 to 0.3 m/sec for translations, and −0.5 to 0.5 rad/s for rotations) at 20 Hz (ROS navigation package), which the RP subscribed and passed to the motor controller for execution. Wireless Recording System. The wireless recording system was built in-house, as described in32,33. In short, it was composed of two wireless headstages of 128-channels (one for each monkey), two bridge receivers (one for each headstage), and one recording computer. Once spikes were sorted in the recording computer, the 16-point spike waveform templates were transmitted to the storage on the headstage. The headstage sampled neural activities of each channel at 31,250 Hz, and the headstage performed spike sorting on an FPGA using a template algorithm, where a spike was detected if the absolute distance between the waveform and the template was within a user-specified threshold. Detected spike occurrences and channel IDs were wirelessly sent to the bridge receiver and then routed to the recording computer through wired ethernet. The recording computer timed the detected spikes (temporal resolution was estimated about 5 ms) and stored this information and sent neuronal data to the system control computer for further processing, such as online decoding of neural signals.
Data analysis. We conducted seven sessions with the monkey pair C-K and four sessions with the pair C-J.
For C-K, each session lasted 99.3 ± 17.3 (mean ± standard deviation) 99.3 minutes and consisted of 80 ± 45 trials. Only the movements of the wheelchair toward the food dispenser (62.8 ± 10.4 minutes) were analyzed; the returns to the starting positions were excluded from the analysis. Each trial began when the wheelchair started at the starting positions and ended when the wheelchair arrived the target location, where the fruit has not being presented to the monkey at the time so that no reaching related activities was included in the analysis. For the first three experimental sessions, monkey K was the passenger, and monkey C was the passenger during the next
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
11
266
www.nature.com/scientificreports/ three sessions. On the last, the seventh session, monkey K was the passenger again. The observer remained in the same corner of the room for the first five sessions, and then the locations of the observer and grape dispenser were swapped. For the C-J pair, the four sessions lasted on average 55.9 minutes (std. 6.4 minutes) and included 48 trials (std. 1.3 trials). The movements of the wheelchair toward the dispenser lasted 31.5 minutes (std. 0.6 minutes). Monkey C was the passenger for the first two sessions, then monkey J became the passenger for the next two sessions. The observer was located in one room corner for the first and fourth sessions, and the opposite corner for the second and third sessions. Modulation depth. Modulation depth was calculated based on linear regression. We sampled the variates (e.g., wheelchair position) with a 100-ms sampling interval within the interval 500 ms before and after time t and we regressed them to the value of neural firing rate at time t. Once the regression model was trained, we computed correlation coefficient between the true and the fitted firing rate time series as the uncorrected modulation depth (r′). To test whether a unit was significantly modulated to a variate, we compared r ′ against the modulation depth r ″ computed from randomly sampled and permuted firing rate series. We performed the permutation test with 1,000 permutations, and the p-values were corrected by false-discovery rate. We considered a unit significantly modulated if the corrected p-value was under 0.05. Note that for each ICS condition, only the firing rate time series that belonged to that condition was permuted and tested against. To compare neuronal modulation under different ICS conditions, the modulation depth r was computed from the same length of data across conditions, where the length was the duration of synchronized episodes for the whole session. Finally, we computed unbiased modulation depth, r = r ′ − mean(r ″), r was used as the metric for modulation depth in all analyses. ANOVA for modulation depth. We assessed the effect of the presence of synchrony episodes (ICS+ vs. ICS−), monkey role (passenger vs. observer), social rank (dominant vs. subordinate), and monkey pair (C-K vs. C-J) on modulation depth using a mixed-designed four-way ANOVA, where ICS episode was a repeated measure obtained from the same unit, while others were non-repeated measures. Modulation depth of each unit was treated as the random effect, while all others were fixed effect. Sums of squares of type 3 were employed when the data was unbalanced. ANOVA for firing rates. ANOVA was designed the same way as that for modulation depth. ICS metric. Bias-corrected distance correlation was originally proposed by Székely and Rizzo81 to quantify the strength of correlation between two random vectors in arbitrarily high dimensions that do not need to be equal. We chose distance correlation over RV coefficient because distance correlation is⁎more sensitive to detect nonlinV n(X, Y) ear dependency82. The distance correlation was computed as R ⁎n(X, Y) = , where V ⁎n(X, Y) is the ⁎ ⁎ V n(X, X)V n(Y, Y)
modified distance covariance between two random vectors X ∈ p and Y ∈ q . This distance correlation test statistic has an asymptotic student t distribution under independence, and thus t-test is used for multivariate independence in high dimension. Also, R ⁎n is always positive if X and Y are correlated. Brain-to-brain correlation was computed as bias-corrected distance correlation between two random vectors, where each random vector was the neural firing rate (0.1 s time bins) of the population of units from each monkey. When analyzing episodic brain-to-brain correlation, the correlation was computed within a 3-s sliding window shifted with 0.1 s steps. Episodic ICS. To test whether an ICS episode was significant, we first computed interbrain correlation for all 0.1 s time steps (see ICS metric above). Then, we permuted the spike trains from one monkey and computed the correlation again for the permuted data, which resulted in a null distribution of ICS (Fig. S1). Lastly, we tested ICS value of each 3 s sliding window against this null distribution to determine whether it is a significant ICS episode (t-test with alpha = 0.05 for a right-tail test, and p-value was corrected by false discovery rate). ICS Probability. ICS probability was calculated as the conditional probability of the occurrence of an ICS episode given the wheelchair position or velocity. To compute the ICS probability as a function of room coordinates of the wheelchair, the room coordinates were binned into a 0.2 m2 grid, where x spanned from 0.1 m to 3.1 m and y spanned from −0.8 m to 1.2 m. The ICS probability was computed for each bin as the count of ICS episodes, divided by the total count of the wheelchair entering the bin. Similarly, the passenger-grape distance and the passenger-observer distance were both binned from 0.4 m to 4 m with 0.2 m step. To compute the ICS probability as a function of the wheelchair kinematics, the rotational velocity was binned from −0.5 to 0.5 rad/s with bin size of 0.05 rad/s; the translational velocity was binned from 0 to 0.3 m/s with bin size of 0.05 m/s. Decoder output correlation. Correlation between the decoder outputs was computed as the bias-corrected distance correlation between the decoded velocities decoded for each monkey. This correlation was also computed within 3-s sliding window shift with 0.1 s steps. Flip-index. Flip-index measured if the neuronal modulation pattern to different room locations symmetrically reflected when the locations of the food dispenser and the observer were swapped. We first computed pixel-wise Spearman correlation between the spatial modulation patterns being compared. Then, we flipped one of the patterns and computed the correlation coefficient again. The flip index of a unit was the difference between these
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
12
267
www.nature.com/scientificreports/ two correlation coefficient values. Note that, because the food dispenser and the observer only swapped locations between different sessions, we assumed units of the same ID number from different sessions were the same units for this analysis (see Supplementary Fig. S5). Also, the flip index was calculated between sessions where the monkeys had the same roles. ANOVA for flip-index. Multi-factor ANOVA assessed the effect of the presence of synchrony episodes (ICS+ vs. ICS−), monkey role (passenger vs. observer), recorded area (M1 vs. PMd), and social rank (dominant vs. subordinate) on flip-index. We used a mixed-designed four-way ANOVA. We treated synchrony episodes as repeated measures from the same units, while the others as non-repeated measures. Flip-index was the only random effect, while others were fixed effect. Sums of squares of type 3 were employed when the data was unbalanced.
References
1. Cook, R., Bird, G., Catmur, C., Press, C. & Heyes, C. Mirror neurons: from origin to function. Behav. Brain Sci. 37, 177–92 (2014). 2. Dunbar, R. I. M. & Shultz, S. Evolution in the Social Brain. Science (80-). 317, 1344–1347 (2007). 3. Caggiano, V. et al. Mirror neurons encode the subjective value of an observed action. Proc. Natl. Acad. Sci. 109, 11848–11853 (2012). 4. Caggiano, V., Fogassi, L., Rizzolatti, G., Thier, P. & Casile, A. Mirror neurons differentially encode the peripersonal and extrapersonal space of monkeys. Science (80-). 324, 403–406 (2009). 5. di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V. & Rizzolatti, G. Understanding motor events: a neurophysiological study. Exp. brain Res. 91, 176–80 (1992). 6. Dushanova, J. & Donoghue, J. Neurons in primary motor cortex engaged during action observation. Eur. J. Neurosci. 31, 386–98 (2010). 7. Fujii, N., Hihara, S. & Iriki, A. Dynamic social adaptation of motion-related neurons in primate parietal cortex. PLoS One 2 (2007). 8. Gallese, V., Fadiga, L., Fogassi, L. & Rizzolatti, G. Action recognition in the premotor cortex. Brain 119(Pt 2), 593–609 (1996). 9. Ifft, P. J., Shokur, S., Li, Z., Lebedev, M. A. & Nicolelis, M. A. L. A brain-machine interface enables bimanual arm movements in monkeys. Sci. Transl. Med 5, 210ra154 (2013). 10. Ishida, H., Nakajima, K., Inase, M. & Murata, A. Shared mapping of own and others’ bodies in visuotactile bimodal area of monkey parietal cortex. J. Cogn. Neurosci. 22, 83–96 (2010). 11. Kraskov, A., Dancause, N., Quallo, M. M., Shepherd, S. & Lemon, R. N. Corticospinal neurons in macaque ventral premotor cortex with mirror properties: a potential mechanism for action suppression? Neuron 64, 922–30 (2009). 12. O’Doherty, J. E. et al. Active tactile exploration using a brain-machine-brain interface. Nature 479, 228–31 (2011). 13. Shepherd, S. V., Klein, J. T., Deaner, R. O. & Platt, M. L. Mirroring of attention by neurons in macaque parietal cortex. Proc. Natl. Acad. Sci. 106, 9489–9494 (2009). 14. Yoshida, K., Saito, N., Iriki, A. & Isoda, M. Representation of others’ action by neurons in monkey medial frontal cortex. Curr. Biol. 21, 249–253 (2011). 15. Rizzolatti, G., Cattaneo, L., Fabbri-Destro, M. & Rozzi, S. Cortical mechanisms underlying the organization of goal-directed actions and mirror neuron-based action understanding. Physiol. Rev. 94, 655–706 (2014). 16. Fabbri-Destro, M. & Rizzolatti, G. Mirror Neurons and Mirror Systems in Monkeys and Humans. Physiology 23, 171–179 (2008). 17. Ferrari, P. F., Rozzi, S. & Fogassi, L. Mirror Neurons Responding to Observation of Actions Made with Tools in Monkey Ventral PremotorCortex. J. Cogn. Neurosci 17, 212–226 (2005). 18. Gallese, V., Keysers, C. & Rizzolatti, G. A unifying view of the basis of social cognition. Trends in Cognitive Sciences 8, 396–403 (2004). 19. Heyes, C. Where do mirror neurons come from? Neurosci. Biobehav. Rev. 34, 575–83 (2010). 20. Iacoboni, M. et al. Cortical mechanisms of human imitation. Science 286, 2526–8 (1999). 21. Oberman, L. M., Pineda, J. A. & Ramachandran, V. S. The human mirror neuron system: A link between action observation and social skills. Soc. Cogn. Affect. Neurosci. 2, 62–66 (2007). 22. Ramachandran, V. S. Mirror neurons and imitation learning as the driving force behind ‘the great leap forward’ in human evolution. (2000). 23. Haroush, K. & Williams, Z. M. Neuronal prediction of opponent’s behavior during cooperative social interchange in primates. Cell 160, 1233–1245 (2015). 24. Yoshida, K., Saito, N., Iriki, A. & Isoda, M. Social error monitoring in macaque frontal cortex. Nat. Neurosci. 15, 1307–1312 (2012). 25. Altmann, S. a. Sociobiology of rhesus monkeys. II. Stochastics of social communication. J. Theor. Biol. 8, 490–522 (1965). 26. Lauer, C. Seasonal variability in spatial defence by free-ranging rhesus monkeys (Macaca mulatta). Anim. Behav. 28, 476–482 (1980). 27. Mitani, J. C. & Rodman, P. S. Territoriality: The relation of ranging pattern and home range size to defendability, with an analysis of territoriality among primate species. Behav. Ecol. Sociobiol. 5, 241–251 (1979). 28. van Schaik, C. P., van Noordwijk, M. A., de Boer, R. J. & den Tonkelaar, I. The effect of group size on time budgets and social behaviour in wild long-tailed macaques (Macaca fascicularis). Behav. Ecol. Sociobiol. 13, 173–181 (1983). 29. Belzung, C. & Anderson, J. R. Social rank and responses to feeding competition in rhesus monkeys. Behav. Processes 12, 307–316 (1986). 30. Brennan, J. & Anderson, J. R. Varying responses to feeding competition in a group of rhesus monkeys (Macaca mulatta). Primates 29, 353–360 (1988). 31. Deutsch, J. C. & Lee, P. C. Dominance and feeding competition in captive rhesus monkeys. Int. J. Primatol. 12, 615–628 (1991). 32. Rajangam, S. et al. Wireless Cortical Brain-Machine Interface for Whole-Body Navigation in Primates. Sci. Rep. 6, 22170 (2016). 33. Schwarz, D. A. et al. Chronic, wireless recordings of large-scale brain activity in freely moving rhesus monkeys. Nat. Methods 11, 670–6 (2014). 34. Allison, T., Puce, A. & McCarthy, G. Social perception from visual cues: Role of the STS region. Trends in Cognitive Sciences 4, 267–278 (2000). 35. De Gelder, B. Towards the neurobiology of emotional body language. Nature Reviews Neuroscience 7, 242–249 (2006). 36. Jellema, T. & Perrett, D. I. Cells in monkey STS responsive to articulated body motions and consequent static posture: a case of implied motion? Neuropsychologia 41, 1728–37 (2003). 37. Mazur, A. A biosocial model of status in face-to-face primate groups. Soc. Forces 64, 377–402 (1985). 38. Dikker, S. et al. Brain-to-Brain Synchrony Tracks Real-World Dynamic Group Interactions in the Classroom. Curr. Biol. 27, 1375–1380 (2017). 39. Gollo, L. L., Mirasso, C., Sporns, O. & Breakspear, M. Mechanisms of Zero-Lag Synchronization in Cortical Motifs. PLoS Comput. Biol. 10, (2014). 40. Hasson, U., Ghazanfar, A. A., Galantucci, B., Garrod, S. & Keysers, C. Brain-to-brain coupling: A mechanism for creating and sharing a social world. Trends Cogn. Sci. 16, 114–121 (2012). 41. Nijholt, A. In Brain-Computer Interfaces 74, 313–335 (Springer, 2015). 42. Schilbach, L. et al. Toward a second-person neuroscience. Behav. Brain Sci. 36, 393–414 (2013). 43. Montague, P. R. et al. Hyperscanning: simultaneous fMRI during linked social interactions. Neuroimage 16, 1159–64 (2002).
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
13
268
www.nature.com/scientificreports/ 44. Anders, S., Heinzle, J., Weiskopf, N., Ethofer, T. & Haynes, J. D. Flow of affective information between communicating brains. Neuroimage 54, 439–446 (2011). 45. King-Casas, B. et al. Getting to know you: reputation and trust in a two-person economic exchange. Science 308, 78–83 (2005). 46. Schippers, M. B., Gazzola, V., Goebel, R. & Keysers, C. Playing charades in the fMRI: Are mirror and/or mentalizing areas involved in gestural communication? PLoS One 4, e6801 (2009). 47. Schippers, M. B., Roebroeck, A., Renken, R., Nanetti, L. & Keysers, C. Mapping the information flow from one brain to another during gestural communication. Proc. Natl. Acad. Sci. USA 107, 9388–93 (2010). 48. Tomlin, D. et al. Agent-specific responses in the cingulate cortex during economic exchanges. Science 312, 1047–50 (2006). 49. Stephens, G. J., Silbert, L. J. & Hasson, U. Speaker-listener neural coupling underlies successful communication. Proc. Natl. Acad. Sci. 107, 14425–30 (2010). 50. Lee, R. F. Emergence of the default-mode network from resting-state to activation-state in reciprocal social interaction via eye contact. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS 2015–Novem, 1821–1824 (2015). 51. Fliessbach, K. et al. Social comparison affects reward-related brain activity in the human ventral striatum. Science 318, 1305–8 (2007). 52. Bilek, E. et al. Information flow between interacting human brains: Identification, validation, and relationship to social expertise. Proc. Natl. Acad. Sci. USA 112, 5207–12 (2015). 53. Lee, R. F., Dai, W. & Dix, W. A decoupled circular-polarized volume head coil pair for studying two interacting human brains with MRI. Conf. Proc. IEEE Eng. Med. Biol. Soc 2010, 6645–6648 (2010). 54. Lee, R. F., Dai, W. & Jones, J. Decoupled circular-polarized dual-head volume coil pair for studying two interacting human brains with dyadic fMRI. Magn. Reson. Med. 68, 1087–96 (2012). 55. Ramakrishnan, A. et al. Computing Arm Movements with a Monkey Brainet. Sci. Rep. 5, 10767 (2015). 56. Rosenboom, D. Biofeedback and the arts: results of early experiments. (Aesthetic Research Centre of Canada: Vancouver, B.C., 1976). 57. Rosenboom, D. The Performing Brain. Comput. Music J. 14, 48–66 (1990). 58. Sobell, N. & Trivich, M. Brainwave drawing game. Delicate Balance: Technics, Culture and Consequences 1989, 360–362 (1989). 59. Brody, E. B. & Rosvold, E. H. Influence of prefrontal lobotomy on social interaction in a monkey group. Psychosom. Med. 14, 406–415 (1952). 60. Sahlins, M. D. The social life of monkeys, apes and primitive man. Hum. Biol. 31, 54–73 (1959). 61. Ferrari, P. F., Maiolini, C., Addessi, E., Fogassi, L. & Visalberghi, E. The observation and hearing of eating actions activates motor programs related to eating in macaque monkeys. Behav. Brain Res. 161, 95–101 (2005). 62. Harlow, H. F. & Yudin, H. C. Social behavior of primates. I. Social facilitation of feeding in the monkey and its relation to attitudes of ascendance and submission. J. Comp. Psychol. 16, 171–185 (1933). 63. Marsh, B. T., Tarigoppula, V. S. A., Chen, C. & Francis, J. T. Toward an autonomous brain machine interface: integrating sensorimotor reward modulation and reinforcement learning. J. Neurosci. 35, 7374–87 (2015). 64. Ramakrishnan, A. et al. Cortical neurons multiplex reward-related signals along with sensory and motor information. Proc. Natl. Acad. Sci. 114, E4841–E4850 (2017). 65. Ramkumar, P., Dekleva, B., Cooler, S., Miller, L. & Kording, K. Premotor and Motor Cortices Encode Reward. PLoS One 11, e0160851 (2016). 66. Darby, C. L. & Riopelle, A. J. Observational learning in the rhesus monkey. J. Comp. Physiol. Psychol. 52, 94–98 (1959). 67. Petrosini, L. et al. Watch how to do it! New advances in learning by observation. Brain Res. Brain Res. Rev. 42, 252–64 (2003). 68. Fujii, N., Hihara, S., Nagasaka, Y. & Iriki, A. Social state representation in prefrontal cortex. Soc. Neurosci. 4, 73–84 (2009). 69. Oosugi, N., Yanagawa, T., Nagasaka, Y. & Fujii, N. Social Suppressive Behavior Is Organized by the Spatiotemporal Integration of Multiple Cortical Regions in the Japanese Macaque. PLoS One 11, e0150934 (2016). 70. Lebedev, M. A. & Wise, S. P. Tuning for the orientation of spatial attention in dorsal premotor cortex. Eur. J. Neurosci. 13, 1002–1008 (2001). 71. Ferrari, P. F., Gallese, V., Rizzolatti, G. & Fogassi, L. Mirror neurons responding to the observation of ingestive and communicative mouth actions in the monkey ventral premotor cortex. Eur. J. Neurosci. 17, 1703–1714 (2003). 72. Haruno, M. & Kawato, M. Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. J. Neurophysiol. 95, 948–959 (2006). 73. Hollerman, J. R., Tremblay, L. & Schultz, W. Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol. 80, 947–63 (1998). 74. Frith, U., Happé, F. & Happe, F. Autism spectrum disorder. Curr. Biol. 15, R786–90 (2005). 75. Lord, C., Cook, E. H., Leventhal, B. L. & Amaral, D. G. Autism spectrum disorders. Neuron 28, 355–63 (2000). 76. Dapretto, M. et al. Understanding emotions in others: mirror neuron dysfunction in children with autism spectrum disorders. Nat. Neurosci. 9, 28–30 (2006). 77. Oberman, L. M. et al. EEG evidence for mirror neuron dysfunction in autism spectrum disorders. Cogn. Brain Res. 24, 190–198 (2005). 78. Perkins, T., Stokes, M., McGillivray, J. & Bittar, R. Mirror neuron dysfunction in autism spectrum disorders. J. Clin. Neurosci. 17, 1239–43 (2010). 79. Théoret, H. et al. Impaired motor facilitation during action observation in individuals with autism spectrum disorder. Curr. Biol. 15, R84–5 (2005). 80. Kohlbrecher, S., Von Stryk, O., Meyer, J. & Klingauf, U. A flexible and scalable SLAM system with full 3D motion estimation. 9th IEEE Int. Symp. Safety, Secur. Rescue Robot. SSRR 2011 155–160, https://doi.org/10.1109/SSRR.2011.6106777 (2011). 81. Székely, G. J. & Rizzo, M. L. The distance correlation t -test of independence in high dimension. J. Multivar. Anal. 117, 193–213 (2013). 82. Josse, J. & Holmes, S. Measures of dependence between random vectors and tests of independence. Literature review. 7383, 1–41 (2013).
Acknowledgements
We thank D. Dimitrov and L. Oliveira for conducting neurosurgeries, T. Phillips for experimental support, and S. Halkiotis for administrative support and proofreading the manuscript. This work was supported by The Hartwell Foundation, NIH (NINDS) R01NS073952 and NIH (NIMH) DP1MH099903 awarded to M.A.L.N. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Author Contributions
P.T., M.A.L. and M.A.L.N. designed the study. P.T. and S.R. performed the experiment. G.L. built hardware including the multielectrode arrays, the grape dispenser, and the wheelchair. P.T. programmed software including the wheelchair navigation system and the experiment control system. P.T. analyzed the data. P.T., M.A.L., and M.A.L.N. wrote the manuscript.
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
14
269
www.nature.com/scientificreports/
Additional Information
Supplementary information accompanies this paper at https://doi.org/10.1038/s41598-018-22679-x. Competing Interests: The authors declare no competing interests. Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the articleâ&#x20AC;&#x2122;s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the articleâ&#x20AC;&#x2122;s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. Š The Author(s) 2018
SCiENtifiC RePorTs | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x
15
270
Key Opinion and Review Articles
271 Neuron, Vol. 19, 219–221, August, 1997, Copyright 1997 by Cell Press
Hebb’s Dream: The Resurgence of Cell Assemblies Miguel A. L. Nicolelis, Erika E. Fanselow, and Asif A. Ghazanfar Department of Neurobiology Duke University Medical Center Durham, North Carolina 27710
Although Donald Hebb’s classic book, The Organization of Behavior (1949), is widely known for its description of a mechanism for synaptic plasticity (the so-called Hebbian synapse), it also contains one of the most influential proposals on how interactions between large populations of neurons underlie brain processes. Central to Hebb’s theory is the concept of the ‘‘cell assembly . . . a diffuse structure comprising cells in the cortex and diencephalon, capable of acting briefly as a closed system, delivering facilitation to other such systems . . .’’ (p. xix). In Hebb’s view, individual neurons did not work in isolation, and consequently, they could not, by themselves, account for any given percept or ability. In fact, he suggested that individual neurons could participate in different cell assemblies and be involved in multiple functions and representations. Although the importance of neural populations in sensorimotor information processing had been recognized earlier (Young, 1802; Sherrington, 1906), Hebb’s work was a landmark because it provided the first elaborated description of mechanisms by which neural populations could underlie a variety of brain functions. Because single neurons can respond to a range of sensory stimuli or participate in multiple motor acts, large neural populations are required for representing the different attributes of a particular stimulus or for producing a given behavior. Population coding schemes have been proposed for several sensory modalities (reviewed by Erickson, 1968), as well as for cortical control of arm movements (Georgopolous et al., 1986) and tectal control of saccadic eye movements (Lee et al., 1988). Nevertheless, until recently, the main approach used to reconstruct both sensory and motor representations was to record, in a serial fashion, the activity of individual neurons and then try to derive a population code. Unfortunately, this approach does not allow one to investigate the potential time-dependent interactions between neurons that may be used by the brain to represent information. The recent advent of new electrophysiological techniques, which currently allow one to record the simultaneous activity of 100–150 neurons, has sparked renewed interest in the properties of neural assemblies and their potential roles in brain function. Therefore, it is not surprising that many laboratories have begun applying neural ensemble recordings to investigate how neuronal populations encode sensory and motor information. Here, we examine some of this recent work, which suggests that Hebb’s view is likely to become the rule rather than the exception. Neural Ensemble Encoding of Sensory Information Temporal interactions between cell assembles in different anatomical locations were a fundamental postulate
Minireview
of Hebb’s theory. Testing this theory would require one to record from populations of neurons distributed across multiple cortical areas and subcortical nuclei. Using a chronic multiple-electrode recording preparation, Nicolelis and coworkers (1995, 1997a) recorded from populations of neurons distributed throughout the trigeminal somatosensory pathway, from the trigeminal brain stem complex and the somatosensory thalamus to the primary somatosensory cortex, in awake, behaving rats. These authors showed that ensembles of single neurons from most of these structures exhibited widespread, synchronous oscillatory firing that began during attentive immobility and predicted the onset of rhythmic whisker movements. These oscillations were detected first in cortex and then spread to subcortical structures. The cortical and subcortical ensembles underlying such rhythmic firing have been found to contain highly distributed representations of tactile information (Ghazanfar and Nicolelis, 1997; Nicolelis et al., 1997a, 1997b). Multivariate statistical analysis (e.g., discriminant analysis and canonical correlation) revealed that in this sensory system, the precise location of a tactile stimulus could only be unambiguously predicted, on a single trial basis, when population rather than single neural responses were taken into account (Nicolelis et al., 1997b). Therefore, these results emphasized that the coordinated activity of large ensembles of neurons, distributed across cortical and subcortical structures, may provide the basis for the encoding of tactile information in mammalian somatosensory systems. Maldonado and Gerstein (1996) were interested in determining the changes in neuronal ensemble dynamics that follow sensory reorganization induced by intracortical microstimulation in the auditory cortex of the rat. Intracortical microstimulation is known to produce a broadening of the receptive fields of cells located at the stimulation site. In addition, neurons recorded from adjacent electrodes have been shown to increase their responsiveness to the best frequency of the cells recorded from the stimulating electrode. The functional relationships between neurons distributed within the primary auditory cortex were assessed using gravity analysis, a method in which the temporal relationships between neuronal spike trains are represented as a series of clusters in a multidimensional space. In this multidimensional space, neurons attract or repel each other depending on the coincidence of their neuronal firing. The responses of up to eight neurons were recorded simultaneously following auditory stimuli and intracortical microstimulation. These experiments demonstrated that the functional clustering of a subset of the simultaneously recorded neurons, obtained during the delivery of the auditory stimuli, could be strengthened following intracortical microstimulation. The formation of a functional cluster of neurons did not necessarily relate to the anatomical distance between the cells. In other words, neurons that were anatomically close did not necessarily have a strong interaction, and neurons that were far apart did not necessarily have weak interactions. These
272 Neuron 220
results showed that neural ensembles can be established transiently, that they are not necessarily composed of neurons within a circumscribed location (e.g., a cortical column), and that membership in an ensemble is mutable as a function of the induced reorganization. Time-dependent encoding of sensory information has also been described in the locust olfactory system. In a series of elegant studies, Laurent and colleagues (Wehr and Laurent, 1996; Laurent et al., 1996) recorded from ensembles of two to five projection neurons in the antennal lobe of the locust during the presentation of various odors to the animalâ&#x20AC;&#x2122;s antenna. Several important findings emerged from these experiments: 1) multiple neurons, distributed throughout the lobe, responded during the presentation of the same odor; 2) different odors could elicit unique responses from a given neuron; and 3) neurons that responded during a given odor presentation did so during specific epochs of the response, corresponding to cycles of field potential oscillations in the mushroom body, which receives input from the antennal lobe projection neurons. Thus, neural assemblies in the antennal lobe respond to each odor with a unique spatiotemporal pattern of firing. Coding of Task Parameters by Hippocampal Ensembles The hippocampus has long been implicated as a major component of a system that is involved in memory and in the representation of spatial information. In a recent study, Deadwyler et al. (1996) investigated how activity in simultaneously recorded CA1 and CA3 ensembles (10 neurons per ensemble) concurrently encodes several task-related events in a delayed-non-match-to-sample lever-press paradigm. Central to this investigation was the use by these authors of discriminant analysis and canonical correlation, which has been adapted for simultaneously recorded neuronal population data sets (Nicolelis et al., 1997b). It was shown that the spatiotemporal patterns of hippocampal ensemble firing encoded four different task-related parameters: the phase of the task (i.e., sample versus non-match phase), errors committed on the non-match phase of the task, the position of the lever being pressed, and the position of the lever presented in the sample phase of the task. It is important to note that while the firing patterns in the hippocampal cells differed from animal to animal, the same four task parameters could be extracted from the ensemble firing patterns in all animals. This indicates that the same dimensions of the behavioral task were encoded by the neuronal ensembles even though their firing patterns were not the same, and emphasizes that the activity of individual neurons is not sufficient for encoding these types of behavioral parameters. Instead, as Hebb postulated, patterns of activity must be analyzed across ensembles of neurons to determine how such information is represented. Dynamic Encoding of Motor Behavior In their pioneering work on the primate motor cortex, Georgopolous and coworkers (1986) elegantly demonstrated that populations of neurons could accurately predict the trajectory of arm movements. In their studies, a neural population vector was derived by pooling together the responses of serially recorded single units in different recording sessions and in different animals.
Although these vectors could be used to predict the direction of arm movement, the potential of using time as a coding dimension was lost by such an approach. This is a relevant issue, since it is conceivable that the same population of cortical motor neurons could use time-dependent coding schemes to represent different attributes of motor behavior. The importance of the time domain in cortical motor coding has recently been demonstrated by Abeles and colleagues who obtained simultaneous recordings of six to eight neurons in the primate frontal cortex while animals performed a delayed-localization task (Abeles et al., 1995; Seidemann et al., 1996). This task required the animals to make arm movements to a remembered visual target that was flashed either left or right of a reference light. By implementing a hidden Markov model to analyze the simultaneously recorded neuronal spike trains, these authors proposed that the cortical neural ensembles go through a sequence of discrete, stable states during the delay period of the task (when the monkey must remember the target location). These stable states were characterized by a specific stationary pattern of relative firing between neurons, which changed abruptly from one state to another. In addition, these states were not time locked to the occurrence of any specific sensory or motor event, and the particular sequence of states could be used to predict, with z90% accuracy, the response of the monkey. If the correlated firing-rate modulations were eliminated from the ensemble activity, but the overall firing rate was preserved, no clear states or sharp transitions between states could be detected by the hidden Markov model. Similarly, if ensembles were formed by neurons recorded serially, the hidden Markov model failed to detect discrete and stable states of neuronal ensemble activity. This work underscores the importance of the temporal domain in ensemble coding and suggests that fundamental information processing at the level of cell assemblies may occur even in the absence of a particular sensory stimulus or motor output. Cell assembly encoding of motor output patterns has also been studied extensively in several invertebrate systems. This work has shown that neurons within a single ganglion can participate in multiple networks that yield different behaviors at different moments in time. For example, it has been shown that some crab stomatogastric ganglion motor neurons can participate in separate feeding rhythms, the gastric and pyloric rhythms (Wieman et al., 1991). Thus, these neurons do not belong to unique stomatogastric ganglion networks, but instead, can change their activity to participate in both motor output patterns. Similarly, Wu et al. (1994) used optical recordings to simultaneously sample the activity of multiple neurons in the abdominal ganglion of Aplysia. In this preparation, large groups of neurons were activated during three gill-related motor behaviors: the gill withdrawal reflex, spontaneous gill contraction, and respiratory pumping. These motor activities were not encoded by dedicated circuits, but instead were controlled by distributed networks of neurons, whose members were selected from a larger neuronal population that participates in the genesis of multiple behaviors.
273 Minireview 221
Future Directions What we are witnessing in modern neurophysiology is increasing empirical support for Hebb’s views on the neural basis of behavior. While there is much more to be learned about the nature of distributed processing in the nervous system, it is safe to say that the observations made in the last 5 years are likely to change the focus of systems neuroscience from the single neuron to neural ensembles. Fundamental to this shift will be the development of powerful analytical tools that allow the characterization of the encoding algorithms employed by distinct neural populations. Currently, this is an area of research that is rapidly evolving. Further demonstration of a causal link between neural ensemble activity patterns and specific sensations or behaviors is necessary to demonstrate the relevance of population coding in the CNS. This issue is being approached in several ways. On one hand, information obtained at the molecular and cellular level is beginning to be applied to the investigation of circuit properties. For instance, ensemble recordings can now be combined with other neurobiological approaches, such as knockout genetics and/or the selective elimination of specific cell types (e.g., McHugh et al., 1996). These techniques will allow us to investigate what role a specific cellular population may play in information coding by large cell assemblies. At the other end of the spectrum, chronic and simultaneous multisite neural ensemble recordings can now be performed in behaving primates (Nicolelis et al., 1996, Soc. Neurosci. abstract). Since these recordings remain stable for many months, this opens the possibility of investigating how the learning of sensorimotor or cognitive tasks impacts the largescale neuronal interactions within and between cortical and subcortical neural ensembles. These and other exciting developments promise to open a new era of investigation in systems neuroscience. Selected Reading Abeles, M., Bergman, H., Gat, I., Meilijson, I., Seidemann, E., Tishby, N., and Vaadia, E. (1995). Proc. Natl. Acad. Sci. USA 92, 8616–8620. Deadwyler, S.A., Bunn, T., and Hampson, R.E. (1996). J. Neurosci. 16, 354–372. Erickson, R.P. (1968). Psych. Rev. 75, 447–465. Georgopolous, A.P., Schwartz, A.B., and Ketter, R.E. (1986). Science 233, 1416–1419. Ghazanfar, A.A., and Nicolelis, M.A.L. (1997). J. Neurophysiol. 78, 506–510. Hebb, D.O. (1949). The Organization of Behavior. (New York: Wiley and Sons). Laurent, G., Wehr, M., and Davidowitz, H. (1996). J. Neurosci. 16, 3837–3847. Lee, C., Rohrer W.H., and Sparks, D.L. (1988). Nature 332, 357–360. Maldonado, P.E., and Gerstein, G.L. (1996). Exp. Brain Res. 112, 431–441. McHugh, T.J., Blum, K.I., Tsien, J.Z., Tonegawa, S., and Wilson, M.A. (1996). Cell 87, 1339–1349. Nicolelis, M.A.L., Baccala, L.A., Lin, R.C.S., and Chapin, J.K. (1995). Science 268, 1353–1358. Nicolelis, M.A.L., Ghazanfar, A.A., Faggin, B., Votaw, S., Oliveira, L.M.O. (1997a). Neuron 18, 529–537. Nicolelis, M.A.L., Lin, R.C.S, and Chapin, J.K. (1997b). J. Neurophysiol. In press.
Seidemann, E., Meilijson, I., Abeles, M., Bergman, H., and Vaadia, E. (1996). J. Neurosci. 16, 752–768. Sherrington, C.S. (1906). The Integrative Action of the Nervous System. (New Haven, CT: Yale University Press). Wehr, M., and Laurent, G. (1996). Nature 384, 162–166. Wieman, J.M., Meyrand, P., and Marder, E. (1991). J. Neurophysiol. 65, 111–122. Wu, J., Cohen, L.B., and Falk, C.X. (1994). Science 263, 820–823. Young, T. (1802). Philos. Trans. R. Soc. Lond. 92, 12–48.
274
insight feature
Actions from thoughts Miguel A. L. Nicolelis
Real-time direct interfaces between the brain and electronic and mechanical devices could one day be used to restore sensory and motor functions lost through injury or disease. Hybrid brain–machine interfaces also have the potential to enhance our perceptual, motor and cognitive capabilities by revolutionizing the way we use computers and interact with remote environments.
After a clever throw in by Tostão, a simple flick of Rivelino’s magic left foot was enough to send the ball soaring into the thin air of the Azteca stadium in Mexico City. As the immaculate white object flew towards the middle of the penalty box on that hot afternoon, the colourful crowd that packed the stands slowly rose in anticipation. They roared, already celebrating, because they had seen that scene a thousand times before: the same graceful black man, dressed in blue shorts and a yellow jersey with the green 10 sewn in the back, defying logic, making fun of physics. The early celebration was warranted. As expected, Pelé floated above all Italian defenders to encounter the ball in mid air, and, with a gentle kiss of a forehead, changed its trajectory towards the net. Brazil had scored the first of its four goals in the final game of the 1970 World Cup and a whole country was about to start dancing in the streets.
he vast range of human abilities and behaviour illustrated here, as well as the gift of remembering the multitude of sensations associated with an instant of joy many decades ago, offer us a glimpse of the awesome repertoire of tasks that the human brain can accomplish. Through mechanisms that still elude our comprehension, the electrical activity of millions of brain cells (neurons) can be translated into precise sequences of skilled movements. Coordinated neuronal activity also provides us with exquisite perceptual and sensorimotor capabilities, illustrated in this example by Pelé’s ability to track the ball’s trajectory and plan the timing of his jump to hit it head on. But this is not all. Highly distributed patterns of neuronal firing underlie our ability to generate expectations about the outcome of a future event, learn the complex laws of nature and create art. One could argue, therefore, that hidden within the intricate principles that govern the way brain circuits operate lies the key to understanding the very essence of what it is to be human. Witnessing the relentless growth of the disciplines that define modern neuroscience, one cannot help wondering what kind of insights, clinical applications and technologies may emerge from brain research in the future, and, more important, how they will impact on our lives. Although many of the imagined possibilities may not be feasible at this time, recent work indicates that some current ideas will come to fruition in the not-so-distant future. Here, I focus on one of these — the development of direct interfaces between machines and the human brain.
T
NATURE | VOL 409 | 18 JANUARY 2001 | www.nature.com
Brain–machine interfaces I propose that the introduction of new methods for measuring large-scale brain activity, new techniques for microstimulating neuronal tissue, and emerging developments in microchip design, computer science and robotics have the potential to coalesce into a new technology devoted to creating interfaces between the human brain and artificial devices. One day, it is conceivable that such technology could allow patients to use brain activity to control electronic, mechanical or even virtual devices, leading to new therapeutic alternatives for restoring lost sensory, motor and even cognitive functions. Although many fundamental neurobiological questions and technical difficulties need to be solved, we can be optimistic about the feasibility of implementing this concept in the next few decades. Indeed, one brain–machine interface — the auditory prosthesis known as the cochlear implant — was introduced years ago and has improved the quality of life of many deaf patients1,2 (see Box 1). Neuroscientists have long relished the possibility of using brain signals to control artificial devices3. As a consequence, there are already many terms in the literature4 to describe devices that could accomplish this goal (for example, brain-actuated technology, neuroprostheses or neurorobots). Here I will refer to these devices collectively as ‘hybrid brain–machine interfaces (HBMIs)’. The word ‘hybrid’ reflects the fact that these applications rely on continuous interactions between living brain tissue and artificial electronic or mechanical devices. My definition of HBMIs incorporates two main types of application. Type 1 devices use © 2001 Macmillan Magazines Ltd
artificially generated electrical signals to stimulate brain tissue in order to transmit some particular type of sensory information or to mimic a particular neurological function. The classic example of this application is an auditory prosthesis. Future applications aimed at restoring other sensory functions, such as vision, by microstimulation of specific brain areas would also belong to this group. In addition, type 1 HBMIs include methods for direct stimulation of the brain to alleviate pain, to control motor disorders such as Parkinson’s disease5, and to reduce epileptic activity by stimulation of cranial nerves6. These last three applications rely on the observation that direct microstimulation of brain tissue can disrupt pathological patterns of brain activity that underlie some neurological disorders. Type 2 HBMIs rely on the real-time sampling and processing of large-scale brain activity to control artificial devices. An example of this application would be the use of neural signals derived from the motor cortex to control the
Box 1 Cochlear implants: the first HBMI Auditory prostheses work by converting features of acoustic signals, such as speech, into patterns of electrical stimuli that are then delivered through an array of chronically implanted electrodes to auditory nerve fibres lying on the basilar membrane of the cochlea. As the basilar membrane contains a representation of sound frequencies, known as a tonotopic map, auditory prostheses deliver high-frequency information to the basal region of the cochlea, and low-frequency signals to the apical region, to mimic normal auditory processing. More than 30,000 deaf patients, ranging in age from 12 months to 80 years, have had such devices successfully implanted2. Although results vary from case to case, even slight improvements in auditory performance have helped people to communicate better and to become more aware of their surrounding environment.
403
275
insight feature movements of a prosthetic robotic arm in real time. Obviously, clinical applications that require reciprocal interaction between the brain and artificial devices will combine both type 1 and 2 HBMIs. The design and implementation of HBMIs will involve the combined efforts of many areas of research, such as neuroscience, computer science, biomedical engineering, very large scale integration (VLSI) design and robotics. I have selected a few current developments in these fields to illustrate below some of the conceptual advances and technologies that will be required to design and implement useful HBMIs. I will then describe two potential clinical applications of such technology that should emerge in the near future: a system to monitor and treat epileptic seizures and a device to control a robotic prosthetic arm. Building a HBMI The first of the many challenges associated with the development of any HBMI is the need to understand better the principles by which neural ensembles encode sensory, motor and cognitive information. This is rapidly becoming one of the main goals of modern neuroscience, but our present knowledge is elementary at best. In the case of motor control, for instance, the areas of the primate brain involved are well known, and considerable information is available on the physiological properties of individual neurons located in each of them. But we know little about how the brain makes use of information from these neurons to generate movements. To design a type 2 HBMI that uses brain-derived signals to control a prosthetic robotic arm, we will need to learn how to sample and decode the motor signals generated by neurons and how to feed them into an artificial device to mimic the intended movement. Recording brain activity It is clear that neurobiological principles will be central in devising a strategy to overcome these hurdles. For example, classic experiments in primates have demonstrated that fundamental parameters of motor control emerge by the collective activation of large distributed populations of neurons in the primary motor cortex (M1). Single M1 neurons are broadly tuned to the direction of force required to generate a reaching arm movement7. In other words, even though these neurons fire maximally before the execution of a movement in one direction, they also fire significantly before the onset of arm movements in a broad range of other directions. Therefore, to compute a precise direction of arm movement, the brain may have to perform the equivalent of a neuronal ‘vote’ or, in mathematical terms, a vector summation of the activity of these broadly tuned neurons7. This implies that to obtain the motor signals required to control an artificial 404
a Technique for multichannel acquisition system
b
c Real-time analysis of brain
Signal processing
d Real-time telemetry interface
Telemetry receiver Three-dimensional artificial limb Visual feedback
e Real-time multichannel mechanical actuator
f Tactile and proprioceptive feedback
Figure 1 Schematic description of the general organization of a type 2 HBMI.
device we will need to sample the activity of many neurons simultaneously and design algorithms capable of extracting motor control signals from these ensembles. Moreover, it will be crucial to investigate how these neural ensembles interact under more complex and ‘real-world’ experimental conditions8 to generate different motor behaviours. These data will be vital to answering basic questions in regard to the development of type 2 HBMIs. For example, what is the minimum neuronal sample required to generate reliable brain-derived control signals? Should these samples be obtained from one or multiple brain areas? Does the same population of neurons code for single or multiple control parameters? Finally, how might neural encoding mechanisms change with time, experience and learning? Figure 1 illustrates the general organization of a type 2 HBMI and depicts some of the technological challenges involved in designing such devices. The first design step involves the selection of a technique (Fig. 1a) that yields reliable, stable and long-term recordings of brain activity that can be used as control signals to drive an artificial device. From recent studies in animals9,10, clinical applications of HBMIs will probably require sampling of large numbers of neurons (in the order of hundreds or thousands) with a temporal resolution of 10–100 ms, depending on the application. Although neuroscientists have long recognized the need to investigate the properties of large neural ensembles11, it is very difficult to obtain reliable, long-term measurements of neural ensemble activity with high spatial and temporal resolution. Starting in the © 2001 Macmillan Magazines Ltd
1940s and 50s with multichannel recordings of scalp electroencephalographic (EEG) activity and of the general electrical activity evoked by movement or sensory stimulation, a variety of metabolic, optical and electrophysiological methods have been introduced for monitoring large-scale brain activity. Modern multichannel electrophysiological recordings are made from arrays of microelectrodes surgically implanted in the brain. They currently allow neurophysiologists to record simultaneously, with a resolution of milliseconds, the extracellular activity of up to 100 individual neurons, distributed across multiple brain structures, in animals carrying out some task or other12. Although future improvements might allow long-term and non-invasive sampling of human neural activity with the same temporal resolution as intracranial recordings, the first generation of HBMIs will probably rely on improved versions of electrophysiological methods, such as multichannel EEG or multielectrode intracranial recordings. Indeed, preliminary studies in paralysed patients have shown that EEG signals can be used to trigger the movement of computer cursors13 or offer a way for patients to communicate14. Unfortunately, less invasive electrophysiological methods, such as scalp EEG recordings that reflect the common electrical activity of millions of neurons in widespread areas of the cortex, lack the resolution to provide the kind of time-varying motor signals needed to control a robotic arm in real time4. Multichannel intracranial recordings of brain activity, obtained by surgical implantation of arrays of microwires within one or more cortical motor areas, will NATURE | VOL 409 | 18 JANUARY 2001 | www.nature.com
276
insight feature
102amplifiers
Differential amplifiers Bandpass filters
Figure 2 A prototype of an instrumentation neurochip for processing brain-derived signals. This chip, containing a portion of the analog signal processing for 16 neural channels, was designed by I. Obeid, H. Aurora, J. Morizio and P. Wolf in the Departments of Biomedical and Electrical & Computer Engineering at the Pratt School of Engineering, Duke University. The mixed-signal CMOS (complementary metal-oxide semiconductor) process used in the design supports digital signal processing modules which will be included in future generations of this device.
therefore be required, with mathematical analysis of the extracellular activity of smaller populations (100–1,000) of neurons providing the raw brain signals for use in most HBMIs10. Despite some degree of recording degradation over time, present technology allows simultaneous sampling of 50–100 neurons, distributed across multiple cortical areas of small primates, to remain viable for several years10,12. Technological advances in multielectrode array design and neural signal instrumentation in the next decade alone are expected to increase the number of neurons that can be recorded simultaneously by at least one order of magnitude. The precise placement of the electrode arrays for intracranial recording may not be as critical to the ability to control an artificial device as was first conjectured. As motor control signals emerge from the distributed activation of large populations of neurons, and as cortical and subcortical neurons are capable of considerable plastic reorganization during adulthood15, electrode arrays targeted to brain areas of interest may suffice in most cases. As subjects learn to interact with artificial devices through HBMIs, it is likely that sampled neurons that were not originally involved in the type of motor control to be mimicked may be recruited into generating the signals required to control artificial devices. Generating the output After selecting a method for acquiring the necessary brain signals, the next challenge is to design the instrumentation (Fig. 1b–d) required for recording and processing these NATURE | VOL 409 | 18 JANUARY 2001 | www.nature.com
signals in real time. Currently, this requires specialized, sizeable and expensive electronic equipment, which can amplify and filter the original signal as well as perform analogto-digital conversion to facilitate further processing and storage of data. To make HBMIs viable, new technologies for portable, wireless-based, multichannel neural signal instrumentation are needed. The central issue of signal conditioning and instrumentation may be solved in the near future by the application of mixed-signal VLSI into the design of neurophysiological instrumentation chips. This technology allows analog and digital signals to coexist in the same microchip, and has the potential to provide the multichannel, programmable and low-noise package required for conditioning brainderived signals for clinical implementation of HBMIs. Moreover, the resulting microchip would be small enough to be chronically implanted in patients and could be powered by replaceable batteries. Such microchips could rely on wireless communication protocols based on a radio frequency link to broadcast neural signals to other components of the HBMI (Fig. 1d–e). Prototypes of dedicated ‘instrumentation neurochips’ (Fig. 2) are currently being developed, although many complex issues must be solved before they can become operational16. For instance, efficient solutions will have to be found to provide enough power for performing analog and digital processing, while still ensuring that signals can be transmitted by telemetry. Thus, battery technology, device packing and the bandwidth of the neural signals, among © 2001 Macmillan Magazines Ltd
other factors, will certainly be important in the design of HBMIs16. Having selected a method for sampling and conditioning brain signals, the next step — and one of the most difficult challenges — is to define a strategy for extracting meaningful control information from neural ensemble activity in real time. Currently, neuroscientists rely on a variety of linear and nonlinear multivariate algorithms, such as discriminant analysis, multiple linear regression and artificial neural networks, to carry out real-time and off-line analysis of neural ensemble data. Preliminary results from animal studies that use these different methods are encouraging, but considerably more experience is needed to apply these techniques in clinical HBMIs. The challenge is to produce algorithms that can combine the activity of large numbers of neurons, which convey different amounts of information, and extract stable control signals, even when the firing patterns of these neurons change significantly across different timescales. Research on areas ranging from automatic sorting algorithms for unsupervised isolation of single neuron action potentials, to the design of real-time pattern recognition algorithms that can handle data from thousands of simultaneously recorded neurons will certainly be required. In the same context, clinical applications of HBMIs will require considerable computational resources. In the not too distant future, new developments in the design of brain-inspired VLSI17, an exciting area of research aimed at modelling neuronal systems in silicon18, may provide the means for achieving the type of efficient realtime neural signal analysis required for HBMIs. This technology may allow pattern recognition algorithms, such as artificial neural networks or realistic models of neural circuits, to be implemented directly in silicon circuits. Among many other technical hurdles, significant work will be required to make these silicon circuits adaptive, perhaps by incorporating learning rules derived from the study of biological neural circuits. This will allow ‘training’ of algorithms as well as ensuring the robustness of the control system. From an implementation point of view, ‘analytical neurochips’ are ideal as they could be interfaced with the instrumentation neurochip and be chronically implanted in the subject. The final component of the idealized HBMI (Fig. 1e–f) is a real-time control interface which uses processed brain signals to control an artificial device. The types of devices used are likely to vary considerably in each application, ranging from elaborate electrical pattern generators to control muscles, to complex robotic and computational devices designed to augment motor skills19. HBMIs for epilepsy control Estimates indicate that about 0.5–2.0% of the population has epilepsy20. About 10–50% of these patients do not respond well to current 405
277
insight feature antiepileptic medications and may not be candidates for surgery. Throughout this century, neuroscientists have used multichannel recording from scalp, brain surface and even chronically implanted intracranial electrodes to investigate the electrophysiological activity that characterizes different types of seizure in humans. By doing so, scientists have not only identified different types of epilepsy, but they have also learned that there are distinct patterns of neurophysiological activity associated with the initiation and establishment of a seizure attack. Several exciting new developments in epilepsy research indicate that the development of an unsupervised HBMI for monitoring, detecting and treating seizure activity may be possible in the next decade (Fig. 3a). First, for certain types of seizure, there seems to be a particular spatiotemporal pattern of cortical activity that appears seconds or even minutes before the full epileptic attack starts21. Recently, a few laboratories have introduced automatic seizure-prediction algorithms that can be applied to intracranial and scalp recordings to forecast the occurrence of a seizure21,22. These and future seizure-prediction algorithms might provide sufficient time (2–5 minutes) to warn the patient of an imminent attack, and to trigger automatic therapeutic intervention before convulsion or loss of consciousness. But what kind of therapy could be triggered that would work in patients who are refractory to epilepsy medication? The answer may lie in another recent development in epilepsy research. Studies in both animals23 and human subjects6 have revealed that electrical stimulation of peripheral cranial nerves, such as the vagus23 and trigeminal24 nerves, can substantially reduce cortical epileptic activity. Moreover, if this peripheral nerve stimulation is applied before the initiation of seizure or during its initial stages, significantly higher reduction of seizure activity can be achieved. From this I believe that a device containing a combination of both type 1 and 2 HBMIs could be designed to function somewhat like a modern heart pacemaker (Fig. 3a). This ‘brain pacemaker’ would rely on arrays of chronically implanted electrodes to search continuously for spatiotemporal patterns of cortical activity indicating an imminent epileptic attack. Instrumentation neurochips would be responsible for all the basic signal-processing operations. They would also provide signals to one or more seizure-prediction algorithms, implemented into analytical neurochips, which would carry out real-time analysis of cortical activity. Once pre-seizure activity patterns were detected, the analytical neurochip could trigger electrical stimulation of one or multiple cranial nerves. In patients who respond to pharmacological therapy, the same stimulator could be used to activate a minipump to deliver one or more anti-epileptic 406
Seizure activity without intervention:
a
Seizure activity terminated by automatic seizure detector: Neural signal from brain
Nerve cuff electrode for seizure-triggered cranial nerve stimulation
Impulse from nerve
Implanted VLSI device for monitoring neural activity and detecting seizure activity
Implanted stimulus generator
Implanted mini-pump for seizure-triggered systemic drug delivery
b Multichannel neural signal processing: Instrumentation and analysis neurochip Implanted microelectrode arrays
Transmission of Computation of neural activity 3D movement via telemetry trajectory
Visual and tactile feedback
Real-time interface to control a robotic prosthetic arm Z X Y 3D arm trajectory
Figure 3 Schematic description of two potential applications of type 2 HBMIs. a, Design of a ‘brain pacemaker’ that monitors neural activity using a VLSI chip designed to detect seizure activity. When seizure activity is detected, the VLSI chip sends a signal to an implanted stimulus generator that drives either a nerve cuff electrode or a mini-pump for drug delivery, either of which can stop the seizure activity. b, HBMI for controlling a robotic prosthetic arm using brain-derived signals. Multiple, chronically implanted, intracranial microelectrode arrays would be used to sample the activity of large populations of single cortical neurons simultaneously. The combined activity of these neural ensembles would then be transformed by a mathematical algorithm into continuous three-dimensional arm-trajectory signals that would be used to control the movements of a robotic prosthetic arm. A closed control loop would be established by providing the subject with both visual and tactile feedback signals generated by movement of the robotic arm.
drugs directly into the blood stream. Recently, a simplified implementation of this concept has been used successfully in rats24, giving hope that a brain pacemaker for seizure monitoring and control in humans may not be far ahead. HBMIs to restore motor function Another clinical application of HBMIs that could emerge in the near future aims at © 2001 Macmillan Magazines Ltd
restoring different aspects of motor function in patients with severe body paralysis, caused primarily by strokes, spinal cord lesions or peripheral degenerative disorders (Fig. 3b). Advances in this rapidly growing field of research indicate that neural signals from healthy regions of the brain could be used to control the movements of artificial prosthetic devices, such as a robotic arm. Preliminary NATURE | VOL 409 | 18 JANUARY 2001 | www.nature.com
278
insight feature findings also demonstrate that paralysed patients can learn to use brain signals obtained from their motor cortex to interact with computers25. Extensive electrophysiological work in primates and imaging studies in humans have shown that multiple interconnected cortical areas in the frontal and parietal lobes are involved in the selection of motor commands that control the production of voluntary arm movements26. Although each of these areas has different degrees of functional specialization, in theory, each of them could be selected as the source of brain signals for controlling the movements of an artificial device. Within each of these cortical areas, different motor parameters, such a force and direction of movement, are coded by the distributed activity of populations of neurons, each of which is typically broadly tuned to one (or more) of these parameters. This indicates that implementations of HBMIs for robotic arm control need to rely on intracranial recordings from large populations of single neurons to derive motor control signals. At a first glance, a random sample of 100–1,000 cortical motor neurons, which represents a reasonable expectation for the yield of multielectrode intracranial recordings in the near future, may look too small to unveil any useful information. But recent neurophysiological experiments dispute this view. For instance, currently one can obtain precise off-line reconstructions of complex three-dimensional arm trajectories by using simple multiple regression techniques to transform the activity of 300-400 serially recorded cortical motor neurons into a neural population vector27. Moreover, experiments in rats9 and primates10 have shown that simple, real-time algorithms, applied to samples of 50–100 simultaneously recorded cortical neurons, can be used to control robotic devices in real time and mimic the type of three-dimensional arm reaching movements produced by primates. Another important issue is that, to achieve seamless interactions with prosthetic devices, patients will have to receive sensory feedback information (for example visual or tactile signals) from the prosthetic limbs. These feedback signals will establish a closed control loop between the brain and artificial devices and will probably help patients learn how to operate HBMIs. Studies in rats have revealed that, if subjects receive visual feedback information as they learn to use brain activity to interact with a robotic arm, and are rewarded for the successful completion of these movements, they progressively cease to produce overt limb movements9. In other words, even though the rats continued to exhibit the patterns of cortical activity required to control the movements of the robotic arm, this motor activity did not result in any significant limb movement. This indicates that motor control signals can be generated by cortical neurons NATURE | VOL 409 | 18 JANUARY 2001 | www.nature.com
without any muscle activity, and hence that paralysed patients might be capable of learning to operate a robotic arm even though they cannot move their own limbs. These observations also raise the intriguing hypothesis that, by establishing a closed control loop with an artificial device (Fig. 3b), the brain could incorporate electronic, mechanical or even virtual objects into its somatic and motor representations, and operate upon them as if they were simple extensions of our own bodies. The fact that the adult cortex is capable of significant functional reorganization (or plasticity) after peripheral and central injuries15, changes in sensory experience28 and learning of new motor skills29 supports this possibility. Indeed, the notion that adult plasticity can dynamically alter the perception of the limits of our own body is corroborated by studies on patients who have undergone limb amputations. Immediately after the amputation, most of these patients experience the sensation that their amputated limb is still present and moving. These ‘phantom limb’ sensations are paralleled by a significant plasticity of body maps in the somatosensory cortex30, the part of the brain that receives and interprets sensory signals from areas such as the skin surface. Instead of remaining silent, the areas in these brain maps that used to represent the amputated limb progressively start to respond to stimulation of neighbouring body regions spared by the amputation. Thus, it is conceivable that tactile feedback signals, generated by the movements of a brain-controlled robotic arm and delivered to the patient’s skin, could be used to incorporate the representation of such an artificial device into cortical and subcortical somatotopic maps. Undoubtedly, years of research will be required and many fundamental technological breakthroughs needed before this comes close to reality. Nevertheless, it seems reasonable to predict that a definitive demonstration of such a phenomenon could trigger a revolution in the way future generations interact with computers, virtual objects and remote environments, by allowing never-before-experienced augmentation of perceptual, motor and cognitive capabilities. Such applications, however, will require the introduction of new, non-invasive methods for sampling brain activity. A final thought Although developing expectations of a distant future is a risky business and may raise unjustified hope that solutions are just around the corner, I cannot avoid ending this brief overview on an optimistic tone. Despite many significant conceptual and technological obstacles, the possibility of developing clinical applications of HBMIs is real and worth pursuing, especially given the potential benefits that they may bring to people afflicted by neurological disorders. At the very least, © 2001 Macmillan Magazines Ltd
research on HBMIs will yield powerful new tools to investigate hypotheses of how large populations of neurons process information and adapt according to changes in experience. Some may argue that one could achieve this goal just by building theoretical models and running computational simulations. Perhaps that is true. But as my good friend Idan Segev, a leading computational neuroscientist, always tells me, there is a subtle but fundamental difference between simulating reality and building it. Those of us who saw Pelé scoring that magic goal on that hot Mexican afternoon in 1970 and dreamed about doing the same thing would certainly agree. Miguel A. L. Nicolelis is in the Departments of Neurobiology, Experimental Psychology, and Biomedical Engineering, Duke University, Durham, North Carolina 27710, USA (e-mail: nicoleli@neuro.duke.edu) 1. Merzenich, M. M., Schindler, D. N. & White, M. W. Laryngoscope 84, 1887–1893 (1974). 2. Pfingst, B. E. in Neural Prostheses for Restoration of Sensory and Motor Function. (eds Chapin, J. K. & Moxon, K. A.) 3–43 (CRC, Boca Raton, 2000). 3. Schmidt, E. M. Ann. Biomed. Eng. 8, 339–349 (1980). 4. Chapin, J. K. & Moxon, K. A. (eds) Neural Prostheses for Restoration of Sensory and Motor Function. (CRC, Boca Raton, 2000). 5. Benabid, A. L. et al. Lancet 337, 403–406 (1991). 6. Uthman, B. M., Wilder, B. J., Hammond, E. J. & Reid, S. A. Epilepsia 31(Suppl. 2), S44–S50 (1990). 7. Georgopoulos, A. P., Swartz, A. B. & Ketter, R. E. Science 233, 1416–1419 (1986). 8. Ghazanfar, A. A. & Hauser, M. D. Trends Cog. Sci. 3, 377–384 (1999). 9. Chapin, J. K., Moxon, K. A., Markowitz, R. S. & Nicolelis, M. A. L. Nature Neurosci. 2, 664–670 (1999). 10. Wessberg, J. et al. Nature 408, 361–365 (2000). 11. Hebb, D. O. The Organization of Behaviour: A Neuropsychological Theory (Wiley, New York, 1949). 12. Nicolelis, M. A. L. et al. Nature Neurosci. 1, 621–630 (1998). 13. Wolpaw, J. R., McFarland, D. J., Neat, G. W. & Forneris, C. A. Electroencephalogr. Clin. Neurophysiol. 78, 252–259 (1991). 14. Schutz, S. et al. Nature 398, 297–298 (1999). 15. Wu, C. W. & Kaas, J. H. J. Neurosci. 19, 7679–7697 (1999). 16. Moxon, K. A., Morizio, J., Chapin, J. K., Nicolelis, M. A. L. & Wolf, P. D. in Neural Prostheses for Restoration of Sensory and Motor Function. (eds Chapin, J. K. & Moxon, K. A.) (CRC, Boca Raton, 2000). 17. Hahnloser, R. H., Sarpeshkar, R., Mahowald, M. A., Douglas, R. J. & Seung, H. S. Nature 405, 947–951 (2000). 18. Mead, C. Analog VLSI and Neural Systems (Addison-Wesley, Reading, MA, 1989). 19. Srinivasan, M. A. in In Virtual Reality: Scientific and Technical Challenges (eds Durlach, N. I. & Mavour, A. S.) 161–187 (National Academy Press, 1994). 20. McNamara, J. O. Nature 399(Suppl.), A15–A22 (1999). 21. Martinerie, J. et al. Nature Med. 4, 1173–1176 (1998). 22. Webber, W. R. S., Lesser, R. P., Richardson, R. T. & Wilson, K. Electroencephalogr. Clin. Neurophysiol. 98, 250–272 (1996). 23. Zabara, J. Epilepsia 33, 1005–1012 (1992). 24. Fanselow, E. E., Reid, A. P. & Nicolelis, M. A. L. J. Neurosci. 20, 8160–8168 (2000). 25. Kennedy, P. R. & Bakay, R. A. NeuroReport 9, 1707–1711 (1998). 26. Wise, S. P., Boussaoud, D., Johnson, P. B. & Caminiti, R. Annu. Rev. Neurosci. 20, 25–42 (1997). 27. Schwartz, A. Science 265, 540–542 (1994). 28. Polley, D. B., Chen-Bee, C. H. & Frostig, R. D. Neuron 24, 623–637 (1999). 29. Laubach, M., Wessberg, J. & Nicolelis, M. A. L. Nature 405, 567–571 (2000). 30. Ramachandran, V. S. Proc. Natl Acad. Sci. USA 90, 10413–10420 (1993). Acknowledgements. I thank J. Chapin, F. Ebner, E. Fanselow, A. Ghazanfar, C. Henriquez, J. Kaas, J. Kralik, D. Krupa, S. Ribeiro, V. de Sa, I. Segev, M. Shuler, S. Simon, J. Wessberg and P. Wolf for comments and suggestions, and E. Fanselow, D. Krupa, P. Beck and J. Wessberg for their help in creating the illustrations for this article. The author’s research on HBMI is funded by NIH, DARPA and ONR grants.
407
279
280
281
282
283
284
285
286
287
288
PERSPECTIVES OPINION
Brain–machine interfaces to restore motor function and probe neural circuits Miguel A. L. Nicolelis Recent studies have shown that it is possible to create functional, bidirectional, real-time interfaces between living brain tissue and artificial devices. It is reasonable to predict that further research on brain–machine interfaces will lead to the development of a new generation of neuroprosthetic devices aimed at restoring motor functions in severely paralysed patients. In addition, I propose that such interfaces can become the core of a new experimental approach with which to investigate the operation of neural systems in behaving animals.
Paralysis, one of the most common and debilitating outcomes of severe damage to the central nervous system, continues to cast a long shadow of hopelessness on millions of lives worldwide. Despite the significant strides made by basic and clinical research, few (if any) therapeutic options are available at present for restoring voluntary motor control of the limbs in patients suffering from extensive traumatic or degenerative lesions of the motor system. The prevalence of severe body paralysis is high, particularly among young adults. For instance, among the leading causes of permanent paralysis, traumatic spinal cord injuries — produced by traffic accidents, acts of violence or falls — account for nearly 11,000 new cases each year in the United States alone1. In all, more than 200,000 patients in the United States live with the motor sequelae of similar injuries1. About half of these patients are
NATURE REVIEWS | NEUROSCIENCE
quadriplegic, which means that, owing to injury to the cervical spinal cord, they cannot move any of their limbs or any other muscle below the neck. Quadriplegics depend on continuous assistance to accomplish even the simplest of motor acts. Whereas most of us take for granted our ability to breathe, eat and drink, a quadriplegic patient usually cannot do any of these without the assistance of a machine (such as a ventilator) or a carer. For this reason, restoring even the smallest of motor skills in these patients can have a profound effect on their quality of life. Several new experimental approaches to the restoration of motor function lost as a result of spinal cord injuries have been proposed2. Most focus on ways to repair the damaged axons that normally mediate communication between cortical (and subcortical) motor neurons and pools of interneurons and α-motor neurons in the grey matter of the spinal cord. Despite promising results, this approach faces large challenges given the difficulty involved in guiding large numbers of severed axons to re-establish their original connections. Parallel to these efforts to repair spinal cord connectivity, recent experimental demonstrations in rodents3,4, primates5–7 and patients8–11 have raised interest in the proposition that neuroprosthetic devices — designed to bypass spinal cord lesions — could be used to restore basic motor functions in patients suffering from severe body paralysis. This approach, first proposed by the neurophysiologist
Edward Schmidt12, assumes that voluntary motor commands can be extracted in real time from the collective electrical activity of populations of cortical or subcortical neurons spared by the underlying illness, and then used to enact motor function either by directly stimulating the patient’s musculature or by controlling the movements of artificial actuators, such as robot arms. FIGURE 1 shows the general design of a cortical neuroprosthetic device aimed at restoring upper limb movements in quadriplegic patients. This device relies on chronic intracranial recordings to obtain large-scale brain activity from motor areas in the cortex. A real-time mathematical model is responsible for extracting a few motor control commands from the raw electrical brain activity. An artificial actuator (in this case, a multi-joint robot arm), controlled by the output of the real-time mathematical model, is used to reproduce the subject’s upper limb movements. Because the robot arm provides continuous feedback information to the subject (not shown), this neuroprosthetic device operates through a closed control loop. Recent review articles discuss this design in greater detail13,14. The device shown in FIG. 1 is also defined by the neurophysiological approach chosen to extract the raw brain activity from which the motor control signals are derived. Over the years, distinct sources of neuronal signals, ranging from scalp electroencephalograms (EEGs)8,10 to intracranial single-unit recordings5,9,13, have been proposed as potential sources of control signals to drive various neuroprostheses. The advantages and disadvantages of each of these and other neurophysiological approaches have been reviewed13. Regardless of the type of brain signal selected, several studies have now indicated the feasibility of building functional interfaces between living brain tissue in experimental animals and various electronic, mechanical and computational devices3–7. To a limited degree, clinical applications of this approach, notably those based on EEG recordings, have also been implemented10.
VOLUME 4 | MAY 2003 | 4 1 7
289
PERSPECTIVES The recent increase in interest in this field of research — commonly referred to as neuroengineering — has been driven by the expectation that various powerful clinical implementations of direct brain–machine interfaces (BMIs) might emerge in the near future. Support for this contention is provided by the tangible success of a variety of implantable brain stimulators, such as cochlear implants for restoring auditory function, deep brain stimulators for pain management and control of motor disorders (such as Parkinson’s disease), and vagal nerve stimulators for treating chronic epilepsy. As more patients have benefited from this approach, the interest in brain stimulation technology has grown significantly. Indeed, investments in a new generation of these devices are rapidly fuelling the emergence of an incipient brain-based biomedical industry. In essence, this process is similar to the phenomenon that led to the translation of basic science discoveries into revolutionary clinical applications in the field of cardiology. That translational process helped to create the backbone of a heart-based biomedical engineering industry, which today generates billions of dollars in revenue. At present, it is not possible to predict the economic impact that a brain-based industry will have in the future. However, judging by present trends, it seems fair to say that brain actuators will have a prominent role in the further development of this industry. Although these arguments might encourage some neurophysiologists to become entrepreneurs, a word of caution is justified. Most BMIs tested in experimental animals are not yet ready to be translated into practical clinical applications. Much basic research must be done to ensure that this approach is safe and can provide sufficient benefits to justify the type of surgical intervention that might be required for BMIs to become fully operational. Furthermore, significant engineering bottlenecks have yet to be overcome 13. These include substantial work in the areas of microelectrode array design and testing, biocompatibility of chronic brain implants, microelectronics (for example, miniaturization of hardware for multichannel neural signal conditioning and telemetry), power management, real-time computational modelling and robotics (a new generation of actuators and sensors). These engineering developments are prerequisites for moving the field from experimental demonstrations to clinical implementations that can achieve the therapeutic benefit envisioned for such devices.
418
| MAY 2003 | VOLUME 4
Figure 1 | Schematic representation of a cortical neuroprosthetic device. In the general design shown, intracranial recordings are used to sample the extracellular activity of a few hundred neurons in frontal and parietal cortical areas that are involved in the planning of arm and hand movements. The combined activity of these neuronal populations is processed, in real time, by a series of simple mathematical models designed to extract motor-control parameters from the raw brain signals. The outputs of these models are used to control the movements of a robot arm that has been designed to allow the patient to enact fundamental upper limb movements.
BMIs and fundamental research
Because of their potential clinical relevance, the potential contribution of BMIs to basic brain research is often neglected. For example, recent findings indicate that BMIs might lead to the definition of various new experimental models, aimed at investigating the real-time operation of neural circuits in behaving animals15. The continuing refinement of electrophysiological, computational and engineering methods for establishing functional interfaces between living brain tissue and artificial devices has the potential to influence experimental models in several other fields of inquiry, such as cellular, computational and cognitive neuroscience4,5,16. A few examples of what the future might bring are already present
in the literature. These include the use of experimental BMIs to investigate how different populations of neurons in a neural circuit contribute to the encoding of motor parameters5,17,18, and an in vitro implementation of hybrid biological and artificial networks to study the cellular properties of complex neural circuits16. In addition, recent findings show that a BMI could be used to optimize the operant training of experimental animals4. These examples illustrate a new approach to investigation of the brain, which I call real-time neurophysiology15. The uniqueness of this approach is that theories of brain function can be tested under the demanding constraints imposed by the need to perform efficiently in real time, or at the same timescale as the animal’s behaviour. An example of this approach is shown in FIG. 2, which illustrates a new experimental platform that is being used in our laboratory to address questions on both neural population coding and neuroprosthetic design. In this system, monkeys learn to produce complex hand movements in response to arbitrary sensory cues. Chronically implanted microwire arrays are used to simultaneously record the extracellular activity of hundreds of single neurons distributed across several cortical and subcortical structures. Using this neurophysiological approach, the dynamics of several neural circuits can be measured simultaneously. Moreover, because neuronal recordings remain stable for long periods5, this approach also allows one to quantify the physiological changes that take place in different components of a neural circuit as animals learn various sensorimotor and cognitive tasks. In this type of experimental apparatus, several real-time models can be used to extract various motor-control parameters — such as direction of hand movement, gripping force, hand velocity, acceleration, three-dimensional position and so on — from the parallel streams of neuronal activity that are recorded as the animal executes various arm movements. The outputs of these models are then used to control the movements of a multi-jointed robot arm, so that one can investigate the type of real-time computation that is required to reproduce the animal’s arm movements in a robot arm. Although successful replication of the animal’s arm movements in a robot does not imply that the motor system works in the way proposed by the real-time model tested in these studies, this experimental apparatus can be a useful tool in ongoing efforts to dissociate motor variables through behavioural training. I envisage that this can be achieved by modifying the strategy used to close the control loop between the robot and the animal (see later discussion).
www.nature.com/reviews/neuro
290
PERSPECTIVES
Real-time predictions through server
ANN and linear model
Client
Data acquisition box LAN
Visual feedback loop Visual feedback loop
Local robot
Somatosensory feedback loop
Figure 2 | Experimental design used to test a closed-loop control brain–machine interface for motor control in macaque monkeys. Chronically implanted microwire arrays are used to sample the extracellular activity of populations of neurons in several cortical motor regions. Linear and nonlinear realtime models are used to extract various motor-control signals from the raw brain activity. The outputs of these models are used to control the movements of a robot arm. For instance, while one model might provide a velocity signal to move the robot arm, another model, running in parallel, might extract a force signal that can be used to allow a robot gripper to hold an object during an arm movement. Artificial visual and tactile feedback signals are used to inform the animal about the performance of a robot arm controlled by brain-derived signals. Visual feedback is provided by using a moving cursor on a video screen to inform the animal about the position of the robot arm in space. Artificial tactile and proprioceptive feedback is delivered by a series of small vibromechanical elements attached to the animal’s arm. This haptic display is used to inform the animal about the performance of the robot arm gripper (whether the gripper has encountered an object in space, or whether the gripper is applying enough force to hold a particular object). ANN, artificial neural network; LAN, local area network.
To complete a closed-loop control BMI, information describing the performance of the robot arm is relayed back to the animal, using artificially generated visual and ‘proprioceptive/tactile’ feedback signals. Visual feedback is delivered on a monitor in front of the animal, and proprioceptive/tactile information can be relayed through an array of vibrotactile elements attached to the animal’s contralateral arm. As microelectrode arrays are implanted in neural circuits that process sensory feedback generated by arm movements, intracranial microstimulation can be used to deliver to the animal feedback information that describes the robot’s performance (not shown). When a BMI is in a closed-loop configuration, the experimenter can manipulate the feedback information that the animal receives in several ways. These manipulations can be designed to permit neuroscientists to address key questions regarding the dynamics of sensory and motor information encoding by distinct populations of neurons. For example, does learning allow a population of neurons
NATURE REVIEWS | NEUROSCIENCE
that normally contributes little to the encoding of a given movement parameter to enhance its contribution? This question could be addressed by first comparing the performance of populations of neurons from distinct cortical areas in controlling the movements of a robot arm. Then, by selecting only neurons from a given cortical area to feed into the realtime model, one could measure whether the contribution of these neurons can be enhanced by visual and/or tactile feedback, which are used to indicate the error between the movements of the animal’s arm and the robot arm. This apparatus could also be used to measure how fast changes in the kinematic properties of the motor actuator (robot arm) or in the task contingency affect the encoding of motor information by distinct populations of neurons19. For example, suppose that animals are rewarded for using their brain activity to make a robot arm move in the same direction as their own hand. By introducing a rotation in the output of the model that converts the animal’s brain activity into robot arm movements, and using visual and
tactile feedback to illustrate the mismatch between the robot arm and the animal’s hand movements, one can test whether populations of neurons that encode information about hand movements adapt to account for the transformation of the model’s output. Results obtained at the single-neuron level indicate that this would occur19–21. The use of sensory feedback or reward in operant conditioning of neural activity has a long tradition in neuroscience20–24. Indeed, the concept of building a BMI to restore motor function was heavily influenced by the early studies of Fetz and colleagues20,21,23,24. These landmark experiments showed that monkeys could learn to increase the firing rate of individual cortical neurons if they were provided with visual or auditory feedback — which signalled the unit firing rate — combined with a food reward for attaining high rates. This conditioning occurred over a few training sessions so that when overtraining was achieved, monkeys could readily increase the firing rates of newly isolated cortical neurons to levels 50–500% higher than their normal rates23. Fetz also showed that removal of feedback and reward led to the return of normal firing rate levels23. The experimental apparatus illustrated in FIG. 2 also makes it possible to test the hypothesis that, with adequate visual and proprioceptive/tactile feedback, animals could not only learn to operate a robot arm efficiently, but they could also incorporate such an artificial device into the body representations that are present in motor and somatosensory cortical and subcortical structures13,25,26. Theoretically, such an assimilation could occur as a result of experience-dependent plastic reorganization, and could lead to significant improvement in the operation of a neuroprosthetic device. Such a demonstration would not only have considerable clinical relevance, but would also raise intriguing neurobiological and robotic questions27. BMIs and distributed neural coding
The examples described above indicate that BMIs could become useful tools for studying the dynamic and distributed nature of neural population coding in behaving animals. The concept of distributed coding has a long history in neuroscience. The initial formulation of this theoretical model — which guides most of the present thinking behind attempts to develop a neuroprosthetic device for restoring motor function — dates back to the work of the eclectic nineteenth-century English physician and Cambridge physicist, Thomas Young28. Young is perhaps best known for the double slit experiment, which led to the
VOLUME 4 | MAY 2003 | 4 1 9
291
Response magnitude
PERSPECTIVES
1
2
Thomas Young (1802) 3 R
O
Y
G
B
V
Stimulus continuum
P
Q
R
S
Figure 3 | Distributed neural coding in colour vision. In 1802, Thomas Young (left) introduced the concept of distributed neural coding in his classic trichromatic theory of colour vision. In his formulation, the combined response profile of only three retinal receptors (middle), tuned to respond to a broad spectrum of light wavelength (right), can account for the representation of any colour in the visible spectrum. P, Q, R, S, colour stimuli.
proposition of the principle of light interference, for his fundamental contributions to the theory of elasticity of materials, and for his efforts to decipher Egyptian hieroglyphics. His sole contribution to neuroscience was of equal stature to his other intellectual adventures. In a paper published in 1802 (REF. 28), Young proposed the trichromatic theory of colour vision (FIG. 3). With no anatomical or functional evidence, Young proposed that the combined action of just three classes of light receptor in the retina (later known as cones) could account for the complete spectrum of colour sensation experienced by humans.Young indicated (FIG. 3) that although these receptors could be specialized to respond maximally to the presence of one of the three main colours (red, blue and yellow), each would also be able to respond, albeit less strongly, to light of different wavelengths. In other words, each receptor would be broadly tuned to a large wavelength spectrum. Young’s formulation predicted that the collective or distributed response pattern of these three retinal receptors could be used to represent the wavelength (or colour) of any light stimulus in the visual spectrum unambiguously. Despite lacking any insight into the structure of the retina or the brain, Young’s ingenious formulation gave rise to the concept of distributed neural coding. In this scheme, the electrical activity of large and spatially distributed populations of neurons — rather than single cells — is responsible for representing the attributes of incoming sensory stimuli, or for generating the motor commands required for the production of a voluntary act. After Young, many neuroscientists contributed to the elaboration of the concept of distributed neural coding. Perhaps the most influential was Donald Hebb. In his classic book The Organization of Behavior 29, published in 1949, Hebb provided a cellular
420
| MAY 2003 | VOLUME 4
analogue to Young’s formulation by describing the brain entity that, according to him, would be responsible for the ‘grunt’ work of computing, storing and representing information in the central nervous system. Hebb proposed that these functions would be carried out by: “...the cell assembly, a diffuse structure comprising [brain] cells in the cortex and diencephalon, capable of acting briefly as a closed system, delivering facilitation to other such systems...”
The main reason that one can seriously consider using neuroprosthetic devices to restore motor function in paralysed patients is that motor information is widely distributed in populations of neurons in the primary motor cortex30 and other motor cortical areas5,17,18. This widespread dispersion of information within and between cortical areas might explain why random samples of relatively small populations of single neurons can provide enough information to reconstruct continuous three-dimensional hand trajectories produced by monkeys that were trained in simple motor tasks5,7. Further experimental evidence indicates that cortical and subcortical neural ensembles
“...preliminary results in our laboratory indicate that by sampling a few hundred neurons simultaneously and differentially weighting their contribution, one can extract multiple control signals from the same population of recorded neurons.”
can simultaneously represent multiple motor and sensory parameters31–33. Although there is debate on how this multiplexing is achieved, preliminary results in our laboratory indicate that by sampling a few hundred neurons simultaneously and differentially weighting their contribution, one can extract multiple control signals from the same population of recorded neurons. Further studies using BMIs in experimental animals could help to answer this and other fundamental questions in neural ensemble physiology. Converging on applications
Because BMIs offer a new way to study distributed neural processing, and because addressing some of these basic questions might influence the design of future neuroprosthetic devices, research on BMIs offers a unique opportunity for promoting convergent areas of investigation for both basic and applied neuroscientists. For example, there is much discussion in the literature regarding how many neurons need to be sampled to build a clinically viable BMI for restoring upper limb movements. Although the original studies suggested that a few hundred neurons could provide an ideal sample for driving such a BMI 5, a couple of laboratories have used small neuronal samples (8–30 neurons) to drive experimental BMIs with some success6,7. Clearly, this question is as pertinent to neuroscientists that are interested in how motor information is encoded in the brain as it is to biomedical engineers that are interested in designing and implementing a clinically relevant BMI. An appropriate answer requires the analysis of a number of factors. Three main arguments challenge the notion that small samples of neurons could be used to drive a clinically relevant BMI. First, to be considered as a viable therapeutic alternative, BMIs will have to produce a significant improvement in the patient’s quality of life. Before now, studies based on reduced samples of neurons have led to limited experimental demonstrations of motor control6,7,9. These include using cortical neuronal activity to control the movements of a computer cursor9 or to reproduce the direction and trajectory of hand movements for brief periods of time6,7. A recent study reported that monkeys operating a closed-loop control BMI driven by the combined activity of about 18 cortical neurons correctly completed 50% or fewer of the task trials7. As control of computer cursors can be achieved with non-invasive methods, such as EEG recordings and electromyogram activity8,10, proponents of BMIs based on small samples of neurons would have to demonstrate
www.nature.com/reviews/neuro
292
PERSPECTIVES much more elaborate and reliable levels of motor control to justify subjecting patients to the neurosurgical procedure that is required to render these neuroprostheses functional. A second argument against the potential clinical relevance of these BMIs is that any small reduction in the original population of recorded cells could impede the reliable function of the neuroprostheses. For instance, any minor postsurgical disruption in electrode properties, leading to an inability to record from the full original population of neurons, could render this type of BMI useless. Natural loss or death of just a couple of neurons could produce a similarly catastrophic effect. Indeed, the normal time-dependent reduction in neuronal yield that characterizes some methods for chronic multi-electrode recordings would lead to the same outcome. Third, relatively small changes in the physiological properties of these small samples of neurons (such as changes in tuning properties) could also reduce the effectiveness of such BMIs. Although adaptive learning algorithms can be used to counteract these changes5,7, variations in patterns of neuronal firing due to changes in attention or arousal can prove difficult to handle in real time, particularly if the sample of cells used to derive a neural population signal is very small. It could be argued that patient training using visual, auditory and tactile feedback signals might enhance the information content of individual neurons or small populations of cells. Even though sensory feedback will probably improve the performance of BMIs and reduce the overall neuronal sample required to operate them, the crucial demonstration that BMIs based on small populations of neurons can maintain high performance for months or years, despite losing a few neurons, is still lacking. As BMIs based on brain implants would have to maintain a high level of daily performance for many years, this important drawback alone almost certainly limits the clinical application of a design based on a small sample of neurons. Neural coding theory and BMI design
Ultimately, I believe that the design of a successful BMI for restoring control of upper limb movements will have to take into account general physiological principles of how motor signals underlying these movements are encoded in the primate brain. Moreover, instead of aiming solely to restore motor functions by controlling a computer cursor, these BMIs must be able to restore fundamental hand or arm movements by using either the patient’s limbs (the most difficult goal) or artificial devices — such as robot arms and specially designed exoskeletons34 — as their
NATURE REVIEWS | NEUROSCIENCE
motor actuators. Including a gripper in these artificial actuators would also be essential. Although these requirements make it more difficult to build these devices, the successful implementation of such a BMI would lead to significant benefits for severely paralysed patients, while providing undisputed clinical justification for the need for surgical intervention.
“Ultimately, I believe that the design of a successful BMI for restoring control of upper limb movements will have to take into account general physiological principles of how motor signals underlying these movements are encoded in the primate brain.” Recently, I proposed a neuroprosthetic design that could overcome the three problems identified in the discussion of BMIs based on small neuronal samples13. This design was based on five ‘physiologically inspired’ principles. The first principle proposes that motor information related to hand movements is represented in a distributed way in several cortical and subcortical structures that define the motor system. The second principle purports that, within each of these areas, multiple motor parameters (position, velocity, force, direction and so on) can be extracted in real time from the electrical activity of populations of neurons. This principle assumes that multiplexing of information by neural ensembles is a ubiquitous property of motor cortical areas in the frontal and parietal lobes. The third principle contends that to reproduce a given hand trajectory in a robot arm, one might need to sample from a small fraction (a few hundred) of all the neurons (several million) in each cortical motor area (and across the entire motor system) that modulate their firing rate before the onset of a hand movement5. As mentioned above, several independent empirical observations support this contention. These findings also raise the hypothesis that there is considerable redundancy in the encoding of motor parameters in each cortical area. In this context, one could conceive a potential coding scheme in which the minimal neuronal mass required to generate an appropriate arm/hand movement in a given trial is selected from a
large pool of cortical neurons that modulate their firing before the onset of a hand movement (Principle 1). As this minimal neuronal mass could be defined by different combinations of individual neurons (Principle 5, below) such a coding scheme would ensure that reliable motor outputs would continue to be produced, even if significant numbers of neurons were lost owing to lesions of the motor system. The fourth principle states that the physiological properties of cortical ensembles are adaptive and can change as a function of experience and training, so the existence of cortical and subcortical plasticity must be taken into account in the design of an efficient BMI. Finally, the fifth principle asserts that the same hand/arm movements can be produced by distinct spatiotemporal patterns of neural ensemble firing. In other words, on a single-trial basis, different combinations of single neurons from several cortical areas, producing distinct spatiotemporal sequences of neuronal firing, can encode the same movement. Experimental evidence supports three of these principles; the third and fifth principles are still hypothetical. However, preliminary evidence obtained in our laboratory indicates that they have merit and should be investigated further. A BMI design that takes into account these five principles would use chronic implants of high-density microwire arrays to sample the extracellular activity of 100–200 neurons from each of 3–5 cortical areas simultaneously. Using this approach, several motor control parameters could be simultaneously extracted from the recorded neuronal sample, enabling patients to achieve more elaborate control of artificial actuators — such as robot arms and grippers — to recreate various arm/hand movements aimed at increasing their independence (for example, feeding without assistance) or at improving their ability to interact with their surrounding environment (for example, controlling a wheelchair). Additional improvements in the ability to control actuators located in remote environments (such as robots in different rooms) could further improve quality of life for these patients. Overall, this BMI design would increase the chances of achieving robust, continuous performance and long-term reliability. By sampling from a large neuronal population from the onset, the performance of such a BMI would be much less affected by eventual problems with individual electrodes, reductions in neuronal samples or changes in the physiological properties of individual neurons. Indeed, loss of all but one of the implants could still provide enough information for the continuous operation of such a cortical neuroprosthesis. VOLUME 4 | MAY 2003 | 4 2 1
293
PERSPECTIVES 4.
Conclusions
During the last five years, a series of experimental studies has demonstrated the feasibility of building neuroprosthetic devices to restore basic motor functions in patients suffering from catastrophic body paralysis. As obstacles to bringing these devices to the clinical arena are overcome, further research on BMIs is also likely to spur the development of various new models to investigate the operation of the neural circuits that will be used as a source of brain signals to drive a new generation of neuroprostheses. The confluence of these two outcomes might lead to profound contributions to the study of distributed neural coding, and the design of new brain-controlled actuators aimed at minimizing the devastating motor impairments that are caused by a large repertoire of neurological disorders.
5.
6.
7.
8.
9.
10. 11.
12.
13.
Miguel Nicolelis is at the Departments of Neurobiology, Biomedical Engineering, and Psychological and Brain Sciences, Box 3209, Bryan Research Building, Room 327E, 101 Research Drive, Duke University Medical Center, Durham, North Carolina 27710, USA. e-mail: nicoleli@neuro.duke.edu doi:10.1038/nrn1105 1.
2.
3.
422
Nobunaga, A. I., Go, B. K. & Karunas, R. B. Recent demographic and injury trends in people served by the model spinal cord injury care systems. Arch. Phys. Med. Rehabil. 80, 1372–1382 (1999). Bomze, H. M., Bulsara, K. R., Iskandar, B. J., Caroni, P. & Skene, J. H. Spinal axon regeneration evoked by replacing two growth cone proteins in adult neurons. Nature Neurosci. 4, 38–43 (2001). Chapin, J. K., Moxon, K. A., Markowitz, R. S. & Nicolelis, M. A. L. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neurosci. 2, 664–670 (1999).
| MAY 2003 | VOLUME 4
14.
15.
16.
17.
18.
19.
20.
Talwar, S. K. et al. Rat navigation guided by remote control. Nature 417, 37–38 (2002). Wessberg, J. et al. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408, 361–365 (2000). Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R. & Donoghue, J. P. Instant neural control of a movement signal. Nature 416, 141–142 (2002). Taylor, D. M., Tillery, S. I. & Schwartz, A. B. Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832 (2002). Wolpaw, J. R., McFarland, D. J., Neat, G. W. & Forneris, C. A. An EEG-based brain-computer interface for cursor control. Electroencephalogr. Clin. Neurophysiol. 78, 252–259 (1991). Kennedy, P. R. & Bakay, R. A. Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport 9, 1707–1711 (1998). Birbaumer, N. et al. A spelling device for the paralysed. Nature 398, 297–298 (1999). Keith, M. W. et al. Tendon transfers and functional electrical stimulation for restoration of hand function in spinal cord injury. J. Hand Surg. 21, 89–99 (1996). Schmidt, E. M. Single neuron recording from motor cortex as a possible source of signals for control of external devices. Ann. Biomed. Eng. 8, 339–349 (1980). Nicolelis, M. A. L. Actions from thoughts. Nature 409, 403–407 (2001). Donoghue, J. P. Connecting cortex to machines: recent advances in brain interfaces. Nature Neurosci. Suppl. 5, 1085–1088 (2002). Nicolelis, M. A. L. & Ribeiro, S. Multi-electrode recordings: the next steps. Curr. Opin. Neurobiol. 12, 602–606 (2002). Le Masson, G., Renaud-Le Masson, S., Debay, D. & Bal, T. Feedback inhibition controls spike transfer in hybrid thalamic circuits. Nature 417, 854–858 (2002). Kalaska, J. F. & Crammond, D. J. Cerebral cortical mechanisms of reaching movements. Science 255, 1517–1523 (1992). Kalaska, J. F., Scott, S. H., Cisek, P. & Sergio, L. E. Cortical control of reaching movements. Curr. Opin. Neurobiol. 7, 849–859 (1997). Mitz, A. R., Godschalk, M. & Wise, S. P. Learningdependent neuronal activity in the premotor cortex: activity during the acquisition of conditional motor associations. J. Neurosci. 1, 1855–1872 (1991). Fetz, E. E. & Baker, M. A. Operantly conditioned patterns of precentral unit activity and correlated responses in adjacent cells and contralateral muscles. J. Neurophysiol. 36, 179–204 (1973).
21. Fetz, E. E. & Finocchio, D. V. Correlations between activity of motor cortex cells and arm muscles during operantly conditioned response patterns. Exp. Brain Res. 23, 217–240 (1975). 22. Olds, J. The limbic system and behavioral reinforcement. Prog. Brain Res. 27, 144–164 (1967). 23. Fetz, E. E. Operant conditioning of cortical unit activity. Science 163, 955–957 (1969). 24. Fetz, E. E. & Finocchio, D. V. Operant conditioning of specific patterns of neural and muscular activity. Science 174, 431–435 (1971). 25. Iriki, A., Tanka, M. & Iwamura, Y. Coding of modified body schema during tool use by macaque postcentral neurones. Neuroreport 7, 2325–2330 (1996). 26. Ishibashi, H. et al. Tool-use learning selectively induces expression of brain-derived neurotrophic factor, its receptor trkB, and neurotrophin 3 in the intraparietal multisensorycortex of monkeys. Cogn. Brain Res. 14, 3–9 (2002). 27. Brooks, R. A. Flesh and Machines: How Robots Will Change Us (Knopf, New York, 2002). 28. Young, T. On the theory of light and colours. Phil. Trans. R. Soc. Lond. 92, 12–48 (1802). 29. Hebb, D. O. Organization of Behavior (John Wiley & Sons, New York, 1949). 30. Georgopoulos, A. P., Schwartz, A. B. & Ketter, R. E. Neuronal population coding of movement direction. Science 233, 1416–1419 (1986). 31. Johnson, M. T. & Ebner, T. J. Processing of multiple kinematic signals in the cerebellum and motor cortices. Brain Res. Rev. 33, 155–168 (2000). 32. Messier, J. & Kalaska, J. F. Covariation of primate dorsal premotor cell activity with direction and amplitude during a memorized-delay reaching task. J. Neurophysiol. 84, 152–165 (2000). 33. Johnson, M. T. V., Mason, C. R. & Ebner, T. J. Central processes for the multiparametric control of arm movements in primates. Curr. Opin. Neurobiol. 11, 684–688 (2001). 34. Gribble, P. L. & Scott, S. H. Overlap of internal models in motor cortex for mechanical loads during reaching. Nature 417, 938–941 (2002).
Acknowledgements I thank S. Simon for his comments on the manuscript and G. Licholai for his help with statistical data on spinal cord injuries.
Online links DATABASE The following term in this article is linked online to: OMIM: http://www.ncbi.nlm.nih.gov/Omim/ Parkinson disease Access to this interactive links box is free online.
www.nature.com/reviews/neuro
294 Review
TRENDS in Neurosciences
Vol.29 No.9
Brain–machine interfaces: past, present and future Mikhail A. Lebedev1 and Miguel A.L. Nicolelis2 1
Department of Neurobiology and Center for Neuroengineering, Duke University, Durham, NC 27710, USA Department of Biomedical Engineering and Department of Psychological and Brain Sciences, Duke University, Durham, NC 27710, USA
2
Since the original demonstration that electrical activity generated by ensembles of cortical neurons can be employed directly to control a robotic manipulator, research on brain–machine interfaces (BMIs) has experienced an impressive growth. Today BMIs designed for both experimental and clinical studies can translate raw neuronal signals into motor commands that reproduce arm reaching and hand grasping movements in artificial actuators. Clearly, these developments hold promise for the restoration of limb mobility in paralyzed subjects. However, as we review here, before this goal can be reached several bottlenecks have to be passed. These include designing a fully implantable biocompatible recording device, further developing real-time computational algorithms, introducing a method for providing the brain with sensory feedback from the actuators, and designing and building artificial prostheses that can be controlled directly by brain-derived signals. By reaching these milestones, future BMIs will be able to drive and control revolutionary prostheses that feel and act like the human arm. Introduction Less than a decade ago, hardly anyone could have predicted that attempts to build direct functional interfaces between brains and artificial devices, such as computers and robotic limbs, would have succeeded so readily, and in the process would have led to the establishment of a new area at the frontier of systems neuroscience. Born as a highly multidisciplinary field, basic research on brain–machine interfaces (BMIs) has moved at a stunning pace since the first experimental demonstration in1999 that ensembles of cortical neurons could directly control a robotic manipulator [1]. Since then, a continuous stream of research papers has kindled an enormous interest in BMIs among the scientific community and the lay public. This interest stems from the considerable potential of this technology for restoration of motor behaviors in severely handicapped patients. Indeed, BMIs have been primarily conceived as a potential new therapy to restore motor control in severely disabled patients, particularly those suffering from devastating conditions such as amyotrophic lateral
Corresponding author: Nicolelis, M.A.L. (nicoleli@neuro.duke.edu) Available online 21 July 2006. www.sciencedirect.com
sclerosis (ALS), spinal cord injury, stroke and cerebral palsy. As this technology advances and the risks of invasive brain recordings decrease, BMIs might also hold promise for amputees. In addition to the systems controlling upperlimb prostheses, BMIs dedicated to the restoration of locomotion and speech are likely to emerge. However, such stellar progress also breeds unrealistic expectations that such a future is just around the corner. Thus, the understandable eagerness in attaining the lofty goal of helping severely disabled patients has to be carefully calibrated by an objective analysis of the current state and future directions of the field. Such analysis indicates that, despite the optimism raised by a barrage of new accomplishments, there are still many issues that preclude straightforward translation of experimental BMIs into clinical applications. Indeed, most of the invasive BMIs have been tested only in experimental animals. Thus, despite recent enthusiasm to move emergent, and in some cases not thoroughly tested, BMI-related technology into clinical trials, much experimentation remains to be done before BMIs can become a safe and efficient rehabilitation tool. Here, we highlight some of the fundamental obstacles faced by BMI research and propose a series of milestones that can transform recent experimental advances into viable clinical applications in the next 10–20 years. The roadmap detailed here takes into account the recent history of the field, the factors that influenced its growth, and a critical analysis of the published work. Non-invasive BMIs Figure 1 depicts a classification of the BMIs (or brain–computer interfaces, BCIs) developed during the past decade. The first feature that distinguishes BMIs is whether they utilize invasive (i.e. intra-cranial) or non-invasive methods of electrophysiological recordings. Non-invasive systems primarily exploit electroencephalograms (EEGs) to control computer cursors or other devices. This approach has proved useful for helping paralyzed or ‘locked in’ patients develop ways of communication with the external world [2–11]. However, despite having the great advantage of not exposing the patient to the risks of brain surgery, EEG-based techniques provide communication channels of limited capacity. Their typical transfer rate is currently 5–25 bits s 1 [2,11]. Although such a transfer rate might not be sufficient to control the movements of an arm or leg prosthesis that has multiple degrees of freedom, past and recent research in this field seems to
0166-2236/$ – see front matter ß 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.tins.2006.07.004
295 Review
TRENDS in Neurosciences
Vol.29 No.9
537
Figure 1. Classification of brain–machine interfaces. Abbreviations: BMI, brain machine interface; EEG, electroencephalogram; LFP, local field potential; M1, primary motor cortex; PP, posterior parietal cortex.
indicate that EEG-based BMIs are likely to continue to offer some practical solutions (e.g. cursor control, communication, computer operation and wheelchair control) for patients in the future. Original attempts to provide subjects with feedback signals derived from their own brain activity were made in the 1960s and 1970s. Primarily, these attempts were aimed at enabling human subjects to gain voluntary control over brain rhythms. Nowlis and Kamiya claimed that, after training with an EEG biofeedback, human subjects acquired an ability to detect their own alpha rhythms [12], and this claim was later scrutinized by Plotkin [13]. Sterman and colleagues came to similar conclusions by utilizing the sensorimotor mu rhythm in cats [14] and humans [15]. In addition, Black operantly conditioned dogs to control their hippocampal theta rhythm [16]. Clearly, the results of these pioneering experiments helped pave the way for the introduction, a few years later, of EEGbased BCIs. Generally, EEG-based BCIs try to decipher the subject’s voluntary intentions and decisions through measurements of the combined electrical activity of massive neuronal populations. As such, both the spatial and temporal resolution of EEGs become limited owing to the overlapping electrical activity generated by different cortical areas. Furthermore, during the passive conductance of these signals through brain tissue, bone and skin, resolution is also lost owing to the low-pass filtering of the EEG signals. www.sciencedirect.com
EEGs are also susceptible to electromyographic (EMG), electrooculographic (EOG) and mechanical artifacts. Despite these well-known shortcomings, EEG techniques can detect modulations of brain activity that correlate with visual stimuli, gaze angle, voluntary intentions and cognitive states. These properties have led to development of several classes of EEG-based systems, which differ according to the cortical areas recorded, the features of EEG signals extracted, and the sensory modality providing feedback to subjects. One class of BCIs makes use of visual evoked potentials (VEPs). These BCIs detect the VEPs that occur when subjects look at particular items on a computer screen [17,18] or attend to them [19]. BCIs based on the P300 evoked potential uncover the subjects’ choices by distinguishing parietal cortex responses to the preferred versus non-preferred stimuli [20–22]. Several BCI designs continuously drive computer cursors. Both slow cortical potentials, recorded over several cortical areas [23], and faster mu (8–12 Hz) and beta (18–26 Hz) rhythms, recorded over sensorimotor cortex [24–26], have been exploited in such BCIs. For example, one such system relies on event-related synchronization and desynchronization of the EEGs associated with motor imagery [25,27]. Training to operate EEG-based BCIs can take many days [2]. Visual feedback is the essential part of such training. Some BCI designs rely on the subjects’ ability to develop control of their own brain activity using biofeedback, whereas others utilize classifier algorithms that
296 538
Review
TRENDS in Neurosciences Vol.29 No.9
recognize EEG patterns related to particular voluntary intentions. Recently, adaptive algorithms that constantly update the classifier parameters during training have been implemented [26]. Several strategies have also been proposed to provide feedback to users of EEG-based BCIs. For instance, virtual-reality systems can provide a realistic feedback that can be efficient for BCI training [28]. In a recent demonstration of this approach, subjects navigated through a virtual environment by imagining themselves walking [29]. In an effort to improve the resolution of brain potentials monitored by the BCIs, more invasive recording methods, such as electrocorticograms (ECoGs) recorded by subdural electrodes, have been introduced. ECoGs sample neuronal activity from smaller cortical areas than conventional EEGs. In addition, they contain higher-frequency gamma rhythms (>30 Hz). Consequently, ECoG-based BCIs are expected to have better accuracy and shorter training times than BCIs based on EEGs [30]. EEG-based BCIs have been implemented as solutions for patients suffering from various degrees of body paralysis. These BCIs (in the case of patients with advanced ALS) enable control of computer cursors, which the patients use to communicate with the external world or to indicate their intentions. The first successful and most well received application of such an approach was based on the utilization of slow cortical potentials to control a computer-aided spelling system [3,31]. BCIs based on mu and beta rhythms have also been tested in severely paralyzed people [32]. One study reported that a tetraplegic patient, aided by a BCI that detected beta waves in his sensorimotor cortex and activated a functional electrical stimulation device, learned to grasp objects using his paralyzed hand [33]. A motor imagery-based system [32], coupled to an implanted neuroprosthesis system [34] (Freehandß ) has been used to help a partially paralyzed patient. In addition, tetraplegic patients were able to gain some degree of control of the P300-based BCI [21]. Off-line analyses showed that P300 potentials can be used to obtain information about stimulus selections made by patients with ALS [22]. In addition to using EEGs, imaging techniques such as functional magnetic resonance imaging (fMRI), have been explored as a new source of brain-derived signals to drive BCIs [35]. Although fMRI-based BCIs are not suitable for everyday use and suffer from temporal delays of several seconds, they have good spatial resolution and, most importantly, can sample the activity of deep brain structures. Recently, fMRI was used to measure brain activation during the operation of a BCI based on slow cortical potentials [4]. Myolectric systems that make use of voluntary activations of unaffected muscles in partially paralyzed subjects and amputees [36–39], and use these signals to control limb prostheses and exoskeletons, present an alternative to the existing non-invasive BCIs. Currently, these systems are more practical for everyday situations than EEGbased BCIs [11]. The details of their operation are beyond the scope of this review. www.sciencedirect.com
In summary, severely and partially paralyzed patients can reacquire basic forms of communication and motor control using EEG-based systems. Yet motor recovery obtained using these systems has been limited, and no clear breakthrough that could significantly enhance the power of EEG-based BCIs in the near future has been reported in the literature [11]. This by no means reduces the clinical utility of such systems. Some of them have improved the quality of life of patients, such as the BCI for spelling [3]. But if the goal of a BMI is to restore movements with multiple degrees of freedom through the control of an artificial prosthesis, the message from published evidence is clear: this task will require recording of highresolution signals from the brain, and this can be done using invasive approaches. Invasive BMIs Invasive BMI approaches are based on recordings from ensembles of single brain cells (also known as single units) or on the activity of multiple neurons (also known as multi-units). These approaches have their roots in the pioneering studies conducted by Fetz and colleagues in the 1960s and 1970s [40–45]. In these experiments, monkeys learned to control the activity of their cortical neurons voluntarily, aided by biofeedback indicating the firing rate of single neurons. A few years after these experiments, Edward Schmidt raised the possibility that voluntary motor commands could be extracted from raw cortical neural activity and used to control a prosthetic device designed to restore motor functions in severely paralyzed patients [46]. Largely owing to technical difficulties associated with obtaining the needed cortical signals and implementing real-time interfaces quickly enough, thorough experimental testing of Schmidt’s proposition took almost two decades to be accomplished. These bottlenecks were passed because of a series of experimental and technological breakthroughs that led to a new electrophysiological methodology for chronic, multi-site, multi-electrode recordings [47–51]. The BMI approach that relies on long-term recordings from large populations of neurons (100–400 units) evolved from experiments carried out in 1995 [47]. After the introduction of such an approach, a series of studies demonstrated that neuronal readout of tactile stimuli could be uncovered using pattern-recognition algorithms, such as artificial neural networks [52,53]. These developments paved the way for the first experiment in which neuronal population activity recorded in behaving rats enacted movements of a robotic device that had a single degree of freedom [1]. Soon after this first demonstration, a similar BMI approach was shown to work in New World [54] and rhesus monkeys [55–58]. As a result of these experimental efforts, in less than six years several laboratories reported BMIs that reproduced primate arm reaching [1,54–58] and the combination of reaching and grasping movements [57], using either computer cursors or robotic manipulators as actuators. During the past three years, most of the published studies on BMIs have been conducted in behaving rhesus monkeys. There are several important differences that distinguish these BMIs (Figure 1). These include:
297 Review
TRENDS in Neurosciences
the number of cortical implants (e.g. uni-site or multi-site recordings); the cortical location of implants (e.g. frontal or parietal cortex, or both); the type of neural signal recorded (local field potentials versus single-unit or multi-unit signals); and the size of the neural sample. With the exception of the BMIs used at Duke University (http://www. duke.edu/), all BMIs tested in monkeys have relied on single cortical site recordings either of local field potentials [59â&#x20AC;&#x201C;62] or of small samples (<30) of neurons or multi-units [55,56,63]. Most of these small-sample, single-area BMIs utilized neural signals recorded in the primary motor cortex [55,56], although one group has focused on BMIs that processed neural signals recorded in the posterior parietal cortex [64]. At Duke University, a BMI strategy has recently been implemented based on single-unit recordings made during intra-operative placement of deep-brain stimulators in Parkinsonian patients [65]. Principles of BMI operation Invasive BMIs rely on the physiological properties of individual cortical and subcortical neurons (or pools of neurons) that modulate their activity in association with movements. First documented four decades ago by Evarts [66â&#x20AC;&#x201C;68], such modulations are highly variable, from neuron to neuron and from trial to trial [69â&#x20AC;&#x201C;72]. Thus, as much as neighboring neurons might display highly distinct firing modulation patterns during the execution of a particular movement, single-neuron firing can vary substantially from one trial to the next, despite the fact that the overt movements remain virtually identical. Yet averaging across many trials reveals fairly consistent firing patterns. By the same token, averaging across large populations of neurons significantly reduces the variability of signals derived from single neurons [54,69]. Extracting motor control signals from the firing patterns of populations of neurons and using these control signals to reproduce motor behaviors in artificial actuators are the two key operations that a clinically viable BMI should perform flawlessly [51,73]. To be accepted by patients, BMI devices will also have to act in the same way and feel the same as the subjectsâ&#x20AC;&#x2122; own limbs. Recent findings suggest that this task might be accomplished by creating conditions under which the brain undergoes experience-dependent plasticity and assimilates the prosthetic limb as if it were part of the subjectâ&#x20AC;&#x2122;s own body. Until recently, such plasticity was achieved using visual feedback. However, a more efficient way to assimilate the prosthetic limb in the brain representation could be to use multiple artificial feedback signals, derived from pressure and position sensors placed on the prosthetic limb. These feedback signals would effectively train the brain to incorporate the properties of the artificial limb into the tuning characteristic of neurons located in cortical and subcortical areas that maintain representations of the subjectâ&#x20AC;&#x2122;s body. We predict that such plasticity will result in sensory and motor areas of the brain representing the prosthetic device. A proposed roadmap for the future of BMI research To achieve the ambitious goal of creating a clinically useful invasive BMI for restoring upper-limb mobility, one has to pass the following key bottlenecks: www.sciencedirect.com
Vol.29 No.9
539
Obtaining stable, very long-term recordings (i.e. over years) of large populations of neurons (i.e. hundreds to thousands) from multiple brain areas. This task encourages development of a new generation of biocompatible 3D electrode matrices that yield thousands of channels of recordings while producing little tissue damage at implantation and minimal inflammatory reaction thereafter. Developing computationally efficient algorithms, that can be incorporated into the BMI software, for translating neuronal activity into high-precision command signals capable of controlling an artificial actuator that has multiple degrees of freedom. Learning how to use brain plasticity to incorporate prosthetic devices into the body representation. This will make the prosthetic feel like the subjectâ&#x20AC;&#x2122;s own limb. Implementing a new generation of upper-limb prosthetics, capable of accepting brain-derived control signals to perform movements with multiple degrees of freedom. We now discuss some potential avenues for addressing the first three of these major challenges. A thorough discussion of the fourth challenge (i.e. engineering a new generation of prosthetic arms) is beyond the scope of this review. Long-term recordings of neuronal activity from multiple brain areas Although recording from single neurons is the first choice of neurophysiologists, multi-unit signals that comprise activity of a few neurons can also be efficiently used in BMI control [57]. In addition, several reports have suggested using local field potentials [59â&#x20AC;&#x201C;62]. It is conceivable that, in future, advanced neuroprosthetic devices will use hybrid solutions in which a combination of several types of neural signals are recorded and processed. Here, however, we focus on using single-unit and multi-unit signals as the primary input to a BMI. This choice raises a fundamental question: how many neurons does a BMI need to sample to produce effective motor outputs? This question, first raised several years ago [51,73], remains a matter of debate. Some groups [55,56,63] have strongly claimed that recordings from a small number of neurons can be sufficient for good performance of a BMI. Selected populations of highly tuned neurons can indeed accurately predict movement parameters [74]. However, highly tuned neurons are rare in a typical random sample of cortical cells. Given that the neuronal yield of all chronic recording techniques is produced by random sampling of neurons, it is unrealistic to expect that a large fraction of these cells will be highly tuned to a particular motor variable. Moreover, it would be even more unrealistic to expect that a small neuronal sample would represent several variables of interest. Therefore, large samples of recorded neurons are preferable, at the very least to enable selection of a sufficient number of highly tuned neurons. Besides, the reason for relying on large neuronal populations goes far beyond the issue of selecting the best performing cells. Both the accuracy [54,57,70] and the
298 540
Review
TRENDS in Neurosciences Vol.29 No.9
reliability [69] of predictions improve considerably with the number of simultaneously recorded neurons, because motor information seems to be represented in the cortex in a highly distributed way. Thus, as the neuronal sample increases in size, errors related to individual neuron firing variability decrease significantly [69]. So looking into the future, it seems unlikely that invasive BMIs based on a small group of neurons will be capable of continuously reproducing in artificial limbs the range of fine movements normally performed by the human arm and hand. Currently, chronically implanted microwire arrays offer the best compromise between safety, recording longevity and neuronal yield required to operate BMIs [48,51, 54,57,58,73]. It is clear that this methodology will continue to be applied in experimental settings, but several significant improvements are required before it becomes fully applicable for long-term (months to years) chronic clinical applications in humans. First and foremost, the broad and challenging issue of biological compatibility [75–80] has to be properly addressed and solved. Second, fully implantable technologies using wireless headstages for amplification of neuronal signals have to be implemented to reduce the risks of infection introduced by the use of cables that connect brain implants to external hardware. Current microelectrode designs typically enable good quality recordings to be made for several months. In certain cases and species, these recordings can last for several years [81]. However, recording quality often deteriorates, probably owing to a process of electrode encapsulation by fibrous tissue and cell death in the vicinity of the electrode [77]. Some authors have proposed that electrodes that contain neurotrophic medium [82–85], or are coated with factors that promote neuronal growth (e.g. nerve growth factor, brain-derived neurotrophic factor or laminin) and various anti-inflammatory compounds (e.g. dexamethazone) [77,86–91], might be a way to cope with encapsulation. Currently, it is unclear whether these approaches will be useful. Efforts to resolve the biocompatibility issues will probably have to be pursued in parallel with the development of new 3D electrode matrices, which should aim to increase the average yield to thousands of neuronal signals per implanted probe. Current alternatives to such microwires (e.g. the Utah probe [92], which implements arrays of rigid, single-ended electrodes) have yet to prove their adequacy to support the haste in which this technology was moved into clinical applications. Judging from the published evidence, such arrays are best suited to sample neuronal activity from flat surfaces of cortical gyri in animal experiments. However, this design might not be suitable for longterm use in human patients. In addition to issues of how electrodes are inserted into the cortex, the inability to sample from deep cortical layers, and many unanswered biocompatibility questions, the reliability of the recording system utilized by currently available probes is also compromised by the continuous stress of a daily routine that involves external cables and plugging and unplugging of external head-stages. These operations carry a risk of causing tissue damage, bleeding and brain infection. Such a risk of failure, which might be tolerable in animal experiments, is unwanted in practical applications www.sciencedirect.com
for humans. Cyberkinetics Neurotechnology Systems (http://www.cyberkineticsinc.com/content/index.jsp) has recently started clinical trials in severely paralyzed patients of a BCI based on a probe developed at the University of Utah (http://www.utah.edu/). Because no peer-reviewed publication has appeared related to this work, the exact outcome of this study remains unknown. From these considerations, it is clear that the issues related to the long-term functionality of implantable microelectrodes, and to the development of fully implantable electronic devices for amplification of a large number of neuronal signals and their wireless transmission to the actuator, are the major technological challenges that will determine the success or failure of future clinical applications of BMI technology. These technological developments are necessary not only to increase the practical usefulness of the BMI (more neurons mean better stability and accuracy) but also to ensure that risks to patient health are minimized. Auspiciously, telemetry transmission methods [93–97] for effective wireless transmission of multichannel neuronal signals have already started to appear in the literature [98,99]. These solutions are currently being tested in animal experiments. Many new ideas of how to improve neuronal recordings have been proposed recently. These range from ceramicbased multi-electrode arrays [100] to nanotechnology probes that access the brain through the vascular system [101]. In this latter design, probes record neuronal activity without compromising brain parenchyma. Undoubtedly, much more testing will be needed to conclude which of these ideas are viable. Developing algorithms for translating neuronal activity into command signals for artificial actuators Currently, neuroscientists are far from obtaining a clear understanding of how motor and cognitive information is processed by the populations of neurons that form large brain circuits. Rate encoding, temporal encoding and population encoding principles have been suggested, and various experimental paradigms, including BMIs, have been developed to test the validity of these concepts. However, precise knowledge of computations performed by brain circuits is not crucial for the construction of clinically relevant BMIs. Mostly, BMI platforms take advantage of the well known correlation between discharges of cortical neurons and motor parameters of interest, and perform a reverse operation: they predict motor parameters from patterns of neuronal firing. Generally, predictions of motor parameters do not signify a causal relationship between the neuronal activity and the generation of movements. One type of correlation between neuronal activity and movement is known as directional tuning [102,103], and correlations of neuronal activity with kinematic [104–106] and kinetic [107–109] parameters of movements have also been described. Although a wealth of linear and nonlinear algorithms for translating neuronal activity into commands to artificial actuators have been suggested [1,54,56,57,70,110– 116], relatively simple multiple linear regression models have proved to be efficient in many practical BMI designs [54,55,57,58,65,117]. In these models, predicted motor
299 Review
TRENDS in Neurosciences
parameters are derived from the weighted sums of neuronal rates of firing, measured at different time points in the past. The number of regressors in the model and the time window used for predictions can be optimized for each concrete BMI application [70,74,117]. Linear methods, such as population vector predictions, can incorporate adaptive algorithms that continuously update the model parameters while the subject trains [56]. Basic research using the BMI paradigm has lent support to some fundamental principles of neural information coding. For example, studies in which several independent linear models were run in parallel revealed that several motor parameters, such as arm position, velocity, acceleration and hand gripping force, could be predicted simultaneously by separate combinations of the activity of the same original pool of neurons [57]. This finding supports the notion that multiple motor parameters are processed by overlapping neuronal ensembles. As a corollary, a single cortical neuron can contribute to several predictions simultaneously. The choice of motor parameters extracted in future clinical BMIs will depend on the main rehabilitation or therapeutic goals of these applications. For example, an experimental BMI for reaching and grasping [57] predicted hand velocity and gripping force that matched the characteristics of a robotic arm equipped with a gripper. In the near future, this design could lead to the implementation of neuroprosthetic devices that help quadriplegic or ‘locked in’ patients to reach and grasp objects in the surrounding space. We can also envisage that BMIs that synthesize speech, based on neuronal signals recorded in intact speech-related regions, could one day help patients suffering from aphasia due to cortical strokes recover their ability to communicate. Future clinical applications might also take advantage of BMIs that predict EMG signals [117]. The main benefit of this design compared with a BMI that predicts kinematic parameters is that the signals of individual muscles can control biologically-inspired devices, which would produce a whole range of actuator stiffness. This is an important property needed for a future generation of prosthetic limbs that should be able to manipulate objects with different physical characteristics. Another powerful future application for BMIs that decode EMGs is the construction of brain–muscle interfaces that directly stimulate the muscles of paralyzed patients and thereby restore mobility by using the patient’s own musculoskeletal apparatus [37,118,119]. Such BMIs are likely to be much more acceptable to patients, particularly because the hardware needed for amplification, transmission and processing of brain-derived control signals, and the muscle stimulators driven by these neural signals, such as the BION [120], can be entirely encased in the patient’s body. In the future, it is conceivable that such BMIs could merge the current cortically driven paradigm with methods and new technologies developed in the field of functional electrical stimulation. Early BMI designs focused on decoding motor parameters from neuronal activity [1,54–56,117]. More recently, it was suggested that BMIs that decode cognitive signals, for example those that decode intended reach direction during the delay periods preceding movement www.sciencedirect.com
Vol.29 No.9
541
execution [64,121–123], could also be efficient. Although this idea is very attractive, a BMI based exclusively on cognitive signals cannot execute continuous control of movement parameters. Instead, it decodes higher-order characteristics of movements, such as reach direction or characteristics of objects being grasped, and delegates lower-order details of motor execution to the actuator controller. Recently, we have proposed that a hybrid BMI, based on a shared control mode of operation [124], can improve the accuracy with which the prosthetic implements the voluntary intentions of the subject. A sharedcontrol mode of operation would be achieved by a combination of high-order brain-derived signals, conveying the subject’s voluntary intentions, and low-level artificial ‘reflex-like’ circuits, designed to improve the precision with which prosthetic limb movements are generated. In the future, BMIs that take advantage of the higherorder neuronal representations of movement-related variables will also emerge. These BMIs will probably derive information from representations of movement sequences [125,126], reference frames [127–130], potential movement targets [131] and simultaneous encoding of multiple spatial variables, such as movement direction, orientation of selective spatial attention, and gaze angle [132,133]. Utilization of this wide array of information will endow BMIs with a much more flexible control of prosthetic limbs. In the same context, we also believe that future BMIs will take advantage of new insights on how neural circuits encode temporal characteristics of movements. Recent studies [134–137] have indicated that a rather distributed representation of temporal information might exist in the brain. Recordings obtained from primary motor and premotor cortical ensembles while monkeys performed selftimed button presses [138] enabled prediction of both the time that had elapsed since the monkey pressed the button and the time until the button would be released. Because any motor behavior has a temporal structure, and because episodes of movement execution are typically intermingled with periods of immobility during which movements are being prepared, a BMI that decodes behavioral time will be able to inhibit movements of the actuator during waiting periods and release the actuator at appropriate times. These operations will be based on the voluntary intentions of the user. Making use of brain plasticity to incorporate prosthetic devices into the body representation Controlling an artificial actuator through a BMI can be thought of as a process somewhat similar to the operation required by subjects to operate tools – a capacity that is inherent only in higher primates such as chimpanzees and humans [139]. Almost 100 years ago [140], Head and Holmes suggested that the ‘body schema’– that is, the internal brain representations of one’s body – could extend itself to include a wielded tool. This idea was validated by the experimental demonstration that cortical neurons extend their visual receptive fields along the length of a rake used by monkeys to retrieve distant objects [141]. Psychophysics experiments also support the notion that tool usage leads to remapping of the ‘body schema’ in
300 542
Review
TRENDS in Neurosciences Vol.29 No.9
humans [142,143]. Accordingly, a recent neuroimaging study [144] described specific activations of the right ventral premotor cortex during manipulation of a myoelectric prosthetic hand. Altogether, these results suggest that long-term usage of an artificial actuator directly controlled by brain activity might lead to substantial cortical and subcortical remapping. As such, this process might elicit the vivid perceptual experience that the artificial actuator becomes an extension of the subject’s body rather than a mere tool. This suggestion is supported by the report of primary sensorimotor cortex activation during perceived voluntary movements of phantom limbs in amputees [145]. Perhaps the most stunning demonstration of tool assimilation by animals was observed when both rats and primates learned to operate an actuator through a BMI, without the need to move their own limbs [1,56–58]. In these experiments, decoding algorithms were initially trained to predict limb movements of animals from the activity of neuronal populations. Remarkably, after these animals started to control the actuator directly using their neuronal activity, their limbs eventually stopped moving, while the animals continued to control the actuator by generating proper modulations of their cortical neurons. Interestingly, during these episodes neuronal tuning to movements of the subject’s own limbs decreased while the animals continued to control the artificial actuator by their brain activity [58]. The most parsimonious interpretation of this finding is that the brain was capable of undergoing a gradual assimilation of the actuator within the same maps that represented the body [57,58]. Neuronal mechanisms mediating such plasticity are far from being understood.
However, it is fair to state that there is a growing consensus in the literature that continuous BMI operations in primates lead to physiological changes in neuronal tuning, which include changes in preferred direction and direction tuning strength of neurons [56–58]. In addition, broad changes in pair-wise neuronal correlation can be detected after BMIs are switched to operate fully under brain-control mode [57,58]. Along with these physiological adaptations of neuronal firing patterns, behavioral performance improves as animals learn to operate BMIs effectively [56–58]. Initial training to operate a BMI is characterized by an increase in neuronal firing rate variance, which cannot be simply explained by changes in limb or actuator movements [146]. As the quality of BMI control improves, initial elevation of neuronal firing variability subsides. Plastic changes in neuronal firing patterns during BMI control, leading to the physiological incorporation of the artificial actuator properties into neuronal space, could account for these changes in firing rate variance. This interpretation is in accord with the theory of optimal feedback control [147–149]. According to this theory, a motor system acts as a stochastic feedback controller that optimizes only those motor parameters that are necessary to achieve the goals of a particular task. During the brain-control mode of operation of a BMI, the goals of a motor task are achieved only by direct brain control of an artificial actuator. Thus, in terms of optimal feedback control theory, neuronal ensembles should adapt their physiological tuning properties to represent better the goal-related variables of the task performed by the BMI.
Figure 2. A BMI with multiple feedback loops being developed at the Duke University Center for Neuroengineering. A rhesus macaque is operating an artificial robotic manipulator that reaches and grasps different objects. The manipulator is equipped with touch, proximity and position sensors. Signals from the sensors are delivered to the control computer (right), which processes them and converts to microstimulation pulses delivered to the sensory areas in the brain of the monkey, to provide it with feedback information (red loop). A series of microstimulation pulses is illustrated in the inset on the left. Neuronal activity is recorded in multiple brain areas and translated to commands to the actuator, via the control computer and multiple decoding algorithms (blue loop). Arm position is monitored using an optical tracking system that tracks the position of several markers mounted on the arm (green loop). We hypothesize that continuous operation of this interface would lead to incorporation of the external actuator into the representation of the body in the brain. Figure designed by Nathan Fitzsimmons. www.sciencedirect.com
301 Review
TRENDS in Neurosciences
Vol.29 No.9
543
instructions provided by cortical microstimulation, and their behavioral performance eventually surpassed the level of performance observed when the vibratory stimulus was applied to their skin. These results suggest that cortical microstimulation might become a useful way to deliver long-term feedback from prosthetic limbs controlled by a BMI, and might contribute to the development of a completely new generation of neuroprosthetic devices for restoring various motor behaviors in severely impaired patients.
Figure 3. How a fully-implantable BMI could restore limb mobility in paralyzed subjects or amputees. Although the details of this system have to be worked out through future research, it is clear that the BMI for human clinical applications should be encased in the patient’s body as much as possible. Wireless telemetry offers a viable solution for this purpose. The prosthesis not only should have the functionality of the human arm in terms of power and accuracy of the actuators, but also should be equipped with the sensors of touch and position from which signals can be transmitted back to the subject’s brain.
Making the prosthetic feel like the subject’s own limb using microstimulation of cortical sensory areas Peripheral tactile and proprioceptive signals contribute to the normal operation of one’s limbs and the perception that they are part of the body [142,143]. For a neuroprosthesis to behave and feel as a natural appendage of the subject’s body, it will have to be instrumented with various sensors that can provide multiple channels of ‘sensory’ information back to the subject’s brain. In most current BMI designs, animal subjects receive sensory information from the actuator through visual feedback [55–58]. Predictions of motor parameters are less stable in the absence of visual feedback [70] than when it is present [55–58]. Curiously, the use of tactile and proprioceptive-like feedback in BMI research remains largely unexplored. Recently, in collaboration with John Chapin, we have started to explore the intriguing possibility of delivering such sensory feedback information, generated in the actuator, to the brain through the use of multi-channel microstimulation of somatosensory cortical areas (Figure 2). Previous studies have shown that monkeys sense microstimulation patterns and can use them to guide their behavioral responses [150,151]. In a recent long-term study, owl monkeys could learn to guide their reaching movements by decoding vibratory stimuli applied to their arms [152]. Next, instead of vibratory stimulation, matching patterns of microstimulation were applied through the electrodes implanted in the primary somatosensory cortex [153]. Monkeys were still able to interpret correctly the www.sciencedirect.com
Concluding remarks Our vision of neuroprosthetic developments that might emerge in the next 10–20 years includes a fully implantable recording system that wirelessly transmits multiple streams of electrical signals, derived from thousands of neurons, to a BMI capable of decoding spatial and temporal characteristics of movements and intermittent periods of immobility, in addition to cognitive characteristics of the intended actions (Figure 3). This BMI would utilize a combination of high-order motor commands, derived from cortical and subcortical neuronal activity, and peripheral low-level control signals, derived from artificial ‘reflex-like’ control loops. Such a shared-control mode of BMI operation would either command an actuator with multiple degrees of freedom or directly stimulate multiple peripheral nerves and muscles through implantable stimulators. Highly instrumented artificial actuators, containing arrays of touch and position sensors, would generate multiple streams of sensory feedback signals that could be directly delivered to cortical and/or subcortical somatosensory areas of the subject’s brain, through spatiotemporal patterns of multi-channel microstimulation. Such closed-loop, hybrid BMIs would get one step closer to the dream of restoring a large repertoire of motor functions to a multitude of patients who currently have very few options for regaining their mobility. References 1 Chapin, J.K. et al. (1999) Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat. Neurosci. 2, 664–670 2 Wolpaw, J.R. et al. (2002) Brain–computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791 3 Birbaumer, N. et al. (1999) A spelling device for the paralysed. Nature 398, 297–298 4 Hinterberger, T. et al. (2005) Neuronal mechanisms underlying control of a brain–computer interface. Eur. J. Neurosci. 21, 3169–3181 5 Kubler, A. et al. (2001) Brain–computer communication: unlocking the locked in. Psychol. Bull. 127, 358–375 6 Kubler, A. et al. (2001) Brain–computer communication: selfregulation of slow cortical potentials for verbal communication. Arch. Phys. Med. Rehabil. 82, 1533–1539 7 Obermaier, B. et al. (2003) Virtual keyboard controlled by spontaneous EEG activity. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 422–426 8 Obermaier, B. et al. (2001) Information transfer rate in a five-classes brain–computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 9, 283–288 9 Sheikh, H. et al. (2003) Electroencephalographic (EEG)-based communication: EEG control versus system performance in humans. Neurosci. Lett. 345, 89–92 10 Wolpaw, J.R. (2004) Brain-computer interfaces (BCIs) for communication and control: a mini-review. Suppl. Clin. Neurophysiol. 57, 607–613
302 544
Review
TRENDS in Neurosciences Vol.29 No.9
11 Birbaumer, N. (2006) Brain–computer-interface research: coming of age. Clin. Neurophysiol. 117, 479–483 12 Nowlis, D.P. and Kamiya, J. (1970) The control of electroencephalographic alpha rhythms through auditory feedback and the associated mental activity. Psychophysiology 6, 476–484 13 Plotkin, W.B. (1976) On the self-regulation of the occipital alpha rhythm: control strategies, states of consciousness, and the role of physiological feedback. J. Exp. Psychol. Gen. 105, 66–99 14 Wyricka, W. and Sterman, M. (1968) Instrumental conditioning of sensorimotor cortex EEG spindles in the waking cat. Psychol Behav 3, 703–707 15 Sterman, M.B. et al. (1974) Biofeedback training of the sensorimotor electroencephalogram rhythm in man: effects on epilepsy. Epilepsia 15, 395–416 16 Black, A.H. (1971) The direct control of neural processes by reward and punishment. Am. Sci. 59, 236–245 17 Middendorf, M. et al. (2000) Brain–computer interfaces based on the steady-state visual-evoked response. IEEE Trans. Rehabil. Eng. 8, 211–214 18 Sutter, E.E. and Tran, D. (1992) The field topography of ERG components in man – I. The photopic luminance response. Vision Res. 32, 433–446 19 Kelly, S.P. et al. (2005) Visual spatial attention control in an independent brain–computer interface. IEEE Trans. Biomed. Eng. 52, 1588–1596 20 Donchin, E. et al. (2000) The mental prosthesis: assessing the speed of a P300-based brain-computer interface. IEEE Trans. Rehabil. Eng. 8, 174–179 21 Piccione, F. et al. (2006) P300-based brain computer interface: reliability and performance in healthy and paralysed participants. Clin. Neurophysiol. 117, 531–537 22 Sellers, E.W. and Donchin, E. (2006) A P300-based brain–computer interface: initial tests by ALS patients. Clin. Neurophysiol. 117, 538– 548 23 Birbaumer, N. et al. (2000) The thought translation device (TTD) for completely paralyzed patients. IEEE Trans. Rehabil. Eng. 8, 190–193 24 Pfurtscheller, G. et al. (2006) Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks. NeuroImage 33, 153–159 25 Pfurtscheller, G. et al. (2003) Graz-BCI: state of the art and clinical applications. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 177–180 26 Wolpaw, J.R. and McFarland, D.J. (2004) Control of a twodimensional movement signal by a noninvasive brain–computer interface in humans. Proc. Natl. Acad. Sci. U. S. A. 101, 17849–17854 27 Pfurtscheller, G. and Lopes da Silva, F.H. (1999) Event-related EEG/ MEG synchronization and desynchronization: basic principles. Clin. Neurophysiol. 110, 1842–1857 28 Bayliss, J.D. and Ballard, D.H. (2000) A virtual reality testbed for brain–computer interface research. IEEE Trans. Rehabil. Eng. 8, 188–190 29 Pfurtscheller, G. et al. (2006) Walking from thought. Brain Res. 1071, 145–152 30 Leuthardt, E.C. et al. (2004) A brain–computer interface using electrocorticographic signals in humans. J Neural Eng 1, 63–71 31 Hinterberger, T. et al. (2003) A brain–computer interface (BCI) for the locked-in: comparison of different EEG classifications for the thought translation device. Clin. Neurophysiol. 114, 416–425 32 Kubler, A. et al. (2005) Patients with ALS can use sensorimotor rhythms to operate a brain–computer interface. Neurology 64, 1775–1777 33 Pfurtscheller, G. et al. (2003) ‘Thought’ control of functional electrical stimulation to restore hand grasp in a patient with tetraplegia. Neurosci. Lett. 351, 33–36 34 Keith, M.W. et al. (1989) Implantable functional neuromuscular stimulation in the tetraplegic hand. J. Hand Surg. (Am.) 14, 524–530 35 Weiskopf, N. et al. (2004) Self-regulation of local brain activity using real-time functional magnetic resonance imaging (fMRI). J. Physiol. (Paris) 98, 357–373 36 Light, C.M. et al. (2002) Intelligent multifunction myoelectric control of hand prostheses. J. Med. Eng. Technol. 26, 139–146 37 Navarro, X. et al. (2005) A critical review of interfaces with the peripheral nervous system for the control of neuroprostheses and hybrid bionic systems. J. Peripher. Nerv. Syst. 10, 229–258 www.sciencedirect.com
38 Okuno, R. et al. (2005) Compliant grasp in a myoelectric hand prosthesis. Controlling flexion angle and compliance with electromyogram signals. IEEE Eng. Med. Biol. Mag. 24, 48–56 39 Zecca, M. et al. (2002) Control of multifunctional prosthetic hands by processing the electromyographic signal. Crit. Rev. Biomed. Eng. 30, 459–485 40 Fetz, E.E. (1969) Operant conditioning of cortical unit activity. Science 163, 955–958 41 Fetz, E.E. (1992) Are movement parameters recognizably coded in activity of single neurons? Behav Brain Sci. 15, 679–690 42 Fetz, E.E. and Baker, M.A. (1973) Operantly conditioned patterns on precentral unit activity and correlated responses in adjacent cells and contralateral muscles. J. Neurophysiol. 36, 179–204 43 Fetz, E.E. and Finocchio, D.V. (1971) Operant conditioning of specific patterns of neural and muscular activity. Science 174, 431– 435 44 Fetz, E.E. and Finocchio, D.V. (1972) Operant conditioning of isolated activity in specific muscles and precentral cells. Brain Res. 40, 19– 23 45 Fetz, E.E. and Finocchio, D.V. (1975) Correlations between activity of motor cortex cells and arm muscles during operantly conditioned response patterns. Exp. Brain Res. 23, 217–240 46 Schmidt, E.M. (1980) Single neuron recording from motor cortex as a possible source of signals for control of external devices. Ann. Biomed. Eng. 8, 339–349 47 Nicolelis, M.A. et al. (1995) Sensorimotor encoding by synchronous neural ensemble activity at multiple levels of the somatosensory system. Science 268, 1353–1358 48 Nicolelis, M.A. et al. (2003) Chronic, multisite, multielectrode recordings in macaque monkeys. Proc. Natl. Acad. Sci. U. S. A. 100, 11041–11046 49 Nicolelis, M.A. et al. (1997) Reconstructing the engram: simultaneous, multisite, many single neuron recordings. Neuron 18, 529–537 50 Nicolelis, M.A. and Ribeiro, S. (2002) Multielectrode recordings: the next steps. Curr. Opin. Neurobiol. 12, 602–606 51 Nicolelis, M.A. (2001) Actions from thoughts. Nature 409, 403–407 52 Ghazanfar, A.A. et al. (2000) Encoding of tactile stimulus location by somatosensory thalamocortical ensembles. J. Neurosci. 20, 3761– 3775 53 Krupa, D.J. et al. (2004) Layer-specific somatosensory cortical activation during active tactile discrimination. Science 304, 1989– 1992 54 Wessberg, J. et al. (2000) Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408, 361– 365 55 Serruya, M.D. et al. (2002) Instant neural control of a movement signal. Nature 416, 141–142 56 Taylor, D.M. et al. (2002) Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832 57 Carmena, J.M. et al. (2003) Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol. 1, E42 58 Lebedev, M.A. et al. (2005) Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain–machine interface. J. Neurosci. 25, 4681–4693 59 Mehring, C. et al. (2003) Inference of hand movements from local field potentials in monkey motor cortex. Nat. Neurosci. 6, 1253–1254 60 Rickert, J. et al. (2005) Encoding of movement direction in different frequency ranges of motor cortical local field potentials. J. Neurosci. 25, 8815–8824 61 Pesaran, B. et al. (2002) Temporal structure in neuronal activity during working memory in macaque parietal cortex. Nat. Neurosci. 5, 805–811 62 Scherberger, H. et al. (2005) Cortical local field potential encodes movement intentions in the posterior parietal cortex. Neuron 46, 347– 354 63 Tillery, S.I. and Taylor, D.M. (2004) Signal acquisition and analysis for cortical control of neuroprosthetics. Curr. Opin. Neurobiol. 14, 758– 762 64 Musallam, S. et al. (2004) Cognitive control signals for neural prosthetics. Science 305, 258–262 65 Patil, P.G. et al. (2004) Ensemble recordings of human subcortical neurons as a source of motor control signals for a brain–machine interface. Neurosurgery 55, 27–35
303 Review
TRENDS in Neurosciences
66 Evarts, E.V. (1966) Pyramidal tract activity associated with a conditioned hand movement in the monkey. J. Neurophysiol. 29, 1011–1027 67 Evarts, E.V. (1968) Relation of pyramidal tract activity to force exerted during voluntary movement. J. Neurophysiol. 31, 14–27 68 Evarts, E.V. (1968) A technique for recording activity of subcortical neurons in moving animals. Electroencephalogr. Clin. Neurophysiol. 24, 83–86 69 Carmena, J.M. et al. (2005) Stable ensemble performance with singleneuron variability during reaching movements in primates. J. Neurosci. 25, 10712–10716 70 Wessberg, J. and Nicolelis, M.A. (2004) Optimizing a linear algorithm for real-time robotic control using chronic cortical ensemble recordings in monkeys. J. Cogn. Neurosci. 16, 1022–1035 71 Stein, R.B. et al. (2005) Neuronal variability: noise or part of the signal? Nat. Rev. Neurosci. 6, 389–397 72 Cohen, D. and Nicolelis, M.A. (2004) Reduction of single-neuron firing uncertainty by cortical ensembles during motor skill learning. J. Neurosci. 24, 3574–3582 73 Nicolelis, M.A. (2003) Brain–machine interfaces to restore motor function and probe neural circuits. Nat. Rev. Neurosci. 4, 417–422 74 Sanchez, J.C. et al. (2004) Ascertaining the importance of neurons to develop better brain–machine interfaces. IEEE Trans. Biomed. Eng. 51, 943–953 75 Dodson, R.F. et al. (1978) Cerebral tissue response to electrode implantation. Can. J. Neurol. Sci. 5, 443–446 76 Schultz, R.L. and Willey, T.J. (1976) The ultrastructure of the sheath around chronically implanted electrodes in brain. J. Neurocytol. 5, 621–642 77 Polikov, V.S. et al. (2005) Response of brain tissue to chronically implanted neural electrodes. J. Neurosci. Methods 148, 1–18 78 Tresco, P.A. et al. (2000) Cellular transplants as sources for therapeutic agents. Adv. Drug Deliv. Rev. 42, 3–27 79 Berry, M. et al. (1999) Cellular Responses to Penetrating CNS Injury. CRC Press 80 Landis, D.M. (1994) The early reactions of non-neuronal cells to brain injury. Annu. Rev. Neurosci. 17, 133–151 81 Sandler, A.J. et al. (2005) Long-term neuronal recordings from nonhuman primates. In 2005 Abstract Viewer and Itinerary Planner, Program No. 402.8, Society for Neuroscience Online (http://sfn.scholarone.com/) 82 Kennedy, P.R. (1989) The cone electrode: a long-term electrode that records from neurites grown onto its recording surface. J. Neurosci. Methods 29, 181–193 83 Kennedy, P.R. and Bakay, R.A. (1998) Restoration of neural output from a paralyzed patient by a direct brain connection. NeuroReport 9, 1707–1711 84 Kennedy, P.R. et al. (2000) Direct control of a computer from the human central nervous system. IEEE Trans. Rehabil. Eng. 8, 198–202 85 Kennedy, P.R. et al. (1992) The cone electrode: ultrastructural studies following long-term recording in rat and monkey cortex. Neurosci. Lett. 142, 89–94 86 Cui, X. et al. (2001) Surface modification of neural recording electrodes with conducting polymer/biomolecule blends. J. Biomed. Mater. Res. 56, 261–272 87 Rahimi, O. and Juliano, S.L. (2001) Transplants of NGF-secreting fibroblasts restore stimulus-evoked activity in barrel cortex of basalforebrain-lesioned rats. J. Neurophysiol. 86, 2081–2096 88 Ignatius, M.J. et al. (1998) Bioactive surface coatings for nanoscale instruments: effects on CNS neurons. J. Biomed. Mater. Res. 40, 264– 274 89 Kam, L. et al. (2002) Selective adhesion of astrocytes to surfaces modified with immobilized peptides. Biomaterials 23, 511–515 90 Cui, X. et al. (2003) In vivo studies of polypyrrole/peptide coated neural probes. Biomaterials 24, 777–787 91 Biran, R. et al. (2003) Directed nerve outgrowth is enhanced by engineered glial substrates. Exp. Neurol. 184, 141–152 92 Rousche, P.J. and Normann, R.A. (1998) Chronic recording capability of the Utah intracortical electrode array in cat sensory cortex. J. Neurosci. Methods 82, 1–15 93 Mohseni, P. et al. (2005) Wireless multichannel biopotential recording using an integrated FM telemetry circuit. IEEE Trans. Neural Syst. Rehabil. Eng. 13, 263–271 www.sciencedirect.com
Vol.29 No.9
545
94 Mackay, S. (1998) Bio-Medical Telemetry: Sensing and Transmitting Biological Information from Animals and Man. Wiley-IEEE Press 95 Knutti, J.W. et al. (1979) An integrated circuit approach to totally implantable telemetry systems. Biotelem. Patient Monit. 6, 95– 106 96 Claude, J.P. et al. (1979) Applications of totally implantable telemetry systems to chronic medical research. Biotelem. Patient Monit. 6, 160– 171 97 Chien, C.N. and Jaw, F.S. (2005) Miniature telemetry system for the recording of action and field potentials. J. Neurosci. Methods 147, 68– 73 98 Bossetti, C.A. et al. (2004) Transmission latencies in a telemetrylinked brain–machine interface. IEEE Trans. Biomed. Eng. 51, 919– 924 99 Morizio, J. et al. (2005) Fifteen-channel wireless headstage system for single-unit rat recordings. In 2005 Abstract Viewer and Itinerary Planner, Program No. 68.4, Society for Neuroscience Online (http:// sfn.scholarone.com/) 100 Moxon, K.A. et al. (2004) Ceramic-based multisite electrode arrays for chronic single-neuron recording. IEEE Trans. Biomed. Eng. 51, 647– 656 101 Llinas, R.R. et al. (2005) Neuro-vascular central nervous recording/ stimulating system: using nanotechnology probes. J Nanopart. Res. 7, 111–127 102 Georgopoulos, A.P. et al. (1988) Primate motor cortex and free arm movements to visual targets in three-dimensional space. II. Coding of the direction of movement by a neuronal population. J. Neurosci. 8, 2928–2937 103 Georgopoulos, A.P. et al. (1986) Neuronal population coding of movement direction. Science 233, 1416–1419 104 Ashe, J. and Georgopoulos, A.P. (1994) Movement parameters and neural activity in motor cortex and area 5. Cereb. Cortex 4, 590–600 105 Moran, D.W. and Schwartz, A.B. (1999) Motor cortical representation of speed and direction during reaching. J. Neurophysiol. 82, 2676– 2692 106 Averbeck, B.B. et al. (2005) Parietal representation of hand velocity in a copy task. J. Neurophysiol. 93, 508–518 107 Sergio, L.E. and Kalaska, J.F. (1998) Changes in the temporal pattern of primary motor cortex activity in a directional isometric force versus limb movement task. J. Neurophysiol. 80, 1577–1583 108 Sergio, L.E. et al. (2005) Motor cortex neural correlates of output kinematics and kinetics during isometric-force and arm-reaching tasks. J. Neurophysiol. 94, 2353–2378 109 Todorov, E. (2000) Direct cortical control of muscle activation in voluntary arm movements: a model. Nat. Neurosci. 3, 391– 398 110 Truccolo, W. et al. (2005) A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. J. Neurophysiol. 93, 1074–1089 111 Brown, E.N. et al. (2004) Multiple neural spike train data analysis: state-of-the-art and future challenges. Nat. Neurosci. 7, 456– 461 112 Brockwell, A.E. et al. (2004) Recursive Bayesian decoding of motor cortical signals by particle filtering. J. Neurophysiol. 91, 1899– 1907 113 Hu, J. et al. (2005) Feature detection in motor cortical spikes by principal component analysis. IEEE Trans. Neural Syst. Rehabil. Eng. 13, 256–262 114 Wu, W. et al. (2004) Modeling and decoding motor cortical activity using a switching Kalman filter. IEEE Trans. Biomed. Eng. 51, 933– 942 115 Kim, S.P. et al. (2003) Divide-and-conquer approach for brain machine interfaces: nonlinear mixture of competitive linear models. Neural Netw. 16, 865–871 116 Kemere, C. et al. (2004) Model-based neural decoding of reaching movements: a maximum likelihood approach. IEEE Trans. Biomed. Eng. 51, 925–932 117 Santucci, D.M. et al. (2005) Frontal and parietal cortical ensembles predict single-trial muscle activity during reaching movements in primates. Eur. J. Neurosci. 22, 1529–1540 118 Degnan, G.G. et al. (2002) Functional electrical stimulation in tetraplegic patients to restore hand function. J. Long Term Eff. Med. Implants 12, 175–188
304 546
Review
TRENDS in Neurosciences Vol.29 No.9
119 Peckham, P.H. and Knutson, J.S. (2005) Functional electrical stimulation for neuromuscular applications. Annu. Rev. Biomed. Eng. 7, 327–360 120 Loeb, G.E. et al. (2001) BION system for distributed neural prosthetic interfaces. Med. Eng. Phys. 23, 9–18 121 Hatsopoulos, N. et al. (2004) Decoding continuous and discrete motor behaviors using motor and premotor cortical ensembles. J. Neurophysiol. 92, 1165–1174 122 Rizzuto, D.S. et al. (2005) Spatial selectivity in human ventrolateral prefrontal cortex. Nat. Neurosci. 8, 415–417 123 Shenoy, K.V. et al. (2003) Neural prosthetic control signals from plan activity. NeuroReport 14, 591–596 124 Kim, H.K. et al. (2006) Continuous shared control stabilizes reach and grasping with brain–machine interfaces. IEEE Trans. Biomed. Eng. 53, 1164–1173 125 Hoshi, E. and Tanji, J. (2004) Differential roles of neuronal activity in the supplementary and presupplementary motor areas: from information retrieval to motor planning and execution. J. Neurophysiol. 92, 3482–3499 126 Lu, X. and Ashe, J. (2005) Anticipatory activity in primary motor cortex codes memorized movement sequences. Neuron 45, 967–973 127 Olson, C.R. (2003) Brain representation of object-centered space in monkeys and humans. Annu. Rev. Neurosci. 26, 331–354 128 Batista, A.P. et al. (1999) Reach plans in eye-centered coordinates. Science 285, 257–260 129 Battaglia-Mayer, A. et al. (2000) Early coding of reaching in the parietooccipital cortex. J. Neurophysiol. 83, 2374–2391 130 Graziano, M.S. and Gross, C.G. (1998) Spatial maps for the control of movement. Curr. Opin. Neurobiol. 8, 195–201 131 Cisek, P. and Kalaska, J.F. (2002) Simultaneous encoding of multiple potential reach directions in dorsal premotor cortex. J. Neurophysiol. 87, 1149–1154 132 Lebedev, M.A. and Wise, S.P. (2001) Tuning for the orientation of spatial attention in dorsal premotor cortex. Eur. J. Neurosci. 13, 1002–1008 133 Boussaoud, D. et al. (1993) Effects of gaze on apparent visual responses of frontal cortex neurons. Exp. Brain Res. 93, 423–434 134 Ivry, R.B. (1996) The representation of temporal information in perception and motor control. Curr. Opin. Neurobiol. 6, 851–857 135 Leon, M.I. and Shadlen, M.N. (2003) Representation of time by neurons in the posterior parietal cortex of the macaque. Neuron 38, 317–327 136 Roux, S. et al. (2003) Context-related representation of timing processes in monkey motor cortex. Eur. J. Neurosci. 18, 1011–1016 137 Matell, M.S. et al. (2003) Interval timing and the encoding of signal duration by ensembles of cortical and striatal neurons. Behav. Neurosci. 117, 760–773
138 O’Doherty, J. et al. (2005) Ensemble representation of time: interhemispheric communication involved? In 2005 Abstract Viewer and Itinerary Planner, Program No. 402.5, Society for Neuroscience Online (http://sfn.scholarone.com/) 139 Breuer, T. et al. (2005) First observation of tool use in wild gorillas. PLoS Biol. 3, e380 140 Head, H. and Holmes, G. (1911) Sensory disturbances from cerebral lesion. Brain 34, 102–254 141 Iriki, A. et al. (1996) Coding of modified body schema during tool use by macaque postcentral neurones. NeuroReport 7, 2325–2330 142 Maravita, A. et al. (2003) Multisensory integration and the body schema: close to hand and within reach. Curr. Biol. 13, R531–R539 143 Gurfinkel, V.S. et al. (1991) Body scheme concept and motor control. Body scheme in the postural automatisms regulation. In Intellectual Processes and Their Modelling, pp 24–53, Nauka 144 Maruishi, M. et al. (2004) Brain activation during manipulation of the myoelectric prosthetic hand: a functional magnetic resonance imaging study. NeuroImage 21, 1604–1611 145 Roux, F.E. et al. (2003) Cortical areas involved in virtual movement of phantom limbs: comparison with normal subjects. Neurosurgery 53, 1342–1352 146 Zacksenhouse, M. et al. (2005) Trends in firing rate statistics mirroring changes in test performance during training with brain machine interfaces. In 2005 Abstract Viewer and Itinerary Planner, Program No. 402.4, Society for Neuroscience Online (http:// sfn.scholarone.com/) 147 Todorov, E. and Jordan, M.I. (2002) Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 5, 1226–1235 148 Scott, S.H. (2004) Optimal feedback control and the neural basis of volitional motor control. Nat. Rev. Neurosci. 5, 532–546 149 Harris, C.M. and Wolpert, D.M. (1998) Signal-dependent noise determines motor planning. Nature 394, 780–784 150 Cohen, M.R. and Newsome, W.T. (2004) What electrical microstimulation has revealed about the neural basis of cognition. Curr. Opin. Neurobiol. 14, 169–177 151 Romo, R. et al. (2000) Sensing without touching: psychophysical performance based on cortical microstimulation. Neuron 26, 273–278 152 Sandler, A. et al. (2004) Primate somatosensorimotor learning: examining cue-related, association-related and motor-related responses in several cortical areas. In 2004 Abstract Viewer and Itinerary Planner, Program No. 884.7, Society for Neuroscience Online (http://sfn.scholarone.com/) 153 Fitzsimmons, N.A. et al. (2005) Long-term behavioral improvements in a reaching task cued by microstimulation of the primary somatosensory cortex. In 2005 Abstract Viewer and Itinerary Planner, Program No. 402.7, Society for Neuroscience Online (http://sfn.scholarone.com/)
Articles of interest in Current Opinion journals The development and modulation of nociceptive circuitry Xu Zhang and Lan Bao Current Opinion in Neurobiology DOI: 10.1016/j.conb.2006.06.002 Seeing sounds: visual and auditory interactions in the brain David A. Bulkin and Jennifer M. Groh Current Opinion in Neurobiology DOI: 10.1016/j.conb.2006.06.008 The decoding of electrosensory systems Eric S. Fortune Current Opinion in Neurobiology DOI: 10.1016/j.conb.2006.06.006 Touching on somatosensory specializations in mammals Kenneth C. Catania and Erin C. Henry Current Opinion in Neurobiology DOI: 10.1016/j.conb.2006.06.010 Lifelong learning: ocular dominance plasticity in mouse visual cortex Sonja B. Hofer, Thomas D. Mrsic-Flogel, Tobias Bonhoeffer and Mark Hübener Current Opinion in Neurobiology DOI: 10.1016/j.conb.2006.06.007 Molecular Trojan horses for blood-brain barrier drug delivery William M. Pardridge Current Opinion in Pharmacology DOI: 10.1016/j.coph.2006.06.001 www.sciencedirect.com
305
PeRsPectives opiNioN
Principles of neural ensemble physiology underlying the operation of brain–machine interfaces Miguel A. L. Nicolelis and Mikhail A. Lebedev
Abstract | Research on brain–machine interfaces has been ongoing for at least a decade. During this period, simultaneous recordings of the extracellular electrical activity of hundreds of individual neurons have been used for direct, real-time control of various artificial devices. Brain–machine interfaces have also added greatly to our knowledge of the fundamental physiological principles governing the operation of large neural ensembles. Further understanding of these principles is likely to have a key role in the future development of neuroprosthetics for restoring mobility in severely paralysed patients. Recent demonstrations of direct, real-time interfaces between living brain tissue and artificial devices, such as computer cursors, robots and mechanical prostheses, have opened new avenues for experimental and clinical investigation1–13. Interest in these brain–machine interfaces (BMIs) has been kindled by the contribution that they may make to the treatment or rehabilitation of patients suffering from severe motor disabilities6,8,9,14–17. As such, BMIs have rapidly become incorporated into the development of ‘neuroprosthetics’, devices that use neurophysiological signals from undamaged components of the central or peripheral nervous system to allow patients to regain motor capabilities. Indeed, several findings already point to a bright future for neuroprosthetics in many domains of rehabilitation medicine6,18–28. For example, scalp electroencephalography (EEG) signals linked to a computer have provided ‘locked-in’ patients with a channel of communication5,19,29–32. BMI technology, based on multi-electrode single-unit recordings — a technique originally introduced in rodents33–36 and later demonstrated in non-human primates1,7,11–13,37–45 — has yet to be transferred to clinical neuroprosthetics. Human trials in which paralysed patients were chronically implanted with cone electrodes5 or
intracortical multi-electrode arrays46 allowed the direct control of computer cursors. However, these trials also raised a number of issues that need to be addressed before the true clinical worth of invasive BMIs can be realized6. These include the reliability, safety and biocompatibility of chronic brain implants and the longevity of chronic recordings, areas that require greater attention if BMIs are to be safely moved into the clinical arena46–48. BMIs provide new insights1,4,6–13 into important questions pertaining to the central issue of information processing by the CNS during the generation of motor behaviours49–60. Many recent review articles have covered BMI methods6,10,25,61,62 and
…in addition to offering hope for a potential future therapy for the rehabilitation of severely paralysed patients, BMIs can be extremely useful platforms to test various ideas for how populations of neurons encode information in behaving animals.
530 | julY 2009 | VoluME 10
their potential implementation in medical rehabilitation18–20,22–25,27,28, and so these issues will not be covered here. Instead, we focus on how modern BMI research has led to the proposal, and in some cases validation, of various physiological principles governing the operation of large populations of cortical neurons in behaving mammals (animals performing a given action or movement). Neuronal ensemble recordings Although the first multi-electrode recording experiments in rhesus monkeys date back to the mid 1950s63,64, the current neurophysiological approach for sampling the extracellular activity of large populations of individual neurons in behaving animals emerged in the early 1980s65–70. At that time, most of the systems neuroscience community considered the single neuron to be the key functional unit of the CNS and, therefore, the main target for neurophysiological investigation71,72. Not surprisingly, the transition to neural ensemble recordings was slow and difficult. In addition to the enormous technological and technical barriers, few systems neurophysiologists saw any advantage in investing effort and resources into this paradigm shift. As a result, the concept of population coding 73–76, first proposed by Young 77 and further popularized by Hebb78, played a distant second fiddle to the single-neuron doctrine71,79–83 for many decades. Today, the weight of evidence supports the idea that distributed ensembles of neurons define the true physiological unit of the mammalian CNS73,84–86. However, this does not mean that neurophysiologists have given up examining the degree to which animal behaviour can be affected by singleneuron activity 87–90. Significant examples of the importance of single-neuron physiology to BMI research include the demonstration that single neurons can be conditioned to produce particular firing patterns if their activity is presented to primates as sensory feedback91–94. In these experiments, the firing of single cells became so well correlated to the desired motor output that primates could use this single-neuron activity to control the movements of a gauge needle93 or drive a functional electrical stimulator to produce an isometric contraction94. www.nature.com/reviews/neuro
© 2009 Macmillan Publishers Limited. All rights reserved
PersPectives During the past 25 years, the introduction of various new electrophysiological33,36–38,41,43,65–70,95–99 and imaging methods100–111 has allowed neurophysiologists to measure the concurrent activity of progressively larger samples of single neurons in behaving animals. Interestingly, the emergence of multi-electrode recordings as a new electrophysiological paradigm
occurred in parallel with the development of BMIs. As researchers started to implant more than one micro-electrode in the brain, it was proposed that single-neuron recordings from the motor cortex might one day provide the source of signals to drive artificial devices designed to restore mobility in paralysed patients112. However, almost two decades went by before the first experiments
a
Real-time analysis of brain activity
Signal processing
Real-time telemetry interface
Three-dimensional artificial limb Telemetry receiver Tactile and proprioceptive feedback
Visual feedback
Real-time multi-channel mechanical actuator
b
c
Upper-limb position
Gripping force
R2 of prediction
0.8
R2 of prediction
0.8
0
1
20
40
60
Number of neurons PMd
M1
S1
SMA
0
1
20
40
60
Number of neurons PP
Figure 1 | Principles of a brain–machine interface. a | A schematicNature of a brain–machine interface Reviews | Neuroscience (BMi) for reaching and grasping. Motor commands are extracted from cortical sensorimotor areas using multi-electrode implants that record neuronal discharges in large ensembles of cortical cells. signal-processing algorithms convert neuronal spikes into the commands to a robotic manipulator. Wireless telemetry can be used to link the BMi to the manipulator. the subject receives visual and somatosensory feedback from the actuator, possibly through the microstimulation of cortical sensory areas. b | Neuronal dropping curves for the prediction of arm movements in rhesus macaques1 calculated for the ensembles recorded in different cortical areas: the dorsal premotor cortex (PMd), the primary motor cortex (M1), the primary somatosensory cortex (s1), the supplementary motor area (sMA) and the posterior parietal cortex (PP). Neuronal dropping curves describe the accuracy (R2) of a BMi’s performance as a function of the size of the neuronal ensemble used to generate predictions. the best predictions were generated by the M1. Prediction accuracy improved with the increase of neuronal ensemble size. c | Predictions of hand gripping force calculated from the activity of the same cortical areas as in part a. image in part a is modified, with permission, from REF. 8 (2001) Macmillan Publishers Ltd. All rights reserved. images in parts b and c are reproduced from REF. 1. NATuRE REVIEWS | NeuroscieNce
were conducted to test the hypothesis54 that highly distributed populations of broadly tuned neurons can sustain the continuous production of motor behaviours in real-time1,11–13,113. FIGURE 1a shows a basic BMI paradigm8 in which the kinematic and dynamic parameters of upper- or lower-limb movements are predicted (or extracted) in real time from neuronal ensemble activity recorded by micro-electrode brain implants. In this context, the term prediction refers to the use of combined electrical neural ensemble activity to estimate time-varying kinematic and dynamic motor parameters a few hundred milliseconds (typically 100–1,000 ms) in the future. Multiple computational models are used to simultaneously extract various motor parameters (such as arm position and velocity, or hand gripping force) in real time from the extracellular activity of frontal and parietal cortical neurons. Computational models are first trained to predict motor parameters from the modulations of neuronal ensemble activity while animals perform motor tasks (typically reaching or grasping movements) with their own limbs. As the result of this training, the models generate a ‘transform function’ that matches neuronal activity patterns to particular movements. Next, the mode of operation is switched to ‘brain control’, in which the time-varying outputs of the computational models control the movements of an artificial device (such as a computer cursor or robot limbs) to reproduce the subject’s voluntary motor intentions6. A somewhat different approach for model training implemented in invasive BMIs in monkeys12 and non-invasive BMIs in humans12,114 is based on a supervised adaptive algorithm that does not require subjects to perform limb movements, but rather adapts the model parameters so that the model output approximates ideal trajectories. principles of neural ensemble physiology The advent of BMI research has advanced the field of multi-electrode recordings. Here we propose a series of principles of neural ensemble physiology (TABLE 1) that have been derived from, or validated by, BMI studies1,6,7,11–13,42,115–119. ultimately, these principles may be used in the development of new neuroprosthetic devices (BOX 1).
The distributed-coding principle. Multielectrode studies in New World13,117,118 and old World monkeys1,42, rats and mice86,120–122 consistently support the idea VoluME 10 | julY 2009 | 531
© 2009 Macmillan Publishers Limited. All rights reserved
306
307
PersPectives that information about single motor parameters is processed within multiple cortical areas. BMI studies1,42 have also revealed that real-time predictions of motor parameters can be obtained from multiple frontal and parietal cortical areas. This widespread representation of motor parameters defines the distributed-coding principle73,84–86. The analysis of neuron-dropping curves (NDCs) illustrates this principle well. NDCs depict a BMI’s prediction accuracy as a function of the number of neurons recorded simultaneously during a given experimental session. NDCs are computed by first measuring the entire neuronal population’s performance and then repeating the calculation after randomly chosen individual neurons are removed (dropped) from the original sample. In essence, NDCs measure the size of neuronal ensembles needed for a given BMI algorithm to achieve a certain level of performance. FIGURE 1b,c shows a series of NDCs that describe the contribution made by populations of neurons, located in different cortical areas, to the simultaneous prediction of multiple time-varying motor parameters during operation of a BMI by a rhesus monkey. This figure shows how the predictions of two such parameters — hand position (FIG. 1b) and gripping force123 (FIG. 1c) — vary as a function of the size of the recorded neuronal population1. A widely distributed representation of each motor parameter does not necessarily mean that equally sized neuronal samples obtained from each of these cortical areas should yield similar levels of predictions1 (FIG. 1b,c). For instance, in the example shown in FIG. 1, the prediction of hand position was, on average, better when randomly sampled populations of M1 neurons were used than
when similar samples of posterior parietal cortex (PP) neurons were used. Moreover, the difference in prediction performance was much smaller between these two cortical areas when gripping force was used as the predicted parameter. However, NDC extrapolation to larger samples13 indicates that, if a sufficiently large sample of PP neurons could be obtained, neural ensembles from the PP could eventually accurately predict both hand position and gripping force. Although the representation of motor parameters is distributed in the cortex, cortical areas nonetheless show a clear degree of specialization (but not in an absolute or strict sense). Additionally, modulations in neuronal activity in different cortical areas that seem to be similar (for example, increases in activity during rightward movements) may underlie different functions in the cortical motor programme transmitted to the spinal cord. The observation of distributed representations of motor parameters obtained in BMI studies corresponds well with the proposition from previous neurophysiological research that brain areas represent information in a holographic manner, and that searching for explicit coding (of force, limb displacement or behavioural context) may be futile124. The single-neuron insufficiency principle. BMI studies have also revealed that, no matter how well tuned a cell is to the behavioural task in question, the firing rate of individual neurons usually carries only a limited amount of information about a given motor parameter 1,13,42. Moreover, the contribution of individual neurons to the encoding of a given motor parameter
table 1 | principles of neural ensemble physiology Principle
explanation
Distributed coding
the representation of any behavioural parameter is distributed across many brain areas
single-neuron insufficiency
single neurons are limited in encoding a given parameter
Multitasking
A single neuron is informative of several behavioural parameters
Mass effect principle
A certain number of neurons in a population is needed for their information capacity to stabilize at a sufficiently high value
Degeneracy principle
the same behaviour can be produced by different neuronal assemblies
Plasticity
Neural ensemble function is crucially dependent on the capacity to plastically adapt to new behavioural tasks
conservation of firing
the overall firing rates of an ensemble stay constant during the learning of a task
context principle
the sensory responses of neural ensembles change according to the context of the stimulus
532 | julY 2009 | VoluME 10
tends to vary significantly from minute to minute125. Reliably predicting a motor variable, and achieving accurate and consistent operation of a BMI for long periods of time, therefore requires simultaneous recording from many neurons, and combining their collective ensemble firing 118. Incidentally, the same single-neuron limitations have been observed in the rat somatosensory 126–128 and gustatory systems86,129,130, and in the corticostriatal system of wild-type and transgenic mice121. We have called this principle the single-neuron insufficiency principle. The insufficiency of single-neuron firing to precisely reproduce a given behavioural output has long been appreciated in studies in which averaging of neuronal activity over many trials was required to quantify a given neuron’s behavioural function131,132. This analytical strategy is typically used when animals have attained a highly stereotyped behavioural performance, after being overtrained in a given task. Despite this caveat, single neurons have often been attributed very specific functions, and their inherent noisiness — clearly verified when single trials are analysed independently — has been disregarded132. In such studies, peri-event time histograms and directional tuning curves have emphasized a consistent relationship between the modulations of the firing rate of a single cell and behavioural parameters. As the attention of neurophysiological investigations started to shift towards ensemble recordings, neuronal variability, as opposed to consistency, came into focus, and neurophysiologists started to realize that modulations in neuronal firing are usually highly transient and plastic86,133–137. This led researchers to question the classic assertion that behavioural parameters are encoded only by the modulation of the firing rate of individual cells, and to the realization that the precise timing and correlations of neural ensemble firing should be taken more seriously65–67,138. usually, in BMIs based on recordings from large neuronal populations, single-neuron noisiness is removed by ensemble averaging. In other words, as the population recorded becomes larger, variability in single-neuron firing declines in importance. A recent study 139 documented significant single-neuron tuning stability over recording sessions that lasted several hours while monkeys performed a reaching task. Although this result initially seemed to contradict the earlier claim that there is single-neuron discharge variability 125, these two points of view proved to be consistent. The study demonstrating tuning stability focused on www.nature.com/reviews/neuro
© 2009 Macmillan Publishers Limited. All rights reserved
PersPectives Box 1 | From ensemble principles to neuroprosthetic development Ultimately, we expect that the identification of principles of neural ensemble physiology will guide the development of a generation of cortical neuroprosthetic devices that can restore full-body mobility in patients suffering from devastating levels of paralysis, due either to traumatic or degenerative lesions of the nervous system. We believe that such devices should incorporate several key design features. First, brain-derived signals should be obtained from multi-electrode arrays implanted in the upper- and lower-limb representations of the cortex, preferably in multiple cortical areas. Custom-designed microchips (also known as neurochips), chronically implanted in the skull, would be used for neural signal-processing tasks. To significantly reduce the risk of infection and damage to the cortex, multi-channel wireless technology would transmit neural signals to a small, wearable processing unit. Such a unit would run multiple real-time computational models designed to optimize the real-time prediction of motor parameters. Time-varying, kinematic and dynamic digital motor signals would be used to continuously control actuators distributed across the joints of a wearable, whole-body, robotic exoskeleton. High-order brain-derived motor commands would then interact with the controllers of local actuators and sensors distributed across the exoskeleton. Such interplay between brain-derived and robotic control signals, known as shared brain–machine control192, would assure both voluntary control and stability of bipedal walking of a patient supported by the exoskeleton. Touch, position, stretch and force sensors, distributed throughout the exoskeleton, would generate a continuous stream of artificial touch and proprioceptive feedback signals to inform the patient’s brain of the neuroprosthetic performance. Such signals would be delivered by multichannel cortical microstimulation directly into the patient’s somatosensory areas. Our prediction is that, after a few weeks, such a continuous stream of somatosensory feedback signals, combined with vision, would allow patients to incorporate, through a process of experience-dependent cortical plasticity, the whole exoskeleton as an extension of their body. These developments are likely to converge into the first reliable, safe and clinically useful cortical neuroprosthetic. To accelerate this process and make this milestone a clinical reality, a worldwide team of neurophysiologists, computer scientists, engineers, roboticists, neurologists and neurosurgeons has been assembled to launch the Walk Again Project, a non-profit, global initiative aimed at building the first cortical neuroprosthetic capable of restoring full-body mobility in severely paralysed patients.
mean firing characteristics, obtained by averaging hundreds of behavioural trials, to extract the preferred movement direction of single neurons. However, this study clearly showed that the firing rate of a single M1 neuron varied significantly from trial to trial (15–35 spikes per second). only by averaging many trials were those authors able to obtain smooth directional tuning curves. The earlier study 125 examined a large population of individual neurons and focused on shorter behavioural epochs, during which they observed considerable variability. The difference between these studies therefore resides mainly in the temporal scale and the analytical procedure used to estimate shortterm versus long-term changes in neuronal tuning properties. In any neuronal population sample there are cells that are better tuned to a given motor parameter of interest. Such neurons are usually called task-related cells140,141. However, even these cells show significant variability in their discharges and need to be combined to produce accurate predictions of motor parameters142. Although single cells are generally insufficient for obtaining accurate BMI predictions, the performance of a single-cell BMI has been shown to improve with training 94. Indeed, in
our own studies using large neural ensembles to drive BMIs, we observed that both the firing patterns of individual cells and the correlation between cells underwent plastic changes that improved BMI accuracy 1,42,119. The neuronal multitasking principle. BMI experiments also indicate that individual neurons, located in each of the cortical areas sampled, can participate in the encoding of more than one parameter at a given moment in time1. In other words, although individual cortical neurons might be better tuned to a given motor parameter, they can still contribute simultaneously to multiple, transient functional neural assemblies and therefore encode several motor parameters at once78. Here we name this the multitasking principle. The multitasking principle, described here for BMI studies, is similar to the multimodal interactions observed previously in sensory and associational cortical areas143–152. However, as most of the BMI literature deals with the motor system, we prefer to use the term ‘multitasking’. In our notation, a multitasking BMI controls several motor parameters simultaneously, for example several degrees of freedom of a multi-joint actuator.
NATuRE REVIEWS | NeuroscieNce
BMI experiments in which monkeys used cortical activity to control the reaching and grasping movements of a robotic manipulator revealed that the firing of single cortical neurons was typically correlated to several motor variables, such as the manipulator position coordinates and its gripping force1. Recent experiments that aimed to use the combined activity of primate cortical neurons to reproduce patterns of bipedal locomotion153 revealed that the firing of single neurons could contribute to the prediction of several motor variables related to leg movements154–156, including the timing of movement onset, as previously observed for hand movements116. The neuronal mass principle. Further analysis of the NDCs shown in FIG. 1b,c shows that parametric reductions in the size of the neuronal population initially produce a minor reduction in overall prediction performance for each motor parameter in each of the sampled cortical areas1,13,42. However, below a certain critical population size the accuracy of the predictions starts to fall more rapidly and, at a certain level (fewer than ~10–20 neurons), becomes poor 1,13. This suggests that BMIs based on recording the activity of just a few neurons are likely to perform poorly. According to the singleneuron insufficiency principle, predictive performance should increase continuously as a function of the growth in neural ensemble size. However, NDCs revealed that when the number of neurons used went above a certain population size (tens of neurons), the amount of predictive information obtained tended to remain virtually constant, regardless of the identity of the individual neurons sampled. This result is attributable to a significant decrease in the variance of NDCs for sufficiently large neuronal samples116. In other words, once a certain critical neuronal mass had been achieved, different, and sufficiently large, random samples of single neurons from a given cortical area (from different layers or different subregions) tended to yield similar levels of predictive information about a given motor parameter 1,42. These results led us to propose the neuronal mass effect principle, which states that to achieve a sufficiently accurate and stable prediction of a given motor parameter, a neural ensemble has to recruit a crucial number of neurons at each moment in time. The neuronal mass needed to achieve stability depends on several factors, including the presence of highly tuned neurons in the population116,142. If these are missing, predictions gradually improve with neuronal sample size, VoluME 10 | julY 2009 | 533
© 2009 Macmillan Publishers Limited. All rights reserved
308
309
PersPectives and the noise in the combined population activity is proportional to the square root of the number of neurons. The critical neuronal mass is also highly dependent on neuronal correlations, which limit the information that the population can contain157,158. Correlation makes neuronal encoding redundant. As a consequence, beyond a certain size the information represented by a neuronal population increases only marginally with the addition of new cells. The minimal size of a neuronal sample needed to effectively control a BMI has become a controversial issue (for contrasting opinions, compare REFS 7,11,12 with REFS 1,6,9,13,42,116). on the basis of demonstrations that involved stereotypical and relatively simple upper-limb movements, several groups have argued that BMIs intended to restore upper-limb mobility could operate using small neuronal samples (<30 neurons)7,11,12,159. Despite this emphasis on the role of small neuronal samples, and results showing improvement in accuracy of small-sample BMIs with training 94, practical BMIs might not perform sufficiently well using signals from only a few neurons, for various reasons. For instance, it is unclear from the studies that used this approach whether small-sample BMIs can sustain the same level of performance over long periods of time11,12,160. Current recording techniques may not allow the sampling of high numbers of highly tuned neurons, or provide the kind of stability needed for such smallsample BMIs to remain effective for many months, let alone for many years. Additionally, small-sample BMIs may not be able to generalize their function to cope with newer or more complex behavioural tasks153. We therefore feel that it is likely that such an increase in behavioural demand will be met by only large neuronal populations. Evidence obtained in our laboratory indicates that this is precisely the case for BMIs aimed at reproducing both upper- and lower-limb movements1,153. The neural degeneracy principle. BMI studies also revealed that a single motor output is often associated with distinct spatiotemporal patterns of neural ensemble firing on the millisecond scale118,161–164. Following the nomenclature introduced by Reeke and Edelman165, this principle, which states that identical behavioural outputs can be produced by distinct functional and transient neural ensembles, has been named the degeneracy principle. Neural degeneracy is similar to neural redundancy in that different combinations
of single neurons belonging to a neural circuit can produce different spatiotemporal firing patterns that end up encoding the same motor outputs166. Degenerate coding has been demonstrated in several neural circuits, including the pyloric network of the lobster, the song control system of the zebra finch and the order-encoding system of the locust164, where it serves to represent low-dimensional information by a high-dimensional neural network in a fault-tolerant way. BMIs based on neuronal ensemble recordings solve a similar problem: they map the activity of several hundred neurons onto the lower number of degrees of freedom of an artificial actuator. In these experiments, we have observed that similar movements, produced either by the animal’s arm or by an artificial actuator, can result from distinct spatiotemporal patterns of neuronal population activity 125. Therefore, if a sufficiently large population of neurons is recorded simultaneously, movements induced by a BMI can be reliably produced in each behavioural trial. Similarly, we observed that stereotypical steps in bipedally walking monkeys were associated with different patterns of motor cortex activations153. It follows from these considerations that the basic proportion between the recorded ensemble size and the number of controlled degrees of freedom should be preserved for BMI applications that require the production of complex motor behaviours in artificial actuators. The plasticity principle. Experiencedependent plasticity in cortical neural ensembles167,168 is essential for primates to learn to operate a BMI. As mentioned above, the strength of a single-neuron correlation to a given motor parameter is typically imprecise, varying as a function of time, internal state and learning, as well as the animal’s expectation of the task outcome and reward118,125,169. Several studies have now documented the occurrence of cortical plasticity as animals learn to operate a BMI1,12,42. This phenomenon is characterized by changes in the tuning properties of individual neurons12,42 and physiological adaptations at the level of neural ensembles, which include changes in firing covariance and spike timing 1. Such changes in neuronal properties are undoubtedly related to basic plasticity mechanisms, such as changes in the strength of synaptic connections and gene expression. However, in BMI experiments such basic mechanisms are difficult to isolate from the population effects. For example, increases in the firing activity of a given neuron can result from multiple
534 | julY 2009 | VoluME 10
factors, such as changes in synaptic strength, increases in excitatory inputs or release of inhibition. Combinations of such factors manifest themselves as changes in neuronal tuning (the correlation of cell firing with a given motor parameter). In BMI studies, similar measurements of neuronal tuning are provided by the weights that prediction models assign to different neurons142 and time-dependent correlations of neuronal rates with kinematic parameters42. As a rule, neuronal tuning tends to be modified and refined as a result of operant conditioning 93,94,161,170–179. In BMI studies, cortical plasticity manifests itself in a series of physiological adaptations. For instance, during the transition from manual to brain control of a BMI1,42 (when animals ceased to use their own limbs and started to control an actuator using their cortical activity directly), a significant portion of the recorded neurons, which were distributed across multiple cortical areas, progressively acquired tuning properties related to the kinematic properties of the robotic device used (FIG. 2). As a result, a fraction of these cortical neurons showed tuning to the kinematic properties of both the animal’s biological arms and the robotic arm (FIG. 2a). Conversely, a subset of the recorded cortical neurons ceased to fire, or to show velocity or direction tuning, when animals stopped producing arm movements and controlled a robotic device without any overt motor behaviour 1,42 (FIG. 2b). Perhaps more surprisingly, a fraction of the recorded cortical neurons showed clear velocity and direction tuning that was related to the movements of the robotic prosthesis but not to the displacement of the animal’s own arms1,42 (FIG. 2c). Such tuning developed and became sharper during the period in which monkeys learned to operate the BMI without execution of overt body movements (brain control mode). The emergence of such tuning may explain why monkeys were able to control both robotic arms and legs using BMIs without generating corresponding body movements. Besides changes in single-neuron tuning properties, a significant increase in firing covariance between pairs of neurons, located within and between multiple cortical areas, has also been observed when animals started operating a BMI without moving their own limbs1 (FIG. 2d). As animals shifted back and forth between using their own limbs or the artificial actuator controlled by the BMI to solve a particular motor task, functional coupling between pairs of cortical neurons adapted dynamically. Interestingly, this www.nature.com/reviews/neuro
© 2009 Macmillan Publishers Limited. All rights reserved
PersPectives increase in neuronal pair covariance was observed not only within a given cortical area, but also between neurons located in distinct cortical fields1. The observation of such a broad repertoire of functional cortical adaptations
during the operation of BMIs supports many far-reaching conclusions. First, they suggest that old World monkeys may be capable of ‘motor imagery’180–183: to imagine, in great detail, a series of complex motor sequences without necessarily producing
a Modulated with hand and robot –1 1.5 Normalized rate
body movements to execute such motor plans. Second, they imply that, at its limit, cortical plasticity may allow artificial tools to be incorporated as part of the multiple functional representations of the body that exist in the mammalian brain. If this proves
b Modulated with hand only
Lag = –400 ms –200 ms
Instant velocity measurement
–0.9
0 ms +100 ms +300 ms
Instant velocity measurement
Lag = –400 ms
1.1 Normalized rate
–200 ms
0 ms +100 ms +300 ms
+500 ms
Hand
+500 ms
Hand
Pole control Hand
0 –14 –14
Brain control with hand movements
Robot Robot 0
Vy (cm per s)
Vy (cm per s)
Hand
14
Brain control without hand movements
14
Vx (cm per s)
d
Lag = –400 ms –200 ms
0 –14 –14
Instant velocity measurement
0 ms +100 ms +300 ms
Brain control with hand movements
Robot
14
Robot 0
Brain control without hand movements
14
Vx (cm per s)
c Enhanced modulation with robot 1.2 –0.6 Normalized rate
Pole control
Neuronal correlation analysis M1ips SMA S1
Brain control with hand movements
Hand control
M1
+500 ms PMd
–14 –14
Robot 0
14
Vx (cm per s)
Brain control without hand movements
Brain control with hand movements
Correlation coefficient 0.3
–0.1 J Neuron #
Vy (cm per s)
0
1
Robot
14
s 1ip MA SM S1
Hand
M
Pole control
d
PM
Hand
Brain control without hand movements
I I Neuron # J
Figure 2 | Neuronal activity during a reaching task. the task illustrated was performed by a rhesus macaque that controlled a robotic actuator using a hand control or through a brain–machine interface (BMi; brain control). During BMi operation, the monkey either continued to move the pole with the hand (brain control with hand movements) or stopped moving its hand (brain control without hand movements). a | Activity of a primary motor cortex (M1) neuron during both pole and brain control. colour-coded diagrams represent neuronal tuning to movement velocity (that is, the average neuronal rate as a function of hand or robot velocity), calculated at different lags (–400 ms to +500 ms) with respect to the time of velocity measurement. the diagrams labelled ‘Hand’ represent neuronal tuning to hand movements, and the diagrams labelled ‘Robot’ represent tuning to robot movements. During brain control without hand movements, this neuron became less tuned to hand movements (row of colour diagrams
labelled ‘Brain control with hand movements: hand’). tuning to robot moveNature Reviews | Neuroscience ments was maximal during brain control without hand movements (row labelled ‘Brain control without hand movements: robot’). b | An M1 neuron modulated only when the monkey moved its hand. c | An M1 neuron that was not modulated during hand movements, but became tuned to the robot movements during brain control without hand movements. d | Analysis of pairwise correlations in firing between the neurons in the recorded ensemble, using data from REF. 1. correlations increased during brain control, especially brain control without hand movements. the highest correlations were between the neurons recorded in the same cortical area. M1ips, primary motor cortex, hemisphere ipsilateral to the working hand; PMd, dorsal premotor cortex; s1, primary somatosensory cortex; sMA, supplementary motor area. images in parts a–c are modified, with permission, from REF. 42 (2005) society for Neuroscience.
NATuRE REVIEWS | NeuroscieNce
VoluME 10 | julY 2009 | 535 © 2009 Macmillan Publishers Limited. All rights reserved
310
311
PersPectives to be true, we would predict that continuous use of a BMI should induce subjects to perceive artificial prosthetic devices, such as prosthetic arms and legs, controlled by a BMI as part of their own bodies. Such a prediction opens the intriguing possibility that the representation of self does not necessarily end at the limit of the body surface, but can be extended to incorporate artificial tools under the control of the subject’s brain. BMI research further stretches this puzzling idea by demonstrating that, once brain activity is recorded and decoded efficiently in real time, its capacity to control artificial devices can undergo considerable modification in terms of temporal, spatial, kinematic and kinetic characteristics, termed scaling 1,12. In other words, not only can a BMI enact voluntary motor outputs faster than the subject’s biological apparatus (temporal scaling), but it can also accomplish motor tasks at a distance from the subject’s own body (spatial scaling), by controlling an actuator that is either considerably smaller (for example, a nano-tool) or considerably larger (for example, a crane) than the subject’s own biological appendices. Recently, another powerful way to induce cortical plasticity has been introduced a Basic amplitude discrimination task
to BMI research: multichannel, cortical microstimulation115,184. FIGURE 3 shows some of the findings obtained when chronic multichannel microstimulation of the primary somatosensory cortex was used to instruct owl monkeys on how to locate food rewards. During several months of microstimulation sessions, these monkeys progressively learned to detect the presence or absence of microstimulation, and to discriminate different temporal patterns of microstimulation pulses that indicated food location115. Moreover, the animals also learned new behavioural contingencies after changes were made in the direction of arm reach instructed by microstimulation. Interestingly, monkeys required less time to master a new set of rules as training progressed and new task contingencies were introduced, which allowed them more practice in handling microstimulation cues (FIG. 3b,c). So after being exposed to an original basic rule, monkeys learned a reversed task much more rapidly and, subsequently, more elaborate contingencies as well115. Although the basic mechanism involved in such ‘rule generalization’ was not uncovered, these results confirm the hypothesis that functional plasticity of cortical tissue can b Temporal discrimination task
be induced by intracortical microstimulation184. This raises the question of whether chronic cortical microstimulation can trigger a process of functional adaptation that leads to the emergence of realistic perceptual experiences. Although there is no definitive answer to this question, it is interesting to note that people who were exposed to chronic patterned cutaneous stimulation, as an artificial replacement strategy for vision, learned to use such artificial sensory input to guide their movements and reported the development of qualitatively new perceptions185,186. Confirmation of new perceptual experiences after prolonged training with microstimulation would certainly be of considerable relevance for the design of future neuroprosthetic devices that aim to restore upper- and lower-limb mobility in severely paralysed patients. With that long-term vision in mind, we have recently started to develop a new paradigm, named brain–machine–brain interface (FIG. 4), that will enable us to test whether monkeys can use neural ensemble activity to control the movements of artificial devices guided by instructions delivered directly to their somatosensory cortices by multichannel microstimulation. c Spatiotemporal discrimination task 150 ms Electrode pair 1 (EP 1)
100 ms 150 ms
100 ms 150 ms
100 Hz
Versus
100 ms
100 Hz
EP 2 EP 3
Versus
EP 4
300 ms No stimulation
Versus
100 Hz
EP 1
200 ms
EP 2 EP 3
0.8 0.7 0.6
Chance level
0.5 0.4
0
5
10
15
20
25
Experiment
30
35
40
Fraction correct
Fraction correct
1 0.9
0.95 0.90 0.85 0.80 0.75 0.70 0.65 0.60 0.55 0.50
Fraction correct
EP 4
Chance level 0
2
4
6
8
Experiment
Figure 3 | Discrimination of spatiotemporal microstimulation patterns by owl monkeys. Microstimulation trains were delivered to the primary somatosensory cortex through chronically implanted multi-electrode arrays. the monkeys responded to microstimulation or its absence by selecting the target of reaching movements. top panels illustrate microstimulation patterns; bottom panels show discrimination accuracy as the function of training day. a | A basic task in which the monkeys detected the presence or absence of microstimulation. the monkeys learned the task in 1 month.
10
12
1 0.95 0.90 0.85 0.80 0.75 0.70 0.65 0.60 0.55 0.50
Chance level 1
2
3
4
5
6
7
Experiment
8
9
10
b | Discrimination of temporal patterns of microstimulation. the monkeys learned the task in 1 week. c | A spatiotemporal discrimination task during Nature Reviews | Neuroscience which waves of microstimulation were delivered through the electrode arrays. the monkeys learned the task in 3 days, and could then discriminate spatiotemporal patterns of cortical microstimulation. Moreover, after prolonged training with microstimulation they learned to interpret new microstimulation patterns faster. Figure is reproduced, with permission, from REF. 115 (2007) society for Neuroscience.
536 | julY 2009 | VoluME 10
www.nature.com/reviews/neuro © 2009 Macmillan Publishers Limited. All rights reserved
PersPectives a Implantation sites
c Receptive fields
b Electrode array
D4
Pa ir PMd
M1
Pa ir
Pa ir
10 mm
4
D2
D5 D1
3
2
D5
S1 Stim pair 1
Pads
D3
D4 D2
D3
Pad
5 mm
d Stimulation pattern
e
Output communication
Signal processing
Decode neural data
Visual feedback
150 µS
50 µA
Real-time task control Target selection task
100 µS 50 µA Input communication
150 µS
Figure 4 | The concept of a brain–machine–brain (BMBi) interface with artificial sensory feedback. in one possible implementation, depending on the presence or absence of microstimulation the monkeys perform brain– machine interface (BMi)-controlled reaching movements in different directions (right or left). initially the monkeys acquire visual targets using a screen cursor moved by a hand-held joystick, and the directional instruction is delivered by mechanical vibration of the joystick handle. Manual control is then replaced by BMi control of cursor movements, and vibration is replaced by cortical microstimulation. a | examples of possible sites of cortical implantation.
The conservation of firing principle. Despite documenting clear and widespread changes in the single-neuron firing rate related to plastic modifications in neuronal velocity and duration tuning, and increases in firing covariance between pairs of cortical neurons, we have also observed that the global firing rate (total number of spikes) of the cortical neural ensembles recorded in our experiments usually remained unchanged as animals learned to operate a BMI1. This principle of neural ensemble firing conservation has also been observed in various other studies — including experiments conducted in New World monkeys, rats and mice — involving distinct cortical areas and various motor and sensory tasks95,120–122,187–190. These studies indicate that maintaining the total
Multi-electrode arrays are placed in the dorsal premotor cortex (PMd), the primary motor cortex (M1) and the primary somatosensory cortex (s1). PMd Natureand Reviews | Neuroscience and M1 arrays are used to extract motor commands, the s1 is the site of microstimulation. b | examples of possible locations of s1 electrodes with respect to a somatotopic map determined using receptive field measurements. the multi-electrode array covers the representation of digits D2–D5 and of the hand pads. c | Receptive fields of the electrodes through which microstimulation is delivered. d | Parameters of microstimulation train (top) and microstimulation pulses. e | schematic of the experiment, as described above.
number of spikes for a range of behaviours could be a pervasive, homeostatis-like mechanism of cortical ensembles. The context principle. Multi-electrode recordings in freely behaving animals have also opened new ways to examine a fundamental question in classic neurophysiology: how neurons respond to sensory stimuli that are applied passively or acquired actively by subjects. A study in behaving rats trained to perform a tactile discrimination task using only their facial whiskers addressed this issue directly 169. This study revealed that neuronal modulations evoked by passively versus actively acquired tactile stimulation were strikingly different in their magnitude, adaptation rate and percentage of excitatory
NATuRE REVIEWS | NeuroscieNce
versus inhibitory sensory evoked responses in the primary somatosensory cortex (FIG. 5). A similar result has since been described in the rat primary gustatory cortex 191 and in the auditory cortex of marmosets39. Such marked neurophysiological differences indicate that the context in which animals sample their surrounding environment can radically alter the way cortical neural ensembles respond to incoming sensory information. Therefore, we named this principle the context principle. Conclusions The principles of neural ensemble physiology described above were either derived from, or confirmed by, a decade of BMI experiments. This demonstrates that, in VoluME 10 | julY 2009 | 537
© 2009 Macmillan Publishers Limited. All rights reserved
312
313
PersPectives
Whisker row
32 1 0
Pneumatic solenoid
32 1 0
B C D E
DC servo Aperture
0
25 50 Stimulus onset time (ms)
Awake rats 32 1 0
Whisker column
75
32 1 0
Passive stimulation
4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15.
Correspondence to M.A.L.N. e‑mail: nicoleli@neuro.duke.edu doi:10.1038/nrn2653 Carmena, J. M. et al. Learning to control a brain– machine interface for reaching and grasping by primates. PLoS Biol. 1, e42 (2003). Chapin, J. K., Moxon, K. A., Markowitz, R. S. & Nicolelis, M. A. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neurosci. 2, 664–670 (1999). Donoghue, J. P. Connecting cortex to machines: recent advances in brain interfaces. Nature Neurosci. 5 (Suppl.), 1085–1088 (2002).
Moving aperture
2
16. 17.
18. 19.
Active discrimination
0
e ak Aw ed tiz he
Miguel A. L. Nicolelis is also at the Edmond and Lily Safra International Institute of Neuroscience of Natal, Rua Professor Francisco Luciano de Oliveira, 2460, Candelária, Natal, Rio Grande do Norte 59066‑060, Brazil and is a Fellow at the Center for Neuroprosthetics, Ecole Polytechnique Federale de Lausanne, 1015 Lausanne, Switzerland.
Ramp and hold
100
t es An
Miguel A. L. Nicolelis and Mikhail A. Lebedev are at the Duke University Center for Neuroengineering and the Department of Neurobiology, Duke University, Durham, North Carolina 27710, USA.
4
150
e ak Aw ed tiz he
addition to offering hope for a potential future therapy for the rehabilitation of severely paralysed patients, BMIs can be extremely useful platforms to test various ideas for how populations of neurons encode information in behaving animals. Together with other methods, research on BMIs has contributed to the growing consensus that distributed neural ensembles, rather than the single neuron, constitute the true functional unit of the CNS responsible for the production of a wide behavioural repertoire.
3.
6 200
Anaesthetized or awake restrained rat
60
Figure 5 | Neuronal responses in rat somatosensory cortex to passively applied stimuli versus active discrimination of the same stimuli. a | schematics illustrating the whiskers stimulated (arranged in rows and columns) and the stimulus timing. Multi-whisker ramp-and-hold stimuli were delivered to anaesthetized (top) or awake restrained (bottom) rats. Large circles represent stimulation of a particular whisker. Arrows show stimulation onsets. b | schematic of stimulus delivery. the aperture was accelerated across the facial whiskers by the pneumatic solenoid and also simultaneously deflected laterally in varying amounts by the direct-current
2.
250
0
Stimulus onset time (ms)
8
Response duration Response magnitude
50
B C D E 0
1.
300
t es An
Whisker row
Whisker column 32 1 0 32 1 0
c
Response magnitude (spikes per stimulus)
b Moving aperture
Anaesthetized rats
Response duration (ms)
a Ramp and hold
servo to accurately replicate the range of whisker deflection dynamics that occurred during active discrimination. c | Mean duration and magnitude of the responses evoked during active discrimination and during Nature Reviews | Neuroscience delivery of passive stimuli to anaesthetized or awake restrained rats. these results indicated that neuronal responses evoked in the primary somatosensory cortex by passively applied stimuli were strikingly different from those evoked by actively acquired tactile stimulation. Figure is reproduced, with permission, from REF. 169 (2004) American Association for the Advancement of science.
Fetz, E. E. Volitional control of neural activity: implications for brain–computer interfaces. J. Physiol. 579, 571–579 (2007). Kennedy, P. R. & Bakay, R. A. Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport 9, 1707–1711 (1998). Lebedev, M. A. & Nicolelis, M. A. Brain–machine interfaces: past, present and future. Trends Neurosci. 29, 536–546 (2006). Musallam, S., Corneil, B. D., Greger, B., Scherberger, H. & Andersen, R. A. Cognitive control signals for neural prosthetics. Science 305, 258–262 (2004). Nicolelis, M. A. Actions from thoughts. Nature 409, 403–407 (2001). Nicolelis, M. A. Brain–machine interfaces to restore motor function and probe neural circuits. Nature Rev. Neurosci. 4, 417–422 (2003). Schwartz, A. B., Cui, X. T., Weber, D. J. & Moran, D. W. Brain-controlled interfaces: movement restoration with neural prosthetics. Neuron 52, 205–220 (2006). Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R. & Donoghue, J. P. Instant neural control of a movement signal. Nature 416, 141–142 (2002). Taylor, D. M., Tillery, S. I. & Schwartz, A. B. Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832 (2002). Wessberg, J. et al. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408, 361–365 (2000). Chapin, J. K. Neural prosthetic devices for quadriplegia. Curr. Opin. Neurol. 13, 671–675 (2000). Donoghue, J. P., Nurmikko, A., Black, M. & Hochberg, L. R. Assistive technology and robotic control using motor cortex ensemble-based neural interface systems in humans with tetraplegia. J. Physiol. 579, 603–611 (2007). Friehs, G. M., Zerris, V. A., Ojakangas, C. L., Fellows, M. R. & Donoghue, J. P. Brain–machine and brain– computer interfaces. Stroke 35, 2702–2705 (2004). Mussa-Ivaldi, F. A. & Miller, L. E. Brain–machine interfaces: computational demands and clinical needs meet basic neuroscience. Trends Neurosci. 26, 329–334 (2003). Birbaumer, N. Breaking the silence: brain–computer interfaces (BCI) for communication and motor control. Psychophysiology 43, 517–532 (2006). Birbaumer, N. & Cohen, L. G. Brain–computer interfaces: communication and restoration of movement in paralysis. J. Physiol. 579, 621–636 (2007).
538 | julY 2009 | VoluME 10
20. Cohen, E. D. Prosthetic interfaces with the visual system: biological issues. J. Neural Eng. 4, R14–R31 (2007). 21. Dobkin, B. H. Brain–computer interface technology as a tool to augment plasticity and outcomes for neurological rehabilitation. J. Physiol. 579, 637–642 (2007). 22. Kubler, A. & Kotchoubey, B. Brain–computer interfaces in the continuum of consciousness. Curr. Opin. Neurol. 20, 643–649 (2007). 23. Kubler, A. & Neumann, N. Brain–computer interfaces — the key for the conscious brain locked into a paralyzed body. Prog. Brain Res. 150, 513–525 (2005). 24. Leuthardt, E. C., Schalk, G., Moran, D. & Ojemann, J. G. The emerging world of motor neuroprosthetics: a neurosurgical perspective. Neurosurgery 59, 1–14 (2006). 25. Lotte, F., Congedo, M., Lecuyer, A., Lamarche, F. & Arnaldi, B. A review of classification algorithms for EEG-based brain–computer interfaces. J. Neural Eng. 4, R1–R13 (2007). 26. Mason, S. G., Bashashati, A., Fatourechi, M., Navarro, K. F. & Birch, G. E. A comprehensive survey of brain interface technology designs. Ann. Biomed. Eng. 35, 137–169 (2007). 27. Pfurtscheller, G. & Neuper, C. Future prospects of ERD/ERS in the context of brain–computer interface (BCI) developments. Prog. Brain Res. 159, 433–437 (2006). 28. Wolpaw, J. R. Brain–computer interfaces as new brain output pathways. J. Physiol. 579, 613–619 (2007). 29. Birbaumer, N. et al. A spelling device for the paralysed. Nature 398, 297–298 (1999). 30. Karim, A. A. et al. Neural internet: web surfing with brain potentials for the completely paralyzed. Neurorehabil. Neural Repair 20, 508–515 (2006). 31. Kennedy, P. R., Kirby, M. T., Moore, M. M., King, B. & Mallory, A. Computer control using human intracortical local field potentials. IEEE Trans. Neural Syst. Rehabil. Eng. 12, 339–344 (2004). 32. Nijboer, F. et al. A P300-based brain–computer interface for people with amyotrophic lateral sclerosis. Clin. Neurophysiol. 119, 1909–1916 (2008). 33. Nicolelis, M. A., Baccala, L. A., Lin, R. C. & Chapin, J. K. Sensorimotor encoding by synchronous neural ensemble activity at multiple levels of the somatosensory system. Science 268, 1353–1358 (1995).
www.nature.com/reviews/neuro © 2009 Macmillan Publishers Limited. All rights reserved
PersPectives 34. Nicolelis, M. A., Lin, R. C., Woodward, D. J. & Chapin, J. K. Induction of immediate spatiotemporal changes in thalamic networks by peripheral block of ascending cutaneous information. Nature 361, 533–536 (1993). 35. O’Keefe, J. & Dostrovsky, J. The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res. 34, 171–175 (1971). 36. Wilson, M. A. & McNaughton, B. L. Dynamics of the hippocampal ensemble code for space. Science 261, 1055–1058 (1993). 37. Baker, S. N. et al. Multiple single unit recording in the cortex of monkeys using independently moveable microelectrodes. J. Neurosci. Methods 94, 5–17 (1999). 38. deCharms, R. C., Blake, D. T. & Merzenich, M. M. A multielectrode implant device for the cerebral cortex. J. Neurosci. Methods 93, 27–35 (1999). 39. Eliades, S. J. & Wang, X. Neural substrates of vocalization feedback monitoring in primate auditory cortex. Nature 453, 1102–1106 (2008). 40. Hatsopoulos, N., Joshi, J. & O’Leary, J. G. Decoding continuous and discrete motor behaviors using motor and premotor cortical ensembles. J. Neurophysiol. 92, 1165–1174 (2004). 41. Jackson, A. & Fetz, E. E. Compact movable microwire array for long-term chronic unit recording in cerebral cortex of primates. J. Neurophysiol. 98, 3109–3118 (2007). 42. Lebedev, M. A. et al. Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain–machine interface. J. Neurosci. 25, 4681–4693 (2005). 43. Nicolelis, M. A. et al. Chronic, multisite, multielectrode recordings in macaque monkeys. Proc. Natl Acad. Sci. USA 100, 11041–11046 (2003). 44. Nicolelis, M. A. et al. Simultaneous encoding of tactile information by three primate cortical areas. Nature Neurosci. 1, 621–630 (1998). 45. Santhanam, G., Ryu, S. I., Yu, B. M., Afshar, A. & Shenoy, K. V. A high-performance brain–computer interface. Nature 442, 195–198 (2006). 46. Hochberg, L. R. et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171 (2006). 47. Patil, P. G., Carmena, J. M., Nicolelis, M. A. & Turner, D. A. Ensemble recordings of human subcortical neurons as a source of motor control signals for a brainmachine interface. Neurosurgery 55, 27–35 (2004). 48. Truccolo, W., Friehs, G. M., Donoghue, J. P. & Hochberg, L. R. Primary motor cortex tuning to intended movement kinematics in humans with tetraplegia. J. Neurosci. 28, 1163–1178 (2008). 49. Bizzi, E., Accornero, N., Chapple, W. & Hogan, N. Arm trajectory formation in monkeys. Exp. Brain Res. 46, 139–143 (1982). 50. Bizzi, E., Mussa-Ivaldi, F. A. & Giszter, S. Computations underlying the execution of movement: a biological perspective. Science 253, 287–291 (1991). 51. Cohen, Y. E. & Andersen, R. A. A common reference frame for movement plans in the posterior parietal cortex. Nature Rev. Neurosci. 3, 553–562 (2002). 52. Evarts, E. V. & Fromm, C. Information processing in the sensorimotor cortex during voluntary movement. Prog. Brain Res. 54, 143–155 (1980). 53. Georgopoulos, A. P. Spatial coding of visually guided arm movements in primate motor cortex. Can. J. Physiol. Pharmacol. 66, 518–526 (1988). 54. Georgopoulos, A. P., Schwartz, A. B. & Kettner, R. E. Neuronal population coding of movement direction. Science 233, 1416–1419 (1986). 55. Kakei, S., Hoffman, D. S. & Strick, P. L. Muscle and movement representations in the primary motor cortex. Science 285, 2136–2139 (1999). 56. Lebedev, M. A. & Wise, S. P. Insights into seeing and grasping: distinguishing the neural correlates of perception and action. Behav. Cogn. Neurosci. Rev. 1, 108–129 (2002). 57. Paz, R., Wise, S. P. & Vaadia, E. Viewing and doing: similar cortical mechanisms for perceptual and motor learning. Trends Neurosci. 27, 496–503 (2004). 58. Polit, A. & Bizzi, E. Processes controlling arm movements in monkeys. Science 201, 1235–1237 (1978). 59. Todorov, E. Optimality principles in sensorimotor control. Nature Neurosci. 7, 907–915 (2004). 60. Wise, S. P., di Pellegrino, G. & Boussaoud, D. The premotor cortex and nonstandard sensorimotor mapping. Can. J. Physiol. Pharmacol. 74, 469–482 (1996). 61. Andersen, R. A., Musallam, S. & Pesaran, B. Selecting the signals for a brain–machine interface. Curr. Opin. Neurobiol. 14, 720–726 (2004).
62. Bashashati, A., Fatourechi, M., Ward, R. K. & Birch, G. E. A survey of signal processing algorithms in brain–computer interfaces based on electrical brain signals. J. Neural Eng. 4, R32–57 (2007). 63. Lilly, J. C. in Biological and Biochemical Bases of Behavior (eds Harlow, H. F. & Woolsey, C. N.) 83–100 (Univ. of Wisconsin Press, Madison, Wisconsin, 1958). 64. Lilly, J. C. Distribution of ‘motor’ functions in the cerebral cortex in the conscious, intact monkey. Science Abstr. 124, 937 (1956). 65. Gerstein, G. L. & Aertsen, A. M. Representation of cooperative firing activity among simultaneously recorded neurons. J. Neurophysiol. 54, 1513–1528 (1985). 66. Gerstein, G. L., Perkel, D. H. & Dayhoff, J. E. Cooperative firing activity in simultaneously recorded populations of neurons: detection and measurement. J. Neurosci. 5, 881–889 (1985). 67. Gerstein, G. L., Perkel, D. H. & Subramanian, K. N. Identification of functionally related neural assemblies. Brain Res. 140, 43–62 (1978). 68. Kruger, J. & Bach, M. Simultaneous recording with 30 microelectrodes in monkey visual cortex. Exp. Brain Res. 41, 191–194 (1981). 69. McNaughton, B. L., Barnes, C. A. & O’Keefe, J. The contributions of position, direction, and velocity to single unit activity in the hippocampus of freelymoving rats. Exp. Brain Res. 52, 41–49 (1983). 70. Shin, H. C. & Chapin, J. K. Mapping the effects of motor cortex stimulation on single neurons in the dorsal column nuclei in the rat: direct responses and afferent modulation. Brain Res. Bull. 22, 245–252 (1989). 71. Barlow, H. B. Single units and sensation: a neuron doctrine for perceptual psychology? Perception 1, 371–394 (1972). 72. Hubel, D. H. & Wiesel, T. N. Early exploration of the visual cortex. Neuron 20, 401–412 (1998). 73. Averbeck, B. B. & Lee, D. Coding and transmission of information by neural ensembles. Trends Neurosci. 27, 225–230 (2004). 74. Covey, E. Neural population coding and auditory temporal pattern analysis. Physiol. Behav. 69, 211–220 (2000). 75. Doetsch, G. S. Patterns in the brain. Neuronal population coding in the somatosensory system. Physiol. Behav. 69, 187–201 (2000). 76. Sakurai, Y. Population coding by cell assemblies — what it really is in the brain. Neurosci. Res. 26, 1–16 (1996). 77. Young, T. On the theory of light and colours. Philos. Trans. R. Soc. Lond. B Biol. Sci. 92, 12–48 (1802). 78. Hebb, D. O. The Organization of Behavior: A Neuropsychological Theory (Wiley, New York, 1949). 79. Barlow, H. B. in The Cognitive Neurosciences (ed. Gazzaniga, M.) 415–435 (MIT Press, Cambridge, 1995). 80. Barlow, H. B. Pattern recognition and the responses of sensory neurons. Ann. NY Acad. Sci. 156, 872–881 (1969). 81. Cajal, R. Histology of the Nervous System of Man and Vertebrates (Oxford Univ. Press, New York, 1899). 82. Hubel, D. H. Eye, Brain and Vision (W. H. Freeman and Company, New York, 1988). 83. Hubel, D. H. & Wiesel, T. N. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. 160, 106–154 (1962). 84. Breakspear, M. & Stam, C. J. Dynamics of a neural system with a multiscale architecture. Philos. Trans. R. Soc. Lond. B Biol. Sci. 360, 1051–1074 (2005). 85. Serences, J. T. & Yantis, S. Selective visual attention and perceptual coherence. Trends Cogn. Sci. 10, 38–45 (2006). 86. Simon, S. A., de Araujo, I. E., Gutierrez, R. & Nicolelis, M. A. The neural mechanisms of gustation: a distributed processing code. Nature Rev. Neurosci. 7, 890–901 (2006). 87. Bichot, N. P., Thompson, K. G., Chenchal Rao, S. & Schall, J. D. Reliability of macaque frontal eye field neurons signaling saccade targets during visual search. J. Neurosci. 21, 713–725 (2001). 88. Brecht, M., Schneider, M., Sakmann, B. & Margrie, T. W. Whisker movements evoked by stimulation of single pyramidal cells in rat motor cortex. Nature 427, 704–710 (2004). 89. Houweling, A. R. & Brecht, M. Behavioural report of single neuron stimulation in somatosensory cortex. Nature 451, 65–68 (2008). 90. Shadlen, M. N. & Newsome, W. T. Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. J. Neurophysiol. 86, 1916–1936 (2001).
NATuRE REVIEWS | NeuroscieNce
91. Fetz, E. E. Operant conditioning of cortical unit activity. Science 163, 955–958 (1969). 92. Fetz, E. E. & Finocchio, D. V. Correlations between activity of motor cortex cells and arm muscles during operantly conditioned response patterns. Exp. Brain Res. 23, 217–240 (1975). 93. Fetz, E. E. & Finocchio, D. V. Operant conditioning of specific patterns of neural and muscular activity. Science 174, 431–435 (1971). 94. Moritz, C. T., Perlmutter, S. I. & Fetz, E. E. Direct control of paralysed muscles by cortical neurons. Nature 456, 639–642 (2008). 95. Eliades, S. J. & Wang, X. Chronic multi-electrode neural recording in free-roaming monkeys. J. Neurosci. Methods 172, 201–214 (2008). 96. Guillory, K. S. & Normann, R. A. A 100-channel system for real time detection and storage of extracellular spike waveforms. J. Neurosci. Methods 91, 21–29 (1999). 97. Mountcastle, V. B., Reitboeck, H. J., Poggio, G. F. & Steinmetz, M. A. Adaptation of the Reitboeck method of multiple microelectrode recording to the neocortex of the waking monkey. J. Neurosci. Methods 36, 77–84 (1991). 98. Musallam, S., Bak, M. J., Troyk, P. R. & Andersen, R. A. A floating metal microelectrode array for chronic implantation. J. Neurosci. Methods 160, 122–127 (2007). 99. Nicolelis, M. A., Ghazanfar, A. A., Faggin, B. M., Votaw, S. & Oliveira, L. M. Reconstructing the engram: simultaneous, multisite, many single neuron recordings. Neuron 18, 529–537 (1997). 100. Grinvald, A. Imaging input and output dynamics of neocortical networks in vivo: exciting times ahead. Proc. Natl Acad. Sci. USA 102, 14125–14126 (2005). 101. Grinvald, A., Frostig, R. D., Siegel, R. M. & Bartfeld, E. High-resolution optical imaging of functional brain architecture in the awake monkey. Proc. Natl Acad. Sci. USA 88, 11559–11563 (1991). 102. Lendvai, B., Stern, E. A., Chen, B. & Svoboda, K. Experience-dependent plasticity of dendritic spines in the developing rat barrel cortex in vivo. Nature 404, 876–881 (2000). 103. Logothetis, N. K., Guggenberger, H., Peled, S. & Pauls, J. Functional imaging of the monkey brain. Nature Neurosci. 2, 555–562 (1999). 104. Nikolenko, V., Poskanzer, K. E. & Yuste, R. Two-photon photostimulation and imaging of neural circuits. Nature Methods 4, 943–950 (2007). 105. Ohki, K., Chung, S., Ch’ng, Y. H., Kara, P. & Reid, R. C. Functional imaging with cellular resolution reveals precise micro-architecture in visual cortex. Nature 433, 597–603 (2005). 106. Ohki, K. et al. Highly ordered arrangement of single neurons in orientation pinwheels. Nature 442, 925–928 (2006). 107. Rainer, G., Augath, M., Trinath, T. & Logothetis, N. K. Nonmonotonic noise tuning of BOLD fMRI signal to natural images in the visual cortex of the anesthetized monkey. Curr. Biol. 11, 846–854 (2001). 108. Siegel, R. M., Duann, J. R., Jung, T. P. & Sejnowski, T. Spatiotemporal dynamics of the functional architecture for gain fields in inferior parietal lobule of behaving monkey. Cereb. Cortex 17, 378–390 (2007). 109. Svoboda, K., Denk, W., Kleinfeld, D. & Tank, D. W. In vivo dendritic calcium dynamics in neocortical pyramidal neurons. Nature 385, 161–165 (1997). 110. Ts’o, D. Y., Frostig, R. D., Lieke, E. E. & Grinvald, A. Functional organization of primate visual cortex revealed by high resolution optical imaging. Science 249, 417–420 (1990). 111. Yuste, R. Fluorescence microscopy today. Nature Methods 2, 902–904 (2005). 112. Schmidt, E. M. Single neuron recording from motor cortex as a possible source of signals for control of external devices. Ann. Biomed. Eng. 8, 339–349 (1980). 113. Isaacs, R. E., Weber, D. J. & Schwartz, A. B. Work toward real-time control of a cortical neural prothesis. IEEE Trans. Rehabil. Eng. 8, 196–198 (2000). 114. Wolpaw, J. R. & McFarland, D. J. Control of a twodimensional movement signal by a noninvasive braincomputer interface in humans. Proc. Natl Acad. Sci. USA 101, 17849–17854 (2004). 115. Fitzsimmons, N. A., Drake, W., Hanson, T. L., Lebedev, M. A. & Nicolelis, M. A. Primate reaching cued by multichannel spatiotemporal cortical microstimulation. J. Neurosci. 27, 5593–5602 (2007). 116. Lebedev, M. A., O’Doherty, J. E. & Nicolelis, M. A. Decoding of temporal intervals from cortical ensemble activity. J. Neurophysiol. 99, 166–186 (2008).
VoluME 10 | julY 2009 | 539 © 2009 Macmillan Publishers Limited. All rights reserved
314
315
PersPectives 117. Santucci, D. M., Kralik, J. D., Lebedev, M. A. & Nicolelis, M. A. Frontal and parietal cortical ensembles predict single-trial muscle activity during reaching movements in primates. Eur. J. Neurosci. 22, 1529–1540 (2005). 118. Wessberg, J. & Nicolelis, M. A. Optimizing a linear algorithm for real-time robotic control using chronic cortical ensemble recordings in monkeys. J. Cogn. Neurosci. 16, 1022–1035 (2004). 119. Zacksenhouse, M. et al. Cortical modulations increase in early sessions with brain–machine interface. PLoS ONE 2, e619 (2007). 120. Costa, R. M. et al. Rapid alterations in corticostriatal ensemble coordination during acute dopaminedependent motor dysfunction. Neuron 52, 359–369 (2006). 121. Dzirasa, K. et al. Dopaminergic control of sleep-wake states. J. Neurosci. 26, 10577–10589 (2006). 122. Lin, S. C., Gervasoni, D. & Nicolelis, M. A. Fast modulation of prefrontal cortex activity by basal forebrain noncholinergic neuronal ensembles. J. Neurophysiol. 96, 3209–3219 (2006). 123. Haykin, S. Adaptive Filter Theory (PrenticeHall, Upper Saddle River, New Jersey, 2002). 124. Fetz, E. E. Are movement parameters recognizably coded in activity of single neurons? Behav. Brain Sci. 15, 679–690 (1992). 125. Carmena, J. M., Lebedev, M. A., Henriquez, C. S. & Nicolelis, M. A. Stable ensemble performance with single-neuron variability during reaching movements in primates. J. Neurosci. 25, 10712–10716 (2005). 126. Ghazanfar, A. A., Krupa, D. J. & Nicolelis, M. A. Role of cortical feedback in the receptive field structure and nonlinear response properties of somatosensory thalamic neurons. Exp. Brain Res. 141, 88–100 (2001). 127. Ghazanfar, A. A. & Nicolelis, M. A. Spatiotemporal properties of layer V neurons of the rat primary somatosensory cortex. Cereb. Cortex 9, 348–361 (1999). 128. Ghazanfar, A. A., Stambaugh, C. R. & Nicolelis, M. A. Encoding of tactile stimulus location by somatosensory thalamocortical ensembles. J. Neurosci. 20, 3761–3775 (2000). 129. de Araujo, I. E. et al. Food reward in the absence of taste receptor signaling. Neuron 57, 930–941 (2008). 130. Soares, E. S. et al. Behavioral and neural responses to gustatory stimuli delivered non-contingently through intra-oral cannulas. Physiol. Behav. 92, 629–642 (2007). 131. Glaser, E. M. & Ruchkin, D. S. Principles of Neurobiological Signal Analysis (Academic Press, New York, 1976). 132. Quian Quiroga, R. & Panzeri, S. Extracting information from neuronal populations: information theory and decoding approaches. Nature Rev. Neurosci. 10, 173–185 (2009). 133. Faisal, A. A., Selen, L. P. & Wolpert, D. M. Noise in the nervous system. Nature Rev. Neurosci. 9, 292–303 (2008). 134. Fontanini, A. & Katz, D. B. Behavioral states, network states, and sensory response variability. J. Neurophysiol. 100, 1160–1168 (2008). 135. Getting, P. A. Emerging principles governing the operation of neural networks. Annu. Rev. Neurosci. 12, 185–204 (1989). 136. Nicolelis, M. A. Computing with thalamocortical ensembles during different behavioural states. J. Physiol. 566, 37–47 (2005). 137. van Beers, R. J., Baraduc, P. & Wolpert, D. M. Role of uncertainty in sensorimotor control. Philos. Trans. R. Soc. Lond. B Biol. Sci. 357, 1137–1145 (2002). 138. Abeles, M. Neural Circuits of the Cerebral Cortex (Cambridge Univ. Press, Cambridge, 1991). 139. Chestek, C. A. et al. Single-neuron stability during repeated reaching in macaque premotor cortex. J. Neurosci. 27, 10742–10750 (2007). 140. Brooks, V. B., Adrien, J. & Dykes, R. W. Task-related discharge of neurons in motor cortex and effects of denatate cooling. Brain Res. 40, 85–88 (1972). 141. Niki, H. & Watanabe, M. Prefrontal unit activity and delayed response: relation to cue location versus direction of response. Brain Res. 105, 79–88 (1976). 142. Sanchez, J. C. et al. Ascertaining the importance of neurons to develop better brain–machine interfaces. IEEE Trans. Biomed. Eng. 51, 943–953 (2004). 143. Ghazanfar, A. A. & Schroeder, C. E. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285 (2006).
144. Graziano, M. S. & Gross, C. G. Spatial maps for the control of movement. Curr. Opin. Neurobiol. 8, 195–201 (1998). 145. Avillac, M., Deneve, S., Olivier, E., Pouget, A. & Duhamel, J. R. Reference frames for representing visual and tactile locations in parietal cortex. Nature Neurosci. 8, 941–949 (2005). 146. Benedek, G., Eordegh, G., Chadaide, Z. & Nagy, A. Distributed population coding of multisensory spatial information in the associative cortex. Eur. J. Neurosci. 20, 525–529 (2004). 147. Bridgeman, B. Multiplexing in single cells of the alert monkeys visual cortex during brightness discrimination. Neuropsychologia 20, 33–42 (1982). 148. Driver, J. & Noesselt, T. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57, 11–23 (2008). 149. Friedrich, R. W., Habermann, C. J. & Laurent, G. Multiplexing using synchrony in the zebrafish olfactory bulb. Nature Neurosci. 7, 862–871 (2004). 150. Lebedev, M. A., Messinger, A., Kralik, J. D. & Wise, S. P. Representation of attended versus remembered locations in prefrontal cortex. PLoS Biol. 2, e365 (2004). 151. Stanford, T. R. & Stein, B. E. Superadditivity in multisensory integration: putting the computation in context. Neuroreport 18, 787–792 (2007). 152. Stein, B. E. & Stanford, T. R. Multisensory integration: current issues from the perspective of the single neuron. Nature Rev. Neurosci. 9, 255–266 (2008). 153. Fitzsimmons, N. A., Lebedev, M. A., Peikon, I. D. & Nicolelis, M. A. Decoding of monkey bipedal walking from cortical neuronal ensembles. Front. Integr. Neurosci. 3, 3 (2009). 154. Alexander, R. M. Bipedal animals, and their differences from humans. J. Anat. 204, 321–330 (2004). 155. Dietz, V. Do human bipeds use quadrupedal coordination? Trends Neurosci. 25, 462–467 (2002). 156. Prilutsky, B. I., Sirota, M. G., Gregor, R. J. & Beloozerova, I. N. Quantification of motor cortex activity and full-body biomechanics during unconstrained locomotion. J. Neurophysiol. 94, 2959–2969 (2005). 157. Narayanan, N. S., Kimchi, E. Y. & Laubach, M. Redundancy and synergy of neuronal ensembles in motor cortex. J. Neurosci. 25, 4207–4216 (2005). 158. Shadlen, M. N. & Newsome, W. T. The variable discharge of cortical neurons: implications for connectivity, computation, and information coding. J. Neurosci. 18, 3870–3896 (1998). 159. Schwartz, A. B., Taylor, D. M. & Tillery, S. I. Extraction algorithms for cortical control of arm prosthetics. Curr. Opin. Neurobiol. 11, 701–707 (2001). 160. Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S. & Schwartz, A. B. Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101 (2008). 161. Cohen, D. & Nicolelis, M. A. Reduction of single-neuron firing uncertainty by cortical ensembles during motor skill learning. J. Neurosci. 24, 3574–3582 (2004). 162. Lashley, K. S. An examination of the “continuity theory” as applied to discrimination learning. J. Gen. Psychol. 26, 241–265 (1942). 163. Lashley, K. S. The mechanism of vision: XV. Preliminary studies of the rat’s capacity for detail vision. J. Gen. Psychol. 18, 123–193 (1938). 164. Leonardo, A. Degenerate coding in neural systems. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 191, 995–1010 (2005). 165. Reeke, G. N. Jr & Edelman, G. M. Selective networks and recognition automata. Ann. NY Acad. Sci. 426, 181–201 (1984). 166. Tononi, G., Sporns, O. & Edelman, G. M. Measures of degeneracy and redundancy in biological networks. Proc. Natl Acad. Sci. USA 96, 3257–3262 (1999). 167. Merzenich, M. M. et al. Topographic reorganization of somatosensory cortical areas 3b and 1 in adult monkeys following restricted deafferentation. Neuroscience 8, 33–55 (1983). 168. Merzenich, M. M. et al. Progression of change following median nerve section in the cortical representation of the hand in areas 3b and 1 in adult owl and squirrel monkeys. Neuroscience 10, 639–665 (1983). 169. Krupa, D. J., Wiest, M. C., Shuler, M. G., Laubach, M. & Nicolelis, M. A. Layer-specific somatosensory cortical activation during active tactile discrimination. Science 304, 1989–1992 (2004). 170. Chen, L. L. & Wise, S. P. Evolution of directional preferences in the supplementary eye field during
540 | julY 2009 | VoluME 10
acquisition of conditional oculomotor associations. J. Neurosci. 16, 3067–3081 (1996). 171. Laubach, M., Wessberg, J. & Nicolelis, M. A. Cortical ensemble activity increasingly predicts behaviour outcomes during learning of a motor task. Nature 405, 567–571 (2000). 172. Li, C. S., Padoa-Schioppa, C. & Bizzi, E. Neuronal correlates of motor performance and motor learning in the primary motor cortex of monkeys adapting to an external force field. Neuron 30, 593–607 (2001). 173. Mitz, A. R., Godschalk, M. & Wise, S. P. Learningdependent neuronal activity in the premotor cortex: activity during the acquisition of conditional motor associations. J. Neurosci. 11, 1855–1872 (1991). 174. Padoa-Schioppa, C., Li, C. S. & Bizzi, E. Neuronal activity in the supplementary motor area of monkeys adapting to a new dynamic environment. J. Neurophysiol. 91, 449–473 (2004). 175. Padoa-Schioppa, C., Li, C. S. & Bizzi, E. Neuronal correlates of kinematics-to-dynamics transformation in the supplementary motor area. Neuron 36, 751–765 (2002). 176. Paz, R., Boraud, T., Natan, C., Bergman, H. & Vaadia, E. Preparatory activity in motor cortex reflects learning of local visuomotor skills. Nature Neurosci. 6, 882–890 (2003). 177. Paz, R. & Vaadia, E. Learning-induced improvement in encoding and decoding of specific movement directions by neurons in the primary motor cortex. PLoS Biol. 2, e45 (2004). 178. Rokni, U., Richardson, A. G., Bizzi, E. & Seung, H. S. Motor learning with unstable neural representations. Neuron 54, 653–666 (2007). 179. Wise, S. P., Moody, S. L., Blomstrom, K. J. & Mitz, A. R. Changes in motor cortical activity during visuomotor adaptation. Exp. Brain Res. 121, 285–299 (1998). 180. de Lange, F. P., Roelofs, K. & Toni, I. Motor imagery: a window into the mechanisms and alterations of the motor system. Cortex 44, 494–506 (2008). 181. Decety, J. The neurophysiological basis of motor imagery. Behav. Brain Res. 77, 45–52 (1996). 182. Jeannerod, M. & Frak, V. Mental imaging of motor activity in humans. Curr. Opin. Neurobiol. 9, 735–739 (1999). 183. Neuper, C., Muller-Putz, G. R., Scherer, R. & Pfurtscheller, G. Motor imagery and EEG-based control of spelling devices and neuroprostheses. Prog. Brain Res. 159, 393–409 (2006). 184. Jackson, A., Mavoori, J. & Fetz, E. E. Long-term motor cortex plasticity induced by an electronic neural implant. Nature 444, 56–60 (2006). 185. Bach-y-Rita, P. & S., W. K. Sensory substitution and the human–machine interface. Trends Cogn. Sci. 7, 541–546 (2003). 186. Segond, H., Weiss, D. & Sampaio, E. Human spatial navigation via a visuo-tactile sensory substitution system. Perception 34, 1231–1249 (2005). 187. Eliades, S. J. & Wang, X. Dynamics of auditory-vocal interaction in monkey auditory cortex. Cereb. Cortex 15, 1510–1523 (2005). 188. Lin, S. C. & Nicolelis, M. A. Neuronal ensemble bursting in the basal forebrain encodes salience irrespective of valence. Neuron 59, 138–149 (2008). 189. Pantoja, J. et al. Neuronal activity in the primary somatosensory thalamocortical loop is modulated by reward contingency during tactile discrimination. J. Neurosci. 27, 10608–10620 (2007). 190. Pereira, A. et al. Processing of tactile information by the hippocampus. Proc. Natl Acad. Sci. USA 104, 18286–18291 (2007). 191. Stapleton, J. R., Lavine, M. L., Nicolelis, M. A. & Simon, S. A. Ensembles of gustatory cortical neurons anticipate and discriminate between tastants in a single lick. Front. Neurosci. 1, 161–174 (2007). 192. Kim, H. K. et al. Continuous shared control stabilizes reach and grasping with brain–machine interfaces. IEEE Trans. Biomed. Eng. 53, 1164–1173 (2005).
Acknowledgements
We thank N. Fitzsimmons for his assistance with designing figures for this manuscript.
FURTHER iNFoRMATioN Miguel A. L. Nicolelis’s homepage: http://www.nicolelislab.net Walk Again Project: http://www.walkagainproject.org All liNks Are AcTive iN The oNliNe PDf
www.nature.com/reviews/neuro © 2009 Macmillan Publishers Limited. All rights reserved
316 CLINICS 2011;66(S1):25-32
Future developments in brain-machine interface research Mikhail A. Lebedev,I Andrew J. Tate,I,II Timothy L. Hanson,I,II Zheng Li,I,II Joseph E. O’Doherty,I,II Jesse A. Winans,I Peter J. Ifft,I Katie Z. Zhuang,I Nathan A. Fitzsimmons,I David A. Schwarz,I Andrew M. Fuller,I Je Hi An,I Miguel A. L. NicolelisI,III I Neurobiology,Duke University, Durham, NC, USA. II Duke University Center for Neuroengineering, Duke University, Durham, NC, USA. III Edmond and Lily Safra International Institute of Neuroscience of Natal
Neuroprosthetic devices based on brain-machine interface technology hold promise for the restoration of body mobility in patients suffering from devastating motor deficits caused by brain injury, neurologic diseases and limb loss. During the last decade, considerable progress has been achieved in this multidisciplinary research, mainly in the brain-machine interface that enacts upper-limb functionality. However, a considerable number of problems need to be resolved before fully functional limb neuroprostheses can be built. To move towards developing neuroprosthetic devices for humans, brain-machine interface research has to address a number of issues related to improving the quality of neuronal recordings, achieving stable, long-term performance, and extending the brain-machine interface approach to a broad range of motor and sensory functions. Here, we review the future steps that are part of the strategic plan of the Duke University Center for Neuroengineering, and its partners, the Brazilian National Institute of Brain-Machine Interfaces and the École Polytechnique Fédérale de Lausanne (EPFL) Center for Neuroprosthetics, to bring this new technology to clinical fruition. KEYWORDS: Brain-machine interface; Neuroprosthetic; Primate; Bipedal locomotion; Intracortical microstimulation; Sensory substitution. Lebedev MA, Tate AJ, Hanson TL, Li Z, O’Doherty JE, Winans JA, Ifft PJ, Zhuang KZ, Fitzsimmons NA, Schwarz DA, Fuller AM, An JH, Nicolelis MAL. Future developments in brain-machine interface research. Clinics. 2011;66(S1):25-32. Received for publication on January 28, 2011; Accepted for publicaiton on January 30, 2011 E-mail: nicoleli@neuro.duke.edu Tel.: 919 684 4580
During the last decade, the field of BMIs has experienced an explosive development.7,9 Hence, it has generated high expectations among neuroscientists, physicians and patients alike, regarding its potential clinical applications. A number of BMI systems have been studied in rodents12 and nonhuman primates.13–17 BMI technology also entered human clinical research where both non-invasive EEGbased systems5,18,19 and invasive BMIs based on brain implants20–22 have been tested. Notwithstanding the success of these pioneering experiments, a number of issues need to be resolved before a fully functional practical neuroprosthetic for long-term use can be built.7 These include: implant biocompatibility issues;23 increasing the number of neural channels of the recording system; improving BMI decoding algorithms; building fully implantable systems; sensorizing neuroprosthetic limbs; and extending the BMI approach to a broader range of motor control tasks, especially tasks that require lower limb control: bipedal walking24 and upright posture control.25 The Duke University Center for Neuroengineering (DUCN) has been at the forefront of BMI research on cortical prosthetic devices for motor rehabilitation since this field emerged about 12 years ago. At the DUCN, we have developed pioneering BMI systems that enact a wide range of motor functions, from arm reaching and grasping 13,17,26 to bipedal locomotion24,27 in a variety of artificial actuators. DUCN researchers were also the first to incorporate artificial somatic sensation in BMIs.28,29 Here, we review the most recent findings of the BMI initiative at the DUCN and discuss
INTRODUCTION Millions of people worldwide suffer from sensorimotor deficits caused by neurologic injuries, diseases or limb loss. According to recent data reported in Medical News Today, five million people in the USA alone currently suffer from some type of severe body paralysis.1 Currently, there is no cure for such devastating cases of paralysis, for example complete spinal cord injury (SCI).2 Meanwhile, treatment is only partially effective in less severe cases.3 Neural prosthetic devices based on brain-machine interfaces (BMIs) hold promise to restore both partial and full body mobility in paralyzed patients.4–10 BMIs bypass the site of the neural lesion and connect the remaining healthy motor areas of the brain, particularly the motor cortex, directly to assistive and prosthetic devices that can take the shape of, for example, robotic limbs or a full body exoskeleton. The main idea behind BMIs is to employ the activity of healthy motor brain areas, which in many cases of paralysis remain capable of generating motor commands despite being disconnected from the body effectors,11 to control artificial tools that restore the patient’s mobility.
Copyright ß 2011 CLINICS – This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http:// creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted noncommercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
25
317 Brain-machine interface research Mikhail AL et al.
CLINICS 2011;66(S1):25-32
task related neurons.24,36 This corresponds to a 5 bits/s bandwidth. SNR can be further improved via time averaging, but this slows down the movements. Realistic and useful movements should have a SNR of about 20 to 30dB. Indeed, we consider this benchmark as the lower bound that one should aim in order to build a neuroprosthetic device for clinical use. Thus, to estimate the number of neurons needed to obtain this level of accuracy, we assume that noise decreases as the inverse of the square root of the number of neurons—an assumption that holds for both Gaussian and Poisson noise sources. Under this assumption, a tenfold increase in the sample of neurons recorded simultaneously would be needed to achieve the 10dB of improvement needed. Therefore, to obtain a control signal with 20dB fidelity (the DUCN short-term goal), recordings involving 1000 neurons are needed; to obtain 40dB (the DUCN long-term goal), 100,000 neurons are needed. In addition, it should be emphasized that commensurately more neurons are required for prosthetic limbs with many degrees of freedom (DOFs). As practical neuroprosthetics for paralyzed subjects will have to cope with at least 4–10 DOF, demands for large samples of recorded neurons will remain a central bottleneck for the development of clinical applications of BMIs for the foreseeable future. Thus, only when a new generation of high-channel count recordings becomes available can practical neuroprosthetics for clinical use be implemented with a reasonable expectation for clinical success. At the DUCN, we expect to conclude the development of such new technology and, as a consequence, launch the testing phase of our first clinical application of a BMI for restoring full body mobility by the summer of 2014. To achieve this goal, we have been developing advanced sensors that sample large-scale extracellular electrical activity from the brain. In our most recent design of multielectrode implants, electrodes are arranged into subsets sitting inside guiding tubes.37 The electrodes within these subsets have different lengths. The guiding tubes are spaced at 0.5–2 mm, and each of them is independently movable. This new three-dimensional (3D) design, named ‘the multi-electrode recording cube’ not only improves the size of the potential neuronal sample recorded simultaneously per probe, but also enhances the implantation capacity of the microelectrodes in the brain tissue as each electrode subset penetrates individually, minimizing cortical dimpling. Currently, we are expanding this new cube design to develop the next generation of implants that will increase the number of potential active recording sites per cube to about 1500. Also, as the size of each implanted microelectrode is of vital importance when so many electrodes are inserted into richly vascularized brain tissue, it is imperative to produce as little tissue displacement as possible during the surgical implantation of these recording devices. This issue will be resolved by removing the structural elements after implantation, thereby freeing the microelectrodes made of smaller diameter microwires than that required to pierce and penetrate the brain tissue. Not only will small diameter microwires minimize the mechanical distortion of nervous tissue, it will also help to minimize microglial and other immune responses to the foreign material. Our approach will allow implantation of smalldiameter wires and thereby avoid failures associated with large-diameter implants. Fine electrodes will be guided into the brain with a strong, stiff tungsten central shaft. Later on,
its perspectives and strategic plan for the near- and long-term future of BMI research.
BMI COMPONENTS The essential components of a BMI system are well captured by the BMI that enacts reaching and grasping.7,9,13,17,26 In this BMI design, a rhesus monkey controls a robotic arm with its motor cortical activity, while visual and/or somatosensory feedback signals from the robot are delivered back to the brain as either natural visual stimuli or, in the case of artificial tactile information, intracortical microstimulation (ICMS) of the primary somatosensory cortex (S1).7,28,29 In these studies, we implant multielectrode arrays in multiple cortical areas of the rhesus monkey’s brain.30 The present generation of chronic multielectrode implants is capable of recording the extracellular electrical activity of hundreds of cortical cells.13,24,26,30 As a result of recent technological developments, this benchmark number is expected to rise to several thousands of simultaneously recorded neurons in the next decade. Recording such largescale neuronal ensemble activity is crucially important for BMI accuracy.9 Concurrent neuronal ensemble activity is processed by BMI decoding algorithms which translate myriad neuronal spikes into continuous signals that drive the robotic arm’s movements, according to the voluntary motor intentions of the subject. The BMI setup also includes the data acquisition system, the computer cluster running multiple decoding models in real time, the robot arm, the visual display and a sensory feedback loop from the actuator to the brain. Below we discuss these key BMI components in more detail.
LARGE SCALE NEURONAL RECORDINGS Our work on BMIs has clearly demonstrated that a large number of recording channels is needed for accurate extraction of motor intentions from the brain.9,13,17,31,32 Shifting from the classic focus on single neurons, today, more and more evidence accumulates in favor of the notion that distributed ensembles of neurons define the true physiologic unit of the mammalian central nervous system.9 During the last two decades, advanced electrophysiological methods have allowed recording from progressively larger samples of single neurons in behaving animals.30,33–35 This methodology has equipped neuroscientists with better tools to study the neurophysiologic principles that define the operation of the cortex,9 and has also made the idea of BMIs practical. Today, the most advanced BMIs developed at the DUCN utilize simultaneous recordings of the extracellular electrical activity of hundreds of individual neurons.30 Neuron-dropping analysis, a technique developed at the DUCN to measure the dependence of parameter extraction accuracy on the number of simultaneously recorded neurons,17 shows that BMI accuracy increases with the size of the neuronal ensemble. Typically, the best extractions of motor parameters are obtained when the activity of populations of neurons from several cortical areas is recorded simultaneously.13,24 Importantly, the number of required neurons increases as more motor parameters are simultaneously extracted.24 Large neural ensembles confer redundancy of control and, hence, reliability.9 In our current research on BMIs that enact arm and leg movements, we can obtain a peak signal-to-noise ratio (SNR) of 10dB using approximately 100
26
318 CLINICS 2011;66(S1):25-32
Brain-machine interface research Mikhail AL et al.
practical BMI applications will be inconvenient to use if the user is required to control each degree of freedom independently. To overcome this problem, we intend to implement in our clinical applications a mode of BMI operation in which the control over an external actuator is shared between the subject’s brain activity and robotic controls. This mode of operation is known as shared control of a BMI.39 During shared control, the subject’s brain is primarily in charge of high-order control of movements (when to initiate movement and where to move), whereas the low-level coordination of the movement is performed by an autonomous controller. Such sharing optimizes BMI performance: the user can control it through their voluntary intentions, whereas the robotic controller assures accuracy and stability.
the shaft will be removed, leaving the electrodes in the brain. Each single electrode shaft will carry 10–20 recording microwires, staggered to cover the targeted nervous tissue. To refer properly to this new technology and to distinguish it from previous approaches, we have coined the term ‘verylarge scale multichannel brain implants’ (VLS-MBI).
DECODING ALGORITHMS Large-scale neuronal activity recorded by our very largescale multichannel brain implants will be processed by realtime BMI encoding algorithms that extract behavioral parameters, for example kinematic parameters of many DOF limb movements. At the DUCN, we employ an integrated BMI suite that combines the neurophysiologic recording and stimulation hardware, as well as the computer cluster that runs behavioral tasks and BMI decoders.24,29 We have incorporated into this BMI suite several neuronal decoders: the unscented Kalman filter,36 the Wiener filter, artificial neural networks, and discrete state Bayesian approaches.38 In particular, the unscented Kalman filter makes Bayesian inferences of the repetitive patterns of the movements performed during arm reaching tasks, as such patterns will occur frequently in the tasks that a practical neuroprosthetic limb has to perform. For example, a prosthetic limb that aids in feeding moves back and forth between the food item and the user’s mouth. The unscented Kalman filter exploits non-linear models of neural tuning and prior knowledge about movement patterns. In addition, it keeps a short history of the state variables—in this case, the desired limb movements. This algorithm also captures complex patterns in the desired movements. At the DUCN, the unscented Kalman filter was tested in a BMI that enacted arm reaching movements and achieved significantly better accuracy as compared to the Kalman filter, the Wiener filter, the population vector method, and the stochastic state point process filter.38 In addition to the unscented Kalman filter, we employ a multiple-model-switching paradigm.24 In this approach, separate submodels are trained to decode particular behavioral states (e.g. the grasping phase of the reach and grasp movement versus the reaching phase, or walking forward versus walking backward), and the state predictor model serves to detect the state and select the appropriate submodel. The simplest switching model works as a combination of three linear decoding models: a model for predicting state 1 (e.g. reaching), a model for state 2 (e.g. grasping) and the paradigm predictor model (the switch). These models are arranged in two layers with the paradigm predictor model controlling a toggle between the two kinematic submodels, which are then shunted to the final output of the switching model. When it is determined that the monkey is performing in state 1, one submodel is used to produce the output, and when state 2 is detected, the other submodel is used. These algorithmic tools allow us to test different neural control modes. In the most straightforward implementation, we predict the kinematics of each joint of the limb and convert them into the actuator configuration. This control model is of interest for several reasons. First, this mode of operation is similar to the operation of the nervous system in controlling natural movements. Second, the number of degrees of freedom that can be achieved by a BMI is of great interest to those working in this field. At the same time,
BRAIN-MACHINE-BRAIN INTERFACE The DUCN research team was the first to add an artificial somatosensory feedback loop to a BMI for arm reaching. In our initial studies, we used spatiotemporal patterns of ICMS to instruct primate arm reaches. In a study conducted in owl monkeys,40 we addressed two issues that are critical for using ICMS as an artificial sensory channel in neural prostheses: (i) whether such artificial sensation can be evoked by multichannel ICMS; and (ii) whether ICMS is suitable for long-term usage in a BMI-like application. We explored the first issue by testing the capacity of owl monkeys to discriminate multichannel ICMS of increasing complexity. We investigated the second issue by testing the long-term efficacy of ICMS over many months. We discovered that owl monkeys could learn to discriminate spatiotemporal patterns of ICMS delivered directly to their S1 and guide their reaching movements based on this discrimination. Owl monkeys were implanted with multielectrode arrays in several cortical areas, and S1 implants were employed to deliver spatiotemporal patterns of ICMS. The behavioral task performed by the owl monkeys progressed from a simple requirement of detecting the presence of ICMS, to the requirement of discriminating spatiotemporal patterns created using four electrode pairs. The monkeys successfully learned these tasks. Interestingly, they learned new microstimulation patterns more rapidly compared with initial training. This result suggests that the monkey brain may have mimicked the operation of this intra-cortical microstimulation paradigm as if it were a new sensory channel. In a study conducted in rhesus monkeys, we employed ICMS to cue monkeys that performed BMI reaching tasks.29 The behavioral tasks consisted of acquiring visual targets with a computer cursor. The monkeys first performed the behavioral tasks manually, using a hand-held joystick, and later controlled the cursor movements directly with the electrical activity of a sample of cortical neurons. Manual performance data were used to train linear decoding models that extracted cursor position from the modulations of populations of simultaneously recorded cortical cells. Once the model parameters were calculated, the mode of operation was switched to brain control during which the joystick was disconnected from the cursor and cursor position became directly controlled by the signals extracted from the animal’s brain. ICMS of S1 was then used to cue monkeys to which direction they had to move their arms during execution of a target choice task. In this experiment, monkeys had to choose among two visually identical targets
27
319 Brain-machine interface research Mikhail AL et al.
CLINICS 2011;66(S1):25-32
discharges,44 would be difficult to incorporate in such implementation. Because of these foreseen difficulties in the straightforward implementation of an artificial position sense, we chose a simpler and more feasible approach in which stimulation of S1 is not initially coupled to the orientation of the limb position, but instead represents 3D spatial locations to which the subject is required to reach. In this approach, the subject starts with learning how ICMS of cortical somatotopic representations of the body is mapped to a 3D space. Once the learning of such mapping is perfected, the same ICMS pattern is used to represent the spatial location of the endpoint of a prosthetic limb, fulfilling the goal of providing a limb neuroprosthetic with a position sense. This experimental design bears similarity to the studies on sensory substitution in which visual information was conveyed by the stimulation of skin surfaces,45–47 with the difference that, instead of using peripheral receptors as the entry point for the artificial sensory channel, we opted to deliver the information directly to the somatosensory cortex or thalamus. We expect that similarly to sensory substitution using peripheral stimulation, training with stimulation of the somatosensory areas of the brain would eventually give rise to an artificial position sense that represents external space. Furthermore, we expect that as such position sense becomes coupled to the position of a virtual image of a monkey arm (monkey avatar), monkeys will develop an artificial proprioception sense capable of guiding their movements thereafter.
based on an instruction, in the format of spatiotemporal pattern of ICMS, delivered directly to the animals’ S1. The monkeys were implanted with multiple microwire arrays in several cortical areas. The dorsal premotor cortex (PMd) and primary motor cortex (M1) arrays were used to extract motor commands, while the primary somatosensory cortex (S1) array was employed as the main target for ICMS. The electrodes chosen for ICMS yielded recorded S1 neurons with clear receptive fields located on the ventral aspect of the second, third and fourth digits and palm pads. Biphasic current pulses were injected into S1 through these electrode pairs synchronously at 30 to 60 Hz. We simultaneously recorded the electrical activity of 50–200 cortical neurons. The monkeys learned to perform in BMI control with and without using the joystick. Moreover, in our recent study,28 ICMS served as an artificial sense of active touch as it conveyed to the monkeys the properties of virtual objects that the actuator (computer cursor or a virtual image of a monkey arm) touched. Two monkeys learned to operate this new apparatus, without any need to move their own limbs or use visual feedback to solve the task. Thus, as a result of converging principles of motor and sensory neuroprosthetics, we have demonstrated the feasibility of moving from a BMI to a brain-machine-brain interface in which artificial actuators and brain tissue are connected bi-directionally, without any meaningful interference or constraint imposed by the physical limits of the subject’s body.
OPTOGENETICS
POSITION SENSE
Our initial approach to sensorizing neuroprosthetic limbs was based on ICMS, which involves delivering small pulses of current through microelectrodes directly into the sensory areas of the brain.28,29,40 Although this technique proved to be efficient in bi-directional BMIs, it has relatively low spatial resolution48 and produces electrical artifacts that saturate neural recording channels,49 causing problems for extracting neural activity both during and after this period, as it typically occludes 5–10 ms of neural data per pulse. ICMS acts on both fibers and neurons, and neurons in the stimulated area may become inhibited instead of being excited.50 A further problem is that ICMS pulses have to be precisely balanced, otherwise an unbalanced current injection could cause damage to the neural tissue stimulated by this procedure. To explore an alternative method for sensorizing neuroprosthetics, we have started to design a cortical multichannel stimulator based on optogenetics—a new technique for the stimulation of neurons. Optogenetics is based on genetically modified ion channels that respond directly to light.51 These light-gated ion channels, such as Channelrhodopsin-2 (Chr-2), allow precise, millisecond control of specific neurons.52,53 This technique eliminates most of the key problems associated with ICMS: there is no associated electrical artifact to interfere with the electrophysiological recordings, nor any tissue damage from the current injection. It also allows for finer control of the spatial pattern of activation. We expect that multichannel optogenetic stimulation combined with multichannel neuronal recordings will allow us to develop an artificial position sense in monkeys and to study the neuronal mechanisms involved in it. Bi-directional BMIs based on such a design
Having created a paradigm for introducing an artificial sense of touch to our BMIs, the next goal of the DUCN is to develop neuroprosthetic limbs that incorporate an artificial sense of position. Position sense is of great importance for clinical applications because ideal prosthetic limbs should be perceived as natural extensions of the users’ bodies. Normally, such positional signals are provided by the afferents of muscles, joints and skin. Afferent information ascends to the brain where it is eventually transformed into representations encoding limb position in different coordinate frames, such as body-centered and external spacecentered coordinates.41,42 The complexity of spatial processing in the brain makes the task of creating an artificial position sense particularly difficult. Theoretically, artificial proprioception could be implemented in a straightforward way by applying microstimulation to the somatosensory cortex or thalamus that reproduces individual joint angles of an artificial arm. The subject would then experience sensation of many DOFs of the artificial arm and would be able to perceive its spatial orientation. However, practical realization of such position sense would be problematic because of numerous uncertainties in choosing the transformation from the stimulation patterns to the joint angles. Given the complexity of cortical processing of proprioceptive information,43 it would be impossible to implement such transformation as a precise mapping from the arm joints to the brain somatosensory map. It is also questionable if the user would be able to transform the stimulation patterns designed to mimic individual joints into a coherent position sense. Additionally, certain centrally generated components of normal position sense, such as corollary
28
320 CLINICS 2011;66(S1):25-32
Brain-machine interface research Mikhail AL et al.
posterior parietal cortex (PPC); and M1.60 We will implant these areas to build the first bimanual BMI. In our bimanual BMI experiment, monkeys have to move two avatar hands within an initial target that appears in the center of the screen. When this task is learned, the monkey will learn a similar behavior, except that two peripheral targets will appear, and the monkeys will have to guide each avatar hand to the separate targets. We expect that monkeys will eventually learn to control such bimanual movements through a BMI, without engaging any overt movements of their limbs.
would be superior to current designs in both the specificity and long-term performance of the sensory loop and the quality of neuronal recordings. We are currently building a prototype optogenetic implant that will simultaneously sample the activity of cortical neuronal ensembles and deliver complex stimulation patterns through multichannel optogenetic stimulation of cortical sensory areas. Our optogenetic arrays consist of both recording electrodes and fibers for optical stimulation. This design uses a novel light delivery system in which light pulses from a laser are directed onto a Digital Micromirror Device. These DMD chips consist of over 150,000 individually movable micrometer size mirrors. Depending on the alignment of these mirrors, they will either allow the light to bypass the chip or reflect it into the components of a multicore fiberoptic, therefore allowing incredibly sophisticated patterns of activation. The multicore fibers consist of 30,000 individual strands. A number of these strands will be attached to movable electrode shafts allowing them to penetrate deep into the cortical columnar structure, thus negating the problem of light scatter by the tissue and allow very specific targeting of individual neurons. After the Chr-2 gene is delivered to the somatosensory and motor cortex of our monkeys through adeno-associated virus vector (AAV) injections performed using implanted cannulae, combined optical-stimulation/recording grids will be inserted in the cortex through the same cannulae. We expect that implanted animals will learn to use the spatiotemporal optogenetic stimulation delivered to their S1 as BMI feedback. We will integrate this artificial sensory channel into our real-time BMI system, in which monkeys control the movements of primate virtual bodies, known as avatars, directly by brain activity while receiving sensory input through spatiotemporal optogenetic stimulation.
BIPEDAL LOCOMOTION Previous BMI studies focused predominantly on the behavioral tasks in which an artificial actuator enacted upper extremity movements, such as reaching and grasping. Except for a few studies,24,61,62 virtually no attempts have been made to translate BMI technology to tasks enacting motor functionality of lower extremities. Yet, deficits or the complete loss of the ability to walk present a considerable problem. Such deficits commonly result from spinal cord injury,63â&#x20AC;&#x201C;65 neurologic diseases66,67 and limb loss.68 Surveys of paraplegic patients showed that they prioritized walking and trunk stability among the most desired motor functions they would like to be restored.69,70 Quadriplegic patients prioritized arm and hand function.69 Thus, developing neural prosthetic devices for restoration of leg mobility is as important as developing neuroprosthetics for resuming arm and hand movements, and for some categories of patients it is their top priority. Such neuroprosthetic devices would clearly have a major impact on the community of patients suffering from leg paralysis. In our treadmill locomotion setup, rhesus macaques walked bipedally on a custom modified treadmill.24 We tracked movements of the right legs of the monkeys using a wireless, video-based tracking system developed in our laboratory.71 We then added a second tracking system to the setup and tracked movements of both legs. Leg kinematics and electromyographies (EMGs) of leg muscles were reconstructed from neuronal ensemble activity in real time using the Wiener filter.13,17,26,72 While the individual neuronal firing rates were highly variable from step to step at the millisecond scale, combining the activity of many neurons using a BMI decoder produced accurate predictions of leg movements. Extraction of neural information performed separately for different cortical areas showed that walking parameters could be predicted using neuronal activity recorded in either M1 or S1 contralateral to the right leg, as well as ipsilateral M1. In these experiments, neuronal ensemble activity was simultaneously recorded from all 384 channels. Using the locomotion setup, we also demonstrated real-time BMI control of bipedal walking in a robot.27 We sent the predictions of monkey leg kinematics to the Advanced Telecommunications Research Institute (ATR), Kyoto, Japan, where our collaborators set their humanoid 51-degree of freedom robot (manufactured by SARCOS, Salt Lake City, UT, USA) to reproduce monkey locomotion patterns. The monkey received visual feedback of the robotâ&#x20AC;&#x2122;s movements on a video screen.
BIMANUAL BMI So far, BMIs for arm movements involved only a single actuator that typically enacted reaching and grasping movements. Future BMIs will have to generate bimanual functionality. In their truest sense, bimanual movements are temporally and spatially coordinated movements of both upper limbs. At the DUCN, we are designing a BMI capable of extracting the motor commands to enact such coordination. The starting point in developing a bimanual BMI is to understand the underlying cortical processes. Bimanual processes activate different subsets of neurons than unimanual tasks.54â&#x20AC;&#x201C;59 As previous BMI research has focused on unimanual tasks, a key component of transitioning to successful bimanual control will be recognizing and adapting to the differences in the cortical representation of this behavior. Unimanual movement initiation and coordination have often been shown to be correlated to neural activity in the contralateral hemisphere of the motor cortex. However, in tasks involving both arms, activated cortical areas are less clearly defined and have greater interhemispheric interactions. Interhemispheric connections of premotor regions have been the subject of several investigations, and several findings point to the fact that these regions exhibit much greater interhemispheric connectivity than M1.59 At least five cortical regions show activation related to inter-limb coordination: dorsal premotor cortex (PMd); cingulate motor area (CMA); supplementary motor area (SMA);
POSTURE AND BALANCE BMI for restoration of lower limb function is fundamentally different from the BMI for upper limbs because it has
29
321 Brain-machine interface research Mikhail AL et al.
CLINICS 2011;66(S1):25-32
to not only replicate basic gait and stepping functions, but also adapt to postural control. In addition to developing a BMI for bipedal locomotion, we have developed a proof of concept BMI for postural control.25 Using this experimental setup, in which monkeys stood on a platform that produced abrupt horizontal movements and evoked postural perturbations, we demonstrated recently that cortical neurons modulate their firing rate in relation to changes in posture. We then applied BMI decoders to extract information about these changes from cortical ensemble activity. The platform was driven either rhythmically, allowing the animal to anticipate the upcoming perturbation, or with a random time delay between movements so that the animal was unable to predict when the next movement would come. Similarly to our treadmill experiments, single unit analysis of the neural activity showed that a vast majority of cortical neurons (mostly in M1 and S1 representations of the legs) were highly modulated in association with compensatory postural reactions. These modulations were directionally tuned: neuronal rates changed differently depending on the direction of platform displacement. The BMI decoder based on a linear model predicted postural disturbances with good signal-to-noise ratios. Importantly, the decoder yielded different results depending on anticipated versus unanticipated platform displacements. Model parameters were different in these two cases, and anticipated displacements were predicted with higher accuracy than anticipated. These results show the feasibility of a BMI for restoration of upright posture. The concrete implementation of such a BMI should be based on shared control rather than relying solely on cortical modulations for postural control.
a cortically driven FES system for the restoration of walking. This novel BMI will be tested in a monkey model of bipedal walking. Our objective is to build a BMI-driven FES system that produces bipedal locomotion patterns by converting cortical ensemble activity into stimulation patterns that drive leg muscles. We suggest that BMI control over bipedal locomotion can be established by recording large-scale cortical activity from the sensorimotor cortex, extracting locomotion patterns from the raw cortical signals and converting them into trains of FES applied to multiple leg muscles.
WHOLE-BODY NEUROPROSTHETIC As follows from our results on BMIs that enact leg movements, BMIs for the whole body are likely to become a real possibility in the near future. We propose the development of a whole-body BMI in which neuronal ensemble activity recorded from multiple cortical areas in rhesus monkeys controls the actuators that enact movements of both upper and lower extremities.37 This BMI will be first implemented in a virtual environment (monkey avatar) and then using a whole-body exoskeleton. In these experiments we will also examine the plasticity of neuronal ensemble properties caused by their involvement in the whole-body BMI control and the ability of cortical ensembles to adapt to represent novel external actuators. Furthermore, we will also explore the ability of an animal to navigate a virtual environment using both physical and neural control that engages both the upper and lower limbs. The first phase of these experiments will be to train the animals to walk in a bipedal manner on a treadmill while assisting the navigation with a hand steering control. We have already built a virtual environment needed for the monkey to navigate using 3D visualization software. Within this environment, the monkey’s body is represented by a life-like avatar. This representation is viewed in the third person by the monkey and employs real-world inverse kinematics to move, allowing the avatar’s limbs to move in close relation to the experimental animal. Initially, the direction that the avatar is facing will be dictated by the monkey moving a handlebar with its hands. As the animal moves the handlebar left or right, the avatar will rotate in the corresponding direction. The avatar’s legs will mimic the exact motion of the monkey’s legs on the treadmill. The simplest task will be for the animal to simply move the avatar forward to an object that represents a reward, a virtual fruit. Virtual fruits will appear at different angular positions relative to the monkey, which will let us measure the neuronal representation of navigation direction and modulations in cortical arm representation related to the steering. The monkey will have to make several steps while steering in the required direction to approach a virtual reward and to obtain an actual reward. The next set of experiments will allow the animal to control the virtual BMI in a manner similar to how we anticipate that the eventual application will be used: with no active movement of the subject’s body parts. The animals will use the neural control of the environment to obtain rewards when they are seated in a monkey chair. We expect that the monkey will be able to generate periodic neural modulations associated with individual steps of the avatar even though it does not perform any actual steps with its own legs.
INTEGRATING FUNCTIONAL ELECTRICAL STIMULATION (FES) TO THE BMI PARADIGM The issue of directly actuating the movements of paralyzed legs has been addressed in many clinical studies in spinal cord injury patients. Robotic orthoses73–75 and functional electrical stimulation (FES) devices76–79 have been introduced as therapies for leg paralysis. FES devices for the lower extremities date back to the 1960s when the first FES application to restore standing was developed.80 In this study, paraplegic subjects were able to stand after FES was applied to their quadriceps and gluteus maximus muscles. Also in the 1960s, FES of the common peroneal nerve was used to correct foot drop during the swing phase of the gait.81 Foot drop stimulators have been commercialized.82,83 This technique is used in a ParastepH system developed by Sigmedics, Inc, Fairborn, OH, USA.84,85 At present, a variety of FES methods have been developed that allow stimulation of multiple muscle groups with surface or intramuscular electrodes. Importantly, FES systems have many positive therapeutic effects on patients with incomplete spinal cord injury (SCI).79 Unfortunately, current FES systems allow for only crude movement such as short-distance ambulation or short-term standing, require support from a walker or similar tool, and are primarily offered as a way to avoid medical complications of prolonged limb inactivity rather than as an alternative to locomotion restoration. Additionally, such systems require manual actuation by the user, resulting in a limited and arguably non-intuitive way to restore leg function. Building on our success in BMIs that enact bipedal locomotion and postural control, we are currently developing
30
322 CLINICS 2011;66(S1):25-32
Brain-machine interface research Mikhail AL et al. 14. Moritz CT, Perlmutter SI, Fetz EE. Direct control of paralysed muscles by cortical neurons. Nature. 2008;456:639–42, doi: 10.1038/nature07418. 15. Taylor DM, Tillery SI, Schwartz AB. Direct cortical control of 3D neuroprosthetic devices. Science. 2002;296:1829–32, doi: 10.1126/science. 1070291. 16. Velliste M, Perel S, Spalding MC, Whitford AS, Schwartz AB. Cortical control of a prosthetic arm for self-feeding. Nature. 2008;453:1098–101, doi: 10.1038/nature06996. 17. Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, et al. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature. 2000;408:361-5, doi: 10.1038/35042582. 18. Allison BZ, Wolpaw EW, Wolpaw JR. Brain-computer interface systems: progress and prospects. Expert Rev Med Devices. 2007;4:463–74, doi: 10. 1586/17434440.4.4.463. 19. Pfurtscheller G, Neuper C. Future prospects of ERD/ERS in the context of brain-computer interface (BCI) developments. Prog Brain Res. 2006;159:433–7, doi: 10.1016/S0079-6123(06)59028-4. 20. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature. 2006;442:164–71, doi: 10.1038/nature04970. 21. Kennedy PR, Bakay RA. Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport. 1998;9:1707–11, doi: 10.1097/00001756-199806010-00007. 22. Patil PG, Carmena JM, Nicolelis MA, Turner DA. Ensemble recordings of human subcortical neurons as a source of motor control signals for a brain-machine interface. Neurosurgery. 2004;55:27–35. 23. Polikov VS, Tresco PA, Reichert WM. Response of brain tissue to chronically implanted neural electrodes. J Neurosci Methods. 2005;148:1– 18, doi: 10.1016/j.jneumeth.2005.08.015. 24. Fitzsimmons NA, Lebedev MA, Peikon ID, Nicolelis MA. Extracting kinematic parameters for monkey bipedal walking from cortical neuronal ensemble activity. Front Integr Neurosci. 2009;3:3, doi: 10. 3389/neuro.07.003.2009. 25. Tate AJ, Lebedev MA, Nicolelis MA. Neural activity associated with changes in posture in rhesus macaques. Paper presented at: Neuroscience 2009 Chicago, IL, USA. 2009. 26. Lebedev MA, Carmena JM, O’Doherty JE, Zacksenhouse M, Henriquez CS, Principe JC, et al. Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface. J Neurosci. 2005;25:4681–93, doi: 10.1523/JNEUROSCI.4088-04.2005. 27. Cheng G, Fitzsimmons NA, Morimoto J, Lebedev MA, Kawato M, Nicolelis MA. Bipedal locomotion with a humanoid robot controlled by cortical ensemble activity. Paper presented at: Annual Meeting of the Society for Neuroscience, 2007 San Diego, California. 28. O’Doherty JE, Ifft PJ, Zhuang KZ, Lebedev MA, NIcolelis MA. Brainmachine-brain interface using simultaneous recording and intracortical microstimulation feedback Paper presented at: Neuroscience 2010, 2010; San Diego, CA, USA. 29. O’Doherty JE, Lebedev MA, Hanson TL, Fitzsimmons NA, Nicolelis MA. A brain-machine interface instructed by direct intracortical microstimulation. Front Integr Neurosci. 2009;3:1–10, doi: 10.3389/neuro.07.020.2009 30. Nicolelis MA, Dimitrov D, Carmena JM, Crist R, Lehew G, Kralik JD, et al. Chronic, multisite, multielectrode recordings in macaque monkeys. Proc Natl Acad Sci U S A. 2003;100:11041–6, doi: 10.1073/pnas. 1934665100. 31. Lebedev MA, Fitzsimmons NA, Drake W, Lehew G, Dimitrov DF, Nicolelis MAL. Decoding Bipedal Locomotion Patterns From Cortical Ensemble Activity in Rhesus Monkeys. Paper presented at: Society for Neuroscience Annual Meeting, San Diego, CA. 2007. 32. Santucci DM, Kralik JD, Lebedev MA, Nicolelis MA. Frontal and parietal cortical ensembles predict single-trial muscle activity during reaching movements in primates. Eur J Neurosci. 2005;22:1529–1540, doi: 10.1111/ j.1460-9568.2005.04320.x. 33. Churchland MM, Yu BM, Sahani M, Shenoy KV. Techniques for extracting single-trial activity patterns from large-scale neural recordings. Curr Opin Neurobiol. 2007;17:609–18, doi: 10.1016/j.conb.2007.11. 001. 34. Nicolelis MA, Baccala LA, Lin RC, Chapin JK. Sensorimotor encoding by synchronous neural ensemble activity at multiple levels of the somatosensory system. Science. 1995;268:1353–8, doi: 10.1126/science. 7761855. 35. Nicolelis MA, Ribeiro S. Multielectrode recordings: the next steps. Curr Opin Neurobiol. 2002;12:602–6, doi: 10.1016/S0959-4388(02)00374-4. 36. Li Z, O’Doherty JE, Hanson TL, Lebedev MA, Henriquez CS, Nicolelis MA. Unscented Kalman filter for brain-machine interfaces. PloS One. 2009;4:e6243, doi: 10.1371/journal.pone.0006243. 37. Winans JA, Tate AJ, Lebedev MA, Nicolelis MA. Extraction of leg kinematics from the sensorimotor cortex representation of the whole body, Paper presented at: Neuroscience 2010, San Diego, CA, USA. 2010. 38. Li Z, O’Doherty JE, Lebedev MA, Nicolelis MA. Closed-loop adaptive decoding using bayesian regression self-training, Paper presented at: Neuroscience 2010, San Diego, CA, USA. 2010. 39. Kim HK, Biggs SJ, Schloerb DW, Carmena JM, Lebedev MA, Nicolelis MAL, et al. Continuous shared control stabilizes reach and grasping with
Finally, we will use the algorithms developed in these experiments to control a full-body monkey exoskeleton in a non-human primate which has been subjected to a spinal cord anesthetic block to produce a temporary and reversible state of quadriplegia. This exoskeleton will encase the monkey’s arms and legs. It will be attached to the monkey using bracelets molded in the shape of the monkey’s limbs. A full body exoskeleton prototype will be utilized. The basic design and controller will be based on the humanoid robot, Computational Brain (CB).27,86 The exoskeleton will provide full-sensory feedback to the BMI set up—joints position/ velocity/torque, ground contacts and orientations. In BMI mode, the exoskeleton will guide the monkey’s limbs with smooth motions while at the same time monitoring its range of motions to ensure it is within the safety limits. This demonstration will provide the first prototype of a neural prosthetic device that would allow paralyzed people to walk again.
CONCLUSIONS BMI technology offers a revolutionary treatment for paralysis. Recent studies suggest that BMIs have the potential to restore mobility to both upper and lower extremities and to enable a range of motor tasks, from arm reaching and grasping, to bipedal locomotion and balance. Moreover, it is feasible to enhance BMIs with an artificial somatosensory feedback either through ICMS or optogenetic stimulation. We envision that multidisciplinary BMI research will lead to the creation of whole-body neural prosthetic devices aimed at restoring full, essential mobility functions to paralyzed patients.
REFERENCES 1. Paddock C. Paralysis affects more Americans than previously thought. Available at: http://www.medicalnewstoday.com/articles/146819.php. 2. Dobkin BH, Havton LA. Basic advances and new avenues in therapy of spinal cord injury. Ann Rev Med. 2004;55:255–82, doi: 10.1146/annurev. med.55.091902.104338. 3. Fouad K, Pearson K. Restoring walking after spinal cord injury. Prog Neurobiol. 2004;73:107–26, doi: 10.1016/j.pneurobio.2004.04.003. 4. Andersen RA, Musallam S, Pesaran B. Selecting the signals for a brainmachine interface. Curr Opin Neurobiol. 2004;14:720–26, doi: 10.1016/j. conb.2004.10.005. 5. Birbaumer N, Cohen LG. Brain-computer interfaces: communication and restoration of movement in paralysis. J Physiol. 2007;579:621–36, doi: 10. 1113/jphysiol.2006.125633. 6. Fetz EE. Volitional control of neural activity: implications for braincomputer interfaces. J Physiol. 2007;579:571–9, doi: 10.1113/jphysiol. 2006.127142. 7. Lebedev MA, Nicolelis MA. Brain-machine interfaces: past, present and future. Trends Neurosci. 2006;29:536–46, doi: 10.1016/j.tins.2006.07.004. 8. Mussa-Ivaldi FA, Miller LE. Brain-machine interfaces: computational demands and clinical needs meet basic neuroscience. Trend Neurosci. 2003;26:329–34, doi: 10.1016/S0166-2236(03)00121-8. 9. Nicolelis MA, Lebedev MA. Principles of neural ensemble physiology underlying the operation of brain-machine interfaces. Nat Rev. 2009;10:530–40, doi: 10.1038/nrn2653. 10. Schwartz AB, Cui XT, Weber DJ, Moran DW. Brain-controlled interfaces: movement restoration with neural prosthetics. Neuron. 2006;52:205–20, doi: 10.1016/j.neuron.2006.09.019. 11. Mattia D, Cincotti F, Astolfi L, de Vico Fallani F, Scivoletto G, Marciani MG, et al. Motor cortical responsiveness to attempted movements in tetraplegia: evidence from neuroelectrical imaging. Clin Neurophysiol. 2009;120:181–9, doi: 10.1016/j.clinph.2008.09.023. 12. Chapin JK, Moxon KA, Markowitz RS, Nicolelis MA. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat Neurosci. 1999;2:664–70, doi: 10.1038/10223. 13. Carmena JM, Lebedev MA, Crist RE, O’Doherty JE, Santucci DM, Dimitrov DF, et al. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol. 2003;1:E42, doi: 10.1371/ journal.pbio.0000042.
31
323 Brain-machine interface research Mikhail AL et al.
40.
41. 42. 43. 44. 45. 46. 47. 48. 49.
50. 51. 52. 53. 54. 55.
56. 57. 58. 59.
60.
61. 62.
CLINICS 2011;66(S1):25-32
brain-machine interfaces. IEEE Trans Biomed Eng. 2005;53:1164–73, doi: 10.1109/TBME.2006.870235. Fitzsimmons NA, Drake W, Hanson TL, Lebedev MA, Nicolelis MA. Primate reaching cued by multichannel spatiotemporal cortical microstimulation. J Neurosci. 2007;27:5593–602, doi: 10.1523/JNEUROSCI. 5297-06.2007. Maravita A, Spence C, Driver J. Multisensory integration and the body schema: close to hand and within reach. Curr Biol. 2003;13:R531–9, doi: 10.1016/S0960-9822(03)00449-4. Matthews PB. Proprioceptors and their contribution to somatosensory mapping: complex messages require complex processing. Can J Physiol Pharmacol. 1988;66:430–8, doi: 10.1139/y88-073. Longo MR, Azanon E, Haggard P. More than skin deep: body representation beyond primary somatosensory cortex. Neuropsychologia. 2010;48:655–68, doi: 10.1016/j.neuropsychologia.2009.08.022. Crapse TB, Sommer MA. Corollary discharge across the animal kingdom. Nat Rev. 2008;9:587–600. Bach-y-Rita P. Tactile vision substitution: past and future. Int J Neurosci. 1983;19:29–36, doi: 10.3109/00207458309148643. Bach-y-Rita P, Collins CC, Saunders FA, White B, Scadden L. Vision substitution by tactile image projection. Nature. 1969;221:963–4, doi: 10. 1038/221963a0. Bach-y-Rita P, Kercel W. Sensory substitution and the human-machine interface. Trends Cogn Sci. 2003;7:541–6, doi: 10.1016/j.tics.2003.10.013. Histed MH, Bonin V, Reid RC. Direct activation of sparse, distributed populations of cortical neurons by electrical microstimulation. Neuron. 2009;63:508–22, doi: 10.1016/j.neuron.2009.07.016. Rolston JD, Gross RE, Potter SM. A low-cost multielectrode system for data acquisition enabling real-time closed-loop processing with rapid recovery from stimulation artifacts. Front Neuroeng. 2009;2:12, doi: 10. 3389/neuro.16.012.2009. Butovas S, Schwarz C. Spatiotemporal effects of microstimulation in rat neocortex: a parametric study using multielectrode recordings. J Neurophysiol. 2003;90:3024–39, doi: 10.1152/jn.00245.2003. Zhang F, Aravanis AM, Adamantidis A, de Lecea L, Deisseroth K. Circuit-breakers: optical technologies for probing neural signals and systems. Nat Rev. 2007;8:577–81, doi: 10.1038/nrn2192. Boyden ES, Zhang F, Bamberg E, Nagel G, Deisseroth K. Millisecondtimescale, genetically targeted optical control of neural activity. Nat Neurosci. 2005;8:1263–8, doi: 10.1038/nn1525. Zhang F, Wang LP, Boyden ES, Deisseroth K. Channelrhodopsin-2 and optical control of excitable cells. Nat Methods. 2006;3:785–92, doi: 10. 1038/nmeth936. Carson RG. Neural pathways mediating bilateral interactions between the upper limbs. Brain Res. 2005;49:641–62, doi: 10.1016/j.brainresrev. 2005.03.005. Kazennikov O, Hyland B, Corboz M, Babalian A, Rouiller EM, Wiesendanger M. Neural activity of supplementary and primary motor areas in monkeys and its relation to bimanual and unimanual movement sequences. Neuroscience. 1999;89:661–74, doi: 10.1016/S0306-4522(98)00348-0. Nakajima T, Mushiake H, Inui T, Tanji J. Decoding higher-order motor information from primate non-primary motor cortices. Technol Health Care. 2007;15:103–10. Obhi SS, Goodale MA. Bimanual interference in rapid discrete movements is task specific and occurs at multiple levels of processing. J Neurophysiol. 2005;94:1861–8, doi: 10.1152/jn.00320.2005. Rokni U, Steinberg O, Vaadia E, Sompolinsky H. Cortical representation of bimanual movements. J Neurosci. 2003;23:11577–86. Rouiller EM, Babalian A, Kazennikov O, Moret V, Yu XH, Wiesendanger M. Transcallosal connections of the distal forelimb representations of the primary and supplementary motor cortical areas in macaque monkeys. Exp Brain Res. 1994;102:227–43, doi: 10.1007/BF00227511. Kermadi I, Liu Y, Rouiller EM. Do bimanual motor actions involve the dorsal premotor (PMd), cingulate (CMA) and posterior parietal (PPC) cortices? Comparison with primary and supplementary motor cortical areas. Somatosens Mot Res. 2000;17:255–71, doi: 10.1080/08990220050117619. Pfurtscheller G, Leeb R, Keinrath C, Friedman D, Neuper C, Guger C, et al. Walking from thought. Brain Res. 2006;1071:145–52, doi: 10.1016/j. brainres.2005.11.083. Prilutsky BI, Sirota MG, Gregor RJ, Beloozerova IN. Quantification of motor cortex activity and full-body biomechanics during unconstrained locomotion. J Neurophysiol. 2005;94:2959–69, doi: 10.1152/jn.00704.2004.
63. Dietz V. Spinal cord lesion: effects of and perspectives for treatment. Neural Plasticity. 2001;8:83–90, doi: 10.1155/NP.2001.83. 64. Rossignol S, Schwab M, Schwartz M, Fehlings MG. Spinal cord injury: time to move? J Neurosci. 2007;27:11782–92, doi: 10.1523/JNEUROSCI. 3444-07.2007. 65. Scivoletto G, Di Donna V. Prediction of walking recovery after spinal cord injury. Brain Res Bull. 2009;78:43–51, doi: 10.1016/j.brainresbull. 2008.06.002. 66. Boonstra TA, van der Kooij H, Munneke M, Bloem BR. Gait disorders and balance disturbances in Parkinson’s disease: clinical update and pathophysiology. Curr Opin Neurol. 2008;21:461–71, doi: 10.1097/WCO. 0b013e328305bdaf. 67. Pearson OR, Busse ME, van Deursen RW, Wiles CM. Quantification of walking mobility in neurological disorders. QJM. 2004;97:463–75, doi: 10. 1093/qjmed/hch084. 68. Pasquina PF, Bryant PR, Huang ME, Roberts TL, Nelson VS, Flood KM. Advances in amputee care. Arch Phys Med Rehabil. 2006;87(Suppl 1): S34–43, doi: 10.1016/j.apmr.2005.11.026. 69. Anderson KD. Targeting recovery: priorities of the spinal cord-injured population. J Neurotrauma. 2004;21:1371–83, doi: 10.1089/neu.2004.21. 1371. 70. Brown-Triolo DL, Roach MJ, Nelson K, Triolo RJ. Consumer perspectives on mobility: implications for neuroprosthesis design. J Rehabil Res Develop. 2002;39:659–69. 71. Peikon ID, Fitzsimmons NA, Lebedev MA, Nicolelis MA. Threedimensional, automated, real-time video system for tracking limb motion in brain-machine interface studies. J Neurosci Methods. 2009;180:224–33, doi: 10.1016/j.jneumeth.2009.03.010. 72. Haykin S. Adaptive Filter Theory. 4th ed. Upper Saddle River, NJ: Prentice Hall; 2002. 73. Colombo G, Joerg M, Schreier R, Dietz V. Treadmill training of paraplegic patients using a robotic orthosis. J Rehabil Res Develop. 2000;37:693–700. 74. Dollar AM, Herr H. Lower Extremity Exoskeletons and Active Orthoses: Challenges and State-of-the-Art. IEEE Trans Robotics. 2008;24:144–58, doi: 10.1109/TRO.2008.915453. 75. Ferris DP, Sawicki GS, Daley MA. A Physiologist’s Perspective on Robotic Exoskeletons for Human Locomotion. Int J HR. 2007;4:507–28. 76. Barbeau H, Ladouceur M, Mirbagheri MM, Kearney RE. The effect of locomotor training combined with functional electrical stimulation in chronic spinal cord injured subjects: walking and reflex studies. Brain Res. 2002;40:274–91, doi: 10.1016/S0165-0173(02)00210-2. 77. Dobkin BH. Brain-computer interface technology as a tool to augment plasticity and outcomes for neurological rehabilitation. J Physiol. 2007;579:637–42, doi: 10.1113/jphysiol.2006.123067. 78. Peckham PH, Keith MW, Freehafer AA. Restoration of functional control by electrical stimulation in the upper extremity of the quadriplegic patient. J Bone Joint Surg. 1988;70:144–8. 79. Thrasher TA, Popovic MR. Functional electrical stimulation of walking: function, exercise and rehabilitation. Ann Readapt Med Phys. 2008;51:452–60. 80. Kantrowitz A. A Report of the Maimonides Hospital. Electronic Physiologic Aids. Brooklyn, NY; 1960. 81. Liberson WT, Holmquest HJ, Scot D, Dow M. Functional electrotherapy: stimulation of the peroneal nerve synchronized with the swing phase of the gait of hemiplegic patients. Arch Phys Med Rehabil. 1961;42:101–5. 82. Taylor PN, Burridge JH, Dunkerley AL, Wood DE, Norton JA, Singleton C, et al. Clinical use of the Odstock dropped foot stimulator: its effect on the speed and effort of walking. Arch Phys Med Rehabil. 1999;80:1577– 83, doi: 10.1016/S0003-9993(99)90333-7. 83. Waters RL, McNeal D, Perry J. Experimental correction of footdrop by electrical stimulation of the peroneal nerve. J Bone Joint Surg. 1975;57:1047–54. 84. Graupe D. An overview of the state of the art of noninvasive FES for independent ambulation by thoracic level paraplegics. Neurol Res. 2002;24:431–42, doi: 10.1179/016164102101200302. 85. Graupe D, Kohn KH. Transcutaneous functional neuromuscular stimulation of certain traumatic complete thoracic paraplegics for independent short-distance ambulation. Neurol Res. 1997;19:323–33. 86. Cheng G, Hyon SH, Morimoto J, Ude A, Hale JG, Colvin G, et al. CB: A humanoid research platform for exploring neuroscience. Adv Robotics. 2007;21:1097–14, doi: 10.1163/156855307781389356.
32
324 J. Schouenborg, M. Garwicz and N. Danielsen (Eds.) Progress in Brain Research, Vol. 194 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 3
Toward a whole-body neuroprosthetic Mikhail A. Lebedev{,{ and Miguel A. L. Nicolelis{,{,},},k,#,* {
Department of Neurobiology, Duke University, Durham, NC, USA Duke Center for Neuroengineering, Duke University, Durham, NC, USA } Department of Biomedical Engineering, Duke University, Durham, NC, USA } Department of Psychology and Neuroscience, Duke University, Durham, NC, USA k Edmond and Lily Safra International Institute of Neuroscience, Natal, Brazil Fellow, Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland {
#
Abstract: Brain–machine interfaces (BMIs) hold promise for the restoration of body mobility in patients suffering from devastating motor deficits caused by brain injury, neurological diseases, and limb loss. Considerable progress has been achieved in BMIs that enact arm movements, and initial work has been done on BMIs for lower limb and trunk control. These developments put Duke University Center for Neuroengineering in the position to develop the first BMI for whole-body control. This whole-body BMI will incorporate very large-scale brain recordings, advanced decoding algorithms, artificial sensory feedback based on electrical stimulation of somatosensory areas, virtual environment representations, and a whole-body exoskeleton. This system will be first tested in nonhuman primates and then transferred to clinical trials in humans. Keywords: brain–machine interface; brain–machine–brain interface; intracortical microstimulation; bidirectional brain–machine interface; neuroprosthetic feedback; artificial sensation; active touch; locomotion; functional electrical stimulation; bimanual; multielectrode implant; primary motor cortex; primary somatosensory cortex; exoskeleton; posture; balance.
2001; Nicolelis and Lebedev, 2009; Schwartz et al., 2006; Wessberg et al., 2000) offer a translational solution to the problem of restoring mobility to millions of people who suffer from paralysis caused by neurological injuries, neurodegenerative diseases, or limb loss (Paddock, 2009). Only limited treatment options are available to these patients, and often their condition cannot be improved or ameliorated (Dobkin and Havton,
Introduction Brain–machine interfaces (BMIs) (Andersen et al., 2004; Birbaumer and Cohen, 2007; Fetz, 2007; Lebedev and Nicolelis, 2006; Nicolelis, *Corresponding author. Tel.: þ1-919-684-4580; Fax: 919.668.0734 E-mail: nicoleli@neuro.duke.edu DOI: 10.1016/B978-0-444-53815-4.00018-2
47
325 48
2004; Fouad and Pearson, 2004). BMIs hold promise to revolutionize the treatment of paralysis, as they strive to repair the damaged neural circuitry by bypassing the site of the lesion and establishing direct neural control of artificial tools by the activity of intact brain areas, such as the primary motor cortex (M1), which in many cases remain capable of generating motor commands despite being disconnected from the body effectors (Mattia et al., 2009). BMI research has expanded rapidly during the past decade (Lebedev and Nicolelis, 2006; Nicolelis and Lebedev, 2009), generating high expectations for potential clinical applications. Proof-of-concept BMIs have been tested in rodents (Chapin et al., 1999), nonhuman primates (Carmena et al., 2003; Moritz et al., 2008; Taylor et al., 2002; Velliste et al., 2008; Wessberg et al., 2000), and human subjects (Allison et al., 2007; Birbaumer and Cohen, 2007; Hochberg et al., 2006; Kennedy and Bakay, 1998; Patil et al., 2004; Pfurtscheller and Neuper, 2006). BMI systems developed at the Duke University Center for Neuroengineering (DUCN) during the past 12 years have made it possible to control many motor functions by neuronal ensemble activity recorded with chronic implants, ranging from arm reaching and grasping movements (Carmena et al., 2003; Lebedev et al., 2005; Wessberg et al., 2000) to bipedal locomotion (Cheng et al., 2007a; Fitzsimmons et al., 2009). Moreover, recently we have demonstrated for the first time brain–machine–brain interfaces (BMBIs) that incorporate somatosensory feedback loops that transmit information from the actuator to the brain (O’Doherty et al., 2009, 2010). These developments have put us in the position to develop the first whole-body neuroprosthetic for severely paralyzed patients.
Whole-body neuroprosthetic Previous BMI studies focused predominantly on behavioral tasks in which an artificial actuator enacted upper extremity movements, such as
reaching and grasping. Except for a few studies (Fitzsimmons et al., 2009; Pfurtscheller et al., 2006; Prilutsky et al., 2005), virtually no attempts have been made to translate BMI technology to tasks enacting motor functionality of lower extremities and the trunk. Yet, deficits or the complete loss of the ability to walk presents a considerable problem for millions of patients worldwide. Such motor deficits commonly result from spinal cord injury (Dietz, 2001; Rossignol et al., 2007; Scivoletto and Di Donna, 2009) and neurological diseases (Boonstra et al., 2008; Pearson et al., 2004). Surveys of paraplegic patients showed that they prioritized walking and trunk stability among desired mobility functions (Anderson, 2004; Brown-Triolo et al., 2002). Quadriplegic patients prioritized arm and hand function (Anderson, 2004). Thus, developing neural prosthetic devices for restoration of leg mobility is as important as developing neuroprosthetics of the arm and hand, and for some categories of patients, it is one of their main priorities in terms of rehabilitation gain. In this context, a whole-body neuroprosthetic device would clearly have a major impact on the community of people suffering from many types of body paralysis. As it defines its strategic mission for the next decade, the DUCN has elected as its main priority to construct, test, and implement a clinical version of a whole-body BMI. We envision a whole-body BMI as a neuroprosthetic system in which neuronal ensemble activity sampled simultaneously from multiple cortical areas controls the actuators that generate movements of both upper and lower extremities (Winans et al., 2010). In theory, the actuators can be implemented as a whole-body exoskeleton or a set of devices for functional electrical stimulation (FES). As the main project, the DUCN has opted for the first solution, building a whole-body exoskeleton in partnership with Dr. Gordon Cheng’s group at the Technical University of Munich. In our efforts to implement the first wholebody neuroprosthetic device for clinical use, the
326 49
first step will be to test a monkey version of such a BMI. This prototype will take into account all findings and technological developments obtained by our laboratory during the past decade. We expect that this whole-body BMI will provide animals with the ability to navigate a virtual environment through the control of a realistic representation of a monkey body. All the movements of this life-like computational avatar will be enacted by the animal’s brain activity. Indeed, in this experimental paradigm, monkeys will learn to control the avatar’s body movements under BMI control and to perform a series of tasks that require both upper and lower limb coordinated movements, including reaching and grasping virtual objects, selecting objects with different texture, bimanual object manipulation, and autonomous bipedal locomotion. Once monkeys achieve a high degree of proficiency in interacting with such a whole-body BMI in a virtual environment, the same technology will be then transferred to control a real whole-body exoskeleton that the animals will wear. These latter primate experiments, in which animals learn to control an exoskeleton through a BMI, will be developed in parallel with the creation of a virtual environment for patients that will use magnetoencephalography to generate the type of brain-derived signals needed for these patients to learn how to operate a realistic avatar of their own body (see Fig. 1). Through this sequence of animal and clinical experiments, we intend to generate a prototype neural prosthetic device for whole-body control which will be then translated to clinical trials and, eventually, enable severely paralyzed people to walk again by 2016. Below, we review recent results obtained at the DUCN that support our contention that a wholebody BMI can be implemented by this deadline.
BMI components The basic components of a BMI system are exemplified by the now classical paradigm that enacts direct control of robotic arm reaching
movements based on the combined cortical activity of hundreds of cortical neurons (Carmena et al., 2003; Lebedev and Nicolelis, 2006; Lebedev et al., 2005; Nicolelis and Lebedev, 2009; Wessberg et al., 2000). In this BMI paradigm, the electrical activity of large populations of motor cortical neurons is recorded by chronically implanted multielectrode arrays and converted into control signals that drive a robotic arm. Concurrently, visual and/or somatosensory feedback signals from the robot are delivered back to the brain as either natural visual stimuli or, as in our recent demonstrations, intracortical microstimulation (ICMS) of the primary somatosensory cortex (S1) (Lebedev and Nicolelis, 2006; O’Doherty et al., 2009, 2010). The processing of neuronal ensemble activity is performed by a series of real-time BMI decoding algorithms which translate a myriad of neuronal spikes into the kinematic and dynamic parameters required to generate the robotic arm’s movements, according to the voluntary motor intentions of the subject. The BMI setup also includes the data acquisition system, the computer cluster running multiple decoding models in real time, the robot arm, the visual display, and a sensory feedback loop from the actuator to the brain.
Large-scale neuronal recordings The major prerequisite for the performance of a neuroprosthetic device to be versatile, accurate, and stable, and to allow simultaneous motor control of both lower and upper extremities, is that multiple brain areas should be implanted and large-scale neuronal activity sampled from those areas simultaneously during operation of a whole-body BMI (Nicolelis and Lebedev, 2009; Nicolelis et al., 2003). During the past two decades, advanced electrophysiological methods have allowed recording from progressively larger samples of single neurons in behaving animals (Churchland et al., 2007; Nicolelis and Ribeiro, 2002; Nicolelis et al., 1995, 2003). With the
327 50
Combined MEG and MRI
Virtual reality environment
Prism workstation
Bluegene supercomputer cluster
Fig. 1. Schematic illustration of a human whole-body neuroprosthetic based on magnetoencephalographic recordings and functional magnetic resonance imaging that enables a paralyzed human to operate a realistic avatar of their own body in a virtual environment. Brain signals are processed by a powerful computer cluster and converted into commands that drive a life-like avatar. This technology will be used to develop and test a brain-controlled whole-body navigation system and will be eventually translated into a mind-operated whole-body exoskeleton.
present generation of planar multielectrode arrays, which can be chronically implanted in the brain, we can record the extracellular electrical activity of several hundreds of cortical cells simultaneously in a behaving rhesus monkey (Carmena et al., 2003; Fitzsimmons et al., 2009; Lebedev et al., 2005; Nicolelis et al., 2003). Due to a rapid progress in the methods for large-scale recordings (Nicolelis et al., 2003), the number of simultaneously sampled
neural channels is expected to rise to several thousands in the next decade. Indeed, recently, we have started to employ a new type of device, called the three-dimensional (3D) recording cube, which provides recording sites throughout the vertical shaft of each microwire bundle. Preliminary results suggest that such a 3D recording cube could allow neuronal yield to reach several tens of thousands of neurons in the near future. Given that
328 51
our previous studies have clearly demonstrated that the accuracy of extraction of motor intentions from the brain improves with the number of neurons recorded (Carmena et al., 2003; Lebedev et al., 2007; Nicolelis and Lebedev, 2009; Santucci et al., 2005; Wessberg et al., 2000), and given that the number of required neurons increases as more motor parameters are simultaneously extracted (Fitzsimmons et al., 2009), the advent of the 3D recording cube, and other technologies like it, could prove vital for the translation of experimental discoveries into clinical practice. Incidentally, the central experimental finding obtained through BMI research, the confirmation that distributed ensembles of neurons define the true physiological unit of the mammalian central nervous system (Nicolelis and Lebedev, 2009), will also play a major role in the road that will take us to clinical fruition. A brief description may help clarify the imposing need for increasing the size of cortical recording samples in order to obtain the level of motor control needed to operate a whole-body exoskeleton. Currently, our most sophisticated BMIs provide a peak signal-to-noise ratio (SNR) of 10 dB using approximately 100 neurons (Fitzsimmons et al., 2009; Li et al., 2009). This corresponds to 5 bits/s bandwidth. In our estimation, SNRs of about 20â&#x20AC;&#x201C;30 dB are needed for a neuroprosthetic device to generate useful movements (see Fig. 2). Assuming that noise decreases as the inverse of the square root of the number of neurons, a tenfold increase in the number of simultaneously recorded neurons is needed to achieve an improvement of 10 dB. A control signal with 20-dB fidelity will then require 1000 neurons recorded simultaneously. For achieving our long-term goal of reaching an SNR of 40 dB, and control a whole-body exoskeleton, recording samples between 50,000â&#x20AC;&#x201C;100,000 cortical neurons will be needed. Further, to control the movements of multiple prosthetic limbs, each of which requires enacting of many degrees-of-freedom (DOF), a whole-body neuroprosthetic will call for even higher neuronal samples. Thus, demands for large samples of neurons, located in multiple
cortical areas, will remain a central bottleneck for the development of clinical neuroprosthetics for the foreseeable future. Another key issue to be dealt with is the question of the longevity of chronic multielectrode recordings. Heretofore, our state-of-the-art multielectrode implants reliably sample large-scale extracellular cortical electrical activity for at least 2 years in rhesus monkeys (Fitzsimmons et al., 2009) and at least 6.5 years in New World monkeys, like the owl monkey (Sandler et al., 2005). We have used a variety of designs for such multielectrode probes. Typically, electrode penetration sites are spaced with a 1-mm separation. The electrodes can be fixed or movable. In our recent design, we arranged stainless steel microwires into subsets (pairs or triplets) sitting inside guiding tubes (Winans et al., 2010). Each subset can be moved independently driven by a microscrew. Given that the electrodes in each subset have different lengths, the array samples from a 3D volume; that is, how we started to test what became known as the 3D recording cube. Currently, we are using this new cube design as a base for the development of the next-generation implants that will increase the number of recording channels to about 1500 per cube. That is because a 10 10 array will gain up to 15 recording contacts per vertical component, using microwire assemblies of different lengths. Further, to minimize the displacement of the nervous tissue and to minimize microglial and other immune responses to the foreign material during the surgical implantation of these recording cubes, we will be removing the structural elements after implantation, leaving only small diameter microwires within the neuronal tissue. Small-diameter microwires will be guided into the brain with a strong central shaft which will be removed later. To distinguish the type of cortical activity obtained with this new technology, we have coined the term very large scale brain activity (VLSBA), meaning extracellular neuronal activity generated by populations of more than 50,000 single neurons recorded simultaneously. By developing the means
329 52 Limb kinematics (current state of the art)
8
SNR (dB)
N
5–10 SNR
> 100
Current state of the art
20 SNR
> 1,000
Poor motor performance
30 SNR
> 10,000
40 SNR
> 100,000
50 SNR
> 1,000,000
SNR (dB)
6 4
Acceptable range
2 0 1
100 200 10 Neurons used to predict, log10(N)
Professional NBA player’s performance on free throws
SNR ≅ C1 + C2 . log10 (N) (C2 = 10 for uncorrelated neurons, C2 = 3.7 for our data)
Fig. 2. An estimation of the dependency of the BMI signal-to-noise ratio (SNR) on the neuronal ensemble size. The scatter plot on the left shows a linear dependency of the SNR (in dB) on the logarithm of the number of neurons recorded. The data from Fitzsimmons et al. (2009) was used to construct this plot. This dependency is in correspondence with the equation shown at the bottom. The table on the right shows the estimates of the number of neurons needed to achieve certain performance benchmarks.
to record from VLSBA, we expect to generate the type of motor control signals that will be capable of driving a whole-body BMI.
BMI decoders The success of any BMI system depends to a significant degree on the choice of BMI decoders that extract motor parameters from the sample of neuronal electrical activity recorded in real time. In our current studies, online processing of large-scale brain activity is achieved through an integrated BMI suite that incorporates the recording and stimulation hardware, as well as a computer cluster employed for all real-time processing of the massive stream of neurophysiological data generated in each of these experimental sections (Fitzsimmons et al., 2009; O’Doherty et al., 2009). This BMI suite can simultaneously run several neuronal decoders, including the unscented Kalman filter (Li et al., 2009), the Wiener filter, multiple artificial neural networks, and discrete state Bayesian algorithms (Li et al.,
2010). Each motor parameter can be extracted by one of these filters or can represent a mixture of the outputs of individual decoders. During the extraction or arm reaching movements, the unscented Kalman filter outperforms the other decoding algorithms (Li et al., 2010). This decoder is based on nonlinear models of neural tuning and prior knowledge about movement patterns. It is also enhanced by a short history of the arm movements. We have also developed a multiple model-switching paradigm (Fitzsimmons et al., 2009) that uses different submodels to extract motor variables in particular behavioral states (e.g., reaching vs. grasping or walking forward vs. walking backward). The simplest switching model consists of three linear decoding models: a model for predicting state 1 (e.g., walking forward), a model for state 2 (e.g., walking backward), and the paradigm predictor model (the switch). When the paradigm predictor determines a particular state, an appropriate submodel is used to produce the output, and when a different state is detected, the other submodel is used.
330 53
In the most straightforward implementation of BMI decoding, the kinematics of several limb joints is extracted from the combined neuronal electrical activity and converted into movements of the particular mechanical, electronic or virtual actuator employed in the experiment at hand. This general approach, however, is not optimal for clinical applications of BMIs because the user is required to control each DOF continuously and independently. In a more practical solution, the control over an external actuator is shared between the subject brainâ&#x20AC;&#x2122;s activity and robotic controls (Kim et al., 2005). During this shared control, the subjectâ&#x20AC;&#x2122;s brain is in charge primarily of high-order control of movements (when to initiate movement and where to move), whereas the low-level coordination of the movement is performed by an autonomous controller. Recently, we have developed a BMI decoder that autonomously adjusts its performance during long-term recordings and compensates for nonstationarities in neuronal inputs (Li et al., 2010). This decoder uses a Bayesian regression selftraining method for updating the parameters for an unscented Kalman filter. To allow updates on subsets of neurons and to allow addition of newly discovered neurons, we approximated the probability distribution on the tuning model parameters using a factorized distribution and computed the Bayesian regression solution. We tested the performance of this filter in rhesus monkeys that learned to perform reaching movements under the BMI control. Over the course of 29 days, Bayesian regression self-training maintained control accuracy better than decoding without updates. This BMI decoder will be used in the whole-body BMI to provide long-term stability.
Bimanual control A whole-body neuroprosthetic will have to enable independent control of two prosthetic arms. Such BMI control has not been achieved before. The majority of BMIs developed so far
involved only a single actuator (computer cursor or a robotic arm) that enacted arm reaching movements under the control of the subjectâ&#x20AC;&#x2122;s brain activity. We are currently exploring the use of BMIs for bimanual actuator control. As in other applications, the starting point in developing a bimanual BMI is to understand the neurophysiological processes that underlie bimanual behaviors in multiple cortical areas. Bimanual operations engage neuronal populations which are different from those engaged by unimanual movements (Kazennikov et al., 1999; Nakajima et al., 2007; Obhi and Goodale, 2005; Rokni et al., 2003; Rouiller et al., 1994). When both arms are involved, motor areas are activated in both hemispheres, and greater interhemispheric interactions occur. Five cortical areas involved in interlimb coordination are of particular interest for bimanual BMIs: dorsal premotor cortex (PMd), cingulate motor area, supplementary motor area, posterior parietal cortex, and M1 (Kermadi et al., 2000). We have conducted preliminary experiments to investigate if rhesus monkeys could learn bimanual control of a life-like avatar. In these experiments, one monkey learned to manipulate two joysticks to move two avatar hands toward virtual objects. This learning occurred rapidly, suggesting that the monkey readily associated itself with the avatar. This observation strengthens our expectation that monkeys will be able to learn to control such bimanual movements through a BMI, without engaging any overt movements of their limbs.
Bipedal locomotion Our laboratory pioneered BMIs that reproduce kinematics of leg movements during bipedal locomotion (Fitzsimmons et al., 2009). Previously, both BMI research and neurophysiological studies in awake, behaving monkeys focused predominantly on the behavioral tasks that involved arm movements and arm representation in the brain.
331 54
Neurophysiology of lower extremity control has been virtually neglected in nonhuman primates. Yet, a complete loss of the ability to walk is commonly caused by spinal cord injury (Dietz, 2001; Rossignol et al., 2007; Scivoletto and Di Donna, 2009), neurological diseases (Boonstra et al., 2008; Pearson et al., 2004), and limb loss (Pasquina et al., 2006). Developing a neuroprosthetic for lower limb control will clearly be very important for the treatment of whole-body paralysis. In our initial study on the BMI for bipedal locomotion, rhesus monkeys walked bipedally on a treadmill (Fitzsimmons et al., 2009). We tracked leg movements using a video-based tracking system developed in our laboratory (Peikon et al., 2009). Leg kinematics and EMGs of leg muscles were extracted from linearly combined cortical ensemble activity in real time using a modified Wiener filter algorithm (Carmena et al., 2003; Haykin, 2002; Lebedev et al., 2005; Wessberg et al., 2000). Recording implants were placed in the leg representation area of M1. We found that the individual neuronal firing rates were highly variable from step to step. However, after the activity of many (several hundreds) neurons was combined using 100-ms bins, accurate extractions of the kinematics of leg movements (X,Y coordinates of the joints, joint angles) were produced. Walking parameters could be extracted using neuronal activity recorded in either M1 or S1 contralateral to the right leg, as well as ipsilateral M1. Simultaneous extraction of many parameters of bipedal locomotion essentially depended on the size of the neuronal population. In this analysis, random neuronal subpopulations were pooled from the entire recorded population and used to predict several locomotion parameters simultaneously. The accuracy of simultaneous predictions was characterized by the normalized accuracy for the least well-predicted parameter. We found that smaller neuronal populations could predict only a few parameters simultaneously, and larger populations were required for predicting many parameters. Moreover, larger
neuronal populations were required to predict complex patterns of walking compared to more simple walking patterns. For example, the number of neurons required to achieve 95% of maximum prediction accuracy for the X position of the ankle clearly increased when walking conditions of higher complexity were required to complete a task (intermittent walking forward and backward, walking at different speeds). On average, 60 neurons were sufficient for predicting constant-speed walking in the forward direction. However, 90 neurons were needed to achieve this level of accuracy for variable-speed, forward walking, 95 neurons were required for extracting backward walking at constant speed, 115 neurons were required for predicting backward walking at variable speed, and extracting variable-speed bidirectional walking required 110 neurons. Using our BMI for extracting patterns of bipedal locomotion, we demonstrated BMI control of bipedal walking in a robot (Cheng et al., 2007a) in real time. To achieve this goal, realtime predictions of monkey leg kinematics, derived from combined cortical activity, were transmitted through a dedicated Internet connection to ATR laboratories, in Kyoto, Japan, where our collaborators led by Cheng and Kawato set their humanoid 51-DOF robot (CB-1, manufactured by SARCOS) to reproduce monkey locomotion patterns. Back at our laboratory at Duke University, the monkey received continuous visual feedback of the robotâ&#x20AC;&#x2122;s movements on a video screen placed in front of the animal. As we had observed in the case of BMI control of upper-limb extremities, after the treadmill was stopped, the monkey continued to use its brain activity to sustain bipedal walking in the robot without moving its own biological legs. This result supports our suggestion that a paralyzed person will be able to control a device for locomotion (e.g., an exoskeleton) with his/her cortical activity alone, without the need to produce overt body movements.
332 55
Posture and balance Whole-body neuroprosthetics will not only have to produce stereotypical stepping but also adapt to postural control. As an advancement toward this goal, we have developed a proof-of-concept BMI for postural control (Tate et al., 2009). In these experiments, monkeys first learned to maintain an upright posture on a platform. Then, the platform moved abruptly, generating a postural perturbation. The platform was driven either periodically, allowing the animal to anticipate the upcoming perturbation, or with a random time delay between movements so that the animal could not anticipate when the next displacement would occur. By analyzing samples of simultaneously recorded cortical neurons, we observed that cells recorded in the leg representation of M1 and S1 clearly modulated their firing rate in response to platform displacement. These modulations were directionally tuned. Linear BMI decoders were applied to extract the kinematics of platform movements from cortical ensemble activity. We found that the decoder performance was different, depending on whether the platform movements were anticipated or unanticipated. Anticipated displacements were extracted with higher accuracy than unanticipated. These results suggest that cortical control over posture and balance can be added to a whole-body BMI. Optimal operation of such control probably can be aided by a shared control scheme (Kim et al., 2005).
Functional electrical stimulation FES that activates the subjectâ&#x20AC;&#x2122;s own muscles may be implemented in future whole-body neuroprosthetics. FES devices have been already introduced to clinical practice as therapies for leg paralysis (Barbeau et al., 2002; Dobkin, 2007; Peckham et al., 1988; Thrasher and Popovic, 2008) along with robotic orthoses (Colombo et al., 2000; Dollar and Herr, 2008; Ferris et al., 2007). The first publications on a FES device for
helping to achieve an upright posture date back to the 1960s (Kantrowitz, 1960). Also in the 1960s, FES of the common peroneal nerve was introduced to correct foot drop during the swing phase of the gate (Liberson et al., 1961). At present, a variety of FES methods are available for stimulation of multiple muscle groups using superficial or intramuscular electrodes. In particular, FES systems have been shown to have positive therapeutic effects in patients with incomplete spinal cord injury (Thrasher and Popovic, 2008). We suggest that FES technology can be incorporated in BMIs for the restoration of walking and balance. In support of this suggestion, we have already obtained reliable predictions of the EMGs of leg muscles during locomotion (Fitzsimmons et al., 2009) from the combined electrical activity of populations of M1 and S1 cortical neurons. Our next objective is to build a BMI-driven FES system that produces bipedal locomotion patterns by converting cortical ensemble activity into stimulation patterns that drive leg muscles. This approach can be used alone or in combination with robotic orthoses, such as exoskeletons.
Sensorized neuroprosthetic Recently, we have reported our findings on the first BMBI. Such a paradigm expands on traditional BMIs by adding an artificial somatosensory feedback channel that can deliver artificially created tactile signals, generated by either real sensors placed in a robotic hand or virtual ones added to an avatar arm, directly to the somatosensory cortex, via ICMS. In our first study on this subject (Fitzsimmons et al., 2007), we investigated whether multichannel ICMS of S1 could be discriminated by owl monkeys and whether ICMS is suitable for long-term usage. Owl monkeys were implanted with multielectrode arrays in several cortical areas, whereas S1 implants were employed to deliver spatiotemporal patterns of ICMS. The behavioral task progressed from a simple requirement of detecting
333 56
the presence of ICMS to the goal of discriminating spatiotemporal patterns created using four electrode pairs. We found that owl monkeys could learn to discriminate spatiotemporal patterns of ICMS of increasing complexity and guide their arm reaching movements based on this discrimination. Moreover, spatiotemporal ICMS was efficient for many months. Interestingly, monkeys got progressively better in learning novel microstimulation patterns. This result suggests that ICMS was incorporated in their brains as a new sensory channel and that they could generalize the general rule applied to communicate directly with their brains. In a series of studies conducted in rhesus monkeys, we used ICMS to guide monkeys’ behavior when they performed BMI-reaching tasks (O’Doherty et al., 2009, 2010). We followed our usual paradigm of training a BMI decoder during a manually performed task and then switched to brain control (Carmena et al., 2003; Lebedev et al., 2005). The tasks consisted of acquiring visual targets with a computer cursor or a life-like avatar hand. We recorded simultaneously the electrical activity of 50–200 neurons in M1 and PMd. The monkeys learned to perform in BMI control with and without using the joystick. The innovative feature of these experiments was that ICMS of the S1 cortex was added to the BMI as an artificial sensory feedback. In the first study that implemented this approach (O’Doherty et al., 2009), ICMS served as a directional cue. It informed the monkeys to which direction they had to move. To avoid an interference of ICMS electrical artifacts with BMI extractions, we segregated the stimulation and recording epochs. Monkeys operated such a BMBI with approximately the same accuracy as reported previously for simpler BMI designs. Moreover, in our recent study (O’Doherty et al., 2010), ICMS served as an artificial sense of active touch, as it conveyed to the monkeys the properties of virtual objects that the actuator (computer cursor or a virtual image of a monkey arm) touched. Monkeys controlled an avatar arm with a BMBI that derived motor commands
from the M1 activity. ICMS patterns were delivered to S1 each time the avatar touched virtual objects. Monkeys learned to search through sets of visually identical objects and select those with particular textures. These results suggest that future clinical neuroprosthetics can implement ICMS feedback to generate somatic perceptions from prosthetic limbs. In our future work, we will incorporate multiple channel ICMS feedback in the BMBIs for locomotion and whole-body control. We will also work on the incorporation of an artificial sense of position. Position sense is very important for clinical applications because ideal prosthetic limbs should feel as if they are natural extensions of the users’ bodies. Normally, positional signals are provided by muscles, joints, and skin afferents. This information ascends to the sensory areas of the brain where it is processed using different coordinate frames, such as body- and external space-centered coordinates (Maravita et al., 2003; Matthews, 1988). Given this complexity of cortical processing of proprioceptive information (Longo et al., 2010), it would be difficult for an artificial position sense to achieve such precise mapping from the arm joints to the brain somatosensory map. Additionally, certain centrally generated components of normal position sense, such as corollary discharges (Crapse and Sommer, 2008), would be difficult to incorporate in such an implementation. Because of these foreseen difficulties in the straightforward implementation of an artificial position sense, we chose a simpler approach in which stimulation of S1 is not initially coupled to the orientation of the limb position but instead represents 3D spatial locations to which the subject is required to reach. In this approach, the subject starts with learning how ICMS of cortical somatotopic representations of the body is mapped to a 3D space. Such experimental design bears similarity to the studies on sensory substitution in which visual information was conveyed by the stimulation of skin surfaces (Bach-y-Rita, 1983; Bach-y-Rita and Kercel, 2003; Bach-y-Rita et al., 1969). We expect that similar to sensory
334 57
substitution using peripheral stimulation, training with stimulation of the somatosensory areas of the brain will eventually give rise to an artificial position sense that represents external space. In addition to ICMS as the method to sensorize a whole-body BMI, optogenetic methods will be probably, extensively used in the future. Optogenetics is based on genetically modified ion channels that respond directly to light (Zhang et al., 2007). These light-gated ion channels, such as Channelrhodopsin-2 (Chr-2), allow precise, millisecond control of specific neurons (Boyden et al., 2005; Zhang et al., 2006). This technique eliminates most of the key problems associated with ICMS: there is no associated electrical artifact to interfere with the electrophysiological recordings, nor any tissue damage from the current injection. It also allows for finer control of the spatial pattern of activation. BMBIs that incorporate optogenetic stimulators would be superior to current designs in both the specificity and the long-term performance of the sensory loop and the quality of neuronal recordings.
A whole-body exoskeleton In our research program, the definitive demonstration of a whole-body neuroprosthetic will involve a subject that is able to use his/her VLSBA to control movements of an exoskeleton that encases the entire body. Currently, we have all components needed for this demonstration. We have demonstrated BMIs for arm reaching and leg locomotion in separate experiments. We have also implanted leg and arm representations of the sensorimotor cortex in both hemispheres (Winans et al., 2010). The exoskeleton technology required for this demonstration already exists, and a prototype for our initial testing in monkeys is close to be completed. In our primate studies, such exoskeleton will be attached to the monkey using bracelets molded in the shape of the monkey’s limbs. The basic design and controller will be based on the humanoid robot, CB-1 (Cheng et al., 2007a,b). This exoskeleton will provide a rich sensory feedback stream to
the BMI setup, including measurements of joints position and velocity, as well as torque, ground contacts, and orientations. In the BMI mode, the exoskeleton will guide the monkey’s limbs with smooth motions, while monitoring its range of motions to ensure it is within the safety limits. This upcoming demonstration will provide the first prototype of a neural prosthetic device that would allow, one day in the near future, paralyzed people to recover a wide range of desired mobility functions. References Allison, B. Z., Wolpaw, E. W., & Wolpaw, J. R. (2007). Braincomputer interface systems: Progress and prospects. Expert Review of Medical Devices, 4, 463–474. Andersen, R. A., Musallam, S., & Pesaran, B. (2004). Selecting the signals for a brain-machine interface. Current Opinion in Neurobiology, 14, 720–726. Anderson, K. D. (2004). Targeting recovery: Priorities of the spinal cord-injured population. Journal of Neurotrauma, 21, 1371–1383. Bach-y-Rita, P. (1983). Tactile vision substitution: Past and future. The International Journal of Neuroscience, 19, 29–36. Bach-y-Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221, 963–964. Bach-y-Rita, P., & Kercel, W. (2003). Sensory substitution and the human-machine interface. Trends in Cognitive Sciences, 7, 541–546. Barbeau, H., Ladouceur, M., Mirbagheri, M. M., & Kearney, R. E. (2002). The effect of locomotor training combined with functional electrical stimulation in chronic spinal cord injured subjects: Walking and reflex studies. Brain Research. Brain Research Reviews, 40, 274–291. Birbaumer, N., & Cohen, L. G. (2007). Brain-computer interfaces: Communication and restoration of movement in paralysis. The Journal of Physiology, 579, 621–636. Boonstra, T. A., van der Kooij, H., Munneke, M., & Bloem, B. R. (2008). Gait disorders and balance disturbances in Parkinson’s disease: Clinical update and pathophysiology. Current Opinion in Neurology, 21, 461–471. Boyden, E. S., Zhang, F., Bamberg, E., Nagel, G., & Deisseroth, K. (2005). Millisecond-timescale, genetically targeted optical control of neural activity. Nature Neuroscience, 8, 1263–1268. Brown-Triolo, D. L., Roach, M. J., Nelson, K., & Triolo, R. J. (2002). Consumer perspectives on mobility: Implications for neuroprosthesis design. Journal of Rehabilitation Research and Development, 39, 659–669.
335 58 Carmena, J. M., Lebedev, M. A., Crist, R. E., O’Doherty, J. E., Santucci, D. M., Dimitrov, D. F., et al. (2003). Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biology, 1, E42. Chapin, J. K., Moxon, K. A., Markowitz, R. S., & Nicolelis, M. A. L. (1999). Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neuroscience, 2, 664–670. Cheng, G., Fitzsimmons, N. A., Morimoto, J., Lebedev, M. A., Kawato, M., & Nicolelis, M. A. L. (2007). Bipedal locomotion with a humanoid robot controlled by cortical ensemble activity. Annual Meeting of the Society for Neuroscience. San Diego, California. Cheng, G., Hyon, S. H., Morimoto, J., Ude, A., Hale, J. G., Colvin, G., et al. (2007). CB: A humanoid research platform for exploring neuroscience. Advanced Robotics, 21, 1097–1114. Churchland, M. M., Yu, B. M., Sahani, M., & Shenoy, K. V. (2007). Techniques for extracting single-trial activity patterns from large-scale neural recordings. Current Opinion in Neurobiology, 17, 609–618. Colombo, G., Joerg, M., Schreier, R., & Dietz, V. (2000). Treadmill training of paraplegic patients using a robotic orthosis. Journal of Rehabilitation Research and Development, 37, 693–700. Crapse, T. B., & Sommer, M. A. (2008). Corollary discharge across the animal kingdom. Nature Reviews. Neuroscience, 9, 587–600. Dietz, V. (2001). Spinal cord lesion: Effects of and perspectives for treatment. Neural Plasticity, 8, 83–90. Dobkin, B. H. (2007). Brain-computer interface technology as a tool to augment plasticity and outcomes for neurological rehabilitation. The Journal of Physiology, 579, 637–642. Dobkin, B. H., & Havton, L. A. (2004). Basic advances and new avenues in therapy of spinal cord injury. Annual Review of Medicine, 55, 255–282. Dollar, A. M., & Herr, H. (2008). Lower extremity exoskeletons and active orthoses: Challenges and state-ofthe-art. IEEE Transactions on Robotics, 24, 144–158. Ferris, D. P., Sawicki, G. S., & Daley, M. A. (2007). A physiologist’s perspective on robotic exoskeletons for human locomotion. International Journal of Humanoid Robotics, 4, 507–528. Fetz, E. E. (2007). Volitional control of neural activity: Implications for brain-computer interfaces. The Journal of Physiology, 579, 571–579. Fitzsimmons, N. A., Drake, W., Hanson, T. L., Lebedev, M. A., & Nicolelis, M. A. L. (2007). Primate reaching cued by multichannel spatiotemporal cortical microstimulation. The Journal of Neuroscience, 27, 5593–5602. Fitzsimmons, N. A., Lebedev, M. A., Peikon, I. D., & Nicolelis, M. A. L. (2009). Extracting kinematic parameters for monkey bipedal walking from cortical neuronal ensemble activity. Frontiers in Integrative Neuroscience, 3, 3.
Fouad, K., & Pearson, K. (2004). Restoring walking after spinal cord injury. Progress in Neurobiology, 73, 107–126. Haykin, S. (2002). Adaptive filter theory (4th ed.). Upper Saddle River, NJ: Prentice Hall. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., et al. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442, 164–171. Kantrowitz, A. (1960). Electronic physiological aids: A report of the Maimonides Hospital, Brooklyn, New York, 75–77. Kazennikov, O., Hyland, B., Corboz, M., Babalian, A., Rouiller, E. M., & Wiesendanger, M. (1999). Neural activity of supplementary and primary motor areas in monkeys and its relation to bimanual and unimanual movement sequences. Neuroscience, 89, 661–674. Kennedy, P. R., & Bakay, R. A. (1998). Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport, 9, 1707–1711. Kermadi, I., Liu, Y., & Rouiller, E. M. (2000). Do bimanual motor actions involve the dorsal premotor (PMd), cingulate (CMA) and posterior parietal (PPC) cortices? Comparison with primary and supplementary motor cortical areas. Somatosensory and Motor Research, 17, 255–271. Kim, H. K., Biggs, S. J., Schloerb, D. W., Carmena, J. M., Lebedev, M. A., Nicolelis, M. A. L., et al. (2005). Continuous shared control stabilizes reach and grasping with brainmachine interfaces. IEEE Transactions on Biomedical Engineering, 53, 1164–1173. Lebedev, M. A., Carmena, J. M., O’Doherty, J. E., Zacksenhouse, M., Henriquez, C. S., Principe, J. C., et al. (2005). Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface. The Journal of Neuroscience, 25, 4681–4693. Lebedev, M. A., Fitzsimmons, N. A., Drake, W., Lehew, G., Dimitrov, D. F., & Nicolelis, M. A. L. (2007). Decoding bipedal locomotion patterns from cortical ensemble activity in rhesus monkeys. In Proceedings of the Society for Neuroscience Annual Meeting, 33, 517–523, San Diego, CA. Lebedev, M. A., & Nicolelis, M. A. L. (2006). Brain-machine interfaces: Past, present and future. Trends in Neurosciences, 29, 536–546. Li, Z., O’Doherty, J. E., Lebedev, M. A., Nicolelis, M. A. L. (2010) Closed-loop adaptive decoding using bayesian regression self-training. Proceedings of the Society for Neuroscience Annual Meeting, vol. 383.101, San Diego, CA, USA. Li, Z., O’Doherty, J. E., Hanson, T. L., Lebedev, M. A., Henriquez, C. S., & Nicolelis, M. A. L. (2009). Unscented Kalman filter for brain-machine interfaces. PLoS One, 4, e6243. Liberson, W. T., Holmquest, H. J., Scot, D., & Dow, M. (1961). Functional electrotherapy: Stimulation of the peroneal nerve synchronized with the swing phase of the gait
336 59 of hemiplegic patients. Archives of Physical Medicine and Rehabilitation, 42, 101–105. Longo, M. R., Azanon, E., & Haggard, P. (2010). More than skin deep: Body representation beyond primary somatosensory cortex. Neuropsychologia, 48, 655–668. Maravita, A., Spence, C., & Driver, J. (2003). Multisensory integration and the body schema: Close to hand and within reach. Current Biology, 13, R531–R539. Matthews, P. B. (1988). Proprioceptors and their contribution to somatosensory mapping: Complex messages require complex processing. Canadian Journal of Physiology and Pharmacology, 66, 430–438. Mattia, D., Cincotti, F., Astolfi, L., de Vico Fallani, F., Scivoletto, G., Marciani, M. G., et al. (2009). Motor cortical responsiveness to attempted movements in tetraplegia: Evidence from neuroelectrical imaging. Clinical Neurophysiology, 120, 181–189. Moritz, C. T., Perlmutter, S. I., & Fetz, E. E. (2008). Direct control of paralysed muscles by cortical neurons. Nature, 456, 639–642. Nakajima, T., Mushiake, H., Inui, T., & Tanji, J. (2007). Decoding higher-order motor information from primate non-primary motor cortices. Technology and Health Care, 15, 103–110. Nicolelis, M. A. L. (2001). Actions from thoughts. Nature, 409, 403–407. Nicolelis, M. A. L., Chapin, J. K., & Lin, R. C. (1995). Development of direct GABAergic projections from the zona incerta to the somatosensory cortex of the rat. Neuroscience, 65, 609–631. Nicolelis, M. A. L., Dimitrov, D., Carmena, J. M., Crist, R., Lehew, G., Kralik, J. D., et al. (2003). Chronic, multisite, multielectrode recordings in macaque monkeys. Proceedings of the National Academy of Sciences of the United States of America, 100, 11041–11046. Nicolelis, M. A. L., & Lebedev, M. A. (2009). Principles of neural ensemble physiology underlying the operation of brain-machine interfaces. Nature Reviews. Neuroscience, 10, 530–540. Nicolelis, M. A. L., & Ribeiro, S. (2002). Multielectrode recordings: The next steps. Current Opinion in Neurobiology, 12, 602–606. Obhi, S. S., & Goodale, M. A. (2005). Bimanual interference in rapid discrete movements is task specific and occurs at multiple levels of processing. Journal of Neurophysiology, 94, 1861–1868. O’Doherty, J. E., Ifft, P. J., Zhuang, K. Z., Lebedev, M. A., & Nicolelis, M. A. L. (2010). Brain-machine-brain interface using simultaneous recording and intracortical microstimulation feedback. In Proceedings of the Society for Neuroscience Annual Meeting. vol. 899.15, San Diego, CA, USA.
O’Doherty, J. E., Lebedev, M. A., Hanson, T. L., Fitzsimmons, N. A., & Nicolelis, M. A. L. (2009). A brainmachine interface instructed by direct intracortical microstimulation. Frontiers in Integrative Neuroscience, 3, 1–10. Paddock, C. (2009). Paralysis affects more Americans than previously thought. In Medical news today. http://www. medicalnewstoday.com/articles/146819.php. Pasquina, P. F., Bryant, P. R., Huang, M. E., Roberts, T. L., Nelson, V. S., & Flood, K. M. (2006). Advances in amputee care. Archives of Physical Medicine and Rehabilitation, 87, S34–S43 quiz S44–S35. Patil, P. G., Carmena, J. M., Nicolelis, M. A. L., & Turner, D. A. (2004). Ensemble recordings of human subcortical neurons as a source of motor control signals for a brain-machine interface. Neurosurgery, 55, 27–35, discussion 35-28. Pearson, O. R., Busse, M. E., van Deursen, R. W., & Wiles, C. M. (2004). Quantification of walking mobility in neurological disorders. QJM, 97, 463–475. Peckham, P. H., Keith, M. W., & Freehafer, A. A. (1988). Restoration of functional control by electrical stimulation in the upper extremity of the quadriplegic patient. The Journal of Bone and Joint Surgery. American Volume, 70, 144–148. Peikon, I. D., Fitzsimmons, N. A., Lebedev, M. A., & Nicolelis, M. A. L. (2009). Three-dimensional, automated, real-time video system for tracking limb motion in brainmachine interface studies. Journal of Neuroscience Methods, 180, 224–233. Pfurtscheller, G., Leeb, R., Keinrath, C., Friedman, D., Neuper, C., Guger, C., et al. (2006). Walking from thought. Brain Research, 1071, 145–152. Pfurtscheller, G., & Neuper, C. (2006). Future prospects of ERD/ERS in the context of brain-computer interface (BCI) developments. Progress in Brain Research, 159, 433–437. Prilutsky, B. I., Sirota, M. G., Gregor, R. J., & Beloozerova, I. N. (2005). Quantification of motor cortex activity and full-body biomechanics during unconstrained locomotion. Journal of Neurophysiology, 94, 2959–2969. Rokni, U., Steinberg, O., Vaadia, E., & Sompolinsky, H. (2003). Cortical representation of bimanual movements. The Journal of Neuroscience, 23, 11577–11586. Rossignol, S., Schwab, M., Schwartz, M., & Fehlings, M. G. (2007). Spinal cord injury: Time to move? The Journal of Neuroscience, 27, 11782–11792. Rouiller, E. M., Babalian, A., Kazennikov, O., Moret, V., Yu, X. H., & Wiesendanger, M. (1994). Transcallosal connections of the distal forelimb representations of the primary and supplementary motor cortical areas in macaque monkeys. Experimental Brain Research, 102, 227–243.
337 60 Sandler, A. J., Kralik, J. D., Dewey, K. S., & Nicolelis, M. A. L. (2005). Long-term neuronal recordings from nonhuman primates. In Society for Neuroscience Annual Meeting 31, 402.8, Washington, DC. Santucci, D. M., Kralik, J. D., Lebedev, M. A., & Nicolelis, M. A. L. (2005). Frontal and parietal cortical ensembles predict single-trial muscle activity during reaching movements in primates. The European Journal of Neuroscience, 22, 1529–1540. Schwartz, A. B., Cui, X. T., Weber, D. J., & Moran, D. W. (2006). Brain-controlled interfaces: Movement restoration with neural prosthetics. Neuron, 52, 205–220. Scivoletto, G., & Di Donna, V. (2009). Prediction of walking recovery after spinal cord injury. Brain Research Bulletin, 78, 43–51. Tate, A. J., Lebedev, M. A., & Nicolelis, M. A. L. (2009). Neural activity associated with changes in posture in rhesus macaques. In Proceedings of the Society for Neuroscience Annual Meeting. vol. 181.7, Chicago, IL, USA. Taylor, D. M., Tillery, S. I., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science, 296, 1829–1832.
Thrasher, T. A., & Popovic, M. R. (2008). Functional electrical stimulation of walking: Function, exercise and rehabilitation. Annales de Réadaptation et de Médecine Physique, 51, 452–460. Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature, 453, 1098–1101. Wessberg, J., Stambaugh, C. R., Kralik, J. D., Beck, P. D., Laubach, M., Chapin, J. K., et al. (2000). Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature, 408, 361–365. Winans, J. A., Tate, A. J., Lebedev, M. A., & Nicolelis, M. A. L. (2010). Extraction of leg kinematics from the sensorimotor cortex representation of the whole body. In Proceedings of the Society for Neuroscience Annual Meeting. vol. 294.7. San Diego, CA, USA. Zhang, F., Aravanis, A. M., Adamantidis, A., de Lecea, L., & Deisseroth, K. (2007). Circuit-breakers: Optical technologies for probing neural signals and systems. Nature Reviews. Neuroscience, 8, 577–581. Zhang, F., Wang, L. P., Boyden, E. S., & Deisseroth, K. (2006). Channelrhodopsin-2 and optical control of excitable cells. Nature Methods, 3, 785–792.
WHO WE ARE
W H AT W E CA N D O
W H E R E W E’ R E H E A D E D
B
LIME Y O ITS N D
338
neuroengineering
Mind in Motion
The idea that paralyzed people might one day control their limbs just by thinking is no longer a Hollywood-style fantasy
By Miguel A. L. Nicolelis
in 2014 billions of viewers worldwide may remember the opening game of the World Cup in Brazil for more than just the goals scored by the Brazilian national team and the red cards given to its adversary. On that day my laboratory at Duke University, which specializes in developing technologies that allow electrical signals from the brain to control robotic limbs, plans to mark a milestone in overcoming paralysis. If we succeed in meeting still formidable challenges, the first ceremonial kick of the World Cup game may be made by a paralyzed teenager, who, flanked by the two contending soccer teams, will saunter onto the pitch clad in a robotic body suit. This suit—or exoskeleton, as we call it—will envelop the teenager’s legs. His or her first steps onto the field will be controlled by motor signals originating in the kicker’s brain and transmitted wirelessly to a computer unit the size of a laptop in a backpack carried by our patient. This computer will be responsi-
58 Scientific American, September 2012
ble for translating electrical brain signals into digital motor commands so that the exoskeleton can first stabilize the kicker’s body weight and then induce the robotic legs to begin the back-and-forth coordinated movements of a walk over the manicured grass. Then, on approaching the ball, the kicker will visualize placing a foot in contact with it. Three hundred milliseconds later brain signals will instruct the exoskeleton’s robotic foot to hook under the leather sphere, Brazilian style, and boot it aloft. This scientific demonstration of a radically new technology, undertaken with collaborators in Europe and Brazil, will convey to a global audience of billions that brain control of machines has moved from lab demos and futuristic speculation to a new era in which tools capable of bringing mobility to patients incapacitated by injury or disease may become a reality. We are on our way, perhaps by the next decade, to technology that links the brain with mechanical, electronic or virtual machines. This development will restore mobility, not only to accident and war victims but also to patients with ALS (also known as Lou Gehrig’s disease), Parkinson’s and other disorders that disrupt motor behaviors that impede arm reaching, hand grasping, locomotion and speech production. Neuroprosthetic devices—or brain-machine interfaces—will also allow scientists to do much more than help the disabled. They will make it possible to explore the world in revolutionary ways by providing healthy human beings with the ability to augment their sensory and motor skills. In this futuristic scenario, voluntary electrical brain waves, the biological alphabet that underlies human thinking, will maneuver large and small robots remotely, control airships from afar, and perhaps even allow the sharing of thoughts and sensations of one individu-
339
Illustration by Kemp Remillard
September 2012, ScientificAmerican.comâ&#x20AC;&#x201A;59
WHO WE ARE
B
LIME Y O ITS N D
340 W H AT W E CA N D O
W H E R E W E’ R E H E A D E D
al with another over what will become a collective brain-based network. Thought Machines
Miguel A. L. Nicolelis has pioneered the field of neuroprosthetics. He is Duke School of Medicine Professor of Neurosciences and co-director of the Duke University Center for Neuroengineering.
In Brief
Brain waves can now control the functioning of computer cursors, robotic arms and, soon, an entire suit: an exoskeleton that will allow a paraplegic to walk and maybe even move gracefully. Sending signals from the brain’s outer rindlike cortex to initiate movement in the exoskeleton represents the state of the art for a number of bioelectrical technologies perfected in recent years. The 2014 World Cup in Brazil will serve as a proving ground for a brain-controlled exoskeleton if, as expected, a handicapped teenager delivers the ceremonial opening kick.
60 Scientific American, September 2012
the lightweight body suit intended for the kicker, who has not yet been selected, is still under development. A prototype, though, is now under construction at the lab of my great friend and collaborator Gordon Cheng of the Technical University of Munich—one of the founding members of the Walk Again Project, a nonprofit, international collaboration among the Duke University Center for Neuroengineering, the Technical University of Munich, the Swiss Federal Institute of Technology in Lausanne, and the Edmond and Lily Safra International Institute of Neuroscience of Natal in Brazil. A few new members, including major research institutes and universities all over the world, will join this international team in the next few months. The project builds on nearly two decades of pioneering work on brain-machine interfaces at Duke—research that itself grew out of studies dating back to the 1960s, when scientists first attempted to tap into animal brains to see if a neural signal could be fed into a computer and thereby prompt a command to initiate motion in a mechanical device. Back in 1990 and throughout the first decade of this century, my Duke colleagues and I pioneered a method through which the brains of both rats and monkeys could be implanted with hundreds of hair-thin and flexible sensors, known as micro wires. Over the past two decades we have shown that, once implanted, the flexible electrical prongs can detect minute electrical signals, or action potentials, generated by hundreds of individual neurons distributed throughout the animals’ frontal and parietal cortices—the regions that define a vast brain circuit responsible for the generation of voluntary movements. This interface has for a full decade used brain-derived signals to generate movements of robotic arms, hands and legs in animal experiments. A critical breakthrough occurred last year when two monkeys in our lab learned to exert neural control over the movements of a computer-generated avatar arm that touched objects in a virtual world but also provided an “artificial tactile” feedback signal directly to each monkey’s brain. The software allowed us to train the animals to
feel what it was like to touch an object with virtual fingers controlled directly by their brain. The Walk Again consortium—assisted by its international team of neuroscientists, roboticists, computer scientists, neurosurgeons and rehabilitation professionals—has begun to take advantage of these animal research findings to create a completely new way to train and rehabilitate severely paralyzed patients in how to use brain-machine interface technologies to regain full-body mobility. Indeed, the first baby steps for our future ceremonial kicker will happen inside an advanced virtual-reality chamber known as a Cave Automatic Virtual Environment, a room with screens projected on every wall, including the floor and ceiling. After donning 3-D goggles and a headpiece that will noninvasively detect brain waves (through techniques known as electroencephalography—EEG—and magnetoencephalography), our candidate kicker— by necessity a lightweight teenager for this first iteration of the technology—will become immersed in a virtual environment that stretches out in all directions. There the youngster will learn to control the movements of a software body avatar through thought alone. Little by little, the motions induced in the avatar will increase in complexity and will ultimately end with fine-motor movements such as walking on a changing terrain or unscrewing a virtual jelly jar top. Plugging into Neurons
the mechanical movements of an exoskeleton cannot be manipulated as readily as those of a software avatar, so the technology and the training will be more complicated. It will be necessary to implant electrodes directly in the brain to manipulate the robotic limbs. We will need not only to place the electrodes under the skull in the brain but also to increase the number of neurons to be “read” simultaneously throughout the cortex. Many of the sensors will be implanted in the motor cortex, the region of the frontal lobe more readily associated with the generation of the motor program that is normally downloaded to the spinal cord, from which neurons directly control and coordinate the work of our muscles. (Some neuroscientists believe that this interaction between mind and muscle may be achieved through a nonin-
Science & Society Picture Library (artificial leg and Paré); Corbis (Civil War officer)
341
vasive method of recording brain activity, like EEG, but that goal has yet to be practically achieved.) Gary Lehew in my group at Duke has devised a new type of sensor: a recording cube that, when implanted, can pick up signals throughout a three-dimensional volume of cortex. Unlike earlier brain sensors, which consist of flat arrays of microelectrodes whose tips record neuronal electrical signals, Lehew’s cube extends sensing microwires up, down and sideways throughout the length of a central shaft. The current version of our recording cubes contains up to 1,000 active recording microwires. Because at least four to six single neurons can be recorded from each microwire, every cube can potentially capture the electrical activity of between 4,000 to 6,000 neurons. Assuming that we could implant several of those cubes in the frontal and parietal cortices—areas responsible for high-level control of movement and decision making— we could obtain a simultaneous sample of tens of thousands of neurons. According to our theoretical software modeling, this design would suffice for controlling the flexibility of movement required to operate an exoskeleton with two legs and to restore autonomous locomotion in our patients. To handle the avalanche of data from these sensors, we are also moving ahead on making a new generation of customdesigned neurochips. Implanted in a patient’s skull along with the microelectrodes, they will extract the raw motor commands needed to manipulate a wholebody exoskeleton. Of course, the signals detected from the brain will then need to be broadcast to the prosthetic limbs. Recently Tim Hanson, a newly graduated Ph.D. student at Duke, built a 128-channel wireless recording system equipped with sensors and chips that can be encased in the cranium and that is capable of broadcasting recorded brain waves to a remote receiver. The first version of these neurochips is currently being tested successfully in monkeys. Indeed, we have recently witnessed the first monkey to operate a brain-machine interface around the clock using wireless transmission of brain signals. We filed in July with the Brazilian government for permission to use this technology in humans. For our future soccer ball kicker, the
chronology
The Long Road to Brain-Controlled Prosthetics Replacement limbs have existed for millennia—a rational response to the need to address war wounds or other types of trauma and birth defects. Today the technology is so sophisticated that an artificial limb can be controlled by electrical signals channeled directly from the brain. 1500–1000 B.C. FIRST HISTORICAL REFERENCE A Hindu holy book written during this period mentioned Vishpala, who had a leg amputation after a wound sustained during battle. She had the limb replaced with an iron version that let her walk and return to her troops.
Fourth Century B.C. Ancient artifact One of the oldest artificial limbs discovered—a copy of which is shown here—was dug up in southern Italy in 1858. Fabricated in about 300 B.C., it was made of copper and wood and designed, it appears, for a belowknee amputee. 14th Century GUNS AND AMPUTATIONS The arrival of gunpowder at the European battlefront greatly amplified the number of injuries sustained by soldiers. In response, in the 16th century Ambroise Paré, the royal surgeon for several French kings, developed techniques to attach both upper and lower limbs to patients and reintroduced the use of ligatures to tie off blood vessels. 1861–1865 Civil War The War between the States resulted in many amputations. One person affected was Brigadier General Stephen Joseph McGroarty, who lost an arm. An influx of government funding and the avail ability of anesthetics that allowed for longer operations improved prosthetic technology during this era. 1963 PRIMITIVE BRAIN INTERFACE José Manuel Rodriguez Delgado implanted a radiocontrolled electrode in the caudate nucleus deep in a bull’s brain and stopped the animal dead in its tracks by pressing a button on a remote transmitter; his device was a predecessor to contemporary brainmachine interfaces.
1969 PIONEERING EXPERIMENTS Eberhard Fetz of the University of Washington performed a study in which monkeys were trained to activate electrical signals in their brain to control the firing of a single neuron, duly recorded by a metal microelectrode.
September 2012, ScientificAmerican.com 61
W H AT W E CA N D O
WHO WE ARE
chronology
1980s LISTENING TO BRAIN WAVES Apostolos Georgopoulos of Johns Hopkins University discovered an electrical firing pattern in the motor neurons of rhesus macaques that occurred when they rotated their arm in a particular direction. Early 1990s PLUGGING IN John Chapin, now at S.U.N.Y. Downstate University, and Miguel A. L. Nicolelis introduced a technique that allowed for simultaneous recording of dozens of widely dispersed neurons using permanently implanted electrodes, thus paving the way for research on brain-machine interfaces.
1997 BETTER MOVES The microprocessor-controlled C-Leg knee prosthesis, which in its current version allows the wearer to turn on customized settings that can be used for activities such as bicycling, was introduced. 1999–2000 GOOD FEEDBACK The Chapin and Nicolelis laboratories published the first description of a brain-machine interface operated by activity from rat brains, whereby the animals sensed the movement through a visual feedback signal. The following year the Nicolelis lab published the first study in which a monkey controlled the movements of a robotic arm using only brain activity.
2008–2011 BLADE RUNNER After failing to qualify for the 2008 Summer Olympics Games, Oscar Pistorius swept the 2008 Summer Paralympic Games and then got to the 400-meter semifinals at the 2011 International Association of Athletics Federations World Championships in Daegu, South Korea.
2011 MONKEY THINK, AVATAR DO Nicolelis’s team at the Duke University Center for Neuroengineering demonstrated that a monkey was able to use thoughts to manipulate the movements of a software avatar.
2012 FROM MY BRAIN TO MY ROBOT ARM John Donoghue of Brown University and his colleagues showed with their BrainGate neural interface system that a subject with a brain implant could manipulate a robotic arm to pick up a drink.
2014 CYBORG OPENING KICK The Nicolelis lab intends to provide an exoskeleton for a handicapped teenager to make the first kick of the opening event of the World Cup in Brazil.
62 Scientific American, September 2012
W H E R E W E’ R E H E A D E D
data from the recording systems will be relayed wirelessly to a small computer processing unit contained in a backpack. Multiple digital processors will run various software algorithms that translate motor signals into digital commands that are able to control moving parts, or actuators, distributed across the joints of the robotic suit, hardware elements that adjust the positioning of the exoskeleton’s artificial limbs. force of brainpower
the commands will permit the exoskeleton wearer to take one step and then another, slow down or speed up, bend over or climb a set of stairs. Some low-level adjustments to the positioning of the prosthetic hardware will be handled directly by the exoskeleton’s electromechanical circuits without any neural input. The space suit–like garment will remain flexible but still furnish structural support to its wearer, a surrogate for the human spinal cord. By taking full advantage of this interplay between brain-derived control signals and the electronic reflexes supplied by the actuators, we hope that our brain-machine interface will literally carry the World Cup kicker along by force of willpower. The kicker will not only move but also feel the ground underneath. The exoskeleton will replicate a sense of touch and balance by incorporating microscopic sensors that both detect the amount of force from a particular movement and convey the information from the suit back to the brain. The kicker should be able to feel that a toe has come in contact with the ball. Our decade-long experience with brainmachine interfaces suggests that as soon as the kicker starts interacting with this exoskeleton, the brain will start incorporating this robotic body as a true extension of his or her own body image. From training, the accumulated experience obtained from this continuous feeling of contact with the ground and the position of the robotic legs should enable movement with fluid steps over a soccer pitch or down any sidewalk. All phases of this project require continuous and rigorous testing in animal experiments before we begin in humans. In addition, all procedures must pass muster with regulatory agencies in Brazil, the U.S. and Europe to ensure proper scientific and ethical re-
courtesy of otto bock healthcare (artificial leg); andrew medichini AP Photo (Pistorius); courtesy of miguel a. l. nicolelis (monkey avatar); courtesy of braingate2.org (BrainGate)
B
LIME Y O ITS N D
342
343
view. Even with all the uncertainties involved and the short time required for the completion of its first public demonstration, the simple idea of reaching for such a major milestone has galvanized Brazilian society’s interest in science in ways rarely seen before. Remote Control
the opening kickoff of the World Cup—or a similar event, say, the 2016 Olympic and Paralympic Games in Rio de Janeiro, if we miss the first deadline for any reason—will be more than just a one-time stunt. A hint of what may be possible with this technology can be gleaned from a two-part experiment already completed with monkeys. As a prelude, back in 2007, our research team at Duke trained rhesus monkeys to walk upright on a treadmill as the electrical activity of more than 200 cortical neurons was recorded simultaneously. Meanwhile Gordon Cheng, then at ATR Intelligent Robotics and Communication Laboratories in Kyoto, built an extremely fast Internet protocol that allowed us to send this stream of neuronal data directly to Kyoto, where it fed the electronic controllers of CB1, a humanoid robot. In the first half of this across-the-globe experiment, Cheng and my group at Duke showed that the same software algorithms developed previously for translating thoughts into control of robotic arms could also convert patterns of neural activity involved in bipedal locomotion to make two mechanical legs walk. The second part of the experiment yielded a much bigger surprise. As one of our monkeys, Idoya, walked on the treadmill in Durham, N.C., our brain-machine interface broadcast a constant stream of her brain’s electrical activity through Cheng’s Internet connection to Kyoto. There CB1 detected these motor commands and began to walk as well, almost immediately. CB1 first needed some support at the waist, but in later experiments it began to move autonomously in response to the brain-derived commands generated by the monkey on the other side of the globe. What is more, even when the treadmill at Duke stopped and Idoya ceased walking, she could still control CB1’s leg movements in Kyoto by merely observing the robot’s legs moving on a live video feed and imagining each step CB1
should take. Idoya continued to produce the brain patterns required to make CB1 walk even though her own body was no longer engaged in this motor task. This transcontinental brain-machine interface demonstration revealed that it is possible for a human or a simian to readily transcend space, force and time by liberating brain-derived commands from the physical limits of the biological body that houses the brain and broadcasting them to a man-made device located far from the original thought that generated the action. These experiments imply that brainmachine interfaces could make it possible to manipulate robots sent into environments that a human will never be able to penetrate directly: our thoughts might operate a microsurgical tool inside the body, say, or direct the activities of a humanoid worker trying to repair a leak at a nuclear plant. The interface could also control tools that exert much stronger or lighter forces than our bodies can, thereby breaking free of ordinary constraints on the amount of force an individual can exert. Linking a monkey’s brain to a humanoid robot has already done away with constraints imposed by the clock: Idoya’s mental trip around the globe took 20 milliseconds— less time than was required to move her own limb. Along with inspiring visions of the far future, the work we have done with monkeys gives us confidence that our plan may be achievable. At the time of this writing, we are waiting to see whether the International Football Association (FIFA), which is in charge of organizing the ceremony, will grant our proposal to have a paraplegic young adult participate in the opening ceremony of the inaugural game of the 2014 World Cup. The Brazilian government—which is still awaiting FIFA’s endorsement—has tentatively supported our application. Bureaucratic difficulties and scientific uncertainties abound before our vision is realized. Yet I cannot stop imagining what it will be like during the brief but historic stroll onto a tropical green soccer pitch for three billion people to witness a paralyzed Brazilian youth stand up, walk again by his or her own volition, and ultimately kick a ball to score an unforgettable goal for science, in the very land that mastered the beautiful game.
m o r e t o e x pl o r e
Controlling Robots with the Mind. Miguel A. L. Nicolelis and John K. Chapin in Scientific American, Vol. 287, No. 4, pages 46–53; October 2002. Cortical Control of a Prosthetic Arm for Self Feeding. Meel Velliste et al. in Nature, Vol. 453, pages 1098– 1101; June 19, 2008. Beyond Boundaries: The New Neuroscience of Connecting Brains with Machines—and How It Will Change Our Lives. Miguel Nicolelis. St. Martin’s Griffin, 2012. Scientific American Online Inspect an exoskeleton prototype at ScientificAmerican. com/sep2012/exoskeleton
September 2012, ScientificAmerican.com 63
344 PUBLISHED: 10 JANUARY 2017 | VOLUME: 1 | ARTICLE NUMBER: 0008
comment
Are we at risk of becoming biological digital machines? Miguel A. L. Nicolelis
I
n Neal Stephenson’s science-fiction novel Snow Crash1, Hiro Protagonist, “a warrior prince in the metaverse” and one of the most skilful hackers of his generation, is on a mission to save humanity from its true neurophysiological doomsday. Hiro’s herculean task is to try to stop a new type of virus, Snow Crash, from infecting the minds of people all over the world and transforming them into mere biological automata, devoid of any trace of real consciousness, free will, agency or individuality. Once uploaded into the brainstem, Snow Crash insidiously hacks the basic machine code that runs in our subcortical limbic system, leading to a complete halt of neocortical subroutines. All this neurobiological mayhem turns the infected person into a perfect zombie, someone incapable of reasoning critically or maintaining his/her natural state of consciousness. Are contemporary humans running a similar risk? Could our constant reciprocal interaction with digital logic (through laptops, tablets, smartphones, all the way to highly immersive virtual reality environments), particularly when it leads to powerfully hedonic experiences, result in the slow compromise or even elimination of some of the behaviours and cognitive aptitudes that represent the most exquisite and cherished attributes of the human condition? Attributes such as our multifaceted social skills, empathy, linguistic semantics, aesthetic sense, artistic expression, intuition, creativity and the ability to improvise solutions to novel contingencies, to name just a few. In other words, could opting for the fast lane of the never-ending highway to full digital immersion and automation — an obvious current trend in our modern society — produce a reduction in human cognitive capabilities that resembles, even if only remotely, the less-than-enticing fate of those infected by Snow Crash?
CREATIVE-IDEA/DIGITALVISION VECTORS/GETTY
The brain can be viewed as an organic computer that can be reprogrammed to incorporate external elements, such as artificial tools. But is there a risk that our increasing reliance on digital devices, such as smartphones, could also be reprogramming our brains and blunting our human attributes?
The brain as an organic computer
Since Alan Turing’s original proposition in the late 1930s of what became known as the universal Turing machine, the debate on whether an animal brain, and in particular the human central nervous system, can be considered a digital computer has raged. Artificial-intelligence researchers who believe that digital machines can acquire human-like intelligence have no qualms in accepting this view at face value. Surprisingly, a few neuroscientists tend to agree with this position and even suggest that animal brains, including ours, can be fully simulated in a digital device. On the other hand, many philosophers, neuroscientists and physicists have staunchly disagreed with this proposition, labelling it as nothing more than wishful thinking or, worse, a form of mystical belief. Recently, Ronald Cicurel and I2 summarized the evolutionary, neurobiological, computational and mathematical arguments that challenge the notion that the operation of the human brain can be reduced to the algorithmic nature of Turing machines. Instead, we proposed that higher-order brains, including ours, constitute a different type of computation
device altogether, which we referred to as an organic computer. One of the distinctions between digital and organic computers is that higher animal brains use a recursive mix of analogue and digital processing to compute and are not only capable of, but highly specialized in, handling semanticrich information, which differs markedly from the classic syntactically rigid type of information on which digital machines rely. Furthermore, in our brains, the concepts of software and hardware cannot be dissociated from each other; both are welded together in the very organic matter that defines the organic computer. As such, brains utilize the coherent blending of their multiple levels of organization — from intracellular protein complexes, to cell membranes, to individual whole cells, to neuronal circuits, all the way to the entire system — to compute.
Programming the human brain
But there is an important caveat. Even though organic computers cannot be reduced to a Turing machine, they can be programmed by many distinct biological processes. For instance, at the most basic level, the expression of numerous genes in the human genome, selected by a multitude of evolutionary events, interact as part of a ‘genetic programme’ that is responsible for assembling the brain’s natural 3D structure during prenatal and postnatal life. This genetic programming guarantees that our brain’s initial physical configuration reflects the same basic neural architecture that evolved in anatomically modern humans about 100,000 years ago. Once we are born, the brain’s programming continues as we grow and learn, through interactions with our social and physical environment. Continuous immersion in human culture and its cornucopia of social interactions further programmes the central nervous system. Indeed, the so-called social brain hypothesis3,4 proposes that the tremendous
NATURE HUMAN BEHAVIOUR 1, 0008 (2017) | DOI: 10.1038/s41562-016-0008 | www.nature.com/nathumbehav
1
. d e v r e s e r s t h g i r l l A . e r u t a N r e g n i r p S f o t r a p , d e t i m i L s r e h s i l b u P n a l l i m c a M 7 1 0 2 ©
345
comment growth in neocortical volume, experienced during the evolution of anthropoid primates into Homo sapiens, is intimately related to the concomitant increase in social complexity that these species experience in their daily routine living in groups. This correlation can be readily visualized when the cortical volume of several primates is plotted against the typical social group size of each species. Accordingly, the significant growth in human cortical volume would explain why our species developed the capacity to handle close social interactions in groups of about 150 individuals, while chimpanzees and baboons limit their bands to 50–55 individuals4.
Reprogramming the human brain
But this is not all. There are other possible ways to programme or even reprogramme the primate brain that we carry between our ears. A decade and a half of intense basic and clinical research with brain/machine interfaces (BMIs) has clearly demonstrated that the primate brain, including our own, is capable of assimilating artificial tools — like robotic or even avatar limbs — as extensions of the brain-built representation of the subject’s own body5, through the phenomenon of cortical plasticity. This means that neuronal space is dedicated to representing the operation of artificial tools5. These findings, together with other results from various studies, indicate that our sense of self can be readily reprogrammed to incorporate external elements. We have recently documented such an extension of the sense of self in severely paralyzed patients trained to employ a BMI to control the movements of an avatar body6. Indeed, such reprogramming through cortical, and probably subcortical, plastic changes might also account for the partial neurological clinical recovery experienced by these patients7. That brings us back to the central issue of this brief Comment article. Even though the brain cannot be reduced to a digital machine, could the human brain simply assimilate and begin to mimic the rigid binary logic and algorithmic mode of operation of digital machines due to the growing overexposure to digital devices and the hedonic response triggered by these interactions, and become a biological digital system? Given the ominous introduction of digital automation in almost all aspects of human life, it is not surprising that a large literature exists describing the impact
2
of the introduction of digital systems on human behaviour and mental skills. In his book8, The Glass Cage: Automation and Us, Nicholas Carr reviews some of these studies, showing that continuous exposure to digital systems can have profound effects on human performance, from the flying skills of airplane pilots, to the pattern recognition ability of radiologists, to the broad sense of creativity of architects.
In our brains, the concepts of software and hardware cannot be dissociated from each other. If I had to propose a hypothesis to account for all these findings, I would volunteer the notion that passive immersion in the digital systems of modern airplanes (in the case of pilots), digital imaging diagnostics (radiologists) and computer-assisted design (architects) may gradually curtail the range and acuity of some mental functions and cognitive skills, such as creativity, insight and the ability to solve novel problems. A study by Betsy Sparrow et al.9 showed that when people believe that a series of statements that they have been asked to remember will be stored online, they perform worse than a control group that relies only on their own biological memory to remember the statements. This suggests that subcontracting some simple mental searches to Google may, after all, reduce our own brain’s ability to store and recall memories reliably. The impact of online social media on our natural social skills is another area in which we may be able to measure the true effects of digital systems on human behaviour. In her book10, Alone Together: Why We Expect More from Technology and Less from Each Other, Sherry Turkle describes her experience interviewing teenagers and adults who are heavy users of texting, social media and other online virtual environments. An intense presence on social media and virtual reality environments can produce significant anxiety, a reduction in real social interactions, lack of social skills and human empathy, and difficulties in handling solitude. Moreover, symptoms and signs of addiction to virtual life are often reported almost casually by some of the interviewees. After reading Turkle’s interviews, I began wondering whether the new ‘always
connected’ routine is overtaxing our cerebral cortex by dramatically expanding the number of people with whom we can closely communicate, almost instantaneously, via the multitude of social media outlets available on the internet. Instead of respecting the group size limit (about 150 individuals) afforded by our cortical volume, we are now in continuous contact with a group of people that could far exceed that neurobiological limit. What are the consequences of this cortical overtaxing? Anxiety, attention, cognitive and even memory deficits?
Homo digitalis
Is the above scenario something we should pay attention to? I think so. If not because of the potential impact on the mental health of this and future generations, but also because of the far-reaching consequences of our increasing interaction with digital systems. For example, at the far limit, I can conceive that this staggering expansion in our online social connectivity is capable of providing a completely new type of selective pressure that may, eventually, bias the evolutionary future of our species. One may begin wondering whether the dawn of ‘Homo digitalis’ is upon us or, more surprisingly, whether he/she is already around, texting and tweeting without being noticed. Evidently, much more research will be needed to address these intriguing questions and the true impact of digital technology on shaping key functions of the human brain. Meanwhile, just in case we need his services in the near future, next time I meet Neal Stephenson, I will be sure to ask for Hiro Protagonist’s phone number and e-mail, Facebook, Twitter and LinkedIn addresses. That is, if I remember to ask. ❐ Miguel A. L. Nicolelis is in the Department of Neurobiology, Box 3209 Duke University, Durham, North Carolina 27710, USA. e-mail: nicoleli@neuro.duke.edu References
1. Stephenson, N. Snow Crash (Bantam Books, 2008). 2. Cicurel, R. M. & Nicolelis, M. A. L. The Relativistic Brain: How It Works and Why It Cannot Be Simulated by a Turing Machine (Kios Press, 2015). 3. Dunbar, R. I. M. & Shultz, S. Science 317, 1344–1347 (2007). 4. Dunbar, R. Grooming, Gossip, and the Evolution of Language (Harvard Univ. Press, 1996). 5. Carmena, J. M. et al. PLoS Biol. 1, e42 (2003). 6. Shokur, S. et al. Sci. Rep. 6, 32293 (2016). 7. Donati, A. R. C. et al. Sci. Rep. 6, 30383 (2016). 8. Carr, N. G. The Glass Cage: Automation and Us (W. W. Norton & Company, 2014). 9. Sparrow, B., Liu, J. & Wegner, D. M. Science 333, 776–778 (2011). 10. Turkle, S. Alone Together: Why We Expect More from Technology and Less from Each Other (Basic Books, 2011).
NATURE HUMAN BEHAVIOUR 1, 0008 (2017) | DOI: 10.1038/s41562-016-0008 | www.nature.com/nathumbehav
. d e v r e s e r s t h g i r l l A . e r u t a N r e g n i r p S f o t r a p , d e t i m i L s r e h s i l b u P n a l l i m c a M 7 1 0 2 ©
346
REVIEW
Electrical Stimulation of the Dorsal Columns of the Spinal Cord for Parkinson’s Disease Amol P. Yadav, PhD1,2 and Miguel A. L. Nicolelis, MD, PhD1,2,3,4,5,6* 1 Department of Neurobiology, Duke University, Durham, North Carolina, USA Duke Center for Neuroengineering, Duke University, Durham, North Carolina, USA 3 Department of Psychology and Neuroscience, Duke University, Durham, North Carolina, USA 4 Department of Biomedical Engineering, Duke University, Durham, North Carolina, USA 5 Department of Neurology, Duke University, Durham, North Carolina, USA 6 Edmond and Lily Safra International Institute of Neuroscience of Natal, Natal, Brazil 2
A B S T R A C T : Spinal cord stimulation has been used for the treatment of chronic pain for decades. In 2009, our laboratory proposed, based on studies in rodents, that electrical stimulation of the dorsal columns of the spinal cord could become an effective treatment for motor symptoms associated with Parkinson’s disease (PD). Since our initial report in rodents and a more recent study in primates, several clinical studies have now described beneficial effects of dorsal column stimulation in parkinsonian patients. In primates, we have shown that dorsal column stimulation activates multiple structures along the somatosensory pathway and desynchronizes the pathological cortico-striatal oscillations responsible for the manifestation of PD
Parkinson’s disease (PD) is a chronic, progressive neurodegenerative disease1-3 resulting from the loss of dopamine-producing brain cells in the substantia nigra pars compacta.4 Patients suffering from PD experience progressive motor impairments,1 which include tremor, rigidity, bradykinesia, and gait instability. It is estimated that approximately 0.4 to 1 million Americans live with PD.5,6 Although dopamine replacement therapy, through administration of the dopamine precursor effectively L-3,4-dihydroxyphenylalanine (L-dopa), ameliorates PD symptoms in the early stages of the disease,7 prolonged use of the drug results in dose-limiting side effects,8,9 reduced efficacy,10,11 and complications
-----------------------------------------------------------*Corresponding author: Dr. Miguel A. Nicolelis, Box 3209 Duke University, Durham, NC 27710; nicoleli@neuro.duke.edu
Relevant conflicts of interests/financial disclosures: Nothing to report. Received: 28 November 2016; Revised: 7 March 2017; Accepted: 10 April 2017 Published online 00 Month 2017 in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/mds.27033
symptoms. Based on recent evidence, we argue that neurological disorders such as PD can be broadly classified as diseases emerging from abnormal neuronal timing, leading to pathological brain states, and that the spinal cord could be used as a “channel” to transmit therapeutic electrical signals to disrupt these C abnormalities. V 2017 International Parkinson and Movement Disorder Society
K e y W o r d s : spinal cord stimulation; Parkinson’s disease; neuronal oscillations; deep brain stimulation; dopamine
such as L-dopa–induced dyskinesia.12,13 In this context, additional therapeutic strategies, such as deep brain stimulation (DBS), that have proven to be effective for treating the main PD motor symptoms have attracted considerable attention in the past few decades.14-17 Despite its unquestionable success, there are some disadvantages associated with DBS. First, it requires a highly invasive and expensive surgical procedure, which targets a very small structure deep in the brain.18,19 Because of this, the neurosurgical procedure for DBS comes associated with a 1.1% risk of death during the operation and 1.2% to 15.2% risk of other major complications.20-23 This means that only very experienced functional neurosurgeons can perform the procedure. Altogether, these factors, in particular the procedure’s invasiveness, limit the procedure to just a small fraction of severely ill PD patients: 1.6% to 4.5%.24-26 Another important issue is that, although DBS produces a very significant improvement of the patient’s tremor and rigidity, it is less effective in treating those
Movement Disorders, Vol. 00, No. 00, 2017
1
347 Y A D A V
A N D
N I C O L E L I S
who suffer primarily from bradykinesia or gait instability and “freezing.”14,27,28 Thus, although DBS greatly improves the quality of life in patients with advanced PD by addressing the disease’s cardinal motor symptoms while reducing levodopa-induced dyskinesias, locomotion impairment can be extremely disabling and severely affect the quality of life of PD patients, even when the primary symptoms are controlled by this surgical therapy. Furthermore, if one considers that standard treatments for bradykinesia and freezing, such as physical therapy, are often ineffective in PD, it becomes clear that novel therapeutic approaches are required to address postural instability and gait disturbance in PD patients.24,29,30 In this context, a significantly less invasive method—epidural dorsal column stimulation (DCS)—has been suggested by our laboratory as an alternative approach for symptomatic treatment of PD. DCS is a well-established therapy for the treatment of chronic neuropathic pain. In the United States, DCS has been approved by the Food and Drug Administration for treating chronic low back and limb pain.31 Back in 2009, our laboratory reported on the initial experimental demonstrations of the potential therapeutic benefit of DCS in multiple rodent models of PD.32 Initially, we demonstrated an acute effect of DCS on akinesia and bradykinesia in rodents.32 Later, the same effect was reproduced in nonhuman primate models.33 Further studies revealed both a long-term motor improvement and a neuroprotective effect on the rat nigrostriatal dopaminergic system of rodents, following chronic DCS.34 Following these initial studies, other laboratories have independently validated our animal findings.35,36 In parallel, during the past 7 years, several independent clinical studies in PD patients with abnormal posture and gait disturbances have demonstrated positive results with DCS.37-43 The central goal of this short review is to cover the recent literature that supports our initial contention that DCS can become a useful new therapy for PD, particularly for those patients who cannot benefit from DBS because their main symptoms are related to gait dysfunction.
Spinal Cord Stimulation Electrical stimulation of the dorsal funiculus of the spinal cord was first used for the treatment of chronic pain by Shealy and colleagues at Case Western Reserve University in 1967.44 Since then it has become a treatment for different pain syndromes such as pain from failed back surgery syndrome or intractable low back pain31,45,46 and also investigated for numerous novel pain syndromes.47-49 The rationale for using DCS in pain treatments was initially provided by the classic gate control theory of pain proposed originally by Melzack and Wall.50 According to the theory, by
2
Movement Disorders, Vol. 00, No. 00, 2017
activating the large diameter, myelinated, nonnociceptive fibers of the dorsal funiculus of the spinal cord, DCS would produce a closing of the “gate” at the dorsal horns, leading to suppression of the noxious stimuli carried by the much smaller unmyelinated C fibers that transmit nociceptive information that contributes to the genesis of pain. Subsequently, other potential mechanisms for the suppression of pain by DCS were uncovered, including the following: 1. Paresthesia triggered by the activation of supraspinal circuits produced by orthodromic activation of dorsal columns (DCs),51,52 2. Changes in the local transmitter systems and the suppression of dorsal horn neurons in the case of neuropathic pain,53,54 and 3. The balance of the oxygen supply and demand by inhibition of sympathetic activity in the case of ischemic pain.51,55 Interestingly, over the years and in parallel with the growing use of this approach for chronic pain, several reports of the potential therapeutic effects of DCS in improving motor symptoms in patients suffering from various motor disorders, such as dystonia,56-59 multiple sclerosis,60 nonparkinsonian tremor,61,62 and painful leg and moving toes syndrome,63,64 have appeared in the literature.65 Although the exact neurophysiological mechanisms involved in relieving the symptoms in these motor disorders by DCS have not been elucidated, computational modeling studies have shown that epidural electrical stimulation of the spinal cord, when applied within the therapeutic range, usually activates DC fibers and the dorsal roots in the vicinity of the stimulating cathode.66 In addition, functional magnetic resonance imaging during DCS application in patients revealed clear modulation of activity in cortical structures, such as primary and secondary somatosensory cortices (S1 and S2), prefrontal cortex, cingulate cortex, insula, and thalamus.67-69 DCS has also been shown to cause changes in c-fos expression in supraspinal structures, which in turn may lead to long-term sustained effects.70,71 Both of these findings are consistent with the neurophysiological and immunohistochemical results obtained in our animal studies with DCS, which show that this approach disrupts pathological oscillations around the cortical-basal ganglia circuitry while inducing a neuroprotective effect on the dopaminergic neurons of the nigrostriatal system.34
Low-Frequency Neuronal Oscillations Are Correlated With PD Symptoms The basal ganglia is involved in the execution of goal-directed behavior in conjunction with the cortex
348 S P I N A L
via an extensive circuitry formed by multiple structures that share feedforward and feedback pathways.72 Multiple clinical studies in parkinsonian patients have detected the presence of pathologically high levels of synchronous oscillatory neuronal and local field potential activity in the basal ganglia, which is particularly prominent in the beta frequency range of 15 to 30 Hz.73-75 Animal models of PD also exhibit similar pathological activity. For example, monkeys treated with the toxin 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP) show a significant increase in the fraction of basal ganglia neurons with oscillatory frequency in the 3 to 8 Hz and 8 to 15 Hz ranges. Often this neuronal activity is correlated with the animal’s tremor.76,77 Multiple reports using the MPTP model have identified neuronal synchronous oscillatory activity in the subthalamic nucleus (STN), internal globus pallidus, external globus pallidus, and substantia nigra pars reticulata neurons.76,78 Similarly, studies involving the 6-hydroxydopamine (6-OHDA) rat model have also demonstrated the presence of beta frequency (15-30 Hz) oscillations in STN and substantia nigra pars reticulata.79-81 Sharott and colleagues80 demonstrate that the power of beta frequency oscillations and their coherence between the STN and cortex was significantly reduced after the administration of apomorphine (a dopamine agonist) in rats, shifting the peak coherence to higher frequencies, a phenomenon we have also observed after 82 L-dopa administration in dopamine-depleted mice. This effect coincides with previous observations in the basal ganglia areas of parkinsonian patients after dopaminergic medication.83,84 In monkeys, dopamine replacement therapy85 and STN inactivation through DBS86 significantly ameliorate MPTP-induced tremor while reducing prominent 8 to 20 Hz oscillations in the basal ganglia. DBS has shown to effectively improve PD symptoms in humans while attenuating synchronous beta band activity in the cortico-basal ganglia network.87-90 It is, however, important to note that the reduction in beta band activity after medication as well as DBS is often correlated with the improvement in motor performance, suggesting that the attenuation of oscillatory and synchronous neural activity has a therapeutic effect on PD symptoms.91-93
C O R D
S T I M U L A T I O N
F O R
P D
thereby causing the alleviation of PD symptoms. Our studies revealed that DCS treated animals exhibited much higher levels of locomotion than dopaminedepleted mice, 6-OHDA lesioned rats, and 6-OHDA lesioned marmosets.32-34 Increased oscillatory power in the 1.5 to 4 Hz and 10 to 15 Hz bands and decreased power in 25 to 55 Hz bands was observed during the dopamine-depleted state in mice.32 DCS created a shift in spectral power from lower to higher frequencies (Fig. 1) and produced a neuronal firing pattern similar to the one observed prior to locomotion initiation.32 The fraction of neurons in the M1 and striatum that were entrained to the Local Field Potential (LFP) activity also dropped considerably after the application of DCS. Although the onset of locomotion on DCS application was delayed by a few seconds, the changes in neuronal activity were almost spontaneous. These findings suggested that DCS created a brain state permissible for locomotion onset. Thereafter, our study using a marmoset monkey model of PD revealed that DCS indeed alleviates akinesia and restores the pathological brain state, defined by abnormal neuronal bursting and oscillatory activity, to normalcy by altering the functional coupling between multiple areas in the cortico-basal gangliathalamic loop.33 During the induced PD state, we observed a significant increase in functional coherence, in the 8 to 15 Hz range, between multiple neural structures that belong to the cortical-basal ganglia circuitry. During DCS, however, this enhanced coherence—which can also be seen as enhanced functional connectivity—was reduced, an effect similar to what was observed after L-dopa administration. DCS resulted in the suppression of 8 to 20 Hz beta band LFP power as well as the beta rhythmicity of neurons
Spinal Cord Stimulation Mechanism for PD For the past 7 years, our laboratory has studied the effects of DCS on PD symptoms in rodents and nonhuman primates. Based on previous evidence that electrical stimulation of the peripheral afferents of a major somatosensory pathway could be used to induce potent cortical desynchronization,94 we proposed that high-frequency synchronous activation of the DC fibers could lead to cortico-striatal desynchronization,
FIG. 1. Dorsal column stimulation (DCS) restores locomotion and desynchronize corticostriatal activity (reprinted with permission; originally published in Fuentes et al.32). Average spectrograms of striatal local field potentials (LFP) (in dopamine-depleted mice) recorded around 300-Hz stimulation events (yellow bar) shown in top row indicates the spectral power (denoted by black trace) shifting to higher frequencies, middle row showing LFP power standardized to first 240 seconds demonstrates desynchronization, whereas the bottom row shows increased locomotion during stimulation “ON” period.
Movement Disorders, Vol. 00, No. 00, 2017
3
349 Y A D A V
A N D
N I C O L E L I S
FIG. 2. Dorsal column stimulation (DCS) reverses weight loss, restores motor function, and protects dopaminergic neurons in a 6-hydroxydopamine (6-OHDA) model of Parkinsonâ&#x20AC;&#x2122;s disease (reprinted with permission; originally published in Yadav et al.34). (A) Changes in body weight after bilateral intrastriatal 6-OHDA lesion with or without DCS treatment. Lesioned, nontreated rats (n 5 8) suffered sustained weight loss with little to no recovery. Lesioned rats with DCS treatment (n 5 7, 30 minutes, 333 Hz continuous DCS during 30 minutes twice a week, starting 7th day, black arrow) recovered body weight significantly faster than nontreated rats. (B) Lesioned rats develop crouched posture resulting in shorter major axis length. DCS treatment restores posture significantly faster than nontreated rats. (C) Representative immunostaining for tyrosine hydroxylase in substantia nigra pars compacta (SNc); scale bar 5 500 um. There was a significant difference between the tyrosine hydroxylase (TH) levels of 6-OHDA and 6OHDA1DCS groups in the SNc. VTA, ventral tegmental area.
in the cortico-basal ganglia-thalamic loop. Detailed analysis revealed that DCS activation of the primary somatosensory cortex via the ventral posterolateral nucleus of the thalamus induces a potent disruption of beta oscillations in multiple cortical and subcortical neural structures via a phase reset mechanism, where the LFP oscillations get phase locked to the incoming DCS pulses.33 This finding corroborated our original hypothesis that DCS desynchronizes corticostriatal oscillations, following the activation of the DCâ&#x20AC;&#x201C;medial lemniscal pathway.
Preliminary Evidence of Neuroprotection In a subsequent series of experiments (Fig. 2), we investigated the long-term effects of DCS on animal body weight, motor symptoms, and survival of nigrostriatal dopaminergic neurons in a chronic rat model of PD using 6-OHDA lesioning.34 This study revealed that the application of DCS at regular intervals led to a progressive improvement in the observed motor impairment, such as gait and postural instability, and an accelerated recovery from weight loss. When compared with untreated control animals, motor improvement was accompanied by a higher density of dopaminergic innervation in the striatum and higher neuronal cell counts in the substantia nigra pars compacta (SNc) of DCS-treated rats. These results suggest that DCS applied in 6-OHDA-treated rats induced both a functional and structural recovery in the nigrostriatal circuitry. Remarkably, such a neuroprotective effect was achieved by delivering DCS for 30 minutes, twice a week only, suggesting that better effects could be produced by more frequent therapy. These results have now been
4
Movement Disorders, Vol. 00, No. 00, 2017
replicated independently by another laboratory,36 which reported that 1 hour/day of DCS, delivered for 16 consecutive days, resulted in the improvement of PD symptoms and the significant preservation of tyrosine hydroxylase (TH) fibers in the striatum and TH-positive neurons in the SNc. Although the cellular mechanism underlying this putative long-term neuroprotective action of DCS remains to be investigated, Shinko and colleagues36 also reported that DCS treatment upregulated the levels of vascular endothelial growth factor, a finding that could account, at least partially, for the observed neuroprotective effects. Support for this idea comes from studies in which intrastriatal injections of glial-derived nerve growth factor (GDNF) and brain-derived neurotrophic factor (BDNF) have shown significant protection or restoration following 6-OHDA or MPTP lesions.95-98 Comparative analysis revealed that GDNF was more effective than BDNF for correcting behavioral deficits and protecting nigrostriatal DA neurons.99 Spieles-Engemann and colleagues100 have shown that STN DBS also exerts a neuroprotective effect on the SN neurons in a 6-OHDA rodent model of PD. A follow-up study from the same group revealed that this effect could be mediated by the increased levels of BDNF in the nigrostriatal system and the primary motor cortex.101 Shinko and colleagues note that upregulation of vascular endothelial growth factor may protect the nigrostriatal dopaminergic system by improving microcirculation in the striatum as a result of enhanced glial proliferation and angiogenesis.36 Although the results showing neuroprotection in animal models using DCS or DBS are promising, there is no clinical evidence supporting this claim, suggesting that further investigation using DCS in parkinsonian humans is necessary.102
63-79
65
43
15
1
1
Agari et al, 201237
Landi et al, 201241
Hassan et al, 201340
68
74
1
Fenelon et. al. 201139
NA
7-31
8
8
5
17
PD with postlaminectomy syndrome pain
PD with chronic neuropathic pain in neck and upper extremities
Advanced PD with chronic intractable leg pain
PD (Hoehn/Yahr stage III and IV) with low back and lower limb pain
PD (Hoehn/Yahr stage IV) with chronic back pain PD with lower back neuropathic pain
PD with moderate to severe motor impairments
Diagnosis
T9-T11
C2
T9-T10
T7-T12
T9-T10
Cervicothoracic
High cervical
Location
60
40
30
5-20
130
130 and 300
Frequency (Hz)
Stimulation
Patient observation
VAS for pain, UPDRS and timed 10 meter walk test
VAS for pain, quality of life, 20 meter walk and UPDRS III
VAS for pain, UPDRS and 7 meter walk and back VAS for pain, UPDRS ADL and motor, TUG, 10 meter walk test
Motor UPDRS, timed 10 meter walk, timed handarm movement test, timed lower limb tapping NAS, UPDRS III, up and go test
Evaluation method
NA
2 years
16 months
12 months
4 sessions two to five weeks apart
NA
10 days
Follow-up
S T I M U L A T I O N
(Continued)
Improvement >50% for VAS, improvement in TUG at 3 months, and 10 metre walk at 3 and 12 months, improvement in UPDRS II (items 12,15) and III (items 28,29,30,31) Improvement of 70% in VAS score for pain, 20% in 20 meter walk time, 60% in quality of life, No change in UPDRS III Improvement in VAS pain score throughout follow-up and timed 10 meter walk, UPDRS score: 28 (early post-operative), 22 (1 year) and 16 (2 year) Improvement in leg pain and left sided resting tremor
>50 improvement in UPDRS scores
Improvement in NAS score but not UPDRS
No improvement
Results
C O R D
1
72
1
Weise et al, 201043
NA
Disease duration, y
S P I N A L
Soltani & Lalkhen 201342
75-77
2
Patientsâ&#x20AC;&#x2122; age, y
Thevathasan et al, 2010103
Study
No. of patients
TABLE 1. Clinical studies of spinal cord stimulation for Parkinsonâ&#x20AC;&#x2122;s disease
350
F O R
Movement Disorders, Vol. 00, No. 00, 2017
P D
5
6
Movement Disorders, Vol. 00, No. 00, 2017
67-80
56-69
3
4
Nishioka & Nakajima 2015107
de Souza et al, 2016108
NA
05-10
NA
07-10
PD (Hoehn/Yahr stage III and IV) with camptocormia (anterior flexion of thoracolumbar spine 45%) PD with severe intractable lower back and lower limb pain Advanced idiopathic PD with significant PIGD after bilateral STN DBS
Chronic low back pain associated with PD Hoehn/ Yahr stage II
Diagnosis
Upper thoracic (T2-T4)
T8-L1
Lower thoracic to upper lumbar
Midthoracic
Location
300
5-65
5 Hz rTSMS
NA
Frequency (Hz)
Stimulation
TUG, TUG-DT, 20 minute walking with/without obstacles, PDQ39, UPDRS III
VAS and WPI for pain, UPDRS and Hoehn/Yahr for PD
thoracolumbar spine flexion angle in standing and seated position
VAS for pain, walking posture and bradykinesia
Evaluation method
6 months
1 year
acute
NA
Follow-up
Improvement of 63.2% for TUG, 54% TUG-DT, 63.3%/58% 20-minutes walking with/without obstacle, 44.7% for PDQ-39, 38.3% for UPDRS III
Improvement in VAS pain score, and UPDRS score
>50% pain relief, improvement in walking posture, no change in bradykinesia Significant improvement in thoracolumbar spine flexion angle in both standing (10.98) and seating (8.18) positions
Results
NAS, Numeric Analog Scale; VAS, Visual Analog Scale; WPI, Widespread Pain Index; ADL, Activities of Daily Living; TUG, Timed-Up-Go; TUG-DT, Timed-Up-Go with Dual Task; PDQ, Parkinson’s Disease Questionnaire; PIGD, postural instability and gait disturbance; UPDRS, Unified Parkinson’s Disease Rating Scale; rTSMS, Repetitive Trans-Spinal Magnetic Stimulation; NA, not available.
50-85
37
Arii et al, 201438
NA
Disease duration, y
A N D
2
Patients’ age, y
Y A D A V
Mitsuyama et al, 2013104
Study
No. of patients
TABLE 1. Continued
351
N I C O L E L I S
352 S P I N A L
Clinical Studies in PD Patients Following our original report showing that highfrequency DCS alleviated akinesia and bradykinesia in rodent models of PD, Thevathasan and colleagues103 investigated the effect of DCS on motor function in 2 patients with advanced PD in a double-blind crossover study. Patient 1 was stimulated at 130 Hz, whereas patient 2 received 300 Hz, and the following assessments were made 10 days postoperatively (see Table 1) using the Unified Parkinsonâ&#x20AC;&#x2122;s Disease Rating Scale (UPDRS) motor scores, timed 10-meter walk, timed hand-arm movements, and timed lower limb in 3 different conditions (off-stimulation, suprathreshold stimulation, and subthreshold stimulation). In this study, DCS failed to provide improvement in all of the assessments. However, after closely analyzing the experimental design of this study, we argued that this clinical study failed to show significant improvements in PD symptoms mainly because of major differences in both the geometry of stimulation electrodes used and the chosen location of the implant along the spinal cord.105 Support for our contention followed soon after, when several clinical groups around the world began to report positive results with DCS in PD patients. For example, a 74-year-old man who had originally been implanted with a DCS electrode in the T9-T10 epidural area for treating back pain because of failed back surgery syndrome, reported increased tremor when the stimulator was turned off.39 Clinical examination revealed that the patient had developed PD-related tremor 8 years after the implantation of DCS. When DCS stimulation was turned on and a 130-Hz stimulus was employed, the patientâ&#x20AC;&#x2122;s UPDRSIII scores (evaluated during 4 separate sessions) improved by almost 50%. Tremor in the upper and lower limbs showed significant improvement during the stimulation-on condition while axial symptoms of gait and posture were also improved. In another case report, a 68-year-old woman implanted with DCS electrodes for postlaminectomy syndrome exhibited dramatic clinical improvements in her parkinsonian tremor in concurrence with relief from leg pain when DCS stimulation was delivered at 60 Hz.42 Another 43-year-old woman with a PD history was implanted with DCS electrodes at the C2 level for neuropathic pain in her neck and upper extremities.40 After years 1 and 2 of DCS treatment, using a stimulation frequency of 40 Hz, her motor UPDRS scores showed significant progressive improvements, leading to a significant reduction in tremor, rigidity, gait imbalance, and neuropathic pain. A total of 15 patients (5 men and 10 women), ranging in age from 63 to 79 years (mean 71.1 years), with complaints of low back and leg pain received DCS implants in the thoracic area T7 to T12,
C O R D
S T I M U L A T I O N
F O R
P D
depending on pain localization.37 Overall, 7 of these patients had previously undergone DBS in the STN. At 1 year after surgery, the pain scores and the UPDRS scores were significantly improved. There was a clear relief of gait disorder and postural instability, indicating that DCS had a remarkable effect on the axial symptoms associated with advanced PD. Another study by Arii and colleagues38 involving 37 PD patients with camptocormia (a treatment-resistant postural abnormality) demonstrated an immediate effect using repetitive trans-spinal magnetic stimulation in the lower thoracic to upper lumbar vertebral area. Landi and colleagues41 also reported significant improvement in gait and postural stability in a patient with advanced PD who had previously undergone STN DBS. Another advanced PD patient implanted with DCS electrodes for chronic back pain showed no improvement in UPDRS scores or locomotion.43 Mitsuyama and colleagues104 reported pain relief as well as significant improvement in walking posture in 2 patients with chronic lumbar pain and PD symptoms of abnormal posture, sagittal imbalance, and difficulty in locomotion. Similarly, 3 PD patients, who also suffered from intractable pain as a result of failed back surgery syndrome and lumbar canal stenosis, showed remarkable improvement in pain as well as rigidity and tremor reflected in their UPDRS III scores.107 Finally, in a more recent study108 conducted by a neurosurgical team at the University of S~ao Paulo, Brazil, in 4 long-term PD patients who had been subjected to DBS 7 to 8 years prior but still exhibited serious problems in locomotion including freezing, at the time of testing, 300 Hz DCS was found to induce a significant improvement in patient gait (Timed-Up-Go [TUG], 50%-65%) and the 20-minute walking test (50%60% on time and 65%-70% on the number of steps). A significant improvement in UPDRS III (38% OFF meds; P 5 .017), Freezing of Gait (FOG) (56%; P 5 .04), and Parkinsonâ&#x20AC;&#x2122;s Disease Questionnaire (PDQ)-39 (mobility 57%, P 5 .003; Activities of Daily Living (ADL) 28%, P 5 .02) was also documented. Based on the data described in the previous paragraph, it is fair to say that, 7 years after our initial report in rodents, a beneficial clinical effect of DCS on alleviating multiple parkinsonian symptoms has been consistently reported by several independent groups. These preliminary clinical studies validated our original proposal that DCS should be further examined as a potential new therapy for PD, particularly in cases involving gait problems and freezing. Encouraging as they are, however, it is important to emphasize that these initial clinical results involved a small and heterogeneous number of patients, tested in an open-label and heterogeneous fashion. For example, in 1 case the patients were >70 years old, whereas in 2 other cases they were aged 43 and 68 years.40,42,103 Studies varied
Movement Disorders, Vol. 00, No. 00, 2017
7
353 Y A D A V
A N D
N I C O L E L I S
locomotor region in the brain stem in the pathophysiology of PD—has gained considerable interest in recent times. Similar to striatal lesions, PPN lesions in rhesus monkeys have resulted in parkinsonian symptoms of bradykinesia, hypokinesia, and flexed posture.109,110 Stimulation of the PPN in PD patients has resulted in significant improvements in axial symptoms, such as gait imbalance and postural instability, suggesting that DCS may also exert its clinical effect by modulation of PPN activity.111-113 Alternatively, other studies have highlighted the moderate response to PPN-DBS during longer evaluation periods and its inadequacy as an alternate target for DBS when not used in conjunction with STN-DBS.27,114,115 However, the fact that PPN has reciprocal connections with the cortex, thalamus, basal ganglia, and spinal cord116 implies that modulation of PPN either by ascending pathways from the DCs or by indirect descending pathways from the cortex might play a crucial role in the alleviation of PD symptoms (Fig. 3).
The Spinal Cord as a Potential Common Neural Pathway for Treating Multiple Brain Disorders With Electrical Stimulation FIG. 3. Illustrative model of the motor circuitry involving the direct and indirect pathways of basal ganglia, thalamus, and cortex and also the pathways from the postural and gait control with projections from the basal ganglia to the pedunculopontine nucleus and the spinal cord. (adapted with permission; originally published in de Andrade et al.106). PPN, pedunculopontine nucleus; Str, striatum; GPe, globus pallidus external segment; GPi, globus pallidus internal segment; STN, subthalamic nucleus; SNc, substantia nigra pars compacta; SNr, substantia nigra pars reticulate.
in the type of electrodes used, stimulation parameters selected, number of sessions performed, and timeline of symptom evaluation. Moreover, most patients also exhibited a broad range of pain symptoms in addition to PD symptoms. Although most patients had relief from symptoms of pain, Thiriez and colleagues65 argue that the effect of DCS on PD motor symptoms in these multiple studies might depend on the patients’ initial quality of response to L-dopa therapy. Thus, although the initial human results seem promising (summarized in Table 1), large, randomized, doubleblind, clinical studies need to be performed to truly assess the potential therapeutic effects of DCS in PD and other motor disorders. It is also important to highlight that the effect of DCS on axial symptoms of PD was noticeable in more recent clinical studies.37,39,40 The role of the pedunculopontine nucleus (PPN)—a part of the mesencephalic
8
Movement Disorders, Vol. 00, No. 00, 2017
The spinal cord serves as a true “neural information highway” transmitting non-noxious as well as nociceptive sensory information from various parts of the body to supra-spinal structures in the brain. Although spinal cord stimulation has been used for the treatment of chronic pain for decades, its role as a channel to deliver therapeutic electrical signals to the cortex and other subcortical targets has never been explored.31 Traditionally, it has been assumed that, to cause cortical or subcortical neuronal modulation, the electrical stimulation has to be applied at the target site, which is evident in earlier approaches where intracortical microstimulation of areas 3b and 3a of the somatosensory cortex was used for creating artificial sensation of flutter and proprioception, respectively.117-119 However, more recently, thalamic stimulation or even peripheral nerve innervation have been used for the same purpose, suggesting that multiple locations along a neural pathway can be used for modulating the activity of the target of interest.120,121 Consistent with this argument, we had previously used electrical nerve stimulation to treat epileptic seizures.94 Recently we reported that closed-loop DCS can also be used to reduce the frequency and duration of seizures in a Pentylenetetrazol (PTZ) rat model of epilepsy, primarily by modulating the theta frequency oscillations in the somatosensory cortex.122 DCS has also been used for the remodeling of corticospinal projections in spinal cord injured rats, leading
354 S P I N A L
to partial recovery of locomotion.123 As mentioned previously, although functional magnetic resonance imaging studies revealed distinct modulation of cortical structures on the administration of DCS, it is also known to cause gene expression changes in upstream pathways.69,71 This suggests that the stimulation of the DCs does indeed result in the electrical signal being transmitted to multiple cortical and subcortical areas of great interest, from a therapeutic point of view, in the human brain. In addition, research on PD patients undergoing tango lessons has shown that the asymmetrical movements involved in this type of dancing could be providing the sensory feedback necessary for improvement in symptoms of balance and gait.124,125 Because proprioceptive signals run primarily through some of the largest myelinated axons that comprise the DC of the spinal cord, the observations that increased proprioceptive feedback is correlated with improvements in PD symptoms strongly support our claim that high-frequency electrical stimulation of the dorsal funiculus of the spinal cord could provide therapeutic effects for PD. Accordingly, future implementation of a closed-loop DCS, which allows intermittent and not only continuous DCS, may provide a better strategy to maximize this beneficial effect.
Pathological Brain States as Neuronal Timing Disorders Synchronization of neuronal oscillations is representative of the temporally precise interactions necessary for neural communication to occur between multiple areas in a neural circuit.126 High-frequency oscillations are observed in local neuronal areas, whereas slow oscillations recruit large networks. The relation between neural circuit architecture and oscillatory patterns is responsible for the dynamic establishment of normal brain operations carried out across spatial and temporal scales.127 Earlier work in human patients using EEG emphasized the role of neural synchrony in cognitive functions, such as attention dependent stimulus selection and working memory tasks that require integration of distributed neural activity.128,129 Although EEG recordings showed that synchronization of rhinal and hippocampal oscillations takes place during memory formation,130 using multielectrode recordings we have demonstrated that hippocampal and prefrontal cortical oscillations synchronize during spatial-cognitive processes.131 Ulhass and colleagues132,133 observed impaired neural synchrony in schizophrenia patients performing a cognitive task particularly reduced phase synchrony in the beta band (20-30 Hz), suggesting the relation between impaired neuronal synchrony and cognitive deficits associated
C O R D
S T I M U L A T I O N
F O R
P D
with schizophrenia. Previous work in our laboratory has shown that, although cross-structural synchronization of oscillations in the limbic circuit plays an important role in the manifestation of anxiety-related behaviors in healthy animals, deficits in the neuronal oscillations can lead to the development of behaviors that are typically characterized as bipolar mania disorder.134 We show that the nucleus accumbens–ventral tegmental area synchrony and nucleus accumbens– amygdala synchrony increases during specific phases of the elevated zero-maze task and that the correlation between synchrony and anxiety-related behaviors changes between healthy and diseased mice.134 EEG and functional magnetic resonance imaging studies of autistic children have also revealed reduced neural synchronization, characterized by a decreased functional connectivity between neural areas,135,136 particularly in the cortical language system during comprehension of sentences137 or between frontal and parietal areas during an executive task.138 Similarly, the reduction of alpha and beta band synchronization has been implicated in patients with Alzheimer’s disease during the resting state,139,140 and the reduction is correlated with the severity of the cognitive deficits.141 Pijnenburg and colleagues142 showed that in Alzheimer’s disease patients, alpha and beta band synchronization decreased during a working memory task when compared with controls. Moreover, our research has demonstrated that neuronal synchronization increases between the amygdala and medial prefrontal cortex circuits in a mouse model of depression.143 Likewise, an increase in mesolimbic cross-structural coherence was correlated with hyperactivity and stereotyped behaviors in a model of hyponoradrenergia.144 These results suggest that many neurological disorders are characterized by pathological levels of neuronal synchronization, occurring in distinct neural circuits involved in key brain functions. In fact, in our decade-long survey of transgenic mice models of neurological/psychiatric disorders, we have always identified a particular brain circuit in which pathological levels of neuronal oscillations or synchronization can be identified and found to correlate with the behavior abnormalities exhibited by each mouse strain. This led us to postulate that, independent of their specific etiology (eg, genetic, specific cellular degeneration, etc.), a large number of intrinsic brain disorders may produce abnormal motor, sensory, cognitive, and emotional outcomes as a result of fundamental disturbances in neuronal timing, taking place in specific brain circuits. Based on this view, we propose that any technique that can reset the pathological neuronal timing of a given neural circuit, bringing it close to a normal dynamic state, may be capable of ameliorating the symptoms, and hence the brain pathology, created by
Movement Disorders, Vol. 00, No. 00, 2017
9
355 Y A D A V
A N D
N I C O L E L I S
such electrophysiological disturbance. Support for this contention came from the observation that neuronal activity recorded in both PD animal models and parkinsonian patients often resembles the type of pathological neuronal hyper-synchronization that is prominent during epileptic seizures.145,146 Our results with animal PD models and, more recently in an animal model of epilepsy,122 indicate that DCS may achieve its major therapeutic effects by disrupting ongoing supraspinal pathological synchronization by simultaneous activation of DC fibers in the spinal cord. Based on a decade of experimental work in animal models of brain disorders, we envision that the spinal cord “neural super highway” may become an optimal route to deliver different types of electrical, optical, and even noninvasive magnetic stimulation to manipulate or even abolish pathological neuronal activity in supraspinal structures. This approach could trigger the establishment of a complete new repertoire of nonpharmacological, low-cost, and less invasive, neurophysiologically inspired therapeutic options for treating a variety of neurological and even psychiatric diseases for which we have very few efficient treatments available today.
Concluding Remarks After the original experimental findings in animal models of PD carried out in the late 2000s, multiple preliminary studies have reported the beneficial clinical effects of DC stimulation in PD patients. DCS effects are not only significant in motor symptoms such as akinesia, bradykinesia, and tremor, but they also provide significant alleviation of axial symptoms, gait and posture impairment, and “freezing,” a range of problems that are difficult to treat with current therapeutic options, such as DBS. Current experimental evidence from our laboratory suggests that DCS modulates activity of subcortical and cortical structures by activating ascending DC fibers of the spinal cord. Although the precise mechanism is not yet clear, high-frequency continuous DCS appears to disrupt the pathological low-frequency hypersynchronized neuronal firing observed in the basal ganglia and motor cortex, which has been associated with PD. Indeed, the neurophysiological effects observed with DCS in experimental animals mimic those observed during dopamine-replacement therapy. In addition, in rodents, DCS also has produced a long-term protective effect on the degeneration of dopaminergic neurons, making it a promising treatment option for early-stage PD along with traditional levodopa therapy. Because of its semi-invasive nature, we propose that in the future DCS may become a potential therapeutic alternative for DBS.
10
Movement Disorders, Vol. 00, No. 00, 2017
References 1.
Fahn S. Description of Parkinson’s disease as a clinical syndrome. Ann N Y Acad Sci 2003;991:1-14.
2.
Goetz CG. The history of Parkinson’s disease: early clinical descriptions and neurological therapies. Cold Spring Harb Perspect Med 2011;1(1):a008862.
3.
Stein E, Bar-Gad I. beta oscillations in the cortico-basal ganglia loop during parkinsonism. Exp Neurol 2013;245:52-59.
4.
Carlsson A. Biochemical and pharmacological aspects of Parkinsonism. Acta Neurol Scand 1972;51(suppl):11-42.
5.
Dorsey ER, Constantinescu R, Thompson JP, et al. Projected number of people with Parkinson disease in the most populous nations, 2005 through 2030. Neurology 2007;68(5):384-386.
6.
Schapira AH, Olanow CW. Neuroprotection in Parkinson disease: mysteries, myths, and misconceptions. JAMA 2004;291(3):358364.
7.
Hornykiewicz O. L-DOPA: from a biologically inactive amino acid to a successful therapeutic agent. Amino Acids 2002;23(13):65-70.
8.
Lloyd KG, Davidson L, Hornykiewicz O. The neurochemistry of Parkinson’s disease: effect of L-dopa therapy. J Pharmacol Exp Ther 1975;195(3):453-464.
9.
Shaw KM, Lees AJ, Stern GM. The impact of treatment with levodopa on Parkinson’s disease. Q J Med 1980;49(195):283-293.
10.
Marsden CD, Parkes JD. “On-off” effects in patients with Parkinson’s disease on chronic levodopa therapy. Lancet 1976;1(7954): 292-296.
11.
Rinne UK. Problems associated with long-term levodopa treatment of Parkinson’s disease. Acta Neurol Scand 1983;95(suppl): 19-26.
12.
Nagatsua T, Sawadab M. L-dopa therapy for Parkinson’s disease: past, present, and future. Parkinsonism Relat Disord 2009; 15(suppl 1):S3-S8.
13.
Rascol O, Payoux P, Ory F, Ferreira JJ, Brefel-Courbon C, Montastruc JL. Limitations of current Parkinson’s disease therapy. Ann Neurol 2003;53(suppl 3):S3-S12; discussion S-5.
14.
Benabid AL. Deep brain stimulation for Parkinson’s disease. Curr Opin Neurobiol 2003;13(6):696-706.
15.
The Deep-Brain Stimulation for Parkinson’s Disease Study. Deepbrain stimulation of the subthalamic nucleus or the pars interna of the globus pallidus in Parkinson’s disease. N Engl J Med 2001; 345(13):956-963.
16.
Kumar R, Lozano AM, Sime E, Lang AE. Long-term follow-up of thalamic deep brain stimulation for essential and parkinsonian tremor. Neurology 2003;61(11):1601-1604.
17.
Lozano AM, Dostrovsky J, Chen R, Ashby P. Deep brain stimulation for Parkinson’s disease: disrupting the disruption. Lancet Neurol 2002;1(4):225-231.
18.
Beric A, Kelly PJ, Rezai A, et al. Complications of deep brain stimulation surgery. Stereotact Funct Neurosurg 2001;77(1-4):73-78.
19.
Hariz MI. Complications of deep brain stimulation surgery. Mov Disord 2002;17(suppl 3):S162-S166.
20.
Fenoy AJ, Simpson RK Jr. Management of device-related wound complications in deep brain stimulation surgery. J Neurosurg 2012;116(6):1324-1332.
21.
Okun MS. Deep-brain stimulation for Parkinson’s disease. N Engl J Med 2012;367(16):1529-1538.
22.
Sillay KA, Larson PS, Starr PA. Deep brain stimulator hardwarerelated infections: incidence and management in a large series. Neurosurgery 2008;62(2):360-366; discussion 6-7.
23.
Zrinzo L, Foltynie T, Limousin P, Hariz MI. Reducing hemorrhagic complications in functional neurosurgery: a large case series and systematic literature review. J Neurosurg 2012;116(1): 84-94.
24.
Krack P, Batir A, Van Blercom N, et al. Five-year follow-up of bilateral stimulation of the subthalamic nucleus in advanced Parkinson’s disease. N Engl J Med 2003;349(20):1925-1934.
25.
Morgante L, Morgante F, Moro E, et al. How many parkinsonian patients are suitable candidates for deep brain stimulation of subthalamic nucleus? Results of a questionnaire. Parkinsonism Relat Disord 2007;13(8):528-531.
356 S P I N A L
C O R D
S T I M U L A T I O N
F O R
P D
26.
Pizzolato G, Mandat T. Deep brain stimulation for movement disorders. Front Integr Neurosci 2012;6:2.
50.
Melzack R, Wall PD. Pain mechanisms: a new theory. Science 1965;150(3699):971-979.
27.
Ferraye MU, Debu B, Fraix V, et al. Effects of pedunculopontine nucleus area stimulation on gait disorders in Parkinson’s disease. Brain 2010;133(Pt 1):205-214.
51.
Linderoth B, Foreman RD. Physiology of spinal cord stimulation: review and update. Neuromodulation 1999;2(3):150-164.
52.
28.
Perlmutter JS, Mink JW. Deep brain stimulation. Annu Rev Neurosci 2006;29:229-257.
Oakley JC, Prager JP. Spinal cord stimulation: mechanisms of action. Spine (Phila Pa 1976). 2002;27(22):2574-2583.
53.
29.
Comella CL, Stebbins GT, Brown-Toms N, Goetz CG. Physical therapy and Parkinson’s disease: a controlled clinical trial. Neurology 1994;44(3 Pt 1):376-378.
D’Mello R, Dickenson AH. Spinal cord mechanisms of pain. Br J Anaesth 2008;101(1):8-16.
54.
Kwakkel G, de Goede CJ, van Wegen EE. Impact of physical therapy for Parkinson’s disease: a critical review of the literature. Parkinsonism Relat Disord 2007;13(suppl 3):S478-S487.
Meyerson BA, Linderoth B. Mode of action of spinal cord stimulation in neuropathic pain. J Pain Symptom Manage 2006;31(4 suppl):S6-S12.
55.
Eliasson T, Albertsson P, Hardhammar P, Emanuelsson H, Augustinsson LE, Mannheimer C. Spinal cord stimulation in angina pectoris with normal coronary arteriograms. Coron Artery Dis 1993;4(9):819-827.
56.
Broseta J, Garcia-March G, Sanchez-Ledesma MJ, Barbera J, Gonzalez-Darder J. High-frequency cervical spinal cord stimulation in spasticity and motor disorders. Acta Neurochir Suppl (Wien) 1987;39:106-111.
57.
Dieckmann G, Veras G. Bipolar spinal cord stimulation for spasmodic torticollis. Appl Neurophysiol 1985;48(1-6):339-346.
58.
Gildenberg PL. Treatment of spasmodic torticollis by dorsal column stimulation. Appl Neurophysiol 1978;41(1-4):113-121.
59.
Waltz JM, Andreesen WH, Hunt DP. Spinal cord stimulation and motor disorders. Pacing Clin Electrophysiol 1987;10(1 Pt 2):180204.
60.
Cook AW, Weinstein SP. Chronic dorsal column stimulation in multiple sclerosis. Preliminary report. N Y State J Med 1973; 73(24):2868-2872.
61.
Fredriksen TA, Bergmann S, Hesselberg JP, Stolt-Nielsen A, Ringkjob R, Sjaastad O. Electrical stimulation in multiple sclerosis. Comparison of transcutaneous electrical stimulation and epidural spinal cord stimulation. Appl Neurophysiol 1986;49(1-2): 4-24.
30.
31.
Compton AK, Shah B, Hayek SM. Spinal cord stimulation: a review. Curr Pain Headache Rep 2012;16(1):35-42.
32.
Fuentes R, Petersson P, Siesser WB, Caron MG, Nicolelis MA. Spinal cord stimulation restores locomotion in animal models of Parkinson’s disease. Science 2009;323(5921):1578-1582.
33.
Santana MB, Halje P, Simplicio H, et al. Spinal cord stimulation alleviates motor deficits in a primate model of Parkinson disease. Neuron 2014;84(4):716-722.
34.
Yadav AP, Fuentes R, Zhang H, et al. Chronic spinal cord electrical stimulation protects against 6-hydroxydopamine lesions. Sci Rep 2014;4:3839.
35.
Brys I, Bobela W, Schneider BL, Aebischer P, Fuentes R. Spinal cord stimulation improves forelimb use in an alpha-synuclein animal model of Parkinson’s disease. Int J Neurosci 2017;127: 28-36.
36.
Shinko A, Agari T, Kameda M, et al. Spinal cord stimulation exerts neuroprotective effects against experimental Parkinson’s disease. PLoS ONE 2014;9(7):e101468.
37.
Agari T, Date I. Spinal cord stimulation for the treatment of abnormal posture and gait disorder in patients with Parkinson’s disease. Neurol Med Chir (Tokyo) 2012;52(7):470-474.
38.
Arii Y, Sawada Y, Kawamura K, et al. Immediate effect of spinal magnetic stimulation on camptocormia in Parkinson’s disease. J Neurol Neurosurg Psychiatry 2014;85(11):1221-1226.
62.
Krauss JK, Weigel R, Blahak C, et al. Chronic spinal cord stimulation in medically intractable orthostatic tremor. J Neurol Neurosurg Psychiatry 2006;77(9):1013-1016.
39.
Fenelon G, Goujon C, Gurruchaga JM, et al. Spinal cord stimulation for chronic pain improved motor function in a patient with Parkinson’s disease. Parkinsonism Relat Disord 2012;18(2):213214.
63.
Raina GB, Piedimonte F, Micheli F. Posterior spinal cord stimulation in a case of painful legs and moving toes. Stereotact Funct Neurosurg 2007;85(6):307-309.
64.
40.
Hassan S, Amer S, Alwaki A, Elborno A. A patient with Parkinson’s disease benefits from spinal cord stimulation. J Clin Neurosci 2013;20(8):1155-1156.
Takahashi H, Saitoh C, Iwata O, Nanbu T, Takada S, Morita S. Epidural spinal cord stimulation for the treatment of painful legs and moving toes syndrome. Pain 2002;96(3):343-345.
65.
41.
Landi A, Trezza A, Pirillo D, Vimercati A, Antonini A, Sganzerla EP. Spinal cord stimulation for the treatment of sensory symptoms in advanced Parkinson’s disease. Neuromodulation 2013; 16(3):276-279.
Thiriez C, Gurruchaga JM, Goujon C, Fenelon G, Palfi S. Spinal stimulation for movement disorders. Neurotherapeutics 2014; 11(3):543-552.
66.
Holsheimer J. Which neuronal elements are activated directly by spinal cord stimulation. Neuromodulation 2002;5(1):25-31.
67.
Kiriakopoulos ET, Tasker RR, Nicosia S, Wood ML, Mikulis DJ. Functional magnetic resonance imaging: a potential tool for the evaluation of spinal cord stimulation: technical case report. Neurosurgery 1997;41(2):501-504.
68.
Rasche D, Siebert S, Stippich C, et al. Spinal cord stimulation in failed-back-surgery-syndrome. Preliminary study for the evaluation of therapy by functional magnetic resonance imaging (fMRI). Schmerz 2005;19(6):497-500, 2-5.
69.
Stancak A, Kozak J, Vrba I, et al. Functional magnetic resonance imaging of cerebral activation during spinal cord stimulation in failed back surgery syndrome patients. Eur J Pain 2008;12(2): 137-148.
70.
Dejongste MJ, Hautvast RW, Ruiters MH, Ter Horst GJ. Spinal cord stimulation and the induction of c-fos and heat shock protein 72 in the central nervous system of rats. Neuromodulation 1998;1(2):73-84.
42.
Soltani F, Lalkhen A. Improvement of Parkinsonian symptoms with spinal cord stimulation: consequence or coincidence? J Neurol Neurosurg Psychiatry 2013;84(11):71.
43.
Weise D, Winkler D, Meixensberger J, Classen J. Effects of spinal cord stimulation in a patient with Parkinson’s disease and chronic back pain. J Neurol 2010;257:S217.
44.
Shealy CN, Mortimer JT, Reswick JB. Electrical inhibition of pain by stimulation of the dorsal columns: preliminary clinical report. Anesth Analg 1967;46(4):489-491.
45.
North RB, Ewend MG, Lawton MT, Piantadosi S. Spinal cord stimulation for chronic, intractable pain: superiority of “multichannel” devices. Pain 1991;44(2):119-130.
46.
Turner JA, Loeser JD, Bell KG. Spinal cord stimulation for chronic low back pain: a systematic literature synthesis. Neurosurgery 1995;37(6):1088-1095; discussion 95-96.
47.
Aly MM, Saitoh Y, Hosomi K, Oshino S, Kishima H, Yoshimine T. Spinal cord stimulation for central poststroke pain. Neurosurgery. 2010;67(3 suppl Operative):ons206-212; discussion ons12.
71.
Cioni B, Meglio M, Pentimalli L, Visocchi M. Spinal cord stimulation in the treatment of paraplegic pain. J Neurosurg 1995; 82(1):35-39.
Maeda Y, Ikeuchi M, Wacnik P, Sluka KA. Increased c-fos immunoreactivity in the spinal cord and brain following spinal cord stimulation is frequency-dependent. Brain Res 2009;1259:40-50.
72.
Haber SN. The primate basal ganglia: parallel and integrative networks. J Chem Neuroanat 2003;26(4):317-330.
73.
Alonso-Frech F, Zamarbide I, Alegre M, et al. Slow oscillatory activity and levodopa-induced dyskinesias in Parkinson’s disease. Brain 2006;129(Pt 7):1748-1757.
48.
49.
Meglio M, Cioni B, Prezioso A, Talamonti G. Spinal cord stimulation (SCS) in deafferentation pain. Pacing Clin Electrophysiol 1989;12(4 Pt 2):709-712.
Movement Disorders, Vol. 00, No. 00, 2017
11
357 Y A D A V
A N D
N I C O L E L I S
74.
Hammond C, Bergman H, Brown P. Pathological synchronization in Parkinson’s disease: networks, models and treatments. Trends Neurosci 2007;30(7):357-364.
75.
Weinberger M, Mahant N, Hutchison WD, et al. Beta oscillatory activity in the subthalamic nucleus and its relation to dopaminergic response in Parkinson’s disease. J Neurophysiol 2006;96(6): 3248-3256.
76.
Wichmann T, Soares J. Neuronal firing before and after burst discharges in the monkey basal ganglia is predictably patterned in the normal state and altered in parkinsonism. J Neurophysiol. 2006;95(4):2120-2133.
77.
Bergman H, Wichmann T, Karmon B, DeLong MR. The primate subthalamic nucleus. II. Neuronal activity in the MPTP model of parkinsonism. J Neurophysiol 1994;72(2):507-520.
78.
Wichmann T, DeLong MR. Models of basal ganglia function and pathophysiology of movement disorders. Neurosurg Clin N Am 1998;9(2):223-236.
79.
Mallet N, Pogosyan A, Sharott A, et al. Disrupted dopamine transmission and the emergence of exaggerated beta oscillations in subthalamic nucleus and cerebral cortex. J Neurosci 2008; 28(18):4795-4806.
80.
Sharott A, Magill PJ, Harnack D, Kupsch A, Meissner W, Brown P. Dopamine depletion increases the power and coherence of beta-oscillations in the cerebral cortex and subthalamic nucleus of the awake rat. Eur J Neurosci 2005;21(5):1413-1422.
81.
Avila I, Parr-Brownlie LC, Brazhnik E, Castaneda E, Bergstrom DA, Walters JR. Beta frequency synchronization in basal ganglia output during rest and walk in a hemiparkinsonian rat. Exp Neurol 2010;221(2):307-319.
82.
Costa RM, Lin SC, Sotnikova TD, et al. Rapid alterations in corticostriatal ensemble coordination during acute dopaminedependent motor dysfunction. Neuron 2006;52(2):359-369.
83.
Brown P, Oliviero A, Mazzone P, Insola A, Tonali P, Di Lazzaro V. Dopamine dependency of oscillations between subthalamic nucleus and pallidum in Parkinson’s disease. J Neurosci 2001; 21(3):1033-1038.
84.
Williams D, Tijssen M, Van Bruggen G, et al. Dopamine-dependent changes in the functional connectivity between basal ganglia and cerebral cortex in humans. Brain 2002;125(Pt 7):1558-1569.
85.
Heimer G, Rivlin-Etzion M, Bar-Gad I, Goldberg JA, Haber SN, Bergman H. Dopamine replacement therapy does not restore the full spectrum of normal pallidal activity in the 1-methyl-4-phenyl1,2,3,6-tetra-hydropyridine primate model of Parkinsonism. J Neurosci 2006;26(31):8101-8114.
94.
Fanselow EE, Reid AP, Nicolelis MA. Reduction of pentylenetetrazole-induced seizure activity in awake rats by seizure-triggered trigeminal nerve stimulation. J Neurosci 2000; 20(21):8160-8168.
95.
Klein RL, Lewis MH, Muzyczka N, Meyer EM. Prevention of 6hydroxydopamine-induced rotational behavior by BDNF somatic gene transfer. Brain Res 1999;847(2):314-320.
96.
Kordower JH, Emborg ME, Bloch J, et al. Neurodegeneration prevented by lentiviral vector delivery of GDNF in primate models of Parkinson’s disease. Science 2000;290(5492):767-773.
97.
Shults CW, Kimber T, Altar CA. BDNF attenuates the effects of intrastriatal injection of 6-hydroxydopamine. Neuroreport 1995; 6(8):1109-1112.
98.
Shults CW, Kimber T, Martin D. Intrastriatal injection of GDNF attenuates the effects of 6-hydroxydopamine. Neuroreport 1996; 7(2):627-631.
99.
Sun M, Kong L, Wang X, Lu XG, Gao Q, Geller AI. Comparison of the capability of GDNF, BDNF, or both, to protect nigrostriatal neurons in a rat model of Parkinson’s disease. Brain Res 2005; 1052(2):119-129.
100.
Spieles-Engemann AL, Behbehani MM, Collier TJ, et al. Stimulation of the rat subthalamic nucleus is neuroprotective following significant nigral dopamine neuron loss. Neurobiol Dis 2010; 39(1):105-115.
101.
Spieles-Engemann AL, Steece-Collier K, Behbehani MM, et al. Subthalamic nucleus stimulation increases brain derived neurotrophic factor in the nigrostriatal system and primary motor cortex. J Parkinsons Dis 2011;1(1):123-136.
102.
Hilker R, Portman AT, Voges J, et al. Disease progression continues in patients with advanced Parkinson’s disease and effective subthalamic nucleus stimulation. J Neurol Neurosurg Psychiatry 2005;76(9):1217-1221.
103.
Thevathasan W, Mazzone P, Jha A, et al. Spinal cord stimulation failed to relieve akinesia or restore locomotion in Parkinson disease. Neurology 2010;74(16):1325-1327.
104.
Mitsuyama T, Goto S, Sasaki T, Taira T., Okada, Y. Spinal cord simulation for chronic lumbar pain in patients with Parkinson’s disease. Stereotact Funct Neurosurg 2013;91:273-273.
105.
Fuentes R, Petersson P, Nicolelis MA. Restoration of locomotive function in Parkinson’s disease by spinal cord stimulation: mechanistic approach. Eur J Neurosci 2010;32(7):1100-1108.
106.
de Andrade EM, Ghilardi MG, Cury RG, et al. Spinal cord stimulation for Parkinson’s disease: a systematic review. Neurosurg Rev 2016;39(1):27-35.
107.
Nishioka K, Nakajima M. Beneficial therapeutic effects of spinal cord stimulation in advanced cases of Parkinson’s disease with intractable chronic pain: a case series. Neuromodulation 2015; 18(8):751-753.
86.
Wichmann T, Bergman H, DeLong MR. The primate subthalamic nucleus. III. Changes in motor behavior and neuronal activity in the internal pallidum induced by subthalamic inactivation in the MPTP model of parkinsonism. J Neurophysiol 1994;72(2):521530.
87.
Giannicola G, Marceglia S, Rossi L, et al. The effects of levodopa and ongoing deep brain stimulation on subthalamic beta oscillations in Parkinson’s disease. Exp Neurol 2010;226(1):120-127.
108.
de Souza CP, Hamani C, Souza CO, et al. Spinal cord stimulation improves gait in patients with Parkinson’s disease previously treated with deep brain stimulation. Mov Disord 2017;32(2):278-282.
88.
Quinn EJ, Blumenfeld Z, Velisar A, et al. Beta oscillations in freely moving Parkinson’s subjects are attenuated during deep brain stimulation. Mov Disord 2015;30(13):1750-1758.
109.
Aziz TZ, Davies L, Stein J, France S. The role of descending basal ganglia connections to the brain stem in parkinsonian akinesia. Br J Neurosurg 1998;12(3):245-249.
89.
Eusebio A, Thevathasan W, Gaynor LD, et al. Deep brain stimulation can suppress pathological synchronisation in parkinsonian patients. J Neurol Neurosurg Psychiatry 2011;82(5):569-573.
110.
Kojima J, Yamaji Y, Matsumura M, et al. Excitotoxic lesions of the pedunculopontine tegmental nucleus produce contralateral hemiparkinsonism in the monkey. Neurosci Lett 1997;226(2):111-114.
90.
Whitmer D, de Solages C, Hill B, Yu H, Henderson JM, BronteStewart H. High frequency deep brain stimulation attenuates subthalamic and cortical rhythms in Parkinson’s disease. Front Hum Neurosci 2012;6:155.
111.
Ferraye MU, Debu B, Pollak P. Deep brain stimulation effect on freezing of gait. Mov Disord 2008;23(suppl 2):S489-S494.
112.
Khan S, Gill SS, Mooney L, et al. Combined pedunculopontinesubthalamic stimulation in Parkinson disease. Neurology 2012; 78(14):1090-1095.
113.
Plaha P, Gill SS. Bilateral deep brain stimulation of the pedunculopontine nucleus for Parkinson’s disease. Neuroreport 2005; 16(17):1883-1887.
114.
Moro E, Hamani C, Poon YY, et al. Unilateral pedunculopontine stimulation improves falls in Parkinson’s disease. Brain 2010; 133(Pt 1):215-224.
115.
Stefani A, Lozano AM, Peppe A, et al. Bilateral deep brain stimulation of the pedunculopontine and subthalamic nuclei in severe Parkinson’s disease. Brain 2007;130:1596-1607.
116.
Pahapill PA, Lozano AM. The pedunculopontine nucleus and Parkinson’s disease. Brain 2000;123(Pt 9):1767-1783.
91.
Kuhn AA, Kempf F, Brucke C, et al. High-frequency stimulation of the subthalamic nucleus suppresses oscillatory beta activity in patients with Parkinson’s disease in parallel with improvement in motor performance. J Neurosci 2008;28(24):6165-6173.
92.
Kuhn AA, Kupsch A, Schneider GH, Brown P. Reduction in subthalamic 8-35 Hz oscillatory activity correlates with clinical improvement in Parkinson’s disease. Eur J Neurosci 2006;23(7): 1956-1960.
93.
Ray NJ, Jenkinson N, Wang S, et al. Local field potential beta activity in the subthalamic nucleus of patients with Parkinson’s disease is associated with improvements in bradykinesia after dopamine and deep brain stimulation. Exp Neurol 2008;213(1): 108-113.
12
Movement Disorders, Vol. 00, No. 00, 2017
358 S P I N A L
C O R D
S T I M U L A T I O N
F O R
P D
133.
Uhlhaas PJ, Singer W. Neural synchrony in brain disorders: relevance for cognitive dysfunctions and pathophysiology. Neuron 2006;52(1):155-168.
134.
Dzirasa K, McGarity DL, Bhattacharya A, et al. Impaired limbic gamma oscillatory synchrony during anxiety-related behavior in a genetic mouse model of bipolar mania. J Neurosci 2011;31(17): 6449-6456.
Romo R, Hernandez A, Zainos A, Salinas E. Somatosensory discrimination based on cortical microstimulation. Nature 1998; 392(6674):387-390.
135.
Belmonte MK, Allen G, Beckel-Mitchener A, Boulanger LM, Carper RA, Webb SJ. Autism and abnormal development of brain connectivity. J Neurosci 2004;24(42):9228-9231.
120.
Heming E, Sanden A, Kiss ZH. Designing a somatosensory neural prosthesis: percepts evoked by different patterns of thalamic stimulation. J Neural Eng 2010;7(6):064001.
136.
Brock J, Brown CC, Boucher J, Rippon G. The temporal binding deficit hypothesis of autism. Dev Psychopathol 2002;14(2):209224.
121.
Marasco PD, Schultz AE, Kuiken TA. Sensory capacity of reinnervated skin after redirection of amputated upper limb nerves to the chest. Brain 2009;132(Pt 6):1441-1448.
137.
122.
Pais-Vieira M, Yadav AP, Moreira D, et al. A closed loop brainmachine interface for epilepsy control using dorsal column electrical stimulation. Sci Rep 2016;6:32814.
Just MA, Cherkassky VL, Keller TA, Minshew NJ. Cortical activation and synchronization during sentence comprehension in high-functioning autism: evidence of underconnectivity. Brain 2004;127:1811-1821.
138.
Just MA, Cherkassky VL, Keller TA, Kana RK, Minshew NJ. Functional and anatomical cortical underconnectivity in autism: evidence from an fMRI study of an executive function task and corpus callosum morphometry. Cerebral Cortex 2007;17(4):951961.
139.
Stam CJ, Montez T, Jones BF, et al. Disturbed fluctuations of resting state EEG synchronization in Alzheimer’s disease. Clin Neurophysiol 2005;116(3):708-715.
140.
Stam CJ, van der Made Y, Pijnenburg YAL, Scheltens P. EEG synchronization in mild cognitive impairment and Alzheimer’s disease. Acta Neurologica Scandinavica 2003;108(2): 90-96.
117.
London BM, Jordan LR, Jackson CR, Miller LE. Electrical stimulation of the proprioceptive cortex (area 3a) used to instruct a behaving monkey. IEEE Trans Neural Syst Rehabil Eng 2008; 16(1):32-36.
118.
Romo R, Hernandez A, Zainos A, Brody CD, Lemus L. Sensing without touching: psychophysical performance based on cortical microstimulation. Neuron 2000;26(1):273-278.
119.
123.
van den Brand R, Heutschi J, Barraud Q, et al. Restoring voluntary control of locomotion after paralyzing spinal cord injury. Science 2012;336(6085):1182-1185.
124.
Duncan RP, Earhart GM. Randomized controlled trial of community-based dancing to modify disease progression in Parkinson disease. Neurorehabil Neural Repair 2012;26(2):132-143.
125.
Hackney ME, Earhart GM. Effects of dance on movement control in Parkinson’s disease: a comparison of Argentine tango and American ballroom. J Rehabil Med 2009;41(6):475-481.
126.
Schnitzler A, Gross J. Normal and pathological oscillatory communication in the brain. Nature Reviews Neuroscience 2005;6(4):285-296.
141.
Jeong J. EEG dynamics in patients with Alzheimer’s disease. Clin Neurophysiol 2004;115(7):1490-1505.
127.
Buzsaki G, Draguhn A. Neuronal oscillations in cortical networks. Science 2004;304(5679):1926-1929.
142.
128.
Singer W. Neuronal synchrony: a versatile code for the definition of relations? Neuron 1999;24(1):49-65.
Pijnenburg YAL, Made YV, van Walsum AMV, Knol DL, Scheltens P, Stam CJ. EEG synchronization likelihood in mild cognitive impairment and Alzheimer’s disease during a working memory task. Clin Neurophysiol 2004;115(6):1332-1339.
129.
Varela F, Lachaux JP, Rodriguez E, Martinerie J. The brainweb: phase synchronization and large-scale integration. Nature Reviews Neuroscience 2001;2(4):229-239.
143.
Dzirasa K, Kumar S, Sachs BD, Caron MG, Nicolelis MA. Cortical-amygdalar circuit dysfunction in a genetic mouse model of serotonin deficiency. J Neurosci 2013;33(10):4505-4513.
130.
Fell J, Klaver P, Lehnertz K, et al. Human memory formation is accompanied by rhinal-hippocampal coupling and decoupling. Nat Neurosci 2001;4(12):1259-1264.
144.
Dzirasa K, Phillips HW, Sotnikova TD, et al. Noradrenergic control of cortico-striato-thalamic and mesolimbic cross-structural synchrony. J Neurosci 2010;30(18):6387-6397.
131.
Dzirasa K, Ramsey AJ, Takahashi DY, et al. Hyperdopaminergia and NMDA receptor hypofunction disrupt neural phase signaling. J Neurosci 2009;29(25):8215-8224.
145.
Brown P. Oscillatory nature of human basal ganglia activity: relationship to the pathophysiology of Parkinson’s disease. Mov Disord 2003;18(4):357-363.
132.
Uhlhaas PJ, Linden DEJ, Singer W, et al. Dysfunctional longrange coordination of neural activity during Gestalt perception in schizophrenia. J Neurosci 2006;26(31):8168-8175.
146.
Jiruska P, de Curtis M, Jefferys JG, Schevon CA, Schiff SJ, Schindler K. Synchronization and desynchronization in epilepsy: controversies and hypotheses. J Physiol 2013;591(4):787-797.
Movement Disorders, Vol. 00, No. 00, 2017
13
359
A Comprehensive Review of the BMI Field
360 Physiol Rev 97: 767– 837, 2017 Published March 8, 2017; doi:10.1152/physrev.00027.2016
BRAIN-MACHINE INTERFACES: FROM BASIC SCIENCE TO NEUROPROSTHESES AND NEUROREHABILITATION Mikhail A. Lebedev and Miguel A. L. Nicolelis Duke University, Durham, North Carolina
L
I. II. III. IV. V. VI. VII. VIII. IX. X. XI. XII. XIII.
INTRODUCTION HISTORY OF BMI RESEARCH BMI CLASSIFICATION REPRESENTATION OF INFORMATION... MULTICHANNEL RECORDING... DECODING OF BRAIN SIGNALS MOTOR CONTROL WITH... NONINVASIVE BMIS BMIS WITH ARTIFICIAL SENSATIONS COGNITIVE BMIS BRAIN-TO-BRAIN INTERFACES... BMI AS A POTENTIAL... CONCLUSION
767 772 779 782 786 792 795 804 806 810 811 814 817
I. INTRODUCTION A. From a Single Neuron to Neural Ensembles The daunting task of unraveling the physiological mechanisms that account for the operation of the human brain, a highly complex and self-adaptive biological system, formed by ⬃100 billion interconnected neurons (37), has become
the true holy grail of neuroscience since the first images of brain circuits were produced, more than 100 years ago, by the skillful hands of Santiago Ramon y Cajal (256, 658). By the time Cajal’s histological approach was complemented by its electrophysiological counterpart, Sir Edgar Adrian’s metal microelectrode approach to record the electrical pulses produced by individual neurons (2), the neuron doctrine (732) had become established as the fundamental theory in the emergent field of brain research. This doctrine purports that individual neurons work as the functional unit of the brain through processing and transmission of electrophysiological signals. Despite the undeniable success of the neuron doctrine, since the origins of modern neuroscience, researchers entertained physiological models of brain function in which populations of neurons performed the fundamental job of generating functions and behaviors. Indeed, the Italian anatomist, Camillo Golgi, who shared the 1906 Nobel Prize in Medicine and Physiology with Ramon y Cajal, was the first to introduce the term neural network, as a way to describe the underlying “functional module” of brain operation proposed in his reticular theory (308). According to Golgi’s view, brain tissue should work pretty much like the heart
0031-9333/17 Copyright © 2017 the American Physiological Society
767
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Lebedev MA, Nicolelis MAL. Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation. Physiol Rev 97: 767– 837, 2017. Published March 8, 2017; doi:10.1152/physrev.00027.2016.—Brain-machine interfaces (BMIs) combine methods, approaches, and concepts derived from neurophysiology, computer science, and engineering in an effort to establish real-time bidirectional links between living brains and artificial actuators. Although theoretical propositions and some proof of concept experiments on directly linking the brains with machines date back to the early 1960s, BMI research only took off in earnest at the end of the 1990s, when this approach became intimately linked to new neurophysiological methods for sampling large-scale brain activity. The classic goals of BMIs are 1) to unveil and utilize principles of operation and plastic properties of the distributed and dynamic circuits of the brain and 2) to create new therapies to restore mobility and sensations to severely disabled patients. Over the past decade, a wide range of BMI applications have emerged, which considerably expanded these original goals. BMI studies have shown neural control over the movements of robotic and virtual actuators that enact both upper and lower limb functions. Furthermore, BMIs have also incorporated ways to deliver sensory feedback, generated from external actuators, back to the brain. BMI research has been at the forefront of many neurophysiological discoveries, including the demonstration that, through continuous use, artificial tools can be assimilated by the primate brain’s body schema. Work on BMIs has also led to the introduction of novel neurorehabilitation strategies. As a result of these efforts, long-term continuous BMI use has been recently implicated with the induction of partial neurological recovery in spinal cord injury patients.
361
LEBEDEV AND NICOLELIS
Even though Hebb’s masterpiece work, The Organization of Behavior (352), published in 1949, launched the modern era of neural population coding in systems neuroscience, few took notice at the time of its publication. By then, neurophysiologists were primarily engaged in characterizing the physiological properties of individual neurons, either using the classical Adrian’s approach of extracellular single neuron recording (2, 372) or a new methodological breakthrough of that time: intracellular single neuron recordings using sharp glass electrodes (92, 503, 781). Certainly, the multiple technological challenges involved in developing techniques for recording simultaneously from large populations of individual brain cells, even in anesthetized animals, let alone awake preparations, kept most experimentalists away from trying to test experimentally the
768
Spatial Resolution (mm; logscale)
102
100
10-2
10-4 10-4
ms
10-2
100
s
102
min
104
hour
106
day
108
year
Temporal Resolution (s; logscale)
B Spatial Resolution (mm; logscale)
Yet, since the late 1940s, a neural network-based view of brain function began to reemerge. Inspired by the pioneering work of Thomas Young on color coding in the early 19th century (881) and that of Charles Sherrington on spinal reflexes at the beginning of the 20th (733), theoreticians, such as Donald Hebb (352), and neurophysiologists, such as John Lilly (499), proposed that the true functional unit of complex brains, such as ours and those of other mammals, is represented, according to Hebb’s own terms, by “. . . a diffuse structure comprising cells in the cortex and diencephalon, capable of acting briefly as a closed system, delivering facilitation to other such systems (352).”
Techniques for Recording/Visualizing
Electrode Techniques for Stimulation/Lesioning
102
100
10-2
10-4 10-4
ms
10-2
100
s
102
min
104
hour
106
day
108
year
Temporal Resolution (s; logscale)
C
Techniques for Stimulation/Lesioning
102
100
10-2
10-4 10-4
ms
10-2
100
s
102
min
104
hour
106
day
108
year
Temporal Resolution (s; logscale)
FIGURE 1. Temporal and spatial resolution for different techniques to study the brain and interact with its circuitry. A: techniques for recording and visualizing. B: electrode techniques for recording and stimulation. C: techniques for stimulation and making lesions. (Adapted with permission from Sejnowski TJ, Churchland PS, Movshon JA. Putting big data to good use in neuroscience. Nature Neurosci 17: 1440 –1441, 2014. Reprinted by permission from Macmillan Publishers Ltd.)
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
During the century of intense work that followed the pioneering discoveries of Ramon y Cajal and Golgi, many other histological, electrophysiological, and imaging methods have been incorporated in the technical arsenal employed by neuroscientists to probe brain function at different levels of spatial and temporal resolution (FIGURE 1,A–C ). For most of this continuous procession of new technological developments, the neuron doctrine continued to flourish. As a result, for the vast majority of the neuroscientific community, the single neuron remained the central focus of systems neuroscience for most of the 20th century (583).
A
Spatial Resolution (mm; logscale)
myocardium, forming a syncytium, or a continuous network of fused or tightly connected neurons (308). Cajal’s histological demonstration of the existence of synaptic clefts between neurons in most of the brain debunked Golgi’s view from an anatomical point of view and, as a consequence, the reticular theory was abandoned and the term network fell out of favor with most neuroscientists. Ironically, many decades later, clusters of neurons tightly linked via gap junctions (155) were identified in some key structures of the mammalian brain, including the inferior olive (506), hippocampus (443), and neocortex (486, 747). Because of the abundant existence of tight junctions, these clusters exchange information through electrotonic coupling. As such, they clearly typify the type of networks envisioned by Golgi.
362
BRAIN-MACHINE INTERFACES ideas about neural populations introduced by Hebb. Thus, if one combines these notorious and, at the time, insurmountable technical difficulties, with the stupendous success and widespread acceptance of Cajal’s neuron doctrine, amplified significantly in the 1960s and 1970s by the enormous impact of the Hubel and Wiesel’s characterization of single neuron physiological properties in the visual cortex, it is no surprise that a neural population view of brain function had to wait for four long decades before it could begin receiving serious experimental attention by neurophysiologists.
B. Emergence of BMIs The modern era of BMIs emerged precisely at the time the methods for chronic multi-electrode recordings were consolidated in rodents and started to make their transition to
B
C
D Cover
High-density connector organizer
Cover
Wireless module Battery Connector module
Base module
FIGURE 2. Multichannel, wireless recordings in rhesus monkeys. A: movable 10 ⫻ 10 microwire arrays. B: photographs showing implant connectors in two monkeys (left and middle) and wireless module (right). C: layered schematic of the 3-D printed modular headcap. D: headcap wireless assembly. [Adapted from Schwarz et al. (711).]
primates, first to New World monkeys (582, 852) and then to rhesus monkeys (114, 463, 580). Around that time, the term BMI was introduced for the first time in the systems neuroscience literature (575). The main goal of these studies was to investigate physiological properties, including the ability of neural ensembles in the sensorimotor cortex to encode information and express plastic adaptations while freely behaving animals learned new motor tasks (460). However, with the publication of a series of original studies, conducted in rats and monkeys in the early 2000s, it soon became apparent that BMIs could also serve as the foundation of a new generation of neuroprosthetic devices aimed at restoring mobility to patients severely paralyzed due to trauma to the nervous systems, notably spinal cord injuries (SCIs) or neurodegenerative diseases. FIGURE 3 shows the original schematic description of this idea, proposed in the early 2000s, as an envisioned direct link between a human brain and a robotic arm. FIGURES 4 AND 5 illustrate how the original BMI control scheme was adapted to become the experimental paradigm employed with rhesus monkeys. For the most part, until today, most laboratories around the world that work with upper limb BMIs
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
769
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Among the experimental studies that rekindled the interest in neural populations in neurophysiology during the 1980s was the pioneering work of Apostolos Georgopoulos on directional coding in the primate motor cortex (292, 295, 299) and John O’Keefe’s discovery of place neurons in the rat hippocampus (600). Georgopoulos’ findings that individual neurons in the primate primary motor cortex are broadly tuned to the direction of arm movement (294, 295, 423, 710) and that populations of such neurons, rather than an individual M1 neuron, have to be pooled together to compute the direction in which a monkey is about to move its arm (292, 299) brought to the forefront the much neglected, if not forgotten, Hebbian’s view of the neural population basis of brain function. By the mid-1990s, the introduction of new electrophysiological methods for chronic multielectrode recordings in freely behaving animals triggered a new phase of neural ensemble physiology (581, 582, 584, 857, 858). In this experimental approach, arrays or bundles of microelectrodes, originally made of fine insulated metal filaments, were chronically implanted across multiple cortical and subcortical structures of the brain of rodents and primates and yielded viable single and multiunit activity for long periods of time, which today can reach several years for the same animal (711) (FIGURE 2). By the early 1990s, this new approach was yielding simultaneous recordings of ⬃12–24 neurons in freely behaving rats. These recordings lasted for several weeks or even months (578, 579, 584). By mid-1995 the yield increased to ⬃50 simultaneously recorded neurons, with the added capability that individual neurons could be recorded in up to five different subcortical and cortical structures that defined a given neural pathway (i.e., the rat trigeminal somatosensory system) (578). By 1999 –2000, this recording benchmark reached 100 neurons, and such simultaneous recordings could be obtained in both awake rats and monkeys (580 –582). At that time the modern concept of BMIs was proposed by John Chapin’s and Miguel Nicolelis’ laboratories working together (124, 575, 852).
A
363
LEBEDEV AND NICOLELIS
continue to employ the elements depicted in FIGURES 3–5, which were originally published about 17 years ago (124, 852). Indeed, most researchers working in the field would consider FIGURE 3 as the standard or operational definition of a BMI, which can be described as a reciprocal link between an animal or human brain and an artificial actuator, such as a virtual or robotic arm or leg, which allows the
FIGURE 4. A brain-machine interface for enabling arm movements and with multiple feedback loops. A rhesus monkey is controlling a robotic arm that reaches and grasps objects. The robotic arm contains touch and position whose signals are processes and converts to microstimulation pulses delivered to the sensory areas in the brain. Neuronal ensemble activity is recorded in multiple brain areas and decoded to generate commands that control the robotic arm. [From Lebedev and Nicolelis (466), with permission from Elsevier.]
770
subject to utilize its own volitional electrical brain activity to control the movements of such an actuator, while receiving continuous feedback information from it. As we will see throughout this review, many aspects of this standard definition have been expanded significantly over the past two decades. For example, as illustrated in FIGURE 5A, large variety of brain signals, ranging from single units (114, 124, 466, 583, 852), to local field potentials (LFPs) (264, 670, 750), electrocorticography (ECoG) (483, 484, 697, 759, 837, 839, 856), electroencephalography (EEG) (7, 71, 141, 169, 358, 412, 627, 865), all the way to magnetic resonance imaging (MRI) signals (666, 745, 755, 847, 879), have been utilized as the source of voluntary motor activity needed to control an artificial actuator. By the same token, BMIs have been now designed to control a large variety of actuators, including computer cursors (114, 283, 463, 692, 725, 794, 864), digital communication systems (10, 71, 101, 247, 545, 559, 672), robotic limbs (114, 152, 360, 463, 795, 817), robotic exoskeletons (156, 456, 467, 836), avatar bodies (151, 377, 435, 598, 657, 835), drones (426, 457), and wheelchairs (169, 511, 553, 656, 890). BMIs have also been coupled with a variety of traditional medical devices (522). For instance, recently a BMI core was used to allow patients to control stimulators that activate their own muscles through functional electrical stimulation (FES) (83, 633) (FIGURE 6). To extract the type of information needed to control such a vast list of actuators, a multitude of mathematical and computational approaches have been proposed as potential real-time decoders of voluntary motor activity generated by the brains of animals and human subjects (49, 449, 490).
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
FIGURE 3. Schematics of a cortical brain-machine interface. Intracranial recordings are employed to sample the extracellular activity of several hundred neurons in multiple cortical areas that are involved in motor control of arm and movements. The combined activity of cortical neuronal ensembles is processed, in real time, by a series of decoders that extract motor parameters from the brain signals. The outputs of these decoders are used to control the movements of a robot arm that allows human patients to perform arm movements. [From Nicolelis (575).]
364
BRAIN-MACHINE INTERFACES
A
Multichannel Neural Recordings Real-time control
BMI Decoder Visual Feedback
Position, Gripping Force
C
Observed
Predicted
Vx
20 0
-20
Vy
20
Task 1
Task 2
0
-20
GF
2 1 0
GF
1
0
Task 3
0
20
Time (s)
40
60
FIGURE 5. The first brain-machine interface for reaching and grasping. A: diagram of the experimental setup. The extracellular electrical activity of neuronal ensembles was simultaneously recorded from multiple cortical areas and then directed to a robotic arm that performed reaching and grasping movements. B: schematics of three motor tasks. In task 1, monkeys placed the cursor over a visual target. In task 2, the joystick was in a fixed position, and the monkeys grasped a virtual object by squeezing the joystick handle. In task 3, monkey placed the cursor over the target then produced a gripping force. C: recordings of motor parameters (blue lines) and their decoding (red lines). From top to bottom: example traces of hand velocity (Vx, Vy) and gripping force (GF). [Adapted from Carmena et al. (114).]
Moreover, a variety of signals and strategies for delivering sensory feedback to subjects operating a BMI have been proposed, including visual feedback through computer screens (114, 377, 692), tactile and proprioceptive cues via haptic displays (128, 597), and even direct intracortical microstimulation (ICMS) applied to the primary somatosensory cortex (S1) (262, 597, 598). To give an idea of how energized the field has become, even BMIs that involve the collaboration of multiple animal brains have been demonstrated in rat and monkeys (614, 616, 657). Notwithstanding these countless innovations, FIGURES 3 AND 4 still are useful to describe the basic elements that constitute a BMI. These include the use of sensors (e.g., multielectrode arrays) to sample large-scale brain activity; multi-channel electronics that amplify, filter, and
digitize these signals; a computational engine responsible for real-time extraction of motor commands from the raw brain activity; an artificial actuator that performs motor tasks; and a stream of feedback signals delivered to the subject’s brain. Merely 17 years after the modern age of BMI research was launched, the tremendous impact of this paradigm in neuroscience can be measured by a variety of metrics. For example, a bibliographic search in Google Scholar using the terms brain-machine interface and a closely related expression, brain-computer interface, yields more than 40,000 publications during the past decade alone. Meanwhile, several of the key experimental and clinical papers in the field have surpassed 1,000 citations. Several prominent books have been published on different aspects of BMIs (61, 106,
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
771
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
B
Robot Arm + Gripper
LEBEDEV AND NICOLELIS
EEG (µV)
Motor imagery
Beta (µV)
Motor imagery
Motor imagery
50
25
0
5
10
15
20
25
30
35
40
45
50
10s
−4
0
5
10
15
20
25
30
40
45
50
35
40
45
50
−8 0
5
10
15
20
25
30
Relax 1
2
3
0
FIGURE 6. Hand grasping induced in a tetraplegic patient by functional electrical stimulation (FES) controlled by an EEG-based brainmachine interface (BMI). The BMI utilized motor imagery of foot movements. FES was applied through surface electrodes. Aided by this BMI, the patient was able to grasp a glass with his paralyzed hand.
210, 577, 586, 660, 862). During the same period, an estimated $800 million United States dollars have been invested in BMI research worldwide. Since the introduction of experimental BMIs in the late 1990s, many applications have emerged for healthy subjects, outside the domains of basic and clinical research. BMIs for computer gaming (6, 9, 257, 342, 489, 493, 531, 532, 775, 874), EEG-based that detect drowsiness in drivers (137, 286, 501, 504, 636, 685, 810), and even BMIs for education (371, 526) are just a few examples of this parallel line of BMI development outside biomedical research. As a result, many started to consider BMIs as a method to augment human neural and physiological functions, such as cognitive abilities (242, 518, 519, 539, 888) and motor performance (148). Despite being very interesting subjects for debate, none of these latter areas of BMI application will be covered here. In this review, we start with a brief history of BMIs, followed by a description of major classes of BMIs. Next, we spell out the major components for building a BMI: sensors to record large-scale brain activity, decoding algorithms for
As mentioned in the introduction, the history of BMIs is intimately related to the effort of developing new electrophysiological methods to record the extracellular electrical activity of large neuronal populations using multi-electrode configurations. Such an essential component of modern BMI architecture was pioneered in the 1950s, by John Cunningham Lilly, then a principal investigator at the National Institutes of Health. In his experiments, Lilly was able to implant 25– 610 electrodes, either on the pial surface of the cortex or intracortically, in adult rhesus monkeys (498) (FIGURE 7). In addition to recording field potentials (25 channels at a time) with these electrodes while animals exhibited a variety of behaviors and states (arm movements, sleep, etc.), Lilly also applied electrical current through those electrodes and elicited movements in both anesthetized and awake monkeys (496). He observed that motor responses could be evoked from many cortical sites, including M1 and S1. Lilly concluded that there was no clear-cut separation between cortical regions presumed to be motor alone or sensory alone. He suggested that these areas be named sensorimotor instead (497). The next intermediary step in the development of the BMI concept can be traced back to the introduction of “EEG biofeedback” or “neurofeedback,” which became very popular in the 1960s and 1970s, in a variety of experimental settings (179, 258, 405, 407, 622, 749, 764, 777, 843, 867). In these studies, subjects were provided with an indicator of their own neural activity, for example, auditory or visual feedback derived from EEG recordings, which assisted them with self-regulating those neural signals. David Nowlis and Joe Kamiya (405, 595, 596) recorded EEGs in animals and human subjects, and converted them into sound. Aided by this type of neurofeedback, subjects gained some level of volitional control over their own EEG activity. Maurice Sterman and his colleagues converted EEGs of epileptic patients into lights and tones, and achieved seizure reduction with this type of neurofeedback training (763– 766). According to Daniel Dennett (193), the first experiment in which human subjects sent brain-derived signals to command an external device was described in 1963 by Grey Walter in a talk to the Osler Society at Oxford University. Since Walter himself did not publish this presentation, Dennett’s anecdote may not be completely accurate or verified.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
35
Threshold
−6
Trigger for FES
II. HISTORY OF BMI RESEARCH A. Early Studies
0
−25
772
extracting behavioral variables from brain signals, the means to deliver sensory feedback to the brain, and external actuators controlled through BMIs. Finally, we will discuss BMI applications to restore mobility and, eventually trigger partial neurological recovery in severely paralyzed patients.
0
−50
Beta power
Motor imagery
365
366
BRAIN-MACHINE INTERFACES
A
B FIGURE 7. Multichannel implant of John Lilly. A: parts of the electrode implant: the lower end of a spear-shaped hardened steel tool that was used for starting bone holes (a, a’); the lower part of the mandrel (b, b’) with a sleeve on the cylindrical lower end (b’); sleeves (c, c’); electrode (d, d’); and sleeve guide (e). B: an X-ray of a monkey skull showing 20 implanted sleeves and one inserted electrode. [Adapted from Lilly (498). Reprinted with permission from AAAS.]
By the end of the 1960s, researchers at the NIH Laboratory of Neural Control started to experiment with the possibility of utilizing recordings from cortical neurons to control artificial actuators (270). They were also interested in using direct connections between brains and external devices to restore hearing to the deaf, walking to the paralyzed, and vision to the blind (701). This NIHled research was conducted with some universities and medical schools participating as subcontractors. Karl Frank, the NIH laboratory head, proclaimed, “We will be engaged in the development of principles and techniques by which information from the nervous system
can be used to control external devices such as prosthetic devices, communications equipment, teleoperators . . . and ultimately perhaps even computers” (269). In their initial study, the NIH team implanted five microelectrodes in the primary motor cortex (M1) of rhesus monkeys and then recorded action potentials generated by 3– 8 M1 neurons, while animals performed a motor task that required them to flex and extend their wrists (374). Since the eventual goal was to convert these neuronal signals into the movements of an external device, the researchers probed whether wrist movements could be predicted from the recorded activity of small neuronal populations. They utilized multiple linear regression as a prediction algorithm. The algorithm took neuronal rates as inputs and returned movement kinematics as the output. A decade of this research eventually resulted in the demonstration of a realtime neural control (702): a rhesus monkey with 12 microelectrodes implanted in M1 for 37 mo learned to move a cursor on an LED display using its own neural activity as a direct source of motor commands. During the late 1960s, Eberhard Fetz and his colleagues conducted experiments in which they utilized the electrical activity of single neurons recorded in monkey M1 as a source of neurofeedback (254). Using this apparatus, monkeys learned to self-regulate their own single-neuron activity. Typically, one neuron was tested at a time. The neuronal electrical firing rate was converted into either auditory (a click for each spike) or visual (deflections of an arrow meter placed in front of the animal) feedback. Monkeys learned to volitionally modulate the activity of each individual M1 neuron to reach a particular level of firing required to obtain a reward (FIGURE 8). While Fetz emphasized the neurofeedback aspect of such operant conditioning experiments, Brindley and Craggs (89, 168) employed epidural recordings of motor cortical field potentials in the frequency band 80 to 250 Hz in baboons to test the possi-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
773
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Published literature, however, confirms that Walter and his colleagues implanted multiple electrodes (up to 1,000) in the cortex of neurological patients and used these implants for monitoring cortical field potentials over a period of several months (831). In the experiments described by Dennett, Walter recorded motor cortex readiness potentials preceding movements, also called Bereitschaftspotentials, while his patients periodically pressed a button to advance slides in a slide projector. The button presses were selfpaced. The readiness potentials led the movement by approximately half a second and were sufficiently strong to be detected by Walter’s recording equipment. Next, Walter succeeded in creating a direct link between each patient’s motor cortex and the projector. In this experimental condition that we would now call brain control, the button was electrically disconnected from the projector and the slides were advanced by motor cortical readiness potentials. It came as a surprise to the patients that the projector responded to their will even before they physically initiated the movement. Although Walter’s experiments can be considered as the first proof of concept of the possibility of building BMIs, he never published these results or interpreted them in the context of BMIs, even though in his earlier career in the 1950s, he conducted research on robots with artificial brains (829, 830).
367
LEBEDEV AND NICOLELIS
A Unit discharge
Meter
Target
Reward
Average Firing rate (impulses/second)
B Operant level
Pellets only
Extinction
Clicks only
Extinction
25
30
Pellets and clicks
30
20
10
0 0
5
10
15
20
35
40
45
Time (minutes)
bility of creating a motor neuroprosthesis that recognized specific movements produced with the arms or legs. Moreover, Craggs (167) used baboons with complete spinal cord transections at the midthoracic level as a model of human paraplegia, while he recorded motor commands directly from the cortical representation of the foot disconnected from its spinal cord projection area. At the same time these laboratories experimented with extraction of motor signals from the brain and/or using them to generate neurofeedback, another line of research focused on ways to deliver information to the brain using electrical stimulation, either applied to peripheral nerves (153, 354) or to the central nervous system (90, 91, 495). This work led to early attempts to build sensory BMIs that strived to restore normal perception to patients suffering with neurological conditions that induced significant sensory deficits. From these pioneering studies, the work on cochlear implants (FIGURE 9) eventually reached the most spectacular results (198, 218, 219, 367, 510, 743, 855). In parallel, some progress was achieved in the development of visual cortical prostheses pioneered by the groups led by Brindley (88, 90, 91) and Dobelle (200 –203). These researchers applied electrical current to the visual cortex of blind patients through grids of surface electrodes. Using this apparatus, blind subjects could perceive light spots, phosphens, and
774
learned to recognize simple visual objects composed of several phosphens. Also in the 1960s, Bach-y-Rita and his colleagues started to develop visual substitution systems for the blind, based on tactile stimulation of the skin on the patient’s back (39, 40). This technique became known as vision substitution by tactile image projection. The apparatus employed, called a haptic display, consisted of 400 solenoid stimulators arranged in a 20 by 20 array. The tactile stimulation was applied to the surface of the patient’s back and attempted to reproduce, through the sense of touch, visual images captured by a video camera (FIGURE 10). After being trained for 10 h, blind patients learned to recognize objects and their positional relationship in a room, as well as landmarks, such as the room’s door frame (40).
B. Explosive Development in the Late 1990s Despite the initial push observed in the 1960s and 1970s, research on direct links between brain and machines experienced a decline during the next 20 years. Clearly, the lack of major technological breakthroughs in the area of multichannel neuronal recordings during that period prevented the field from taking off and fulfilling the auspicious poten-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
FIGURE 8. Operant conditioning of the discharge rate of a single motor cortex neuron. A: a monkey was operantly conditioned to increase the neuron’s rate until it reached a target level, and a reward was delivered. The feedback of the neuron’s firing was provided using either a visual meter or auditory clicks, one slick per spike. B: after the training, the monkey could increase the neuronal rate even in the absence of the feedback. However, removal of reinforcement extinguished the neuronal rate increase. [Adapted from Fetz (254). Reprinted with permission from AAAS.]
368
BRAIN-MACHINE INTERFACES External Transmitter Implanted Receiver/ Stimulator
Reference Electrode Microphone, BatteryPack, and Speech Processor
Intracochlear Electrodes
tial envisioned by the early proponents of this research program. Around the mid-1990s, however, the required technological innovation materialized with the introduction of a new multi-electrode design that allowed bundles or planar arrays to be built using flexible and insulated metal filaments, known as microwires (108, 578, 584, 585). John K. Chapin, one of the pioneers of microwire implants, discovered that if a blunt tip was left exposed in an otherwise insulated metal microwire, made of stainless steel, individual singleunits could be recorded for many weeks or even months from the forepaw representation area of M1 and S1 of freely moving rats (125, 734). A few years later, Miguel Nicolelis and Chapin chronically implanted multiple bundles and/or arrays of Teflon-coated stainless steel microwires in the ventral posterior medial nucleus (VPM) of the thalamus of adult rats and recorded the simultaneous electrical signals produced by up to 24 of these thalamic neurons in awake and freely moving rats (579, 584). A year later, the same authors reported simultaneous multi-site recordings, obtained from chronic implants that included not only VPM, but also multiple key subcortical structures that define the rat trigeminal system, such as S1, different thalamic and brain stem nuclei and even the trigeminal ganglion of the same subjects (578). Using this new ap-
proach, these researchers recorded the extracellular activity of up to 48 neurons distributed across multiple subcortical and cortical relays of the rat trigeminal somatosensory system, in awake, freely behaving rats. These experiments marked the first time simultaneous neuronal population activity, originating from multiple processing levels of a mammalian sensory system, was measured in the same animal subject. Three years later the same technique was validated in primates, allowing the Nicolelis Lab at Duke University to record from multiple cortical areas in awake owl monkeys (582). Since the recording properties of chronic microwire implants lasted for several months in both rats and owl monkeys, Chapin and Nicolelis found this technique suitable for testing the concept of linking the brains of rats and monkeys to artificial actuators and investigating whether these animals could learn to control external devices using only their brain electrical activity. In 1999, Chapin and Nicolelis published their first BMI study where rats learned to use the combined extracellular activity of up to 46 neurons to control the uni-dimensional movements of a lever that delivered water, collected from a water dropper, to the animal’s mouth (124). Initially, rats were trained to press a bar to generate the lever movements. As rats learned this task, a principal component analysis algorithm, implemented us-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
775
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
FIGURE 9. Diagram of a cochlear implant. (Reprinted with permission from MED-EL Medical Electronics GmbH, Innsbruck, Austria.)
LEBEDEV AND NICOLELIS
369
Curiously, also in 1999, without knowledge of the work carried out by Chapin and Nicolelis in animals, a group led by Niels Birbaumer at the University of Tubingen in Germany pioneered their version of a direct link between a brain and a computer in locked-in patients (71). Birbaumer chose to name this paradigm brain-computer interface (BCI), a term that had been introduced in the literature by Jacques Vidal in 1973 (823). Birbaumer’s BCI allowed locked-in patients to communicate with the external world using slow cortical potentials recorded via a noninvasive technique, EEG recordings, to control computer software for spelling. Using this system, locked-in patients became capable of writing messages on the computer.
ing analog electronics, was employed to transform the combined cortical activity of the recorded neuronal population into a continuous motor control signal that moved the lever. To the total surprise of Chapin and Nicolelis, not only did the rats learn to operate the lever efficiently to drink their daily water allotment, but, in a few trials, the animals would simply stop moving their forepaws and still successfully use their brain activity alone to move the lever and receive their water reward. A year later, the Nicolelis laboratory demonstrated that owl monkeys could utilize the simultaneously recorded electrical activity of close to 100 cortical neurons, distributed across multiple frontal and parietal cortical fields, to control the two- and three-dimensional movements of a multiple degree of freedom robot arm (852). This study also introduced a new analytical method and graphic representation, the neuronal dropping curve (NDC), which would become a standard representation to depict the neurophysiological results of BMI studies (FIGURE 11) (114, 283, 377, 462, 465, 562, 690, 692). In 2000, in a review paper commissioned by Nature, Nicolelis dubbed the paradigm Chapin and he had implemented as a brainmachine interface or BMI, the first time the now traditional term was used to refer to real-time links between living brains and artificial devices (575).
776
In 2004, the Nicolelis laboratory also reported the first demonstration that ensembles of subcortical neurons, recorded intraoperatively with microwire bundles, could be employed to extract hand movements in awake and conscious human subjects (620). These recordings were obtained during a neurosurgical procedure in which Parkinsonian patients received a deep brain stimulator. During a brief intraoperative period of 10 –15 min, up to 50 neurons located in the subthalamic nucleus and thalamic motor nuclei were recorded while these patients played a one-dimensional video game by exerting a gripping force with one hand. This study revealed that the same computational algorithm, multi-linear regression, employed by Wessberg et al. (852) and Carmena et al. (114) in owl and rhesus monkeys, respectively, could be used to extract hand movement patterns from human subcortical signals.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
FIGURE 10. Vision substitution system developed by Paul Bach-yRita and his colleagues. The system included a digitally sampled television camera, control electronics, and a 400-point matrix array of tactile stimulators mounted on a dental chair. The tactile stimulation projected images to the back of blind subjects. [From Bach-yRita et al. (40).Reprinted by permission from Macmillan Publishers Ltd.]
Overall, the original papers by Chapin, Nicolelis, and Birbaumer mark the beginning of the modern age of research on BMIs. In the case of intracranial BMIs, the focus of this review, advances in multi-electrode recording methods, combined with the introduction of faster digital computers running new computational algorithms for extracting motor signals from brain-derived signals, triggered a phase of very fast growth in the field. Thus, following the original demonstrations in rats and New World monkeys, the next important milestone of the field was the translation of the BMI paradigm to rhesus monkeys, the conventional experimental animal model for exploring neurophysiology of advanced motor behaviors and cognition. In a quick succession, three different groups published their results in this primate species (114, 725, 794). In a span of 12 mo, the BMI paradigm in rhesus monkeys incorporated the use of a series of novel actuators, such as a computer cursor (725, 794), and a robot arm capable of producing both arm reaching movements and hand grasping (114). With this latter addition, the Nicolelis laboratory demonstrated that the same pool of recorded cortical neurons could be employed to simultaneously extract hand gripping force and arm position and velocity from multiple frontal and parietal cortical areas in awake rhesus monkeys (114, 463).
370
BRAIN-MACHINE INTERFACES
A
B
Monkey 1 1
Monkey 2 1
Linear model fit, R2
All neurons 0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
20
C
40
60
80
100
PP
MI
0.6
PMd
0.4
0.3
0.3
0.2
0.2
0.1
0.1 10
60
80
100
120
0.5
ipsi MI
0
40
0.7
0.4
0
20
20
30
Number of neurons
40
50
0
0
10
20
30
40
FIGURE 11. Neuron-dropping curves (NDCs). In this study, owl monkeys performed one-dimensional movements with a joystick. Joystick position was decoded from the activity of cortical neuronal populations using a linear algorithm. Decoding accuracy was measured as coefficient of determination, R2. NDCs plot R2 as a function of neuronal ensemble size. They were constructed by calculating R2 for the entire neuronal population, then removing one neuron from the population and calculating R2 again, and so on until only one neuron was left. Extrapolated NDCs for even larger populations were constructed using a hyperbolic function. A and B: NDCs (thick lines) and hyperbolic extrapolations (thin lines) for all neurons in monkeys 1 and 2, respectively. C and D: NDCs calculated separately for each recorded cortical area in monkeys 1 and 2, respectively. [From Wessberg et al. (852).]
50
Number of neurons
A couple of years later, in 2006, the group led by John Donoghue reported the operation of a BMI in one patient chronically implanted with a different multichannel recording technology approved for human trials (362). Their implant, named the Utah probe, was a 10 ⫻ 10 array of silicon-etched rigid needles (111, 537, 593). The Utah probes were inserted in the cortex ballistically using a pneumatic gun (680), the method adopted for this implant to avoid the “bed of nails effect,” where a slowly inserted dense electrode array produces cortical dimpling and trauma (76, 111, 668). A total of two patients were implanted with this device in 2006 (362), one of whom experienced implant malfunctions for the first 6 mo followed by 2 mo of recordings. Both patients used a BMI to control two-dimensional movements of a computer cursor. As discussed below, recordings with Utah array suffer from biocompatibility issues (220, 251, 679), which in our opinion should preclude them for further use in human subjects. Shortly after implantation, this probe can produce a significant tissue lesion and, hence, become encapsulated by glia and protein deposits, as a result of the local inflammatory reaction. Usually, this process renders the Utah probe unusable for single-unit recordings after a few weeks/months (134). Groups that rely on this probe usually resort to the utilization of a threshold-crossing method to detect useful neuronal activity (134, 803), a maneuver that discards wellestablished neuronal recording quality criteria (580) and increases the likelihood of recording noise instead of neuronal spikes, by mistakenly recording mechanical, electrical and EMG artifacts as if they represented valid neural activ-
ity. As such, this recording technology cannot be considered as the final solution for clinical BMI applications, even though currently this method is commonly used in human trials. Practical solutions will clearly require better recording reliability, stability, and longevity standards to become accepted by patients and clinicians. To achieve better biocompatibility of a brain implant, in 1989, Philip Kennedy implanted an ALS patient with a neurotrophic electrode loaded with nerve growth factors that induced growth of nerve fibers into the electrode tip (418, 420, 422). The study reported that the patient learned to produce on/off neural control signals that were detected by the electrode. While the same group continues this research until now (98, 327), neurotrophic electrodes were not adopted by other groups and their effectiveness remains difficult to evaluate. In addition to a few more clinical demonstrations of BMIs that controlled computer cursors (3, 731), upper limb robotic prostheses (152, 360) or an FES system for the hand (83) using either populations of M1 or other cortical neurons, several innovations were incorporated to the traditional BMI approach over the past several years. In 2009, the Nicolelis laboratory published the first BMI approach to decode kinematics of bipedal walking in rhesus monkeys (261) (FIGURE 12). Two years later, in 2011, the same laboratory implemented, for the first time, a method for multi-channel ICMS as a tool to deliver direct tactile feedback to the subject’s somatosensory cortex in a BMI setup (598)(FIGURE 13). This new
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
777
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
0.5
0
D
PMd
0.6
0
120
MI
0.7
Linear model fit, R2
All neurons
0.8
LEBEDEV AND NICOLELIS
A
Inte
rne
Neural Data
Kin
B
nk
Position Trace
cm
em
atic
Dat
-20 0 Time (sec) 30
a
Monkey 1
Monkey 2
Variable Speed Tracking and Predictions 0.14
Slow Forward Walking, SNR = 6.44
Actual
Predicted
(m)
Ankle X Coordinate
-0.06 0.15
Fast Forward Walking, SNR = 7.87
(m) -0.1 0.15
Variable Speed Forward Walking, SNR = 6.22
In parallel with these developments, considerable efforts have been made by many laboratories to design better real-time decoding algorithms. Krishna Shenoy and his colleagues (692) reported a “high-performance BMI” that relied on the strategy of flashing potential targets in a rapid succession on a computer screen. They recorded from small neuronal populations in dorsal premotor cortex and found that the firing of these neurons reflected the target location. Target locations could be decoded from these neuronal firing modulations using recording intervals as short as 250 ms, which allowed the BMI to reach an information transfer rate of 6.5 bits/s. The authors argued that the observed neuronal responses represented the monkeys’ motor preparatory activity for arm reaching movements rather than merely visual responses to the targets, and therefore could be useful for controlling a motor BMI that automatically directs the cursor to the target once its location is determined. The same group developed an improved algorithm for continuous cursor control (307). The improvement was achieved using the “recalibrated feedback intention-trained Kalman filter” that was trained using both the cursor position in screen coordinates and an estimate of intended velocity based on the relative location of the cursor and target.
(m) -0.2
0
Time (s)
FIGURE 12. Decoding kinematics of bipedal walking from cortical ensemble activity. A: diagram of the experimental setup, consisting of a treadmill, video tracking system, neural recording system (Plexon, USA) and a computer for real-time decoding of neural activity. B: video frames depicting step cycles of two monkeys. C: tracking (blue line) and decoding (red line) of ankle position at different treadmill speeds. [Adapted from Fitzsimmons et al. (261).]
paradigm was named a brain-machine-brain interface (BMBI). In those experiments, rhesus monkeys performed an active tactile discrimination task using a BMBI that both generated motor commands and delivered artificial tactile feedback directly to the animal’s brain. To perform the exploration task, monkeys had to control the movements of a virtual hand, using their motor cortical activity alone, to scan the surfaces of up to three visually
778
50
Moving in the same direction, our laboratory achieved considerable improvement of real-time decoding by employing an unscented Kalman filter that used position, velocity, and speed as state variables and incorporated nonlinear relationships between the neuronal rates and these variables (491). More recently, Jose Carmena and his colleagues (728, 729) reported an improved temporal resolution for an adaptive decoder that modeled spikes as a point process. Overall, these efforts resulted in a large variety of BMI decoding algorithms from which designers can choose depending on the requirements for their experimental or future clinical implementations. While the research on intracranial BMIs has been conducted mostly in animals for many decades and only recently has started to expand into clinical trials in human patients, noninvasive BMIs, pioneered by Jacques Vidal in the early 1970s (823, 824) and introduced to
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
C
identical virtual objects shown on a computer screen. The position of the virtual objects changed with every new trial. As the virtual hand touched each virtual object, temporally patterned ICMS was applied to the animal’s primary somatosensory cortex to mimic the virtual textures. To receive a juice reward, the monkey had to find an object associated with a particular virtual texture, and hold the virtual hand over that object. After a few weeks of training, two monkeys learned to perform this task at levels similar to those attained when the control signal to the virtual arm came from the joystick that the animals moved with their own biological hands.
t Li
20
371
372
BRAIN-MACHINE INTERFACES
A
B Movement decoding M1
Active exploration task
Brain control
S1 10 mm
Hand control
C U
e pp
rm
Di
Artificial tactile encoding
D
ra
gi
ts
S1
5 mm
UAT
RAT
5
x position
400 Hz
200 Hz
–5 –10 106
Neuron
E
0
1 0
0.5
1
1.5
2
2.5
Time (s)
clinical practice by Niels Birbaumer (71), have undergone a considerable expansion (14, 74, 246, 358, 815, 863). BMIs for humans are often referred to as BCIs, including both invasive and noninvasive systems. It is worth noting that the first publication on a human controlling a robot with EEG activity dates back to 1988 (85). In this report, subjects issued start and stop commands to a robot by closing and opening their eyes, the well-known procedure to activate and deactivate alpha waves (60, 587). Recently, a motor imagery-based BCI was used by the Walk Again Project, an international nonprofit research consortium, to allow complete paraplegic patients to use EEG to control the start and stop sessions of bipedal walking of a lower limb robotic exoskeleton (737). Overall, BCI research yielded many practical applications, such as BCIs that use EEG signals to control computer cursors (240, 864, 865), computer-assisted spellers (102, 245, 247, 421, 545, 832), wheelchairs (121, 411, 511, 553, 811, 833), and exoskeletons that restore bipedal walking (156, 836).
III. BMI CLASSIFICATION A. Classification by Function Several BMI classification schemes have been proposed heretofore. One scheme classifies BMIs according to the physiological function they are intended to emulate. Here, BMI systems are commonly categorized as follows: 1) motor, 2) sensory, 3) sensorimotor (or bidirectional), and 4) cognitive. The recent introduction by our laboratory of BMIs that incorporate multiple brains of different subjects (614, 616, 657) adds one more BMI class, which we named Brainet (657). Motor BMIs reproduce motor functions, such as upper (114, 152, 817, 852) and lower limb (261) movements or whole body navigation (656, 873). Sensory BMIs aim at reproducing sensations, while sensorimotor BMIs combine the motor and sensory components in a single application (57, 597, 598). Cognitive BMIs (19) enable higher-order
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
779
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
FIGURE 13. Brain-machine-brain interface. A: diagram of experimental setup. Monkey was seated in front of a computer screen showing an avatar arm an multiple targets. Motor commands were decoded from motor cortex activity. Artificial tactile feedback was produced by intracortical microstimulation applied to primary somatosensory cortex. B: cortical location of microelectrode implants. C: microelectrodes used for microstimulation (accented in red). D: avatar arm position for a representative trial. The monkey first placed the avatar hand over the unrewarded artificial texture (UAT), then ultimately selected rewarded artificial texture (RAT). Vertical gray bars correspond to the periods of microstimulation; insets indicate stimulation frequency. E: raster display of motor cortex discharges for the same trial; spikes were not detected during microstimulation delivery because of the stimulation-induced artifacts. Only the periods void of microstimulation were used for neural decoding. [Adapted from O’Doherty et al. (598).]
LEBEDEV AND NICOLELIS
Based on our own BMI work, we have argued for more than a decade that highly distributed neuronal ensembles represent the true physiological unit of the nervous system (583), a proposition that is supported by an extensive literature (107, 109, 253, 302, 339, 477, 648, 686, 687). In a distributed neural circuitry, there are no exclusive functional specializations at the level of individual neurons. Rather, single neurons multiplex several functions, and the best decoding can be achieved by sampling large numbers of neurons from multiple brain areas, simultaneously. We have proposed, therefore, that as the BMI field advances, this new theoretical view of brain function will lead to almost universal acceptance of a distributed principle of neuronal activity sampling in BMI applications. Since our first BMI studies (852), we have consistently demonstrated that large-scale neuronal recordings from multiple cortical areas are imperative for building BMIs that are versatile, robust, efficient, and clinically relevant (466, 575, 583). Another step in this direction was made when the BMBI paradigm was introduced (598). In a BMBI, motor and sensory streams of information are handled simultaneously to facilitate sensorimotor processing. In the same context, the concept of Brainets takes multitasking BMI designs to the next level by combining multiple brains into a higher order computational entity (614, 616, 657).
B. Classification by the Level of Invasiveness Since the inception of modern-era BMIs, two major approaches have dominated the field: intracranial or invasive (124, 852) and noninvasive (71) systems. Accordingly, it is common to classify BMIs by their level of invasiveness. This division is important for a variety of reasons, the major one
780
being a safety concern. Invasive BMIs require a neurosurgery procedure that involves opening the scalp and skull and penetrating the brain tissue, albeit only for a few millimeters, for systems relying on cortical signals. These procedures carry a risk of tissue damage and/or infection, particularly if the implant is not fully contained within the body and has external parts, like wires connected to extracranial recording hardware, as was the case in recent clinical trials in humans (83, 152, 360, 362). Noninvasive BMIs, on the other hand, do not carry such risks and can be implemented rather easily. For example, in the case of EEG recordings, the electrodes are simply placed on the scalp surface (588) through an easy and safe procedure, particularly if dry sensors (136, 266, 287, 329, 785) are used. Since patient safety is of paramount importance, noninvasive BMIs are currently the default choice for clinical applications. Yet, their recording quality is insufficient in many cases, causing EEG-based BMIs to be rather slow systems. Indeed, EEGs represent attenuated and filtered brain activity, which combines synchronous electrical signals, produced by many millions of neurons. Since these signals have to travel through bone and skin prior to reaching the scalp sensors, EEG signals lack fine spatial resolution and do not provide the kind of precise task-related neuronal signals that can be obtained from intracranial recordings (466). Similar limitations characterize all noninvasive recording methods that measure neuronal signals at a distance from their source. In contrast, in invasive BMIs, recording sensors are brought close to the very source of generation of neural activity: the single neurons that code information through trains of action potentials (157, 318, 487). Current intracranial BMIs usually employ extracellular recording methods that allow one to sample and discriminate action potentials generated by hundreds of individual cortical neurons (466, 583, 711, 767). The more microelectrodes are implanted; the more neurons can be sampled simultaneously. Moreover, the same microelectrodes can also record local field potentials (LFPs), which represent combined potentials of large (on the order of tens of thousands) neuronal populations (192, 289, 373, 398, 571). Additionally, the same implanted microelectrodes that are used for recordings can also be used for the delivery of electrical microstimulation that, depending on the stimulated area, can influence sensory, motor or cognitive processing (41, 45, 48, 146, 704, 705, 719, 770). An intermediary approach, known as electrocorticography (ECoG) can be considered as a semi-invasive method since it requires a craniotomy but does not involve sensors penetrating the nervous tissue. ECoGs are recorded with a grid of electrodes placed on the brain’s surface; dura mater may be left intact (i.e., epidural ECoG) or open to allow closer contact between the electrodes and the cortex (subdural ECoG) (24, 94, 357, 530, 665, 696, 828). ECoG recordings
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
brain functions, such as memory (62), attention (279, 512), and decision-making (343, 564). Finally, Brainets involve the implementation of shared BMIs requiring the participation of multiple subjects since these systems require the combination of electrical activity of multiple brains simultaneously to operate properly (657). Although seemingly straightforward, this classification scheme follows the traditional labeling of brain areas as motor, sensory, or higher order (associative), a simplified description, which is fundamentally wrong in most cases. There is mounting evidence against such a parcellation scheme of brain functions and in favor of a more distributed mode of information processing in the primate cortex (96, 253, 497, 583, 675). Thus sensory and motor signals are typically multiplexed by cortical neurons, including the areas considered to be purely motor or purely sensory (470). Indeed, even the primary motor and somatosensory cortical areas have been shown to concurrently process both motor and somatosensory information (229, 277, 497, 500, 647), and also represent visual signals (738), reward amount (529), and even cognitive processes, such as mental rotation (297, 300) and encoding serial order of stimuli (115).
373
374
BRAIN-MACHINE INTERFACES have better spatial and temporal resolution than EEG, but they cannot be used to reliably detect single-neuron spikes. Chronic ECoG recordings can last many years in animals (105, 123, 131, 508, 682) and humans (845, 870). Overall, ECoG has many advantages compared with EEG, and it is not as invasive as penetrating implants. Still, it remains controversial whether or not ECoG-based BMIs can rival single units in terms of BMI performance and accuracy. A variety of intracranial and invasive approaches are reviewed below in the section on neural recording methods.
C. Classification by the Origin of Neural Signal
Our laboratory has long advocated recording simultaneously from multiple cortical areas as an efficient way to increase both the amount of information processed by a BMI and its performance and versatility (114, 377, 466, 583). We have routinely recorded from four to eight frontal and parietal cortical areas in rhesus monkeys to operate a variety of BMIs and observed that any of those cortical areas provide useful information. The best BMI performance was usually achieved when neuronal signals from multiple frontal and parietal cortical areas were combined. Lately, interest has increased to BMI systems that utilize subcortical recordings. These systems, in principle, could capture neural processing in cortico-subcortical loops (11) related to motor control (188, 189, 677), sensory processing (44, 396, 397, 447, 528), motivation (706 –709), and skill learning (72, 212, 356, 689). It is noteworthy that subcortical recordings from the ventrolateral thalamus (VL) were utilized in the pioneering BMI study by Chapin et al. in 1999 (124) where VL neurons contributed to the BMI control of lever movements. Additionally, Patil et al. (620) relied on subcortical recordings from the subthalamic nucleus (STN) and ventral intermediate/ventral oralis posterior motor thalamus (Vim/Vop) to demonstrate, for the first time, that human patients could utilize real-time algorithms previously tested in monkeys to generate one-dimensional
In the future, subcortical BMIs could contribute to treatment of neural conditions caused by disorders of subcortical processing, such as Parkinson’s disease. As a step in this direction, we analyzed pathological signs in neuronal populations in Vim/Vop and STN recorded in Parkinsonian patients (336), while they controlled a computer cursor by opening and closing their hands. Their task was to point to screen targets with the cursor. Vim/Vop and STN neuronal populations responded to target onset, and hand movements, as well as being correlated with hand tremor. BMI decoders extracted movement kinematics from the STN population activity even when those populations exhibited tremor-related oscillations. These findings indicate that, in the future, BMIs based on subcortical recordings could be used for monitoring signs of neurological diseases, evaluating medical treatments and even delivering real-time rehabilitation therapies, via implanted devices, without the need for continuous supervision.
D. Classification by BMI Design Development during the last two decades resulted in several well-established BMI designs. Two broad classes of BMIs are represented by the so-called independent (endogenous) and dependent (exogenous) systems. Although this terminology is usually applied to noninvasive BMIs (i.e., BCIs), it is also applicable to intracranial BMIs. In an independent BMI, subjects self-initiate actions, for example, by imagining movements (604, 631, 651, 698) or even assisting themselves with overt movements of the limbs (114, 307, 463, 852). Such imagery and self-generated movements are controlled voluntarily by the subjects and, in principle, could be performed independently from any external stimuli (although some external stimuli are usually involved). Dependent BMIs, as the name suggests, critically depend on the presence of an external stimulus and the triggered neural responses to this stimulus (246, 473, 717, 832). For example, a P300 EEG- or ECoG-based BCI monitors cortical responses to computer screen events and detects a stronger response to the stimulus attended by the subject (101, 209,
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
781
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Since the late 1990s when the modern BMI concept and design was introduced, the majority of BMIs have utilized neural signals recorded from cortical areas of animals or human subjects (71, 114, 124, 152, 362, 377, 794, 852, 864). Such an abundance of cortical BMIs is not surprising because the cortex is the largest and most advanced brain structure, which is also the easiest to access with the recording sensors. Among cortical areas utilized in BMIs, M1 is the most commonly implanted area (114, 725, 852) because neuronal discharges in M1 are clearly correlated with different movement parameters (227, 294, 554). Additionally, recordings from premotor cortex are employed in BMIs as a source of motor commands to control ongoing movements (114) and preparatory signals that reflect motor planning before movements have started (692).
movements of a computer cursor (336). More recently, Koralek et al. (442) employed cortical and striatal recordings in awake, behaving rats to construct a BMI for abstract skill learning. In these experiments, rats learned to control the pitch of an auditory stimulus through a BMI. This learning was accompanied by increases in neuronal firing in the striatum. Additionally, stronger correlations developed between the cortical and striatal neurons. These findings further supported the claim, made in early BMI studies (114, 124, 463), that brain plasticity, in this case, corticostriatal plasticity, plays a major role in learning to control a BMI. This is an important conclusion for the development of BMIs, since it means that severely disabled patients may be able to learn novel motor skills, through BMI training.
LEBEDEV AND NICOLELIS 257, 545, 635, 805). A very similar design, mentioned above, was implemented in an intracranial BMI that evaluated premotor cortex responses to visual targets that rapidly flashed on the screen (692). The improvement in performance was achieved because the decoder received the information at the time of stimuli presentation and started to analyze neuronal data precisely after this event. Such utilization of the timing of external events is characteristic of all dependent BMIs. For example, BMIs that incorporate instructed delay tasks rely on a precise sequence of computergenerated task events for decoding (564, 731). While such supervised operation speeds up the decoding and makes it more reliable, dependent BMIs can operate only for a given set of rules, a property that clearly limits the user’s autonomy in choosing motor outputs.
Another criterion to classify BMIs is whether a subject performs overt movements while performing a motor task using the BMI. Early BMI demonstrations relied on overt movements to train the decoder and to operate a BMI (114, 261, 463, 852), whereas the next generation of BMIs excluded overt movements from both training and operation phases (377, 794). The goal of this latter modification was to mimic more properly the conditions of paralyzed patients who cannot produce overt movements. Withholding overt movements during BMI control dramatically changes neuronal ensemble activity patterns (463, 583), including a transient increase in correlation between cortical neurons, which tends to subside with further training (377). Overall, having a monkey control a BMI without moving its own limbs is a much more difficult task for the animal compared with the BMI control assisted by overt behaviors. Some recent studies still allow monkeys to assist themselves in the BMI operation by producing overt limb movements (239, 307, 761). While these reports downplay the significance of this component (307), it is possible that the presence of overt movements contributed to the improvement in BMI decoding. We suggest that overt behaviors should be better monitored in BMI studies and compared with the changes in BMI performance. In this context, we should also mention that the requirement to withhold limb movements, while producing actuator movements only through a BMI, defines a particular motor task by itself; one that resembles the well-known
782
In the early days of the field, the number of neurons needed for accurate BMI performance control used to be a controversial issue. During the early 2000s, research groups led by Donoghue (723, 725) and Schwartz (794) suggested that recording from just a few cortical neurons simultaneously could be sufficient for achieving good BMI control. In contrast, our laboratory has always argued that large neuronal ensembles are required to achieve optimal BMI performance (114, 261, 377, 466, 468, 583, 852). We reasoned that single neurons represent behavioral parameters of interest only partially and in a noisy way, and that combining contributions from many neurons both increases the information content and improves the signal to noise ratio of decoding. Additionally, it is easier to select neurons with the properties needed for decoding when there is a sufficiently large neuronal sample from which to select. As time passed, this dispute was resolved in favor of the large neuronal samples. Even the former proponents of small ensembles have now switched their approaches to embrace large neuronal samples as the only viable approach for clinically-relevant BMI applications (152, 360, 362).
IV. REPRESENTATION OF INFORMATION BY NEURONS AND THEIR ENSEMBLES A. Tuning of Single Neurons For the past six decades, an extraordinarily large neurophysiological literature has accumulated around the subject
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
In addition to independent and dependent BMIs, the term passive BMI (or passive BCI) has been recently introduced to describe a system that performs useful decoding of neural signals without considerable mental efforts of the subjects (885, 887). Passive BMIs could, for example, improve human interactions with a technical system by monitoring and decoding neural signals representing cognitive and emotional states, while making appropriate adjustments to the technical system.
instructed delay task, where cortical neuronal firing modulations (143, 171, 535, 846, 859), and even spinal cord interneurons (653), change their activity in the absence of overt limb movements. Similar no-go requirements can be found in BMI task designs. For example, Ganguly et al. (284) attached monkeys’ arms to a KINARM apparatus and required the animals to maintain constant arm position on each trial, while moving a screen cursor using a BMI; any arm movements cancelled the trial. These experimental settings are virtually identical to a classical instructed delay task, with the exception that in the instructed delay task, the arm eventually moves to the target, whereas in the BMI task, arm movements were not required. Such a requirement to pay attention to the arm position is different from other studies where animals were not encouraged to pay attention to the arm position, which was irrelevant for the BMI performance (114, 377, 463). This difference is important because the results of such experiments are often interpreted in terms of incorporating an external effector into the brain’s own representation of the subject’s body (466). To validate such an interpretation, it matters whether the subject refocused attention to a new effector or continued to attend to the arm position and possibly used it as a reference for a BMI-controlled cursor.
375
376
BRAIN-MACHINE INTERFACES
The existence of a correlation between neuronal firing rate and a given behavioral variable is classically referred to as neuronal tuning. When one says that a neuron is tuned to a behaviorally relevant parameter, this simply means that the neuronal rate is correlated with that parameter, and this correlation is consistent. In this context, two key physiological properties of neuronal circuits have accounted for the feasibility of creating functional BMIs which can generate consistent motor outputs. First, BMIs have benefited from one of the early discoveries of neural ensemble physiology: that the trial-to-trial variability in firing rates of single neurons, which is often described as neuronal noise, can be significantly compensated by recording simultaneously from ensembles of neurons (253, 292, 462, 467, 583). In other words, combining contributions from many neurons reduces the uncorrelated noise produced by individual neurons while leaving intact the consistent component of firing modulations, best represented by the entire neuronal population. The second physiological property that proved to be indispensable for the proper operation of BMIs is the occurrence of neuronal plasticity, i.e., the ability of neurons to continuously adapt their tuning when exposed to novel task contingencies and external world statistics (see below). The relevant literature on neuronal tuning starts with the work of Edward Evarts who pioneered the investigation of the physiological properties of single M1 neurons in the mid-1960s (233, 235, 236). By using sharp-tip electrodes, Evarts sequentially recorded extracellular electrical activity of single M1 neurons, while his monkeys remained awake and performed a variety of motor tasks (227, 228, 230 – 232, 234, 238). These now classical experiments revealed that M1 neurons were tuned to parameters such as muscle force and joint torque. For example, an M1 neuron would increase its firing rate when the monkey pulled a lever and decrease firing when the monkey pushed it. As such, M1 neuronal firing modulations coded the next push or pull movements performed in a task trial. Combining neuronal
firing rates from many trials of each kind allowed Evarts to describe an average response curve, which was named perievent time histogram (PETH). Evarts’ original findings triggered a major push in systems neurophysiology, resulting in the widespread use of his technique for single-unit recording to characterize the tuning properties of individual neurons in various areas of the rhesus monkey’s brain. In the case of the motor cortex, the next breakthrough was produced by the seminal discovery of Apostolos Georgopoulos and his colleagues that M1 neurons exhibit broad tuning to the direction of arm movement (295, 299, 423, 710). To reach this conclusion, the Georgopoulos laboratory measured the firing discharge patterns of individual M1 neurons, while their monkeys performed arm reaching movements that started at an initial, central location and ended on peripheral targets (FIGURE 14). Analysis of these center-out movements revealed that the single neuron’s firing rate peaked for a particular movement direction, called preferred direction, and decreased gradually when the direction deflected from the preferred one. Georgopoulos graphically represented the relationship between neuronal rate and movement angle as the directional tuning curve (290, 710). He also proposed a cosine fit, where the neuronal rate was proportional to the cosine of the angular deviation from the preferred direction. Further exploration conducted by many groups into the activities of cortical neurons during motor behaviors revealed many additional characteristics of neuronal tuning, such as representation of sensory signals (464, 505, 565), sensorimotor transformations (194, 297, 298, 400, 401, 860), simultaneous encoding of the motor goal and the direction of spatial attention (471), representation of multiple motor plans (143), and even encoding of cognitive variables by M1 neurons (291). With the accumulation of these findings, it became progressively clear that the next step toward the understanding of brain encoding would require sampling the activity of populations of neurons recorded simultaneously, instead of recording one neuron at a time. In the case of BMIs, this step was imperative because single neurons or small neuronal samples (10 –30 neurons) could not sustain BMI operational accuracy and stability.
B. Neuronal Ensemble Physiology At present, it has become conventional for BMI studies to report that decoding of behavioral parameters from neuronal activity improves with an increase in neuronal sample (114, 261, 377, 463, 466, 468, 583, 852). The relationship between the sample size and decoding accuracy is depicted by a NDC (FIGURE 11), an analysis introduced by our lab-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
783
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
of how ethologically meaningful information is encoded by individual neurons and neuronal ensembles (106, 142, 237, 293, 401, 578, 583, 671). Despite these extensive observations, we are still far from understanding the physiological mechanisms guiding the dynamical operation of neural circuits in mammals. This fundamental lack of knowledge, however, has not precluded BMI research from adopting an empirical approach, where parameters of interest are extracted from neuronal signals. Any decoding approach is always based on the existence of some degree of correlation between neural activity and those parameters of interest, but rarely on an unequivocal establishment of a causal relationship or a clear knowledge of the neural mechanisms involved. Yet, as we previously argued (466, 583), empirical approaches employed in BMI research could help us to uncover fundamental neurophysiological principles governing the operation of brain circuits.
377
LEBEDEV AND NICOLELIS
A -500
-500
0
500
0
1000
500
1000
-500
90°
0
500
1000
0° 0
500
-500
0
1000
500
-500
1000
-500
0
T
-500
B
0
C
500
0
500
500
1000
1000
M
1000
60
90°
180°
0°
Impulse/Sec
40
20
270°
0 45°
135°
225°
315°
Direction of Movement FIGURE 14. Broad tuning to movement direction of a motor cortex neuron. In this study of Apostolos Georgopoulos and colleagues, monkeys performed center-out arm movements in different directions. A: raster displays of the neuron activity for eight movement directions. B: arm trajectories. C: directional tuning curve showing neuronal firing rate as a function of movement angle. [Adapted from Georgopoulos et al. (294).]
oratory in 2000 (852). Following this original publication, this analysis quickly became a standard in the literature on neural decoding (50, 78, 283, 552, 564).
784
Based on a decade of BMI studies in our laboratory, we have proposed a series of principles of neural ensemble physiology, derived primarily from the analyses of NDCs
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
-500
378
BRAIN-MACHINE INTERFACES
Table 1. Principles of neural ensemble physiology Principle
Explanation
Distributed coding Single neuron insufficiency Multi-tasking Mass effect principle
Representation of any behavioral parameter is distributed across many brain areas Single neurons are limited in encoding a given parameter A single neuron is informative of several behavioral parameters A certain number of neurons in a population is needed for their information capacity to stabilize at a sufficiently high value The same behavior can be produced by different neuronal assemblies Neural ensemble function is critically dependent on the ability to plastically adapt to new behavioral tasks Overall firing rates of an ensemble stays constant Sensory responses of neural ensemble changes according to the context of the stimulus
Degeneracy principle Plasticity Conservation of firing Context principle
(583, 586) (TABLE 1). Several of these principles can be immediately derived from the typical NDC, which shows a rapid rise in decoding accuracy for small neuronal samples followed by a slower rise for larger populations. Accordingly, the single-neuron insufficiency principle states that individual neurons carry low amounts of information. Conversely, the neuronal mass principle states that a given neuronal ensemble should reach a certain size, in terms of number of neuronal elements, for decoding accuracy to stabilize. At this point, decoding performance does not change substantially when a few neurons are added or removed. This stabilization of decoding accuracy should not be confused with saturation. Further improvement of decoding is possible, but a substantial increase in neuronal sample is needed to achieve an appreciable effect. The level of information grows as a function of the logarithm of the neuronal sample size. FIGURE 15 shows NDCs for several cortical areas and a variety of BMI experiments conducted in rhesus monkeys. It follows from the analysis of these NDCs that typically each cortical area contains some level of useful information, regarding the decoding of a given motor parameter. Accordingly, the distributed-coding principle postulates that information is represented by the cortex in a distributed
A
B
Velocity
0.6
0.4
0.2
0
Next, the neuronal multitasking principle proposes that an individual neuron can simultaneously represent multiple behavioral parameters, for example, arm kinematics and gripping force exerted by the hand (114). Consistent with this statement, a recent review by Fusi et al. (280) confirmed that neuronal responses often represent combinations of behavioral parameters. Fusi et al. (280) suggested that such high-dimensional representation is computationally advantageous compared with a population of highly specialized neurons because it allows the nervous system to generate a huge number of potential responses from the inputs to the
Gripping Force 0.8
R2 of Prediction
R2 of Prediction
0.8
way. This means that, for a given behavioral parameter, neurons distributed within multiple cortical areas participate in the representation and processing of that parameter. As an illustration, a study that employed multielectrode recordings from dorsal premotor cortex (PMd) and ventral premotor cortex (PMv) (68) showed that both areas represented the kinematics of arm reaching and characteristics of hand grasping, the conclusion that favored the distributedcoding principle and rejected a dual-channel hypothesis that attributed the representation of reaching to PMd and the representation of grasping to PMv (673).
1
20
40
Number of Neurons
60
0.6
0.4
PMd M1 S1 SMA PP
0.2
0
1
20
40
Number of Neurons
FIGURE 15. Neuronal dropping curves (NDCs) for the decoding of arm velocity and hand gripping force in rhesus monkeys. NDCs were calculated separately for the neuronal populations recorded in different cortical areas: dorsal premotor (PMd), primary motor (M1), primary somatosensory (S1), supplementary motor (SMA), and posterior parietal (PP). A: decoding of velocity. B: decoding of gripping force. [From Carmena et al. (114).]
60
Physiol Rev â&#x20AC;˘ VOL 97 â&#x20AC;˘ APRIL 2017 â&#x20AC;˘ www.prv.org
785
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
[From Nicolelis and Lebedev (583).]
LEBEDEV AND NICOLELIS neuronal population. Indeed, our neural degeneracy principle supports the hypothesis of Fusi et al. by proposing that a given neuronal ensemble can generate many behavioral outputs and, conversely, that a given behavioral output can be encoded by a variety of neural ensembles. According to our view, such a principle, which is similar to degeneracy observed in the genetic code (221, 853), improves the robustness and flexibility of neural encoding (480).
Within the constraints of the conservation of firing principle, the context principle states that information representations by neural ensembles depend on behavioral context. Neuronal response to an event would be different depending on the set of circumstances that surround the event, for instance, whether the animal is awake or anesthetized when the same sensory stimulus is delivered. Differences in neuronal firing patterns produced under these two different conditions, according to our view, reflect the brain’s ability to contextualize information (527). Finally, the plasticity principle states that neuronal populations modify their properties when an organism adapts to novel conditions or learns new behavioral tasks, including BMI tasks. Our findings from BMI studies are consistent with these neural ensemble principles. From more than a decade of BMI research, we have learned that the best and most optimal way to extract a motor parameter from brain signals is to rely on concurrent recordings of the activity of large populations of neurons, distributed across multiple cortical areas. Clearly, this sample should include a large population of neurons located in M1, the cortical area from which most reliable kinematic and dynamic information can be obtained. Yet, surprisingly, many nonprimary frontal and parietal areas can contribute meaningfully to a BMI that continuously controls movements of an external actuator (3, 114, 852). On the basis of these findings, it is fair to say that BMI research has contributed decisively to consolidate a new view of cortical processing, one that departs from the classical dogma, proposed by the neuron doctrine, where a single neuron is considered the true functional unit of the brain, to one in which distributed neuronal populations
786
assume that key physiological role. Put in other words, BMI research truly vindicated the Hebbian view of the brain. Among multiple contributions to a better understanding of cortical ensemble physiology (583), BMI research has led to several demonstrations of new types of cortical plasticity. Indeed, since the early BMI studies, it became clear that it was only through cortical plastic adaptations that subjects could learn to control a BMI and improve their overall motor performance over time (114, 283, 377, 794). Eventually, this cortical plasticity led to the assimilation or incorporation of external actuators, such as robotic arms and legs, as if they were true extensions of the subject’s body representation that is known to exist in the brain (466, 583). BMI-associated cortical plasticity is manifested by changes in directional tuning patterns of individual neurons (114, 283, 463), alterations in temporal patterns of neuronal discharges (884), and a transient increase in the correlation between neurons within and between multiple frontal and parietal cortical areas (114, 377, 583). All these physiological adaptations mean that neuronal space in the cortex becomes devoted to representing a variety of properties associated with the artificial actuators employed by a BMI.
V. MULTICHANNEL RECORDING TECHNOLOGY A. Microwire Recording Cubes Developing methodology for reliable recordings from large populations of brain neurons is essential for further advances in our understanding of neuronal ensemble physiology and for the development of more practical BMI applications (466, 581, 583, 711). So far, the major achievements in this field have been associated with multielectrode implant methods. We start by discussing the technology employed by our laboratory for the past 25 years because it has produced the highest neuronal yield and postimplant longevity reported in the literature so far (711). Thus, in our hands, a typical multielectrode implant was originally defined, in the mid1990s, by a two-dimensional grid composed of small-diameter (12–50 m) Teflon or isonel-insulated metal microwires (452, 581, 711). This technology evolved over nearly three decades of research, through experiments in rats, mice, monkeys, and human subjects. Optimal parameters were then obtained empirically over years of experimentation. Our latest configuration of this technology is defined by so-called volumetric, movable implants, named recording cubes, introduced by our laboratory over the past 3 years (FIGURE 2) (711). A recording cube is built by first constructing a grid of polyimide guiding tubes. The grid has a 10 ⫻ 10 arrangement with 1 mm spacing between the adjacent tubes. This spacing is optimal for monkey record-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
An additional clarification of the relationship between neuronal ensemble physiological patterns and associated behaviors is provided by the conservation of firing principle. This principle explains that, due to the metabolic constraint created by a fixed brain energy budget, the overall discharge rate of a neuronal population should always remain constant. Thus, if some cells increase their firing rate to encode a motor parameter, others should reduce their activity proportionally, to allow the energy consumption to remain around a set limit. This principle is closely related to the free energy principle which states that the brain minimizes its free energy when exchanging information with the environment (273–275, 806).
379
380
BRAIN-MACHINE INTERFACES ings; denser grids can be harmful to cortical tissue. The guiding tubes are contained in a three-dimensional printed plastic case that also holds an array of miniature screws utilized to move the microelectrodes. Each guiding tube accommodates 3–10 microelectrodes. These microelectrodes can have different lengths, which allow their placement at different depths in both cortical and subcortical areas. For that reason, these cubes allow us to obtain volumetric recordings from whatever cortical/subcortical structure is sampled.
Although in the past we experimented with a variety of microelectrode materials, for example, tungsten for the shafts and gold-plated tips (478), we currently use only polyimide insulated stainless steel microelectrodes, 30 –50 m in diameter for recordings in monkeys, because they were found to have the best longevity and quality of recordings. For recordings in rats, tungsten microwires are useful because they allow recordings for up to 6 mo and do not induce any significant neuronal death or tissue inflammation, although recording quality deteriorates with time due to glial encapsulation of the microelectrodes (271). The quality of the implantation surgery is a key factor in defining the recording array performance and its long-term reliability (607). In our laboratory, primate surgery is conducted in strict sterile conditions under general anesthesia. The animal is placed in a stereotaxic apparatus, and craniotomies are made over the areas of interest. Dura mater is removed inside the craniotomy, and the guiding tubes are placed in light contact with the pia mater. The recording cube is then fixed to the skull with dental cement; stainless steel and ceramic bone screws serve as anchors. The implant is then encased in a three-dimensional printed protective cap, which can also house the components of our wireless recording system (FIGURE 2).
We usually place recording cubes over both hemispheres, including areas such as the primary motor (M1), primary somatosensory (S1), dorsal premotor (PMd), supplementary motor (SMA), and posterior parietal (PP) cortical areas. With this approach to cortical recordings, we have routinely recorded from 300-1,700 neurons in a single animal per day; good recording quality and high neuronal yield typically have continued for several years (FIGURE 16). Indeed, at the time of this writing, two of our monkeys completed more than 7 yr of cortical recordings after the original implantation surgery. The same microelectrodes can be used for both electrophysiological recordings and electrical microstimulation for many months (597–599). Viable recordings with implanted microwire arrays for 7 yr were also reported by the Rizzolatti laboratory (453). We recently designed multielectrode implants suitable for chronic implantation into subcortical structures in rhesus monkeys, such as the neostriatum, thalamus, and the hippocampus. In this design, subsets of 10 –30 microwires form bundles, staggered at 1–1.5 mm, which are inserted through individual guiding tubes. Each guiding tube is inserted into the brain at a depth of 5–15 mm (depending on the target structure). The microelectrode bundle is passed through the length of the guiding tube, and then travels an additional 5–10 mm to cover the area of interest.
B. Utah Array While multielectrode implants employing individually movable microwire bundles can be placed at a wide range of cortical and subcortical depths, several laboratories have used an array constructed of 100 rigid microelectrodes in a
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
787
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
During penetration into the brain, the microwires contained inside each tube move as a single set. This set includes the longest, leading microelectrode that penetrates the brain first. The leading microelectrode has a conical tip, whereas the other microwires in the set have either conical or cut angle tips. This configuration of the tip shape allows each microwire set to penetrate through the monkey pia. Each recording cube is compact (surface area of 0.22 mm2 per recording channel) and light (11.6 g). Since a 10 ⫻ 10 tube grid can contain up to 10 microwires per tube, each recording cube provides a total of 1,000 potential recording sensors. We typically record 1–2 individual neurons per microwire, which provides a potential sample of 1,000 –2,000 neurons per recording cube implanted. Since we usually implant 4 – 8 such recording cubes in a single monkey, the potential neuronal sample that could be recorded in each monkey ranges between 4,000 and 16,000 neurons.
Microelectrode penetrations are performed 1–2 wk after the surgery under ketamine anesthesia. Generally, it takes 1–2 wk to insert 500-1,500 microelectrodes in each animal. The penetrations are performed slowly and over multiple days to minimize tissue damage. Importantly, during each penetration, only a small number of microelectrodes are moved to avoid dimpling of the cortex and a bed of nails effect, where an electrode array cannot penetrate because the pressure is evenly distributed among many electrodes. Depending on the recording cube design, rotation of one miniature screw brings in motion the microelectrodes located in 1– 4 guiding tubes. Electrophysiological recordings are conducted simultaneously with each penetration to confirm that the microelectrodes gradually move through the cortical layers as the miniature screws are rotated. We usually penetrate the cortex with the microelectrodes spaced no closer than 2 mm, and perform penetrations with other microelectrode subsets on a different day. Once the microelectrodes are placed in their designated locations, they are never moved afterwards. Several months later, the guiding tubes are encapsulated by connective tissue, and eventually the spaces of the craniotomies may get ossified.
381
LEBEDEV AND NICOLELIS
Monkey N
06/2011
07/2010
Monkey M Monkey C Monkey G Monkey I Monkey O Monkey K Monkey Cl
25
20
15
10
2008
5
2009
2010
2011
2012
2013
2014
06/2013
09/2009
Year
FIGURE 16. Number of neurons recorded by microwire implants over time. The graph shows the average number of units sampled per 32-channel connector. Data from eight monkeys are presented. Sample waveforms are shown for one of the monkeys (monkey M), for different dates after the implantation surgery. [From Schwarz et al. (711).]
fixed arrangement, known as the Utah array or Utah probe (111). The array is micromachined from silicon. Each silicon needle is ⬃1.5 mm long. The needles’ shafts are coated with polyimide, whereas their sharpened tips are coated with platinum. The spacing between neighboring needles is 0.4 mm. Insulated gold wires make electrical contacts to the back sides of the needles. Originally, attempts at slowly lowering the Utah array into cortical tissue failed because such a dense array produced cortical dimpling, due to the bed of nails effect, resulting in an incomplete penetration of some of the individual needles (111). To overcome this problem, the inventors of this device adopted an impact insertion procedure whereby the array is pushed into the cortex at a high speed, though a pneumatic gun (680, 681). An examination of the long-term recording performance of the Utah probe in the cat cortex showed that, 6 mo after the implantation, 40% of the needles could not record neuronal activity, most likely because of fibrous encapsulation (679). Indeed, extensive fibrous tissue was detected on the explanted probes.
788
Currently, the Utah array is the only microelectrode implant approved by the United States Food and Drug Administration (FDA) for human use. The array has been employed in several clinical studies that involved examination of epileptic patients (594, 808, 850) and BMI operation (3, 83, 152, 360, 361). Yet, instead of sampling single-unit activity, most of these studies employed recordings of multiunit activity defined as electrical signals that crossed a certain voltage threshold. The degree of contamination of this signal by electrical and mechanical artifacts was not reported, and the possibility of electrical cross-talk between the channels was not analyzed. In defense of the poor quality of recordings (single-unit recordings are considered a gold standard in the field), an argument was put forward that simple threshold-crossing is sufficient for BMI control (134). While this practical consideration could be valid for some implementations, the threshold-crossing method is prone to confusing noise and artifacts with real neuronal activity. That can easily lead to the undesirable outcome that artifacts contaminate the signal used for decoding and controlling artificial actuators. These are very real possibilities that could occur in clinical studies of BMIs, particularly
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Average units per connector
30
382
BRAIN-MACHINE INTERFACES in those in which human subjects, who suffer from quadriplegia, can still move their heads. Artifacts generated by head muscle contractions or overall head movement could be accepted by a threshold-crossing method as representing valid neuronal signals. Supporting this possible scenario, a curious example of controlling a BMI by EMG artifacts sampled by an EEG cap was recently reported (160).
C. Improving Multielectrode Implants While microwire arrays currently represent the most practical solution for large-scale, multi-area recordings from both cortical and subcortical structures, there is ongoing research into new technologies. One direction of this research is aimed at minimizing the micro-movements between the implant and the brain that could result in tissue damage and inflammation. One solution is to have the implant float with the brain instead of being anchored to the skull. In the design, pioneered by Gualtierotti and Bailey in 1968 (325), a lightweight implant is tethered by a flexible cable. Several implementations of this idea have been developed (563, 574, 758). Another solution to decrease the motion between brain tissue and microelectrodes is offered by polymer-based microelectrodes that are more flexible than microwires and exert less strain on the brain tissue (4, 344, 451, 788). Such flexible arrays require specialized insertion techniques that temporarily stiffen the microelectrode shafts with biodegradable materials (344, 441) or polymers (754, 789), which are then dissolved in the tissue. Promising results were recently obtained using a sinusoidal probe anchored to a three-dimensional spheroid tip to minimize the movements between the probe and the brain (754). The other type of flexible electrodes contains magnetic materials (214, 383). These electrodes are inserted using an external magnetic field.
Tetrodes represent a popular solution to improve the quality of single-unit isolation. A tetrode is composed of four twisted microwires with blunt tips (389, 664, 857). The microwires have differing lengths, which allows sampling neuronal potentials in a small three-dimensional space surrounding the microwire tips. Each neuronal waveform sampled by a tetrode yields a different amplitude for each microwire, due to the differences in distance from the neuron to each of the four microelectrode tips. This simple geometrical arrangement allows better sorting of single units based on the extra spatial sampling dimensions added to the process. Recently, tetrode designs have been applied to multiarea recordings in primates, where a microdrive advanced up to six guiding tubes containing tetrodes to several brain areas in awake rhesus monkeys, including primary motor cortex, prefrontal cortex, neostriatum, and hippocampus (693). Ongoing research on new materials could further improve both microelectrode recording properties and biocompatibility. Keefer et al. (416) reported that coating of tungsten and stainless steel electrodes with carbon nanotubes improved the recordings by decreasing the microelectrode impedance and enhancing charge transfer. Suyatin et al. (779) developed an electrode based on gallium phosphide nanowires with the sensor composed of a deposited metal film. Overall, nanomaterials are a promising research area because they can provide better biocompatibility (66, 433, 446), spatial resolution (215, 799), and electrical properties (32, 118). Using pure carbon nanotube probes is another promising step in this development (880).
D. Neurotrophic Electrode Neurotrophic electrodes were developed by Philip Kennedy in the late 1980s to early 1990s (418, 420, 422) as an endeavor to produce a long-lasting solution for the recordings of brain activity in paralyzed and locked-in patients to aid the communication of these patients with the external world. The idea was to make these electrodes biocompatible using neurotrophic factors that evoke growth of neu-
Physiol Rev â&#x20AC;˘ VOL 97 â&#x20AC;˘ APRIL 2017 â&#x20AC;˘ www.prv.org
789
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Several biocompatibility issues have been reported for Utah arrays. The available data on human cortical tissue responses to Utah arrays indicate that these implants cause tissue reactions, such as microhemorrhaging, microglia activation, and long-term inflammation with the level of severity depending on the tissue damage during the implantation (251). Some of these unwanted effects could be caused by micro-movements between the brain and the needles. In addition to these biocompatibility issues, the Utah probe is suitable for recordings only from flat cortical surfaces, not from the sulci. For the flat cortical surfaces, recordings cannot be obtained from sites located deeper than 1.5 mm. Because of these shortcomings, the Utah probe cannot be considered as the final solution for human implants despite its current use in clinical trials. There is a growing consensus in the literature that research should continue into developing better recording technologies suitable for humans (251, 271, 391, 624, 646).
Several microelectrode designs have explored the possibility of placing recording points along an electrode shaft to increase neuronal yield and achieve recordings from a volume of the nervous tissue. This design is exemplified by the NeuroNexus array, also known as the Michigan Probe (23, 567, 822, 844). This is an array composed of silicon-based planar electrodes with multiple recording sites. The number of electrodes per shank and the number of shanks can be configured. The array can be used for recordings from local populations of neurons and for recordings from different cortical and hippocampal layers simultaneously (174, 548). Recordings with these arrays remain good during the early recording days, after which the recording quality usually deteriorates (409, 695).
LEBEDEV AND NICOLELIS rites into the recording tip. The electrode was made of a hollow glass cone that contained three or four golden, Teflon-insulated wires glued to the cone walls. The cone was inserted in the cortex at a 45° angle to the surface. Histological analysis of rat (418) and monkey (420) implants showed that neurites grew into the cone and became myelinated. The tissue inside the cone contained axons, axodendritic synapses, blood vessels, and oligodendrocytes, whereas no microglial cells were detected. Bipolar recordings were conducted from pairs of wires. Kennedy and his colleagues reported that the cone electrode remained functional during the entire implantation period, which was 15 mo in the monkey (420, 422), 16 mo in the rat (418), and longer than 4 yr in humans (47).
In the next study, a neurotrophic electrode was placed in a speech-related area of the left precentral gyrus of a locked-in patient (327). The patient was paralyzed, with the motor output limited to the ability to produce slow vertical eye movements. The recorded potentials were transmitted using a wireless link to a speech synthesizer. The neuronal rates were converted into formants that represented small sets of continuous sounds (1-s long vowels “uh”, “iy”, “a”, or “oo”) using a Kalman filter. The sound served as auditory feedback. Aided with this feedback, the patient achieved a success rate of up to 70% on a three-vowel task.
790
Notwithstanding the significance of these results, the exact nature of electrical potentials recorded by the neurotrophic electrode remains unclear. From the information provided in these human studies, it is difficult to tell whether the signal contained neuronal spikes or field potentials picked by the microwires. To date, no other group reported using these or similar recording methods.
E. Neural Dust Neural dust is a recording method that utilizes small (10 – 100 m) sensors (“dust”) that detect extracellular neuronal potentials and communicate them via an ultrasonic link to an interrogator placed under the skull (720). Each sensor contains a set of electrodes for recording neuronal activity, metal-oxide-semiconductor (CMOS) circuitry that amplifies the signal, and a piezoelectric transducer that converts electrical potentials into ultrasound. The interrogator uses ultrasound to both power the dust particles and examine their state. The interrogator also communicates with the extracranial components of the system. While the main advantage of neural dust is the absence of microelectrode shafts that could be traumatic to the nervous tissue, the methodology has not been developed yet for injecting these particles into the brain. Additionally, fundamental concerns remain regarding effects of implantation, signal quality, separation of multichannel signals, and recordings longevity. There have been no recordings yet of cortical potentials using these devices, only recordings of large compound nerve potentials and EMGs (721). Therefore, the viability of this new method for cortical and subcortical recordings still needs to be demonstrated.
F. Endovascular Electrodes Using brain blood vessels as entry points for brain recording probes is an attractive possibility because this method could allow placing the recording sensors close to neurons without breaking the blood-brain barrier, by inserting thin electrodes in capillaries (507). Endovascular stent electrodes have been used in cardiology for several decades. For example, Mirowski et al. (546) developed in 1980 an endovascular defibrillator for monitoring cardiac electrical activity and delivering defibrillating discharges when ventricular fibrillation was detected. Since that time, several modifications of such an endovascular cardiac electrode have been proposed (699, 700). The same electrode insertion methods have been used for endovascular recordings of neural activity (84, 350, 712, 797). For example, Boniface and Antoun (80) recorded EEG activity using a Teflon-coated endovascular guide wire that was inserted in the middle cerebral artery in human
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Several BMIs were implemented with this implant. A patient suffering from brain stem stroke received the implant in an unspecified cortical area and was able to control a computer cursor with the recorded signals (419). The recordings continued for more than 17 mo. The patient had residual facial movements and eye movements with nystagmus. Correlation of the recorded signals with mouth, face, and eye movements were noticed over the 4 mo following the implantation. The patient stopped making these movements afterwards and learned to control the X direction of a computer cursor. The cursor moved to the right when the recorded signal increased, but did not move when the signal decreased. After the cursor reached the right edge of the screen, it returned to the leftmost position and shifted downward (i.e., a carriage return). With this simple control, the patient improved in two tasks: 1) moving the cursor to a screen icon and staying on it for 2 s to produce synthetic speech using phrases such as “Hello, my name is JR,” “I feel uncomfortable,” “I feel too cold,” “I feel too warm,” “Please help me,” and “I am in pain”; and 2) using a screen keyboard to spell phrases that are printed on the screen or vocalized by a speech synthesizer. The patient reached a spelling rate of three letters per minute. Kennedy and his colleagues claimed the patient could dissociate neural activity from the facial EMG activity when controlling the cursor. It was also implied that the recorded signal was not contaminated by the EMGs and other artifacts.
383
384
BRAIN-MACHINE INTERFACES epileptic patients undergoing preoperative carotid artery assessment. Recently Oxley et al. (613) conducted multichannel recordings with an array of stent electrodes, called stentrodes, which they implanted into the superficial cortical veins overlying the sheep motor cortex. Blood vessels as thin as 1.7 mm in diameter were implanted with the stentrodes, yielding brain signals that were comparable to epidural ECoG recordings. Recordings in freely moving animals continued for 190 days. The authors proposed that such their endovascular system could be used to detect seizures in epileptic patients, operate BMIs, and deliver electrical stimulation.
G. Optical Recordings Optical imaging is based on voltage-sensitive (322, 323, 621, 793) and calcium-sensitive (320, 748, 771) fluorescent dyes, and genetically encoded calcium indicators (516, 523, 892) whose signals are monitored using video recordings. A particularly powerful method, two-photon excitation laser scanning microscopy, allows minimally invasive, three-dimensional sampling with submicrometer resolution (353, 591, 782). Although these techniques require filling neurons with a dye, hardly a practical procedure for human clinical applications, several optical imaging-based BMIs have been already demonstrated. Clancy et al. (145) conducted experiments in mice with genetically encoded calcium indicator gCaMP6f in layers 2 and 3 in M1 or S1. Recordings were conducted using two photon imaging. Head-fixed mice were trained to control the pitch of a sound by modulating activity of ⬃20 optically recorded neurons. It took the animals 8 days to learn to control sound generation with this BMI. Ziv et al. (896) performed calcium imaging in freely behaving mice whose hippocampal tissue was virally infected and coexpressed GCaMP3 and CaMKII in the same neurons. A miniature (1.9 g) integrated fluorescence microscope (304) was employed for two photon imaging. While the mice explored their environment, place fields of thousands of hippocampal neurons were tracked over several weeks. Next, Bayesian decoding was employed to reconstruct the
H. Electrocorticographic Grids ECoG grids, containing several tens to several hundred electrodes, allow minimally invasively recordings of multichannel field potentials from large cortical territories (172, 357, 483, 544). A recent trend in this approach was to miniaturize the electrodes and increase their density (79, 542, 825, 838, 895), leading to improvements in spatial resolution of the recordings. The efficiency of recordings with high-density ECoG can be improved using electronics embedded in the grids. For example, Viventi et al. (825) developed ECoG grids with embedded flexible nano-membrane transistors that performed amplification and multiplexing. This technology reduced the number of connecting wires while increasing the number of recording channels to several thousands. Fu et al. (278) developed flexible mesh electronics with micrometer components and bending properties matching those of neural tissue. The mesh consisted of 16 electrodes and could be injected in the mouse brain using a syringe. The implant yielded stable recordings of LFPs and singleunit activity for 8 mo. The same probe was used for longterm electrical stimulation.
I. Amplification, Processing, and Transmission of Neuronal Activity A modern neuronal recording system typically includes several signal processing components. The preamplifier is usually placed near the subject’s head and is often called the headstage. It performs an initial amplification (typically with the gain from 1 to 20) and decreases the output impedance. This preamplification is needed to reduce the noise added at the next signal transmission stage. Depending on the system, the headstage output remains analog (601) or is digitized (135, 340, 711). The headstage may also perform signal multiplexing for the reduction of the number of cables in a tethered system. The headstage is connected, via a tethered or wireless link, to an external processing unit that performs further amplification and/or filtering. Next, the neural signals are digitized if they were not digitized by the headstage. The digitized signals are then sent to a computer for further processing, including neural decoding for running BMI tasks. While tethered systems were used in early BMIs (114, 124, 725, 794, 852) and are employed in human clinical trials (3,
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
791
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Nanoscale electronics is a promising method for recordings from brain capillaries (507). Masayuki Nakao, Rodolfo Llinas, and their colleagues developed nanotechnology probes composed of insulated Wollaston platinum wires, 0.6 m in diameter (507, 842). The feasibility of these recordings was demonstrated in the frog spinal cord. The sensors were introduced to the bloodstream through a polyimide tube, 90 –300 m in diameter. The wires “sailed” within a blood vessel until they straightened and could be used for recordings.
animal location within the environment from the optical recordings. The decoder was trained on the recording data collected on one day, and tested on the data the same day or different days. The decoding accuracy was the highest for the same day, but declined modestly for the different days. The authors explained these results by the dynamical changes in the hippocampal place fields.
LEBEDEV AND NICOLELIS
VI. DECODING OF BRAIN SIGNALS A. Principles of Neural Decoding Modern real-time BMI computational algorithms, or decoders, are employed for transforming neuronal activity into signals suitable for direct communication of the subject’s brain with artificial actuators. Such decoders can employ a large variety of statistical and machine-learning methods. BMI decoding algorithms belong to the class of multiple-input and multiple-output (MIMO) models (432), where multiple inputs are provided by the neural recording channels and multiple outputs correspond to the behavioral variables controlled by the BMI and/or signals for communication with the external world. A decoder applies a transform algorithm to neuronal inputs to calculate the output variables. In many cases, the transform algorithm has many independent parameters that need to be mapped to a much smaller list of output variables. Setting the values of these parameters is called decoder training. There are different methods to train a decoder. The most traditional one requires sampling an initial segment of input data from which correlations between neuronal signals and behavioral variables of interest are determined. The decoder performance is
792
evaluated using the comparison of the actual behavioral parameters and the values derived by the decoder from the neuronal signals. After the decoder reaches high performance for the training segment, the BMI mode of operation can begin (851). At this point, the parameters needed to control an external device are derived solely by real-time decoding of the incoming neural activity. The original BMI experiment performed by Carmena et al. (114) (FIGURE 5) can be used as an example of the traditional training paradigm to create a BMI decoder. The experiment started with monkeys using one hand to operate a joystick linked to a robotic arm. The joystick movements were translated into the reaching movements performed by the robot. Additionally, monkeys activated the robot gripper by squeezing the joystick handle. This hand-control phase lasted for ⬃15 min. Neuronal recordings obtained during this period provided the training data for a linear decoding algorithm, called the Wiener filter (see below), which represented the robotic arm kinematics and the gripping force as weighted sums of the neuronal discharge rates. Once the Wiener filter was trained, the operation was switched to BMI control, where the joystick was either electronically disconnected from the robot or in some experiments physically removed from the setup. At this point, the monkeys used the Wiener filter’s outputs to directly control the robotic arm’s reaching and grasping movements. This type of BMI operation was called brain control. The shift from the training phase to brain control is often accompanied by changes in the patterns of overt movements produced by monkeys (114, 124, 463). Often, animals tend to diminish or eliminate their arm movements when operating a BMI in brain-control mode. Furthermore, the animals’ cortical neuronal firing patterns change significantly during the transition from the training period to brain-control mode. These physiological changes are manifested as increases in correlation between cortical neurons, within and between cortical areas, (114, 583), and alterations in the directional tuning of individual neurons (283, 463). If these changes occur, the decoder performance may deteriorate because it was trained under different behavioral conditions. To mitigate this problem, several strategies have been designed to adapt to these new behavioral and physiological conditions (114, 180, 283, 492, 609, 794). For example, the initial training period can be eliminated and the adaptation could begin from arbitrary decoder settings and continue throughout the experiment (283). Additionally, the initial training can be conducted without any overt movements, using only passive observations by the subject (377, 800, 827) and/or their mental imagery of movements (360, 362). As mentioned above, after the original introduction of linear models for neural decoding in BMIs (374, 852), many other decoding algorithms have been proposed over the past decade. Indeed, describing a new BMI real-time com-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
83, 360), the increasing numbers of simultaneously sampled neural channels, the need to study BMIs while animals freely behave and the eventual goal of making BMIs fully implantable have prompted the development of wireless technologies. FIGURE 2 shows our recently developed wireless BMI, which has a 512–1024 channel capacity (711). This system includes four modules: 1) digitizing headstages housed in a monkey headcap, 2) wireless transceivers also housed in the headcap, 3) a wireless-to-wired bridge for bidirectional signal transfer, and 4) client software. Each headstage samples 32 channels, and each transceiver module connects to 4 headstages for a total of 128 channels handled by one transceiver. Up to four transceivers have been demonstrated to work simultaneously in rhesus monkeys. Recently, we found a solution to add 4 more transceivers for a total of 1024 channels. The bidirectional wireless link serves to transmit neural information to the external units and to set the headstage/transceiver parameters by the operator. Spike sorting operations are performed by the transceiver, which reduces the amount of neural information transmitted wirelessly. The headstage and transceiver are powered by a lithium-ion cell, which is housed in the headcap and operates for 30 h continuously. This wireless, multichannel recording system is connected to a BMI suite. The system has been already used in a variety of BMI tasks, ranging from BMI control of a cart by a monkey housed in its home cage (711), to a monkey performing BMI whole-body navigation tasks while seated in a motorized wheelchair (656). Several multichannel, wireless recording systems have been developed by other groups, as well (81, 119, 135, 340, 429, 550, 557, 602, 783).
385
386
BRAIN-MACHINE INTERFACES putational decoding strategy is the theme of a growing literature in the field. Below we provide a brief summary of the main BMI decoders reported in the literature.
B. Linear Decoders
Since its introduction, the population vector approach to neural decoding has been very influential even though its original description was not based on simultaneous recordings from many neurons. Instead, Georgopoulos and his colleagues (295, 299) employed traditional single-electrode neurophysiology to record from a single neuron or, rarely, a few neurons at a time. Artificial neuronal populations were assembled from sequentially recorded neurons on different days, and population vectors were calculated for those populations. Although Georgopoulos later employed a sevenelectrode apparatus for his motor cortical recordings (36, 173, 570), he did not record neuronal samples large enough to enable real-time decoding of arm movements using the population vector. In the classical Georgopoulos paradigm, the neuronal preferred directions were derived from a behavioral paradigm called a center-out task. In this task, monkeys were required to perform arm reaching movements from an initial, central location, to a set of peripheral locations arranged in two(294) or three-dimensional (299) space. Once the preferred directions were determined, the motor task could remain the same, or a more sophisticated task could be introduced. For example, Georgopoulos et al. (297) employed a cognitive task where monkeys had to perform a 90-degree mental rotation from the location of a visual stimulus to the motor target instructed by that stimulus. The population vector
This early work on population-vector decoding contained an estimation of how many neurons would be needed to decrease noise in the extraction of arm kinematics from M1 ensembles (295). The decoding noise was expressed as the variability of the calculated population vector direction for different realizations of single-trial ensemble firings. This parameter was then plotted against the population size. The curve showed an initial rapid decrease in variability as the population size increased, and then followed a much slower rate of decrease after the population reached the size of 150 neurons. The slow decrease continued until all 475 neurons recorded in that study were included in the population. These estimations are consistent with our analysis of neuronal dropping curves constructed for simultaneously recorded populations of cortical neurons (467, 583). Notwithstanding the elegance and theoretical importance of the population vector approach, this decoding method is suboptimal because the weights given to different neurons are chosen intuitively and without any provision for minimizing decoding errors. Additionally, the approach where the algorithm parameters are derived from one task (centerout task) and then applied to a different motor task (e.g., mental rotation) may result in additional errors because neuronal tuning properties could be different in the new task. Such a change in neuronal tuning is consistent with the context principle discussed above: under different conditions, cortical neurons tend to exhibit different activity patterns. Another decoding algorithm, the Wiener filter, has been successfully employed in various BMI studies to extract limb kinematics and other behavioral variables from neuronal population activity (114, 283, 284, 432, 468, 656, 852). The Wiener filter is an optimal linear decoder set to minimize mean-square error (347, 485, 854). In a typical implementation, the Wiener filter output for time t is computed as a weighted sum of neuronal rates sampled at several time points, called taps or lags, preceding t. Ten taps with 100 ms spacing is a typical setting in monkey BMI experiments (114, 463). The filter weights are computed by applying matrix transformations to the training data that include the recordings of neuronal rates and behavioral variables of interest. The total number of Wiener filter weights depends on the population size and the number of taps. For neuronal population of size N and number of taps T, the number of weights is equal to N multiplied by T. The number of weights, or free parameters, is referred to as the dimensionality of the decoder. An excessive number of weights may be harmful for decoding accuracy because of the problem
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
793
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Linear decoders compute the output variables as weighted sums of the recorded neuronal rates (114, 794, 851, 852). Humphrey, Schmidt, and Thompson first proposed this idea in 1970 (374). They successfully demonstrated that movement parameters can be reconstructed from the recordings of the firing rates of multiple neurons using multiple linear regression. Schmidt and his colleagues then ran this linear decoding in real time (702, 703). Georgopoulos was a notable proponent of the theory that such weighted summation constitutes a fundamental mechanism by which cortical neuronal populations represent motor variables (295, 297, 299). Georgopoulos’ theory expressed the contribution of each neuron to population encoding as a vector that pointed to the so-called preferred direction of that neuron (294, 710). The preferred direction was defined as the direction for which the neuronal firing rate was maximal. Next, the individual-neuron unit vectors, multiplied by the discharge rate of the corresponding neuron, were summed to form a population vector. The population vector turned out to be an excellent method to continuously extract the direction of arm movement from the activity of a population of M1 neurons.
proved to be informative of the representation of mental rotation by M1 neurons even though the neuronal preferred directions were derived from a simpler, center-out task.
LEBEDEV AND NICOLELIS known in many disciplines as overfitting (38, 338, 345) or the curse of dimensionality (821). An overfitted decoder may start fitting data to noise. Such fitting may appear to work perfectly for the training period, but the decoder would then fail when applied to a different data segment. To reduce overfitting, BMI decoding algorithms incorporate such methods as regularization and dimensionality reduction (432).
C. Kalman Filter The Kalman filter (403, 404) is another popular algorithm for BMI decoding (432, 491, 620, 724, 869) that was previously employed in numerous engineering applications (319). Like the Wiener filter, the Kalman filter takes multichannel neuronal signals as inputs and returns predictions of behavioral variables as its output. The filter models neuronal inputs as observations and the behavioral variables as state variables. The state variables may include, for example, the position and velocity of the arm. The Kalman filter updates states in discrete time steps (usually 50 –100 ms). Each update consists of two calculations. The first calculation, called predict step, provides an estimate of the next state from the previous state based on a state transition model, also called a movement model in BMI applications. For a robotic arm movement, for example, the next state can be estimated based on the previous position and velocity. The second calculation, called update step, performs an adjustment of this estimated state using the observed neuronal rates. This calculation employs an observation model, or neuronal tuning model, that represents neuronal rates as a function of state variables. The directional tuning curve is an example of such an observation model as it describes neuronal firing rates as a function of movement direction. During the update step, the observation model converts the estimation of state into an estimation of expected neuronal rates. Then, the estimated neuronal rates are compared with the actual recorded neuronal rates, and the state is adjusted to accommodate this difference. This final computation of the state forms the filter output.
794
While several groups have reported that the Kalman filter outperforms other linear decoders (724, 869), there is also a report of a very similar performance of these methods for a different dataset (432). Thus, currently, there is no general recommendation on which algorithm should be chosen for BMI decoding. Instead, choices should be made after concrete conditions and requirements are examined. A few years ago, our laboratory introduced an improvement to the Kalman-based BMI decoder, based on the unscented Kalman filter (UKF) (491). This algorithm was designed for handling nonlinear observation and state transition models (392). The UKF is relevant for neural decoding because there are nonlinearities in the relationship between neuronal rates and limb kinematics, such as the one relating neuronal tuning to speed in addition to velocity (492, 554). In our implementation, an nth order UKF included two novel features: 1) a nonlinear model of neural tuning which incorporated absolute values of velocity and radius, and 2) an addition of n-1 recent states to the state variables. This new decoder outperformed the classical Kalman filter and the Wiener filter when applied to the data from a center-out task and a target tracking task. Moreover, the UKF outperformed the other decoders when used for real-time BMI control of movements of a computer cursor. Following this study, we have used the UKF in a number of BMI studies, including a BMI for controlling one avatar arm (598) or two avatar arms simultaneously (377).
D. Point-Process Models Point-process models of neuronal spiking activity employ a likelihood function to describe the probability of a neuron to produce a spike. The likelihood function depends on such parameters as the neuronal spiking history, activity of the other neurons in the population, external stimuli, and behaviors (144, 809). An analog of the Kalman filter can be formulated using a point-process model of the observation state (97, 222, 490). Indeed, both the Kalman filter and point-process decoders utilize the concept of state to describe both the decoded variables and neural activity. State transitions are described by statistical models (450). Although computationally demanding, point-process decoders in certain cases offer better temporal resolution compared with neural decoders that decrease the resolution by down sampling the neuronal activity into bins (with a typical bin width of 50 –100 ms) (461, 728 –730, 807, 816, 872).
E. Artificial Neural Networks Artificial neural networks (ANNs) were introduced as decoders in the early BMI studies of the late 1990s (124, 852). Since then many ANNs have been used in both invasive and
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Over the past 17 years, our laboratory has employed multiple Wiener filters, running in parallel, for generating multiple BMI outputs. For example, Carmena et al. (114) implemented two Wiener filters that generated x and y components of the robot velocity, and a third filter that generated the robot’s gripping force (FIGURE 5). The same approach was subsequently used to extract kinematics of lower limb motion during bipedal locomotion (261), decode arm EMGs from cortical ensemble activity (694), decode cortical representation of time intervals (468), and control whole-body navigation by monkeys seated in a motorized wheelchair (656). Overall, this is a powerful and easily tractable decoding method.
387
388
BRAIN-MACHINE INTERFACES noninvasive BMIs, including a multilayer perceptron (22, 161, 369, 428, 431, 691), adaptive logic network (444, 445), tree-based neural network (380), and learning-vectorquantization (402, 460, 632). Krishna Shenoy and his colleagues (776) have introduced a decoding method based on a dynamical ANN, called recurrent neural network (RNN). In this algorithm, neuronal activity is considered as a function of its history in addition to being related to motor parameters. In Shenoy’s study, the RNN continuously decoded the kinematics of center-out arm reaching movements from monkey M1 activity. This decoding scheme outperformed the velocity Kalman filter in the same task.
Adaptive decoders make adjustments that improve BMI performance while the subject continuously operates a BMI. While the decoder adapts, changes occur in the brain itself owing to neuronal plasticity. The first adaptive decoder introduced in BMI literature was a coadaptive algorithm implemented to improve real-time conversion of the activity of monkey M1 neurons into the three-dimensional center-out movements of a cursor (794). The cursor position was generated by a population-vector decoder. The coadaptive algorithm adjusted the population vector weights after each trial to bring the BMI-generated trajectories closer to the ideal trajectories connecting the cursor’s initial position to the target. Following that study, many adapting algorithms have been developed. Li et al. (492) developed a Bayesian regression self-training algorithm that updated the settings of a UKF. That BMI utilized neuronal ensemble recordings from multiple cortical areas in rhesus monkeys to control two-dimensional cursor movements. The adaptive algorithm monitored the decoder output and periodically updated the UKF neuronal tuning model based on the detected changes. The updates were performed using Bayesian linear regression. The online performance of this algorithm was tested in 11 experimental sessions that spanned 29 days. The initial parameters of the decoder were trained on the first day of recordings, and the evolution of these parameters was performed by the adapting algorithm without any retraining sessions. The adaptive decoder secured stable BMI performance, whereas the performance deteriorated if the unchanged initial decoder was used. Dangi et al. (181) used a similar two-dimensional reaching task as a test bed for developing a general framework for selecting parameters to adapt and the adaptation timescale. They also developed tools that evaluated convergence properties of adaptive algorithms. Several studies introduced supervised learning algorithms that used the information about target location to adap-
Justin Sanchez and his colleagues (197) employed reinforcement learning as a BMI adapting algorithm. In this approach, actions that maximized the reward were selected through trial and error. Trial outcome (i.e., presence or absence of reward) served as a scalar signal that was utilized for parameter updates at the end of each trial. Two studies from this group (520, 640) showed that actor-critic reinforcement learning could quickly recover decoding accuracy when neural inputs were lost or shuffled. Moreover, the same group showed that the reinforcement signal for such learning could be derived from the recordings in nucleus accumbens (520).
G. Discrete Classifiers Discrete classifiers convert neuronal activity into discrete choices. These decoders are commonly used in noninvasive BMIs, where subjects generate a limited number of outputs, often just two (333, 569, 639). Discrete classifiers have been used in some intracranial BMIs, as well (343, 692), including systems that combined both discrete and continuous decoders (261, 816). The mathematical algorithms for discrete classification include linear discriminant analysis (LDA) (259, 288, 629), support vector machine (56, 288, 780), ANNs (359), multilayer perceptron (43, 75), hidden Markov models (603, 655), k nearest neighbors classifier (177), and nonlinear Bayesian classifiers (191).
VII. MOTOR CONTROL WITH INTRACRANIAL BMIs A. Theories of Motor Control as a Foundation for Motor BMIs Motor BMIs extract motor commands from a sample of neuronal activity and send this control information to external devices that execute the movements imagined by the operator. At the basic science level, these systems are in-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
795
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
F. Adaptive Decoders
tively improve BMI performance. Kowalski et al. (448) employed a naive adaptive BMI for this purpose. The system jointly analyzed neuronal patterns and user intent of reaching to the target. Shanechi and Carmena (728) used a similar idea of analyzing the user intent. They developed an adaptive optimal feedback-controlled point process decoder that derived subjects’ intentions from the relative position of the cursor and target. This adaptive scheme improved the performance even when the decoder’s initial parameters were set to arbitrary numbers. Along similar lines, Suminski et al. (774) developed a kinetic decoder that continuously adapted joint torques based on the discrepancy between the target location and hand position. The updates were performed using gradient descent.
LEBEDEV AND NICOLELIS tended to investigate the physiological properties of motor circuits, theories on neuronal encoding, and the impact of learning and plasticity on neuronal ensembles. From a clinical point of view, BMIs primarily aim to restore crucial motor behaviors, such as arm movements or locomotion, to patients suffering from devastating levels of body paralysis, as a result of brain trauma or degenerative neurological diseases.
Several theories of motor control have influenced the design and experimentation with BMI. For instance, the concept of body schema, a quite important concept for modern BMI research as we discuss below, was originally proposed by Head and Holmes one century ago (351). According to Head and Holmes’ original formulation, the brain creates an internal model of the body, the body schema, which governs motor activities and perceptions. The body schema is constantly updated by streams of sensory information. With the emergence of BMI research, the concept of body schema was not only investigated but acquired a complete new angle; one in which artificial prosthetic limbs, con-
796
trolled directly by the patient’s own brain activity during the utilization of a BMI, are believed to be assimilated into the body schema as extensions of the subject’s biological body, through the process of plasticity triggered by BMI long-term usage (466). Modern theories of motor control are rooted in the ideas of Head and Holmes and, as such, have become the subject of investigation by BMI research. The internal model theory (310, 415, 459, 866) describes the motor system as being defined by two components: the controlled object (e.g., a body part or the entire body) and the controller. The controller uses an internal model to program future motor states. When the object movement is executed, the controller compares the expected state with the actual sensory feedback from the controlled object. If a discrepancy between the expected and actual state is detected, the controller issues a correction command. It has been suggested that to be efficient, BMIs should perform similar forward planning based on an internal model (175, 309). Another popular motor control theory, Feldman’s equilibrium point theory (249, 250) suggests a possible neural mechanism to implement the controller. In this theory, higher-order motor centers manage the position of an equilibrium point for the limb, and the limb is brought to the equilibrium point by a spinal servo mechanism. BMIs with a similar separation between the higher-order and low-order controls have been proposed. In this design, called shared-control BMI, high-order motor commands are extracted from cortical activity, whereas the low-order execution is delegated to a robotic controller, which handles the “equilibrium point” using Feldman’s terminology (427, 634). Optimal feedback control is yet another popular motor control theory (263, 801, 802). This theory describes an optimal strategy for using multiple biomechanical degrees of freedom to achieve the goal of a motor action. The strategy is based on stochastic optimal feedback control that corrects deviations in the degrees of freedom that define task goals, while allowing variability in task-irrelevant dimensions. The theory explains such phenomena as motor variability, error corrections, and motor synergies. Several BMI decoders that implement optimal feedback control have been proposed (59, 729, 730). These BMIs estimate the performance error by comparing the current location of an actuator with the planned trajectory estimated from the neuronal signals. A correction is then issued in the appropriate dimensions.
B. BMI Control of Virtual and Robotic Limbs Motor BMIs that enable upper limb functionality, for example, a BMI for arm reaching and grasping (114) (FIGURE 5), have received particular attention because of the obvious
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
The design of existing motor BMIs, in many ways, matches current theories of the motor system layout and operation. The first design issue is where in the brain to record neural activity that would be converted into motor commands to an external device. This issue is closely related to the assessment of functional roles of different brain areas and the types of neural processing they perform. Since the motor system in primates is defined by a highly interconnected network of cortical, subcortical, and spinal structures, in general, there are many brain areas that could provide inputs for motor BMIs. Classically, the motor system is described as being formed by a hierarchy, in which cortical motor areas are presumed to handle advanced or higher order functions, for example, dexterous hand movements. Meanwhile, lower-order subcortical areas are presumed to manage less complex, automated motor acts. In this motor hierarchy, the spinal cord has been traditionally believed to handle low-order functions, such as reflexes (733) and central pattern generators (CPGs) (328). Reflexes are automated, and often unconscious motor responses to sensory stimuli. In contrast, voluntary movements are prepared and executed under cortical control. They may be related to external stimuli, but may also originate in the mind rather than being caused by sensory inputs. While there are merits to the classification of motor activities into less advanced, automated responses and more advanced, voluntary actions (65), our work has repeatedly shown that there is a constant flow of information between cortical, subcortical, and spinal structures during the execution of motor behaviors (576, 578, 617, 894). In this distributed view of the motor system, there is no clear-cut separation between high-order and low-order processing. In support of our view, practically any motor task involves a mixture of voluntary and reflex activities (158).
389
390
BRAIN-MACHINE INTERFACES key importance of arm movements in our daily life. The first BMI of this type operated in an open-loop mode, i.e., without any sensory feedback from the BMI-controlled actuator (852). In that experiment, while New World monkeys manipulated a joystick, their cortical activity was decoded and converted into the movements of a robotic arm using an Internet protocol. The monkeys did not see the robot, which was located in a different state many hundreds of miles from the animal. All subsequent BMI demonstrations utilized a robotic arm operated in a closed-loop mode (114, 463, 794, 817), where monkeys received visual feedback of the robotic arm movements and could correct their performance errors.
Two monkeys initially executed a motor task, placing a computer cursor to the center of a moving circle that served as a target, manually (FIGURE 5B). To do this, monkeys grasped a joystick and shifted it in different directions; the joystick position was translated into the cursor position on the screen. Later stages of this behavioral task also required that the monkeys apply gripping force to the joystick handle, at the end of the reaching movement, so that they could imitate grasping the virtual target. Next, monkeys learned to control the reach and grasp movements of the robotic arm equipped with a gripper. Since the joystick was connected to the robot arm, when the monkey moved the joystick and applied hand gripping force to it, the robot arm and gripper reproduced these movements. The visual feedback of the robot movements was delivered to the computer screen, where the robot position was represented by a circular cursor and the gripping force was represented by the cursor diameter. The virtual targets that the robot had to reach and grab were represented by circles of varying diameters. To win a fruit juice reward, monkeys had to move the robot, place its gripper over the virtual target, and then produce the correct level of gripping force, to match the cursor diameter with the diameter of the target. While monkeys practiced these reach and grasp task, the firing rates of ⬃100 cortical neurons, distributed across the cortical areas mentioned above, were fed into multiple Wiener filter algorithms so that multiple parameters could be
Several other groups developed BMIs for arm reaching, as well. Schwartz and his colleagues have explored the possibility of performing BMI control over reaching in three dimensions (3D). In one study, they trained monkeys to wear stereoscopic goggles that displayed 3D movements of a cursor (794). In the beginning of each trial, the cursor was positioned in the center of this virtual reality display. A spherical target then appeared at a random 3D location, and the monkeys acquired it with the cursor to receive a reward. During the manual control mode, monkeys waved their hands in the air to move the cursor. The hand movements were monitored by a video tracking system. During the brain control mode, the cursor was moved by cortical activity processed by a decoding algorithm. Initially, the researchers attempted to train a population vector decoder using the manual performance data, and then use that decoder for brain control. However, after realizing that this control was not sufficiently accurate, they sought an adaptive algorithm that would improve the performance. Their adaptive decoder, called coadaptive movement prediction algorithm, adjusted the decoder parameters so that the trajectories generated by the BMI were brought closer to the ideal linear trajectories connecting the initial central position and the target. Building on these results, the Schwartz laboratory developed a BMI for monkey self-feeding (817). For this purpose, they used a robotic arm equipped with a gripper that picked a piece of food and brought it to the monkey’s mouth. These experimental settings resembled the previous study of Lebedev and Wise where a robotic manipulator brought food to monkeys (471). In Schwartz’s study, the robot was controlled by a linear decoder that transformed cortical neuronal population activity into the velocity of the robot’s end point. The gripper’s opening and closing was commanded
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
797
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
The main components of a motor BMI that controls a robotic arm are featured in the system developed in our laboratory in 2003 (114) (FIGURE 5). Here, rhesus monkeys learned to control reaching and grasping movements performed by a robotic arm by using only the combined electrical activity of cortical ensembles recorded with multielectrode arrays, built from flexible Teflon-coated microwires chronically implanted in multiple cortical areas, including M1, S1, PMd, supplementary motor area (SMA), and posterior parietal cortex (PPC). These cortical areas were chosen because they belong to the frontal-parietal cortical circuitry that controls goal-directed arm and hand movements (20, 399, 470).
generated continuously to control the reach and grasp movements of the robotic arm. In brain-control mode, the joystick was electrically disconnected from the robot and the outputs of the Wiener filters defined the robot movements. The monkeys continued to manipulate the joystick with their hands although it was disconnected from the system. After the monkeys perfected this brain control assisted by the joystick movements, the joystick was removed from the setup. At this stage, to receive its fruit juice reward, the monkey could no longer rely on the well-trained joystick task. Instead, they had to learn to control the robot with their own cortical activity without assisting themselves with arm movements. The performance errors were initially high, but then decreased as the monkeys practiced in the brain control without hand movements. Learning to control this BMI was accompanied by a transient increase in correlation between the simultaneously recorded cortical neurons (114, 583), within and between cortical areas, and by changes in neuronal tuning to the robot arm movements (463).
LEBEDEV AND NICOLELIS by cortical activity, as well. The authors reported a curious type of learning during these BMI operations: monkeys learned to start opening the gripper before it reached the target. They could do this without risking dropping a piece of food because marshmallows that stuck to the gripper were used as rewards. Although the monkeys possibly focused less on controlling the feeder because of the sticky rewards, this observation illustrates that BMI control, like normal motor control, can undergo adaptation.
John Donoghue’s group at Brown conducted several BMI studies in implanted humans. In these studies, paralyzed patients were implanted with the Utah array in the M1. The patients learned BMI control of a screen cursor (361) or a robotic arm (361). One of the patients learned to grasp a coffee bottle with a robotic hand and, somewhat slowly (more than 1 min per trial), bring it to her mouth. The slowness of operation was possibly related to a deteriorated quality of neuronal recordings. The study did not document the number of neurons performing the control. Instead, it reported that neuronal electrical signals were picked up by 96 recording channels. A simple threshold crossing procedure was used to detect multiunit activity. The decoding was performed using a Kalman filter that was initially trained to predict robot hand displacements as the patients observed the movements of the robotic arm and imagined themselves controlling those movements. The Kalman filter decoder was iteratively adjusted during the phase of BMI control. To ease the learning, the patient’s performance was corrected by computer commands that brought the robot arm closer to the optimal trajectory. This procedure, called “error attenuation” consisted of decreasing the robot movement commands orthogonal to the trajectory connecting the robot to the target. The contribution from the error
798
While the experiments of Donoghue and his colleagues have demonstrated that patients with upper-limb paralysis can employ their cortical activity, recorded from the arm and hand representations in MI, to control the reaching and grasping movements performed by a robotic arm, several key questions remain regarding the nature of this control. The videos from their experiments show that the subjects could move their heads. In some of the trials, they clearly tracked the robot displacement with head movements. Arm movements accompanying the grasp command to the robot are also noticeable in the videos. These observations suggest that cortical neuronal activity related to head movements, which, in these patients, likely expanded beyond the original head representation before the trauma or disease, could have been involved in controlling the robot arm, in addition to the newly created cortical representation of the robot. Overall, the role of assistive overt behaviors in BMI control, i.e., movements of body parts and the eyes that could be used to generate neural inputs for a BMI, is often neglected or downplayed in the literature. Certainly, this topic will require more scrutiny in the future. Historically, neurophysiological experiments strived for maximal control of unwanted overt behaviors. For example, neurophysiologists have developed an instructed delay task where an animal is not allowed to produce any motor output while preparing a movement (860). However, even if an instructeddelay task is well-learned, and no overt movement occurs, motor preparation still causes activation of spinal circuits involved in low-level motor control (653). Therefore, even a very clean BMI experiment, where the subject does not move the limbs or eyes, may involve activation of both higher-order brain areas that drive the BMI, and low-order subcortical and spinal regions. This is not a problem for practical BMI implementations, but rather an issue that needs to be better understood. For practical BMIs, even the presence of overt behaviors can be useful because they could improve the subject’s performance. Indeed, BMIs that mix several brain-derived signals with the signals representing
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Following these demonstrations of unimanual BMI control, the Nicolelis laboratory took the next logical step in this research by demonstrating a BMI that controlled two artificial arms simultaneously (377). In that study, rhesus monkeys viewed two virtual arms on a computer screen, and commanded their reaching movements using a cortical BMI that utilized the extracellular activity of ⬃400 neurons sampled in multiple areas of both hemispheres, including M1, S1, PMd, and SMA. Cortical ensemble activity was converted into bimanual movements using a UKF that treated the kinematic parameters of both arms as parts of the same state model. The decoder was trained either using a joystick task where monkeys moved the virtual arms with two joysticks or through a passive observation task that required monkeys to watch the virtual arms move on the screen. Eventually, the monkeys were able to control the virtual arms by their cortical activity without moving their own arms. This learning was accompanied by widespread cortical plasticity that manifested itself by an increase in cortical responses to the observation of virtual arm movements and by changes in pairwise correlations between neurons.
attenuation routine was gradually decreased and eventually removed. The robot hand state was controlled using an LDA classifier that, similarly to the velocity decoder, was trained using observations of the robot movements combined with motor imagery. The drinking task was further assisted by a preprogrammed sequence of actions. First, the LDA classifier commanded an automated impedance-controlled grasping and lifting the bottle. Second, the same classifier stopped the movements of the robot arm and pronated the robot wrist to point the bottle toward the patient. Third, the robot wrist was brought to its initial position and arm movements allowed, and fourth, the bottle lowered to the table and released. Such a mode of operation, where control functions are distributed between a BMI operator and the robotic controller, is referred to in the literature as shared control (223, 379, 427).
391
392
BRAIN-MACHINE INTERFACES overt behaviors, for example, eye movements or EMGs, are called hybrid BMIs (337, 387, 476, 626, 833).
Altogether, these clinical studies demonstrated the feasibility of implementing cortically controlled BMIs to reproduce upper limb movements. They also exposed a number of issues that preclude immediate translation of these systems into the clinical arena (466). One issue is the requirement for practical neural prostheses to be fully implantable. Wired implants are suitable for animal experiments and short-term clinical trials, but not for devices aimed at serving as long-term clinical solutions. In a practical clinical system, implanted electrodes and preamplifiers should be fully contained under the scalp while wireless technology is used to transfer large-scale recorded neural signals. Furthermore, implant biocompatibility remains a problematic issue. The utilization of the Utah probe, in both monkeys and human subjects, has repeatedly shown that the quality of neural recordings tends to deteriorate with time due to electrode encapsulation and neuronal tissue loss, likely as a result of the tissue injury caused by the electrodes. Finally, there are many challenges for real-time decoding algorithms, which currently are limited to small sets of motor behaviors.
C. BMI for Walking During the last two decades of explosive BMI development, research focused mostly on controlling neuroprosthetic devices that mimic upper limb functions. Yet, tens of millions of people worldwide suffer from paralysis of the lower limbs as a result of trauma to the spinal cord or neurodegenerative diseases that affect the peripheral nervous system. Additionally, there are millions of lower limb amputees and patients who suffer from neurological disorders that affect gait, such as Parkinson’s disease. A cortically driven BMI for decoding of bipedal walking was first developed by our laboratory (261) (FIGURE 12). In
In a second series of experiments, using a custom-designed internet connection, the Nicolelis group transmitted the output of their BMI to a humanoid robot built by Gordon Cheng and Mitsuo Kawato at The Advanced Telecommunications Research (ATR) Institute in Kyoto, Japan (133). The humanoid robot received continuous signals from the BMI through an optimized internet link that minimized the transmission delay. An image of the walking robot was projected to the screen mounted in front of the monkey. Initially, the robot was suspended over a treadmill. In later experiments, monkey cortical activity was employed to induce controlled bipedal walking of the same robot on the floor (414). After this study, decoding of kinematics of monkey quadrupedal walking from cortical activity was demonstrated by other groups, again with good precision (267, 268, 711). Additionally, leg EMGs during standing and squatting were extracted from monkey M1 activity (889). For the case of quadrupedal locomotion, Capogrosso et al. (112) recently reported a “brain-spine interface” that alleviated gait deficits in rhesus macaques with unilateral spinal cord injuries. They implanted rhesus monkeys with multielectrode arrays placed in the leg area of M1, contralateral to the subsequent SCI site. Electrical stimulation was applied epidurally to dorsal roots to produce extensions and flexions of the leg weakened by the SCI. Monkeys learned to volitionally control the paralyzed leg using the interface that converted cortical neuronal activity into the spinal stimulation patterns. The authors argued that, because they recorded in M1 representation of the affected leg, the extracted motor commands represented intentions to move that leg. A careful examination of their methodology, however, raises several pivotal concerns. First, they employed a decoder training procedure that relied on the presence of overt movements in the affected leg that exhibited “residual hip or knee oscillations.” Clearly, this could be related to the mechanical perturbations caused by the movements of the intact limbs, rather than voluntary attempts to execute
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
799
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
In their clinical trials, Schwartz and his colleagues further improved the accuracy BMI control exerted by a paralyzed human. They recorded from close to 200 neurons in an individual with tetraplegia implanted with multielectrode arrays in the motor cortex (152). As predicted in the early 2000s by our lab (114, 852), the increase in the number of simultaneously recorded neurons led to an improvement in the BMI performance. The subject gained control of an anthropomorphic robotic arm that performed skillful and coordinated reaching and grasping movements, like reaching to a knob and then turning it clockwise or counterclockwise. Like in Donoghue’s experiments, many of these experiments utilized assisted BMI control, where the subjects’ errors were corrected by the controller to facilitate learning. The subjects were eventually able to operate without that assistance.
these experiments, two macaque monkeys were trained to walk bipedally on a treadmill while holding a bar with their hands to assist balance. Next, the monkeys were implanted with multielectrode arrays placed in the regions of M1 and S1 representing the lower limbs. The neuronal ensemble recordings conducted with these implants showed that, while monkeys walked on the treadmill, cortical neuronal discharges were correlated with the stepping movements. Owing to these correlations, the Nicolelis lab researchers could extract multiple lower limb kinematic parameters from the cortical recordings. Multiple Wiener filters were used for that purpose, which extracted 3D position of the hip, knee, and ankle joints, as well as the EMGs of leg muscles. The decoding reconstructed movement patterns of both forward and backward walking.
LEBEDEV AND NICOLELIS steps with the paralyzed leg. Second, the neuronal modulations in the M1 ipsilateral to the lesion could represent the movements of the intact limbs that could move normally. Such representation could occur as the result of cortical plasticity following the SCI and maintained by cortico-cortical connections (393). Thus factors different from the monkey’s true intention to move the paralyzed leg could underlie the cortical modulations that triggered electrical stimulation of the dorsal roots and evoked the artificial steps. In addition to assisting disabled people to regain the ability to walk, BMIs can be employed as a rehabilitation method (205). This latter approach, which has employed mainly noninvasive BMIs, will be discussed below.
Currently, the wheelchair is the main assistive device that enables navigation to people suffering from paralysis. Our laboratory also pioneered an intracranial BMI for wheelchair control (656). In this study, two rhesus monkeys were trained to control a robotic wheelchair, while being seated on top of it, by the activity of their cortical neuronal ensembles. Monkeys were chronically implanted with microelectrode arrays in multiple areas of both hemispheres. Neuronal ensemble activity in these areas was recorded using our wireless recording system (711). Wiener filters were used as decoders. Each experimental session started with a decoder training session, where the robotic wheelchair was driven by the computer; monkeys remained passive observers of these movements. During this passive navigation, two Wiener filters were trained to extract wheelchair kinematics from cortical activity. Such decoding was possible because cortical neurons were tuned to the wheelchair movements. One Wiener filter extracted translational velocity of the wheelchair (movements forward and backwards), whereas the other extracted rotational velocity (leftward and rightward rotations). Following the training session, the mode of operation was switched to brain control, where the monkeys’ cortical activity was now mapped into the wheelchair’s translation and rotation velocities. The behavioral task consisted of driving the wheelchair toward a food dispenser that delivered grapes as a reward. As the monkeys trained, their ability to navigate the wheelchair with cortical signals improved. Additionally, performance on the wheelchair navigation task resulted in the emergence of a representation of the distance to reward location, a tuning property that resembled hippocampal place cells, in the primary motor and somatosensory cortical areas. This representation was totally unrelated to the settings of the decoder. While our BMI converted M1 and S1 activity directly into whole body navigation commands, without the need for
800
Overall, these studies have demonstrated that intracranial BMIs could drive a prosthetic device that enabled whole body mobility. Such a device could be used to restore mobility to severely paralyzed patients in the future.
E. BMIs That Utilize FES FES of peripheral nerves is a promising approach to restore motor functions to paralyzed subjects. FES-based BMIs aim to use the subject’s own brain activity to control the delivery of electrical stimulation to his/her own muscles that would then move their limbs. Over the past decade and a half, some progress has been reported with such BMIs. The initial evidence of the feasibility of BMIs that mimic muscle activity was provided by the demonstrations that EMGs of arm (558, 642, 694) and leg (261) muscles could be extracted from the activity of cortical neuronal populations. Additionally, studies in healthy human subjects showed that a multichannel FES could produce near-normal hand movement patterns (390, 713). The first demonstration of a BMI with FES output was done by Pfurtscheller and colleagues who aided a tetraplegic patient with a FES device attached to his forearm (630, 633). The FES was controlled by bursts of cortical beta activity (18 –25 Hz) recorded by EEG electrodes placed over the patient’s sensorimotor cortex while he tried to imagine moving his foot. After some practice, the subject learned to grasp objects using this device.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
D. BMI for Whole Body Navigation
any intermediary overt behaviors, an alternative approach to enable such navigation is to have monkeys steer a motorized wheelchair with a joystick. Recently, it has been shown that monkeys can perform such steering to navigate a complex maze (226). Moreover, one study has shown a transition from joystick control to BMI control of a wheelchair (494). In that study, neuronal ensemble activity was recorded using cortical arrays implanted in the arm representation of M1. Monkeys were initially trained to steer the wheelchair with a joystick. While they did so, a decoder was trained to classify the joystick steering commands based on M1 activity. Next, the mode of operation was switched to brain control, where the steering command was derived from cortical activity. Finally, the authors demonstrated that, like in our study, the decoder could be trained without the joystick movements. Curiously, activity patterns of some M1 neurons changed dramatically after the mode of operation was switched from joystick control to brain control. Using the joystick in the context of BMI control of whole body navigation somewhat resembles the previous implementations of BMIs for arm reaching (114, 377). However, an important difference is that subjects have to learn a spatial transformation from the arm to the joystick movements (757).
393
394
BRAIN-MACHINE INTERFACES Several demonstrations of BMI-controlled FES have been accomplished in monkey studies. Eberhard Fetz’ group temporarily paralyzed monkeys’ hands with an anesthetic blockade (556) and then employed the firing rate of neurons located in the primary motor cortex (M1) to control an FES device that evoked wrist torques. Visual feedback of the torque was provided by a screen cursor. Monkeys successfully learned to control this BMI. Moreover, M1 neurons, which were initially poorly associated with hand movements in a manual task, later on developed task-related modulations during the BMI control.
Bouton et al. (83) demonstrated a BMI with FES in a paralyzed human with an intracranial multielectrode implant. The study subject suffered from a C5/C6 complete, nonspastic quadriplegia, resulting from a diving accident. A Utah array was implanted in the hand area of M1, which allowed recordings from up to 50 single units simultaneously. The recordings continued for 350 days, and 33 units were isolated by the end of the study. During the training session, the subject attempted to produce six wrist and hand movements. These movements were impaired by the paralysis but could be evoked by FES. The FES was delivered using a 130-electrode array of surface electrodes embedded in a sleeve that was wrapped around the forearm. Neuronal population activity was converted into FES patterns using multiple simultaneous neural decoders based on a nonlinear kernel method with a non-smooth support vector machine. During the brain control mode of operation, the subject was required to generate hand movement that matched the cue shown on a computer screen. Following training, he managed to perform up to 70% of trials correctly. Notwithstanding the success of these demonstrations, using FES to restore movements meets a number of difficulties, such as muscle fatigue (224, 305, 796) and difficulties in achieving good accuracy of evoked movements without sensory feedback of force and position (25, 385, 818).
F. Neuronal Plasticity in Motor BMIs Subjects usually experience difficulties when they are first introduced to brain-control mode of BMI operation. Yet, over time, they improve their performance with continuous practice. Such improvements have similar mechanisms as
In general, BMI control of an artificial actuator has much in common with the neurophysiological mechanisms involved in learning to use and become proficient in tool handling, operations known to evoke brain plasticity (67, 195, 378, 524, 525). This likeness can be easily verified by reviewing the experiments conducted by Atsushi Iriki’s laboratory. In their fundamental experiments on primate tool usage, Iriki et al. (378) trained macaque monkeys to reach toward distant objects, which could not be accomplished by using their arms alone, by utilizing an external tool: a rake. Before monkeys could use the rake, the researchers measured the receptive fields of multimodal neurons in the posterior parietal cortex. Prior to the use of the artificial tool, these parietal cortical neurons exhibited both tactile and visual receptive fields (RFs) related to the animal’s hand: while the tactile RF was located on the hand skin, the visual RF was circumscribed to the visual space that closely surrounds the hand, the so-called peri-personal space. After the monkeys practiced and became proficient in the task of retrieving grapes with the rake, Iriki et al. observed that the visual RFs of the parietal neurons expanded to include the entire length of the rake, in addition to the peri-personal space around the animal’s hand. The Iriki laboratory interpreted these results as a suggestion that these cortical adaptations represented modifications of the animal’s body schema that resulted in the incorporation of the rake as an extension of the animal’s arm, as seen from the brain’s own point of view (586). Long-term operation of BMIs that control the movements of artificial actuators, robotic or even virtual arms, leads to similar brain remapping of the receptive fields of cortical neurons located in multiple motor and somatosensory areas, as described by Iriki in their experiments with tool usage. Several studies from our laboratory reported neuronal plasticity during learning to operate BMIs, starting with the study by Carmena et al. (114) that showed changes in neuronal tuning curves accompanied by changes in correlation between neurons as monkeys learned to operate a BMI that enacted reaching and grasping movements. Changes in neuronal tuning were further investigated by Lebedev et al. (463), and Zacksenhouse et al. (884) reported stronger cortical firing modulations during learning of BMI tasks, which decreased after monkeys learned. Transient increases in correlations between neurons, associated with learning a
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
801
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Lee Miller and colleagues also demonstrated brain control with an FES device (225, 641). Their FES system was controlled by populations of ⬃100 motor cortical neurons recorded with chronically implanted microelectrode arrays in the monkey M1. Hand paralysis in these animals was induced by an anesthetic block of the median and ulnar nerves at the elbow level. After the voluntary motor control of the hand was extinguished, the researchers activated forearm muscles with FES driven by the M1 signals. Monkeys were able to perform object grasping with this neural prosthesis.
learning of new motor skills (1, 69, 212, 213, 356, 439, 460, 547, 726). As such, many authors have proposed that neuronal plasticity is essential for BMIs to work properly in both animal and human subjects (114, 170, 195, 204, 324, 463, 466, 583, 612). As a matter of fact, some authors have gone as far as to implicate cortical plasticity triggered by BMI operations as the key mechanism through which subjects could assimilate prosthetic limbs or even other actuators, such as virtual limbs, as extensions of the subject’s body schema created by the brain (466, 586, 738).
LEBEDEV AND NICOLELIS
Evaluation of changes in neuronal tuning during BMI operations has several caveats. The main factor that should be considered in such an analysis is that, during brain control, neuronal tuning properties no longer depend on the brain circuitry alone, but also essentially depend on the decoder settings. Indeed, the decoder uses a transfer function or an algorithm to translate the activity pattern of each neuron into a contribution to actuator movements. In the case a Wiener filter is used for decoding, the contribution of a neuron to a given degree of freedom is defined by the weight assigned to that neuron. For example, if a neuron is set to have rightward tuning, the decoder translates the discharge rate of that neuron into increments of the x-coordinate of the actuator, while no contribution is made to the y-coordinate. This decoder-assigned tuning may be different from the true neurophysiological properties of the neuron. Say, the neuron has switched to representing the y-coordinate and now matches the user’s intention to change the actuator’s y-coordinate and/or responds to the visual of the y-coordinate. Despite this new representation, the neuron will still contribute to the x-coordinate only because of the original decoder settings. In another scenario, the neuron fires at random and does not represent any intention or feedback, but still has a directionally tuned contribution established by the decoder. The interpretation of neuronal tuning during BMI control is further complicated by the ensemble properties of the neurons contributing to the decoding. Consider a Wiener filter (for simplicity with just one tap) applied to a population of randomly firing neurons. An assessment of directional properties of each neuron in the population would show cosine tuning with a preferred direction defined by the Weiner filter weights assigned to the x and y dimensions. In this example, neuronal tuning during BMI control does not necessarily match any neuronal representation of the user motor intention and/or sensory feedback from the actuator; the tuning only corresponds to the decoder settings. The next level of complexity to the analysis of tuning properties
802
of a neuron during BMI control is brought by interferences from the other neurons. The BMI output is produced by many neurons, not just by the neuron whose tuning is being assessed. Therefore, when the firing of one neuron is compared with the BMI output, for example, cursor trajectory, this is effectively a comparison of activity of one neuron with a variable composed from the activity of many other neurons. Consequently, the tuning assessment critically depends on the relative contribution of different neurons. Two extreme cases can be considered: 1) the neuron in question has a very strong contribution to the decoder output whereas the contribution of the other neurons is relatively small, and 2) the neuron’s contribution is very small and the decoder output is dominated by the other neurons. In the first case, the neuron’s tuning will mostly represent the decoder settings for the reasons explained above. In the second case, the neuron’s tuning will reflect the relationship of that neuron’s firing to the actuator position generated by the other neurons, irrespective of the decoder weights assigned to that neuron. Furthermore, correlated firing between neurons may have strong effects on the BMI output and consequently on the tuning of individual neurons. For example, if activity of a weakly contributing neuron is correlated with the activity of strongly contributing neurons, the tuning properties of the former will be very similar to those of the latter. Overall, although characterizing neuronal patterns during BMI control using tuning curves is helpful to reveal some basic features (114, 317, 463, 598), interpretation of such tuning characteristics is not trivial. The pitfalls of neuronal tuning analysis for BMIs can be illustrated by the study of Ganguly and Carmena (283) that attempted to characterize the formation of new “cortical maps” as the result of learning to control a BMI. In that study, monkeys performed a two-dimensional center-out task using a BMI based on the recordings from small (⬃15 neurons) M1 ensembles. The small-ensemble activity was translated into cursor position using a Wiener filter with 10 taps. The study claimed that if the decoder is trained on day one and fixed afterwards, M1 neurons would plastically adapt to improve BMI performance and form a “cortical map.” The authors argued that the same M1 ensemble could simultaneously hold several “cortical maps” corresponding to different decoder. The “cortical map” was defined as a set of directional tuning curves, one per neuron. A close examination of this analysis reveals that neuronal tuning was determined differently from how it was set by the decoding algorithm. The 10-tap Wiener filter (100 ms bin width) effectively assigned 10 tuning curves for each neuron, one per tap. Yet, the authors chose to compute one tuning curve per neuron, which was derived either from a relatively short time window (200 ms) or a long one (2 s). The tuning curves were normalized to change from ⫺1 to 1, which made it impossible to compare tuning strength in different neurons. Factors like the relationship of these tuning curves to the fixed decoder settings, relative contribu-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
bimanual BMI task, were confirmed by our laboratory (377). Overall, these studies showed that, after the mode of operation is switched to brain control, neuronal activity patterns markedly changed, both at the level of individual neurons and their populations. Changes in neuronal tuning were observed even when monkeys continued to perform arm movements during brain control. In this case, neuronal tuning to movements of their own arms weakened, and the neurons started to represent the BMI-controlled actuator instead (463). Moreover, neuronal tuning to the actuator remained even when monkeys stopped moving their own arms (114, 377, 463). At the population level, switching to brain control was associated with increased synchrony between the neurons and, consequently, with many neurons having very similar preferred directions (114, 377, 583, 598).
395
396
BRAIN-MACHINE INTERFACES
In the above examples, the major difficulty in evaluating neuronal tuning during BMI control is related to a somewhat circular approach: the actuator position is first generated from neuronal activity using a mathematical algorithm, and then an attempt is made to determine the relationship between the neuronal patterns and actuator movements once again, and to extract the features in this relationship that are not explainable merely by the decoder settings. This difficulty can be avoided if neuronal tuning is assessed based on parameter that is not generated by the decoder and can be manipulated independently of the decoder settings. Such an analysis was conducted in our study of a BMI for bimanual movements (377). In that study, we evaluated neuronal tuning to target position, the parameter that unrelated to the decoder settings. Monkeys controlled 2D movements of two virtual hands using a BMI; a separate target was designated for each hand. Since the target positions were not included in the decoder variables, neuronal tuning to the targets could not be a consequence of the decoder settings. A k-nearest neighbor (k-NN) classifier
UKF decoder was used to extract the screen locations of targets from cortical ensemble activity on each behavioral trial, and the percentage of correct classifications was used as a measure of representation strength. The locations of the targets for both virtual hands were clearly represented by the cortical neuronal ensemble. These representations persisted during brain-control trials and passive observation trials. The passive observation trajectories did not change day to day. Therefore, we used them to assess long-term changes in the neuronal responses to the virtual hands. This analysis was valid because passive observation trials did not involve BMI control. The analysis was conducted offline by applying a UKF decoder to the neuronal recordings and using decoding accuracy as a measure of tuning strength. We found a clear improvement in decoding accuracy across the training days. Several studies evoked learning (and related plasticity) by altering BMI decoder settings and observing behavioral and neural adaptions to such manipulations. Thus Chase et al. (127) examined a BMI that generated 2D cursor position from monkey M1 activity using a linear decoder (127). Next, they applied a rotational transformation to the contribution to the BMI output from a subset of neurons. Although this manipulation initially resulted in curved cursor trajectories, their monkeys adapted to the new condition and straightened the trajectories. The analysis of neuronal responses showed that the entire neuronal population contributed to that adaptation, not only the neuronal subset with perturbed BMI outputs. Using a similar manipulation, Ganguly and Carmena (283) perturbed BMI output by randomly shuffling neuronal inputs to a fixed Wiener filter. Their monkeys successfully adapted to that perturbation. Sadtler et al. (684) devised a method that made adaptation to BMI control particularly difficult. They applied a factor analysis to extract correlated neuronal responses and represent them as an intrinsic manifold, a subspace in a multidimensional space of population firing rates. The authors found that monkeys successfully learned to control the BMI with the inputs taken from the manifold, but learned with great difficulty if the inputs came from the outside of the manifold. In other words, monkeys adapted to a new decoder if it did not require them to alter the original structure of neuronal correlations. Although this study seems to suggest an existence of strong synergies between the neurons in an ensemble, there is also an alternative explanation. The study utilized a threshold crossing method for detecting multiunit spikes, a method prone to inclusion of noise into the spike data. The noise most likely ended up outside the intrinsic manifold, so in the outside-of-manifold task monkeys were asked to control the BMI with noise, obviously a task impossible to learn. In addition to brain plasticity induced by learning to operate motor BMIs, plasticity occurs after training with sen-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
803
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
tion of different neurons to BMI output, and neuronal correlations were not considered. As training on the BMI task continued for 9 –19 days, the tuning curves changed during the initial training days and later stabilized, which was interpreted as the formation of a “cortical map.” These changes were paralleled by a clear evolution of cursor movements, which started as highly convoluted trajectories resembling random walk, but changed to almost straight center-out trajectories during the late training days. This meant that the early and late tuning curves were generated from very different cursor movements, which by itself can explain the differences that appear as changes in neuronal tuning. For example, the 2-s time window most definitely contained movements in very different directions during the early days, and represented a more uniform sample during the late days, an analysis that is guaranteed to generate different looking tuning curves. Therefore, the seemingly paradoxical result that neuronal tuning curves changed for a fixed decoder that presumably would have kept them very stable, most likely reflected changes in cursor movement patterns rather than any meaningful characteristics of neuronal representation of the external actuator. Given these considerations, that study’s conclusion regarding the emergence of a “stable cortical map” appears questionable. A more plausible conclusion is that both BMI output and the underlying neuronal patterns changed during learning. More data would be needed to evaluate if there was any change in the cortical representation of the actuator movements resulting from learning to control the BMI. Specifically, answers to the following questions would be needed: 1) how the neuronal tuning curves are affected by the decoder settings; 2) what other factors affect the tuning besides the decoder settings; and 3) how the tuning characteristics could be compared for datasets with very different actuator trajectories.
LEBEDEV AND NICOLELIS sory BMIs (see the sections below on sensory and bidirectional BMIs). Studies conducted in our laboratory (341, 798) enabled rats to perceive infrared light using a BMI that converted the signals from the head-mounted infrared sensors into ICMS of the rat primary somatosensory cortex (S1). Learning to use this BMI resulted in the emergence of a representation of infrared light in S1. Moreover, this new representation coexisted with S1 representation of the rat whiskers.
VIII. NONINVASIVE BMIs A. EEG-Based BMIs EEG-based systems are the most popular noninvasive BMIs, which have been thoroughly studied in humans, in both healthy subjects and patients. While approaches to EEG decoding are somewhat different from those used for extracting motor commands from streams of neuronal spikes, the general principles of neuronal ensemble physiology (583, 586) still apply to these applications. For example, decoding accuracy improves when more EEG channels are added (141). Although EEG signals are prone to be contaminated with many sources of noise, including facial EMG, electrooculogram (EOG), and all sorts of movement artifacts, and despite the fact that EEG recordings do not yield detailed motor information compared with intracranial single-unit recordings, EEG-based BMIs have been successfully implemented in both normal and disabled human subjects to enact motor commands and provide communication channels. In particular, these EEG-based BMIs have been extremely useful in allowing “locked in” patients, those suffering from a complete level of body paralysis, as a result of a neurodegenerative disorder such as amyotrophic lateral sclerosis (or Lou Gehrig’s disease), to regain the ability to communicate with the external world, using EEG based spelling devices (71–73, 129, 184, 358, 364, 455, 481, 573, 606, 635, 718, 740, 742). The BMI classification into independent (endogenous) or dependent (exogenous) systems (see sect. III) is particularly
804
distinct for EEG-based systems. In independent BMIs, subjects perform volitional mental tasks, for example, motor imagery, that evoke changes in their EEG rhythms (3, 5, 17, 27–29, 31, 51, 77, 408, 560, 611, 632, 654, 811, 813, 826). The EEG bands typically employed in such BMIs are slow cortical potentials, mu (8 –12 Hz), beta (18 –30 Hz) and gamma (30 –70 Hz) waves (14, 74, 815, 863). The majority of EEG-based BMIs translate EEG activity into discrete choices (17, 110, 159, 355, 370, 667, 669, 722, 840), but continuous control is also possible with independent EEGbased BMIs (211, 864, 893). Dependent EEG-based BMIs utilize computer screens or LED displays as sources of visual stimuli that evoke EEG responses. Users modulate these responses to produce a BMI output (10, 42, 70, 74, 101, 117, 521). Classification algorithms identify cortical responses to screen stimuli to which participants attend (overtly or covertly). For example, a decoder based on visual evoked potentials (VEPs) (162, 196, 216) exploits the fact that VEPs are stronger in the visual areas when subjects attend to the stimulus and/or look at it (147, 216, 555, 566). The P300 component of the response to a visual stimulus, also called P3 (206, 637), has been particularly popular in BMI designs (10, 15, 16, 34, 54, 209, 243, 246, 257, 285, 363, 406, 521, 538, 545, 590, 717, 735, 736, 787, 832) because of its high sensitivity to subjects’ reaction to the stimulus in “oddball” paradigms, where a person is required to detect a target within a train of irrelevant stimuli (12, 206 –208, 778). Auditory-based (58, 332, 410, 440, 455, 643) and tactile-based (95, 514, 814) P300 interfaces have been implemented, as well. Another popular BMI design is the one that utilizes VEPs generated by rapidly occurring stimuli (up to 60 Hz) (8, 42, 117, 122, 186, 187, 386, 395, 473, 481, 502, 515, 868, 891). Such VEPs are called steady-state visual evoked potentials (SSVEPs). In a typical SSVEP-based BMI, multiple flickering objects are shown on the screen; the flicker frequency is unique for each object. Subjects look at a specific object to issue a BMI command. Hybrid schemes for EEG-based BMIs have also been developed (561, 626). Such BMIs combine several decoding principles, for example, motor imagery combined with SSVEPs (13, 100) or steady-state somatosensory evoked potentials (7), functional near-infrared spectroscopy (fNIRS) with asynchronous sensorimotor rhythms, P300 with SSVEP (154), and eye position with EEG decoding (337, 387, 833). Such hybrid BMIs are more versatile and accurate compared with BMIs that use only one control mode. Overall, EEG-based BMIs have successfully achieved many significant milestones. These include spelling devices (10, 71, 101, 117, 132, 438, 559, 608, 619, 672, 876, 882), speech generators (190, 326, 549), BMIs for control of a humanoid robot (55, 103, 120, 138, 301), telepresence systems (130, 223, 804), BMI-controlled wheelchairs (121,
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Notwithstanding the progress in studies on BMI-induced neuronal plasticity, this research is still at the initial stage. Much more work will be needed to further clarify our understanding of this phenomenon. Yet, if one would have to choose one consensual view in this field, it is the assumption that without the occurrence of some level of cortical plasticity, BMIs would not be able to operate as successfully as they do. In other words, after a decade and a half of intense work, BMIs certainly owe their prominence in systems neuroscience to the exuberant propensity of the adult mammalian cortex to adapt itself to new task contingencies, particularly when exposed to rich feedback signals.
397
398
BRAIN-MACHINE INTERFACES magnetometer that is more sensitive than SQUID (871). MEG has better temporal and spatial resolution compared with EEG, but they can be used only in magnetically shielded facilities.
Notwithstanding these successes, it has been noted that many publications on EEG-based BCIs do not contain information on how EEG artifacts were handled (244). This is a serious problem because artifacts not only contaminate EEG recordings, but they could serve as a source of control signal for a BCI. Curiously, users can easily utilize their facial EMGs, produced by clenching the teeth and recorded with regular EEG electrodes placed on the scalp, to control a robotic arm. In fact, this EMG control outperforms EEG control in the same settings (160).
Salmelin and Hari (688) reported suppression of the murhythm, recorded with MEG, by thumb movements. Georgopoulos and his colleagues (296) employed a 248-sensor MEG to reconstruct arm movements from MEG recordings using a linear decoder. The first real-time MEG-based BMI was developed by Lal et al. (458); the system applied a binary classifier to MEG mu-rhythm. Mellinger et al. (541) used a similar design of a MEG-based BMI to classify mu and beta rhythms. MEG-based BMIs have been implemented in tetraplegic (413) and stroke (104) patients.
While EMG artifacts can be partially filtered out from EEG recordings by removing high-frequency signals (46, 311), mechanical artifacts strongly affect the EEG low-frequency range (331). Low-frequency mechanical artifacts can jeopardize the performance of BCIs that are based on slow cortical potentials. For example, José Contreras-Vidal and his colleagues employed linear regression decoders to reconstruct three-dimensional hand kinematics (86) and leg kinematics during treadmill walking (652) from slow cortical potentials. No artifact removal was performed, which opens a possibility that mechanical artifacts could influence the reconstruction. Strong objections to these results were expressed by Castermans et al. (116) who showed that EEGs recorded during treadmill walking were contaminated by the harmonics of the stepping frequency. In addition to questioning the reconstruction of movements from a low-delta EEG band, this study found that mechanical artifacts covered a wide range of EEG frequencies, so artifact removal by frequency filtering appeared to be unreliable. Yet another study (33) claimed that the employment of linear regression methods to reconstruct movements from cortical slow potentials was statistically invalid, since similar reconstructions could be obtain from both real and random EEG data.
C. fNIRS-Based BMIs
B. Magnetoencephalography-Based BMIs
Sitaram et al. (746) employed multiple optodes (four illuminators and four detectors for each hemisphere) placed over the motor cortex. The subjects performed finger tapping with the left hand or right hand, or imagined these movements. Support vector machine (SVM) and Hidden Markov Model (HMM) algorithms allowed this BMI to achieve greater than 80% accuracy in decoding which finger, right or left, was moving.
In addition to electrical fields emitted by the brain, noninvasively recorded brain magnetic fields have also provided signals for BMIs (104, 413, 458, 541). Magnetoencephalography (MEG) detects weak magnetic fields produced by the electrical currents generated mainly by cortical neurons (149, 150, 334). Brain magnetic fields are very weak, on the order of picoTesla, or 10 million times less than the Earth’s magnetic field. To detect these tiny signals, one needs to employ very sensitive magnetometers. Historically, the copper induction coil was the first magnetometer (150), later replaced by superconducting quantum interference devices (SQUIDs) (149). Recently, atomic magnetometry was introduced based on a spin-exchange-relaxation-free (SERF)
fNIRS applies light in the near-infrared range (600-1,000 nm) through the skull to detect changes in oxyhemoglobin and deoxyhemoglobin concentrations in the brain blood flow (252, 388). fNIRS-based BMIs have been gaining popularity recently (366, 533, 568, 744). fNIRS measures cortical metabolic activity with a spatial resolution of ⬃1 cm and temporal resolution on the order of 100 ms. However, the delay between neural activity and blood oxygenation changes is several seconds. The first fNIRS-based BMI was demonstrated by Coyle et al. in 2004 (166). That BMI extracted brain signals related to motor imagery when patients were asked to squeeze a ball. For this purpose, a single optode was placed over the motor cortex representation of the patient’s hand (EEG coordinate C3). Increases in oxyhemoglobin and decreases in deoxyhemoglobin were detected when subjects imagined the contralateral hand movements. The neurofeedback generated by the patient’s motor imagery was provided by a variable-diameter circle shown on the screen. The BMI had 75% accuracy in recognizing the patient’s imagined movements. Approximately 5 s were required to register the change in the hemodynamic response.
A number of studies employed fNIRs to recognize prefrontal cortex activity related to performing mental arithmetic (52, 53, 375, 649, 650). Additionally, fNIRs of prefrontal cortex activity was employed to decode subjective preference (517), music imagery (241, 649, 650), and emotional states (786).
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
805
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
169, 281, 411, 511, 663, 833), operation of a hand orthosis under BMI control (434, 436, 534, 610, 628), and BMIcontrolled leg exoskeletons (156, 205, 282, 425, 456, 768, 812).
LEBEDEV AND NICOLELIS
Interestingly, NIRS methods can detect not only slow responses related to hemodynamics but also fast responses (with a millisecond temporal resolution) related to the scattering of light by neuronal membranes (315, 762, 861). This signal is weak, however, and has not been used for singletrial detection of neural activity.
D. Functional MRI Functional MRI (fMRI) is another method for measuring brain hemodynamic responses (165, 276, 509, 772). fMRIbased BMIs derive their control signals from blood oxygen level dependent (BOLD) activity measured with an MRI scanner (744, 745, 847). fMRI has a relatively low temporal resolution (1–2 s), and the lag between neuronal activity and BOLD response is on the order of 3– 6 s. The main advantage of fMRI-based systems is their spatial resolution that allows monitoring the entire activity of the brain. fMRI-based BMIs typically use slices with 5 mm thickness; each slice is represented by a 64 ⫻ 64 image with 3- to 4-mm voxels. fMRI-based BMIs usually utilize visual feedback to display to subjects their own brain activity. For this purpose, various displays have been used, including functional brain maps (877), scrolling graphs of BOLD activity (139, 848), and virtual reality (745). The great advantage of fMRI is the ability to target a localized area as the source of a BMI signal. Weiskopf et al. (848) and Caria et al. (113) utilized imaging of the anterior cingulated cortex to develop a BMI for self-regulation of emotional processing. Additionally, BMIs have been developed for self-regulation of activity in the supplementary motor area (SMA) and Broca area (745). Several studies have employed fMRI-based BMIs to control computer cursors, avatars, and robotic arms. In the study of Yoo et al. (879), subjects were asked to perform several mental tasks (sequential number subtraction, covert speech,
806
and imagery of right or left hand clenching) to generate BOLD activity that drove a cursor through a two-dimensional maze. Yukiashu Kamitani and colleagues extracted fMRI representation of individual finger movements and drove a robotic hand with those signals, as reported in Scientific American (365). In Lee et al. (472), vertical and horizontal movements of a robotic arm were generated from fMRI signals. Cohen et al. (151) demonstrated control of a whole body human avatar in virtual reality by an fMRIbased BMI driven by motor imagery. Although control of prosthetic limbs from an fMRI scanner is not applicable to everyday use, such systems can be useful as rehabilitation tools. Additionally, fMRI-based BMIs can be potentially used as clinical devices for treating neurological conditions, such as stroke (185, 877), chronic pain (140), emotional disorders (113), and psychiatric disorders (430).
IX. BMIs WITH ARTIFICIAL SENSATIONS A. Restoration of Sensations Sensory BMIs enable the flow of information from the external world to be delivered back to the subject’s brain (57, 199, 466, 469, 583, 678). These systems strive to repair damage to sensory neural circuitry. In principle, sensory BMIs could interfere with different levels of neural sensory processing, from peripheral receptors to the spinal cord, brain stem nuclei, thalamus, cortical sensory areas, and cerebellum. For the development of efficient sensory BMIs, it is important to understand that sensory processing does not depend only on a unidirectional flow of information from the peripheral receptors or sensory organs to hierarchically higher processing stages. Top-down modulatory signals (e.g., describing influences to the lower-order areas from the higherorder brain regions) are vital for sensory processing in awake subjects (303, 306, 454, 464, 617, 739). Such modulatory signals are essential during the execution of voluntary movements (126, 572, 714 –716, 756, 760) and active sensory exploration of the environment (176, 314, 454, 615, 617). Sensory impairments can take many forms, from a complete loss of sensation caused by destruction of peripheral receptors and nerves to impairments of certain aspects of sensory processing that occur when cortical or subcortical areas are damaged. For example, following extensive lesions to the primary visual cortex, patients (163, 769, 849) and monkeys (164) do not perceive visual stimuli, but may retain an ability to utilize visual information. This phenomenon, called blindsight, is mediated by subcortical visual structures like the superior colliculus. Additionally, damage to cortical areas of the so-called ventral visual stream produces deficits of visual object recognition, whereas damage to the dorsal stream areas impairs spatial visual processing and
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
As mentioned above, fNIRS can be used in combination with other neural recording methods to create hybrid BMIs. For example, hybrid EEG-fNIRS BMIs have improved the speed of fNIR thanks to the EEG’s superior temporal resolution. In addition, this hybrid method has allowed the generation of a larger repertoire of commands because different types of brain-related signals are employed in the discrimination performed by this BMI. For example, Fazli et al. (248) recorded cortical sensorimotor rhythms simultaneously with EEG and fNIRs methods, which resulted in better classification of motor imagery. Khan et al. (424) positioned the fNIRS sensors over the prefrontal cortex, whereas the EEG electrodes were placed over the motor cortex. This BMI processed 1) brain activity generated by mental arithmetic, which was detected from prefrontal recordings, and 2) motor commands generated by hand tapping, which were extracted from motor cortical activity.
399
400
BRAIN-MACHINE INTERFACES visually guided movements (312, 313, 346). While such peculiar sensory disabilities could likely be treated with BMIs in the future, current implementations of sensory BMIs deal mostly with cases of damage to peripheral sensory receptors, sensory nerves, or spinal tracts. In these cases, there is a loss of normal sensation, but higher-order sensory areas remain intact and could still process sensory information if it is delivered to them using a BMI. Accordingly, sensory BMIs attempt to mitigate the devastating effects of peripheral lesions by linking these intact brain areas to artificial sensors.
Tactile sensations evoked by electrical stimulation of the surface of the postcentral cortex without eliciting movements were first described in 1909 by Harvey Cushing (178); they were later extensively studied by Wilder Penfield (623). Penfield’s patients most often reported sensations of numbness or tingling, rarely pain. The modern era in this research started with the experiments of Ranulfo Romo et al. (676) who employed small currents injected through a microelectrode, the method called ICMS, to evoke tactile sensations in monkeys. Romo’s monkeys started with learning a sensory discrimination task where they compared two vibrotactile stimuli applied to their hands one after another. The animal reported, by pressing a button with an opposite hand, which of the two vibrations had a higher frequency. Next, the first stimulus in the sequence remained vibrotactile, whereas the second one was an ICMS train applied to S1. The task was again to compare the frequencies at which the stimuli were presented. Surprisingly, monkeys began to successfully compare the vibrotactile and ICMS patterns with very little practice. This result suggested that sensations resembling skin vibration could be evoked artificially with ICMS of S1. Romo et al. (676) penetrated S1, with a microelectrode placed at a new location every day; they did not implant those microelectrodes. With this method, they could not study long-term changes in the ICMS-induced artificial sensations. A long-term study of ICMS effects with implanted microelectrodes was conducted by our laboratory (260) (FIGURE 17). The experiments were conducted in owl monkeys chronically implanted with cortical microelectrode arrays. The experimental task consisted of having animals reach and open one of two doors. Animals were searching for a piece of food that was hidden behind one of the doors. In each trial, the location of the food was cued by an ICMS
Talbot et al. (784) asked if sensations from different skin locations could be evoked using ICMS. They applied ICMS through different microelectrodes of the arrays implanted in the hand representation of S1. Monkeys reported with eye movements where on the hand they felt the stimulus. Predictably, these experiments confirmed the well-known S1 somatotopic organization (394). Additionally, it was determined that monkeys could discriminate ICMS intensity and match it to the pressure applied to the hand using a mechanical probe. Virtually the same experiment was recently conducted in a tetraplegic patient implanted with a Utah array in S1 (262). The patient correctly matched cortical stimulation sites to different hand locations. Stimulation applied through ECoG grids has been shown to evoke somatosensory sensations, as well (589, 638). This stimulation method evoked sensations of tingling, numbness, and temperature. Electrical stimulation of both the precentral and postcentral locations was effective in producing these sensory effects. Several studies explored peripheral nerve stimulation as a method to provide humans with artificial tactile sensations. One study (791) employed peripheral nerve cuff electrodes implanted in two patients with arm amputation for more than 1 yr. Patterned electrical stimulation of the nerves produced touch perceptions in the phantom hands that the patients described as being natural (tapping, pressure, moving touch, and vibration); the sensations changed with modifications in the stimulation pattern. These artificial sensations improved the subjects’ performance with a prosthetic hand. In the other study (183), phantom hand sensations were evoked in amputees using electrical stimulation of the median or ulnar nerve delivered through a 96-channel Utah
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
807
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
We will focus on sensory BMIs that enable artificial tactile sensations. In such BMIs, electrical stimulation is usually used to reactivate sensory responses (260, 598, 676, 784). Additionally, several papers have recently employed optogenetic methods to induce somatosensory sensations (536, 875). Stimulation can be applied to somatosensory cortex (368, 598, 676, 784), thalamus (182, 321, 479, 605), and peripheral sensory nerves (183, 662, 683, 791).
train. Progressively, more complex ICMS patterns were employed as the animals learned novel tasks. Monkeys were first required to simply detect the presence of ICMS. Next, they had to discriminate temporal patterns of ICMS, and finally they discriminated spatiotemporal ICMS patterns delivered through multiple electrodes. Although it took several weeks for monkeys to learn the initial, simple task, their ability to interpret new ICMS patterns clearly improved after several months of training with ICMS. After this initial learning phase, animals could acquire a new and more difficult task in just a few days. This result indicated that ICMS progressively generated a new sense, some sort of artificial touch sensation, that monkeys could readily utilize. In fact, it seems that some degree of generalization was achieved by these monkeys after a few months of training, which allowed them to learn new tasks that involved ICMS faster than when they were naive in terms of experiencing ICMS. We also conducted a study in rhesus monkeys where ICMS of the S1 instructed the animals about the direction of joystick movement they had to produce in a trial (597). Like the owl monkey experiment, rhesus monkeys learned that task after several days of training.
LEBEDEV AND NICOLELIS
A
B Microstimulation
Transparent barrier
D
F E
G
808
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Food Reward hidden behind one of two doors
C
401
402
BRAIN-MACHINE INTERFACES arrays, which remained implanted for 30 days. The same stimulation approach was utilized to reproduce sensations from a hand prosthesis that performed grasping tasks (662).
Using rats as an experimental model, Thompson et al. (798) showed that ICMS of S1 could be used to substitute or augment the animal’s natural vision. Their BMI allowed rats to use the S1 cortex to perceive, or “touch,” otherwise invisible infrared light. Light from infrared (IR) sources was detected by head-mounted sensors and converted into ICMS applied to the rats’ S1 representation of their facial whiskers. Initially, rats took 4 wk to learn to use a BMI with a single infrared detector to find reward ports that emitted infrared light. An upgrade to this system included four IR sensors that provided a panoramic infrared vision (341). Using this system, a new group of rats took only 3 days on average to find the same infrared sources. After rats learned to utilize this BMI, electrophysiological recordings revealed that S1 neurons developed multimodal receptive fields that represented both somatosensory responses from the facial whiskers and infrared light generated in the animal’s surroundings. These results showed that, even in adult animals, primary cortical areas can incorporate new sensory representations, leading to the emergence of multiple and overlapping sensory maps simultaneously sustained by the same neuronal populations. Sensory substitution through haptic stimulation of the subject’s body is an alternative to using the neurostimulation approach, which is particularly relevant to neurorehabilitation. A study by the Walk Again consortium (737) used haptic stimulation to restore the sensation of autonomous walking to paraplegic patients. For this purpose, a new
In addition to electrical stimulation, optogenetic stimulation has been steadily gaining popularity (875), so it is plausible that this method will be used in sensory BMIs in the future. Another stimulation method employs ultrasound (475, 674). Recently, implantable microcoils have been developed for magnetic stimulation (474).
B. Brain-Machine-Brain Interface Brain-machine-brain interfaces (BMBIs), also called bidirectional BMIs, perform both the extraction of motor command signals from raw brain activity and the delivery of sensory feedback to the brain (57, 255, 469, 583) or peripheral nerves (543). This approach was pioneered by our laboratory (597, 598) (FIGURE 13). In our experiments, rhesus monkeys were chronically implanted with microelectrode arrays in M1 and S1. M1 implants were used for the extraction of motor commands, and ICMS was delivered through S1 implants. The motor loop of this BMBI controlled movements of an avatar arm shown on a computer screen placed in front of the animals. Monkeys used this avatar arm to actively explore a set of virtual objects (2 or 3 gray circles) rendered in the virtual space they searched. The objects were visually identical but differed in terms of their artificial
FIGURE 17. Long-term experiments in owl monkeys on reaching movements cued by intracortical microstimulation of somatosensory cortex (S1). A: diagram of the experimental task. After a barrier was lifted, monkeys reached toward one of two doors; a food pellet was behind one of them. B: location of cortical implants. S1 implant was used to deliver microstimulation. C: microstimulation parameters. D–F: stimulation patterns. The first task (E) required reaching to the right if a sequence of microstimulation pulses was delivered. If no stimulation was applied, monkeys reached to the left. In the second task (G), the rule was reversed: monkeys reached leftward in response to microstimulation. The third task (D) employed two different temporal patterns of microstimulation. The fourth task (F) used spatiotemporal patterns of microstimulation produced using four pairs of implanted microwires. [Adapted from O’Doherty et al. (597).]
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
809
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Several somatosensory BMIs have been demonstrated in rats, taking advantage of the exquisite tactile skills of these animals. A study of John Chapin’s laboratory (790) reported a BMI that guided rat navigation through threedimensional structures. Steering cues were provided by ICMS of S1, whereas locomotion was reinforced by ICMS applied to the medial forebrain bundle, a structure known to be part of the brain’s reward system. A human operator used this BMI to steer rats over complex terrains. Venkatraman and Carmena (820) developed an active sensing paradigm based on ICMS of rat S1. ICMS was delivered when a whisker crossed a spatial location designated as a target. Rats were rewarded for localizing the invisible target and crossing it several times with the whisker.
paradigm was developed for reproducing lower limb somatosensory feedback in paraplegics by substituting sensations generated by a haptic display placed on patients’ forearms for the normal sensation generated by walking legs. Initially, leg movements were simulated by making an avatar of the patients move in an immersive virtual reality environment. Patients used goggles to observe their avatars moving on different ground surfaces while a haptic display was used to deliver a wave of tactile stimulation to the skin of their forearms. The use of this haptic display induced patients to experience the perception of walking on various surfaces: grass, a paved street, or beach sand. Moreover, patients perceived leg movements during the swing phase of the avatar legs and experienced the perception of their feet rolling on the floor, despite the fact that their legs were completely paralyzed. These results showed that virtual reality training combined with haptic stimulation resulted in the assimilation of the virtual lower limbs in the body representation present in the patients’ brains. These findings suggest that, in the future, the addition of rich haptic feedback to rehabilitation devices will be essential to restore realistic perceptual experience in paralyzed patients.
LEBEDEV AND NICOLELIS
Several recent studies have implemented bidirectional interfaces with peripheral nerves. Davis et al. (183) demonstrated real-time control of a robotic finger by amputees using multielectrode recordings from the median or ulnar nerves. The decoding was performed by a Kalman filter. The same electrodes were used to deliver sensory feedback using electrical stimulation. The other study (662) reported a myoelectric interface that amputees could control using surface EMGs to produce grasping movements using a robotic hand. Grasp force feedback, produced by robotic sensors, was delivered using intrafascicular stimulation of the median and ulnar nerves; stimulation intensity was proportional to the sensor signal. This bidirectional setup enabled the subjects to maintain three force levels without looking at the robotic hand. In addition to BMBIs that provide sensory feedback from an external actuator, several recent demonstrations of closedloop activity-dependent stimulation can be described as BMBIs. In these systems, neuronal activity is recorded from a brain area and then converted into a pattern of electrical stimulation delivered to the same or a different area. Such feedback loops may serve different purposes. Andrew Jackson and his colleagues at Eberhard Fetz’s laboratory employed a neural implant to form and strengthen an artificial connection between two sites in the motor cortex of freely behaving monkeys (381, 382). The implant triggered electrical stimulation in one cortical location with neuronal discharges recorded from a different site. Several days of operation of this implant produced a stable cortical reorganization that was evident from the changes in wrist movements evoked by electrical stimulation applied to each site. Wrist movements evoked from the implant’s recording site started to resemble those evoked from the stimulation site, which indicated that a Hebbian potentiation of synaptic connections occurred for the artificial connection. The au-
810
thors suggested that this approach could be used for neurorehabilitation in the future. Lucas and Fetz (513) employed EMG-triggered cortical stimulation to induce a similar targeted reorganization of cortical motor output. They observed that the stimulated cortical site became associated with the activity of the recorded muscle, even though that particular muscle was not represented by neurons in that cortical location previously (513). Yet another study by the Fetz laboratory (592) demonstrated that spinal stimulation, triggered from cortical spikes, could modify the strength of corticospinal connections in a manner consistent with spike-timing-dependent plasticity. Several studies have shown that closed-loop stimulation systems can lead to partial recovery of function in neurological conditions resulting from injury or disease. Guggenmos et al. (330) employed a functional bridge connecting motor and somatosensory areas of the rodent brain to promote recovery of motor skills after traumatic brain injury. McPherson et al. (540) used EMG-triggered spinal stimulation to facilitate recovery after spinal cord injury in rats. Overall, these studies showed that BMBIs could be used to plastically modify neural connectivity and promote functional recovery. Our laboratory developed a closed-loop stimulation system for epilepsy control (618). In that study, rats were treated with pentylenetetrazole to provoke epileptic seizures. The system detected the seizure episodes in cortical LFPs, and applied electrical stimulation to rat spinal cord to suppress the seizures. This approach reduced the frequency of seizure episodes by 44%. In the future, a similar approach may prove useful for the treatment for drugresistant epilepsy. A stimulation system has been suggested as a potential prosthetic system for improving memory (62, 63). In these studies, a multiple-input, multiple-output model reproduced the associations between CA3 and CA1 regions of the rat hippocampus. Neuronal ensemble recordings were conducted in CA3 and CA1 of rats performing a delayed-nonmatchto-sample memory task. A nonlinear MIMO was trained to predict CA1 activity based on CA3 patterns. The predicted patterns of activity were then delivered to CA1, using electrical stimulation through the same electrodes that recorded neuronal spikes. The stimulation improved task performance in normal rats and restored performance in rats with a pharmacological block of hippocampal synaptic transmission. The authors suggested that this approach could be used to restore long-term memory function in patients with damage to hippocampus and its interconnected structures.
X. COGNITIVE BMIs Cognitive BMIs or cognitive neural prostheses deal with brain activity related to higher-order functions, as opposed
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
texture. Monkeys had to assess the objects’ texture by using the BMBI to scan their avatar hands over the surface of the virtual objects. When a monkey’s avatar hand came into virtual contact with the surface of a given object, a pattern of ICMS was delivered to the hand representation of the animal’s S1. Monkeys had to identify a specific virtual texture using this BMBI and then hold the avatar hand over it to obtain a fruit juice reward. The implementation of this BMBI had a caveat: since ICMS evoked electrical artifacts that occluded the neuronal spikes, recordings and stimulation could not be conducted simultaneously. This issue was solved by switching from recording to stimulating every 50 ms. Although this approach resulted in a loss of some neuronal data, the BMBI still performed well because several hundred neurons were recorded simultaneously. A similar BMBI was reported by Richard Andersen’s group (437). In that system, ICMS was applied to S1, whereas BMI control commands were extracted from PPC.
403
404
BRAIN-MACHINE INTERFACES to more simple motor and sensory functions (625, 792). Although the distinction between higher-order and lowerorder functions is not clear cut, by convention BMIs are called cognitive if they work in the domains of cognitive states (348, 886), executive functions (349), decision making (21, 335, 343), memory (62, 63), attention (35, 417), and language (98, 99, 327, 482, 834, 839).
In our laboratory, a BMI approach was employed to extract decisions involved in reprogramming a motor goal (376). Monkeys performed a center-out task, where they moved a cursor towards screen targets using a joystick. Neuronal ensemble activity was recorded from M1 and S1 arm representations. Monkeys started the trials by placing the cursor at the screen center. Next, a target appeared at an offcenter location. In some trials, that initial target appeared for 50 –250 ms, and then it was replaced by a new target at a different screen location. We found that both the emergence of the decision to move to the initial target and the new decision to cancel that motor plan and move to a new target could be decoded from population M1 activity. Target locations were decoded using an LDA classifier. This analysis showed that M1 activity initially represented the first target, then simultaneously represented both targets, and eventually shifted to represent the new target only. Based on these findings, we proposed that such decoding of covert motor planning could improve motor BMIs by equipping them with the capacity to detect motor decisions early and inhibiting them if the user decides to cancel a prepared action or chooses a different one. In the other study, we decoded representation of time from M1 and PMd activity in the absence of over behavior (468). Monkeys were trained to perform self-timed button presses, where they touched a button with their hands, maintained contact for 3– 4 s, and then released the button. We found that, while monkeys self-timed the required interval and did not produce any movements, their M1 neurons exhibited ramping activity patterns. We then employed a Wiener filter to
Motor imagery, widely used in noninvasive BMI (5, 28, 272, 551, 773, 840), can be considered a cognitive component of a BMI. Richard Anderson’s group recently decoded motor-imagery from the intracranial PPC recordings in a tetraplegic human (3). PPC is engaged in higher-order aspects of motor behaviors (18). In the tetraplegic subject, motor imagery clearly activated different PPC neurons, depending on the specific action being imagined (e.g., imagining hand movement to the mouth or ear, imagining shoulder rotation, etc.). Additionally, PPC neurons responded to the imagery of movement goal, movement trajectory, and the type of movement. All these variables were successfully decoded from the activity of PPC neuronal populations. Moreover, the subject learned to control a robotic arm by imagining movements. As intracranial recording methods become more routine in clinical studies, we will probably see a rapid development in BMIs related to human cognitive processes. Advances in this new domain will likely contribute to the emergence of new clinical applications for BMIs, as well as the incorporation of fundamental knowledge about the neurophysiological involved in higher brain functions.
XI. BRAIN-TO-BRAIN INTERFACES AND BRAINETS The growth of BMI research gave rise to a large variety of spin off experimental paradigms. In one of the variations of the classical BMI approach (FIGURE 18), multiple animal (or human) brains can be connected to each other to establish a direct brain-to-brain communication linkage, called brain-to-brain interface (BTBI) (616). In the other variation, several individual brains collaborate on a common motor task by establishing a network of brains, or a Brainet (614, 657) (FIGURE 19). By definition, BTBIs allow multiple animals to exchange information using a protocol that incorporates both neural recording and stimulation. The pioneering BTBI was implemented in rats (616). In that study, the first animal performed the role of information encoder, and the second animal was the decoder of a simple binary message. The binary message represented the encoder rat performing a two-choice behavioral task (active tactile discrimination or responses to a visual stimulus). The encoder rat’s neuronal firing rates, recorded from either the S1 or M1, depending on whether the rat performed a tactile discrimination or a visuomotor task, underwent a sigmoid transform and then were converted into patterns of ICMS
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
811
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Several intracranial cognitive BMIs have been developed, mostly dealing with decoding of different aspects of motor decisions during the periods of immobility preceding movement onsets. For example, Hasegawa et al. (343) decoded go versus no-go decisions, and the prepared saccade direction from the activity of monkey superior colliculus neurons (343). Musallam et al. (564) decoded the representation of expected rewards and motor decision from the neural activity recorded in the cortical parietal reach region. In that study monkeys were engaged in an instructed-delay task where they prepared an arm movement, but withheld it for several seconds. In the beginning of each behavioral trial, the monkey was shown a cue that indicated what kind of reward would be given. A Bayesian algorithm was applied to decode expected reward and target location simultaneously.
decode the representation of temporal intervals from these M1 patterns. Such decoding of action timing could be useful for developing BMIs that enact typical motor behaviors where movements are intermingled with periods of immobility.
405
LEBEDEV AND NICOLELIS Sigmoid Transform
ICMS
ƒ(x)
Number Pulses
M1 neural ensemble
ZScore Correct Lever
Correct Lever
Feedback
(2nd reward)
Decoder
FIGURE 18. Brain to brain interface. Two rats participated in the experiment, the encoder rat and decoder rat. The flow of information between the animals is shown by arrows. The encoder rat responded to a visual stimulus provided by an LED by pressing one of two levers. Activity of an M1 neuronal population activity was recorded while the encoder performed the task. This activity underwent a sigmoid transform to generate a microstimulation pattern, which was delivered to the somatosensory cortex of the decoder rat. The decoder animal had to select the same lever. The encoder rat received an additional reward if the decoder rat performed correctly. [From Pais-Vieira et al. (616).]
applied to S1 or M1 of the decoder rat. This latter animal could be located next to or far apart from the encoder. On average, the decoder reproduced the behavioral choices of the encoder rat in about 70% of the trials. During operation of the BTBI, the encoder rat received an additional reinforcement. Pais-Vieira and colleagues noticed that following an error by the decoder rat, the encoder rat adapted both its behavior and cortical activity to generate cleaner neuronal signals to be broadcast to its partner. Invariably, the decoder rat performed better after this encoder’s adaption. In the next study by Pais-Vieira et al. (614), several rat brains were connected to a network of brains - named a Brainet - that performed several elementary computations, like discrimination of ICMS patterns by several rats simultaneously to improve overall discrimination accuracy, or retaining information in their collective memory by transferring it from rat to rat (614). ICMS served as an input to such a Brainet, while the output was derived from cortical activity of the participating animals. Essentially, the Brainet acted as an organic computer that processed input data through a network of living brains. These initial publications were followed by a number of studies by different groups unified by a common theme of connecting of the brains of different organisms. Yoo et al. (878) connected the brain of a human to the spinal cord of a rat. The human operated an SSVEP-based BMI to generate “go” commands to an anesthetized rat. The command was executed by applying transcranial focused ultrasound to the rat motor cortex, causing the movement of the ani-
812
mal’s tail (878). In another study BTBI connected two different species (488). The human attempted to make the cockroach walk along an S-shape track, and succeeded in 20% of cases. In the other study (727), the premotor cortex of one monkey was linked to the spinal cord of a second, anesthetized primate. The second monkey’s hand was attached to a joystick. The first monkey generated motor intention commands while looking at a computer screen that showed a cursor and a target of movement. This intention command was extracted from premotor cortex activity and translated into a stimulation pattern applied to the spinal cord of the second monkey, causing the joystick movement, which in turn moved the cursor on the first monkey’s screen. A proof of concept study (265) showed that gene expression can be controlled by brain signals. In that study, a human operating an EEG-based BMI optogenetically controlled the expression of designer cells. The designer cells were either in culture or in subcutaneous implants in mice. Several BTBIs have been demonstrated in humans. Grau et al. (316) had one human subject, the emitter, operate a motor-imagery EEG-based BMI. The binary output of that BMI was delivered to the brain of the second subject, the receiver, using TMS pulses applied to the visual cortex. Depending on whether the transmitted signal was “1” or “0,” a robot placed the TMS coil over the area where stimulation induced or did not induce a conscious perception of phosphenes. Information transfer rates of 3 and 2 bits per minute were achieved for the BMI performance and message transmission, respectively. Rao et al. (661) employed a very similar BTBI design, with the dif-
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Encoder
406
BRAIN-MACHINE INTERFACES
A
B
X
SHARED CONTROL TASK
Y 50%
Z Visual feedback
decoder
Visual feedback Movements decoded from brain 1
decoder
Controls XY
50%
Movements decoded from brain 2
decoder
Controls YZ Controls XZ
Monkey M 410-501 neurons
D
PARTITIONED CONTROL TASK
X position
PARTITIONED CONTROL TASK Monkey K 140-156 neurons
Y position
Visual feedback
Visual feedback
X position decoded from brain 1
Y position decoded from brain 2
share X control
Z Y
Z X
Monkey M 410-501 neurons
share Z control
X
Monkey C 196-214 neurons
Monkey M 410-501 neurons
Y share Y control
Monkey C 196-214 neurons
FIGURE 19. Monkey brainet. A: diagram of the experimental setup. Up to three monkeys were seated in monkey chairs in separate rooms. Each monkey faced a computer screen that displayed a virtual avatar arm. The behavioral task consisted of reaching screen targets with the avatar arm. The avatar arm was controlled jointly by several monkeys. B: shared control task, where each of two participating monkeys contributed 50% to the (X,Y) position of the virtual arm. Locations of microelectrode arrays are shown below the task diagram. C: partitioned control task, where one monkey contributed to X position of the avatar arm, whereas the other monkey contributed to Y position. D: a 3-monkey task. Each monkey performed a two-dimensional task, and all three together controlled 3-dimensional movements of the avatar arm. [From Ramakrishnan et al. (657).]
ference that TMS was applied to the motor cortex of the second subject. Accordingly, the second subject responded with a TMS-induced hand movement that produced a touchpad press. While the studies reviewed above emphasized direct communication between different brains, our laboratory recently demonstrated several Brainets that emphasized cooperation of multiple subjects to achieve a common motor goal of a typical upper limb BMI (657). In that study, two or three monkeys shared control of the movements of an avatar arm using their combined cortical activity. Three Brainet designs were tested. The first design, called shared-control Brainet, merged the outputs of two monkey brains. Cortical activity of each monkey was processed by a separate decoder. The decoder outputs were then averaged to set the coordinates of the avatar arm. Performance improvement was achieved because the averaging of contributions from
both monkeys enhanced the signal and suppressed the noise. In the second design, called partitioned control Brainet, two monkeys performed together again, but they had different tasks. The first monkey generated neural control commands to move the avatar arm in the horizontal dimension, while the other monkey controlled the vertical dimension. In that Brainet, performance improved because each monkey made fewer errors in the simple, one-dimensional task. In the third Brainet design, named a triad Brainet, three monkeys cooperatively controlled three-dimensional movements of an avatar arm. Yet, each monkey performed a two-dimensional task, and all animals were unaware that the cooperative task was three-dimensional in nature. That design modeled a “super-brain” that, by combining the brain activity of three individual brains into a single computing system, handles a higher-order task while individual brains have lower-order contributions.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
813
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
C
Monkey O 156-229 neurons
407
LEBEDEV AND NICOLELIS Somewhat similar cooperative systems have been developed using EEG-based controls by several humans. These include a BMI for spacecraft navigation controlled by two users (644), BMIs for group decision making (217, 645, 883), and a cooperative BMI for movement planning (841).
XII. BMI AS A POTENTIAL NEUROREHABILITATION THERAPY
To date, noninvasive BMIs have been used as neurorehabilitation tools primarily in clinical studies focused on stroke victims. The main assumption motivating these studies has been that practice with a BMI that mimics movements of a paralyzed limb could facilitate brain plasticity
A
Mean improvement after 10 months of training
B
Much less research has been conducted on the effectiveness of BMI training in patients with SCI. In the first long-term study of this kind, Donati et al. (205) conducted BMI training of eight chronic paraplegic patients in a multi-stage rehabilitation paradigm, aimed at restoring bipedal locomotion through robotic lower limb orthoses. The core of this paradigm was based on the utilization of an EEG-based BMI that allowed patients to control multiple actuators, ranging from avatar bodies to two types of robotic walkers: a commercially available gait robotic system (Lokomat) (384) and a custom-designed lower limb exoskeleton. In addition to the traditional visual feedback, this BMI was also coupled with a haptic display system that delivered continuous streams of tactile information to the skin of the patients’ forearm. These artificial tactile/proprioceptive sig-
Example of expansion of the zone of partial preservation (ZPP) for sensory assessment in two ASIA A patients Right Side
Left Side
Number of dermatomes
7 6
Normal sensation
5
Altered sensation
ZPP Patient P1
4
Patient P6
3 2 1
gr
2 0.
gr 0
2.
gr 0
4.
r
gr
0g
10
Pa
30
in
0
Start of training
After 10 months of training
Start of training
After 10 months of training
FIGURE 20. Sensory improvement after neurorehabilitation training. A: average sensory improvement (mean ⫾ SE over all patients) after 10 mo training. B: example of improvement on Zone of Partial Preservation for sensory evaluation for two patients. [From Donati et al. (205).]
814
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Since the late 1990s, when BMI research began in earnest, the field has focused primarily on achieving two major goals: 1) to establish a new paradigm to investigate the dynamic physiological properties of distributed neural circuits in behaving animals, and 2) to explore the possibility of creating new assisted technologies, aimed at restoring upper, lower, or full body mobility in severely paralyzed patients. For the past decade, the focus on developing clinical applications based on BMIs has increased markedly, as noted throughout this review. Yet, no one had anticipated that this paradigm could provide benefits beyond the commonly stated goal of assisting patients in regaining mobility through the employment of a new generation of brain-controlled prosthetic or orthotic devices. Thanks to recent clinical studies, however, a third potential future application of this paradigm has been introduced: the use of BMIs as a neurorehabilitation tool (26, 27, 82, 204, 205, 737, 741, 751, 752, 819).
and contribute to some level of motor recovery. For example, stroke patients can learn to operate an MEG-based BMI by modulating their rhythm recorded in the hemisphere ipsilateral to the lesion (104). In this study, the BMI opened and closed an orthosis that was attached to the paralyzed hand. This learning did not cause noticeable clinical improvements. However, long-term BMI training combined with physical therapy resulted in clear motor recovery (93, 659). As shown by the analysis of motor evoked potentials (MEPs), the recovery was related to enhanced neuronal activity in the hemisphere ipsilateral to the stroke site (87). Similar results were demonstrated by a study that combined a BMI-controlled robot with robot-assisted physical therapy (27, 30). Combining BMI training with virtual reality resulted in clinical improvements as well (64). Additionally, a combination of BMI control with transcranial direct current stimulation (tDCS) showed positive clinical results (753).
408
BRAIN-MACHINE INTERFACES nals were generated either when the avatar body walked on a virtual surface, or when the patients walked with the help of the robotic devices. In the latter case, pressure sensors applied to the plantar surface of the robotic feet were responsible for generating signals depicting the feet’s contact with the ground during bipedal walking. Closing the control loop with this haptic display led patients to experience vivid lower limb phantom sensations, which included the illusion of experiencing leg movements even when they were operating the avatar body while remaining immobile
themselves. Moreover, using the information delivered by the haptic display applied to the skin surface of their forearms, six out of eight patients could discriminate between three different types of surface in which the avatar body walked (e.g., sand, grass, and asphalt). Despite being completely paraplegic, immobile from the level of the spinal cord lesion down, lesions ranging from (T4-T11), since the day of their spinal cord lesions (3–13 years earlier), and lacking any somatic sensation below the
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
FIGURE 21. Lower limb motor recovery. A: details of the EMG recording procedure in SCI patients. A1: raw EMG for the right gluteus maximus muscle for patient P1 is shown at the top of the topmost graph. The lower part of this graph depicts the envelope of the raw EMG, after the signal was rectified and low pass filtered at 3 Hz. Gray shaded areas represent periods where the patient was instructed to move the right leg, while the blue shaded areas indicate periods of left leg movement. Red areas indicate periods where patients were instructed to relax both legs. A2: all trials over one session were averaged (mean ⫾ SD envelopes are shown) and plotted as a function of instruction type (gray envelope ⫽ contract right leg; blue ⫽ contract left leg; red ⫽ relax both legs). A3: below the averaged EMG record, light green bars indicate instances in which the voluntary muscle contraction (right leg) was significantly different (t-test, P ⬍ 0.01) than the baseline (periods where she/he was instructed to relax both legs). Dark green bars depict periods in which there was a significant difference (P ⬍ 0.01) between muscle contraction in the right versus the left leg. B: EMG envelops and t-tests for all recording sessions, involving four muscles, for all eight patients: left and right gluteus maximus (GMx) and rectus femoris proximal (RFP) muscles. Color convention and figure organization follows the one of A. Data were collected after 7 mo of training for all patients and for all but patients P2 and P8 after 12 mo. [From Donati et al. (205).]
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
815
LEBEDEV AND NICOLELIS level of the lesion, after a 12 mo period of training with this BMI paradigm, all patients exhibited a very significant partial neurological recovery, which was characterized by the following: 1) an average expansion of 5 dermatomes, in pinprick, nociceptive sensation, in the zone of partial preservation,1 (below the level of the lesion) (FIGURE 20, A AND B); 2) an average 1–2 dermatome expansion in fine touch (FIGURE 20A); 3) significant improvement in proprioception and vibration perception below the level of the lesion; 4) recuperation of voluntary control of multiple muscles below the level of the SCI lesion, as measured by EMG
1
recordings and direct force measurements. In some cases, patients regained the ability to produce multi-joint leg movements (FIGURE 21, A AND B); 5) marked improvement in the walking index; 6) an improvement in thoracic lumbar control (FIGURE 22B); and 7) restoration of peristaltic and bowel movements, bladder control, and improvement cardiovascular function (FIGURE 22C). Because of this substantial neurological recovery, 50% of the eight patients were upgraded from a complete paraplegia (ASIA A n ⫽ 7, ASIA B n ⫽ 1) to a partial paraplegia classification (ASIA C) at the end of 12 mo of training with this BMI-based protocol (FIGURE 22A). Longitudinal analysis of EEG recordings obtained from these patients during the 12-mo training period reveled that this partial sensory, motor, and visceral recovery was paralleled by an expan-
FIGURE 22. Clinical and functional improvements. A: patients with ASIA classification improvements: four patients changed ASIA classification over the course of the neuro-rehabilitation training, three moved from ASIA A to C and one moved from ASIA B to C. ASIA A is characterized by absence of both motor and sensory functions in the lowest sacral area; ASIA B by the presence of sensory functions below the neurological level of injury, including sacral segments S4 –S5 and no motor function is preserved more than three levels below the motor level on either side of the body; ASIA C by the presence of voluntary anal sphincter contraction, or sacral sensory sparing with sparing of motor function more than three levels below the motor level, majority of key muscles have muscle grade less than 319. B: thoracic-lumbar control scale evaluates quantitatively motor skill of the thoracolumbar region. Score ranges between 0 and 65. It has 10 items that consider supine, prone, sitting, and standing postures. In the present study, the last item (orthostatic position) was scored 0 due to the limitations of the pathology. C: correlation between average time spent in a standing position in orthostatic or gait training (mean ⫾ SE, values are average hours per month) and mean frequency for bowel function (values calculated per month and z-scored per patient). [From Donati et al. (205).]
816
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
The area of the body, measured in dermatomes and myotomes, below the level of the spinal cord lesion, in patients classified as having a clinically complete lesion, that remained partially innervated.
409
410
BRAIN-MACHINE INTERFACES sion of the representation of lower limbs in their primary sensorimotor cortex. Based on these results, Donati et al. (205) proposed that a combination of cortical and spinal cord plasticity, triggered by chronic use of a BMI that provided rich visuo-tactile feedback, may have rekindled remaining axons that survived the original spinal cord injury. Overall, the study by Donati et al. (205) raises the concrete possibility that the future goals of BMI research may include the possibility of creating therapeutic procedures aimed at inducing some degree of neurological recovery in patients suffering from incomplete SCIs. As such, BMIs may become a true neurorehabilitation paradigm for these patients, instead of a mere assistive technology.
6. Ahn M, Lee M, Choi J, Jun SC. A review of brain-computer interface games and an opinion survey from researchers, developers and users. Sensors 14: 14601–14633, 2014. 7. Ahn S, Ahn M, Cho H, Chan Jun S. Achieving a hybrid brain-computer interface with tactile selective attention and motor imagery. J Neural Eng 11: 066004, 2014. 8. Akce A, Norton JJ, Bretl T. An SSVEP-based brain-computer interface for text spelling with adaptive queries that maximize information gain rates. IEEE Trans Neural Syst Rehabil Eng 23: 857– 866, 2015. 9. Akhtar A, Norton JJ, Kasraie M, Bretl T. Playing checkers with your mind: an interactive multiplayer hardware game platform for brain-computer interfaces. Conf Proc IEEE Eng Med Biol Soc 2014: 1650 –1653, 2014. 10. Akram F, Han HS, Kim TS. A P300-based brain computer interface system for words typing. Comput Biol Med 45: 118 –125, 2014. 11. Alexander GE, DeLong MR, Strick PL. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci 9: 357–381, 1986.
After a decade and a half of intense development, BMI research is currently witnessing a very rapid growth towards a broad range of potential clinical applications. This trend was originally driven by the expectation that BMIs may provide fundamental assistive tools for people who suffer from motor and/or sensory deficits. Recently, this expectation has been upgraded to reflect the possibility that BMIs may also become a new neurorehabilitation therapy that takes advantage of the phenomenon of brain plasticity to induce partial neurological recovery in severely disabled patients.
ACKNOWLEDGMENTS
13. Allison BZ, Brunner C, Kaiser V, Muller-Putz GR, Neuper C, Pfurtscheller G. Toward a hybrid brain-computer interface based on imagined movement and visual attention. J Neural Eng 7: 26007, 2010. 14. Allison BZ, Wolpaw EW, Wolpaw JR. Brain-computer interface systems: progress and prospects. Expert Rev Med Devices 4: 463– 474, 2007. 15. Aloise F, Arico P, Schettini F, Riccio A, Salinari S, Mattia D, Babiloni F, Cincotti F. A covert attention P300-based brain-computer interface: Geospell. Ergonomics 55: 538 –551, 2012. 16. Aloise F, Schettini F, Arico P, Leotta F, Salinari S, Mattia D, Babiloni F, Cincotti F. P300-based brain-computer interface for environmental control: an asynchronous approach. J Neural Eng 8: 025025, 2011. 17. An B, Ning Y, Jiang Z, Feng H, Zhou H. Classifying ECoG/EEG-based motor imagery tasks. Conf Proc IEEE Eng Med Biol Soc 1: 6339 – 6342, 2006. 18. Andersen RA, Buneo CA. Intentional maps in posterior parietal cortex. Annu Rev Neurosci 25: 189 –220, 2002.
Address for reprint requests and other correspondence: Miguel A. Nicolelis, Box 103905, Duke University, Durham, NC 27710 (e-mail: nicoleli@neuro.duke.edu).
19. Andersen RA, Burdick JW, Musallam S, Pesaran B, Cham JG. Cognitive neural prosthetics. Trends Cogn Sci 8: 486 – 493, 2004. 20. Andersen RA, Cui H. Intention, action planning, and decision making in parietal-frontal circuits. Neuron 63: 568 –583, 2009.
DISCLOSURES
21. Andersen RA, Hwang EJ, Mulliken GH. Cognitive neural prosthetics. Annu Rev Psychol 61: 169 –190, 2010.
No conflicts of interest, financial or otherwise, are declared by the authors.
22. Anderson CW, Devulapalli SV, Stolz EA. EEG signal classification with different signal representations. In: Neural Networks for Signal Processing V. Proceedings of the 1995 IEEE Workshop. New York: IEEE, 1995, p. 475– 483. 23. Anderson DJ, Najafi K, Tanghe SJ, Evans DA, Levy KL, Hetke JF, Xue X, Zappia JJ, Wise KD. Batch fabricated thin-film electrodes for stimulation of the central auditory system. Biomed Eng IEEE Trans 36: 693–704, 1989.
REFERENCES 1. Adams JA. Historical review and appraisal of research on the learning, retention, and transfer of human motor skills. Psychol Bull 101: 41, 1987.
24. Anderson NR, Blakely T, Schalk G, Leuthardt EC, Moran DW. Electrocorticographic (ECoG) correlates of human arm movements. Exp Brain Res 223: 1–10, 2012.
2. Adrian EDA. The Mechanism of Nervous Action: Electrical sStudies of the Neurone. Philadelphia, PA: Univ. of Pennsylvania Press, 1932, p. x.
25. Andrews B, Baxendale R, Barnett R, Phillips G, Yamazaki T, Paul J, Freeman P. Hybrid FES orthosis incorporating closed loop control and sensory feedback. J Biomed Eng 10: 189 –195, 1988.
3. Aflalo T, Kellis S, Klaes C, Lee B, Shi Y, Pejsa K, Shanfield K, Hayes-Jackson S, Aisen M, Heck C, Liu C, Andersen RA. Neurophysiology. Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Science 348: 906 –910, 2015. 4. Agorelius J, Tsanakalis F, Friberg A, Thorbergsson PT, Pettersson LME, Schouenborg J. An array of highly flexible electrodes with a tailored configuration locked by gelatin during implantation: initial evaluation in cortex cerebri of awake rats. Front Neurosci 9: 331, 2015. 5. Ahn M, Jun SC. Performance variation in motor imagery brain-computer interface: a brief review. J Neurosci Methods 243: 103–110, 2015.
26. Ang CS, Sakel M, Pepper M, Phillips M. Use of brain computer interfaces in neurological rehabilitation. Br J Neurosci Nursing 7: 523–528, 2011. 27. Ang KK, Chua KS, Phua KS, Wang C, Chin ZY, Kuah CW, Low W, Guan C. A randomized controlled trial of EEG-based motor imagery brain-computer interface robotic rehabilitation for stroke. Clin EEG Neurosci 46: 310 –320, 2015. 28. Ang KK, Guan C, Chua KS, Ang BT, Kuah C, Wang C, Phua KS, Chin ZY, Zhang H. A clinical study of motor imagery-based brain-computer interface for upper limb robotic rehabilitation. Conf Proc IEEE Eng Med Biol Soc 2009: 5981–5984, 2009.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
817
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
12. Alexander JE, Porjesz B, Bauer LO, Kuperman S, Morzorati S, O’CONNORSJ, Rohrbaugh J, Begleiter H, Polich J. P300 hemispheric amplitude asymmetries from a visual oddball task. Psychophysiology 32: 467– 475, 1995.
XIII. CONCLUSION
LEBEDEV AND NICOLELIS 29. Ang KK, Guan C, Chua KS, Ang BT, Kuah CW, Wang C, Phua KS, Chin ZY, Zhang H. A clinical evaluation of non-invasive motor imagery-based brain-computer interface in stroke. Conf Proc IEEE Eng Med Biol Soc 2008: 4178 – 4181, 2008.
50. Batista AP, Yu BM, Santhanam G, Ryu SI, Afshar A, Shenoy KV. Cortical neural prosthesis performance improves when eye position is monitored. IEEE Trans Neural Syst Rehabil Eng 16: 24 –31, 2008.
30. Ang KK, Guan C, Phua KS, Wang C, Zhou L, Tang KY, Ephraim Joseph GJ, Kuah CW, Chua KS. Brain-computer interface-based robotic end effector system for wrist and hand rehabilitation: results of a three-armed randomized controlled trial for chronic stroke. Front Neuroeng 7: 30, 2014.
51. Batula AM, Ayaz H, Kim YE. Evaluating a four-class motor-imagery-based optical brain-computer interface. Conf Proc IEEE Eng Med Biol Soc 2014: 2000 –2003, 2014.
31. Ang KK, Guan C, Wang C, Phua KS, Tan AH, Chin ZY. Calibrating EEG-based motor imagery brain-computer interface from passive movement. Conf Proc IEEE Eng Med Biol Soc 2011: 4199 – 4202, 2011. 32. Ansaldo A, Castagnola E, Maggiolini E, Fadiga L, Ricci D. Superior electrochemical performance of carbon nanotubes directly grown on sharp microelectrodes. ACS Nano 5: 2206 –2214, 2011. 33. Antelis JM, Montesano L, Ramos-Murguialday A, Birbaumer N, Minguez J. On the usage of linear regression models to reconstruct limb kinematics from low frequency EEG signals. PLoS One 8: e61976, 2013.
52. Bauernfeind G, Leeb R, Wriessnegger SC, Pfurtscheller G. Development, set-up and first results for a one-channel near-infrared spectroscopy system/Entwicklung, Aufbau und vorläufige Ergebnisse eines Einkanal-Nahinfrarot-Spektroskopie-Systems. Biomedizinische Technik 53: 36 – 43, 2008. 53. Bauernfeind G, Steyrl D, Brunner C, Muller-Putz GR. Single trial classification of fNIRS-based brain-computer interface mental arithmetic data: a comparison between different classifiers. Conf Proc IEEE Eng Med Biol Soc 2014: 2004 –2007, 2014. 54. Bayliss JD, Inverso SA, Tentler A. Changing the P300 brain computer interface. Cyberpsychol Behav 7: 694 –704, 2004. 55. Bell CJ, Shenoy P, Chalodhorn R, Rao RP. Control of a humanoid robot by a noninvasive brain-computer interface in humans. J Neural Eng 5: 214 –220, 2008. 56. Bennett KP, Campbell C. Support vector machines: hype or hallelujah? ACM SIGKDD Explorations Newsletter 2: 1–13, 2000.
35. Astrand E, Wardak C, Ben Hamed S. Selective visual attention to drive cognitive brain-machine interfaces: from concepts to neurofeedback and rehabilitation applications. Front Syst Neurosci 8: 144, 2014.
57. Bensmaia SJ, Miller LE. Restoring sensorimotor function through intracortical interfaces: progress and looming challenges. Nature Rev Neurosci 15: 313–325, 2014.
36. Averbeck BB, Chafee MV, Crowe DA, Georgopoulos AP. Parietal representation of hand velocity in a copy task. J Neurophysiol 93: 508 –518, 2005.
58. Bentley AS, Andrew CM, John LR. An offline auditory P300 brain-computer interface using principal and independent component analysis techniques for functional electrical stimulation application. Conf Proc IEEE Eng Med Biol Soc 2008: 4660 – 4663, 2008.
37. Azevedo FA, Carvalho LR, Grinberg LT, Farfel JM, Ferretti RE, Leite RE, Jacob Filho W, Lent R, Herculano-Houzel S. Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain. J Comp Neurol 513: 532–541, 2009.
59. Benyamini M, Zacksenhouse M. Optimal feedback control successfully explains changes in neural modulations during experiments with brain-machine interfaces. Front Syst Neurosci 9: 71, 2015.
38. Babyak MA. What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosom Med 66: 411– 421, 2004. 39. Bach-y-Rita P. Tactile vision substitution: past and future. Int J Neurosci 19: 29 –36, 1983. 40. Bach-y-Rita P, Collins CC, Saunders FA, White B, Scadden L. Vision substitution by tactile image projection. Nature 221: 963–964, 1969. 41. Bak M, Girvin JP, Hambrecht FT, Kufta CV, Loeb GE, Schmidt EM. Visual sensations produced by intracortical microstimulation of the human occipital cortex. Med Biol Eng Comput 28: 257–259, 1990. 42. Bakardjian H, Tanaka T, Cichocki A. Optimization of SSVEP brain responses with application to eight-command Brain-Computer Interface. Neurosci Lett 469: 34 –38, 2010. 43. Balakrishnan D, Puthusserypady S. Multilayer perceptrons for the classification of brain computer interface data. In: Proceedings of the IEEE 31st Annual Northeast Bioengineering Conference. New York: IEEE, 2005, p. 118 –119. 44. Bares M, Rektor I. Basal ganglia involvement in sensory and cognitive processing. A depth electrode CNV study in human subjects. Clin Neurophysiol 112: 2022–2030, 2001. 45. Bari BA, Ollerenshaw DR, Millard DC, Wang Q, Stanley GB. Behavioral and electrophysiological effects of cortical microstimulation parameters. PLoS One 8: e82170, 2013. 46. Barlow JS. EMG artifact minimization during clinical EEG recordings by special analog filtering. Electroencephalogr Clin Neurophysiol 58: 161–174, 1984. 47. Bartels J, Andreasen D, Ehirim P, Mao H, Seibert S, Wright EJ, Kennedy P. Neurotrophic electrode: method of assembly and implantation into human motor speech cortex. J Neurosci Methods 174: 168 –176, 2008.
60. Berger H. Über das elektrenkephalogramm des menschen. Eur Arch Psychiatr Clin Neurosci 87: 527–570, 1929. 61. Berger TW, Chapin JK, Gerhardt GA, McFarland DJ, Principe JC, Soussou WV, Taylor DM, Tresco PA. Brain-Computer Interfaces: An International aAssessment of rResearch and Development Trends. New York: Springer Science & Business Media, 2008. 62. Berger TW, Hampson RE, Song D, Goonawardena A, Marmarelis VZ, Deadwyler SA. A cortical neural prosthesis for restoring and enhancing memory. J Neural Eng 8: 046017, 2011. 63. Berger TW, Song D, Marmarelis VZ, LaCoss J, Wills J, Gerhardt GA, Granacki JJ, Hampson RE, Deadwyler SA. Reverse engineering the brain: a hippocampal cognitive prosthesis for repair and enhancement of memory function. In: Neural Engineering. New York: Springer, 2013, p. 725–764. 64. Bermudez I, Badia S, Garcia Morgade A, Samaha H, Verschure PF. Using a hybrid brain computer interface and virtual reality system to monitor and promote cortical reorganization through motor activity and motor imagery training. IEEE Trans Neural Syst Rehabil Eng 21: 174 –181, 2013. 65. Bernstein NA. The Coordination and Regulation of Movements. New York: Pergamon, 1967. 66. Berthing T, Bonde S, Sørensen CB, Utko P, Nygård J, Martinez KL. Intact mammalian cell function on semiconductor nanowire arrays: new perspectives for cell-based biosensing. Small 7: 640 – 647, 2011. 67. Berti A, Frassinetti F. When far becomes near: remapping of space by tool use. J Cogn Neurosci 12: 415– 420, 2000. 68. Best MD. The Involvement of Premotor Cortex in Executing Reach to Grasp Movements. Chicago: Univ. of Chicago, 2016. 69. Bilodeau EA, Bilodeau IM. Motor-skills learning. Annu Rev Psychol 12: 243–280, 1961.
48. Bartlett JR, Doty RW. An exploration of the ability of macaques to detect microstimulation of striate cortex. Acta Neurobiol Exp 40: 713–727, 1980.
70. Bin G, Gao X, Yan Z, Hong B, Gao S. An online multi-channel SSVEP-based braincomputer interface using a canonical correlation analysis method. J Neural Eng 6: 046002, 2009.
49. Bashashati A, Fatourechi M, Ward RK, Birch GE. A survey of signal processing algorithms in brain-computer interfaces based on electrical brain signals. J Neural Eng 4: R32–57, 2007.
71. Birbaumer N, Ghanayim N, Hinterberger T, Iversen I, Kotchoubey B, Kubler A, Perelmouter J, Taub E, Flor H. A spelling device for the paralysed. Nature 398: 297–298, 1999.
818
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
34. Arboleda C, Garcia E, Posada A, Torres R. P300-based brain computer interface experimental setup. Conf Proc IEEE Eng Med Biol Soc 2009: 598 – 601, 2009.
411
412
BRAIN-MACHINE INTERFACES 72. Birbaumer N, Hummel FC. Habit learning and brain-machine interfaces (BMI): a tribute to Valentino Braitenberg’s “Vehicles”. Biol Cybern 108: 595– 601, 2014. 73. Birbaumer N, Murguialday AR, Cohen L. Brain-computer interface in paralysis. Curr Opin Neurol 21: 634 – 638, 2008.
93. Broetz D, Braun C, Weber C, Soekadar SR, Caria A, Birbaumer N. Combination of brain-computer interface training and goal-directed physical therapy in chronic stroke: a case report. Neurorehabil Neural Repair 24: 674 – 679, 2010. 94. Broseta J, Barcia-Salorio JL, Lopez-Gomez L, Roldan P, Gonzalez-Darder J, Barbera J. Burr-hole electrocorticography. Acta Neurochir Suppl 30: 91–96, 1980.
74. Birbaumer N, Weber C, Neuper C, Buch E, Haapen K, Cohen L. Physiological regulation of thinking: brain-computer interface (BCI) research. Prog Brain Res 159: 369 – 391, 2006.
95. Brouwer AM, van Erp JB. A tactile P300 brain-computer interface. Front Neurosci 4: 19, 2010.
75. Bishop CM. Neural Networks for Pattern Recognition. New York: Oxford Univ. Press, 1995.
96. Brovelli A, Ding M, Ledberg A, Chen Y, Nakamura R, Bressler SL. Beta oscillations in a large-scale sensorimotor cortical network: directional influences revealed by Granger causality. Proc Natl Acad Sci USA 101: 9849 –9854, 2004.
76. Bjornsson CS, Oh SJ, Al-Kofahi YA, Lim YJ, Smith KL, Turner JN, De S, Roysam B, Shain W, Kim SJ. Effects of insertion conditions on tissue strain and vascular damage during neuroprosthetic device insertion. J Neural Eng 3: 196 –207, 2006.
97. Brown EN, Frank LM, Tang D, Quirk MC, Wilson MA. A statistical paradigm for neural spike train decoding applied to position prediction from ensemble firing patterns of rat hippocampal place cells. J Neurosci 18: 7411–7425, 1998. 98. Brumberg JS, Nieto-Castanon A, Kennedy PR, Guenther FH. Brain-computer interfaces for speech communication. Speech Commun 52: 367–379, 2010.
78. Blazquez PM, Fujii N, Kojima J, Graybiel AM. A network representation of response probability in the striatum. Neuron 33: 973–982, 2002.
99. Brumberg JS, Wright EJ, Andreasen DS, Guenther FH, Kennedy PR. Classification of intended phoneme production from chronic intracortical microelectrode recordings in speech-motor cortex. Front Neurosci 5: 65, 2011.
79. Bleichner M, Freudenburg Z, Jansma J, Aarnoutse E, Vansteensel M, Ramsey N. Give me a sign: decoding four complex hand gestures based on high-density ECoG. Brain Struct Funct 221: 203–216, 2016.
100. Brunner C, Allison BZ, Krusienski DJ, Kaiser V, Muller-Putz GR, Pfurtscheller G, Neuper C. Improved signal processing approaches in an offline simulation of a hybrid brain-computer interface. J Neurosci Methods 188: 165–173, 2010.
80. Boniface S, Antoun N. Endovascular electroencephalography: the technique and its application during carotid amytal assessment. J Neurol Neurosurg Psychiatry 62: 193– 195, 1997.
101. Brunner P, Ritaccio AL, Emrich JF, Bischof H, Schalk G. Rapid communication with a “P300” matrix speller using electrocorticographic signals (ECoG). Front Neurosci 5: 5, 2011.
81. Borghi T, Bonfanti A, Zambra G, Gusmeroli R, Spinelli A, Baranauskas G. A compact multichannel system for acquisition and processing of neural signals. In: Proceedings of 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. New York: IEEE, 2007, p. 441– 444.
102. Brunner P, Schalk G. Toward a gaze-independent matrix speller brain-computer interface. Clin Neurophysiol 122: 1063–1064, 2011.
82. Bortole M, Controzzi M, Pisotta I, Úbeda A. BMIs for motor rehabilitation: key concepts and challenges. In: Emerging Therapies in Neurorehabilitation. New York: Springer, 2014, p. 235–247. 83. Bouton CE, Shaikhouni A, Annetta NV, Bockbrader MA, Friedenberg DA, Nielson DM, Sharma G, Sederberg PB, Glenn BC, Mysiw WJ, Morgan AG, Deogaonkar M, Rezai AR. Restoring cortical control of functional movement in a human with quadriplegia. Nature 533: 247–250, 2016. 84. Bower MR, Stead M, Van Gompel JJ, Bower RS, Sulc V, Asirvatham SJ, Worrell GA. Intravenous recording of intracranial, broadband EEG. J Neurosci Methods 214: 21–26, 2013. 85. Bozinovski S, Sestakov M, Bozinovska L. Using EEG alpha rhythm to control a mobile robot. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society. New York: IEEE, 1988. 86. Bradberry TJ, Gentili RJ, Contreras-Vidal JL. Reconstructing three-dimensional hand movements from noninvasive electroencephalographic signals. J Neurosci 30: 3432– 3437, 2010.
103. Bryan M, Green J, Chung M, Chang L, Scherer R, Smith J, Rao RP. An adaptive brain-computer interface for humanoid robot control. In: Humanoid Robots (Humanoids), 2011 11th IEEE-RAS International Conference on IEEE. New York: IEEE, 2011, p. 199 –204. 104. Buch E, Weber C, Cohen LG, Braun C, Dimyan MA, Ard T, Mellinger J, Caria A, Soekadar S, Fourkas A. Think to move: a neuromagnetic brain-computer interface (BCI) system for chronic stroke. Stroke 39: 910 –917, 2008. 105. Bullara LA, Agnew WF, Yuen TG, Jacques S, Pudenz RH. Evaluation of electrode array material for neural prostheses. Neurosurgery 5: 681– 686, 1979. 106. Buzsaki G. Large-scale recording of neuronal ensembles. Nat Neurosci 7: 446 – 451, 2004. 107. Buzsáki G. Neural syntax: cell assemblies, synapsembles, readers. Neuron 68: 362– 385, 2010. 108. Buzsaki G, Bickford RG, Ryan LJ, Young S, Prohaska O, Mandel RJ, Gage FH. Multisite recording of brain field potentials and unit activity in freely moving rats. J Neurosci Methods 28: 209 –217, 1989. 109. Buzsáki G, Chrobak JJ. Temporal structure in spatially organized neuronal ensembles: a role for interneuronal networks. Curr Opin Neurobiol 5: 504 –510, 1995.
87. Brasil F, Curado M, Agostini M, Liberati G, Garcia-Cossio E, Broetz D, Witkowski M, Birbaumer N, Soekadar S. MEP as predictor of motor recovery in chronic stroke patients after a 4-week daily physical therapy. In: Organization for Human Brain Mapping, Human Brain Mapping Annual Meeting, Beijing, China, 2012.
110. Cabrera AF, Farina D, Dremstrup K. Comparison of feature selection and classification methods for a brain-computer interface driven by non-motor imagery. Med Biol Eng Comput 48: 123–132, 2010.
88. Brindley GS. Sensations produced by electrical stimulation of the occipital poles of the cerebral hemispheres, and their use in constructing visual prostheses. Ann R Coll Surg Engl 47: 106 –108, 1970.
111. Campbell PK, Jones KE, Huber RJ, Horch KW, Normann RA. A silicon-based, threedimensional neural interface: manufacturing processes for an intracortical electrode array. Biomed Eng IEEE Trans 38: 758 –768, 1991.
89. Brindley GS, Craggs MD. The electrical activity in the motor cortex that accompanies voluntary movement. J Physiol 223: 28P–29P, 1972.
112. Capogrosso M, Milekovic T, Borton D, Wagner F, Moraud EM, Mignardot JB, Buse N, Gandar J, Barraud Q, Xing D. A brain-spine interface alleviating gait deficits after spinal cord injury in primates. Nature 539: 284 –288, 2016.
90. Brindley GS, Lewin WS. The sensations produced by electrical stimulation of the visual cortex. J Physiol 196: 479 – 493, 1968. 91. Brindley GS, Lewin WS. The visual sensations produced by electrical stimulation of the medial occipital cortex. J Physiol 194: 54 –55P, 1968. 92. Brock LG, Coombs JS, Eccles JC. The recording of potentials from motoneurones with an intracellular electrode. J Physiol 117: 431– 460, 1952.
113. Caria A, Veit R, Sitaram R, Lotze M, Weiskopf N, Grodd W, Birbaumer N. Regulation of anterior insular cortex activity using real-time fMRI. Neuroimage 35: 1238 –1246, 2007. 114. Carmena JM, Lebedev MA, Crist RE, O’Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MA. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol 1: E42, 2003.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
819
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
77. Blakely TM, Olson JD, Miller KJ, Rao RP, Ojemann JG. Neural correlates of learning in an electrocorticographic motor-imagery brain-computer interface. Brain Comput Interfaces 1: 147–157, 2014.
LEBEDEV AND NICOLELIS 136. Chi YM, Jung TP, Cauwenberghs G. Dry-contact and noncontact biopotential electrodes: methodological review. IEEE Rev Biomed Eng 3: 106 –119, 2010.
116. Castermans T, Duvinage M, Cheron G, Dutoit T. About the cortical origin of the low-delta and high-gamma rhythms observed in EEG signals during treadmill walking. Neurosci Lett 561: 166 –170, 2014.
137. Chin-Teng L, Che-Jui C, Bor-Shyh L, Shao-Hang H, Chih-Feng C, Wang IJ. A real-time wireless brain-computer interface system for drowsiness detection. IEEE Trans Biomed Circuits Syst 4: 214 –222, 2010.
117. Cecotti H. A self-paced and calibration-less SSVEP-based brain-computer interface speller. IEEE Trans Neural Syst Rehabil Eng 18: 127–133, 2010.
138. Choi B, Jo S. A low-cost EEG system-based hybrid brain-computer interface for humanoid robot navigation and recognition. PLoS One 8: e74583, 2013.
118. Cellot G, Cilia E, Cipollone S, Rancic V, Sucapane A, Giordani S, Gambazzi L, Markram H, Grandolfo M, Scaini D. Carbon nanotubes might improve neuronal performance by favouring electrical shortcuts. Nature Nanotechnol 4: 126 –133, 2009.
139. Christopher deCharms R, Christoff K, Glover GH, Pauly JM, Whitfield S, Gabrieli JD. Learned regulation of spatially localized brain activation using real-time fMRI. Neuroimage 21: 436 – 443, 2004.
119. Chae M, Liu W, Yang Z, Chen T, Kim J, Sivaprakasam M, Yuce M. A 128-channel 6mW wireless neural recording ic with on-the-fly spike sorting and uwb tansmitter. In: 2008 IEEE International Solid-State Circuits Conference-Digest of Technical Papers. New York: IEEE, 2008, p. 146 – 603.
140. Christopher deCharms R, Maeda F, Glover GH, Ludlow D, Pauly JM, Soneji D, Gabrieli JD, Mackey SC. Control over brain activation and pain learned by using real-time functional MRI. Proc Natl Acad Sci USA 102: 18626 –18631, 2005.
120. Chae Y, Jeong J, Jo S. Toward brain-actuated humanoid robots: asynchronous direct control using an EEG-based BCI. IEEE Trans Robotics 28: 1131–1144, 2012.
141. Cincotti F, Mattia D, Aloise F, Bufalari S, Astolfi L, De Vico Fallani F, Tocci A, Bianchi L, Marciani MG, Gao S, Millan J, Babiloni F. High-resolution EEG techniques for brain-computer interface applications. J Neurosci Methods 167: 31– 42, 2008.
121. Chai R, Ling SH, Hunter GP, Tran Y, Nguyen HT. Brain-computer interface classifier for wheelchair commands using neural network with fuzzy particle swarm optimization. IEEE J Biomed Health Inform 18: 1614 –1624, 2014.
142. Cisek P, Kalaska JF. Neural mechanisms for interacting with a world full of action choices. Annu Rev Neurosci 33: 269 –298, 2010.
122. Chang MH, Park KS. Frequency recognition methods for dual-frequency SSVEP based brain-computer interface. Conf Proc IEEE Eng Med Biol Soc 2013: 2220 –2223, 2013. 123. Chao ZC, Nagasaka Y, Fujii N. Long-term asynchronous decoding of arm motion using electrocorticographic signals in monkeys. Front Neuroeng 3: 3, 2010.
143. Cisek P, Kalaska JF. Simultaneous encoding of multiple potential reach directions in dorsal premotor cortex. J Neurophysiol 87: 1149 –1154, 2002. 144. Citi L, Ba D, Brown EN, Barbieri R. Likelihood methods for point processes with refractoriness. Neural Computat 26: 237–263, 2014.
124. Chapin JK, Moxon KA, Markowitz RS, Nicolelis MA. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat Neurosci 2: 664 – 670, 1999.
145. Clancy KB, Koralek AC, Costa RM, Feldman DE, Carmena JM. Volitional modulation of optically recorded calcium signals during neuroprosthetic learning. Nat Neurosci 17: 807– 809, 2014.
125. Chapin JK, Woodward DJ. Distribution of somatic sensory and active-movement neuronal discharge properties in the MI-SI cortical border area in the rat. Exp Neurol 91: 502–523, 1986.
146. Clark RW, Luschei ES. Short latency jaw movement produced by low intensity intracortical microstimulation of the precentral face area in monkeys. Brain Res 70: 144 – 147, 1974.
126. Chapin JK, Woodward DJ. Somatic sensory transmission to the cortex during movement: gating of single cell responses to touch. Exp Neurol 78: 654 – 669, 1982.
147. Clark VP, Hillyard SA. Spatial selective attention affects early extrastriate but not striate components of the visual evoked potential. J Cogn Neurosci 8: 387– 402, 1996.
127. Chase SM, Kass RE, Schwartz AB. Behavioral and neural correlates of visuomotor adaptation observed through a brain-computer interface in primary motor cortex. J Neurophysiol 108: 624 – 644, 2012.
148. Clausen J. Moving minds: ethical aspects of neural motor prostheses. Biotechnol J 3: 1493–1501, 2008.
128. Chatterjee A, Aggarwal V, Ramos A, Acharya S, Thakor NV. A brain-computer interface with vibrotactile biofeedback for haptic information. J Neuroeng Rehabil 4: 40, 2007. 129. Chaudhary U, Birbaumer N. Communication in locked-in state after brainstem stroke: a brain-computer-interface approach. Ann Transl Med 3: S29, 2015. 130. Chella A, Pagello E, Menegatti E, Sorbello R, Anzalone SM, Cinquegrani F, Tonin L, Piccione F, Prifitis K, Blanda C. A BCI teleoperated museum robotic guide. In: Complex, Intelligent and Software Intensive Systems, CISIS’09 International Conference. New York: IEEE, 2009, p. 783–788. 131. Chen Q, Peng H, Jiang C, Feng H. Off-line experiments and analysis of independent brain– computer interface. Sheng Wu Yi Xue Gong Cheng Xue Za Zhi 23: 478 – 482, 2006. 132. Chen X, Wang Y, Nakanishi M, Gao X, Jung TP, Gao S. High-speed spelling with a noninvasive brain-computer interface. Proc Natl Acad Sci USA 112: E6058 – 6067, 2015. 133. Cheng G, Fitzsimmons N, Morimoto J, Lebedev M, Kawato M, Nicolelis M. Bipedal locomotion with a humanoid robot controlled by cortical ensemble activity. Abstr Soc Neurosci 22, 2007. 134. Chestek CA, Gilja V, Nuyujukian P, Foster JD, Fan JM, Kaufman MT, Churchland MM, Rivera-Alvidrez Z, Cunningham JP, Ryu SI, Shenoy KV. Long-term stability of neural prosthetic control signals from silicon cortical arrays in rhesus macaque motor cortex. J Neural Eng 8: 045005, 2011. 135. Chestek CA, Gilja V, Nuyujukian P, Kier RJ, Solzbacher F, Ryu SI, Harrison RR, Shenoy KV. HermesC: low-power wireless neural recording system for freely moving primates. IEEE Trans Neural Syst Rehabil Eng 17: 330 –338, 2009.
820
149. Cohen D. Magnetoencephalography: detection of the brain’s electrical activity with a superconducting magnetometer. Science 175: 664 – 666, 1972. 150. Cohen D. Magnetoencephalography: evidence of magnetic fields produced by alpharhythm currents. Science 161: 784 –786, 1968. 151. Cohen O, Koppel M, Malach R, Friedman D. Controlling an avatar by thought using real-time fMRI. J Neural Eng 11: 035006, 2014. 152. Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler-Kabara EC, Weber DJ, McMorland AJ, Velliste M, Boninger ML, Schwartz AB. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381: 557–564, 2013. 153. Collins WR Jr, Nulsen FE, Randt CT. Relation of peripheral nerve fiber size and sensation in man. Arch Neurol 3: 381–385, 1960. 154. Combaz A, Van Hulle MM. Simultaneous detection of P300 and steady-state visually evoked potentials for hybrid brain-computer interface. PLoS One 10: e0121481, 2015. 155. Connors BW, Long MA. Electrical synapses in the mammalian brain. Annu Rev Neurosci 27: 393– 418, 2004. 156. Contreras-Vidal JL, Grossman RG. NeuroRex: a clinical neural interface roadmap for EEG-based brain machine interfaces to a lower body robotic exoskeleton. Conf Proc IEEE Eng Med Biol Soc 2013: 1579 –1582, 2013. 157. Cordeau JP, Gybels J, Jasper H, Poirier LJ. Microelectrode studies of unit discharges in the sensorimotor cortex: investigations in monkeys with experimental tremor. Neurology 10: 591– 600, 1960. 158. Cordo PJ, Gurfinkel VS. Motor coordination can be fully understood only by studying complex movements. Prog Brain Res 143: 29 –38, 2004.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
115. Carpenter AF, Georgopoulos AP, Pellizzer G. Motor cortical encoding of serial order in a context-recall task. Science 283: 1752–1757, 1999.
413
414
BRAIN-MACHINE INTERFACES 159. Corralejo R, Hornero R, Alvarez D. Feature selection using a genetic algorithm in a motor imagery-based Brain Computer Interface. Conf Proc IEEE Eng Med Biol Soc 2011: 7703–7706, 2011.
181. Dangi S, Orsborn AL, Moorman HG, Carmena JM. Design and analysis of closed-loop decoder adaptation algorithms for brain-machine interfaces. Neural Comput 25: 1693–1731, 2013.
160. Costa Á, Hortal E, Iáñez E, Azorín JM. A supplementary system for a brain-machine interface based on jaw artifacts for the bidimensional control of a robotic arm. PloS One 9: e112352, 2014.
182. Davis KD, Kiss ZH, Luo L, Tasker RR, Lozano AM, Dostrovsky JO. Phantom sensations generated by thalamic microstimulation. Nature 391: 385–387, 1998. 183. Davis TS, Wark HA, Hutchinson DT, Warren DJ, O’Neill K, Scheinblum T, Clark GA, Normann RA, Greger B. Restoring motor control and sensory feedback in people with upper extremity amputations using arrays of 96 microelectrodes implanted in the median and ulnar nerves. J Neural Eng 13: 036001, 2016.
162. Courchesne E, Hillyard SA, Galambos R. Stimulus novelty, task relevance and the visual evoked potential in man. Electroencephalogr Clin Neurophysiol 39: 131–143, 1975.
184. De Massari D, Matuz T, Furdea A, Ruf CA, Halder S, Birbaumer N. Brain-computer interface and semantic classical conditioning of communication in paralysis. Biol Psychol 92: 267–274, 2013.
163. Cowey A, Stoerig P. The neurobiology of blindsight. Trends Neurosci 14: 140 –145, 1991.
185. De Vries S, Mulder T. Motor imagery and stroke rehabilitation: a critical discussion. J Rehab Med 39: 5–13, 2007.
164. Cowey A, Stoerig P. Visual detection in monkeys with blindsight. Neuropsychologia 35: 929 –939, 1997.
186. Dehzangi O, Jafari R. Time-varying and simultaneous frequency stimulation for multiclass SSVEP-based brain-computer interface. Conf Proc IEEE Eng Med Biol Soc 2015: 1757–1760, 2015.
165. Cox RW, Jesmanowicz A, Hyde JS. Real-time functional magnetic resonance imaging. Magn Reson Med 33: 230 –236, 1995. 166. Coyle S, Ward T, Markham C, McDarby G. On the suitability of near-infrared (NIR) systems for next-generation brain-computer interfaces. Physiol Measurement 25: 815, 2004. 167. Craggs MD. Cortical control of motor prostheses: using the cord-transected baboon as the primate model for human paraplegia. Adv Neurol 10: 91–101, 1975. 168. Craggs MD. Electrical activity of the motor cortex associated with voluntary movements in the baboon. J Physiol 237: 12P–13P, 1974. 169. Craig DA, Nguyen HT. Adaptive EEG thought pattern classifier for advanced wheelchair control. Conf Proc IEEE Eng Med Biol Soc 2007: 2544 –2547, 2007. 170. Cramer SC, Sur M, Dobkin BH, O’Brien C, Sanger TD, Trojanowski JQ, Rumsey JM, Hicks R, Cameron J, Chen D, Chen WG, Cohen LG, deCharms C, Duffy CJ, Eden GF, Fetz EE, Filart R, Freund M, Grant SJ, Haber S, Kalivas PW, Kolb B, Kramer AF, Lynch M, Mayberg HS, McQuillen PS, Nitkin R, Pascual-Leone A, Reuter-Lorenz P, Schiff N, Sharma A, Shekim L, Stryker M, Sullivan EV, Vinogradov S. Harnessing neuroplasticity for clinical applications. Brain 134: 1591–1609, 2011. 171. Crammond DJ, Kalaska JF. Neuronal activity in primate parietal cortex area 5 varies with intended movement direction during an instructed-delay period. Exp Brain Res 76: 458 – 462, 1989. 172. Crone NE, Sinai A, Korzeniewska A. High-frequency gamma oscillations and human brain mapping with electrocorticography. Prog Brain Res 159: 275–295, 2006. 173. Crowe DA, Chafee MV, Averbeck BB, Georgopoulos AP. Participation of primary motor cortical neurons in a distributed network during maze solution: representation of spatial parameters and time-course comparison with parietal area 7a. Exp Brain Res 158: 28 –34, 2004.
187. Dehzangi O, Nathan V, Zong C, Lee C, Kim I, Jafari R. A novel stimulation for multi-class SSVEP-based brain-computer interface using patterns of time-varying frequencies. Conf Proc IEEE Eng Med Biol Soc 2014: 118 –121, 2014. 188. DeLong MR. Activity of basal ganglia neurons during movement. Brain Res 40: 127– 135, 1972. 189. DeLong MR, Alexander GE, Georgopoulos AP, Crutcher MD, Mitchell SJ, Richardson RT. Role of basal ganglia in limb movements. Hum Neurobiol 2: 235–244, 1984. 190. Denby B, Schultz T, Honda K, Hueber T, Gilbert JM, Brumberg JS. Silent speech interfaces. Speech Commun 52: 270 –287, 2010. 191. Denison DG. Bayesian Methods for Nonlinear Classification and Regression. New York: Wiley, 2002. 192. Denker M, Roux S, Linden H, Diesmann M, Riehle A, Grun S. The local field potential reflects surplus spike synchrony. Cereb Cortex 21: 2681–2695, 2011. 193. Dennett DC. Consciousness Explained. Boston: Little, Brown, 1991, p. xiii. 194. Di Pellegrino G, Wise SP. Visuospatial versus visuomotor activity in the premotor and prefrontal cortex of a primate. J Neurosci 13: 1227–1243, 1993. 195. Di Pino G, Maravita A, Zollo L, Guglielmelli E, Di Lazzaro V. Augmentation-related brain plasticity. Front Syst Neurosci 8: 109, 2014. 196. Di Russo F, Martínez A, Sereno MI, Pitzalis S, Hillyard SA. Cortical sources of the early components of the visual evoked potential. Hum Brain Mapping 15: 95–111, 2002. 197. DiGiovanna J, Mahmoudi B, Fortes J, Principe JC, Sanchez JC. Coadaptive brainmachine interface via reinforcement learning. IEEE Trans Biomed Eng 56: 54 – 64, 2009.
174. Csicsvari J, Henze DA, Jamieson B, Harris KD, Sirota A, Barthó P, Wise KD, Buzsáki G. Massively parallel recording of unit and local field potentials with silicon-based electrodes. J Neurophysiol 90: 1314 –1323, 2003.
198. Djourno A, Eyriès C. Prothese auditive par excitation electrique a distance du nerf sensoriel a laide dun bobinage inclus a demeure. Presse Médicale 65: 1417, 1957.
175. Cui H. Forward prediction in the posterior parietal cortex and dynamic brain-machine interface. Front Integr Neurosci 10: 35, 2016.
199. Dobelle WH. Artificial vision for the blind. The summit may be closer than you think. ASAIO J 40: 919 –922, 1994.
176. Cullen KE. Sensory signals during active versus passive movement. Curr Opin Neurobiol 14: 698 –706, 2004.
200. Dobelle WH, Mladejovsky MG. Phosphenes produced by electrical stimulation of human occipital cortex, and their application to the development of a prosthesis for the blind. J Physiol 243: 553–576, 1974.
177. Cunningham P, Delany SJ. k-Nearest neighbour classifiers. Multiple Classifier Systems 1–17, 2007. 178. Cushing H. A note upon the faradic stimulation of the postcentral gyrus in conscious patients. Brain 32: 44 –53, 1909. 179. Dahl WD. An alpha rhythm feedback control unit. Rep US Nav Med Res Lab 5848: 20, 1962. 180. Dangi S, Gowda S, Moorman HG, Orsborn AL, So K, Shanechi M, Carmena JM. Continuous closed-loop decoder adaptation with a recursive maximum likelihood algorithm allows for rapid performance acquisition in brain-machine interfaces. Neural Comput 26: 1811–1839, 2014.
201. Dobelle WH, Mladejovsky MG, Evans JR, Roberts TS, Girvin JP. “Braille” reading by a blind volunteer by visual cortex stimulation. Nature 259: 111–112, 1976. 202. Dobelle WH, Mladejovsky MG, Girvin JP. Artifical vision for the blind: electrical stimulation of visual cortex offers hope for a functional prosthesis. Science 183: 440 – 444, 1974. 203. Dobelle WH, Quest DO, Antunes JL, Roberts TS, Girvin JP. Artificial vision for the blind by electrical stimulation of the visual cortex. Neurosurgery 5: 521–527, 1979. 204. Dobkin BH. Brain-computer interface technology as a tool to augment plasticity and outcomes for neurological rehabilitation. J Physiol 579: 637– 642, 2007.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
821
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
161. Costa EJ, Cabral EF. EEG-based discrimination between imagination of left and right hand movements using adaptive gaussian representation. Med Eng Physics 22: 345– 348, 2000.
LEBEDEV AND NICOLELIS 205. Donati AR, Shokur S, Morya E, Campos DS, Moioli RC, Gitti CM, Augusto PB, Tripodi S, Pires CG, Pereira GA, Brasil FL, Gallo S, Lin AA, Takigami AK, Aratanha MA, Joshi S, Bleuler H, Cheng G, Rudolph A, Nicolelis MA. Long-term training with a brainmachine interface-based gait protocol induces partial neurological recovery in paraplegic patients. Sci Rep 6: 30383, 2016.
227. Evarts EV. Activity of motor cortex neurons in association with learned movement. Int J Neurosci 3: 113–124, 1972.
206. Donchin E. Event-related brain potentials: a tool in the study of human information processing. In: Evoked Brain Potentials and Behavior. New York: Springer, 1979, p. 13– 88.
229. Evarts EV. Brain mechanisms in movement. Sci Am 229: 96 –103, 1973.
207. Donchin E, Cohen L. Averaged evoked potentials and intramodality selective attention. Electroencephalogr Clin Neurophysiol 22: 537–546, 1967. 208. Donchin E, Coles MG. Is the P300 component a manifestation of context updating. Behav Brain Sci 11: 357– 427, 1988. 209. Donchin E, Spencer KM, Wijesinghe R. The mental prosthesis: assessing the speed of a P300-based brain-computer interface. IEEE Trans Rehabil Eng 8: 174 –179, 2000. 210. Dornhege G. Toward Brain-Computer Interfacing. Boston: MIT Press, 2007.
212. Doyon J, Bellec P, Amsel R, Penhune V, Monchi O, Carrier J, Lehericy S, Benali H. Contributions of the basal ganglia and functionally related brain structures to motor learning. Behav Brain Res 199: 61–75, 2009. 213. Doyon J, Penhune V, Ungerleider LG. Distinct contribution of the cortico-striatal and cortico-cerebellar systems to motor skill learning. Neuropsychologia 41: 252–262, 2003. 214. Dryg ID, Ward MP, Qing KY, Mei H, Schaffer JE, Irazoqui PP. Magnetically inserted neural electrodes: tissue response and functional lifetime. Neural Syst Rehab Eng IEEE Trans 23: 562–571, 2015. 215. Duan X, Gao R, Xie P, Cohen-Karni T, Qing Q, Choe HS, Tian B, Jiang X, Lieber CM. Intracellular recordings of action potentials by an extracellular nanoscale field-effect transistor. Nature Nanotechnol 7: 174 –179, 2012. 216. Eason RG. Visual evoked potential correlates of early neural filtering during selective attention. Bull Psychon Soc 18: 203–206, 1981. 217. Eckstein MP, Das K, Pham BT, Peterson MF, Abbey CK, Sy JL, Giesbrecht B. Neural decoding of collective wisdom with multi-brain computing. NeuroImage 59: 94 –108, 2012. 218. Eddington DK. Speech discrimination in deaf subjects with cochlear implants. J Acoust Soc Am 68: 885– 891, 1980. 219. Eddington DK. Speech recognition in deaf subjects with multichannel intracochlear electrodes. Ann NY Acad Sci 405: 241–258, 1983. 220. Edell DJ, Toi VV, McNeil VM, Clark LD. Factors influencing the biocompatibility of insertable silicon microshafts in cerebral cortex. IEEE Trans Biomed Eng 39: 635– 643, 1992. 221. Edelman GM, Gally JA. Degeneracy and complexity in biological systems. Proc Natl Acad Sci USA 98: 13763–13768, 2001. 222. Eden UT, Frank LM, Barbieri R, Solo V, Brown EN. Dynamic analysis of neural encoding by point process adaptive filtering. Neural Computat 16: 971–998, 2004. 223. Escolano C, Antelis JM, Minguez J. A telepresence mobile robot controlled with a noninvasive brain-computer interface. IEEE Trans Syst Man Cybern B Cybern 42: 793– 804, 2012. 224. Eser PC, Donaldson NN, Knecht H, Stussi E. Influence of different stimulation frequencies on power output and fatigue during FES-cycling in recently injured SCI people. IEEE Trans Neural Syst Rehab Eng 11: 236 –240, 2003. 225. Ethier C, Oby ER, Bauman M, Miller LE. Restoration of grasp following paralysis through brain-controlled stimulation of muscles. Nature 485: 368 –371, 2012. 226. Etienne S, Guthrie M, Goillandeau M, Nguyen TH, Orignac H, Gross C, Boraud T. Easy rider: monkeys learn to drive a wheelchair to navigate through a complex maze. PloS One 9: e96275, 2014.
822
228. Evarts EV. Activity of pyramidal tract neurons during postural fixation. J Neurophysiol 32: 375–385, 1969.
230. Evarts EV. Contrasts between activity of precentral and postcentral neurons of cerebral cortex during movement in the monkey. Brain Res 40: 25–31, 1972. 231. Evarts EV. Motor cortex reflexes associated with learned movement. Science 179: 501–503, 1973. 232. Evarts EV. Precentral and postcentral cortical activity in association with visually triggered movement. J Neurophysiol 37: 373–381, 1974. 233. Evarts EV. Pyramidal tract activity associated with a conditioned hand movement in the monkey. J Neurophysiol 29: 1011–1027, 1966. 234. Evarts EV. Relation of pyramidal tract activity to force exerted during voluntary movement. J Neurophysiol 31: 14 –27, 1968. 235. Evarts EV. Temporal patterns of discharge of pyramidal tract neurons during sleep and waking in the monkey. J Neurophysiol 27: 152–171, 1964. 236. Evarts EV, Bental E, Bihari B, Huttenlocher PR. Spontaneous discharge of single neurons during sleep and waking. Science 135: 726 –728, 1962. 237. Evarts EV, Fromm C. Information processing in the sensorimotor cortex during voluntary movement. Prog Brain Res 54: 143–155, 1980. 238. Evarts EV, Tanji J. Gating of motor cortex reflexes by prior instruction. Brain Res 71: 479 – 494, 1974. 239. Even-Chen N, Stavisky SD, Kao JC, Ryu SI, Shenoy KV. Auto-deleting brain machine interface: Error detection using spiking neural activity in the motor cortex. Conf Proc IEEE Eng Med Biol Soc 2015: 71–75, 2015. 240. Fabiani GE, McFarland DJ, Wolpaw JR, Pfurtscheller G. Conversion of EEG activity into cursor movement by a brain-computer interface (BCI). IEEE Trans Neural Syst Rehabil Eng 12: 331–338, 2004. 241. Falk TH, Guirgis M, Power S, Chau TT. Taking NIRS-BCIs outside the lab: towards achieving robustness against environment noise. IEEE Trans Neural Syst Rehab Eng 19: 136 –146, 2011. 242. Farah MJ, Wolpe PR. Monitoring and manipulating brain function: new neuroscience technologies and their ethical implications. Hastings Center Report 34: 35– 45, 2004. 243. Farwell LA, Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol 70: 510 – 523, 1988. 244. Fatourechi M, Bashashati A, Ward RK, Birch GE. EMG and EOG artifacts in brain computer interface systems: a survey. Clin Neurophysiol 118: 480 – 494, 2007. 245. Fazel-Rezai R, Abhari K. A comparison between a matrix-based and a region-based P300 speller paradigms for brain-computer interface. Conf Proc IEEE Eng Med Biol Soc 2008: 1147–1150, 2008. 246. Fazel-Rezai R, Allison BZ, Guger C, Sellers EW, Kleih SC, Kubler A. P300 brain computer interface: current challenges and emerging trends. Front Neuroeng 5: 14, 2012. 247. Fazel-Rezai R, Gavett S, Ahmad W, Rabbi A, Schneider E. A comparison among several P300 brain-computer interface speller paradigms. Clin EEG Neurosci 42: 209 – 213, 2011. 248. Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Muller KR, Blankertz B. Enhanced performance by a hybrid NIRS-EEG brain computer interface. Neuroimage 59: 519 –529, 2012. 249. Feldman AG. Once more on the equilibrium-point hypothesis ( model) for motor control. J Motor Behav 18: 17–54, 1986. 250. Feldman AG, Levin MF. The equilibrium-point hypothesis-past, present and future. In: Progress in Motor Control. New York: Springer, 2009, p. 699 –726.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
211. Doud AJ, Lucas JP, Pisansky MT, He B. Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface. PLoS One 6: e26322, 2011.
415
416
BRAIN-MACHINE INTERFACES 251. Fernandez E, Greger B, House PA, Aranda I, Botella C, Albisua J, Soto-Sanchez C, Alfaro A, Normann RA. Acute human brain responses to intracortical microelectrode arrays: challenges and future prospects. Front Neuroeng 7: 24, 2014. 252. Ferrari M, Mottola L, Quaresima V. Principles, techniques, and limitations of near infrared spectroscopy. Can J Appl Physiol 29: 463– 487, 2004. 253. Fetz EE. Are movement parameters recognizably coded in the activity of single neurons? Behav Brain Sci 15: 679 – 690, 1992.
274. Friston K. The free-energy principle: a unified brain theory? Nat Rev Neurosci 11: 127–138, 2010. 275. Friston K, Kilner J, Harrison L. A free energy principle for the brain. J Physiol Paris 100: 70 – 87, 2006. 276. Friston KJ, Fletcher P, Josephs O, Holmes A, Rugg M, Turner R. Event-related fMRI: characterizing differential responses. Neuroimage 7: 30 – 40, 1998.
254. Fetz EE. Operant conditioning of cortical unit activity. Science 163: 955–958, 1969.
277. Fromm C, Evarts EV. Pyramidal tract neurons in somatosensory cortex: central and peripheral inputs during voluntary movement. Brain Res 238: 186 –191, 1982.
255. Fetz EE. Restoring motor function with bidirectional neural interfaces. Prog Brain Res 218: 241–252, 2015.
278. Fu TM, Hong G, Zhou T, Schuhmann TG, Viveros RD, Lieber CM. Stable long-term chronic brain mapping at the single-neuron level. Nature Methods 13: 875– 882, 2016.
256. Finger S. Origins of Neuroscience: A History of Explorations Into Brain Function. New York: Oxford Univ. Press, 1994, p. xviii.
279. Fuchs T, Birbaumer N, Lutzenberger W, Gruzelier JH, Kaiser J. Neurofeedback treatment for attention-deficit/hyperactivity disorder in children: a comparison with methylphenidate. Appl Psychophysiol Biofeedback 28: 1–12, 2003.
257. Finke A, Lenhardt A, Ritter H. The MindGame: a P300-based brain-computer interface game. Neural Netw 22: 1329 –1333, 2009.
259. Fisher RA. The use of multiple measurements in taxonomic problems. Ann Eugenics 7: 179 –188, 1936. 260. Fitzsimmons NA, Drake W, Hanson TL, Lebedev MA, Nicolelis MA. Primate reaching cued by multichannel spatiotemporal cortical microstimulation. J Neurosci 27: 5593– 5602, 2007. 261. Fitzsimmons NA, Lebedev MA, Peikon ID, Nicolelis MA. Extracting kinematic parameters for monkey bipedal walking from cortical neuronal ensemble activity. Front Integr Neurosci 3: 3, 2009. 262. Flesher SN, Collinger JL, Foldes ST, Weiss JM, Downey JE, Tyler-Kabara EC, Bensmaia SJ, Schwartz AB, Boninger ML, Gaunt RA. Intracortical microstimulation of human somatosensory cortex. Sci Transl Med 8: 361ra141–361ra141, 2016. 263. Flint RD, Scheid MR, Wright ZA, Solla SA, Slutzky MW. Long-term stability of motor cortical activity: implications for brain machine interfaces and optimal feedback control. J Neurosci 36: 3623–3632, 2016. 264. Flint RD, Wright ZA, Scheid MR, Slutzky MW. Long term, stable brain machine interface performance using local field potentials and multiunit spikes. J Neural Eng 10: 056005, 2013. 265. Folcher M, Oesterle S, Zwicky K, Thekkottil T, Heymoz J, Hohmann M, Christen M, El-Baba MD, Buchmann P, Fussenegger M. Mind-controlled transgene expression by a wireless-powered optogenetic designer cell implant. Nature Commun 5: 5392, 2014. 266. Fonseca C, Silva Cunha JP, Martins RE, Ferreira VM, Marques de Sa JP, Barbosa MA, Martins da Silva A. A novel dry active electrode for EEG recording. IEEE Trans Biomed Eng 54: 162–165, 2007. 267. Foster JD, Nuyujukian P, Freifeld O, Gao H, Walker R, Ryu SI, Meng TH, Murmann B, Black MJ, Shenoy KV. A freely-moving monkey treadmill model. J Neural Eng 11: 046020, 2014. 268. Foster JD, Nuyujukian P, Freifeld O, Ryu SI, Black MJ, Shenoy KV. A framework for relating neural activity to freely moving behavior. Conf Proc IEEE Eng Med Biol Soc 2012: 2736 –2739, 2012. 269. Frank K. Some approaches to the technical problem of chronic excitation of peripheral nerve. Ann Otol Rhinol Laryngol 77: 761–771, 1968. 270. Frank K. Use of neural signals to control external device. Neurosci Res Prog Bull 9: 113–118, 1971. 271. Freire MA, Morya E, Faber J, Santos JR, Guimaraes JS, Lemos NA, Sameshima K, Pereira A, Ribeiro S, Nicolelis MA. Comprehensive analysis of tissue preservation and recording quality from chronic multielectrode implants. PLoS One 6: e27554, 2011.
281. Galán F, Nuttin M, Lew E, Ferrez PW, Vanacker G, Philips J, Millán JdR. A brainactuated wheelchair: asynchronous and non-invasive brain-computer interfaces for continuous control of robots. Clin Neurophysiol 119: 2159 –2169, 2008. 282. Gancet J, Ilzkovitz M, Cheron G, Ivanenko Y, van der Kooij H, van der Helm F, Zanow F, Thorsteinsson F. MINDWALKER: a brain controlled lower limbs exoskeleton for rehabilitation. Potential applications to space. In: 11th Symposium on Advanced Space Technologies in Robotics and Automation; European Space Agency’s European Space Research and Technology Centre. Noordwijk: The Netherlands, April 12–14, 2011. 283. Ganguly K, Carmena JM. Emergence of a stable cortical map for neuroprosthetic control. PLoS Biol 7: e1000153, 2009. 284. Ganguly K, Dimitrov DF, Wallis JD, Carmena JM. Reversible large-scale modification of cortical networks during neuroprosthetic control. Nat Neurosci 14: 662– 667, 2011. 285. Ganin I, Shishkin S, Kaplan AY. A P300 BCI with stimuli presented on moving objects: four-session single-trial and triple-trial tests with a game-like task design. PLoS One 8: e77755, 2013. 286. Garces Correa A, Orosco L, Laciar E. Automatic detection of drowsiness in EEG records based on multimodal analysis. Med Eng Phys 36: 244 –249, 2014. 287. Gargiulo G, Bifulco P, McEwan A, Nasehi Tehrani J, Calvo RA, Romano M, Ruffo M, Shephard R, Cesarelli M, Jin C, Mohamed A, van Schaik A. Dry electrode bio-potential recordings. Conf Proc IEEE Eng Med Biol Soc 2010: 6493– 6496, 2010. 288. Garrett D, Peterson DA, Anderson CW, Thaut MH. Comparison of linear, nonlinear, and feature selection methods for EEG signal classification. IEEE Trans Neural Syst Rehab Eng 11: 141–144, 2003. 289. Gawne TJ. The local and non-local components of the local field potential in awake primate visual cortex. J Comput Neurosci 29: 615– 623, 2010. 290. Georgopoulos AP. Cortical mechanisms subserving reaching. Ciba Found Symp 132: 125–141, 1987. 291. Georgopoulos AP. Neural aspects of cognitive motor control. Curr Opin Neurobiol 10: 238 –241, 2000. 292. Georgopoulos AP. Population activity in the control of movement. Int Rev Neurobiol 37: 103–119, 1994. 293. Georgopoulos AP, Carpenter AF. Coding of movements in the motor cortex. Curr Opin Neurobiol 33: 34 –39, 2015. 294. Georgopoulos AP, Kalaska JF, Caminiti R, Massey JT. On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. J Neurosci 2: 1527–1537, 1982.
272. Friedrich EV, Scherer R, Neuper C. Long-term evaluation of a 4-class imagery-based brain-computer interface. Clin Neurophysiol 124: 916 –927, 2013.
295. Georgopoulos AP, Kettner RE, Schwartz AB. Primate motor cortex and free arm movements to visual targets in three-dimensional space. II. Coding of the direction of movement by a neuronal population. J Neurosci 8: 2928 –2937, 1988.
273. Friston K. The free-energy principle: a rough guide to the brain? Trends Cogn Sci 13: 293–301, 2009.
296. Georgopoulos AP, Langheim FJ, Leuthold AC, Merkle AN. Magnetoencephalographic signals predict movement trajectory in space. Exp Brain Res 167: 132–135, 2005.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
823
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
258. Finley WW, Smith HA, Etherton MD. Reduction of seizures and normalization of the EEG in a severe epileptic following sensorimotor biofeedback training: preliminary study. Biol Psychol 2: 189 –203, 1975.
280. Fusi S, Miller EK, Rigotti M. Why neurons mix: high dimensionality for higher cognition. Curr Opin Neurobiol 37: 66 –74, 2016.
LEBEDEV AND NICOLELIS 297. Georgopoulos AP, Lurito JT, Petrides M, Schwartz AB, Massey JT. Mental rotation of the neuronal population vector. Science 243: 234 –236, 1989.
321. Grill WM, Simmons AM, Cooper SE, Miocinovic S, Montgomery EB, Baker KB, Rezai AR. Temporal excitation properties of paresthesias evoked by thalamic microstimulation. Clin Neurophysiol 116: 1227–1234, 2005.
298. Georgopoulos AP, Massey JT. Cognitive spatial-motor processes. 1. The making of movements at various angles from a stimulus direction. Exp Brain Res 65: 361–370, 1987.
322. Grinvald A, Frostig R, Lieke E, Hildesheim R. Optical imaging of neuronal activity. Physiol Rev 68: 1285–1366, 1988.
299. Georgopoulos AP, Schwartz AB, Kettner RE. Neuronal population coding of movement direction. Science 233: 1416 –1419, 1986.
323. Grinvald A, Hildesheim R. VSDI: a new era in functional imaging of cortical dynamics. Nature Rev Neurosci 5: 874 – 885, 2004.
300. Georgopoulos AP, Taira M, Lukashin A. Cognitive neurophysiology of the motor cortex. Science 260: 47–52, 1993.
324. Grosse-Wentrup M, Mattia D, Oweiss K. Using brain-computer interfaces to induce neural plasticity and restore function. J Neural Eng 8: 025004, 2011.
301. Gergondet P, Druon S, Kheddar A, Hintermüller C, Guger C, Slater M. Using braincomputer interface to steer a humanoid robot. In: Robotics and Biomimetics (ROBIO), 2011 International Conference. New York: IEEE, p. .192–197
325. Gualtierotti T, Bailey P. A neutral buoyancy micro-electrode for prolonged recording from single nerve units. Electroencephalogr Clin Neurophysiol 25: 77– 81, 1968.
302. Gerstein GL, Bedenbaugh P, Aertsen AM. Neuronal assemblies. Biomed Eng IEEE Trans 36: 4 –14, 1989.
304. Ghosh KK, Burns LD, Cocker ED, Nimmerjahn A, Ziv Y, El Gamal A, Schnitzer MJ. Miniaturized integration of a fluorescence microscope. Nature Methods 8: 871– 878, 2011. 305. Giat Y, Mizrahi J, Levy M. A musculotendon model of the fatigue profiles of paralyzed quadriceps muscle under FES. IEEE Trans Biomed Eng 40: 664 – 674, 1993. 306. Gilbert CD, Sigman M. Brain states: top-down influences in sensory processing. Neuron 54: 677– 696, 2007. 307. Gilja V, Nuyujukian P, Chestek CA, Cunningham JP, Yu BM, Fan JM, Churchland MM, Kaufman MT, Kao JC, Ryu SI, Shenoy KV. A high-performance neural prosthesis enabled by control algorithm design. Nat Neurosci 15: 1752–1757, 2012. 308. Golgi C. Sulla fina anatomia degli organi centrali del sistema nervoso. Firenze: Giunti, 1995, p. 264. 309. Golub MD, Byron MY, Chase SM. Internal models engaged by brain-computer interface control. In: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. New York: IEEE, 2012, p. 1327–1330.
326. Guenther FH, Brumberg JS. Brain-machine interfaces for real-time speech synthesis. Conf Proc IEEE Eng Med Biol Soc 2011: 5360 –5363, 2011. 327. Guenther FH, Brumberg JS, Wright EJ, Nieto-Castanon A, Tourville JA, Panko M, Law R, Siebert SA, Bartels JL, Andreasen DS, Ehirim P, Mao H, Kennedy PR. A wireless brain-machine interface for real-time speech synthesis. PLoS One 4: e8218, 2009. 328. Guertin PA. The mammalian central pattern generator for locomotion. Brain Res Rev 62: 45–56, 2009. 329. Guger C, Krausz G, Allison BZ, Edlinger G. Comparison of dry and gel based electrodes for p300 brain-computer interfaces. Front Neurosci 6: 60, 2012. 330. Guggenmos DJ, Azin M, Barbay S, Mahnken JD, Dunham C, Mohseni P, Nudo RJ. Restoration of function after brain damage using a neural prosthesis. Proc Natl Acad Sci USA 110: 21177–21182, 2013. 331. Gwin JT, Gramann K, Makeig S, Ferris DP. Removal of movement artifact from high-density EEG recorded during walking and running. J Neurophysiol 103: 3526 – 3534, 2010. 332. Halder S, Hammer EM, Kleih SC, Bogdan M, Rosenstiel W, Birbaumer N, Kubler A. Prediction of auditory and visual p300 brain-computer interface aptitude. PLoS One 8: e53513, 2013. 333. Halder S, Rea M, Andreoni R, Nijboer F, Hammer EM, Kleih SC, Birbaumer N, Kubler A. An auditory oddball brain-computer interface for binary choices. Clin Neurophysiol 121: 516 –523, 2010.
310. Golub MD, Byron MY, Chase SM. Internal models for interpreting neural population activity during sensorimotor control. eLife 4: e10015, 2015.
334. Hämäläinen M, Hari R, Ilmoniemi RJ, Knuutila J, Lounasmaa OV. Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain. Rev Modern Physics 65: 413, 1993.
311. Goncharova II, McFarland DJ, Vaughan TM, Wolpaw JR. EMG contamination of EEG: spectral and topographical characteristics. Clin Neurophysiol 114: 1580 –1593, 2003.
335. Hampton AN, O’Doherty JP. Decoding the neural substrates of reward-related decision making with functional MRI. Proc Natl Acad Sci USA 104: 1377–1382, 2007.
312. Goodale MA, Milner AD. Separate visual pathways for perception and action. Trends Neurosci 15: 20 –25, 1992.
336. Hanson TL, Fuller AM, Lebedev MA, Turner DA, Nicolelis MA. Subcortical neuronal ensembles: an analysis of motor task association, tremor, oscillations, and synchrony in human patients. J Neurosci 32: 8620 – 8632, 2012.
313. Goodale MA, Milner AD, Jakobson L, Carey D. A neurological dissociation between perceiving objects and grasping them. Nature 349: 154 –156, 1991. 314. Grant RA, Mitchinson B, Fox CW, Prescott TJ. Active touch sensing in the rat: anticipatory and regulatory control of whisker movements during surface exploration. J Neurophysiol 101: 862– 874, 2009. 315. Gratton G, Fabiani M. Shedding light on brain function: the event-related optical signal. Trends Cogn Sci 5: 357–363, 2001. 316. Grau C, Ginhoux R, Riera A, Nguyen TL, Chauvat H, Berg M, Amengual JL, PascualLeone A, Ruffini G. Conscious brain-to-brain communication in humans using noninvasive technologies. PLoS One 9: e105225, 2014. 317. Green AM, Kalaska JF. Learning to move machines with the mind. Trends Neurosci 34: 61–75, 2011. 318. Green JD. A simple microelectrode for recording from the central nervous system. Nature 182: 962, 1958. 319. Grewal MS. Kalman Filtering. New York: Springer, 2011. 320. Grewe BF, Langer D, Kasper H, Kampa BM, Helmchen F. High-speed in vivo calcium imaging reveals neuronal network activity with near-millisecond precision. Nature Methods 7: 399 – 405, 2010.
824
337. Haofei W, Xujiong D, Zhaokang C, Shi BE. Hybrid gaze/EEG brain computer interface for robot arm control on a pick and place task. Conf Proc IEEE Eng Med Biol Soc 2015: 1476 –1479, 2015. 338. Harrell FE, Lee KL, Mark DB. Tutorial in biostatistics multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics Med 15: 361–387, 1996. 339. Harris KD. Neural signatures of cell assembly organization. Nature Rev Neurosci 6: 399 – 407, 2005. 340. Harrison RR, Kier RJ, Chestek CA, Gilja V, Nuyujukian P, Ryu S, Greger B, Solzbacher F, Shenoy KV. Wireless neural recording with single low-power integrated circuit. IEEE Trans Neural Syst Rehabil Eng 17: 322–329, 2009. 341. Hartmann K, Thomson EE, Zea I, Yun R, Mullen P, Canarick J, Huh A, Nicolelis MA. Embedding a panoramic representation of infrared light in the adult rat somatosensory cortex through a sensory neuroprosthesis. J Neurosci 36: 2406 –2424, 2016. 342. Hasan BA, Gan JQ. Hangman BCI: an unsupervised adaptive self-paced Brain-Computer Interface for playing games. Comput Biol Med 42: 598 – 606, 2012. 343. Hasegawa RP, Hasegawa YT, Segraves MA. Neural mind reading of multi-dimensional decisions by monkey mid-brain activity. Neural Networks 22: 1247–1256, 2009.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
303. Ghazanfar AA, Krupa DJ, Nicolelis MA. Role of cortical feedback in the receptive field structure and nonlinear response properties of somatosensory thalamic neurons. Exp Brain Res 141: 88 –100, 2001.
417
418
BRAIN-MACHINE INTERFACES 344. Hassler C, Guy J, Nietzschmann M, Staiger JF, Stieglitz T. Chronic intracortical implantation of saccharose-coated flexible shaft electrodes into the cortex of rats. In: Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE. New York: IEEE, 2011, p. 644 – 647. 345. Hawkins DM. The problem of overfitting. J Chem Info Comput Sci 44: 1–12, 2004. 346. Haxby JV, Grady CL, Horwitz B, Ungerleider LG, Mishkin M, Carson RE, Herscovitch P, Schapiro MB, Rapoport SI. Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proc Natl Acad Sci USA 88: 1621–1625, 1991.
369. Huan NJ, Palaniappan R. Neural network classification of autoregressive features from electroencephalogram signals for brain-computer interface design. J Neural Eng 1: 142–150, 2004. 370. Huang D, Lin P, Fei DY, Chen X, Bai O. Decoding human motor activity from EEG single trials for a discrete two-dimensional cursor control. J Neural Eng 6: 046005, 2009. 371. Huang J, Yu C, Wang Y, Zhao Y, Liu S, Mo C, Liu J, Zhang L, Shi Y. FOCUS: enhancing children’s engagement in reading by using contextual BCI training sessions. In: Proceedings of the 32nd annual ACM conference on Human factors in Computing Systems, 2014, p. 1905142–1908.
347. Haykin SS. Adaptive Filter Theory. Upper Saddle River, NJ: Pearson, 2014, p. xvii. 348. Haynes JD, Rees G. Decoding mental states from brain activity in humans. Nature Rev Neurosci 7: 523–534, 2006.
372. Hubel DH. Tungsten microelectrode for recording from single units. Science 125: 549 –550, 1957. 373. Huberdeau D, Walker H, Huang H, Montgomery E, Sarma SV. Analysis of local field potential signals: a systems approach. Conf Proc IEEE Eng Med Biol Soc 2011: 814 – 817, 2011.
350. He BD, Ebrahimi M, Palafox L, Srinivasan L. Signal quality of endovascular electroencephalography. J Neural Eng 13: 016016, 2016.
374. Humphrey DR, Schmidt EM, Thompson WD. Predicting measures of motor performance from multiple cortical spike trains. Science 170: 758 –762, 1970.
351. Head H, Holmes G. Sensory disturbances from cerebral lesions. Brain 34: 102–254, 1911.
375. Hwang HJ, Lim JH, Kim DW, Im CH. Evaluation of various mental task combinations for near-infrared spectroscopy-based brain-computer interfaces. J Biomed Optics 19: 077005, 2014.
352. Hebb DO. The Organization of Behavior: A Neuropsychological Theory. Mahwah, NJ: Erlbaum, 2002. 353. Helmchen F, Fee MS, Tank DW, Denk W. A miniature head-mounted two-photon microscope: high-resolution brain imaging in freely moving animals. Neuron 31: 903– 912, 2001. 354. Hensel H, Boman KK. Afferent impulses in cutaneous sensory nerves in human subjects. J Neurophysiol 23: 564 –578, 1960. 355. Herman P, Prasad G, McGinnity TM. Investigation of the type-2 fuzzy logic approach to classification in an EEG-based brain-computer interface. Conf Proc IEEE Eng Med Biol Soc 5: 5354 –5357, 2005. 356. Hikosaka O, Nakamura K, Sakai K, Nakahara H. Central mechanisms of motor skill learning. Curr Opin Neurobiol 12: 217–222, 2002. 357. Hill NJ, Gupta D, Brunner P, Gunduz A, Adamo MA, Ritaccio A, Schalk G. Recording human electrocorticographic (ECoG) signals for neuroscientific research and realtime functional cortical mapping. J Vis Exp pii: 3553, 2012. 358. Hinterberger T, Kubler A, Kaiser J, Neumann N, Birbaumer N. A brain-computer interface (BCI) for the locked-in: comparison of different EEG classifications for the thought translation device. Clin Neurophysiol 114: 416 – 425, 2003. 359. Hiraiwa A, Shimohara K, Tokunaga Y. EEG topography recognition by neural networks. IEEE Eng Med Biol Mag 9: 39 – 42, 1990. 360. Hochberg LR, Bacher D, Jarosiewicz B, Masse NY, Simeral JD, Vogel J, Haddadin S, Liu J, Cash SS, van der Smagt P, Donoghue JP. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485: 372–375, 2012. 361. Hochberg LR, Donoghue JP. Sensors for brain-computer interfaces. IEEE Eng Med Biol Mag 25: 32–38, 2006.
376. Ifft PJ, Lebedev MA, Nicolelis MA. Reprogramming movements: extraction of motor intentions from cortical ensemble activity when movement goals change. Front Neuroeng 5: 16, 2012. 377. Ifft PJ, Shokur S, Li Z, Lebedev MA, Nicolelis MA. A brain-machine interface enables bimanual arm movements in monkeys. Sci Transl Med 5: 210ra154, 2013. 378. Iriki A, Tanaka M, Iwamura Y. Coding of modified body schema during tool use by macaque postcentral neurones. Neuroreport 7: 2325–2330, 1996. 379. Iturrate I, Montesano L, Minguez J. Shared-control brain-computer interface for a two dimensional reaching task using EEG error-related potentials. Conf Proc IEEE Eng Med Biol Soc 2013: 5258 –5262, 2013. 380. Ivanova I, Pfurtscheller G, Andrew C. AI-based classification of single-trial EEG data. In: Engineering in Medicine and Biology Society, 1995, IEEE 17th Annual Conference. New York: IEEE, 1995, p. 703–704. 381. Jackson A, Baker SN, Fetz EE. Tests for presynaptic modulation of corticospinal terminals from peripheral afferents and pyramidal tract in the macaque. J Physiol 573: 107–120, 2006. 382. Jackson A, Mavoori J, Fetz EE. Long-term motor cortex plasticity induced by an electronic neural implant. Nature 444: 56 – 60, 2006. 383. Jaroch DB, Ward MP, Chow EY, Rickus JL, Irazoqui PP. Magnetic insertion system for flexible electrode implantation. J Neurosci Methods 183: 213–222, 2009. 384. Jezernik S, Colombo G, Keller T, Frueh H, Morari M. Robotic orthosis lokomat: A rehabilitation and research tool. Neuromodulation: Technol Neural Interface 6: 108 – 115, 2003. 385. Jezernik S, Wassink RG, Keller T. Sliding mode closed-loop control of FES controlling the shank movement. IEEE Trans Biomed Eng 51: 263–272, 2004.
362. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, Branner A, Chen D, Penn RD, Donoghue JP. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442: 164 –171, 2006.
386. Jia C, Gao X, Hong B, Gao S. Frequency and phase mixed coding in SSVEP-based brain– computer interface. IEEE Trans Biomed Eng 58: 200 –206, 2011.
363. Hoffmann U, Vesin JM, Ebrahimi T, Diserens K. An efficient P300-based brain-computer interface for disabled subjects. J Neurosci Methods 167: 115–125, 2008.
387. Jiang J, Zhou Z, Yin E, Yu Y, Hu D. Hybrid Brain-Computer Interface (BCI) based on the EEG and EOG signals. Biomed Mater Eng 24: 2919 –2925, 2014.
364. Holz EM, Botrel L, Kaufmann T, Kubler A. Long-term independent brain-computer interface home use improves quality of life of a patient in the locked-in state: a case study. Arch Phys Med Rehabil 96: S16 –26, 2015.
388. Jobsis FF. Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science 198: 1264 –1267, 1977.
365. Hornyak T. Thinking of child’s play. Sci Am 295: 30, 2006. 366. Hoshi Y. Functional near-infrared spectroscopy: current status and future prospects. J Biomed Optics 12: 062106 – 062109, 2007.
389. Jog M, Connolly C, Kubota Y, Iyengar D, Garrido L, Harlan R, Graybiel A. Tetrode technology: advances in implantable hardware, neuroimaging, and data analysis techniques. J Neurosci Methods 117: 141–152, 2002. 390. Johnson LA, Fuglevand AJ. Mimicking muscle activity with electrical stimulation. J Neural Eng 8: 016009, 2011.
367. House WH. Cochlear implants. Ann Otol Rhinol Laryngol 85: 3–91, 1976. 368. Houweling AR, Brecht M. Behavioural report of single neuron stimulation in somatosensory cortex. Nature 451: 65– 68, 2008.
391. Jorfi M, Skousen JL, Weder C, Capadona JR. Progress towards biocompatible intracortical microelectrodes for neural interfacing applications. J Neural Eng 12: 011001, 2014.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
825
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
349. Haynes JD, Sakai K, Rees G, Gilbert S, Frith C, Passingham RE. Reading hidden intentions in the human brain. Curr Biol 17: 323–328, 2007.
LEBEDEV AND NICOLELIS 392. Julier SJ, Uhlmann JK. New extension of the Kalman filter to nonlinear systems. In: AeroSense’97 International Society for Optics and Photonics, 1997, p. 182141–193.
417. Kelly SP, Lalor E, Finucane C, Reilly RB. A comparison of covert and overt attention as a control option in a steady-state visual evoked potential-based brain computer interface. Conf Proc IEEE Eng Med Biol Soc 7: 4725– 4728, 2004. 418. Kennedy PR. The cone electrode: a long-term electrode that records from neurites grown onto its recording surface. J Neurosci Methods 29: 181–193, 1989.
394. Kaas JH, Nelson RJ, Sur M, Lin CS, Merzenich MM. Multiple representations of the body within the primary somatosensory cortex of primates. Science 204: 521–523, 1979.
419. Kennedy PR, Bakay RA, Moore MM, Adams K, Goldwaithe J. Direct control of a computer from the human central nervous system. IEEE Trans Rehabil Eng 8: 198 – 202, 2000.
395. Kaczmarek P, Salomon P. Towards SSVEP-based, portable, responsive Brain-Computer Interface. Conf Proc IEEE Eng Med Biol Soc 2015: 1095–1098, 2015.
420. Kennedy PR, Bakay RA, Sharpe SM. Behavioral correlates of action potentials recorded chronically inside the Cone Electrode. Neuroreport 3: 605– 608, 1992.
396. Kaji R. Basal ganglia as a sensory gating devise for motor control. J Med Invest 48: 142–146, 2001. 397. Kaji R, Murase N. Sensory function of basal ganglia. Mov Disord 16: 593–594, 2001.
421. Kennedy PR, Kirby MT, Moore MM, King B, Mallory A. Computer control using human intracortical local field potentials. IEEE Trans Neural Syst Rehabil Eng 12: 339 – 344, 2004.
398. Kajikawa Y, Schroeder CE. How local is the local field potential? Neuron 72: 847– 858, 2011.
422. Kennedy PR, Mirra SS, Bakay RA. The cone electrode: ultrastructural studies following long-term recording in rat and monkey cortex. Neurosci Lett 142: 89 –94, 1992.
399. Kalaska JF. From intention to action: motor cortex and the control of reaching movements. Adv Exp Med Biol 629: 139 –178, 2009.
423. Kettner RE, Schwartz AB, Georgopoulos AP. Primate motor cortex and free arm movements to visual targets in three-dimensional space. III. Positional gradients and population coding of movement direction from various movement origins. J Neurosci 8: 2938 –2947, 1988.
401. Kalaska JF, Scott SH, Cisek P, Sergio LE. Cortical control of reaching movements. Curr Opin Neurobiol 7: 849 – 859, 1997. 402. Kalcher J, Flotzinger D, Pfurtscheller G. A new approach to a brain-computer-interface (BCI) based on learning vector quantization (LVQ3). In: Engineering in Medicine and Biology Society, 14th Annual International Conference of the IEEE. New York: IEEE, 1992, p. 1658 –1659. 403. Kalman RE. A new approach to linear filtering and prediction problems. J Basic Eng 82: 35– 45, 1960. 404. Kalman RE, Bucy RS. New results in linear filtering and prediction theory. J Basic Eng 83: 95–108, 1961. 405. Kamiya J. Biofeedback training in voluntary control of EEG alpha rhythms. Calif Med 115: 44, 1971. 406. Kaplan AY, Shishkin SL, Ganin IP, Basyul IA, Zhigalov AY. Adapting the P300-based brain-computer interface for gaming: a review. IEEE Trans Computat Intell 5: 141–149, 2013. 407. Kaplan BJ. Biofeedback in epileptics: equivocal relationship of reinforced EEG frequency to seizure reduction. Epilepsia 16: 477– 485, 1975. 408. Karklinsky M, Flash T. Timing of continuous motor imagery: the two-thirds power law originates in trajectory planning. J Neurophysiol 113: 2490 –2499, 2015. 409. Karumbaiah L, Saxena T, Carlson D, Patil K, Patkar R, Gaupp EA, Betancur M, Stanley GB, Carin L, Bellamkonda RV. Relationship between intracortical electrode design and chronic recording function. Biomaterials 34: 8061– 8074, 2013. 410. Kathner I, Ruf CA, Pasqualotto E, Braun C, Birbaumer N, Halder S. A portable auditory P300 brain-computer interface with directional cues. Clin Neurophysiol 124: 327–338, 2013. 411. Kaufmann T, Herweg A, Kubler A. Toward brain-computer interface based wheelchair control utilizing tactually-evoked event-related potentials. J Neuroeng Rehabil 11: 7, 2014. 412. Kauhanen L, Jylanki P, Lehtonen J, Rantanen P, Alaranta H, Sams M. EEG-based brain-computer interface for tetraplegics. Comput Intell Neurosci 2007: 23864, 2007. 413. Kauhanen L, Nykopp T, Lehtonen J, Jylanki P, Heikkonen J, Rantanen P, Alaranta H, Sams M. EEG and MEG brain-computer interface for tetraplegic patients. IEEE Trans Neural Syst Rehabil Eng 14: 190 –193, 2006. 414. Kawato M. Brain controlled robots. HFSP J 2: 136 –141, 2008. 415. Kawato M. Internal models for motor control and trajectory planning. Curr Opin Neurobiol 9: 718 –727, 1999. 416. Keefer EW, Botterman BR, Romero MI, Rossi AF, Gross GW. Carbon nanotube coating improves neuronal recordings. Nature Nanotechnol 3: 434 – 439, 2008.
826
424. Khan MJ, Hong MJ, Hong KS. Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface. Front Hum Neurosci 8: 244, 2014. 425. Kilicarslan A, Prasad S, Grossman RG, ContreVras-idal JL. High accuracy decoding of user intentions using EEG to control a lower-body exoskeleton. In: 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New York: IEEE, 2013 p. 5606 –5609. 426. Kim BH, Kim M, Jo S. Quadcopter flight control using a low-cost hybrid interface with EEG-based classification and eye tracking. Comput Biol Med 51: 82–92, 2014. 427. Kim HK, Biggs SJ, Schloerb DW, Carmena JM, Lebedev MA, Nicolelis MA, Srinivasan MA. Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Trans Biomed Eng 53: 1164 –1173, 2006. 428. Kim SP, Rao YN, Erdogmus D, Sanchez JC, Nicolelis MA, Principe JC. Determining patterns in neural activity for reaching movements using nonnegative matrix factorization. EURASIP J Appl Signal Processing 2005: 3113–3121, 2005. 429. Kim S, Bhandari R, Klein M, Negi S, Rieth L, Tathireddy P, Toepper M, Oppermann H, Solzbacher F. Integrated wireless neural interface based on the Utah electrode array. Biomed Microdevices 11: 453– 466, 2009. 430. Kim S, Birbaumer N. Real-time functional MRI neurofeedback: a tool for psychiatry. Curr Opin Psychiatry 27: 332–336, 2014. 431. Kim SP, Sanchez JC, Erdogmus D, Rao YN, Wessberg J, Principe JC, Nicolelis M. Divide-and-conquer approach for brain machine interfaces: nonlinear mixture of competitive linear models. Neural Netw 16: 865– 871, 2003. 432. Kim SP, Sanchez JC, Rao YN, Erdogmus D, Carmena JM, Lebedev MA, Nicolelis MA, Principe JC. A comparison of optimal MIMO linear and nonlinear models for brainmachine interfaces. J Neural Eng 3: 145–161, 2006. 433. Kim W, Ng JK, Kunitake ME, Conklin BR, Yang P. Interfacing silicon nanowires with mammalian cells. J Am Chem Soc 129: 7228 –7229, 2007. 434. King CE, Dave KR, Wang PT, Mizuta M, Reinkensmeyer DJ, Do AH, Moromugi S, Nenadic Z. Performance assessment of a brain-computer interface driven hand orthosis. Ann Biomed Eng 42: 2095–2105, 2014. 435. King CE, Wang PT, Chui LA, Do AH, Nenadic Z. Operation of a brain-computer interface walking simulator for individuals with spinal cord injury. J Neuroeng Rehabil 10: 77, 2013. 436. King CE, Wang PT, Mizuta M, Reinkensmeyer DJ, Do AH, Moromugi S, Nenadic Z. Noninvasive brain-computer interface driven hand orthosis. Conf Proc IEEE Eng Med Biol Soc 2011: 5786 –5789, 2011. 437. Klaes C, Shi Y, Kellis S, Minxha J, Revechkis B, Andersen RA. A cognitive neuroprosthetic that uses cortical stimulation for somatosensory feedback. J Neural Eng 11: 056024, 2014.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
393. Jurkiewicz MT, Mikulis DJ, McIlroy WE, Fehlings MG, Verrier MC. Sensorimotor cortical plasticity during recovery following spinal cord injury: a longitudinal fMRI study. Neurorehab Neural Repair 21: 527–538, 2007.
400. Kalaska JF. Parietal cortex area 5 and visuomotor behavior. Can J Physiol Pharmacol 74: 483– 498, 1996.
419
420
BRAIN-MACHINE INTERFACES 438. Kleih SC, Herweg A, Kaufmann T, Staiger-Salzer P, Gerstner N, Kubler A. The WINspeller: a new intuitive auditory brain-computer interface spelling application. Front Neurosci 9: 346, 2015. 439. Kleim JA, Barbay S, Nudo RJ. Functional reorganization of the rat motor cortex following motor skill learning. J Neurophysiol 80: 3321–3325, 1998. 440. Klobassa DS, Vaughan TM, Brunner P, Schwartz NE, Wolpaw JR, Neuper C, Sellers EW. Toward a high-throughput auditory P300-based brain-computer interface. Clin Neurophysiol 120: 1252–1261, 2009.
459. Lalazar H, Vaadia E. Neural basis of sensorimotor learning: modifying internal models. Curr Opin Neurobiol 18: 573–581, 2008. 460. Laubach M, Wessberg J, Nicolelis MA. Cortical ensemble activity increasingly predicts behaviour outcomes during learning of a motor task. Nature 405: 567–571, 2000. 461. Lawhern V, Wu W, Hatsopoulos N, Paninski L. Population decoding of motor cortical activity using a generalized linear model with hidden states. J Neurosci Methods 189: 267–280, 2010. 462. Lebedev MA. How to read neuron-dropping curves? Front Syst Neurosci 8: 102, 2014. 463. Lebedev MA, Carmena JM, O’Doherty JE, Zacksenhouse M, Henriquez CS, Principe JC, Nicolelis MA. Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface. J Neurosci 25: 4681– 4693, 2005.
442. Koralek AC, Jin X, Long JD, 2nd Costa RM, Carmena JM. Corticostriatal plasticity is necessary for learning intentional neuroprosthetic skills. Nature 483: 331–335, 2012.
464. Lebedev MA, Denton JM, Nelson RJ. Vibration-entrained and premovement activity in monkey primary somatosensory cortex. J Neurophysiol 72: 1654 –1673, 1994.
443. Kosaka T, Hama K. Gap junctions between non-pyramidal cell dendrites in the rat hippocampus (CA1 and CA3 regions): a combined Golgi-electron microscopy study. J Comp Neurol 231: 150 –161, 1985.
465. Lebedev MA, Messinger A, Kralik JD, Wise SP. Representation of attended versus remembered locations in prefrontal cortex. PLoS Biol 2: e365, 2004.
444. Kostov A, Polak M. Parallel man-machine training in development of EEG-based cursor control. IEEE Trans Rehab Eng 8: 203–205, 2000. 445. Kostov A, Polak M. Prospects of computer access using voluntary modulated EEG signal. In: Proc ECPD Symposium on Brain & Consciousness Belgrade, Yugoslavia, 1997, p. 233203–236. 446. Kotov NA, Winter JO, Clements IP, Jan E, Timko BP, Campidelli S, Pathak S, Mazzatenta A, Lieber CM, Prato M. Nanomaterials for neural interfaces. Adv Materials 21: 3970 – 4004, 2009. 447. Kotz SA, Schwartze M, Schmidt-Kassow M. Non-motor basal ganglia functions: a review and proposal for a model of sensory predictability in auditory language perception. Cortex 45: 982–990, 2009. 448. Kowalski KC, He BD, Srinivasan L. Dynamic analysis of naive adaptive brain-machine interfaces. Neural Comput 25: 2373–2420, 2013. 449. Koyama S, Chase SM, Whitford AS, Velliste M, Schwartz AB, Kass RE. Comparison of brain-computer interface decoding algorithms in open-loop and closed-loop control. J Comput Neurosci 29: 73– 87, 2010. 450. Koyama S, Eden UT, Brown EN, Kass RE. Bayesian decoding of neural spike trains. Ann Inst Statistical Math 62: 37–59, 2010. 451. Kozai TDY, Kipke DR. Insertion shuttle with carboxyl terminated self-assembled monolayer coatings for implanting flexible polymer neural probes in the brain. J Neurosci Methods 184: 199 –205, 2009. 452. Kralik JD, Dimitrov DF, Krupa DJ, Katz DB, Cohen D, Nicolelis MA. Techniques for long-term multisite neuronal ensemble recordings in behaving animals. Methods 25: 121–150, 2001. 453. Krüger J, Caruana F, Rizzolatti G. Seven years of recording from monkey cortex with a chronically implanted multiple microelectrode. Front Neuroeng 3: 6, 2010. 454. Krupa DJ, Wiest MC, Shuler MG, Laubach M, Nicolelis MA. Layer-specific somatosensory cortical activation during active tactile discrimination. Science 304: 1989 – 1992, 2004. 455. Kubler A, Furdea A, Halder S, Hammer EM, Nijboer F, Kotchoubey B. A braincomputer interface controlled auditory event-related potential (p300) spelling system for locked-in patients. Ann NY Acad Sci 1157: 90 –100, 2009. 456. Kwak NS, Muller KR, Lee SW. A lower limb exoskeleton control system based on steady state visual evoked potentials. J Neural Eng 12: 056009, 2015. 457. LaFleur K, Cassady K, Doud A, Shades K, Rogin E, He B. Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain-computer interface. J Neural Eng 10: 046003, 2013. 458. Lal TN, Schröder M, Hill NJ, Preissl H, Hinterberger T, Mellinger J, Bogdan M, Rosenstiel W, Hofmann T, Birbaumer N. A brain computer interface with online feedback based on magnetoencephalography. In: Proceedings of the 22nd International Conference on Machine Learning 2005, p. 465– 472.
466. Lebedev MA, Nicolelis MA. Brain-machine interfaces: past, present and future. Trends Neurosci 29: 536 –546, 2006. 467. Lebedev MA, Nicolelis MA. Toward a whole-body neuroprosthetic. Prog Brain Res 194: 47– 60, 2011. 468. Lebedev MA, O’Doherty JE, Nicolelis MA. Decoding of temporal intervals from cortical ensemble activity. J Neurophysiol 99: 166 –186, 2008. 469. Lebedev MA, Tate AJ, Hanson TL, Li Z, O’Doherty JE, Winans JA, Ifft PJ, Zhuang KZ, Fitzsimmons NA, Schwarz DA, Fuller AM, An JH, Nicolelis MA. Future developments in brain-machine interface research. Clinics 66 Suppl 1: 25–32, 2011. 470. Lebedev MA, Wise SP. Insights into seeing and grasping: distinguishing the neural correlates of perception and action. Behav Cogn Neurosci Rev 1: 108 –129, 2002. 471. Lebedev MA, Wise SP. Tuning for the orientation of spatial attention in dorsal premotor cortex. Eur J Neurosci 13: 1002–1008, 2001. 472. Lee JH, Ryu J, Jolesz FA, Cho ZH, Yoo SS. Brain-machine interface via real-time fMRI: preliminary study on thought-controlled robotic arm. Neurosci Lett 450: 1– 6, 2009. 473. Lee PL, Sie JJ, Liu YJ, Wu CH, Lee MH, Shu CH, Li PH, Sun CW, Shyu KK. An SSVEP-actuated brain computer interface using phase-tagged flickering sequences: a cursor system. Ann Biomed Eng 38: 2383–2397, 2010. 474. Lee SW, Fallegger F, Casse BDF, Fried SI. Implantable microcoils for intracortical magnetic stimulation. Sc Adv 2: e1600889, 2016. 475. Lee W, Kim H, Jung Y, Song IU, Chung YA, Yoo SS. Image-guided transcranial focused ultrasound stimulates human primary somatosensory cortex. Sci Rep 5: 8743, 2015. 476. Leeb R, Sagha H, Chavarriaga R, Millan Jdel R. A hybrid brain-computer interface based on the fusion of electroencephalographic and electromyographic activities. J Neural Eng 8: 025011, 2011. 477. Legendy C. On the scheme by which the human brain stores information. Math Biosci 1: 555–597, 1967. 478. Lehew G, Nicolelis MAL. State-of-the-art microwire array design for chronic neural recordings in behaving animals. In: Methods for Neural Ensemble Recordings, edited by Nicolelis MAL. Boca Raton, FL: CRC, 2008. 479. Lenz F, Seike M, Richardson R, Lin Y, Baker F, Khoja I, Jaeger C, Gracely RH. Thermal and pain sensations evoked by microstimulation in the area of human ventrocaudal nucleus. J Neurophysiol 70: 200 –212, 1993. 480. Leonardo A. Degenerate coding in neural systems. J Comp Physiol A Neuroethol Sens Neural Behav Physiol 191: 995–1010, 2005. 481. Lesenfants D, Habbal D, Lugo Z, Lebeau M, Horki P, Amico E, Pokorny C, Gomez F, Soddu A, Muller-Putz G, Laureys S, Noirhomme Q. An independent SSVEP-based brain-computer interface in locked-in syndrome. J Neural Eng 11: 035002, 2014. 482. Leuthardt EC, Gaona C, Sharma M, Szrama N, Roland J, Freudenberg Z, Solis J, Breshears J, Schalk G. Using the electrocorticographic speech network to control a brain-computer interface in humans. J Neural Eng 8: 036004, 2011.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
827
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
441. Köhler P, Linsmeier CE, Thelin J, Bengtsson M, Jörntell H, Garwicz M, Schouenborg J, Wallman L. Flexible multi electrode brain-machine interface for recording in the cerebellum. In: Engineering in Medicine and Biology Society 2009 Annual International Conference of the IEEE, p. 536 –538.
LEBEDEV AND NICOLELIS 483. Leuthardt EC, Miller KJ, Schalk G, Rao RP, Ojemann JG. Electrocorticography-based brain computer interface–the Seattle experience. IEEE Trans Neural Syst Rehabil Eng 14: 194 –198, 2006. 484. Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW. A brain-computer interface using electrocorticographic signals in humans. J Neural Eng 1: 63–71, 2004. 485. Levinson N. The Wiener (root mean square) error criterion in filter design and prediction. J Math Physics 25: 261–278, 1946. 486. Lewis TJ, Rinzel J. Dynamics of spiking neurons connected by both inhibitory and electrical coupling. J Comput Neurosci 14: 283–309, 2003. 487. Li CL, Jasper H. Microelectrode studies of the electrical activity of the cerebral cortex in the cat. J Physiol 121: 117–140, 1953. 488. Li G, Zhang D. Brain-computer interface controlled cyborg: establishing a functional information transfer pathway from human brain to cockroach brain. PloS One 11: e0150667, 2016.
490. Li Z. Decoding methods for neural prostheses: where have we reached? Front Syst Neurosci 8: 129, 2014. 491. Li Z, O’Doherty JE, Hanson TL, Lebedev MA, Henriquez CS, Nicolelis MA. Unscented Kalman filter for brain-machine interfaces. PLoS One 4: e6243, 2009.
506. Llinas R, Baker R, Sotelo C. Electrotonic coupling between neurons in cat inferior olive. J Neurophysiol 37: 560 –571, 1974. 507. Llinas RR, Walton KD, Nakao M, Hunter I, Anquetil PA. Neuro-vascular central nervous recording/stimulating system: Using nanotechnology probes. J Nanoparticle Res 7: 111–127, 2005. 508. Loeb GE, Walker AE, Uematsu S, Konigsmark BW. Histological reaction to various conductive and dielectric films chronically implanted in the subdural space. J Biomed Mater Res 11: 195–210, 1977. 509. Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A. Neurophysiological investigation of the basis of the fMRI signal. Nature 412: 150 –157, 2001. 510. Loizou PC. Introduction to cochlear implants. IEEE Eng Med Biol Mag 18: 32– 42, 1999. 511. Long J, Li Y, Wang H, Yu T, Pan J, Li F. A hybrid brain computer interface to control the direction and speed of a simulated or real wheelchair. IEEE Trans Neural Syst Rehabil Eng 20: 720 –729, 2012. 512. Lubar JF. Neurofeedback for the management of attention-deficit/hyperactivity disorders. In: Biofeedback: A Practitioner’s Guide, edited by Schwartz MS. New York: Guildford, 1995. 513. Lucas TH, Fetz EE. Myo-cortical crossed feedback reorganizes primate motor cortex output. J Neurosci 33: 5261–5274, 2013.
492. Li Z, O’Doherty JE, Lebedev MA, Nicolelis MA. Adaptive decoding for brain-machine interfaces through Bayesian parameter updates. Neural Comput 23: 3162–3204, 2011.
514. Lugo ZR, Rodriguez J, Lechner A, Ortner R, Gantner IS, Laureys S, Noirhomme Q, Guger C. A vibrotactile p300-based brain-computer interface for consciousness detection and communication. Clin EEG Neurosci 45: 14 –21, 2014.
493. Liao LD, Chen CY, Wang IJ, Chen SF, Li SY, Chen BW, Chang JY, Lin CT. Gaming control using a wearable and wireless EEG-based brain-computer interface device with novel dry foam-based sensors. J Neuroeng Rehabil 9: 5, 2012.
515. Luo A, Sullivan TJ. A user-friendly SSVEP-based brain-computer interface using a time-domain classifier. J Neural Eng 7: 26010, 2010.
494. Libedinsky C, So R, Xu Z, Kyar TK, Ho D, Lim C, Chan L, Chua Y, Yao L, Cheong JH. Independent mobility achieved through a wireless brain-machine interface. PloS One 11: e0165773, 2016.
516. Lütcke H, Murayama M, Hahn T, Margolis DJ, Astori S, Meyer S, Göbel W, Yang Y, Tang W, Kügler S. Optical recording of neuronal activity with a genetically-encoded calcium indicator in anesthetized and freely moving mice. Front Neural Circuits 4: 9, 2010.
495. Libet B, Alberts WW, Wright EW Jr, Delattre LD, Levin G, Feinstein B. Production of threshold levels of conscious sensation by electrical stimulation of human somatosensory cortex. J Neurophysiol 27: 546 –578, 1964.
517. Luu S, Chau T. Decoding subjective preference from single-trial near-infrared spectroscopy signals. J Neural Eng 6: 016003, 2008.
496. Lilly JC. Correlations between neurophysiological activity in the cortex and shortterm behavior in the monkey. Biol Biochem Bases Behav 83–100, 1958. 497. Lilly JC. Distribution of motor functions in the cerebral cortex in the conscious intact monkey. Science 124: 937, 1956.
518. Madan CR. Augmented memory: a survey of the approaches to remembering more. Front Syst Neurosci 8: 30, 2014. 519. Maguire GQ, McGee EM. Implantable brain chips? Time for debate. Hastings Center Report 29: 7–13, 1999.
498. Lilly JC. Electrode and cannulae implantation in the brain by a simple percutaneous method. Science 127: 1181–1182, 1958.
520. Mahmoudi B, Pohlmeyer EA, Prins NW, Geng S, Sanchez JC. Towards autonomous neuroprosthetic control using Hebbian reinforcement learning. J Neural Eng 10: 066005, 2013.
499. Lilly JC. Instantaneous relations between the activities of closely spaced zones on the cerebral cortex; electrical figures during responses and spontaneous activity. Am J Physiol 176: 493–504, 1954.
521. Mak JN, Arbel Y, Minett JW, McCane LM, Yuksel B, Ryan D, Thompson D, Bianchi L, Erdogmus D. Optimizing the P300-based brain-computer interface: current status, limitations and future directions. J Neural Eng 8: 025003, 2011.
500. Lilly JC. Mental effects of reduction of ordinary levels of physical stimuli on intact, healthy persons. Psychiatr Res Rep Am Psychiatr Assoc 5: 1–28, 1956.
522. Mak JN, Wolpaw JR. Clinical applications of brain-computer interfaces: current state and future prospects. IEEE Rev Biomed Eng 2: 187–199, 2009.
501. Lin CT, Chen YC, Huang TY, Chiu TT, Ko LW, Liang SF, Hsieh HY, Hsu SH, Duann JR. Development of wireless brain computer interface with embedded multitask scheduling and its application on real-time driver’s drowsiness detection and warning. IEEE Trans Biomed Eng 55: 1582–1591, 2008.
523. Mank M, Griesbeck O. Genetically encoded calcium indicators. Chem Rev 108: 1550 – 1564, 2008.
502. Lin YP, Wang Y, Jung TP. A mobile SSVEP-based brain-computer interface for freely moving humans: the robustness of canonical correlation analysis to motion artifacts. Conf Proc IEEE Eng Med Biol Soc 2013: 1350 –1353, 2013.
525. Maravita A, Spence C, Driver J. Multisensory integration and the body schema: close to hand and within reach. Curr Biol 13: R531–R539, 2003.
524. Maravita A, Iriki A. Tools for the body (schema). Trends Cogn Sci 8: 79 – 86, 2004.
503. Ling G, Gerard RW. The normal membrane potential of frog sartorius fibers. J Cell Physiol 34: 383–396, 1949.
526. Marchesi M, Riccò B. BRAVO: a brain virtual operator for education exploiting braincomputer interfaces. In: CHI’13 Extended Abstracts on Human Factors in Computing Systems. New York: ACM, 2013, p. 3091–3094.
504. Liu NH, Chiang CY, Hsu HM. Improving driver alertness through music selection using a mobile EEG to detect brainwaves. Sensors 13: 8199 – 8221, 2013.
527. Maren S, Phan KL, Liberzon I. The contextual brain: implications for fear conditioning, extinction and psychopathology. Nat Rev Neurosci 14: 417– 428, 2013.
505. Liu Y, Denton JM, Nelson RJ. Neuronal activity in primary motor cortex differs when monkeys perform somatosensory and visually guided wrist movements. Exp Brain Res 167: 571–586, 2005.
528. Markus Z, Eordegh G, Paroczy Z, Benedek G, Nagy A. Modality distribution of sensory neurons in the feline caudate nucleus and the substantia nigra. Acta Biol Hung 59: 269 –279, 2008.
828
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
489. Li J, Liu Y, Lu Z, Zhang L. A competitive brain computer interface: multi-person car racing system. Conf Proc IEEE Eng Med Biol Soc 2013: 2200 –2203, 2013.
421
422
BRAIN-MACHINE INTERFACES 529. Marsh BT, Tarigoppula VSA, Chen C, Francis JT. Toward an autonomous brain machine interface: integrating sensorimotor reward modulation and reinforcement learning. J Neurosci 35: 7374 –7387, 2015.
551. Mokienko OA, Chervyakov AV, Kulikova SN, Bobrov PD, Chernikova LA, Frolov AA, Piradov MA. Increased motor cortex excitability during motor imagery in braincomputer interface trained subjects. Front Comput Neurosci 7: 168, 2013.
530. Martens S, Bensch M, Halder S, Hill J, Nijboer F, Ramos-Murguialday A, Schoelkopf B, Birbaumer N, Gharabaghi A. Epidural electrocorticography for monitoring of arousal in locked-in state. Front Hum Neurosci 8: 861, 2014.
552. Montijn JS, Vinck M, Pennartz CM. Population coding in mouse visual cortex: response reliability and dissociability of stimulus tuning and noise correlation. Front Comput Neurosci 8: 58, 2014.
531. Martisius I, Damasevicius R. A prototype SSVEP based real time BCI gaming system. Comput Intell Neurosci 2016: 3861425, 2016.
553. Moore MM. Real-world applications for brain-computer interface technology. IEEE Trans Neural Syst Rehabil Eng 11: 162–165, 2003.
532. Mason SG, Bohringer R, Borisoff JF, Birch GE. Real-time control of a video game with a direct brain– computer interface. J Clin Neurophysiol 21: 404 – 408, 2004.
554. Moran DW, Schwartz AB. Motor cortical representation of speed and direction during reaching. J Neurophysiol 82: 2676 –2692, 1999.
533. Matthews F, Pearlmutter BA, Wards TE, Soraghan C, Markham C. Hemodynamics for brain-computer interfaces. IEEE Signal Processing Magazine 25: 87–94, 2008.
555. Morgan S, Hansen J, Hillyard S. Selective attention to stimulus location modulates the steady-state visual evoked potential. Proc Natl Acad Sci USA 93: 4770 – 4774, 1996.
534. Mattia D, Pichiorri F, Molinari M, Rupp R. Brain computer interface for hand motor function restoration and rehabilitation. In: Towards Practical Brain-Computer Interfaces. New York: Springer, 2012, p. 131–153.
556. Moritz CT, Perlmutter SI, Fetz EE. Direct control of paralysed muscles by cortical neurons. Nature 456: 639 – 642, 2008.
536. May T, Ozden I, Brush B, Borton D, Wagner F, Agha N, Sheinberg DL, Nurmikko AV. Detection of optogenetic stimulation in somatosensory cortex by non-human primates-towards artificial tactile sensation. PloS One 9: e114529, 2014. 537. Maynard EM, Nordhausen CT, Normann RA. The Utah intracortical Electrode Array: a recording structure for potential brain-computer interfaces. Electroencephalogr Clin Neurophysiol 102: 228 –239, 1997. 538. McFarland DJ, Sarnacki WA, Townsend G, Vaughan T, Wolpaw JR. The P300-based brain-computer interface (BCI): effects of stimulus rate. Clin Neurophysiol 122: 731– 737, 2011. 539. McGee EM. Bioelectronics and Implanted devices. In: Medical Enhancement and Posthumanity. New York: Springer, 2008, p. 207–224. 540. McPherson JG, Miller RR, Perlmutter SI. Targeted, activity-dependent spinal stimulation produces long-lasting motor recovery in chronic cervical spinal cord injury. Proc Natl Acad Sci USA 112: 12193–12198, 2015. 541. Mellinger J, Schalk G, Braun C, Preissl H, Rosenstiel W, Birbaumer N, Kubler A. An MEG-based brain-computer interface (BCI). Neuroimage 36: 581–593, 2007. 542. Mendoza G, Peyrache A, Gámez J, Prado L, Buzsaki G, Merchant H. Recording extracellular neural activity in the behaving monkey using a semi-chronic and highdensity electrode system. J Neurophysiol 116: 563–574, 2016. 543. Micera S, Navarro X. Bidirectional interfaces with the peripheral nervous system. Int Rev Neurobiol 86: 23–38, 2009. 544. Miller KJ, Shenoy P, Miller JW, Rao RP, Ojemann JG. Real-time functional brain mapping using electrocorticography. Neuroimage 37: 504 –507, 2007. 545. Mirghasemi H, Fazel-Rezai R, Shamsollahi MB. Analysis of p300 classifiers in brain computer interface speller. Conf Proc IEEE Eng Med Biol Soc 1: 6205– 6208, 2006.
557. Morizio J, Irazoqui P, Go V, Parmentier J. Wireless headstage for neural prosthetics. In: Conference Proceedings 2nd International IEEE EMBS Conference on Neural Engineering. New York: IEEE, 2005, p. 414 – 417. 558. Morrow MM, Miller LE. Prediction of muscle activity by populations of sequentially recorded primary motor cortex neurons. J Neurophysiol 89: 2279 –2288, 2003. 559. Movahedi MM, Mehdizadeh A, Alipour A. Development of a Brain Computer Interface (BCI) Speller System Based on SSVEP Signals. J Biomed Phys Eng 3: 81– 86, 2013. 560. Muller-Putz GR, Daly I, Kaiser V. Motor imagery-induced EEG patterns in individuals with spinal cord injury and their impact on brain-computer interface accuracy. J Neural Eng 11: 035011, 2014. 561. Muller-Putz GR, Schreuder M, Tangermann M, Leeb R, Millan Del RJ. The hybrid Brain-Computer Interface: a bridge to assistive technology? Biomed Tech (Berl) 2013, doi: 10.1515/bmt-2013-4435. 562. Mulliken GH, Musallam S, Andersen RA. Decoding trajectories from posterior parietal cortex ensembles. J Neurosci 28: 12913–12926, 2008. 563. Musallam S, Bak MJ, Troyk PR, Andersen RA. A floating metal microelectrode array for chronic implantation. J Neurosci Methods 160: 122–127, 2007. 564. Musallam S, Corneil BD, Greger B, Scherberger H, Andersen RA. Cognitive control signals for neural prosthetics. Science 305: 258 –262, 2004. 565. Mushiake H, Inase M, Tanji J. Neuronal activity in the primate premotor, supplementary, and precentral motor cortex during visually guided and internally determined sequential movements. J Neurophysiol 66: 705–718, 1991. 566. Näätänen R, Michie PT. Early selective-attention effects on the evoked potential: a critical review and reinterpretation. Biol Psychol 8: 81–136, 1979. 567. Najafi K, Wise KD, Mochizuki T. A high-yield IC-compatible multichannel recording array. Electron Devices IEEE Trans 32: 1206 –1211, 1985. 568. Naseer N, Hong KS. fNIRS-based brain-computer interfaces: a review. Front Hum Neurosci 9: 172, 2015.
546. Mirowski M, Reid PR, Mower MM, Watkins L, Gott VL, Schauble JF, Langer A, Heilman M, Kolenik SA, Fischell RE. Termination of malignant ventricular arrhythmias with an implanted automatic defibrillator in human beings. N Engl J Med 303: 322–324, 1980.
569. Naseer N, Hong MJ, Hong KS. Online binary decision decoding using functional near-infrared spectroscopy for the development of brain-computer interface. Exp Brain Res 232: 555–564, 2014.
547. Mitz AR, Godschalk M, Wise SP. Learning-dependent neuronal activity in the premotor cortex: activity during the acquisition of conditional motor associations. J Neurosci 11: 1855–1872, 1991.
570. Naselaris T, Merchant H, Amirikian B, Georgopoulos AP. Spatial reconstruction of trajectories of an array of recording microelectrodes. J Neurophysiol 93: 2318 –2330, 2005.
548. Mizuseki K, Buzsaki G. Theta oscillations decrease spike synchrony in the hippocampus and entorhinal cortex. Philos Trans R Soc Lond B Biol Sci 369: 20120530, 2014.
571. Nelson MJ, Pouget P. Do electrode properties create a problem in interpreting local field potential recordings? J Neurophysiol 103: 2315–2317, 2010.
549. Mohanchandra K, Saha S, Lingaraju G. EEG based brain computer interface for speech communication: principles and applications. In: Brain-Computer Interfaces. New York: Springer, 2015, p. 273–293.
572. Nelson R. Set related and premovement related activity of primate primary somatosensory cortical neurons depends upon stimulus modality and subsequent movement. Brain Res Bull 21: 411– 424, 1988.
550. Mohseni P, Najafi K, Eliades SJ, Wang X. Wireless multichannel biopotential recording using an integrated FM telemetry circuit. IEEE Trans Neural Syst Rehab Eng 13: 263– 271, 2005.
573. Neuper C, Muller GR, Kubler A, Birbaumer N, Pfurtscheller G. Clinical application of an EEG-based brain-computer interface: a case study in a patient with severe motor impairment. Clin Neurophysiol 114: 399 – 409, 2003.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
829
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
535. Mauritz KH, Wise SP. Premotor cortex of the rhesus monkey: neuronal activity in anticipation of predictable environmental events. Exp Brain Res 61: 229 –244, 1986.
LEBEDEV AND NICOLELIS 574. Neves H, Orban G, Koudelka-Hep M, Stieglitz T, Ruther P. Development of modular multifunctional probe arrays for cerebral applications. In: Neural Engineering, CNE’07 3rd International IEEE/EMBS Conference. New York: IEEE, 2007, p. 104 –109. 575. Nicolelis MA. Actions from thoughts. Nature 409: 403– 407, 2001. 576. Nicolelis MA. Beyond maps: a dynamic view of the somatosensory system. Braz J Med Biol Res 29: 401– 412, 1996.
596. Nowlis DP, Wortz EC. Control of the ratio of midline parietal to midline frontal EEG alpha rhythms through auditory feedback. Percept Mot Skills 37: 815– 824, 1973. 597. O’Doherty JE, Lebedev MA, Hanson TL, Fitzsimmons NA, Nicolelis MA. A brainmachine interface instructed by direct intracortical microstimulation. Front Integr Neurosci 3: 20, 2009.
577. Nicolelis MA. Methods for Neural Ensemble Recordings. Boca Raton, FL: CRC, 2007.
598. O’Doherty JE, Lebedev MA, Ifft PJ, Zhuang KZ, Shokur S, Bleuler H, Nicolelis MA. Active tactile exploration using a brain-machine-brain interface. Nature 479: 228 – 231, 2011.
578. Nicolelis MA, Baccala LA, Lin RC, Chapin JK. Sensorimotor encoding by synchronous neural ensemble activity at multiple levels of the somatosensory system. Science 268: 1353–1358, 1995.
599. O’Doherty JE, Lebedev MA, Li Z, Nicolelis MA. Virtual active touch using randomly patterned intracortical microstimulation. IEEE Trans Neural Syst Rehabil Eng 20: 85– 93, 2012.
579. Nicolelis MA, Chapin JK. Spatiotemporal structure of somatosensory responses of many-neuron ensembles in the rat ventral posterior medial nucleus of the thalamus. J Neurosci 14: 3511–3532, 1994.
600. O’Keefe J, Dostrovsky J. The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res 34: 171–175, 1971.
581. Nicolelis MA, Ghazanfar AA, Faggin BM, Votaw S, Oliveira LM. Reconstructing the engram: simultaneous, multisite, many single neuron recordings. Neuron 18: 529 – 537, 1997. 582. Nicolelis MA, Ghazanfar AA, Stambaugh CR, Oliveira LM, Laubach M, Chapin JK, Nelson RJ, Kaas JH. Simultaneous encoding of tactile information by three primate cortical areas. Nat Neurosci 1: 621– 630, 1998. 583. Nicolelis MA, Lebedev MA. Principles of neural ensemble physiology underlying the operation of brain-machine interfaces. Nat Rev Neurosci 10: 530 –540, 2009. 584. Nicolelis MA, Lin RC, Woodward DJ, Chapin JK. Dynamic and distributed properties of many-neuron ensembles in the ventral posterior medial thalamus of awake rats. Proc Natl Acad Sci USA 90: 2212–2216, 1993. 585. Nicolelis MA, Lin RC, Woodward DJ, Chapin JK. Induction of immediate spatiotemporal changes in thalamic networks by peripheral block of ascending cutaneous information. Nature 361: 533–536, 1993. 586. Nicolelis MAL. Beyond Boundaries: The New Neuroscience of Connecting Brains With Machines–And How It Will Change Our Lives. New York: Times Books/Henry Holt, 2011, p. 353 p. 587. Niedermeyer E. Alpha rhythms as physiological and abnormal phenomena. Int J Psychophysiol 26: 31– 49, 1997. 588. Niedermeyer E, Lopes da Silva FH. Electroencephalography Basic Principles, Clinical Applications, and Related Fields. Philadelphia, PA: Lippincott Williams & Wilkins, 2005. 589. Nii Y, Uematsu S, Lesser RP, Gordon B. Does the central sulcus divide motor and sensory functions. Cortical mapping of human hand areas as revealed by electrical stimulation through subdural grid electrodes. Neurology 46: 360 –367, 1996. 590. Nijboer F, Sellers EW, Mellinger J, Jordan MA, Matuz T, Furdea A, Halder S, Mochty U, Krusienski DJ, Vaughan TM, Wolpaw JR, Birbaumer N, Kubler A. A P300-based brain-computer interface for people with amyotrophic lateral sclerosis. Clin Neurophysiol 119: 1909 –1916, 2008. 591. Nikolenko V, Poskanzer KE, Yuste R. Two-photon photostimulation and imaging of neural circuits. Nature Methods 4: 943–950, 2007. 592. Nishimura Y, Perlmutter SI, Eaton RW, Fetz EE. Spike-timing-dependent plasticity in primate corticospinal connections induced during free behavior. Neuron 80: 1301– 1309, 2013. 593. Nordhausen CT, Maynard EM, Normann RA. Single unit recording capabilities of a 100 microelectrode array. Brain Res 726: 129 –140, 1996. 594. Normann RA, Greger B, House P, Romero SF, Pelayo F, Fernandez E. Toward the development of a cortically based visual neuroprosthesis. J Neural Eng 6: 035001, 2009. 595. Nowlis DP, Kamiya J. The control of electroencephalographic alpha rhythms through auditory feedback and the associated mental activity. Psychophysiology 6: 476 – 484, 1970.
830
601. Obeid I, Morizio JC, Moxon KA, Nicolelis MA, Wolf PD. Two multichannel integrated circuits for neural recording and signal processing. IEEE Trans Biomed Eng 50: 255– 258, 2003. 602. Obeid I, Nicolelis MA, Wolf PD. A multichannel telemetry system for single unit neural recordings. J Neurosci Methods 133: 33–38, 2004. 603. Obermaier B, Guger C, Neuper C, Pfurtscheller G. Hidden Markov models for online classification of single trial EEG data. Pattern Recogn Lett 22: 1299 –1309, 2001. 604. Obermaier B, Neuper C, Guger C, Pfurtscheller G. Information transfer rate in a five-classes brain-computer interface. IEEE Trans Neural Syst Rehabil Eng 9: 283–288, 2001. 605. Ohara S, Weiss N, Lenz FA. Microstimulation in the region of the human thalamic principal somatic sensory nucleus evokes sensations like those of mechanical stimulation and movement. J Neurophysiol 91: 736 –745, 2004. 606. Oken BS, Orhan U, Roark B, Erdogmus D, Fowler A, Mooney A, Peters B, Miller M, Fried-Oken MB. Brain-computer interface with language model-electroencephalography fusion for locked-in syndrome. Neurorehabil Neural Repair 28: 387–394, 2014. 607. Oliveira LMO, Dimitrov D. Surgical techniques for chronic implantation of microwire arrays in rodents and primates. In: Methods for Neural Ensemble Recordings, edited by Nicolelis MAL. Boca Raton, FL: CRC, 2008. 608. Orhan U, Erdogmus D, Roark B, Purwar S, Hild KE, 2nd Oken B, Nezamfar H, Fried-Oken M. Fusion with language models improves spelling accuracy for ERPbased brain computer interface spellers. Conf Proc IEEE Eng Med Biol Soc 2011: 5774 – 5777, 2011. 609. Orsborn AL, Dangi S, Moorman HG, Carmena JM. Closed-loop decoder adaptation on intermediate time-scales facilitates rapid BMI performance improvements independent of decoder initialization conditions. IEEE Trans Neural Syst Rehabil Eng 20: 468 – 477, 2012. 610. Ortner R, Allison BZ, Korisek G, Gaggl H, Pfurtscheller G. An SSVEP BCI to control a hand orthosis for persons with tetraplegia. IEEE Trans Neural Systems and Rehab Eng 19: 1–5, 2011. 611. Ortner R, Irimia DC, Scharinger J, Guger C. A motor imagery based brain-computer interface for stroke rehabilitation. Stud Health Technol Inform 181: 319 –323, 2012. 612. Oweiss KG, Badreldin IS. Neuroplasticity subserving the operation of brain-machine interfaces. Neurobiol Dis 83: 161–171, 2015. 613. Oxley TJ, Opie NL, John SE, Rind GS, Ronayne SM, Wheeler TL, Judy JW, McDonald AJ, Dornom A, Lovell TJ, Steward C, Garrett DJ, Moffat BA, Lui EH, Yassi N, Campbell BC, Wong YT, Fox KE, Nurse ES, Bennett IE, Bauquier SH, Liyanage KA, van der Nagel NR, Perucca P, Ahnood A, Gill KP, Yan B, Churilov L, French CR, Desmond PM, Horne MK, Kiers L, Prawer S, Davis SM, Burkitt AN, Mitchell PJ, Grayden DB, May CN, O’Brien TJ. Minimally invasive endovascular stent-electrode array for high-fidelity, chronic recordings of cortical neural activity. Nature Biotechnol 34: 320 –327, 2016. 614. Pais-Vieira M, Chiuffa G, Lebedev M, Yadav A, Nicolelis MA. Building an organic computing device with multiple interconnected brains. Sci Rep 5: 11869, 2015. 615. Pais-Vieira M, Kunicki C, Tseng PH, Martin J, Lebedev M, Nicolelis MA. Cortical and thalamic contributions to response dynamics across layers of the primary somatosensory cortex during tactile discrimination. J Neurophysiol 114: 1652–1676, 2015.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
580. Nicolelis MA, Dimitrov D, Carmena JM, Crist R, Lehew G, Kralik JD, Wise SP. Chronic, multisite, multielectrode recordings in macaque monkeys. Proc Natl Acad Sci USA 100: 11041–11046, 2003.
423
424
BRAIN-MACHINE INTERFACES 616. Pais-Vieira M, Lebedev M, Kunicki C, Wang J, Nicolelis MA. A brain-to-brain interface for real-time sharing of sensorimotor information. Sci Rep 3: 1319, 2013. 617. Pais-Vieira M, Lebedev MA, Wiest MC, Nicolelis MA. Simultaneous top-down modulation of the primary somatosensory cortex and thalamic nuclei during active tactile discrimination. J Neurosci 33: 4076 – 4093, 2013. 618. Pais-Vieira M, Yadav AP, Moreira D, Guggenmos D, Santos A, Lebedev M, Nicolelis MA. A closed loop brain-machine interface for epilepsy control using dorsal column electrical stimulation. Sci Rep 6: 32814, 2016. 619. Pan J, Li Y, Gu Z, Yu Z. A comparison study of two P300 speller paradigms for brain-computer interface. Cogn Neurodyn 7: 523–529, 2013. 620. Patil PG, Carmena JM, Nicolelis MA, Turner DA. Ensemble recordings of human subcortical neurons as a source of motor control signals for a brain-machine interface. Neurosurgery 55: 27–35, 2004. 621. Patrick J, Valeur B, Monnerie L, Changeux JP. Changes in extrinsic fluorescence intensity of the electroplax membrane during electrical excitation. J Membr Biol 5: 102–120, 1971.
639. Placidi G, Petracca A, Spezialetti M, Iacoviello D. Classification strategies for a singletrial binary Brain Computer Interface based on remembering unpleasant odors. Conf Proc IEEE Eng Med Biol Soc 2015: 7019 –7022, 2015. 640. Pohlmeyer EA, Mahmoudi B, Geng S, Prins NW, Sanchez JC. Using reinforcement learning to provide stable brain-machine interface control despite neural input reorganization. PloS One 9: e87253, 2014. 641. Pohlmeyer EA, Oby ER, Perreault EJ, Solla SA, Kilgore KL, Kirsch RF, Miller LE. Toward the restoration of hand use to a paralyzed monkey: brain-controlled functional electrical stimulation of forearm muscles. PloS One 4: e5924, 2009. 642. Pohlmeyer EA, Solla SA, Perreault EJ, Miller LE. Prediction of upper limb muscle activity from motor cortical discharge during reaching. J Neural Eng 4: 369, 2007. 643. Pokorny C, Klobassa DS, Pichler G, Erlbeck H, Real RG, Kubler A, Lesenfants D, Habbal D, Noirhomme Q, Risetti M, Mattia D, Muller-Putz GR. The auditory P300based single-switch brain-computer interface: paradigm transition from healthy subjects to minimally conscious patients. Artif Intell Med 59: 81–90, 2013.
623. Penfield W, Boldrey E. Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain 60: 389 – 443, 1937.
644. Poli R, Cinel C, Matran-Fernandez A, Sepulveda F, Stoica A. Towards cooperative brain-computer interfaces for space navigation. In: Proceedings of the 2013 International Conference on Intelligent User Interfaces. New York: ACM, 2013, p. 149 –160.
624. Perge JA, Homer ML, Malik WQ, Cash S, Eskandar E, Friehs G, Donoghue JP, Hochberg LR. Intra-day signal instabilities affect decoding performance in an intracortical neural interface system. J Neural Eng 10: 036004, 2013.
645. Poli R, Valeriani D, Cinel C. Collaborative brain-computer interface for aiding decision-making. PLoS One 9: e102693, 2014.
625. Pesaran B, Musallam S, Andersen RA. Cognitive neural prosthetics. Curr Biol 16: R77– 80, 2006. 626. Pfurtscheller G, Allison BZ, Bauernfeind G, Brunner C, Solis Escalante T, Scherer R, Zander TO, Mueller-Putz G, Neuper C, Birbaumer N. The hybrid BCI. Front Neurosci 4: 30, 2010. 627. Pfurtscheller G, Flotzinger D, Pregenzer M, Wolpaw JR, McFarland D. EEG-based brain computer interface (BCI). Search for optimal electrode positions and frequency components. Med Prog Technol 21: 111–121, 1995. 628. Pfurtscheller G, Guger C, Müller G, Krausz G, Neuper C. Brain oscillations control hand orthosis in a tetraplegic. Neurosci Lett 292: 211–214, 2000. 629. Pfurtscheller G, Lopes Da Silva FH. Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin Neurophysiol 110: 1842–1857, 1999. 630. Pfurtscheller G, Müller GR, Pfurtscheller J, Gerner HJ, Rupp R. “Thought”-control of functional electrical stimulation to restore hand grasp in a patient with tetraplegia. Neurosci Lett 351: 33–36, 2003.
646. Polikov VS, Tresco PA, Reichert WM. Response of brain tissue to chronically implanted neural electrodes. J Neurosci Methods 148: 1–18, 2005. 647. Porter LL. Somatosensory input onto pyramidal tract neurons in rodent motor cortex. Neuroreport 7: 2309 –2315, 1996. 648. Pouget A, Dayan P, Zemel R. Information processing with population codes. Nature Rev Neurosci 1: 125–132, 2000. 649. Power SD, Falk TH, Chau T. Classification of prefrontal activity due to mental arithmetic and music imagery using hidden Markov models and frequency domain nearinfrared spectroscopy. J Neural Eng 7: 026002, 2010. 650. Power SD, Kushki A, Chau T. Towards a system-paced near-infrared spectroscopy brain-computer interface: differentiating prefrontal activity due to mental arithmetic and mental singing from the no-control state. J Neural Eng 8: 066004, 2011. 651. Prasad G, Herman P, Coyle D, McDonough S, Crosbie J. Applying a brain-computer interface to support motor imagery practice in people with stroke for upper limb recovery: a feasibility study. J Neuroeng Rehabil 7: 60, 2010.
631. Pfurtscheller G, Neuper C. Future prospects of ERD/ERS in the context of braincomputer interface (BCI) developments. Prog Brain Res 159: 433– 437, 2006.
652. Presacco A, Goodman R, Forrester L, Contreras-Vidal JL. Neural decoding of treadmill walking from noninvasive electroencephalographic signals. J Neurophysiol 106: 1875–1887, 2011.
632. Pfurtscheller G, Neuper C. Motor imagery and direct brain-computer communication. Proc IEEE 89: 1123–1134, 2001.
653. Prut Y, Fetz EE. Primate spinal interneurons show pre-movement instructed delay activity. Nature 401: 590 –594, 1999.
633. Pfurtscheller J, Rupp R, Muller GR, Fabsits E, Korisek G, Gerner HJ, Pfurtscheller G. [Functional electrical stimulation instead of surgery? Improvement of grasping function with FES in a patient with C5 tetraplegia] Unfallchirurg 108: 587–590, 2005.
654. Qin L, Ding L, He B. Motor imagery classification by means of source analysis for brain-computer interface applications. J Neural Eng 1: 135–141, 2004.
634. Philips J, Millán JdR, Vanacker G, Lew E, Galán F, Ferrez PW, Van Brussel H, Nuttin M. Adaptive shared control of a brain-actuated simulated wheelchair. In: 2007 IEEE 10th International Conference on Rehabilitation Robotics. New York: IEEE, 2007, p. 408 – 414. 635. Piccione F, Giorgi F, Tonin P, Priftis K, Giove S, Silvoni S, Palmas G, Beverina F. P300-based brain computer interface: reliability and performance in healthy and paralysed participants. Clin Neurophysiol 117: 531–537, 2006. 636. Picot A, Charbonnier S, Caplier A. On-line automatic detection of driver drowsiness using a single electroencephalographic channel. Conf Proc IEEE Eng Med Biol Soc 2008: 3864 –3867, 2008. 637. Picton TW. The P300 wave of the human event-related potential. J Clin Neurophysiol 9: 456 – 479, 1992.
655. Rabiner LR. A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77: 257–286, 1989. 656. Rajangam S, Tseng PH, Yin A, Lehew G, Schwarz D, Lebedev MA, Nicolelis MA. Wireless cortical brain-machine interface for whole-body navigation in primates. Sci Rep 6: 22170, 2016. 657. Ramakrishnan A, Ifft PJ, Pais-Vieira M, Byun YW, Zhuang KZ, Lebedev MA, Nicolelis MA. Computing Arm Movements with a Monkey Brainet. Sci Rep 5: 10767, 2015. 658. Ramon y Cajal S. Histology of the Nervous System of Man and Vertebrates. New York: Oxford Univ. Press, 1995. 659. Ramos-Murguialday A, Broetz D, Rea M, Läer L, Yilmaz Ö, Brasil FL, Liberati G, Curado MR, Garcia-Cossio E, Vyziotis A. Brain-machine interface in chronic stroke rehabilitation: a controlled study. Ann Neurol 74: 100 –108, 2013.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
831
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
622. Pelletier KR, Peer E. Developing a biofeedback model: alpha EEG feedback as a means for pain control. Int J Clin Exp Hypn 25: 361–371, 1977.
638. Pistohl T, Ball T, Schulze-Bonhage A, Aertsen A, Mehring C. Prediction of arm movement trajectories from ECoG-recordings in humans. J Neurosci Methods 167: 105– 114, 2008.
LEBEDEV AND NICOLELIS 660. Rao RP. Brain-Computer Interfacing: An Introduction. Cambridge, UK: Cambridge Univ. Press, 2013. 661. Rao RP, Stocco A, Bryan M, Sarma D, Youngquist TM, Wu J, Prat CS. A direct brain-to-brain interface in humans. PLoS One 9: e111332, 2014. 662. Raspopovic S, Capogrosso M, Petrini FM, Bonizzato M, Rigosa J, Di Pino G, Carpaneto J, Controzzi M, Boretius T, Fernandez E. Restoring natural sensory feedback in realtime bidirectional hand prostheses. Science Transl Med 6: 222ra219, 2014. 663. Rebsamen B, Guan C, Zhang H, Wang C, Teo C, Ang MH, Burdet E. A brain controlled wheelchair to navigate in familiar environments. IEEE Trans Neural Syst Rehab Eng 18: 590 –598, 2010. 664. Recce M, O’Keefe J. The tetrode: a new technique for multi-unit extracellular recording. Soc Neurosci Abstr 1250, 1989. 665. Reid SA. Toward the ideal electrocorticography array. Neurosurgery 25: 135–137, 1989.
667. Renfrew M, Cheng R, Daly JJ, Cavusoglu M. Comparison of filtering and classification techniques of electroencephalography for brain-computer interface. Conf Proc IEEE Eng Med Biol Soc 2008: 2634 –2637, 2008. 668. Rennaker RL, Street S, Ruyle AM, Sloan AM. A comparison of chronic multi-channel cortical implantation techniques: manual versus mechanical insertion. J Neurosci Methods 142: 169 –176, 2005. 669. Rezaei S, Tavakolian K, Nasrabadi AM, Setarehdan SK. Different classification techniques considering brain computer interface applications. J Neural Eng 3: 139 –144, 2006. 670. Rickert J, Oliveira SC, Vaadia E, Aertsen A, Rotter S, Mehring C. Encoding of movement direction in different frequency ranges of motor cortical local field potentials. J Neurosci 25: 8815– 8824, 2005. 671. Rickert J, Riehle A, Aertsen A, Rotter S, Nawrot MP. Dynamic encoding of movement direction in motor cortical neurons. J Neurosci 29: 13870 –13882, 2009.
682. Rouse AG, Williams JJ, Wheeler JJ, Moran DW. Cortical adaptation to a chronic micro-electrocorticographic brain computer interface. J Neurosci 33: 1326 –1330, 2013. 683. Saal HP, Bensmaia SJ. Biomimetic approaches to bionic touch through a peripheral nerve interface. Neuropsychologia 79: 344 –353, 2015. 684. Sadtler PT, Quick KM, Golub MD, Chase SM, Ryu SI, Tyler-Kabara EC, Byron MY, Batista AP. Neural constraints on learning. Nature 512: 423– 426, 2014. 685. Sahayadhas A, Sundaraj K, Murugappan M. Drowsiness detection during different times of day using multiple features. Australas Phys Eng Sci Med 36: 243–250, 2013. 686. Sakurai Y. Brain-machine interfaces can accelerate clarification of the principal mysteries and real plasticity of the brain. Front Syst Neurosci 8: 104, 2014. 687. Sakurai Y. How do cell assemblies encode information in the brain? Neurosci Biobehav Rev 23: 785–796, 1999. 688. Salmelin R, Hari R. Spatiotemporal characteristics of sensorimotor neuromagnetic rhythms related to thumb movement. Neuroscience 60: 537–550, 1994. 689. Salmon DP, Butters N. Neurobiology of skill and habit learning. Curr Opin Neurobiol 5: 184 –190, 1995. 690. Sanchez JC, Carmena JM, Lebedev MA, Nicolelis MA, Harris JG, Principe JC. Ascertaining the importance of neurons to develop better brain-machine interfaces. IEEE Trans Biomed Eng 51: 943–953, 2004. 691. Sanchez JC, Erdogmus D, Rao Y, Principe JC, Nicolelis M, Wessberg J. Learning the contributions of the motor, premotor, and posterior parietal cortices for hand trajectory reconstruction in a brain machine interface. In: Neural Engineering, 2003 Conference Proceedings First International IEEE EMBS Conference. New York: IEEE, 2003, p. 59 – 62. 692. Santhanam G, Ryu SI, Yu BM, Afshar A, Shenoy KV. A high-performance brain-computer interface. Nature 442: 195–198, 2006. 693. Santos L, Opris I, Fuqua J, Hampson RE, Deadwyler SA. A novel tetrode microdrive for simultaneous multi-neuron recording from different regions of primate brain. J Neurosci Methods 205: 368 –374, 2012.
672. Rivet B, Cecotti H, Perrin M, Maby E, Mattout J. Adaptive training session for a P300 speller brain-computer interface. J Physiol 105: 123–129, 2011.
694. Santucci DM, Kralik JD, Lebedev MA, Nicolelis MA. Frontal and parietal cortical ensembles predict single-trial muscle activity during reaching movements in primates. Eur J Neurosci 22: 1529 –1540, 2005.
673. Rizzolatti G, Camarda R, Fogassi L, Gentilucci M, Luppino G, Matelli M. Functional organization of inferior area 6 in the macaque monkey. Exp Brain Res 71: 491–507, 1988.
695. Saxena T, Karumbaiah L, Gaupp EA, Patkar R, Patil K, Betancur M, Stanley GB, Bellamkonda RV. The impact of chronic blood-brain barrier breach on intracortical electrode function. Biomaterials 34: 4703– 4713, 2013.
674. Rodenkirch C, Schriver B, Wang Q. Brain-machine interfaces: restoring and establishing communication channels. In: Neural Engineering. New York: Springer, 2016, p. 227–259.
696. Schalk G. Can electrocorticography (ECoG) support robust and powerful brain-computer interfaces? Front Neuroeng 3: 9, 2010.
675. Rolls ET. Parallel distributed processing in the brain: implications of the functional architecture of neuronal networks in the hippocampus. In: Parallel Distributed Processing: Implications for Psychology and Neurobiology, edited by Morris RGM. New York: Clarendon, 1989. 676. Romo R, Hernández A, Zainos A, Salinas E. Somatosensory discrimination based on cortical microstimulation. Nature 392: 387–390, 1998. 677. Romo R, Scarnati E, Schultz W. Role of primate basal ganglia and frontal cortex in the internal generation of movements. II. Movement-related activity in the anterior striatum. Exp Brain Res 91: 385–395, 1992. 678. Rothschild RM. Neuroengineering tools/applications for bidirectional interfaces, brain-computer interfaces, and neuroprosthetic implants: a review of recent progress. Front Neuroeng 3: 112, 2010. 679. Rousche PJ, Normann RA. Chronic recording capability of the Utah Intracortical Electrode Array in cat sensory cortex. J Neurosci Methods 82: 1–15, 1998. 680. Rousche PJ, Normann RA. A method for pneumatically inserting an array of penetrating electrodes into cortical tissue. Ann Biomed Eng 20: 413– 422, 1992. 681. Rousche PJ, Normann RA. A system for impact insertion of a 100 electrode array into cortical tissue. In: Proceedings of the 12th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Philadelphia, PA. Piscataway, NJ: IEEE, 1990.
832
697. Schalk G, Kubanek J, Miller KJ, Anderson NR, Leuthardt EC, Ojemann JG, Limbrick D, Moran D, Gerhardt LA, Wolpaw JR. Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. J Neural Eng 4: 264 –275, 2007. 698. Scherer R, Schloegl A, Lee F, Bischof H, Jansa J, Pfurtscheller G. The self-paced graz brain-computer interface: methods and applications. Comput Intell Neurosci 79826, 2007. 699. Scherlag BJ, Yamanashi WS, Schauerte P, Scherlag M, Sun YX, Hou Y, Jackman WM, Lazzara R. Endovascular stimulation within the left pulmonary artery to induce slowing of heart rate and paroxysmal atrial fibrillation. Cardiovasc Res 54: 470 – 475, 2002. 700. Scherlag MA, Scherlag BJ, Yamanashi W, Schauerte P, Goli S, Jackman WM, Reynolds D, Lazzara R. Endovascular neural stimulation via a novel basket electrode catheter: comparison of electrode configurations. J Intervent Cardiac Electrophysiol 4: 219 –224, 2000. 701. Schmidt EM. Cortical control of robotic devices and neuromuscular stimulators. In: Neurobionics: An Interdisciplinary Approach to Substitute Impaired Functions of the Human Nervous System, edited by Bothe H-W, Samii M, Eckmiller R. Amsterdam: NorthHolland/Elsevier, 2013. 702. Schmidt EM. Single neuron recording from motor cortex as a possible source of signals for control of external devices. Ann Biomed Eng 8: 339 –349, 1980. 703. Schmidt EM, Bak MJ, McIntosh JS, Thomas JS. Operant conditioning of firing patterns in monkey cortical neurons. Exp Neurol 54: 467– 477, 1977.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
666. Renaud P, Joyal C, Stoleru S, Goyette M, Weiskopf N, Birbaumer N. Real-time functional magnetic imaging-brain-computer interface and virtual reality promising tools for the treatment of pedophilia. Prog Brain Res 192: 263–272, 2011.
425
426
BRAIN-MACHINE INTERFACES 704. Schmidt EM, McIntosh JS. Microstimulation mapping of precentral cortex during trained movements. J Neurophysiol 64: 1668 –1682, 1990.
727. Shanechi MM, Hu RC, Williams ZM. A cortical-spinal prosthesis for targeted limb movement in paralysed primate avatars. Nature Commun 5: 3237, 2014.
705. Schmidt EM, McIntosh JS. Microstimulation of precentral cortex with chronically implanted microelectrodes. Exp Neurol 63: 485–503, 1979.
728. Shanechi MM, Orsborn A, Moorman H, Gowda S, Carmena JM. High-performance brain-machine interface enabled by an adaptive optimal feedback-controlled point process decoder. Conf Proc IEEE Eng Med Biol Soc 2014: 6493– 6496, 2014.
706. Schultz W. Reward functions of the basal ganglia. J Neural Transm 123: 679 – 693, 2016. 707. Schultz W, Apicella P, Ljungberg T, Romo R, Scarnati E. Reward-related activity in the monkey striatum and substantia nigra. Prog Brain Res 99: 227–235, 1993. 708. Schultz W, Apicella P, Scarnati E, Ljungberg T. Neuronal activity in monkey ventral striatum related to the expectation of reward. J Neurosci 12: 4595– 4610, 1992. 709. Schultz W, Tremblay L, Hollerman JR. Reward prediction in primate basal ganglia and frontal cortex. Neuropharmacology 37: 421– 429, 1998. 710. Schwartz AB, Kettner RE, Georgopoulos AP. Primate motor cortex and free arm movements to visual targets in three-dimensional space. I. Relations between single cell discharge and direction of movement. J Neurosci 8: 2913–2927, 1988.
712. Sefcik RK, Opie NL, John SE, Kellner CP, Mocco J, Oxley TJ. The evolution of endovascular electroencephalography: historical perspective and future applications. Neurosurg Focus 40: E7, 2016. 713. Seifert HM, Fuglevand AJ. Restoration of movement using functional electrical stimulation and Bayes’ theorem. J Neurosci 22: 9465–9474, 2002. 714. Seki K, Fetz EE. Gating of sensory input at spinal and cortical levels during preparation and execution of voluntary movement. J Neurosci 32: 890 –902, 2012. 715. Seki K, Perlmutter SI, Fetz EE. Sensory input to primate spinal cord is presynaptically inhibited during voluntary movement. Nat Neurosci 6: 1309 –1316, 2003. 716. Seki K, Perlmutter SI, Fetz EE. Task-dependent modulation of primary afferent depolarization in cervical spinal cord of monkeys performing an instructed delay task. J Neurophysiol 102: 85–99, 2009.
730. Shanechi MM, Williams ZM, Wornell GW, Hu RC, Powers M, Brown EN. A real-time brain-machine interface combining motor target and trajectory intent using an optimal feedback control design. PloS One 8: e59049, 2013. 731. Shenoy KV, Meeker D, Cao S, Kureshi SA, Pesaran B, Buneo CA, Batista AP, Mitra PP, Burdick JW, Andersen RA. Neural prosthetic control signals from plan activity. Neuroreport 14: 591–596, 2003. 732. Shepherd GM. Foundations of the Neuron Doctrine. New York: Oxford Univ. Press, 2016, p. xxvi. 733. Sherrington CS. The Integrative Action of the Nervous System. New York: C. Scribner’s Sons, 1906, p. xvi. 734. Shin HC, Chapin JK. Movement induced modulation of afferent transmission to single neurons in the ventroposterior thalamus and somatosensory cortex in rat. Exp Brain Res 81: 515–522, 1990. 735. Shishkin SL, Ganin IP, Basyul IA, Zhigalov AY, Kaplan AY. N1 wave in the P300 BCI is not sensitive to the physical characteristics of stimuli. J Integr Neurosci 8: 471– 485, 2009. 736. Shishkin SL, Ganin IP, Kaplan AY. Event-related potentials in a moving matrix modification of the P300 brain-computer interface paradigm. Neurosci Lett 496: 95–99, 2011. 737. Shokur S, Gallo S, Moioli RC, Donati AR, Morya E, Bleuler H, Nicolelis MA. Assimilation of virtual legs and perception of floor texture by complete paraplegic patients receiving artificial tactile feedback. Sci Rep 6: 32293, 2016. 738. Shokur S, O’Doherty JE, Winans JA, Bleuler H, Lebedev MA, Nicolelis MA. Expanding the primate body schema in sensorimotor cortex by virtual touches of an avatar. Proc Natl Acad Sci USA 110: 15121–15126, 2013.
717. Sellers EW, Krusienski DJ, McFarland DJ, Vaughan TM, Wolpaw JR. A P300 eventrelated potential brain-computer interface (BCI): the effects of matrix size and inter stimulus interval on performance. Biol Psychol 73: 242–252, 2006.
739. Siegel M, Körding KP, König P. Integrating top-down and bottom-up sensory processing by somato-dendritic interactions. J Computat Neurosci 8: 161–173, 2000.
718. Sellers EW, Ryan DB, Hauser CK. Noninvasive brain-computer interface enables communication after brainstem stroke. Sci Transl Med 6: 257re257, 2014.
740. Silvoni S, Cavinato M, Volpato C, Ruf CA, Birbaumer N, Piccione F. Amyotrophic lateral sclerosis progression and stability of brain-computer interface communication. Amyotroph Lateral Scler Frontotemporal Degener 14: 390 –396, 2013.
719. Semework M. Microstimulation: principles, techniques, and approaches to somatosensory neuroprosthesis. Crit Rev Biomed Eng 43: 61–95, 2015. 720. Seo D, Carmena JM, Rabaey JM, Maharbiz MM, Alon E. Model validation of untethered, ultrasonic neural dust motes for cortical recording. J Neurosci Methods 244: 114 –122, 2015. 721. Seo D, Neely RM, Shen K, Singhal U, Alon E, Rabaey JM, Carmena JM, Maharbiz MM. Wireless recording in the peripheral nervous system with ultrasonic neural dust. Neuron 91: 529 –539, 2016. 722. Serdar Bascil M, Tesneli AY, Temurtas F. Multi-channel EEG signal feature extraction and pattern recognition on horizontal mental imagination task of 1-D cursor movement for brain computer interface. Australas Phys Eng Sci Med 38: 229 –239, 2015. 723. Serruya M, Hatsopoulos N, Fellows M, Paninski L, Donoghue J. Robustness of neuroprosthetic decoding algorithms. Biol Cybern 88: 219 –228, 2003.
741. Silvoni S, Ramos-Murguialday A, Cavinato M, Volpato C, Cisotto G, Turolla A, Piccione F, Birbaumer N. Brain-computer interface in stroke: a review of progress. Clin EEG Neurosci 42: 245–252, 2011. 742. Silvoni S, Volpato C, Cavinato M, Marchetti M, Priftis K, Merico A, Tonin P, Koutsikos K, Beverina F, Piccione F. P300-based brain-computer interface communication: evaluation and follow-up in amyotrophic lateral sclerosis. Front Neurosci 3: 60, 2009. 743. Simmons FB, Mongeon CJ, Lewis WR, Huntington DA. Electrical stimulation of acoustical nerve and inferior colliculus. Arch Otolaryngol 79: 559 –568, 1964. 744. Sitaram R, Caria A, Birbaumer N. Hemodynamic brain-computer interfaces for communication and rehabilitation. Neural Networks 22: 1320 –1328, 2009. 745. Sitaram R, Caria A, Veit R, Gaber T, Rota G, Kuebler A, Birbaumer N. FMRI braincomputer interface: a tool for neuroscientific research and treatment. Comput Intell Neurosci 25487, 2007.
724. Serruya M, Shaikhouni A, Donoghue J. Neural decoding of cursor motion using a Kalman filter. In: Advances in Neural Information Processing Systems 15: Proceedings of the 2002 Conference. Cambridge, MA: MIT Press, 2003, p. 133.
746. Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, Shimizu K, Birbaumer N. Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface. Neuroimage 34: 1416 –1427, 2007.
725. Serruya MD, Hatsopoulos NG, Paninski L, Fellows MR, Donoghue JP. Instant neural control of a movement signal. Nature 416: 141–142, 2002.
747. Sloper JJ. Gap junctions between dendrites in the primate neocortex. Brain Res 44: 641– 646, 1972.
726. Shadmehr R, Wise SP. The Computational Neurobiology of Reaching and Pointing: A Foundation for Motor Learning. Cambridge, MA: MIT Press, 2005.
748. Smetters D, Majewska A, Yuste R. Detecting action potentials in neuronal populations with calcium imaging. Methods 18: 215–221, 1999.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
833
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
711. Schwarz DA, Lebedev MA, Hanson TL, Dimitrov DF, Lehew G, Meloy J, Rajangam S, Subramanian V, Ifft PJ, Li Z, Ramakrishnan A, Tate A, Zhuang KZ, Nicolelis MA. Chronic, wireless recordings of large-scale brain activity in freely moving rhesus monkeys. Nat Methods 11: 670 – 676, 2014.
729. Shanechi MM, Orsborn AL, Carmena JM. Robust brain-machine interface design using optimal feedback control modeling and adaptive point process filtering. PLoS Comput Biol 12: e1004730, 2016.
LEBEDEV AND NICOLELIS 749. Smith KU, Ansell SD. Closed-loop digital computer system for study of sensory feedback effects of brain rhythms. Am J Phys Med 44: 125–137, 1965. 750. So K, Dangi S, Orsborn AL, Gastpar MC, Carmena J. Subject-specific modulation of local field potential spectral power during brain-machine interface control in primates. J Neural Eng 11: 026002, 2014. 751. Soekadar SR, Birbaumer N, Slutzky MW, Cohen LG. Brain-machine interfaces in neurorehabilitation of stroke. Neurobiol Dis 83: 172–179, 2015. 752. Soekadar SR, Cohen LG, Birbaumer N. Clinical brain-machine interfaces. Cogn Plast Neurol Disorders 347, 2014. 753. Soekadar SR, Witkowski M, Garcia Cossio E, Birbaumer N, Cohen L. Learned EEGbased brain self-regulation of motor-related oscillations during application of transcranial electric brain stimulation: feasibility and limitations. Front Behav Neurosci 8: 93, 2014. 754. Sohal HS, Jackson A, Jackson R, Clowry GJ, Vaisilevskiy K, O’Neill A, Baker S. The Sinusoidal Probe: a new approach to improve electrode longevity. Front Neuroeng 7: 10, 2014.
756. Soso M, Fetz E. Responses of identified cells in postcentral cortex of awake monkeys during comparable active and passive joint movements. J Neurophysiol 43: 1090 – 1110, 1980. 757. Spiers HJ, Maguire EA. Neural substrates of driving behaviour. Neuroimage 36: 245– 255, 2007. 758. Spieth S, Brett O, Seidl K, Aarts A, Erismis M, Herwik S, Trenkle F, Tätzner S, Auber J, Daub M. A floating 3D silicon microprobe array for neural drug delivery compatible with electrical recording. J Micromech Microeng 21: 125001, 2011. 759. Spuler M, Walter A, Ramos-Murguialday A, Naros G, Birbaumer N, Gharabaghi A, Rosenstiel W, Bogdan M. Decoding of motor intentions from epidural ECoG recordings in severely paralyzed chronic stroke patients. J Neural Eng 11: 066008, 2014.
772. Strangman G, Culver JP, Thompson JH, Boas DA. A quantitative comparison of simultaneous BOLD fMRI and NIRS recordings during functional brain activation. Neuroimage 17: 719 –731, 2002. 773. Sugata H, Hirata M, Yanagisawa T, Matsushita K, Yorifuji S, Yoshimine T. Common neural correlates of real and imagined movements contributing to the performance of brain-machine interfaces. Sci Rep 6: 24663, 2016. 774. Suminski AJ, Fagg AH, Willett FR, Bodenhamer M, Hatsopoulos NG. Online adaptive decoding of intended movements with a hybrid kinetic and kinematic brain machine interface. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New York: IEEE, 2013, p. 1583–1586. 775. Sung Y, Cho K, Um K. A development architecture for serious games using BCI (brain computer interface) sensors. Sensors 12: 15671–15688, 2012. 776. Sussillo D, Nuyujukian P, Fan JM, Kao JC, Stavisky SD, Ryu S, Shenoy K. A recurrent neural network for closed-loop intracortical brain-machine interface decoders. J Neural Eng 9: 026027, 2012. 777. Suter S. Independent biofeedback self-regulation of EEG alpha and skin resistance. Biofeedback Self Regul 2: 255–258, 1977. 778. Sutton S, Braren M, Zubin J, John E. Evoked-potential correlates of stimulus uncertainty. Science 150: 1187–1188, 1965. 779. Suyatin DB, Wallman L, Thelin J, Prinz CN, Jörntell H, Samuelson L, Montelius L, Schouenborg J. Nanowire-based electrode for acute in vivo neural recordings in the brain. PloS One 8: e56673, 2013. 780. Suykens JA, Vandewalle J. Least squares support vector machine classifiers. Neural Processing Lett 9: 293–300, 1999. 781. Svaetichin G. Electrophysiological investigations on single ganglion cells. Acta Physiol Scand Suppl 24: 1–57, 1951. 782. Svoboda K, Yasuda R. Principles of two-photon excitation microscopy and its applications to neuroscience. Neuron 50: 823– 839, 2006.
760. Starr A, Cohen LG. “Gating” of somatosensory evoked potentials begins before the onset of voluntary movement in man. Brain Res 348: 183–186, 1985.
783. Szuts TA, Fadeyev V, Kachiguine S, Sher A, Grivich MV, Agrochão M, Hottowy P, Dabrowski W, Lubenov EV, Siapas AG. A wireless multi-channel neural amplifier for freely moving animals. Nature Neurosci 14: 263–269, 2011.
761. Stavisky SD, Kao JC, Nuyujukian P, Ryu SI, Shenoy KV. A high performing brainmachine interface driven by low-frequency local field potentials alone and together with spikes. J Neural Eng 12: 036009, 2015.
784. Tabot GA, Dammann JF, Berg JA, Tenore FV, Boback JL, Vogelstein RJ, Bensmaia SJ. Restoring the sense of touch with a prosthetic hand through a brain interface. Proc Natl Acad Sci USA 110: 18279 –18284, 2013.
762. Stepnoski R, LaPorta A, Raccuia-Behling F, Blonder G, Slusher R, Kleinfeld D. Noninvasive detection of changes in membrane potential in cultured neurons by light scattering. Proc Natl Acad Sci USA 88: 9382–9386, 1991.
785. Taheri BA, Knight RT, Smith RL. A dry electrode for EEG recording. Electroencephalogr Clin Neurophysiol 90: 376 –383, 1994.
763. Sterman MB. EEG biofeedback: physiological behavior modification. Neurosci Biobehav Rev 5: 405– 412, 1981.
786. Tai K, Chau T. Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface. J Neuroeng Rehab 6: 1, 2009.
764. Sterman MB. Neurophysiologic and clinical studies of sensorimotor EEG biofeedback training: some effects on epilepsy. Semin Psychiatry 5: 507–525, 1973.
787. Takano K, Komatsu T, Hata N, Nakajima Y, Kansaku K. Visual stimuli for the P300 brain-computer interface: a comparison of white/gray and green/blue flicker matrices. Clin Neurophysiol 120: 1562–1566, 2009.
765. Sterman MB, Friar L. Suppression of seizures in an epileptic following sensorimotor EEG feedback training. Electroencephalogr Clin Neurophysiol 33: 89 –95, 1972.
788. Takeuchi S, Suzuki T, Mabuchi K, Fujita H. 3D flexible multichannel neural probe array. J Micromech Microeng 14: 104, 2003.
766. Sterman MB, Macdonald LR, Stone RK. Biofeedback training of the sensorimotor electroencephalogram rhythm in man: effects on epilepsy. Epilepsia 15: 395– 416, 1974.
789. Takeuchi S, Yoshida Y, Ziegler D, Mabuchi K, Suzuki T. Parylene flexible neural probe with micro fluidic channel. In: Micro Electro Mechanical Systems, 17th IEEE International Conference. New York: IEEE, 2004, p. 208 –211.
767. Stevenson IH, Kording KP. How advances in neural recording affect data analysis. Nat Neurosci 14: 139 –142, 2011.
790. Talwar SK, Xu S, Hawley ES, Weiss SA, Moxon KA, Chapin JK. Rat navigation guided by remote control. Nature 417: 37–38, 2002.
768. Stienen AH, Hekman EE, Van Der Helm FC, Van Der Kooij H. Self-aligning exoskeleton axes through decoupling of joint rotations and translations. IEEE Trans Robotics 25: 628 – 633, 2009.
791. Tan DW, Schiefer MA, Keith MW, Anderson JR, Tyler J, Tyler DJ. A neural interface provides long-term stable natural touch perception. Science Transl Med 6: 257ra138, 2014.
769. Stoerig P, Cowey A. Blindsight in man and monkey. Brain 120: 535–559, 1997.
792. Tankus A, Fried I, Shoham S. Cognitive-motor brain-machine interfaces. J Physiol 108: 38 – 44, 2014.
770. Stoney SD Jr, Thompson WD, Asanuma H. Excitation of pyramidal tract cells by intracortical microstimulation: effective extent of stimulating current. J Neurophysiol 31: 659 – 669, 1968.
793. Tasaki I, Watanabe A, Sandlin R, Carnay L. Changes in fluorescence, turbidity, and birefringence associated with nerve excitation. Proc Natl Acad Sci USA 61: 883, 1968.
771. Stosiek C, Garaschuk O, Holthoff K, Konnerth A. In vivo two-photon calcium imaging of neuronal networks. Proc Natl Acad Sci USA 100: 7319 –7324, 2003.
794. Taylor DM, Tillery SI, Schwartz AB. Direct cortical control of 3D neuroprosthetic devices. Science 296: 1829 –1832, 2002.
834
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
755. Sokunbi MO, Linden DE, Habes I, Johnston S, Ihssen N. Real-time fMRI brain-computer interface: development of a “motivational feedback” subsystem for the regulation of visual cue reactivity. Front Behav Neurosci 8: 392, 2014.
427
428
BRAIN-MACHINE INTERFACES 795. Taylor DM, Tillery SI, Schwartz AB. Information conveyed through brain-control: cursor versus robot. IEEE Trans Neural Syst Rehabil Eng 11: 195–199, 2003.
817. Velliste M, Perel S, Spalding MC, Whitford AS, Schwartz AB. Cortical control of a prosthetic arm for self-feeding. Nature 453: 1098 –1101, 2008.
796. Tepavac D, Schwirtlich L. Detection and prediction of FES-induced fatigue. J Electromyogr Kinesiol 7: 39 –50, 1997.
818. Veltink PH. Sensory feedback in artificial control of human mobility. Technol Health Care 7: 383–391, 1999.
797. Thomke F, Stoeter P, Stader D. Endovascular electroencephalography during an intracarotid amobarbital test with simultaneous recordings from 16 electrodes. J Neurol Neurosurg Psychiatry 64: 565, 1998.
819. Venkatakrishnan A, Francisco GE, Contreras-Vidal JL. Applications of brain-machine interface systems in stroke recovery and rehabilitation. Curr Physical Med Rehab Reports 2: 93–105, 2014.
798. Thomson EE, Carra R, Nicolelis MA. Perceiving invisible light through a somatosensory cortical prosthesis. Nat Commun 4: 1482, 2013.
820. Venkatraman S, Carmena JM. Active sensing of target location encoded by cortical microstimulation. IEEE Trans Neural Syst Rehabil Eng 19: 317–324, 2011.
799. Tian B, Cohen-Karni T, Qing Q, Duan X, Xie P, Lieber CM. Three-dimensional, flexible nanoscale field-effect transistors as localized bioprobes. Science 329: 830 – 834, 2010.
821. Verleysen M, François D. The curse of dimensionality in data mining and time series prediction. In: Computational Intelligence and Bioinspired Systems. New York: Springer, 2005, p. 758 –770.
800. Tkach D, Reimer J, Hatsopoulos NG. Observation-based learning for brain-machine interfaces. Curr Opin Neurobiol 18: 589 –594, 2008.
822. Vetter RJ, Williams JC, Hetke JF, Nunamaker EA, Kipke DR. Chronic neural recording using silicon-substrate microelectrode arrays implanted in cerebral cortex. Biomed Eng IEEE Trans 51: 896 –904, 2004.
801. Todorov E. Optimality principles in sensorimotor control. Nature Neurosci 7: 907– 915, 2004.
803. Todorova S, Sadtler P, Batista A, Chase S, Ventura V. To sort or not to sort: the impact of spike-sorting on neural decoding performance. J Neural Eng 11: 056005, 2014. 804. Tonin L, Leeb R, Tavella M, Perdikis S, Millán JdR. The role of shared-control in BCI-based telepresence. In: Systems Man and Cybernetics (SMC), 2010 IEEE International Conference. New York: IEEE, 2010, p. 1462–1466. 805. Townsend G, LaPallo BK, Boulay CB, Krusienski DJ, Frye GE, Hauser CK, Schwartz NE, Vaughan TM, Wolpaw JR, Sellers EW. A novel P300-based brain-computer interface stimulus presentation paradigm: moving beyond rows and columns. Clin Neurophysiol 121: 1109 –1120, 2010. 806. Tozzi A, Zare M, Benasich AA. New perspectives on spontaneous brain activity: dynamic networks and energy matter. Front Hum Neurosci 10: 247, 2016. 807. Truccolo W. Stochastic models for multivariate neural point processes: Collective dynamics and neural decoding. In: Analysis of Parallel Spike Trains. New York: Springer, 2010, p. 321–341. 808. Truccolo W, Donoghue JA, Hochberg LR, Eskandar EN, Madsen JR, Anderson WS, Brown EN, Halgren E, Cash SS. Single-neuron dynamics in human focal epilepsy. Nature Neurosci 14: 635– 641, 2011. 809. Truccolo W, Eden UT, Fellows MR, Donoghue JP, Brown EN. A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. J Neurophysiol 93: 1074 –1089, 2005. 810. Tsai PY, Hu W, Kuo TB, Shyu LY. A portable device for real time drowsiness detection using novel active dry electrode system. Conf Proc IEEE Eng Med Biol Soc 2009: 3775– 3778, 2009. 811. Tsui CS, Gan JQ, Hu H. A self-paced motor imagery based brain-computer interface for robotic wheelchair control. Clin EEG Neurosci 42: 225–229, 2011. 812. Tucker MR, Olivier J, Pagel A, Bleuler H, Bouri M, Lambercy O, del Millán J R, Riener R, Vallery H, Gassert R. Control strategies for active lower extremity prosthetics and orthotics: a review. J Neuroeng Rehab 12: 1, 2015. 813. Vallabhaneni A, He B. Motor imagery task classification for brain computer interface applications using spatiotemporal principle component analysis. Neurol Res 26: 282– 287, 2004. 814. van der Waal M, Severens M, Geuze J, Desain P. Introducing the tactile speller: an ERP-based brain-computer interface for communication. J Neural Eng 9: 045002, 2012.
824. Vidal JJ. Real-time detection of brain events in EEG. Proc IEEE 65: 633– 641, 1977. 825. Viventi J, Kim DH, Vigeland L, Frechette ES, Blanco JA, Kim YS, Avrin AE, Tiruvadi VR, Hwang SW, Vanleer AC. Flexible, foldable, actively multiplexed, high-density electrode array for mapping brain activity in vivo. Nature Neurosci 14: 1599 –1605, 2011. 826. Vuckovic A, Osuagwu BA. Using a motor imagery questionnaire to estimate the performance of a Brain-Computer Interface based on object oriented motor imagery. Clin Neurophysiol 124: 1586 –1595, 2013. 827. Wahnoun R, He J, Tillery SIH. Selection and parameterization of cortical neurons for neuroprosthetic control. J Neural Eng 3: 162, 2006. 828. Walker AE, Johnson HC, Marshall C. Electrocorticography. Bull Johns Hopkins Hosp 84: 583, 1949. 829. Walter WG. An imitation of life. Sci Am 182: 42– 45, 1950. 830. Walter WG. A machine that learns. Sc Am 185: 60 – 63, 1951. 831. Walter WG, Crow HJ. Depth Recording from the Human Brain. Electroencephalogr Clin Neurophysiol 16: 68 –72, 1964. 832. Wang C, Guan C, Zhang H. P300 brain-computer interface design for communication and control applications. Conf Proc IEEE Eng Med Biol Soc 5: 5400 –5403, 2005. 833. Wang H, Li Y, Long J, Yu T, Gu Z. An asynchronous wheelchair control by hybrid EEG-EOG brain-computer interface. Cogn Neurodyn 8: 399 – 409, 2014. 834. Wang L, Zhang X, Zhang Y. Extending motor imagery by speech imagery for braincomputer interface. Conf Proc IEEE Eng Med Biol Soc 2013: 7056 –7059, 2013. 835. Wang PT, King CE, Chui LA, Do AH, Nenadic Z. Self-paced brain-computer interface control of ambulation in a virtual reality environment. J Neural Eng 9: 056016, 2012. 836. Wang S, Wang L, Meijneke C, van Asseldonk E, Hoellinger T, Cheron G, Ivanenko Y, La Scaleia V, Sylos-Labini F, Molinari M, Tamburella F, Pisotta I, Thorsteinsson F, Ilzkovitz M, Gancet J, Nevatia Y, Hauffe R, Zanow F, van der Kooij H. Design and control of the MINDWALKER exoskeleton. IEEE Trans Neural Syst Rehabil Eng 23: 277–286, 2015. 837. Wang W, Collinger JL, Degenhart AD, Tyler-Kabara EC, Schwartz AB, Moran DW, Weber DJ, Wodlinger B, Vinjamuri RK, Ashmore RC, Kelly JW, Boninger ML. An electrocorticographic brain interface in an individual with tetraplegia. PLoS One 8: e55344, 2013.
815. Vaughan TM, Heetderks WJ, Trejo LJ, Rymer WZ, Weinrich M, Moore MM, Kubler A, Dobkin BH, Birbaumer N, Donchin E, Wolpaw EW, Wolpaw JR. Brain-computer interface technology: a review of the Second International Meeting. IEEE Trans Neural Syst Rehabil Eng 11: 94 –109, 2003.
838. Wang W, Degenhart AD, Collinger JL, Vinjamuri R, Sudre GP, Adelson PD, Holder DL, Leuthardt EC, Moran DW, Boninger ML. Human motor cortical activity recorded with Micro-ECoG electrodes, during individual finger movements. In: 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. New York: IEEE, 2009, p. 586 –589.
816. Velliste M, Kennedy SD, Schwartz AB, Whitford AS, Sohn JW, McMorland AJ. Motor cortical correlates of arm resting in the context of a reaching task and implications for prosthetic control. J Neurosci 34: 6011– 6022, 2014.
839. Wang W, Degenhart AD, Sudre GP, Pomerleau DA, Tyler-Kabara EC. Decoding semantic information from human electrocorticographic (ECoG) signals. Conf Proc IEEE Eng Med Biol Soc 2011: 6294 – 6298, 2011.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
835
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
802. Todorov E, Jordan MI. Optimal feedback control as a theory of motor coordination. Nature Neurosci 5: 1226 –1235, 2002.
823. Vidal JJ. Toward direct brain-computer communication. Annu Rev Biophys Bioeng 2: 157–180, 1973.
LEBEDEV AND NICOLELIS 840. Wang Y, Hong B, Gao X, Gao S. Implementation of a brain-computer interface based on three states of motor imagery. Conf Proc IEEE Eng Med Biol Soc 2007: 5059 –5062, 2007.
nology: a review of the first international meeting. IEEE Trans Rehabil Eng 8: 164 –173, 2000.
841. Wang Y, Jung TP. A collaborative brain-computer interface for improving human performance. PloS One 6: e20422, 2011.
864. Wolpaw JR, McFarland DJ. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc Natl Acad Sci USA 101: 17849 – 17854, 2004.
842. Watanabe H, Takahashi H, Nakao M, Walton K, Llinás RR. Intravascular neural interface with nanowire electrode. Electronics Commun Jpn 92: 29 –37, 2009.
865. Wolpaw JR, McFarland DJ, Neat GW, Forneris CA. An EEG-based brain-computer interface for cursor control. Electroencephalogr Clin Neurophysiol 78: 252–259, 1991.
843. Weber ES. Autogenic training and EEG biofeedback training in coronary heart disease. J Med Soc NJ 71: 927–931, 1974.
866. Wolpert DM, Ghahramani Z, Jordan MI. An internal model for sensorimotor integration. Science 269: 1880, 1995.
844. Weiland JD, Anderson DJ. Chronic neural stimulation with thin-film, iridium oxide electrodes. Biomed Eng IEEE Trans 47: 911–918, 2000.
867. Woodruff DS. Relationships among EEG alpha frequency, reaction time, and age: a biofeedback study. Psychophysiology 12: 673– 681, 1975.
845. Weinand ME, Hermann B, Wyler AR, Carter LP, Oommen KJ, Labiner D, Ahern G, Herring A. Long-term subdural strip electrocorticographic monitoring of ictal deja vu. Epilepsia 35: 1054 –1059, 1994.
868. Wu CH, Chang HC, Lee PL, Li KS, Sie JJ, Sun CW, Yang CY, Li PH, Deng HT, Shyu KK. Frequency recognition in an SSVEP-based brain computer interface using empirical mode decomposition and refined generalized zero-crossing. J Neurosci Methods 196: 170 –181, 2011.
846. Weinrich M, Wise SP, Mauritz KH. A neurophysiological study of the premotor cortex in the rhesus monkey. Brain 107: 385– 414, 1984.
848. Weiskopf N, Veit R, Erb M, Mathiak K, Grodd W, Goebel R, Birbaumer N. Physiological self-regulation of regional brain activity using real-time functional magnetic resonance imaging (fMRI): methodology and exemplary data. Neuroimage 19: 577– 586, 2003. 849. Weiskrantz L. Blindsight revisited. Curr Opin Neurobiol 6: 215–220, 1996. 850. Weiss SA, McKhann G Jr, Goodman R, Emerson RG, Trevelyan A, Bikson M, Schevon CA. Field effects and ictal synchronization: insights from in homine observations. Front Hum Neurosci 7: 2013. 851. Wessberg J, Nicolelis MA. Optimizing a linear algorithm for real-time robotic control using chronic cortical ensemble recordings in monkeys. J Cogn Neurosci 16: 1022– 1035, 2004. 852. Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, Kim J, Biggs SJ, Srinivasan MA, Nicolelis MA. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408: 361–365, 2000. 853. Whitacre J, Bender A. Degeneracy: a design principle for achieving robustness and evolvability. J Theor Biol 263: 143–153, 2010. 854. Wiener N. Extrapolation, Interpolation, and Smoothing of Stationary Time Series. Cambridge, MA: MIT Press, 1949. 855. Wilson BS, Dorman MF. Cochlear implants: a remarkable past and a brilliant future. Hear Res 242: 3–21, 2008. 856. Wilson JA, Felton EA, Garell PC, Schalk G, Williams JC. ECoG factors underlying multimodal control of a brain-computer interface. IEEE Trans Neural Syst Rehabil Eng 14: 246 –250, 2006. 857. Wilson MA, McNaughton BL. Dynamics of the hippocampal ensemble code for space. Science 261: 1055–1058, 1993. 858. Wilson MA, McNaughton BL. Reactivation of hippocampal ensemble memories during sleep. Science 265: 676 – 679, 1994. 859. Wise SP. The primate premotor cortex: past, present, and preparatory. Annu Rev Neurosci 8: 1–19, 1985. 860. Wise SP, di Pellegrino G, Boussaoud D. The premotor cortex and nonstandard sensorimotor mapping. Can J Physiol Pharmacol 74: 469 – 482, 1996. 861. Wolf M, Wolf U, Choi JH, Gupta R, Safonova LP, Paunescu LA, Michalos A, Gratton E. Functional frequency-domain near-infrared spectroscopy detects fast neuronal signal in the motor cortex. Neuroimage 17: 1868 –1875, 2002. 862. Wolpaw J, Wolpaw EW. Brain-Computer Interfaces: Principles and Practice. New York: Oxford Univ. Press, 2012. 863. Wolpaw JR, Birbaumer N, Heetderks WJ, McFarland DJ, Peckham PH, Schalk G, Donchin E, Quatrano LA, Robinson CJ, Vaughan TM. Brain-computer interface tech-
836
869. Wu W, Shaikhouni A, Donoghue JP, Black MJ. Closed-loop neural control of cursor motion using a Kalman filter. Conf Proc IEEE Eng Med Biol Soc 6: 4126 – 4129, 2004. 870. Wyler AR, Walker G, Somes G. The morbidity of long-term seizure monitoring using subdural strip electrodes. J Neurosurg 74: 734 –737, 1991. 871. Xia H, Baranga ABA, Hoffman D, Romalis M. Magnetoencephalography with an atomic magnetometer. Appl Physics Lett 89: 211104, 2006. 872. Xu K, Wang Y, Wang F, Liao Y, Zhang Q, Li H, Zheng X. Neural decoding using a parallel sequential Monte Carlo method on point processes with ensemble effect. BioMed Res Int 2014: 2014. 873. Xu Z, So RQ, Toe KK, Ang KK, Guan C. On the asynchronously continuous control of mobile robot movement by motor cortical spiking activity. In: Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE. New York: IEEE, 2014, p. 3049 –3052. 874. Yang L, Leung H. An online BCI game based on the decoding of users’ attention to color stimulus. Conf Proc IEEE Eng Med Biol Soc 2013: 5267–5270, 2013. 875. Yazdan-Shahmorad A, Diaz-Botia C, Hanson TL, Kharazia V, Ledochowitsch P, Maharbiz MM, Sabes PN. A large-scale interface for optogenetic stimulation and recording in nonhuman primates. Neuron 89: 927–939, 2016. 876. Yin E, Zhou Z, Jiang J, Yu Y, Hu D. A dynamically optimized SSVEP brain-computer interface (BCI) Sspeller. IEEE Trans Biomed Eng 62: 1447–1456, 2015. 877. Yoo SS, Jolesz FA. Functional MRI for neurofeedback: feasibility study on a hand motor task. Neuroreport 13: 1377–1381, 2002. 878. Yoo SS, Kim H, Filandrianos E, Taghados SJ, Park S. Non-invasive brain-to-brain interface (BBI): establishing functional links between two brains. PloS One 8: e60410, 2013. 879. Yoo SS, Fairneny T, Chen NK, Choo SE, Panych LP, Park H, Lee SY, Jolesz FA. Brain-computer interface using fMRI: spatial navigation by thoughts. Neuroreport 15: 1591–1595, 2004. 880. Yoon I, Hamaguchi K, Borzenets IV, Finkelstein G, Mooney R, Donald BR. Intracellular neural recording with pure carbon nanotube probes. PloS One 8: e65715, 2013. 881. Young T. A Course of Lectures on Natural Philosophy and the Mechanical Arts. New York: Johnson Reprint Corp., 1971. 882. Yu T, Li Y, Long J, Li F. A hybrid brain-computer interface-based mail client. Comput Math Methods Med 2013: 750934, 2013. 883. Yuan P, Wang Y, Gao X, Jung TP, Gao S. A collaborative brain-computer interface for accelerating human decision making. In: International Conference on Universal Access in Human-Computer Interaction. New York: Springer, 2013, p. 672– 681. 884. Zacksenhouse M, Lebedev MA, Carmena JM, O’Doherty JE, Henriquez C, Nicolelis MA. Cortical modulations increase in early sessions with brain-machine interface. PLoS One 2: e619, 2007. 885. Zander TO, Jatzev S. Detecting affective covert user states with passive brain-computer interfaces. In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops. Amsterdam, The Netherlands: ACII, 2009.
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
847. Weiskopf N, Mathiak K, Bock SW, Scharnowski F, Veit R, Grodd W, Goebel R, Birbaumer N. Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI). IEEE Trans Biomed Eng 51: 966 –970, 2004.
429
430
BRAIN-MACHINE INTERFACES 886. Zander TO, Kothe C. Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general. J Neural Eng 8: 025005, 2011. 887. Zander TO, Kothe C, Jatzev S, Gaertner M. Enhancing human-computer interaction with input from active and passive brain-computer interfaces. In: Brain-Computer Interfaces. New York: Springer, 2010, p. 181–199. 888. Zehr EP. Future think: cautiously optimistic about brain augmentation using tissue engineering and machine interface. Front Syst Neurosci 9: 72, 2015. 889. Zhang H, Ma C, He J. Predicting lower limb muscular activity during standing and squatting using spikes of primary motor cortical neurons in monkeys. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology. New York: IEEE, 2010, p. 4124 – 4127. 890. Zhang R, Li Y, Yan Y, Zhang H, Wu S, Yu T, Gu Z. Control of a Wheelchair in an Indoor Environment Based on a Brain-Computer Interface and Automated Navigation. IEEE Trans Neural Syst Rehabil Eng 24: 128 –139, 2016. 891. Zhang Y, Xu P, Liu T, Hu J, Zhang R, Yao D. Multiple frequencies sequential coding for SSVEP-based brain-computer interface. PLoS One 7: e29519, 2012.
892. Zhao Y, Araki S, Wu J, Teramoto T, Chang YF, Nakano M, Abdelfattah AS, Fujiwara M, Ishihara T, Nagai T. An expanded palette of genetically encoded Ca2⫹ indicators. Science 333: 1888 –1891, 2011. 893. Zhu X, Guan C, Wu J, Cheng Y, Wang Y. Bayesian method for continuous cursor control in eeg-based brain-computer interface. Conf Proc IEEE Eng Med Biol Soc 7: 7052–7055, 2005. 894. Zhuang KZ, Lebedev MA, Nicolelis MA. Joint cross-correlation analysis reveals complex, time-dependent functional relationship between cortical neurons and arm electromyograms. J Neurophysiol 112: 2865–2887, 2014. 895. Zippo AG, Romanelli P, Torres Martinez NR, Caramenti GC, Benabid AL, Biella GEM. A novel wireless recording and stimulating multichannel epicortical grid for supplementing or enhancing the sensory-motor functions in monkey (Macaca fascicularis). Front Syst Neurosci 9: 73, 2015. 896. Ziv Y, Burns LD, Cocker ED, Hamel EO, Ghosh KK, Kitch LJ, El Gamal A, Schnitzer MJ. Long-term dynamics of CA1 hippocampal place codes. Nature Neurosci 16: 264 – 266, 2013.
Downloaded from http://physrev.physiology.org/ by 10.220.33.4 on May 9, 2017
Physiol Rev • VOL 97 • APRIL 2017 • www.prv.org
837
431