the neuroethics and law issue | volume 05 issue 03 | february 2018
Neurorights: When is it Legitimate to Gain Access to or Interfere with Another Person’s Brain? Ariana Tang pg. 08
The History of Neurocriminology Admissibility of Neurotechnologies in Court Marietou Daou pg. 10
Dorsa Rafiei pg. 09
INTERNEURON
© 2018 INTERNEURON UOFT ALL RIGHTS RESERVED
2
Volume 5. Issue 3. February 2018
table of contents 04
editor’s note + editorial board
06
neurolaw: its history and implications pascale tsai
07
“guilty or not guilty” in the light of neuroscience ayesha kawser
08
neurorights ariana tang
09
history of neurocriminology dorsa rafiei
10
admissibility of neurotechnologies in court marietou daou
12
to what extent can ethical concerns be compromised? aisha adil
14
to interfere or to not interfere: the extent of neuroscience application saloni gupta
15 16
ethical issues in child neurology aisha adil neuroscience and public policy bailey mcmaster
18
wanted: neuroscience students in law school carol chen
19
alexa, AI, and amendments yun kim free will: a brief glance and perspective david tran
20
3
INTERNEURON
Editor’s Note It is with great pleasure that Parandis and I, along with our entire Interneuron executive team, present to you issue 3 of this year’s quarterly cycle.
Parandis & Waleed Co-Editors-in-Chief
As with any issue of Interneuron, this quarter, we spent a considerable amount of time debating what constitutes the most pressing area of neuroscience that we should bring to your attention. Since our last issue focused on neuropsychiatry and mental health, offering a very personal perspective of the brain, this issue, we decided to take a more macroscopic viewpoint by providing a more social perspective of the brain, the kind of conundrums it creates for us, and how we can solve them, using the tools of neuroscience. It is no mystery that as we are progressing towards a better understanding of ourselves, our society and institutions are facing an urgent need to adapt in order to better accommodate the very people it was created to serve. There can be no better example than the legal system itself, where eyewitness testimony, once a gold standard for evidence, is now gradually being understood as unreliable. Such changes have profound implications for both us as a society and as individuals, yet none of these realizations would have ever come to limelight had it not been the tireless work of neuroscientists and psychologists working to understand the human brain and behaviour. It is for this reason we would like to dedicate issue 3 to address some of the ethical and legal challenges faced by society today and how neuroscience is playing a key role in solving them.
Contributors Aisha Adil Carol Chen Marietou Daou Saloni Gupta Ayesha Kawser Yun Kim Bailey McMaster Dorsa Rafiei Ariana Tang David Tran Pascale Tsai
4
In the end, both Parandis and I would like to thank all our contributors and our executives for their tireless efforts, without whom this issue would not have been possible.
Volume 5. Issue 3. February 2018
editorial board 2017-2018
parandis kazemi | EIC
clara hong | submissions
waleed khan | EIC
belinda hoang | layout
wazaira khan | submissions
andrea macanovic | submissions
kayla liu | layout
amar dholakia | marketing
5
INTERNEURON
Its History and Implications Pascale Tsai Neuroscience, the multidisciplinary study of the brain and nervous system, began over 100 years ago, with the work of scientists such as Aristotle and Hippocrates [1,2]. Over time, stemming from the goal of understanding the basic functions of the brain as an organ, the development of technologies like CT scans and fMRIs allowed for the examination of the human brain [3]. Discoveries of the mechanisms of human action by the brain as well as the conscious decision-making process led towards the advancement of neuroethics. Currently, neuroethicists aim to consider how human responses may be controlled by the brain before conscious awareness, and in following, what it means to be an acting human being [3]. With neuroethics, researchers seek to understand how individuals form intentions, and think of themselves and others. Over the past 10 years, the integration of neuroscientific principles with ethical research has branched into neurolaw, an interdisciplinary field linking the scientific study of the brain with the legal system [4]. Specifically, neurolaw explores the effects of discoveries in neuroscience on legal rules. While the goal of a legal system across different nations is relative to justice and the realization of human knowledge, neuroscience is based on more absolute propositions of memory, addiction, and motivation [5]. Neuroscience is based on experimental evidence, while law is considered to be a science of the humanities, built upon societal obligations [5]. Consequently, through the intersection of the two fields, and consideration of the role of the brain in human decisions, more equitable and fair legal decisions can be made in trial. Neurolawyers are tasked with using technologies like radio-imagery, as well as memory detection and polygraph tests, to study criminal neuropsychology [5]. By doing so, they examine the neurobiology and potential causes of human behaviors and incriminating activities. The use of neuroscientific principles and ethics in the legal system henceforth addresses issues like the responsibility of the person for their behaviour, their mental state at the time of a criminal act, and the effects of their emotions on their memory, behaviour and motivational drive [4]. Neurolaw has broadened the amount of examinable evidence in legal procedures, allowing for greater substantiation of the claims and decisions made during criminal investigation.
References
1. Bernacer, J., & Murillo, J. I. (2014). The Aristotelian conception of habit and its contribution to human neuroscience. Frontiers in Human Neuroscience, 8, 883. http://doi.org/10.3389/fnhum.2014.00883 2. Breitenfeld, T., Jurasic, M.J. (2014). Hippocrates: The forefather of neurology. Neurol Sci, 9, 1349-1352. doi: 10.1007/s10072-014-1869-3. 3. Gazzaniga, M.S. (2008). The law and neuroscience. Elsevier, 60, 412-415. http://dx.doi.org/10.1016/j.neuron.2008.10.022 4. Jones, O.D., Marois, R., Farah, M.J., Greely, H.T. (2013). Law and neuroscience. Journal of Neuroscience, 33(45), 17624-17630. doi: 10.1523/JNEUROSCI.3254-13.2013 5. Petoft, A. (2015). Neurolaw: A brief introduction. Iranian Journal of Neurology, 14(1), 53–58.
6
Volume 5. Issue 3. February 2018
“GUILTY
OR NOT GUILTY” in The Light of Neuroscience Ayesha Kawser
What do you expect from the legal system in response to criminal activity? Can crime be forgiven if there is significant evidence of abnormal brain activity? Even if there is important data that proves a crime to have been committed because of mental deterioration, to what extent will you believe it? Neurolaw, a complex intersection of neuroscience and law, focuses on neurological techniques to establish biological cause for a certain atypical behavior [2]. According to Nita Farhany, a professor at Duke University, around 20% of court cases from 2005-2012 have had criminal defendants pleading innocent because there was evidence of mental impairment [1]. Criminal defense lawyers have often been implicated in “ineffective assistance of counsel” on the basis of ignored neuropsychological tests [1]. If any criminal is deprived of their constitutional rights because an incompetent lawyer ignored contributions like DNA testing, then it can be labelled as “ineffective assistance of counsel” [1]. Nita Farhany also pointed out that many lawyers are using neuroscience inappropriately to avoid punishment for their clients, even in high-profile cases such as homicide, rape and murder [1]. Another researcher at Harvard University, Joshua Buckholtz, says that much of neuroscientific data has been interpreted within individual dif-
ferences [1]. Individual differences mean when any difference is visible in one person compared to other people who share many similar qualities [1]. For this reason, a certain neurological result from a single person cannot explain the same behavior in a different person [1]. According to Stephen Morse, a professor at the University of Pennsylvania, neuroscience is not being used in law as expected since neuropsychological data is only used to support behavior that can be “seen” [2]. For example, a brain-damaged person’s crime can be justified under the law, whereas the same person’s neuroscientific data indicating a damaged brain will not be of any use if there is no observed deviant behavior, even though there might have been a crime committed by him/her [2]. Although there are many contradictory perspectives, neuroscience is affecting the law around the world quite rapidly [1]. The application of neuroscience in the field of law is vast, but not that analytical and accurate until now [2]. However, because of the speed of development of neurolaw and researchers’ efforts, neurolaw promises us a future where crime can be critically and precisely investigated further in the light of neuroscience.
References
[1]Sukel, K. (2013). Beyond Sentencing: How Neuroscience Has Already Changed the Legal System. [online] Dana.org. Available at: http:// dana.org/News/Beyond_Sentencing__How_Neuroscience_Has_Already_Changed_the_Legal_System/ [Accessed 25 Jan. 2018]. [2]Malcolm, L. and Willis, O. (2016). Can neurolaw change the criminal justice system?. [online] Radio National. Available at: http://www.abc.net.au/radionational/programs/allinthemind/can-neurolaw-change-the-criminal-justice-system/7323610 [Accessed 25 Jan. 2018].
7
INTERNEURON
By Ariana Tang
Neurorights
When is it Legitimate to Gain Access to or Interfere with Another Person’s Brain?
What would happen if we could gain an all-access pass to someone’s brain? The Netflix anthology series Black Mirror has posed this question; without giving away too many details, untethered access to the mind has its own dark side. While it seems to be in the realm of science fiction, advances in neuroscience have made this an increasingly possible reality. Enter neurorights: a new category of human rights. Neurorights is comprised of 4 fundamental human rights 1) Cognitive liberty – it is based on the concept of “freedom of thought”. It involves 3 “dimensions” – the right to choose whether or not to use technologies to alter the brain; and the ability to change one’s mind regarding this decision, protection of interventions into other minds for promotion of neural integrity, and the promotion of cognitive liberty¹. 2) Mental privacy – this involves protection against unauthorized access into the brain, and to prevent spread of this information1. It is essentially the “right to privacy”, but the personal information is your own mind¹. 3) Mental integrity – this is the right to protect the mind from being harmed by any means, and to guarantee the right for individuals with mental health issues to access whatever treatment is needed¹. 4) Psychological continuity – this is the right to preservation of one’s identity and psychological state¹. Essentially, it disallows the ability to interfere with what makes you, you; including your thoughts, your hobbies, dislikes and so on¹. This sounds like something that will be helpful in the future – i.e. to prevent robots from turning the human race into their footstools. However, access to the human mind is already starting to become a reality. For fans of the movie Matilda, imagine being able to move objects by just thinking of moving them, Matilda-style. This is the idea behind a brain-computer interface, which enables people to interact with their environment via brain signals – including electroencephalography (EEG), electrocorticography (ECoG), and signals from other regions of the brain². While research is relatively young, it is rapidly expanding: Tesla’s Elon Musk recently announced the start of a new company, Neuralink, which aims to create a brain-computer interface, and he is not the first (here’s a fun comic with more details: https://waitbutwhy. com/2017/04/neuralink-cleanversion. html). Research has shown its poten-
8
tial, especially in movement rehabilitation³. However, allowing a computer to essentially read the mind has raised some important questions. For example, what would happen if hackers could hijack this interface and force the user to perform tasks against their own will? This puts the individual’s neurorights at great risk. Unfortunately, this has been seen with the best of medical intentions – an example is a study involving deep brain stimulation in Parkinsonian patients. Although they displayed significant improvements in their Parkinsonian symptoms, there were unintended side effects such as feeling “like a robot/electronic doll”, feelings of unfamiliarity, loss of a feeling of purpose in life, and marital issues⁴. These novel neurotechnologies could nevertheless be revolutionary in areas like law. For instance, a study investigating the relationship between
anterior cingulate cortex activity and reoccurring criminal behavior concluded a potential neurocognitive biomarker for antisocial behavior⁵. It also opens up a whole new world of possibilities for crime prevention. One could essentially “read” the mind of a suspect and prevent events such as terrorist attacks. However, this sounds like, and could potentially lead to a dystopian world, reminiscent of George Orwell’s “1984”. “Thou canst not touch the freedom of my mind” is an oft-quoted line from Comus by John Milton in reference to neurorights. It communicates the fortress-like, autonomic quality that is so commonly associated with the mind: one that neurorights aims to hold the integrity of. Until neurotechnologies no longer endanger the right for us to think freely, we should proceed with great caution.
References 1. Ienca M, Andorno R. Towards new human rights in the age of neuroscience and neurotechnology. Life Sciences, Society and Policy. 2017 Apr 26;13:5. 2. Shih JJ, Krusienski DJ, Wolpaw JR. Brain-Computer Interfaces in Medicine. Mayo Clin Proc. 2012 Mar;87(3):268–79. 3. Daly JJ, Wolpaw JR. Brain–computer interfaces in neurological rehabilitation. The Lancet Neurology. 2008 Nov 1;7(11):1032–43. 4. Aharoni E, Vincent GM, Harenski CL, Calhoun VD, Sinnott-Armstrong W, Gazzaniga MS, et al. Neuroprediction of future rearrest. PNAS. 2013 Apr 9;110(15):6223–8. 5. Schüpbach M, Gargiulo M, Welter ML, Mallet L, Béhar C, Houeto JL, et al. Neurosurgery in Parkinson disease: a distressed mind in a repaired body? Neurology. 2006 Jun 27;66(12):1811–6.
Volume 5. Issue 3. February 2018
History of Neurocriminology By Dorsa Rafiei
“
What is it that makes someone commit a crime? Is it free will? Or is it out of the i n d i v i d u a l’ s control, and instead due to genetic and/ or neurological predispositions?
”
Imagine we lived in a world where all eighteen-year-old males were obligated to undergo neuroimaging tests in an attempt to predict how likely they are to engage in criminal activity. This provocative and controversial idea was suggested by Adrian Raine, a leading neurocriminologist, in his book The Anatomy of Violence (1). During the 19th century, Cesare Lombroso, the father of scientific criminology, hypothesized that biological factors (both physical and psychological) served as predictors for criminal behavior (2). Since then, there has been great debate over whether criminals are born or formed. What is it that makes someone commit a crime? Is it free will (3)? Or is it out of the individual’s control, and instead due to genetic and/or neurological predispositions (1)? This is where the study of neurocriminology comes in, by looking into the neurobiology behind the crime. Since the 1980s, there have been major advances in neuroimaging technology that have allowed researchers, such as Raine, to study the brains of violent offenders (2). These offenders are commonly diagnosed with antisocial personality disorder (APD), or psychopathy, and are characterized by aggressiveness, impulsivity, emotional deficits, lack of empathy, and manipulative behavior (1). They go after anything and anyone, looking to quench their thirst
for stimulation while showing no remorse (1). Brain scans have shown that these individuals have reduced grey matter in the prefrontal cortex and reduced amygdala volume, accounting for their lack of executive control and their emotionless behavior, respectively (1). It is not so simple, though. Other neurobiological factors have also been tightly linked to APD, such as hormones (namely high testosterone and low cortisol), neurotransmitters (e.g. low serotonin), genes such as MAO-A, alongside social and environmental factors (4). In spite of the need for improvements in neuroimaging and a deeper understanding of brain circuits, there is promise in the way that we can predict and prevent crime by incorporating neuroscience into law (4). The outlined neurobiological characteristics can not only be used to assess whether someone is predisposed to criminal activity, but also their risk of re-offending and which offenders would benefit the most from rehabilitation programs or pharmacological intervention (1,4). Furthermore, it can impact prison sentencing depending on whether the offender is “responsible” for his behavior (4). As Raine suggests, neurocriminology still has a long way to go in the way that it influences law, but holds great predictive power and has major implications in preventing violence.
References: Dahl O. Neurocriminology: The Disease behind the Crime [Internet]. DUJS Online. 2018. Available from: http://dujs.dartmouth.edu/2013/11/neurocriminology-the-disease-behind-the-crime/#.WmyxnDZjrwy Are murderers born or made? [Internet]. BBC News. 2018. Available from: http://www.bbc.com/news/magazine-31714853 Cook G. Secrets of the Criminal Mind [Internet]. Scientific American. 2018. Available from: https://www.scientificamerican.com/article/secrets-criminal-mind-adrian-raine/ Glenn A, Raine A. Neurocriminology: implications for the punishment, prediction and prevention of criminal behaviour. Nature Reviews Neuroscience. 2013;15(1):5463
9
INTERNEURON
BY Marietou Daou
Admissibility of
Neurotechnologies in Court
In June 2008, a woman in Mumbai, India was convicted for the murder of her fiancĂŠ based on neuroscientific evidence that she had experiential knowledge of poisoning him (3). This conviction was overruled afterwards, but this case marked an early example of the use of neurotechnologies in court. Neuroimaging has been widely used in clinical and research settings. Now its application is expanding in other areas of society, including marketing and the judicial system. Experiences and knowledge leave a distinctive trace in our brains that can be detected by neuroimaging techniques such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) (3). Based on these technologies, US researchers developed different methods to assess a brain signature for lying, including brain fingerprinting, the guilty knowledge test, and the concealed information test (3). The brain fingerprinting test looks for knowledge of evidences or events rather than testing for lies. It is based on the use of the EEG technology
10
to record electrical neural signals in response to visual stimuli or questions. The goal is to assess prior knowledge and memory through the identification of neural signatures for memories; these brain signatures indicate the recognition of pieces of information1. Meanwhile, the guilty knowledge test is based on fMRI and aims to distinguish the neural substrates that correspond to truth-telling from the neural substrates that correspond to lie-telling (3). Theoretically, these approaches should be more reliable than lie-detection machines like polygraphs: polygraphs rely on physiological responses such as heart rate and blood pressure and are more susceptible to confounding variables. The applications of neurotechnologies in court are multiple: in fact they can be used to assess thoughts and intentions, distinguish between intentions and behaviour, identify subconscious (forgotten) witnesses’ memories and jurors’ biases, and even predict likelihood of recidivism. Therefore, there is potential for these technologies not only in criminal cases but also in civil cases
References
Volume 5. Issue 3. February 2018
and contracts stipulations. Despite their potential, neurotechnologies have not yet been introduced widely in courts. First, it is important to establish how reliable such technologies are, and to guarantee their accuracy test set-up is fundamental; in particular we have to consider how the question or the stimuli are presented and how the brain signals are interpreted. That is, we need to consider how we distinguish between the actual memory for experiencing or witnessing something and the memory of something being told or thought. However, since this becomes subject to varying interpretations, it is apparent that neurotechnologies are not as bullet-proof as they may seem. Moreover, given the complexity of neural substrates, it is evident that experiential knowledge does not necessarily mean guilt; for example, it has been shown that the same circuits activated while performing an action are activated while thinking of performing it. Therefore it is important for the jury to understand how to properly weigh the neuroscientific evidence presented among the rest of the evidence. Researchers, concerned with the potential bias neuroscientific evidence could have on jurors, conducted an experiment to assess how accurate such a concern was. However, a mock trial-based research showed that the use of neurotechnologies is not sufficient for a conviction but can contribute to build a strong case in conjunction with the context in which is introduced (3). Even though the mock trial ruled out
the potential for prejudice, there still are some concerns that may prevent the established use of brain fingerprinting in courts. For instance, this neuroscientific advancement might result in the need for formulation of a new human rights code on “neurorights”; if the subjects were unwilling to undergo EEG or fMRI testing, would it be necessary to acquire a warrant? Could it be considered as a more sophisticated form of torture, i.e. one less violent but much more pervasive? This is the start of a conversation about “cognitive liberty” and the right of the legal system to threaten individual privacy and mental freedom (4). Cognitive liberty is defined as “the right of individuals to make free and competent decisions about the use of neurotechnologies” (2). On the other hand, a right to mental privacy would protect individuals against the unconsented intrusion by third parties into their brain data as well as against the unauthorized collection of that data (2). Perhaps the invisible invasiveness of neurotechnologies might be in opposition to the principle against self-incrimination, which is a right recognized in some countries not “to an accusation or charge of crime; to involve oneself or another [person] in a criminal prosecution or the danger thereof ” (1). All things considered, there is a great potential in the field of neurolaw, but we still have a long way to go and various issues to sort out before brain fingerprinting and other neurotechnologies can be used as a reliable neurolegal resources.
1. Black’s Law Dictionary (5th ed.). 1979. p. 690. 2. Ienca, M. and Ienca, M. (2018). Do We Have a Right to Mental Privacy and Cognitive Liberty? [online] Scientific American Blog Network. Available at: https://blogs.scientificamerican.com/ observations/do-we-have-a-right-to-mental-privacy-and-cognitive-liberty/ [Accessed Jan 2018]. 3. Neuroscience News. (2018). If a Brain Can Be Caught Lying, Should We Admit That Evidence to Court? [online] Available at: http://neurosciencenews.com/neuroimaging-lying-court-7249/ [Accessed Jan 2018]. 4. Rosen, J. (2018). Neuroscience - Law - The Brain on the Stand - Jeffrey Rosen. [online] Nytimes.com. Available at: http://www.nytimes.com/2007/03/11/magazine/11Neurolaw.t.html [Accessed Jan 2018]. 11
INTERNEURON
What makes studying the brain special? Often referred to as the “seat of the mind”, the brain is the final common destination for expressing our mental activities1. However, given that it is the most complex organ, comprising of numerous interdependent parts and functions, the brain is very challenging to study, especially when attempting to experiment and conduct research on it. Introducing any form of intervention may result in several consequences; cognitive, emotional, physical, or even consequences on one’s own personality1. As neuroscience research is moving forward, our potential for understanding the brain in ways never understood before is as well. However, many ethical concerns are being voiced over the methods used to conduct such research— how is research on animal models being translated into human models? How is consent being obtained from patients with severe mental deficits or from vulnerable populations? How is neuroscience research being used for drug testing by private-sector companies? Studies involving mice face several neuroethical challenges especially when neural-mediated treatment is being tested with no existing human disease in need of treatment2. Invasive animal work also involves directly manipulating animals’ natural cognitive and emotional mech-
anisms2. In the case of humans and mice specifically, one of the fundamental challenges is that it is similar enough to humans to justify its own validity as an experimental model, but it is not the same in its capacity for pain and suffering as humans2. However, much of the ethical concerns commonly appoint obtaining informed consent as one of the major challenges in biomedical research. Many clinical brain research participants have cognitive and/or emotional impairments, rendering them as easy targets for coercion or persuasion given they may not be fully aware of what they are consenting to2,3. In 2015, the BRAIN initiative project was upstarted to map and understand the brain to learn more of its functions3. The Presidential Commission for the Study of Bioethical Issues, in regards to this project, voiced the crucial need for ensuring patients have a legal substitute decision maker3. While such a viable option is available for obtaining consent, children are also under this vulnerable population and require a decision maker. The Bucharest Early Intervention Project raised several, serious ethical concerns for its study on comparing cognitive development in socially deprived children4,5. Abandoned children growing up in Romanian orphanages were compared to children (with similar conditions)
To What Extent Can Ethical Concerns Be Compromised in Neuroscience Research?
Aisha Adil
12
Volume 5. Issue 3. February 2018
who moved into foster care4,5. Researchers deduced that early institutionalization of children reared negative outcomes on cognitive development as children who moved into foster care had better cognitive development than those institutionalized5. However, these children were doubly vulnerable; they were unable to give informed consent and were institutionalized5. Although seemingly unethical, researchers proved the study be ethical under certain conditions; the children would not have been worse conditions if they had not partaken in the study, and the Romanian government was informed of the study’s results who has since restricted the institutionalization of children under 2 unless severely handicapped5. Drug testing and addiction research on the other hand present manifold ethical problems. Whether addicted individuals are abstaining or not, they are still compromised in their ability to control cravings and avoiding harmful inducements1. Moreover, testing brain-targeting drugs approved for adults in children or adolescents heightens the risks and harmful effects these children and adolescents may experience. Similar stances can be taken for the elderly given that brain development varies across different life stages1.
However, several reports have indicated that private-sector businesses targeting drug testing, test drugs on undocumented immigrants and individuals who lack proper treatment and care5. In parts of the United States, there are “professional” subjects for drug testing—subjects who are economically disadvantaged5. These actions are clear violations of the Belmont report, a guideline for administering proper ethical conduct on research subjects as these businesses are selecting only “undesirable” individuals for drug-testing research5. However, neuroethicist, Judy Illes, from the University of British Columbia, states that “private sector research has no generally required ethical guidelines and is not obligated to follow the Belmont report…”5. Additionally, Illes points out that there is little oversight over private research. This raises questions for the societal implications such research may have—are the results from private research really applicable to the general population? What impacts can these studies have on individuals and society as a whole? Especially in studies focusing on the brain and behavior, generalizing results from such studies can involve changes in law enforcement, employment, educational institutions and more who may not always use that knowledge for the best interest of individuals1.
References
1. Leshner, AI. Ethical Issues in Taking Neuroscience Research from Bench to Bedside. The Dana Foundation [Internet]. 2004 Oct 1 [cited 2017 Jan 27]. Available from: http://www.dana.org/Cerebrum/Default.aspx?id=39167 2. Cassady, HJ. What’s Special about the Ethical Challenges of Studying Disorders with Altered Brain Activity?. Curr Topics Behav Neurosci. 2015;19: 137–157. DOI: 10.1007/7854_2014_333 3. Loria, K. We’re learning more about the brain than we’ve ever known before, but that raises three serious ethical questions. Business Insider [Internet]. 2015 Mar 26 [cited 2017 Jan 26]. Available from: http://www.businessinsider.com/brain-initiative-ethical-issues-in-neuroscience-2015-3 4. Nelson, CA. Cognitive recovery in socially deprived young children: the Bucharest Early Intervention Project. Science. 2007;318(5858):1937-40. DOI: 10.1126/science.1143921 5. The ethical neuroscientist. Nat. Neurosci. 2008;11:239. DOI: 10.1038/nn0308-239
13
INTERNEURON
To Interfere or To Not Interfere: The Extent of Neuroscience Application
Saloni Gupta
You’re about to write an exam and you haven’t studied much. You didn’t show up to class and you’re regretting just about every decision you made this semester. As you’re herded into EX100, your friend presses a little blue pill in your hand and says, “Relax - here’s your 4.0.” Do you take it? In a world where such ‘smart pills’ are propelling research on cognitive enhancers, it becomes vital that ethical limits and restrictions are established in order to preserve morality within the discipline of neuroscience. In response to this, the Dana Foundation met in 2002 to create the discipline of neuroethics. William Safire defines it as “the examination of what is right and wrong, good and bad about the treatment of, perfection of, or unwelcome invasion of and worrisome manipulation of the human brain.” (1) This seemingly new discipline marries cognitive science, neuroscience, and philosophy to address old problems with neuroscientific interference, whether it’s the lobotomies of the 1960s or fMRI-based mind reading. The discipline divides itself into the ‘neuroscience of ethics’ and the ‘ethics of neuroscience.’(2) While the former is based on understanding the biological mechanisms behind morality or values, the latter evaluates the morality of using neuroscience to affect people. Contentious debates regarding ethics and morality are taking place regarding both current and advancing neurotechnologies. For instance, a drug for ADHD, Ritalin, is commonly abused by students to enhance their focus. (2) Should a student be able to artificially enhance their confidence or motivation? What if they could enhance their memory or public speaking skills? As these become realities, ethicists worry of the impact and influence of neuroscience within non-clinical, unregulated settings such as these. Cognitive liberty remains another area of concern to neuroethicists. The development of neuroscience brings with it the potential to interfere with the conscious. And while autonomy and privacy are elementary to us, such potential for interference changes these assumptions. While it would be great to remove an implicit racial bias or to learn what a potential terrorist’s intentions are (2), if indeed we could use fMRIs to manipulate thoughts or to predict future actions (3), then what would become of freedom of thought? What would become of the right to privacy? The modification of the ‘self ’ is perhaps the most ambiguous part of the field. Despite centuries of research, the scientific community is still unable to identify which qualities and characteristics make us individual. If we could artificially induce characteristics such as humor or love, we would be blurring the lines even further. The future of neuroscience brings the possibility of CRISPR and optogenetics methods where we can noninvasively edit and control neurons (2). These possibilities lead to yet another important question: who in our society can be trusted to control and handle such power? At a time where individuals can be manipulated by such neurotechnologies, it is imperative that the exciting field of neuroethics continues to explore these questions, ensuring that the field of neuroscience prospers and serves the good.
References
1. Neuroethics: Mapping the Field [Internet]. Dana.org. 2018 [cited 28 January 2018]. Available from: http://dana.org/Cerebrum/2002/Neuroethics__Mapping_the_Field/ 2. Roskies A. Neuroethics [Internet]. Plato.stanford.edu. 2018 [cited 28 January 2018]. Available from:https://plato.stanford.edu/entries/neuroethics/#RiseScopNeur 3. Illes J, Bird S. Neuroethics: a modern context for ethics in neuroscience. Trends in Neurosciences. 2006;29(9):511-517.
14
Volume 5. Issue 3. February 2018
Ethical Issues in A Review by Aisha Adil Approaching ethics in child neurology systemically is an area very few have explored. David Coulter examines the tools needed to identify and tackle moral issues that may arise in paediatric neuroscience research. Overall, Coulter argues physicians’ need for an “ethical toolkit” where physicians can withdraw knowledge from the major ethical theories needed for current medical practice1. Utilitarianism suggests physicians act in a way that renders a net beneficial outcome when accounting the negative and positive consequences of an act or situation. While seemingly an intuitive approach, Coulter remarks that physicians are challenged in applying this principle in paediatric medical practice. If medical support is removed from a child in a vegetative state, for example, the child will be harmed because they will not be able to live1. However, this act may benefit the family in their grieving, and benefit society by providing resources for other patients1. Furthermore, this theory can be used to justify the killing of defective newborns1. Though, this act itself presents morally objectionable stances from individuals who hold differing moral beliefs1. Coulter also notes this theory’s failure to protect minorities1. Children with rare neurological conditions can be unheeded for their needs because in seeking the greater benefit for the majority, using scarce medical resources for these patients will be costly relative to children with common disorders1. Conversely, the deontological theory focuses on achieving universal decisions that can be applied in relevant situations1. While being dutiful and obligatory, two key emphases of this theory, Coulter points out that the intricacies of human relationships are
disregarded; a conflict with care-based ethical approaches1. Care-based ethics appoint moral value to emotions, patients’ relationships, and the level of interdependence between patients and their relationships, a much-needed approach in child neurology1. Coulter also notes principlism, an approach focussing on beneficence, nonmaleficence, justice, and autonomy. While beneficence requires physicians to act in the best of interest of the patient, nonmaleficence emphasizes the avoidance of harm done unto patients. Autonomy appoints individuals as rationally capable of governing their own actions and decisions, however, children are not considered autonomous and require their decisions to be made by legal guardians1. Child neurologists struggle in harmonizing the conditions of these principles, the opinions about neurological and medical data, and opinions about the ethical issues pertaining to each unique case1. However, the physician’s own opinion about ethical issues associated with any patient’s case is not the most important—it is usually the child’s family as well as the views of extended family, friends and/ or religious or cultural views1. Here, Coulter notes that child neurologists and affiliated physicians and nurses are challenged in ensuring their own ethical concerns are not imposed or do not override the patient’s family and friends’ opinions1. Coulter also notes that while accounting for these theoretical frameworks, physicians are challenged in their decisions given that they rely on their previous experiences with other patients and must think in different perspectives to offer the “best fit” treatment1.
References
1. Coulter DL. Ethical Issues in Child Neurology. Swaimans Pediatric Neurology. 5th ed. Saunders; 2012. e246. doi:10.1016/b978-1-4377-0435-8.00083-4
15
INTERNEURON
Neuroscience & As the study of neuroscience continues to expand into the medical field, it can only be expected that the results of these studies extend to other fields as well. This can be seen in public policy issues like mental health law, family law, and drug policy. In these highly contentious issues, neuroimaging and neuroscience studies are frequently cited to explain the reasoning behind policy-makers’ decisions. However, the use of these studies is also often criticized. This article looks at three public policy matters that neuroscience has played a role in. With the upcoming legalization of cannabis in Canada, there is the highly controversial ongoing debate of legal minimum age for use. The Federal Task Force on Cannabis Legalization and Regulation recommended the age of 18, to discourage youth from unregulated supply and reducing the number of youth criminalized for cannabis possession and use (1). For policy-makers who seek to set the age higher than 18, neuroscience is a tool that is often used to make their argument. The main piece of evidence used is that cannabis can be harmful to developing brains. In response, neuroscientists criticise this generalization of what research truly shows, as it does not address confounding factors such as alcohol and poly-substance use. While the neuroscience literature does support that
age of onset of use and frequency of use in young ages may be harmful, this science should not be misused. Rather than having the information relayed by policy-makers, neuroscientists should be involved in the discussion around policy and preventative action to ensure neuroscience is communicated truthfully and effectively (2). The field of family law has also seen an increase in the use of neuroscience. As explained by Schore and McIntosh, this is primarily based in interpersonal neurobiology and attachment theory. These two models are highly interconnected in their explanation of infant development. Interpersonal neurobiology explains how relationships early in life shape the development trajectory of the brain. Attachment theory is a psychological theory with the essential tenet that an infant needs a relationship with a primary caregiver to successfully achieve social and emotional development. Both theories, from a psychological and neurological standpoint, state that development of the child’s brain occurs both pre- and postnatally. Prenatally, the mother’s cortisol levels pass through the placenta to the baby, influencing stress levels in the fetus. After birth, the mother typically regulates the stress response of the child through consistent and predictable care. After the first year, both the primary and secondary care-givers play a
References 1. A framework for the legalization and regulation of cannabis in Canada: the final report of the Task Force on Cannabis Legalization and Regulation. Ottawa: Health Canada; 2016. 2. Haines-Saah RJ, Jenkins EK. Setting the Legal Age for Access to Cannabis in Canada: Bridging Neuroscience, Policy, and Prevention. Neuropsychopharmacology. 2018;43(1):213–4.
16
Volume 5. Issue 3. February 2018
PUBLIC POLicy
By Bailey McMaster
role, adding in up-regulation of playful and happy mood states. These attachments allow the infant brain to develop on a healthy trajectory, especially until the end of the massive brain growth spurt between ages three and four. Disruption of these relationships can be detrimental in the child’s future ability to react in stressful situations and control aggression. In the context of family law, these findings can be essential in the decision-making about custody. From a neuroscientific perspective, 50/50 care is not sufficient in allowing the child to make these essential connections early in life. Custody of the child should go to the caregiver that can most effectively provide consistent and predictable care and can intuitively be aware of the child’s needs and address them accordingly. In cases of abuse and neglect, the removal of the child from stress-inducing situations is especially important. The high levels of cortisol associated with these situations are likely to hinder the child’s ability later in life to deal with stressful situations as well (3). Finally, the field of mental health law has been using neuroscience in one of its most highly contentious issues. McSherry states that with many countries enforcing laws regarding compulsory detention and substituted decision-making for people who have a severe mental illness, some ar-
gue that these laws are restrictive, give into stereotypes about people with mental impairments, and violate their human rights. The use of neuroscience in mental illness seeks to lead to better treatment options, a better understanding of the underlying causes, and reduction of stigma surrounding people with mental illness. However, when translating this information to the mental health law context, the use of neuroscience faces criticism. With the neuroscientific description of mental impairment as dysfunctional brain processes affecting decision-making, the laws surrounding substituted decision-making seem justified. It is argued that using neuroscience in issues this complex is reductive, as psychosocial factors can be equally important in these illnesses. Some have criticized neuroscience for its allure, as people believe dependency on the science has developed beyond where the field truly stands, impacting the judgment of its use (4). In public policy, the introduction of neuroscience can have both positive and negative consequences. As public policy uses neuroscience in future disputes, the spread of this information should be regulated by experts in the field, as scientific communication can be misleading and intimidating when misused.
References 3. Schore A, McIntosh J. Family Law And The Neuroscience Of Attachment, Part I. Family Court Review. 2011;49(3):501–12. 4. McSherry B. Decision-Making, Legal Capacity and Neuroscience: Implications for Mental Health Laws. Laws. 2015;4(4):125–38.
17
INTERNEURON
The study of the human brain has been rapidly growing in popularity in the past decade.1 In this age of “brain culture,” law schools are now incorporating the study of neuroscience to better understand human behaviour and improve legal rules in court. Having a background in neuroscience is highly useful for the critical mindset needed for law school. In court, lawyers attempt to determine the hardest questions about the reason and mentality behind people’s actions: how and why a person behaved the way they did, the accuracy of a person’s memory, and the person’s mental state at the time of act. Science students are taught to critically think and draw conclusions from evidence, and law schools look for those same analytical and logical thinking skills in law students.2 Understanding the neural and biological mechanisms of decision-making can make you a more fair and effective lawyer. For in-
stance, one model used in third-party punishment behaviour in courts involved decoding the network of prefrontal, temporoparietal, and corticolimbic brain regions. Neuroscience-based predictions can improve your ability to predict a suspect’s future violent behavior, current states of mind, and even deceptive or emotional tactics.3 Law schools are now adding courses on how to use neuroimaging evidence to support case arguments. In 2010, the EEG evidence showing brain abnormalities for Grady Nelson, who had murdered his wife, saved him from an almost inevitable death sentence.4 Neuroimaging technologies can reveal the early signs of mental illness, the immaturity of an adolescent’s brain, and physical evidence of an emotional response to a traumatic event.5 Scientists are being hired by law schools and courts to improve their teaching curriculum and investigate the fairness of the judicial system.
One cognitive research study showed that using visual aids increased critical-thinking skills in law students.6 Law schools are collaborating with medical research groups to investigate the use of drugs and treatments on enhancing students’ analytical skills and even improving witnesses’ memories in the court.7 An increasing number of schools are also offering courses and joint programs in neuroscience and law. In 2012, the University of Wisconsin-Madison and University of Vanderbilt both launched joint J.D. and PhD programs in neuroscience and law. Top universities such as Harvard and Stanford have added multidisciplinary programs combining law, science and technology in recent years.8 Unsure of what to do with a neuroscience degree? Having that strong science foundation may just be the right key in the lock to law school.
Wanted: Neuroscience Students in Law School Carol Chen Image source: www.rcc.int References
18
1. Andy Wai Kan Yeung, Tazuko K. Goto, W. Keung Leung. (2017). The Changing Landscape of Neuroscience Research, 2006–2015: A Bibliometric Study. Frontiers in Neuroscience, 11 DOI: 10.3389/fnins.2017.00120 2. Jones O, Marois R, Farah MJ, Greely H. (2013). Law and Neuroscience. Journal of Neuroscience 33(45): 17624-17630. DOI: https://doi.org/10.1523/JNEUROSCI.3254-13.2013 3. Buckholtz JW, Asplund CL, Dux PE, Zald DH, Gore JC, Jones OD, Marois R. (2008). Neuron. The neural correlates of third-party punishment.http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1715509, 60, pp 930–940,doi:10.1016/j.neuron.2008.10.016, pmid:19081385. 4. Miller, G. (2010). Brain Exam May Have Swayed Jury Sentencing Convicted Murder. Science. http://www.sciencemag.org/news/2010/12/brain-exam-may-have-swayed-jury-sentencing-convicted-murderer 5. Meixner JB, Rosenfeld JP. (2011). A mock terrorism application of the P300-based concealed information test. Psychophysiology 48:149–154, doi:10.1111/j.1469-8986. 2010.01050.x 6. Burgess, H. (2011). Depending the Discourse Using the Legal Mind’s Eye: Lessons from Neuroscience and Psychology that Optimize Law School Learning. Hein Online. http://heinonline.org. myaccess.library.utoronto.ca/HOL/Page?handle=hein.journals/qlr29&start_page=1&collection=journals&id=3 7. Greely H, Sahakian B, Harris J, Kessler RC, Gazzaniga M, Campbell P, Farah MJ. (2008). Towards responsible use of cognitive-enhancing drugs by the healthy. Nature 456:702–705, doi:10.1038/456702a, pmid:19060880. 8. J.D./Ph.D. Programs in Laws and Neuroscience. (2012). Vanderbilt University. http://lawneuro.org/blog/2012/03/19/jdphd-programs-in-law-neuroscience/
Volume 5. Issue 3. February 2018
Alexa, AI, and Amendments by Yun Kim On November 22nd, 2015, a man was found dead in the backyard hot tub of James Bates. Bates claimed innocence, suggesting an accidental drowning, but detailed inspection found signs of a violent struggle. With no witnesses and Bates as the sole suspect, investigators turned towards an unlikely source – Bates’ Amazon Echo, which had been streaming music near the scene¹. The Echo, colloquially “Alexa”, is a home speaker which uses weak artificial intelligence (AI) to respond to vocal commands. Given the potential for evidence, prosecutors sought a warrant to obtain recordings made by Bates’ Alexa. On February 17th, 2016, Amazon filed a motion to quash the warrant, and within it contained a striking proposition: that Alexa’s recordings and responses to user queries were protected under the First Amendment². Amazon’s ninety-page motion argued that despite being an AI, Alexa’s responses revealed personal information about user interests and thus deserved constitutional protection². While the Amendment is known for its free speech clauses, it also grants “the right to receive information and ideas” with freedom from government inquiry³. In the investigation of former President Bill Clinton, the Supreme Court required heightened justification for Monica Lewinsky’s book purchases under similar Amendment concerns⁴. To be clear, Amazon’s motion argues Amendment protection for Alexa’s responses, but not Alexa as an entity. Alexa is inherently editorial; its capabilities as weak AI are narrow and programmer-defined. This contrasts with hypothetical strong AI, which if created, would possess near human-level sentience⁵. Debate continues over whether strong AI is achievable, but Amazon’s motion poses an interesting question: are First Amendment protections extendable to future strong AI? Textually, the Amendment poses no specific restrictions to legal persons. Indeed, the landmark Citizens
United v. FEC established precedent by extending protection to corporations and labour unions⁶. The amendment states that “Congress shall make no law … abridging the freedom of speech”, premised on the mistrust of governmental power and its regulation of content⁶. Indifferent to its origin, the Amendment grants protection to speech as a deterring mechanism against undemocratic governing. Furthermore, major legal theories justifying speech protection are often separated from the individual. Philosopher Alexander Meiklejohn proposed that self-governance of liberal democracies necessitates free information, since it facilitates fair and informed electorate representation⁷. Similarly, John Milton and John Stuart Mill proposed that free speech creates a necessary “marketplace of ideas”, contending that truth from competitive discourse emerges only under transparency⁸ ⁹. Both arguments, often cited as major legal justifications, hold the value of speech from the ideas and corresponding social value it creates, rather than the entities from which it originates⁷. For many however, the idea of speech is interconnected with qualities that are uniquely human. Protected speech, for instance, is often seen as means to preserve individual autonomy and dignity¹⁰ ¹¹. Legal theorists question the extent that AI can possess such attributes, which many argue are distinctly human¹². Furthermore, extension of AI rights can raise practical questions, such as whether AIs can be mentally ill, or whether they can be held legally liable¹³. Legal scholars recognize potential flexibility for AIs like Alexa in the First Amendment. Ultimately, however, AI rights for speech may be affected by our perceptions of AI “humanness”. The extent to which AIs can develop human qualities, and the way our perspectives of speech and humanisms change, will shape how readily AI rights are adopted in the future.
References 1. Christopher Mele. Bid for Access to Amazon Echo Audio in Murder Case Raises Privacy Concerns. The New York Times [Internet]. 2016 Dec 28 [cited 2017 Jan 25]. Available from: https://www. nytimes.com/2016/12/28/business/amazon-echo-murder-case-arkansas.html 2. Memorandum of Law in Support of Amazon’s Motion to Quash Search Warrant, State of Arkansas v. James Bates, No. CR-2016-370-2 (Ark. Cir. Feb. 17, 2017) 3. Stanley v. Georgia, 394 U.S. 557, 564 (1969) 4. Grand Jury Subpoena to Kramerbooks & Afterwords Inc., 26 Media L. Rep. (BNA) 1599, 1601 (D.D.C. 1998) 5. Ray Kurzweil. The Singularity Is Near: When Humans Transcend Biology. New York: Penguin; (2006). 6. Citizens United v. Fed. Election Comm, 558 U.S. 310, 130 S. Ct. 876, 175 L. Ed. 2d 753 (2010) 6. Alexander Meiklejohn. Political Freedom: The Constitutional Powers of The People. New York: Harper; (1960). 8. John Milton. Areopagitica. New York: AMS Press; (1971). 9. John Stuart Mill. On Liberty. London: Longman, Roberts & Green; (1859). 10. Maxine Goodman. Human Dignity in Supreme Court Constitutional Jurisprudence. Neb. L. Rev. 2005;84. 11. Christina E. Wells, Reinvigorating Autonomy: Freedom and Responsibility in the Supreme Court’s First Amendment Jurisprudence. Harv. C.R.-C.L. L. Rev. 1997;32:159. 12. Jason Iuliano. Do Corporations Have Religious Beliefs?. IND. L.J. 2015;90:47, 71 13. Samir Chopra & Laurence F. White. A Legal Theory for Autonomous Artificial Agents. Ann Arbor: University of Michigan Press; (2011).
19
INTERNEURON
A Brief Glance and Perspective David Tran
By far one of the most significant concepts that defines us as a species is our capacity of free will. Free will, being the power to not only have a choice in decision, but to be able to freely and morally make any decision under only your volition. It has become one of the most fundamentally embedded concepts in our ideologies, religions, and cultures. From our legal systems or to our concepts of morality, the concept of free will is based on our assumed possession of three things: our volition in whether to do something, when to do something, and what it is we want to do (2).
Determinism: The Illusion of Free Will
Our knowledge and understanding of the brain has experienced a remarkable change over the past number of decades. The theory of determinism and its many derivatives have arisen due to our stronger understanding of the brain. Determinism, in essence, states that all things happen due to a previous cause, including our own thoughts and actions. Every thought you’ve ever had, or every idea, concept, or statement you’ve ever come up with, is therefore a result of chemical and physical effects due to pre-existing causes. Many researchers have come up with evidence to show that perhaps our free will is an aftermath of an unconscious cause (1, 2, 4). The concept of free will is then claimed to be no more than an illusion. Experiments have shown that brain activity for simple tasks is detected before the conscious thought of doing them occurs (1, 4). If the thought truly does come after the action’s been decided, as many researchers claim, then free will is also suggested to be a human interpretation of our actions. In other words, free will can be defined as an interpretation that our conscious thoughts are the cause of our actions and choices (6). But is this enough to justify that what we do is the fault of our neurology? Can we be held accountable for anything we do?
Ethical Implications: Criminal Responsibility and Individual Accountability
Though we are most certainly not ready to conclude that we have no free will, one’s mental capabilities in the judicial setting is growingly becoming a significant factor. Every year there are approximately 250 – 300 cases in Canada where the defendant was proven to not be criminally responsible due to pre-existing medical ailments that affected behaviour (5).
20
Volume 5. Issue 3. February 2018
To be clear, this represents less than 1% of all criminal cases in Canada. However, this raises many ethical implications and questions that require discussion. Defendants deemed not mentally responsible for their actions may typically have a mental disorder that prevents them from being able to make moral decisions. If a certain neuropathology may predispose someone to aggressive behaviour, where might one draw the line between moral responsibility and neural accountability?
Why We Want Free Will to be Real
In 2016, Adam Bear and Paul Bloom at Yale University conducted an experiment which found that certain perceptual events can unconsciously affect an individual’s decision-making process (1). They called it the postdictive illusion of choice, suggesting that a conscious decision arises after the unconscious choice has already been made. In spite of a relatively recent surge in theories and evidence supporting the radical view that free will does not exist, there also exists appropriate discussion for why one would find this to be so radical. Studies have shown that the human psychological need for a feeling of control is desirable and motivational (3). Losing this feeling of control has been shown to diminish motivations for tackling environmental and social obstacles (2). As such, a contributive aspect of our free will stems from our knowing that we have it. Coming to a belief that we don’t have it could potentially lead to impulsiveness and lack of motivation to tackle life issues, and furthermore, a loss of our self-worth.
Conclusion
A discussion of free will in and of itself is not as simple as one might have expected at first glance. Whether we have it or not, it’s clear we have only touched the tip of the iceberg of this discussion. Surely it will be fascinating to hear what kinds of findings we may have in the future that’ll prove or disprove its existence. However, free will is merely a concept meant to describe what it could be that makes us who we are. We are in control of our actions because we believe we are in control. And that statement in of itself is capable of opening a whole other door of discussion.
References
1. Bear, A., & Bloom, P. (2016). A simple task uncovers a postdictive illusion of choice. Psychological Science, 27(6), 914-922. doi:10.1177/0956797616641943 2. Brass, M., Lynn, M., Demanet, J., & Rigoni, D. (2013). Imaging volition: What the brain can tell us about the will. Experimental Brain Research, 229(3), 301-312. doi:10.1007/s00221-013-3472-x 3. Leotti, L. A., Iyengar, S. S., & Ochsner, K. N. (2010). Born to choose: The origins and value of the need for control. doi://doi.org/10.1016/j.tics.2010.08.001 4. Libet, B., Gleason, C. A., Wright, E. W., & Pearl, D. K. (1983). Time of conscious intention to act in relation to onset of cerebral activity (readiness-potential). Brain, 106(3), 623-642. doi:10.1093/brain/106.3.623 5. Miladinovic, Zoran., & Lukassen, Jennifer. (2018). Verdicts of not criminally responsible on account of mental disorder in adult criminal courts, 2005/2006 - 2011/2012. Retrieved Feb 3, 2018 from http://www.statcan.gc.ca/pub/85-002-x/2014001/article/14085-eng.htm#a7. 6. Wegner, D. M., & Wheatley, T. (1999). Apparent mental causation: Sources of the experience of will. American Psychologist, 54(7), 480-492. doi:10.1037/0003-066X.54.7.480
21
INTERNEURON
Get Involved! Interneuron is always looking for contributors! If you are interested in joining us as a writer, artist, or photographer:
Like our Facebook page:
https://www.facebook.com/UTInterneuron/
Send us an email:
interneuron.utoronto@gmail.com
Tweet us:
@InterneuronUofT
Visit our website:
https://interneuronuoft.wordpress.com
Address: 7 Hart House Cir, Toronto, ON M5S 3H3
22
Volume 5. Issue 3. February 2018
23
INTERNEURON
24