VOLUME
Neural Communication in Canines By: Jonathan Tran
19 • ISSUE 1 • FALL 2020
How COVID-19 Has Changed the Way We Communicate By: Andrew Lowrance
The Universality of Facial Expressions By: Sagar Gupta
PENNSCIENCE PennScience is a peer-reviewed journal of undergraduate research and related content published by the Science and Technology Wing at The University of Pennsylvania and advised by a board of faculty members. PennScience presents relevant science features, interviews, and research articles from many disciplines, including the biological sciences, chemistry, physics, mathematics, geological science, and computer science. PennScience is funded by the Student Activities Council. For additional information about the journal including submission guidelines, visit www.pennscience.org or email pennscience@gmail.com.
EDITORS IN CHIEF Neelu Paleti
Helen Jiang
Faculty Advisors Dr. M. Krimo Bokreta Dr. Jorge Santiago-Aviles Editing Managers:
Writing Managers:
Editing:
Writing:
Mehek Dedhia Brian Song
Andrew Lowrance Magnolia Wang
Business Managers:
Design Manager:
Glen Kahan Angela Yang
BIANCA VAMA
Anushka Dasgupta Samuel Gavronski Abigail Granger Hyunil Kim Natalie Kim BRIAN LEE Rajat-Ramesh Sarika Rau Daniel Rodriguez Mary Tran
DaHyeon Choi Kevin Guo Sagar Gupta Rebecca Nadler Michelle Paolicelli Jonathan Tran
Technology Manager: Grace Lee
2
PENNSCIENCE JOURNAL | FALL 2020
Business: Elena Cruz-Adames Sneha Sebastian
Design: Jessica Hao Lynne Kim Amara Okafor Ethan Seto
Dear Readers, On behalf of the entire PennScience team, we are excited to present the Fall 2020 issue of the nineteenth issue of the PennScience Journal of Undergraduate Research. For many years, PennScience has explored a variety of topics across the fields of science, technology, and engineering, and this year we focused on the idea of Communication. We are grateful to have such talented writers who have taken this topic in many novel directions. Andrew Lowrance discusses the role of science communication in the age of COVID-19, while Rebecca Nadler examines how the physician-patient relationship can be maintained in the age of the digitization of healthcare. Sagar Gupta focuses on the topic of facial expressions and the changes we see during this pandemic. DaHyeon Choi reviews the importance of Natural Language Processing in computers for human communication. Kevin Guo explores the similarities and differences of emotions and communication between animals and humans. Michelle Paolicelli explains how hermit crabs and ants inspire communication during organ transplant and air traffic control processes. Finally, Jonathan Tran investigates neural communication in canines, relating it to communication between humans. We received many excellent original research submissions, and we are proud to present the work of Madeline Cook from Drexel University, who discusses how machine learning can be used to determine the minimum inhibitory concentration of medicines in order to prevent antibiotic resistance when treating patients. While this past semester has had its share of challenges, we are extremely proud of our PennScience team for completing this journal all while working remotely. We are grateful to have such a wonderful team of students from the writing, editing, design, business, and technology committees. This journal would not have been possible without their dedicated, passionate work this semester. We would also like to thank the Science and Technology Wing of the King’s Court College House, as well as the Student Activities Council for their generous funding that made this possible. Finally, a very big thank you to our faculty mentors, Dr. Krimo Bokreta and Dr. Jorge Santiago-Aviles for their constant support and enthusiasm for this journal. Last but not least, we would like to thank all of our readers! We hope you enjoy reading this issue. Sincerely, Neelu Paleti (C’21) and Helen Jiang (C’22) Co-Editors-in-Chief
FALL 2020 | PENNSCIENCE JOURNAL
3
FEATURE
:
4
PENNSCIENCE JOURNAL | FALL 2020
FALL 2020 | PENNSCIENCE JOURNAL
5
FEATURE
H o w C OV I D - 1 9 H a s C h a n g e d t h e D e s i g n e d b y : Way We C o m m u n i c a t e Wr i t t e n b y :
B i a n c a Va m a
If the onset of COVID-19 has demonstrated anything, it is that, along with necessary preparation from the public to socially distance, the disbursement of scientific information has proven critical for allowing trust in the process of remediating the ongoing pandemic. With the pandemic growing to over 65 million cases worldwide and 14.2 million cases in the U.S. alone today1, it has become imperative for governments, institutions, and media groups to guarantee the disbursement of information that is both critical to understanding the pandemic as well as ensuring public health recommendations are practiced. It is more important than ever to make sure accurate information is relayed from the right sources. But how exactly is this accomplished and why is it necessary? How Scientific Communication is Accomplished in a Virtual World There are various examples which illustrate the need to adapt to the pandemic. One could for instance look to China, which has instituted CAST2, the China Association for Science and Technology. This system served as a one-stop shop for helping citizens navigate through the COVID-19 pandemic by providing key insights and public health recommendations in real-time. Such a system serves as a revolutionary step in the way in which information is translated to the public. CAST, for instance, organized a series of articles and live sessions from respected scientists to communicate most frequently asked questions about the virus, masks, and appro6
PENNSCIENCE JOURNAL | FALL 2020
Andrew Lowrance
priate behavior in the pandemic. One such member is Xiangyu Jin, a textile researcher from Donghua University in Shanghai. Ziangyu helped to convey to the public the importance of wearing masks to mitigate the spread of the virus. More interestingly, CAST’s success as a communication platform stems from its implementation. The platform can be characterized as a “flattened crisis management structure.” This structure focuses less on internal authority within the organization and more so on maintaining regular contact points with various national and provincial associations. This is particularly noteworthy as other systems, such as that in the U.S., have utilized a hierarchical approach in the disbursement of information primarily from the chief executive’s office. And, similar to China’s CAST, there have been efforts bolstered by private groups to accomplish this same task elsewhere, such as in Philadelphia, Pennsylvania.
Sarah McAunlty, who is based in Philadelphia and serves as a science communicator, launched Skype a Scientist. This is a private initiative that serves to connect researchers with individuals, classrooms, and public groups to provide additional information regarding the ongoing pandemic. This initiative is ultimately a bolstered effort by instructors, researchers, and individuals alike to ensure regular communication with young individuals around the world is maintained with the growing physical dissociation caused by the pandemic. This platform speaks to the growing mistrust in science in the U.S., which is only bolstered by the pandemic3. What’s important to note is that this system of regular communication is not unfounded. Research
suggests that the transition to social distancing and virtual interactions has led to psychological effects in individuals, which are now propagated on a global scale. As people begin to become dissociated with one another, either mentally or physically, thereby relying on technological means to meet social needs, there is an associated mistrust4 that is developed. Why a Means for Scientific Communication is Psychologically Necessary Communication has always been a hallmark attribute of beings, founded on centuries of developed psychology grounded on the social experience. This means that communication is critical to the development and maintenance of, amongst many things, healthy relationships. Now, in light of the COVID-19 pandemic and the need to do the exact opposite of our socially-attributed traits, this bears a question of to what degree individuals can adapt. Tim Levine, Ph.D., a professor in the Department of Communication Studies at the University of Alabama at Birmingham, illustrates that the most obvious changes to interpersonal interaction come from less face-to-face interaction. On average, this
decrease in social interaction may in turn increase the frequency of social isolation. Research finds one impact of social isolation is increased suspicion of others, triggering a type of defensive mechanism. This defensiveness in turn leads one down a spiral of increased isolation and suspicion, creating what is known as a “self-fulfilling prophecy5.” Now, why would social isolation lead one to become suspicious of others to begin with? With interpersonal communication, there is an attributable activation of oxytocin, commonly known as the love hormone. This neurochemical provides the experiencer a sense of security in the individual they are with, like receiving a hug from a loved one. Now, with the transition to online platforms of communication, the release of oxytocin is limited, leading to an associated decreased sense of security6. Though online platforms do facilitate interpersonal interaction, virtual simulation is not as effective as physical contact. Thus, people begin to become suspicious. This analysis is, at best, a broad generalization of the process of social interaction. They are nonetheless critical to understanding how vitally important the issue of communication is to both scientists and our well-being as a global community. This is why platforms like Skype a Scientist and China’s CAST system have been instituted. As cases continue to ravage countries, developed and developing, many have turned to virtual tools to combat what is a second looming global health crisis: the dissociation of how people interact. As governments, private entities, and researchers alike institute means to maintain public confidence in the scientific institutions that will help fight against the pandemic, so too does it fall upon the people to ensure that COVID-19 does not fundamentally change the way in which we communicate. References
1.CDC COVID Data Tracker. (n.d.). Retrieved from https://covid.cdc.gov/ covid-data-tracker/#cases_casesper100klast7days 2.Science communication in the COVID-19 pandemic. (n.d.). Retrieved from https://www.nature.com/articles/d42473-020-00329-z?utm_source=facebook&utm_medium=social&utm_campaign=bc-cast-2020 3.Forrester, N. (2020, July 09). How the coronavirus pandemic is changing virtual science communication. Retrieved from https://www.nature.com/ articles/d41586-020-02075-0 4.How has COVID-19 affected the way we communicate? - News. (n.d.). Retrieved from https://www.uab.edu/news/research/item/11542-how-hascovid-19-affected-the-way-we-communicate 5.id. 6. Han, R. T., Kim, Y., Park, E., Kim, J. Y., Ryu, C., Kim, H. Y., . . . Na, H. S. (2018, August 14). Long-Term Isolation Elicits Depression and Anxiety-Related Behaviors by Reducing Oxytocin-Induced GABAergic Transmission in Central Amygdala. Retrieved from https://www.ncbi.nlm. nih.gov/pmc/articles/PMC6104450/
FALL 2020 | PENNSCIENCE JOURNAL
7
FEATURE
Maintaining the Physician-Patient Relationship in the Digitalization of Healthcare Written by: Rebecca Nadler Designed by: Amara Okafor
In February 2020, the United States’ Centers for Disease Control and Prevention issued social distancing guidelines in response to the coronavirus disease 2019 (COVID-19) pandemic. This amplified the already increasing prevalence of telehealth services, which are the methods of delivering healthcare via telecommunications technology.1 The number of telehealth visits surged by 50% between January and March of 2020 compared to the same period in 2019.2 Notably, 93% of those visits were reported to be conducted for conditions other than COVID-19. Given that telehealth is a newfound asset in a vast range of specialities, it is important that doctors consider methods of optimization for the future. Rapid growth of the telemedicine industry has elevated it to become a staple in the healthcare sector, but due to technological barriers to communication, this is not without its challenges. From a patient perspective, telehealth involves live video conferencing with a healthcare provider in lieu of an in-person appointment. Even though virtual appointments have been utilized to provide care for inmates, 8
PENNSCIENCE JOURNAL | FALL 2020
military personnel, and patients living in rural settings, the public has generally overlooked the capacities of telehealth due to skepticism of its nontraditional nature.1 It allows for another modality of communication between patients and providers, but communication barriers that arise in the absence of face-to-face conversations may make these types of communication incomparable. A study of 27 patients with type 2 diabetes mellitus that experienced clinical video telehealth (CVT) through Veteran Affairs Health Care elucidated benefits from increased efficiency as well as concerns that the providerpatient relationship would be diminished when care was given via video calls. The patients collectively remarked that increased efficiency and accessibility makes CVT attractive.3 It requires no travel and minimal waiting time, partially eliminating the burdens of cost and time from travel. According to the Altarum Institute, patients spend an average of 45 minutes traveling and waiting for care, which is typically unnecessary for telehealth visits.4 CVT is especially beneficial for patients living in rural
“The limitations of telehealth software further complicate the ability for patient-physician communication to be replicated remotely.” areas with limited access to healthcare providers, and has improved the patients’ access to and quality of clinical care.5 However, many patients perceived that providers paid less attention to them, stated they were less comfortable asking questions, and expressed difficulties in establishing a relationship with their healthcare providers.3 There are also questions of confidentiality in a telemedicine setting as privacy and security guidelines are reevaluated in the midst of a nationwide public health emergency.
The limitations of telehealth software further complicate the ability for patientphysician non-verbal communication to be replicated remotely. It is estimated that 55% of emotional communication takes place through nonverbal cues,3 presenting an impediment to maintaining the typical patient-provider relationship from a face-to-face environment.
Studies of nonverbal communication skills in medical care have reported that doctors’ nonverbal behaviors are related to better medical outcomes,6 suggesting a tangible impact of physician communication. Further studies have revealed that doctors with better communication skills are able to recognize and diagnose patients’ illnesses more accurately, leading to greater patient satisfaction and better health outcomes.6 Patient concerns centered around the disadvantages of telehealth maintain that behavioral cues such as eye contact and body language are harder to convey in telehealth. Thus, an understanding of physician attention and attitude may be lost through a screen. Eye contact between doctors and their patients was deemed to be important for cultivating comfort and trust in 86.1% of patients surveyed after receiving care.7 Apprehension about physician engagement during appointments conducted via telehealth was often compounded by inaccurate perceptions of nonverbal cues, as it was worsened by a perceived lack of eye contact
during the calls. As a display of attentiveness, regular eye contact from doctors has been proved essential for building good rapport with patients, especially elderly individuals.8 In a study of elderly patients at a University of Chicago medical center, researchers found that eye contact is critical for patient-centered communication as common functional FALL 2020 | PENNSCIENCE JOURNAL
9
impairments such as hearing deficits results in a greater reliance on eye contact.9 This reigns true on a digital platform for patients of all ages because of the disparities patients’ access to high-speed internet connection, which is often reflected in the sound quality of the device used for an appointment. In the realm of communication barriers, physical touch as a means of comforting an individual also plays a significant role in developing the relationships between providers and patients. Oxytocin, a hormone associated with increased levels of social interaction, well-being, and anti-stress effects, is released upon sensory stimulation. It bolsters the anti-stress effects particularly in situations of low intensity stimulation like touch,10 which may be used during in-person appointments to console patients. In fact, a study of 20 family practices throughout Ontario, Canada found that the majority of patients surveyed found a touch to the arm or hand to be comforting and healing.11 However, a physiological response to a display of compassion is impossible in a telehealth setting. Nonetheless, with technological adaptations to this evolving healthcare landscape, it is possible to recreate many aspects of the patient-provider relationship that are commonly found in the clinic. Adapting to the virtual environment will likely require healthcare workers to direct their gaze toward their cameras rather than their screens in order to recreate a sense of eye contact. Further suggestions aimed toward increasing the quality of care include explicit conveyance of prior knowledge and patient history,3 an exploration of patient goals and concerns, and the development of ‘webside’ manner that is analogous to the traditional bedside manner that practitioners are expected to uphold. Researchers concur that the development of patient education materials designed to prepare patients for their appointments can also facilitate the transition to telemedicine.3 The telehealth platform demands that patients play an active role in maintaining their wellbeing, utilizing active communication to ask questions 10
PENNSCIENCE JOURNAL | FALL 2020
and express concerns FEATURE throughout appointments. Patient feedback is essential for improving the existing telehealth platform, and such interview-style studies are currently transforming practitioners’ training. In the digital era, telemedicine provides an unprecedented ease of accessibility to patients that otherwise would not be able to address their health concerns. While there are limitations to communication via video conferencing, patient insight continues to advance virtual practices. Ultimately, an expression of shared power and responsibility in one’s treatment plan is paramount to the preservation of the patientphysician relationship.
The
UNIVERSALITY
of FACIAL EXPRESSIONS
Written by: Sagar Gupta Designed by: Ethan Seto
FALL 2020 | PENNSCIENCE JOURNAL 11
H
umans have thousands of facial expressions, the most common being fear, anger, surprise, happiness, sadness, and disgust.1 Just twenty independent muscles in the face can make a wide variety of expressions.2 As expected, there is a great deal of variation between humans in the number and type of facial muscles. For example, some can be missing or asymmetric in size. However, those necessary for the common facial expressions are conserved across all humans. The debate about the universality of facial expressions provided further evidence in favor of evolution.
“ JUST TWENTY INDEPENDENT MUSCLES IN THE FACE CAN MAKE A WIDE VARIETY OF EXPRESSIONS
”
DARWIN & EKMAN VS. THE WORLD Charles Darwin was the first to argue in favor of the universality of facial expressions. In his book, The Expressions of the Emotions in Man and Animals (1872), Darwin argued that since humans share a common origin, they express their emotions in similar manners.3 In doing so, he was heavily criticized by leading scientists in the field. Darwin’s ideas were in direct contradiction with those of the leading facial anatomist Sir Charles Bell, who said that God gave humans unique emotions only they could express.4 In the one hundred years following Darwin’s claims, experts began to gather evidence against him. Analysis of the description of faces in Chinese literature and anecdotal examples showed that emotional facial behavior showed variation between cultures.5, 6,
he studied were industrialized, and facial expressions could have been learned from mass media such as television rather than an innate emotional response. Ekman needed to find another way to prove Darwin right.
FEATURE
NEW FINDINGS IN PAPUA NEW GUINEA Dr. Ekman and his colleague Dr. Wallace Friesen realized that the only way to prove the universality hypothesis was to find a group of people unaffected by mass media. Thus, in the early ‘70s they trekked out to a remote, primitive tribe in Papua New Guinea to prove Darwin’s ideas. Armed with just a set of six flashcards, Ekman talked to the members of the group who had no contact with Western or Eastern modern cultures.8 The subjects were told to match the facial expression of a Western person to a simple story that would evoke a clear emotional response. For instance, “A child has died,” would evoke a sad response. In a similar manner, Ekman captured the faces of the Fore tribe and asked Westerners to match them to emotions.9 The results were clear: “particular facial behaviors are universally associated with particular emotions.” With this experiment, Ekman provided concrete evidence and finally convinced the scientific community that Darwin was correct. “ PARTICULAR FACIAL BEHAVIORS ARE UNIVERSALLY ASSOCIATED WITH PARTICULAR EMOTIONS ”
7
However, in the late ‘60s a young professor, Dr. Paul Ekman, challenged these ideas and supported Darwin. He claimed that facial expressions were not culture-specific and set out to disprove the claims of his predecessors. He went across the globe to different cultures, ranging from the United States to Japan, in order to collect samples of common facial expressions: fear, anger, surprise, happiness, sadness, and disgust. Then, he asked volunteers from different regions to describe the expressions of people from across the globe. Ekman found that people could identify the emotions at a rate significantly greater than that of chance. Despite his global findings, critics dismissed them because the cultures 12
PENNSCIENCE JOURNAL | FALL 2020
COVID-19 IMPACT Although the universality of facial expressions makes it easier to communicate with people from other cultures, the COVID-19 pandemic has raged across the globe, causing people to wear masks whenever they interact with others. This common practice has had a significant impact on nonverbal communication. We rely primarily on one’s mouth and eyes to identify the emotion on someone’s face.10 The effect of mask-wearing in nonverbal communication is especially pronounced in healthcare settings. Research suggests that doctors that wear masks are seen as less empathetic and caring.11 Despite these setbacks, we can still effectively com-
municate if we adapt to the crisis by using body language more frequently alongside gestures.12 Gestures, bodily actions in accordance with our speech, would allow us to reinforce what we are saying. Thus, it allows people to stay on the same page when talking.13 For instance, if we want to reply affirmatively, we can give a thumbs up or head nod to mirror what is said. Another way to help bridge the communication gap is to have open and attentive body language.14 This ensures that the other person feels heard. We can lean forward and be active listeners. Additionally, keeping arms uncrossed and palms open helps convey a welcoming environment.15 Eye contact can also be used, but should not be continuous as this can come across as dominant and make the other person feel uncomfortable. Eyes are especially important as they are all people see when one is wearing a mask. As a result, we tend to emphasize the information more. Research shows that we can use eyes to interpret people’s emotions accurately.16 So, it is critical that we are cognizant of how we express ourselves with our eyes. Essentially, we should replace the facial nonverbal communication with other kinds of nonverbal communication such as gestures, body language, and eye contact. CONCLUSION Nonverbal communication is crucial to humans. Thus, it is no surprise that the main emotions have been found to be universal. Dr. Ekman and his colleagues worked hard to please critics and became pioneers in the field of facial expression analysis. Their research spurred interest in the field of nonverbal communication. With the progression of COVID-19, the fundamental ideas in this field are being used to help us communicate. Nonverbal communication goes beyond facial expressions. By using gestures, body language, and eye contact, we
can take steps to remedy the situation going forward. More research is needed to determine the effectiveness of these techniques, but initial results are promising. REFERENCES [1] Black, M. J., & Yacoob, Y. Recognizing Facial Expressions in Image Sequences Using Local Parameterized Models of Image Motion. International Journal of Computer Vision, 25(1), 23-48. [2] Gray, H., & Goss, C. M. (1966). Anatomy of the human body (28th ed.). Philadelphia, PA: Lea & Febiger. [3] Darwin, Charles. The Expression of the Emotions in Man and Animals. London: J. Murray, 1872. [4] Matsumoto, D., Hwang, H. C., & Frank, M. G. (Eds.). (2016). APA handbook of nonverbal communication. American Psychological Association. [5]Klineberg, O. (1938). Emotional expression in Chinese literature. The Journal of Abnormal and Social Psychology, 33(4), 517–520. [6] Labarre, W. (1947). The cultural basis of emotions and gestures. Journal of Personality, 16, 49–68. [7] Birdwhistell, R. L. (1970). Kinesics and context: Essays on body motion communications. Philadelphia: University of Pennsylvania Press. [8] Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 11, 124-129. [9] Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial displays of emotion. Science, 164, 86–88. [10] Wegrzyn, M., Vogt, M., Kireclioglu, B., Schneider, J., Kissler, J. (2017). Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS One, 12(5):e0177239. [11] Wong, C. K., Yip, B. H., Mercer, S., Griffiths, S., Kung, K., Wong, M. C., Chor, J., & Wong, S. Y. (2013). Effect of facemasks on empathy and relational continuity: a randomised controlled trial in primary care. BMC family practice, 14, 200. [12] Carbon, C. (2020). Wearing Face Masks Strongly Confuses Counterparts in Reading Emotions. Frontiers in Psychology, 11. [13] Heid, M. (2020, May 21). I See You but I Don’t: How Masks Alter Human Connection. [14] Kerr, A., & Bylund, C. (2020, Aug 17). 10 Ways to Improve Patient Interactions While Wearing a Mask. [15] Morgan, N. (2011, Sep 9). Body Language Quick Takes: How to Spot Openness. [16] Lee, D.H., & Anderson, A. K. (2017). Reading What the Mind Thinks From How the Eye Sees. Psychological Science, 28(4), 494–503.
FALL 2020 | PENNSCIENCE JOURNAL 13
f Speech and Machines Machines
FEATURE
Written by: DaHyeon Choi Designed by: Amara Okafor Humans are social animals. We use language as our primary means of communication; as such, language is a significant component of our lives. There are countless languages all around the globe, but regardless of where and by whom a language is used, they all contribute to the exchange and recording of information. Machines, too, have their own languages for communication and information storage. As machines and humans cannot understand each other directly, many researchers today study the field of Natural Language Processing in order to bridge the gap. The ultimate goal of NLP is to create a machine capable of communicating with humans directly. Natural Language Processing (NLP) is a branch of computer science that focuses on building machines that understand and respond to text or voice data similar to the way humans do. It is a field that is considered to be part of machine learning and artificial intelligence. NLP is essential to building machines that can directly understand and respond to natural languages (i.e. human languages), and is a field that is historically entwined with a variety of other disciplines including computer science, neuroscience, and linguistics. Creating a machine
14
PENNSCIENCE JOURNAL | FALL 2020
capable of understanding human language requires both knowledge of how humans communicate and knowledge of how that can be defined in computational terms. THE ORIGIN It would be difficult to proclaim an exact starting point in NLP history. However, it would be the most accurate to say that the actual study of linguistics in computer science began with Alan Turing,1 the father of computers. In 1950, Turing described a test for a true “thinking� machine. This Turing test, or more accurately, the Imitation Game,2 specifies that for a machine to exhibit intelligent behavior equivalent to a human, it must be able to hold an intelligent natural language conversation with a human to the point where if a bystander were to read or hear the conversation, the machine would be indifferentiable from the man. This famous Turing test marks the beginning of the study of intelligent machines. SECONDARY FIELDS OF STUDY
In a seemingly much different field of study,
neuroscientists Hodgin and Huxley demonstrated the brain’s usage of a neuron network. A neuron network describes the model of the brain that connects and passes information through neurons. Though seemingly disconnected from Turing’s idea of thinking machines, these discoveries in neuroscience became the inspiration for Artificial Intelligence (AI), neural networks, NLP, and the evolution of computers. Linguistics, led by Noam Chomsky, is also another field that made contributions to the development of NLP. While linguistics began in the early 1900s, Noam Chomsky revolutionized linguistics through publishing his book Syntactic Structures in 1957.
Syntactic Structures
presents an extension of the
contemporary generative grammar — the theory that human language is shaped by a set of innate universal grammar principles (i.e., that language has a biological basis). It was one of the first publications at that time that offered a biological perspective on linguistics. Through his revolutionization of linguistic concepts, Chomsky concluded that for a computer to understand a human language, its grammar would have to be altered to allow extraction of information.3 Chomsky’s work, therefore, significantly influenced work on the computer and the brain. COMPUTER SCIENCE In the computer science field, John McCarthy released the computer language LISP (Locator / Identifier Separation Protocol) in 1958. ELIZA,4 the first computer-based ‘chatbot’ was developed in 1964. Following these developments in computer science, the U.S. National Research Council (NRC) created the Automatic Language Processing Advisory Committee (ALPAC) in the same year, which was tasked with
evaluating the progress of Natural Language Processing research.1 Surprisingly, the ALPAC halted funding for NLP and AI in 1968. People had unrealistically high hopes in these fields and were largely disappointed by the slow progress of research and invention. Subsequently, NLP and AI experienced an ‘ice age’ from 1966 to 1980. However, the end of this ‘ice age’ marked a notable change in our approach to NLP. The 1980s became a point of revolution; researchers moved from the early “handwritten” systems to statistical models. Then, as neural networks developed, NLP entered its Neural NLP era. Neural NLP differs from the previous form of statistical NLP because rather than relying on simple statistical data, it employed
the complex multilayered form of neural networks to process information. NLP TODAY There are many different areas within NLP. Typically called NLP tasks, they include text and speech processing (understanding human speech) ,morphological analysis (breaking down different forms of words), and syntactic analysis (grammar). Advancement in NLP, therefore, advances both linguistics and computer science at the same time. The sheer amount of knowledge needed to be incorporated for research in this field is the reason why accessible public resources for NLP are necessary. One such important resource is the Natural Language Toolkit (NLTK),5 developed by Steven Bird and Edward Loper in the Department of Computer and Information Science at the University of Pennsylvania. NLTK is
FALL 2020 | PENNSCIENCE JOURNAL 15
a platform for those interested in building Python programs to work with human language. Its initial release was in 2001.6 It has been used extensively for educational purposes, especially for students majoring in computer science or linguistics. The Bidirectional Encoder Representations from Transformers (BERT) developed by Jacob Devlin and his colleagues at Google, in 2019, is a similarly significant resource for NLP researchers and developers. BERT is a Transformer-based machine learning technique for NLP pre-training.7 With its state-of-the-art performances in several NLP tasks, BERT is often used for research and development. BERT is currently used for google search queries in more than 70 languages.8 It has also won the Best Long Paper Award at the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).9 Another recent arrival is Generative PreTrained Transformer 3 (GPT-3),10 an autoregressive language model that uses deep-learning to generate human-like text. An autoregressive language model uses a probability distribution over past sequences of words to predict current outputs. The third of the GPT series, GPT-3, was introduced in May 2020 by OpenAI. It is known for its remarkable ability in producing human-like text. University of Massachusetts Amherst, addresses the potential environmental impacts of NLP research itself. With modern advances in NLP involving deep neural networks, which have high costs for training, NLP research now requires substantial computational resources, which in turn consumes considerable energy. Therefore, associated financial and environmental costs of training new models are fairly high. This study by Strubell et. al. characterizes the dollar cost and carbon emissions that result from training state-of-the-art NLP neural networks. The researchers calculated these impacts by converting the kilowatts of energy required to train a variety of neural networks into approximate carbon emissions and electricity costs. For example, the cost of training a BERT model (described above) using V100x64 hardware can consume roughly 12,041.51 W of power and around $3,751 to $12,571 USD, as well as creating 1438 lbs of CO2e. This study and others like it assist the research and development of NLP models by raising questions about cost-efficiency and developmental impact.
16
PENNSCIENCE JOURNAL | FALL 2020
CONCLUSION FEATURE The significance of NLP lies in its application. Today, NLP is applied to translation, search queries, web mining, and more. In the future we may see the machines Alan Turing once described, perfectly indistinguishable from humans in speech. Of course, we still have a long way to go. NLP is a technology that can improve many aspects of modern human life, such as internet surfing, customer service, and even healthcare information. As such, it is only beneficial to keep track of its developments and inventions in this field. We can only look forward to the changes a machine capable of human communication will bring.
References
1. Foote, K. D. (2019, May 22). A Brief History of Natural Language Processing (NLP). Retrieved December 08, 2020, from https://www.dataversity.net/a-brief-history-of-nat ural-language-processing-nlp/ 2. Oppy, G., & Dowe, D. (2020, August 18). The Turing Test. Retrieved December 08, 2020, from https://plato.stanford.edu/entries/ turing-test/ 3. Chomsky, N. (1957). Syntactic Structures. Gravenhage: Mouton. 4. Weizenbaum, Joseph. (January 1966). ELIZA – A Computer Program For the Study of Natural Language Communication Between Man and Machine. Communica tions of the ACM; Volume 9, Issue 1, 36-45 5. Natural Language Toolkit. (n.d., NLTK Project). Retrieved December 08, 2020, from http://www.nltk.org/ 6. Bird, S., & Loper, E., & Klein, E., & Baldridge, J. (2008). Multidisciplinary Instruction with the Natural Language Toolkit. Proceedings of the Third Workshop on Issues in Teaching Computational Linguistics (TeachCL-08), 62–70 7. Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing. (2018, November 02). Retrieved December 08, 2020, from https://ai.googleblog.com/2018/11/open-sourc ing-bert-state-of-art-pre.html 8. Shwartz, B. (2020, October 15). Google: BERT now used on almost every English query. Search Engine Land 9. North American Chapter of the Association for Computational Linguistics. N. (2019, April 10). Best Paper Awards. Retrieved December 08, 2020, from https://naacl.org/naa cl-hlt-2019/blog/best-papers/ 10. Brockman, Greg, et al. “OpenAI API.” OpenAI, OpenAI, 28 Sept. 2020, openai.com/blog/openai-api/. 11. Strubell, Emma & Ganesh, Ananya & Mccallum, Andrew. (2019). Energy and Policy Considerations for Deep Learning in NLP. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3645–3650
Do Animals Share the Same Feelings That We Do? Written by: Kevin Guo Designed by: Bianca Vama
As I locked my car in the parking lot and entered the racquet club, grains in hand, I saw them. Those pure white, long-necked creatures of goodwill paddling their feet furiously against the water. Those two swans that I visit every month dashed to my presence, almost honking in excitement as I took out their favorite snacks. It almost reminded me of when I was little, always joyous that I would get to eat my favorite peanut butter and jelly sandwich. But those swans are not always just happy. Sometimes they attacked each other, aggressively hunted for where the food went, and even turned their heads away when I brought them lettuce. These exhilarating experiences have made me wonder: do animals experience emotions just like humans do? People often define emotions as feelings, something to help them control their behavior based on their circumstances or current mood. However, the term “emotions” has become such a broad topic that encompasses many aspects of our
psychological identity. In the case of animals, there are two different types of emotions to understand: simple emotions (i.e. sad, happy, angry, fearful) and complex emotions (jealousy, vengeance, humiliation). Many proponents of the claim that animals can experience emotions at the same level as humans refer to the philosophy of inference to the best explanation, which explains that one cannot dismiss what he or she cannot directly observe.4 Even though animals may not always convey emotions the same way humans do, there is evidence to suggest they can indeed experience simpler emotions. After the removal of several mammalian species’ brains, most cognitive scientists have observed that several of them have strikingly similar sizes and functional areas to humans. For example, gorillas and dolphins – two of the most intelligent animals behind humans – have two distinct hemispheres, frontal lobes, and a similar number of folds in their brains compared to their human counterparts. Since the frontal lobe of most brains is often involved in emotional processing and expression, it would be reasonable to surmise that animals can indeed harbor similar emotions to those of a human.4 However, not everyone agrees with these findings or their conclusions. Some detractors argue that we simply ascribe our human emotions onto the animals’ actions or their responses, similar to how we sometimes anthropomorphize nonliving objects such as cars or trees. These detractors sometimes point to operative conditioning as the reason for animals’ motivations for actions, citing experiments such as Pavlov’s dogs and their salivation in response to a bell ringing after conditioning. Even though the animals may seem like they are experiencing joy to the sound of food coming, they may in fact be responding from neural activity in their reward centers and thus be acting on impulse rather than emotion. Furthermore, they claim that human and animal brains are not similar due to humans’ nearly 86 billion active neuron usage compared to less than a billion for other mammals. This suggests that they may not be able to comprehend emotions at least not on the same level as that of humans.2 Even though the debate is ongoing, and both sides of the aisle are valid, there is a
FALL 2020 | PENNSCIENCE JOURNAL 17
consensus that animals at least have consciences that give them the capacity to derive emotions from their personalities. According to the 2012 Cambridge Declaration FEATURE on Consciousness, where a prominent group of cognitive neuroscientists convened at The University of Cambridge to determine the existence of a conscious being in non-human animals, “The neural substrates of emotions do not appear to be confined to cortical structures. In fact, subcortical neural networks aroused during affective states in humans are also critically important for generating emotional behaviors in animals … Wherever in the brain one evokes instinctual emotional behaviors in non-human animals, many of the ensuing behaviors are consistent with experienced feeling states, including those internal states that are rewarding and punishing.”1 This resolves the idea that animals may be responding to conditioning; they are actually displaying emotions even if in a less obvious fashion.
Some scientists believe that animals can experience emotions at the same level as humans. In addition to being able to experience a wide range of simple emotions, animals are also capable of more complex emotions and behaviors such as jealousy and empathy. Take, for instance, the rescuer humpback whales. In the last 66 years, there have been more than 115 cases of humpback whales protecting other smaller marine species from killer whales. They have even occasionally interacted with humans in fighting against these predatory animals. Since it is well known that humpbacks are capable of sophisticated thinking, decision-making, problem-solving, and communication, it seems likely that the humpback whale species has developed empathic responses to creatures in need of help.3 This altruism is not the only case of animals exhibiting complex emotions and behaviors, however. During the 2000s, famous “elephant whisperer” Lawrence Anthony tended to and rehabilitated several wild elephants that were about to be killed and released them into the wild. Three days after Anthony passed away on March 7, 2012, the same elephants returned to his home and stood around his resting place, apparently mourning their beloved friend. They have since returned every year in remembrance of their caretaker.5
18
PENNSCIENCE JOURNAL | FALL 2020
Regardless of the true answer, there is no doubt that animals can simulate some of our feelings and sometimes even become lifelong companions. So next time you play around with your pet or play music for the neighbor’s dog, the joy and happiness you experience is mutual.
References
1. Low, P. (2012, July 12). The Cambridge Declaration on Consciousness (1143470847 860609076 C. Koch, 1143470848 860609076 B. V. Swinderen, 1143470849 860609076 D. Edelman, 1143470850 860609076 D. Reiss, & 1143470851 860609076 J. Panksepp, Eds.). Retrieved January 2, 2021, from http://fcmconference.org/img/CambridgeDeclarationOnConsciousness.pdf 2. Perry, P., Poliza, P., Henry, P., & Kristof, P. (2019, November 08). Yes, Animals Think And Feel. Here’s How We Know. Retrieved January 02, 2021, from https://www.nationalgeographic.com/news/2015/07/150714-animal-dog-thinking-feelings-brain-science/ 3. Stokstad, E. (2017, July 26). Why did a humpback whale just save this seal’s life? Retrieved January 02, 2021, from https://www. sciencemag.org/news/2016/07/why-did-humpback-whale-just-save-seals-life 4.Thagard, P. (2017, November 14). Do Animals Have Emotions? A Debate. Retrieved January 02, 2021, from https://www. psychologytoday.com/us/blog/hot-thought/201711/do-animals-have-emotions-debate 5.Valentine, J. (2012, May 17). Wild Elephants Mourn Death of Elephant Whisperer. Retrieved January 02, 2021, from https://www. onegreenplanet.org/news/wild-elephants-mourn-death-of-elephant-whisperer/
FALL 2020 | PENNSCIENCE JOURNAL 19
Organ Transplants and Air Traffic Control:
FEATURE
How Hermit Crabs and Ants Inspire Us to Communicate More Effectively
Written by: Michelle Paolicelli Designed by: Jessica Hao
The social cues and norms of human communication have long been studied and scrutinized. It is recognized that human communication is unique to and more complex than any other species on the planet, setting us apart. However, does this mean that our methods of communication are superior? Other species’ notably deeper history speaks to their success in adapting and evolving to conditions over time. Certain physical attributes may be partly responsible for this survival, but perhaps it is also due in part to their own methods of communication within their populations and communities. To better understand how the communication methods of other species, namely hermit crabs and ants, has aided their survival throughout the ages, we must first be willing to expand our understanding of the definition of communication [3,4] . The word communication connotes speaking and writing or otherwise relaying information through words and direct signals to someone else. In the professional world, clear and effective communication is highly valued. Oftentimes, the best communication occurs when everyone relevant is informed and when exchanges are made frequently and promptly. One would not consider information only being shared with a few select members of a group to be a good example of communication. In most situations, this would likely lead to confusion and lower productivity. However, looking beyond human social and professional constructs at certain species of ants, it becomes clear that 20
PENNSCIENCE JOURNAL | FALL 2020
such a hierarchical method can prove to be efficient and sustainable[1]. Human corporations and businesses often rely on a hierarchical structure with a central commanding authority. Ants, however, are able to achieve coordinated behavior without this centralized control. At a more local level, a few select ants within the colony will serve as hubs of information while most simply carry out their designated duties without receiving direct communication from other parts of the colony. This leads to greater efficiency and allows the colony to stay connected globally without disturbing activity locally. In this structure, most ants have few interactions, while the select few ants have many interactions. This leads to a network structure similar to what is seen in the air traffic control industry[1]. In this structure, local cues from air-traffic control towers direct the movement and activity of planes in the vicinity. Individual planes interact solely with the air-traffic controllers rather than coordinating with all the other planes nearby one-by-one. This mirrors the coordination of activity in ant colonies and is not only more efficient, but also safer than typical one-to-one communication methods. Having ants, analogous to air traffic control towers, serve as hubs for information also allows for rapid dissemination of information if there is an immediate threat to the colony[1]. Another form of communication in animals that humans can learn from involves the exchange of resources rather than information.
Hermit crabs grow out of their shells multiple times during their lives and need to find a new one in a timely manner in order to survive.Typically, it can be competitive and difficult to find a shell of appropriate size and condition[2]. As a result, hermit crabs may have to resort to undersized or damaged shells, making them more vulnerable to predators and less likely to be successful. The solution to this is found in synchronous and asynchronous vacancy chains. In these vacancy chains, hermit crabs will systematically pass down shells in groups, creating a wider selection of shells for the crabs to choose from and increasing the likelihood of a good fit. In synchronous vacancy chains, a group of hermit crabs will line up in size order and exchange shells with the crabs adjacent to them. This is akin to organ transplant chains created by humans. The chain method serves as a win-win situation for all involved because they are able to rid themselves of an unwanted shell while simultaneously acquiring a new one, thus minimizing risk and vulnerability. Asynchronous vacancy chains are a less social affair. One begins when a hermit crab finds a suitable empty shell and discards their own current shell. At a later time, another hermit crab will come upon the recently vacated shell and discard their own shell. The chain will continue until a shell that is either too small or damaged for any crab to find it suitable[2]. These asynchronous chains are less beneficial and efficient than their synchronous counterpart because there is no guarantee that a discarded shell will be of suitable size and condition for the next crab that comes upon it. More broadly, vacancy chain theory is also used to describe how limited resources can be reused and passed down. Human applications of vacancy chains are found in the housing and job market. These chains are marked by interdependent events that lead to the systematic propagation of the vacancy down a socioeconomic order[2,6]. The ultimate result of these human vacancy chains is the same as the vacancy chains found in hermit crab populations: the greatest
It is clear that while human communication has evolved tremendously, there is still much to learn from the communication methods used by other species. Innovation is a key part of human nature, but there is still much to learn from species that have continued to thrive on a changing planet without artificial innovations. The term biomimicry encapsulates this ideal of using nature as a model for human systems and processes [5]. Although certain similar human and animal processes may have developed analogously rather than directly inspiring each other, inspiration from nature can often lead to new and better human innovation. Some may scoff at the idea that the communication methods of ants and hermit crabs could possibly have any relevance to the sophistication of human communication, but nature has proved its resilience and ingenuity time and time again making it a worthy muse for the innovators of tomorrow.
References
1. Pinter-Wollman, N., Wollman, R., Guetz, A., Holmes, S., & Gordon, D. (2011, November 7). The effect of individual variation on the structure and function of interaction networks in harvester ants. Retrieved 2020, from https://www.ncbi.nlm. nih.gov/pmc/articles/PMC3177612/ 2. Rotjan, R., Chabot, J., & Lewis, S. (2010, April 01). Social context of shell acquisition in Coenobita clypeatus hermit crabs. Retrieved 2020, from https://academic.oup.com/beheco/article/21/3/639/220022 3. Khan, S. (2018). Animal communication (article) | Ecology. Retrieved 2020, from https://www.khanacademy.org/science/ ap-biology/ecology-ap/responses-to-the-environment/a/animal-communication 4. Krause, J. (2015, January). Animal Social Networks. Retrieved 2020, from https://oxford.universitypressscholarship. com/view/10.1093/acprof:oso/9780199679041.001.0001/acprof-9780199679041?rskey=wcjqZj 5. Wey, T., Blumstein, D., Shen, W., & Jordรกn, F. (2007, December 26). Social network analysis of animal behaviour: A promising tool for the study of sociality. Retrieved 2020, from https://www.sciencedirect.com/science/article/abs/pii/ S0003347207004393 6. Chase, I. D. (1991). Vacancy Chains. Retrieved 2020, from https://www.jstor.org/stable/2083338?seq=1
FALL 2020 | PENNSCIENCE JOURNAL 21
FEATURE
1 2
3
4
.
:
22
PENNSCIENCE JOURNAL | FALL 2020
:
5
6
7
FALL 2020 | PENNSCIENCE JOURNAL 23
0 2 20 e c n e i c S n n Pe t h g i l h g Hi in h c r a e s Re 24
PENNSCIENCE JOURNAL | FALL 2020
Predicting Minimum Inhibitory Concentrations with Machine Learning By Madeline Cook Advisors: Dr. Andrew Cohen, Dr. Gail Rose, Taha Valizadeh Aslani Drexel University Philadelphia, PA
ABSTRACT
Antibiotic resistance is a significant problem affecting people’s health and is only worsening with time. The ideal method for addressing this problem is to utilize the minimum amount of a drug necessary so that bacteria do not become resistant. Current efforts to predict this minimum via culture testing are inefficient. This paper looks at a developing method using machine learning to identify the genomic makeup of a bacteria and then utlizing that information to predict the minimum inhibitory concentration of a given antibiotic to treat a bacteria. The machine learning classification techniques applied are decision trees and random forests. These methods were found to have a two-fold dilution accuracy of 92% and yielded a decision tree simple enough for practical use. Further development in this field could lead to significant medical advances, improved patient health, and fiscal savings. INTRODUCTION
When repeatedly treated with the same antibiotic, bacteria can develop a resistance to medications that were once able to successfully treat them. This consequence is called antimicrobial resistance and as a result the amount and type of antibiotic needed to treat a bacteria are constantly increasing. This leads to increased medical expenditures and higher mortality rates, because more antibiotics are needed and may not be available or administered in time. Minimum inhibitory concentration (MIC) is the minimum amount of antibiotic needed in μg/mL to inhibit the visible growth of a bacteria. This paper examines a data set of 895 experimental trials with obtained MIC values for a specific combination of antibiotic type and genomic sequence. The data set was obtained from Dr. Gail Rosen’s lab at Drexel University. Dr. Rosen’s lab downloaded the genomes and MIC values from the Pathosystems Resource Integration Center (PATRIC) database [1]. 33 types of antibiotics were included in the trial as well as the frequency values for 64 different 3-mers, three element sequences from within the larger genomic pattern of the bacteria being tested. Dr. Rosen’s lab converted the genomes to 3-mers using KMC3, a tool for working with k-mer databases [2]. Figure 1 is a histogram showing the frequency of MIC values from the data set for all antibiotics and 3-mer combinations of the bacteria Staphylococcus aureus.
Figure 1: Plot of the frequency of log2 MIC values for various antibiotics. Generated in MATLAB by Ben Teperov By gathering data about the genomic makeup of a specific bacteria and searching for patterns between the frequency of a specific genomic sequence in that bacteria, the antibiotic used, and the observed MIC, the MIC to treat other instances of the same type bacteria with a specific antibiotic can be determined. Furthermore, by utilizing machine learning for this purpose, entire genomic sequences can now be assessed and at significantly faster speeds than what is currently seen using the conventional culture-based method [3]. This method increases the amount of data which can be analyzed, the speed at which it is done, and the accuracy of predictions; all of these factors contribute to improved outcomes for patients and room for further applications of machine learning within the medical space. FALL 2020 | PENNSCIENCE JOURNAL 25
RELATED WORK
METHODOLOGY
Work related to this paper can be partitioned into two categories: works with a similar goal of using machine learning to determine minimum inhibitory concentrations and works with different goals but similar methodologies such as classifiers or feature vectors used. Dr. Gail Rosen’s lab was attempting to achieve similar accuracy results to the paper “Developing an in silico minimum inhibitory concentration panel test for Klebsiella pneumoniae” by M. Nguyen [3], but had been unable to do so. Nguyen looked at the bacteria Klebsiella pneumoniae and used the genomic makeup of various bacteria samples in conjunction with the applied antibiotic to try to predict MIC values. Nguyen’s study consisted of only 20 antibiotics and used an XGBoost based machine, achieving two-fold dilution accuracy of 92%. This study was an expansion of an earlier work, “Using Machine Learning To Predict Antimicrobial MICs and Associated Genomic Features for Nontyphoidal Salmonella” [4], which also utilized machine learning and knowledge of genomic features to predict MIC values for Nontyphoidal Salmonella. This earlier work consisted of just 15 antibiotics and also utilized and XGBoost-based learning machine, achieving two-fold dilution accuracy of 95%. The work done in this paper builds upon and differs from [3] and [4] because it evaluates 33 antibiotics, a different bacteria (Staphylococcus aureus), and uses decision trees and random forests as the classifier in order to achieve results similar to those in the aforementioned papers.
Works still within this field of microbiology, which applied the classification methods of either a decision tree or a random forest and used genomic sequences as features include: “The Use of Machine Learning Methodologies to Analyse Antibiotic and Biocide Susceptibility in Staphylococcus aureus” [5], “Evaluation of parameters affecting performance and reliability of machine learning-based antibiotic susceptibility testing from whole genome sequencing data” [6], and “Understanding and predicting ciprofloxacin minimum inhibitory concentration in Escherichia coli with machine learning” [7]. [5], [6], and [7] influenced the work done in this paper by providing model designs and comparisons for decision trees using pruning, random forests using feature selection, and measurement metrics such as dilution accuracy, MSE, and R2 correlation values. 26
PENNSCIENCE JOURNAL | FALL 2020
RESEARCH
The desired outputs, or classifications, were the MIC values in units of μg/ mL with a scale of log2; therefore an MIC value of 5 within the scope of this project would in reality be 32 μg/mL. In order to maintain simplicity within this document MIC values will be referred to as their value used within computation and without units. The 33 antibiotics used in the trials were entered into the feature vector using One Hot Encoding. This method gave the antibiotic used more weight in the feature vectors; rather than the antibiotic being represented by one feature of 65, it was represented over 33 features out of 97 total features. The remainder of the feature vector for each trial was the frequency value of the 64 3-mers found in the specific bacteria sample for that trial. A sample view of this data set is shown in Table I. In table I, the first column is the antibiotic name which is alternatively represented using decimal, binary, and one hot encoding in the 3rd, 4th, and 5th-38th columns, the second column is the MIC value output, and the last 3 columns on the right (and the remaining columns not shown on screen) represent the frequency of the 3-mer listed in the top row. The table below shows 39 of the available 895 rows and 40 of the available 102 columns. The software used is MATLAB 2019a due to its robust Statistics and Machine Learning Toolbox. A decision tree was chosen as the method of classification because its sequential splitting procedure functions well in scenarios with many possible classes; this was ideal for this experiment given that the possible classes were any numbers between the maximum MIC value of 8 and a minimum MIC value of -4.0589. Furthermore, decision trees are speedily trained, require minimal memory, and demonstrate high generalization accuracy. Lastly from a nonprogrammer perspective, the decision tree was selected for its ease of interpretability so that it can be used by medical or science professionals without a strong background in programming.
Upon increased research into work already completed calculating MIC values by machine learning techniques, the decision tree approach was expanded to a random forest approach based on the assertion that it yields better results [8,3]. Using MATLAB a forest of 100 classification trees was generated and then the trees were aggregated using
Table I: View of Inputs and Outputs in Excel File.
a mixture of boosting and bagging algorithms. The data points were randomly assigned to training and test sets of 447 points each categorizing this experiment as supervised learning. Outliers were not removed because they signify real but underrepresented strains that exist in nature. Optimization techniques were first performed on a single classification tree generated by the MATLAB command fitctree using the training data as inputs. The optimization methods employed on classification tree included: manual feature selection, Predictor Selection, automatic hyperparameter optimization, pruning, and pruning with cross validation. Manual feature selection was performed by creating a not optimized classification tree and then using MATLAB’s Predictor Importance command to find with zero importance, meaning that the feature had no impact to the tree’s overall MSE. MATLAB computes predictor importance by executing a split at every predictor, summing the resulting change to MSE, and dividing that sum by the total number of branch nodes. The predictor importance estimates found are shown in Figure 2. In Figure 2, predictors with an importance of 0 have no effect on the classification outcome; predictors x1-x33 correspond to the antibiotic used, while x34-x97 correspond to 3-mer frequencies. As Figure 2 shows, the most important factor in determining MIC value is the antibiotic used, shown as predictors 0-33. Other important factors are specific genomic sequences that play large roles in antimicrobial resistance. Genomic sequences whose frequency values had 0 importance, or no effect on the MIC value prediction, were removed to simplify
Figure 2: MATLAB’s Unbiased Predictor Importance Estimates. computation and attribute more weight to the more important sequences. Predictor Selection is a builtin fitctree parameter and a form of feature selection with the overall goal of minimizing the p value in a chi-squared test measuring independence between feature vectors and the assigned classification. This independence or lack thereof is calculated comparing all possible splits in the tree to the split configuration that has been predetermined as optimal by growing out the tree. The best split returns the highest predictor association estimate. The association values found by this test are shown in Figure 4.
FALL 2020 | PENNSCIENCE JOURNAL 27
Figure 3: MATLAB’s Predictor Association Estimates. Figure 3 measures the relationship among features in determining a classification; a score of 0 signifies complete independence. As is shown here, predictors x1-x33, which represent the antibiotic used, are independent features. The dark blue rectangular sections indicating an association value of 0, show the independence of antibiotic chosen with every other feature; in other words the genomic sequence frequencies in the bacteria are not related to the antibiotic used to treat the bacteria. Furthermore, the diagonal yellow line, indicates the association value of 1, high dependence between each feature and itself. The more revealing section of this association matrix is the bottom right. This region of the graph shows the dependence of among genomic sequence frequencies. If two genomic sequence frequencies are extremely dependent then it is likely that one of those features can be removed, improving the ratio of data points to feature vector dimensions which leads to better model accuracy. Automatic hyperparameter optimization tests several combinations of tree parameters such as maximum number of splits, minimum leaf size, and split criterion. Pruning is the process of growing a tree to full length and then lessening the number of nodes by removing those that do not affect overall performance; pruning by cross validation uses the same procedure but the pruning level is determined by performance on data previously unseen to the machine measured by MATLAB’s cvLoss command. The various optimization techniques tested here were a combination of methods addressed in class and methods found to be successful in a similar study on Staphylococcus aureus using decision trees performed as part of the BIOHYPO European Project [5]. 28
PENNSCIENCE JOURNAL | FALL 2020
Trees were first trained using the above optimization RESEARCH parameters to create a model; this model was then used with MATLAB’s predict command to generate MIC values for the data contained in the test set. The difference between the true and predicted MIC values for the test data was then calculated. The first performance metric utilized was mean squared error because while general classification is the algorithmic goal, the true goal is to generate classifications as close in numerical value to the true MIC value as possible. MSE was used as a measurement of the distance between the computed classification and the true MIC value, and a classification tree with no attempt made to optimization was used as a reference. The second performance metric utilized was two-fold dilution percentage. According to FDA standards an MIC value is acceptable for use if it is within the range 2 times the ideal MIC value and ½ the ideal MIC value [3]. Utilizing the performances of the classification tree as measured by MSE, several optimization techniques were combined and applied to the random forest. Three random forest (also called a tree ensemble) configurations were applied using MATLAB’s fitcensemble command. In all three configurations the reduced feature vectors found to be advantageous by the individual classification trees were used as the inputs to the tree ensemble. This ideal set of features was applied, because similarly to how One Hot Encoding assigns more importance to antibiotic type, genomic sequence selection applies additional weight to the genomic regions that are more important in determining MIC value [4]. The optimization techniques applied included automatic optimization hyperparameters this time with the added parameter of tree aggregation which included several types of boosting and bagging and a pruned tree ensemble. A random forest with no attempt at optimization was used as reference for error values. RESULTS & DISCUSSION The first set of results pertain to trials conducted using only one decision tree. All methods resulted in the same two-fold dilution accuracy, so mean squared error (MSE) served as the deterministic metric. Table II shows a comparison of the results. Mean squared error was used to measure
how close the predicted MIC value was to the true MIC value and two-fold dilution accuracy denotes how many predicted values were within the FDA standards for safe use.
doctor and patient should feel comfortable given that so much information is considered in deciding how much antibiotic to administer. The findings from the singular decision trees were expanded to a random forest approach. Using the results from Table II, the optimization techniques of optimized hyperparameters and pruning were applied to random forests. The feature vector inputs to these forests used the reduced feature selection found to be optimal in the decision tree approach. The results are shown below in Table III. Mean squared error was used to measure how close the predicted MIC value was to the true MIC value and twofold dilution accuracy denotes how many predicted values were within the FDA standards for safe use.
Table II: Mean Squared Error and Two-Fold Dilution Accuracy of Attempted Optimization Techniques for Decision Trees. The best results were obtained by Pruning based on Cross Validation, and the resulting tree is shown in Figure 4.
Table III: Mean Squared Error and Two-Fold Dilution Accuracy of Attempted Optimization
The variable names x## in Figure 4 The best random forest was achieved by correspond to the features that represent either MATLAB’s pruning parameter. However, the best the antibiotic used or the frequency of a genomic results achieved overall, with respect to MSE, came sequence. This tree produces minimal error and from Pruning by Cross Validation used to create the is ideal for use by those without a programming decision tree in Figure 4. All configurations tested background due to its simple flow-chart like structure had high MSE values; this is attributed to the fact and few nodes. Although the structure of the tree that 3-mers do not provide enough information to in Figure 4 is complex, its flow-chart like structure capture the specific biological features that cause still allows for easy use. Additionally, both the Figure 4: Decision Tree Result using Pruning by Cross Validation.
FALL 2020 | PENNSCIENCE JOURNAL 29
antibiotic resistance. This is not a cause for concern because the MSE values only serve to compare amongst classification methods, whereas accuracy of the classifier is measured by double dilution. Classification methods attempted by other members of the team but not discussed here include XGBoost and Neural Networks; by comparing results the best performing classifier overall was identified as the decision tree with Pruning by Cross Validation. The accuracy as measured by two-fold dilution percentage by the decision tree pruned by cross validation (now to be referred to as the optimal decision tree) was equal to that of 92% found in the study using Random Forests and XGBoost to determine MIC values for Klebsiella pneumoniae [3]. However, the accuracy in predicting the exact MIC value is significantly smaller; the optimal decision tree had an accuracy of 48.32%, while the aforementioned study boasted an accuracy of 69% [3]. The confusion matrix in Table IV shows an alternative representation for the optimal decision tree accuracy. In the below Confusion Matrix, the blue color scheme represents accurate predictions, while the beige represents inaccurate predictions; the darker a color the higher the percentage of accuracy of inaccuracy. The numerical values for true and predicted class represent MIC values. The presence of more beige than blue in Table IV shows that this model struggles to exactly predict MIC values. Specifically, it often overestimates
the number of MIC values that are whole numbers; the RESEARCH magnitude of predictions, accurate or inaccurate, for the numbers -1, 0, 1, and 2, far outweigh the predictions of anything else. One possible explanation for the lower accuracy found in this paper when compared to similar studies is the use of shorter k-mers; for the trials used in this paper k-mers of length 3 were used whereas in the study with Klebsiella pneumoniae k-mers of length 10 were used. In most studies the minimum k-mer lengths are either 4 or 8, and the increase of accuracy in direct correlation to k-mer lengths has been cited to improve accuracy by as much as 6% [4]. Although this optimal decision tree model does not achieve exact accuracy, due to its simplicity and sufficient two-fold distribution accuracy still provides a very useful tool for medical professionals. CONCLUSION This paper applied supervised learning to predict MIC values based on applied antibiotic and genomic sequence frequency. Using several optimization techniques applied to both decision trees and random forests a singular decision tree pruned by cross validation was found to be a sufficient model. The models found in this study predicted MIC values within a threshold that is not only approved by the FDA, but also sufficiently small to still save doctor’s money by improving the conservativeness with which they can administer
Table IV: Confusion Matrix Generated Using MATLAB for Optimal Decision Tree.
30
PENNSCIENCE JOURNAL | FALL 2020
medicine and to improve patient outcomes by still lessening the amount of antibiotic they receive. Furthermore, the methods used in this paper show the level of simplicity with which these values can be calculated with or without a strong background in programming. For further research performed into antimicrobial resistance I would recommend k-mer sequences of at least length 10 and a data set with a larger number of trials; this will increase the amount of information each feature tells you about the bacteria being treated and create a better feature dimensionality to data point ratio. Through the application of these techniques doctors can have a better metric for determining minimum amount of antibiotic they can administer to a patient but still trust to be effective. By consistently administering the minimum amount of an antibiotic possible the growth of antibiotic resistance can be effectively minimized. REFERENCES [1]Davis JJ, Wattam AR, Aziz RK, Brettin T, Butler R, Butler RM, Chlenski P, Conrad N, Dickerman A, Dietrich EM, Gabbard JL, Gerdes S, Guard A, Kenyon RW, Machi D, Mao C, Murphy-Olson D, Nguyen M, Nordberg EK, Olsen GJ, Olson RD, Overbeek JC, Overbeek R, Parrello B, Pusch GD, Shukla M, Thomas C, VanOeffelen M, Vonstein V, Warren AS, Xia F, Xie D, Yoo H, Stevens R. The PATRIC Bioinformatics Resource Center: expanding data and analysis capabilities. Nucleic Acids Res. 2020 Jan 8;48(D1):D606-D612. doi: 10.1093/nar/gkz943. PMID: 31667520. PMCID: PMC7145515. [2] Marek Kokot, Maciej Długosz, Sebastian Deorowicz, KMC 3: counting and manipulating k-mer statistics, Bioinformatics, Volume 33, Issue 17, 01 September 2017, Pages 2759–2761, https://doi. org/10.1093/bioinformatics/btx304 [3]M. Nguyen et al., “Developing an in silico minimum inhibitory concentration panel test for Klebsiella pneumoniae,” (in eng), Sci Rep, vol. 8, no. 1, p. 421, 01 2018, doi: 10.1038/s41598-01718972-w. [4]M. Nguyen et al., “Using Machine Learning To Predict Antimicrobial MICs and Associated Genomic Features for Nontyphoidal <em>Salmonella</em>,” Journal of Clinical Microbiology, vol. 57, no. 2, pp. e01260-18, 2019, doi: 10.1128/jcm.01260-18. [5]J. R. Coelho et al., “The Use of Machine Learning Methodologies to Analyse Antibiotic and
Biocide Susceptibility in Staphylococcus aureus,” PLOS ONE, vol. 8, no. 2, p. e55582, 2013, doi: 10.1371/journal.pone.0055582. [6]A. L. Hicks, N. Wheeler, L. Sánchez-Busó, J. L. Rakeman, S. R. Harris, and Y. H. Grad, “Evaluation of parameters affecting performance and reliability of machine learning-based antibiotic susceptibility testing from whole genome sequencing data,” PLOS Computational Biology, vol. 15, no. 9, p. e1007349, 2019, doi: 10.1371/journal.pcbi.1007349. [7]B. Á. Pataki et al., “Understanding and predicting ciprofloxacin minimum inhibitory concentration in <em>Escherichia coli</em> with machine learning,” bioRxiv, p. 806760, 2019, doi: 10.1101/806760. [8]N. C. Gordon et al., “Prediction of <span class=”named-content genus-species” id=”namedcontent-1”>Staphylococcus aureus</span> Antimicrobial Resistance by Whole-Genome Sequencing,” Journal of Clinical Microbiology, vol. 52, no. 4, p. 1182, 2014, doi: 10.1128/JCM.0311713. [9]J. C. Hyun, E. S. Kavvas, J. M. Monk, and B. O. Palsson, “Machine learning with random subspace ensembles identifies antimicrobial resistance determinants from pan-genomes of three pathogens,” PLOS Computational Biology, vol. 16, no. 3, p. e1007608, 2020, doi: 10.1371/journal. pcbi.1007608.
FALL 2020 | PENNSCIENCE JOURNAL 31
Interested in joining PennScience? We Want You! Contact www.pennscience.org PennScience is sponsored by the Science and Technology Wing at the University of Pennsylvania
32
PENNSCIENCE JOURNAL | FALL 2020