CLEAR Journal December 2017 Edition by Simple Groups

CLEAR DECEMBER 2017

CLEAR Journal (Computational Linguistics in Engineering And Research) M. Tech Computational Linguistics, Dept. of Computer Science and Engineering, Govt. Engineering College, Sreekrishnapuram, Palakkad678633 www.simplegroups.in simplequest.in@gmail.com

Editorial………………………………………… 4

Chief Editor Shine S Assistant Professor Dept. of Computer Science and Engineering Govt. Engineering College, Sreekrishnapuram, Palakkad678633

A Note on Deep Learning.......................................8

Editors Bhavya K Muhammed Shameem K Pavithra C P Pradeep T Sreelakshmi K Cover page and Layout Muhammed Shameem K

CLEAR March 2018 Invitation………………………………………27 Last word………………………………………28

Sreelakshmi K AI and Self Driving Cars.............................................13 Anisha T S Dragon personal digital assistant........................................16 Pavithra C P Hebbian learning.........................19 Lakshmi K Long Short Term Memory RNNs ..........................................22 Deepa C A

CLEAR DECEMBER 2017

Dear Readers, This edition of CLEAR Journal contains articles related to some interesting topics like Deep Learning, Self Driving Cars using Artificial Intelligence, Dragon Personal Digital Assistant, Hebbian Learning and Long Short Term Memory RNNs. In our last edition, we concentrated mainly on articles related to some trending topics like Common Sense Reasoning, Human Computer Dialogue Systems, Lexical Simplification and Sorting of Jumbled Story lines. The good responses we received from our readers reflect the wider acceptability and growing awareness of language processing among the technical community, which is definitely gratifying for the entire CLEAR team. On this hopeful prospect, I proudly present this edition of CLEAR Journal to our faithful readers and look forward to your opinions and criticisms.

Best Regards, Shine S (Chief Editor)

CLEAR DECEMBER 2017

LAUNCHING OF SIMPLE GROUPS ACTIVITIES

The inaugural ceremony of the launching of SIMPLE Groups Activities for the academic year 2017-2018 was held on 13th November 2017. The function was presided by Dr. Rafeeque P C, Head Of the Department CSE and inaugurated by Dr. P C Reghuraj, Principal, GEC Palakkad. Felicitation speech was given by Shine S, PG coordinator and A. R. Jayan, Convener, HRD Cell. Several teaching and non teaching staffs and student representatives from all B.Tech batches attended the program. The details of activities planned for this academic year were discussed in the function.

CLEAR DECEMBER 2017

WORKSHOP ON PYTHON A Workshop on GUI Programming using Python was organized at GEC, Sreekrishnapuram on 15th November 2017. The workshop was conducted at M.Tech Computational Linguistics Lab for S3 and S5 B.Tech CSE students. GUI Programming using Tkinter was introduced to the students using many example programs.

WORKSHOP ON R LANGUAGE One-day workshop on R Language was held at GEC, Sreekrishnapuram on 17th November 2017. The workshop was conducted at M.Tech Computational Linguistics Lab for First Year M.Tech Students. Datatypes, variables, decision making, loops and function concepts in R language were discussed using example programs.

CLEAR DECEMBER 2017

PUBLICATIONS  "Approaches for Translating Natural Language Query into Structured Query Language: A Survey", Varsha E and Dr. Rafeeque P C published paper in Second International Conference on Systems, Energy and Environment (ICSEE 2017) held at Government College of Engineering, Kannur.  "A Survey of Intention Detection Methodologies from User Queries", Sreelakshmi K and Dr. Rafeeque P C, published paper in Second International Conference on Systems, Energy and Environment (ICSEE 2017) held at Government College of Engineering, Kannur and secured the award for Best Paper.  "A Survey of Question Answering Systems by using various Methodologies", Sruthy K G and Naseer C, published paper in Second International Conference on Systems, Energy and Environment (ICSEE 2017) held at Government College of Engineering, Kannur.

INTERNSHIP  Jenny George and Nasreen K V have been selected for internship program at SysG Soft Solutions, Kakkanad, Kochi.  Ayishathahira C H, Shabna Nasser and Sruthy K Bhaskaran have been selected for internship program at UVJ Technologies, Kakkanad, Kochi.  Sreelakshmi K and Varsha E have been selected for internship program at Lymbyc, Bangalore.

Ph.D SELECTION  Reji Rahmath K and Rekha Raj C T of M.Tech Computational Linguistics, 2013-2015 batch have been selected for Ph.D at IIT, Palakkad.  Reshma O K of M.Tech Computational Linguistics, 2012-2014 batch has been selected for Ph.D at NIT, Calicut.

CLEAR DECEMBER 2017

A Note on Deep Learning Sreelakshmi K M.Tech Computational Linguistics Government Engineering College, Palakkad sreelakshmiknarayanan@gmail.com Deep learning can be at first seem as an intimidating idea, yet it's becoming progressively essential nowadays. The field of deep learning and artificial intelligence is at all the rage today, as computers get closer to delivering various capabilities like a human does. A wide range of products like Apple’s Siri, Amazon’s Alexa, Google Now, Microsoft’s Cortana etc. make use of this relevant area of technology to provide a proactive experience to its users.

Artificial Intelligence is the all encompassing umbrella that covers everything from Good Old Fashion AI all the way to latest trends in technology like Deep Learning. Deep learning is a subset of artificial intelligence, which includes most logic and rule-based systems designed to

CLEAR DECEMBER 2017

solve problems. Within AI, we have machine learning, which uses a set of algorithms that make use of given data to make and improve the decision making process. Within machine learning we have deep learning, which can make sense of the given data using multiple layers of abstraction. Artificial Neural Networks is where the field of Deep Learning had its genesis from. Deep learning is a form of machine learning that uses a model of computing that's very much inspired by the structure of the brain. A deep learning network can be considered as a stack of neural networks arranged one above the other. A neural network, in its simplest form, is composed of multiple layers of interconnected nodes that are arranged in a manner similar to neurons in the human brain. The training data is fed into the input layer, which in turn feed the data into any number of hidden layers where weighted functions are applied. Finally the neural network spits out an answer through the output layer. If the answer is accurate, then we won’t need many changes. However, if the answer is wrong based on pre-identified training data, the network will automatically make small changes to its weights. This is referred to as the backpropagation of errors 8

in a supervised learning environment. As the network is fed with more data over more training cycles, it learns from its mistakes and gets better at approximating the outcome.

Now, just think of a layer of many such artificial neural networks and the performance of such a computational model. The main advantage of the deep learning based model is that, as the amount of the data increases, the performance of such systems increases. But when we consider the scalability of other learning algorithms in machine learning with the amount of data, they reaches a saturation point of performance and wonâ&#x20AC;&#x2122;t go beyond that even if the amount of training data is increased. The results of deep learning models gets better with more training data and larger models. These in turn require more computation to train the system. Better algorithms, new insights and improved techniques also helps us to achieve better results.

CLEAR DECEMBER 2017

Another benefit of deep learning models is their ability to perform automatic feature extraction from raw data, also called feature learning. Deep learning algorithms seek to exploit the unknown structure in the input distribution in order to discover good representations, often at multiple levels, with higher-level learned features defined in terms of lower-level features. Automatically learning features at multiple levels of abstraction allow a system to learn complex functions mapping the input to the output directly from data, without depending completely on human-crafted features. Just think of the time it saves us because of its self feature learning property!. Deep learning excels on problem domains where the inputs (also output) are analog. Meaning, they are not a few quantities in a tabular format but instead are images of pixel data, documents of text data or files of audio data. That is why, these techniques are applied a lot in various applications of natural language processing, image processing etc. Deep neural networks have an architecture that can be adapted to new problems relatively simply. 9

Every problem cannot be converted to a mathematical form i.e. mathematical formulation (getting equations based on variables), is not possible for every problem. Deep learning wins the race there. Along with imparting a state-of-art accuracy for most of the problems Statistical learning model can solve, it can also learn even if a concrete mathematical formulation is not present. Due to its improved data processing models, deep learning generates actionable results when solving data science tasks. While machine learning works only with labelled data, deep learning supports unsupervised learning techniques that allow the system become smarter on its own. The capacity to determine the most important features allows deep learning to efficiently provide data scientists with concise and reliable analysis results. Deep learning is an approach that models human abstract thinking (or at least represents an attempt to approach it) rather than using it. However, this technology has a set of significant disadvantages despite all its benefits. Deep learning is a quite resource-demanding technology. It requires more powerful GPUs, high-performance graphics processing units, large amounts of storage to train the models, etc. Furthermore, this technology needs more time to train in comparison with traditional machine learning. In deep learning, the training process is based on analysing large amounts of data. But fast-moving and streaming input data provides little time for ensuring an efficient CLEAR DECEMBER 2017

training process. That is why data scientists have to adapt their deep learning algorithms in the way neural networks can handle large amounts of continuous input data. Another important disadvantage of deep learning software is that it is incapable of providing arguments why it has reached a certain conclusion. Unlike in case of traditional machine learning, we can not follow an algorithm to find out why your system has decided that it is a cat on a picture, not a dog. To correct errors in DL algorithms, you have to revise the whole algorithm. For a deep neural network as there is no proper defined mathematical model, the learning is said to be spread across the hidden units, and the learning process is considered a Black-Box there. Despite all its challenges, deep learning discovers new improved methods of unstructured big data analytics for those with the intention to use it. Indeed, various fields can gain significant benefits from using deep learning within their tasks of data processing. Deep learning stands behind numerous innovations like self-driving cars, both voice and image recognition, etc. This technology has occupied multiple aspects of human lives. Now let us see some of the applications of deep learning. Automatic colorization of black and white images is the problem of adding colour to black and white photographs. Deep learning can be used to use the objects and their context within the photograph to colour the image, much like a human operator might approach the problem. Generally the approach involves the use of very large 10

convolutional neural networks and supervised layers that recreate the image with the addition of colour. Automatically adding sounds to silent movies is the problem of synthesizing sounds to match a silent video. This application makes use of both convolutional neural networks and LSTM recurrent neural networks. The system is trained using 1000 examples of video with sound of a drum stick striking different surfaces and creating different sounds. A deep learning model associates the video frames with a database of pre-rerecorded sounds in order to select a sound to play that best matches what is happening in the scene. Automatic machine translation is a task where given words, phrase or sentence in one language is automatically translated into another language. It includes automatic translation of text and automatic translation of images. Text translation can be performed without any pre-processing of the sequence, allowing the algorithm to learn the dependencies between words and their mapping to a new language. Stacked networks of large LSTM recurrent neural networks are used to perform this translation. Convolutional neural networks are used to identify images that have letters and where the letters are in the scene. Once identified, they can be turned into text, translated and the image recreated with the translated text. This is often called instant visual translation. Automatic handwriting generation is a task where given a corpus of handwriting examples, the system generates new CLEAR DECEMBER 2017

handwriting for a given word or phrase. From the hand written corpus the relationship between the pen movement and the letters is learned and new examples can be generated ad hoc using deep learning techniques. Automatic text generation is an interesting task, where a corpus of text is learned and from this model new text is generated, wordby-word or character-by-character. The model is capable of learning how to spell, punctuate, form sentences and even capture the style of the text in the corpus. Large recurrent neural networks are used to learn the relationship between items in the sequences of input strings and then generate text. More recently LSTM recurrent neural networks are demonstrating great success on this problem using a character-based model, generating one character at time. Automatic game playing is a task where a model learns how to play a computer game based only on the pixels on the screen. This very difficult task is the domain of deep reinforcement models and is the breakthrough that DeepMind (now part of google) is renown for achieving. Object classification and detection in photographs requires the classification of objects within a photograph as one of a set of previously known objects. Deep learning based models can be applied for this task and achieve better results than state-of-art models. Automatic image caption generation is the task where given an image the system must generate a caption that describes the contents of the image. Once you can detect objects in 11

photographs and generate labels for those objects, you can see that the next step is to turn those labels into a coherent sentence description. Generally, the systems involve the use of very large convolutional neural networks for the object detection in the photographs and then a recurrent neural network like an LSTM to turn the labels into a coherent sentence. Over the last few years Deep Learning was applied to hundreds of problems, ranging from computer vision to natural language processing. In many cases Deep Learning outperformed previous work. Deep Learning is heavily used in both academia to study intelligence and in the industry in building intelligent systems to assist humans in various tasks and this is obviously a promising area for future research. References [1] “Deep Learning in a Nutshell”, http://nikhilbuduma.com/2014/12/29/deeplearning-in-a-nutshell/ , Accessed Online on – 21 December 2017. [2] “Why Deep Learning, and Why Now”, https://www.datanami.com/2017/01/30/deep -learning-now/ , Accessed Online on – 20 December 2017. [3] “Why Deep Learning Matters and What’s Next for Artificial Intelligence”, https://blog.algorithmia.com/ai-why-deeplearning-matters/ , Accessed Online on – 20 December 2017.

is-deep-learning/ , Accessed Online on – 21 December 2017. [5] “Deep Learning: Definition, Benefits, and Challenges”,https://stfalcon.com/en/blog/pos t/deep-learning-benefits-and-challenges , Accessed Online on – 21 December 2017. [6] “8 Inspirational Applications of Deep Learning”,https://machinelearningmastery.c om/inspirational-applications-deep-learning/ , Accessed Online on – 21 December 2017.

Machine learning shines in galaxy too Researchers had known that seven planets were orbiting the star. But Google Artificial Intelligence -- which enables computers to "learn" -- looked at archival data obtained by NASA's planet-hunting Kepler telescope and uncovered the eighth planet. Scientists trained a computer how to differentiate between images of cats and dogs. They refined their approach to identify exoplanets in Kepler data based on the change in light when a planet passed in front of its star. The neural network learned to identify these by using signals that had been vetted and confirmed in Kepler's planet catalog. In order to help find weaker signals of potential planets that researchers had missed, the neural network was trained to look for weak signals in star systems that were known to support multiple planets

[4] “What is Deep Learning?”, https://machinelearningmastery.com/what-

CLEAR DECEMBER 2017

AI and Self-Driving Cars Anisha T S M.Tech Computational Linguistics Government Engineering College, Palakkad anishats21@gmail.com

A self-driving car (otherwise called a driverless car, autonomous car etc.) and unmanned ground vehicle is a vehicle that is fit for detecting its condition and exploring without human information. Self-driving cars use a variety of techniques to detect their surroundings, such as radar, laser light, GPS, odometer and vision. History Experiments have been conducted on automating driving since at least the 1920s but took place in the 1950s. The first truly autonomous prototype cars appeared in the 1980s, with Carnegie Mellon Universityâ&#x20AC;&#x2122;s Navlab and ALV projects in 1984 and Mercedes-Benz and Bundeswehr University Munich's EUREKA Prometheus Project in 1987. In 2017 Audi stated that its latest A8 would be autonomous at up to speeds of 60 km/h using its "Audi AI". The driver would not have to do safety checks such as frequently gripping the steering wheel. The Audi A8 was claimed to be the first production car to reach level 3 autonomous driving and Audi would be the first manufacturer to use laser scanners in addition to cameras and ultrasonic sensors for their system. On the 7th November 2017, Waymo announced that it had begun testing driverless cars without a safety driver at the driver position.

CLEAR DECEMBER 2017

Companies like Toyota and Ford have dedicated billions of dollars in Research and Development for this technology. Services like Uber and Lyft, who currently pay humans to drive, would soon deploy an entire fleet of self driving cars. In a few years, we would see self driving cars being sold to regular consumers. But there is still a fear surrounding it. Perhaps it's because most people don't know how they work. How do they work? When we humans are in the driverâ&#x20AC;&#x2122;s seat, we are observing our environment by simultaneously receiving an input of our surroundings and processing it in order to make a decision on which way to turn the steering wheel and when to step on the brake. A self driving car is usually outfitted with GPS Unit, an inertial navigation system, and a range of sensors. It uses the positional information from GPS and navigation system to localize itself and the sensor data to build an internal map of its environment. Once it has its position on the internal map of the world down, it can then use that map to find the optimal path, which would avoid all kinds of obstacles, to its destination. What I just described is a very high level description of how self driving cars work. Moving toward true AI, deep learning is a set of algorithms in machine learning that 13

attempt to model high-level data concepts by using architectures of multiple non-linear transformations. Various deep learning architectures such as deep neural networks (DNNs), convolution neural networks (CNNs), and deep belief networks are being applied to several fields such as computer vision, automatic speech recognition, natural language processing, and music/audio signal recognition where they have proven to be astoundingly responsive and accurate. Artificial Intelligence in Self Driving Cars Artificial intelligence, AI is going to have a very major impact on the economy in the near future, roughly 5 to 10 years from now. Robotics is one thing that comes to our mind, but its market is not big enough to justify such a big investment. Chatbots may be another market; it’s not a big enough market to justify these big investments. But when we think about cars, it has a huge market and caters to virtually any human being. It is an ideal platform for AI because if you want to have a self driving car, you would need sensors that understand the world at perception levels that are very close to that of humans and understand driving negotiations as to how we merge into traffic. In short, we would need a car that can be considered a competitor to human intelligence. In order to enable a car to drive on its own, we need to cover three technological pillars: 1. Sensing We all have senses such as seeing, hearing, and touching, all of which collects signal, which are then sent to the brain and directs our bodies to take certain actions. In the CLEAR DECEMBER 2017

same way, we need to enable self driving cars to have sensors that cover 360 degrees using cameras and sensors for redundancy like radar scanners. The laser scanners then take the data from all these sensors and go through a very powerful high-performing computing device in order to build an environmental model that can:  Tell where all the entities around the car are, such as pedestrians, cyclists, road turns, signals, symbols, barriers, etc.  Tell what these entities are doing or what their statuses are.  Tell about the path, such as if this path is a highway. 2. Mapping Mapping should not be confused with navigational map (Google map). Instead, they are maps that are high definition and are very detailed. Once you position your vehicle inside this map at a very high accuracy (in the range of 10 cm), you know everything about the roadway, the drivable paths, the delimiter, etc., the only thing that you don’t know about is where the road users are. That data is provided by sensing. These maps should reflect reality in a very short time, which means, the moment these environments change, the map must be updated immediately – today's maps are not updated at such a fast rate. This brings up the topic of using crowd sourcing to build maps. This is a very hard thing to do, but it's very much necessary. Without these detailed maps, you cannot reliably propel a self driving car.

3. Driving Policy This is where serious intelligence needs to be embedded into the driving platform. The car must be able to tell what pedestrians and other vehicles look like and what they're doing. Not only so, self driving car must understand driving policy, which is mostly about negotiation. It is all about building a strategy as to when we can merge into traffic lanes. Mind you, this skill does not come naturally to us — it often takes both driving lessons and driving experiences. This is a sophisticated negotiation and since it is already difficult for humans, it is going to be even more difficult for computers, because we are talking about broad intelligence which involves understanding which vehicles to give way to, which vehicles to take the way from, what roads are often used to merge into traffic. Also, driving policy changes according to locations because traffic negotiation differs from city to city. This has to be taken into consideration when the driving policy is being fed to the system. The period from 2015 to 2017, driving assist have been used. Driving Assist is a technology used to prevent accidents. This

technology has been used to help a handsfree vehicle stay in lane. With this technology, the driver still needs to stay alert. It is not really autonomous driving because the system can make mistakes. It doesn’t have 360 degree awareness. It doesn’t see everything. It is not designed to add all possible crash situations, which is why the driver needs to stay alert. Starting from 2018, we will be talking about Highly Autonomous Driving. These are real handsfree driving with limited scenarios, where it only works on major highways. Highly autonomous driving means that the driver does not have to have his or her eye on the road. The system will provide a grace period, between 10 to 30 seconds, for the driver to take over – if the driver doesn't, the system will know how to safely move the car aside and stop it.

References [1] " Living my anxiety dream: Taking a ride in a Google self-driving car", https://www.theguardian.com/technology/sel f-driving-cars , Accessed online on 20 December 2017.

Some Interesting facts    

Only about 10% of the world’s currency is physical money, the rest only exists on computers There are more than 5000 new computer viruses are released every month If there was a computer as powerful as the human brain, it would be able to do 38 thousand trillion operations per second and hold more than 3580 terabytes of memory The password for the computer controls of nuclear tipped missiles of the U.S was 00000000 for eight years

CLEAR DECEMBER 2017

Dragon Personal Digital Assistant Pavithra C P M.Tech Computational Linguistics Government Engineering College, Palakkad pavithracp029@gmail.com The first PDA was released in 1984 by Psion, the Organizer. Early PDA’s was devices having full keyboard and touch screen, which was also known as PALMs. The concept of virtual assistant was first developed by Joseph Weizenbaum of MIT in the late 60s. The first chatterbot was “ELIZA”. The principle used in the working of virtual assistant is:  



Natural Language processing: To understand user’s speech input. Automatic speech Recognition: To understand command according to user’s input. Artificial intelligence: To learn things from user and to store all information about behaviour and relations of user. Inter process communication : To get important information from other software applications

The hardware requirements for PDAs is, a phone with a touch screen interface, phone RAM should be of a minimum 512 MB, internet connectivity and the phone should have USB debugging mode for development and testing purposes. The operating system should be android 4.1/win, 8.1/IOS 6 or higher, the kernel version should be 3.0.16 or higher and support of other basic applications like maps, calendar, camera, web connection etc..

CLEAR DECEMBER 2017

Any virtual assistant basically consists of three layers: 1.

Speech to text : A piece of software used to converts audio to text.

Text analysing: Converted text is just letters for computer. So a piece of software converts text to something that is understandable for computer. Then the computer understands the command, so virtual assistant converts this text to computer commands. Virtual personal assistants maps the words to functions and parameters to create a command that computer can understand. (e.g. Reminder)

Interpret commands : This layer, map computer command to serve through internet. Simultaneously speech is evaluated locally. A local recogniser communicates with server to judge whether command will be best handle locally or not. (e.g. play music, restaurant reservation, movie rating etc.)

There have been multiple approaches to building personal assistant software, based on how the user enters the tasks and how the system interprets them. And there are many PDAs applications now-a-days. Dragon mobile assistant is one of the latest application of personal assistant. 16



Dragon Assistant holds the power of Nuance’s speech recognition technology, offering the freedom for flexibility to control the desktop applications using voice. With Dragon Assistant, one can speak commands to do lot of things on desktop such as search the web, post updates in social networks, check email and more. Dragon Assistant turns the power of talk into action. And it consists of more than 170 content providers that work seamlessly with applications to complete tasks, instead of just showing search results. The most important features of these app includes : 



Voice Biometrics: Open and close applications, put the computer to sleep and adjust settings with the accuracy of Natural Language Understanding, even when the computer is offline. Search the web: Search for reviews, make a reservation, get directions and get answers to just about anything. Dragon Assistant pulls up the actual application, such as Yelp or a real map, or replies with an answer, instead of pulling up a search results page.

CLEAR DECEMBER 2017



Get Social medias: Check and update social networks with simple voice commands. Email dictation and manage calendars: Dictate and send emails or search calendar for greater efficiency. Voice notifications : Dragon Mobile Assistant will read aloud your Facebook status updates, messages, incoming calls, alerts and upcoming appointments. You can also respond to any of these activities using your voice. Control playlists: Intuitive voice commands allows to select and play specific songs, artists and movies in personal library, as well as directly from online sources such as Spotify, Hulu, Netflix or YouTube.

The personal assistant can likewise help to manage schedule via things like setting reminders for events and adding appointments to our calendar. The company says its app learns the voice print so that it will only accept spoken commands from us and no one else. The app also supports what Nuance calls an Attentive Mode, which wakes the assistant up and has it listening even if our device’s screen is turned off. This is particularly useful while driving. The main pros of using PDAs are, these applications make small and smart handheld devices to combine multiple features, these allows to export and import data, store various information, make list, recognize voice commands, control various application etc.… The cons of using these includes,



 

Listing problem: PDAs get problem to process wrong pronounced words and miscellaneous words. Silent mode support: It gives response in voice output thus it doesn’t work properly in silent mode. Navigation languages: Most of PDAs can understand only English language. Internet access: It needs internet connection to give desired output.

Personal digital assistants are very effective way to organize the day to day schedules. The Virtual digital assistance provides an intelligent computer secretarial service for mobile professionals. The new service is based on convergence of internet, speech recognition technology and mobile technologies. The PDA minimizes the interruption of the user, improves the utilization of his time, and provides a single point of communication for all his messages, contacts, schedule, and source of information. Now there are many smart personal digital assistant applications available in market for various device platforms. These new software applications are performing really well than PDA devices as they provided with all resources of Smartphone. PDAs are also reliable than human personal assistant because, PDAs are more portable and can use them any time. They also have a lot of information than any assistant as they are connected with internet.

CLEAR DECEMBER 2017

This virtual and personal assistant, which is capable of emulating both logical and emotional interest in the interaction with its user, is a missing link in business support as it falls within the category of user support, supporting the end user in their needs and efforts. Bringing these ideas and technologies together would lead the way for a truly useful virtual personal assistant. Logical aspects make the system usable, however the combination of logical with emotional aspects make such a system worthwhile, and so ultimately useful for the user. References [1] "Personal digital assistant", https://en.wikipedia.org/wiki/Personal_digit al_assistant, Accessed online on 20 December 2017.

[2] "Advantages vs. disadvantages of owning a PDA" , http://www.ehow.co.uk/list_6012597_advan tages-vs_-disadvantages-owning-pda.html, Accessed online on 21 December 2017.

[3] "Simplify your mobile life" , https://www.nuance.com/mobile/mobileapplications/dragon-mobile-assistant.html, Accessed online on 21 December 2017. [4] " A whole new conversation" , https://www.nuance.com/dragon/dragonassistant.html, Accessed online on 21 December 2017.

Hebbian Learning Lakshmi K M.Tech Computational Linguistics Government Engineering College, Palakkad lakshmikskp@gmail.com

The aim of unsupervised learning is to present a neural net with raw data and allow the net to make its own representation of the data – hopefully retaining all information which we human finds important. Unsupervised learning in neural net is generally realised by using a form of Hebbian learning which is based on a proposal by Donald Hebb who wrote: “When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.” Hebbian Learning is the (hypothetical) process by which activity-dependent, longterm synaptic modifications organize neurons into functional networks, called cell assemblies and, subsequently, cell assemblies into phase sequences. A cell assembly is able to exhibit a stereotypical pattern of activation (reverberatory activity), which constitutes the neural mode of representation of elementary information. The activation of a cell assembly can facilitate, or even elicit, activity in other cell assemblies which have become associated through learning, i.e., through interassembly synaptic strengthening. The

CLEAR DECEMBER 2017

sequential activation of cell assemblies is a phase sequence, and constitutes the neural mode of integration of information. This functional organization underlies the brain’s ability to represent complex situations and, thereby, to produce flexible and contextually appropriate behaviour. Hebbian learning is one of the oldest learning algorithms, and is based in large part on the dynamics of biological systems. A synapse between two neurons is strengthened when the neurons on either side of the synapse (input and output) have highly correlated outputs. In essence, when an input neuron fires, if it frequently leads to the firing of the output neuron, the synapse is strengthened. Following the analogy to an artificial system, the tap weight is increased with high correlation between two sequential neurons. In more familiar terminology, that can be stated as the Hebbian Learning rule: 1. If two neurons on either side of a synapse (connection) are activated simultaneously (i.e. synchronously), then the strength of that synapse is selectively increased. This rule is often supplemented by:

2. If two neurons on either side of a synapse are activated asynchronously, then that synapse is selectively weakened or eliminated; so that chance coincidences do not build up connection strengths. Then, in the notation used for perceptrons, that can be written as the weight update: Δwij = η.outj.ini Here, η is a learning rate coefficient. In some cases, the Hebbian rule needs to be modified to counteract unconstrained growth of weight values, which takes place when excitations and responses consistently agree in sign. This corresponds to the Hebbian learning rule with saturation of the weights at a certain, pre-set level. Single Layer Network with Hebb Rule Learning of a set of input-output training vectors is called a HEBB NET The use of Hebbian learning method in neural networks is as shown in figure below:

Hebbian theory concerns how neurons might connect themselves to become engrams. The theory is often summarized by Siegrid Löwel's phrase: "Cells that fire together, wire together." However, this summary should not be taken literally. Hebb emphasized that cell A needs to "take part in firing" cell B, and such causality can only occur if cell A fires just before, not at the same time as, cell B. This important aspect of causation in Hebb's work foreshadowed what is now known about spike-timingdependent plasticity, which requires temporal precedence. The theory attempts to explain associative or Hebbian learning, in which simultaneous activation of cells leads to pronounced increases in synaptic strength between those cells, and provides a biological basis for errorless learning methods for education and memory rehabilitation. In the study of neural networks in cognitive function, it is often regarded as the neuronal basis of unsupervised learning. If the inputs to a system cause the same pattern of activity to occur repeatedly, the set of active elements constituting that pattern will become increasingly strongly inter-associated. That is, each element will tend to turn on every other element and (with negative weights) to turn off the elements that do not form part of the pattern. To put it another way, the pattern as a whole will become 'auto-associated'.

There is strong physiological evidence that, this type of learning does take place in the region of the brain known as hippocampus.

CLEAR DECEMBER 2017

Hebbian for the analysed neuronal

theory has been the primary basis conventional view that, when from a holistic level, engrams are nets or neural networks. A 20

variation of Hebbian learning that takes into account phenomena such as blocking and many other neural learning phenomena is the mathematical model of Harry Klopf. Klopf's model reproduces a great many biological phenomena, and is also simple to implement. Applications: Experiments on Hebbian synapse modification mechanisms at the central nervous system synapses of vertebrates are

much more difficult to control than are experiments with the relatively simple peripheral nervous system synapses studied in marine invertebrates. Much of the work on long-lasting synaptic changes between vertebrate neurons (such as long-term potentiation) involves the use of nonphysiological experimental stimulation of brain cells. However, some of the physiologically relevant synapse modification mechanisms that have been studied in vertebrate brains do seem to be examples of Hebbian processes

Facebook’s new “proactive detection” This is software to save lives. Facebook’s new “proactive detection” artificial intelligence technology will scan all posts for patterns of suicidal thoughts, and when necessary send mental health resources to the user at risk or their friends, or contact local first-responders. By using AI to flag worrisome posts to human moderators instead of waiting for user reports, Facebook can decrease how long it takes to send help. Facebook previously tested using AI to detect troubling posts and more prominently surface suicide reporting options to friends in the U.S. Now Facebook is will scour all types of content around the world with this AI, except in the European Union, where General Data Protection Regulation privacy laws on profiling users based on sensitive information complicate the use of this tech. The social media giant already has mechanisms for flagging posts from people thinking about harming themselves. The new feature is intended to detect such posts before anyone reports them. It reports the AI tools use pattern recognition to identify signals, like comments asking if someone is OK. The Facebook team will then be alerted and will work to quickly get the person help. The company is also using AI to prioritize the order that flagged posts are sent to its human moderators so they can quickly alert local authorities.

CLEAR DECEMBER 2017

Long Short Term Memory RNNs Deepa C A Research Scholar, Department of Computer Science and Engineering Government Engineering College, Palakkad deepa.ca0@gmail.com

Introduction Assume we are reading a story, or a news event , we understand each word based on our understanding of previous words. We keep in memory the necessary information needed and throw the unwanted or irrelevant information. This indicates that our thoughts have persistence. Even though neural networks are created to mimic human brain, they canâ&#x20AC;&#x2122;t do this task. Recurrent Neural Networks address this issue. Recurrent Neural Networks (RNN) used to map input sequences to output sequences, for applications such as sequence recognition, production, or time series prediction. All the applications mentioned above require a system that will store and update context information. Context information means the information computed from the past inputs and useful to produce desired outputs. Recurrent Neural Networks are well suited for those tasks, because they have an internal state that can represent context information. The cycles in the graph of a recurrent network allow it to keep information about past inputs for an amount of time, that is not fixed a priori, but rather depends on its weights and on input data. In contrast, static networks, i. e, networks with no recurrent connection, even if they include delays, such as time delay in neural network, have a finite CLEAR DECEMBER 2017

impulse. Therefore it cannot store a bit of information for an indefinite period of time. RNNs are networks with loops in them, allowing information to persist. Recurrent Neural Networks Unlike feed-forward networks, Recurrent networks, take input not just the current input example they see, but also what they have perceived previously in time. Hereâ&#x20AC;&#x2122;s a diagram of an early, simple recurrent network proposed by Elman, where the BTSXPE at the bottom of the drawing represents the input example in the current moment, and CONTEXT UNIT represents the output of the previous moment.

Figure 1: Simple Recurrent Network The decision of a recurrent net at time step t1 affects the decision taken by it one moment later, at time step t. So recurrent networks have two sources of input, the present and 22

the recent past, which combine to determine how they respond to new data, much as we do in life. Figure 2: Chunk of RNN A

these fields yield incredible success. These incredible success is due to the use of LSTMs(Long Short Term Memory), a very special kind of recurrent neural network which works, for many tasks, much much better than the standard version. Why LSTM? The key idea behind RNNs is that they are able to connect previous information to the present task. It works well in some situations where we only need to look at recent information to perform the present task.

In the above diagram, a chunk of neural network, A, looks at some input xt and outputs a value ht. A loop allows information to be passed from one step of the network to the next. A recurrent neural network can be thought of as multiple copies of the same network, each passing a message to a successor. When we are unrolling the loop, it may appear like in the following diagram.

For example, consider a language model trying to predict the next word based on the previous ones. If we are trying to predict the last word in “the clouds are in the sky,” we don’t need any further context – it’s pretty obvious the next word is going to be sky. In such cases, where the gap between the relevant information and the place that it’s needed is small, RNNs can learn to use the past information.

Figure 4: Storing recent context in RNN Figure 3: Unrolled RNN This chain-like nature of the recurrent neural networks reveals that they are related to sequences and lists. RNNs can be applied to a variety of problems, such as, speech recognition, language modelling, translation, image captioning etc. Applying RNNs to CLEAR DECEMBER 2017

Consider another example, where the network need to store more context. Consider trying to predict the last word in the text “I grew up in China… I speak fluent Chinese.” Recent information suggests that the next word is probably the name of a language, but if we want to narrow down 23

which language, we need the context of France, from further back. It’s entirely possible for the gap between the relevant information and the point where it is needed to become very large. RNNs become unable to learn to connect the information as the gap grows. Theoretically, RNNs are can absolutely handle long term dependencies. The user can carefully take parameters for such examples and solve those issues. Unfortunately in practice, RNNs are not capable to learn long term dependencies. Long Short Term Memory networks Long Short Term Memory networks – usually just called “LSTMs” – are a special kind of RNN, capable of learning long-term dependencies. Remembering information for long periods of time is practically their default behaviour. All recurrent neural networks have the form of a chain of repeating modules of neural network. In standard RNNs, this repeating module will have a very simple structure, such as a single tanh layer.

Figure 5: Repeating module in a standard RNN

structure. Instead of having a single neural network layer, there are four, interacting in a very special way.

Figure 6: Repeating module in LSTM Figure 6 shows the repeating module in an LSTM with four interacting layers. In Figure 6, each line carries an entire vector, from the output of one node to the inputs of others. The pink circles represent pointwise operations, like vector addition, while the yellow boxes are learned neural network layers. Lines merging denote concatenation, while a line forking denote its content being copied and the copies going to different locations. The horizontal line running through the top of the diagram is known to be cell state, which is the key idea of LSTMs. LSTM has the ability to remove or add information to the cell state, carefully regulated by structures called gates. Gates are a way to optionally let information through. They are composed of a sigmoid Neural Network layer and a pointwise multiplication operation.

LSTMs also have this chain like structure, but the repeating module has a different CLEAR DECEMBER 2017

Figure 7: The sigmoid layer The sigmoid layer outputs numbers between zero and one describing how much of each component should be let through. An output 0 indicates, let nothing through, while an output 1 indicates that let everything through. Briefly, an LSTM has three of these gates, in order to protect and control the cell state. Step-by-step Analysis of LSTM Step1: Decide what information is to be throw away from the cell state. This decision is made by a forget gate layer, which is a sigmoid layer. It looks at ht−1 and xt, and outputs a number between 0 and 1 for each number in the cell state Ct−1. A 1 represents “completely keep this” while a 0 represents “completely get rid of this.” Consider a language modelling problem, in which the next word is predicted using the information from previous words. In such a problem, let the cell state might include the gender of the present subject, so that the correct pronouns can be used. When it encounters a new subject, it’s necessary to forget the gender of the old subject.

CLEAR DECEMBER 2017

Figure 8: Forget gate

Step2: Decide what new information is going to be stored in the cell state. This step has two parts. 1. Sigmoid layer, i. e., the input gate layer decides which values want to be updated. 2. A tanh layer creates a vector of new candidate values, Ct, that would be added to the state. After that combine these two to create an update to the state.

Figure 9: decides which values to update

Step3: Update the old cell state Ct-1, into the new cell state Ct

For the language model example, since it just saw a subject, it might want to output information relevant to a verb, in case that’s what is coming next. For example, it might output whether the subject is singular or plural, so that we know what form a verb should be conjugated into if that’s what follows next. This is a simple LSTM. Almost every paper involving LSTMs uses a slightly different version. References:

Figure 10: Updating the cell state Step 4: Decide what are going to output. This output will be based on the cell state, but will be a filtered version. First, run a sigmoid layer which decides what parts of the cell state are going to output. Then, put the cell state through tanh (to push the values to be between −1 and 1) and multiply it by the output of the sigmoid gate, so that network only outputs the parts decided by it.

[1] Y. Bengio et. Al, “Learning Long term Dependencies with Gradient Descent”, IEEE Transactions on Neural Networks, Vol 5, No. 2, Mar 1994. [2] “Understanding LSTM networks”, http://colah.github.io/posts/2015-08Understanding-LSTMs/, visited on 21.12.2017 [3] “The Unreasonable Effectiveness of Recurrent Neural Networks”, http://karpathy.github.io/2015/05/21/rnneffectiveness/, visited on 21.12.2017 [4] “Exploring LSTMs” , http://blog.echen.me/2017/05/30/exploringlstms/, visited on 21.12.2017

Figure 11: Output of LSTM

CLEAR DECEMBER 2017

M.Tech Computational Linguistics Dept. of Computer Science and Engg, Govt. Engg. College, Sreekrishnapuram Palakkad www.simplegroups.in simplequest.in@gmail.com

SIMPLE Groups Students Innovations in Morphology Phonology and Language Engineering

Article Invitation for CLEAR- March-2018 We are inviting thought-provoking articles, interesting dialogues and healthy debates on multifaceted aspects of Computational Linguistics, for the forthcoming issue of CLEAR (Computational Linguistics in Engineering And Research) Journal, publishing on March 2018. The suggested areas of discussion are:

The articles may be sent to the Editor on or before 10th March, 2018 through the email simplequest.in@gmail.com. For more details visit: www.simplegroups.in Editor,

Representative,

CLEAR Journal

SIMPLE Groups

CLEAR DECEMBER 2017

Hello world, This issue of CLEAR Journal contains articles about some interesting topics like Deep Learning, Self Driving Cars using Artificial Intelligence, Dragon Personal Digital Assistant, Hebbian Learning and Long Short Term Memory RNNs. Deep learning have some how managed to capture the full attention of technologists and researchers and it is definitely a promising area for future studies. Artificial intelligence which aims to provide the best human like experience from machines, provides the best support for building systems like self driving cars that can be considered as a competitor to human intelligence. Personal digital assistants have conquered the daily life of humans by proving a proactive experience and Dragon mobile assistant is one of the latest application in this category. Hebbian learning is one of the oldest learning algorithms, and is based on the dynamics of biological systems. Long Short Term Memory (LSTM) is a special kind of RNN, capable of learning long-term dependencies and it finds application in many NLP tasks. The articles are written with the anticipation of shedding light on some interesting areas related to computational linguistics. CLEAR is thankful to all who have given their valuable time and effort for providing us with their precious ideas and suggestions. Simple group invites more aspirers in this field. Wish you all the success in your future endeavoursâ&#x20AC;Ś!!! Sreelakshmi K

CLEAR DECEMBER 2017