Introduction to Artificial Neural Network and Machine Learning

Artificial Neural Networks and Machine Learning Mujeeb Rehman O. First sem M.tech Dept.of CSE Govt. Engg. College Sreekrishnapuram

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

omrehman@gmail.com

December 20, 2011

ANN & Learning

December 20, 2011

1 / 35

Overview:

• Biological Neural Network • Artificial Neural Network : A brief introduction • Benefits of Artificial Neural Networks • Basic Models of Artificial Neural Networks • Types of Activation Function • McCulloch-Pitts model • Learning Process in ANN :Error correction Learning and Hebbian Learning • Types of Learning • An Example of supervised learning :Radial Basis Function Network

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

2 / 35

Biological Neural Network:

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

3 / 35

Artificial Neural Network : A brief introduction

Human brain is highly complex, nonlinear and parallel computer.

The Concept of Artificial Neural Network has been motivated by Human brain. “A Neural network is a massively parallel distributed processor made up of simple processing units, which has a natural propensity for storing experiential knowledge and making it available for use[1].�

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

4 / 35

Benefits of Artificial Neural Networks:

1

Nonlinearity.

2

Input-Output Mapping.

3

Adaptivity.

4

Evidential Response.

5

Contextual Information.

6

Fault Tolerance.

7

VLSI Implementability.

8

Uniformity of Analysis and Design.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

5 / 35

Basic Models of Artificial Neural Networks:

Basic Models are specified by the three basic entities, namely: 1

Synaptic interconnection

2

Training or Learning

3

Activation function

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

6 / 35

Models based on Synaptic interconnection:

The Synapse is a bulb-like organ which is used to interconnect neurons.These types of model based on the interconnection of neurons (i.e., nodes). There exist five basic types of neuron connection architectures.

Single-layer feed-forward network Multilayer feed-forward network Single node with its own feedback Single-layer recurrent network Multilayer recurrent network

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

7 / 35

Models based on Synaptic interconnection(continued): Single-layer feed-forward network

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

8 / 35

Models based on Synaptic interconnection(continued):

Multilayer feed-forward network

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

9 / 35

Models based on Synaptic interconnection(continued):

Single node with its own feedback

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

10 / 35

Models based on Synaptic interconnection(continued): Single-layer recurrent network

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

11 / 35

Models based on Synaptic interconnection(continued):

Multilayer recurrent network

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

12 / 35

Types of Activation Function:

Activation function is an integration function(say f) is associated with input.

1

Identity function: f(x) = x, ∀x

2

Binary step function: f(x) = {1 if x ≥ θ ; 0 if x < θ }

3

Bipolar step function: f(x) = {1 if x≥ θ ; -1 if x < θ }

4

Sigmoidal function: Binary sigmoidal function: f(x) = Bipolar sigmoidal function: f(x)

5

1 1+e −λx −λx = 1−e 1+e −λx

Ramp function: f(x) = { 1 if x > 1 ; x if 0 ≤ x ≤ 1 ; 0 if x < 0 }

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

13 / 35

McCulloch-Pitts Model:

The McCulloch-Pitts model was the earliest neural network.

The neurons are connected by directed weighted paths.

Activation function is binary.

The weight may be excitatory or inhibitory

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

14 / 35

McCulloch-Pitts Neuron(continued): Architecture:

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

15 / 35

McCulloh-Pitts Neuron(continued):

Architecture: x1 to xn possess excitatory weighted connection. xn+1 to xn+m possess inhibitory weighted connection.

Activation function is f(yin ) = { 1 if ≥ θ ; 0 if < θ } For inhibition : θ > nw − p.

For excitation : kw ≥ θ > (k − 1)w.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

16 / 35

Learning Process:

Learning is a process by which the free parameters of a neural network are adapted through a process of stimulation by the environment in which the network is embedded. The type of learning is determined by the manner in which the parameter changes take place[1].

The learning process consist of events like: 1.The Neural Network is stimulated.

2.The Neural Network undergoes changes with respect to the stimulation.

3.The Neural Network responds in a new way.

Learning can be broadly classified as : 1) Parameter Learning and 2) Structure learning.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

17 / 35

Learning Process:

There are five Basic Learning Rules: 1

Error-Correction learning

2

Memory-Based learning

3

Hebbian learning

4

Competitive learning

5

Boltzmann learning

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

18 / 35

Learning Process(continued): 1. Error correction learning: Let k be neuron driven by a signal vector X(n). yk (n) be response obtained from k. dk (n) be desired response from k. so, Control mechanism ek (n) = dk (n) â&#x2C6;&#x2019; yk (n) Aim is to minimize cost fn. Ξ(n) = 21 ek 2 (n).

By adjusting the weights we reaches a study states. for minmizing Ξ(n) we use delta rule, â&#x2C6;&#x2020;wkj = Ρek (n)xj (n) where , wkj (n) = weight or k exited by xj (n) X (n)

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

19 / 35

Hebbian Learning:

Theorem When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic changes takes place in one or both cells such that Aâ&#x20AC;&#x2122;s efficiency as one of the cells firing B, is increased[1][2].

We can expand and rephrase it as a two-part rule: 1. If two neurons on either side of a synapse are activated simultaneously, then the strength of the synapse is increased.

2. If two neurons on either side of a synapse are activated asynchronously, then the synapse is weakened.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

20 / 35

Hebbian Learning(continued):

There are four mechanisms that characterize a Hebbian synapse: 1

Time-dependent mechanism.

2

Local mechanism.

3

Interactive mechanism.

4

Conjunctional or correlational mechanism.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

21 / 35

Hebbian Learning(continued):

The adjustment applied to weight wkj at time step n is expressed in general form : â&#x2C6;&#x2020;wkj (n) = F (yk (n), xj (n))

(1)

where, yk = post-synaptic signal xj = pre-synaptic signal

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

22 / 35

Hebbian Learning(continued):

Hebb’s hypothesis: ∆wkj (n) = ηyk (n)xj (n)

(2)

where, η = a +ve constant that determines the rate of learning.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

23 / 35

Hebbian Learning(continued):

Covariance hypothesis: ∆wkj = η(xj − x 0 )(yk − y 0 )

(3)

where, x 0 = the time-averaged values of pre-synaptic signal. y 0 = the time-averaged values of post-synaptic signal.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

24 / 35

Types of Learning:

Mainly Three types of learning are there: 1

Supervised learning.

2

Unsupervised learning.

3

Reinforcement learning.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

25 / 35

Radial Basis Function Network:

The RBF based on curve fitting problem in a high dimensional space .

Radial-basis function is a set of function provided by hidden units, that constitute an arbitrary basis for the input pattern when they are expanded into the hidden space.

The basic RBF network consist of three layers, 1. The input layer made up of source nodes.

2. The second layer made up of only hidden layer.

3. The third layer consist of output nodes.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

26 / 35

Radial Basis Function Network(continued):

Cover’s Theorem on the separability of patterns:

Theorem A complex pattern-classification problem cast in high-dimensional space nonlinearity is more likely to be linearly separable than in low-dimensional space[1].

Let X be the set of N patterns x1 , x2 , ........xN . ∀xi ∈ X belonging either X1 or X2 . for each x ∈ X we can define a function ϕ(x) as, ϕ(x) = [ϕ1 (x), ϕ2 (x), ..........ϕm (x)]T

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

(4)

December 20, 2011

27 / 35

Radial Basis Function Network(continued): Cover’s Theorem on the separability of patterns: if x is a vector in an m0 -dimensional input space. Then ϕ(x) maps to m1 -dimensional vector space.

A dichotomy {X1 ,X2 } is said to be ϕ-separable if ∃ an m1 -dimensional vector w such that, w T ϕ(x) > 0

(5)

if x ∈ X1 w T ϕ(x) < 0

(6)

if x ∈ X2 The hyperplane defined by the equation,

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

w T ϕ(x) = 0

ANN & Learning

(7)

December 20, 2011

28 / 35

Radial Basis Function Network(continued):

Gaussian function: The most commonly used radial-basis function is a Gaussian function.

In a RBF network, r is the distance from the cluster centre.

The equation represents a Gaussian bell-shaped curve.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

29 / 35

Radial Basis Function Network(continued): Distance measure: For each neuron in the hidden layer, the weights represent the co-ordinates of the centre of the cluster.

Therefore, when that neuron receives an input pattern, X, the distance is found using the following equation: v u n uX rj = t (xj − wij )2

(8)

i=1

Width of hidden unit basis function:

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ϕj = e

(−

Pn 2 i=1 (xj −wij ) ) 2σ 2

ANN & Learning

(9)

December 20, 2011

30 / 35

Radial Basis Function Network(continued):

Example : An often quoted example which shows how the RBF network can handle a non-linearly separable function is the exclusive-or problem.

One solution has 2 inputs, 2 hidden units and 1 output.

The centres for the two hidden units are set at c1 = 0, 0 and c2 = 1,1 and the value of radius σ is chosen such that 2 σ 2 = 1.

The inputs are x, the distances from the centres squared are r, and the outputs from the hidden units are ϕ.

When all four examples of input patterns are shown to the network, the outputs of the two hidden units are shown in the following table.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

31 / 35

Radial Basis Function Network(continued):

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

32 / 35

Summary:

The concept of ANN is based on human brain mechanism.

Models of ANN is based on three entities.They are synaptic interconnection, learning rule and their activation function.

Based on synaptic interconnection we can model them in five ways.

Five types of activation functions are there.

McChulloch-pitts model was the first ever neuron model.

Learning process means, adjusting the free parameters of a neural network to get a desired out put.

There are five basic rules associated with learning process.

And three types of learning process.

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

33 / 35

References

Simon Haykin, “ Neural Network” Prentice-hall of india Pvt. Ltd. , 2008 S. N. Shivanadam , S. N. Deepa “ Principles of Soft Computing” Wiley India pvt. Ltd., 2007

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

ANN & Learning

December 20, 2011

34 / 35

Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )

Thank you

ANN & Learning

December 20, 2011

35 / 35