Artificial Neural Networks and Machine Learning Mujeeb Rehman O. First sem M.tech Dept.of CSE Govt. Engg. College Sreekrishnapuram
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
omrehman@gmail.com
December 20, 2011
ANN & Learning
December 20, 2011
1 / 35
Overview:
• Biological Neural Network • Artificial Neural Network : A brief introduction • Benefits of Artificial Neural Networks • Basic Models of Artificial Neural Networks • Types of Activation Function • McCulloch-Pitts model • Learning Process in ANN :Error correction Learning and Hebbian Learning • Types of Learning • An Example of supervised learning :Radial Basis Function Network
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
2 / 35
Biological Neural Network:
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
3 / 35
Artificial Neural Network : A brief introduction
Human brain is highly complex, nonlinear and parallel computer. The Concept of Artificial Neural Network has been motivated by Human brain. “A Neural network is a massively parallel distributed processor made up of simple processing units, which has a natural propensity for storing experiential knowledge and making it available for use[1].�
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
4 / 35
Benefits of Artificial Neural Networks:
1
Nonlinearity.
2
Input-Output Mapping.
3
Adaptivity.
4
Evidential Response.
5
Contextual Information.
6
Fault Tolerance.
7
VLSI Implementability.
8
Uniformity of Analysis and Design.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
5 / 35
Basic Models of Artificial Neural Networks:
Basic Models are specified by the three basic entities, namely: 1
Synaptic interconnection
2
Training or Learning
3
Activation function
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
6 / 35
Models based on Synaptic interconnection:
The Synapse is a bulb-like organ which is used to interconnect neurons.These types of model based on the interconnection of neurons (i.e., nodes). There exist five basic types of neuron connection architectures. Single-layer feed-forward network Multilayer feed-forward network Single node with its own feedback Single-layer recurrent network Multilayer recurrent network
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
7 / 35
Models based on Synaptic interconnection(continued): Single-layer feed-forward network
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
8 / 35
Models based on Synaptic interconnection(continued):
Multilayer feed-forward network
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
9 / 35
Models based on Synaptic interconnection(continued):
Single node with its own feedback
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
10 / 35
Models based on Synaptic interconnection(continued): Single-layer recurrent network
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
11 / 35
Models based on Synaptic interconnection(continued):
Multilayer recurrent network
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
12 / 35
Types of Activation Function:
Activation function is an integration function(say f) is associated with input. 1
Identity function: f(x) = x, ∀x
2
Binary step function: f(x) = {1 if x ≥ θ ; 0 if x < θ }
3
Bipolar step function: f(x) = {1 if x≥ θ ; -1 if x < θ }
4
Sigmoidal function: Binary sigmoidal function: f(x) = Bipolar sigmoidal function: f(x)
5
1 1+e −λx −λx = 1−e 1+e −λx
Ramp function: f(x) = { 1 if x > 1 ; x if 0 ≤ x ≤ 1 ; 0 if x < 0 }
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
13 / 35
McCulloch-Pitts Model:
The McCulloch-Pitts model was the earliest neural network. The neurons are connected by directed weighted paths. Activation function is binary. The weight may be excitatory or inhibitory
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
14 / 35
McCulloch-Pitts Neuron(continued): Architecture:
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
15 / 35
McCulloh-Pitts Neuron(continued):
Architecture: x1 to xn possess excitatory weighted connection. xn+1 to xn+m possess inhibitory weighted connection. Activation function is f(yin ) = { 1 if ≥ θ ; 0 if < θ } For inhibition : θ > nw − p. For excitation : kw ≥ θ > (k − 1)w.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
16 / 35
Learning Process:
Learning is a process by which the free parameters of a neural network are adapted through a process of stimulation by the environment in which the network is embedded. The type of learning is determined by the manner in which the parameter changes take place[1]. The learning process consist of events like: 1.The Neural Network is stimulated. 2.The Neural Network undergoes changes with respect to the stimulation. 3.The Neural Network responds in a new way. Learning can be broadly classified as : 1) Parameter Learning and 2) Structure learning.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
17 / 35
Learning Process:
There are five Basic Learning Rules: 1
Error-Correction learning
2
Memory-Based learning
3
Hebbian learning
4
Competitive learning
5
Boltzmann learning
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
18 / 35
Learning Process(continued): 1. Error correction learning: Let k be neuron driven by a signal vector X(n). yk (n) be response obtained from k. dk (n) be desired response from k. so, Control mechanism ek (n) = dk (n) â&#x2C6;&#x2019; yk (n) Aim is to minimize cost fn. Ξ(n) = 21 ek 2 (n). By adjusting the weights we reaches a study states. for minmizing Ξ(n) we use delta rule, â&#x2C6;&#x2020;wkj = Ρek (n)xj (n) where , wkj (n) = weight or k exited by xj (n) X (n)
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
19 / 35
Hebbian Learning:
Theorem When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic changes takes place in one or both cells such that Aâ&#x20AC;&#x2122;s efficiency as one of the cells firing B, is increased[1][2]. We can expand and rephrase it as a two-part rule: 1. If two neurons on either side of a synapse are activated simultaneously, then the strength of the synapse is increased. 2. If two neurons on either side of a synapse are activated asynchronously, then the synapse is weakened.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
20 / 35
Hebbian Learning(continued):
There are four mechanisms that characterize a Hebbian synapse: 1
Time-dependent mechanism.
2
Local mechanism.
3
Interactive mechanism.
4
Conjunctional or correlational mechanism.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
21 / 35
Hebbian Learning(continued):
The adjustment applied to weight wkj at time step n is expressed in general form : â&#x2C6;&#x2020;wkj (n) = F (yk (n), xj (n))
(1)
where, yk = post-synaptic signal xj = pre-synaptic signal
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
22 / 35
Hebbian Learning(continued):
Hebb’s hypothesis: ∆wkj (n) = ηyk (n)xj (n)
(2)
where, η = a +ve constant that determines the rate of learning.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
23 / 35
Hebbian Learning(continued):
Covariance hypothesis: ∆wkj = η(xj − x 0 )(yk − y 0 )
(3)
where, x 0 = the time-averaged values of pre-synaptic signal. y 0 = the time-averaged values of post-synaptic signal.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
24 / 35
Types of Learning:
Mainly Three types of learning are there: 1
Supervised learning.
2
Unsupervised learning.
3
Reinforcement learning.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
25 / 35
Radial Basis Function Network:
The RBF based on curve fitting problem in a high dimensional space . Radial-basis function is a set of function provided by hidden units, that constitute an arbitrary basis for the input pattern when they are expanded into the hidden space. The basic RBF network consist of three layers, 1. The input layer made up of source nodes. 2. The second layer made up of only hidden layer. 3. The third layer consist of output nodes.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
26 / 35
Radial Basis Function Network(continued):
Cover’s Theorem on the separability of patterns:
Theorem A complex pattern-classification problem cast in high-dimensional space nonlinearity is more likely to be linearly separable than in low-dimensional space[1]. Let X be the set of N patterns x1 , x2 , ........xN . ∀xi ∈ X belonging either X1 or X2 . for each x ∈ X we can define a function ϕ(x) as, ϕ(x) = [ϕ1 (x), ϕ2 (x), ..........ϕm (x)]T
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
(4)
December 20, 2011
27 / 35
Radial Basis Function Network(continued): Cover’s Theorem on the separability of patterns: if x is a vector in an m0 -dimensional input space. Then ϕ(x) maps to m1 -dimensional vector space. A dichotomy {X1 ,X2 } is said to be ϕ-separable if ∃ an m1 -dimensional vector w such that, w T ϕ(x) > 0
(5)
if x ∈ X1 w T ϕ(x) < 0
(6)
if x ∈ X2 The hyperplane defined by the equation,
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
w T ϕ(x) = 0
ANN & Learning
(7)
December 20, 2011
28 / 35
Radial Basis Function Network(continued):
Gaussian function: The most commonly used radial-basis function is a Gaussian function. In a RBF network, r is the distance from the cluster centre. The equation represents a Gaussian bell-shaped curve.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
29 / 35
Radial Basis Function Network(continued): Distance measure: For each neuron in the hidden layer, the weights represent the co-ordinates of the centre of the cluster. Therefore, when that neuron receives an input pattern, X, the distance is found using the following equation: v u n uX rj = t (xj − wij )2
(8)
i=1
Width of hidden unit basis function:
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ϕj = e
(−
Pn 2 i=1 (xj −wij ) ) 2σ 2
ANN & Learning
(9)
December 20, 2011
30 / 35
Radial Basis Function Network(continued):
Example : An often quoted example which shows how the RBF network can handle a non-linearly separable function is the exclusive-or problem. One solution has 2 inputs, 2 hidden units and 1 output. The centres for the two hidden units are set at c1 = 0, 0 and c2 = 1,1 and the value of radius σ is chosen such that 2 σ 2 = 1. The inputs are x, the distances from the centres squared are r, and the outputs from the hidden units are ϕ. When all four examples of input patterns are shown to the network, the outputs of the two hidden units are shown in the following table.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
31 / 35
Radial Basis Function Network(continued):
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
32 / 35
Summary:
The concept of ANN is based on human brain mechanism. Models of ANN is based on three entities.They are synaptic interconnection, learning rule and their activation function. Based on synaptic interconnection we can model them in five ways. Five types of activation functions are there. McChulloch-pitts model was the first ever neuron model. Learning process means, adjusting the free parameters of a neural network to get a desired out put. There are five basic rules associated with learning process. And three types of learning process.
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
33 / 35
References
Simon Haykin, “ Neural Network” Prentice-hall of india Pvt. Ltd. , 2008 S. N. Shivanadam , S. N. Deepa “ Principles of Soft Computing” Wiley India pvt. Ltd., 2007
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
ANN & Learning
December 20, 2011
34 / 35
Mujeeb Rehman O. (G. E. C. Sreekrishnapuram )
Thank you
ANN & Learning
December 20, 2011
35 / 35