170 by ides editor

Poster Paper Proc. of Int. Conf. on Advances in Computer Engineering 2011

Implementation of Backpropagation Algorithm in Reconfigurable Hardware Nisha Thomas, Mercy Mathew Centre for Development of Advanced Computing Thiruvananthapuram, India nishathomas_26@yahoo.co.in Department of Electronics and Communication Engineering Carmel Engineering College, Pathanamthitta, India mercy13@gmail.com Abstract— Artificial Neural Networks (ANN) is highly parallel, highly interconnected systems. Such characteristics make their implementations very challenging due to the large amount of hardware required. This paper proposes the implementation of a neural network and back propagation algorithm on a reconfigurable hardware. The design and implementation of a nonlinear neuron is performed using 3 different activation functions. Finally the implementation of a feed forward neural network as well as back propagation training algorithm is discussed. The design entry is done in VHDL and simulated using Modelsim SE 6.4 design suite from Mentor Graphics. It is then synthesized and implemented using Xilinx ISE 10.1 targeted towards Xilinx QPro Virtex FPGA.

where y represents the output of the neuron, pj represents the inputs , wj represents the weight coefficients and f represents the activation function used. A. Activation Functions The behavior of ANN depends both on the weights and the activation function used. Activation function, also called transfer function defines the input output function for the units. This function may be step, ramp, sigmoid etc. 1) Step Activation Function The step function is the simplest of a non linear activation function and is based on the “all nothing” response of biological neurons. Step function can be defined as

Keywords- FPGA, Neural Network, Activation Function, Backpropagation Algorithm.

I. INTRODUCTION Artificial neural networks (ANNs) have been used successfully in applications including pattern recognition, pattern classification and control problems. Due to their adaptive, self organizing and non linear features, neural networks are very attractive for signal processing applications [1], [2], [3]. Their processing capabilities are based on their highly parallel and interconnected architecture[4].This makes their implementation challenging due to the large requirement of computational resources. So far, ANN’s have been implemented on multiprocessor platforms [2],[4],[5],[6] and different architectures like hyper cubes[7],[8] tree meshes[9], SIMD arrays[10],network of workstations[11] etc. Due to the technological advancement in the field of programmable devices, field programmable gate arrays (FPGA’s) have been attractive for realizing ANN’s [12]. It offers flexibility in terms of reprogramming as well as higher performance. The work discussed in this paper proposes the implementation of 3 different activation functions, as well as a feed forward neural network [13]. Also the implementation of back propagation learning algorithm using VHDL is discussed. These are implemented in Xilinx QPro Virtex family FPGA’s and the results are analyzed in terms of chip utilization.

where u is the sum of all inputs and ¸ is the threshold level. 2) Ramp Activation Function The ramp activation function produces an output which gradually grows from zero to a maximum level. In order to minimize the hardware resources, a unit slope ramp is implemented according to the expression

3) Sigmoid Activation Function Sigmoid activation functions are widely used in ANN, due to their monotonic character and derivability which makes them suitable for training algorithms. The sigmoid function cannot be easily implemented in digital hardware as it consists of an exponential series. Even though several approximations [13] of sigmoid functions exist, here we have adopted the kwan approximation [14], which is basically a quadratic equation, defined as follows.

where L depends on the level of saturation and Hs is defined as 185

Poster Paper Proc. of Int. Conf. on Advances in Computer Engineering 2011

Step1. Read the first input pattern and the associated output pattern. Also assign Converge =True Step2. For the input layer, assign the corresponding element in the input vector as the net input to each unit. The output for each unit is its net input. Step3. For the first hidden layer units- calculate the net input and output using the following equations.

Here ² and ¸ are the parameters for setting the slope and gain respectively. Sigmoid function is simulated in Modelsim SE 6.4 and the results obtained are plotted in a spreadsheet as shown in Fig 1. B. Neural Network Here we have chosen a feed forward network with 3 layers, having 3 input neurons in each layer. All neurons in the same layer are handled in parallelism. x1, x2 ,x3 are the inputs, and w is weight vector. The clock signals drive both the network layers. The 8-bit outputs of hidden layer neurons y1, y2, y3 are applied as input to output layer neuron. One approach for implementing this feed forward neural network is by using only one input to load all the weight inputs as shown in Fig.2. This helps to save the chip pins and hence reduce the area. The nine weights coefficients are stored in a ROM and are shifted into the network in a sequential manner until each register is loaded with its respective weight. The weights are then multiplied by the inputs and accumulated to get the desired output. Here the sigmoid activation function is implemented using the lookup table approach.

where w 0 refers to the bias weights. Repeat step 3 for all the subsequent hidden layers. Step4. For the output layer units- calculate the net input and output using (5) and (6) Step5. Check whether the difference between the target and the output pattern within the tolerance? If no, then set Convergence = False Step6. For each output unit calculate its error  j as  j = ( t j - o j ) o j ( 1 o j ) .

(7)

Step7. For the last hidden layer calculate the error for each unit as k w . k = o j ( 1  o j )  kj k

(8)

Repeat step 7 for all the subsequent hidden layers. Step8. For all layers update weights for each unit as w ij ( n+1) =  ( j o i )  w ij .

(9)

where η is the learning rate and α is the momentum term added to prevent the oscillatory nature of the weight changes. Step9. Check whether the last pattern has reached? If so, check whether Converge = = True? If yes, stop. Else go to next step. Step10. Read the next input pattern and associated output pattern. Go to step 2.

Figure 1. Sigmoid function (kwan approximation)

C. Back Propagation Algorithm A neural network can be taught to perform a particular task by using certain procedures. The back propagation (BP) algorithm [15], [16] is one of the supervised training algorithms for multilayered feed forward neural networks. .The proposed algorithm for back propagation is as follows.

III. IMPLEMENTATION RESULTS Simulation of neural network as well back propagation algorithm is done using Modelsim SE 6.4. Much iteration is required for the simulation of back propagation algorithm to make the error signal equal to zero. The implementation results of the proposed feed forward neural network as well as the backpropagation algorithm is listed in table 1 and table II respectively They are implemented using Xilinx ISE 10.1 and is targeted towards Xilinx QPro Virtex FPGAs. IV. CONCLUSION In this work an FPGA implementation of the neural network and back propagation algorithm is proposed. The FPGA implementation of the step, ramp and the sigmoid function is done and their performance is compared for nonlinearity. From the comparison we obtained that the sigmoid function is giving more nonlinearity than the other functions. As the next step we have implemented a feed forward neural network

Figure 2. Block diagram of the proposed neural network

186

Poster Paper Proc. of Int. Conf. on Advances in Computer Engineering 2011 REFERENCES

and back propagation algorithm on Xilinx QPro Virtex device. It is found that the implementation of neural networks using hardware description languages offer flexibility, ease of programming and high degree of parallelism. Neural networks are invaluable applications where formal analysis would be difficult or impossible, such as pattern recognition and nonlinear system identification and control. Also its FPGA implementation can be performed to obtain better performance than the existing technologies.

[1] B. H. Juang, S. Y. Kung, and C. A. Kamm, Eds., “Neural Networks for Signal Processing”, Proc. 1991 IEEE Workshop, Princeton, NJ, 1991,pp. 7–185. [2] B. Kosko, Ed., “Neural Networks for Signal Processing”, Englewood Cliffs, NJ: Prentice Hall, 1992. [3] V. Kumar, A. Grama, A. Gupta, and G. Karypis,” Introduction to Parallel Computing—Design and Analysis of Algorithms”, New York: Benjamin/Cummings, 1994. [4] Poliac M., Zanetti J., Salerno D., “Performance Measurements of Seismocardiogram Interpretation Using Neural Networks”, Computer in Cardiology, IEEE [5] E. Deprit, “Implementing Recurrent Backpropagation on the Connection Machine,” Neural Networks, vol. 2, pp. 295–314, 1989. [6] J. A. Feldman et al., “Computing with Structured Connectionist Networks,”Commun. ACM, vol. 31, no. 2, pp. 170–187, 1988. [7] V. Kumar, S. Shekhar, and M. B. Amin, “A Scalable Parallel Formulation of the Backpropagation Algorithm for Hypercubes and Related Architectures,”IEEE Trans. Parallel Distrib. Syst., vol. 5, pp. 1073–1090, Oct. 1994. [8] Zhou, R. Chellappa, A. Vaid, and R. K. Jenkins, “Image Restoration using a Neural Network,” IEEE Trans. Acoust., Speech, Signal Processing,vol. 36, pp. 1141–1151, July 1988. [9] S. P. Luttrell, “Image Compression using a Multilayer Neural Network,”Pattern Recognit., vol. 10, pp. 1–7, 1989. [10] S. Shams and K. W. Przytula, “Implementation of Multilayer Neural Networks on Parallel Programmable Digital Computers,” in M. Bayoumi,Ed., Parallel Algorithms and Architectures for DSP Applications.Norwell, MA: Kluwer, 1991, pp. 225–253. [11] S. Suresh, S.N. Omkar, and V. Mani,” Parallel Implementation of Back-Propagation Algorithm in Networks of Workstations”, IEEE Transactions on parallel and distributed systems, vol. 16, no. 1, January 2005. [12] Rucket U., Funke A. and Pintaske C., “Accelerator board for Neural Associative Memories”, Neurocomputing, Vol.5,No.1, 1993 pp 39-49. [13] S. Haykin, “Neural Networks: A Comprehensive Foundation”, Prentice- Hall, Englewood Cliffs, New Jersey, 1999. [14] Kwan H. K., “simple sigmoid like activation function suitable for digital hardware implementation” , Electronic Letters, Vol. 28, July. [15] R. Gadea, J. Cerda, F. Ballester, A. Mocholi, “Artificial Neural Network Implementation on a Single FPGA of a Pipelined On-line Back Propagation”, Proceedings of the 13th International Symposium on System Synthesis (ISSS’00), pp 225-230, Madrid, Spain. [16] S. Y. Kung, Digital Neural Networks. Englewood Cliffs, NJ, Prentice-Hall, 1993.

ACKNOWLEDGMENT We wish to thank Mrs. Roshni V S, Centre for Development of Advanced computing, Thiruvanathapuram for giving us an opportunity to do this work. T ABLE I. DEVICE UTILIZATION SUMMARY OF NEURAL NETWORK

TABLE II. DEVICE UTILIZATION SUMMARY OF BACK PROPAGATION ALGORITHM

187