MJ Journal on Numerical Analysis 1 (1) (2017) 1-7 Website: www.math-journals.com/index.php/JNA doi: 10.14419/jna.v1i1.87 Research paper
Numerical solution of singular perturbation problems by using collocation neural networks Luma Naji Mohammed Tawfiq * Department of Mathematics, College of Education for pure science / Ibn Al-Haitham, University of Baghdad *Corresponding author E-mail: dr.lumanaji@yahoo.com
Abstract In this paper we propose a collocation neural network (CNN) for solving some well-known classes of singular perturbation problems. The proposed approach is based on an supervised combined neural networks. Firstly, the trial solutions of nano singularly perturbed problems are written in the form of collocation neural networks containing adjustable parameters (the weights and biases); results are then optimized by the back propagation with the Levenberg-Marquardt training algorithm. The proposed network is tested on series of nano-singular perturbation problems and the results are reported. Afterward, these results are compared with the analytic solution demonstrating the efficiency and applicability of the proposed network. Keywords: Singularly Perturbed Problems; Artificial Neural Network; Back Propagation; Levenberg- Marquardt Training Algorithm. AMS Subject Classification: 32S, 34B, 65Y05, 68M07, 68N30, 68T01, 68T05, 34L05, 34B15.
1. Introduction Singularly perturbed problems (SPP) are common in applied sciences and engineering. They often occur in, for example, fluid dynamics, quantum mechanics, chemical reactions, electrical networks, etc. A well known fact is that the solution of such problems has a multi scale character, i.e., there are thin transition layers where the solution varies very rapidly, while away from the layers the solution behaves regularly and varies slowly. For a detailed discussion on the analytical and numerical treatment of such problems one may refer to the Malley [1], Doolan et al. [2], Roos et al. [3], and Miller et al. [4]. Numerically, the presence of the perturbation parameter leads to difficulties when classical numerical techniques are used to solve such problems, this is due to the presence of the boundary layers in these problems, see for example [1], [5]. Even in the case when only the approximate solution is required. Many methods have been developed so far solving Singularly perturbed boundary value problems (SPBVP) , nowadays there is a new way of computing denominated artificial intelligence which through different methods is capable of managing the imprecision's and uncertainties that appear when trying to solve problems related to the real world, offering strong solution and of easy implementation. One of those techniques is known as Artificial Neural Networks (ANN). Inspired, in their origin, in the functioning of the human brain, and entitled with some intelligence. These are the combination of a great amount of elements of process–artificial neurons interconnected that operating in a parallel way get to solve problems related to aspects of classification. The construction of any given ANN we can identify, depending on the location in the network, three kind of computational neurons: input, output and hidden. This paper is organized as follows: the next section define the Nano-Singularly perturbed problems. In section 3, we describe the structure of ANN, in section 4, we describe the structure of CNN, description of the method gave in section 5. Section 6, introduce CNN models. Section 7, report our numerical finding accuracy of suggested network, finally conclusions gave in the last part of the paper.
Copyright Š 2016 Luma Naji Mohammed Tawfiq. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
2
MJ Journal on Numerical Analysis
2. Nanosingularly perturbed problems The term "perturbation problem" is generally used in mathematics when one deals with the following situation: There is a family of problems depending on a small parameter 0, which we denote by P, when = 0, we have the reduced problem Po. We want to study the relationship between the solution of P and the solution of Po under appropriate assumptions. The perturbation problem (PP) may consist of an ordinary differential equation, or a system of differential equations, dong with some given conditions, which illustrate the problem. Thus, the general form of the 2nd order singularly perturbed problems (SPP): (1) Where f are n-dimensional vector functions, x is a scalar variable in a given interval. A perturbation problem (1) is called SPP if ϵ→ 0, the solution y(x) converges to yo(x) only in some interval of x, but not throughout the entire interval, thus giving rise to an "boundary layers" phenomena at both end-points [6]. A perturbation problem (1) is called nanosingular perturbed problems (NSPP) if 0 ˂ ϵ ˂˂˂ 1.
3. Structure of neural network In an ANN expressions structure, architecture or topology, express the way in which computational neurons are organized in the network. Particularly, these terms are focused in the description of how the nodes are connected and in how the information is transmitted through the network. As it has been mentioned, the distribution of computational in the following: Number of levels or layers: neurons in the neural network is done forming levels or layers of a determined number of nodes each one. As there are input, output and hidden neurons, we can talk about an input layer, an output layer and single layer or multilayer hidden layers. By the peculiarity of the behavior of the input nodes some authors consider just two kinds of layers in the ANN, the hidden and the output. Connection patterns: Depending on the links between the elements of the different layers. the ANN can be classified as: totally connected, when all the outputs from a level get to all and each one of the nodes in the following level, if some of the links in the network are lost, then we say that the network is partially connected. Information flow: Another classification of the ANN is obtained by considering the direction of the flow of the through the layers, when any output of the neurons is input of neurons of the same level or preceding levels, the network is described as feed forward. In counter position if there is at least one connected exit as entrance of neurons of previous levels or of the same level, including themselves, the network is denominated of feedback. [7]
4. Structure of multilayer collocation neural network The structure of the multilayered collocation neural network (CNN) which will be involved later in the controller design is investigated in this section. The design of three-layered collocation neural network is as shown in Figure 1. It consists of an input node vector X ϵ Rp, a hidden layer of p activity functions with tansig. activation function in each hidden neuron and an output node vector Y ϵ Rm.
Fig. 1: Three-Layer Neural Network
There are many different training algorithms, but the most often used training algorithm is the back propagation (BP) rule [8]. ANN is trained to map a set of input data by iterative adjustment of the weights. Information from inputs is fed forward through the network to optimize the weights between neurons. Optimization of the weights is made by backward propagation of the error during training phase. The ANN reads the input and output values in the training data set and changes the value of the weighted links to reduce the difference between the predicted and target (observed) values. The error in prediction is minimized across many training cycles (iteration or epoch) until network reaches specified level of accuracy. A complete round of forward backward passes and weight adjustments using all input output pairs in the data set is called an
MJ Journal on Numerical Analysis
3
epoch or iteration [9]. In order to perform a supervised training we need a way of evaluating the ANN output error between the actual and the expected outputs. A popular measure is the mean squared error (MSE) or root mean squared error (RMSE) [10].
5. Description of the method This section illustrate how our approach can be used to the approximation solution of the nanosingular perturbation problems (NSPP) of the form: (2) Where x Ďľ D, D ďƒŒ R denoted the domain and y(x) is the solution to be computed. If yt(x, p) denoted a trial solution with adjustable parameters p, the problem is transformed to a discretize from: (3) In our proposed approach, the trial solution đ?‘Śđ?‘Ą employs an CNN and the parameters p correspond to the weights and biases of the neural architecture.
6. Collocation neural network models In this section, we describe solution of NSPP using suggested network, we choose a form for the trial function yt (x) such that it satisfies the BC’s. This is achieved by writing it as a sum of two terms: (4) Where N(x, p) is suggested neural network with parameters p and n input units fed with the input vector x. The term A (x) contains no adjustable parameters and satisfies the BC. The second term G is constructed so as not to contribute to the BC, since y t(x) satisfy them. This term can be formed by using suggested network whose weights and biases are to be adjusted in order to deal with the minimization problem. To illustrate the method, we will consider the 2nd order nanosingular perturbation problems: (5) Where đ?‘Ľâˆˆ [đ?‘Ž, đ?‘?] and the BC: (đ?‘Ž) = đ??´, đ?‘Ś(đ?‘?) = đ??ľ, 0˂ɛ˂˂˂1; a trial solution can be written as: (6) Where đ?‘ (đ?‘Ľ, đ?‘?) is the output of the CNN with one input unit for đ?‘Ľ and weights đ?‘?. The error quantity to be minimized is given by:
(7) Where the xi Ďľ [a, b]. Since,
(8) It is straight forward to compute the gradient of the error with respect to the parameters p.
7. Applications In this section, we apply collocation Neural Networks (CNN) for the computation of NSPP, using a multilayer CNN having one hidden layer with 5 hidden units (neurons) and one linear output unit. The sigmoid activation of each hidden unit is tansig, the analytic solution ya (x) was known in advance.
4
MJ Journal on Numerical Analysis
Example 1 Consider the following 2nd order nanosingular perturbation problem:
BC′s (Dirishlit case): y (0) = 0, y (1) = 0, x [0, 1] and the analytic solution [11]: -
,
According to the equation (6) the trial neural form of the solution is taken to be:
, and = 1/16 The CNN trained using a grid of equidistant points with h = 0.1 in [0, 1]. Table (1) gives the initial weight and bias of the designer network. The neural results introduced in Table (2) and Table (3) gives the performance of the train with epoch and time. Table 1: Initial Weight and Bias of the CNN of Example 1 x 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
ya(x) Analytic solution yt(x) CNN solution The error E(x) |yt(x) ya(x)| 0 0.000144313758481479 0.000144313758481479 -0.219412696003221 -0.219369172837636 4.35231655843138e-05 -0.173232211741820 -0.173283238665853 5.10269240329986e-05 0.0100016868358891 0.00999116115285745 1.05256830316980e-05 0.191859943153080 0.191912594092090 5.26509390099406e-05 0.265802228834080 0.265835892163686 3.36633296059152e-05 0.191859943153080 0.183094864940238 0.00876507821284245 0.0100016868358891 0.00999792580648937 3.76102939977918e-06 -0.173232211741820 -0.182303355937198 0.00907114419537800 -0.219412696003221 -0.355593603011004 0.136180907007783 0 -0.495435448656505 0.495435448656505 Table 2: Analytic and Neural Solution of Example 1, H = 0.1 Weights and bias for CNN Net.IW{1,1} Net.LW{2,1} Net.B{1} 0.1297 0.8133 0.4623 0.5522 0.7185 0.2875 0.14906 0.5008 0.9263 0.2585 0.8939 0.4963 0.5287 0.9507 0.8773 Table 3: The Performance of the Train with Epoch and Time, H = 0.1 Train Function Trainlm
Performance of train 6.16e-33
Epoch 34
Time 0:00:00
Msereg. 0.021613145304009
Now, if we take a grid of equidistant points with h = 0.05 in [0, 1], to get rid of the problems of early stopping then the results of CNN is better. Table (4) gives the initial weight and bias of the designer network when h= 0.05, and the neural results introduced in Table (5) and Table (6) gives the performance of the train with epoch and time. Table 4: Initial Weight and Bias of the CNN for Example 1 when H = 0.05
Net.IW{1,1} 0.7451 0.74385 0.2659 0.1134 0.0489
Weights and bias for CNN Net.LW{2,1} 0.2761 0.6573 0.6267 0.5741 0.5741
Table 5: Analytic and Neural Solution of Example 1 when H = 0.05
Net.B{1} 0.4093 .2843 0.8340 0.7573 0.8659
MJ Journal on Numerical Analysis
x 0 0.0500 0.1000 0.1500 0.2000 0.2500 0.3000 0.3500 0.4000 0.4500 0.5000 0.5500 0.6000 0.6500 0.7000 0.7500 0.8000 0.8500 0.9000 0.9500 1.0000
5
Analytic solution ya(x) 0 -0.149554961840330 -0.219412696003221 -0.222179020048938 -0.173232211741820 -0.0898457279954018 0.0100016868358891 0.108991923360310 0.191859943153080 0.246664275350676 0.265802228834080 0.246664275350676 0.191859943153080 0.108991923360310 0.0100016868358891 -0.0898457279954015 -0.173232211741820 -0.222179020048938 -0.219412696003221 -0.149554961840330 0
yt(x) CNN solution -2.87928462738840e-07 -0.146322260638484 -0.219411550872578 -0.222661660095107 -0.173242207598580 -0.0898279961423314 0.00989916043776440 0.108969204281101 0.191949600419003 0.246711528039770 0.265762604646483 0.246645604513010 0.191898536683671 0.108994726787815 0.00996165159421200 -0.0898113708349849 -0.173243743793646 -0.222564362335055 -0.219411841967667 -0.147016173999807 -1.83948325238603e-07
The error E(x) |yt(x) ya(x)| 2.87928462738840e-07 0.00323270120184607 1.14513064244925e-06 0.000482640046168353 9.99585675981463e-06 1.77318530703852e-05 0.000102526398124747 2.27190792088661e-05 8.96572659234984e-05 4.72526890941682e-05 3.96241875963987e-05 1.86708376657962e-05 3.85935305911023e-05 2.80342750424989e-06 4.00352416771443e-05 3.43571604166648e-05 1.15320518266349e-05 0.000385342286117141 8.54035553915500e-07 0.00253878784052336 6.80675195374736e-08
Table 6: The Performance of the Train with Epoch and Time, H= 0.05 Train Function Trainlm
Performance of train 6.33e-10
Epoch 205
Time 0:00:02
Msereg. 7.416547969532035e-07
Again solve this example with ϵ = 1/64, that is NSPP and we take a grid of equidistant points with h = 0.05 in [0, 1], to get rid of the problems of early stopping of the network. Table (7) gives the initial weight and bias of the designer network, the neural results introduced in Table (8) and Table (9) gives the performance of the train with epoch and time. Table 7: Initial Weight and Bias of the CNN for Example1, ϵ = 1/64, H= 0.05 Weights and bias for CNN Net.LW{2,1} 0.9582 0.4522 0.8885 0.4648 0.5852
Net.IW{1,1} 0.8266 0.6118 0.9146 0.8841 0.2453
Net.B{1} 0.0116 0.7284 0.4153 0.1186 0.5962
Table 8: The Performance of the Train with Epoch and Time, H= 0.05, ϵ = 1/64 Train Function Trainlm
Performance of train 3.17e-09
Epoch 270
Time 0:00:03
Msereg. 1.758532369007406e-06
Example 2 Consider the following nonlinear 2nd order nanosingular perturbed problem:
With BC: y (0) 0, y (1) = 1/2 and x [0, 1]. The analytic solution is [12]:
According to the equation (6) the trial neural form of the solution is taken to be: yt = ½ x + The CNN trained using a grid of ten equidistant points in [0, 1] and = 10-7. The neural and analytic results introduced in Table (10), the weight and bias of the designer network given in Table (11) and Table (12) gives the performance of the train with epoch and time. Table 9: Analytic and Neural Solution of Example 1, when H = 0.05, ϵ = 1/64
6
MJ Journal on Numerical Analysis
x 0 0.0500 0.1000 0.1500 0.2000 0.2500 0.3000 0.3500 0.4000 0.4500 0.5000 0.5500 0.6000 0.6500 0.7000 0.7500 0.8000 0.8500 0.9000 0.9500 1.0000
Analytic solution ya(x) 0 -0.304932720419843 -0.454583880156263 -0.491686018109092 -0.451018685146743 -0.362232180541477 -0.251107348163182 -0.139802989450818 -0.0465159812423089 0.0151160402765120 0.0366189934736865 0.0151160402765119 -0.0465159812423088 -0.139802989450818 -0.251107348163182 -0.362232180541477 -0.451018685146743 -0.491686018109091 -0.454583880156263 -0.304932720419843 0
The error E(x) |yt(x) ya(x)| 1.93756838773318e-06 0.00375455745880271 3.48476754282734e-05 0.000113172277080187 0.000146936978118628 5.66771910881503e-05 6.06013719962051e-05 6.79609992610686e-05 8.13896157401267e-06 1.48563147918564e-05 0.000153821013327442 8.55514170037056e-06 6.94715228265891e-06 0.000209047223341474 2.78994127128618e-06 3.48001326900271e-05 1.35740301288045e-06 0.00196682757690070 0.00479074086163672 8.21453943811168e-07 1.64402206903613e-07
yt(x) CNN solution -1.93756838773318e-06 -0.301178162961040 -0.454549032480834 -0.491799190386172 -0.450871748168624 -0.362288857732565 -0.251167949535178 -0.139735028451557 -0.0465241202038829 0.0151011839617201 0.0367728144870140 0.0151245954182123 -0.0465229283945915 -0.139593942227476 -0.251104558221910 -0.362266980674167 -0.451020042549756 -0.493652845685992 -0.459374621017899 -0.304931898965899 -1.64402206903613e-07
Table 10: Analytic and Neural Solution of Example2 x 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Analytic solution ya(x) 0 0.909090909090909 0.833333333333333 0.769230769230769 0.714285714285714 0.666666666666667 0.625000000000000 0.588235294117647 0.555555555555556 0.526315789473684 0.500000000000000
The error E(x) |yt(x) ya(x)| 1.69516981588211e-06 7.05190694949565e-07 1.04938964579082e-05 3.97818557831275e-05 8.80079920984045e-05 7.92887707286027e-05 2.95098530584959e-05 0.000224512180094338 0.000237875006939503 8.61008455401713e-05 0.000791656859063683
yt(x) CNN solution 1.69516981588211e-06 0.909091614281604 0.833343827229791 0.769190987374986 0.714373722277813 0.666587377895938 0.625029509853059 0.588459806297741 0.555793430562495 0.526229688628144 0.499208343140936
Table 11: Initial Weight and Bias of the CNN for Example 2 Weights and bias for CNN Net.LW{2,1} 0.5710 0.1769 0.9574 0.2653 0.9246
Net.IW{1,1} 0.2238 0.3736 0.0875 0.6401 0.1806
Net.B{1} 0.1841 0.7258 0.3704 0.8416 0.7342
Table 12: The Performance of the Train with Epoch and Time Train Function Trainlm
Performance of train 5.05e-32
Epoch 260
Time 0:00:03
Msereg. 6.402611650035661e-08
8. Conclusion In this paper, some well-known classes of nanosingular perturbed problem were investigated by using a collocation neural network (CNN). We used supervised combined neural networks containing adjustable parameters; also, the results were optimized by using combined neural network. We trained the Neural Network, and after that we could obtain the result for every point. We reported results and compared this method with the analytic solution, the results show that the solutions are so accurate in these problems.
Reference [1] R. E. O’Malley, Singular Perturbation Methods for Ordinary Differential Equations, Springer Verlag, New York, 1991.
https://doi.org/10.1007/978-1-4612-0977-5. [2] E. P. Doolan, J. J. H. Miller and W. H. A. Schilders, Uniform Numerical Methods for Problems with Initial and Boundary Layers,
Boole Press, Dublin, Ireland, 1980.
MJ Journal on Numerical Analysis
7
[3] H. G. Roos, M. Stynes and L. Tobiska, Numerical Methods for Singularly Perturbed Differential Equations, Springer, Berlin, 1996.
https://doi.org/10.1007/978-3-662-03206-0. [4] J. J. H. Miller, E. O’Riordan and G. I. Shishkin, Fitted Numerical Methods for Singular Perturbation Problems, World Scientific,
Singapore, 1996. https://doi.org/10.1142/2933. R. E. O’Malley, Introduction to Singular Perturbations, Academic Press, New York, 1974. M. Jianzhong, Some Singular Singularly Perturbed Problems, Calgary, Alberta, 1997. S. Haykin , Neural networks: A comprehensive foundation,1993. A. I. Galushkin, Neural Networks Theory, Springer, Berlin, Germany, 2007. M. H. Ali, Design fast feed forward neural networks to solve two point boundary value problems, MSc thesis, University of Baghdad, College of Education for science/Ibn Al-Haitham, 2012. [10] A. Ghaffari, H. Abdollahi, M. R. Khoshayand, I. S. Bozchalooi, A. Dadgar, and M. Rafiee-Tehrani, Performance comparison of neural network training algorithms in modeling of bimodal drug delivery, International Journal of Pharmaceutics, Vol. 327, No. 12, 2006, pp:126–138. https://doi.org/10.1016/j.ijpharm.2006.07.056. [11] F. Ma and L. J. Fu, Principle of multi-time scale order reduction and its application in AC/DC hybrid power systems. In International Conference on Electrical Machines and Systems, 2008, pp: 3951 – 3956. [12] M. K. Kadalbajoo and D. Kumar, Initial value technique for singularly perturbed two point boundary value problems using an exponentially fitted finite difference scheme, I.I.T Kanpur, Kanpur, India, 2009. [5] [6] [7] [8] [9]