Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES Vol No. 2, Issue No. 2, 181 - 186
A MODIFIED NONLINEAR CONJUGATE GRADIENT METHOD FOR SOLVING NONLINEAR REGRESSION PROBLEMS k min f ( x k d k ),
Emmanuel Nwaeze
(3)
Department of Mathematics, University of Ilorin, Ilorin, Nigeria.
d k is a vector in the form
Abstract—In regression analysis, scientists face the
k 0 g k 1 , d k 1 g k 1 k d k , k 1
T
e-mail: nwaezeema@yahoo.com
problem of constructing a nonlinear mathematical function that
in which g k g ( x k ) f ( x k ) and
has the best fit to a series of experimental data points. Thus in
the CGM, i.e. different
Method (MNCGM) for solving the nonlinear regression Problems without linearizing the form of the function. Testing
the MNCGM on some problems (including polynomial regression
constructing a function that has the best fit to given experimental data points. The results obtained also show that the method is
1
g kT ( g k g k 1 )
PkT1 ( g k g k 1 )
2
3
A
automation.
I. INTRODUCTION
The Modified Nonlinear Conjugate Gradient Method (MNCGM) seeks to minimize an objective function f
IJ
f (x),
(1)
x k 1 x k k d k , k 0, 1, 2,...
(2)
N
R N is an N-dimensional Euclidean space
and f is n – times differentiable.
gk
g k 1
amenable to nonlinear regression analysis and computer
where x R,
determine different CGMs. For
instance, the following are formulae for some popular CGMs:
problems with many parameters) confirms the possibility of
min
s
k is a parameter of
ES
this paper, we present a Modified Nonlinear Conjugate Gradient
(4)
2
2
[3]
g kT ( g k g k 1 ) g k 1
4
g k 1 T
[5]
2
[7]
2
d k yk
[1]
Where y k g k 1 g k and
|| . || denotes the Euclidean
norm. The global convergence properties of the above
g (x) is the gradient vector. variants of CGM have already been established by many
A conjugate Gradient Method (CGM) for solving (1) uses an
authors
iterative scheme:
Zoutendijk[8].The cubic interpolation technique is among the
where
x0 R N is an initial point, k , the step size at
iteration k, is defined by
ISSN: 2230-7818
including
Powell[10],
Dai
and
Yuan[1]
and
most widely applicable and useful procedure for establishing convergence results of CGM. This technique (cubic interpolation) minimizes the one-dimensional function
@ 2011 http://www.ijaest.iserp.org. All rights Reserved.
Page 181
Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES Vol No. 2, Issue No. 2, 181 - 186
( ) f ( x k d k )
(5)
through an iterative procedure:
ALGORITHM I: (
are step lengths of the procedure.
( k 1 ) ( k ) k 1 k
cubic interpolation technique extensively in solving nonlinear optimization problems. Dai and Yuan used Wolfe conditions to establish the global convergence of a nonlinear conjugate gradient method as stated in the algorithm below. 1.
2
T
d k yk
(7)
becomes available. Bunday and Garside[2] and others used
,
g k 1
such that 0
1 2
2
f ( x k k d k ) optimize f ( x k d k ),
(6)
p 2 sign( k k 1 ) p1 ( k 1 ) ( k ) ,
4
k
Hence the value of
( k ) p 2 p1 k 1 k ( k k 1 ) , ( k ) ( k 1 ) 2 p 2
p1 ( k 1 ) ( k ) 3
and
, y k g k 1 g k )
The Least-Squares Polynomial of degree m:
Pn ( x) a0 a1 x a 2 x 2 . . . a n x n ,
(9)
where a 0 , a1 , ..., a n are constants to be determined
Step 2 Compute an k 0 that satisfies Wolfe conditions .
and
Step 3 Let x k 1 x k k d k . If g k 1 0 then stop. Step 4 Compute k by 4 and generate d k 1 by (1.4),
(10)
Pn ( x) bx a
where
function (model) of the form:
Pn ( x) f ( x1 , x2 , ...,xm , a0 , a1 , a 2 , ..., a n )
3.
A
P (dependent variable) and f is nonlinear
b0 b1 x b2 x 2 .. . br x r
,
(11)
to be determined.
analysis, we find the best-fitting function to a given set of residuals) of the points from the function. The sum of the
a 0 a1 x a 2 x 2 . . . a n x n
where a 0 , a1 , ..., a n ; b0 , b1 , ..., br are constants
with respect to the regression parameters. In regression points by minimizing the sum of the squares of the offsets (the
The Least-Squares Rational function of degree n + r:
Pn ( x)
x1 , x 2 , ...,x m (independent
IJ
a and b are constants to be determined.
(8)
where ( a 0 , a1 , a 2 , ..., a n ) are regression parameters to a
variables) versus
The Least-Squares Exponential function
Pn ( x) be ax
Nonlinear regression involves a general mathematical
set of m tabulated values of
n m 1.
ES
k k 1, go to step 2.
2.
T
Step 1 Given x1 v , d 1 g 1 , k 1, if g1 0 then stop.
4.
The Multivariable Nonlinear function
continuous differentiable quantity. NCGM is then used to
a 0 a1 x1 a 2 x 2 . . . a n x n (12) b0 b1 x1 b2 x 2 .. . br x r where a 0 , a1 , ..., a n ; b0 , b1 , ..., br are constants
minimize the sum of the squares of the offsets. The best-fit
to be determined.
squares of the offsets is used instead of the offset absolute values because this allows the residuals to be treated as a
function
that
approximates
a
set
of
data
points:
{ ( xi , yi ) | i 1, 2, ..., m } may take one of the following forms, depending on the relationship between the
Pn ( x)
Using any of the model functions above, the Modified Nonlinear Conjugate Gradient Method seeks to minimize the offset (error function)
x i and y i
values.
ISSN: 2230-7818
@ 2011 http://www.ijaest.iserp.org. All rights Reserved.
Page 182
Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES Vol No. 2, Issue No. 2, 181 - 186
m
F (a) y i Pn ( x i ) or 2
PROBLEM I:
i 1
m
F (a, b) y i Pn ( xi )
2
(13)
degree 3 for the data below.
i 1
over a, b where a and b . n
ALGORITHM II i. Input initial values x0 and d 0 g 0 . ii. Repeat:
1
2
3
4
yi
0
12
156
174
408
f ( x) 10 x 5x 2 7 x 3 , n = 3, m = 5)
0
by (10) Compute new point:
PROBLEM II: Construct the least squares polynomial of degree 10 for the data below.
xi
0
0.1
0.2
0.3
0.4
0.5
yi
5
5.266
5.484
5.676
5.872
6.138
0.6
0.7
0.8
0.9
1
1.1
6.635
7.745
10.292
15.962
28
52.337
ES
Find step length k such that
f ( x k k d k ) min f ( x k d k ) ,
f ( x) 5 3 x 4 x 2 7 x 3 9 x 4 6 x 5
8 x 6 3x 7 4 x 8 10 x 9 x 10
(n = 10, m = 12)
x k 1 x k k d k c.
0
T
The Modified Nonlinear Conjugate Gradient Method is Algorithm (I) driven by cubic interpolation line search procedure (10). This method inherits Dai and Yuan[1] convergence results of Algorithm (I) since cubic interpolation technique satisfies Wolfe conditions implicitly. The Algorithm is as follows.
b.
xi
r
II. MODIFIED NONLINEAR CONJUGATE GRADIENT METHOD
a.
Construct the least squares polynomial of
Update search direction:
Dk 1 g k 1 k d k ,
PROBLEM III: Construct the least squares exponential function of the form (10) for the data below.
2
A
k
g k 1 T
d k yk y k g k 1 g k d.
Check for optimality of
g m is so small that x m
is an acceptable estimate of the optimal point optimal set
x * of f . If not
k k 1.
III.
NUMERICAL EXPERIMENTS
The following constitute the test problems for the CGM on which Algorithm II was implemented in Visual Basic 6.0:
ISSN: 2230-7818
1
1.25
1.5
1.75
2
yi
5.1
5.79
6.53
7.45
8.46
n = 2, m = 5
g:
IJ
Terminate iteration at step m when
xi
PROBLEM IV: Construct the least squares nonlinear function of the form (1.12) for the data below (n = 10, m = 12).
x1 x2
yi 5
0
1
2
3
4
9.5
8.5
7.5
6.5
5.5
-4.5
0.234
0.079
0.023
-0.013
6
7
8
9
4.5
3.5
2.5
1.5
0.5
-0.046
-0.08
-0.143
-0.275
-0.982
@ 2011 http://www.ijaest.iserp.org. All rights Reserved.
Page 183
Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES Vol No. 2, Issue No. 2, 181 - 186
Time taken = 0.82 seconds.
ES
T
SOLUTION TO PROBLEM I
P3 ( x) 0.000096 9.998656 x 4.999029 x 2 6.999837 x3 P3 (3) 174.0004
Time taken = 0.45 seconds.
P10 ( x) 5.000068 2.959044 x 3.371729 x 2
P3 (5) 799.997298
3.597307 x 3 1.12348 x 4 0.105908 x 5
3.356655 x 6 5.336969 x 7 5.086751x 8 3.829213 x 9 3.434991x 10
IJ
A
SOLUTION TO PROBLEM II
ISSN: 2230-7818
P10 (2) 7683.00162 P10 (0.35) 5.77066 P10 (1) 27.99988
SOLUTION TO PROBLEM III
@ 2011 http://www.ijaest.iserp.org. All rights Reserved.
Page 184
Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES Vol No. 2, Issue No. 2, 181 - 186
P( x)
0.119524 0.090346 x1 0.075362 x 2 0.132537 0.231999 x1 x 2
P(6, 3.6) 0.08000128 P(5.5, 4) 0.06335777 IV. REMARK ON NUMERICAL RESULTS A cursory look at the tables reveals that
Time taken = 1.06 seconds.
results
obtained with the new conjugate gradient method is in close
P( x) 3.06697e 0.506879x P(1.5) 6.560118 P(3) 14.031814
agreement with the given data. Attainment of global optimum is possible with the new method.
V. CONCLUSION
T
SOLUTION TO PROBLEM IV
Herein, we present a Modified Nonlinear Conjugate Gradient Method (MNCGM) for solving nonlinear regression problems to scientists and engineers. Testing the MNCGM on
ES
some problems (including polynomial regression problems with many parameters) confirms the possibility of obtaining the global optimum of the objective function with high accuracy. The results obtained also show that the method is amenable to nonlinear regression analysis and computer
IJ
A
automation.
REFERENCES
[1]
[2]
[3] [4]
Y.H. Dai and Y. Yuan : “A nonlinear conjugate gradient method with a strong global convergence property”. SIAM J. Optim. 10, 177-182 (1999) B. D. Bunday and G. R. Garside : “Optimization methods in Pascal”. Printed and bound in Britain by Edward Arnold ltd, London (1987) R. Fletcher and G. M. Reeves : “Function minimization by conjugate gradients”. Computer J. 7, 149-154 (1964) E. Polak, “Optimization: Algorithms and Consistent Approximations”, vol. 124 of Applied Mathematical Sciences, Springer, New York, NY, USA, 1997.
Time taken = 0.41 seconds.
ISSN: 2230-7818
@ 2011 http://www.ijaest.iserp.org. All rights Reserved.
Page 185
Emmanuel Nwaeze (IJAEST) INTERNATIONAL JOURNAL OF ADVANCED ENGINEERING SCIENCES AND TECHNOLOGIES Vol No. 2, Issue No. 2, 181 - 186
[5]
[6]
[7]
[8]
M. Hestenes, and E. Stiefel: “Method of conjugate gradients for solving linear systems”. J. Res. Nat. Bur. Standards 49, 409-436 (1952) M. Ledvij “Curve Fitting Made Easy”. Industrial Physicist 9, 24-27, Apr./May 2003. E. Polak and G. Ribiére: Note sur la convergence de directions conjugées. Rev. Francaise Informat Recherche Opertionelle, 3e Année, 16, 35-43 (1969) G. Zoutendijk “Nonlinear programming, Nonlinear Programming”, J. Abadie, Ed., pp. 37–86, North-Holland, Amsterdam,
[10]
Squares Problems”. Englewood Cliffs, NJ: Prentice-Hall, 1974.
ES
[9]
The Netherlands, 1970. C. Lawson and R. Hanson “Solving Least
T
computational methods, in Integer and
M. J. D. Powell “Restart procedures for the conjugate gradient method”, Mathematical Programming, vol. 12, no. 2, pp. 241– 254, 1977.
[11] P. Lancaster and K. Šalkauskas “Curve and
IJ
A
Surface Fitting: An Introduction”. London: Academic Press, 1986.
ISSN: 2230-7818
@ 2011 http://www.ijaest.iserp.org. All rights Reserved.
Page 186