Introduction to numerical linear algebra and optimisation
PHILIPPE
G. CIARLET
Uniuersite Pierre et Marie Curie. Paris with the assistance of Bernadette Miara and Jean-Marie Thomas for the exercises Translated by A. Buttigieg, S.J. Campion Hall, Oxford
_=-_~~_.,
..... ~.....CAMBRIDG~_E •••• :::
UNIVERSITY PR"
IS11W.OO ~
MOTIJA..usrtOl WD.. ••• ,AM \JS'll\) y vs •••••• ~O(J_
Il-_----~'_
Published by the Press Syndicate The Pitt Building, Trumpington 40 West 20th Street, New 10 Stamford Road, Oak leigh,
of the University of Cambridge Street, Cambridge CB2 I RP York, NY 10011, USA Melbourne 3166, Australia
published in French as Introduction a l'analvse numerique matricielle et a /'optimisation and Exercises d'analyse numerique matricielle et d'optimisation in the series Mathematiques appliquees pour la maltrise, editors P.G. Ciariet and l.L. Lions, Masson. Pairs,
Originally
1982 and ~~ Masson, First published
Editeur,
Paris, 1982, 1986
in English by Cambridge
Introduction
University
Press 1989 as
to numerical linear algebra and optimisation Reprinted
English translation
Š
1991, 1995
Cambridge
University
Press 1989
British Library cataloguing in publication data
0170509
o9
~GO
Ciarlet, Philippe, G. Introduction to numerical linear algebra and optimisation. (Cambridge texts in applied mathematics; v. 2). I. Equations - Numerical solutions 2. Mathematical optimisation I. Title II. Miara, Bernadette III. Thomas, lean-Marie IV. Introduction a l'analyse nurnerique matricielle et a I'optimisation. English 512.9'42 QA218
-
Library of Congress cataloguing in publication data
201l
Ciarlet, Philippe, G. . Introduction a l'analyse numerique matricielle et a l'optimisation. English] Introduction to numerical linear algebra and optimisation/Philippe G. Ciariet, with the assistance of Bernadette Miara and Jean-Marie Thomas for the exercises; translated by A. Buttigieg. p. em. Translation of: Introduction l'analyse numerique matricielle et l'optimisation and Exercises d'analyse numerique matricielle et d'optimisation. Bibliography: p. Includes index. ISBN 0 521 32788 1. ISBN 0 521 33984 7 (pbk.) I. Algebra, Linear. 2. Numerical calculations. 3. Mathematical optimization. I. Miara, Bernadette. II. Thomas, Jean-Marie. III. Title. IV. Title: Numerical linear algebra and optimisation. QA I84.C525 1988 512'.5-dc 19 87-35931 CIP
a
a
ISBN 0 521 32788 I hard covers ISBN 0 521 33984 7 paperback TM
Transferred
to di ital
'qBltOTECA ARTURO AGUERO CHAVES SEOE DE OCCIDENTE UNtVERStDAO DE COSTA RtCA
rinting
200 I
Contents
Preface
I Numerical linear algebra 1 A summary or results on matrices Introduction 1.1 Key definitions and notation 1.2 Reduction of matrices 1.3 Special properties of symmetric and Hermitian matrices 1.4 Vector and matrix norms 1.5 Sequences of vectors and matrices
1 1 2 10 15 20 32
2 General results in the numerical analysis or matrices Introduction 2.1 The two fundamental problems; general observations on the methods in use 2.2 Condition of a linear system 2.3 Condition of the eigenvalue problem
37 37
3 Sources of problems in the numerical analysis of matrices Introduction 3.1 The finite-difference method for a one-dimensional boundary-value problem 3.2 The finite-difference method for a two-dimensional boundary-value
~~
37 45 58 64 64 65
~
3.3 The finite-difference method for time-dependent boundary-value problems 3.4 Variational approximation of a one-dimensional boundary-value problem 3.5 Variational approximation of a two-dimensional boundary-value pro~m 3.6 Eigenvalue problems 3.7 Interpolation and approximation problems
1~ 110 115
4 Direct methods for the solution of linear systems Introduction 4.1 Two remarks concerning the solution of linear systems
124 124 125
86 94
viii
Contents
4.2 Gaussian elimination 4.3 The LU factorisation of a matrix 4.4 The Cholesky factorisation and method 4.5. The QR factorisation of a matrix and Householder's method
127 138 147 152
5 Iterative methods for the solution of linear systems Introduction 5.1 General results on iterative methods 5.2 Description of the methods of Jacobi, Gauss-Seidel and relaxation 5.3 Convergence of the Jacobi, Gauss-Seidel and relaxation methods
159 159 159 163 171
6 Methods for the calculation of eigenvalues and eigenvectors Introduction 6.1 The Jacobi method 6.2 The Givens-Householder method 6.3 The QR algorithm 6.4 Calculation of eigenvectors
186 186 187 196 204 212
II Optimisation 7 A review of differential calculus. Some applications Introduction 7.1 First and second derivatives of a function 7.2 Extrema of real functions: Lagrange multipliers 7.3 Extrema of real functions: consideration of the second derivatives 7.4 Extrema of real functions: consideration of convexity 7.5 Newton's method
216 216 218 232 240 241 251
8 General results on optimisation. Some algorithms Introduction 8.1 The projection theorem; some consequences 8.2 General results on optimisation problems 8.3 Examples of optimisation problems 8.4 Relaxation and gradient methods for unconstrained problems 8.5 Conjugate gradient methods for unconstrained problems 8.6 Relaxation, gradient and penalty-function methods for constrained problems
266 266 268 277 286 291 311
9 Introduction to non-linear programming Introduction 9.1 The Farkas lemma 9.2 The Kuhn-Tucker conditions 9.3 Lagrangians and saddle points. Introduction to duality 9.4 Uzawa's method
330 330 332 336 350 360
10 Linear programming Introduction 10.1 General results on linear programming
321
369 369 370
Contents 10.2 Examples of linear programming 10.3 The simplex method 10.4 Duality and linear programming
Bibliography and comments Main notations used Index
problems
IX
374 378 400 411 422
428
1 Preface to the English edition
My main purpose in writing this textbook was to give, within reasonable limits, a thorough description, and a rigorous mathematical analysis, of some of the most commonly used methods in Numerical Linear Algebra and Optimisation. Its contents should illustrate not only the remarkable efficiency of these methods, but also the interest per se of their mathematical analysis. If the first aspect should especially appeal to the more practically oriented readers and the second to the more mathematically oriented readers, it may be also hoped that both kinds of readers could develop a common interest in these two complementary aspects of Numerical Analysis. This textbook should be of interest to advanced undergraduate and beginning graduate students in Pure or Applied Mathematics, Mechanics, and Engineering. It should also be useful to practising engineers, physicists, biologists, economists, etc., wishing to acquire a basic knowledge of, or to implement, the basic numerical methods that are constantly used today. In all cases, it should prove easy for the instructor to adapt the contents to his or her needs and to the level of the audience. For instance, a three hours per week, one-semester, course can be based on Chapters 1 to 6, or on Chapters 7 to 10, or on Chapters 4 to 8. The mathematical prerequisites are relatively modest, especially in the first part. More specifically, I assumed that the readers are already reasonably familiar with the basic properties of matrices (including matrix computations) and of finite-dimensional vector spaces (continuity and differentiability of functions of several variables, compactness, linear mappings). In the second part, where various results are presented in the more general settings of Banach or Hilbert spaces, and where differential calculus in general normed vector spaces is often used, all relevant definitions and results are precisely stated wherever they are needed. Besides, the text is written in such a way that, in each case, the reader not familiar with
these more abstract situations can, without any difficulty, 'stay in finite-dimensional spaces' and thus ignore these generalisations (in this spirit, weak convergence is used for proving only one 'infinite-dimensional' result, whose elementary 'finite-dimensional' proof is also given). This textbook mentioning.
has some
features
which,
in my opinion,
are worth
The combination in a single volume of Numerical Linear Algebra and Optimisation, with a progressive transition, and many cross-references, between these two themes; A mathematical level slowly increasing with the chapter number; A considerable space devoted to reviews of pertinent background material; A description of various practical problems, originating in Physics, M echanics, or Economics, whose numerical solution requires methods from Numerical Linear Algebra or Optimisation; Complete proofs are given of each theorem; Many exercises or problems conclude each section. The first part (Chapters Algebra. It contains:
1 to 6) is essentially devoted to Numerical Linear
A review of all those results about matrices and vector or matrix norms that will be subsequently used (Chapter I); Basic notions about the conditioning of linear systems and eigenvalue problems (Chapter 2); A review of various approximate methods (finite-difference methods, finite element methods, polynomial and spline interpolations, least square approximations, approximation of 'small' vibrations) that eventually lead to the solution of a linear system or of a matrix eigenvalue problem (Chapter 3); A description and a mathematical analysis of some of the fundamental direct methods (Gauss, Cholesky, Householder; cf. Chapter 4) and iterative methods (Jacobi, Gauss-Seidel, relaxation; cf. Chapter 5) for solving linear systems; A description and a mathematical analysis of some of the fundamental methods (Jacobi, Givens-Householder, OR, inverse iteration) for computing the eigenvalues and eigenvectors of matrices (Chapter 6). The second part (Chapters 7 to 10) is essentially devoted to Optimisation. It contains: review of all relevant prerequisites about differential calculus in normed vector spaces (Chapter 7) and about Hilbert spaces (Chapter 8);
A thorough
Preface
X1l1
A progressive introduction to Optimisation, through analyses of Lagrange multipliers, of extrema and convexity of real functions, and of Newton's method (Chapter 7); A description of various linear and nonlinear problems whose approximate solution leads to minimisation problems in IR",with or without constraints (Chapters 8 and 10); A description and mathematical analysis of some of the fundamental algorithms of Optimisation theory - relaxation methods, gradient methods (with optimal, fixed, or variable, parameter), conjugate gradient methods, penalty-function methods (Chapter 8), Uzawa's method (Chapter 9), simplex method (Chapter 10). An introduction to duality theory - Farkas lemma, Kuhn-Tucker conditions, Lagrangians and saddle-points, duality in linear programming (Chapters 9 and 10). More complete descriptions introductions to each chapter.
of the topics treated
are found
in the
CAMBRIDGE UNIVERSITY
PRESS