Advanced Calculus with Applications in Statistics

by André I. Khuri

Overview

Designed to help motivate the learning of advanced calculus by demonstrating its relevance in the field of statistics, this successful text features detailed coverage of optimization techniques and their applications in statistics while introducing the reader to approximation theory. The Second Edition provides substantial new coverage of the material, including three new chapters and a large appendix that contains solutions to almost all of the exercises in the book. Applications of some of these methods in statistics are discusses. 

Product Details

ISBN-13:	9780471391043
Publisher:	Wiley
Publication date:	11/18/2002
Series:	Wiley Series in Probability and Statistics , #360
Edition description:	REV
Pages:	704
Product dimensions:	6.24(w) x 9.67(h) x 1.45(d)

About the Author

ANDRE I. KHURI, PhD, is a Professor in the Department of Statistics at the University of Florida, Gainesville.

Read an Excerpt

Advanced Calculus with Applications in Statistics

---

By André I. Khuri

John Wiley & Sons

ISBN: 0-471-39104-2

---

Chapter One

An Introduction to Set Theory

The origin of the modern theory of sets can be traced back to the Russian-born German mathematician Georg Cantor (1845-1918). This chapter introduces the basic elements of this theory.

1.1. THE CONCEPT OF A SET

A set is any collection of well-defined and distinguishable objects. These objects are called the elements, or members, of the set and are denoted by lowercase letters. Thus a set can be perceived as a collection of elements united into a single entity. Georg Cantor stressed this in the following words: "A set is a multitude conceived of by us as a one."

If x is an element of a set A, then this fact is denoted by writing x [element of] A. If, however, x is not an element of A, then we write x [??] A. Curly brackets are usually used to describe the contents of a set. For example, if a set A consists of the elements [x.sub.1], [x.sub.2],..., [x.sub.n], then it can be represented as A = {[x.sub.1], [x.sub.2],..., [x.sub.n]}. In the event membership in a set is determined by the the satisfaction of a certain property or a relationship, then the description of the same can be given within the curly brackets. For example, if A consists of all real numbers x such that [chi square] > 1, then it can be expressed as A = {x|[chi square] > 1}, where the bar | is used simply to mean "such that." The definition of sets in this manner is based on the axiom of abstraction, which states that given any property, there exists a set whose elements are just those entities having that property.

Definition 1.1.1. The set that contains no elements is called the empty set and is denoted by [empty set].

Definition 1.1.2. A set A is a subset of another set B, written symbolically as A [subset] B, if every element of A is an element of B. If B contains at least one element that is not in A, then A is said to be a proper subset of B.

Definition 1.1.3. A set A and a set B are equal if A [subset] B and B [subset] A. Thus, every element of A is an element of B and vice versa.

Definition 1.1.4. The set that contains all sets under consideration in a certain study is called the universal set and is denoted by [Omega].

1.2. SET OPERATIONS

There are two basic operations for sets that produce new sets from existing ones. They are the operations of union and intersection.

Definition 1.2.1. The union of two sets A and B, denoted by A [union] B, is the set of elements that belong to either A or B, that is,

A [union] B = {x|x [element of] A or x [element of] B}.

This definition can be extended to more than two sets. For example, if [A.sub.1], [A.sub.2],..., [A.sub.n] are n given sets, then their union, denoted by [[union].sup.n.sub.i=1][A.sub.i], is a set such that x is an element of it if and only if x belongs to at least one of the [A.sub.i] (i = 1, 2,..., n).

Definition 1.2.2. The intersection of two sets A and B, denoted by A [intersection] B, is the set of elements that belong to both A and B. Thus

A [intersection] B = {x|x [element of] A and x [element of] B}.

This definition can also be extended to more than two sets. As before, if [A.sub.1], [A.sub.2],..., [A.sub.n] are n given sets, then their intersection, denoted by [[intersection].sup.n.sub.i]=1 [A.sub.i], is the set consisting of all elements that belong to all the [A.sub.i] (i = 1, 2,..., n).

Definition 1.2.3. Two sets A and B are disjoint if their intersection is the empty set, that is, A [intersection] B = [empty set].

Definition 1.2.4. The complement of a set A, denoted by [bar.A], is the set consisting of all elements in the universal set that do not belong to A. In other words, x [element of] [bar.A] if and only if x [??] A.

The complement of A with respect to a set B is the set B - A which consists of the elements of B that do not belong to A. This complement is called the relative complement of A with respect to B.

From Definitions 1.1.1-1.1.4 and 1.2.1-1.2.4, the following results can be concluded:

Result 1.2.1. The empty set [empty set] is a subset of every set. To show this, suppose that A is any set. If it is false that [empty set] [subset] A, then there must be an element in [empty set] which is not in A. But this is not possible, since [empty set] is empty. It is therefore true that [empty set] [subset] A.

Result 1.2.2. The empty set [empty set] is unique. To prove this, suppose that [[empty set].sub.1] and [[empty set].sub.2] are two empty sets. Then, by the previous result, [[empty set].sub.1] [subset] [[empty set].sub.2] and [[empty set].sub.2] [greater than or equal to] [[empty set].sub.2]. Hence, [[empty set].sub.1] = [[empty set].sub.2].

Result 1.2.3. The complement of [empty set] is [Omega]. Vice versa, the complement of [Omega] is [empty set].

Result 1.2.4. The complement of [bar.A] is A.

Result 1.2.5. For any set A, A [union] [bar.A] = [Omega] and A [intersection] [bar.A] = [empty set].

Result 1.2.6. A - B = A - A [intersection] B.

Result 1.2.7. A [union] (B [union] ITLITL) = (A [union] B) [union] ITLITL.

Result 1.2.8. A [intersection] (B [intersection] ITLITL) = (A [intersection] B) [intersection] ITLITL).

Result 1.2.9. A [union] (B [intersection] ITLITL) = (A [union] B) [intersection] (A [union] ITLITL).

Result 1.2.10. A [intersection] (B [union] ITLITL) = (A [intersection] B) [union] (A [intersection] ITLITL).

Result 1.2.11. [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII].

Result 1.2.12. [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII].

Definition 1.2.5. Let A and B be two sets. Their Cartesian product, denoted by A x B, is the set of all ordered pairs (a, b) such that a [element of] A and b [element of] B, that is,

A x B = {(a, b)|a [element of] A and b [element of] B}.

The word "ordered" means that if a and c are elements in A and b and d are elements in B, then (a, b) = (c, d) if and only if a = c and b = d.

The preceding definition can be extended to more than two sets. For example, if [A.sub.1], [A.sub.2],..., [A.sub.n] are n given sets, then their Cartesian product is denoted by [x.sup.n.sub.i=1][A.sub.i] and defined by

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII.]

Here, ([a.sub.1], [a.sub.2],..., [a.sub.n]), called an ordered n-tuple, represents a generalization of the ordered pair. In particular, if the [A.sub.i] are equal to A for i = 1, 2,..., n, then one writes [A.sup.n] for [x.sup.n.sub.i=1]A.

The following results can be easily verified:

Result 1.2.13. A x B = [empty set] if and only if A = [empty set] or B = [empty set].

Result 1.2.14. (A [union] B]) x ITLITL = (A x ITLITL) [union] (B x ITLITL).

Result 1.2.15. (A [intersection] B) x ITLITL = (A x ITLITL) [intersection] x (B x ITLITL).

Result 1.2.16. (A x B) [intersection] (ITLITL x D) = (A [intersection] ITLITL) x (B [intersection] D).

1.3. RELATIONS AND FUNCTIONS

Let A x B be the Cartesian product of two sets, A and B.

Definition 1.3.1. A relations [rho] from A to B is a subset of A x B, that is, [rho] consists of ordered pairs (a, b) such that a [element of] A and b [element of] B. In particular, if A = B, then [rho] is said to be a relation in A.

For example, if A = {7, 8, 9} and B = {7, 8, 9, 10}, then [rho] {(a, b)|a < b, a [element of] A, b [element of] B} is a relation from A to B that consists of the six ordered pairs (7, 8), (7, 9), (7, 10), (8, 9), (8, 10), and (9, 10).

Whenever [rho] is a relation and (x, y) [element of] [rho], then x and y are said to be [rho]-related. This is denoted by writing x p y.

Definition 1.3.2. A relation [rho] in a set A is an equivalence relation if the following properties are satisfied:

1. [rho] is reflexive, that is, a [rho] a for any a in A.

2. [rho] is symmetric, that is, if a [rho] b, then b [rho] a for any a, b in A.

3. [rho] is transitive, that is, if a [rho] b and b [rho] c, then a [rho] c for any a, b, c in A.

If [rho] is an equivalence relation in a set A, then for a given [a.sub.0] in A, the set

ITLITL([a.sub.0]) = {a [element of] A|[a.sub.0] [rho] a},

which consists of all elements of A that are [rho]-related to [a.sub.0], is called an equivalence class of [a.sub.0].

Result 1.3.1. a [element of] ITLITL(a) for any a in A. Thus each element of A is an element of an equivalence class.

Result 1.3.2. If ITLITL([a.sub.1]) and ITLITL([a.sub.2]) are two equivalence classes, then either ITLITL([a.sub.1]) = ITLITL([a.sub.2]), or ITLITL([a.sub.1) and ITLITL([a.sub.2]) are disjoint subsets.

It follows from Results 1.3.1 and 1.3.2 that if A is a nonempty set, the collection of distinct [rho]-equivalence classes of A forms a partition of A.

As an example of an equivalence relation, consider that a [rho] b if and only if a and b are integers such that a - b is divisible by a nonzero integer n. This is the relation of congruence modulo n in the set of integers and is written symbolically as a [equivalent to] b (mod n). Clearly, a [equivalent to] a (mod n), since a - a = 0 is divisible by n. Also, if a [equivalent to] b (mod n), then b [equivalent to] a (mod n), since if a - b is divisible by n, then so is b - a. Furthermore, if a [equivalent to] _ b (mod n) and b [equivalent to] c (mod n), then a [equivalent to] c (mod n). This is true because if a - b and b - c are both divisible by n, then so is (a - b) + (b - c) = a - c. Now, if [a.sub.0] is a given integer, then a [rho]-equivalence class of [a.sub.0] consists of all integers that can be written as a = [a.sub.0] + kn, where k is an integer. This in this example ITLITL([a.sub.0]) is the set {[a.sub.0] + kn|k [element of] J}, where J denotes the set of all integers.

Definition 1.3.3. Let [rho] be a relation from A to B. Suppose that [rho] has the property that for all x in A, if x [rho] y and x [rho] z, where y and z are elements in B, then y = z. Such a relation is called a function.

Thus a function is a relation [rho] such that any two elements in B that are [rho]-related to the same x in A must be identical. In other words, to each element x in A, there corresponds only one element y in B. We call y the value of the function at x and denote it by writing y = f(x). The set A is called the domain of the function f, and the set of all values of f(x) for x in A is called the range of f, or the image of A under f, and is denoted by f(A). In this case, we say that f is a function, or a mapping, from A into B. We express this fact by writing f: A [right arrow] B. Note that f(A) is a subset of B. In particular, if B = f(A), then f is said to be a function from A onto B. In this case, every element b in B has a corresponding element a in A such that b = f(a).

Definition 1.3.4. A function f defined on a set A is said to be a one-to-one function if whenever f([x.sub.1]) = f([x.sub.2]) for [x.sub.1], [x.sub.2] in A, one has [x.sub.1] = [x.sub.2]. Equivalently, f is a one-to-one function if whenever [x.sub.1] [not equal to] [x.sub.2], one has f([x.sub.1])[not equal to]f([x.sub.2]).

Continues...

---

Excerpted from Advanced Calculus with Applications in Statistics by André I. Khuri Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

---

Table of Contents

Preface xv

Preface to the First Edition xvii

1. An Introduction to Set Theory 1

1.1. The Concept of a Set 1

1.2. Set Operations 2

1.3. Relations and Functions 4

1.4. Finite Countable and Uncountable Sets 6

1.5. Bounded Sets 9

1.6. Some Basic Topological Concepts 10

1.7. Examples in Probability and Statistics 13

Further Reading and Annotated Bibliography 15

Exercises 17

2. Basic Concepts in Linear Algebra 21

2.1. Vector Spaces and Subspaces 21

2.2. Linear Transformations 25

2.3. Matrices and Determinants 27

2.3.1. Basic Operations on Matrices 28

2.3.2. The Rank of a Matrix 33

2.3.3. The Inverse of a Matrix 34

2.3.4. Generalized Inverse of a Matrix 36

2.3.5. Eigenvalues and Eigenvectors of a Matrix 36

2.3.6. Some Special Matrices 38

2.3.7. The Diagonalization of a Matrix 38

2.3.8. Quadratic Forms 39

2.3.9. The Simultaneous Diagonalization of Matrices 40

2.3.10. Bounds on Eigenvalues 41

2.4. Applications of Matrices in Statistics 43

2.4.1. The Analysis of the Balanced Mixed Model 43

2.4.2. The Singular-Value Decomposition 45

2.4.3. Extrema of Quadratic Forms 48

2.4.4. The Parameterization of Orthogonal Matrices 49

Further Reading and Annotated Bibliography 50

Exercises 53

3. Limits and Continuity of Functions 57

3.1. Limits of a Function 57

3.2. Some Properties Associated with Limits of Functions 63

3.3. The o O Notation 65

3.4. Continuous Functions 66

3.4.1. Some Properties of Continuous Functions 71

3.4.2. Lipschitz Continuous Functions 75

3.5. Inverse Functions 76

3.6. Convex Functions 79

3.7. Continuous and Convex Functions in Statistics 82

Further Reading and Annotated Bibliography 87

Exercises 88

4. Differentiation 93

4.1. The Derivative of a Function 93

4.2. The Mean Value Theorem 99

4.3. Taylor’s Theorem 108

4.4. Maxima and Minima of a Function 112

4.4.1. A Sufficient Condition for a Local Optimum 114

4.5. Applications in Statistics 115

4.5.1. Functions of Random Variables 116

4.5.2. Approximating Response Functions 121

4.5.3. The Poisson Process 122

4.5.4. Minimizing the Sum of Absolute Deviations 124

Further Reading and Annotated Bibliography 125

Exercises 127

5. Infinite Sequences and Series 132

5.1. Infinite Sequences 132

5.1.1. The Cauchy Criterion 137

5.2. Infinite Series 140

5.2.1. Tests of Convergence for Series of Positive Terms 144

5.2.2. Series of Positive and Negative Terms 158

5.2.3. Rearrangement of Series 159

5.2.4. Multiplication of Series 162

5.3. Sequences and Series of Functions 165

5.3.1. Properties of Uniformly Convergent Sequences and Series 169

5.4. Power Series 174

5.5. Sequences and Series of Matrices 178

5.6. Applications in Statistics 182

5.6.1. Moments of a Discrete Distribution 182

5.6.2. Moment and Probability Generating Functions 186

5.6.3. Some Limit Theorems 191

5.6.3.1. The Weak Law of Large Numbers ŽKhinchine’s Theorem. 192

5.6.3.2. The Strong Law of Large Numbers ŽKolmogorov’s Theorem. 192

5.6.3.3. The Continuity Theorem for Probability Generating Functions 192

5.6.4. Power Series and Logarithmic Series Distributions 193

5.6.5. Poisson Approximation to Power Series Distributions 194

5.6.6. A Ridge Regression Application 195

Further Reading and Annotated Bibliography 197

Exercises 199

6. Integration 205

6.1. Some Basic Definitions 205

6.2. The Existence of the Riemann Integral 206

6.3. Some Classes of Functions That Are Riemann Integrable 210

6.3.1. Functions of Bounded Variation 212

6.4. Properties of the Riemann Integral 215

6.4.1. Change of Variables in Riemann Integration 219

6.5. Improper Riemann Integrals 220

6.5.1. Improper Riemann Integrals of the Second Kind 225

6.6. Convergence of a Sequence of Riemann Integrals 227

6.7. Some Fundamental Inequalities 229

6.7.1. The Cauchy_Schwarz Inequality 229

6.7.2. Hólder’s Inequality 230

6.7.3. Minkowski’s Inequality 232

6.7.4. Jensen’s Inequality 233

6.8. Riemann_Stieltjes Integral 234

6.9. Applications in Statistics 239

6.9.1. The Existence of the First Negative Moment of a Continuous Distribution 242

6.9.2. Transformation of Continuous Random Variables 246

6.9.3. The Riemann_Stieltjes Representation of the Expected Value 249

6.9.4. Chebyshev’s Inequality 251

Further Reading and Annotated Bibliography 252

Exercises 253

7. Multidimensional Calculus 261

7.1. Some Basic Definitions 261

7.2. Limits of a Multivariable Function 262

7.3. Continuity of a Multivariable Function 264

7.4. Derivatives of a Multivariable Function 267

7.4.1. The Total Derivative 270

7.4.2. Directional Derivatives 273

7.4.3. Differentiation of Composite Functions 276

7.5. Taylor’s Theorem for a Multivariable Function 277

7.6. Inverse and Implicit Function Theorems 280

7.7. Optima of a Multivariable Function 283

7.8. The Method of Lagrange Multipliers 288

7.9. The Riemann Integral of a Multivariable Function 293

7.9.1. The Riemann Integral on Cells 294

7.9.2. Iterated Riemann Integrals on Cells 295

7.9.3. Integration over General Sets 297

7.9.4. Change of Variables in n-Tuple Riemann Integrals 299

7.10. Differentiation under the Integral Sign 301

7.11. Applications in Statistics 304

7.11.1. Transformations of Random Vectors 305

7.11.2. Maximum Likelihood Estimation 308

7.11.3. Comparison of Two Unbiased Estimators 310

7.11.4. Best Linear Unbiased Estimation 311

7.11.5. Optimal Choice of Sample Sizes in Stratified Sampling 313

Further Reading and Annotated Bibliography 315

Exercises 316

8. Optimization in Statistics 327

8.1. The Gradient Methods 329

8.1.1. The Method of Steepest Descent 329

8.1.2. The Newton_Raphson Method 331

8.1.3. The Davidon_Fletcher_Powell Method 331

8.2. The Direct Search Methods 332

8.2.1. The Nelder_Mead Simplex Method 332

8.2.2. Price’s Controlled Random Search Procedure 336

8.2.3. The Generalized Simulated Annealing Method 338

8.3. Optimization Techniques in Response Surface Methodology 339

8.3.1. The Method of Steepest Ascent 340

8.3.2. The Method of Ridge Analysis 343

8.3.3. Modified Ridge Analysis 350

8.4. Response Surface Designs 355

8.4.1. First-Order Designs 356

8.4.2. Second-Order Designs 358

8.4.3. Variance and Bias Design Criteria 359

8.5. Alphabetic Optimality of Designs 362

8.6. Designs for Nonlinear Models 367

8.7. Multiresponse Optimization 370

8.8. Maximum Likelihood Estimation and the EM Algorithm 372

8.8.1. The EM Algorithm 375

8.9. Minimum Norm Quadratic Unbiased Estimation of Variance Components 378

8.10. Scheffè’s Confidence Intervals 382

8.10.1. The Relation of Scheffè’s Confidence Intervals to the F-Test 385

Further Reading and Annotated Bibliography 391

Exercises 395

9. Approximation of Functions 403

9.1. Weierstrass Approximation 403

9.2. Approximation by Polynomial Interpolation 410

9.2.1. The Accuracy of Lagrange Interpolation 413

9.2.2. A Combination of Interpolation and Approximation 417

9.3. Approximation by Spline Functions 418

9.3.1. Properties of Spline Functions 418

9.3.2. Error Bounds for Spline Approximation 421

9.4. Applications in Statistics 422

9.4.1. Approximate Linearization of Nonlinear Models by Lagrange Interpolation 422

9.4.2. Splines in Statistics 428

9.4.2.1. The Use of Cubic Splines in Regression 428

9.4.2.2. Designs for Fitting Spline Models 430

9.4.2.3. Other Applications of Splines in Statistics 431

Further Reading and Annotated Bibliography 432

Exercises 434

10. Orthogonal Polynomials 437

10.1. Introduction 437

10.2. Legendre Polynomials 440

10.2.1. Expansion of a Function Using Legendre Polynomials 442

10.3. Jacobi Polynomials 443

10.4. Chebyshev Polynomials 444

10.4.1. Chebyshev Polynomials of the First Kind 444

10.4.2. Chebyshev Polynomials of the Second Kind 445

10.5. Hermite Polynomials 447

10.6. Laguerre Polynomials 451

10.7. Least-Squares Approximation with Orthogonal Polynomials 453

10.8. Orthogonal Polynomials Defined on a Finite Set 455

10.9. Applications in Statistics 456

10.9.1. Applications of Hermite Polynomials 456

10.9.1.1. Approximation of Density Functions and Quantiles of Distributions 456

10.9.1.2. Approximation of a Normal Integral 460

10.9.1.3. Estimation of Unknown Densities 461

10.9.2. Applications of Jacobi and Laguerre Polynomials 462

10.9.3. Calculation of Hypergeometric Probabilities Using Discrete Chebyshev Polynomials 462

Further Reading and Annotated Bibliography 464

Exercises 466

11. Fourier Series 471

11.1. Introduction 471

11.2. Convergence of Fourier Series 475

11.3. Differentiation and Integration of Fourier Series 483

11.4. The Fourier Integral 488

11.5. Approximation of Functions by Trigonometric Polynomials 495

11.5.1. Parseval’s Theorem 496

11.6. The Fourier Transform 497

11.6.1. Fourier Transform of a Convolution 499

11.7. Applications in Statistics 500

11.7.1. Applications in Time Series 500

11.7.2. Representation of Probability Distributions 501

11.7.3. Regression Modeling 504

11.7.4. The Characteristic Function 505

11.7.4.1. Some Properties of Characteristic Functions 510

Further Reading and Annotated Bibliography 510

Exercises 512

12. Approximation of Integrals 517

12.1. The Trapezoidal Method 517

12.1.1. Accuracy of the Approximation 518

12.2. Simpson’s Method 521

12.3. Newton_Cotes Methods 523

12.4. Gaussian Quadrature 524

12.5. Approximation over an Infinite Interval 528

12.6. The Method of Laplace 531

12.7. Multiple Integrals 533

12.8. The Monte Carlo Method 535

12.8.1. Variation Reduction 537

12.8.2. Integrals in Higher Dimensions 540

12.9. Applications in Statistics 541

12.9.1. The Gauss_Hermite Quadrature 542

12.9.2. Minimum Mean Squared Error Quadrature 543

12.9.3. Moments of a Ratio of Quadratic Forms 546

12.9.4. Laplace’s Approximation in Bayesian Statistics 548

12.9.5. Other Methods of Approximating Integrals in Statistics 549

Further Reading and Annotated Bibliography 550

Exercises 552

Appendix. Solutions to Selected Exercises 557

General Bibliography 652

Index 665

Editorial Reviews

"This is an exceptional book, which I would recommend for anyone beginning a career in statistical research." (Journal of the American Statistical Association, September 2004)

From the Publisher

An application-oriented and highly rigorous introduction to the central themes of advanced calculus for statistics students, with enough theoretical explanation to be suitable for mathematics students. Topics include an introduction to set theory, linear algebra, limits and continuity, differentiation, infinite sequences and series, integration, multidimensional calculus, optimization in statistics, and approximation of functions. Exercises and an annotated bibliography are included. Annotation c. Book News, Inc., Portland, OR (booknews.com)

Booknews