Table of Contents
Preface 1 Introduction to Probability 1
1.1 Introduction: Why Study Probability? 1
1.2 The Different Kinds of Probability 2
Probability as Intuition 2
Probability as the Ratio of Favorable to Total Outcomes (Classical Theory) 3
Probability as a Measure of Frequency of Occurrence 4
Probability Based on an Axiomatic Theory 5
1.3 Misuses, Miscalculations, and Paradoxes in Probability 7
1.4 Sets, Fields, and Events 8
Examples of Sample Spaces 8
1.5 Axiomatic Definition of Probability 15
1.6 Joint, Conditional, and Total Probabilities; Independence 20
Compound Experiments 23
1.7 Bayes’ Theorem and Applications 35
1.8 Combinatorics 38
Occupancy Problems 42
Extensions and Applications 46
1.9 Bernoulli Trials–Binomial and Multinomial Probability Laws 48
Multinomial Probability Law 54
1.10 Asymptotic Behavior of the Binomial Law: The Poisson Law 57
1.11 Normal Approximation to the Binomial Law 63
Summary 65
Problems 66
References 77
2 Random Variables 79
2.1 Introduction 79
2.2 Definition of a Random Variable 80
2.3 Cumulative Distribution Function 83
Properties of F X(x) 84
Computation of F X(x) 85
2.4 Probability Density Function (pdf) 88
Four Other Common Density Functions 95
More Advanced Density Functions 97
2.5 Continuous, Discrete, and Mixed Random Variables 100
Some Common Discrete Random Variables 102
2.6 Conditional and Joint Distributions and Densities 107
Properties of Joint CDF F XY (x, y) 118
2.7 Failure Rates 137
Summary 141
Problems 141
References 149
Additional Reading 149
3 Functions of Random Variables 151
3.1 Introduction 151
Functions of a Random Variable (FRV): Several Views 154
3.2 Solving Problems of the Type Y = g(X) 155
General Formula of Determining the pdf of Y = g(X) 166
3.3 Solving Problems of the Type Z = g(X, Y ) 171
3.4 Solving Problems of the Type V = g(X, Y ), W = h(X, Y ) 193
Fundamental Problem 193
Obtaining f VW Directly from f XY 196
3.5 Additional Examples 200
Summary 205
Problems 206
References 214
Additional Reading 214
4 Expectation and Moments 215
4.1 Expected Value of a Random Variable 215
On the Validity of Equation 4.1-8 218
4.2 Conditional Expectations 232
Conditional Expectation as a Random Variable 239
4.3 Moments of Random Variables 242
Joint Moments 246
Properties of Uncorrelated Random Variables 248
Jointly Gaussian Random Variables 251
4.4 Chebyshev and Schwarz Inequalities 255
Markov Inequality 257
The Schwarz Inequality 258
4.5 Moment-Generating Functions 261
4.6 Chernoff Bound 264
4.7 Characteristic Functions 266
Joint Characteristic Functions 273
The Central Limit Theorem 276
4.8 Additional Examples 281
Summary 283
Problems 284
References 293
Additional Reading 294
5 Random Vectors 295
5.1 Joint Distribution and Densities 295
5.2 Multiple Transformation of Random Variables 299
5.3 Ordered Random Variables 302
Distribution of area random variables 305
5.4 Expectation Vectors and Covariance Matrices 311
5.5 Properties of Covariance Matrices 314
Whitening Transformation 318
5.6 The Multidimensional Gaussian (Normal) Law 319
5.7 Characteristic Functions of Random Vectors 328
Properties of CF of Random Vectors 330
The Characteristic Function of the Gaussian (Normal) Law 331
Summary 332
Problems 333
References 339
Additional Reading 339
6 Statistics: Part 1 Parameter Estimation 340
6.1 Introduction 340
Independent, Identically Distributed (i.i.d.) Observations 341
Estimation of Probabilities 343
6.2 Estimators 346
6.3 Estimation of the Mean 348
Properties of the Mean-Estimator Function (MEF) 349
Procedure for Getting a δ-confidence Interval on the Mean of a Normal
Random Variable When σ X Is Known 352
Confidence Interval for the Mean of a Normal Distribution When σX Is Not
Known 352
Procedure for Getting a δ-Confidence Interval Based on n Observations on
the Mean of a Normal Random Variable when σ X Is Not Known 355
Interpretation of the Confidence Interval 355
6.4 Estimation of the Variance and Covariance 355
Confidence Interval for the Variance of a Normal Random
variable 357
Estimating the Standard Deviation Directly 359
Estimating the covariance 360
6.5 Simultaneous Estimation of Mean and Variance 361
6.6 Estimation of Non-Gaussian Parameters from Large Samples 363
6.7 Maximum Likelihood Estimators 365
6.8 Ordering, more on Percentiles, Parametric Versus Nonparametric Statistics 369
The Median of a Population Versus Its Mean 371
Parametric versus Nonparametric Statistics 372
Confidence Interval on the Percentile 373
Confidence Interval for the Median When n Is Large 375
6.9 Estimation of Vector Means and Covariance Matrices 376
Estimation of μ 377
Estimation of the covariance K 378
6.10 Linear Estimation of Vector Parameters 380
Summary 384
Problems 384
References 388
Additional Reading 389
7 Statistics: Part 2 Hypothesis Testing 390
7.1 Bayesian Decision Theory 391
7.2 Likelihood Ratio Test 396
7.3 Composite Hypotheses 402
Generalized Likelihood Ratio Test (GLRT) 403
How Do We Test for the Equality of Means of Two Populations? 408
Testing for the Equality of Variances for Normal Populations:
The F-test 412
Testing Whether the Variance of a Normal Population Has a
Predetermined Value: 416
7.4 Goodness of Fit 417
7.5 Ordering, Percentiles, and Rank 423
How Ordering is Useful in Estimating Percentiles and the Median 425
Confidence Interval for the Median When n Is Large 428
Distribution-free Hypothesis Testing: Testing If Two Population are the
Same Using Runs 429
Ranking Test for Sameness of Two Populations 432
Summary 433
Problems 433
References 439
8 Random Sequences 441
8.1 Basic Concepts 442
Infinite-length Bernoulli Trials 447
Continuity of Probability Measure 452
Statistical Specification of a Random Sequence 454
8.2 Basic Principles of Discrete-Time Linear Systems 471
8.3 Random Sequences and Linear Systems 477
8.4 WSS Random Sequences 486
Power Spectral Density 489
Interpretation of the psd 490
Synthesis of Random Sequences and Discrete-Time Simulation 493
Decimation 496
Interpolation 497
8.5 Markov Random Sequences 500
ARMA Models 503
Markov Chains 504
8.6 Vector Random Sequences and State Equations 511
8.7 Convergence of Random Sequences 513
8.8 Laws of Large Numbers 521
Summary 526
Problems 526
References 541
9 Random Processes 543
9.1 Basic Definitions 544
9.2 Some Important Random Processes 548
Asynchronous Binary Signaling 548
Poisson Counting Process 550
Alternative Derivation of Poisson Process 555
Random Telegraph Signal 557
Digital Modulation Using Phase-Shift Keying 558
Wiener Process or Brownian Motion 560
Markov Random Processes 563
Birth—Death Markov Chains 567
Chapman—Kolmogorov Equations 571
Random Process Generated from Random Sequences 572
9.3 Continuous-Time Linear Systems with Random Inputs 572
White Noise 577
9.4 Some Useful Classifications of Random Processes 578
Stationarity 579
9.5 Wide-Sense Stationary Processes and LSI Systems 581
Wide-Sense Stationary Case 582
Power Spectral Density 584
An Interpretation of the psd 586
More on White Noise 590
Stationary Processes and Differential Equations 596
9.6 Periodic and Cyclostationary Processes 600
9.7 Vector Processes and State Equations 606
State Equations 608
Summary 611
Problems 611
References 633
Chapters 10 and 11 are available as Web chapters on the companion
Web site at http://www.pearsonhighered.com/stark.
10 Advanced Topics in Random Processes 635
10.1 Mean-Square (m.s.) Calculus 635
Stochastic Continuity and Derivatives [10-1] 635
Further Results on m.s. Convergence [10-1] 645
10.2 Mean-Square Stochastic Integrals 650
10.3 Mean-Square Stochastic Differential Equations 653
10.4 Ergodicity [10-3] 658
10.5 Karhunen—Lo‘eve Expansion [10-5] 665
10.6 Representation of Bandlimited and Periodic Processes 671
Bandlimited Processes 671
Bandpass Random Processes 674
WSS Periodic Processes 677
Fourier Series for WSS Processes 680
Summary 682
Appendix: Integral Equations 682
Existence Theorem 683
Problems 686
References 699
11 Applications to Statistical Signal Processing 700
11.1 Estimation of Random Variables and Vectors 700
More on the Conditional Mean 706
Orthogonality and Linear Estimation 708
Some Properties of the Operator ˆE 716
11.2 Innovation Sequences and Kalman Filtering 718
Predicting Gaussian Random Sequences 722
Kalman Predictor and Filter 724
Error-Covariance Equations 729
11.3 Wiener Filters for Random Sequences 733
Unrealizable Case (Smoothing) 734
Causal Wiener Filter 736
11.4 Expectation-Maximization Algorithm 738
Log-likelihood for the Linear Transformation 740
Summary of the E-M algorithm 742
E-M Algorithm for Exponential Probability
Functions 743
Application to Emission Tomography 744
Log-likelihood Function of Complete Data 746
E-step 747
M-step 748
11.5 Hidden Markov Models (HMM) 749
Specification of an HMM 751
Application to Speech Processing 753
Efficient Computation of P[E | M] with a Recursive
Algorithm 754
Viterbi Algorithm and the Most Likely State Sequence
for the Observations 756
11.6 Spectral Estimation 759
The Periodogram 760
Bartlett’s Procedure—-Averaging Periodograms 762
Parametric Spectral Estimate 767
Maximum Entropy Spectral Density 769
11.7 Simulated Annealing 772
Gibbs Sampler 773
Noncausal Gauss—Markov Models 774
Compound Markov Models 778
Gibbs Line Sequence 779
Summary 783
Problems 783
References 788
Appendix A Review of Relevant Mathematics A-1
A.1 Basic Mathematics A-1
Sequences A-1
Convergence A-2
Summations A-3
Z-Transform A-3
A.2 Continuous Mathematics A-4
Definite and Indefinite Integrals A-5
Differentiation of Integrals A-6
Integration by Parts A-7
Completing the Square A-7
Double Integration A-8
Functions A-8
A.3 Residue Method for Inverse Fourier Transformation A-10
Fact A-11
Inverse Fourier Transform for psd of Random Sequence A-13
A.4 Mathematical Induction A-17
References A-17
Appendix B Gamma and Delta Functions B-1
B.1 Gamma Function B-1
B.2 Incomplete Gamma Function B-2
B.3 Dirac Delta Function B-2
References B-5
Appendix C Functional Transformations and Jacobians C-1
C.1 Introduction C-1
C.2 Jacobians for n = 2 C-2
C.3 Jacobian for General n C-4
Appendix D Measure and Probability D-1
D.1 Introduction and Basic Ideas D-1
Measurable Mappings and Functions D-3
D.2 Application of Measure Theory to Probability D-3
Distribution Measure D-4
Appendix E Sampled Analog Waveforms and Discrete-time Signals E-1
Appendix F Independence of Sample Mean and Variance for Normal
Random Variables F-1
Appendix G Tables of Cumulative Distribution Functions: the Normal,
Student t, Chi-square, and F G-1
Index I-1
