The Variational Bayes Method in Signal Processing / Edition 1

The Variational Bayes Method in Signal Processing / Edition 1

by Vaclav #midl, Anthony Quinn

View All Available Formats & Editions

ISBN-10: 3540288198

ISBN-13: 9783540288190

Pub. Date: 12/16/2005

Publisher: Springer Berlin Heidelberg

This is the first book-length treatment of the Variational Bayes (VB) approximation in signal processing. It has been written as a self-contained, self-learning guide for academic and industrial research groups in signal processing, data analysis, machine learning, identification and control. It reviews the VB distributional approximation, showing that tractable


This is the first book-length treatment of the Variational Bayes (VB) approximation in signal processing. It has been written as a self-contained, self-learning guide for academic and industrial research groups in signal processing, data analysis, machine learning, identification and control. It reviews the VB distributional approximation, showing that tractable algorithms for parametric model identification can be generated in off-line and on-line contexts. Many of the principles are first illustrated via easy-to-follow scalar decomposition problems. In later chapters, successful applications are found in factor analysis for medical image sequences, mixture model identification and speech reconstruction. Results with simulated and real data are presented in detail. The unique development of an eight-step "VB method", which can be followed in all cases, enables the reader to develop a VB inference algorithm from the ground up, for their own particular signal or image model.

Product Details

Springer Berlin Heidelberg
Publication date:
Signals and Communication Technology Series
Edition description:
Product dimensions:
9.21(w) x 6.14(h) x 0.63(d)

Table of Contents

Introduction     1
How to be a Bayesian     1
The Variational Bayes (VB) Method     2
A First Example of the VB Method: Scalar Additive Decomposition     3
A First Choice of Prior     3
The Prior Choice Revisited     4
The VB Method in its Context     6
VB as a Distributional Approximation     8
Layout of the Work     10
Acknowledgement     11
Bayesian Theory     13
Bayesian Benefits     13
Off-line vs. On-line Parametric Inference     14
Bayesian Parametric Inference: the Off-Line Case     15
The Subjective Philosophy     16
Posterior Inferences and Decisions     16
Prior Elicitation     18
Conjugate priors     19
Bayesian Parametric Inference: the On-line Case     19
Time-invariant Parameterization     20
Time-variant Parameterization     20
Prediction     22
Summary     22
Off-line Distributional Approximations and the Variational Bayes Method     25
Distributional Approximation     25
How to Choose a Distributional Approximation     26
Distributional Approximation as anOptimization Problem     26
The Bayesian Approach to Distributional Approximation     27
The Variational Bayes (VB) Method of Distributional Approximation     28
The VB Theorem     28
The VB Method of Approximation as an Operator     32
The VB Method     33
The VB Method for Scalar Additive Decomposition     37
VB-related Distributional Approximations     39
Optimization with Minimum-Risk KL Divergence     39
Fixed-form (FF) Approximation     40
Restricted VB (RVB) Approximation     40
Adaptation of the VB method for the RVB Approximation     41
The Quasi-Bayes (QB) Approximation     42
The Expectation-Maximization (EM) Algorithm     44
Other Deterministic Distributional Approximations     45
The Certainty Equivalence Approximation     45
The Laplace Approximation     45
The Maximum Entropy (MaxEnt) Approximation     45
Stochastic Distributional Approximations     46
Distributional Estimation     47
Example: Scalar Multiplicative Decomposition     48
Classical Modelling     48
The Bayesian Formulation     48
Full Bayesian Solution      49
The Variational Bayes (VB) Approximation     51
Comparison with Other Techniques     54
Conclusion     56
Principal Component Analysis and Matrix Decompositions     57
Probabilistic Principal Component Analysis (PPCA)     58
Maximum Likelihood (ML) Estimation for the PPCA Model     59
Marginal Likelihood Inference of A     61
Exact Bayesian Analysis     61
The Laplace Approximation     62
The Variational Bayes (VB) Method for the PPCA Model     62
Orthogonal Variational PCA (OVPCA)     69
The Orthogonal PPCA Model     70
The VB Method for the Orthogonal PPCA Model     70
Inference of Rank     77
Moments of the Model Parameters     78
Simulation Studies     79
Convergence to Orthogonal Solutions: VPCA vs. FVPCA     79
Local Minima in FVPCA and OVPCA     82
Comparison of Methods for Inference of Rank     83
Application: Inference of Rank in a Medical Image Sequence     85
Conclusion     87
Functional Analysis of Medical Image Sequences     89
A Physical Model for Medical Image Sequences     90
Classical Inference of the Physiological Model      92
The FAMIS Observation Model     92
Bayesian Inference of FAMIS and Related Models     94
The VB Method for the FAMIS Model     94
The VB Method for FAMIS: Alternative Priors     99
Analysis of Clinical Data Using the FAMIS Model     102
Conclusion     107
On-line Inference of Time-Invariant Parameters     109
Recursive Inference     110
Bayesian Recursive Inference     110
The Dynamic Exponential Family (DEF)     112
Example: The AutoRegressive (AR) Model     114
Recursive Inference of non-DEF models     117
The VB Approximation in On-Line Scenarios     118
Scenario I: VB-Marginalization for Conjugate Updates     118
Scenario II: The VB Method in One-Step Approximation     121
Scenario III: Achieving Conjugacy in non-DEF Models via the VB Approximation     123
The VB Method in the On-Line Scenarios     126
Related Distributional Approximations     127
The Quasi-Bayes (QB) Approximation in On-Line Scenarios     128
Global Approximation via the Geometric Approach     128
One-step Fixed-Form (FF) Approximation     129
On-line Inference of a Mixture of AutoRegressive (AR) Models      130
The VB Method for AR Mixtures     130
Related Distributional Approximations for AR Mixtures     133
The Quasi-Bayes (QB) Approximation     133
One-step Fixed-Form (FF) Approximation     135
Simulation Study: On-line Inference of a Static Mixture     135
Inference of a Many-Component Mixture     136
Inference of a Two-Component Mixture     136
Data-Intensive Applications of Dynamic Mixtures     139
Urban Vehicular Traffic Prediction     141
Conclusion     143
On-line Inference of Time-Variant Parameters     145
Exact Bayesian Filtering     145
The VB-Approximation in Bayesian Filtering     147
The VB method for Bayesian Filtering     149
Other Approximation Techniques for Bayesian Filtering     150
Restricted VB (RVB) Approximation     150
Particle Filtering     152
Stabilized Forgetting     153
The Choice of the Forgetting Factor     154
The VB-Approximation in Kalman Filtering     155
The VB method     156
Loss of Moment Information in the VB Approximation     158
VB-Filtering for the Hidden Markov Model (HMM)     158
Exact Bayesian filtering for known T      159
The VB Method for the HMM Model with Known T     160
The VB Method for the HMM Model with Unknown T     162
Other Approximate Inference Techniques     164
Particle Filtering     164
Certainty Equivalence Approach     165
Simulation Study: Inference of Soft Bits     166
The VB-Approximation for an Unknown Forgetting Factor     168
Inference of a Univariate AR Model with Time-Variant Parameters     169
Simulation Study: Non-stationary AR Model Inference via Unknown Forgetting     173
Inference of an AR Process with Switching Parameters     173
Initialization of Inference for a Stationary AR Process     174
Conclusion     176
The Mixture-based Extension of the AR Model (MEAR)     179
The Extended AR (EAR) Model     179
Bayesian Inference of the EAR Model     181
Computational Issues     182
The EAR Model with Unknown Transformation: the MEAR Model     182
The VB Method for the MEAR Model     183
Related Distributional Approximations for MEAR     186
The Quasi-Bayes (QB) Approximation     186
The Viterbi-Like (VL) Approximation     187
Computational Issues     188
The MEAR Model with Time-Variant Parameters     191
Application: Inference of an AR Model Robust to Outliers     192
Design of the Filter-bank     192
Simulation Study     193
Application: Inference of an AR Model Robust to Burst Noise     196
Design of the Filter-Bank     196
Simulation Study     197
Application in Speech Reconstruction     201
Conclusion     201
Concluding Remarks     205
The VB Method     205
Contributions of the Work     206
Current Issues     206
Future Prospects for the VB Method     207
Required Probability Distributions     209
Multivariate Normal distribution     209
Matrix Normal distribution     209
Normal-inverse-Wishart (NiW [subscript A, Omega]) Distribution     210
Truncated Normal Distribution     211
Gamma Distribution     212
Von Mises-Fisher Matrix distribution     212
Definition     213
First Moment     213
Second Moment and Uncertainty Bounds     214
Multinomial Distribution     215
Dirichlet Distribution     215
Truncated Exponential Distribution     216
References     217
Index     225

Customer Reviews

Average Review:

Write a Review

and post it to your social network


Most Helpful Customer Reviews

See all customer reviews >