Dynamic Probabilistic Systems, Volume II: Semi-Markov and Decision Processes

by Ronald A. Howard

Overview

This book is an integrated work published in two volumes. The first volume treats the basic Markov process and its variants; the second, semi-Markov and decision processes. Its intent is to equip readers to formulate, analyze, and evaluate simple and advanced Markov models of systems, ranging from genetics and space engineering to marketing. More than a collection of techniques, it constitutes a guide to the consistent application of the fundamental principles of probability and linear system theory.
Author Ronald A. Howard, Professor of Management Science and Engineering at Stanford University, continues his treatment from Volume I with surveys of the discrete- and continuous-time semi-Markov processes, continuous-time Markov processes, and the optimization procedure of dynamic programming. The final chapter reviews the preceding material, focusing on the decision processes with discussions of decision structure, value and policy iteration, and examples of infinite duration and transient processes. Volume II concludes with an appendix listing the properties of congruent matrix multiplication.

Product Details

ISBN-13:	9780486152004
Publisher:	Dover Publications
Publication date:	01/18/2013
Series:	Dover Books on Mathematics , #2
Sold by:	Barnes & Noble
Format:	eBook
Pages:	576
File size:	30 MB
Note:	This product may take a few minutes to download.

Table of Contents

The Discrete-Time Semi-Markov Process     577
The Formal Model     577
The Interval Transition Probabilities     584
Transform Analysis     589
An Alternate Formulation-Conditional Transition Probabilities     598
Flow Graph Analysis     603
Counting Transitions     604
Entrance and Destination Probabilities     611
Transient Processes     621
Duration     629
First Passage Times     634
State Occupancies     644
The General Discrete-Time Semi-Markov Process     657
Random Starting     663
Basic Markovian Equivalents     678
Conclusion     683
The Continuous-Time Semi-Markov Process     687
The Formal Model     687
The Car Rental Example     692
The Interval Transition Probabilities     694
The Exponential Transform     695
Transform Analysis of the Continuous-Time Semi-Markov Process     709
Alternate Formulations     714
Flow Graph Analysis     718
Counting Transitions     720
Entrance and Destination Probabilities     723
Transient Processes     728
First Passage Times     733
State Occupancies     736
The General Continuous-Time Semi-Markov Process     740
Random Starting     741
The Continuous-Time Renewal Process     745
Conclusion     763
Continuous-Time Markov Processes     769
Defining Relationships     769
Interval Transition Probabilities of the Continuous-Time Markov Process     774
A Continuous-Time Taxicab Problem and Other Examples     777
Flow Graph Analysis     789
Interpretation as Competing Exponential Processes     793
Continuous-Time Birth and Death Processes     797
Transient Processes and First Passage Times     814
The Infinite-State Continuous-Time Markov Process     818
Counting Transitions     827
Processes with Partial Information: Inference     829
The Continuous-Time Chapman-Kolmogorov Equations     841
Conclusion     843
Rewards     851
The Reward Structure for Continuous-Time Processes     851
The Continuous-Time Semi-Markov Reward Process with Discounting     854
The Continuous-Time Semi-Markov Reward Process without Discounting     861
The Car Rental Example Again      875
The Reward Structure for Discrete-Time Processes     880
The Discrete-Time Semi-Markov Reward Process with Discounting     882
The Discrete-Time Semi-Markov Reward Process without Discounting     884
The Discrete-Time Car Rental Example     886
The Effect of Lapsed Time on Expected Rewards     894
Rewards in Transient Processes     905
Concluding Remarks     910
Dynamic Programming     917
The Structure of Sequential Decision Processes     917
The Solution Concepts of Dynamic Programming     919
A Production Scheduling Example     921
An Action-Timing Problem     935
The Dynamic Programming Formalism     949
Multiplicative Rewards     955
Conclusion     959
Semi-Markov Decision Processes     965
The Decision Structure     965
Value Iteration     971
Policy Iteration     983
Policy Iteration with Discounting     1005
Policy Iteration in Transient Processes     1017
Examples of Infinite Duration Processes     1025
Examples of Transient Processes     1058
Conclusion     1094
Notation      1101
Appendix
Properties of Congruent Matrix Multiplication     1107
References
Index     I-1