Approximate Dynamic Programming: Solving the Curses of Dimensionality / Edition 1

Hardcover (Print)
Buy New
Buy New from
Used and New from Other Sellers
Used and New from Other Sellers
from $176.99
Usually ships in 1-2 business days
Other sellers (Hardcover)
  • All (4) from $176.99   
  • New (2) from $215.00   
  • Used (2) from $176.99   
Sort by
Page 1 of 1
Showing 1 – 3 of 4
Note: Marketplace items are not eligible for any coupons and promotions
Seller since 2014

Feedback rating:



New — never opened or used in original packaging.

Like New — packaging may have been opened. A "Like New" item is suitable to give as a gift.

Very Good — may have minor signs of wear on packaging but item works perfectly and has no damage.

Good — item is in good condition but packaging may have signs of shelf wear/aging or torn packaging. All specific defects should be noted in the Comments section associated with each item.

Acceptable — item is in working order but may show signs of wear such as scratches or torn packaging. All specific defects should be noted in the Comments section associated with each item.

Used — An item that has been opened and may show signs of wear. All specific defects should be noted in the Comments section associated with each item.

Refurbished — A used item that has been renewed or updated and verified to be in proper working condition. Not necessarily completed by the original manufacturer.

Buy with Confidence. Excellent Customer Support. We ship from multiple US locations. No CD, DVD or Access Code Included.

Ships from: Fort Mill, SC

Usually ships in 1-2 business days

  • Standard, 48 States
Seller since 2014

Feedback rating:


Condition: New
Brand new.

Ships from: acton, MA

Usually ships in 1-2 business days

  • Standard, 48 States
  • Standard (AK, HI)
Seller since 2010

Feedback rating:


Condition: New
Hardcover New 0470171553 New Condition ~~~ Right off the Shelf-BUY NOW & INCREASE IN KNOWLEDGE...

Ships from: Geneva, IL

Usually ships in 1-2 business days

  • Canadian
  • International
  • Standard, 48 States
  • Standard (AK, HI)
  • Express, 48 States
  • Express (AK, HI)
Page 1 of 1
Showing 1 – 3 of 4
Sort by


A complete and accessible introduction to the real-world applications of approximate dynamic programming

With the growing levels of sophistication in modern-day operations, it is vital for practitioners to understand how to approach, model, and solve complex industrial problems. Approximate Dynamic Programming is a result of the author's decades of experience working in large industrial settings to develop practical and high-quality solutions to problems that involve making decisions in the presence of uncertainty. This groundbreaking book uniquely integrates four distinct disciplines—Markov design processes, mathematical programming, simulation, and statistics—to demonstrate how to successfully model and solve a wide range of real-life problems using the techniques of approximate dynamic programming (ADP). The reader is introduced to the three curses of dimensionality that impact complex problems and is also shown how the post-decision state variable allows for the use of classical algorithmic strategies from operations research to treat complex stochastic optimization problems.

Designed as an introduction and assuming no prior training in dynamic programming of any form, Approximate Dynamic Programming contains dozens of algorithms that are intended to serve as a starting point in the design of practical solutions for real problems. The book provides detailed coverage of implementation challenges including: modeling complex sequential decision processes under uncertainty, identifying robust policies, designing and estimating value function approximations, choosing effective stepsize rules, and resolving convergence issues.

With a focus on modeling and algorithms in conjunction with the language of mainstream operations research, artificial intelligence, and control theory, Approximate Dynamic Programming:

  • Models complex, high-dimensional problems in a natural and practical way, which draws on years of industrial projects
  • Introduces and emphasizes the power of estimating a value function around the post-decision state, allowing solution algorithms to be broken down into three fundamental steps: classical simulation, classical optimization, and classical statistics
  • Presents a thorough discussion of recursive estimation, including fundamental theory and a number of issues that arise in the development of practical algorithms
  • Offers a variety of methods for approximating dynamic programs that have appeared in previous literature, but that have never been presented in the coherent format of a book

Motivated by examples from modern-day operations research, Approximate Dynamic Programming is an accessible introduction to dynamic modeling and is also a valuable guide for the development of high-quality solutions to problems that exist in operations research and engineering. The clear and precise presentation of the material makes this an appropriate text for advanced undergraduate and beginning graduate courses, while also serving as a reference for researchers and practitioners. A companion Web site is available for readers, which includes additional exercises, solutions to exercises, and data sets to reinforce the book's main concepts.

Read More Show Less

Product Details

  • ISBN-13: 9780470171554
  • Publisher: Wiley, John & Sons, Incorporated
  • Publication date: 10/5/2007
  • Series: Wiley Series in Probability and Statistics Series , #703
  • Edition description: Older Edition
  • Edition number: 1
  • Pages: 488
  • Product dimensions: 7.24 (w) x 10.20 (h) x 1.24 (d)

Meet the Author

WARREN B. POWELL, PhD, is Professor of Operations Research and Financial Engineering at Princeton University, where he is founder and Director of CASTLE Laboratory, a research unit that works with industrial partners to test new ideas found in operations research. The recipient of the 2004 INFORMS Fellow Award, Dr. Powell has authored more than 160 published articles on stochastic optimization, approximate dynamicprogramming, and dynamic resource management.
Read More Show Less

Table of Contents

Preface to the Second Edition xi

Preface to the First Edition xv

Acknowledgments xvii

1 The Challenges of Dynamic Programming 1

1.1 A Dynamic Programming Example: A Shortest Path Problem, 2

1.2 The Three Curses of Dimensionality, 3

1.3 Some Real Applications, 6

1.4 Problem Classes, 11

1.5 The Many Dialects of Dynamic Programming, 15

1.6 What Is New in This Book?, 17

1.7 Pedagogy, 19

1.8 Bibliographic Notes, 22

2 Some Illustrative Models 25

2.1 Deterministic Problems, 26

2.2 Stochastic Problems, 31

2.3 Information Acquisition Problems, 47

2.4 A Simple Modeling Framework for Dynamic Programs, 50

2.5 Bibliographic Notes, 54

Problems, 54

3 Introduction to Markov Decision Processes 57

3.1 The Optimality Equations, 58

3.2 Finite Horizon Problems, 65

3.3 Infinite Horizon Problems, 66

3.4 Value Iteration, 68

3.5 Policy Iteration, 74

3.6 Hybrid Value-Policy Iteration, 75

3.7 Average Reward Dynamic Programming, 76

3.8 The Linear Programming Method for Dynamic Programs, 77

3.9 Monotone Policies*, 78

3.10 Why Does It Work?**, 84

3.11 Bibliographic Notes, 103

Problems, 103

4 Introduction to Approximate Dynamic Programming 111

4.1 The Three Curses of Dimensionality (Revisited), 112

4.2 The Basic Idea, 114

4.3 Q-Learning and SARSA, 122

4.4 Real-Time Dynamic Programming, 126

4.5 Approximate Value Iteration, 127

4.6 The Post-Decision State Variable, 129

4.7 Low-Dimensional Representations of Value Functions, 144

4.8 So Just What Is Approximate Dynamic Programming?, 146

4.9 Experimental Issues, 149

4.10 But Does It Work?, 155

4.11 Bibliographic Notes, 156

Problems, 158

5 Modeling Dynamic Programs 167

5.1 Notational Style, 169

5.2 Modeling Time, 170

5.3 Modeling Resources, 174

5.4 The States of Our System, 178

5.5 Modeling Decisions, 187

5.6 The Exogenous Information Process, 189

5.7 The Transition Function, 198

5.8 The Objective Function, 206

5.9 A Measure-Theoretic View of Information**, 211

5.10 Bibliographic Notes, 213

Problems, 214

6 Policies 221

6.1 Myopic Policies, 224

6.2 Lookahead Policies, 224

6.3 Policy Function Approximations, 232

6.4 Value Function Approximations, 235

6.5 Hybrid Strategies, 239

6.6 Randomized Policies, 242

6.7 How to Choose a Policy?, 244

6.8 Bibliographic Notes, 247

Problems, 247

7 Policy Search 249

7.1 Background, 250

7.2 Gradient Search, 253

7.3 Direct Policy Search for Finite Alternatives, 256

7.4 The Knowledge Gradient Algorithm for Discrete Alternatives, 262

7.5 Simulation Optimization, 270

7.6 Why Does It Work?**, 274

7.7 Bibliographic Notes, 285

Problems, 286

8 Approximating Value Functions 289

8.1 Lookup Tables and Aggregation, 290

8.2 Parametric Models, 304

8.3 Regression Variations, 314

8.4 Nonparametric Models, 316

8.5 Approximations and the Curse of Dimensionality, 325

8.6 Why Does It Work?**, 328

8.7 Bibliographic Notes, 333

Problems, 334

9 Learning Value Function Approximations 337

9.1 Sampling the Value of a Policy, 337

9.2 Stochastic Approximation Methods, 347

9.3 Recursive Least Squares for Linear Models, 349

9.4 Temporal Difference Learning with a Linear Model, 356

9.5 Bellman’s Equation Using a Linear Model, 358

9.6 Analysis of TD(0), LSTD, and LSPE Using a Single State, 364

9.7 Gradient-Based Methods for Approximate Value Iteration*, 366

9.8 Least Squares Temporal Differencing with Kernel Regression*, 371

9.9 Value Function Approximations Based on Bayesian Learning*, 373

9.10 Why Does It Work*, 376

9.11 Bibliographic Notes, 379

Problems, 381

10 Optimizing While Learning 383

10.1 Overview of Algorithmic Strategies, 385

10.2 Approximate Value Iteration and Q-Learning Using Lookup Tables, 386

10.3 Statistical Bias in the Max Operator, 397

10.4 Approximate Value Iteration and Q-Learning Using Linear Models, 400

10.5 Approximate Policy Iteration, 402

10.6 The Actor–Critic Paradigm, 408

10.7 Policy Gradient Methods, 410

10.8 The Linear Programming Method Using Basis Functions, 411

10.9 Approximate Policy Iteration Using Kernel Regression*, 413

10.10 Finite Horizon Approximations for Steady-State Applications, 415

10.11 Bibliographic Notes, 416

Problems, 418

11 Adaptive Estimation and Stepsizes 419

11.1 Learning Algorithms and Stepsizes, 420

11.2 Deterministic Stepsize Recipes, 425

11.3 Stochastic Stepsizes, 433

11.4 Optimal Stepsizes for Nonstationary Time Series, 437

11.5 Optimal Stepsizes for Approximate Value Iteration, 447

11.6 Convergence, 449

11.7 Guidelines for Choosing Stepsize Formulas, 451

11.8 Bibliographic Notes, 452

Problems, 453

12 Exploration Versus Exploitation 457

12.1 A Learning Exercise: The Nomadic Trucker, 457

12.2 An Introduction to Learning, 460

12.3 Heuristic Learning Policies, 464

12.4 Gittins Indexes for Online Learning, 470

12.5 The Knowledge Gradient Policy, 477

12.6 Learning with a Physical State, 482

12.7 Bibliographic Notes, 492

Problems, 493

13 Value Function Approximations for Resource Allocation Problems 497

13.1 Value Functions versus Gradients, 498

13.2 Linear Approximations, 499

13.3 Piecewise-Linear Approximations, 501

13.4 Solving a Resource Allocation Problem Using Piecewise-Linear Functions, 505

13.5 The SHAPE Algorithm, 509

13.6 Regression Methods, 513

13.7 Cutting Planes*, 516

13.8 Why Does It Work?**, 528

13.9 Bibliographic Notes, 535

Problems, 536

14 Dynamic Resource Allocation Problems 541

14.1 An Asset Acquisition Problem, 541

14.2 The Blood Management Problem, 547

14.3 A Portfolio Optimization Problem, 557

14.4 A General Resource Allocation Problem, 560

14.5 A Fleet Management Problem, 573

14.6 A Driver Management Problem, 580

14.7 Bibliographic Notes, 585

Problems, 586

15 Implementation Challenges 593

15.1 Will ADP Work for Your Problem?, 593

15.2 Designing an ADP Algorithm for Complex Problems, 594

15.3 Debugging an ADP Algorithm, 596

15.4 Practical Issues, 597

15.5 Modeling Your Problem, 602

15.6 Online versus Offline Models, 604

15.7 If It Works, Patent It!, 606

Bibliography 607

Index 623

Read More Show Less

Customer Reviews

Be the first to write a review
( 0 )
Rating Distribution

5 Star


4 Star


3 Star


2 Star


1 Star


Your Rating:

Your Name: Create a Pen Name or

Barnes & Review Rules

Our reader reviews allow you to share your comments on titles you liked, or didn't, with others. By submitting an online review, you are representing to Barnes & that all information contained in your review is original and accurate in all respects, and that the submission of such content by you and the posting of such content by Barnes & does not and will not violate the rights of any third party. Please follow the rules below to help ensure that your review can be posted.

Reviews by Our Customers Under the Age of 13

We highly value and respect everyone's opinion concerning the titles we offer. However, we cannot allow persons under the age of 13 to have accounts at or to post customer reviews. Please see our Terms of Use for more details.

What to exclude from your review:

Please do not write about reviews, commentary, or information posted on the product page. If you see any errors in the information on the product page, please send us an email.

Reviews should not contain any of the following:

  • - HTML tags, profanity, obscenities, vulgarities, or comments that defame anyone
  • - Time-sensitive information such as tour dates, signings, lectures, etc.
  • - Single-word reviews. Other people will read your review to discover why you liked or didn't like the title. Be descriptive.
  • - Comments focusing on the author or that may ruin the ending for others
  • - Phone numbers, addresses, URLs
  • - Pricing and availability information or alternative ordering information
  • - Advertisements or commercial solicitation


  • - By submitting a review, you grant to Barnes & and its sublicensees the royalty-free, perpetual, irrevocable right and license to use the review in accordance with the Barnes & Terms of Use.
  • - Barnes & reserves the right not to post any review -- particularly those that do not follow the terms and conditions of these Rules. Barnes & also reserves the right to remove any review at any time without notice.
  • - See Terms of Use for other conditions and disclaimers.
Search for Products You'd Like to Recommend

Recommend other products that relate to your review. Just search for them below and share!

Create a Pen Name

Your Pen Name is your unique identity on It will appear on the reviews you write and other website activities. Your Pen Name cannot be edited, changed or deleted once submitted.

Your Pen Name can be any combination of alphanumeric characters (plus - and _), and must be at least two characters long.

Continue Anonymously
Sort by: Showing all of 2 Customer Reviews
  • Anonymous

    Posted February 9, 2009

    No text was provided for this review.

  • Anonymous

    Posted June 6, 2012

    No text was provided for this review.

Sort by: Showing all of 2 Customer Reviews

If you find inappropriate content, please report it to Barnes & Noble
Why is this product inappropriate?
Comments (optional)