Uncertain Judgements: Eliciting Experts' Probabilities / Edition 1

Hardcover (Print)
Used and New from Other Sellers
Used and New from Other Sellers
from $53.14
Usually ships in 1-2 business days
(Save 37%)
Other sellers (Hardcover)
  • All (14) from $53.14   
  • New (8) from $62.98   
  • Used (6) from $53.14   

Overview

Elicitation is the process of extracting expert knowledge about some unknown quantity or quantities, and formulating that information as a probability distribution. Elicitation is important in situations, such as modelling the safety of nuclear installations or assessing the risk of terrorist attacks, where expert knowledge is essentially the only source of good information. It also plays a major role in other contexts by augmenting scarce observational data, through the use of Bayesian statistical methods. However, elicitation is not a simple task, and practitioners need to be aware of a wide range of research findings in order to elicit expert judgements accurately and reliably. Uncertain Judgements introduces the area, before guiding the reader through the study of appropriate elicitation methods, illustrated by a variety of multi-disciplinary examples.

This is achieved by:

  • Presenting a methodological framework for the elicitation of expert knowledge incorporating findings from both statistical and psychological research.
  • Detailing techniques for the elicitation of a wide range of standard distributions, appropriate to the most common types of quantities.
  • Providing a comprehensive review of the available literature and pointing to the best practice methods and future research needs.
  • Using examples from many disciplines, including statistics, psychology, engineering and health sciences.
  • Including an extensive glossary of statistical and psychological terms.

An ideal source and guide for statisticians and psychologists with interests in expert judgement or practical applications of Bayesian analysis, Uncertain Judgements will also benefit decision-makers, risk analysts, engineers and researchers in the medical and social sciences.

Read More Show Less

Editorial Reviews

From the Publisher
“This book, written by a group of expert statisticians and psychologists, provides an introduction to the subject and a detailed overview of the existing literature. The book guides the reader through the design of an elicitation method and details examples from a cross section of literature in the statistics, psychology, engineering and health sciences disciplines.”  (Zentralblatt Math, 1 August 2013)

"This is an interesting, well-written book that will be valuable to any decision maker who much rely on expert judgments, any statistician who uses Bayesian statistics, and any researcher who wishes to understand the field of elicitation." (Journal of the American Statistical Association, March 2009)

"This book provides an excellent introduction and working reference to the subject of its title and should be an invaluable aid to producers and consumers of expert opinion." (Biometrics, September 2008)

"I recommend 'Uncertain Judgements' as an excellent source for a wide variety of research." (Psychometrika, March 2008)

“…will be of interest to those who are concerned with or interested primarily in the practicalities of modeling expert judgement and opinion.” (International Journal of Marketing, January 2007)

Read More Show Less

Product Details

  • ISBN-13: 9780470029992
  • Publisher: Wiley
  • Publication date: 9/14/2006
  • Series: Statistics in Practice Series, #35
  • Edition number: 1
  • Pages: 338
  • Product dimensions: 6.28 (w) x 9.17 (h) x 0.96 (d)

Meet the Author

Professor Anthony O’Hagan is the Director of The Centre for Bayesian Statistics in Health Economics at the University of Sheffield. The Centre is a collaboration between the Department of Probability and Statistics and the School of Health and Related Research (ScHARR). The Department of Probability and Statistics is internationally respected for its research in Bayesian statistics, while ScHARR is one of the leading UK centres for economic evaluation.

Prof O’Hagan is an internationally leading expert in Bayesian Statistics.

Co-authors:

Professor Paul Gathwaite – Open University, Prof of Statistics, Maths and Computing

Dr Jeremy Oakley – Sheffield University

Professor John Brazier – Director of Health Economics Group, University of Sheffield

Dr Tim Rakow – University of Essex, Psychology Department

Dr Alireza Daneshkhah – University of Sheffield, Medical Statistics Department

Dr Jim Chilcott - School of Health Research, University of Sheffield, Department of OR

Read More Show Less

Table of Contents

Preface xi

1 Fundamentals of Probability and Judgement 1

1.1 Introduction 1

1.2 Probability and elicitation 1

1.2.1 Probability 1

1.2.2 Random variables and probability distributions 3

1.2.3 Summaries of distributions 5

1.2.4 Joint distributions 7

1.2.5 Bayes’ Theorem 8

1.2.6 Elicitation 9

1.3 Uncertainty and the interpretation of probability 10

1.3.1 Aleatory and epistemic uncertainty 10

1.3.2 Frequency and personal probabilities 11

1.3.3 An extended example 12

1.3.4 Implications for elicitation 14

1.4 Elicitation and the psychology of judgement 14

1.4.1 Judgement – absolute or relative? 15

1.4.2 Beyond perception 18

1.4.3 Implications for elicitation 20

1.5 Of what use are such judgements? 20

1.5.1 Normative theories of probability 21

1.5.2 Coherence 21

1.5.3 Do elicited probabilities have the desired interpretation? 22

1.6 Conclusions 24

1.6.1 Elicitation practice 24

1.6.2 Research questions 24

2 The Elicitation Context 25

2.1 How and who? 25

2.1.1 Choice of format 25

2.1.2 What is an expert? 26

2.2 The elicitation process 27

2.2.1 Roles within the elicitation process 28

2.2.2 A model for the elicitation process 28

2.3 Conventions in Chapters 3 to 9 31

2.4 Conclusions 31

2.4.1 Elicitation practice 31

2.4.2 Research question 31

3 The Psychology of Judgement Under Uncertainty 33

3.1 Introduction 33

3.1.1 Why psychology? 33

3.1.2 Chapter overview 34

3.2 Understanding the task and the expert 35

3.2.1 Cognitive capabilities: the proper view of human information processing? 35

3.2.2 Constructive processes: the proper view of the process? 36

3.3 Understanding research on human judgement 37

3.3.1 Experts versus the rest: the proper focus of research? 37

3.3.2 Early research on subjective probability: ‘conservatism’ in Bayesian probability revision 38

3.4 The heuristics and biases research programme 38

3.4.1 Availability 39

3.4.2 Representativeness 41

3.4.3 Do frequency representations remove the biases attributed to availability and representativeness? 46

3.4.4 Anchoring-and-adjusting 47

3.4.5 Support theory 49

3.4.6 The affect heuristic 51

3.4.7 Critique of the heuristics and biases approach 52

3.5 Experts and expertise 52

3.5.1 The heuristics and biases approach 53

3.5.2 The cognitive science approach 53

3.5.3 ‘The middle way’ 54

3.6 Three meta-theories of judgement 55

3.6.1 The cognitive continuum 56

3.6.2 The inside versus the outside view 56

3.6.3 The naive intuitive statistician metaphor 58

3.7 Conclusions 58

3.7.1 Elicitation practice 58

3.7.2 Research questions 59

4 The Elicitation of Probabilities 61

4.1 Introduction 61

4.2 The calibration of subjective probabilities 62

4.2.1 Research methods in calibration research 67

4.2.2 Calibration research: general findings 68

4.2.3 Calibration research in applied settings 72

4.2.4 A case study in probability judgement: calibration research in medicine 74

4.3 The calibration of subjective probabilities: theories and explanations 77

4.3.1 Explanations of probability judgement in calibration tasks 77

4.3.2 Theories of the calibration of subjective probabilities 79

4.4 Representations and methods 82

4.4.1 Different modes for representing uncertainty 83

4.4.2 Different formats for eliciting responses 87

4.4.3 Key lessons 89

4.5 Debiasing 89

4.5.1 General principles for debiasing judgement 90

4.5.2 Managing noise 91

4.5.3 Redressing insufficient regressiveness in prediction 92

4.5.4 A caveat concerning post hoc corrections 94

4.6 Conclusions 95

4.6.1 Elicitation practice 95

4.6.2 Research questions 95

5 Eliciting Distributions – General 97

5.1 From probabilities to distributions 97

5.1.1 From a few to infinity 98

5.1.2 Summaries 99

5.1.3 Fitting 100

5.1.4 Overview 100

5.2 Eliciting univariate distributions 100

5.2.1 Summaries based on probabilities 100

5.2.2 Proportions 104

5.2.3 Other summaries 105

5.3 Eliciting multivariate distributions 107

5.3.1 Structuring 107

5.3.2 Eliciting association 108

5.3.3 Joint and conditional probabilities 111

5.3.4 Regression 112

5.3.5 Many variables 113

5.4 Uncertainty and imprecision 114

5.4.1 Quantifying elicitation error 114

5.4.2 Sensitivity analysis 115

5.4.3 Feedback and overfitting 116

5.5 Conclusions 118

5.5.1 Elicitation practice 118

5.5.2 Research questions 119

6 Eliciting and Fitting a Parametric Distribution 121

6.1 Introduction 121

6.2 Outline of this Chapter 122

6.3 Eliciting opinion about a proportion 124

6.4 Eliciting opinion about a general scalar quantity 132

6.5 Eliciting opinion about a set of proportions 137

6.6 Eliciting opinion about the parameters of a multivariate normal distribution 139

6.7 Eliciting opinion about the parameters of a linear regression model 142

6.8 Eliciting opinion about the parameters of a generalised linear model 145

6.9 Elicitation methods for other problems 147

6.10 Deficiencies in existing research 149

6.11 Conclusions 150

6.11.1 Elicitation practice 150

6.11.2 Research questions 151

7 Eliciting Distributions – Uncertainty and Imprecision 153

7.1 Introduction 153

7.2 Imprecise probabilities 153

7.3 Incomplete information 156

7.4 Summary 160

7.5 Conclusions 160

7.5.1 Elicitation practice 160

7.5.2 Research questions 160

8 Evaluating Elicitation 161

8.1 Introduction 161

8.1.1 Good elicitation 161

8.1.2 Inaccurate knowledge 161

8.1.3 Automatic calibration 162

8.1.4 Lessons of the psychological literature 163

8.1.5 Outline of this Chapter 163

8.2 Scoring rules 163

8.2.1 Scoring rules for discrete probability distributions 165

8.2.2 Scoring rules for continuous probability distributions 169

8.3 Coherence, feedback and overfitting 171

8.3.1 Coherence and calibration 171

8.3.2 Feedback and overfitting 173

8.4 Conclusions 176

8.4.1 Elicitation practice 176

8.4.2 Research questions 177

9 Multiple Experts 179

9.1 Introduction 179

9.2 Mathematical aggregation 180

9.2.1 Bayesian methods 180

9.2.2 Opinion pooling 181

9.2.3 Cooke’s method 184

9.2.4 Performance of mathematical aggregation 185

9.3 Behavioural aggregation 186

9.3.1 Group elicitation 186

9.3.2 Other methods of behavioural aggregation 188

9.3.3 Performance of behavioural methods 190

9.4 Discussion 190

9.5 Elicitation practice 191

9.6 Research questions 191

10 Published Examples of the Formal Elicitation of Expert Opinion 193

10.1 Some applications 193

10.2 An example of an elicitation interview – eliciting engine sales 193

10.3 Medicine 195

10.3.1 Diagnosis and treatment decisions 195

10.3.2 Clinical trials 199

10.3.3 Survival analysis 201

10.3.4 Clinical psychology 202

10.4 The nuclear industry 204

10.5 Veterinary science 206

10.6 Agriculture 207

10.7 Meteorology 208

10.8 Business studies, economics and finance 209

10.9 Other professions 212

10.10 Other examples of the elicitation of subjective probabilities 213

11 Guidance on Best Practice 217

12 Areas for Research 223

Glossary 227

Bibliography 267

Author Index 307

Index 313

Read More Show Less

Customer Reviews

Be the first to write a review
( 0 )
Rating Distribution

5 Star

(0)

4 Star

(0)

3 Star

(0)

2 Star

(0)

1 Star

(0)

Your Rating:

Your Name: Create a Pen Name or

Barnes & Noble.com Review Rules

Our reader reviews allow you to share your comments on titles you liked, or didn't, with others. By submitting an online review, you are representing to Barnes & Noble.com that all information contained in your review is original and accurate in all respects, and that the submission of such content by you and the posting of such content by Barnes & Noble.com does not and will not violate the rights of any third party. Please follow the rules below to help ensure that your review can be posted.

Reviews by Our Customers Under the Age of 13

We highly value and respect everyone's opinion concerning the titles we offer. However, we cannot allow persons under the age of 13 to have accounts at BN.com or to post customer reviews. Please see our Terms of Use for more details.

What to exclude from your review:

Please do not write about reviews, commentary, or information posted on the product page. If you see any errors in the information on the product page, please send us an email.

Reviews should not contain any of the following:

  • - HTML tags, profanity, obscenities, vulgarities, or comments that defame anyone
  • - Time-sensitive information such as tour dates, signings, lectures, etc.
  • - Single-word reviews. Other people will read your review to discover why you liked or didn't like the title. Be descriptive.
  • - Comments focusing on the author or that may ruin the ending for others
  • - Phone numbers, addresses, URLs
  • - Pricing and availability information or alternative ordering information
  • - Advertisements or commercial solicitation

Reminder:

  • - By submitting a review, you grant to Barnes & Noble.com and its sublicensees the royalty-free, perpetual, irrevocable right and license to use the review in accordance with the Barnes & Noble.com Terms of Use.
  • - Barnes & Noble.com reserves the right not to post any review -- particularly those that do not follow the terms and conditions of these Rules. Barnes & Noble.com also reserves the right to remove any review at any time without notice.
  • - See Terms of Use for other conditions and disclaimers.
Search for Products You'd Like to Recommend

Recommend other products that relate to your review. Just search for them below and share!

Create a Pen Name

Your Pen Name is your unique identity on BN.com. It will appear on the reviews you write and other website activities. Your Pen Name cannot be edited, changed or deleted once submitted.

 
Your Pen Name can be any combination of alphanumeric characters (plus - and _), and must be at least two characters long.

Continue Anonymously

    If you find inappropriate content, please report it to Barnes & Noble
    Why is this product inappropriate?
    Comments (optional)