Utilization-Focused Evaluation

by Michael Quinn Patton, Charmagne E. Campbell-Patton

Overview

Founded in 1965, SAGE is a leading independent academic and professional publisher of innovative, high-quality content. Known for our commitment to quality and innovation, SAGE has helped inform and educate a global community of scholars, practitioners, researchers, and students across a broad range of subject areas.

Product Details

ISBN-13:	9781544379456
Publisher:	SAGE Publications
Publication date:	11/29/2021
Edition description:	Fifth Edition
Pages:	432
Product dimensions:	8.00(w) x 10.00(h) x (d)

About the Author

Michael Quinn Patton is author of more than a dozen books on evaluation including Qualitative Research & Evaluation Methods, 4th ed (2015), Blue Marble Evaluation (2020), Principles-Focused Evaluation (2018), Facilitating Evaluation (2018) and Developmental Evaluation (2011). Based in Minnesota, he was on the faculty of the University of Minnesota for 18 years and is a former president of the American Evaluation Association (AEA). Michael is a recipient of the Alva and Gunnar Myrdal Evaluation Practice Award, the Paul F. Lazarsfeld Evaluation Theory Award, and the Research on Evaluation Award, all from AEA He has also received the Lester F. Ward Distinguished Contribution to Applied and Clinical Sociology Award from the Association for Applied and Clinical Sociology. In 2021 he received the first Transformative Evaluator Award from EvalYouth. He is an active speaker, trainer, and workshop presenter who has conducted applied research and evaluation on a broad range of issues and has worked with organizations and programs at the international, national, state, provincial, and local levels. Michael has three children--a musician, an engineer, and an evaluator--and four grandchildren. When not evaluating, he enjoys exploring the woods and rivers of Minnesota, where he lives.

Charmagne Campbell-Patton is second-generation evaluator and twelfth generation European settler based in Mnisota Makoce. Her professional experience spans fifteen years of program design, implementation and evaluation work across a range of fields, including youth engagement, education, criminal and juvenile justice and philanthropy. She specializes in qualitative research methods and utilization-focused developmental evaluation. Charmagne holds a BA in Political Science from Grinnell College and an MA in International Peace and Conflict Resolution from American University's School of International Service. She has authored several articles and blog posts, but this is her first book. A global citizen with deep local roots, Charmagne resides near the Mississippi River on Dakota land in Minneapolis with her husband, two young children and cat.

Table of Contents

Preface xv

Acknowledgments xxiii

About the Authors xxiv

Part 1 What Utilization-Focused Evaluation is and Why It Matters 1

Chapter 1 The What, Why, How, Who, When, and Where of Utilization-Focused Evaluation 3

What Is Program Evaluation? 4

Utilization-Focused Evaluation 5

What Is Utilization-Focused Evaluation? 5

Why U-FE? 6

How Is U-FE Done? 6

Who Is U-FE for? 7

When Is U-FE Done? 8

Where Is U-FE Done? 8

Questioning as an Inquiry and Design Framework 9

Intended Use by Intended Users 10

Alternatives and Barriers to Utilization-Foe used Evaluation 13

Evaluation Use as a Value Proposition 15

Summary and Conclusion 15

Chapter 2 The Essence of Utilization-Focused Evaluation Expressed as Minimum Specifications 19

MIN SPEC 1 Honor the Personal Factor: Identify and Engage Primary Intended Users 21

A Personal-Factor Exemplar: Engaging Educational Leaders in Evaluation 22

MIN SPEC 2 Be Purpose-Driven: Focus on Priority Intended Uses 23

A Purpose-Driven Exemplar: Evaluation for Improvement 26

MIN SPEC 3 Facilitate Process Use: Be Active, Reactive, Interactive, and Adaptive in Engaging Users in All Aspects of the Evaluation 27

The Impacts of Process Use 32

Process Use as Capacity Building 32

Process as Outcome 33

A Process Use Exemplar: Organization-Wide Learning 33

MIN SPEC 4 Take a Full-Journey Stance: Focus on Use From Beginning to the End 34

Readiness 34

Pay Attention Along the Way 36

Endings: Follow Up to Support Utilization 36

Full-Journey Stance Premise 36

Readiness: Baseline Associations With and Perceptions of Evaluation 37

A Full-Journey Stance Exemplar: Developmental Evaluation as Maori 39

MIN SPEC 5 Adapt to Context Changes: When the Context for an Evaluation Changes, the Evaluation May Have to Change 40

A Contextual Adaptation Exemplar: The 2020 Global Health Pandemic 42

Summary and Conclusion 44

Chapter 3 Utilization-Focused Evaluative Thinking 47

How We Make Decisions 47

Evaluation Logic 49

The Logic of Evaluation-With a U-FE Twist 50

Statistical Analysis as Rigorous Thinking 51

Evaluative Thinking as a Form of Reasoned Argument 54

Rigorous Evaluative Thinking 55

Utilization-Focused Evaluative Thinking 58

Intersection of the Personal and Professional 60

Summary and Conclusion 61

Chapter 4 The High Stakes of Evaluation Use, Nonuse, and Misuse 65

Evaluation for Accountability 65

Organizational Culture and Evaluation Use 66

Determining What Works 67

Using Local Evaluations 68

Beyond Good Intentions 69

Politics and Evaluation 69

The High Stakes of Evaluation Use 71

Speak Truth to Power, but What Happens When Those With Power Aren't Listening 71

The 2020 Global Coronavirus Pandemic: Political Lessons About Evaluation Use 72

Clear and Credible Communications 73

Pandemic Evaluation Contributions When Evaluations Were Used 75

Timeliness of Evaluation Use 77

Evaluation Use in the Information Age 77

Use of Knowledge in Daily Life 78

Summary and Conclusion 78

Part 2 Utilization-Focused Evaluation Principles 83

Chapter 5 Guide to Utilization-Focused Evaluation Principles 85

Principles Versus Rules and Recipes 85

Effectiveness Principles 86

Utilization-Focused Evaluation Principles 86

Overarching and Operating Principles 87

GUIDE Framework for Principles 88

GUIDE Applied to Utilization-Focused Evaluation 88

Alignment of Personal and Professional Principles 90

Principles-Focused Evaluation Applied to Utilization-Focused Evaluation 90

Summary and Conclusion 91

Chapter 6 Contextualize: U-FE Operating Principle 1 95

A Situation Analysis Example 96

The Situation Analysis Challenge 98

Force Field Analysis 100

Additional Factors to Assess 100

Distinguishing Simple, Complicated and Complex Situations, and Interventions 102

Simple Evaluation Challenges 102

Complicated Evaluation Challenges 103

Complex Evaluation Challenges 103

Special Contextual Challenges 107

Summary and Conclusion 110

Chapter 7 Personalize: U-FE Operating Principle 2 113

10 Tips for Identifying Primary Intended Users 115

Examples of Working With Primary Intended Users 121

Facilitation Challenges Working With Intended Users 122

Diversions Away From Primary Intended Users 122

Summary and Conclusion 127

Chapter 8 Customize: U-FE Operating Principle 3 129

Purpose Decisions Connected to Situation Analysis 130

Intended Uses Menu: Six Alternative Evaluation Purposes 132

Purpose 1 Judging Overall Merit, Worth, and Significance 133

Purpose 2 Learning and improvement 135

Purpose 3 Accountability 137

Purpose 4 Monitoring for Internal Management and Oversight. 140

Purpose 5 Generating Knowledge and Lessons 142

Purpose 6 Supporting Innovation and Adaptation: Developmental Evaluation 144

Applying Purpose and Use Distinctions 146

From Purpose to Data Collection 148

Examples of How Purpose Affects Use 150

Making Menu Selections: Connecting Decisions to Uses 152

Summary and Conclusion 152

Chapter 9 Align: U-FE Operating Principle 4 155

Focusing Evaluation Questions 156

Generate Actionable Questions 157

Three Lessons to Guide Question Formulation 159

Three U-FE Question Formulation Challenges 161

Generate Actionable Questions 162

Skill in Question Formulation 162

The Focus in Utilization-Focused Evaluation 162

Aligning Users' Questions With Design and Methods 164

Controversy About U-FE Approach to Methods Decision-Making 164

Principles of Methodological Engagement With Primary Intended Users 165

Facilitation Challenges In Making Methods Decisions With Primary Intended Users 168

Finding the Sweet Spot 168

Involving Intended Users in a Technical Decision 168

Summary and Conclusion 169

Chapter 10 Prioritize: U-FE Operating Principle 5 173

From MIN SPEC to Principle 173

Preparation and Readiness 174

Simulate Use of Findings Before Data Collection 176

An Ancient Practice Example 177

Fabricating Data for Simulation 179

Ongoing Attention to Use During Data Collection 179

Challenges Along the Way 180

Making Sense of Findings 181

Four Steps to Generating Useful Findings 181

Answering the What Question: Utilization-Focused Analysis With Intended Users 182

Preparing for Analysis 183

Facilitating Analysis 185

Facilitating Interpretation 186

Rendering Judgment 187

Generating Recommendations 189

Putting It All Together 189

Findings From Data Analysis 190

Interpretation 190

Judgment 190

Recommendation Options 190

Summary and Conclusion 190

Chapter 11 Engage Users: U-FE Operating Principle 6 195

Utilization-Focused Evaluation Facilitation 195

Tips for Being Active-Reactive-Interactive-Adaptive 197

Facilitation Tips Summary 201

Regional Economic Development Agency Facilitation Example 202

Facilitation Principles 204

Five Operating Principles for Evaluation Facilitation 204

Principles-Foe used Evaluation Example 209

Summary and Conclusion 210

Chapter 12 Strategize Process Use: U-FE Operating Principle 7 213

Process Use Defined 214

Five Process Use Facilitation Tasks 215

Process Use for Organization-Wide Learning: An Illuminative Example 220

Evaluation Models That Spotlight Process Use 221

Evaluation Approaches That Center on User Engagement 221

Principles for Use in Guiding Collaborative Approaches to Evaluation 223

The Power of Group Processes 226

Changed Evaluator Role: From Independent Judge to Social Change Facilitator 227

Process Use as Intervention-Oriented Evaluation 227

Intervention-Oriented Evaluation 230

Challenges to Process Use 230

Being Thoughtful and Ethical 231

Summary and Conclusion 232

Chapter 13 Adapt: U-FE Operating Principle 8 237

Focus on Being Adaptive 237

Utilization-Focused Evaluation for Adaptive Management 238

Six Examples of Evaluation Adaptation 239

Adapting to a Critical Community Incident 239

Targeting Disparities 239

Program Participants Change 240

Going to Scale 240

Adapting to Change in Program Resources 240

Adapting to COVID-19 Trauma 240

Evaluator as Adaptive Problem Solver 240

Adapting Evaluation Designs and Implementation 241

Adaptation as Mindset 242

Risk Assessment to Anticipate Adaptation 242

Turbulent Environments and Changing Goals 243

Adaptability as Agility 244

Agile Evaluation 244

The Agile Manifesto 246

Creating the Conditions for Evaluation Adaptability 248

Summary and Conclusion 249

Chapter 14 Mobilize Follow-Through: U-FE Operating Principle 9 253

Utilization-Focused Reporting 254

Fear of Negative Findings 255

Prospect Theory and Evaluation 256

Generating Actionable Recommendations 257

10 Guidelines for Generating Useful Recommendations 258

Follow-Through: Completing The Endgame 260

Follow-Through as an Evaluation Standard of Excellence 260

Follow-Through in Practice: Endgame Examples 261

The Challenge of Meeting Information Needs and Expectations 262

Example of Follow-Through at an Organizational Level 263

10 Follow-Through Lessons 264

Summary and Conclusion 265

Chapter 15 Reflect and Learn: U-FE Operating Principle 10 269

Reflective Practice 270

Single-, Double-, and Triple-Loop Learning 271

Reflexivity 272

How Evaluators Influence Cultural Competence and Responsiveness 273

The Utilization-Focused Evaluator's Role and Values 274

Learning Over Time: Building Momentum for Use Across Evaluations 276

The Utilization-Foe used Evaluation Flywheel 276

The Momentum of the Flywheel 279

Forward Motion 279

Summary and Conclusion 280

Chapter 16 Learning Breakthroughs in Applying Utilization-Focused Evaluation Principles 285

Lessons Learned About Utilization-Focused Evaluation Principles 285

From Steps to Principles 296

Maintain Evaluation Use as the Quintessential Criterion of Evaluation Excellence 299

Summary and Conclusion 300

Part 3 New, Emergent, and Urgent Directions for Utilization-Focused Evaluation 303

Chapter 17 The High Stakes of Evaluative Thinking and Reality-Testing 305

Evaluation in the Context of a Posttruth World 305

Original High Hopes for Evaluative Thinking 307

Evaluation and Rationality 307

Reality Check: Evaluations Largely Unused 308

The High Stakes of Evaluative Thinking 308

Assessing What Is Good 310

The Banality of Goodness 310

The Banality of Evaluation 311

Thoughtlessness in a Posttruth World 312

Infodemic 312

Thinking Globally 315

Evaluation in the Epoch of the Anthropocene 315

Toward a Balanced Assessment 316

Blue Marble Evaluation 317

Thinking Glocally 318

Summary and Conclusion 319

Chapter 18 Using Evaluation to Contribute to a More Equitable and Sustainable World 321

Values and Valuing 322

Stake's Position on Evaluators' Values 323

Morality and Ethics of Evaluation 325

Whom Does an Evaluation-and an Evaluator-Serve? 326

Contributing to Social Betterment as an Intended Use 326

Serving the Common Good 327

Equity: No One Left Behind 328

Evaluation Commitments to Equity 329

Evaluator Competence 330

Contextualization (U-FE Principle 1): Equity Situation Analysis 331

Equity Through a Social Justice Lens 332

An Evaluation of Systemic Racism 333

Equity-Focused Evaluation 335

U-FE and Equity 336

Sustainability 336

Contextualization: Sustainability Situation Analysis 337

Responses to the Climate Emergency 337

Sustainability as a Universal Evaluation Criterion 338

U-FE and Sustainability 339

Equity and Sustainability as Mutually Reinforcing Criteria 339

Facilitating U-FE Consideration of the Criteria of Equity and Sustainability 340

Utilization-Foe used Evaluation Approach to Facilitating Criteria Priorities 341

Summary and Conclusion 342

Chapter 19 Evaluation for Transformation 345

Transformation Means "No One Left Behind" 346

Transformation, Transformation, Transformation 346

Contextualizing: Situation Analysis 346

Evaluating Transformation 348

Evaluate Transformation Fidelity 348

Utilization-Foe used Evaluation for Transformation Exemplar 353

Principles-Focused Transformation and Evaluation 355

Systems Thinking Integrated With Evaluative Thinking 357

The World's First Blue Marble Evaluation 357

Future Utilization-Focused Directions for Transforming Evaluation to Evaluate Transformation 358

Evaluation for a Transformed Future 359

Syndemic 360

Summary and Conclusion 360

Chapter 20 Utilization-Focused Evaluation for Transformation: Toward a More Thoughtful, Equitable, and Sustainable World 363

Five Decades of Utilization-Foe used Evaluation and Forward Into the Future 368

The Evolution of Evaluation Practice 368

The Evolution of Utilization-Focused Evaluation 369

Transformation Wisdom 371

References 373

Index 389