Table of Contents
Preface ix
1 What This Book Is About 3
What Is Program Evaluation? 3
Types of Program Evaluations 8
Basic Characteristics of Program Evaluation 13
Relation of Program Evaluation to the General Field of Policy Analysis 15
Assessing Government Performance: Program Evaluation and Performance Measurement 15
A Brief History of Program Evaluation 17
What Comes Next 19
Key Concepts 20
Do It Yourself 20
2 Defensible Program Evaluations: Four Types of Validity 26
Defining Defensibility 26
Types of Validity: Definitions 27
Types of Validity: Threats and Simple Remedies 28
Basic Concepts 47
Do It Yourself 48
3 Internal Validity 51
The Logic of Internal Validity 51
Making Comparisons: Cross Sections and Time Series 54
Threats to Internal Validity 55
Summary 63
Three Basic Research Designs 64
Rethinking Validity: The Causal Model Workhorse 66
Basic Concepts 68
Do It Yourself 69
A Summary of Threats to Internal Validity 70
4 Randomized Field Experiments 73
Basic Characteristics 73
Brief History 74
Caveats and Cautions About Randomized Experiments 76
Types of RFEs 79
Issues in Implementing RFEs 92
Threats to the Validity of RFEs: Internal Validity 96
Threats to the Validity of RFEs: External Validity 100
Threats to the Validity of RFEs: Measurement and Statistical Validity 101
Conclusion 101
Some Cool Examples of RFEs 102
Basic Concepts 103
Do It Yourself: Design a Randomized Field Experiment 104
5 The Quasi Experiment 110
Defining Quasi-Experimental Designs 110
The One-Shot Case Study 111
The Posttest-Only Comparison-Group (PTCG) Design 113
The Pretest-Posttest Comparison-Group (PTPTCG) (The Nonequivalent Control-Group) Design 119
The Pretest-Posttest (Single-Group) Design 123
The Single Interrupted Time-Series Design 125
The Interrupted Time-Series Comparison-Group (TTSCG) Design 131
The Multiple Comparison-Group Time-Series Design 134
Summary of Quasi-Experimental Design 135
Basic Concepts 136
Do It Yourself 137
6 The Nonexperimental Design: Variations on the Multiple Regression Theme 143
What Is a Nonexperimental Design? 143
Back to the Basics: The Workhorse Diagram 144
The Nonexperimental Workhorse Regression Equation 146
Data for the Workhorse Regression Equation 148
Interpreting Multiple Regression Output 149
Assumptions Needed to Believe That b Is a Valid Estimate of B [E(b) = B] 164
Assumptions Needed to Believe the Significance Test for b 184
What Happened to the R2? 190
Conclusion 191
Basic Concepts 192
Introduction to Stata 194
Do It Yourself: Interpreting Nonexperimental Results 197
7 Designing Useful Surveys for Evaluation 209
The Response Rate 210
How to Write Questions to Get Unbiased, Accurate, Informative Responses 217
Turning Responses into Useful Information 224
For Further Reading 233
Basic Concepts 233
Do It Yourself 234
8 Summing It Up: Meta-Analysis 239
What Is Meta-Analysis? 239
Example of a Meta-Analysis: Data 240
Example of a Meta-Analysis: Variables 241
Example of a Meta-Analysis: Data Analysis 242
The Role of Meta-Analysis in Program Evaluation and Causal Conclusions 243
For Further Reading 244
Index 247
About the Author 253