Table of Contents
Preface xv
Chapter 1 Introduction 1
1.1 Drawing Inferences from Intuitions, Anecdotes, and Correlations 2
1.2 Experiments as a Solution to the Problem of Unobserved Confounders 5
1.3 Experiments as Fair Tests 7
1.4 Field Experiments 8
1.5 Advantages and Disadvantages of Experimenting in Real-World Settings 13
1.6 Naturally Occurring Experiments and Quasi-Experiments 15
1.7 Plan of the Book 17
Suggested Readings 18
Exercises 18
Chapter 2 Causal Inference and Experimentation 21
2.1 Potential Outcomes 21
2.2 Average Treatment Effects 23
2.3 Random Sampling and Expectations 26
2.4 Random Assignment and Unbiased Inference 30
2.5 The Mechanics of Random Assignment 36
2.6 The Threat of Selection Bias When Random Assignment Is Not Used 37
2.7 Two Core Assumptions about Potential Outcomes 39
2.7.1 Excludability 39
2.7.2 Non-interference 43
Summary 44
Suggested Readings 46
Exercises 46
Chapter 3 Sampling Distributions, Statistical Inference, and Hypothesis Testing 51
3.1 Sampling Distributions 52
3.2 The Standard Error as a Measure of Uncertainty 54
3.3 Estimating Sampling Variability 59 3.4- Hypothesis Testing 61
3.5 Confidence Intervals 66
3.6 Sampling Distributions for Experiments That Use Block or Cluster Random Assignment 71
3.6.1 Block Random Assignment 71
3.6.1.1 Matched Pair Design 77
3.6.1.2 Summary of the Advantages and Disadvantages of Blocking 79
3.6.2 Cluster Random Assignment 80
Summary 85
Suggested Readings 86
Exercises 86
Appendix 3.1 Power 93
Chapter 4 Using Covariates in Experimental Design and Analysis 95
4.1 Using Covariates to Rescale Outcomes 96
4.2 Adjusting for Covariates Using Regression 102
4.3 Covariate Imbalance and the Detection of Administrative Errors 105
4.4 Blocked Randomization and Covariate Adjustment 109
4.5 Analysis of Block Randomized Experiments with Treatment Probabilities That Vary by Block 116
Summary 121
Suggested Readings 123
Exercises 123
Chapter 5 One-Sided Noncompliance 131
5.1 New Definitions and Assumptions 134
5.2 Denning Causal Effects for the Case of One-Sided Noncompliance 13 137
5.2.1 The Non-interference Assumption for Experiments That Encounter Noncompliance 138
5.2.2 The Excludability Assumption for One-Sided Noncompliance 140
5.3 Average Treatment Effects, Intent-to-Treat Effects, and Complier Average Causal Effects 141
5.4 Identification of the CACE 143
5.5 Estimation 149
5.6 Avoiding Common Mistakes 152
5.7 Evaluating the Assumptions Required to Identify the CACE 15 155
5.7.1 Non-interference Assumption 155
5.7.2 Exclusion Restriction 156
5.8 Statistical Inference 157
5.9 Designing Experiments in Anticipation of Noncompliance 161
5.10 Estimating Treatment Effects When Some Subjects Receive "Partial Treatment" 164
Summary 165
Suggested Readings 167
Exercises 168
Chapter 6 Two-Sided Noncompliance 173
6.1 Two-Sided Noncompliance: New Definitions and Assumptions 175
6.2 ITT, ITTD, and CACE under Two-Sided Noncompliance 179
6.3 A Numerical Illustration of the Role of Monotonicity 181
6.4 Estimation of the CACE: An Example 185
6.5 Discussion of Assumptions 189
6.5.1 Monotonicity 190
6.5.2 Exclusion Restriction 191
6.5.3 Random Assignment 192
6.5.4 Design Suggestions 192
6.6 Downstream Experimentation 193
Summary 204
Suggested Readings 206
Exercises 206
Chapter 7 Attrition 211
7.1 Conditions Under Which Attrition Leads to Bias 215
7.2 Special Forms of Attrition 219
7.3 Redefining the Estimand When Attrition Is Not a Function of Treatment Assignment 224
7.4 Placing Bounds on the Average Treatment Effect 226
7.5 Addressing Attrition: An Empirical Example 230
7.6 Addressing Attrition with Additional Data Collection 236
7.7 Two Frequently Asked Questions 241
Summary 243
Suggested Readings 244
Exercises 244
Appendix 7.1 Optimal Sample Allocation for Second-Round Sampling 248
Chapter 8 Interference between Experimental Units 253
8.1 Identifying Causal Effects in the Presence of Localized Spillover 256
8.2 Spatial Spillover 260
8.2.1 Using Nonexperimental Units to Investigate Spillovers 264
8.3 An Example of Spatial Spillovers in Two Dimensions 264
8.4 Within-Subjects Design and Time-Series Experiments 273
8.5 Waitlist Designs (Also Known as Stepped-Wedge Designs) 276
Summary 281
Suggested Readings 283
Exercises 283
Chapter 9 Heterogeneous Treatment Effects 289
9.1 Limits to What Experimental Data Tell Us about Treatment Effect Heterogeneity 291
9.2 Bounding Var (τ) and Testing for Heterogeneity 292
9.3 Two Approaches to the Exploration of Heterogeneity: Covariates and Design 296
9.3.1 Assessmg Treatment-by-Covariate Interactions 296
9.3.2 Caution Is Required When Interpreting Treatment-by-Covariate Interactions 299
9.3.3 Assessing Treatment-by-Treatment Interactions 303
9.4 Using Regression to Model Treatment Effect Heterogeneity 305
9.5 Automating the Search for Interactions 310
Summary 310
Suggested Readings 312
Exercises 313
Chapter 10 Mediation 319
10.1 Regression-Based Approaches to Mediation 322
10.2 Mediation Analysis from a Potential Outcomes Perspective 325
10.3 Why Experimental Analysis of Mediators Is Challenging 328
10.4 Ruling Out Mediators? 330
10.5 What about Experiments That Manipulate the Mediator? 331
10.6 Implicit Mediation Analysis 333
Summary 336
Suggested Readings 338
Exercises 338
Appendix 10.1 Treatment Postcards Mailed to Michigan Households 343
Chapter 11 Integration of Research Findings 347
11.1 Estimation of Population Average Treatment Effects 350
11.2 A Bayesian Framework for Interpreting Research Findings 353
11.3 Replication and Integration of Experimental Findings: An Example 358
11.4 Treatments That Vary in Intensity: Extrapolation and Statistical Modeling 366
Summary 377
Suggested Readings 378
Exercises 379
Chapter 12 Instructive Examples of Experimental Design 383
12.1 Using Experimental Design to Distinguish between Competing Theories 384
12.2 Oversampling Subjects Based on Their Anticipated Response to Treatment 387
12.3 Comprehensive Measurement of Outcomes 393
12.4 Factorial Design and Special Cases of Non-interference 395
12.5 Design and Analysis of Experiments In Which Treatments Vary with Subjects' Characteristics 400
12.6 Design and Analysis of Experiments In Which Failure to Receive Treatment Has a Causal Effect 406
12.7 Addressing Complications Posed by Missing Data 410
Summary 414
Suggested Readings 415
Exercises 416
Chapter 13 Writing a Proposal, Research Report, and Journal Article 425
13.1 Writing the Proposal 426
13.2 Writing the Research Report 435
13.3 Writing the Journal Article 440
13.4 Archiving Data 442
Summary 444
Suggested Readings 445
Exercises 445
Appendix A Protection of Human Subjects 447
A.1 Regulatory Guidelines 447
A.2 Guidelines for Keeping Field Experiments within Regulatory Boundaries 449
Appendix B Suggested Field Experiments for Class Projects 453
B.1 Crafting Your Own Experiment 453
B.2 Suggested Experimental Topics for Practicum Exercises 455
References 461
Index 479
