Test Development: Fundamentals for Certification and Evaluation

by Melissa Fein

Overview

Test Development: Fundamentals for Certification and Evaluation provides practical guidelines for the classical approach to test development – specifically for certification and credentialing. Most of the existing high-quality books on the topic of criterion-referenced test (CRT) development are written for professional statisticians and psychometricians. And the books written for the general reader tend to lack accurate, user-friendly coverage of some of the most critical topics. This book serves as an accessible resource on test development that does not require any technical background or expertise. Part I, Conceptual Principles, covers the process of CRT development in narrative form. Some of the topics covered include:

defining content domain as related to job analysis and assessment objectives

item writing and scoring

understanding validity and reliability

conducting item analysis

defining and differentiating the major standard setting method

preparing for and understanding issues with test administration.

Part II, the Technical Appendix, contains the computational instructions and supplementary background material. This do-it-yourself manual is a valuable resource for training program developers, credential program developers, as well as other relevant human resources personnel and upper management who need to be on board for change management related to testing programs.

Product Details

ISBN-13:	9781607286769
Publisher:	Association for Talent Development
Publication date:	10/09/2012
Sold by:	Barnes & Noble
Format:	eBook
Pages:	296
File size:	1 MB

About the Author

Melissa Fein, PhD, has graduate degrees in statistics, psychology, and economics, along with coaching certification, and first started working in the field of training evaluation in 1988. She began private consulting on industrial psychology/evaluation issues in 1996, and for a number of years was affiliated with the Maryland Assessment Research Center for Education Success (MARCES), at the University of Maryland, College Park, where she has taught a variety of courses in measurement, statistics, and evaluation.

Table of Contents

Preface ix

Part I Conceptual Principles 1

1 Introduction 3

Purpose of This Book 4

Content Coverage of This Book 5

References and Resources 8

2 Credentialing 9

Introduction 10

Motivations Underlying the Development of Credentialing Programs 11

Differentiating Between Types of Credentials 13

Components of the Credentialing Process 20

Accreditation 26

Summary 28

References and Resources 29

Related Topics in the Technical Appendix 32

3 Evaluation & CRT Development 33

Introduction 34

Formative and Summative Evaluation 34

Designs for Measuring Program Effectiveness 35

Kirkpatrick Model 36

Logic Modeling 37

Integration of Evaluation Frameworks: Hybrid Models 38

Relevance of CRTs to Evaluation in HPT/HPI 40

CRT Development and Monitoring Training Vendors 40

Summary 42

References and Resources 42

4 Approaches to Test Development 45

Introduction 46

Classical Test Theory 46

Item Response Theory 48

Computerized Adaptive Testing 49

Computerized Testing 50

IRT Resources 50

Sample Size 51

Using Consulting Services vs. In-House Developers 51

Summary 55

References and Resources 56

5 Defining Content Domain (Part I)—Job Analysis 57

Introduction 58

Stage I: Initial Task Inventory 58

Stage II: Survey Tool Development 62

Stage III: Survey Administration 65

Stage IV: Instructional Design and CRT Content Weighting 67

Summary 69

References and Resources 69

6 Denning Content Domain (Part II)—Assessment Objectives and the Table of Specifications 71

Introduction 72

Domains and Taxonomies 73

Writing Assessment/Learning Objectives 82

Table of Specifications 88

Summary 89

References and Resources 92

7 Item Writing and Scoring 93

Introduction 94

Fidelity and Authenticity 94

Item Types 102

General Guidelines for Item Writing 103

Multiple-Choice Items 105

True/False Items 108

Traditional Matching Item Sets and Extended Matching Items 110

Testes 111

Performance Assessment 111

Scoring Keys 113

Rater Training and Sources of Bias 116

Item Shells and Item Banks 119

Additional Item Format Issues 120

Organizing the Item-Writing Team: Item Writing and Reviews 121

Summary 122

References and Resources 123

Related Topics in the Technical Appendix 125

8 Validity 127

Introduction 128

Face Validity 129

Content Validity 130

Decision Validity (Criterion-Related Validity) 132

Survey Fatigue and Decision Validity 134

Summary 135

References/Resources 136

Related Topics in the Technical Appendix 136

9 Reliability 137

Introduction 138

Coefficient-a 140

Mastery Decision Consistency 141

Equivalent/Alternate Forms 142

Rater Agreement 142

Reliability of a Two-Component Test 144

Strategies for Enhancing Reliability Measurement 145

Summary 146

Additional Notes on Reliability 147

References and Resources 147

Related Topics in the Technical Appendix 148

10 Item Analysis 149

Introduction 150

Item Difficulty 150

Item Discrimination 150

Distractor Analysis 152

Instructional Sensitivity 153

Examinee Sample and Item Statistics 154

Summary 155

References and Resources 155

Related Topics in the Technical Appendix 155

11 Standard Setting 157

Introduction 158

Process Issues 158

Bookmark Procedure 162

Angoff Variations 165

Additional Standard Setting Methods: Contrasting Groups and Borderline Group 167

Standard Setting in Rubric Development 168

Standard Setting and Program Evaluation 169

Imperfection of Cut Scores and the Standard Error of Measurement 169

Summary 171

References and Resources 172

Related Topics in the Technical Appendix 173

12 Test Administration Issues 175

Introduction 176

Test Directions 176

Compiling and Piloting the Test 177

Time Limits 178

Accommodations 178

Reading Level 179

Test Security 179

Testing Formats 180

Adverse Impact 181

Score Reporting 183

References and Resources 184

Related Topics in the Technical Appendix 184

13 A Conclusion of Caveats 185

Part II Technical Appendix 187

14 Levels of Measurement 189

Overview of Levels of Measurement 189

Levels of Measurement (LOM) and Item Scores 191

LOM and Test Scores 191

References and Resources 192

15 Mathematical Tools and Microsoft Excel 193

Mathematical Tools 194

LOM and Correlation Coefficients 202

References and Resources 202

16 Classical Item Analysis 203

Item Difficulty 203

Item Discrimination 208

Distractor Analysis 214

References and Resources 216

17 Reliability 217

Computing Coefficient-a (alpha) 217

Computing Rater Agreement 222

Computing Parallel Forms Reliability 223

Computing Mastery Decision Consistency 225

Creation of a Composite Score 227

Reliability Computation of Two-Part CRTs 228

References and Resources 229

18 Standard Error of Measurement (SEM) 231

Computation 231

Error Bands 232

Implications 233

19 Validity 235

Item Labels: Coding Items 235

Item Development Worksheet 237

Computing Decision Validity 239

Level 2 Foundations: Computing Predictive Validity and Kirkpatrick Level 3 241

References and Resources 245

20 Scaled Scores 247

21 Standard Setting 249

Ordering Items by Difficulty 249

Median Computation: Borderline Groups 250

SME Credentials Form 251

Standard Setting Facilitator's Worksheet 253

Confidentiality Agreement Components 254

Credential Team Leader's Worksheet 255

22 Job Analysis 257

Hints for Running Focus Groups for J/PA 257

Hints for J/PA Interviews 258

Job Analysis, DIF Scores, and Content Weighting 259

Conclusion: Overview of the Criterion-Referenced Test (CRT) Development Process 265

Publishers Note 268

About the Author 269

Index 271