Developing Tests and Questionnaires for a National Assessment of Educational Achievement

by Prue Anderson, George Morgan

Overview

'Developing Tests and Questionnaires for a National Assessment of Educational Achievement' is the second in the 'National Assessments of Educational Achievement' series. It is designed to help build capacity in carrying out technically adequate assessments of national levels of student achievement. It introduces readers to the activities involved in the development of achievement tests, and includes developing an assessment framework, writing multiple choice and constructed response type items, pretesting, producing test booklets, and handscoring items.

A section on questionnaire construction features designing questionnaires, writing questions, coding responses, and linking questionnaire and test score data. The final section covers the development of a test administration manual, selecting test administrators, and contacting sampled schools.

A companion CD contains examples of released items from national and international tests, sample questionnaires, and administrative manuals.

The book should serve as a useful resource for those interested in educational assessment and evaluation, especially for personnel engaged in national assessments of student achievement.

'National Assessments of Educational Achievement'
Effective assessment of the performance of educational systems is a key component in developing policies to optimize the development of human capital around the world. The five books in the National Assessments of Educational Achievement series introduce key concepts in national assessments of student achievement levels, from policy issues to address when designing and carrying out assessments through test development, questionnaire design, sampling, organizing and carrying out data collection, data cleaning, statistical analysis, report writing, and using results to improve educational quality.

Product Details

ISBN-13:	9780821374979
Publisher:	World Bank Publications
Publication date:	06/13/2008
Series:	National Assessments of Educational Achievement , #2
Pages:	188
Product dimensions:	6.80(w) x 9.90(h) x 0.40(d)

Table of Contents

Preface     xi
About the Authors and Editors     xiii
Acknowledgments     xv
Abbreviations     xvii
Constructing Achievement Tests
Introduction     3
Developing an Assessment Framework     9
The Test Blueprint or Table of Specifications     10
Validity     16
Test Language     16
Item Format     17
Student Population to Be Assessed     24
Reporting Results     24
Contexts     25
Item Writing     27
Item Difficulty     29
Item Bias     30
Stimulus Material     30
Item Format     33
Practice Items     45
Item Layout and Design     46
The Item-Writing Team     52
Item Panels     55
Other Reviewers     57
Tracking Items     58
Pretesting Items     61
Designing the Pretest Form     64
Printing and Proofreading the Pretest     68
Implementing the Pretest     70
Scoring the Pretest     71
Reliability     76
Selecting Test Items     79
Producing the Final Test     85
Designing the Final Test     85
Printing and Proofreading     88
Hand-Scoring Test Items     91
Constructing Questionnaires     97
Designing Questionnaires     99
Questionnaire Content     102
Questionnaire Blueprint     105
Questionnaire Items     105
Item Format     108
Language of the Questionnaire     109
Respondents     109
Questionnaire Administration     110
Data Analysis Plan     111
Item Writing for Questionnaires     113
Questions     114
Statements     114
Response Categories     115
Managing Sensitive Issues     117
Questionnaire Layout     117
Reviewing Questionnaires     119
Coding Questionnaire Responses     121
Preparing Questionnaires for Data Entry     123
Coding Missing or Ambiguous Responses     123
Matching Questionnaires and Test Data     125
Student Questionnaires     125
Parent Questionnaires     126
Teacher and Head-Teacher Questionnaires     126
Designing a Manual for Test Administration     127
The Test Administrators' Manual     129
Contents of the Manual     130
Use of the Manual     131
Features of a Manual     132
How Much Detail Is Necessary?     132
Practice Questions     134
Tryout     134
Review     137
The Test Administrator     139
Choice of Test Administrator     139
Following Instructions     140
Quality Assurance     142
Administrator's Checklist     142
Informing Schools About the National Assessment     145
Glossary     147
Further Reading     153
Examples of Test and Questionnaire Items and Administration Manuals on CD     157
Index     163
Boxes
Papua New Guinea Mathematics Curriculum     11
New Zealand English Curriculum     11
Examples of Multiple-Choice Items     18
Example of a Closed Constructed-Response Item     19
Examples of Open-Ended Short-Response Items     19
Example of an Essay Prompt     20
Example of Irrelevant Stimulus Material     32
Example of an Item with Inaccurate or Misleading Information      33
Example of a Multiple-Choice Item     34
Punctuation in Complete Sentences     35
Punctuation in a List     35
Minimizing Reading     36
Item with a Negative Stem     36
Poorly Paired Distractors     37
Dealing with Pairs in Distractors     37
Confusing Open-Ended Item with Unclear Directions     40
Good Example of a Closed Constructed-Response Item     41
Item with Partial Credit     42
Example of an Open-Ended Response Item with Scoring Guide     43
Example of a Closed-Constructed Item with Scoring Guide     44
Using Pictures to Reduce Words     48
Keeping Images Simple     49
Labeling Graphs Clearly     49
Labeling Maps Clearly     50
Leaving Space in the Stimulus Material     51
Example of Style Sheet for Item Writers     54
Example of an Item in Multiple-Choice and Open-Ended Format     71
Example of a Data Entry Sheet for a Pretest     73
Example of a Test Cover Page     86
Attitudes and Values Questionnaire Blueprint     106
Poor Alignment of Boxes and Response Categories     118
Better Alignment of Boxes and Response Categories      118
Grayscale Coding Example     122
Example Treating Items as Separate Categories for Data Entry     122
Administration Manual Instructions     133
Information for Teachers and Principals     134
Administration of Practice Items     135
Administration Checklist: An Example from the Philippines     143
Figures
National Assessment Organizational Chart     4
Overview of National Assessment Activities     5
Example of Circular Linking of Items     65
Model for Vertically Linking Items     66
Guide to CD Materials on Tests, Questionnaires, and Manuals     158
Tables
National Assessment Stages in Test Development and Questionnaire Design     6
Blueprint for a Middle Primary Mathematics Test     13
TIMSS Mathematics Blueprint, Grades 3 and 4     14
Papua New Guinea Mathematics Content Blueprint     15
Advantages and Limitations of Item Formats     23
Papua New Guinea Mathematics Test-Item Formats     24
Link Items in Two Reading Units     67
Part of a Spreadsheet for Tracking Items across Forms     68
Example of Output from Analysis of a Multiple-Choice Item     80
Example of Output from Analysis of an Open-Ended Partial-Credit Item     81
Components of Questionnaire Development     101
Functions of Reading in an International Study: Weights Used to Create Two New Variables, "Reading for a Utilitarian Purpose" and "Reading for Enjoyment"     107