Observational Measurement of Behavior

by Paul J. Yoder

Overview

An essential textbook for anyone preparing to be a researcher, this comprehensive volume introduces graduate students to key principles of observational measurement of behavior. Based on a course the highly respected authors taught at Vanderbilt University and the University of Minnesota, this text delves deeply into a highly effective approach to observational measurement: systematic observation.

Students will master both the theoretical principles of systematic observation and recommended research methods and techniques. They'll learn from practical examples that illustrate complex concepts, clear explanations of recommended research methods, definitions of key terms, and exercises and assignments that help them practice putting principles into action. Online companion materials include two free licenses for proprietary observational software that students can use to complete the exercises and assignments in this book.

Ideal for use in research methodology courses in diverse fields—including special education, communication sciences, psychology, and social work—this fundamental graduate text will prepare future researchers to skillfully collect, summarize, and communicate their observations of children's behavior.

STUDENTS WILL:

• Fully understand key methods of observational research and measurement
• Get comprehensive information on both foundational and advanced topics
• Learn from real-world examples based on the authors' extensive experience
• Apply specific recommendations for effective techniques and best practices

SELECTED TOPICS COVERED: validity and reliability * representativeness * measurement theory * behavior sampling and coding * observer training * metrics of observational variables * modifying and designing coding manuals * sequential analysis * generalizability theory

ONLINE COMPANION MATERIALS: To enhance their courses, instructors will get a full package of online materials, including two licenses for observational software, video clips students can use to practice coding behaviors, a suggested schedule for a semester-long course, exercises for students, and assignments with corresponding grading rubrics.

Product Details

ISBN-13:	9781681252469
Publisher:	Brookes Publishing
Publication date:	05/01/2018
Edition description:	New Edition
Pages:	272
Product dimensions:	7.00(w) x 10.00(h) x 0.00(d)

About the Author

Paul J. Yoder, Ph.D., Professor, Department of Special Education, Peabody College, Vanderbilt University, Nashville, Tennessee 37203

Dr. Paul Yoder has been studying the transition from prelinguistic to linguistic communication in multiple populations with disabilities for over two decades. He is a co-designer of Milieu Communication Teaching and has contributed to several studies examining the efficacy of this treatment. He teaches methods and measurement at Vanderbilt University.

Primary research activities of Frank J. Symons, Ph.D., are supported by the National Institute of Child Health and Human Development (NICHD), and they focus on improving the assessment and treatment of severe self-injurious behavior among individuals with developmental disabilities and pervasive developmental disorders. Dr. Symons was a research scientist at the Frank Porter Graham Child Development Center at the University of North Carolina at Chapel Hill and a postdoctoral fellow at the John F. Kennedy Center at the Peabody College of Vanderbilt University in Nashville, Tennessee. He is the co-author of Behavioral Observation: Technology and Applications in Developmental Disabilities (Paul H. Brookes Publishing Co., 2000).

Read an Excerpt

Read an excerpt
Chapter 1: Introduction to Systematic Observation and Measurement Contexts

Table of Contents

About the Online Companion Materials
About the Authors
Preface

• The Scope of This Book
• Topics and Corresponding Chapters
• The Book’s Iterative Teaching Style
• Using the Online Companion Materials

Acknowledgments

Section I Foundational Topics
Chapter 1 Introduction to Systematic Observation and Measurement Contexts

• Systematic Observation Using Count Coding
• Alternatives to Systematic Observation
• Ways to Quantify Observations
• The Rationale for Systematic Observation Using Count Coding
• The Importance of Falsifiable Research Questions or Hypotheses
• Objects of Measurement: The Continuum of Context-Dependent Behaviors to Generalized Person Characteristics
• Context-Dependent Behaviors
• Generalized Person Characteristics
• Generalized Behavioral Tendencies
• Skills
• Judging the Relative Scientific Value of Different Measures
• Reliability
• Validity
• Ecological Validity
• Representativeness
• Conclusions and Recommendations

Chapter 2 Validation of Observational Variables

• The Changing Concept of Validation
• Consequences of Not Attending to Validation
• Overview: Types of Validity by Objects of Measurement and Purposes
• Content Validation
• Varying Importance Ascribed to Content Validation
• Weaknesses of Content Validation
• Sensitivity to Change
• Influences on Sensitivity to Change
• Weakness of Sensitivity to Change as Way to Judge a Variable’s Validity
• Criterion-Related Validation
• The Primary Appeal of Criterion-Related Validation
• Weaknesses of Criterion-Related Validation
• Construct Validation
• Convergent Validity
• Discriminative Validation Evidence
• Nomological Validation Evidence
• Weaknesses of Convergent Validity
• Methods That Combine Convergent and Divergent Validity
• Multitrait, Multimethod (MTMM) Validation
• Confirmatory Factor Analysis as a Method of Validation
• Putting It All Together With Literature Synthesis
• An Implicit Weakness of Science?
• Conclusions and Recommendations

Chapter 3 Estimating Stable Measures of Generalized Person Characteristics Through Systematic Observation

• A Brief Overview of Measurement Theory
• Why Stable Estimates Maximize Convergent Construct Validity
• Two Ways to Stabilize Observational Measures
• Estimating Stable Skills Through Observation
• Definition of Measurement Context
• How Controlling Influential Contextual Variables Stabilizes Skill Estimates
• Why Skills Are Often Assessed in Clinics or Labs
• Estimating Stable Generalized Behavioral Tendencies Through Observation
• Representativeness, Revisited
• Definition of Contextual Measurement Error
• Contextual Measurement Error in Measures of Generalized Behavioral Tendencies
• How Averaging Scores Across Contexts Improves Measures of Generalized Behavioral Tendencies
• Naturalness of Observations and Representativeness, Revisited
• Computing Stability Coefficients
• Conclusions and Recommendations

Chapter 4 Designing or Adapting Coding Manuals

• Definition of a Coding Manual
• Deciding Whether to Write a New Coding Manual
• Recommended Steps for Modifying or Designing Coding Manuals
• Define Start and Stop Coding Rules
• Conceptually Define the Object of Measurement
• Define the Highest Level of Codable Behavior
• Determine the Level of Distinction Coders Have to Make
• Organize the Coded Categories into Mutually Exclusive Sets
• Decide How to Use Physically Based and/or Socially Based
Definitions
• Define the Lowest-Level Categories
• Determine Sources of Conceptual and Operational Definitions
• Define Segmenting Rules
• The Potential Value of Flowcharts
• Recommended Length of Coding Manuals
• Conclusions and Recommendations

Chapter 5 Coding

• The Elements of an Observational Measurement System Behavior Sampling
• The Superordinate Distinctions: Continuous Versus Intermittent
• The Subordinate Distinctions: Continuous Versus Intermittent
• Timed-Event Sampling
• Event Sampling
• Interval Sampling
• Types of Interval Sampling
• Whole-Interval Sampling
• Momentary-Interval Sampling
• Partial-Interval Sampling
• Summary of Interval Sampling
• Which Dimension of Behavior Should Be Estimated
• Summary of Behavior Sampling
• Participant Sampling
• Focal Sampling
• Multiple-Pass Sampling
• Conspicuous Sampling
• Reactivity
• When to Code Relative to When the Behavior Occurs
• Live Coding
• Coding From Recorded Sessions
• Recording Coding Decisions
• Paper and Pencil
• Observational Software
• Conclusions and Recommendations

Chapter 6 Common Metrics of Observational Variables

• Definition of Metric
• Quantifiable Dimensions of Behavior
• Proportion Metrics
• How Proportion Metrics Change the Meaning of Observational Variables
• Scrutinizing Proportions
• An Implicit Assumption of Proportion Metrics
• Testing Whether the Data Fit the Assumption of Proportion Metrics
• Consequences of Using a Proportion When the Data Do Not Fit the Assumption
• Alternative Methods to Control Influential Contextual Variables
• Statistical Control
• Procedural Control
• Aggregate Measures of Generalized Person Characteristics
• Weighted Counts
• Unit-Weighted Aggregates
• Group Analysis of Observational Variables
• Transforming the Metric
• Bootstrapping
• Analyzing Count Variables
• Conclusions and Recommendations

Chapter 7 Observer Training and Preventing Observer Drift

• Point-by-Point Agreement and Disagreement
• Point-by-Point Agreement of Interval-Sampled Data
• Point-by-Point Agreement of Timed-Event Data
• Discrepancy Matrices
• Discrepancy Discussions
• Using Discrepancy Discussions to Train Observers
• Creating Criterion-Coding Standards
• Training Observers: Remaining Steps
• Preventing Observer Drift
• Choosing a Method of Selecting Sessions for Agreement Checks
• Preventing or Addressing Observer Drift: Remaining Steps
• Conclusions and Recommendations

Chapter 8 Interobserver Reliability of Observational Variables

• General Principles of Interobserver Reliability Estimation
• Single-Case Design Concepts of Interobserver Reliability
• Session-Level Agreement Indices
• Summary-Level Agreement
• Point-by-Point Agreement
• Base Rate and All Indices of Point-by-Point Agreement
• Summary of Point-by-Point Agreement Indices
• Group-Design Concepts of Interobserver Reliability
• A Sample-Level Reliability Index: Intraclass Correlation
• Why Session-Level Reliability Is Insufficient for Group-Design Studies
• The Interpretation of IMB SPSS Software Output for ICC
• The Relation Between Interobserver Agreement and ICC
• The Special Case of Fidelity of Treatment Data
• Selection of Interobserver Reliability Index
• Consequences of Low or Unknown Interobserver Reliability
• Conclusions and Recommendations

Section II Advanced Topics
Chapter 9 Introduction to Sequential Analysis

• About the Terminology Used in This Chapter
• Sequential Versus Nonsequential Variable Metrics
• Requirements for Sequential Analysis
• Why Sequential Associations Are Insufficient for Causal Inferences
• Coded Units and Contingency Tables
• Four Types of Sequential Analysis
• Event Lag
• Event Lag With Pauses
• Concurrent Interval
• Interval Lag
• Observational Software for Sequential Analysis
• The Need to Control for Chance
• Indices of Sequential Association
• Transitional Probabilities
• Risk Difference
• Yule’s Q
• Relative Advantages and Disadvantages Across Indices
• Conclusions and Recommendations

Chapter 10 Research Questions Involving Sequential Associations

• Sequential Analysis in Within-Group and Between-Groups Designs
• Testing the Significance of Mean Sequential Associations
• Testing Between-Groups Differences in Mean Sequential Associations
• Testing Within-Group Differences in Mean Sequential Associations
• Testing Summary-Level Associations Between Participant Characteristics and Sequential Associations
• Sequential Analysis in Single-Case Designs
• The Meaning of Contingency in Behavior Analysis
• Why Significance Testing Is Controversial at the Individual Participant Level
• Types of Within-Participant Research Questions and Methods to Address Them
• Descriptive Questions to Inform or Supplement Single-Case Experiments
• Transitional Probability Comparisons and Contingency Space Analysis
• Contingency Indices as Dependent Variables in Single-Case Experimental Designs
• Contingency Indices as Procedural Fidelity Measures in Single-Case Experimental Designs
• Data Sufficiency for Sequential Analysis
• Consequences of Insufficient Data
• Defining Sufficient Data for Estimating Sequential Associations
• Proposed Solutions When Data Are Insufficient
• Conclusions and Recommendations

Chapter 11 Generalizability Theory

• The Scope of This Chapter
• Overview of G Theory and Definition of Terms
• A Sample Observer-by-Context G and D Study
• The Rationale for Preferring the Absolute G Coefficient
• Sample Applications of D Studies
• An Ongoing Controversy
• Conclusions and Recommendations

Section III Putting It All Together
Chapter 12 Summary of Recommendations for Best Practices in Observational Measurement

• Identify Research Questions and Objects of Measurement
• Validate Observational Variables
• Design or Adapt Coding Manuals
• Select Each Component of the Coding Enterprise
• Select Observational Variable Metrics
• Train Observers
• Prevent, Detect, and Address Observer Drift
• Estimate, Report, and Interpret Interobserver Reliability
• Use Sequential Analysis to Address Research Questions Involving Sequential Associations or Contingencies
• Apply Generalizability Theory to Improve Reliability of Observational
• Measures of Generalized Person Characteristics

Glossary
Index

Editorial Reviews

“Yoder, Lloyd and Symons have given us an observational methods opus that is rooted in the real world of human behavior. Their book is thorough, systematic, appropriately nuanced, concise and remarkably interesting. This book is a gift to present and future generations of behavioral scientists who's research is dependent on the observational measurement of human behavior.

Steven F. Warren