Evaluation Essentials: From A to Z
346
Evaluation Essentials: From A to Z
346Hardcover(Third Edition)
-
SHIP THIS ITEMAvailable for Pre-Order. This item will be released on October 10, 2024PICK UP IN STORE
Store Pickup available after publication date.
Available within 2 business hours
Related collections and offers
Overview
Product Details
| ISBN-13: | 9781462555444 |
|---|---|
| Publisher: | Guilford Publications, Inc. |
| Publication date: | 10/10/2024 |
| Edition description: | Third Edition |
| Pages: | 346 |
| Product dimensions: | 6.00(w) x 9.00(h) x 0.00(d) |
About the Author
Read an Excerpt
CHAPTER 1
SECTION A
What Is Evaluation?
Evaluation is taking place everywhere around us. You most certainly have engaged in evaluation within the past day. But what is evaluation? The popular definition of evaluation, according to the dictionary, is "to ascertain the value or amount of." Indeed, you do this all the time. When you go to the store to make purchases, you determine the value of things. You might ask, "Is it worth it?" You look at the cost of an item and determine whether, in fact, its value to you exceeds the cost.
Perhaps the most common kind of evaluation that you might engage in is product evaluation. If you are looking to buy a new flat-screen television, you examine several different products to gather information about their technical specifications, size, attractiveness, and the cost. You make a valuation judgment. Sometimes these judgments are done at an instinctive level. You might just look at competing products and make a decision, all the while having processed data in your head, perhaps unknowingly, about what you believe to be the differences among the products.
Sometimes we might be more systematic in our evaluations. I recall that when my wife and I bought our first house, we listed the attributes that we thought were essential. Some items we considered to be necessary and other items were viewed as optional, but preferred. All of these attributes were listed on a piece of paper and we developed columns for each of the three competing houses, and performed ratings with respect to each of the characteristics. Then, the "evaluation model" became somewhat more sophisticated. We indicated those dimensions that needed to be present in order to be considered (e.g., three bedrooms). This was a "necessary, but not sufficient" list. We then further differentiated among the houses by addressing additional ways of appraising the data. Values or weightings were attached to each of the dimensions. We needed to decide which ones, for example, were more important and then provided a weight for each. The question was asked: What are the weightings for each — the relative importance? Was having an additional bathroom more important than whether the house was landscaped well? How much more important? If landscaping was weighted at "1," would an extra bathroom be a "2" or a "3"? Thus in a way we were doing an evaluation based on a number of criteria weighted differentially based on our view of their relative importance.
A house is a product — evaluating products is one kind of evaluation. You might also evaluate people — a personnel evaluation. You could make judgments about whether you would like to develop a friendship with an individual or whether a particular painter or electrician seems trustworthy and dependable. If you are in a position where you supervise personnel who work for you, you are engaged in evaluation, or you might need to make a decision about which of several applicants for a position should be hired. Personnel evaluations, again, require an appraisal, or an evaluation, including making judgments about relative value. Sometimes, these kinds of decisions are made based on impressions — just instinct. Other times, those making these decisions are more systematic in performing these evaluations.
A third kind of evaluation is policy evaluation. Policies are general directions for action without necessarily having a particular program or plan in mind. For example, at the everyday level of evaluation, one might be evaluating a potential policy decision of whether to go on a diet. Because no specific diet plan is necessarily in mind, it is a policy being evaluated — not a program. This policy evaluation might consider things such as what are the potential benefits from commencing this policy — this course of action? In doing this, you might consider what you know about the relationship between being overweight and good health. You might ask, "Is following this course of action compatible with my lifestyle, and if not, is that acceptable? And, what are the costs to me — either in dollars or in terms of modifications — that I would need to make in my lifestyle if I were to pursue that course of action or policy?"
Another kind of evaluation is program evaluation. Before discussing program evaluation, it is important that I add a brief side note. In program evaluation, evaluators can gather data about personnel (teachers, caseworkers, students, clients, etc.), but the focus is not to make judgments about these individuals. Products might also be a part of the program that is being evaluated. Data might also be gathered about products, but the primary purpose is not evaluating the products. Rather, evaluators are interested in using this information collectively to better understand the program in which participants are involved.
Now let us consider the nature of program evaluation. Suppose that you wish to enroll your child in a preschool program and need to make a choice about which one to select. Let me make the example simpler by assuming that you have become convinced of the benefits of the Montessori preschool approach, but there are three schools within easy driving distance that all claim to be "Montessori." In doing this evaluation, you might visit the three schools and observe in the classrooms, but what do you look for? One approach is to focus on whether the scholars are truly adhering to the procedure established as being appropriate for and typical of Montessori schools — that is, are they truly implementing a Montessori approach? You might look at the activities in which children are engaged and the ratio of adults to children. You might look at the number and type of manipulatives available. All of these are relevant things to be examined, but if you wish to be systematic, you should select the kinds of things that are typically a part of a Montessori program — those that follow the Montessori philosophy. After you have compiled a list of elements or activities, you must consider the possible ways to see whether those things are actually taking place. In other words, you want to evaluate whether a Montessori approach is truly being implemented — whether the program is really operating in a Montessori way. Thus you would want to examine: Does multiaged grouping take place? Are there work centers? Are areas of study interlinked? Do children have a 3-hour work period available? Are the teachers Montessori trained?
You might also want to do an evaluation not focusing simply on the extent to which appropriate implementation had occurred, but on outcomes. The focus of an outcome evaluation is to obtain a good picture of the success of the program. Was the program successful? Is there a basis for a judgment that the program is worthwhile? Let us consider the kind of criteria you could use for making such a judgment. Perhaps there were expectations that students would have developed certain prereading skills. Possibly you were concerned about whether students would be better able to engage in socialization with their peers. Certainly, you anticipated that students would be better able to maintain appropriate behavior. While these and other outcome measures might be initially identified, the evaluation of the program should take heed of other program accomplishments that you might not have thought about: Are the children happy? Have they increased in maturity? Do they have a love of learning?
To summarize, I have talked about evaluating products, personnel (or individuals), policy, and programs. In this book, I focus on program evaluations.
PROFESSIONAL PROGRAM EVALUATION
Now let me separate the examples given above, which are everyday evaluations, from what I call professional evaluation. As you have seen, there is great variation in the way that everyday evaluation takes place. These informal, nonprofessional evaluations range from somewhat systematic (perhaps even — or almost — "professional") to almost instinctual. For example, the listing of criteria and weighting them for relative importance as in the evaluation of various houses discussed above was relatively systematic. At the other extreme of everyday evaluations are those that are almost instinctual — a decision based on "I just had a gut feeling."
To be "professional," evaluation must be conducted in a systematic way. In essence, it is an inquiry involving the gathering and assessment of information in a planned and methodical way. Some authors use the term "disciplined" to describe activities such as professional evaluation and other forms of inquiry that are conducted in a systematic way. In this sense, disciplined inquiry refers to engaging in a procedure that is objective and one in which others are able to easily discern the steps that were taken. For these reasons, disciplined inquiry leads to findings or conclusions that have credibility. The manner in which the study was conducted must be so complete that the recipient of the evaluation has little doubt that the results are meaningful. Disciplined inquiries must set in place procedures to carefully control potential errors in reasoning, and to ensure systematic data collection and analysis of data. Credibility is established by paying heed to these potential sources of error and eliminating them, or at minimum, exploring what they are and how they could influence the findings.
EVALUATION AND RESEARCH
Both professional evaluation and "research" are forms of disciplined inquiry. How do they differ? Sometimes the two are virtually indistinguishable. This is particularly true when considering evaluations performed by those who consider evaluation basically as a kind of applied research, but many other evaluators (me included) view them to be quite different.
The main distinguishing characteristic between research and evaluation is that the former seeks conclusions and the latter leads to decisions. Research seeks to add to the body of knowledge (typically of or pertaining to a particular academic discipline). Implicit in the concept of "knowledge" is that it is applicable across settings, across geography, and across time. By this I mean that the findings seek to be applicable to like programs anywhere, and be as valid in a year (or 2 or 3) as they are now. Research seeks to gain generalizable knowledge. Evaluations (as I wish to describe them) address the here and now (this program, at this time) and attempt to provide insights that might lead to program improvement decisions. Evaluations recognize that there could be differences among programs that might even have the same name — they could differ in goals or emphases or in their context — including the people involved and the particular situation. Evaluators respect these differences in context and seek to gain understandings only about the specific program being studied.
Another important distinction between research and evaluation is "who asks the questions." Evaluation seeks to answer questions posed by, and of importance to, a client, program, or community. Generally, researchers define the question they seek to answer and seek conclusions that add to understandings about the knowledge base.
Let me explore with you the way that disciplined inquiry is applied to an evaluation situation. In the example given earlier in this section, I discussed the evaluation of Montessori schools. In that situation, the data collected would need to be justified as relevant indicators of the characteristics of the program by carefully studying the Montessori philosophy and other writings to discern the program elements that must be present to categorize a program as Montessori. The procedures for amassing data would need to be considered nonarbitrary — that is, they must be well-defined. What precisely does a particular program characteristic look like? You will need to consider: How will I unambiguously know when I see it? — that is, how would I know that multiage grouping is taking place? In essence, what characteristics should be present?
Furthermore, the person(s) gathering the data should be considered free of initial bias (or at least those biases should be specified as part of the evaluation). A legitimate professional evaluator should not enter the process with a predisposition to saying one or another program is best. Also, the way in which data are analyzed should be reasonable, easy to follow, and free of error. It should be patently clear how pieces of information (data) were analyzed (meaning added together, or in some other way compared), or otherwise refined into more meaningful descriptions of results. Finally, the findings should be justified solely by the data. Evaluators should not take a broad leap to conclusions beyond the specific data of the study.
EVALUATION DEFINITION
For those in need of a formal definition, let me provide one. I will be brief. Formal definitions, detailed descriptions, and ponderous writing are not in keeping with the focus of this volume. Rather, I prefer to explain by offering examples and by raising rhetorical questions that lead the reader (you) to think about evaluation.
So here we go. Most simply put, evaluators state that evaluation is considering the merit and worth of an entity. This, in fact, is a statement of the goal of evaluation. The goal is to consider value in a systematic way. This valuing consists of two aspects. As you have seen, a part of it is the determination of the merit — which is the intrinsic value of the entity being studied. The dictionary describes merit as intrinsic rightness or goodness "apart from formalities, emotional considerations, etc." Intrinsic goodness? What does intrinsic mean when I am talking about a program? If a program does well — that is, does what it is supposed to do — it has merit. But is it sufficiently meritorious to satisfy the needs of a particular context? Think of my house-buying example. If a house has a large ultramodern bathroom, then as a bathroom it might be considered meritorious but not have greater worth to me as a house buyer, because "ultramodern bathrooms" are not something I value. A flat-screen television could have all kinds of fancy attachments. That television might have merit but if I do not value the attachments, it does not have "worth." Thus one must consider the extrinsic aspects of what is being evaluated. While the program may be meritorious, we ask, what is its worth within our context? Is the high merit exhibited valuable within the particular program's context? We seek to value or evaluate through both merit and worth considerations.
The above provides a definition of evaluation based on its goal. Note, however, that I have stated that evaluation, along with research, is a disciplined inquiry. Thus we need to consider the process for reaching the stage of being able to judge merit and worth. This process requires systematic, unbiased context-sensitive behavior. In a sense, then, the various sections that I present in this volume constitute my process definition of what I call context-sensitive evaluation.
A CONFUSION OF TERMS
Now let me deal with some of the confusing terms associated with evaluation. "Assessment" is a term that is often used synonymously with "evaluation," but it is different. Another term that we often hear is "appraisal." A brief clarification is in order. This is my interpretation. Each of these three terms involve valuing (judging merit or worth). Evaluation is the favored term when we talk of judging a program. Assessment is employed when one refers to the clients of a program. This is particularly true in the education field where we are constantly confronted with things like state assessment tests and national assessment of education. In each of these cases, we are assessing students. Appraisal, I believe, is more relevant when we talk about program staff. Think of teacher appraisal, for example. Summary: We evaluate programs; we assess client knowledge; and we appraise staff.
Another related term is "testing." I view testing as different in nature from the above. Testing is the process used for giving tests. Tests are instruments used for gathering data. They do not, in and of themselves, include a "considering," or a valuing component. They may subsequently be given value and enable judgments to be made. Thus I consider testing as a means of gathering the data for assessing, appraising, or evaluating.
Enough said.
EVALUATION PURPOSES
Another issue: Evaluation writers tend to make the distinction between what they call "formative" evaluation and "summative" evaluation. Formative evaluation generally takes place during the early stages of program implementation. Formative evaluation is conducted in order to provide information for program improvement, which generally means that the evaluation information would provide an indication of how things are going. The evaluation information, for example, would highlight problems related to whether program activities were being conducted — and being conducted in a proper manner. Formative evaluation might also provide some early indication about whether it is likely that program outcomes — the goals of the program — are potentially achievable. Did some early testing of clients show that they were not making a sufficient level of intended progress? Formative evaluation is generally conducted primarily to benefit in-house staff — that is, it is information for those who are conducting the program so that they can make improvements. Such improvements might refer to modifications to ensure that the original program plan is complied with or could suggest changes in the program as conceived. However, it might take place over extended periods of time; in such instances, this continuous formative evaluation is focused on the ongoing development of a program or innovation in more complex settings. Some theorists refer to this as "developmental evaluation."
(Continues…)
Excerpted from "Evaluation Essentials From A to Z"
by .
Copyright © 2018 The Guilford Press.
Excerpted by permission of The Guilford Press.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
Table of Contents
Overview Session A. What Is Evaluation? Session B. Why Do Evaluations? Session C. Who Does Evaluations? Session D. Who Are the Users of an Evaluation? Session E. How Do You Strengthen Relationships with Users? Session F. How Do You Describe the Program? Session G. How Do You “Understand” the Program? Session H. What Are the Questions and Issues to Be Addressed? Session I. What Are the Sociopolitical and Community Contexts of the Program? Session J. What Is the Organizational Context of the Program? Session K. What Are Instruments for Collecting Quantitative Data? Session L. What Are Instruments for Collecting Qualitative Data? Session M. How Do Data Collection Issues Impact Potential Evaluability? Session N. Are the Questions Evaluable? Session O. How Do We Plan for Process-Focused Evaluation? Session P. How Do We Plan for Outcome-Focused Evaluation? Session Q. How Do We Manage the Evaluation? Session R. How Are Quantitative Data Analyzed? Session S. How Are Qualitative Data Analyzed? Session T. How Are Analyzed Data Used to Answer Questions? Session U. How Are Evaluation Results Reported? Session V. What Is the Evaluator’s Role in Evaluation Use? Session W. What Are the Evaluation Standards and Codes of Behavior? Session X. Contracting for Evaluations Session Y. How Are Costs Analyzed? Session Z. Context Sensitive Evaluation: Attaining Evaluation Use Appendix A. Use Factors: Relationship to Research Compilations Appendix B. How Can You Embark on a Program to Learn More about Evaluation? Appendix C. An Evaluation Lesson, by “Unknown Student” Index About the AuthorsInterviews
Graduate students and instructors in education, psychology, social work, nursing, management, and public policy; applied researchers who need a refresher on conducting evaluations; educational administrators and program administrators who use evaluations in their work. Serves as a text for graduate-level evaluation courses or as a core book in applied research methods courses.