Collaborative Evaluations: Step-by-Step, Second Edition is a comprehensive guide for evaluators who aim to master collaborative practice. Liliana Rodríguez-Campos and Rigoberto Rincones-Gómez present their Model for Collaborative Evaluations (MCE) with its six major components: identify the situation, clarify the expectations, establish a collective commitment, ensure open communication, encourage effective practices, and follow specific guidelines.
Fully updated to reflect the state-of-the-art in the field, each core chapter addresses one component of the model, providing step-by-step guidance, as well as helpful tips for successful application. To further demonstrate the utility of the MCE, this new edition includes recurring vignettes about several evaluators and clients, illustrating frequent questions and specific challenges that arise when evaluators take a collaborative approach. Drawing on a wide range of collaborative evaluations conducted in the business, nonprofit, and education sectors, this precise and easy-to-understand guide is ideal for students and practitioners who want to use its tools immediately.
|Publisher:||Stanford University Press|
|Product dimensions:||6.10(w) x 9.20(h) x 0.80(d)|
About the Author
Liliana Rodríguez-Campos is Associate Professor and Director of the Center for Research, Evaluation, Assessment, and Measurement at the University of South Florida.
Rigoberto Rincones-Gómez is Associate Vice President for Institutional Research, Planning, and Effectiveness at Broward College.
Read an Excerpt
Collaborative Evaluations Step-by-Step
By Liliana Rodríguez-Campos Rigoberto Rincones-Gómez
Stanford University PressCopyright © 2013 Liliana Rodríguez-Campos and Rigoberto Rincones-Gómez
All right reserved.
Chapter OneIdentify the Situation
The situation is a combination of formal and informal circumstances determined by the relationships that surround and sustain the collaborative evaluation. This component is very important because the success of the evaluation will depend, to a large degree, on how well the situation is understood. Although you and the collaboration members (CMs) may be able to influence the environment in some way, keep in mind that you do not have total control over it. Understanding the situation is the foundation for all the evaluation decisions and prepares you to anticipate opportunities and threats throughout the collaborative process.
The way you perceive the situation will determine the approach you will take to achieve the desired results. Those evaluation results will be based on specific situational characteristics, such as resources (for instance, people or time) and a particular setting (for example, physical, social, political, economic, geographical, or demographic). Rather than simply accepting the perception that the client has of the situation, you should gain a broad sense of the situation on your own. In addition, instead of just stopping with what the client wants, pursue what the client needs, because the difference between "want" and "need" is your value added (Weiss, 2009).
An early indication of the situation with regard to the potential constraints and benefits needed to support the collaborative evaluation (such as funds, staff, materials, scope, and context) helps you better manage it and be prepared to overcome barriers. It also helps to determine strategies to deal with any concerns raised and to develop a beneficial evaluation plan. To gain a better understanding about the situation surrounding the evaluation you may conduct observations of the evaluand and stakeholders in their day-to-day activities. Always have multiple inside and outside sources for gathering helpful information, in order to understand the evaluand culture (for example, interview key stakeholders or review documents and reports).
The situation sets the foundation for everything that follows in the collaborative evaluation. Among other resources, the time needed to identify the situation varies depending on the complexity of each evaluation, which also leads to the amount of effort invested in it (for example, the number of meetings needed). Because people are going to collaborate within the evaluation, it is also necessary to determine what level of collaboration will be required depending on the situation. The better you understand the situation and the more fully you analyze it, the easier your collaborative evaluation will be (and the application of the remaining components of the Model for Collaborative Evaluations [MCE]).
This chapter provides you with step-by-step suggestions on how to identify information in order to understand the current nature of the evaluand, its surroundings, and the scope for the collaborative evaluation. In other words, we show you how to carefully identify the situation. In addition, we present information on how to prioritize the activities toward which you need to focus your resources in order to accomplish clear evaluation results. Thus the MCE component presented in this chapter is divided into the following subcomponents: (a) identify stakeholders, (b) identify logic model elements, (c) identify potential SWOTs, (d) identify the evaluation scope, and (e) identify critical evaluation activities (see Figure I.2).
A stakeholder is a person who is interested in the collaborative evaluation because he or she may be directly or indirectly affected by its results (see Figure 1.1). Identify early the individuals who have at least some interest or stake in the evaluation and how their roles can help to accomplish the overall goal. The audience is a person, or group of persons, who receives the evaluation results (for example, a written or an oral evaluation report).
There are five types of stakeholders who are associated with almost any evaluand: (a) policymakers, such as governing board members; (b) administrators or managers, those who direct and administer the evaluand; (c) practitioners, those who operate the evaluand; (d) primary consumers, those who use the evaluation results, such as evaluation clients; and (e) secondary consumers, those who are affected by what happens to primary consumers, such as community groups (Fitzpatrick, Sanders, & Worthen, 2011). After you have a comprehensive knowledge of the stakeholders, then you are ready to identify the CMs.
Identifying the stakeholders is critical for the collaborative evaluation because their specific needs must be understood in order for them to be addressed in a useful way. Therefore, make a favorable first impression to develop strong relationships from the very beginning. From your first meeting, make direct eye contact, display excellent manners, and use appropriate body language in order to develop effective rapport.
Your goal is to understand who the stakeholders are, what they want and need, and how the evaluand is being affected by them. As a result, you can identify and select specific stakeholders to become CMs who will be involved in depth in the collaborative effort (see Figure 1.1). Of course, always consider the views of the broader range of stakeholders who also collaborate in a less active way.
Identify General Stakeholders
As we mentioned in the Introduction, the MCE is a systematic approach with feedback mechanisms that allow you to come back to this subcomponent and increase (or decrease) the number of identified stakeholders as needed. You may have to seek new CMs, in response to changing needs (for example, a negative climate due to an inappropriate cross-section of members). We suggest the following steps to identify general stakeholders:
1. Be clear that your responsibility is to meet, or even better, exceed your client's expectations while ensuring that particular interests of all relevant stakeholders are considered throughout a balanced collaborative evaluation.
2. Learn as much as you can about your evaluation client, and later about other potential stakeholders, in order to establish a beneficial relationship. For example, read from the evaluand's or client's website and other sources of information.
3. Think carefully and write down all the questions you need the client, and later other potential stakeholders, to answer (for instance, who needs to be informed, who is to receive the evaluation results, and whose collaboration is crucial).
4. Seek clarification (with a courteous attitude) to your questions when meeting with the client, and later with other potential stakeholders, as feasible. For example, if doubts arise after a meeting, then seek to clarify all those doubts.
5. Listen rather than just making statements or educating about the evaluation process (avoid interrupting). For example, wait silently and patiently during pauses, because generally the most important information follows them.
6. Identify diverse points of view (such as the strongest supporters and opponents) in order to diffuse any resistance toward the collaborative evaluation and ensure that everyone's contributions will be adequately considered.
7. Make a preliminary list with the client's assistance (and others as applicable) of the key stakeholders affected by the evaluation. For example, distinguish who would gain or lose the most from the evaluation under different scenarios.
8. Invite your client to review and add or delete names from the preliminary stakeholder list. By doing this, you can have a better idea of who the key stakeholders really are and their possible contributions to the evaluation as future CMs.
9. Gather feedback on a regular basis using a previously agreed-upon system (including meetings, emails, and surveys as feasible). This will help in monitoring the stakeholders' identification process and making updates as needed.
10. Provide the client and other stakeholders as appropriate with a summary report of this experience (for example, an updated list of the stakeholders) as a baseline for identifying the situation of the collaborative evaluation.
Identify Collaboration Members
Now that you have a list of the stakeholders, identify the CMs who will work jointly with you to help with particular evaluation tasks in order to achieve the evaluation vision (see Chapter 3 on how to establish a shared evaluation vision). The CMs should want to be members, be able to fulfill specific evaluation requirements, and work well together. Also, the client must agree with the selected CMs and allow, as applicable, time from their regular duties for them to participate.
This step is critical for the collaborative evaluation process because the CMs can help ensure their own (and other people's) involvement and consequent use of the evaluation results. In other words, the use of the evaluation results will be improved by involving stakeholders in the evaluation process. Specifically, their buy-in proves to be very helpful when evaluation recommendations are being implemented. We suggest the following steps to identify the CMs:
1. Make a preliminary list of essential characteristics that are desirable in the CMs. With this goal in mind, you will be able to later match specific stakeholders to this list while being prepared to be flexible in the selection process.
2. Agree with your client, and other key stakeholders as feasible, which stakeholders from the list can have the most active involvement in the collaborative evaluation, becoming potential CMs (including the client or not, as appropriate).
3. Create, with the potential CMs' help, a calendar that shows their availability during the different evaluation stages and how they see themselves fitting into the collaboration. For example, determine who will be available and when.
4. Learn about each potential CM in terms of their individual characteristics (such as strengths and weaknesses) before making a decision on which individuals you are going to officially invite to become CMs of the evaluation.
5. Identify which of the potential CMs may require special training to enhance their evaluation skills. Sometimes you may not find enough qualified CMs, so they will need appropriate training to build specific evaluation skills.
6. Agree on the final CMs with the client, once you have met with all the key stakeholders, as feasible. Be sure that major stakeholders are represented (that is, specific characteristics) across the main areas affected by the evaluation.
7. Ensure that you and the client are making a fair decision on the CMs' selection (for example, that you have all the relevant facts to make a decision) and on how to match their skills and abilities to the expectations of the collaborative evaluation.
8. Determine if the selected CMs agree to formally collaborate as such in the evaluation, and get started with those who are willing to do it. In other words, find who is genuinely interested to collaborate in the evaluation efforts.
9. Consider the morale of the non-selected stakeholders, who may feel disappointed with the selection. For example, ensure open communication with them, because they can provide important feedback for the collaborative evaluation.
10. Gather feedback on a regular basis using a previously agreed-upon system, such as meetings, and summarize it in a written format (including an updated list of the CMs) so it is available to each CM and other stakeholders as appropriate.
Carefully select CMs with varied characteristics to facilitate involvement within the collaborative evaluation and achievement of its vision. Through the years, we have noticed that differences in education, authority, or other personal characteristics can result in lack of involvement by those who are at the end of the scale (typically the lower end) in regard to those characteristics. Hence, keep in mind that the same evaluation may have a different set of CMs for each of its main needs.
For large, more complex evaluations you may need to create several groups of CMs, so that greater diversity of expertise can be represented. Avoid creating groups larger than six members, because then people tend to feel less individual responsibility for the evaluation results. Be aware that sometimes there may be some poor fits among the CMs selected even though you have been very careful with the identification process. Learn from this experience and continuously monitor the CMs' commitment, performance, and satisfaction by obtaining accurate information (see Chapter 4 on how to ensure immediate feedback).
IDENTIFY LOGIC MODEL ELEMENTS
A logic model is a tool that visually shows, through a graphic illustration or picture, how a particular initiative (such as an evaluand or evaluation) is intended or perceived to occur through logical relationships. It is the logic model elements and their connections (how the elements are linked to each other) that provide a complete picture of the initiative.
A logic model can be used to identify the basic theoretical framework of an initiative and to clarify what should be measured and when. For example, how is the evaluand supposed to work? Is the evaluand working as depicted in the logic model? A logic model is like a roadmap that clearly shows the connection of interdependent elements in order to achieve specific outcomes (Rincones-Gómez, 2009). In general, a logic model consists of the following elements: input, activity, output, and outcome (see Figure 1.2).
Input. This includes all the resources invested and used by an initiative in orYder to achieve its activities, outputs, and outcomes. In general, resources are the facilities, equipment, materials, staff, information, and money. For example, inputs are items provided internally (such as volunteers) and by an external source (such as grant funding).
Activity. This is what an initiative offers or does with the inputs in order to lead to specific outputs. For example, activities may include delivering services and developing products. At this stage, you and the CMs do not know yet if the activity has made a difference to the specific stakeholders, but the activity needs to be executed in order for outputs and outcomes to occur as intended.
Output. This is the direct result of the activities, and it is related to the number of products or services delivered by an initiative in measurable terms. For example, output may be the number of computers sold or the number of participants who attended a particular professional development workshop. Thus the outputs refer to what has been done by an initiative. Outputs lead to outcomes in which you and the CMs will know whether the output has made a difference to the specific stakeholders.
Outcome. This is the effect or change that an initiative makes on stakeholders (individuals, groups, or communities) as a consequence of its outputs. In particular, you and the CMs want to know what intended or unintended difference has been made for stakeholders. Outcomes can be short term, medium term or long term (also called impact or final value). For example, outcomes can be specific changes on stakeholders' attitudes, skills, knowledge, awareness, status, motivation, and behavior.
Next time you (or the CMs) need to diagram a simple logic model you may follow the example provided in Figure 1.2. However, when you need to diagram a more complex logic model, the presence of many arrows can become extremely confusing. For this reason, we created the logic model structure shown in Figure 1.3, which includes a label or identifier for each box to show the relationship. In this type of logic model representation, you write at the left-hand side of each box the label belonging to the previous connection that is helping generate the box you are currently considering. For example, I.1 represents the input 1 or "Instructors," A.1 represents the activity 1 or "Facilitate training class," and O.1 represents the output 1 or "Number of participants graduated."
Excerpted from Collaborative Evaluations Step-by-Step by Liliana Rodríguez-Campos Rigoberto Rincones-Gómez Copyright © 2013 by Liliana Rodríguez-Campos and Rigoberto Rincones-Gómez. Excerpted by permission of Stanford University Press. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
Table of Contents
ContentsForeword by James R. Sanders....................xi
1 Identify the Situation....................9
Identify Logic Model Elements....................15
Identify Potential SWOTs....................21
Identify the Evaluation Scope....................28
Identify Critical Evaluation Activities....................38
2 Clarify the Expectations....................50
Clarify the Role of the Evaluator....................51
Clarify the Role of the CMs....................57
Clarify the Evaluand Criteria and Standards....................64
Clarify the Evaluation Process....................68
Clarify the Evaluation Budget....................75
3 Establish a Collective Commitment....................82
Establish a Shared Evaluation Vision....................83
Establish Recommendations for Positive Actions....................88
Establish Means Toward Conflict Resolution....................93
Establish Decision-Making Procedures....................98
Establish Reward Options....................104
4 Ensure Open Communication....................110
Ensure Active Participation....................111
Ensure Careful Listening....................114
Ensure the Message Is Clear....................118
Ensure Immediate Feedback....................122
Ensure the Need for Change Is Justified....................127
5 Encourage Effective Practices....................134
Encourage Appreciation for Individual Differences....................134
Encourage Fairness and Sincerity....................148
Encourage Teaching by Example....................156
Encourage Flexibility and Creativity....................161
6 Follow Specific Guidelines....................168
Follow Guiding Principles for Evaluators....................168
Follow Evaluation Standards....................173
Follow the Collaboration Guiding Principles....................178
Model for Collaborative Evaluations Checklist....................189
About the Authors....................247