How do you develop an evaluation plan?
This is Part 3 of the Program Evaluation Whitepaper series. The previous papers covered what you should consider before starting a program evaluation and how to get started. Be sure to read those papers before diving into writing your plan.
What have you already determined?
For an evaluation, you should have done the following:
- Developed your outcome statements
- Mapped out your outcome progression sequence
- Identified your performance indicators (and targets if you have baseline information already)
- Discussed with your team and leadership which outcomes to initially pursue (Note: We suggest that you start out tracking only a few outcomes and then adding others as you feel more confident in your ability). That is, what are your “need to knows”?
So what now? The next step in the process is to get started writing the evaluation plan.
How do you get started?
- Outcomes – the broader change you want to see
- Indicators – what you will use to measure
- Data collection strategy – how you will measure
- Notes – references to possible survey measures (It’s good to note whether the data collection method will need a considerable amount of time and money for the creation of the assessment tool in addition to the actual collection and analysis of the information. You can include “low,” “medium,” or “high.”)
- Outcome stage – short-, medium-, or long-term outcomes (refer to the logic model)
We propose something that looks like this:
At the end of this document, you will see an example of how all of this is put together.
What about data collection?
Aside from inputting the outcomes and indicators, answer the following questions in order to complete the plan:
- What is the source of the evaluation information?
- Program director, staff, program participants, etc.
- What methods will be used?
- Surveys, interviews, focus groups, observations, etc.
- Note: document review (records, reports, census data, health records, etc.) is closer to activities and outputs for implementation evaluation
- Who will collect the data?
- Program administrators, staff, external consultants, etc.
- When will the data be collected?
- Ongoing, quarterly, annually, etc.
- Who will summarize/analyze the information?
- Program administrators, staff, external consultants, etc.
- What is the annual cost for this evaluation?
- You may not know this upfront, especially if you are new to evaluations. It would be a good idea to track costs for a specified time in order to estimate costs going forward.
If you are new to research and evaluation, you may need to consider adding account details fit for research and development expenses. Add as much detail as possible (even down to geographic location) without it becoming too laborious to track. By creating this detail, you will be able to better understand the breakdown of costs, variances between projects, differences in costs between states or countries (if you are an international nonprofit), etc. Pivot tables can be your best friend. Suggestions for details include:
- Salary and benefits (your staff’s time)
- Purchased materials (cost of paper, licenses, etc.)
- Independent contractors (data collection or analysis experts, etc.)
- Equipment (tablets, phones, etc.)
- Travel (airfare, gas, mileage, rental cars, etc.) – consider organizing this information on separate lines
- Other details relevant to your nonprofit
When considering surveys and quantitative data, always:
- Provide context for the participants. This includes:
- Who you are surveying
- Why you are surveying
- What you will use the information for
- How long the survey should take
- Whether the information collected will be kept confidential or anonymous, as well as who will have access to the data
- How to express your gratitude for their participation
- Consider the time it takes to take the survey; participants will be less inclined to honestly engage in a survey that takes longer than 10 minutes—be sure to time it!
- Include a healthy amount of demographic data. In doing so, you set yourself up for better data analysis. By collecting this basic demographic information, you will be able to determine for which categories of beneficiaries your program, procedures, and policies are working well and for which they are not.
- First Name, Last Name – if anonymous: DDMM (birthday), XXX (first 3 letters of last name)
- Age
- Gender
- Marital Status (if applicable)
- Education Level
- Race/Ethnicity
- Other elements that could be included: Salary Range, Employment Status, Family Size
- Consider how you will measure or chart the growth: yes/no answers do not allow for much outcome change (growth), so it is better to write questions that have scale (Likert) responses
- Ask only what is necessary
- Test the survey beforehand to make sure questions are clear
If you are submitting your outcomes for any type of grant (foundation or federal), get some extra help.
This includes submitting the actual grant proposal as well. The stipulations for data collection and analysis are far more restrictive and demanding. You risk losing your funding or not being considered for the grant if you do not have a sound data collection and analysis plan in place.
Outcomes data rarely give you reasons as to why the outcomes have occurred.
Outcomes data merely identify what works well and what does not. This might be the reason many nonprofits do not engage with measuring outcomes. The question of, “What if what we are doing isn’t really making a difference?” is enough to throw any manager, CEO, or board into panic mode. But rather than being afraid of these results, nonprofit leaders should embrace them as a way of improving and bettering their programs. When you identify what is not working well, you can take a step back, redesign, and try again. This process of redesign should include qualitative research with various stakeholders (beneficiaries, administrators, providers, etc.) to pinpoint the breakdown in service or design. The “A-ha” moment is worth it.
What do you do now?
- Make sure your outcomes and indicators are clear and defined.
- Make sure you have only included “need to know” items, especially if you are new to this process.
- Brainstorm data collection methods for obtaining your data.
- Evaluate the time and energy required for these efforts.
- Put all of the information together into the Evaluation Plan.
- Review.
- Make a timeline. Be sure to include the necessary preparation time for any instrument development required. Timelines can get very complicated depending on how much information you wish to include.
- Review again.
- Start the evaluation process.
Evaluation Plan Outline for Data Collection (example)
Evaluation Plan for Data Collection (blank)
Sources
From “Evaluation Plan Workbook,” Innovation Network, URL: http://www.innonet.org/resources/eval-plan-workbook
From “Building a Common Outcome Framework to Measure Nonprofit Performance,“ Urban Institute, URL: http://www.urban.org/sites/default/files/alfresco/publication-pdfs/411404-Building-a-Common-Outcome-Framework-To-Measure-Nonprofit-Performance.PDF
From “Candidate Outcome Indicators: Youth Mentoring Program,” Urban Institute and the Center for What Works, URL: http://www.urban.org/sites/default/files/youth_mentoring.pdf
