Program Evaluation

INTRODUCTION

What difference does your program make in the lives of youth? This section will introduce the steps involved in designing and executing an evaluation that can measure some of the positive changes you make in the lives of your participants. There are resources and examples in this section for both longer-term, more in-depth programs and short-term or one-time experience programs.

You might have different purposes in mind for your evaluation, including helping determine if you are achieving your mission, clarifying how you impact youth, facilitating continuous program improvement, providing accountability and assisting with communication and fundraising efforts.

An effective evaluation is based on a solid understanding of the connection between what you DO (your program activities) and what you CAUSE TO HAPPEN (your program outcomes). A theory of change that helps clarify that connection is an important foundation step to a successful evaluation.

EvaluationSAMPLE Evaluation Report

This is an excerpt from a program’s annual report showing their evaluation data summary.

Impact EvaluationSAMPLE Program Evaluation Overview

This document gives an overview of an organization’s evaluation process.

WHAT CHANGES AS A RESULT OF YOUR PROGRAM?

Your program outcomes are a description of the changes you hope occur as a result of youth’s experiences with your program. Indicators are things you can see, hear or read to help you know that your program is contributing to transforming youths lives. Examples of program outcomes include increasing youths’ ability to set and achieve goals, increasing youths’ ability to work with others and increasing youths’ environmental stewardship.

Outcomes and IndicatorsGUIDE Outcomes and Indicators for Long and Short Term Programs

This document gives detailed examples of outcomes and indicators for both long and short term programs.

WHAT DATA WILL YOU COLLECT?

You will want to collect data on each of your indicators. Possible data sources include: information from kids themselves (through interviews, answers to questions on forms, etc.); systematic observations by staff; and third-party data (e.g., school records).

Guide Sources of DataGUIDE Sources of Data (Methodology)

This document shows the different sources of data that you might consider using and gives some background on the different choices.

HOW WILL YOU COLLECT DATA?

Make plans for when and how you collect data. Depending on your outcomes and indicators, you might need to collect data before kids start the program, during the program, at the end of the program and/or months or years after kids complete the program. Your data collection tools can include surveys, registration forms, observation forms and the instructions about how to use them.

Data Collection ToolsGUIDE / TEMPLATES Data Collection Tools

This document explains the key indicators collected through the tools created by TYO, and explains how each tool should be administered with students.

DATA MANAGEMENT AND CALCULATIONS

You need to organize your data in a way that makes data analysis possible. One easy way to do that is to put your data into an Excel spreadsheet. Always use one row per participant and be sure to enter your data in the same way for each person. You can also use Excel to calculate the statistics that help you summarize your data. (e.g., averages, % change, etc.).

Data ManagementSAMPLE Explanation of Data Management, Coding and Input

This document describes how to analyze survey questions for a longer-term program model and what data to enter into a spreadsheet making analysis possible.

CodebookSAMPLE Codebook and Analysis Guide

This document is a guide that goes along with a data management system (e.g. Excel spreadsheet) for a longer-term program model.

MAKING SENSE OF YOUR DATA

It is important to take time to make sense of your data. Is the data what you expect? Or are any surprises? Note if there are differences from year to year or from school to school. If you can, share an initial analysis with stakeholders like program graduates and staff. This is also a good time for staff and stakeholders to share and record stories that help paint a picture of the story that the data tells.

Making SenseGUIDE Making Sense of Your Program Evaluation Data

This document lists several potential evaluations challenges with suggestions for how to address them.

REPORTS AND OTHER WAYS TO SHARE YOUR DATA

Tailor your reports and communications about your data to your purpose for evaluation. One kind of report is a good fit to help funders understand your program. A different kind of report may support program improvement. You and your participants have done all this work to collect data, be sure to use it!

Program EvaluationSAMPLE Evaluation Report

This is an excerpt from a program’s annual report showing their evaluation data summary.