In a recent technical assistance online meeting, I was talking with several community groups about various data collection and analysis related to their program implementation and intended outcomes. I raised the importance of sifting through all the various data in order to extract actionable information useful for the program staff, community stakeholders, and intended beneficiaries of the program. I was reminded of a past blog post from Seth Godin that I shared with them. Quite succinctly, he reminds us about the importance of disseminating evaluation findings that can both be understood and story-like.
When there’s simply data, it’s all noise. It’s impossible for a human being to absorb data without a narrative.
Once we figure out how to turn your features and ideas and benefits and effort into a story, though, it becomes information. And then we can act on it.
We have a story problem. All of us do. We’re not doing a good job of developing the empathy to turn all the data we’ve assembled into a story that others can understand.
In Developing Monitoring and Evaluation Frameworks, Markiewicz and Patrick (2016) laid out the relationship of program development and implementation to five domains of evaluation questions. In doing so, they’ve contributed to the discussion in the previous post about what evaluation is for. At the very least, drilling down the domains of evaluation questions alter the types of program decisions to be considered and answered.
The following table, adapted from their work, summarizes the five domains of evaluation questions aligned to five program components.
Domains of Evaluation Questions
Description of Domains
Planning and design
Assessing the appropriateness of the program’s design
-Suitability of program design in context -Fit of program with program theory and/or logic -Testing of underlying assumptions -Extent program meets the priorities and needs of key stakeholders
Assessing program effectivenessin meeting it’s objectives, its value, and quality
– Fidelity of implementation -Achievement of program objectives -Assessment of the quality and value of the program
Examining efficiencyand fidelity in program implementation
-Conversion of inputs to outputs and outputs to results -Governance and management
Establishing impact: intended and unintended, and the degree to which change is attributable to the program
-Changes (results) produced by the program, intended and unintended, direct and indirect
Sustainability of results
Identifying ongoing sustainablebenefits from the program
In a recent blog post published during the AEA 2019 Conference, Cameron Norman, initiated a list of things that evaluation is for. Recalling this list was helpful during a recent family gathering when I was asked about work. Rather than focus on the means of evaluation, I found myself talking more about the ends of evaluation which coincided with many of the things on his list shared below. Evaluation is for…
Seeing the future
Design and innovation
Asking better questions
Speaking truth to power
Honoring our work
Leading system change
Telling stories about who we are as a people
Promoting health and preventing harm
Reinforcing democratic ideals
Make sure to check out Cameron’s blog post that elaborates on each of these.
Using released items from state and national mathematics assessments can be helpful in designing a grade level assessment instrument for students to be used as one source of data in the evaluation of teacher professional development or student mathematics interventions.
The list below identifies sources of released assessment items that might be aligned with the content of your program. This list will be updated as additional sources are identified.
Based on the book Eva the Evaluator by Roger Miranda, Wendy Tackett’s enactment helps Eva understand all the different roles that an evaluator might take on in an evaluation project. This light-hearted video and the book in which it is based on demystify the complex work of evaluation.
A program is any set of organized activities supported by a set of resources to achieve a specific and intended result (p. 3)
Inherent in this rather broad definition are three important moving parts:
Set of resources (inputs)
Set of organized activities (activities described by outputs)
Specific and intended result (outcomes and impacts)
Each of these provide parameters that define and focus a program and its evaluation. Yet, embedded in the definition are the qualifiers- organized, specific, and intended. Activities must have some sort of underlying organization. Perhaps the activities are organized developmentally, sequenced by difficulty, or arranged in some manner as a result of previous research. Results must be specific and intended. That is, there is a desired end that can be clearly articulated so as to justify the means.
A program exists when the qualifiers (organized, specific, and intended) are understood and there is a logical relationship between resources, activities, and outcomes. This logical arrangement is often described in a program’s logic model. Without understood qualifiers and an articulated logic, a program is not a program, but rather a loose set of activities.
Programs are designed and implemented to create change, often throughout multiple levels of a program; but at the very basic level, change occurs in individuals. Quoting Albert Wenger, “change creates information.” Information about changes that occurred during a program is essential to evaluate a program.
According to Radhakrishna and Relado (2009), a program might influence change in a program participant’s knowledge, attitude, skills, aspirations (KASA), and behavior. Another common set of individual outcomes called KAP include knowledge, attitudes, and practices (for example, see Chaplowe (2008)). At their intersection, changes in:
what participants understand (knowledge),
what participants use to base their decisions (attitudes/beliefs), and
what participants eventually enact (practices/behaviors)
lead to useful information about the potential impact of a program.
Interviewing multiple individuals for a project is time consuming and scheduling those interviews also can be time consuming. There are several online applications that can match available interview times between interviewers and interviewees; however, I created an easy, quick, and no-cost system to schedule interview times using a Google Doc as described below.
I created a Google Doc that listed available day/times I was available for interviews. This Google Doc was shared with all interviewees at the same time using a shared link giving them permission to view but not edit the online document.
I easily and quickly edited the online document with days/times that became available or unavailable as my time commitments changed. At some level, using a Google Doc in this manner was a quick way to develop an instant website describing my availability. I used an email similar to the one below to share this process with those I was planning to interview.
I’d like to schedule an hour phone conversation to talk with you about [name of program].
This online document suggests several day/time possibilities. Please reply to this email with your preferred choices. If none will work, please suggest a couple others. I’ll confirm in a return email.
I’m really looking forward to talking with you about [name of program].
Evaluations vary as much as programs do (i.e., different activities, duration, outcomes, et cetera) underscoring the importance of wisely choosing the focus of an evaluation. Step 4 in the guide “Choosing the focus of your evaluation” describes a five tier approach that is summarized below (pages 13-16). Determining an appropriate evaluation focus is largely dependent on a program’s maturity and developmental stage.
Tier 1- Conduct a needs assessment to address how the program can best meet needs
Tier 2- Document program services to understand how program services are being implemented
Tier 3- Clarify the program to see if the program is being implemented as intended
Tier 4- Make program modifications to improve the program
Tier 5- Assess program impact to demonstrate program effectiveness
As you can see, evaluation can and should coincide with a program throughout its lifespan. These tiers are useful to help design an evaluation plan and to determine appropriate methods of data collection and analysis.
Out of school programming often seeks to develop children’s’ outcomes beyond those purely academic. Working with program leaders to align program activities/goals with specific outcomes can be a challenge. In many cases, program activities are determined prior determining program outcomes. In these instances, understanding components of positive youth development can be useful to reflect on the relationship and alignment of program activities and program outcomes.
First, the Oregon State University 4-H Youth Development Program developed a Positive Youth Development Inventory (PYDI) intended to assess PYD changes in students ages 12-18. The collection of 55 Likert scale items measures the latent constructs of: