Programming and Domains of Evaluation Questions

In Developing Monitoring and Evaluation Frameworks, Markiewicz and Patrick (2016) laid out the relationship of program development and implementation to five domains of evaluation questions. In doing so, they’ve contributed to the discussion in the previous post about what evaluation is for. At the very least, drilling down the domains of evaluation questions alter the types of program decisions to be considered and answered.

The following table, adapted from their work, summarizes the five domains of evaluation questions aligned to five program components.

Program Component Domains of Evaluation Questions Description of Domains
Planning and designAssessing the appropriateness of the program’s design-Suitability of program design in context
-Fit of program with program theory and/or logic
-Testing of underlying assumptions
-Extent program meets the priorities and needs of key stakeholders
Objectives Assessing program effectiveness in meeting it’s objectives, its value, and quality – Fidelity of implementation
-Achievement of program objectives
-Assessment of the quality and value of the program
ImplementationExamining efficiency and fidelity in program implementation-Conversion of inputs to outputs and outputs to results
-Governance and management
ResultsEstablishing impact: intended and unintended, and the degree to which change is attributable to the program-Changes (results) produced by the program, intended and unintended, direct and indirect
Sustainability of resultsIdentifying ongoing sustainable benefits from the program-Continuation of program benefits

Defining a Program

What makes a program a program?

A recent webinar led by Boris Volkov hosted by AEA’s Organizational Learning & Evaluation Capacity Building Topic Interest Group gave me pause to think about important qualities of a program.  His talk titled From Purist to Pragmatist: Expanding Our Approaches to Building Evaluation Capacity opened with a shared definition of program, evaluation, and program evaluation. He used the CDC (2011) definition that states:

A program is any set of organized activities supported by a set of resources to achieve a specific and intended result (p. 3)

Inherent in this rather broad definition are three important moving parts:

  1. Set of resources (inputs)
  2. Set of organized activities (activities described by outputs)
  3. Specific and intended result (outcomes and impacts)

Each of these provide parameters that define and focus a program and its evaluation. Yet, embedded in the definition are the qualifiers- organized, specific, and intended. Activities must have some sort of underlying organization. Perhaps the activities are organized developmentally, sequenced by difficulty, or arranged in some manner as a result of previous research. Results must be specific and intended. That is, there is a desired end that can be clearly articulated so as to justify the means.

A program exists when the qualifiers (organized, specific, and intended) are understood and there is a logical relationship between resources, activities, and outcomes. This logical arrangement is often described in a program’s logic model. Without understood qualifiers and an articulated logic, a program is not a program, but rather a loose set of activities.

Areas of Interest

As indicated in the tagline of this blog, I’m interested in the design, implementation, and evaluation of educational programs. This post, to be refined over time, serves as a framework describing relevant topics that will be explored in the future.

About program design:

  1. Teacher preparation, induction, and development
  2. Mathematics professional development
  3. College readiness and persistence
  4. School-community intersection
  5. Community health

About program implementation:

  1. Fidelity of implementation

About program evaluation:

  1. Evaluation capacity building
  2. Empowerment evaluation
  3. Developmental evaluation
  4. Monitoring and evaluation
  5. Evaluation questions and rubrics
  6. Theory of change and logic models

What other topics intersect or are aligned with these topics that would be helpful to add to this list?