evaluation design

Evaluation Questions

The foundation of good research, including evaluation, is a well-written question in which quantitative and/or qualitative research methods attempt to answer. In the case of evaluation, questions take on different value-oriented frames such as appropriateness, effectiveness, efficiency, impact, or sustainability.

The three resources below and those added to the Twitter discussion capture different qualities related to writing good evaluation questions.

  1. A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions (FSG) This guide offers practical considerations about stakeholders at the beginning of an evaluation as well as the conclusion of an evaluation.
  2. Evaluation Questions Library Guide (CDC)  This library guide aggregates resources that provide (1) an introduction to evaluation questions, (2) developing and using evaluation questions, and (3) linking evaluation questions to approaches, designs, or criteria.
  3. Evaluation Questions Checklist for Program Evaluation from The Evaluation Center at Western Michigan University   This checklist describes ways in which evaluation questions should be (1) evaluative, (2) pertinent, (3) reasonable, (4) specific, (5) answerable, and (6) complete.
evaluation design programs

Programming and Domains of Evaluation Questions

In Developing Monitoring and Evaluation Frameworks, Markiewicz and Patrick (2016) laid out the relationship of program development and implementation to five domains of evaluation questions. In doing so, they’ve contributed to the discussion in the previous post about what evaluation is for. At the very least, drilling down the domains of evaluation questions alter the types of program decisions to be considered and answered.

The following table, adapted from their work, summarizes the five domains of evaluation questions aligned to five program components.

Program Component Domains of Evaluation Questions Description of Domains
Planning and designAssessing the appropriateness of the program’s design-Suitability of program design in context
-Fit of program with program theory and/or logic
-Testing of underlying assumptions
-Extent program meets the priorities and needs of key stakeholders
Objectives Assessing program effectiveness in meeting it’s objectives, its value, and quality – Fidelity of implementation
-Achievement of program objectives
-Assessment of the quality and value of the program
ImplementationExamining efficiency and fidelity in program implementation-Conversion of inputs to outputs and outputs to results
-Governance and management
ResultsEstablishing impact: intended and unintended, and the degree to which change is attributable to the program-Changes (results) produced by the program, intended and unintended, direct and indirect
Sustainability of resultsIdentifying ongoing sustainable benefits from the program-Continuation of program benefits
evaluation design

Program Evaluation Tiers

The Harvard Family Research Project offers several useful evaluation resources. The guide Afterschool Evaluation 101: How to Evaluate an Expanded Learning Program is one of their valuable resources useful to provide non-evaluators an overview of program evaluation. The guide is organized around nine steps:

  1. Determine the evaluation’s purpose
  2. Developing a logic model
  3. Assessing your program’s capacity for evaluation
  4. Choosing the focus of your evaluation
  5. Selecting the evaluation design
  6. Collecting data
  7. Analyzing data
  8. Presenting evaluation results
  9. Using evaluation data

Evaluations vary as much as programs do (i.e., different activities, duration, outcomes, et cetera) underscoring the importance of wisely choosing the focus of an evaluation. Step 4 in the guide “Choosing the focus of your evaluation” describes a five tier approach that is summarized below (pages 13-16). Determining an appropriate evaluation focus is largely dependent on a program’s maturity and developmental stage.

  • Tier 1- Conduct a needs assessment to address how the program can best meet needs
  • Tier 2- Document program services to understand how program services are being implemented
  • Tier 3- Clarify the program to see if the program is being implemented as intended
  • Tier 4- Make program modifications to improve the program
  • Tier 5- Assess program impact to demonstrate program effectiveness

As you can see, evaluation can and should coincide with a program throughout its lifespan. These tiers are useful to help design an evaluation plan and to determine appropriate methods of data collection and analysis.