Volunteers set up water distribution

Generating Evidence

Photo Credit: Shutterstock.com

Help build our shared knowledge of effective Strategies to address homelessness.

Generating evidence—that is, conducting rigorous research—is essential for learning the most effective methods to prevent and reduce homelessness. Results from impact evaluations, especially randomized evaluations, can both  inform administrators about the effectiveness of their programs and also instruct the broader field on the best ways to target and invest resources.

While the body of existing literature on effective approaches to homelessness reduction is growing, important research questions remain to be answered.

If you are a nonprofit or public sector leader in the housing or homelessness sector interested in evaluating the impact of your program or policy, we hope the modules below will provide you with foundational information and tools to generate agency buy-in, create a shared language among research teams, and set high-quality evaluations up for success.

Before Jumping In: Researcher-Practitioner Partnerships

At J-PAL North America, we believe that a strong research partnership is one of the best tools for conducting a rigorous impact evaluation, particularly when carrying out a randomized evaluation. That's why we work to connect service providers and government leaders with academic researchers to conduct rigorous, policy-relevant randomized evaluations. Each brings an essential expertise to the relationship without which an evaluation cannot be successful. 

Ven diagram showing benefit of research expertise, policymaker expertise, and overlap where evidence can thrive.

While critical, these relationships can be challenging to form and and even harder to maintain. Check out the following resources for insights into this model and guidance on successful partnerships.

Resources on Successful Research Partnerships

Impact Evaluations: Foundations

An impact evaluation is a systematic assessment of whether a program or policy has its intended effect (or impact) on a specific outcome. While there are several methods of estimating the effectiveness of a program—all with differing degrees of credibility—they all share several common fundamentals.

The following modules provide an introduction to these fundamentals of a good impact evaluation.

Module 1: Overview of Impact Evaluations

Learning Objectives:

By interacting with the materials in this module, we hope that you will be able to…

  1. Understand the value of impact evaluations for evidence-based decisionmaking;
  2. Define key terminology such as “impact,” and “counterfactual”;
  3. Recognize common impact evaluation methods; and
  4. Articulate your research questions.

Watch: Introducing Impact Evaluations

Read: 

  • Glossary: Module 1 (PDF)
  • Overview of Impact Evaluation Methods (PDF)

Practice:

  • Laying the Groundwork for an Impact Evaluation: Part 1 (PDF | Drive)

Module 2: Creating a Theory of Change

Learning Objectives:

By interacting with the materials in this module, we hope that you will be able to…

  1. Recognize the importance of theories of change for both developing and evaluating a program;
  2. Identify the key components of a theory of change;
  3. Practice thinking through a theory of change and, if applicable, refine your research question accordingly.

Watch: Introducing a Theory of Change 

Read:

  • Glossary: Module 2 (PDF)
  • Theory of Change: Example (PDF)

Practice:

  • Laying the Groundwork for an Impact Evaluation: Part 2 (PDF | Drive)

Learn More:

Module 3: Data and Measurement

Learning Objectives:

By interacting with the materials in this module, we hope that you will be able to…

  1. Understand the importance of using good data to measure different components of your theory of change;
  2. Distinguish between primary and secondary sources of data;
  3. Define key terms, such as validity and reliability; and
  4. Recognize common sources of data used in housing and homelessness evaluations.

Watch: Introducing Data and Measurement

Read:

  • Glossary: Module 3 (PDF)

Practice:

  • Laying the Groundwork for an Impact Evaluation: Part 3 (PDF | Drive)

Learn More:

BONUS! Explore: Housing Data [External]

Module 4: Fundamentals of Ethical Research

Learning Objectives:

By interacting with the materials in this module, we hope that you will be able to…

  • Identify the three key components of research ethics, as outlined by the Belmont Report;
  • Understand the role of Institutional Review Boards in protecting research participants and promoting ethical research; and
  • Recognize specific ethical considerations when working with unhoused populations.

Watch: Introducing Research Ethics 

Read:

  • Glossary: Module 4 (PDF)

Practice:

Learn More:

Module 5: Understanding Statistical Power

Learning Objectives:

By interacting with the materials in this module, we hope that you will be able to…

  • Define statistical power and its importance in high-quality impact evaluations;
  • Understand the relationship between power, sample size, and effect size; and
  • Gain practice using a power calculator.

Watch: Introducing Statistical Power 

Read:

  • Glossary: Module 5 (PDF)
  • Six Rules of Thumb for Determining Sample Size and Statistical Power (PDF)

Practice:

Learn More:

Randomized Evaluations: Key Components and Considerations

Randomized evaluations are a type of impact evaluation that, due to their use of random assignment, are uniquely positioned to identify the causal impacts of a program. This information—whether a program caused a change in an outcome—is critical for making evidence-informed decisions about programs and policies. Despite their value, randomized evaluations can be challenging to execute and are not the best fit for every context or research question. 

The following modules provide an introduction to the basic premise behind randomized evaluations and the instances in which they may be an appropriate method for ascertaining the impact of a program or policy.

Module 6: Overview of Randomized Evaluations

Learning Objectives:

By interacting with the materials in this module, we hope that you will be able to…

  • Understand the unique advantage of randomized evaluations to identify causal impacts;
  • Identify the fundamental characteristics of an RCT; and
  • Define key terminology, such as random assignment.

Watch: Introducing Randomized Evaluations 

Read:

  • Glossary: Module 6 (PDF)
  • Review: Overview of Impact Evaluation Methods (PDF)

Practice:

  • Is a randomized evaluation the right fit? Part 1 (PDF | Drive)

Learn More:

Module 7: Opportunities for Randomization

Learning Objectives:

By interacting with the materials in this module, we hope that you will be able to…

  • Identify instances when a randomized evaluation may be an appropriate research method to use;
  • Recognize common randomization strategies; 
  • Become familiar with how these strategies have been applied in prior housing and homelessness research; and 
  • Think about whether a randomized evaluation might be a good fit to answer your research questions.

Watch: Introducing Randomization Designs

Read:

  • Glossary: Module 7 (PDF)
  • Common Randomization Designs (PDF)

Practice:

  • Is a randomized evaluation the right fit? Part 2 (PDF | Drive)

Learn More:

Diving Deeper

Interested in learning more about impact evaluations? Try these more in-depth resources and training opportunities:

If you are interested in diving even deeper into these materials, contact [email protected] to inquire about a custom training.

Get Connected

Ready to rigorously evaluate your program with a randomized evaluation? Contact us at [email protected] to discuss opportunities to be connected to an academic researcher in J-PAL North America's network.