Strengthening capacity for evidence generation: J-PAL’s evaluation incubators

This is the fourth blog in a series illustrating stories of how J-PAL’s training courses have built new policy and research partnerships and strengthened existing ones to advance evidence-informed decision-making. The first blog in the series highlights two meaningful examples from J-PAL’s Evaluating Social Programs course participants, the second showcases how custom courses strengthened government partnerships, and the third illustrates how online course participants applied learnings at their organizations.

J-PAL’s evaluation incubators use different combinations of training, resources, and technical support for varying lengths of time to guide participants through designing and implementing a randomized evaluation of their own program. Training lays the foundation for the incubators: participants learn the nuts and bolts of how to conduct a rigorous impact evaluation and are then able to better engage with other incubator elements, leading to more fruitful research collaborations.

To support evaluations with high potential for impact, our incubators usually focus on a particular region or sector and are often linked to specific funding opportunities or initiatives. This year, J-PAL held four incubators across three regions focusing on a range of issues, from expanding labor market opportunities in Brazil to Covid-19 recovery in the United States. Incubator participants typically apply with a project and research idea already in mind, such as a group of officials from Pierce County, Washington looking to assess the impact of a multifaceted eviction prevention program. 

Coupled with technical support from J-PAL staff on project and proposal development, participants use their training knowledge to actively engage in planning a randomized evaluation for their target program. Highly promising projects, in which a randomized evaluation may be feasible to implement, often go on to be matched with J-PAL affiliated researchers to carry out the study together. This blog showcases three examples of how training activities within J-PAL incubators helped partners better engage with the research design process and generate evidence to guide program decision-making.

Expanding research avenues to prevent gender-based violence in Peru

J-PAL Latin America and the Caribbean (LAC) has strengthened partnerships with local organizations through incubator courses, building long-term relationships to inform critical policy issues. One example is an ongoing collaboration with Peru’s Ministry of Women and Vulnerable Populations (MIMP) and Innovations for Poverty Action (IPA) which began in 2016 to identify the best strategies to confront gender-based violence (GBV) through scientific evidence. 

As a first step, J-PAL LAC and IPA hosted a two-day incubator workshop at the Ministry headquarters in Lima. Approximately 36 government officials from key areas across the Ministry participated in the sessions, with MIMP's senior management, including the former Vice-Minister of Women Ana María Mendieta, committed to the workshop's success. Facilitated by IPA and J-PAL LAC staff, the workshop provided theoretical and practical sessions to enhance participants’ skills in evidence use and generation. It also paved the way to identifying projects with high potential for impact in preventing violence against women that could be rigorously evaluated. 

Since the workshop, the Ministry, IPA, J-PAL, and several J-PAL affiliated researchers with a focus on gender in the region, including Jorge Agüero and Erica Field, have joined forces to evaluate and improve the design of MIMP’s GBV prevention programs. Currently, four impact evaluations are taking place to develop a learning cycle in Peru, including an evaluation of an innovative program to help men regulate their emotions and reduce the perpetration of intimate partner violence.

Connecting research and implementation for collaborations in Europe

J-PAL Europe has organized several incubators in recent years to advance research in specific areas, like promoting social inclusion in Europe, and to equip partners with the tools to explore collaborations on randomized evaluations. Incubators have also served to strengthen partnerships with various development agencies, such as an online incubator with the German Development Agency which included designing an impact evaluation across five project teams in East and West Africa. J-PAL Europe also collaborated with the French Fund for Innovation in Development (FID) to add an incubator module to this year’s Summer School on Development Methodologies targeting FID projects interested in launching their own impact evaluations.

Most recently, the Displaced Livelihoods Initiative (DLI) hosted its first in-person incubator in October 2023 focused on impact evaluations in forced displacement settings. The three-day workshop combined lectures on the fundamentals of impact evaluation and group work sessions where participants could apply the knowledge acquired to their individual projects. One of the participating organizations, Building Markets, an NGO that supports refugee-led businesses in Turkey, is partnering with a research team, including a J-PAL affiliated researcher, to better understand the impact of their programming and offers a compelling example of what the incubator hopes to achieve. The NGO took part in the incubator to better understand and refine the design of its impact evaluation, including their theory of change and outcome indicators related to social cohesion between refugees and host communities. 

"Joining this incubator enabled my ideas to take root and flourish, nurtured by the collective brilliance of curious minds. It allowed me to connect the dots between research and implementation, and was useful to deepen my understanding of research concepts."

Bora Arican, Turkey Country Director of Building Markets

Leveling up engagement to promote wildfire defensive practices in the United States

J-PAL North America runs evaluation incubators through multiple initiatives, focusing on topics such as housing stability and state and local government partnerships. In 2021, Bob Horton, former Chief Executive of Jackson County Fire District 3 in Oregon state and current Deputy Director of the Western Fire Chiefs Association, applied to be an incubator partner through the State and Local Innovation Initiative (SLI) to receive six months of individualized technical assistance support, flexible funding, and training opportunities. Oregon has been facing increasingly destructive wildfires, which disproportionately impact low-income residents, and Bob was motivated to evaluate different strategies to encourage resident uptake of “defensible space” strategies that help create a buffer between homes and wildfires, like clearing out flammable vegetation near the home. He wanted to make sure the district’s scarce government funding was going to programs that worked but felt that his team did not have the administrative capacity or specialized knowledge to explore rigorous evaluation strategies without additional support. 

In addition to meeting regularly with J-PAL North America staff to design a randomized evaluation, Bob attended J-PAL’s Evaluating Social Programs course as a key training component of the incubator to build knowledge and skills in rigorous evaluation. Bob shared that attending the course helped him take his learnings from technical assistance to a new level. Through the incubator, the team refined their research question, identified opportunities for randomization, explored existing evidence on wildfire risk mitigation strategies, and connected with researchers in J-PAL’s network. The incubator project grew into a funded pilot matched with two J-PAL affiliated researchers who are now working with the Jackson County Fire Department and Western Fire Chiefs Association to test interventions to increase adoption of defensible space.

"The training gave me a framework and process to apply evaluation principles to our incubator project and take a closer look at the other things we were spending money on."

Bob Horton, former Chief Executive of Jackson County Fire District 3 and current Deputy Director of the Western Fire Chiefs Association

A holistic approach for project incubation

What sets incubator courses apart from other J-PAL trainings are the integrated activities that work in tandem to strengthen organizational capacities for evidence generation to inform decision-making. Training helps incubator participants build their impact evaluation toolkit and engage more deeply with the technical support they receive from J-PAL staff and researchers to design and implement rigorous evaluations.

If you are interested in learning more about our evaluation incubators, please contact one of our regional office teams. For incubators in Latin America and the Caribbean, contact our LAC office at [email protected] and visit our website in Spanish. For information on the DLI Incubator in Europe, contact our initiative team at [email protected]. For more information on the LEVER Evaluation Incubator in North America, contact the SLI team at [email protected].