Motivating new education research and scale for COVID-19 recovery

J-PAL’s Post-Primary Education Initiative has funded more than 140 projects since its inception in 2013. We reflect on what we’ve learned through this research—and where we go from here.

Around the world, approximately 200 million students of secondary school age are out of school, and only 36 percent of tertiary school age students are enrolled in higher education. Of those in school, many reach young adulthood without developing the most basic literacy, numeracy, and life skills. 

Large surveys in countries like India, Kenya, Tanzania, and Uganda suggest that these learning gaps can trace back to primary school, when children initially fail to master basic primary level skills. 

It is also plausible that such learning gaps stem during post-primary education, depending on the relevance of the curriculum, the quality of the teaching, or the types of pedagogical innovations designed to reach marginalized students. 

In 2013, J-PAL affiliates and staff members conducted a global review of the evidence on post-primary education to better understand the discrepancies in learning outcomes. They found limited research that could help education stakeholders effectively identify and address the specific barriers restricting the delivery of quality, relevant, and equitable post-primary education. 

In response to these gaps, J-PAL launched the Post-Primary Education (PPE) Initiative, with support from the Douglas B. Marshall, Jr. Family Foundation, Echidna Giving, Google.org, the John D. and Catherine T. MacArthur Foundation, and the U.K. Department for International Development (now known as the U.K. Foreign, Commonwealth & Development Office). 

The PPE Initiative has sought to generate rigorous, policy-relevant research on open questions across secondary, tertiary, and vocational/entrepreneurial education in low- and middle-income countries. In doing so, PPE has funded randomized evaluations that develop and test innovations across nine priority areas and two cross-cutting themes. 

Over the past seven years, the PPE Initiative has allocated over $11 million across 14 regular funding cycles and one COVID-responsive off-cycle round. In total, the PPE Initiative has funded 140 full, pilot, and proposal development research projects to date. 

In addition to research grants, the PPE Initiative has funded four policy outreach grants in Argentina, the Dominican Republic, Ghana, and Turkey to support the dissemination of rigorous evidence on topics such as soft skills and education technology.

Contributions to global knowledge

The PPE Initiative has made great strides to increase the quantity, quality, and diversity of rigorous research produced.
 

Looking across PPE’s key priority areas and cross-cutting themes, since the release of J-PAL’s PPE evidence review in 2013, the PPE Initiative has doubled the number of existing PPE randomized studies focused on teachers.

Similarly, the number of PPE evaluations on vocational/entrepreneurial education has risen from 1 to 23, the number of pedagogy-focused studies has increased from 2 to 22, and PPE evaluations on topics such as curriculum, governance, and student motivation/effort have since emerged.

The PPE Initiative has funded research in 31 countries across all regions of the world. 

Tangible impacts on policy

Several drivers of impact that play a role in improving access to and quality of primary education also help motivate and inform our knowledge and understanding of post-primary education. As such, PPE-funded studies have not only added to the global evidence in education but have also contributed to key open questions in post-primary education and led to findings applicable to policy decisions and scale-ups. We highlight a few examples.

Improving school participation

For a start, the growing global evidence on school participation suggests that when making decisions about investing in education, parents and students often weigh the expected costs and benefits of school enrollment and attendance. Findings from a PPE- and U.S. Department of Labor co-funded randomized evaluation in Peru contributed to one of these lessons in particular: policies and programs that address perception gaps or make the benefits of education more salient increase school attendance at low cost. 

J-PAL affiliates Francisco Gallego (Pontificia Universidad Católica de Chile) and Christopher Neilson (Princeton) with Oswaldo Molina (Universidad del Pacífico) collaborated with Innovations for Poverty Action and the Peruvian Ministry of Education to evaluate at scale the effects of informing students about the returns to schooling through telenovela-style videos and an intensive app-based information campaign. 

Results suggest that the program was effective at changing students’ educational plans and lowering dropout rates by 1.8 percentage points, all at a cost of less than US$0.05 per student. 

The Peruvian government has since adopted and financed the program, which is now part of the national curriculum and has scaled to all urban public schools with full class days. With the ongoing COVID pandemic, the Peruvian government is currently broadcasting the program on national television as part of existing distance learning programs. 

This evaluation built on the successes of a similar program in the Dominican Republic, also evaluated by Neilson and J-PAL affiliate James Berry (University of Delaware) with Lucas Coffman (Harvard University) and Daniel Morales (IDEICE). The program decreased student dropout rates, increased students’ national test results, and has since been scaled up by the national government to reach roughly 92 percent of all public secondary schools.

Leveraging education technology to close student achievement gaps

Moreover, the growing body of evidence on student learning indicates that dedicating a portion of teaching time to tailoring instruction to students’ learning levels is an effective and cost-effective approach to closing student achievement gaps and improving overall learning outcomes. A PPE-funded evaluation in Delhi, India, contributes to this body of evidence by demonstrating how education technology can be leveraged to deliver personalized instruction to students.

J-PAL affiliates Karthik Muralidharan (University of California, San Diego) and Abhijieet Singh (Stockholm School of Economics) with Alejandro Ganimian (New York University) evaluated the impact of vouchers to attend a personalized, technology-aided afterschool instruction program, called Mindspark, on learning outcomes. 

Results showed that the 4.5-month intervention improved students’ math and Hindi test scores by 0.37 and 0.23 standard deviations, respectively, thereby demonstrating some of the largest impacts on middle-school learning seen in education research. 

Building on these promising results from the after-school supplemental education model, Muralidharan and Singh are also evaluating a subsequent scalable model in Rajasthan, India, which is implemented in schools and substitutes classroom time with the software. This is an ongoing 3-year evaluation, with encouraging evidence of significant positive impacts after one and two years.  

Strengthening girls’ empowerment

Furthermore, lessons from a PPE-funded evaluation of the Empowerment and Livelihood for Adolescents (ELA) program in Sierra Leone have answered key open questions related to the long-term effects of social and economic inputs on women’s empowerment. This, in turn, has contributed to a growing body of evidence on women's agency which indicates that soft and life skills training programs, often bundled with other components, can improve adolescent girls’ self-efficacy, confidence, and attitudes toward restrictive gender norms. 

Conducted by J-PAL affiliates Oriana Bandiera (London School of Economics) and Imran Rasul (University College London) in collaboration with Niklas Buehren (World Bank), Markus Goldstein (World Bank), and Andrea Smurra (University College London), the evaluation unexpectedly coincided with the Ebola outbreak in 2014-15. 

In highly disrupted comparison villages, the crisis resulted in younger girls spending significantly more time with men, leading to a rise in out-of-wedlock pregnancies and a persistent 17 percentage point drop in school enrollment post-crisis. 

These adverse effects on enrollment were halved in ELA villages, allowing girls to spend time away from men, thereby reducing out-of-wedlock pregnancies by 7 percentage points and enabling them to re-enroll in school post-crisis. A long-term follow-up in 2019-20 showed that impacts persisted, and girls from ELA villages were more likely to select partners who were more educated and more averse to gender-based violence.

Results from this study have become increasingly important in the COVID-19 context, when girls are particularly vulnerable to dropout and other pandemic-related challenges.

Measuring the long-term benefits of secondary education

Finally, given the limited rigorous and causal evidence on the impact of education on long-term development outcomes, PPE has funded a number of projects that track the impact of secondary education and vocational programs on policy-relevant outcomes, such as employment, marriage age, fertility, and wages, among others. 

One evaluation conducted in Ghana by J-PAL affiliates Esther Duflo (MIT), Pascaline Dupas (Stanford), and Michael Kremer (Harvard) found that providing scholarships for secondary school led to improvements in educational attainment, knowledge, skills, and preventative health behaviors, 11 years after the scholarships were awarded. Women who received the scholarships delayed marriage and pregnancy compared to their non-recipient peers. 

Findings from this study have also contributed to a broader body of knowledge on how reducing costs, such as eliminating school fees or providing small incentives like free uniforms, can lead to large increases in school participation. 

As very few studies measure the causal impact of education on long-term outcomes, results from such studies not only fill key evidence gaps but also strengthen the evidence base on the causal returns to education.

Looking ahead to a broader Initiative

Finding ways to respond to the COVID-19 pandemic and support school systems as they reopen is a pressing challenge that must be met in the context of severe budget constraints, insufficient crisis preparation, and persistent gender and socioeconomic gaps. After seven years focused on post-primary education, we are excited to announce two broader requests for proposals, supported by the Douglas B. Marshall Jr. Family Foundation and Echidna Giving, to address these challenges. 

The first request for proposals will launch the week of November 9 and will be run in conjunction with J-PAL’s Innovation in Government Initiative. Proposals seeking funding for either research or scale-ups in education will be eligible to apply to this one-time joint call.

These rounds will broaden in scope to include all grade levels from early childhood to tertiary education, but will narrow in focus to generate particularly timely and relevant knowledge to inform COVID response and recovery. We are especially seeking proposals focused on remote/distance learning, girls’ education, and remedial education. Proposals for research or scale-ups of strategies that improve other aspects of education systems to build resilience against future crises are also encouraged. 

In the future, J-PAL aims to launch a larger expanded research initiative which funds policy-relevant research across all grade levels. Stay tuned for updates!

At J-PAL, we often talk about the importance of using evidence to design and implement effective policies and programs, and many governments are working to make it a priority. But what does this look like in practice? How can we best support governments in finding and applying the evidence they need to make informed decisions?

We manage the Government Partnership Initiative (GPI), a J-PAL fund that supports governments in using evidence in policy design and decisions. This takes many forms, from temporarily placing J-PAL staff in a government office to support them in scaling up an evidence-informed program to developing new platforms to help governments use their administrative data more effectively. While J-PAL manages GPI’s day-to-day operations, the initiative is funded through generous support from longstanding J-PAL partner Community Jameel, as well as several other foundations and individuals dedicated to maximizing the positive impact that governments can have.

Since it launched in 2015, GPI has contributed to at least two scale-ups of evidence-informed policies in social protection and education in India and Zambia and improved systems for evidence use in Chile, India, and Peru. Our experience illustrates some promising strategies for closing the gap between evidence and policy in government. We’ve found that relatively small investments dedicated to putting evidence into practice can help some of the biggest players in poverty reduction use evidence to invest their limited funds more effectively.

This post is the first in a series on practical lessons learned from the first two years of GPI partnerships. Before we get into details, let’s start at the beginning:

Why create a fund to increase governments’ use of data and evidence?

Developing country governments support social programs that reach millions of people, with budgets that dwarf those of foreign aid agencies and foundations. Like many institutions, they sometimes spend money on ineffective programs or lose program resources due to implementation challenges.

By identifying ways to improve existing programs and pointing policymakers to more effective approaches, evidence from rigorous impact evaluations can help governments allocate scarce resources to programs that can have a bigger impact on the lives of the poor.

Since J-PAL was founded, we have built many long-term partnerships with governments to generate and use rigorous evidence in policy decisions. J-PAL's network of affiliated researchers and offices has conducted over fifty randomized evaluations in collaboration with governments, in addition to dozens of long-term capacity building and policy partnerships (for example, see GPI Co-Chair and J-PAL Executive Director Iqbal Dhaliwal's recent op-ed about J-PAL South Asia's decade of work with governments in India).

When governments decide to use data and evidence to improve policy, the results can be powerful. Of the 300 million people reached by scale-ups of programs evaluated and found effective by J-PAL affiliated professors, over 80 percent were reached by programs implemented or commissioned by governments.

What is the Government Partnership Initiative?

Recognizing that increasing the effectiveness of public spending is one of the biggest levers we have to reduce poverty worldwide, and given our positive experience working with governments over the past decade, we launched GPI in 2015 with a mission to build long-term partnerships between researchers and governments to increase the use of evidence in policy.

GPI provides funding to support three types of activities:

1. New randomized evaluations and pilot studies designed to answer governments’ priority questions; 
2. Technical assistance to governments to scale up an effective program based on existing evidence; and 
3. Technical assistance to governments to create systems and institutions that can help make evidence-informed policymaking more the norm.

What has GPI supported so far?

GPI has awarded US$2 million over the past two years to support 28 partnerships in 15 countries. This represents just 24 percent of the US$8.3 million requested by the 67 governments and researchers that have applied for funds.

Two measures of GPI's progress are our partnerships’ contributions to scale-ups of evidence-informed policies, and to improved systems for data and evidence use. We also track the number of policy-relevant research projects that are funded by or emerge from partnerships supported by GPI.

In all the partnerships GPI supports, J-PAL affiliated researchers and staff could not contribute to meaningful policy change without the leadership and dedication of our government partners, who are leading the way in making evidence-informed policy a reality, as well as the generous support of many funders and implementing partners.

SCALE-UPS

Most of our partnerships are still ongoing, with the potential to inform and improve policy in 2018 and beyond. But to date, the GPI portfolio has already contributed to at least two governments scaling evidence-informed policies that reach at least 48 million people.

A national funds flow reform in India

In India, GPI provided funds to place a J-PAL South Asia staff member in the Ministry of Rural Development to support them in scaling up a financial reform in a national social protection program for the poor, the Mahatma Gandhi National Rural Employment Guarantee Scheme (MGNREGS).

In 2012, J-PAL affiliated researchers Abhijit Banerjee (MIT), Esther Duflo (MIT), Clément Imbert (University of Warwick), and Rohini Pande (Harvard), collaborated with Santhosh Mathew, an official in the Government of Bihar, to conduct a randomized evaluation of a financial reform that disbursed MGNREGS funds to local governments on a reimbursement basis, rather than in advance. The reform reduced losses due to corruption and inefficiency by 24 percent, saving the government roughly US$6 million. The study was funded by the International Growth Centre and the International Initiative for Impact Evaluation.

This study formed the basis for the collaboration between J-PAL and the Ministry of Rural Development, where Santhosh Mathew began serving as Joint Secretary in 2013, marking an opportunity to leverage learning from the study nationwide. He collaborated with J-PAL affiliated researchers to share the research results and emerging policy insights with key central government ministries. The GPI-funded staff member supported this work by writing policy memos and coordinating meetings to generate buy-in for the scale-up, and developing a policy implementation plan with detailed steps, workflows, and timelines.

The research and policy outreach informed the Indian Union Cabinet’s approval of a national reform of MGNREGS funds flow in 2015. The reform allowed beneficiary payments to be made through a new National Electronic Fund Management System directly from the central government to the beneficiary’s bank account. The Cabinet note cites the evaluation as part of the rationale for this decision—highlighting the reduced leakage and improved service delivery generated through fund-flow reforms.

The reform has been scaled up in 21 states and one union territory, where the MGNREGS program reached over 48 million households last year. The authors of the Bihar study examined the rollout of the national reform and found that program expenditures fell by 18 percent, similar to the effect found in the original study.

“I could not have done this on my own. It’s the powerful research but most important—far more important—is the support over the last six years that I got from the J-PAL team. They have been a dedicated team whose only job is to help people like me to push so that when I need a new document it’s redone. When I need to make a presentation, it’s they who do it for me.”
– Santhosh Mathew, former Joint Secretary, Ministry of Rural Development, India

Teaching at the Right Level in Zambia

In Zambia, GPI provided funds to J-PAL Africa to support the Ministry of General Education in designing and piloting a remedial education program based on randomized evaluations from India, Ghana, and Kenya. GPI funds supported learning journeys between India and Zambia and helped support independent process monitoring of the government’s pilot program in 80 schools.

The Zambian remedial education program, named “Catch Up,” is based on the Teaching at the Right Level (TaRL) pedagogical approach. The program aims to improve the basic literacy and numeracy skills of primary school students by grouping them according to their initial level of knowledge and providing them with instruction tailored to their current learning level.

Pratham, a large education-focused NGO based in India, developed and refined the TaRL approach over the last two decades. Pratham has collaborated with J-PAL affiliated researchers since the early 2000s to evaluate of several different versions of the approach in India. Results showed that tailoring instruction to the level of the child was a highly effective and cost-effective way to improve student learning, and Pratham has since scaled it up in many states in India.

Some of the GPI funds in Zambia supported learning journeys, enabling Zambian officials to travel to India to see Pratham implement the TaRL approach firsthand. GPI funds also supported Pratham staff travel to Zambia to train the ministry on implementing the TaRL model. Zambia's design additionally drew on global evidence on adjusting teaching to the level of the child, including from the TCAI program in Ghana and from a Kenya program that tracked students by ability.

As a result of this work, the Zambian government launched an 80-school pilot of the Catch Up program, trying out multiple versions to identify the Catch Up model that was best for their school system. The GPI grant supported a collaboration between Innovations for Poverty Action (IPA), J-PAL Africa, and UNICEF to conduct an independent process monitoring of the pilot program to assess the viability of a broader scale-up to more schools across the country.

In August 2017, after the completion of the pilot and process monitoring, the Ministry decided to scale the program to approximately 1,800 schools across Zambia over the next three years. The evidence-based program will reach approximately 280,000 students in grades 3-5 with support from USAID’s Development Innovation Ventures and the USAID Zambia Mission. Several additional partners provided support to the pilot and/or scale-up, including UNICEF, IPA, Pratham, VVOB, and Zambia Education Sector Support Technical Assistance.

GPI funds helped meet a critical policy window. The Ministry was interested in finding a remedial education (or "Catch Up") program, but did not know about effective approaches from randomized evaluations or have the flexible funding to adapt and monitor a Catch Up program to their context. By meeting this need, the Ministry was able to test the program for itself, monitor the implementation, and make decisions about how to scale it up. Other partners (such as USAID) were also watching the pilot and only decided to invest after they saw that this program could be successfully implemented in Zambia.

IMPROVED SYSTEMS FOR EVIDENCE USE

While individual evaluations and scale-ups are important, GPI's ultimate goal is to institutionalize the use of evidence in policymaking, making it more the norm rather than the exception.

GPI grants have contributed to several improved systems for data and evidence use that encourage, require, or incentivize greater generation and/or use of evidence in the policy process. Below are just three examples:

1. In Chile, the Ministry of Economy’s Innovation Fund worked with J-PAL LAC to create new system for reviewing potential investments requested by other departments within the government. The system requires applicants to include a theory of change and review of existing evidence from past impact evaluations when seeking funds. They also launched a new public bid for impact evaluations of the Innovation Fund’s investments and reserved a portion of program funds for evaluation.

“In the fund, this system created the perception that it is necessary to base yourself on evidence to make public policies.”
– Antonio Martner Sota, Research Coordinator, Innovation Fund for Competitiveness, Ministry of Economy, Development, and Tourism

2. In India, GPI is supporting staff working on J-PAL South Asia’s institutional partnership with the Government of Tamil Nadu to create an evidence-informed approach to policymaking across its departments. To date, the government has supported 15 policy-driven research projects across nine departments, using over US$4 million of its own funding and collaborating with more than 34 academic researchers. Recently, the state issued new guidelines requiring all programs in sectors of strategic importance with budgets greater than US$22 million undergo impact evaluations. The state’s Planning Department has also created a dedicated fund for program evaluation of US$1.5 million per year.  Using funds from GPI, J-PAL South Asia staff also provide capacity-building support to a data analytics unit set up by the Government to allow them to make real-time decisions based on the administrative data they already collect.

3. In Peru, the Ministry of Women and Vulnerable Populations refined their approach to reducing domestic violence to focus more on violence prevention after J-PAL LAC created an evidence repository summarizing key findings from impact evaluations on reducing violence. As a result of this partnership, the Ministry is also currently preparing a randomized evaluation with J-PAL affiliated researcher Erica Field to evaluate the impact of a community program to prevent domestic violence. 

"With this repository it was clear that one of the most important themes in the area of violence was prevention of violence. Our focus changed based on this evidence.”- Director, Monitoring and Evaluation, Ministry of Women and Vulnerable Populations

ONGOING PARTNERSHIPS

GPI has twenty ongoing partnerships, many of which have the potential to inform and improve policy in the future. For example, six of the 16 institutional partnerships supported by GPI have already contributed to an improved system for data and evidence use or led to a new evaluation on a government’s priority question.

Research partnerships often represent the first time a government or department is investing in evaluation, and often lay the groundwork for future collaborations that move beyond a single project. For example, the partnership described above with the Ministry of Rural Development in India began with a single evaluation in Bihar and contributed to a national reform after several years of collaboration.

What are we learning about building a culture of evidence use in government?

Our experience with these partnerships has created a rich base of internal knowledge on how to most effectively work with governments to apply evidence to policy in meaningful ways. We’re working to summarize and share this knowledge in an effort to further expand GPI’s impact in 2018. Stay tuned for our next post in this series to learn some early insights from our interviews with our government partners.

For more information about GPI, including opportunities for foundations and individuals to support this important work, please contact [email protected].

This is the first of two posts about GPI. Read the second post in this series.

Despite large advances in school enrollment over the last decade, approximately 262 million children, adolescents, and youth between the ages of 6 and 17 are still out of school, according to UNESCO. In Latin America and elsewhere, school dropout is a key obstacle to universal school attendance: in the region, 15.5 percent of students dropped out during their time in primary school in 2016.

How do policymakers begin to tackle such a large and complex issue? And, if policymakers do find a possible solution, how do they know it can successfully and cost-effectively work when implemented at scale?

Long-term partnerships between governments and researchers, based on the use and institutionalization of rigorous evidence, may be the answer.

On December 3, 2018, education policymakers and researchers from the Dominican Republic, Chile, Colombia, Peru, and the United States joined representatives from the Inter-American Development Bank, Teach for All, and Inicia Educación to discuss strategies for establishing these types of partnerships. Co-sponsored by J-PAL’s Latin America & the Caribbean (LAC) office, together with the Dominican Institute for Educational Quality Research and Evaluation (IDEICE) and Princeton University, the event highlighted the successes of education partnerships between governments and researchers throughout Latin America. Successful examples included the evaluation and scale-up of information dissemination policies in education, as well as the strengthening of government data centers and impact evaluation labs in Chile, the Dominican Republic, and Peru.

Implementing evidence-based education policies at scale

National-level education programs that aim to reduce school dropout by providing information to students about the returns to staying in school have proved promising in some contexts.

In the Dominican Republic, J-PAL affiliated researchers James Berry (University of Delaware) and Christopher Neilson (Princeton University), together with researchers Lucas Coffman (Harvard University) and Daniel Morales (IDEICE), evaluated a program that transmits information about the returns to education to middle-school students through the use of different forms of media, including telenovela-style videos, posters, and tablet-based information campaigns.

Preliminary results suggest the program reduced school dropout rates by between 2.5 and 3 percentage points after a year—resulting in approximately 6,500 more students staying in school. In addition, for children who received the information, it improved schooling outcomes in national examination scores by between 0.05 and 0.12 standard deviations.

Similarly, in Peru, J-PAL affiliates Francisco Gallego (Pontificia Universidad Católica de Chile), and Neilson, together with Oswaldo Molina (Universidad del Pacífico), collaborated with Innovations for Poverty Action (IPA) and the Ministry of Education’s MineduLAB to study the effects of the “Choosing a Better Future (DFM)” program. DFM also spread information about returns to schooling through videos, in addition to including more intensive, in-person information delivery to students and their parents.

Consistent with the preliminary results from the Dominican Republic, DFM decreased school dropout rates by 18.8 percent in urban areas and corrected students’ and parents’ perceptions about the returns to education. Furthermore, the intervention reduced child labor by 15 percent for girls in urban areas.

Both programs were found to be cost-effective—the cost of videos per students in Peru was only US$0.05, and increases in national examination scores in the Dominican Republic cost between US$78.5 and US$124.4 per standard deviation. Given their impact and cost-effectiveness, both programs are respectively being scaled-up by the Peruvian and Dominican governments to ensure that all public schools are reached.

Institutionalizing evidence use

Beyond these policies, partnerships between government and academia have resulted in the promotion and institutionalization of administrative data use within governments.

During the event, Juan Ariel Jimenez (Vice Minister of Development Policy of the Ministry of the Presidency, Dominican Republic), Juan Pablo Silva (former Vice Minister of the Ministry of Education in Peru), Ryan Cooper (DIPRES Impact Evaluation Coordinator, Chile), and Elianny Medina (Cabinet for Social Policy Coordination, Dominican Republic) discussed ongoing efforts throughout the region to embed innovation and impact evaluation labs and administrative data centers within governments. These units include Peru’s MineduLAB—which pilots and evaluates the effectiveness of a number of innovations in education policy—as well as the Gabinete de Coordinación de Politícas Sociales (GCPS) evidence lab and a Data Center of the State in the Dominican Republic, which harmonize administrative data across government bodies and generate evidence for social policies.

The gathering in Santo Domingo was a concrete reminder of how long-term partnerships between government and researchers can yield substantial results; either in the form of cost-effective and scalable policies that are proven to be effective in a local context or in the increased use of administrative data to ensure that policies are grounded in evidence. Such data-oriented initiatives are examples of J-PAL LAC’s long-term efforts to forge and build evidence-based government partnerships to improve the effectiveness of programs in education and more, across the continent.