What have we learned about building a culture of data and evidence use in government?
This is the second of two posts about GPI. Read the first post in this series.
Earlier this month, we wrote about how partnerships funded by J-PAL’s Government Partnership Initiative (GPI) have helped increase governments’ use of data and evidence in policymaking in places like Chile, India, Peru, and Zambia.
One common theme across all of GPI’s partnerships is that they are led by a champion in government who is making evidence-informed policy a priority. We could not contribute data and evidence to policy decisions without their leadership and dedication.
But in addition to great leadership, what have we learned about how governments are building a culture of using data and evidence in policymaking? And about how J-PAL and other evidence-to-practice organizations can best support them in these efforts?
To start to answer these questions, we conducted in-depth interviews with stakeholders from eleven of our government partnerships in Brazil, Chile, Colombia, and Peru. We’ll share the full findings later this year, but in the meantime, some early insights about what’s working have emerged.
1. Make it someone’s job to use evidence.
Applying evidence to design better policies and programs takes time and resources. In GPI, we've seen that when an official or department is committed to prioritizing evidence-informed policy, allocating even a small amount of resources and personnel time to using evidence is an important way to start to build a culture to support data-driven and evidence-informed decisions.
Some governments can and do hire staff to do this on their own, but others don’t have the resources. In these cases, temporarily placing policy and research staff to work within a government office or closely with them to achieve a particular goal can be effective (for example, see the India case from our last post). Working in a government office can be an effective way to build trust, better understand their priorities, and spot policy windows where evidence could be useful.
Embedded staff can also be useful to government officials, who may not have the time to compile the data and evidence they need to make policy decisions, by producing summaries of existing evidence or doing quick analyses of administrative data.
This model is not new; organizations like the Overseas Development Institute and others have embedded technical staff in governments for years. We’re just adding our voice to the chorus of organizations saying that it can work, even (and maybe especially) as a way to drive the use of evidence in policy.
In hiring these staff, we’ve found that the most important qualities are previous government experience in the country or region, knowledge of existing evidence and research methods, data analysis skills, and an enthusiasm for supporting their government colleagues and being responsive to their needs. Some governments have gone on to hire these temporary staff as full-time employees, including one of our partners in Brazil.
2. Help governments make better use of the data they already collect.
Many governments collect vast amounts of data, but need support to organize it in a usable format and identify use cases to improve program management. Creating user-friendly charts, maps, tables, and dashboards with key stats on program implementation, or combining previously disconnected datasets, can help officials see how their data can be useful in driving better program management decisions.
Administrative data can also be used to run low-cost evaluations that can demonstrate the usefulness of testing and refining programs. Several of our government partners highlighted the importance of improving their administrative data systems to accelerate their transition to evidence-informed policymaking.
In Brazil, GPI funded the salary for a research associate to sit inside a state security department to help organize and analyze its data more effectively. She helped them develop a platform that allows police to view real-time data on crime, police presence, and environmental factors, to inform decisions about dispatching officers.
Research associate in Brazil analyzes data for a state security department
3. Invest in long-term partnerships.
Having a long-term presence in a country or a long-term relationship with a government fosters greater trust and mutual understanding. Long-term partnerships between governments and researchers can help civil servants understand the different ways that evidence can be used and help researchers make their research more useful for policy.
By building a reputation and knowledge among many people in a government, long-term partnerships allow us to identify and respond quickly to policy windows where evidence could be useful, know the right people to work with, and maintain partnerships despite transfers of key officials or administration changes after elections.
Working together on multiple projects, and not just impact evaluations, is also more conducive to evidence use and institutional learning than a one-time evaluation.
In Peru, J-PAL and Innovations for Poverty Action used a GPI grant to begin collaborating with the Ministry of Women and Vulnerable Populations in 2016. Our first project was to create an evidence repository of all relevant completed impact evaluations related to the Ministry’s priority questions on reducing domestic violence, which the Ministry used to inform its strategy on reducing domestic violence.
As a result of this work, the Ministry developed a new randomized evaluation with J-PAL affiliated researcher Erica Field to evaluate the impact of one of the Ministry’s flagship domestic violence prevention programs.
4. Take on quick wins that can build trust and demand for evidence.
Starting a relationship with simple, low-cost evaluations, descriptive data analyses, or trainings tailored to government priorities can open the door for future collaboration by demonstrating the usefulness of data, evaluation, and existing evidence.
Collaborating with government schools that train civil servants can also help make training resources available to a much larger number of agencies.
Recently GPI funded a partnership with Brazil’s National School of Public Administration (ENAP) to develop an online course on evaluating social programs in Portuguese. The course, which is freely available on ENAP’s online learning platform for all Brazilian citizens, will make our capacity-building materials more widely available. Given that more than 1,000 Brazilian civil servants recently completed the course, we are enthusiastic about this model for engaging a wide range of officials across agencies.
5. Dedicating a small sum of money to evidence use can ease large constraints.
When directed to the right people at the right time, even small amounts of money can reduce seemingly large barriers to using data and evidence in decisions.
Often, small sums can allow either governments or organizations like J-PAL to hire someone whose job it is to use evidence, collect relevant data, or invest in the software they need to process information. Providing quick-turnaround grants after an evaluation is complete can also support evidence-informed scale-ups or reforms, or help set up a government lab that incentivizes greater use of evidence. Governments may not be able to fund these activities due to budget constraints, lengthy contracting requirements, or conflicting priorities, and a small grant can have a big impact.
For example, in India, just US$70,000 of GPI support enabled J-PAL South Asia to support a government partner’s efforts to lead the national scale-up of a funds-flow reform to a major government program.
Stay tuned
We'll be sharing more insights from GPI throughout 2018. Based on what we’ve learned so far, we hope to expand our investments in two successful models going forward:
Embedding J-PAL staff within governments to support the use of evidence in policy design in key sectors like education, health, and social protection.
Working with governments to make the administrative data they already collect more useful for day-to-day decision-making and evaluation.
We're always looking for new partners in these efforts. If you're interested in learning more about J-PAL's Government Partnership Initiative, email us at [email protected].
Authored By
At J-PAL, we often talk about the importance of using evidence to design and implement effective policies and programs, and many governments are working to make it a priority. But what does this look like in practice? How can we best support governments in finding and applying the evidence they need to make informed decisions?
At J-PAL, we often talk about the importance of using evidence to design and implement effective policies and programs, and many governments are working to make it a priority. But what does this look like in practice? How can we best support governments in finding and applying the evidence they need to make informed decisions?
We manage the Government Partnership Initiative (GPI), a J-PAL fund that supports governments in using evidence in policy design and decisions. This takes many forms, from temporarily placing J-PAL staff in a government office to support them in scaling up an evidence-informed program to developing new platforms to help governments use their administrative data more effectively. While J-PAL manages GPI’s day-to-day operations, the initiative is funded through generous support from longstanding J-PAL partner Community Jameel, as well as several other foundations and individuals dedicated to maximizing the positive impact that governments can have.
Since it launched in 2015, GPI has contributed to at least two scale-ups of evidence-informed policies in social protection and education in India and Zambia and improved systems for evidence use in Chile, India, and Peru. Our experience illustrates some promising strategies for closing the gap between evidence and policy in government. We’ve found that relatively small investments dedicated to putting evidence into practice can help some of the biggest players in poverty reduction use evidence to invest their limited funds more effectively.
This post is the first in a series on practical lessons learned from the first two years of GPI partnerships. Before we get into details, let’s start at the beginning:
Why create a fund to increase governments’ use of data and evidence?
Developing country governments support social programs that reach millions of people, with budgets that dwarf those of foreign aid agencies and foundations. Like many institutions, they sometimes spend money on ineffective programs or lose program resources due to implementation challenges.
By identifying ways to improve existing programs and pointing policymakers to more effective approaches, evidence from rigorous impact evaluations can help governments allocate scarce resources to programs that can have a bigger impact on the lives of the poor.
Since J-PAL was founded, we have built many long-term partnerships with governments to generate and use rigorous evidence in policy decisions. J-PAL's network of affiliated researchers and offices has conducted over fifty randomized evaluations in collaboration with governments, in addition to dozens of long-term capacity building and policy partnerships (for example, see GPI Co-Chair and J-PAL Executive Director Iqbal Dhaliwal's recent op-ed about J-PAL South Asia's decade of work with governments in India).
When governments decide to use data and evidence to improve policy, the results can be powerful. Of the 300 million people reached by scale-ups of programs evaluated and found effective by J-PAL affiliated professors, over 80 percent were reached by programs implemented or commissioned by governments.
What is the Government Partnership Initiative?
Recognizing that increasing the effectiveness of public spending is one of the biggest levers we have to reduce poverty worldwide, and given our positive experience working with governments over the past decade, we launched GPI in 2015 with a mission to build long-term partnerships between researchers and governments to increase the use of evidence in policy.
GPI provides funding to support three types of activities:
- New randomized evaluations and pilot studies designed to answer governments’ priority questions;
- Technical assistance to governments to scale up an effective program based on existing evidence; and
- Technical assistance to governments to create systems and institutions that can help make evidence-informed policymaking more the norm.
What has GPI supported so far?
GPI has awarded US$2 million over the past two years to support 28 partnerships in 15 countries. This represents just 24 percent of the US$8.3 million requested by the 67 governments and researchers that have applied for funds.
Two measures of GPI's progress are our partnerships’ contributions to scale-ups of evidence-informed policies, and to improved systems for data and evidence use. We also track the number of policy-relevant research projects that are funded by or emerge from partnerships supported by GPI.
In all the partnerships GPI supports, J-PAL affiliated researchers and staff could not contribute to meaningful policy change without the leadership and dedication of our government partners, who are leading the way in making evidence-informed policy a reality, as well as the generous support of many funders and implementing partners.
SCALE-UPS
Most of our partnerships are still ongoing, with the potential to inform and improve policy in 2018 and beyond. But to date, the GPI portfolio has already contributed to at least two governments scaling evidence-informed policies that reach at least 48 million people.
A national funds flow reform in India
In India, GPI provided funds to place a J-PAL South Asia staff member in the Ministry of Rural Development to support them in scaling up a financial reform in a national social protection program for the poor, the Mahatma Gandhi National Rural Employment Guarantee Scheme (MGNREGS).
In 2012, J-PAL affiliated researchers Abhijit Banerjee (MIT), Esther Duflo (MIT), Clément Imbert (University of Warwick), and Rohini Pande (Harvard), collaborated with Santhosh Mathew, an official in the Government of Bihar, to conduct a randomized evaluation of a financial reform that disbursed MGNREGS funds to local governments on a reimbursement basis, rather than in advance. The reform reduced losses due to corruption and inefficiency by 24 percent, saving the government roughly US$6 million. The study was funded by the International Growth Centre and the International Initiative for Impact Evaluation.
This study formed the basis for the collaboration between J-PAL and the Ministry of Rural Development, where Santhosh Mathew began serving as Joint Secretary in 2013, marking an opportunity to leverage learning from the study nationwide. He collaborated with J-PAL affiliated researchers to share the research results and emerging policy insights with key central government ministries. The GPI-funded staff member supported this work by writing policy memos and coordinating meetings to generate buy-in for the scale-up, and developing a policy implementation plan with detailed steps, workflows, and timelines.
The research and policy outreach informed the Indian Union Cabinet’s approval of a national reform of MGNREGS funds flow in 2015. The reform allowed beneficiary payments to be made through a new National Electronic Fund Management System directly from the central government to the beneficiary’s bank account. The Cabinet note cites the evaluation as part of the rationale for this decision—highlighting the reduced leakage and improved service delivery generated through fund-flow reforms.
The reform has been scaled up in 21 states and one union territory, where the MGNREGS program reached over 48 million households last year. The authors of the Bihar study examined the rollout of the national reform and found that program expenditures fell by 18 percent, similar to the effect found in the original study.
“I could not have done this on my own. It’s the powerful research but most important—far more important—is the support over the last six years that I got from the J-PAL team. They have been a dedicated team whose only job is to help people like me to push so that when I need a new document it’s redone. When I need to make a presentation, it’s they who do it for me.”
– Santhosh Mathew, former Joint Secretary, Ministry of Rural Development, India
Teaching at the Right Level in Zambia
In Zambia, GPI provided funds to J-PAL Africa to support the Ministry of General Education in designing and piloting a remedial education program based on randomized evaluations from India, Ghana, and Kenya. GPI funds supported learning journeys between India and Zambia and helped support independent process monitoring of the government’s pilot program in 80 schools.
The Zambian remedial education program, named “Catch Up,” is based on the Teaching at the Right Level (TaRL) pedagogical approach. The program aims to improve the basic literacy and numeracy skills of primary school students by grouping them according to their initial level of knowledge and providing them with instruction tailored to their current learning level.
Pratham, a large education-focused NGO based in India, developed and refined the TaRL approach over the last two decades. Pratham has collaborated with J-PAL affiliated researchers since the early 2000s to evaluate of several different versions of the approach in India. Results showed that tailoring instruction to the level of the child was a highly effective and cost-effective way to improve student learning, and Pratham has since scaled it up in many states in India.
Some of the GPI funds in Zambia supported learning journeys, enabling Zambian officials to travel to India to see Pratham implement the TaRL approach firsthand. GPI funds also supported Pratham staff travel to Zambia to train the ministry on implementing the TaRL model. Zambia's design additionally drew on global evidence on adjusting teaching to the level of the child, including from the TCAI program in Ghana and from a Kenya program that tracked students by ability.
As a result of this work, the Zambian government launched an 80-school pilot of the Catch Up program, trying out multiple versions to identify the Catch Up model that was best for their school system. The GPI grant supported a collaboration between Innovations for Poverty Action (IPA), J-PAL Africa, and UNICEF to conduct an independent process monitoring of the pilot program to assess the viability of a broader scale-up to more schools across the country.
In August 2017, after the completion of the pilot and process monitoring, the Ministry decided to scale the program to approximately 1,800 schools across Zambia over the next three years. The evidence-based program will reach approximately 280,000 students in grades 3-5 with support from USAID’s Development Innovation Ventures and the USAID Zambia Mission. Several additional partners provided support to the pilot and/or scale-up, including UNICEF, IPA, Pratham, VVOB, and Zambia Education Sector Support Technical Assistance.
GPI funds helped meet a critical policy window. The Ministry was interested in finding a remedial education (or "Catch Up") program, but did not know about effective approaches from randomized evaluations or have the flexible funding to adapt and monitor a Catch Up program to their context. By meeting this need, the Ministry was able to test the program for itself, monitor the implementation, and make decisions about how to scale it up. Other partners (such as USAID) were also watching the pilot and only decided to invest after they saw that this program could be successfully implemented in Zambia.
IMPROVED SYSTEMS FOR EVIDENCE USE
While individual evaluations and scale-ups are important, GPI's ultimate goal is to institutionalize the use of evidence in policymaking, making it more the norm rather than the exception.
GPI grants have contributed to several improved systems for data and evidence use that encourage, require, or incentivize greater generation and/or use of evidence in the policy process. Below are just three examples:
1. In Chile, the Ministry of Economy’s Innovation Fund worked with J-PAL LAC to create new system for reviewing potential investments requested by other departments within the government. The system requires applicants to include a theory of change and review of existing evidence from past impact evaluations when seeking funds. They also launched a new public bid for impact evaluations of the Innovation Fund’s investments and reserved a portion of program funds for evaluation.
“In the fund, this system created the perception that it is necessary to base yourself on evidence to make public policies.”
– Antonio Martner Sota, Research Coordinator, Innovation Fund for Competitiveness, Ministry of Economy, Development, and Tourism
2. In India, GPI is supporting staff working on J-PAL South Asia’s institutional partnership with the Government of Tamil Nadu to create an evidence-informed approach to policymaking across its departments. To date, the government has supported 15 policy-driven research projects across nine departments, using over US$4 million of its own funding and collaborating with more than 34 academic researchers. Recently, the state issued new guidelines requiring all programs in sectors of strategic importance with budgets greater than US$22 million undergo impact evaluations. The state’s Planning Department has also created a dedicated fund for program evaluation of US$1.5 million per year. Using funds from GPI, J-PAL South Asia staff also provide capacity-building support to a data analytics unit set up by the Government to allow them to make real-time decisions based on the administrative data they already collect.
3. In Peru, the Ministry of Women and Vulnerable Populations refined their approach to reducing domestic violence to focus more on violence prevention after J-PAL LAC created an evidence repository summarizing key findings from impact evaluations on reducing violence. As a result of this partnership, the Ministry is also currently preparing a randomized evaluation with J-PAL affiliated researcher Erica Field to evaluate the impact of a community program to prevent domestic violence.
"With this repository it was clear that one of the most important themes in the area of violence was prevention of violence. Our focus changed based on this evidence.”- Director, Monitoring and Evaluation, Ministry of Women and Vulnerable Populations
ONGOING PARTNERSHIPS
GPI has twenty ongoing partnerships, many of which have the potential to inform and improve policy in the future. For example, six of the 16 institutional partnerships supported by GPI have already contributed to an improved system for data and evidence use or led to a new evaluation on a government’s priority question.
Research partnerships often represent the first time a government or department is investing in evaluation, and often lay the groundwork for future collaborations that move beyond a single project. For example, the partnership described above with the Ministry of Rural Development in India began with a single evaluation in Bihar and contributed to a national reform after several years of collaboration.
What are we learning about building a culture of evidence use in government?
Our experience with these partnerships has created a rich base of internal knowledge on how to most effectively work with governments to apply evidence to policy in meaningful ways. We’re working to summarize and share this knowledge in an effort to further expand GPI’s impact in 2018. Stay tuned for our next post in this series to learn some early insights from our interviews with our government partners.
For more information about GPI, including opportunities for foundations and individuals to support this important work, please contact [email protected].
This is the first of two posts about GPI. Read the second post in this series.
One of our fundamental goals at J-PAL Latin America & Caribbean (LAC) is building government capacity to generate and use scientific evidence in order to achieve more effective public policies. But are policymakers and civil servants in Latin America interested in learning more about these topics? Last year we saw evidence that in Brazil, the answer is yes.
In partnership with ENAP (the National School of Public Administration), J-PAL LAC’s Training team and Brazil team developed a MOOC (Massive Open Online Course) on Impact Evaluation of Social Programs and Policies. This course is a Portuguese version of J-PAL 101x: Evaluating Social Programs (also available in Spanish and English).
The partnership took off when J-PAL LAC approached ENAP—which had 340,000 students throughout Brazil enrolled in their online courses in 2017—to develop a Portuguese-language course relevant for a Brazilian audience, particularly government officials. The idea became a reality thanks to funds from J-PAL’s Government Partnership Initiative (GPI).
In 2017, J-PAL LAC took charge of developing new content and adapting existing material to the Brazilian context. J-PAL affiliated professors Claudio Ferraz, Bruno Ferman, Leonardo Bursztyn, and Marco Gonzalez-Navarro, along with invited professor Cristine Pinto and J-PAL LAC staff, recorded videos for the course. Evaluations led by J-PAL affiliates were used to develop examples, exercises, and case studies. ENAP hosted our staff and professors, recorded the ten lectures, edited the videos, and uploaded the course on its online education platform.
Watch the course trailer video:
The 30-hour course is free and open to both civil servants and the general public. The course was piloted over six weeks in November and December 2017 and the inaugural class had 5,790 students enrolled, most of them government officials.
Participants in the course learn why evaluations matter and how evaluations can be used to rigorously measure the impact of social policies and programs. Through a combination of lectures and case studies from real-world evaluations, topics of discussion include the benefits of measuring impact and methods of randomized impact evaluations; techniques for determining an appropriate sample size and evaluation design; threats that can undermine the integrity of evaluation results; and strategies for translating evaluation results into policy action.
The course is designed for people with a variety of backgrounds: managers and researchers from governments, foundations, and non-governmental organizations, as well as trained economists seeking to develop new knowledge in randomized evaluations. This made the course a good fit for Brazilian civil servants.
Luciana Ferreira de Almeida, who works at the Secretary of Budget and Planning at the State of Rio de Janeiro, took the course and also disseminated the opportunity among her colleagues in government.
"My interest in the course came from the perception that we have to create a culture of evaluation and evidence in the State to guide policy", Luciana said. "The government has the duty of directing public resources in the most effective way, strengthening and scaling up programs that really make a positive impact for people. However, very often the selection of programs is based on impressions rather than evidence, which results in waste of money in a State in fiscal crisis."
Approximately 25 percent of the enrolled cohort, or 1,488 students, completed the course—one of the highest completion rates to date among online courses offered by J-PAL—indicating that impact evaluation is a topic of interest among Brazilian civil servants.
Among the students who finished the course is Ronaldo Souza, director at the Secretary of Evaluation and Information Management (SAGI) at the Ministry of Social Development in Brazil.
"The course material is comprehensive and updated. The use of Brazilian and Latin American case studies get the theory closer to our reality," Ronaldo said. "The need to develop evaluations of social programs should be widely disseminated, not only to show positive impacts, but also to optimize the design, make corrections and scale up the access of programs based on this process."
This is just the beginning. In 2018, the online course will be offered regularly as part of ENAP’s academic calendar, and as a result, more government officials and citizens will have the opportunity to learn about the importance of measuring impact and using rigorous evidence to improve social programs.
We hope that the course contributes to fostering a culture of evidence and evaluation inside the Brazilian government. Watch the course preview and see more information about future courses.