What Have We Learned? Improving Development Policy Through Impact Evaluation
CGD and J-PAL take stock of the evidence and impact evaluation movement and its promise for improving social policy in developing countries.
In 2006, CGD released a working group report titled "When Will We Ever Learn? Improving Lives Through Impact Evaluation." It described an evaluation gap and proposed an international effort to systematically build evidence on “what works” in development with the aim of improving the effectiveness of social programs. Ten years later, this event reflected on progress toward these goals. Despite challenges, impact evaluations have generated knowledge about poverty and public policy which have led to better programs. Hundreds of millions of people across the world have benefited from programs that have been designed based on the findings of rigorous evaluations.
View a recording of the event:
Sessions
Opening Remarks
Amanda Glassman, Chief Operating Officer and Senior Fellow, Center for Global Development
Amanda Glassman reflected on the exponential growth in the use of rigorous evidence over the past decade and highlighted challenges for further consideration.
Keynote Address: Shifting Opinions, Changing Practices
Abhijit Banerjee, Ford Foundation International Professor of Economics, MIT and Director, J-PAL
Abhijit Banerjee discussed some of the ways in which evidence generated from impact evaluation has shifted broad policy conversations including microcredit, pricing of preventive health products, building stable livelihoods for the ultra-poor, and education reform. Abhijit stressed the importance of building a theoretical framework to inform policy change and ensuring that impact evaluations are “intellectually exciting” and test these theories empirically.
Panel I: Learning Together—Shaping Policy through Partnership
Moderator: Rachel Glennerster, Executive Director, J-PAL
Using Identification Cards to Improve National Social Assistance in Indonesia
When resources are constrained, how can governments ensure that they are delivering social protection programs in the most cost-effective way? Researchers partnered with the Indonesian government to design and test how the information printed on identification cards and the channels throught which the cards were distributed affected program outcomes. Government-issued identification cards, in combination with community outreach and advertisements, improved households’ access to a subsidized rice program, and have subsequently been used as a means for beneficiaries to access cash transfers, as well. Overall, the scale-up of these findings have improved access to targeted social programs for 66 million people.
The policymaker: Bambang Widianto, Executive Secretary of TNP2K (National Team for Acceleration of Poverty Reduction, Office of Vice President), Government of Indonesia (via video recording)
The researcher: Rema Hanna, Jeffrey Cheah Professor of South-East Asia Studies, Harvard Kennedy School and Scientific Director, J-PAL Southeast Asia
View a recording of Bambang Widianto and Rema Hanna discussing this research partnership and its policy implications:
Enhancing Delivery through Financial Reform of India’s Largest Social Program
Social programs in developing countries are often hindered by corruption, especially by the flow of funds from the central government to beneficiaries. For the Mahatma Gandhi National Rural Employment Guarantee Scheme (MGNREGS) in India, the central government disburses funds to local governments based on projected spending. In Bihar, policymakers and researchers tested the effect of an information technology reform that linked the flow of funds to actual expenditures and reduced the number of officials involved in the process. This research informed the approval a national reform in 2015 and, to date, 21 states have adopted new systems for making MGNREGS wage payments directly to beneficiaries.
The policymaker: Santhosh Mathew, Joint Secretary, Ministry of Rural Development, Government of India
The researcher: Abhijit Banerjee, Ford Foundation International Professor of Economics, MIT and Director, J-PAL
View a recording of Santhosh Mathew and Abhijit Banerjee discussing this research partnership and its policy implications:
Improving Education Outcomes in Kenya
Although great gains have been achieved for school enrollment, education outcomes such as literacy and numeracy skills have proven more difficult to improve. Researchers partnered with Kenya’s Ministry of Education, Science, & Technology to develop the Kenya Primary Math and Reading (PRIMR) Initiative, which uses evidence-based instructional improvement methods to increase students’ basic literacy and numeracy skills. This initiative has been applied to more than 23,000 schools to reach 5.2 million children across Kenya.
The policymaker: Darius Mogaka, Director of Policy, Partnerships and East African Community Affairs, Ministry of Education, Science, & Technology, Government of Kenya (via video recording)
The researcher: Amber Gove, Director of Research, RTI International
View a pre-recorded statement by Darius Mogaka discussing this research partnership and its policy implications:
Panel II: What Have We Learned about Investing in Impact Evaluation and Evidence?
Moderator: Justin Sandefur, Senior Fellow, Center for Global Development
Funding organizations can serve a catalytic role in investing in impact to evaluation and scaling up programs informed by evidence. Representatives from several donors shared what they have learned about investing in evidence and possible directions for the future. They discussed opportunities for additional funding, as well as the usefulness of expanding support for complementary activities such as small-scale pilots and technical assistance for government-led scale-ups.
Amanda Glassman, Chief Operating Officer and Senior Fellow, Center for Global Development
Markus Goldstein, Lead Economist, Africa Gender Innovation Lab, World Bank
Emmanuel Jimenez, Director, International Impact Evaluation Initiative (3ie)
Michael Kremer, Gates Professor of Developing Societies in the Department of Economics, Harvard University and Scientific Director, Development Innovation Ventures (DIV), USAID
Announcement of Evaluation Award
William Savedoff, Senior Fellow, Center for Global Development
Donor organizations have played a catalytic role in the support for and use of rigorous evidence. However, impact evaluations continue to account for a relatively small portion of overall development research. William Savedoff shared findings from his review of impact evaluation funders and announced the creation of an evaluation award to commend progressive donor organizations for their support of evidence-based policy.
Closing Remarks
Rachel Glennerster, Executive Director, J-PAL
Rachel Glennerster reflected on the day’s discussions, highlighting a few major takeaways. First, not every evaluation will work or lead to policy change, but a “few big wins” can justify an entire research portfolio. Second, donors can play a catalytic role, and open, flexible support structures can unleash innovation and encourage bottom-up collaboration between policymakers and researchers. Rachel reiterated the substantial opportunities for future research, as well as the need to continue to ensure that existing learning informs policy decisions around the world. The ecosystem of organizations focusing on these efforts continue to grow, and Rachel invited those in attendance to continue this conversation to ensure another successful decade of promoting evidence-based policymaking.