Replication support for J-PAL evaluations now underway

February 6, 2017

Over the next three years, J-PAL plans to conduct 36 re-analyses of data from randomized evaluations in collaboration with the original researchers. This work aims to make academic research more reproducible and to reward researchers for conducting replications, which strengthens the reliability of the results. A new team of J-PAL graduate student fellows will trace the steps of the publishing researchers by using the original data to rewrite the analysis code and determine whether the primary results can be replicated. This collaborative approach occurs before work is published, to avoid mistakes and contribute to the credibility of results. Fellowships are supported through a grant from the Alfred P. Sloan Foundation and the Laura and John Arnold Foundation, and contribute to J-PAL’s broader goals to promote research transparency and reproducibility.

We asked J-PAL’s first research transparency graduate fellows, Cesar Huaroto and Odyssia Ng, about their interest in research transparency and their experiences in research more broadly.

Why are you interested in research transparency?

Cesar: I think that transparency is critical in order to increase trust between academia and policymakers. In development economics, there have been many important improvements on research design but there is still work to do on other aspects, such as transparency.
Odyssia: Research in development economics can have a tremendous impact on real policies in developing countries. We must therefore ensure that research findings are of the utmost quality and that their limitations are clearly outlined. Transparency is necessary to achieve these goals

What got you interested in development economics?

Cesar: I am from a developing country [Peru] and I have seen the consequences of poverty and inequality. I think that development economics is gaining more space to improve society and help policymakers make adequate choices, and I want to help with that.
Odyssia: The potential to come up with solutions to solve real world problems affecting the poorest of the poor is what got me interested in development economics. An undergraduate class I took at UC Berkeley introduced me to the world of RCTs, and I learned the importance of rigorous impact evaluations to improve the effectiveness of anti-poverty programs.

What is your "dream evaluation"? (It doesn't have to be feasible!)

Cesar: I think that the most important word for an evaluation is “usefulness”. The results may be disappointing or the implementation might have had many unforeseen problems, but there are many things that can be done by learning from those experiences too. Evaluation is very helpful in that sense, since it gives us objective feedback.
Odyssia: The topic RCTs cannot easily address is the role of political institutions and political leaders in developmentwe shouldn't randomize political regimes across countries! But if we could, determining how different political regimes and institutions affect economic development would help answer many long-standing questions in development economics.

What is your most memorable story from the field?

Cesar: When I was working on the collection of data of a longitudinal survey (more than 8 years after the baseline) from a poor family in rural Peru I had the chance to visit them in the company of a surveyor who also interviewed them on the first round. It was a magical experience since only by listening to their conversation was I able to understand what the more important changes were for this family in those years. There I realized the power of experience and communication in research.
Odyssia: I have many stories, but my most memorable day in the field was my first one. On our way from Accra to the field, my team and I stopped by the city's slums to pick up our driver. Although I had spent years learning about people with no access to water or electricity, I was still extremely struck by the difference in living conditions, and kept trying to make sense of what I was seeing. It is one thing to read extensively about extreme poverty, and another to see it for yourself.

What are your future goals (in relation to research transparency and development economics)?

Cesar: I aim to increase the knowledge and experience of my colleagues and students about the important role of replication and adequate research (pre-analysis plans, etc.). Also, I would like to improve the relationship between academics and policymakers by making research more transparent and accessible for them.
Odyssia: My future goal is to use the knowledge and tools I have gained from my doctoral studies to help design, implement and evaluate anti-poverty programs in the developing world. I would like to promote a tighter link between the insights we gain from research and the way these are used by development practitioners in the field.

Cesar Huaroto is currently a PhD student on Economics at Universidad Nacional de La Plata (UNLP) and holds master's and a bachelor's degree in economics from Pontificia Universidad Católica del Perú (PUCP). He has previous experience both in academia and in the public sector. He has recently been working as an Evaluation Specialist at the MineduLab within the Ministry of Education in Peru, the first institution in Latin America that collaborates with external researchers to develop low-cost randomized evaluations on education policy. Additionally, he is co-author of a replication study funded by 3ie. He has previously worked in the Ministry of Development and Social Inclusion (MIDIS) of Peru and different research centers in Peru such as Group for Analysis for Development (Grade) and the Peruvian Studies Institute (IEP), where he worked as an Assistant Researcher.
Odyssia Ng is a PhD student in the Department of Economics at Stanford University, studying development economics. She has always been fascinated by randomized evaluations, enjoys field work, and has worked on impact evaluations in Kenya and Ghana. Her research focuses on microfinance contracts for farmers in Western Kenya. Before her studies at Stanford, Odyssia obtained her Bachelor's degree in Economics and Mathematics at the University of St. Andrews.

Learn more about J-PAL’s three-year goals for improving research transparency and explore J-PAL’s related resources >>