Replication support for J-PAL evaluations now underway
Over the next three years, J-PAL plans to conduct 36 re-analyses of data from randomized evaluations in collaboration with the original researchers. This work aims to make academic research more reproducible and to reward researchers for conducting replications, which strengthens the reliability of the results. A new team of J-PAL graduate student fellows will trace the steps of the publishing researchers by using the original data to rewrite the analysis code and determine whether the primary results can be replicated. This collaborative approach occurs before work is published, to avoid mistakes and contribute to the credibility of results. Fellowships are supported through a grant from the Alfred P. Sloan Foundation and the Laura and John Arnold Foundation, and contribute to J-PAL’s broader goals to promote research transparency and reproducibility.
We asked J-PAL’s first research transparency graduate fellows, Cesar Huaroto and Odyssia Ng, about their interest in research transparency and their experiences in research more broadly.
Why are you interested in research transparency?
Cesar: I think that transparency is critical in order to increase trust between academia and policymakers. In development economics, there have been many important improvements on research design but there is still work to do on other aspects, such as transparency.
Odyssia: Research in development economics can have a tremendous impact on real policies in developing countries. We must therefore ensure that research findings are of the utmost quality and that their limitations are clearly outlined. Transparency is necessary to achieve these goals
What got you interested in development economics?
Cesar: I am from a developing country [Peru] and I have seen the consequences of poverty and inequality. I think that development economics is gaining more space to improve society and help policymakers make adequate choices, and I want to help with that.
Odyssia: The potential to come up with solutions to solve real world problems affecting the poorest of the poor is what got me interested in development economics. An undergraduate class I took at UC Berkeley introduced me to the world of RCTs, and I learned the importance of rigorous impact evaluations to improve the effectiveness of anti-poverty programs.
What is your "dream evaluation"? (It doesn't have to be feasible!)
Cesar: I think that the most important word for an evaluation is “usefulness”. The results may be disappointing or the implementation might have had many unforeseen problems, but there are many things that can be done by learning from those experiences too. Evaluation is very helpful in that sense, since it gives us objective feedback.
Odyssia: The topic RCTs cannot easily address is the role of political institutions and political leaders in development—we shouldn't randomize political regimes across countries! But if we could, determining how different political regimes and institutions affect economic development would help answer many long-standing questions in development economics.
What is your most memorable story from the field?
Cesar: When I was working on the collection of data of a longitudinal survey (more than 8 years after the baseline) from a poor family in rural Peru I had the chance to visit them in the company of a surveyor who also interviewed them on the first round. It was a magical experience since only by listening to their conversation was I able to understand what the more important changes were for this family in those years. There I realized the power of experience and communication in research.
Odyssia: I have many stories, but my most memorable day in the field was my first one. On our way from Accra to the field, my team and I stopped by the city's slums to pick up our driver. Although I had spent years learning about people with no access to water or electricity, I was still extremely struck by the difference in living conditions, and kept trying to make sense of what I was seeing. It is one thing to read extensively about extreme poverty, and another to see it for yourself.
What are your future goals (in relation to research transparency and development economics)?
Cesar: I aim to increase the knowledge and experience of my colleagues and students about the important role of replication and adequate research (pre-analysis plans, etc.). Also, I would like to improve the relationship between academics and policymakers by making research more transparent and accessible for them.
Odyssia: My future goal is to use the knowledge and tools I have gained from my doctoral studies to help design, implement and evaluate anti-poverty programs in the developing world. I would like to promote a tighter link between the insights we gain from research and the way these are used by development practitioners in the field.
