A year in blogs: Reflecting on 2019 through the eyes of staff, affiliates, and partners

As 2019 comes to a close, we reflect back on key accomplishments, lessons learned, and the partnerships that have made it possible to advance J-PAL North America’s mission of reducing poverty in the region. From compelling new evidence and reviews of over 160 rigorous evaluations, to the Evaluation Toolkit and building the capacity of policymakers, all the accomplishments we have spotlighted have been made possible through our partnership with the broader evidence-based policymaking community. Together with our committed network of state and local governments, implementing partners, funders, researchers, and study participants—without whom none of this would be possible—we have continued working to advance the evidence-based policymaking movement. And we’ve learned a lot along the way.

Photo: Shutterstock.com

Catalyzing rigorous research

At J-PAL North America, we are passionate about tackling critical questions on poverty alleviation through rigorous research. We want to know what works and why and focus on the most policy-relevant questions. This year, we launched twelve new randomized evaluations and nine pilot studies. In addition to catalyzing new studies, completed studies by J-PAL affiliates produced compelling results that shed light on some of today’s biggest policy questions. Several completed studies are highlighted below.

Research suggests that growing up in high-poverty neighborhoods negatively affects a number of important life outcomes. This August, preliminary results from a rigorous evaluation of the Creating Moves to Opportunity (CMTO) project found that access to housing mobility services increased the share of families who moved to higher-opportunity neighborhoods by 40 percentage points. These results suggest that improving housing choice may be an effective strategy to reduce housing segregation.

To evaluate whether workplace wellness programs, which have dramatically increased in popularity in recent years, actually promote healthy behaviors and reduce medical spending, J-PAL affiliated researchers conducted randomized evaluations of two programs in the United States. Results showed that the programs improved a small subset of health behaviors, but had no impact on health, health care spending and utilization, or employment-related outcomes. This was possibly due to the fact that employees most likely to benefit (those with poor health habits) participated at a lower rate.

Health care spending accounts for almost one-fifth of the US economy. In 2014, Medicare spent over $4 billion on high-cost diagnostic imaging alone—research suggests that up to 30 percent were unnecessary. To reduce inappropriate imaging, beginning in January 2021, Medicare will no longer reimburse providers for high-cost scans unless they are ordered using a clinical decision support (CDS) system. With the impending mandate in mind, a randomized evaluation was conducted on the impact of a CDS system on the number of high-cost medical imaging orders in Wisconsin and Illinois. The CDS system reduced the number of scans targeted by the software, but did not impact the total number of low- or high-cost scans.

To address the many barriers to completing community college, Chicago nonprofit, One Million Degrees (OMD), provided wraparound support to low-income community college students using a model that targets academic, professional, personal, and financial needs of students. Early findings from a randomized evaluation found that program participants experienced a 25 percent increase in full-time enrollment, a 35 percent increase in persistence, and a 47 percent increase in full-time persistence. The effects of OMD predict positive program effects on degree attainment.

Photo: Shutterstock.com

In 2019, J-PAL North America also focused on launching a body of research in new critical areas, including housing and homelessness, crime and violence prevention, and work of the future.

To expand the base of rigorous evidence on strategies that reduce and prevent homelessness and promote housing stability, we are supporting randomized evaluations on the impact of several interventions, including targeted financial assistance, eviction diversion, and rapid re-housing, on homelessness. Similarly, our crime and violence prevention work supports randomized evaluations of strategies that aim to increase the equity and functionality of the criminal justice system. One recent study assesses the impact of various types of text message reminders on rates of failure to appear in court.

Technology is rapidly changing the nature of work in North America and has potentially profound implications for workers and the labor market. Launched in April of this year, the Work of the Future Initiative supports randomized evaluations of strategies and innovations that address these changes in a way that benefits workers and the overall economy. A research agenda identifies four key areas where research is needed, including programs designed to reskill workers, strategies to improve traditional and non-traditional educational institutions, interventions to facilitate worker transitions between occupations and industries, and more.

Photo: J-PAL

Translating Research into Action

In addition to catalyzing new research, J-PAL North America works to ensure rigorous research conducted by our affiliated network helps policymakers make evidence-based decisions. We do this by synthesizing research findings, identifying policy insights that decision-makers can apply, and promoting promising programs for scale. Our staff members have also traveled to over ten states and presented at over eight conferences to share evidence with leading decision-makers.

Recent years have seen widespread buzz and excitement around the potential for technology to help students learn. But which programs actually deliver? To shed light on the effectiveness of various education technology interventions, J-PAL North America published the Education Technology Evidence Review, which summarized key findings and policy lessons from 126 rigorous studies that assessed the impact of technology-based nudges, social psychology interventions, online and in-person instruction, and educational software, and more. Key lessons include that educational software programs designed to help students develop particular skills have shown enormous promise in improving learning outcomes, particularly in math, while initiatives that expand access to computers and internet alone generally do not improve K-12th grade students’ grades and test scores.

More than 500,000 people experience homelessness on a given night and 1.4 million people pass through emergency shelters in a given year in the United States. The last several decades have seen rigorous evaluation of interventions like the Housing First approach resulting in significant policy changes. To draw key policy lessons from this body of research, J-PAL North America published an Evidence Review on Reducing and Preventing Homelessness that highlights key findings from forty rigorous evaluations on policies that seek to provide housing stability and identifies important questions for future research to address. Key insights include that comprehensive interventions providing a range of assistance can prevent homelessness among families at risk of losing their homes, permanent supportive housing increases housing stability for individuals with severe mental illness and veterans experiencing homelessness, and Housing Choice Vouchers help low-income families avoid homelessness and maintain housing stability.

With the goal of translating research into policy action, J-PAL North America’s Evidence to Scale team also systematically reviewed our evidence base and identified three programs to prioritize for scale-up, with the aim of improving many more lives in the years to come.

Photo: J-PAL

Building Researcher and Practitioner Research Capacity

One of J-PAL North America’s core activities is capacity building—equipping researchers and policymakers with the tools to be better producers and users of evidence. In 2019, our team developed research resources, including the Evaluation Toolkit, and built the capacity of state and local governments through multiple trainings on rigorous evaluations of programs and policies.

Drawing from our own lessons learned and guidance from researchers and research organizations, J-PAL North America developed an Evaluation Toolkit, which provides practical guidance for designing, implementing, and communicating about evaluations. The Toolkit provides information relevant to both those who are only beginning to think about conducting a randomized evaluation to those that are already in the weeds of one. In addition to written research resources, J-PAL North America expanded our Short-Term Research Management (STReaM) program: this year we supported six projects.

This year, J-PAL North America worked closely with state and local leaders to build evaluation capacity. In partnership with the Pew-MacArthur Results First Initiative, J-PAL North America staff members conducted a training series for state and local policymakers interested in rigorously evaluating their programs and policies. Topics covered included concepts related to measuring impact and practical design considerations that go into randomized evaluations of programs and policies. To strengthen partnerships and share lessons on communicating research results, the State and Local Innovation Initiative also hosted the Building Evidence, Advancing Policy, Impacting Lives Convening, which brought together over seventy government leaders, researchers, and funders. Eight state and local governments presented on randomized evaluations currently under development, received feedback from research and policy experts on their research plans, and discussed ways to communicate research results to produce meaningful policy impact.

As illustrated by the policy-relevant studies in The Lessons of Administrative Data Brief, administrative data can help reduce research costs, offer opportunities for long-term follow-up on intervention impacts, and improve research accuracy. In 2019, J-PAL North America provided practical guidance to researchers and policymakers interested in using administrative data, through resources like our Administrative Data Catalog and Evaluation Toolkit. This work complements the Innovations in Data and Experiments for Action (IDEA) Initiative, an effort led by J-PAL Global to increase the use of administrative data in research.

Thank you for your partnership in 2019 and for helping to make this work possible. Together, we are improving the lives of those who experience poverty in North America, and we look forward to continuing this important work together in 2020.

An earlier version of this blog was posted in The Week. 

More than a decade ago, I met Abhijit Banerjee and Esther Duflo to discuss joining a small center they had started at the Massachusetts Institute of Technology (MIT) called PAL, or the Poverty Action Lab (it was subsequently renamed J-PAL to recognize a generous endowment to MIT by its alum Mohammed Jameel). After my Master’s at the Delhi School of Economics, I had joined the IAS, attracted by the incredible satisfaction of working in development policy in the field. But I was often puzzled at how so many of our key decisions were not based on any semblance of rigorous evidence. After a few years working in the private sector, I was itching to come back to my roots. Talking to them in J-PAL’s three-room office with less than 10 staffers back then, I was drawn to the huge potential of their very simple vision: What if policymakers in governments, NGOs, and foundations could base their decisions on scientific evidence rather than instincts, ideology, or inertia? 

Even though J-PAL was then only five years old, pioneers like Abhijit, Esther, Michael Kremer, and Rohini Pande (both also J-PAL affiliates) had already launched a number of field research studies in India and Kenya that would cement the role of randomized evaluations, or RCTs, as an incredibly important tool in the field of economics to understand what poverty alleviation programs work or not, and why. And J-PAL, along with our partner organization Innovations for Poverty Action (IPA), had begun to work with a number of other economists to run their research in many countries to expand the knowledge base of what works or not in global development. The dilemma they faced when I met them was even more challenging: How do we make sure that the results of this research do not just stay in academic journals but actually make their way to the policymakers who need it most in their decision making? How could such evidence inform programs that could be scaled up to reach millions?

I am thrilled that Esther and Abhijit, along with longtime J-PAL affiliate Michael Kremer, were just awarded the 2019 Nobel Memorial Prize in Economic Sciences for “their experimental approach to alleviating global poverty.” But while a lot of attention the past few days has been focused on their experimental methodologies, the policy action that has resulted from their research is equally significant—the Nobel award specifically mentions their work to fight poverty in practice, stating that “as a direct result of one of their studies, more than five million Indian children have benefitted from effective programmes of remedial tutoring in schools. Another example is the heavy subsidies for preventive healthcare that have been introduced in many countries.” The brilliance of Abhijit and Esther was in recognizing that this new research that they and a generation of amazing colleagues was producing would not by itself be sufficient to reach policymakers; more would need to be done to actually make a difference in the lives of millions.

Their response was to found and put their considerable political capital into not just raising funds for J-PAL but to recruit talented staff and provide the creative vision for it to be that agent of change. Over the past decade, J-PAL has moved way beyond just being a research center at MIT, but to take on work on the policy outreach and capacity building sides. We were able to do so thanks to the support of a coalition of additional funders such as the Gates, Hewlett, and MacArthur foundations and the Omidyar Network.

One of J-PAL's first partnerships with a state government in India was with Mr. Modi. This picture shows the signing of an MoU with the Government of Gujarat by John Floretta (Director of Policy, J-PAL).

As a result, J-PAL not only supports innovative research worldwide, but also forms strong partnerships with governments, NGOs, and donors to ensure that the knowledge generated by this research is understood and used to address the trickiest challenges of global poverty. This includes, for example, encouraging the use of preventive health products or tailoring instruction to students’ learning levels to increase learning. We also support governments and NGOs in scaling up programs found to be most effective.

Finally, J-PAL staff also help build the capacity of government officials and practitioners to understand and use evidence through customized in-person trainings around the world. To make this knowledge even more accessible, Abhijit and Esther lead a push to create a suite of online courses hosted by MIT’s edX platform that now reaches thousands of learners worldwide.

At the core of our work are our regional offices, which implement our affiliates’ field projects, conduct trainings, and work with policymakers to make sure they have all the information they need to move toward a culture of evidence-informed policy. Our first regional office was established at the Institute for Financial Management and Research (IFMR) in India to support J-PAL’s work in South Asia in 2007. Five more regional offices have since been established to support ongoing research, policy outreach, and training activities in Europe (at the Paris School of Economics in France), Latin America and the Carribean (at the Pontifícia Universidad Católica in Chile), Africa (at the University of Cape Town in South Africa), Southeast Asia (at the Institute for Economic and Social Research at the University of Indonesia), and North America (at MIT in the United States).

Investing in this infrastructure has paid off. J-PAL’s affiliated professors have designed and carried out almost 1,000 randomized impact evaluations to test and improve the effectiveness of social programs in more than 50 countries. J-PAL’s research covers traditional sectors such as agriculture, education, or health, as well as newer, cross-cutting sectors such as crime and violence, firms and the private sector, or gender. On the policy front, scale-ups of programs evaluated and found to be successful have to-date reached 400 million people worldwide, including the Teaching at the Right Level approach with the Indian NGO Pratham and an identity card program for a massive food subsidy program in Indonesia.

As global poverty continues to affect hundreds of millions worldwide, our work is more urgent now than ever. J-PAL is responding to this critical need by launching cutting-edge initiatives like the Innovations in Data and Experiments for Action Initiative (IDEA). IDEA supports governments, firms, and non-profits who want to make their administrative data accessible; analyze it; and partner with researchers to design innovative programs, evaluate program impact through randomized experiments, and scale up successful programs.

Abhijit, Esther, and Michael receiving this year’s Nobel Prize in Economics is a recognition of the creativity and value of the methodology of randomized evaluations. But it is equally a recognition of how the outputs from this methodology can improve the lives of millions of people worldwide. This required not just the genius of the researchers but the hard work of thousands of field research, policy, and training staff, the commitment of hundreds of governments and NGOs to evaluate and question the impact of their own programs, hundreds of thousands of program participants who also agreed to be surveyed, and the support of visionary funders. 

In this journey, I feel particularly fortunate to have discovered two incredible human beings as my dear friends. Abhijit is one of the smartest people I have ever met, and not just at the weekly MIT-Harvard development seminars where he has this uncanny ability to see three steps ahead of most people in the room as to where the presenter’s model is headed, all while multi-tasking with his email and sometimes even phone calls! But more importantly for me, this also extends to discussions with him on policy or field implementation issues—his ability to see the big picture, predict how people will respond, and suggest solutions has been invaluable not just for J-PAL but also for our partner organizations. For the past five years, Abhijit and I have been co-Chairs of J-PAL’s Innovations in Government Initiative (IGI) that supports creative partnerships between governments, researchers, and donors to find innovative solutions to key policy challenges worldwide. Every single one of those projects, whether in Mexico, India, or Kenya has benefited from his insights.  

And Esther is one of the most relentless, hard working, and visionary persons I have met. She never accepts something as “not possible” and challenges herself and everyone around her to think out of the box and aim higher. Thanks to her, we have applied for moonshot multi-million dollar grants that helped us fund a generation of research projects in areas as diverse as measurement of corruption, strengthening democracy, or improving delivery of urban services. But Esther’s abilities don’t just stop at big-picture thinking. She has outstanding management instincts. For the past eight years, Esther and I have been co-Scientific Directors of J-PAL’s largest regional office for South Asia, which runs dozens of projects in India in partnership with state governments like Gujarat (launched when Prime Minister Modi was the Chief Minister), Haryana, Punjab, and Tamil Nadu. The office also runs capacity building work with various government departments. As you can imagine, this creates numerous day-to-day management and operational challenges. Despite being so busy with her own research and editing responsibilities, Esther manages to find time to help brainstorm and provide solutions that are as creative as her research!

I remember coming home from that first meeting with Abhijit and Esther over a decade ago and saying to my wife and fellow economist Gita Gopinath that “this is a Nobel-winning idea, but it is going to require a lot of work and a long time.” Her response was, “With people like Abhijit and Esther, you never know, it might be sooner than you think.” She could not have been more right!

Today, J-PAL and IPA are excited to announce the creation of the new Datahub for Field Experiments in Economics and Public Policy. With this new dataverse, we will be housing all datasets published on The Dataverse Project by our respective organizations in one centralized location. 

Our organizations have made impressive strides in recent years in publishing data: we have made available more than 140 datasets from studies about poverty and development conducted by our researchers around the world. This joint effort will bring together our two dataverses. In doing so, we aim to facilitate the search and discovery of new and existing datasets pertaining to field experiments carried out by our organizations and affiliated researchers. 

This dataverse will strengthen IPA and J-PAL’s commitment to research transparency, specifically data publication, and facilitate the reuse of research data. It will also enable us to increase our efforts in building more consistent and robust metadata. The dataverse is open to J-PAL/IPA partner organizations interested in joining and supporting further expansion of these efforts.

Anyone can visit the Datahub to search for and download existing datasets to support their own research or replicate existing research. Most datasets include original data from the study; programming code written to run the analysis; instructional documentation on the data, code, and variables; and the survey documents used to collect the data. We outlined a few reasons why researchers should publish their data in a previous blog post earlier this year.

Some examples of published data include:

Recently published

• Measuring Productivity: Lessons from Tailored Surveys and Productivity Benchmarking. Data and code for a study that explores different ways to measure productivity. It uses detailed data from 219 rug-making firms in Fowa, Egypt collected from 2010 to 2014 to explore how well these firms produce output, revenue and quality. They document how these varied productivity measures compare to the most commonly used measures in the literature.

• Clinical Decision Support for High-Cost Imaging: a Randomized Clinical Trial. Data collected from health providers who ordered low-cost imaging scanners from November 2015 to October 2016. The code analyzes the effect of a software that provides information on high-cost images on the amount of scan orders by health providers.

• Follow the money not the cash: Comparing methods for identifying consumption and investment responses to a liquidity shock Data, code and surveys used for data collection to measure the impacts of liquidity shocks on spending in a 2010-2015 study out of the Filipino cities Manila and Luzon. 

• Getting to the Top of Mind: How Reminders Increase Savings Data and code for the publication tables are included for this study on the ability of reminder messages to increase commitment attainment for recently opened savings-account holders. This study was conducted from 2006-2008 in a mix of rural and urban areas within the Philippines, Bolivia and Peru. 

Most downloaded

• The Diffusion of Microfinance. A dataset containing information on the demographic and social network in 43 villages from 2006 to 2007 in South India before and after an introduction of microfinance. The code is used to answer how participation in a microfinance program diffuses through social networks.

• Measuring the Impact of Microfinance in Hyderabad, India. A dataset that provides information on 2,800 households living in slums in Hyderabad, Andhra Pradesh from 2005 to 2007. Information was collected on household composition, education, employment, asset ownership, decision-making, expenditure, borrowing, saving, and any businesses currently operated by the household or stopped within the last year.

• A Multifaceted Program Causes Lasting Progress for the Very Poor: Evidence From Six Countries A dataset that provides results from six randomized control trials of an integrated approach to improve livelihoods among the poor in 6 countries: Peru, Honduras, Ghana, Ethiopia, Pakistan and India from 2007-2014. Information was collected on consumption, food security, assets, finance, time use, income and revenues, mental health, and women’s decision-making. 

• Underinvestment in a Profitables Technology: the case of Seasonal Migration in Bangladesh A dataset containing information on the results of a 2008-2011 financial intervention in northwestern Bangladesh intended to promote out-migration to nearby urban areas during the lean season before harvest in order to mitigate famine. 
   

Explore the Datahub >>