Partnering with J-PAL North America: Policymaker and Practitioner Perspectives

December 13, 2016

J-PAL North America aims to reduce poverty by ensuring that policy is informed by scientific evidence. Our affiliated professors and staff work with implementing partners throughout the region, including nonprofits and governments, to design and carry out randomized evaluations that can answer critical questions in the fight against poverty.

Our partnerships start by identifying the priority questions our partners most want to answer. We then work together to design high-quality, feasible randomized evaluations. J-PAL North America also invests in building long-term collaborations with our partners, in order to help them build their capacity for creating evidence and using the results to inform their decision-making.

We asked two of our partners to share why they partnered with J-PAL North America and what they've learned in the process.

Benefits Data Trust (BDT)

Benefits Data Trust (BDT), a nonprofit partner, is collaborating with J-PAL North America to identify effective outreach strategies to enroll low-income households into benefits. In 2014, BDT connected with J-PAL North America staff and affiliated researchers Amy Finkelstein (MIT) and Matthew Notowidigdo (Northwestern) about conducting an evaluation to better understand the impact of BDT’s work. BDT, J-PAL North America, and the researchers agreed that, at that time, there was little rigorous evidence available to address a key policy question: What interventions are most effective at helping unenrolled but eligible households successfully apply for the Supplemental Nutrition Assistance Program (SNAP)?

BDT and J-PAL North America worked together to design and implement an ongoing randomized evaluation of the effect of informational mailings and application assistance on SNAP enrollment in Pennsylvania. Researchers will study the effect of two different interventions—a low-intensity informational mailing and high-intensity outreach with SNAP application assistance—on SNAP enrollment. Within each intervention group, the researchers will also study different variations in the design and wording of the mailings. The knowledge generated by the evaluation will help BDT understand which outreach activities are most effective at connecting eligible households to SNAP and will enable BDT to better target its efforts in the future.

Rachel Cahill, Director of Policy at BDT, described her experience partnering with J-PAL North America to design an evaluation that will answer important questions about BDT’s work:

Our partnership with J-PAL has provided tremendous value in terms of learning about and improving our program. The researchers helped us design new outreach letters for the evaluation based on marketing and psychology literature, which very quickly generated an increased response rate.  The improvement was significant enough that we did not even wait until the end of the evaluation; we thought about how to incorporate this letter design in other states besides Pennsylvania immediately. The research team has also donated their time to provide technical assistance on other data analyses not directly connected to the study.

Through this partnership, we have learned a lot about doing research, which will make us smarter in the future. For example, we had to think creatively about the design of the evaluation, which includes a high-intensity group, a low-intensity group, and a control group. Although a design that included a dozen different types of outreach letters would allow us to explore the effect of many different variations on the outreach strategy, there was a limit to how many different treatment arms we could manage with fidelity.

Our hypothesis is that the high-touch intervention will increase take-up more than the light-touch intervention and certainly more than the control group. Many of the people we talk to know that SNAP exists. They just can’t imagine going through all the paperwork and enrollment procedures to access the benefits they qualify for. If we can demonstrate that the high-touch intervention increased take-up more than the letters alone, the evaluation could shift what we see as a common misperception that low program take-up is just an issue of awareness.

Ginger Zielinskie, President of Benefits Data Trust, described the value of the organization’s partnership with J-PAL.

Benefits Data Trust’s model of outreach and application assistance offers an efficient and cost-effective means of enrolling low-income individuals into the benefits for which they are eligible.  The J-PAL randomized evaluation will provide an added level of rigor to optimize our outreach strategies, which in turn could strengthen our case to partners and potential partners.

South Carolina Department of Health and Human Services (SCDHHS)
Through our U.S. Health Care Delivery Initiative, J-PAL North America is partnering with the South Carolina Department of Health and Human Services (SCDHHS) to design a randomized evaluation of a nurse home-visiting program for low-income mothers delivered by the nonprofit organization Nurse-Family Partnership (NFP). Prior evaluations showed that NFP improves outcomes for mothers and children. SCDHHS, NFP, and J-PAL North America believed it was important to evaluate NFP in a modern context, for a different population, and at a larger scale.

South Carolina is expanding NFP through an innovative pay-for-success initiative. But despite a substantial increase in the number of mothers NFP serves annually, the program does not have sufficient resources to serve all of the women in South Carolina who are eligible. J-PAL North America helped SCDHHS design an evaluation that will randomly assign applicants on a rolling basis over four years to either a treatment group of 4,000 women that are offered access to the program or a control group of 2,000 women that are not, facilitating a rigorous evaluation of the impact of NFP on mothers’ and children’s health outcomes. The randomized evaluation is led by J-PAL affiliate Katherine Baicker (Harvard), J-PAL North America Deputy Director Mary Ann Bates, J-PAL North America Research Manager Michelle Woodford, and Annetta Zhou.

Christian Soura, Director of SCDHHS, described his experience working with J-PAL North America to design and implement a large-scale evaluation with important implications for state policy:

Our partnership with J-PAL allowed for the development of the first randomized evaluation of NFP in South Carolina. The State of South Carolina and the Nurse-Family Partnership (NFP) are working together to expand a proven home-visiting model to serve thousands of additional first-time mothers and babies. As part of this common effort, NFP is piloting a new approach that is intended to significantly reduce the cost of the home visiting intervention. Although the NFP model has had favorable results from several randomized evaluations in the past, we agreed that it was important to subject the revised approach to a rigorous evaluation as well. 

J-PAL’s staff and affiliates have an excellent problem-solving approach and have worked hand-in-hand with us to develop a research design for our project that will produce meaningful results for the academic and policy communities while considering the operational needs of those engaged in front-line service delivery. Integrating the procedures for randomizing participants was a new challenge for both DHHS and the service providers in our project. Fortunately, J-PAL’s team has been committed to providing regular training to all new nurses who are participating in this evaluation. This has included spending a full day with each new nursing team to walk through the process of enrolling moms in the project and answering their questions about how to collect background information and consent, while also handling the new nurses’ questions and concerns. J-PAL brings a down-to-earth, accessible perspective to this training and the ongoing support of enrollment workers—preparing those front-line workers critical to the project’s success in a manner that otherwise would be very challenging for our own staff to deliver.

One of the most striking things about J-PAL is their commitment to not only implement a high-quality randomized evaluation, but also assist project partners with successful operations. J-PAL designed a baseline survey of participants and has helped project partners adjust the intake survey over time as new data needs emerged, while also helping us consider how those changes would affect participants and the implementation of the evaluation. For example, at one point, the project’s billing team needed additional information on the specific funding source that was covering the cost of home visiting services for each individual enrollee. Though not directly tied to the evaluation, J-PAL worked quickly to integrate new questions into the intake survey and communicate results to the agency, making our own work more efficient.

We will use the results of the evaluation in several ways. These results will determine the amount of the “success payments” that the state will pay NFP for reducing preterm births and child injuries, increasing birth spacing, and targeting service delivery into the rural and disadvantaged communities that need these services the most. South Carolina is administering this project under authority of a five-year waiver granted by the federal Centers for Medicare and Medicaid Services. The state will use the evaluation results to decide whether to seek a renewal of the waiver (and if so, under what conditions), or whether these services might be candidates for more permanent inclusion in the State Plan for Medical Assistance.

Through this project, NFP is testing a lower-cost variant of its longstanding model. The results of the study will give us insight into the effectiveness of this revised approach, which we will compare to past studies of the original intervention.  Even after the initial five-year waiver period has expired, J-PAL will continue to follow the project’s participants so that they can produce a longitudinal study of the effectiveness of this model; this will be enormously valuable for researchers and public policymakers for years to come.

J-PAL North America is honored to work with both BDT and SCDHHS in generating rigorous evidence that can answer important questions in the fight against poverty. To learn more about J-PAL North America, visit www.povertyactionlab.org/na.

Read the interview transcript from Rachel Cahill, Policy Director at BDT, here.

Contact: J-PAL North America Policy Associate Todd Hall at [email protected].