Promoting rigorous, actionable research in maternal and newborn health in the United States

Researchers from J-PAL’s network partnered with the South Carolina Department of Health and Human Services (SC DHHS) and the National Service Office for Nurse-Family Partnership and Child First on a rigorous evaluation of the Nurse-Family Partnership (NFP) program in South Carolina. As research on the program’s impact on birth outcomes concludes, we sat down to debrief with Jordan Desai, chief of quality at SC DHHS; William (Bill) Thorland, senior research fellow at the National Service Office for Nurse-Family Partnership and Child First; and Margaret (Maggie) McConnell, lead researcher on the study and associate professor of global health economics at Harvard T.H. Chan School of Public Health. Together, they bring us behind the scenes of the research partnership, discuss what challenges and opportunities they’ve experienced, and highlight what they look forward to learning as the study progresses.

Why was SC DHHS and the National Service Office for Nurse-Family Partnership and Child First interested in conducting a randomized evaluation of the NFP program in South Carolina? 

Bill: Conducting rigorous evaluations of NFP has been essential to the program’s success for the past five decades. In embarking on the first US Pay for Success project on maternal and child health, we acknowledged the need to test this new innovative financing mechanism and rapid scaling with a randomized evaluation. 

Jordan: Approximately 60 percent of the births in the state of South Carolina are covered through SC DHHS’ Healthy Connections Medicaid program. Like many other states, SC DHHS is interested in looking for data-driven innovations that can improve birth outcomes for the residents we serve. This project’s randomized evaluation was appealing to the agency because it offered new insight into the effectiveness of specific interventions.

Maggie, why were you interested in working with NFP and SC DHHS on this evaluation? How is this study contributing to the existing evidence base on interventions to support families during pregnancy and early childhood?

Maggie: In the United States, low-income families face significant challenges during pregnancy and early childhood that set the stage for lifelong inequities. Our research team recognized the value of learning about a flagship federally-funded program’s impact on outcomes in this population at this critical time. Our early findings underscore growing evidence that improving birth outcomes with behavior-change focused interventions during pregnancy can be challenging and that innovative thinking is needed to address the substantial inequities we see in the study. 

How did the research partnership come about? What was exciting, as well as challenging, about partnering on this evaluation?

Maggie: Initial conversations about the project started nearly a decade ago! Launching a complex project with many partners and sustaining engagement with the work over this long period required tremendous commitment from all partners but especially NFP and the state of South Carolina. Their commitment to building evidence enabled the launch of the largest randomized evaluation of nurse home visiting to date.

Bill: We were excited to partner on the largest randomized controlled trial of NFP and serve more families in South Carolina. Being able to target a large enough sample of the population to enroll in the trial was challenging for enrollment but essential to being able to detect statistical differences. It also afforded a unique opportunity to assess program effectiveness during a time when significant changes were occurring, such as the rapid decline in teenage birth rates and the expansion of community services and public health initiatives focused on maternal-child health.

SC DHHS was one of the first states to use a 1915b Medicaid waiver mechanism with an embedded randomized trial. What is this waiver and why is it important for future innovation and learning?

Jordan: Generally, 1915b waivers provide states with flexibility in how services can be tailored to specific population needs within a managed care delivery system. This waiver authority provides states with important flexibility by allowing them freedom to pursue innovative projects targeted toward specific populations.

Maggie: In the context of this project, with the support of a network of partners and philanthropic donors, South Carolina demonstrated the feasibility of embedding a randomized evaluation into a 1915b waiver. Through this integration, the state is a national leader and provides a pathway forward for other states to rigorously evaluate the impacts of their innovation.

The first academic paper of the study⁠—focusing on pregnancy, birth, and postpartum outcomes⁠—was just published. How do you foresee these results informing the NFP model and policies to support pregnant individuals?

Bill: Previous internal evaluations of NFP national data showed the potential for program impact on some measures of birth outcomes among specific sub-populations of families being served by Nurse-Family Partnership. Additional analysis among families served in South Carolina is needed to better understand how the families and conditions in the state during this time may have differed from what has been previously seen nationally. Ultimately, this informs decisions for more targeted enrollment practices and community-specific outcome expectations to guide the implementation of the program. This study further shows us we need to provide better support for nurses to reach and serve those families facing the most complex challenges and barriers to resources.

This study will also enable us to learn more about how NFP affects a range of life outcomes in several areas of life over time. What additional outcomes do you look forward to learning about as the study progresses? 

Jordan: SC DHHS is interested in learning about longer-term outcomes for the families involved in the study. The population served in this study, new mothers and children, makes up the majority of the state’s Medicaid population. As such, SC DHHS is particularly interested in data that could help the state better understand additional social determinants of health needs for this specific population.

Maggie: In the first few years of analysis, our research will focus on outcomes that are experienced during pregnancy, postpartum, infancy, and toddlerhood that can be observed in administrative data records. Our research team looks forward to exploring longer-term outcomes, such as educational and economic attainment, which may provide a more nuanced picture of how program participation changed families’ trajectories. The State of South Carolina is a national leader in its ability to link data at the state level for research, which will enable ongoing learning from the study even long after study participants have graduated from NFP’s services.   

Bill: We appreciate J-PAL’s commitment to more rigorous evaluations, and we look forward to seeing other findings that can inform additional opportunities to improve the program.  

What advice would you give to other organizations considering conducting a randomized evaluation of their policy or program?

Bill: Large, randomized evaluations can bring challenges. The rapid enrollment period placed a lot of stress on the nurses and took away from the resources needed to better enroll families who could most benefit. It can also be particularly difficult for providers, like the NFP nurses, to know there are potential families who could benefit, but are assigned to the control group. Nevertheless, we believe evaluations are essential to ensuring your program is learning and adapting to the changing needs of the population you intend to serve.
 

This spring, the United States Health Care Delivery Initiative (HCDI) hosted its second convening, HCDI @ 8, during which researchers and implementing organizations shared their experiences evaluating programs using randomized trials. As we look to the future of health care delivery, we reflect on these discussions, highlighting why an organization may want to evaluate their program with a randomized evaluation. 

Choosing to evaluate a social program can be a significant decision for an implementing organization. Randomized evaluations reveal the efficacy of a program. At the outset, neither the researcher nor the implementing partner know whether the evaluation will find that the program has the intended impact. Evaluations take time and resources to plan, execute, and analyze, with few guarantees. This then begs the question, why would an implementing organization choose to evaluate their program?

Why evaluate your program?

Sharing their experiences during a panel discussion, panelists highlighted several reasons why they were motivated to undertake an evaluation. To make decisions about effectively using finite resources, the University of Illinois Urbana-Champaign worked with David Molitor and his co-authors to test the impact of a workplace wellness program for employees and its impact on health outcomes. Through evaluation, they were able to find that workplace wellness programs do not improve employee health outcomes and therefore chose not to invest in a program of their own. 

Allison Sesso of RIP Medical Debt, an organization that buys and forgives medical debt, joined as Executive Director during the planning stages of the evaluation of their program, and was very pleased that she had. To Sesso, the primary goal of RIP Medical Debt is to do what is in the best interest of the people they are serving and to ensure that their services are actually helping people.

Additionally, evaluation gives credibility to the program and can aid in attracting donors and new board members. Evaluation shows that “you’re willing to take the leap and…[to look] at yourself in the mirror.” Sesso also said that she is able to leverage the media attention given towards the study as an opportunity to promote the program and its policy implications.

How can partnerships benefit evaluations and programs?

Strong partnerships are vital to conducting a successful randomized evaluation of a program or policy. Partner organizations bring key insights on program operations and the people they serve, while researchers bring technical expertise. Conversations between researchers and on the ground staff make for stronger, more meaningful evaluations that answer the questions that will best help policymakers and practitioners make key policy decisions. J-PAL helps to facilitate the collaboration between the research team and implementing team and provides technical support. 

Allison Hess of Food Fresh Farmacy, which offers a food as medicine approach to supporting people with diabetes who are experiencing food insecurity, echoed the sentiment that J-PAL, which connected her organization with researchers, was vital in providing technical support and securing funding. Hess admitted that she was initially hesitant about going ahead with an evaluation given the small team and limited resources of Food Fresh Farmacy and the challenges of working in health care delivery during the Covid-19 pandemic. Nevertheless, she found that the researchers and J-PAL were “instrumental early on with really laying out the expectations, what our responsibilities were going to be...and also help to secure funding.” Dr. John Bulger, also of Food Fresh Farmacy, noted that collaboration with the researchers and J-PAL “added things to the program that we would have never done” and that “no matter what the result ends up being, [the organization] will be much better for it and [the] patients will be better for it.”

How can evaluations influence policy and decision-making?

Randomized evaluations are important first steps in shaping policy. As an example of this, Dr. Fatima Cody Stanford highlighted a study she completed with HCDI Co-Chair Dr. Marcella Alsan on public health messages about Covid-19 and the effect on Black and Latinx individuals’ knowledge and information-seeking behavior. The randomized evaluation revealed that for Black participants, viewing a message from a Black physician increased information-seeking behavior. These results build on Dr. Alsan and her co-authors’ patient-provider race concordance study which found that Black men who were treated by a Black doctor were more likely to take-up preventive services.

The number of Black physicians in the United States remains low, with only a 4 percent increase in the last 120 years. These studies highlight the importance of diversifying the physician workforce, as well as possible messaging solutions in the interim. Since the completion of these studies, the American Medical Association has strengthened efforts to increase diversity in medical schools and residencies and cited the patient-provider race concordance study in explaining this policy change.

Evaluating a program can be a positive experience that can empower your organization to influence decision-making, but it can be difficult to know where to start and for an implementing organization to find the right researcher and vice versa. J-PAL offers matchmaking services to pair researchers with practitioners, supports study design and implementation, and holds requests for proposals to apply for research funds. If you are interested in learning more about how we could support an evaluation of your program, please reach out to [email protected]. 

Baby’s First Years (BFY), a J-PAL North America-funded study, recently published results showing that monthly cash support impacts infant brain activity. In an interview with J-PAL staff, J-PAL affiliated professor Lisa Gennetian (Duke University) and researcher Kimberly Noble (Teachers College, Columbia University) highlight the collaboration across academic disciplines, discuss policy implications of the early findings, and reflect on results to come.

BFY is the first causal study to examine the relationship between poverty reduction and brain development in young children. What motivated the idea for this research? 

Lisa: The seeds for this study came from an observation about both social science and neuroscience literature. Decades of social science research—from randomized controlled trials to clever quasi-experimental designs—point to the negative effects of poverty on children’s development. But this evidence speaks less about the period from birth to age three, which is increasingly recognized as an important time in early development. This is in part because of how hard it is to get objective measures of children’s development during those years. 

In the last decade or so, neuroscience literature has shown correlations between children’s early environments—including the socioeconomic status of households—and brain activity. This neuroscience research raises important questions about children’s brain development being adaptive and reactive.  However, this research cannot untangle cause from correlation, leaving open the question of how much socioeconomic status of families, and family income in particular, impacts children’s brain functioning. 

This study was designed to bridge the gap between these two distinct social science and neuroscience literatures and to address a yet-to-be-answered question about the causal impact of poverty reduction during children’s earliest years of brain development. 

The critical component of this study that allows for causal conclusions is randomization.  Can you explain how you designed BFY to measure the impact of monthly cash support on early childhood development and what role randomization played in the design?

Kimberly: We’ve known for a long time that children growing up in families that face economic disadvantage, that is often also coupled with financial exclusion and racism, are at risk for a host of negative outcomes. Yet there are many people who would say it’s not poverty: it’s other things associated with poverty, or it’s the choices that families are making. Regardless of where you sit ideologically, from a scientific perspective, the best way to address that question is through a randomized evaluation, or randomized controlled trial. We knew we couldn’t (and wouldn’t) randomize families to economic hardship, but we could offer families with low incomes the opportunity to be randomized into a trial receiving unconditional cash as a means of poverty reduction. 

Lisa: We came together as a team over a decade ago with no question that we needed to design a randomized evaluation to answer these questions. The tricky part for our team was not whether we should do it, but how to design it. 

Kimberly: The group that we call the “high-cash gift group” receives $333 per month, and the group that we call the “low-cash gift group” receives $20 per month. The amounts were chosen because 1) literature linked a difference in annual income of about $4,000 to positive academic, employment, and health outcomes, and 2) it was an amount that we thought held some policy relevance because it was similar to other social services and benefits that these families were likely to qualify for.

BFY is an interdisciplinary undertaking, involving neuroscience, economics, psychology, and social policy researchers. How has the collaboration between fields affected the research design or execution? 

Lisa: Greg Duncan was the champion who pulled this team together—he wanted to bridge neuroscience to social science and invited us to the table. Having a champion with a vision and an extensive network across disciplines is so important to making any multi- and inter-disciplinary research design work. 

This collaboration has also been successful because of the hard work of the principal investigators with a shared understanding on the contribution of this study to scholarship about the impacts of poverty, and therefore, evidence to inform policy. We have a high-level theoretical framework that crosses disciplines and has guided the study’s data collection. We’ve always envisioned this research as looking across children’s development. Especially when thinking about the cumulative effects of things like caregiver choices and the child’s home environment—this is not a lens that an economist would necessarily bring to the table, so the interdisciplinary framework is really essential.

What do you see as the policy implications of these early results? How do you hope the findings will contribute to policies that increase family well-being and economic stability?

Lisa: Even though we’ve worked together for over a decade, this study is still in the beginning phase of analyzing data and understanding the findings. No one anticipated there would be something like the expanded child tax credit (CTC) in 2021 (and, to clarify, this study is not an evaluation of the expanded CTC). Having said that, the early results can speak to unconditioned monthly cash, which mimics the expanded CTC in the spirit of cash to families but also mimics child benefits or allowances available in many other countries. Just to note to readers, one key difference between the BFY cash gift and the expanded CTC is that the expanded CTC was available to every child in the household under the age of seventeen, while the BFY cash gift is one monthly amount irrespective of the number of children.  There are many other differences related to the design and disbursement of the BFY cash gift too.

The BFY cash gift is a predictable monthly promise to families, no strings attached. In policy circles, I feel we lose sight of stability and predictability. Again, this is where the interdisciplinary perspective of child development is so valuable. This cash gift provides stable income. It doesn’t matter if you got married; it doesn’t matter what assets you’ve accumulated—it’s uncoupled from changing circumstances or eligibility determinations. All of that is stickiness in our existing economic cushion in the US and we have good reasons to believe that this isn’t the best way to think about investments in children.

You note in the paper that these findings come just one year into a multi-year study. What additional outcomes will your team be looking at in the months and years to come?

Lisa: We go back every year to talk to the families at about the time of the focal child’s birthday, and we’ve had excellent retention rates, balanced across high and low cash gift groups and locations. By this summer, we will have four data points: just after birth, age one, age two, and age three. We have information on spending, labor supply, household composition, maternal well-being, the child’s home environment, material hardship, income and benefits, and relationships with partners. We are analyzing that data now. In the next several months, those findings will be out in the public through working papers and peer-reviewed publications. 

Kimberly: When we first enrolled moms, it was a three-year study. After we pivoted to phone-based data collection due to the pandemic, our funders generously agreed to extend the cash gift for an additional year, enabling us to push our capstone wave of in-person data collection to the children’s fourth birthdays. This data collection is set to kick off this summer, and we are optimistic that we’ll be able to invite families into our university lab to collect data over the course of twelve months. We will conduct cognitive and behavioral assessments with the kids and measure brain activity again, and hopefully in a much larger sample. We will also measure stress hormones from both moms and kids, mom’s cognitive bandwidth, environmental influences on genes, and other indicators of physical health.

We’re also raising funds to extend the cash gifts an additional two years, which will get us to the child’s sixth birthday—essentially the entire period before the start of formal schooling. Our hope is to follow these kids much longer-term, ideally through adulthood.