Understanding the role and importance of spillover effects

In part four of J-PAL North America’s ten-year anniversary blog series, we discuss how credible evidence from randomized evaluations is informative in identifying effective strategies to reduce poverty, regardless of the impact estimate. We share what we have learned from evaluations with positive results and from evaluations with null results to provide key lessons for making all research results meaningful.

The power of randomized evaluations to generate credible, clear evidence has transformed the policymaking space, where policymakers, government leaders, and practitioners increasingly prioritize evidence-based policymaking. Randomized evaluations credibly measure the impact of programs and policies because random assignment ensures that systematic differences, such as income or gender, do not drive changes in outcomes between people who do and do not receive a program. Because of this, randomization allows for causal conclusions, which means researchers can credibly say that a program causes observed changes in outcomes. Randomization also minimizes the need for complex statistical methods to estimate impact, so results from randomized evaluations provide clear, easy to understand evidence that can be applied to policies and programs. 

Credible results from randomized evaluations can equip decision-makers to go beyond trends in social policy and meaningfully improve lives. J-PAL North America has supported nearly 200 randomized evaluations on a variety of policy-relevant topics to build a credible body of evidence on what strategies actually improve lives and what do not. Though positive results from randomized evaluations are exciting, sometimes we learn that interventions do not work as intended. While this can be challenging, null results, as well as negative results, are critical in building our understanding of what works in reducing poverty. 

New evidence can transform the narrative around a programmatic area

Whether randomized evaluations show that a program or policy causes positive impact or has no impact, new evidence can transform theories of change about what works in a particular area, such as health, education, or criminal justice.

In the Baby’s First Years study, researchers found that providing monthly cash payments to families experiencing poverty positively impacted infant brain activity. Prior to this study, research in both social science and neuroscience theorized about the impact of poverty on child development, but evidence was scarce on developmental outcomes for children under the age of three. Qualitative work also supported defining the scope of child-related purchases and quantifying household size, which further contextualizes theories of change about child development, household income, and poverty. Evidence from this evaluation to date has demonstrated that providing unrestricted cash, instead of targeting other factors associated with poverty or targeting personal choice, can change infant brain activity patterns—patterns shown to be associated with the development of cognitive skills.

Additionally, null results can promote innovation by giving insight into what mechanisms of a program work and why. In the case of Health Care Hotspotting, researchers investigated the impact of intensive wrap-around support on hospital readmission rates among individuals with high health care usage who have complex medical needs. While this model has received national attention as a promising intervention, researchers ultimately found no impact of this intensive support program on six-month hospital readmission rates. This could be because the intervention served a younger population with more complex and diverse needs than other programs that had been previously evaluated. While these null results are challenging, they ultimately prompted an institutional shift in thinking by the implementing organization, Camden Coalition, where program leaders are currently thinking through what mechanisms of their program work for their particular population and why. 

Credible positive and null results guide strategic scaling

Sometimes new evidence prompts scaling back of programs and policies. In the past decade, J-PAL North America supported evaluations of workplace wellness programs, where researchers sought to understand the impact of these programs on employee health and health care costs. Previous observational analyses suggested varying degrees of positive effects, but rigorous evidence was needed to build validity for these estimates. Ultimately, randomized evaluations found that a fairly typical workplace wellness program delivered across many worksites by a large employer had limited to no impact within the first year, and follow-up analyses found similar results after three years. While disappointing, this insight can guide policymakers in decision-making about scaling incentives, such as tax subsidies for these types of programs, and propel exploration of more effective approaches to bolster employee health.

On the other hand, new evidence can be a catalyst for scaling up effective programs and policies. In the education sector, there are numerous debates on what to prioritize to improve student learning outcomes. Randomized evaluations can be particularly effective in politicized contexts like this because clear and credible results communicate across political divides to share what interventions actually work. By 2020, randomized evaluations had consistently provided evidence of the effectiveness of tutoring on improving student learning, so J-PAL North America conducted an evidence review of 96 tutoring randomized evaluations, synthesizing existing evidence to inform the education sector about which programmatic elements of tutoring programs create high impact for students.

Leveraging actionable insights from the evidence review, J-PAL North America supports scaling tutoring programs through researcher-practitioner partnerships and policy. The recently launched Tutoring Evaluation Accelerator (TEA) will support tutoring programs across the United States to implement evidence-based programs and conduct evaluations in new contexts. Disseminating results about the impact of tutoring resulted in policy actions that increased access to tutoring. Citing J-PAL North America research, the White House encouraged states to allocate American Rescue Plan funds for tutoring programs. Following conversations between J-PAL North America staff and the California Governor’s office about the impressive potential of high-dosage tutoring, the state passed a bill in 2021 that included $460 million for hiring paraprofessional tutors. This evidence also influenced the 2021 creation of a $5 million high-impact tutoring program in Colorado, where advocates drew from J-PAL North America’s evidence review to inform the characteristics of tutoring outlined in the legislation.

Results from randomized evaluations can also provide insights for effective program implementation by improving understanding of which elements of programs are effective. Positive results can help the iteration of program delivery to maximize effectiveness. In the criminal legal space, J-PAL North America supported researchers in New York City in an evaluation that found text message reminders decreased court non appearance by 26 percent. As a result of this evidence, New York City courts now send text message reminders to all summons recipients who provide a cell phone number.  

Trusting relationships ensure that credible evidence creates positive change 

All credible research results can be used to champion innovation and learning. Because null and negative results may be disappointing or even challenge our beliefs, they must be shared thoughtfully with practitioners and policy makers. This underscores the importance of building trusting relationships between researchers and partners, as discussed in part three of this blog series. Ensuring researchers and partners are invested in partnerships plays a key role in ensuring that all members are open to learning from results once they become available. J-PAL supports teams in sharing null results with partners and disseminating null results publicly, so that lessons learned can inform effective decision making. 

Sharing null results also contributes to creating a culture in the policymaking space that celebrates and learns from failure. In reflecting on results, AJ Gutierrez, co-founder of Saga Education shares, “After evaluation, it can be disheartening to find no evidence of impact, but when we don’t know the effects of an intervention these null results provide new information for learning that builds on what researchers know. J-PAL’s courage in sharing results helps build momentum in policy spaces and tells stories that need to be told.” 

In J-PAL North America’s ten-year anniversary blog series, we reflect on some of the most impactful randomized evaluations and bodies of research that our organization has supported over the past decade. We also celebrate the tremendous contributions of our researcher network and the policymakers and practitioners who have made this research possible. Part one kicks off the series with reflections from our scientific leadership. Part two explores the role of study design and implementation. Part three dives into effective collaboration between researchers and practitioners. Part five considers how policy can be informed at scale.

This spring, the United States Health Care Delivery Initiative (HCDI) hosted its second convening, HCDI @ 8, during which researchers and implementing organizations shared their experiences evaluating programs using randomized trials. As we look to the future of health care delivery, we reflect on these discussions, highlighting why an organization may want to evaluate their program with a randomized evaluation. 

Choosing to evaluate a social program can be a significant decision for an implementing organization. Randomized evaluations reveal the efficacy of a program. At the outset, neither the researcher nor the implementing partner know whether the evaluation will find that the program has the intended impact. Evaluations take time and resources to plan, execute, and analyze, with few guarantees. This then begs the question, why would an implementing organization choose to evaluate their program?

Why evaluate your program?

Sharing their experiences during a panel discussion, panelists highlighted several reasons why they were motivated to undertake an evaluation. To make decisions about effectively using finite resources, the University of Illinois Urbana-Champaign worked with David Molitor and his co-authors to test the impact of a workplace wellness program for employees and its impact on health outcomes. Through evaluation, they were able to find that workplace wellness programs do not improve employee health outcomes and therefore chose not to invest in a program of their own. 

Allison Sesso of RIP Medical Debt, an organization that buys and forgives medical debt, joined as Executive Director during the planning stages of the evaluation of their program, and was very pleased that she had. To Sesso, the primary goal of RIP Medical Debt is to do what is in the best interest of the people they are serving and to ensure that their services are actually helping people.

Additionally, evaluation gives credibility to the program and can aid in attracting donors and new board members. Evaluation shows that “you’re willing to take the leap and…[to look] at yourself in the mirror.” Sesso also said that she is able to leverage the media attention given towards the study as an opportunity to promote the program and its policy implications.

How can partnerships benefit evaluations and programs?

Strong partnerships are vital to conducting a successful randomized evaluation of a program or policy. Partner organizations bring key insights on program operations and the people they serve, while researchers bring technical expertise. Conversations between researchers and on the ground staff make for stronger, more meaningful evaluations that answer the questions that will best help policymakers and practitioners make key policy decisions. J-PAL helps to facilitate the collaboration between the research team and implementing team and provides technical support. 

Allison Hess of Food Fresh Farmacy, which offers a food as medicine approach to supporting people with diabetes who are experiencing food insecurity, echoed the sentiment that J-PAL, which connected her organization with researchers, was vital in providing technical support and securing funding. Hess admitted that she was initially hesitant about going ahead with an evaluation given the small team and limited resources of Food Fresh Farmacy and the challenges of working in health care delivery during the Covid-19 pandemic. Nevertheless, she found that the researchers and J-PAL were “instrumental early on with really laying out the expectations, what our responsibilities were going to be...and also help to secure funding.” Dr. John Bulger, also of Food Fresh Farmacy, noted that collaboration with the researchers and J-PAL “added things to the program that we would have never done” and that “no matter what the result ends up being, [the organization] will be much better for it and [the] patients will be better for it.”

How can evaluations influence policy and decision-making?

Randomized evaluations are important first steps in shaping policy. As an example of this, Dr. Fatima Cody Stanford highlighted a study she completed with HCDI Co-Chair Dr. Marcella Alsan on public health messages about Covid-19 and the effect on Black and Latinx individuals’ knowledge and information-seeking behavior. The randomized evaluation revealed that for Black participants, viewing a message from a Black physician increased information-seeking behavior. These results build on Dr. Alsan and her co-authors’ patient-provider race concordance study which found that Black men who were treated by a Black doctor were more likely to take-up preventive services.

The number of Black physicians in the United States remains low, with only a 4 percent increase in the last 120 years. These studies highlight the importance of diversifying the physician workforce, as well as possible messaging solutions in the interim. Since the completion of these studies, the American Medical Association has strengthened efforts to increase diversity in medical schools and residencies and cited the patient-provider race concordance study in explaining this policy change.

Evaluating a program can be a positive experience that can empower your organization to influence decision-making, but it can be difficult to know where to start and for an implementing organization to find the right researcher and vice versa. J-PAL offers matchmaking services to pair researchers with practitioners, supports study design and implementation, and holds requests for proposals to apply for research funds. If you are interested in learning more about how we could support an evaluation of your program, please reach out to [email protected]. 

Mireille Jacobson (University of Southern California), Weston Merrick (Minnesota Management and Budget or MMB), and Adam Sacarny (Columbia University) sit down with J-PAL staff to discuss their recent paper in Health Affairs. The paper reports on the results of their randomized evaluation assessing how various letters affected physicians’ use of Minnesota’s prescription monitoring program (PMP). The PMP is an electronic database that tracks controlled substance prescriptions in the state. They discuss the results, broader implications, and future opportunities for evaluation. 

This evaluation was awarded funding and technical support through J-PAL North America’s State and Local Evaluation Incubator and received Research Management Support. In part one of this blog series, published in the summer of 2021, Weston and Adam discussed their research partnership and the development of the study.  

What were the key outcomes or results of the study? 

Adam: The first main outcome was whether the clinicians in the study searched the prescription monitoring program (PMP). On this outcome, we see pretty substantial and durable effects. Search rates rose by about four percentage points from the most effective letters, and that effect lasted at least eight months.

The second outcome was whether the clinicians changed their co-prescribing of opioids with two other classes of medications, benzodiazepines and gabapentinoids. We focus on co-prescribing because these drug classes have interactions with opioids that can increase the risk of overdose. We don't detect any effects there, which probably reflects that prescribing is a lot harder to change than PMP use. 

Mireille: We also saw that some clinicians who received letters made new PMP accounts, as they didn’t previously even have an account, even though they should have been searching the PMP based on both the law in the state and the best clinical practice. So these letters brought a hard-to-reach population into using the PMP who may not have otherwise engaged with it. 

What are the broader implications of the results?

Adam: Every state has or is setting up a PMP, but the effects of these programs from recent meta-analyses look a bit disappointing. One of the reasons might be that many clinicians don't use them. What our study shows is that simple letters can move the needle on clinician engagement with PMPs. As a result, these letters could be of interest to other PMPs or even healthcare organizations that want to get their clinicians to use the PMP.

In addition, while changing prescribing with letters seems to be more difficult, getting clinicians to use the PMP could still be useful because it could help clinicians become better informed about their prescribing. They then have a reason to inform patients about the risks that they might face from taking these prescriptions, like overdose, and tell them about naloxone, the overdose reversal medication. 

How are these results informing future policy choices? 

Weston: The Minnesota Board of Pharmacy was an extremely engaged partner in this work. They care a lot about making sure that the PMP is an effective system for providers to use. They used the results of this study to create and inform some automated messages sent through the PMP, using the behavioral lessons we used in this study. 

The Board of Pharmacy is also very interested in looking at further interventions to increase the likelihood that folks are using the PMP and using it correctly to prescribe in a safe way. 

More generally, we've started to realize that these nudge messages are about improving the customer experience of accessing the government. There's a reason why these prescribers aren’t using the PMP— maybe they don't know about it, maybe they don't know how to sign up, maybe they forgot their login, whatever it may be. This simple letter was able to prompt them to better use a service that was available to them. This is just one example of how we can continue working to improve access to, and use of, government services across the state. 

Are you considering future collaborations in this partnership?

Adam: We’re certainly trying to build on these successes and use the groundwork already laid to launch new projects. I’m hoping we can work on interventions that not only try to stop risky prescribing but try to promote beneficial care. Weston and I and our collaborators have been talking and trying to think about both sides of the issue—not only reducing bad things but also increasing good things—and how nudge-type interventions can be useful there.

Mireille: Yes—and all of these strategies are fundamentally about trying to mitigate the harms of the opioid epidemic, from every angle possible, whether through better prescribing or better access to treatment. 

Weston, more generally, how has this partnership impacted the ways you and your team think about the role of evidence? 

Weston: This project was a really important example of a research partnership that worked really well. We were able to leverage outside resources and expertise to identify a meaningful policy question, use our existing data to answer that question, change policy, and then publish results in a journal so that places outside of Minnesota can learn from our experience. Having that kind of example allows us to go back to state leaders and say “this is why we should do an evaluation, and this is what you’ll get back,” which makes it much easier to start future projects that can meaningfully impact the health and well-being of Minnesotans. 

Any last thoughts?

Weston: I'll just end on the fact that we're so grateful to the Board of Pharmacy and PMP staff to endeavor on this project with us. They've been a really wonderful partner and we’re hoping to work with them more on future projects.

Adam: I would second that—we are really grateful that the PMP was willing to devote their time to this rigorous evaluation. 

Mireille: I also want to emphasize that it was invaluable to have active collaborators, like Weston and Ian Williamson, who work in the government. They were able to help manage the implementation process given their knowledge of organizational constraints but, more importantly, were able to help design and implement a project that is relevant not only for academia but also for policy.

This piece is part of an ongoing series highlighting research partnerships with state and local government agencies fostered through J-PAL North America’s State and Local Evaluation Incubator. The second & third pieces feature Shasta County's evaluation of text message reminders to reduce failure to appear in court.