Understanding the role and importance of spillover effects

A new J-PAL evidence wrap-up highlights three randomized evaluations that consider the spillover effects of three health care interventions. In this post, we discuss what spillover effects are, a summary of the highlighted studies, and the benefits to policymakers in considering spillover effects on a program or policy. 

Spillover effects of health care interventions

Programs and policies often target a specific population, but they frequently affect other populations as well. These are often referred to as spillover effects. For example, every year, millions of people are prescribed antipsychotic drugs for uses not approved by the United States Food and Drug Administration despite evidence suggesting such off-label use is associated with significant harm. With the goal of limiting the over-prescription of these drugs, the Centers for Medicare and Medicaid Services conducted a randomized evaluation to test the impact of sending warning letters to doctors prescribing a high volume of prescriptions to Medicare patients. Researchers found that these letters not only limited the number of prescriptions doctors gave to Medicare patients but also prescribed these antipsychotic drugs less frequently to privately insured patients.

Spillover effects can occur for many reasons. For example, a non-targeted population could learn about a program, which could alter their behavior or decisions they make, or the targeted population could be affected by a program or policy, and the changes caused by the program can indirectly affect another group of people. 

Spillovers can be positive or negative. Consider the following hypothetical examples:

• A job training program can be effective in making participants more competitive job applicants. If few jobs of this type are available in an area, the program may also displace other workers from jobs they may have otherwise gotten.
• A school tutoring program could provide needed support to struggling students. This could decrease the burden on teachers and cause more teachers to stay at a once-challenging school. At the same time, the program could also attract higher-quality teachers to teach at the school.
• A childhood immunization campaign could increase vaccination rates for children in a specific area. The parents who bring their children into health clinics could then develop relationships with healthcare professionals, causing them to also seek preventive care for themselves.

Evidence from randomized evaluations on spillover effects of health care interventions

As organizations design a new policy or program, it’s important to consider possible spillover effects. Randomized evaluations are one of the most credible tools we have to measure these effects. Researchers can isolate a program's impact on the intended population and identify how non-targeted populations are affected. Other evaluation methods could misinterpret a spillover effect as a systematic difference between the treatment and comparison groups, leading to misleading results. 

The US health care system faces numerous issues related to efficiency and quality. Many organizations are creating innovative solutions to attempt to address these issues. However, if spillover effects aren’t considered in the design and evaluation, policymakers and practitioners will be unable to see a more complete picture of the true impact of a program or policy.  A J-PAL evidence wrap-up highlights three randomized evaluations analyzing spillover effects of health care interventions related to payment reform, insurance eligibility, and provider-facing behavioral interventions. Researchers found that some of these interventions affected non-targeted populations in a similar way as the targeted populations. For example, medical providers were unlikely to tailor treatment strategies based on a patient’s insurance plan, which meant that adjustments that a major insurance provider (such as Medicare) made to their plan affected the care received by patients under different insurance providers. 

Benefits of understanding spillover effects 

Spillover effects provide insights into the broader systemic impact of a program and can reveal how interventions interact with existing social, economic, and health systems. By considering and accounting for spillover effects, policymakers and practitioners can improve the effectiveness and outcomes of their programs. This offers policymakers a more comprehensive understanding of how programs fit into the larger context. 

For example, some policymakers worried that expanded Medicaid eligibility for adults would increase the number of children enrolling in Medicaid by so much that they wouldn’t have the resources to insure all eligible applicants. On the other hand, some policymakers valued the idea that expanding Medicaid may also promote coverage of children. A study examining these spillover effects from a randomized evaluation in Oregon showed that newly eligible parents were more likely to enroll their children slightly sooner than they would otherwise, but the effect faded after just one year.

Analyses that identify the spillover effects of health care interventions can provide decision-makers with a clearer picture of a program’s effects, not just for those directly targeted, but also for others who may be impacted by a program’s far-reaching effects. The simple, credible, and transparent design of randomized evaluations makes them particularly useful in accomplishing this. When feasible and ethical, it is important to identify opportunities to use randomized evaluations to measure spillover effects so policymakers and practitioners can use rigorous evidence to design and implement effective and equitable health care interventions.