Strengthening randomized evaluations with qualitative research, part 4: Oregon health insurance experiment
For the final part of our blog series on incorporating qualitative research into randomized evaluations, we spoke with Associate Professor of Social Work and co-author of the Oregon Health Insurance experiment, Heidi Allen, about how in-depth interviews with study participants helped the research team interpret some of the study’s results.
In 2008, Oregon held a lottery to select an additional group of low-income, uninsured adults into its Medicaid program. Around 90,000 applied for 10,000 openings, providing an opportunity for J-PAL North America Scientific Co-Chair Amy Finkelstein, J-PAL affiliated professor Kate Baicker, Allen, and coauthors to conduct a randomized evaluation to understand the impact of providing health insurance to the uninsured.
The study found that participants who received Medicaid coverage experienced increased health care use, reduced financial strain, and improved mental health and self-reported health. To better understand the causal mechanisms behind these quantitative results, a staff of interviewers led by Allen conducted 120 structured interviews with study participants who recently gained Medicaid coverage.
The interviews were especially useful in helping researchers understand the quantitative results that were more unexpected. For example, many in the broader health policy community believed that providing health insurance to low-income individuals would result in a decrease in emergency department use. However, the results of the study showed that Medicaid coverage actually increased the use of health-care services across the board, including the emergency department. Through the interviews with participants, researchers started to uncover some of the potential factors contributing to the sustained increase in emergency department use.
“There were several things driving this increase. First, we learned that there were many individuals who should have been seeking care but were not. Some of these individuals described themselves as healthy despite having serious, chronic health problems or were generally hesitant to seek care due to cost concerns. Receiving health insurance removed a financial access barrier that may have prevented them from going to the emergency department in the past. On the other hand, we also started hearing how some primary care providers actually told their patients to go to the emergency department. One study participant shared that her primary care provider directed her to the ER since her blood sugar was dangerously high. All these stories helped us understand why we saw an increase in emergency departments.”
According to Allen, another finding that was surprising was that many study participants believed they had emergency-only coverage. Initially, this made little sense to the researchers since the coverage study participants received was a comprehensive health insurance plan. The researchers decided to probe a little deeper to understand the root cause of the confusion.
“We learned that one of the very first things many people did when they got their insurance card was call a dentist and say, ‘I just got the Oregon Health Plan, can I make an appointment?’ And the receptionist at the dental clinic would tell them that the Oregon Health Plan was emergency-only coverage, which is the case for dental care. But hearing this led many study participants to believe that their insurance only covered emergency care for health services too. I don't know that this drove any large effects in the study, but it was really interesting to hear that there was an unexpected kind of administrative complexity that I wouldn't have guessed prior to actually talking to people.”
Study participants also reported significant improvements in health. However, these findings were not coupled with clinically significant improvements in objective health measures like hypertension, diabetes, obesity, and behaviors like smoking. This posed another puzzle for researchers, but some potential explanations were found in participants’ stories.
“Initially, we thought when participants received health insurance, they’d gain access to a provider who could diagnose their health issues, and with this diagnostic information, they could start treatment that led to improved health outcomes. The interviews helped us understand that the pathway of receiving health insurance to improved health wasn’t as simple as we had thought. Having a good patient-provider relationship is a key factor in whether a patient’s health improves. For many participants, it took seeing multiple providers before they found someone they really liked working with. Once participants established a good relationship with a provider, they often prioritized what they were going to work on. And they might have more immediate needs like a broken ankle that needs surgery, so they may not address their diabetes or their obesity or smoking first.”
For Allen, conducting this qualitative follow-up research was critically important to making meaning out of the quantitative results, especially the findings that were surprising. These conversations with patients helped researchers understand where Medicaid was working for people and where people ran into barriers.
“The Oregon study constantly drives me to think about how we can make Medicaid work better for people, how we can improve the places where we lose efficiency and effectiveness in this causal chain from getting coverage to having positive health outcomes. There are probably multiple points where we could make slight modifications and see meaningful improvements. I think what's been helpful about our qualitative work is that it provides a pathway to think through real people's experience, rather than just partisan expectations of the experience. The interviews also drove home the humanity behind the numbers which isn’t to be taken lightly.”
Part one of this four-part blog series highlights the value of incorporating qualitative methods into randomized evaluations and outlined specific tips for researchers. Part two talks about how qualitative research helped motivate and shaped the central question and hypothesis for a study on racial concordance between physicians and patients. Part three looks at how Creating Moves to Opportunity randomized evaluation embedded qualitative research methods into its study design. Part five highlights the value of qualitative research in providing a deeper understanding of mothers' experiences in the Baby's First Years study.
Authored By
Economic theory offers ambiguous predictions for the impact of expanding health insurance on health-care use, economic well-being, and health outcomes, and it is difficult to separate the effects of insurance from confounding factors such as income and initial health in observational studies. In a series of evaluations, researchers took advantage of a lottery that gave low-income uninsured adults the chance to apply for Medicaid in the United States to examine the impact of health insurance on these outcomes over the first two years. They found that Medicaid coverage increased health care use, including use of preventive services and visits to emergency departments; reduced financial strain; reduced depression, and improved self-reported health. However, they did not find evidence that Medicaid coverage improved physical health or affected employment.
For more information about the Oregon Health Insurance Experiment evaluations, see: http://www.nber.org/oregon/
We've also put together a quiz of individual narratives from Oregonians who participated in the Medicaid lottery, describing their experience of how Medicaid affected their health care, financial security, and health. These are all true stories, but only half are consistent with the prevailing experience of most people. Try your hand at identifying which one aligns with the evidence here.
This project is registered on the AEA RCT Registry.
Policy issue
While the recent healthcare reform law in the United States will expand public health insurance programs, it is unclear both from economic theory and from existing observational evidence what effects such an expansion will have. For example, expanding health insurance might increase emergency department use by lowering the cost of visits borne by patients, or it might decrease use by increasing access to physician office visits that replace emergency department care. Similarly, health insurance could increase employment through improved health and fewer disruptions from health emergencies, or it could discourage employment if eligibility is contingent upon having income below a threshold. It is difficult to obtain empirical evidence on the causal impacts of health insurance on these outcomes because enrollees differ in many ways from the uninsured, for example in initial income and health, and these differences can contribute to the observed differences in outcomes between the two groups.
In a series of ongoing evaluations, principal investigators Katherine Baicker and Amy Finkelstein, along with other researchers, examine the effects of a Medicaid expansion in Oregon on a wide range of outcomes. The researchers conducted this evaluation by taking advantage of a lottery that gave randomly selected low-income, uninsured adults on a waiting list a chance to apply for Medicaid, while those not selected did not have that chance.
Context of the evaluation
Medicaid is a program jointly administered by the states and the federal government in the United States, designed to provide insurance coverage to low-income and disabled Americans. States have a great deal of flexibility in determining eligibility. As part of the Affordable Care Act health insurance reform, states have the option to expand Medicaid to all low-income adults. Twenty-seven states, including the District of Columbia, chose to expand Medicaid in 2014 and several states are still debating this choice.
The state of Oregon's Oregon Health Plan (OHP) Standard Medicaid program did cover low-income, uninsured, non-disabled adults who were ineligible for other public health insurance. It included comprehensive health benefits covering doctor visits, hospital stays for major illnesses, and prescription medication with no copayments. Enrolled individuals paid US$0-20 each month for this coverage, depending on their income. In 2004, the state closed the program to new enrollees because of budgetary limits.
Details of the intervention
In 2008, the state of Oregon decided that the OHP Standard program could accommodate 10,000 new enrollees. Oregon state health officials correctly anticipated that the demand for OHP Standard would far exceed the number of slots available and decided that a lottery was the fairest way to allocate these scarce slots. The state conducted extensive outreach. Of the about 75,000 individuals who signed up, the lottery randomly selected about 30,000 winners between March and September 2008. These selected individuals comprised the treatment group. Those who filled out an application that demonstrated their eligibility within 45 days of winning the lottery1 (about 30 percent of the winners) were enrolled in OHP Standard. Those who were not selected by the lottery formed the comparison group.
Researchers used a combination of administrative data sources, in-person interviews, and mail surveys to collect information on earnings, financial hardship, health-care use, insurance coverage, medical history, and physical health, over an approximately two-year period after the lottery. The study is the first randomized evaluation that has ever been conducted on the impact of Medicaid.
Results and policy lessons
The results measure the impact of Medicaid coverage by comparing the individuals who won the lottery to the individuals not selected by the lottery (the comparison group), under the assumption that the only reason these groups differ is because of the increased Medicaid coverage among those selected by the lottery. Because winning the lottery only increased the probability of Medicaid coverage by about 25 percentage points, the effect of Medicaid coverage is found by multiplying the effect of winning the lottery by four.
Medicaid increased the use of health-care services. Administrative hospital and emergency department records showed that, over about an 18-month period, Medicaid increased the probability of hospital admission by 2.1 percentage points (a 30 percent increase relative to the comparison group2 ) and the number of emergency department visits per person by 0.41 visits (a 40 percent increase). This included, in particular, increases in visits to the emergency department for conditions considered likely to be non-emergent and treatable by primary care. Survey results indicated that Medicaid also increased outpatient visits and prescription drug use.
Medicaid increased the use of recommended preventive care services as well. For example, Medicaid more than doubled the likelihood that women over age forty had mammograms. Self-reported access to and quality of care also improved with Medicaid coverage.
Medicaid diminished financial hardship. Medicaid reduced the likelihood of having any unpaid medical bills that were sent to collection agencies by 6.4 percentage points (a 23 percent decrease). It also reduced several other measures of financial hardship such as having any medical debt at the time of the interview and having to borrow money or skip paying bills in the past year to pay for medical bills. Catastrophic expenditures, defined as out-of-pocket medical expenditures in excess of 30 percent of household income, were nearly eliminated.
Medicaid reduced rates of depression and improved self-reported health, but had no statistically significant effect on physical health measures. Specifically, Medicaid did not have a statistically significant effect on measured blood pressure, cholesterol, or glycated hemoglobin (a measure of diabetes). However, Medicaid did increase the diagnosis of diabetes and use of diabetes medication. Given limits in the sample size of diabetic people, the study was not able to rule out potential improvements in glycated hemoglobin one would have expected to see with the increased medication use. On the other hand, the study was able to rule out declines in blood pressure one would have expected to see based on prior quasi-experimental evaluations of the effects of Medicaid. While long-run effects may differ from those found over this two-year study period, these physical health measures were chosen explicitly because clinical trials have shown that they can respond to medication within this time frame.
Medicaid reduced rates of depression by 9 percentage points (a 31 percent decrease) and increased the likelihood of self-reporting health as good, very good, or excellent (as opposed to fair or poor) by 13 percentage points (a 24 percent increase).
Medicaid had no statistically significant effect on individuals' employment or earnings. The employment rate among the comparison group was about 55 percent. The study was able to rule out a decline in employment due to Medicaid of greater than 4.4 percentage points or an increase greater than 1.2 percentage points.
To be eligible for OHP Standard, individuals must be 19-64 years old, an Oregon resident who is a US citizen or legal immigrant, ineligible for other public health insurance, and uninsured for the past six months. Individuals must also earn less than the federal poverty level, and have assets worth no more than US$2,000.
All reported percent changes indicate the percent increase or decrease caused by Medicaid relative to the control group.
This year’s Health Care Delivery Initiative (HCDI) workshop brought together researchers and health care practitioners to strengthen their working knowledge of randomized evaluations and lay a foundation for developing strong research partnerships.
“Trust and communication lay the foundation for a good working relationship,” shared Aaron Truchil of Camden Coalition of Health Care Providers (CCHP) with over 40 audience members, including academics and innovators in health care delivery.
Truchil’s keynote address—referencing his organization’s partnership with J-PAL affiliated researchers to evaluate CCHP’s care management program—kicked off this year’s Health Care Delivery Initiative (HCDI) workshop.
The workshop brought together researchers and health care practitioners to strengthen their working knowledge of randomized evaluations and lay a foundation for developing the type of strong research partnerships Aaron described in his keynote.
Participants included the four finalists of J-PAL North America’s US Health Care Delivery Innovation Competition, who are interested in rigorously evaluating their programs that address the social determinants of health.
Building knowledge of running randomized evaluations in the real world
Following Truchil’s keynote, attendees engaged in three interactive workshops tailored to open questions raised by Innovation Competition finalists, ranging from refining their research questions to strategizing different ways to design a study to best analyze a program’s impact.
“I learned a lot about randomization beyond how we were thinking about it before,” said Erika Ferguson of the North Carolina Department of Health and Human Services, an Innovation Competition finalist.
Participants then heard from three researchers—J-PAL affiliates Rebecca Dizon-Ross (University of Chicago) and Wes Yin (University of California, Los Angeles), as well as Marika Cabral (University of Texas, Austin)—in a candid Q&A conversation on designing and executing a successful randomized evaluation.
This included hearing their perspectives on the power of randomized evaluations, as seen in Dizon-Ross’ answer:
Karis Grounds from 2-1-1 San Diego, another Innovation Competition finalist, agreed, adding “We know RCTs [randomized controlled trials] are so important, and we learned how to communicate with different audiences—from social service providers, health providers, to researchers—to make that case.”
Laying foundations for strong research partnerships
The researcher panel set the stage for the next day’s conversations, where Innovation Competition finalists met with researchers to introduce their programs and discuss possible research designs in greater detail.
Throughout the conversations, Dizon-Ross’s words about impact evaluation rang true: tweaks in study design discussed in these conversations are about more than just methodology; they help answer fundamental questions that will ultimately improve people’s lives.
Mark Schweyer of California Health & Wellness, who is partnering with Innovation Competition finalist MedZed, said of the conversations, “Collaborations and partnerships are paramount. We couldn’t have been able to get much farther [in our research design] without everyone’s help in the room.”
The event didn’t end with the closing remarks. In the coming months, Innovation Competition finalists will work with J-PAL staff to develop their research proposals further and continue conversations with researchers. Competition finalists and partnered researchers from J-PAL’s network can then jointly apply for evaluation funding.
As participant Noah Pearce of Aurora Health Care noted, “We just couldn’t have done this over the phone.”
The Oregon Health Insurance Experiment found that covering the uninsured with Medicaid increased the use of health care, including primary care, hospitalizations, and emergency room visits; diminished financial strain; and reduced depression. There was no statistically significant impact on physical health measures, employment, or earnings.