Strengthening randomized evaluations with qualitative research, Part 3: Creating moves to opportunity
In part three of our qualitative research blog series on incorporating qualitative research into randomized evaluations, we learn more about how researchers conducting the Creating Moves to Opportunity (CMTO) project embedded qualitative research methods into their study and what factors made conducting high-quality, interdisciplinary research feasible.
Since 2018, MDRC, J-PAL affiliates Peter Bergman, Raj Chetty, Nathaniel Hendren, Lawrence Katz, and Christopher Palmer, along with sociologist and qualitative research expert Stefanie DeLuca, have been conducting a randomized evaluation of Creating Moves to Opportunity, a housing mobility program in Seattle and King County, Washington. The study, conducted with approximately 1,300 families, aims to understand CMTO’s impact on helping families move to neighborhoods with lower rates of poverty and more opportunities for upward income mobility.
In Phase One of the study, qualified low-income families with at least one child under fifteen were drawn from the Housing Choice Vouchers waitlist and offered an opportunity to enroll in the study. Participants were then randomly selected to receive CMTO services—including customized search and landlord engagement assistance from family and housing navigators and short-term financial assistance—or receive the housing authorities’ standard services.
Results from Phase One of the study found that CMTO-participating families were more likely to move to higher-opportunity neighborhoods than families who only received standard services. To help interpret these results and understand participating families’ experiences, the research team conducted qualitative analyses to complement the quantitative results. Led by DeLuca, a team of research staff carried out in-depth interviews with 161 participating families (from both the treatment and control groups) using an approach that borrowed some elements typically used in quantitative research to ensure the collection of high-quality, representative data.
“Using the program administrative data, we pulled a stratified random sample of families participating in CMTO to interview and achieved an 80 percent response rate. Those are two relatively unusual things to do in interview studies, but it helped ensure that the particular patterns that emerged from the data were representative of the families in the program.”
According to DeLuca, having a research team with the capacity and flexibility to follow up and be on-site when necessary was critical to the success of the qualitative research component of CMTO.
“Achieving an 80 percent response rate took more than just well-spaced regular phone calls. It required a research team with the capacity to be on-site and commit to really connecting with the study participants, especially with the door-knocking component of recruitment. It makes a huge different for people to see you as interested in their stories, not as telemarketers. With interviews taking up to two to four hours, being on-site also meant we could conduct follow-up interviews in case the interviewer wasn’t able to get through everything the first time. It also allowed us to collect ethnographic observations of the neighborhood and see the rhythm of the household, meet family members and friends—even run errands with families when needed.
We also used pairs when possible for interviews and had team members memorize the interview guide. With one team member conducting the interview without looking at a script and another paying attention to anything that got missed and noting that at the end, we had the benefit of an organic, more natural conversation alongside systematic data collection. While team members trained coders and designed codebooks, the bulk of the coding was also done by people who weren't interviewers, so the data wasn’t coded through the lens of assumptions and preconceived notions that someone present at the interview might have. Finally, we had several reliability procedures in place, including having the interviews coded by at least two different people, and a third person to note inconsistencies to be resolved.”
Participant interview findings helped to highlight the aspects of the program that led families to move to higher-opportunity neighborhoods.
“Through the interviews, we were able to better understand what was happening from the participants’ point of view and what mechanisms were at play. Previously, many believed that interventions focused on providing more information and monetary resources would encourage families to move to neighborhoods with more opportunities for upward income mobility. But what we see from CMTO is that those two types of resources are not sufficient to explain the success of this program. What came through so clearly in the narratives of the participants was how important it was to have that support from the housing navigators who could boost confidence and provide customized assistance based on the specific needs of the families.”
Ultimately, for DeLuca, incorporating qualitative research into randomized evaluations is about providing opportunities to check assumptions and to see a bigger picture of what might be driving the impact of a program or policy.
“The joining of disciplines like economics and sociology for randomized evaluations can provide an opportunity to see something you otherwise wouldn’t see. Sociologists are often trained to think about barriers to social mobility and wellbeing and tend to focus a bit less on the decision-making process of individuals. In contrast, economists tend to emphasize decision-making quite a bit and might give less attention to the context in which the decisions are being made. By bringing the two together, researchers can leverage the theories and tools of each discipline and, hopefully, conduct a more policy-relevant and consequential study.”
Part one of this four-part blog series highlights the value of incorporating qualitative methods into randomized evaluations and outlined specific tips for researchers. Part two talks about how qualitative research helped motivate and shaped the central question and hypothesis for a study on racial concordance between physicians and patients. Part four discusses how qualitative research helped the Oregon Health Insurance Experiment research team make sense of some of the study’s results. Part five highlights the value of qualitative research in providing a deeper understanding of mothers' experiences in the Baby's First Years study.
Authored By
J-PAL affiliate and Director of Opportunity Insights Raj Chetty reflects on partnering with J-PAL North America to evaluate an intervention called Creating Moves to Opportunity, which aims to help lower-income families in the United States to move to higher-opportunity areas.
Raj Chetty is the William A. Ackman Professor of Public Economics at Harvard University and a J-PAL affiliated researcher. Chetty was one of the Principal Investigators for the Creating Moves to Opportunity (CMTO) project, an ongoing collaboration between J-PAL-affiliated researchers and public housing authorities to introduce and evaluate interventions to “create moves to opportunity” for low-income families.
Chetty is also the Director of Opportunity Insights, a research center based at Harvard University, with the mission to identify solutions to help more children rise out of poverty and achieve better life outcomes in the United States. In response to COVID-19, Chetty and the Opportunity Insights Team developed The Opportunity Insights Economic Tracker, a publicly available platform that tracks real time economic activity at a granular level using anonymized private sector data. Using data from the tracker, Opportunity Insights researchers have outlined key insights on household spending patterns, the impact of policy efforts to date, as well as long-term solutions to the current economic crisis.
CMTO beginnings
Often as a researcher, you have an idea about something that can make a difference in the world. One such idea emerged from observational data we were collecting at Opportunity Insights. The data showed how children’s chances of rising up out of poverty varied sharply across different neighborhoods in Seattle. Yet we also noticed that, in spite of receiving housing vouchers (governmental rental assistance worth about $1,500 a month), low-income families tended to live in low-opportunity areas where poverty was likely to persist across generations.
This led us to the question: Why did low-income families tend to be segregated into low-opportunity areas where their kids are unlikely to escape poverty? Did they not want to live in other parts of Seattle that might be further away from their families or jobs and were perfectly happy to live where they resided? Or did they face barriers—like complicated housing voucher inspection processes and regulations, or lack of information and assistance in the housing search process—that prevented them from moving to higher upward mobility areas?
Answering these questions could help figure out how to get lower-income families to move to higher-opportunity areas, which could have a transformational impact on the lives of the children from these families. So we designed a simple intervention called the Creating Moves to Opportunity (CMTO) pilot that provided a set of services to families who applied for housing vouchers in Seattle. Through the CMTO pilot, randomly selected households were paired with a housing navigator to help them throughout the search process, connect them to landlords, and provide a small amount of financial assistance at critical points in the process.
Partnering with J-PAL North America
The challenge in turning this sort of idea into a study, especially for an individual researcher, is that it takes an enormous amount of work—work that often falls outside of the traditional domain of scientific research. In order to carry out a study to measure the pilot’s impact on the lives of families on the ground, it was critical for us to identify the right partners with the expertise and experience necessary to bring the idea to fruition.
Given its reputation, J-PAL quickly rose to the top of the list of potential partners. The number of studies we've seen come out of J-PAL have been some of the most important randomized evaluations in social science. We look to J-PAL for top-quality execution and for high fidelity in that execution. There are many logistical issues and challenges that researchers might not anticipate, and running an experiment in the real world requires specialized tools and juggling a lot of moving parts. J-PAL complements the technical expertise of researchers by working through such hurdles to actually create change in the field.
For the CMTO project, success also depended on our ability to work collaboratively not only with other academics, but also with people who were doing this work on the ground. J-PAL North America helped us determine that Public Housing Authorities (PHAs) were best placed to pilot the CMTO intervention and test them early on. Partnering with PHAs was what ultimately allowed us to test timely and policy-relevant questions related to housing.
We also needed to ensure that, throughout the project, all stakeholders were given a voice. This included housing authority staff who had spent their lives thinking about these issues, tenants who were actually seeking to make these moves, and landlords who faced various concerns themselves. J-PAL North America helped bring all of these groups together to have a constructive dialogue on how to design a program that could have the greatest impact.
Additionally, when you're doing an experiment of this scale, it typically takes several years from planning to implementation to see results. This requires keeping track of compliance with local regulations, meeting diverse stakeholder needs, and taking all the necessary steps to launch the study. J-PAL North America provided added capacity during critical phases of project development and management to oversee these details and ensure all stakeholders were coordinated in our execution. And with a foot in both the policy door and the academic door, J-PAL North America staff also served as a helpful resource in translating between researchers and practitioners.
Through our partnership with J-PAL North America, we went from an idea discussed in our offices, to ultimately visiting Seattle and meeting families who had moved to these different neighborhoods. And the most poignant moment during all of this was receiving a thank you card from a seven-year-old child who felt that her life had been transformed as a result of this pilot study.
Learning from preliminary results
The initial data from phase one of the study showed that the CMTO pilot really changed where families choose to live. In the first year of the study, we found that families who received the additional support services in the pilot were much more likely to move to high-opportunity areas. The rate of families moving to high opportunity places increased from fourteen percent in the control group to 55 percent in the treatment group. The majority of the children whose families participated in the pilot are now growing up in places where we estimate that they will go on to earn an average of $200,000 or more over their lifetimes as a result of this intervention.
To me, the data also gave a positive message about the world. It told us that segregation in America and many other countries is not the result of deep-rooted preferences that landlords or tenants hold, but rather barriers that can be overcome through changes in policy.
In phase two of the study, we are attempting to understand the mechanisms through which phase one had such a great impact. We want to understand, in greater detail, exactly why that bundle of interventions was so successful. We will also determine how we can optimize the program to reduce costs so we can make the program available to more families throughout the United States.
The case for more evidence—and the research partnerships that generate it
Evidence can play a pivotal role in reshaping policy debates. Political discourse often tends towards people's ideologies or prior convictions about what may or may not work. But when rigorous research brings evidence to bear on a politically charged issue, it can completely change the tenor of the conversation. As we’ve learned from CMTO, evidence can shift the focus from “What do I think is most important based on my initial beliefs?” to “What does the research show is best for our kids?”
Rigorous research results can bring people from different backgrounds together, cut through debates, and make real progress. This is the power of evidence, and the power of collaborative research partnerships that can help us put evidence into action.
Neighborhoods matter for the well-being of residents. Moving from lower-opportunity neighborhoods to higher-opportunity neighborhoods has been shown to improve later-life outcomes for children whose families move and may reduce the intergenerational persistence of poverty. Motivated by these findings, researchers are partnering with Seattle Housing Authority (SHA) and King County Housing Authority (KCHA) in Washington to evaluate whether mobility services are effective at encouraging families with children to move to higher-opportunity neighborhoods. Preliminary results from the first year of the ongoing study demonstrate that the mobility services significantly increased the share of families who moved to higher-opportunity areas, suggesting that residential segregation is largely driven by barriers to housing choice.
Policy issue
The United States budgeted over $18 billion in 2017 for the Housing Choice Voucher program (commonly referred to as Section 8). The Housing Choice Voucher program, administered by Public Housing Authorities (PHAs), provides eligible low-income families with rental assistance to pay for decent, safe, and sanitary housing in units they select.3 Previous research conducted by J-PAL affiliates Raj Chetty (Harvard), Nathaniel Hendren (Harvard), and Lawrence Katz (Harvard) on the Moving to Opportunity (MTO) project, which randomly assigned the offer of housing vouchers to families living in low-opportunity neighborhoods, found that young children (under 13 years of age) who moved to a higher-opportunity neighborhood with the voucher had substantially improved life outcomes nearly two decades later. Young children who moved to an opportunity neighborhood before age 13 earned 31 percent more on average in adulthood and were 32 percent more likely to attend college.4 However, as of 2017 only approximately 15 percent of Housing Choice Voucher (HCV) recipients lived in neighborhoods where fewer than 10 percent of residents were poor.5 There is growing interest in identifying effective strategies to help families move to higher-opportunity areas. There are a variety of hypotheses for why more families with vouchers do not choose to live in high-opportunity neighborhoods, including housing and moving costs, lack of information about high-opportunity neighborhoods, landlord constraints, and discrimination. What are effective strategies to increase families’ ability to move to higher-opportunity neighborhoods?
Context of the evaluation
In the Seattle metro area in Washington State, children who grow up in different neighborhoods have substantial differences in life outcomes and economic mobility. Preliminary assessments by the Seattle Housing Authority and the King County Housing Authority have shown that many families with HCVs currently live in low-opportunity neighborhoods.
This evaluation is part of the broader Creating Moves to Opportunity project that aims to better understand how to facilitate moves to higher-opportunity neighborhoods for families with young children. The study is being conducted with approximately 1,300 low-income families who have received HCVs in lotteries through SHA or KCHA.6 To be eligible for the study, families must meet HCV eligibility requirements (including income levels, criminal history, and debt owed to housing authorities) at the relevant PHA and have at least one child under 15 in their household. In this evaluation, higher-opportunity neighborhoods were selected based on the underlying data from the Opportunity Atlas as well as logistical input from the PHAs to ensure sufficient affordable housing stock availability in selected areas and ease of communication about opportunity areas to families.
Details of the intervention
Researchers are evaluating the effectiveness of mobility services that aim to help low-income families with children move to higher-opportunity neighborhoods. Eligible families who participate in the study are randomly assigned to either the treatment or control group.
During the first phase of this evaluation, treatment families have access to all of the services outlined below. Researchers will use preliminary data gathered in the first phase of the project to refine interventions for a second phase where families will be randomly assigned to multiple treatment groups receiving refined interventions and a control group. In both phases, families in the control group have access to standard services provided by their respective PHA.
The mobility services include interventions for both families and landlords that are primarily implemented by a nonprofit contractor. The family-facing services include:
- Rental application coaching to help families prepare to lease, including resolving or preparing families to discuss poor credit or rental histories, and coaching on how to work with landlords;
- Opportunity area education, including information on schools, amenities, and community resources;
- Housing search assistance to support families’ in seeking out rental units, including referrals to units where landlords are interested in leasing to families participating in CMTO; and
- Flexible financial assistance that families can use towards application fees, security deposits, and moving costs.
The landlord-facing services include:
- An expedited lease-up process to minimize administrative delays for landlords who lease properties to families participating in the study in opportunity neighborhoods.
- A damage mitigation fund that provides landlords with up to $2,000 in reimbursement for damages above and beyond the security deposit of units leased by program participants.
- Additional outreach conducted by contractor staff to landlords in opportunity neighborhoods to increase the number of units available to families.
Results and policy lessons
Preliminary results from phase one of the study demonstrate that the mobility services substantially increased the share of families who chose to move to higher-opportunities. Fifty-four percent of families who received the mobility services chose to move to high-opportunity areas, compared to 14 percent of families who received standard services in the control group (a 286 percent increase).
The mobility services did not have any impact on the percentage of families who successfully use their voucher to lease up in a unit. The mobility services therefore changed where families moved, not whether they moved.
On average, there were no differences between families in the treatment and control groups on measures of housing quality such as unit size or age. There was also no difference in families’ distance to their prior location or proximity to jobs. However, the average monthly rent was $186 higher for families in the treatment group compared to the control group. After moving, families in the treatment group were 25 percentage points more likely to report being satisfied with their new neighborhood than those in the control group.
The preliminary results demonstrate that many low-income families live in lower-opportunity areas not because of any preference for such neighborhoods, but rather because of barriers which prevent them from moving to higher-opportunity areas. Qualitative evidence from interviews with a subset of families confirmed that many families would prefer to move to higher-opportunity neighborhoods, but various barriers prevent them from doing so. The interviews suggest that the mobility services were particularly effective due to the program’s ability to customize service according to each family’s specific needs and circumstances.
The researchers suggest that implementing similar housing mobility programs for families with Housing Choice Vouchers across the United States could significantly reduce residential segregation and improve economic mobility.
Chetty, Raj, Nathaniel Hendren, and Lawrence F. Katz. 2016. "The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment." American Economic Review, 106 (4): 855-902.
J-PAL Policy Briefcase. 2015. “Moving to Opportunity.” Cambridge, MA: Abdul Latif Jameel Poverty Action Lab.
J-PAL North America reflects on using qualitative research methods in randomized evaluations, and summarize a few practical tips for those interested in integrating a qualitative approach into their studies.
Randomized evaluations allow researchers to measure the impact of programs and policies on a range of outcomes. Using this approach in North America, J-PAL researchers have recently examined a wide range of topics, including the effects of Medicaid on rates of health care utilization and the impact of a housing mobility program on the likelihood of families moving to lower-poverty neighborhoods.
But what mechanisms are driving the effects of these programs and policies? How did the context, design, and implementation of the program or policy influence the result? If replicated in a different context, will the program have the same effects? Is the study asking the right question?
Researchers can often collect quantitative data and design evaluations to shed light on these types of questions, but there’s always more to learn. Qualitative methods, such as direct observation, in-depth interviews, and focus groups, allow researchers to dive into these questions by examining participants’ beliefs, attitudes, experiences, and perspectives. Data gleaned from these methods can help researchers gain insight into potential mechanisms or barriers, generate new hypotheses and questions, and understand the stories behind the quantitative results.
For decades, social science scholars within anthropology, sociology, and psychology have employed qualitative methods. In recent years, many researchers within the traditionally quantitative field of economics have also incorporated qualitative methods into their studies and built teams with qualitative expertise to strengthen their research.
From our conversations with several researchers who've conducted and relied on qualitative research methods as part of J-PAL-supported randomized evaluations, we've summarized a few practical tips for those interested in integrating a qualitative approach into their studies:
- While developing your randomized evaluation, don't discount questions that can be best addressed through qualitative methods. These questions may challenge certain assumptions or shed light on mechanisms, contexts, or outcomes that quantitative methods may not fully capture. For example, researchers may want to gain insight into the experience of staff implementing a particular program to identify the challenges and barriers they faced, understand their perception of the program’s successes or shortcomings, and identify potential obstacles to longer-term implementation or scale-up. While this may be difficult to assess in a survey, focus groups and qualitative interviews could provide valuable insights.
- Account for qualitative research in study proposals and budgets. Qualitative research can require a high time commitment and can benefit from the support of specialized team members.
- Cultivate relationships with implementing partners. Forming a strong relationship with implementing partners is one key component to a successful and policy-relevant study and can help build a foundation for conducting qualitative research. Implementing organizations interact closely with study participants and often play instrumental roles in shaping the design and implementation of randomized evaluations. They are also well-placed to help researchers determine the best approaches to carrying out the qualitative parts of a study.
- Diversify your research team. Consider building a research team of individuals from different disciplines. Scholars of psychology, anthropology, sociology, and social work often have extensive experience with qualitative methods and bring valued perspectives that economists may be missing.
This blog series highlights three examples of J-PAL research teams using qualitative research methods to inform and strengthen the design, implementation, and analysis of their randomized evaluations. For part two of the series, we interviewed Professor of Public Policy and US Health Care Delivery Initiative Co-Chair Dr. Marcella Alsan about how qualitative research helped motivate and shaped the central question and hypothesis for a study on racial concordance between physicians and patients. In part three, we spoke with Professor of Sociology & Social Policy Stefanie Deluca about how the Creating Moves to Opportunity randomized evaluation, a study she co-led, embedded qualitative research methods into its study design. Part four features a conversation with Associate Professor of Social Work and Oregon Health Insurance Experiment co-author, Heidi Allen, on how qualitative research helped the research team make sense of some of the study’s results. The series concludes with part five, where we spoke with researchers from the the Baby's First Years study about the value of qualitative research in providing a deeper understanding of mothers' experiences.