LEVER Partner Spotlight: Ricardo Basurto-Dávila on the County of San Diego’s Shallow Rent Subsidy program evaluation

In this Partner Spotlight series, J-PAL North America and Results for America highlight the government leaders and decision makers who are harnessing data, evaluation, and evidence to advance economic mobility via our Leveraging Evaluation and Evidence for Equitable Recovery (LEVER) programming.

This piece features the work of Ricardo Basurto-Dávila, the Chief Evaluation Officer in the County of San Diego’s Office of Evaluation, Performance and Analytics. In this post, we share how Ricardo is using resources from LEVER to advance the committed culture of learning and improvement in San Diego.

Can you describe your role in the County of San Diego and how it connects to data and evidence generation and use? 
I'm the chief evaluation officer for the County of San Diego. I lead the Office of Evaluation, Performance, and Analytics, which provides leadership, coordination, and capacity building for the County in our efforts to build a culture of learning and evidence-based policy making.

What have been some of your biggest priorities in that role?
A main one is our strategic research plan, which we have a mandate to update annually and execute on. The one from this year identifies six initial research and evaluation projects for our office. It also has a learning agenda with 33 high-priority questions that we developed by engaging with about 40 departments and agencies across the county to build a shared vision that will guide our work in years to come. 

We're additionally going to develop an inventory of all programs in the county and the level of evidence underpinning them. That will help us to identify which programs are aligned to our learning agenda and which programs may benefit from more evidence regarding their effectiveness or efficiency, which can shape future research priorities.

How did engaging with the two-part LEVER workshop series on Evidence and Evaluation Foundations shape your work?
I was actually only there “live” for workshop two. I was unable to attend the first one, but I worked through Results for America’s self-assessment tool from that workshop on my own. It helped me realize that we’re in a pretty good spot in San Diego and have a lot of foundational pieces in place. But it also showed me where there’s more for us to do and how we specifically can continue working to reach the ideal state. 

After completing that assessment, I became even more interested in engaging with LEVER. I made sure that I’d be able to attend the second workshop precisely because the self-assessment showed me there was real value.

Do you have any advice for other government leaders seeking to do this kind of work?
One recommendation is to start by clarifying ideas and building consensus around what your goals are and what you’re trying to achieve. Most of us in government really want things to work, and we want to help improve the quality of life for the people in our communities. It’s important to show that data and evaluation are tools to remove uncertainty and provide more information to inform critical decisions.

Another recommendation is to think about the full picture of how this work will happen. It’s critical that research and evaluation efforts align with a government’s strategic priorities and come directly from department and agency staff. It's not easy to do something like this piecemeal or in a silo. You need to be very deliberate about putting the structure in place to have a team that will have capacity and expertise, collaborate with staff and agencies who can guide your priorities, and reach the people who will be making decisions.

I’d also recommend the self-assessment tools from the LEVER workshops for any governments interested in advancing their capacity and culture of evaluation and evidence—they really helped us identify areas of strength and specific things to pursue next.

Would you recommend LEVER programming to other government leaders who are interested in more effectively utilizing data and evidence in their work?
Absolutely. We had a peer convening a few weeks ago with some other jurisdictions, and I was struck by the variety and range of capacities and areas of focus. LEVER hasn’t tried to get us to just fit one mold. J-PAL North America and Results For America are working with everyone to assess where we are, then supporting us the way we each need to move forward. That combination of flexibility and expertise is the critical thing, and exactly what I was hoping for.

Raj Chetty is the William A. Ackman Professor of Public Economics at Harvard University and a J-PAL affiliated researcher. Chetty was one of the Principal Investigators for the Creating Moves to Opportunity (CMTO) project, an ongoing collaboration between J-PAL-affiliated researchers and public housing authorities to introduce and evaluate interventions to “create moves to opportunity” for low-income families.

Chetty is also the Director of Opportunity Insights, a research center based at Harvard University, with the mission to identify solutions to help more children rise out of poverty and achieve better life outcomes in the United States. In response to COVID-19, Chetty and the Opportunity Insights Team developed The Opportunity Insights Economic Tracker, a publicly available platform that tracks real time economic activity at a granular level using anonymized private sector data. Using data from the tracker, Opportunity Insights researchers have outlined key insights on household spending patterns, the impact of policy efforts to date, as well as long-term solutions to the current economic crisis.

CMTO beginnings

Often as a researcher, you have an idea about something that can make a difference in the world. One such idea emerged from observational data we were collecting at Opportunity Insights. The data showed how children’s chances of rising up out of poverty varied sharply across different neighborhoods in Seattle. Yet we also noticed that, in spite of receiving housing vouchers (governmental rental assistance worth about $1,500 a month), low-income families tended to live in low-opportunity areas where poverty was likely to persist across generations.

This led us to the question: Why did low-income families tend to be segregated into low-opportunity areas where their kids are unlikely to escape poverty? Did they not want to live in other parts of Seattle that might be further away from their families or jobs and were perfectly happy to live where they resided? Or did they face barriers—like complicated housing voucher inspection processes and regulations, or lack of information and assistance in the housing search process—that prevented them from moving to higher upward mobility areas?

Answering these questions could help figure out how to get lower-income families to move to higher-opportunity areas, which could have a transformational impact on the lives of the children from these families. So we designed a simple intervention called the Creating Moves to Opportunity (CMTO) pilot that provided a set of services to families who applied for housing vouchers in Seattle. Through the CMTO pilot, randomly selected households were paired with a housing navigator to help them throughout the search process, connect them to landlords, and provide a small amount of financial assistance at critical points in the process.

Partnering with J-PAL North America

The challenge in turning this sort of idea into a study, especially for an individual researcher, is that it takes an enormous amount of work—work that often falls outside of the traditional domain of scientific research. In order to carry out a study to measure the pilot’s impact on the lives of families on the ground, it was critical for us to identify the right partners with the expertise and experience necessary to bring the idea to fruition.

Given its reputation, J-PAL quickly rose to the top of the list of potential partners. The number of studies we've seen come out of J-PAL have been some of the most important randomized evaluations in social science. We look to J-PAL for top-quality execution and for high fidelity in that execution. There are many logistical issues and challenges that researchers might not anticipate, and running an experiment in the real world requires specialized tools and juggling a lot of moving parts. J-PAL complements the technical expertise of researchers by working through such hurdles to actually create change in the field.

For the CMTO project, success also depended on our ability to work collaboratively not only with other academics, but also with people who were doing this work on the ground. J-PAL North America helped us determine that Public Housing Authorities (PHAs) were best placed to pilot the CMTO intervention and test them early on. Partnering with PHAs was what ultimately allowed us to test timely and policy-relevant questions related to housing.

We also needed to ensure that, throughout the project, all stakeholders were given a voice. This included housing authority staff who had spent their lives thinking about these issues, tenants who were actually seeking to make these moves, and landlords who faced various concerns themselves.  J-PAL North America helped bring all of these groups together to have a constructive dialogue on how to design a program that could have the greatest impact.

Additionally, when you're doing an experiment of this scale, it typically takes several years from planning to implementation to see results. This requires keeping track of compliance with local regulations, meeting diverse stakeholder needs, and taking all the necessary steps to launch the study. J-PAL North America provided added capacity during critical phases of project development and management to oversee these details and ensure all stakeholders were coordinated in our execution. And with a foot in both the policy door and the academic door, J-PAL North America staff also served as a helpful resource in translating between researchers and practitioners.

Through our partnership with J-PAL North America, we went from an idea discussed in our offices, to ultimately visiting Seattle and meeting families who had moved to these different neighborhoods. And the most poignant moment during all of this was receiving a thank you card from a seven-year-old child who felt that her life had been transformed as a result of this pilot study.

Learning from preliminary results

The initial data from phase one of the study showed that the CMTO pilot really changed where families choose to live. In the first year of the study, we found that families who received the additional support services in the pilot were much more likely to move to high-opportunity areas. The rate of families moving to high opportunity places increased from fourteen percent in the control group to 55 percent in the treatment group. The majority of the children whose families participated in the pilot are now growing up in places where we estimate that they will go on to earn an average of $200,000 or more over their lifetimes as a result of this intervention. 

To me, the data also gave a positive message about the world. It told us that segregation in America and many other countries is not the result of deep-rooted preferences that landlords or tenants hold, but rather barriers that can be overcome through changes in policy.

In phase two of the study, we are attempting to understand the mechanisms through which phase one had such a great impact. We want to understand, in greater detail, exactly why that bundle of interventions was so successful. We will also determine how we can optimize the program to reduce costs so we can make the program available to more families throughout the United States.

The case for more evidence—and the research partnerships that generate it

Evidence can play a pivotal role in reshaping policy debates. Political discourse often tends towards people's ideologies or prior convictions about what may or may not work. But when rigorous research brings evidence to bear on a politically charged issue, it can completely change the tenor of the conversation. As we’ve learned from CMTO, evidence can shift the focus from “What do I think is most important based on my initial beliefs?” to “What does the research show is best for our kids?”

Rigorous research results can bring people from different backgrounds together, cut through debates, and make real progress. This is the power of evidence, and the power of collaborative research partnerships that can help us put evidence into action.

In the United States, failure to appear (FTA)—when those awaiting court processing in the community fail to appear in court for arraignment—is a misdemeanor offense that can lead to additional fines and arrest for otherwise minor infractions. Current research shows that misdemeanor charges leading to incarceration perpetuate the cycle of domestic poverty. Lowering FTA rates may reduce misdemeanor charges and subsequent incarcerations that contribute to institutional income barriers. 

Through a 2019 J-PAL North America Evaluation Incubator (then “Innovation Competition”), Shasta County Superior Court connected with J-PAL affiliated researchers Emily Owens (University of California, Irvine) and CarlyWill Sloan (West Point) to evaluate strategies to reduce FTA rates in Shasta County, California. Their study, which was awarded funding from J-PAL North America, evaluates the impact of a text messaging program on increasing court appearances for both housed and unhoused populations in Shasta County. 
We spoke with Principal Investigator Emily Owens and Project Manager Shawn Watts of Shasta County Superior Court to hear more about their researcher-practitioner partnership. 

Why was Shasta County interested in evaluating an intervention to reduce FTA in court? 

Shawn Watts: For approximately fifteen years, the jail in Shasta County has operated at capacity, with the court currently experiencing FTA rates above 50 percent.
Approximately 30 percent of those who FTA in our county are experiencing homelessness, so we were interested in creating interventions that could apply to a large portion of our FTA population.  We typically have trouble communicating with those experiencing homelessness because they don’t have a stable address, and traditional mail is the primary means of communication from the court to defendants.  We hoped that in our effort to reduce FTAs among this population, we also could develop better ways to communicate to all defendants.

What drew you to take on this project as a Principal Investigator? 

Emily Owens: Navigating the criminal justice system is hard, especially when someone has pressing financial, health, or family issues. Identifying straightforward ways to help people get to their court date can make a big difference in helping defendants resolve their cases. There is evidence that simple reminders can make a difference in people’s ability to get to court, but that evidence has usually come from studies in big cities and has explicitly excluded vulnerable people without stable housing. I was interested in working with the Shasta County Superior Court to minimize the extent to which people become ensnared in the justice system because of their economic circumstances.
I was also intrigued by the Court’s interest in doing outreach to this particular population. How would people respond to the Court collaborating with other service providers to remind them of their court dates? I was curious to find out, especially in a rural context like Shasta County. 

How did the research question and intervention change throughout the technical assistance process? 

Emily: Shawn and the research team brainstormed possible outreach partners and ways to increase court attendance. We talked with the police, a local health care provider, and ultimately decided to work with the Good News Rescue Mission.  We ended up focusing on this partnership based on a combination of implementation feasibility and impact size.  
After reviewing the background literature, it became clear that we first needed to test the applicability of current findings about text message reminders and FTAs within our population. Furthermore, it wasn’t obvious that text messages would not reduce FTAs among people experiencing homelessness.  A first-order question was to figure out if the existing FTA research on the role of simple reminders in encouraging court appearances was generalizable to non-urban and higher-income contexts.

Preliminary results indicate that text message reminders decrease the FTA rate within Shasta County in general. However, initial results also reveal that text messaging does not reduce FTA amongst community members experiencing homelessness. How has this project changed how you think about FTA interventions and how you are thinking about future research? 

Emily: I was glad that we were able to track how many people received the reminder text messages.  It was striking to me how there is a strong push for data-driven approaches to criminal justice reform that rely on administrative data, yet our RCT found that the contact information the Court had about people was frequently just not right.  Moving forward, I am going to think critically about where administrative records are “sourced” (in the case of phone numbers, these are provided by someone who is being arrested), and what incentives the administering agency has to ensure data accuracy (for instance, before this evaluation, the Court hadn’t been as focused on getting accurate cell phone numbers). 

Did the Shasta County court system make any changes to its processes as a result of this randomized evaluation? 

Shawn: Yes.  Before conducting this evaluation, we were aware that texting defendants may reduce FTAs, but we were unsure if texting could be effective in a rural setting such as ours, particularly among those experiencing homelessness.  This study showed us that texting could reduce FTAs in our general population, especially if we had reliable cell phone numbers for our defendants, although it also showed us that texting was unlikely to reduce the number of FTAs in the unhoused population. Consequently, we have purchased a texting module for our new case management system, and as soon as it is operational we will text all defendants about their upcoming court dates.  We also have increased our efforts to improve our cell phone information quality by encouraging our clerks, the District Attorney’s office, Probation staff, and law enforcement to gather cell phone numbers during each contact with a defendant.

This piece is the second of a two-part blog series highlighting our research partnership in Shasta County. In the first article, we spoke with Project Manager Shawn Watts of Shasta County Superior Court to discuss the process of designing a randomized evaluation through the State and Local Evaluation Incubator. 

This work is also part of an ongoing series highlighting research partnerships with state and local government agencies fostered through J-PAL North America’s State and Local Evaluation Incubator. The first piece features Minnesota Management and Budget and the Minnesota Board of Pharmacy’s evaluation of the state’s prescription monitoring program (PMP).