The LA Homelessness Evaluation Network: Lessons learned from supporting organizations build evidence and evaluation capacity

On one night in January 2022, over 580,000 people were experiencing homelessness in the United States. Roughly one-third of this population was living in California, with over one-tenth—more than 65,000 people—in Los Angeles (LA) County alone. Given the scale of homelessness in LA and the urgency for solutions for those in need, it is more important than ever to generate evidence to address unanswered questions on what works best to combat homelessness and why. J-PAL North America is partnering with the Conrad N. Hilton Foundation to offer LA-based homeless service providers a supportive “community of practice” around rigorous research. Through this effort, we aim to bolster organizations’ ability to generate and use evidence to inform decision-making as they support communities experiencing homelessness. 

Homelessness in Los Angeles

LA County is home to the largest number of individuals experiencing homelessness in any of the nation’s Continuums of Care and the largest number of unsheltered people in the country, with many sleeping in tents, makeshift shelters, and vehicles. Homelessness affects thousands of individuals and families each day who are in need of a safe place to call home and who face a myriad of related challenges such as financial stability, health, and safety. 

This vast and growing issue, which disproportionately affects Black, Latino/a, and Indigenous communities, prompted LA City Mayor Karen Bass to declare a state of emergency on homelessness when she took office last year. This decree enables more resources to go toward fighting homelessness, and highlights the need for further evaluation on homelessness reduction and prevention strategies to ensure resources are used effectively. Emergency declarations on homelessness have since been issued by LA County and other cities in the region.
 
There are multiple drivers of homelessness in the LA region, including a lack of affordable housing, high costs of living, and the need for increased support for individuals with mental illness and substance use disorders. It is estimated that LA County has a rental vacancy of just 3.5 percent and faces a shortage of nearly 500,000 affordable homes. In the city of LA, it is estimated that for every 207 people who find stable housing each day, another 227 become homeless. Some of LA’s zoning policies also limit the ability to build multi-family housing, which contributes to the tight market. Building affordable housing can be an expensive and lengthy process, with some affordable housing developments in California costing more than $1 million per apartment to build. 

Gaps in evidence related to homelessness reduction and prevention 

Across LA County’s 88 cities, there is an extensive network of organizations and stakeholders working diligently to support unhoused families and individuals. Yet they face several challenges in conducting this work, including resource and capacity constraints and a lack of rigorous evidence about which policies and programs are the most effective. In order to maximize the impact of the emergency declarations in the region and most effectively allocate resources to combat homelessness, it is critical that policymakers and service providers invest in and implement evidence-informed programs. 

Various evaluations have identified some promising programs related to homelessness reduction and prevention that have informed policy change. For example, the Housing First model—which prioritizes the provision of permanent housing with support and no preconditions—has grown in popularity in recent decades, in part due to evidence on its effectiveness. 

There are numerous open and testable research questions about the best ways to combat homelessness, particularly for different populations and in different contexts. In addition to existing evidence-informed models, organizations and government agencies are trying out new, innovative solutions. For example, Mayor Bass issued an executive directive to explore turning city property into temporary and permanent housing in order to increase housing supply at lower costs. Innovations such as this are essential for combatting such a complex challenge. They also create opportunities to generate data and share lessons on what works.

Supporting LA-based service providers to generate policy-relevant evidence 

To lay the foundation for evidence generation and use among LA-based homeless service providers, J-PAL North America—in partnership with the Conrad N. Hilton Foundation—is creating a supportive “community of practice” around rigorous research: the LA Homelessness Evaluation Network. This model builds from J-PAL North America’s previous partnership engagement work, including the Bay Area Evaluation Incubator, which supports Bay Area service providers in implementing and evaluating cash transfer programs. 

Over the next year, J-PAL North America will work to establish a network of LA-based organizations that are well-positioned to design and develop randomized evaluations of their programs that can ultimately help address open and critical questions related to homelessness prevention and housing stability. LA Homelessness Evaluation Network participants will attend workshops on impact evaluation and ethical evaluation design, receive small group support from J-PAL staff in scoping potential evaluation ideas, and have opportunities for peer learning and connection. Activities will be informed by and tailored to the organizations’ needs, being mindful of the day-to-day challenges organizations are facing as they work tirelessly to address homelessness in their communities.

Those interested in learning more about this engagement can contact Bridget Mercier, J-PAL North America’s policy and training manager, for more information. Information on using rigorous impact evaluations to answer policy questions related to homelessness and housing stability can be found on our website. If you would like to receive updates on our work in LA as well as other projects, subscribe to our Addressing Homelessness and Housing Stability newsletter. 

This post was originally published on Housing Matters, an initiative of the Urban Institute.

Understanding the impact and effectiveness of housing programs and policies is critical to ensuring they achieve their goals and help the people they serve. For example, these considerations include which types of housing and housing modifications should be prioritized to help support older adults as they age.

To improve people’s outcomes, policymakers should be investing in evidence-based programs. Randomized evaluations are one rigorous method researchers often use to evaluate the impact of certain policies.

To demystify randomized evaluations and underscore their value for policymakers and practitioners, we spoke with Bridget Mercier, policy and training manager at J-PAL North America, a regional office of the Abdul Latif Jameel Poverty Action Lab (J-PAL), a research lab based at the Massachusetts Institute of Technology that focuses on this type of research. J-PAL helps to match researchers and practitioners who are interested in impact evaluation, and they help build capacity for organizations hoping to do randomized evaluations.

What is unique about randomized evaluations as a method?
Randomized evaluations are often considered the most rigorous type of impact evaluation. Generally, studies try to find cause and effect, but this is difficult because you can’t observe what would have happened if a program or policy did not take place. Being able to estimate this difference helps us compare the impacts for a group that received an intervention with those that did not, establishing what is known as a counterfactual.

In randomized evaluations, people are randomly assigned to receive the program or to a comparison group, and because they’re assigned at random, we can be confident they don’t differ systematically at the start of the study. This means both groups will have similar characteristics, including those we can easily measure and those we can’t. So we know we can attribute the outcomes to the intervention itself, rather than any underlying factors.

As part of our work in the California Bay Area investigating the impact of cash transfers on housing stability, randomization ensures the group that receives the cash transfer isn’t different in terms of observable characteristics, like age or household size, than the group that didn’t receive the cash transfer, or in characteristics such as motivation or self-selection, which are more difficult to observe. When we have relevant policy questions, randomization can give us trustworthy answers. Researchers also work to ensure their randomized evaluations are optimized to detect impact.

How can randomized evaluation contribute to the evidence base on housing?
The scale of homelessness is immense; nearly 600,000 people are experiencing homelessness on a given night, and more than a million access shelter services over the course of a year. Yet lots of folks are working tirelessly to end homelessness. Those working on issues of homelessness and housing stability need to know if what they’re doing is working. Specifically, randomized evaluations can generate rigorous evidence to determine which strategies are most effective at preventing or reducing homelessness. While randomized evaluations aren’t the best for every housing question, there are many ways that they have been helpful to shifting narratives, especially on issues of housing instability and homelessness.

In J-PAL North America’s evidence review on reducing and preventing homelessness, we summarize results from 40 rigorous evaluations related to housing and homelessness. The review highlights strategies proven to be effective, such as Housing First, which prioritizes immediate housing without preconditions. Evidence has played a fundamental role in the shift toward the Housing First approach, with several randomized evaluations showing that a Housing First approach and permanent supportive housing can more effectively help people who are experiencing chronic homelessness than the traditional shelter system or transitional housing.

Our evidence review also describes where more research is needed, such as identifying which bundles of services and housing supports are most effective, and for whom. There are many open and testable questions in the field and opportunities for randomized evaluations.

Why should policymakers care about randomized evaluations?
Policymakers should care about randomized evaluations because they are one of the most powerful tools to determine if a policy or program is effective. Understanding program impact is vital to ensuring money is spent on programs that make a difference, especially given the size and scope of homelessness. We also know systematic inequality puts certain people at a disadvantage—such as people of color, members of the LGBTQ community, and survivors of domestic violence—and the findings from randomized evaluations may help us advance more equitable policies by assessing how specific policies or programs may be more or less effective for different populations.

Randomized evaluations can also help us understand why we saw certain results by examining mechanisms for impact. This can help policymakers refine their models and help them apply learnings to new contexts.

How have the results from randomized evaluation affected people they study?
Insights gained from randomized evaluations have shifted narratives and policy agendas, such as the Housing First approach and permanent supportive housing, as well as housing choice vouchers. In the case of Housing First, cities and states began implementing Housing First and permanent housing on a wider scale after randomized evaluations demonstrated reductions in homelessness. Randomized evaluations have also demonstrated that housing vouchers can be effective in both reducing homelessness and increasing opportunities for economic mobility. For example, long-term findings from the 1990 Moving To Opportunity study led to policy changes in recent years at both the local and federal levels, expanding to channel over $50 million to housing mobility services to help families move to neighborhoods with greater economic opportunities and expand housing choices for thousands of low-income families in the United States.

As we learn more about what works to reduce and prevent homelessness, J-PAL aims to translate research into action by supporting the expansion and scale up of effective programs to reach more people and have a greater impact.

What are some of the challenges of randomized evaluations?
Ethics are a top priority for any research, and there are ethical considerations when conducting a randomized evaluation. For example, the comparison group shouldn’t be blocked from a program or support they would otherwise have access to (especially one that we already know is effective based on prior research). The comparison group receives “treatment as usual,” or can receive a different version of the intervention rather than nothing at all. When resources are scarce and it isn’t possible to serve everyone, using a lottery may be a fair and ethical way to allocate services, thus creating the opportunity for a comparison group.

Not every research question can be effectively or ethically answered using a randomized evaluation, and we believe the research question itself should determine the best method of evaluation. We don’t encourage randomized evaluations when they aren’t appropriate, such as when there are enough existing resources to serve everyone and we already know the program works.

However, in many cases we don’t know if an intervention is effective, and there are often not enough resources to serve everyone. In these instances, a randomized evaluation can be helpful to provide this evidence or shift resources away from ineffective programs, depending on the results. But of foremost importance is that the rights and welfare of study participants must be maintained.

What recommendations do you have for nonresearchers reading studies employing randomized evaluations? What would you tell people to look out for to understand the findings?
Focus on the applicability of findings to your setting. Consider the context of the evaluation at hand, especially when reviewing studies to inform local policy. It’s important to think about generalizability from one place to another. A framework we use at J-PAL asks a number of important questions, such as is there a similar problem and why did this solution work? What are the local conditions and underlying behavioral factors? What does local implementation look like? Using these questions can help us understand what the policy and program implications of each study may be in a different context.

Statistical power is also critical in randomized evaluations; it tells you how likely you are to detect meaningful changes. Without adequate statistical power in a study, we won’t learn very much, so we need to consider the statistical power that each study brings. A key element of statistical power is sample size, or the total number of people or units in a study. Larger samples are more likely to be representative and more likely to capture impacts of the population. When the sample size is too small, we’re not able to tell if the observed outcome is a result of the program or pure chance.

The Bay Area Evaluation (BAE) Incubator supports Bay Area service providers with implementing and evaluating cash transfer programs to assess their impact on homelessness and housing stability. In part two of our BAE Incubator partner series, Compass Family Services reflects on their experiences as an Incubator partner. Nicole Moler, Impact Analyst, discusses how the BAE Incubator reinforces Compass’s culture of evidence and tells prospective Incubator partners to “go for it.”

Compass Family Services is a community-based organization serving San Francisco families experiencing or at risk of homelessness. It has supported vulnerable San Franciscans for over 100 years. As a member of our Impact + Learning team, my main goal is to learn; to better understand the impact of our work and to iterate on programs to better serve our families. However, before the BAE Incubator, we didn’t have the expertise to understand our impact with the rigor of a randomized evaluation. 

When we applied to the BAE Incubator last summer, we hoped to take our evaluation capacity to the next level. Our goal was to take all of the information we had—all of the assumptions about what we thought we knew—and really test it using rigorous data. 

Our staff is very excited about the idea of cash transfers, and many believe that it will be the best thing for our families in rapid re-housing. We’ve had the opportunity to give small amounts of funds to some clients before and have seen the difference it can make but haven’t had the data to back up our observations. We are excited to explore randomized evaluation because we want to really know if cash transfers will work at a larger scale. At the same time, Compass has a desire to make sure the evaluation is done carefully, ethically, and rigorously. This is where J-PAL’s expertise comes in.

As a naturally skeptical person, I wasn’t exactly sure what to expect from the BAE Incubator. Moreover, while we were excited to dig deeper into rigorous evaluation, we were also a bit overwhelmed by the prospect of taking on a randomized evaluation given the many moving pieces. Because of these concerns, I have really appreciated that the Incubator has been an intentional, steady process. From the beginning, the focus on simply determining feasibility made me feel comfortable. Since then, each subsequent component has assured me that we are building the right blocks. 

The training we received through the BAE Incubator provided the opportunity to get in the weeds about some of the more technical concepts related to both randomized evaluations and evaluation in general. We even shared some training materials with other programs at Compass, which helped reinforce the data-driven culture of evidence we’re building. We also took advantage of the opportunity to attend J-PAL’s week-long Evaluating Social Programs course, which sparked several additional RCT ideas. (For another time, perhaps!)

The Incubator also includes regular, small group technical assistance sessions. When we meet with J-PAL staff members Amanda and Anisha, we have the chance to dig into questions specific to our design. They’ve helped us build confidence in both the feasibility of our project and our capacity to carry it out. They acknowledge that there can and will be challenges, but also that there are often ways to address them. The technical assistance has helped us think through potential unintended consequences of this project so that we can address them proactively. While there are times I feel as though we can’t do this, the next time we meet, I am reassured that we can. 

Through this engagement, we’ve really been able to sharpen our technical skills. We’re also learning how to ask the right questions to advance our evaluation goals, particularly when working directly with program staff. These tools help us not only think through the randomized evaluation we’re designing but also enhance our other work. We’ve been able to plant evaluation seeds across Compass’s many programs and spark conversations with staff about rigorous data.

For any organizations thinking about applying to the J-PAL North America Housing Stability Evaluation Incubator, I say go for it! They have a deliberate, transparent, and thoughtful process focused on determining feasibility. And if a randomized evaluation of your program is not feasible, that’s okay! They’ll still help you add tools to your toolbox. J-PAL staff provide space for organizations to test out ideas and ask questions. Don’t be afraid: trust the process.

Part one of this blog series explores existing evidence on cash transfers and highlights the need for further evaluation in the context of homelessness reduction and prevention in the United States. In parts three and four, Abode Services and Hamilton Families reflects on their experiences as an Incubator partner

The Homelessness and Housing Stability team at J-PAL North America will be accepting Letters of Interest for the Housing Stability Evaluation Incubator from August 1–October 17, 2022. Public housing authorities, continuums of care, nonprofit organizations, and other service providers working to reduce and prevent homelessness are encouraged to apply.