Beyond the Incubator: Building a community of practice

The Bay Area Evaluation (BAE) Incubator supports Bay Area service providers with implementing and evaluating cash transfer programs to assess their impact on homelessness and housing stability. In part two of our BAE Incubator partner series, Compass Family Services reflects on their experiences as an Incubator partner. Nicole Moler, Impact Analyst, discusses how the BAE Incubator reinforces Compass’s culture of evidence and tells prospective Incubator partners to “go for it.”

Compass Family Services is a community-based organization serving San Francisco families experiencing or at risk of homelessness. It has supported vulnerable San Franciscans for over 100 years. As a member of our Impact + Learning team, my main goal is to learn; to better understand the impact of our work and to iterate on programs to better serve our families. However, before the BAE Incubator, we didn’t have the expertise to understand our impact with the rigor of a randomized evaluation. 

When we applied to the BAE Incubator last summer, we hoped to take our evaluation capacity to the next level. Our goal was to take all of the information we had—all of the assumptions about what we thought we knew—and really test it using rigorous data. 

Our staff is very excited about the idea of cash transfers, and many believe that it will be the best thing for our families in rapid re-housing. We’ve had the opportunity to give small amounts of funds to some clients before and have seen the difference it can make but haven’t had the data to back up our observations. We are excited to explore randomized evaluation because we want to really know if cash transfers will work at a larger scale. At the same time, Compass has a desire to make sure the evaluation is done carefully, ethically, and rigorously. This is where J-PAL’s expertise comes in.

As a naturally skeptical person, I wasn’t exactly sure what to expect from the BAE Incubator. Moreover, while we were excited to dig deeper into rigorous evaluation, we were also a bit overwhelmed by the prospect of taking on a randomized evaluation given the many moving pieces. Because of these concerns, I have really appreciated that the Incubator has been an intentional, steady process. From the beginning, the focus on simply determining feasibility made me feel comfortable. Since then, each subsequent component has assured me that we are building the right blocks. 

The training we received through the BAE Incubator provided the opportunity to get in the weeds about some of the more technical concepts related to both randomized evaluations and evaluation in general. We even shared some training materials with other programs at Compass, which helped reinforce the data-driven culture of evidence we’re building. We also took advantage of the opportunity to attend J-PAL’s week-long Evaluating Social Programs course, which sparked several additional RCT ideas. (For another time, perhaps!)

The Incubator also includes regular, small group technical assistance sessions. When we meet with J-PAL staff members Amanda and Anisha, we have the chance to dig into questions specific to our design. They’ve helped us build confidence in both the feasibility of our project and our capacity to carry it out. They acknowledge that there can and will be challenges, but also that there are often ways to address them. The technical assistance has helped us think through potential unintended consequences of this project so that we can address them proactively. While there are times I feel as though we can’t do this, the next time we meet, I am reassured that we can. 

Through this engagement, we’ve really been able to sharpen our technical skills. We’re also learning how to ask the right questions to advance our evaluation goals, particularly when working directly with program staff. These tools help us not only think through the randomized evaluation we’re designing but also enhance our other work. We’ve been able to plant evaluation seeds across Compass’s many programs and spark conversations with staff about rigorous data.

For any organizations thinking about applying to the J-PAL North America Housing Stability Evaluation Incubator, I say go for it! They have a deliberate, transparent, and thoughtful process focused on determining feasibility. And if a randomized evaluation of your program is not feasible, that’s okay! They’ll still help you add tools to your toolbox. J-PAL staff provide space for organizations to test out ideas and ask questions. Don’t be afraid: trust the process.

Part one of this blog series explores existing evidence on cash transfers and highlights the need for further evaluation in the context of homelessness reduction and prevention in the United States. In parts three and four, Abode Services and Hamilton Families reflects on their experiences as an Incubator partner

The Homelessness and Housing Stability team at J-PAL North America will be accepting Letters of Interest for the Housing Stability Evaluation Incubator from August 1–October 17, 2022. Public housing authorities, continuums of care, nonprofit organizations, and other service providers working to reduce and prevent homelessness are encouraged to apply.

The Bay Area Evaluation (BAE) Incubator supports Bay Area service providers with implementing and evaluating cash transfer programs to assess their impact on homelessness and housing stability. In part three of our BAE Incubator partner series, Abode Services reflects on their experiences as an Incubator partner. Vivian Wan, chief operating officer, discusses how the Incubator is enabling Abode to expand its impact by elevating the voices of their program participants.
 
Abode Services was founded in 1987 as an emergency shelter and has grown into one of the California Bay Area’s largest providers of housing for people experiencing homelessness. At the heart of our mission is the firm belief that housing is the solution to end homelessness and that anyone can secure, maintain, and thrive in housing if given the right support. 
 
Abode Services views homelessness as a systemic and structural problem that must be addressed with evidence-based interventions that can be effectively scaled. As such, we are constantly growing and seeking ways to expand our impact. To do so requires both a commitment to the honest assessment of our programs and a willingness to adapt programs when required. Expanding our impact involves investing in evidence-based practices and trying promising new programs. Abode’s partnership with J-PAL North America through the BAE Incubator will enable us to implement a new cash transfer program and assess its impact through a rigorous evaluation.
 
Cash transfers are a promising solution that could be of tremendous benefit to the people we serve. They are also closely aligned with one of Abode’s core values: self-determination and choice. While services can be a critical component to help people experiencing homelessness find and retain housing, there are still some paternalistic concepts baked into service provision. For example, there’s a common notion that many people are poor or homeless due to their poor choices, and service providers should be making decisions on behalf of their participants. We reject this notion. We believe that people know what is best for themselves and their families and should have the power and resources to make their own decisions. We believe cash transfers, which can empower participants to make their own decisions, can lead to better outcomes. And, in a housing market as formidable as the Bay Area, we believe that cash transfers can greatly improve housing stability and other measures of wellness among the households who receive them. 
 
To test this theory, we are working with J-PAL North America and its affiliated researchers to design a robust impact evaluation. We firmly believe in measuring the effects of our program and understand that randomized evaluations are uniquely positioned to rigorously quantify a program’s impact. In a field where resources are limited and the demand for assistance far outstrips the supply, it is critical to know if and how our programs are working and to ensure that valuable resources are going toward the programs that are most impactful and cost-effective.
 
A key component of the BAE Incubator is the opportunity to connect with expert researchers in J-PAL’s network to design and potentially carry out a randomized evaluation. J-PAL North America Senior Research Manager Amanda Lee did a wonderful job matching us with researchers at the Wilson Sheehan Lab for Economic Opportunities (LEO) at the University of Notre Dame. When working with J-PAL to find the right partner, we shared that we really wanted a research team with a thorough understanding of housing and homeless services, a track record of collaboration with non-academic partners, and with whom there would be a true collaborative, candid, and mutually respectful partnership. LEO has exceeded even those high standards and has been an outstanding partner.
 
While we are still in the early stages of designing this program, our current plan is to provide a random selection of households in four to five counties who are exiting a rapid re-housing program with unconditional monthly payments for one year. Tying cash transfers to rapid re-housing provides a sample size that should adequately power the study and allow us to measure how outcomes vary across race and ethnicity, which is too often understudied with respect to program performance. We know that homelessness affects Black, Indigenous, and people of color disproportionately due to longstanding discrimination, structural racism, and unequal access to opportunities. But, as a field, we have less of an understanding of the racial disparities in program outcomes and if there are things that we, as providers and advocates, can do to better serve households of color and address those disparities. We hope that this evaluation will produce meaningful insights on how to best support people of color experiencing homelessness.
 
This program and evaluation are being designed with support from not only LEO and J-PAL North America, but also participants of our programs. Elevating the voices of the people we serve to inform program design advances racial justice and ensures that the program is responsive to their needs. In a focus group, current rapid re-housing participants shared many useful insights and reflections, many of which we have incorporated into the cash transfer program and study design. For example, we have decided to have larger payments to start, then taper over time, based on their stated difficulty of paying for “big ticket” items, like car repairs, in their first months in rapid re-housing. In response to their concerns about potential disappointment among study participants in the comparison group, we have adjusted our consent process and plan to offer a modest payment to control participants.
 
The partnerships developed through the BAE Incubator have provided Abode with an extensive array of tools to bolster our evaluation capabilities even beyond this specific project. LEO has partnered with us to design the best evaluation for our research question, and J-PAL North America has provided invaluable support along the way. We have truly appreciated this experience and all of the resources we have received to date.

Part one of this blog series explores existing evidence on cash transfers and highlights the need for further evaluation in the context of homelessness reduction and prevention in the United States. In parts two and four, Compass Family Services and Hamilton Families reflects on their experiences as an Incubator partner.

The Homelessness and Housing Stability team at J-PAL North America will be accepting Letters of Interest for the Housing Stability Evaluation Incubator from August 1–October 17, 2022. Public housing authorities, continuums of care, nonprofit organizations, and other service providers working to reduce and prevent homelessness are encouraged to apply.
 

The Bay Area Evaluation (BAE) Incubator supports Bay Area service providers with implementing and evaluating cash transfer programs to assess their impact on homelessness and housing stability. In part four of our BAE Incubator partner series, Hamilton Families reflects on their experiences as an Incubator partner. Chris Constantine, Director of Data & Evaluation, explains the difference a guaranteed monthly income could make for their clients.

One of the big questions on everyone’s mind at Hamilton Families is, “How do we effectively break the cycle of homelessness?” because it truly is a cycle. Since 1985, we’ve been working with families to end their homelessness and, in that time, we’ve learned a lot. We know that to end homelessness, we need to tackle the systems that reinforce generational poverty and erect a social safety net that families can rely on to catch them in case they stumble during their journey toward housing stability. 

Family homelessness tends to be the result of a sudden event that could have been prevented had a safety net been in place. For instance, a couple months ago, a mother in our Rapid Re-Housing program had her car towed, and her case manager reached out looking for options to pay the $500 fee to remove  the car from the impound lot. We did not have the infrastructure to pay the fee directly, and it took us three days to secure external funding to support her. In just that short time, the fees piled up and the cost to retrieve her car doubled—a 200 percent “poverty tax.”

Family homelessness disproportionately impacts people of color and single mothers: of the 800+ families in our programs, over 80 percent of families we serve are Black and Brown, and three out of four families are led by single mothers. The average income of families entering our Rapid Re-Housing program is only $1,500 a month, and families must make difficult tradeoffs because they’re struggling to afford life’s necessities like food, clothing, transportation, and rent. A small, sudden event like a parking ticket has the potential to completely derail a family’s stability. It can prevent parents from getting to work, causing them to lose their job which means they can’t pay for bills or rent. This in turn can lead to eviction. Suddenly they’re experiencing homelessness, all because of a parking ticket. Like toppling dominoes, “small” events like these cascade, causing families to experience homelessness every single day.

Situations like these are exactly why Hamilton Families is launching an unconditional cash payment study through J-PAL North America’s BAE Incubator. This is our most ambitious intervention to end family homelessness since the creation of our Rapid Re-Housing rental subsidy program in 2006. Our randomized evaluation will help us determine whether temporary, unrestricted cash assistance to families in our Rapid Re-Housing program yields positive, long-term housing stability outcomes. We also hope to measure participants’ feelings of empowerment and agency. I often think about how that mother’s situation could have been different had she been empowered to pay that fee independent of staff intervention.

Through technical assistance, J-PAL North America staff have done a great job of keeping our project on track and thinking through all of the necessary components needed for approval from the Institutional Review Board. They have also provided us with landscape analyses of cash transfer studies in the United States and around the world. With J-PAL North America’s support, we’re taking the best practices of past and current cash transfer pilot programs while using a peer-reviewed, evidence-based, step-by-step approach to help uplift the families we are permanently housing. 

I started off at Hamilton Families doing direct service work as a Case Manager for three years before moving into research. I’ve witnessed firsthand how necessary cash assistance is for families who are trying to recover their lives. The organization is now at the point where we can build capacity to implement a program like this. We can build the internal knowledge, systems, and structures to responsibly test the effects of direct cash transfers as a homelessness intervention that can benefit families experiencing homelessness in the long-term. It has truly been a full circle moment for me. Now, because of our collaboration with J-PAL North America, we have access to a credible research partner who will be able to help us reach a broader audience with our findings and our impact. 

I think about our impact a lot, and the stories we hope to hear from impacted families after the end of this study.

While we have tremendous hope about the impact of this program, we expect to encounter our fair share of challenges: combating stigma around giving money to low-income folks, the considerations when providing cash assistance to some and not to others, and the operational requirements of running a multiple-year study. But breaking the cycle of homelessness has never been easy, and J-PAL North America is helping us address these challenges. Wherever we look to lift up our community, having a safe place to call home is a crucial part of the solution. We all want safe and healthy neighborhoods for ourselves and our children. Researching and testing a diverse mix of interventions across the Bay Area—including cash transfers—helps us create safer communities and work together to strengthen safety nets and break the cycle of homelessness.

Part one of this blog series explores existing evidence on cash transfers and highlights the need for further evaluation in the context of homelessness reduction and prevention in the United States. In parts two and three, Compass Family Services and Abode Services reflect on their experiences as Incubator partners.

The Homelessness and Housing Stability team at J-PAL North America will be accepting Letters of Interest for the Housing Stability Evaluation Incubator from August 1–October 17, 2022. Public housing authorities, continuums of care, nonprofit organizations, and other service providers working to reduce and prevent homelessness are encouraged to apply.