Behind the scenes: Chelsea Eats

This post summarizes a panel discussion from J-PAL Global and J-PAL North America’s 2024 Evaluating Social Programs (ESP) course. ESP is a weeklong course that trains policymakers and practitioners on conducting randomized program evaluations.

In response to the Covid-19 pandemic and its profound economic consequences, public and private leaders in Chelsea, Massachusetts created Chelsea Eats. This unconditional cash transfer program provided over 2,000 Chelsea residents with US$200–US$400 per month (depending on household size) for nine months. Program administrators used a lottery to select recipients, enabling researchers to run a randomized evaluation assessing the impact of the cash transfers. In this piece, J-PAL affiliated researcher Jeffrey Liebman (Harvard) and former City Manager of Chelsea, Thomas Ambrosino, discuss the motivation, design, and evaluation of Chelsea Eats.

Can you begin by setting the scene for the program and evaluation we’re discussing today?

Tom: Chelsea is a small, working class city north of Boston. Many of its residents work in the service industry and were considered essential workers, unable to stay home during the shut-downs of the early pandemic. So Covid-19 exposure was a big concern and rates were very high. Unemployment hit our community hard as well. Moreover, many residents were not able to access the expanded unemployment benefits or stimulus checks at the time due to immigration status.

Chelsea also has a very strong sense of community and collective responsibility. When fears of widespread food insecurity set in, community leaders quickly developed a grassroots food distribution program at an unprecedented scale. Unfortunately the need was even greater than initially assessed and it became unsustainable and inefficient to distribute food this way. We therefore decided to pivot to providing cash assistance for recipients to purchase their own groceries, or whatever else they deemed necessary.

How did you decide you wanted to not only run the program, but also evaluate it? Why was the evaluation component important to you, your constituents, and other stakeholders?

Tom: To start, there were requirements from some of the program’s funders to be able to demonstrate the impact of their investment. Also, a number of stakeholders were a bit skeptical about how the money would be spent, so we needed to track spending behaviors. But more than that, it was important for us to know whether what we were doing was making a difference.

Can you share more about your partnership and the timeline for this project? 

Tom: This project got off the ground very quickly. By the time we brought on Jeff, we were planning to launch in about three weeks. So logistically speaking, we were seeking a research partner who could actually support this project in such a short period of time. But more importantly than that, I wanted to make sure we were partnering with a compassionate and empathetic team who could treat people who experienced trauma from the Covid-19 pandemic with dignity. Spending time with Jeff, it was clear to me that he and his team understood they were dealing with a delicate situation, and they did so with care.

Jeff: Usually we do prefer a bit longer of a runway when setting up research partnerships, but when we heard about this opportunity we knew we needed to do it. The Chelsea team was already committed to holding a lottery for allocating the cash assistance, so that provided an ideal opportunity for a randomized evaluation. Another thing that made Chelsea such a great collaboration was that there was a strong relationship and a lot of trust between the community and city officials. Finally, as Tom mentioned, there was a firm commitment to understanding the impact of the program. They hoped it worked but were open to learning from evaluation results either way. That’s key in a research partnership.

Can you speak a bit more about the design of the evaluation? How did the randomized design and surveying work?

Tom: Individuals and families were eligible to enter our lottery if they lived in Chelsea and had a household income below 30 percent of the area median income. People who were in greater need or in a higher risk situation (e.g., caretakers, people with disabilities, and those ineligible for other food assistance programs) were given additional lottery tickets. We also thought a lottery was a fair way to allocate the funds, since we unfortunately knew we couldn’t provide the money to everyone. Over 3,600 people applied for a Chelsea Eats cash assistance card and 2,213 won the lottery and received the cards.

Jeff: As mentioned, the lottery made doing the randomized evaluation pretty straightforward. That said, we did have a lot of work to do around deciding what outcomes to measure and how. The inherent flexibility of cash transfers means that they can impact a wide range of outcomes. Our main focus was the direct impacts of the cash on food satisfaction, food security, diet quality, and financial distress. We also looked at some downstream effects on health and school attendance. And finally, because these are the questions policymakers tend to ask about cash transfers, we looked at employment and how people spent the funds.

We used administrative data to assess spending patterns, school attendance, and health care utilization. The other outcomes were primarily measured using surveys. We did six surveys over the course of the study, offered in English and Spanish, and respondents could choose between answering online or over the phone. We found that surveying folks regularly was helpful in building trust and achieving high response rates.

And what were the evaluation results?

Tom: While the program was running, it was really helpful for me to receive data from Jeff’s team. We were able to see that people were spending money on food as we intended, and we could see money was going into Chelsea businesses, which helped the program gain more support from some of the initially skeptical stakeholders. This helped us extend the program from six months to nine months.

Jeff: The cash transfer succeeded in accomplishing the things the city designed the program for: people spent more money on food, they were happier with the food they were consuming, and they increased their intake of total calories, fruit, vegetables, and unprocessed meat. Cash transfer recipients also had lower levels of financial distress. There was no impact on school attendance.

In our surveys, we did not find an impact on physical or mental health. However, when we linked to administrative medical records, we did find impacts on various health outcomes. Those who received the cash transfer had fewer emergency department visits, including those related to behavioral health or substance use, fewer admissions to the hospital from the emergency department, and more outpatient visits to subspecialists than those who did not receive the cash transfer. 

How did you share these results back with the community, and how have they been used?

Jeff: We shared the results with the community through a few channels. For example, we presented at a community poster session at Chelsea High School where Chelsea students and researchers like us shared studies happening in the Chelsea community. We regularly briefed city officials about what we were finding. The Boston Globe also reported on our results. I also brought insights from how Chelsea implemented their cash transfer program to other cities that were asking my Government Performance Lab for advice on setting up their own basic income programs.

Tom: It was really helpful to have these results to show the community and City Council that the program was successful and this was money well-spent. This helped us run a second wave of the program in 2023. 

For more on Chelsea Eats, watch the J-PAL video for an overview of the study results and visit this page for more details.