Since launching the State and Local Innovation Initiative in 2015, J-PAL North America has supported the development and implementation of randomized evaluations in partnership with thirteen state and local governments: Baltimore, MD; California; King County, WA; Massachusetts; Minneapolis, MN; New Mexico; Pennsylvania; Philadelphia, PA; Puerto Rico; Rochester, NY; Santa Clara, CA; South Bend, IN; and South Carolina.
These partners are part of a growing movement among state and local governments to use rigorous evidence to improve the effectiveness of their policies and programs and outcomes for people experiencing poverty. Read more about our partnerships below.
Projects in Development
California Franchise Tax Board
The California Franchise Tax Board is interested in using randomized evaluations to identify effective strategies for lifting low-income Californians out of poverty by encouraging eligible households to claim the state’s Earned Income Tax Credit (CalEITC). Modeled after the federal EITC, one of the nation’s largest anti-poverty programs, the CalEITC provides cash to low-income, working Californians. However, not all families who are eligible claim of the credit. The Franchise Tax Board is interested in improving outreach to eligible filers, for example by sending informational letters or personalized communications. The Franchise Tax Board is also interested in testing behavioral nudges that encourage honesty and accuracy among tax filers, which might help close the annual “tax gap”—the difference between what taxpayers owe and what they actually pay—which is estimated to be over $10 billion in California alone.
Massachusetts Department of Transitional Assistance
The Commonwealth of Massachusetts Department of Transitional Assistance (DTA) is developing a randomized evaluation to test the effectiveness of a text messaging intervention at reducing churn in the Supplemental Nutritional Assistance Program (SNAP). SNAP churn is when a case closes and then resumes receipt of benefits within a period of four months. Some eligible SNAP participants fail to complete administrative processes required to maintain benefits resulting in benefits closing and administrative costs for renewal. The texting intervention will send reminders to simplified reporting SNAP clients who have an upcoming reevaluation process requirement. Drawing from behavioral science principles, the texts will be sent at each step of the SNAP reevaluation process and will provide information on how to complete the process. The evaluation will measure the effect of this intervention on the probability of on-time reevaluation processes and other administrative outcomes.
New Mexico Legislative Finance Committee
The New Mexico Legislative Finance Committee is partnering with J-PAL North America to evaluate the effectiveness of the state’s early college high schools. New Mexico high school graduates enroll in higher education above the national average but are more likely to require remedial coursework, and the state ranks among the lowest in the nation for bachelor’s degree attainment. Early college high schools, which have the mission of improving college preparedness and providing students the opportunity to obtain college credits and degrees while they are enrolled in high school, represent one potential solution to this challenge. When there are more applicants to an early college high school than there are available seats, early college high schools may use lotteries to determine admissions. By comparing outcomes for students who gained admissions through a lottery to those who did not, the Legislative Finance Committee can rigorously measure the impact of early college high schools on a variety of long-term outcome, including postsecondary degree attainment and employment outcomes.
Randomized Evaluations Funded Through the Initiative
Increasing the Take-Up of Cal Grant Awards Through Improved Notification Letters
Researcher: Jesse Rothstein
Each year, over 150,000 California high school students receive letters notifying them that they qualify for Cal Grants, grant aid for college that is assigned based on family income and high school GPA. Less than two-thirds claim their awards. The California Student Aid Commission (CSAC) believes this is in part because some students do not understand their eligibility. This study will investigate whether a simplified letter, in terms of both format and the use of language that behavioral science research suggests would nudge students to take action, will induce higher take-up of the Grants and will consider how different letters impact students’ college choices decision making.
In 2017-18, students were randomly assigned (at the high school level) to receive CSAC’s usual notification letter or one of two variants designed with behavioral science principles in mind. CSAC is also working to modify its IT systems to randomly vary the number of follow-up e-mails students receive. Primary outcomes of interest include whether students create accounts in CSAC’s online portal, enroll in college, and claim their awards, as well as which type of college (community college vs. 4-year) they choose. Analysis of these outcomes will occur beginning in Spring 2018. We are working with CSAC to design higher-touch interventions to be implemented in 2018-19.
Youth Homelessness Prevention: A Randomized Controlled Trial
Researchers: David Phillips, James Sullivan
This project will be the largest randomized control trial of homelessness prevention services to date and will compare two different approaches to homelessness prevention. We will use a lottery to measure the effectiveness of (1) a combined program of progressive case management and flexible financial assistance relative to (2) only flexible financial assistance. This project will be conducted in the context of the Youth and Family Homelessness Prevention Initiative (YFHPI) in King County, Washington, where lawmakers are interested in assessing the effectiveness of the program's case management component. The results of this study will also be informative to policymakers and service providers in other communities that are interested in the most effective means of homelessness prevention. Enrollment began in May of 2018 and will continue until we have enrolled 600 participants. Our primary outcomes will be measures of housing stability including indicators of homelessness and changes in address; secondary outcomes may include family welfare, criminal justice and employment outcomes.
Pilot: Creating Moves to Opportunity in the Twin Cities
Researchers: Nathan Hendren, Christopher Palmer
The goal of Creating Moves to Opportunity in the Twin Cities is to compare the effectiveness of using tenant-based Section 8 Housing Choice Vouchers with mobility supports to using project-based vouchers to provide opportunities for low-income families with children to move to and persist in opportunity neighborhoods. In contrast to tenant-based vouchers, families awarded project-based vouchers in opportunity areas need not search for housing because the voucher is already assigned to a unit. Families on the waiting lists of partner agencies the Minneapolis Public Housing Authority and the Metropolitan Council Housing and Redevelopment Agency will be invited to participate in this study. Those who choose to participate will be randomly assigned to the tenant-based treatment group, the project-based treatment group, or the control group. Mobility supports provided to both treatment groups will include education, pre- and post-move counseling, marketability coaching, and moving assistance funds. The primary outcomes will be to determine (1) if project-based vouchers are more cost-effective than tenant-based vouchers in increasing the number of families who move to high opportunity neighborhoods, and (2) whether the type of voucher (project-based vs. tenant-based) impacts persistence in opportunity areas.
Sara Heller (University of Pennsylvania), the City of Philadelphia Mayor’s Office, and the Philadelphia Youth Network are conducting a randomized evaluation to evaluate the impact of WorkReady, a summer jobs program for disadvantaged youth. Recent evidence from random-assignment studies shows that summer jobs programs in New York City and Chicago dramatically reduce violence involvement among participants, but have small, if any, effects on education and employment. The Philadelphia study is intended to 1) assess how generalizable the prior findings are by testing the crime, employment, and school effects of a different summer jobs program in a new setting, and 2) expand tests for program effects to socially-costly correlates of violence that may also be affected: mental health, substance abuse, teen pregnancy, housing instability, and child maltreatment. To study these questions, researchers will allocate about 1,000 of the 8,000 summer WorkReady slots by lottery. Researchers will track youth in a range of administrative data, as well as collect some supplementary qualitative evidence about youth experiences. This project received full evaluation funding in Spring 2017.
Improving and Sustaining Management Practices in Public Schools: Evidence from Puerto Rico
Researchers: Gustavo J. Bobonis, Marco González-Navarro, Daniela Scur
Improvements in management practices have been credited with generating stunning productivity improvement in the private sector. Although their adoption in the education sector has been slow, management-level interventions have been shown to lead to improvements in school quality and students’ academic achievement. However, adoption of these practices in the public sector is often not sustained in the long run. Promoting improvements in school management practices is thus an amenable area for policy intervention, given the need to identify ways to achieve this sustainably and at scale.
Our proposed study is a collaboration with the Puerto Rico Department of Education (PRDE) to experimentally evaluate the impact of a large-scale principal training program among its school directors. We will measure impacts on managerial practices collecting detailed longitudinal data to measure short and long-term adoption, as well as impacts on student achievement using standardized test scores data. The study will help inform whether improvements in managerial practices can be implemented in schools at scale, their long-term sustainability, and their effects on student learning.
More than 43 million people in the United States live in poverty. Poverty is particularly pervasive in the city of Rochester, NY where rates are more than twice the national average. The Rochester-Monroe Anti-Poverty Initiative (RMAPI), in partnership with the New York Governor's State Anti-Poverty Task Force, is piloting a targeted program designed to reduce poverty in the Rochester-Monroe region. Extensive community research led RMAPI to pilot an adult mentor/navigator program that addresses the issues identified as pervasive in the city of Rochester: a knowledge gap about available services and how to navigate them, and a need for coordinated services. In order to measure the pilot's success and make decisions about scaling the program up, RMAPI has asked the Wilson Sheehan Lab for Economic Opportunities (LEO) from the University of Notre Dame to run a randomized evaluation of the pilot program, Bridges to Success (BTS). This evaluation will test the hypothesis that providing a professional mentor/navigator program in a targeted area of concentrated poverty will increase economic mobility for program participants, resulting in improved self-sufficiency. Results from this study will inform policy decisions to scale up and expand the program in the city of Rochester, and will be disseminated to policy makers and providers in other regions to inform the design or replication of future anti-poverty initiatives. This project received full evaluation funding in Spring 2017.
Santa Clara County’s Office for Supportive Housing (OSH) is introducing a Rapid Re-Housing program (RRH) for single adults in cooperation with HomeFirst, a local non-profit provider. To measure the program’s success and make decisions about scaling the program up, OSH and HomeFirst have been working with the Wilson Sheehan Lab for Economic Opportunities (LEO) and J-PAL to launch a randomized controlled trial (RCT) evaluation of the program. This evaluation will test the hypothesis that providing RRH to homeless single adults will improve housing stability leading to decreases in homeless shelter entry, housing moves, and hospital visits for program participants. This study will also stratify the random assignment of study participants by the score on the county’s risk tool to explore for whom RRH works best. We will randomly select 360 participants for the program and approximately 360 participants for a control group referred to usual care. Results from this RCT will inform policy decisions to scale up and expand the program in the County of Santa Clara and will be disseminated to policy makers and providers in other regions to inform the design or replication of future housing initiatives.
Quantifying the Impact of Jobs for America's Graduates
Researchers: William Evans, Sarah Kroeger, Patrick Turner
What is the effect of completing the Jobs for America’s Graduates (JAG) Multi-Year Dropout Prevention Program on low-income 11th to 12th graders’ academic and economic outcomes? The City of South Bend, in partnership with researchers from J-PAL and LEO, will conducted a randomized controlled trial to analyze the effect of the Jobs for America’s Graduates (JAG) Multi-Year Dropout Prevention Program on education and labor market outcomes.This study is intended to inform the City of how to best optimize JAG services for South Bend youth, and measure the return on investment of the program for the purposes of understanding its relative sustainability and efficiency. We will recruit a total sample of 500 students and randomize 250 students into the treatment group and 250 to control. The primary outcomes of interest are on-time high school completion and GPA. Additional key outcomes include a broad range of indicators of labor market success, including: college application, enrollment, persistence, and completion, and employment and earnings. The City hopes that this evaluation will provide positive and significant results on the specific impact of JAG on South Bend students and make a case to expand this program to more students in South Bend schools.
In South Carolina, the Medicaid program is administered through Managed Care Organizations (MCOs), which offer different health care plans to Medicaid beneficiaries. These plans differ in their generosity, network coverage, and other attributes, and they are ranked by the state using a system of “star ratings.” The system of MCOs offers choices to health care consumers and allows plans to compete for consumers. In South Carolina, when consumers do not make an active plan choice, the state uses an algorithm to assign plans to consumers automatically. Starting earlier this year (in 2017), this auto-assignment is now being made using an explicitly random process. We propose to use this randomized assignment feature to study the effect of plan assignment on patient outcomes such as health care utilization and health care expenditures (both overall and by category). This prospective analysis will be complemented with a retrospective analysis that takes advantage of the state’s historical quasi-random round-robin assignment procedure to allocate households to plans. Additionally, we propose to combine the analysis of the randomly assigned population with the population that made active choices to try to distinguish between treatment and selection in accounting for which plans perform better.