Since launching the State and Local Innovation Initiative in 2015, J-PAL North America has supported the development and implementation of randomized evaluations in partnership with eighteen state and local governments: Baltimore, MD; California; Franklin County, OH; King County, WA; Massachusetts; Minneapolis, MN; Minnesota; New Mexico; Pennsylvania; Philadelphia, PA; Puerto Rico; Rochester, NY; Santa Clara, CA; Shasta County, CA; South Bend, IN; South Carolina; Virginia; and Washington, DC.
These partners are part of a growing movement among state and local governments to use rigorous evidence to improve the effectiveness of their policies and programs and outcomes for people experiencing poverty. Read more about our partnerships below.
Projects in Development
California Department of State Hospitals
The California Department of State Hospitals (DSH) is partnering with J-PAL North America to develop an impact evaluation of the Pre-trial Felony Mental Health Diversion Program, an intervention targeting individuals with serious mental illness who are likely to be found incompetent to stand trial. Participants would be diverted out of the criminal justice system and into wrap-around community treatment services. Historically, when a defendant’s competency is questioned by their defense attorney, prosecuting attorney, or judge, the individual’s court proceedings are suspended. If a defendant has committed a felony and is found Incompetent to Stand Trial (IST), they can receive treatment at a DSH facility. In the new felony pre-trial diversion program, approximately twenty counties will be receiving funding to offer diversion to some of these IST’s. The impact evaluation will measure the effect of the diversion programs on criminal justice and mental health outcomes.
California Franchise Tax Board
The California Franchise Tax Board is interested in using randomized evaluations to identify effective strategies for lifting low-income Californians out of poverty by encouraging eligible households to claim the state’s Earned Income Tax Credit (CalEITC). Modeled after the federal EITC, one of the nation’s largest anti-poverty programs, the CalEITC provides cash to low-income, working Californians. However, not all families who are eligible claim of the credit. The Franchise Tax Board is interested in improving outreach to eligible filers, for example by sending informational letters or personalized communications. The Franchise Tax Board is also interested in testing behavioral nudges that encourage honesty and accuracy among tax filers, which might help close the annual “tax gap”—the difference between what taxpayers owe and what they actually pay—which is estimated to be over $10 billion in California alone.
Massachusetts Department of Transitional Assistance
The Commonwealth of Massachusetts Department of Transitional Assistance (DTA) is developing a randomized evaluation to test the effectiveness of a text messaging intervention at reducing churn in the Supplemental Nutritional Assistance Program (SNAP). SNAP churn is when a case closes and then resumes receipt of benefits within a period of four months. Some eligible SNAP participants fail to complete administrative processes required to maintain benefits resulting in benefits closing and administrative costs for renewal. The texting intervention will send reminders to simplified reporting SNAP clients who have an upcoming reevaluation process requirement. Drawing from behavioral science principles, the texts will be sent at each step of the SNAP reevaluation process and will provide information on how to complete the process. The evaluation will measure the effect of this intervention on the probability of on-time reevaluation processes and other administrative outcomes.
Minnesota Board of Pharmacy
The Minnesota Board of Pharmacy is partnering with J-PAL North America to identify ways to increase the use of Minnesota’s prescription monitoring program (PMP) and measure the impact of that increased use on prescribers’ rate of controlled substance prescriptions and other tertiary outcomes. The PMP database maintains a secure record of all controlled substance prescriptions, and more frequent use of the database may help avoid prescribing to individuals misusing opioids and, instead, allow prescribers to make referrals to treatment services. One idea for a randomized evaluation would test the impact of sending behaviorally-informed mailings or electronic notifications to nudge prescribers of controlled substances to use the database. A second idea for evaluation would test the impact of integrating access to the PMP database into the clinical workflow of a major healthcare system via their electronic medical record (EMR) system. The proposed evaluations would measure the impact of these interventions on the frequency with which practitioners prescribe opioids, their practice specialty, the dosage prescribed, change in prescribing (lower dose, change in drug type), and other measures related to access or use of the PMP.
Minnesota State Court Administrator's Office
The Minnesota State Court Administrator’s Office is working with Minnesota Management and Budget and J-PAL North America to test the impact of different text message reminders on failures to appear (FTAs) in court among individuals with court dates for criminal hearings or eviction hearings. FTAs are costly for both individuals and the court system; When defendants fail to appear, law enforcement must locate, arrest, transport, and jail the individual, all at taxpayer expense. Previous research has found that text message reminders can reduce failures to appear for arraignment for criminal defendants in New York City, with the size of the reduction depending on the content of the messages. Minnesota’s largest county piloted a text reminder program in 2017 and, after promising results, the Minnesota Judicial Branch decided to make these e-reminders available statewide. Court users statewide will be able to opt-in to receive text and/or email reminders of their upcoming court appearances. This evaluation will measure the impact of the different text message content and timing on appearances for scheduled hearings, warrants issued, and eviction outcomes.
New Mexico Legislative Finance Committee
The New Mexico Legislative Finance Committee is partnering with J-PAL North America to evaluate the effectiveness of the state’s early college high schools. New Mexico high school graduates enroll in higher education above the national average but are more likely to require remedial coursework, and the state ranks among the lowest in the nation for bachelor’s degree attainment. Early college high schools, which have the mission of improving college preparedness and providing students the opportunity to obtain college credits and degrees while they are enrolled in high school, represent one potential solution to this challenge. When there are more applicants to an early college high school than there are available seats, early college high schools may use lotteries to determine admissions. By comparing outcomes for students who gained admissions through a lottery to those who did not, the Legislative Finance Committee can rigorously measure the impact of early college high schools on a variety of long-term outcome, including postsecondary degree attainment and employment outcomes.
Shasta County, California Superior Court
The Shasta County Superior Court is working with J-PAL North America to test strategies for reducing the likelihood that defendants awaiting court processing in the community fail to appear (FTA) for their arraignment. FTAs can be costly for both individuals and the court system; when an individual FTAs, a warrant is issued. Law enforcement is made aware of the FTA warrants, but defendants currently are not notified that warrants have been issued by the Court. Thus, outstanding warrants may also result in defendants’ further entanglement with the criminal justice system without their awareness. Previous research suggests that behavioral nudges, like text messages and redesigned summons, can reduce FTAs. This partnership will explore several different ideas for reducing FTAs in Shasta County to test whether these interventions are effective at reducing FTAs.
Virginia Department of Social Services
The Virginia Department of Social Services (VDSS) is developing a randomized evaluation to test the impact of texting Virginia residents who are eligible for the Earned Income Tax Credit (EITC) on take-up rates of the EITC. The EITC is a federal tax credit designed to support low to moderate-income working families. It encourages and promotes work and off-sets federal payroll tax and income taxes. VDSS currently sends reminder notifications via mail and email to clients who are potentially eligible for the EITC, but not everyone who is eligible for the EITC takes up the benefit; In 2018, the EITC take-up rate among eligible Virginia residents was about 80 percent. VDSS would like to conduct an RCT to test the effectiveness of texting and new messages to generate greater rates of tax filing and EITC claims for eligible people.
The city of Washington, D.C. is partnering with J-PAL North America to to design, pilot and evaluate less-expensive transportation options for low-income residents. In D.C. and elsewhere, low-income households are the most likely to be burdened by the costs of transportation, the most likely to forego using transit due to cost, and the least likely to have multiple travel options or employer subsidies. Once any potential options have been identified, the city is interested in rigorously measuring the impact of any lower-cost options on mobility as well as measures of social and economic well-being.
Randomized Evaluations Funded Through the Initiative
Increasing the Take-Up of Cal Grant Awards Through Improved Notification Letters
Researcher: Jesse Rothstein
Each year, over 150,000 California high school students receive letters notifying them that they qualify for Cal Grants, grant aid for college that is assigned based on family income and high school GPA. Less than two-thirds claim their awards. The California Student Aid Commission (CSAC) believes this is in part because some students do not understand their eligibility. This study will investigate whether a simplified letter, in terms of both format and the use of language that behavioral science research suggests would nudge students to take action, will induce higher take-up of the Grants and will consider how different letters impact students’ college choices decision making.
In 2017-18, students were randomly assigned (at the high school level) to receive CSAC’s usual notification letter or one of two variants designed with behavioral science principles in mind. CSAC is also working to modify its IT systems to randomly vary the number of follow-up e-mails students receive. Primary outcomes of interest include whether students create accounts in CSAC’s online portal, enroll in college, and claim their awards, as well as which type of college (community college vs. four-year) they choose. Analysis of these outcomes will occur beginning in Spring 2018. We are working with CSAC to design higher-touch interventions to be implemented in 2018-19.
Pilot: Using Remote Sensing to Reduce Vehicle Emissions in California
Researcher: Michael Greenstone
Particulate matter (PM) air pollution presents a substantial threat to human health, disproportionately impacting low-income populations. The transportation sector – heavy-duty trucks in particular - is a major PM contributor. However, enforcing vehicle emissions regulations has proven prohibitively costly worldwide. We propose utilizing novel remote sensing technology to detect high emitters at substantially reduced cost. We will leverage these new data in a randomized trial to determine the impact of remote monitoring on regulatory compliance. In the proposed pilot, we will test the technology, build institutional capacity, generate data, and field-test the proposed randomized intervention. In the full-scale experiment, we plan to identify high-emitting trucks in California, then randomly assign half to receive letters that (1) inform fleet owners their vehicle is likely in violation of California’s emissions standards, and (2) specify a compliance deadline with an explicit penalty for failing to comply. We will measure impacts using remote-sensed emissions readings and administrative compliance data.
Franklin County, Ohio
Evaluation of Permanent Supportive Housing for Justice-Involved Frequent Utilizers
Researchers: David Phillips, Jim Sullivan
This project will examine the effect of receiving permanent supportive housing services on housing, health, labor market, and criminal justice outcomes for individuals who repeatedly interact with the criminal justice system. The FUSE (Frequent Users Systems Engagement) Initiative, a new program in Franklin County, OH (Columbus), offers permanent supportive housing (PSH) to individuals with a history of many jail stays. “Frequent utilizers” of the criminal justice system are often affected by one or more adverse conditions that make it difficult to attain stability and increase the likelihood of recidivism. The County, in partnership with implementing partners and the Wilson Sheehan Lab for Economic Opportunities (LEO) at the University of Notre Dame, is conducting a randomized controlled trial to quantify the impact of PSH. The initial, pilot phase of the study will include 84 eligible individuals in the County. The study will track several key outcomes, including housing stability and homelessness, emergency care use, the cost of healthcare services, public benefit usage, and recidivism. This pilot would provide the basis for a larger scale evaluation as the program expands.
Youth Homelessness Prevention: A Randomized Controlled Trial
Researchers: David Phillips, James Sullivan
This project will be the largest randomized control trial of homelessness prevention services to date and will compare two different approaches to homelessness prevention. We will use a lottery to measure the effectiveness of (1) a combined program of progressive case management and flexible financial assistance relative to (2) only flexible financial assistance. This project will be conducted in the context of the Youth and Family Homelessness Prevention Initiative (YFHPI) in King County, Washington, where lawmakers are interested in assessing the effectiveness of the program's case management component. The results of this study will also be informative to policymakers and service providers in other communities that are interested in the most effective means of homelessness prevention. Enrollment began in May of 2018 and will continue until we have enrolled 600 participants. Our primary outcomes will be measures of housing stability including indicators of homelessness and changes in address; secondary outcomes may include family welfare, criminal justice, and employment outcomes.
Pilot: Creating Moves to Opportunity in the Twin Cities
Researchers: Nathan Hendren, Christopher Palmer
The goal of Creating Moves to Opportunity in the Twin Cities is to compare the effectiveness of using tenant-based Section 8 Housing Choice Vouchers with mobility supports to using project-based vouchers to provide opportunities for low-income families with children to move to and persist in opportunity neighborhoods. In contrast to tenant-based vouchers, families awarded project-based vouchers in opportunity areas need not search for housing because the voucher is already assigned to a unit. Families on the waiting lists of partner agencies the Minneapolis Public Housing Authority and the Metropolitan Council Housing and Redevelopment Agency will be invited to participate in this study. Those who choose to participate will be randomly assigned to the tenant-based treatment group, the project-based treatment group, or the control group. Mobility supports provided to both treatment groups will include education, pre- and post-move counseling, marketability coaching, and moving assistance funds. The primary outcomes will be to determine (1) if project-based vouchers are more cost-effective than tenant-based vouchers in increasing the number of families who move to high opportunity neighborhoods, and (2) whether the type of voucher (project-based vs. tenant-based) impacts persistence in opportunity areas.
Sara Heller (University of Pennsylvania), the City of Philadelphia Mayor’s Office, and the Philadelphia Youth Network are conducting a randomized evaluation to evaluate the impact of WorkReady, a summer jobs program for disadvantaged youth. Recent evidence from random-assignment studies shows that summer jobs programs in New York City and Chicago dramatically reduce violence involvement among participants, but have small, if any, effects on education and employment. The Philadelphia study is intended to 1) assess how generalizable the prior findings are by testing the crime, employment, and school effects of a different summer jobs program in a new setting, and 2) expand tests for program effects to socially-costly correlates of violence that may also be affected: mental health, substance abuse, teen pregnancy, housing instability, and child maltreatment. To study these questions, researchers will allocate about 1,000 of the 8,000 summer WorkReady slots by lottery. Researchers will track youth in a range of administrative data, as well as collect some supplementary qualitative evidence about youth experiences. This project received full evaluation funding in Spring 2017.
Improving and Sustaining Management Practices in Public Schools: Evidence from Puerto Rico
Researchers: Gustavo J. Bobonis, Marco González-Navarro, Daniela Scur
Improvements in management practices have been credited with generating stunning productivity improvement in the private sector. Although their adoption in the education sector has been slow, management-level interventions have been shown to lead to improvements in school quality and students’ academic achievement. However, adoption of these practices in the public sector is often not sustained in the long run. Promoting improvements in school management practices is thus an amenable area for policy intervention, given the need to identify ways to achieve this sustainably and at scale.
Our proposed study is a collaboration with the Puerto Rico Department of Education (PRDE) to experimentally evaluate the impact of a large-scale principal training program among its school directors. We will measure impacts on managerial practices collecting detailed longitudinal data to measure short and long-term adoption, as well as impacts on student achievement using standardized test scores data. The study will help inform whether improvements in managerial practices can be implemented in schools at scale, their long-term sustainability, and their effects on student learning.
More than 43 million people in the United States live in poverty. Poverty is particularly pervasive in the city of Rochester, NY where rates are more than twice the national average. The Rochester-Monroe Anti-Poverty Initiative (RMAPI), in partnership with the New York Governor's State Anti-Poverty Task Force, is piloting a targeted program designed to reduce poverty in the Rochester-Monroe region. Extensive community research led RMAPI to pilot an adult mentor/navigator program that addresses the issues identified as pervasive in the city of Rochester: a knowledge gap about available services and how to navigate them, and a need for coordinated services. In order to measure the pilot's success and make decisions about scaling the program up, RMAPI has asked the Wilson Sheehan Lab for Economic Opportunities (LEO) from the University of Notre Dame to run a randomized evaluation of the pilot program, Bridges to Success (BTS). This evaluation will test the hypothesis that providing a professional mentor/navigator program in a targeted area of concentrated poverty will increase economic mobility for program participants, resulting in improved self-sufficiency. Results from this study will inform policy decisions to scale up and expand the program in the city of Rochester, and will be disseminated to policy makers and providers in other regions to inform the design or replication of future anti-poverty initiatives. This project received full evaluation funding in Spring 2017.
The Impact of Rapid Rehousing on Homelessness: A Randomized Controlled Trial
Researchers: David Phillips, James Sullivan
Santa Clara County’s Office for Supportive Housing (OSH) is introducing a Rapid Re-Housing program (RRH) for single adults in cooperation with HomeFirst, a local non-profit provider. To measure the program’s success and make decisions about scaling the program up, OSH and HomeFirst have been working with the Wilson Sheehan Lab for Economic Opportunities (LEO) and J-PAL to launch a randomized controlled trial (RCT) evaluation of the program. This evaluation will test the hypothesis that providing RRH to homeless single adults will improve housing stability leading to decreases in homeless shelter entry, housing moves, and hospital visits for program participants. This study will also stratify the random assignment of study participants by the score on the county’s risk tool to explore for whom RRH works best. We will randomly select 360 participants for the program and approximately 360 participants for a control group referred to usual care. Results from this RCT will inform policy decisions to scale up and expand the program in the County of Santa Clara and will be disseminated to policy makers and providers in other regions to inform the design or replication of future housing initiatives.
Quantifying the Impact of Jobs for America's Graduates
Researchers: William Evans, Sarah Kroeger, Patrick Turner
What is the effect of completing the Jobs for America’s Graduates (JAG) Multi-Year Dropout Prevention Program on low-income 11th to 12th graders’ academic and economic outcomes? The City of South Bend, in partnership with researchers from J-PAL and LEO, will conducted a randomized controlled trial to analyze the effect of the Jobs for America’s Graduates (JAG) Multi-Year Dropout Prevention Program on education and labor market outcomes.This study is intended to inform the City of how to best optimize JAG services for South Bend youth, and measure the return on investment of the program for the purposes of understanding its relative sustainability and efficiency. We will recruit a total sample of 500 students and randomize 250 students into the treatment group and 250 to control. The primary outcomes of interest are on-time high school completion and GPA. Additional key outcomes include a broad range of indicators of labor market success, including: college application, enrollment, persistence, and completion, and employment and earnings. The City hopes that this evaluation will provide positive and significant results on the specific impact of JAG on South Bend students and make a case to expand this program to more students in South Bend schools.
In South Carolina, the Medicaid program is administered through Managed Care Organizations (MCOs), which offer different health care plans to Medicaid beneficiaries. These plans differ in their generosity, network coverage, and other attributes, and they are ranked by the state using a system of “star ratings.” The system of MCOs offers choices to health care consumers and allows plans to compete for consumers. In South Carolina, when consumers do not make an active plan choice, the state uses an algorithm to assign plans to consumers automatically. Starting earlier this year (in 2017), this auto-assignment is now being made using an explicitly random process. We propose to use this randomized assignment feature to study the effect of plan assignment on patient outcomes such as health care utilization and health care expenditures (both overall and by category). This prospective analysis will be complemented with a retrospective analysis that takes advantage of the state’s historical quasi-random round-robin assignment procedure to allocate households to plans. Additionally, we propose to combine the analysis of the randomly assigned population with the population that made active choices to try to distinguish between treatment and selection in accounting for which plans perform better.
Virginia Work Requirements Experiment
Researchers: Colin Grey, Mary Beth Landrum, Timothy Layton, Adam Leive, Ben Sommers
Work requirements are increasingly common in major public assistance programs, initially implemented in cash support (TANF) and food assistance (SNAP) programs, and starting in 2018 in Medicaid. Proponents of work requirements contend that these policies increase beneficiary engagement in work and community activities and may lead to better health. Opponents of work requirements suggest that many low-resource households will lose much-needed benefits, without commensurate improvements in employment. There is limited evidence on the causal impact of work requirements in modern public assistance programs, even as more than a dozen states have proposed new Medicaid work requirements in the past year. We propose a randomized-controlled trial (RCT) of work requirements in Virginia, leveraging state support for the RCT evaluation. We will be able to separate the causal impact of work requirements from another new feature of Virginia’s Medicaid program: monthly premiums. We will study impacts of both policies on employment, earnings, insurance status, and access to care using a mix of administrative and survey-collected data.