State and Local Innovation Initiative – Introduction

PDF version

The value of rigorous evidence

State and local leaders are actively innovating and experimenting with approaches to address complex policy challenges. Yet too often, they must act without the benefit of rigorous evidence about what has been tried and proven elsewhere, or of learning after the fact which of their own policies and programs were most effective.

The goal of the J-PAL State and Local Innovation Initiative is to assist state and local governments in using randomized evaluations to generate new and widely applicable lessons about which programs and policies work, which work best, and why.

When governments partner with researchers to conduct high-quality and policy-relevant evaluations, the results can be powerful. Randomized evaluations can have an impact beyond those individuals who participate directly by generating evidence about what works, which decision-makers can use in making program improvements and in scaling up effective programs to reach more people.

What follows are two examples of how government leaders have partnered with researchers to examine an important policy issue: the impact of summer jobs programs on the young people who participate. In Chicago, researchers found that the summer jobs programs led to meaningful reductions in violence—an important finding for policymakers. In turn, the results from Chicago inspired policymakers in the City of Philadelphia to apply to the first round of the J-PAL State and Local Innovation Initiative to learn more about the impact of their own youth summer employment programs.

Fighting crime with jobs

While public funds have been devoted to youth summer employment programs for decades in the United States, it is only recently that these programs have been rigorously evaluated.

In Chicago, researcher Sara Heller partnered with Chicago’s Department of Family and Support Services to study the impact of the One Summer Chicago Plus (OSC+) program. From the start, OSC+ was designed primarily as a violence-reduction intervention, and young people were recruited from 13 Chicago high schools with the highest numbers of students at risk of involvement in violence. To fairly allocate the limited number of program slots, researchers ran a lottery to determine which of the young people who applied would be offered a slot in the program and which would be placed on a waitlist. Local community organizations placed participants in non-profit and government jobs and assigned each group of 10 participants to an adult mentor, who helped them deal with barriers to employment and issues in the workplace.

Of the young people who were offered the opportunity to participate in the summer jobs program, some were randomly assigned to 25 hours per week of work, while others were randomly assigned to 15 hours per week of work and spent the remaining 10 hours participating in social-emotional learning sessions. These social-emotional learning sessions were based on cognitive behavioral therapy principles and designed to help participants understand and manage thoughts, emotions, and behaviors that might interfere with their employment.

To measure the impact of the program, researchers matched program application records with existing administrative data from Chicago Public Schools and the Chicago Police Department. Receiving an offer to participate in OSC+ led to a decrease in violent-crime arrests by 43 percent over a 16 month follow-up period, and this decrease in violent crime was very similar for young people assigned to the jobs-only program and for those assigned to receive a job and the social-emotional learning sessions. On the other hand, there were no decreases in arrests for non-violent crimes like property theft or drug offenses, nor any change in the rates of school attendance.

The decline in violent-crime arrests became greater in the months following the end of the program, when young people returned to school. These results suggest that the mechanism through which OSC+ reduced violent crime was psychological and behavioral, rather than a simple story of youth being kept busy and out of trouble. By providing participants with the tools to manage and control their emotions and behavior in the workplace, OSC+ may have also helped them to learn how to defuse situations with the potential for violence.

The success of the OSC+ program was heralded by the media in Chicago and attracted philanthropic support to expand the program. In the years since the initial evaluation in 2012, the program has been expanded to serve more young people, and additional evaluations are ongoing to figure out which parts of the program are most effective and for which participants.

The findings from Chicago inspired officials in the government of the City of Philadelphia to learn more about the impact of their own youth employment programs. WorkReady Philadelphia is a cross-sector, city-wide initiative dedicated to improving the economic outcomes of the region’s youth by attracting, aligning, and investing resources in coordinated, youth-workforce-development strategies. The WorkReady summer program offers educationally enriched work opportunities to in-school and out of school youth ages 12-24. Participants complete a six-week paid work experience that fosters the acquisition of skills through work-based learning.

In the spring of 2016, the City of Philadelphia and the Philadelphia Youth Network (PYN), the managing partner of WorkReady, applied to and were selected to participate in the first round of the J-PAL State and Local Innovation Initiative. The City and PYN received funding and technical assistance to design a randomized evaluation of the WorkReady Program, in partnership with Professor Heller. By leveraging existing administrative data, they hope to explore the impact of summer youth employment on important measures of well-being such as academic achievement, future employment, substance abuse, and pregnancy.

How you can get involved

In addition to Philadelphia, four other jurisdictions were selected in the first round of the J-PAL State and Local Innovation Initiative: Pennsylvania, Puerto Rico, South Carolina, and Rochester, New York. The City of Baltimore, Maryland; King County, Washington; and Santa Clara County; California were selected in the second round of the initiative in June 2017.  J-PAL North America anticipates announcing the next round Request for Letters of Interest from state and local leaders interested in designing and implementing randomized evaluations in Fall 2017.

Why randomize?

In a randomized evaluation, participants and non-participants are randomly selected from a sample of eligible program participants. This randomization ensures that the individuals who participate in a program and those who do not are similar, on average, including in ways that cannot be measured (such as motivation or family support). As a result, we can be confident that any later differences between participants and non-participants are the result of the program, and not some other factor.

When does a randomized evaluation make sense?

Because this initiative is focused on helping state and local leaders design and implement randomized evaluations, applicants must be must be open to working with J-PAL North America to look for ways to include random assignment (for example, a random lottery) in the implementation of the program or policy they are interested in testing. If you have a program that is oversubscribed, a lottery can be a fair method for determining who gets access. Alternatively, if you have a program with low take-up rates, it may be possible to randomly encourage some of those who are eligible to sign up for the program. Pilot programs, or programs that are being expanded to new locations, can provide opportunities to randomly select who receives the program first. Randomized evaluations do not require that you withhold services from the people who would benefit most: those who are neediest or most deserving can be guaranteed access to your program, with randomization limited to those people on either side of your eligibility criteria, for whom the potential benefit may be less clear.

What kinds of policy questions could a randomized evaluation help answer?

The following is a list of some policy questions for which rigorous evidence could help to inform decision-making. This list is not meant to be exhaustive, and we welcome applicants to identify the policy questions that are most relevant in their jurisdictions.

Possible policy questions Examples of approaches that state and local governments have considered or implemented
Children and families – How can we prevent child abuse and neglect and domestic violence, reduce out-of-home placements, and improve the welfare of children and families?
  • Home visiting
  • Parent education and coaching
  • Child support enforcement
  • Family strengthening programs
  • Teen pregnancy prevention
Crime – How can we reduce juvenile incarceration, reduce recidivism, and prevent crime?
  • Services for at-risk youth and those aging out of the juvenile justice system
  • Juvenile diversion programs
  • Re-entry programs for ex-offenders
  • Probation/parole services
Education – How can we improve student performance and increase college enrollment and persistence?
  • Preschool and other early childhood programs
  • In-school or afterschool tutoring
  • Coaching or mentoring
  • Information and assistance with applying for financial aid
  • Emergency financial assistance for college students
Employment – How can we increase earnings and employment?
  • Summer youth employment
  • Apprenticeship programs
  • Subsidized employment programs
  • Job search assistance
  • Career retraining for the unemployed
Energy and environment – How can we reduce energy use, increase energy efficiency, and reduce pollution?
  • Weatherization assistance
  • Home energy use reports
  • Promoting biking, walking, or other alternative forms of transportation
Health – How can we improve people’s physical and mental health and lower health care costs?
  • Reducing the over-prescription of opioids
  • Coordinated care for patients with complex needs
  • Incentives payments for healthy behaviors (e.g. smoking cessation, weight loss)
  • Assistance with selecting and enrolling in health plans
Household finance – How can we help people increase their savings and reduce their debt?
  • Financial education
  • Encouraging take-up of the Earned Income Tax Credit (EITC) and other tax benefits
  • Subsidized savings accounts
Housing and homelessness – How can we increase housing mobility and reduce homelessness?
  • Housing vouchers
  • Supportive housing for individuals and families experiencing chronic homelessness
  • Rapid rehousing
  • Emergency assistance for individuals and families at risk of homelessness
Governance – How can we increase participation in the political and policy process and improve the efficiency of service delivery?
  • Encouraging voter participation
  • Improving inspection services
  • Improving tax enforcement

For more information and instructions for how to apply, please visit povertyactionlab.org/stateandlocal

PDF iconAn Introduction to the J-PAL State and Local Innovation Initiative