An Introduction to the J-PAL State and Local Innovation Initiative: Creating and Using Rigorous Evidence to Address Important Social Issues
The value of rigorous evidence
State and local leaders are actively innovating and experimenting with approaches to address complex policy challenges. Yet too often, they must act without the benefit of rigorous evidence about what has been tried and proven elsewhere, or of learning after the fact which of their own policies and programs were most effective.
The goal of the J-PAL State and Local Innovation Initiative is to assist state and local governments in using randomized evaluations to generate new and widely applicable lessons about which programs and policies work, which work best, and why.
When governments partner with researchers to conduct high-quality and policy-relevant evaluations, the results can be powerful. Randomized evaluations can have an impact beyond those individuals who participate directly by generating evidence about what works, which decision-makers can use in making program improvements and in scaling up effective programs to reach more people.
What follows are two examples of how government leaders have partnered with researchers to examine an important policy issue: the impact of summer jobs programs on the young people who participate. In Chicago, researchers found that the summer jobs programs led to meaningful reductions in violence—an important finding for policymakers. In turn, the results from Chicago inspired policymakers in the City of Philadelphia to apply to the first round of the J-PAL State and Local Innovation Initiative to learn more about the impact of their own summer youth employment programs.
Fighting crime with jobs
While public funds have been devoted to summer youth employment programs for decades in the United States, it is only recently that these programs have been rigorously evaluated.
In Chicago, researcher Sara Heller partnered with Chicago’s Department of Family and Support Services to study the impact of the One Summer Chicago Plus (OSC+) program. From the start, OSC+ was designed primarily as a violence-reduction intervention, and young people were recruited from 13 Chicago high schools with the highest numbers of students at risk of involvement in violence. To fairly allocate the limited number of program slots, researchers ran a lottery to determine which of the young people who applied would be offered a slot in the program and which would be placed on a waitlist. Local community organizations placed participants in non-profit and government jobs and assigned each group of 10 participants to an adult mentor, who helped them deal with barriers to employment and issues in the workplace.
Of the young people who were offered the opportunity to participate in the summer jobs program, some were randomly assigned to 25 hours per week of work, while others were randomly assigned to 15 hours per week of work and spent the remaining 10 hours participating in social-emotional learning sessions. These social-emotional learning sessions were based on cognitive behavioral therapy principles and designed to help participants understand and manage thoughts, emotions, and behaviors that might interfere with their employment.
To measure the impact of the program, researchers matched program application records with existing administrative data from Chicago Public Schools and the Chicago Police Department. Receiving an offer to participate in OSC+ led to a decrease in violent-crime arrests by 43 percent over a 16 month follow-up period, and this decrease in violent crime was very similar for young people assigned to the jobs-only program and for those assigned to receive a job and the social-emotional learning sessions. On the other hand, there were no decreases in arrests for non-violent crimes like property theft or drug offenses, nor any change in the rates of school attendance.
The decline in violent-crime arrests became greater in the months following the end of the program, when young people returned to school. These results suggest that the mechanism through which OSC+ reduced violent crime was psychological and behavioral, rather than a simple story of youth being kept busy and out of trouble. By providing participants with the tools to manage and control their emotions and behavior in the workplace, OSC+ may have also helped them to learn how to defuse situations with the potential for violence.
The success of the OSC+ program was heralded by the media in Chicago and attracted philanthropic support to expand the program. In the years since the initial evaluation in 2012, the program has been expanded to serve more young people, and additional evaluations are ongoing to figure out which parts of the program are most effective and for which participants.
The findings from Chicago inspired officials in the government of the City of Philadelphia to learn more about the impact of their own youth employment programs. WorkReady Philadelphia is a cross-sector, city-wide initiative dedicated to improving the economic outcomes of the region’s youth by attracting, aligning, and investing resources in coordinated, youth-workforce-development strategies. The WorkReady summer program offers educationally enriched work opportunities to in-school and out of school youth ages 12-24. Participants complete a six-week paid work experience that fosters the acquisition of skills through work-based learning.
In the spring of 2016, the City of Philadelphia and the Philadelphia Youth Network (PYN), the managing partner of WorkReady, applied to and were selected to participate in the first round of the J-PAL State and Local Innovation Initiative. The City and PYN received funding and technical assistance to design a randomized evaluation of the WorkReady Program, in partnership with Professor Heller. By leveraging existing administrative data, they hope to explore the impact of summer youth employment on important measures of well-being such as academic achievement, future employment, substance abuse, and pregnancy.
How you can get involved
Through our Request for Letters of Interest, announced once each year, J-PAL North America invites state and local governments to submit proposals describing how rigorous evidence would help answer an important policy question. Selected state and local governments receive technical support from J-PAL staff, flexible project development funding, and connections with J-PAL’s network of academic researchers. State and local governments that have partnered with a researcher from J-PAL’s network to design a high-quality randomized evaluation can apply for additional funding to carry out the evaluation.
In a randomized evaluation, participants and non-participants are randomly selected from a sample of eligible program participants. This randomization ensures that the individuals who participate in a program and those who do not are similar, on average, including in ways that cannot be measured (such as motivation or family support). As a result, we can be confident that any later differences between participants and non-participants are the result of the program, and not some other factor.
When does a randomized evaluation make sense?
Because this initiative is focused on helping state and local leaders design and implement randomized evaluations, applicants must be must be open to working with J-PAL North America to look for ways to include random assignment (for example, a random lottery) in the implementation of the program or policy they are interested in testing. If you have a program that is oversubscribed, a lottery can be a fair method for determining who gets access. Alternatively, if you have a program with low take-up rates, it may be possible to randomly encourage some of those who are eligible to sign up for the program. Pilot programs, or programs that are being expanded to new locations, can provide opportunities to randomly select who receives the program first. Randomized evaluations do not require that you withhold services from the people who would benefit most: those who are neediest or most deserving can be guaranteed access to your program, with randomization limited to those people on either side of your eligibility criteria, for whom the potential benefit may be less clear.
What kinds of policy questions could a randomized evaluation help answer?
The following is a list of some policy questions for which rigorous evidence could help to inform decision-making. This list is not meant to be exhaustive, and we welcome applicants to identify the policy questions that are most relevant in their jurisdictions.
|Possible policy questions||Examples of approaches that state and local governments have considered or implemented|
|Children and families – How can we prevent child abuse and neglect and domestic violence, reduce out-of-home placements, and improve the welfare of children and families?||
|Crime – How can we reduce juvenile incarceration, reduce recidivism, and prevent crime?||
|Education – How can we improve student performance and increase college enrollment and persistence?||
|Employment – How can we increase earnings and employment?||
|Energy and environment – How can we reduce energy use, increase energy efficiency, and reduce pollution?||
|Health – How can we improve people’s physical and mental health and lower health care costs?||
|Household finance – How can we help people increase their savings and reduce their debt?||
|Housing and homelessness – How can we increase housing mobility and reduce homelessness?||
|Governance – How can we increase participation in the political and policy process and improve the efficiency of service delivery?||
For more information and instructions for how to apply, please visit povertyactionlab.org/stateandlocal