J-PAL North America Evidence Portal
J-PAL North America’s Evidence Portal provides a snapshot of the research in North America by J-PAL affiliates and invited researchers across nine social policy areas related to poverty alleviation. To date, J-PAL affiliated researchers have conducted over 200 ongoing or completed randomized evaluations in North America to answer critical policy questions.
Based on a J-PAL review of six randomized evaluations, simplifying the college application process and supporting students through the transition to college can increase college enrollment and persistence.
- Personalized assistance and fee waivers are key elements of effective programs designed to smooth the application process.
- Merely providing generic information about college is typically not enough to encourage college enrollment. However, timely and specific reminders, like text message reminders to admitted students about tasks required for matriculation, can help students enroll in college at higher rates.
- Students with limited access to other supports benefited the most from support with the transition to college.
- Most students who enrolled under these programs stayed in college after matriculating.
- For more information, see J-PAL’s policy bulletin on Simplifying Barriers Along the Bridge to College.
Comprehensive case management improved persistence and graduation rates among low-income students at a community college in Fort Worth, Texas.
- Researchers conducted a randomized evaluation to measure the impact of a comprehensive case management program offered at a community college in Fort Worth, Texas in 2013 called Stay the Course, which included mentoring and emergency financial assistance, on students’ academic outcomes.
- Six semesters after the program started, students assigned to participate in the program were 5.6 percentage points more likely to be enrolled in school from a baseline of 44 percent (a 12.7 percent increase).
- These large impacts were largely driven by the program’s effect on female participants.
- Outcomes for students who received only financial assistance were not significantly different from students enrolled in the control group, suggesting that financial assistance alone may not be sufficient to help low-income students persist in school.
Based on a J-PAL review of 126 experimental studies, some uses of education technology hold promise in improving college enrollment and success.
- Technology-based nudges that encourage specific, one-time actions—such as text message reminders to complete course registrations—can have meaningful, if modest, impacts on a variety of education-related outcomes, often at extremely low-costs.
- For example, timely and specific reminders, like text message reminders to admitted students about tasks required for matriculation, can help students enroll in college at higher rates.
- Combining online and in-person instruction can work as well as traditional in-person only classes, which suggests blended learning may be cost-effective. Students in online-only courses, however, tend to perform worse than students in in-person-only courses.
- For more information, see J-PAL North America’s evidence review on education technology.
Intensive individualized tutoring significantly improved students’ grades in math and increased the likelihood of passing high school classes in Chicago Public Schools.
- Students who participated in the Match Tutoring program in the 2013-2014 school year in Chicago learned an extra one to two years’ worth of math beyond what their peers in the control group learned in one academic year.
- The large test score gain per dollar spent relative to previous interventions designed to support disadvantaged students demonstrates that it is possible to substantially and cost-effectively boost academic achievement for disadvantaged students, even once they reach adolescence.
- Following this evaluation, SAGA Innovations, the same organization that developed the Match tutoring model, has expanded its individualized tutoring program to serve 2,500 students in New York City and Chicago.
Based on a review of evidence by J-PAL staff and affiliated researchers, charter schools vary widely in their estimated impacts on student outcomes.
- A common feature of charter schools with positive effects has been the adoption of a No Excuses educational approach. This approach is characterized by strict and clear disciplinary policies, mandated intensive tutoring, longer instruction times, frequent teacher feedback, and high expectations for students.
- Most No Excuses schools have located in urban areas, and urban charter schools have tended to attract students who would have otherwise attended poor-performing traditional public schools. As a result, it is difficult to disentangle whether the substantial performance gains associated with attending a No Excuses charter school in an urban area are due to the No Excuses approach or to the low quality of the public school alternatives.
- In urban areas, the effect of charter school attendance was larger for black students, Hispanic students, and previously poor-performing students.
- Some No Excuses charter schools in urban areas have been shown to have positive impacts on longer-term outcomes, including improving college preparation and enrollment, reducing teen pregnancy, and reducing incarceration among male students.
- A study in Texas that randomly assigned traditional public schools to implement a set of practices associated with the most effective charter schools found that these practices increased math achievement.
- For more information, see J-PAL’s policy bulletin on Charter School Lotteries in the United States.
Based on a literature review by J-PAL staff and affiliated researchers, some uses of education technology hold promise in improving learning outcomes in early childhood and K-12 education.
- Initiatives that expand access to computers and internet alone generally do not improve students’ academic outcomes, but do increase computer usage and improve computer proficiency.
- Educational software designed to help students develop particular skills at their own rate of progress have shown promise in improving learning outcomes, particularly in math.
- Technology-based nudges that encourage specific, one-time actions—such as suggesting questions parents should ask their children—can have meaningful, if modest, impacts on a variety of education-related outcomes, often at extremely low-costs.
- For more information, see J-PAL North America’s evidence review on education technology.
In Tennessee kindergarten classrooms, smaller class sizes raised future college attendance, and higher quality classrooms improved college attendance and future earnings.
- In the late 1980’s, the Student/Teacher Achievement Ratio (STAR) project randomly assigned Tennessee kindergarten students and teachers to classes with differing characteristics, enabling researchers to conduct a long-term follow-up to measure the effects of class size, teacher quality, and class quality on future student outcomes.
- A smaller class size increased college attendance, and having a kindergarten teacher with over ten years of experience increased students’ average annual wage earnings by $1,093.
- Similarly, an improvement in class quality, which the study defined as the difference in standardized exam scores between a student’s classmates and students in the same grade, increased average annual income earned between ages 25 and 27.
- The positive effects of early childhood education on test scores diminish over time, but positive effects on earnings persist through adulthood.
Behavioral tools such as goal setting and reminders motivated parents in Chicago to engage often and productively with their children.
- In 2014, researchers evaluated the impact of providing small behavioral tools—including a goal-setting website, text message reminders, and social rewards such as congratulatory texts—to encourage parents to read with their children.
- The behavioral tools and information increased reading efforts among parents in the PACT program, particularly among parents who scored lower on a test that measured parents' patience levels.
Reducing the cost of health care through an expansion of Medicaid in Oregon caused people to seek more health services and reduced financial strain.
- Following a lottery in 2008 that gave low-income uninsured adults the chance to apply for Medicaid in Oregon, researchers studied the impact of health insurance in a series of evaluations.
- They found that Medicaid coverage increased health care use, including use of preventive services and visits to emergency departments, reduced depression, and improved self-reported health over the first two years. However, they did not find evidence that Medicaid coverage improved physical health.
- Medicaid coverage also reduced financial strain for individuals; however, they did not find evidence that Medicaid coverage affected employment.
- For more information, see the J-PAL policy briefcase on Insuring the Uninsured.
Two randomized evaluations in Oregon and Colorado suggest that adopting low-cost, behaviorally-informed mass outreach approaches could substantially increase insurance coverage of vulnerable populations.
- Additional evidence on Medicaid from Oregon demonstrated that improved communication and low-cost “nudges,” such as behaviorally-informed postcards and automated telephone outreach, increased Medicaid enrollment among likely eligible groups by 3.5 percentage points from a base of 38 percent (a 10 percent increase), at a cost of approximately $125 per new enrollee. This increase may be particularly durable among hard-to-reach, vulnerable populations, and these interventions may be as effective as more resource-intensive, individualized outreach strategies.
- Another study in Colorado evaluated the impact of mailing and e-mailing information about a complex set of insurance plan options on the state Marketplace website. While consumers who received the information were more likely to shop for plans on the Marketplace website, they were not more likely to switch plans.
Providing vision screening and free eyeglasses improved educational outcomes in Florida elementary schools.
- Providing vision screening and free eyeglasses to low-income elementary school children in Florida in 2011 increased achievement on standardized test scores, though providing the screening without the free eyeglasses had no effect.
- The education gains from the intervention faded out within a year and a half for most students, which the researchers suggest may be due, in part, to students losing, breaking, or not continuing to wear their glasses.
Seeing a black male doctor significantly increased the take-up of preventive health services among black men in Oakland, California.
- In 2017, researchers examined the impact of race concordance (when the race of a patient matches that of the patient’s physician) and incentives on the take-up of preventive health services by black men.
- For black men, seeing a black male doctor significantly boosted demand for all preventive health services, and especially for more invasive tests. For instance, seeing a black doctor increased take-up of blood pressure measurement among black men by 11 percentage points, compared to an average take-up rate of 72 percent for patients who were seen by a non-black doctor (an increase of 15 percent).
In the first year of an ongoing evaluation in the United States, bundled payments–an alternative Medicare payment model–for knee and hip replacements reduced health care utilization without impacting quality of care.
- One of the primary alternative payment models for medical care in the United States is bundled payments, where one payment is made for all services related to a specific episode of care.
- In the first year of an ongoing study across 196 metropolitan statistical areas, bundled payments for knee and hip replacements starting in 2016 reduced health care utilization, with no evidence of harm to health care quality or change in the volume or mix of patients treated.
- Once all payments were factored in, there was no evidence that this bundled payment model had an effect on total Medicare spending, but this may change as hospitals’ incentives change over the course of the study.
- A separate research team conducted a two-year analysis of the ongoing study and found similar results.
A workplace wellness program at the University of Illinois at Urbana-Champaign did not change participants’ health behaviors in the first year of an ongoing study.
- Workplace wellness programs are broadly defined as employer-offered programs to promote health or prevent disease.
- In 2016, researchers implemented a large-scale workplace wellness program at the University of Illinois at Urbana-Champaign to evaluate its effects on employee health behaviors, self-reported health status, productivity, and total medical expenditures.
- In the first year of this ongoing study, the researchers found that financial incentives increased participation in the program, although only to a point. The workplace wellness program itself, however, did not affect any measured health or cost outcome.
A J-PAL review of three randomized evaluations found that cognitive behavioral therapy can be an effective approach to reduce criminal behavior among young men.
- Cognitive behavioral therapy (CBT) programs, which aim to reduce self-destructive behaviors by teaching individuals to evaluate and modify the way they think and the decisions they make, have reduced violent crime arrest rates, recidivism rates, and high school dropout rates among Chicago youth.
- When delivered in schools, CBT also increased graduation rates by three percentage points from a base of 34 percent (a 9 percent increase), even though the reduction in crime among this group did not persist beyond the yearlong program.
- CBT may have been effective because it changed participants’ decision-making processes. In the United States, students in CBT programs learned to slow down their decision-making to better assess situations and choose appropriate responses.
- For more information, see J-PAL’s policy bulletin on Practicing Choices, Preventing Crime.
Based on a J-PAL review of three randomized evaluations in Chicago and New York City, summer job programs offer considerable promise in reducing violent crime among youth.
- Summer youth employment programs that provided minimum wage summer jobs to youth reduced arrests for violent crimes in Chicago and reduced incarceration and premature deaths in New York City.
- Summer jobs programs provided employment to youth who would otherwise have difficulty finding a job. However, youth who participated in summer jobs programs did not have higher employment or earnings after the program year.
- The violence-prevention impact of summer jobs extended beyond the program summer. In Chicago, the bulk of the decline in violence occurred after the program ended, and in New York City, the reduction in death rates continued over a number of years after the program summer.
- For more information, see the J-PAL policy briefcase Stopping a Bullet with a Summer Job.
Simple, low-cost behavioral nudges helped reduce the likelihood that a person with a court summons failed to appear in court in New York City, and consequently reduced pre-trial detention.
- Researchers partnered with policymakers in New York City in 2016 and found that text message reminders were able to reduce failures to appear by 10 percentage points from a baseline of 38 percent (a 26 percent decrease), translating to 3,700 fewer arrest warrants per year.
An intensive reentry and recidivism program in Wisconsin helped formerly incarcerated individuals successfully reintegrate into society.
- Researchers examined whether a program in Wisconsin that provided inmates with services both prior to and after their release could improve their employment prospects and reduce recidivism.
- The services prior to release were offered starting in 2009 and included access to a community employment program, vocational and soft-skills training, remedial education, participation in restorative justice circles, treatment for drug and alcohol abuse, a cognitive therapy program, and meetings with a coordinated care team. After release, services included access to substance-abuse treatment, participation in the community employment program, and subsidized employment for six months.
- One year later, the program increased employment and reduced the likelihood of re-arrest. Among ex-offenders in the treatment group, 81 percent reported earnings at some point during their first twelve months out of prison, compared to 59 percent of ex-offenders in the control group. However, earnings among the people who were formerly incarcerated who were employed were low and left many below the poverty line; Median annual earnings for ex-offenders in the treatment group were US$2,690 compared to US$462 for ex-offenders in the control group.
- Since many different services were offered to members of the treatment group, additional investigation is needed to understand which services are most effective and why.
Two randomized evaluations point to considerable discrimination in labor markets in the United States and Canada.
- In Boston and Chicago, researchers examined the level of racial discrimination in the United States labor market in the early 2000s by randomly assigning identical résumés to have black-sounding or white-sounding names and observing the impact on requests for interviews from employers. Results found that résumés with white-sounding names received 50 percent more callbacks than those with black-sounding names, indicating that many employers discriminate against applicants they perceive as being black.
- In Toronto and Montreal, researchers randomly manipulated thousands of résumés in 2008 to measure the effects that foreign experience and having a name of Chinese, Indian, Pakistani, or Greek origin (all large immigrant groups in Canada) have on callback rates from employers. Changing only the name on a résumé from an English-sounding name to an Indian, Pakistani, or Chinese name decreased the likelihood of a callback by 4.4 percentage points from a base of 15.7 percent (a 28 percent decrease). Résumés with English-sounding names received more callbacks than those with Chinese, Indian, Pakistani, or Greek names. Work experience in Canada increased responses for résumés with non-English-sounding names, but callback rates were still lower than those with English-sounding names.
Long-term follow-up studies from two randomized evaluations suggest that childhood factors, including neighborhood quality and early childhood education, impact employment outcomes later in life.
- The US Department of Housing and Urban Development (HUD) launched the Moving to Opportunity (MTO) project in 1994 to test the impact of offering housing vouchers to families living in high-poverty neighborhoods with the condition that they move to neighborhoods with lower concentrations of poverty. Researchers found that children whose families moved to lower poverty neighborhoods when they were young have higher incomes in adulthood. However, moving to a lower-poverty neighborhood had no effect on employment or income for those who were adults when they moved.
- In Tennessee, researchers measured the effect of class size, teacher quality, and classroom quality on earnings and other future outcomes for children participating in the Student/Teacher Achievement Ratio (STAR) project. The study found that more experienced teachers increased future earnings of their students, and that higher quality classrooms also improved future earnings.
Moving from high-poverty housing projects to lower-poverty neighborhoods improved the later-life outcomes for children and reduced the intergenerational persistence of poverty.
- The US Department of Housing and Urban Development (HUD) launched the Moving to Opportunity (MTO) project in 1994 to test the impact of offering housing vouchers to families living in high-poverty neighborhoods with the condition that they move to neighborhoods with lower concentrations of poverty.
- The MTO study found that families that moved to lower-poverty neighborhoods were healthier and happier, although moving to a lower-poverty neighborhood had no effect on employment or income for those who were adults when they moved.
- Children who were under 13 when their families moved to lower-poverty neighborhoods were more likely to attend college and had substantially higher incomes in adulthood.
- However, moving may be disruptive to children, and children who were 13 or older when their parents moved to lower-poverty areas had slightly lower educational and earnings outcomes relative to children in the control group.
- For more information, see J-PAL’s policy briefcase on Moving to Opportunity.
Two studies of legal assistance programs in Massachusetts suggest that more research is needed on which legal tactics are most effective for improving the legal outcomes of renters facing eviction.
- Researchers evaluated two approaches to reducing evictions in 2010 in a Massachusetts District Court. They compared the impact of offering full legal representation to offering more limited legal services, where a licensed attorney offered a reduced set of legal services during a critical time in the case. They found that individuals offered full legal representation, with attorneys employing confrontational legal tactics, were more likely to maintain the right to live in the unit and paid less in settlements relative to the comparison group.
- However, in an evaluation at a Massachusetts Housing Court in 2009, occupants assigned to full representation by attorneys who employed facilitative, non-confrontational legal tactics had an equal chance of retaining the right to occupy the property as those offered the limited assistance representation.
- Since researchers were unable to isolate the impact of the intensity of legal services from the style of the attorneys, more research is needed to determine the effectiveness of full legal representation relative to limited legal services as well as the relative benefits of confrontational versus non-confrontational tactics.
Providing payday borrowers in the United States with information about the true costs of payday loans reduced demand for payday loans.
- In 2008 and 2009, researchers provided payday borrowers in the United States information about the true costs of payday loans, in order to find out whether they would respond by changing their demand for the product.
- Better informed individuals significantly reduced both the frequency that consumers took out payday loans and the borrowing amounts, suggesting that getting consumers to think more broadly about the decision to take out a payday loan may result in a reduction in the amount and frequency of payday borrowing.
Repeated notifications with simple, highly relevant information improved US taxpayers’ likelihood of claiming their eligible benefits through the Earned Income Tax Credit (EITC).
- In partnership with the US Internal Revenue Service (IRS), researchers tested the effectiveness of different messages to taxpayers who were eligible for, but did not claim, the Earned Income Tax Credit.
- While simple, targeted messages increased take-up of the EITC, information designed to reduce stigma and perceptions of the economic costs of filing paperwork had no effect on benefit claims.
- Based on the results of this study, the IRS revised the notices sent to taxpayers on a national scale.
Small financial incentives and changes in social networks increased individuals’ likelihood to enroll in a retirement plan offered by a large university in the United States.
- In 2001, individuals at a large university were promised small financial incentives to attend a benefits fair where they were given information about different benefit plans offered by their employer. The researchers found that the small financial incentives successfully induced employees to attend the benefits fair.
- In offices where only some individuals received the promise of an incentive, those who did not receive the promise were just as likely to enroll in a Tax Deferred Account retirement plan as those who did receive it.
- The evaluation results suggest that an individual's decision to participate in a Tax Deferred Account may be affected by their social network and not just directly-received information on the account's benefits.
Higher matching rates for IRA contributions significantly increased IRA participation and contributions among H&R Block clients in St. Louis.
- In 2005, researchers tested the impact of offering clients different matching rates for contributions to their Individual Retirement Arrangements (IRA).
- They found higher matching rates significantly increased IRA participation and contributions.
- The combination of financial incentives, tax preparer assistance, and the opportunity to use part of an income tax refund to save could generate increases in both the efficacy of federal tax incentives and the willingness of households to contribute to retirement savings accounts.
Nudges with social comparisons reduced household energy use by small but significant amounts among households served by 12 different utility companies across the United States.
- Home energy reports successfully reduced home energy consumption, based on a study of nearly 600,000 households served by 12 different utility companies across the United States from 2008 to 2013. The reports, which compared a household's energy use to that of 100 neighbors and included advice for conserving energy and saving money, reduced household energy consumption by an average of 0.6 kWh/day from a baseline of 30 kWh/day (a 2 percent reduction).
- While effective, home reports alone will not be enough to substantially reduce carbon emissions or address water scarcity. For example, if the entire US population received home energy reports and their effects were lasting and the same at scale, they would achieve about 0.8–1.3 percent of the carbon emissions reductions the United States pledged at the Paris Climate Accord.
Results from three randomized evaluations suggest that information alone is often not enough to encourage investments in energy efficiency.
- Providing fuel economy information to potential car buyers in both dealership and online settings had no effect on consumer purchasing behavior.
- Providing information about and subsidies for compact fluorescent lightbulbs (CFLs) increased consumers’ willingness to purchase CFLs in an online survey, but providing information alongside a subsidy (in the form of a coupon) to customers in stores did not increase the use of the coupon.
- Similarly, researchers found that providing information alone did not increase demand for energy efficient appliances, but discounts did.
Providing subsidized energy audits and retrofits to low-income households in Michigan reduced household energy consumption, although the upfront costs of the program outweighed the realized energy savings.
- An evaluation from 2010 to 2012 of the federal Weatherization Assistance Program (WAP), which provides free energy audits and free energy-efficiency retrofits to low-income households, found that the costs of the program substantially outweighed the benefits. The projected savings from the program’s engineering model were three times higher than the actual savings.
- Promotional letters and audit subsidies caused higher rates of participation in energy efficiency audits. However, few households made subsequent investments in energy efficient home solutions, and the cost savings from these investments fell short of projections.
Two randomized evaluations from Connecticut and Michigan demonstrated that personalized outreach and behavioral science principles increased voter turnout.
- Based on findings from a large randomized evaluation in New Haven, Connecticut that took place before the 1998 general election, personal canvassing was much more effective in increasing voter turnout than phone calls or direct mail. Successful personal contact through canvassing raised the probability of voter turnout by 8.7 percentage points relative to those in the comparison group who would likely have been reached with canvassing.
- In a different study of a 2007 election in Michigan, researchers evaluated whether disclosure of past participation had a stronger impact on turnout when it reminded voters of instances in which they previously voted or of instances in which they failed to vote. The results suggest that mailings disclosing past voting behavior had strong positive effects on voter turnout, and these effects were stronger when voters were told of a recent election in which they failed to vote. Overall, the findings suggest that social pressure from receiving any kind of information about past turnout led to a 5.2 percentage point increase in the number of votes cast from a baseline of 22.5 percent (a 23 percent increase).
In Connecticut, informing people who were previously incarcerated for felony charges of their voting eligibility reduced the negative impact of incarceration on political participation.
- In 2012, researchers evaluated the impact of mailings that informed people who were formerly incarcerated for felony charges of their eligibility to vote on political participation.
- The mailing increased political participation, including voter registration and voting, particularly among people formerly incarcerated for felonies that were active voters prior to incarceration.
Contact: J-PAL North America Policy Manager Vincent Quan.