How randomized evaluations build evidence to inform workforce program design, policy, and investment

Posted on:
A group of workers in blue jumpsuits; one writing on whiteboard

This post originally appeared on WorkRise, a research-to-action network hosted by the Urban Institute.

A key driver of income inequality in the United States is disparate access to high-quality jobs. Many workers, especially those without traditional four-year college degrees, are virtually shut out of opportunities that offer not just strong wages but also opportunities for career advancement and economic mobility.

Increasing equal access to such jobs is critical. But what strategies could expand opportunities for workers? Where should governments, nonprofits, private employers, and funders focus their resources to maximize impact? And how do we ensure Black and Latinx workers, who face the most barriers to earning college degrees and accessing high-wage opportunities, have an equitable shot at economic mobility and financial security?

One approach is sectoral employment programs, which train jobseekers for employment in high-growth, high-opportunity industries. The Pursuit Fellowship is one such program, equipping adults from under-resourced communities with the skills to build careers in the high-wage technology sector. Pursuit is partnering with J-PAL North America to conduct a randomized evaluation of the fellowship with support from WorkRise and other funders. This project seeks to generate rigorous evidence about the fellowship’s impact on earnings and other worker outcomes. Lessons from the evaluation can inform future workforce research, policymaking, and investments.

J-PAL North America and WorkRise recently convened a panel to discuss Pursuit’s program and evaluation, existing evidence on sectoral employment training, and the role of rigorous research in guiding the development of pathways to high-quality jobs. Speakers included Lawrence Katz, Elisabeth Allison Professor of Economics at Harvard University and co-scientific director at J-PAL North America; Jukay Hsu, cofounder and chief executive officer at Pursuit; and Maurice Jones, chief executive officer of OneTen, an organization whose mission is to hire, promote, and advance 1 million Black Americans into family-sustaining careers. Kristen Titus, executive director of the Cognizant Foundation, moderated the discussion.

Here are four key takeaways.

Americans without four-year college degrees face tremendous barriers to opportunity

Access to high-wage job opportunities in the United States is largely predicated on holding a four-year college degree. In the mid-20th century, most workers experienced rapidly growing wages in a period of shared prosperity, Katz explained. Since 1980, people with different levels of education have experienced widely divergent trends in earnings.

This growing divide means fewer paths to middle-class jobs, financial security, and prosperity for people without equal access to higher education or who don’t thrive in traditional postsecondary programs without further support, Katz said.

Using four-year degrees as an essential credential creates barriers to opportunity that disproportionately hurt Black and Latinx workers, Jones added. More than 70 percent of jobs that pay $40,000 or more require a degree. But, because of persistent and historic racial inequality that has excluded Black and Latinx Americans from economic and educational opportunities, three-quarters of Black workers and more than 80 percent of Latinx workers in the United States do not have four-year degrees.

“The bottom line is we have a credential in our country that is literally a systemic barrier to most of us earning our way into the middle class,” Jones said. “We’ve got to move to a skills-first approach, we’ve got to understand that there are multiple ways to come about those skills. The four-year credential is one way; it should not be the only way.”

Sectoral employment training programs are a promising pathway to quality jobs

Evidence of the effectiveness of sectoral employment programs is strong. A recent J-PAL North America review summarizing results from four studies of nine distinct sectoral employment programs found that such programs significantly increase earnings in the year participants completed their training and in subsequent years.

Moreover, these earnings gains are among the largest found in evaluations of US training programs. The gains are driven by getting people into higher-paying jobs that have higher earnings in the short term and strong upward trajectories over time.

The review also finds the most effective programs include a combination of several core elements: upfront screening for applicants on their basic skills and motivation, career readiness training, wraparound support services for participants, strong connection to employers in the target field, and occupational skills training for high-wage sectors that lead to industry-recognized certificates. These factors help explain why sectoral employment programs have stood out from other training programs with less comprehensive models that have not produced such impressive results.

Overall, the impact of sectoral employment programs is promising, and the evidence is compelling because it comes from a series of randomized evaluations, Katz explained.

Here are how randomized evaluations of the programs worked: Programs had a limited number of slots to fill and more applicants than they could serve. The slots were allocated through random assignment, as in a lottery, which ensured that people who received access to the program were not, on average, systematically different from those who did not. Therefore, subsequent differences in employment, earnings, credentialing, and other outcomes can be attributed to participation in the training programs and not other factors

Evidence from randomized evaluations can generate key lessons to drive system-wide change

Questions remain despite the strength and promise of evidence on sectoral employment programs. These questions include how such programs affect different groups of workers (for example, people without high school diplomas), what range of industries and salaries they can successfully prepare workers for, and how they can be sustainably financed.

The WorkRise-funded evaluation of the Pursuit Fellowship program seeks to contribute valuable evidence on some of these questions. But what motivated Pursuit’s desire to evaluate? Pursuit already has promising nonexperimental results showing that participants’ average annual income climbs from $18,000 before joining the program to $85,000 after graduating—so why participate in a randomized evaluation?

“Outcomes are fundamental,” Pursuit cofounder Jukay Hsu said. “We all care about creating great outcomes for people, for families, for employers, for communities. Without evidence and outcomes, there is no point in doing the work daily, and I think we should be really rooted to that.”

Hsu noted that a randomized evaluation will also enable Pursuit to share lessons with other programs, policymakers, and investors and funders, facilitating much-needed learning.

“We want to understand the counterfactual, we want to understand how this influences policy,” Hsu said. “And it’s not just to make our work better at Pursuit. Hopefully, we can work with others to take part of our model or adopt others so we can support development and growth in the field overall.”

Dismantling barriers will require many innovative ideas and rigorous research to understand impact

All panelists echoed this last point, agreeing that addressing income inequality and barriers to job opportunities will require systems change, a playbook of evidence-based strategies, and engagement with many different actors in the labor market. “The work really is about building an ecosystem,” Jones said.

“We need multiple options that work,” he added. “We definitely need 1,000 flowers to bloom to reach the talent and maximize the opportunity that we have here.”

Building an evidence-based playbook of strategies to equitably increase economic security is imperative to address multiple challenges and barriers to opportunity facing workers. Many innovative ideas and program models exist; achieving greater worker prosperity will require testing those models, as illustrated by J-PAL North America’s evaluation of the Pursuit Fellowship, and sharing those lessons broadly.

To learn more about J-PAL North America’s research and evaluation capacity focused workers and workforce, please reach out to Toby Chaiken, policy and training manager of the J-PAL North America Worker Prosperity Initiative. You can also subscribe to the Worker Prosperity newsletter.

Learn more about J-PAL North America’s evaluation of the Pursuit Fellowship.

Learn more about WorkRise’s current request for proposals funding pilot studies that will generate actionable evidence on pathways for economic mobility.

Authored By