Combining AI and Human Judgement to Improve Labor Market Matching

Credible skills certification can improve employment outcomes–especially for low-income workers who often lack credible resume signals. But such assessments only matter if hiring managers both trust and effectively use them. Yet, the link between assessment and job performance is often ambiguous. Managers may not know which skills are empirically most relevant for the position, or how to interpret the assessment scores. To this end, the implementing partner for this project uses AI to evaluate applicant fit to various working-class jobs for clients using their existing employees' data. Yet, this leaves hiring managers with high context uncertainty: the best barista at a coffee chain's airport location may be very different from their best barista at a rural location. This uncertainty produces what the researchers refer to as big data's "last mile" problem; many of the attributes, such as team dynamics or customer personalities, needed to fully personalize algorithmic recommendations, are incredibly difficult to capture within datasets. This is true for AI applications in various social sectors.

To resolve this uncertainty, the researchers propose a method to improve AI output interpretability. During a five-month pilot, a random subset of hiring managers will receive enhanced reports comparing applicants with similar and dissimilar existing employees. This treatment should help managers interpret an applicant's fit for the specific position. Using Apli's access to employee performance data, they will evaluate immediate employment outcomes and decision-making patterns. This pilot will assess effect sizes for several outcomes to inform power calculations for a future randomized evaluation. It will also refine a survey capturing hiring managers' beliefs to supplement administrative data. Beyond the study's practical value, it will advance our conceptual understanding of uncertainty in hiring. Distributional implications include uncertainty's disproportionate impact on underrepresented populations, as uncertainty interacts with hiring managers' implicit biases.