The Effect of Academic Achievement Awards on College Success in Canada

Researchers:
Joshua Angrist
Tyler Williams
Fieldwork by:
Location:
Ontario, Canada
Sample:
1271 students
Timeline:
2008 - 2008
Target group:
  • Higher education and universities
  • Students
Outcome of interest:
  • Student learning
Intervention type:
  • Coaching and mentoring
  • Scholarships
AEA RCT registration number:
AEARCTR-0001876
Partners:

As many as 40 percent of U.S. and 30 percent of Canadian undergraduates have yet to finish college six years after matriculating—delays which incur both private and social costs. In an effort to boost their students’ grades, most universities deploy an array of support services, but there is little credible evidence that these efforts justify their cost. Researchers examined the impact of a program that offered financial merit awards combined with academic support services on the learning outcomes of Canadian university students. Results suggest that the impact of such incentives was modest.

Policy issue

As many as 40 percent of U.S. and 30 percent of Canadian undergraduates have yet to finish college six years after matriculating. These delays, which are typically due to a failure to meet academic degree requirements, incur both private and social costs. Students who take longer to complete college pay a higher cost in foregone earnings, and those who fail to complete a degree program forgo the benefit of having that credential in the job market. At public institutions, slow progress towards a degree can be costly for taxpayers. In an effort to boost their students’ grades, most universities deploy an array of support services, but there is little credible evidence that these efforts justify their cost. This evaluation measured the impact of extra academic support services combined with financial merit awards for weaker students on student performance in Canada.

Context of the evaluation

The Opportunity Knocks (OK) program was an award program piloted at the Ontario commuter campus of a large Canadian public university. The six-year completion rate on this campus is about 73 percent, with an entering class of about 2,500 students. The OK program provided both financial awards for course grades, as well as mentoring support to participants. The program application was open to all first- and second-year university students who had requested financial aid, had an email address, had a high school GPA recorded in the university administrative information system, and had enrolled for at least 1.5 credits in the 2008 fall semester. Students who applied to the program had an average high school GPA of 82.5 percent, and less than half spoke English as a first language.

Details of the intervention

Researchers used a randomized evaluation to measure the impact of the OK program’s financial awards and academic support services on student achievement. All eligible students were invited to apply to the program by completing an intake survey. Of the 1,271 eligible students who completed the survey, 400 first and second year students were randomly selected to participate—100 male and 100 female students in each year. The remaining students, who completed the survey but were not invited to participate in the OK program, served as a control group.

OK program participants earned $100 for each class in which they attained a grade of 70 percent or better, and an additional $20 for each percentage point above 70 percent. Rewards were paid out at the end of every term. In addition, these students were randomly assigned and contacted by peer advisors—paid and trained upper-class students who provided advice about study strategies, time management, and university bureaucracy as well as information about OK scholarships. Advisors emailed advisees once every two to three weeks, whether or not the advisees responded. Program engagement was high; by year’s end, at least 86 percent of OK participants had checked their scholarship earnings on the OK website or emailed a peer advisor or program staff.

Data for this study came primarily from university records containing information on applicants, enrolled students, and course grades. Researchers supplemented this with a baseline survey used to identify the population eligible for random assignment.

Results and policy lessons

The OK program was popular with participants; sign-up rates and program engagement were high and participants in follow-up focus group interviews were enthusiastic about the experience. However, overall program effects on achievement were modest. Comparing the OK scholarship earnings of participants to the hypothetical earnings of students in the control group, researchers found that second-year OK students earned about $180 more at the end of the year than the control group. The number of courses in which students earned grades of at least 70 percent also increased for second-year OK students by about half a course more than for the control group. However, the only significant impact on overall GPA was found for second-year men, who experienced a 0.27 increase on a four-point scale, compared to second-year men in the control group.

The effect on scholarship earnings was larger for OK students who better understood the scholarship formula linking grades to awards. To gauge students’ understanding of the scholarship formula, the baseline survey included two questions that required students to calculate award amounts in various scenarios. Of the students who answered both questions correctly, second-year OK participants earned $245 more at the end of the year than non-participants. However, these increased earnings came from an increase in grades earned around the 70 percent threshold, which was not enough to improve students’ GPA overall.

The effect of the program on longer-term academic outcomes was also modest. In the fall semester of 2009 after the end of the program, first-year students saw a slight increase in GPA but the number of courses with grades over 70 percent did not change. One explanation for the muted effectiveness of merit scholarships may be that students with grades at or below 70 percent have trouble developing effective study strategies.