Student Coaching: How Far Can Technology Go?
- Higher education and universities
- Students
- Student learning
- Coaching and mentoring
- Digital and mobile
Nearly half of college students in the United States fail to complete their bachelors’ degree within six years. One-on-one coaching and tutoring can lead to improvements in student outcomes, but are far more expensive than technology-driven alternatives. Researchers conducted a randomized evaluation to test the impact of three different coaching methods on academic outcomes: one-on-one in-person coaching, text messaging, and an online values and goals exercise. One-on-one coaching substantially improved student outcomes, while text messaging and the online exercise had no detectable impact.
Policy issue
Throughout North America, there is rising concern about college completion rates. Though college enrollment increased between 1970 and 1999, college completion fell by 25 percent over the same period. More recent research indicates that only 56 percent of students who pursue a university degree actually complete it within six years. Prior research has shown that one-on-one interventions can have a meaningful impact on student outcomes. However, these one-on-one interventions are often resource-intensive, expensive to implement, and difficult to scale. This study evaluates the role of technology in mimicking the high-impact interventions that have proven effective in the past at much lower cost.
Context of the evaluation
The study takes place across the three campuses of the University of Toronto, a large and prestigious public research university in Canada. The intervention was implemented for all students enrolled in first-year economics courses in the fall of 2015. Approximately half of the study participants were female, 24 percent were first-generation students, and 43 percent were international students. The average age of participants was 18.5. About 30 percent lived on campus. Participants had incoming high school grades of 87 percent on average.
Details of the intervention
Researchers conducted a randomized evaluation to test the impact of three student coaching methods on student outcomes. All students in the study were asked to do a baseline “warm up” exercise that varied by group assignment. Participants were randomly assigned to one of four groups:
- Online Only: This group participated in an online module in which students were prompted to think about their motivations for the upcoming year and their general goals for the future. The exercise was intended to make students’ objectives and goals more salient to them at the beginning of the course.
- Email/Text Messaging: This group was offered an email and/or text messaging program, in addition to participating in the online module. All students in the program received weekly emails and some elected to receive text messages. Students could reply to these messages asking for assistance or guidance on school-related issues. The email/text messages were meant to serve as a “virtual coach” to students, providing personalized assistance when students asked questions or sought help.
- Coaching: The third group was provided with a one-on-one coach (an older undergraduate student). Students in this group were assigned to coaches who were available to meet with them in person or via Skype/phone. Coaches developed personal relationships with students and provided personalized and proactive attention.
- Control group: The students in the control group received a personality test at the beginning of the semester and were sent the results via email.
Results and policy lessons
In-person coaching substantially improved student outcomes, while the lower-cost, technology-based interventions had no detectable impact. The study suggests the importance of pro-active, individualized engagement in student coaching programs.
In-person coaching increased grades by approximately 5 percentage points (a 7 percent increase from a base of 69 points1 ) and increased GPA by 0.35 grade points (a 14 percent increase from a base of 2.38). Improvements in grades over the winter semester were larger than during the fall semester, indicating that the impact of the coaching grew over time. Coaching also reduced extremely low grades. Students who received coaching were 8 percentage points less likely to earn a grade below 60 percent – a D or below (a 40 percent decrease from a base of 21.6 percent). By contrast, the researchers found no impact of either the online exercise or the text messaging intervention.
A common element across numerous successful coaching programs is regular student-coach interaction, either through mandatory meetings or regular proactive initiation of contact by coaches. In this study, coaches actively initiated contact with students and built trust by providing personalized support. By contrast, the text messaging intervention did not initiate contact with individual students to specifically ask how they were doing or provide an opportunity to build a relationship. In ongoing work, the researchers are testing a virtual coaching model that seeks to combine the proactive, personalized outreach of the coaching model with the lower-cost technology enabled platform of the text messaging intervention.