PDF version

Structured Study Time, Self-Efficacy, and Tutoring

Location: Online through a Massive Open Online Course (MOOC)
Sample: 19,694 online course participants
Timeline:
2014
Research Initiative: Post-Primary Education Initiative
Target Group: 
Students
Outcome of Interest: 
Student learning
Intervention Type: 
Digital and mobile
Information
AEA RCT Registration Number: 
Partners:

Massive online courses have the potential to make quality higher education accessible to a much larger public, but they have been plagued by low retention rates. In this study, researchers evaluated scalable methods to improve student retention and performance in an online course, with the goal of improving meaningful access to this resource.

Policy Issue 

Massive Open Online Courses (MOOCs) have expanded in number and popularity but have exhibited relatively low retention and completion rates. MOOCs are potentially at a disadvantage compared to in-person instruction in several ways. First, standard education provides a structured study environment (classes, assignments, etc.) which most students may need to successfully complete a course. Second, standard in-person education provides one-on-one interactions with an instructor who may be able to spot an individual’s problems or questions and address them before she gets completely lost. Third, students can come from all over the world and those students who are far away or do not share the same language may not have the confidence that they really should succeed. However, there is little evidence on scalable interventions designed to address these barriers and whether they can lead to increased retention, completion, and learning. Can blocking out regular study time to interact with online courseware yield better retention and performance? Could extra tutoring from a teaching assistant lead to learning gains? Could encouraging messages course boost marginalized groups’ performance?

Context of the Evaluation 

Researchers ran their evaluation as part of the edX course “The Challenges of Global Poverty” in spring 2014. This is a course that the researchers ran once before in 2013 and provides an introduction to the main topics in global poverty as conceptualized by leading economists and political scientists. Anyone can access the course for free online. The language of the lectures is English and the course is run over 13 weeks. Participants who complete the course can obtain a verified certificate for a fee. In 2014, nearly 20,000 students registered for the course. Of these, 24 percent were active, 6 percent achieved a grade of at least 50 percent, and 6 percent were active on forums.

Photo: Shutterstock.com

Details of the Intervention 

Researchers conducted a series of randomized evaluations to measure the impact of three sets of interventions on online course retention, interaction, completion, and exam grades.

Structured study time: A randomly chosen subset of students had the option to commit to a regular study time. Students who opted in were required to record the time or times that they planned to dedicate to the course each week. Researchers also tested the impact of various enforcement mechanisms to help students commit to studying. One enforcement mechanism was a message sent to a random subset of students stating that the course staff can monitor whether students were using their designated time to study by looking at the timestamps of when students log in and log out. The second was a set of email reminders sent either one-third of the way through the course, two-thirds of the way through the course, or both. These reminders encouraged students to stick to their committed study time and indicated how closely they had been adhering to that time.

Self-efficacy messages: Self-efficacy refers to someone’s belief that they have the capacity to act in ways that achieve specific performance goals. Sharing information with students as to who performs well in online courses is thought to boost the effort of students and their expectations of what they can achieve. The course entrance survey included self-efficacy messages that provided factual information on who did well in the course the previous year. Students were randomly allocated to see either no message or one of three messages: (1) a generic message (2) a message related to females performing well in the course, (3) a message related to non-native English speakers performing well in the course.

Tutoring: All students that enrolled in the course were offered the opportunity to enter a lottery for tutoring. Of those that signed up, 500 were randomly selected to receive tutoring with a group of 20 other students. Tutoring services consisted of weekly online group review sessions, availability for individual questions over email (on assignments or on lectures) on a weekly basis, and a final exam group review session. The tutor played the role that teaching assistants play in in-person education. Researchers monitored the level of engagement between tutors and tutees, as well as the effect of having access to a tutor on eventual performance.

Results and Policy Lessons 

There was no significant impact of assignment of regular study time on various measures of engagement with the course, including hours spent on course, total activity count, and total sessions. The researcher estimated all three outcomes separately for the full course, the first half only, and the second half only.

For the self-efficacy and tutoring interventions, the researchers examined eight primary outcomes, including course completion, certificate earned, fraction completed, hours spent, overall grade, final exam grade, attempted final exam, and final exam grade conditional on attempting. There was no effect for either intervention on any of the eight outcomes.

The researchers also measured the impact of being assigned to a tutor on intensity of interaction with the staff and the course. Those assigned to a tutoring group were more likely to have any interaction with staff (with tutor or on forum) and any engagement on the forum or interaction with a tutor, compared to those not offered the tutor. However, there was no impact on other measures of engagement, including number of interactions with the staff on the forum, any interaction with staff on the forum, any activity on the forum, and number of posts on the forum.