The Medium-Term Educational Effects of Short-Term Literacy Tutoring in Chile
A growing body of literature has shown that remedial education programs using non-teaching staff such as parents and volunteers can improve learning outcomes for students at low-performing schools. Researchers conducted a randomized evaluation to gauge whether a short-term reading tutoring program could improve a range of student educational outcomes in the short- and medium-term (up to eight years after the program). They found that the program increased language test scores in the short term and improved several outcomes in the medium term, including reducing students’ probability of dropout, increasing their probability of graduating on time from primary and secondary school, and improving attendance, grades and test scores. These effects were concentrated among students with a higher risk of dropping out.
Policy issue
Low levels of learning, if not addressed early in a child’s school career, can worsen over time in terms of both cognitive and non-cognitive skills. A growing body of literature has shown that remedial education programs using non-teaching staff such as parents and volunteers can improve learning outcomes. Furthermore, these programs may have different effects on students who have different levels of educational and social opportunities. However, there is relatively less research into the effect that these short-term, volunteer-based programs have on educational outcomes in the medium term (for this study, up to eight years after the program).
Context of the evaluation
Although Chile has a well-developed public primary and secondary education system, most of the students who go on to tertiary institutions come from the highest income bracket in the country. In 2009, one sixth of college-aged youth from the poorest tenth of the population were enrolled in tertiary education, compared to more than half of the richest tenth.
One substantial barrier to seeking tertiary education for those in lower income categories is the high tuition costs for post-secondary education: for households in the lowest income decile, one year of public university tuition cost 141 percent of their average annual income in 2003. Moreover, financial aid is merit-based, so for poor students, good grades and high scores on the college entrance exam are especially important for financing their education. Learning difficulties early in their educational career, if not addressed, could translate into significant disadvantages as they pursue higher education.
Details of the intervention
Researchers conducted a randomized evaluation to test the impact of short-term tutoring programs on educational outcomes including test scores, daily attendance, and dropout in the following years. In order to improve reading comprehension and attitudes toward reading in 4th grade students from vulnerable schools, the Chilean Ministry of Education partnered with the Fundación para la Superación de la Pobreza (FSP) in 2010 to implement Servicio País en Educación (SPE). The Chilean Ministry of Education selected 85 public and voucher schools in ten low-to-middle income counties in the Santiago and Biobio regions that had scored below the average on the language portion of Chile’s national standardized test in the previous year. From these 85 schools, 45 schools (with 2,752 students) were randomly selected to receive SPE and the remaining 40 schools (with 3,377 students) served as comparison schools. Students at these schools had a household monthly income of approximately $500 per month, and 62 percent of the students were classified as vulnerable.
SPE brought college student volunteers into primary schools to conduct fifteen weekly 90-minute reading sessions during school hours with 4th grade students split into small groups of five or six students. This program took place over three months. The sessions included a mix of traditional stories and instructional texts appropriate for the students’ age. Volunteers were managed by a paid employee of FSP stationed permanently in each program school to support implementation and provide assistance on effective teaching techniques.
Tutors were volunteer college students who typically did not have any particular training in teaching. They were from the same communities as the students and received one day of training before the program began.
The implementation of the program was monitored through random visits by research staff to observe tutoring sessions. Researchers also surveyed students, asking them to evaluate their improvements in reading and writing, the quality of the tutors, and their relationships with their tutors, on a 0-3 scale.
In follow-up surveys, students filled out a Reading Comprehension and Texts Production Test (CLPT) to measure skills, and a Taste for Reading (GPL) questionnaire to measure attitudes toward reading. Lastly, researchers conducted medium-term follow-up by collecting data from the Chilean Ministry of Education on students’ enrollment, attendance, and test scores in the 8th and 10th grade.
Results and policy lessons
The program was only slightly associated with higher levels of short-term learning, but substantially increased attendance and grades, and reduced the probability of dropout, years after the program’s completion. These results were found despite the fact that the average school hosted 12 sessions and the average student attended 7.9 out of the intended 15 sessions.
Short-term learning: Researchers found a moderate short-term effect of tutoring programs on student learning. Students who attended schools that were selected to receive the intervention on average performed 0.06 standard deviations better on a composite score of the CLPT test, and 0.11 standard deviations better on its reading comprehension section relative to the comparison group. However, no meaningful change was found in performance on the GPL test, which measured interest in reading.
Dropout and school progression: Students in the intervention group had a 0.8 percentage point lower probability of dropping out by 2011, the year after the intervention took place (when the overall dropout rate was 2 percent), and a 1.8 percentage point lower probability of dropping out by 2017, when they would have been in 11th grade (and the overall dropout rate was 16 percent). These effects are concentrated among those with higher ex-ante probability of dropout (predicted through students’ familial, academic, and socioeconomic backgrounds). These students had a 3.7 percentage point lower chance of dropout by 2017, seven years after receiving the intervention. Students in the intervention group also had a 4.5 percentage point higher probability of graduating on time from primary school (with a sample-wide probability of 79 percent) and a 3.2 percentage point higher probability of graduating on time from 11th grade (sample-wide probability of 58 percent), with the effect again concentrated among those at higher risk of dropout.
Attendance: Students in the intervention group had 0.8 percentage point higher primary school attendance (with overall attendance of 89 percent) and 1.1 percentage point higher secondary school attendance (with overall attendance of 82 percent). These effects were also largest among those at highest ex-ante risk of dropout, who had 1.5 percentage point higher primary school attendance and 2.2 percentage point higher secondary school attendance when in the intervention group. Researchers suggest these estimates may be undercounting the true impact on attendance given more students in the comparison group dropped out of school than in the intervention group.
Grades and test scores: Receiving tutoring in one's school improved student grades by 0.09 units relative to the comparison group (with grades determined along a 1-to-7 point scale, with average primary school grades in the full sample of 5.32). However, the tutoring program had no meaningful impact on secondary school grades. Researchers also found small increases in 8th and 10th grade standardized test scores in both math and reading among the intervention group, but these increases were only statistically meaningful among those at highest risk of dropout.
Influence of tutor quality: During the tutoring program, students rated both the quality of instruction and the quality of relationship with their tutor on a 0-to-3 scale. Students who rated the quality of relationship with their tutor higher performed better in both short-term test scores and medium-term educational performance. However, student-assessed quality of instruction had no relationship with their educational outcomes.
Studying medium-term effects of educational interventions is crucial to understanding the cost-effectiveness of public policies. In Chile, researchers found that a tutoring program improved educational outcomes in both the short-term and the medium-term. Producing 1.39 learning-adjusted years of schooling per US$100 spent (the approximate program cost per student), this program is in the mid-range of cost-effectiveness compared to existing programs once its medium-term effects were considered. Such results suggest that even temporary and narrowly focused educational interventions could have the potential to influence longer-term outcomes.
Cabezas, Verónica, José Cuesta, and Francisco Gallego. “Does Short-Term School Tutoring have Medium-Term Effects? Experimental Evidence from Chile.” Working Paper, September 2021.