Scripted Teacher Lessons on Student Learning in Chile
Higher quality education, particularly for children from lower socioeconomic backgrounds, is critical for offering students equal opportunities and preventing the continued transmission of poverty. Researchers partnered with the Chilean Ministry of Education to evaluate the impact of pre-packaged classroom materials and standardized directions for teachers on teaching quality and student learning. The program improved student learning outcomes substantially, with equal benefits for both boys and girls, and particularly positive impacts for children from higher-income backgrounds.
Around the world, an estimated 356 million children live in poverty, with low school quality contributing to roughly 250 million children failing to develop basic literacy and numeracy skills.
Higher quality education, particularly for children from lower socioeconomic backgrounds, is
critical for offering students equal opportunities and preventing the continued transmission of
poverty. However, educators often grapple with how to effectively improve school quality.
Teachers can have a large impact on student performance, but existing research has been
unsuccessful in identifying how best to improve teacher quality. Could pre-packaged classroom
materials and standardized directions for teachers help improve teaching quality and thereby
increase student learning?
Context of the evaluation
Over the past 20 years, Chile’s primary school enrollment and retainment rates have increased drastically, approaching near universal access to primary education. Yet in terms of student achievement, Chile underperforms in international assessments when compared to other OECD countries. One major reason for low academic performance is wide-ranging socioeconomic disparities, which in turn are reflected in differences in school type, school location, and school resources.
In 2011, the Chilean Ministry of Education launched Plan Apoyo Compartido (PAC), a targeted educational policy providing technical and pedagogical support to schools historically performing below average in Chile’s standardized Education Quality Measurement System national examinations (SIMCE). As part of PAC, teachers are provided with detailed classroom guides and scripted lecture materials in efforts to standardize pedagogical materials and close academic achievement gaps between the lowest-income student population and the national average.
Details of the intervention
Researchers partnered with the Chilean Ministry of Education to evaluate the impact of PAC on student math, language, and science test scores and teacher-student interactions in underperforming Chilean schools. In doing so, they sought to understand the differential effects of academic outcomes based on students’ gender and socioeconomic status and their schools’ socioeconomic status.
From 2011 to 2012, the PAC program was implemented in six-week cycles and consisted of five key components: (1) standardized planning tools and pedagogical materials for teachers; (2) manuals to guide program implementation across schools; (3) student evaluations to guide teaching and monitor student progress on an ongoing basis; (4) the promotion of class planning and frequent classroom observations to provide teacher feedback; and (5) frequent internal school staff meetings to discuss student progress.
The PAC program was accompanied by the development of two support teams–one internal and one external to each school–which worked closely together. The first team, the Education Leadership Team (ELE), comprised the school principal, the head of the school’s technical and pedagogic office, and two distinguished teachers. The second group, the Team of Technical and Pedagogic Advisors (ATP), consisted of three representatives from the regional Department of Education to offer external support to the ELE teams. Every six to seven weeks, each ATP visited their assigned schools to advise ELE teams on the use of PAC teaching materials, discuss the schools’ strengths and weaknesses, and evaluate students’ test scores and overall progress.
Of 843 eligible public and subsidized private schools that had performed below the national math and language SIMCE average and had at least 20 students per grade level, researchers randomly allocated 648 schools received the PAC program, while the remaining 195 schools formed the comparison group.
Researchers gathered information from SIMCE math, language, and science test scores; the ATPs’ school visits and related findings; the Classroom Assessment Scoring System (CLASS)’s scores of teacher-student interaction quality; and two Ministry-designed instruments intended to evaluate the extent to which teachers and schools integrated the program into their classrooms.
For the CLASS measurements, in 2012, 137 PAC and comparison schools were randomly invited to have their grade 4 classrooms videotaped for four full lessons. Trained coders watched and analyzed the videotapes and assigned each recording a score based on the quality of teacher-student interactions.
In terms of the two Ministry-designed instruments, the first instrument involved observing and coding class videotapes, similar to CLASS protocol, based on key aspects of the PAC program. The second instrument involved interviewing the head of each school’s technical and pedagogic office (JUTP) and a set of teachers to assess the extent of PAC implementation.
Results and policy lessons
Overall, Chile’s PAC program improved student learning outcomes substantially, with equal benefits for both boys and girls, and particularly positive impacts for children from higher-income backgrounds. Researchers also found that the quality of teacher-student interactions was positively correlated with the performance of low-income students, even though the PAC program did not affect these interactions.
Student Learning: In 2011, students’ reading test scores improved by 0.10 standard deviations in PAC schools. By 2012, in PAC schools, students’ reading, math, and science test scores improved between 0.09 and 0.13 standard deviations relative to comparison schools. The effects were larger in 2012 than 2011, indicating that the program matured over time and was better embedded in the implementing schools. Overall, the program resulted in persistent and improved academic impacts over time.
Results based on students’ gender and family income indicated the same effects for boys and girls, and positive and significant effects particularly for students from relatively higher-income families in PAC schools. In fact, the academic impacts for students from higher-income backgrounds was approximately twice those of the students from low-income families: reading scores improved by 0.17 standard deviations for higher-income students and only 0.09 standard deviations for lower-income students, while math scores improved by 0.14 and 0.07 standard deviations respectively. Furthermore, the PAC program was most successful in schools with higher socioeconomic statuses, with test scores improving by 0.20 standard deviations among students from more well-off PAC schools.
Teacher-Student Interactions: Researchers found that the CLASS observation process improved students’ reading, math, and science scores by 0.23, 0.18, and 0.21 standard deviations respectively. Researchers hypothesize that CLASS observations improved test scores because teachers improved their productivity and behavior when being monitored. Researchers also found that CLASS scores positively correlated with students’ performance, and particularly for those from lower-income backgrounds, because better student-teacher interactions improve student learning. These results are potentially consistent with the finding that teachers have a causal impact on student performance. For example, results imply that moving a lower-income student from the bottom 2 percent of teachers to the top 2 percent can improve low-income students’ academic outcomes by between 0.6 and 0.8 standard deviations. That being said, PAC did not cause significant improvements in CLASS scores on teacher-student interaction quality, which may explain why low-income students were more modestly impacted by PAC.
Program Fidelity: Compared to non-PAC schools, PAC schools were more likely to have their annual planning performed by either PAC authorities or the ELE team and less likely to leave curriculum design to teachers. Relative to non-PAC schools, PAC schools are also 23 percentage points more likely to use scripted workbooks in class.
Bassi, Marina, Costas Meghir, and Ana Reynoso. “Education Quality and Teaching Practices.” The Economic Journal 130: 1937-1965. doi: https://doi.org/10.1093/ej/ueaa022.