The Impact of Online Lesson Plans on Student Learning in the United States
More than 90 percent of middle and high school teachers report using the internet to source instructional materials, but there is a dearth of rigorous evidence on the effects of providing teachers with access to online materials. Researchers evaluated the impact of providing teachers with access to high-quality online lessons and implementation support on student achievement in mathematics. Providing teachers with access to the lessons, reminders, and implementation support significantly increased student achievement. Researchers also found suggestive evidence that providing access to the lessons alone had a positive effect on math scores.
Policy issue
Teachers have a sizable effect on student achievement as measured by student test scores. These effects persist and are predictive of long-term outcomes including graduation rates, college attendance, and subsequent earnings. National efforts to improve education quality such as No Child Left Behind and The Every Student Succeeds Act have included specific policies aimed at improving teacher effectiveness.
Teaching is a multitask job that requires skill to not only plan lessons but also deliver them effectively while managing a classroom. To meet these distinct obligations, teachers are increasingly turning to the internet for instructional materials: over 90 percent of middle and high school teachers report accessing lessons online, and teachers in schools with a larger proportion of low-income students are more likely to use online resources relative to their peers.
The rapid growth of online lesson plans presents both an opportunity and a challenge for policymakers. Using high-quality, off-the-shelf lessons can provide a minimum level of lesson quality and free up teacher time that would have been dedicated to lesson planning. However, accessing high-quality lessons often requires funding and search time, and there is a dearth of rigorous evidence on the effects of providing teachers with access to online materials.
This study investigates the impact of providing teachers with free access to an inquiry-based, online math curriculum and implementation support on student learning outcomes in three Virginia school divisions.
Context of the evaluation
Mathalicious (now Citizen Math) is an online math curriculum for grades 6–12 with activities that encourage students to think critically about the way the world works and contextualize lessons in real-world situations. These off-the-shelf lessons are informed by education theory on inquiry-based instruction to help students retain new concepts and promote classroom engagement and deep understanding rather than rote memorization.
Mathalicious and the research team partnered with three school divisions near Richmond, Virginia: Chesterfield County Public Schools, Henrico County Public Schools, and Hanover County Public Schools. Each of these divisions had previously sought to buy Mathalicious licenses for their teachers and were offered additional licenses free of charge in exchange for participation in the evaluation. For the 2013–2014 school year, the three participating districts had average or above average passing rates on math assessments.1
Details of the intervention
Researchers conducted a randomized evaluation to test the impact of providing teachers with access to high-quality, off-the-shelf online lessons on student achievement in mathematics. Three-hundred sixty-three math teachers serving 27,613 middle school students (grades 6–8) across three school divisions were assigned to one of two intervention groups or a control group. School divisions were able to request that a subset of teachers be guaranteed to receive a license and randomly assigned to one of the intervention groups.
Teachers in the first intervention group received access to Mathalicious lesson plans and ongoing support through Project Groundswell, an online platform where teachers could talk about lesson implementation with Mathalicious developers and other teachers through private discussion boards. Teachers in this group were invited to an in-person kickoff event where Mathalicious personnel reviewed online materials and provided support logging into the platform. Teachers were also encouraged to use the lessons via email reminders throughout the semester and were provided access to webinars.
Teachers in the second intervention group received a Mathalicious license and had access to the basic technical support features available to all users. They did not receive email reminders and were not invited to the discussion boards.
Teachers in the control group did not receive a license or other additional support.
To measure students’ math achievement, researchers used student-level data from the math portion of the Virginia Standards of Learning (SoL) assessment for each division for the academic years 2012–2013 and 2013–2014. To observe the effect of the provision of lesson plans on student attitudes and perceptions, researchers surveyed students in the middle and at the end of the academic year.
Results and policy lessons
Providing teachers with access to both Mathalicious lessons and ongoing support led to a significant increase in student test scores. Researchers also found suggestive evidence that providing access to Mathalicious lessons alone had a positive effect on math scores.
Student achievement: Students of teachers who received the full treatment (access to the lessons and extra supports) earned test scores that were .086 standard deviations higher than those of the control group. Students of teachers who received access to the lessons alone earned test scores that were .060 standard deviations higher than those of the control group, though results for the lessons-only group were not consistently significant across different statistical models.
Researchers also suggest that off-the-shelf lessons might be most effective for lower-performing teachers and first- or second-year teachers, finding an inverse relationship between classroom quality and lesson impact.
Student attitudes and perceptions: Students of teachers in the full treatment group more frequently reported feeling that math had real life applications, that their teachers promoted deep conceptual understanding, and that their teachers spent more one-on-one time with students than did students of teachers in the control group. Student perceptions of teachers who received only the license were not significantly different from those of students whose teachers did not receive a license.
Cost effectiveness: Researchers estimated that the full cost of the program (lessons and supports) was $431 per teacher and that the test score effect of .09 standard deviations would generate roughly $4,500 per student and $405,000 per teacher in present value of students’ future earnings.
Policy Implications: These findings show that providing teachers with access to high-quality, off-the-shelf lessons on the internet, in combination with supportive services, may be a viable strategy to improve student math scores.
Jackson, Kirabo and Alexey Makarin. 2018. “Can Online Off-the-Shelf Lessons Improve Student Outcomes? Evidence from a Field Experiment.” American Economic Journal: Economic Policy 10(3): 226-254. doi: https://doi.org/10.1257/pol.20170211