The Impact of an Online Math Learning Platform on Test Scores and Attitudes Towards Math in Brazil
Little rigorous evidence exists regarding the impact of technological approaches to improving education quality. Researchers are conducting a randomized evaluation to measure the impacts of the Khan Academy online math platform on students’ test scores and attitudes towards math in Brazilian public schools.
While much progress has been made in increasing access to primary education over the past few decades, improving education quality and learning outcomes remains a key issue for policymakers. To address this issue, governments, civil society organizations, and private sector actors have adopted programs that aim to use technology to improve education quality. Examples of such policies include expanding access to computers and the Internet in public schools and the development of computer assisted learning programs for students. However, previous research has found that simply expanding access to computers in school does not lead to positive impacts on test scores or other education outcomes such as dropout and attendance rates. Interventions that actively encourage technology use have found more positive results, but little rigorous evidence exists on the impact of using interactive online platforms on education outcomes.
Context of the evaluation
A 2015 report by the OECD found that Brazilian students scored an average of 377 on the Program for International Student Assessment (PISA) mathematics examinations, significantly below the OECD average of 490. However, according to the Brazilian Education Census, approximately 53 percent of public schools in Brazil in 2014 (corresponding to 81 percent of students) had computers with internet access for students to use. Given this relatively high prevalence of computers in public schools, encouraging the use of online tools that promote better learning processes and complement the expansion of computer access in schools may be an effective strategy to improve learning outcomes in Brazil.
One example of an online tool aiming to increase education quality is the Khan Academy platform, which offers instructional videos and personalized exercises for a variety of subjects. Founded in 2005, Khan Academy has reached over ten million people worldwide with its online content. The math platform complements school curricula by providing individual students with re-watchable video lessons, personalized interactive online exercises, and detailed automated reports on student progress and performance. In 2014, the Lemann Foundation translated and adapted a Portuguese version of Khan Academy’s math platform and began promoting its use in Brazilian public schools.
Details of the intervention
In partnership with the Lemann Foundation, researchers are conducting a randomized evaluation to measure the impacts of the Khan Academy platform on primary school students’ math scores and attitudes towards math in Brazil. The evaluation focuses on 5th and 9th grade classrooms of 166 public primary schools in five cities across the country that voluntarily agreed to participate in the program. The research team randomly assigned half of the classrooms to receive the Khan Academy program. The other half of classrooms did not receive the program, and served as the comparison group. The Lemann Foundation provided computers for all schools in the evaluation that had fewer than one computer for every two students, including in comparison schools.
For the 2017 academic year, classrooms in the Khan Academy program will replace some traditional math classes with Khan Academy’s personalized instructional videos and exercises under teacher supervision. Teachers in these classrooms will receive a one-day training session to learn to use the platform as well as a series of professional development visits, and each student will receive a unique log-in for the program. Classrooms in the comparison group will receive standard math instruction without the Khan Academy Program.
To measure impacts on student learning, the research team will collect math test scores from the national Prova Brasil examinations at the end of the school year for all schools. In addition, they will survey students about their attitudes toward math before and after the first year of implementation to measure the program’s impact on students’ attitudes.
Results and policy lessons
Study ongoing; results forthcoming.
Ferman, Bruno, Lucas Finamor, and Lycia Lima. "Are Public Schools Ready to Integrate Math Classes with Khan Academy." Working Paper, June 2019.