Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India
We study the impact of a personalized technology-aided after-school instruction program in middle-school grades in urban India using a lottery that provided winners free access to the program. Lottery winners scored 0.37 standard deviations higher in math and 0.23 standard deviations higher in Hindi over just a 4.5-month period. IV estimates suggest that attending the program for 90 days would increase math and Hindi test scores by 0.6 standard deviations and 0.39 standard deviations respectively. We find similar absolute test score gains for all students, but much greater relative gains for academically-weaker students. Our results suggest that well-designed technology-aided instruction programs can sharply improve productivity in delivering education.
Developing countries have made impressive progress in improving school enrollment andcompletion in the last two decades, yet learning levels remain low. In India, for example, over 60 percent of children aged 6-14 cannot read at the second grade level, despite primary school enrollment rates of over 95 percentand substantial increases in education spending.1 In response to this learning crisis, a great deal of funding has been committed to support programs that use information and communications technologies (ICT) in the classroom, promote computer literacy, and develop innovative curricula in electronic formats. However, evidence to date on the impacts and cost-effectiveness of education technology is mixed, with some research suggesting no or negative impacts.2
Context of the evaluation
Researchers partnered with Educational Initiatives to study the “Mindspark” program. Mindspark is a computer-assisted learning (CAL) software that provides students with personalized instruction. Mindspark uses a set of games, videos, and activities that pull from a database of over 45,000 questions to test students and provide explanations and feedback. A key feature of the Mindspark program is its ability to use data to identify the learning level of every student, deliver customized content targeted at this level, and dynamically adjust to the student’s progress. Mindspark can be delivered through desktop computers, laptops, and tablets, and it can be implemented online, in school classrooms, or in after-school programs.
Researchers evaluated a version of Mindspark delivered through three stand-alone centers in Delhi, and included 619 students—mostly between grades 6 and 9—from government-run secondary schools in low-income neighborhoods. This version of the program provided students with 45 minutes of the CAL software and 45 minutes of instructor-led small group instruction. Children signed up for the program by selecting a 90-minute slot, which included about 15 students and ran six days a week. Typically, parents pay INR 200 (US$3) per month to send their children to the program.
Details of the intervention
Researchers conducted a randomized evaluation to test the impact of the Mindspark program on student test scores in mathematics and Hindi. Among the 619 students recruited for participation, around half were offered a voucher for free attendance at a Mindspark center starting in late 2015. During the self-driven learning period, each child was assigned to a computer with software that provided customized activities in math, Hindi and English. During the small group instruction, teaching assistants covered core concepts that were not customized to each student’s learning level, and provided time for students to complete homework assignments. To measure the impact of the program on student achievement, researchers tested students in math and Hindi at the beginning and end of the program—a gap of about 4.5 months—at the Mindspark centers.
Results and policy lessons
Among students offered the free voucher, 58 percent attended the Mindspark program. The Mindspark program increased learning levels across all groups of students and was cost-effective compared to other instruction types.
Learning Gaps: Prior to the program, students in the sample were on average several grade levels behind, and this gap increased for higher grades. The average student in grade 6 was an estimated 2.5 years behind in math; by grade 9, this deficit increased to 4.5 years. The bottom third of students in the comparison group made nearly no learning improvements in the 2015-2016 school year.
Center Test Scores: The program improved performance in both math and Hindi across multiple grade levels. Students offered a voucher scored 0.36 standard deviations higher in math, improving by twice as much as students in the comparison group. Students who received the voucher also scored 0.22 standard deviations higher in Hindi, improving by 2.5 times as much as students in the comparison group. Researchers estimate that attending Mindspark for 90 days would raise math and Hindi test scores by 0.59 and 0.36 standard deviations, respectively. Impacts did not vary significantly by level of initial achievement, gender or wealth, implying that the program was equally effective in teaching all students. However, the relative impact was much greater for weaker students, since their rate of progress under standard classroom settings was close to zero.
School Test Scores: Researchers also used administrative data on student tests scores at their schools to measure their performance in material at their official grade level. In Hindi, for which students were initially less behind and Mindspark presented material at students’ official grade levels, the program had positive effects. In math, the program had no effect. Researchers hypothesized that since students were usually several years behind in math, the school exams would still be beyond their learning level, even if they had made some improvements.
Cost Effectiveness: Mindspark was cost effective in comparison to other common alternatives. The per-student monthly cost of the program was around INR 1000 (around US$15) per month, compared to a cost of around INR 1500 (US$22) per month in spending per student at the public schools from where the students came in Delhi. Researchers expect that the program cost per student would decrease to under US$2 if it were scaled up to a larger number of students.
Use of Results:These evaluation results are being used by the program implementers, Educational Initiatives, to set up potential scale-ups of the intervention in government schools in multiple states. These scale-ups, which are still at the design stage, are expected to integrate Mindspark into classroom settings within government schools.
Muralidharan, Karthik, Abhijeet Singh and Alejandro J. Ganimian. 2019. "Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India." American Economic Review 109 (4): 1426-60.