Isolating the effects of personalization in a targeted instruction program in India
Are the effects of targeted instruction programs—such as Teaching at the Right Level or Computer Adaptive Learning (CAL) software—indeed driven by personalization? In partnership with an education assessment firm, researchers conducted a randomized evaluation to isolate the impact of computer-based individualized instruction in India. After nine months, lower-performing students in grades six to eight learned 22 percent of a standard deviation more in math if offered materials that matched their learning levels. For the average student, individualized instruction did not affect math learning. The results suggest that individualized instruction is most beneficial for low performers.
Policy issue
As in other low-income countries, many Indian students lag behind expected grade-level performance,1 and this gap between students’ expected and actual performance tends to widen starting in primary school. Also, learning levels often vary dramatically between students, even in the same classroom, and teachers can find it challenging to provide targeted instruction that matches the individual needs of each student. School closures during the Covid-19 pandemic exacerbated these issues, which many experts describe as an acute learning crisis.
Meanwhile, in many low- and middle-income countries, tailoring instruction to students’ learning levels has emerged as a promising strategy to close learning gaps—whether through teacher-led interventions that promote differentiated instruction (e.g., “Teaching at the Right Level”) or computer-adaptive learning software (e.g., “Mindspark”). However, as such programs usually bundle different components, and since they often also imply additional study time for students, prior studies have been unable to tie their effects on learning to individualized instruction. With this in mind, what are the isolated impacts of personalized instruction on student learning?
Context of the evaluation
Rajasthan is a populous Indian state, with three-quarters of its residents living in rural areas. Its average education levels roughly mirror those in the rest of India: for example, a similar percentage of sixth graders can subtract a two-digit number by another two-digit number (25 percent in Rajasthan, compared to 24 percent across rural India).2
Within rural areas across India, including Rajasthan, the Indian Ministry of Human Resource Development establishes “model schools” to promote education in disadvantaged rural areas. They teach students in grades six to twelve, English is the language of instruction, and they follow a national curriculum instead of a state curriculum. In 2017, when the evaluation took place, there were 134 model schools in Rajasthan; 64 percent of the state’s district sub-divisions had at least one such school.
Model schools admit students through a selective entrance exam. Thus, their students have higher average achievement compared to traditional government schools. Nevertheless, in the model schools sampled in this evaluation, there are large gaps between expected and actual performance at each grade level. For example, in math, the average sixth grader is two grade levels behind national standards.
A leading assessment firm, Educational Initiatives (EI), developed a computer-adaptive math learning software called Mindspark, over a ten-year period. It has been used by over 500,000 students. When students first log into Mindspark, they take a diagnostic test, which identifies what they know and can do and the areas where they can improve. Then, the software presents them with exercises appropriate for their preparation level based on the diagnostic test. The difficulty and topic covered by subsequent exercises dynamically adjust to each student’s progress. Other research has documented the positive impacts of this software on student learning, both in after-school centers in Delhi and in Rajasthan’s government schools. However, the software comprises many different features, and it is unclear to what extent individualized instruction drives its effectiveness.
Details of the intervention
In partnership with EI and over a period of nine months in 2017, the researchers evaluated the impact of the individualization component of Mindspark on learning outcomes for 1,528 students in grades six to eight from fifteen model schools across Rajasthan.
The researchers randomly assigned students into two groups:
- Comparison group: These students used a version of Mindspark without the individualized instruction component; in other words, they received instruction that matched their enrolled grade level.
- Intervention group: These students used a version of Mindspark with the individualized instruction component.
Students used Mindspark in school during their regular math classes over a period of nine months.
The researchers’ randomization approach was such that within each grade level classroom, there was a random mix of students assigned to the comparison or intervention group. Because the only thing distinguishing those students who received individualized instruction from others that did not is chance, any differences in learning between them can be attributed to personalization.
Results and policy lessons
There were no meaningful differences in learning outcomes for the average student who received the fully personalized version of Mindspark relative to those who used the grade-level version of the software. However, access to personalized instruction significantly improved the math achievement of those students who were initially low-performing. Those students in the intervention group who started the program in the bottom 25 percent of performers in their grade saw a positive effect of 0.22 standard deviations in their math learning.
These gains were concentrated in the topics and skills that featured more frequently in the software. They were driven by students’ improved ability concerning number concepts and operations (as opposed to geometry and measurement or data-related concepts). These positive impacts were also driven by greater improvements in applied math learning and problem-solving (as opposed to rote learning). The researchers measured students’ math ability with paper-based assessments (not on computers or tablets); therefore, the study’s measure of math achievement is not confounded with students’ ability to engage with technology.
The typical student used the software for 329 minutes across the nine months of the study. This usage level is low compared to that of an out-of-school version of Mindspark from another evaluation. There was also considerable variation in the level of interaction from week to week. The researchers suggested that this may have been related to challenges the schools faced regarding classroom or computer availability and teacher coordination.
Overall, these results suggest that low-performing students may stand to benefit the most from individualized instruction, at least in the context of computer-adaptive learning software. This is the first study to document this relationship through an experimental evaluation in a developing country. More broadly, studies like this also indicate the potential for individualized learning and targeted instruction—either with or without education technologies like Mindspark—to help lead to improved educational outcomes for students who are low-performing.
1 de Barros, Andreas, and Alejandro J. Ganimian. “The Foundational Math Skills of Indian Children.” Economics of Education Review 92 (2023): 102336. https://doi.org/10.1016/j.econedurev.2022.102336.
2 ASER (2019). Annual Status of Education Report 2018 (Rural). Provisional Report, Pratham, New Delhi.