Increasing Student Learning through a Phone-based Program during Covid-19 in Botswana

Researchers:
Noam Angrist
Caton Brewster
Moitshepi Matsheng
Fieldwork by:
Location:
Botswana
Sample:
4,550 households
Chronologie:
2020 - 2021
Target group:
  • Children
  • Students
Outcome of interest:
  • Student learning
Intervention type:
  • Nudges and reminders
  • Tracking and remedial education
AEA RCT registration number:
AEARCTR-0006044

Partners:

School closures—due to viruses, teacher strikes, natural disasters, and more—can  result in large learning losses for students. In these situations, low-income families face additional challenges, such as lack of internet access, raising the need for low-technology, inexpensive solutions to support children’s learning. In Botswana, researchers were able to rapidly evaluate a phone-based remote learning program aimed at keeping children engaged with learning during the Covid-19 pandemic. Students who received weekly math exercises through SMS messages, as well those who received complementary phone calls to go over the content, increased their math skills after four weeks. A second phase of the study is ongoing, with the goal of measuring the impact of tailored instruction, through customized text messages, on student’s learning.

Policy issue

As a result of the Covid-19 pandemic, education systems around the world have halted in-person activities, leaving over one billion students out of schools.1 Evidence from both high- and low-income countries suggests that school closures–which also occur during summer holidays, public health crises, and weather shocks–often result in large learning losses for students.2 With children unable to attend school, existing educational inequalities become stark, with high-income families having access to alternative sources of instruction—such as books, computers, internet, and smart phones—while many low-income families do not.3 To address the exacerbated learning crisis, policymakers across the globe have been looking for rapid, cost-effective, low-tech solutions, such as phone and simple SMS messages, to reduce learning losses at scale. However, can these types of interventions actually impact children’s learning? If so, how cost-effective and scalable are they?

Context of the evaluation

In Botswana, where this evaluation took place, primary school enrollment is around 90 percent. However, learning levels are low and stagnating.4 A recent survey of basic literacy and numeracy showed that 32 percent of students in grade 5 could not do subtraction and 44 percent could not read a story in English. In an effort to improve learning levels in Botswana, a number of organizations, led by the Botswana Ministry of Basic Education, have been supporting the scale-up of an education program called Teaching at the Right Level (TaRL) to all primary schools. This program, which has been found to be effective in several contexts, involves regrouping students by their actual learning level rather than the grade-level curricula. By March 2020, TaRL had reached over 15 percent of primary schools in the country. 

Although the direct effects of the Covid-19 pandemic have been minimal in Botswana–with limited reported cases and deaths through July 2020–pre-emptive social distancing measures have been adopted by the government, which severely impacted educational and social services systems. Schools across the country were closed from March to June 2020. Shortly after reopening, a new wave of Covid-19 cases prompted a subsequent school closure. Similar waves of reopening and closing are expected in the coming months. As of July, schools were running a double-shift system—half of the students attend school in the morning, while the other half attend in the afternoon–drastically reducing the total time in school for each student. Despite the government’s efforts to keep students engaged—through learning programs on national television and radio stations—families have expressed demands for additional remote educational activities for their children.

young-boy-using-cell-phone
Photo: Shutterstock.com

Details of the intervention

In partnership with the NGO Young 1ove and the Ministry of Basic Education in Botswana, researchers conducted a randomized evaluation to assess the effectiveness of a low-tech intervention on improving students’ learning during school closure. Given the Covid-19 pandemic’s demand for real-time policy response, the research team was able to conduct the evaluation and generate rigorous results in around two months. 

Using phone numbers collected from primary schools for students in grades 3-5, the research team identified 4,550 households who were willing to receive remote learning support via mobile phone. Each household was randomly assigned to one of three groups:

  1. Weekly SMS message only: Households in this group received a weekly SMS containing several simple math problems (“problem of the week”). 
  2. Weekly SMS message + phone call: In addition to the SMS messages, households in this group also received a 15-20-minute phone call, in which facilitators answered any questions related to the “problem of the week” and provided additional practice questions. The calls required both parents and children to be present, as a way to encourage engagement and maintain accountability.
  3. Comparison group: Households in this group did not receive any type of phone learning support.

After four weeks, facilitators conducted phone surveys with parents and students from half of all households to collect information on their engagement in the educational activities, parents’ perceptions of their child’s learning, and student learning outcomes, as measured through standardized test scores. 

After week four, researchers randomly selected a subset of the households from the SMS only and SMS + phone groups to receive an additional support program: targeted instruction. In this subsequent intervention, which is ongoing, students receive text messages tailored to their individual learning levels over a period of four weeks. For example, students who know addition will receive subtraction problems to push them to a higher level of learning, whereas students who know multiplication will be sent division problems. Through weekly phone surveys, facilitators will collect students’ answers to the “problem of the week,” which will be used to target subsequent weekly SMS messages. At the end of the tenth week, the survey team will conduct a second round of surveying and data collection to evaluate these additional components of the intervention.

Results and policy lessons

Both the SMS-only and SMS + phone calls interventions had positive impacts on students’ learning, particularly for lower-performing students. The programs also led to higher parental engagement in educational activities, and improved the accuracy of parents’ perceptions of their child’s learning level.

Both SMS-only and SMS + phone calls had large impacts on students’ learning: After the first four weeks, students in households that received weekly text messages with math problems showed a 13 percent improvement on learning outcomes (a 0.16 standard deviation increase in scores from an average of 1.73 in the comparison group). Meanwhile, students who received additional support through phone calls experienced a 24 percent increase in their learning outcomes (a 0.29 standard deviation increase). This translates into a reduction in innumeracy of 34 percent among the SMS-only group, and 52 percent for the SMS + phone calls group (only 19 and 14 percent of students were innumerate in the two groups after the evaluation, respectively, compared to 29 percent in the comparison group).

The programs were also effective in closing previous learning gaps: When comparing students who had previously received learning support in math–measured by participation in the Teaching at the Right Level program–with those who had not, researchers found that the second group experienced higher gains in learning from the remote phone interventions after four weeks. A similar trend was observed when comparing girls and boys: boys, who started with lower numeracy levels, benefitted more from the SMS and phone support than girls. This suggests that low-tech learning interventions have the potential to close gaps in learning between higher- and lower-performing students. It also shows how targeting the program to students who need it the most might be important to enable its scale-up. 

Parents were more engaged on their children’s learning: In households that received SMS-only and SMS + phone calls, parents were 7 and 12 percentage points more likely, respectively, to spend time engaging with their child on educational activities after four weeks. This represents a 10 to 18 percent increase in parental engagement, from 67 percent in the comparison group. Additionally, parents in treated households improved the accuracy of their perceptions of their children’s learning levels. Parents in the SMS-only group were 7 percentage points more likely to correctly estimate their child’s learning level, while those in the SMS + phone calls group improved their perception by 11 percentage points–a 23 and 35 percent increase from a comparison group average of 31 percent, respectively. Accurate perceptions are important as they enable families to target learning activities to their child’s level more effectively. The results show, however, how hard it can be for parents to know the “right level” of their children learning, which emphasize the need for targeted instruction–part of the second phase of the study.

Both interventions were relatively cost-effective and easy to scale: Researchers estimate that the total costs of sending weekly SMS messages equated to US$2.13 per child reached, while SMS + phone calls costed US$14 per child reached. Given the learning effects of both programs, this corresponds to US$13.3 per standard deviation gain in learning for the SMS-only group and US$48.28 per standard deviation gain in learning for the SMS + phone calls group. It is important to highlight that a portion of these costs are fixed costs to set up the program, e.g. costs to collect phone numbers, set up new infrastructure, conduct training, and collect routine monitoring data, suggesting likely even lower costs when scaling up. Although more expensive, the SMS + phone calls intervention seemed more effective in promoting student learning. Together, these results show that low-technology interventions, such as SMS messages and phone calls, can be a cost-effective and scalable way to prevent learning losses during school closures, especially among low-performing students. 

The results have been incorporated directly into day-to-day programming by Young 1ove and the Ministry of Basic Education in Botswana is exploring a national scale-up. In addition, the research team is exploring replication trials in 3-5 countries in the coming months, including with NGOs, governments and multilateral institutions such as the World Bank.

A second round of data collection will be conducted by the end of the tenth week. Analysis is ongoing, and results will be forthcoming.

Angrist, Noam, Peter Bergman, Caton Brewster, and Moitshepi Matsheng. "Stemming Learning Loss During the Pandemic: A Rapid Randomized Trial of a Low-Tech Intervention in Botswana." CSAE Working Paper WPS/202013, August 2020.

1.

UNESCO. 2020. “Education: From disruption to recovery.” https://en.unesco.org/covid19/educationresponse

2.
Andrabi, Tahir, Benjamin Daniels, and Jishnu Das. 2020. “Human Capital Accumulation and Disasters: Evidence from the Pakistan Earthquake of 2005.” RISE Working Paper. Cooper, Harris, Barbara Nye, Kelly Charlton, James Lindsay, and Scott Greathouse. 1996. “The effects of summer vacation on achievement test scores: A narrative and meta-analytic review.”  Review of Educational Research 66, no. 3: 227-268. Jaume, David, and Alexander Willén. 2019. “The long-run effects of teacher strikes: evidence from Argentina.” Journal of Labor Economics 37, no. 4: 1097-1139. Slade, Timothy S., Benjamin Piper, Zikani Kaunda, Simon King, and Hibatalla Ibrahim. 2017. “Is ‘summer’ reading loss universal? Using ongoing literacy assessment in Malawi to estimate the loss from grade-transition breaks.” Research in Comparative and International Education 12, no.4: 461-485. 
3.
Bacher-Hicks, Andrew, Joshua Goodman, and Christine Mulhern. 2020. “Inequality in Household Adaptation to Schooling Shocks: Covid-Induced Online Learning Engagement in Real Time.” NBER Working Paper Series. Chetty, Raj, John N. Friedman, Nathaniel Hendren, and Michael Stepner. 2020. “How did COVID-19 and stabilization policies affect spending and employment? a new real-time economic tracker based on private sector data.” National Bureau of Economic Research Working Paper No. w27431.
4.
UNESCO Institute for Statistics (UIS). 2014. “Botswana.” http://uis.unesco.org/en/country/bw