Applying the generalizability framework: Adopting Teaching at the Right Level in Zambia
How can governments learn from evidence generated from randomized evaluations?
Over the last three years, J-PAL Africa has supported the Zambian Ministry of General Education to pilot and scale up the Catch Up Program, with the help of Pratham, Innovations for Poverty Action, UNICEF Zambia, VVOB – education for development, the USAID Zambia Mission, and USAID Development Innovation Ventures. Catch Up is a remedial education program modeled off the effectiveness of the Teaching at the Right Level (TaRL) approach but grounded in the Zambian context.
Our support to the Zambian government has been guided by a framework for how evidence can inform policy across contexts, as presented in Bates and Glennerster (2017). The framework describes how to apply evidence from one context to another by using a combination of randomized evaluations, theory, descriptive data, and process monitoring. To guide this process, it puts forward the following questions:
- What is the disaggregated theory behind the program?
- How strong is the evidence for the required general behavior change?
- Do the local conditions hold for the theory to apply?
- What is the evidence that the implementation process can be carried out well?
Teaching at the Right Level: Evidence, theory, and generalizable behavior
Several studies conducted by J-PAL affiliated researchers and others over the last twenty years on how to improve learning outcomes shed light on a key generalizable lesson in many contexts: teachers tend to teach to the top of the class.
Given the structure of education systems across many parts of the world, this is unsurprising. Many teachers are confronted with classes comprised of students with a wide variety of learning needs, dense and ambitious curricula, and high-stakes primary school leaving exams, which incentivize teachers to move at the pace of the fastest learners.
Teaching at the Right Level, an approach pioneered by Indian NGO Pratham, targets the root of this behavior. The approach groups children by homogenous learning needs; dedicates time to building basic skills rather than focusing solely on the curriculum; and regularly assesses student performance, rather than relying only on end-of-year exams.
Over the last 15 years, together Pratham and J-PAL have rigorously tested TaRL’s underlying theory of change through six randomized evaluations in India and a growing body of research in Africa, which find that when TaRL is successfully implemented, learning outcomes improve.
Catch Up: Context matters
Teaching at the Right Level is not an intervention that can be applied universally. Rather, it is an approach for working with education systems to help them adjust more flexibly to children’s needs.
Several important conditions existed in Zambia that allowed TaRL to take root:
- Zambia had a clear need for the program. Zambia ranked last in the 2011 Southern and East Africa Consortium for Monitoring Education Quality (SACMEQ) in numeracy and tied last with Malawi in literacy.
- Learning levels within each classroom are varied, and students have little recourse to learn basic skills if they have not mastered them in the foundational years.
- Education stakeholders were candid in their acknowledgment of the problem of basic skills and were open to thinking about solutions from other countries.
With support from J-PAL, the ministry chose three different evidence-based implementation models to pilot in the country.
While the evidence on TaRL is strong, it also shows how implementation failures lead to reduced impact. Before fully rolling out the program, the partner team supported the ministry to test whether the implementation could work effectively and sustainably in this new context. This work included:
- Piloting in 80 schools, with monitoring by government officials and independent monitors,
- Developing a data collection system and helping leaders react to this data, and
- Building a system of review meetings into the approach.
Given the demonstrated need for a basic skills intervention and the pilot’s implementation success, the ministry committed to scaling Catch Up to 1,800 schools by 2020. The scale-up plan includes options for iterative learning to ensure the program maintains effectiveness at scale and continues to improve over time.
Interest in Teaching at the Right Level continues to grow throughout Africa. From September 26-27, 2018, governments and organizations from across Africa interested or implementing TaRL-inspired interventions will attend a J-PAL and Pratham-hosted Teaching at the Right Level conference for the launch of the new TaRL website. The website will house useful implementation focused resources, as well as information on upcoming TaRL events and will provide a platform for those interested in TaRL to engage. The conference will provide an opportunity for the TaRL Community of Practice to share implementation lessons, strengthen design, and more effectively collaborate.
Authored By
Building on over a decade of rigorous research around the world and two years of iterative learning in Zambia, the Ministry of General Education will scale up a program called Catch Up to approximately 1,800 schools in Zambia over the next three years. Based on the Teaching at the Right Level (TaRL) approach pioneered by the Indian NGO Pratham, Catch Up aims to improve basic literacy and numeracy skills of primary school students in grades 3–5. The program drew on support from a number of collaborating partners, with J-PAL Africa working together with Innovations for Poverty Action (IPA), Pratham, UNICEF, USAID’s Development Innovation Ventures, USAID Zambia, VVOB, and Zambia Education Sector Support Technical Assistance. The partners’ assistance included sharing global evidence on remedial education, shaping the design of Catch Up by mapping evidence to the Zambian context, supporting pilot implementation and monitoring, and assessing viability for scale up.
The ministry hopes Catch Up will address low-learning levels in the country. A 2014 national assessment found that 68 percent of grade 2 learners were unable to read a single word in their local language. Grade 5 students also scored poorly in local language and mathematics. Since 2015, J-PAL Africa has worked with the Zambian government in leveraging the evidence on TaRL to help design, pilot, and expand the Catch Up program. Together with ministry officials and cooperating partners, J-PAL Africa identified Zambia as an appropriate context for TaRL following extensive scoping of the country’s education sector that revealed low-learning outcomes among students in early grades, large variation of learning levels in the classroom, and government commitment to improving students’ basic skills.
J-PAL Africa, with several cooperating partners including Pratham and IPA Zambia, created a working group to develop the program’s various components. A subsequent learning journey to India allowed ministry officials to observe a TaRL program in action under Pratham and to meet with Indian government officials leading the program.
Over the past year, the government piloted Catch Up in eighty schools. To design the content and structure of the pilot, the Catch Up working group combined evidence on earlier TaRL programs with local knowledge of the particular demands and constraints of the Zambian education space. Together with Pratham, the working group developed teaching materials, assessment tools, classroom activities, and monitoring and mentoring processes tailored for the Zambian context. It also identified three potential models for students in grades 3–5, and piloted and assessed the performance of each during October 2016–July 2017. The Ministry of General Education, the Global Partnership for Education, UNICEF, and J-PAL’s Government Partnership Initiative provided financial support to the pilot. VVOB, a Flemish education NGO, provided technical support for implementation. IPA led an independent process monitoring of the pilot. ZESSTA contributed technical assistance and in-kind support.
In the coming months, the ministry’s Catch Up working group will feed the lessons from the pilot into the implementation of the program at scale. The next phase will start in January 2018 with support from USAID DIV and the USAID Zambia Mission.
The application of the Teaching at the Right Level (TaRL) approach in Zambia, through the Ministry of General Education’s Catch Up program, provides an opportunity to capture key aspects of TaRL in an African context.
TaRL pioneer Pratham, together with J-PAL Africa, worked closely with the Ministry of General Education to produce a set of videos that transport the viewer into a Catch Up classroom and highlight essential aspects of the TaRL approach.
At its core, TaRL is an education intervention that targets teaching to the basic skill level of the child. TaRL teachers assess learners’ basic reading and mathematics levels, group them according to level (rather than age or grade), and spend a period of the day or year using engaging, relatable activities to teach basic skills.
Rigorous impact evaluations over the last 17 years have demonstrated the effectiveness of the Teaching at the Right Level approach.
The videos were developed with the primary goal of strengthening training as Catch Up moves to scale in Zambia.
As with any program, as it expands, key principles of the approach risk getting diluted through multiple rounds of training and implementation, which could affect the success of the program. J-PAL Africa and Pratham created the videos in an effort to further ensure that information gets delivered accurately and consistently to teachers and mentors.
As interest in TaRL grows, we hope these videos will also serve as a useful tool for policymakers and organizations across Africa, helping to deepen their understanding of the TaRL methodology—and spark their interest in learning more about adapting and building their own TaRL programs.
For those seeking a more tangible TaRL experience than an academic paper or policy report, the videos provide a window into the energy and feel of a TaRL classroom. However, they are just one component of a broader set of resources and support systems designed to help policymakers and organizations understand and implement TaRL and should be used together with other pieces of process support.
Pratham and J-PAL Africa work collaboratively to provide this process support to stakeholders interested in building TaRL interventions in Africa.
The Catch Up program was piloted in 2016/17 and is being scaled up by the Ministry of General Education to 1800 schools over the next three years. The program drew on support from a number of collaborating partners, with J-PAL Africa working together with Innovations for Poverty Action (IPA), Pratham, UNICEF, USAID’s Development Innovation Ventures, USAID Zambia, VVOB, and Zambia Education Sector Support Technical Assistance.
The TaRL videos were filmed in two Catch Up pilot districts, Pemba in Southern Province and Katete in Eastern Province. Thank you to Makomba Primary School and Kazulabowa Primary School for making it possible for us to document Catch Up classes in action!
Listen in on our next TaRL webinar on training best practices, scheduled for 6 March 2018, to learn more about the TaRL approach.