Student Learning 

 
You can read about our methodology for estimating cost-effectiveness here

Numerous strategies to improve student learning have been evaluated, and found to have widely different impacts. These different strategies also incur drastically different costs, and some programs therefore achieve learning gains with much greater cost-effectiveness than others.

The cost-effectiveness analysis (CEA) presented on the previous page compares the impact of different programs on student test scores measured against the costs of running those programs. All of the impact estimates are drawn from randomized evaluations, and the costs of running the programs have been gathered from a variety of sources, including academic papers, conversations with field staff, and program budgets.

J-PAL has adopted a standard methodology for conducting CEA from the perspective of a policymaker who has a particular policy objective in mind and is trying to decide between a range of different options for achieving that policy objective. In order to provide comparable cost-effectiveness estimates across these options, a standard methodology must be applied to calculate both impacts and costs. Basic details of our methodology are included below.

Impacts

  • Impacts are measured in terms of standard deviation changes in student test scores. Standard deviations measure how much individual test scores change as a result of a program compared to the average test score of the comparison group. For example, a 0.2 SD change would move a child from the 50th to the 58th percentile. In the education literature, an increase of less than 0.1 SD is typically considered to be a small effect, while an increase of more than 0.3 SD is considered a large effect, and an increase of more than 0.5 SD would be a very large effect. (A forthcoming J-PAL publication will provide further discussion on test scores as a measure of student learning).
  • A ten percent discount rate is applied to impacts incurred over multiple years (after year 1) to account for how an end user of the program would trade off between the value of the services this year versus next year.

Costs

  • The calculations include only the incremental cost of adding a new education program, under the assumption that many of the fixed costs of running a school system will be incurred even in the absence of the program in question.
  • The analysis assumes that policymakers care not just about the costs incurred by their organization or government ministry, but also about costs imposed on beneficiaries and society as a whole. We therefore include the following costs, when relevant:
    • Beneficiary time when a direct requirement of the program, e.g. time involved in traveling to and attending program meetings
    • Goods or services that were provided for free (in situations where free procurement would not necessarily be guaranteed if the program were replicated in a new context)
    • The monetary value of both cash and in-kind transfers
  • Inflation is calculated using GDP deflators. When calculating the average inflation from the base year to the year of analysis, we assume that program costs are incurred on the first day of each year.
  • Costs are expressed in terms of 2011 USD, with local currencies exchanged using standard (not PPP) exchange rates.
  • A ten percent discount rate is applied for costs incurred over multiple years in order to adjust for the choice a funder faces between incurring costs this year, or deferring expenditures to invest for a year and then incurring costs the next year.

When converting costs to dollars in the year of analysis (in this case, 2011 USD), J-PAL applies a standard order of operations to address inflation, exchange rates, and present value: (1) local cost data is exchanged into US dollars using the exchange rate from the year the costs were incurred, (2) these costs are deflated back to the real value in base year prices using the average annual US inflation rate, (3) a ten percent discount rate is applied to take the present value for costs incurred after the first year, and (4) then the average US inflation rate is used to inflate costs forward to the year of analysis. This particular order of operations is not necessarily better than any other, the important thing is that an order be selected and consistently applied to all programs in an analysis.

For more details on the assumptions made in this analysis, please see Dhaliwal et al. 2013.

Underlying Calculations & Sensitivity Analysis

CEA does not by itself provide enough information for a policymaker to make any decisions regarding resource allocation, but it can serve as a useful starting point for assessing the efficacy of different programs and their relevance to a particular situation. When the calculations are done at a highly disaggregated level, with assumptions about key factors such as program take-up or unit costs laid out explicitly, it is easier to gain insights into which programs are likely to provide the greatest value for money in a particular situation, and the key factors to which these outcomes are most sensitive. 

The analysis includes 27 education programs whose impact on student learning was rigorously measured using a randomized evaluation and where the study authors have made detailed cost information available. We wish to gratefully acknowledge the contributions of the researchers whose work we are drawing on in providing us with original cost data about the programs that were evaluated, and in working with us to develop the cost-effectiveness models. Their support and input has been essential in creating such detailed cost-effectiveness models.

The underlying calculations for the CEA of programs aimed at improving student test scores are available below. For any questions or comments, please contact Conner Brannen (cbrannen@mit.edu).

  1. Unconditional cash transfers in Malawi (Baird, McIntosh, and Özler 2011)
  2. Minimum conditional cash transfers in Malawi (Baird, McIntosh, and Özler 2011)
  3. Girls' merit scholarships in Kenya (Kremer, Miguel, and Thornton 2009)
  4. Village-based schools in Afghanistan (Burde and Linden 2012)
  5. Providing earnings information in Madagascar (Nguyen 2008)
  6. Reducing class size in Kenya (Duflo, Dupas, and Kremer 2012)
  7. Textbooks in Kenya (Glewwe, Kremer, and Moulin 2009)
  8. Flipcharts in Kenya (Glewwe et al. 2004)
  9. Reducing class size in India (Banerjee et al. 2007)
  10. Building/improving libraries in India (Borkum, He, and Linden 2013)
  11. School committee grants in Indonesia (Pradhan et al. 2012)
  12. School committee grants in the Gambia (Blimpo and Evans 2011)
  13. Adding computers to classrooms in Colombia (Barrera-Osorio and Linden 2009)
  14. One Laptop per Child program in Peru (Cristia et al. 2012)
  15. Diagnostic feedback for teachers in India (Muralidharan and Sundararaman 2012)
  16. Read-a-Thon in the Philippines (Abeberese, Kumler, and Linden 2007)
  17. Individually-paced computer-assisted learning in India (Banerjee et al. 2007)
  18. Extra contract teacher + streaming by achievement in Kenya (Duflo, Dupas, and Kremer 2011; 2012)
  19. Remedial education in India (Banerjee et al. 2007)
  20. Streaming by achievement (Duflo, Dupas, and Kremer 2011)
  21. Contract teachers in Kenya (Duflo, Dupas, and Kremer 2012)
  22. Teacher incentives in Kenya (Glewwe, Ilias, and Kremer 2010)
  23. Camera monitoring and pay incentives in India (Duflo, Hanna, and Ryan 2012)
  24. Training school committees in Indonesia (Pradhan et al. 2012)
  25. Grants and training for school committees in the Gambia (Blimpo and Evans 2011)
  26. School committee elections and linkage to local government in Indonesia (Pradhan et al. 2012)
  27. Linking school committees to local government in Indonesia (Pradhan et al. 2012)

You can also download the a full workbook of all calculations.

Please note, all calculations are a work in progress and are subject to change.

Read more

See more detailed information about the projects in this analysis.

This figure shows the cost-effectiveness of programs that aim to improve student learning. The cost-effectiveness of each program is measured as the ratio of the aggregate impact of the program—the average test score improvement per student multiplied by the number of students impacted—to the aggregate cost of implementing the program. The numbers presented in the graph represent the total number of standard deviations gained across any sample size for US$100. Cost-effectiveness analysis does not, by itself, provide sufficient information to inform any policy or investment decisions, but it can serve as a useful starting point in the decision-making process by highlighting trends in the types of programs that tend to be the most cost-effective in general.

When access to education is extremely limited, getting children into school can lead to large learning gains. Opening new schools in rural areas of Afghanistan (4) had the greatest impact on test scores of any program in this bulletin and was also quite cost-effective.

In areas with high enrollment rates, it is less clear that increasing attendance on its own is a cost-effective strategy, although improving student motivation to attend and learn can be very cost-effective. Incentivizing students by giving scholarships to the best performing children (3) is a cost-effective strategy to increase both children's time in school as well as their subsequent test scores. In Malawi, providing cash transfers conditional on school attendance (2) did increase test scores, but was less cost-effective than some other approaches. Unconditional transfers (1) increased attendance but not test scores.

Incentivizing students and their families by providing information on the higher wages earned by those with more years of schooling is a very cost-effective way to improve student attendance and test score performance. The program in Madagascar (5), which was very low-cost because it used existing estimates of the effects of education on wages, was the second most cost-effective program included in this analysis. While there is a wide confidence range around the impact estimate for this approach, the program would still be very cost-effective even using the most conservative estimate of effectiveness.

Providing “more of the same” resources is generally insufficient to improve learning when unaccompanied by other reforms. Providing additional teachers to reduce class sizes had no effect on student test scores in India (9) or Kenya (6). Nonteacher inputs, such as flip charts (8) or textbooks (7a), similarly had no impact on average test scores. Even programs in which schools are given discretionary grants to purchase the inputs they feel the students need most have little impact, if any, on student learning. One grant program in India (28) had a positive impact on test scores after the first year, but this impact was offset by a reduction in household education spending. After the second year, any impact of the program had disappeared.

Interventions that direct instruction toward children’s actual learning levels are the most consistently effective at improving learning outcomes, and are also very cost-effective. If a school has more than one class per grade, then reassigning students to classes by initial learning level—often known as streaming—costs very little, improves test scores, and is therefore extremely cost-effective (20). Even if it is necessary to hire a new contract teacher to allow the class to be divided, streaming is still cost-effective (18). Providing targeted help for students in the lower half of their class (19), as well as computer programs that allow for self-paced learning, also appear to be quite cost-effective (17).

Incentives for teachers can lead to significant learning gains if they are objectively administered and structured in such a way as to discourage "teaching to the test." Linking teachers' salaries to their attendance—objectively monitored through daily photos of the teachers with their students—was both an effective, and cost-effective, strategy for improving student test scores (23). But when incentives are tied to student learning outcomes, there may be a danger of “teaching to the test.” An in-kind incentive program in Kenya (21) raised test scores after the second year of the program, but the improvement seems to be driven only by an increase in test preparation. One year after the program ended, any impact had disappeared. However, another program in India (29) that linked teachers' pay with their students test score performance led to test score gains that seem to represent an actual, more durable, increase in learning. Teachers’ pay was either linked to school-wide performance or the performance of the individual teacher’s own students. In the first year, the two types of incentives were equally effective. However, the individual bonuses were much more effective in the second year, and were more cost-effective than the group bonuses in both years.

Adding an extra teacher on a short-term contract can produce significant learning gains at a relatively low cost. Many teachers in developing countries have poor incentives and absenteeism rates are high. Contract teachers—who are hired and held accountable by the local community and whose contracts can be terminated if they perform poorly—are often more likely to attend school and extend more effort when in the classroom then their civil service counterparts. Contract teachers are often paid only a fraction of the salary of civil-service teachers, which makes such programs extremely cost-effective. If we assume the contract teacher is used to replace the civil-service teacher, this intervention, in principle, saves money and therefore may be considered infinitely cost-effective (21).

Providing additional resources or information to the community may be insufficient to improve education quality if community members are not given a clear avenue or sufficient power to affect change. Providing schools or communities with grants to purchase classroom materials has no lasting impact on student learning (11-12). However, when the funds are combined with programs to empower the community, students see large improvements in test scores at a low cost. Programs in Indonesia that aimed to increase the legitimacy and authority of the local school committee (which was already receiving additional funds to spend on educational materials) led to significant learning gains and was highly cost-effective (26-27).

Return to top