Incentivized community grants for aid effectiveness

Enabling local communities to allocate development funds to programs and activities that best meet their needs could more effectively improve outcomes for the neediest individuals and avoid wasteful spending. Providing performance bonuses based on communities’ past success at improving health and education outcomes is one way to make social spending even more effective. A randomized evaluation of Indonesia’s community block grant program, Generasi, conducted by J-PAL affiliate Benjamin Olken (MIT) and co-authors Junko Onishi and Susan Wong from 2007 to 2009 found that community block grants linked to performance were more effective in improving maternal and child health outcomes than non-incentivized grants. Based on this evidence, the Government of Indonesia moved exclusively to a system of incentivized grants in 2010. The version of the Generasi program ran from 2010 to 2018 and reached 4.9 million people including 2.8 million women.¹ Researchers Benjamin Olken, Audrey Sacks, and co-authors conducted a follow-up evaluation from 2016 to 2018 to study the longer-term effects of the Generasi program, and the results informed two new programs that aim to reduce childhood stunting in Indonesia.

The Problem

What are the best ways to design performance-based aid programs?

In recent decades, there has been an increased drive among governments, international organizations, and NGOs to strengthen the link between development aid and performance. While linking aid to performance may create incentives to improve effort and mobilize additional resources, there are potential downsides as well. For example, communities or individuals may reallocate effort toward targeted indicators at the expense of other goals. There is also a risk that performance-based aid will drive budgets to be directed to richer or otherwise better-performing locations.

In 2007, the Indonesian government began a pilot program to test community-based block grants—a new approach to improving health and education. Under the program, known as Generasi, villages received annual block grants which they could allocate to activities covering twelve health, nutrition, and education indicators. The twelve indicators represented activities that were within direct control of villages—such as the number of children who receive immunizations or prenatal and postnatal care, and the number of children enrolled and attending school—rather than long-term outcomes, such as test scores or infant mortality.

The Research

Community block grants improved health and education in Indonesian villages and adding performance incentives sped up improvements in health, particularly in contexts where baseline health was low.

J-PAL affiliated researchers and coauthors worked with the Government of Indonesia’s Ministry of Home Affairs to evaluate the effectiveness of Generasi as it was being rolled out in villages across the country between 2007 and 2009. In some randomly selected communities, a portion of the subsequent year’s grant was allocated based on their relative performance on twelve health, nutrition, and education indicators. Researchers found that health indicators improved more for incentivized communities than for non-incentivized communities, particularly in less developed provinces, but found no effect of incentives on education. In total, between 50–75 percent of the impact of the block grant program on health indicators can be attributed to the performance incentives. Overall the results showed that Generasi was effective at increasing basic health service utilization and speeding up improvements in health in poorer areas where baseline service delivery and health indicators were low.

To learn more about the program and the 2007-2009 evaluation, see the evaluation summary.

After scaling up the Generasi program in 2010, the Government of Indonesia began to roll out a much larger community development program called the Village Fund starting in 2014. The Village Fund was meant to gradually replace Generasi. Building on the interest of the government to generate evidence on whether the Generasi program model should be integrated into the new Village Fund scheme, J-PAL affiliated researchers and coauthors continued their collaboration with the Government of Indonesia to conduct a long-term evaluation of Generasi from 2016 to 2018.

Based on the long-term evaluation, several of the positive health and education impacts researchers found in 2009 did not persist. Researchers found that this was most likely due to substantial improvements in access to health that had occurred across Indonesia since 2009. Yet, nine years after the program began, families in Generasi communities were still more likely to complete infant weight checks at community health centers and attend maternal health and parenting classes. Additionally, in the least developed areas, Generasi continued to be effective in getting people to use community health services, increasing both immunization and the uptake of Vitamin A supplements.²

Taken together, the two evaluations suggest that the Generasi model of incentivized community block grants may be most relevant for accelerating the achievement of basic health indicators in places where baseline levels of health and service delivery are low, and that this program model may become less necessary over time once health indicators in these communities improve.

Learning from the long-term evaluation, in 2018, researchers recommended the Government of Indonesia prioritize other programs that were evaluated and found to be effective at reducing stunting instead, such as the national conditional cash transfer program.

From Research to Action

Building an evaluation into the program design enabled researchers to engage the government at all stages and resulted in the rapid scale-up of the research pilot.

The Government of Indonesia decided to scale up the Generasi program in 2010 following the original evaluation completed in 2009, reaching 4.9 million people.

Following the Government of Indonesia’s decision to scale up incentivized block grants within Generasi in 2010, the US government’s Millennium Challenge Corporation (MCC) signed a US$600 million compact with the Government of Indonesia to fund projects to reduce poverty. Citing the evaluation results, MCC pledged over US$130 million of the compact to support the next phase of Generasi, including revising the program to further target stunting and nutrition outcomes, which were identified as priorities by the government.²

The revised program, known as Generasi Plus, reinforced the community incentives system originally piloted under the program and added performance incentives for individual service providers to ensure that the supply of health services met communities’ demands. The Government included an evaluation component that focused on measuring household-level impacts, as well as the effect of incentives for communities and health workers.

Although the Generasi program ended in 2018, the results of the long-run evaluation were taken up by the World Bank in designing their new program to reduce stunting (Investing in Nutrition and Early Years). In particular, the new program built upon the Generasi facilitation model and citizen engagement tools.³ Additionally, within the guide for villages to tackle stunting, Indonesia’s Ministry of Village, Disadvantaged Regions, and Transmigration recommends that village leaders invest more in community health centers and early childhood education, which is in line with the findings of the evaluation. The guide also cites the Generasi program as a point of reference for villages to learn from when designing programs to decrease stunting.

Conducting long-term impact evaluations is essential for learning whether programs continue to have an impact as programs, policies, and contexts change. The process can inform how to adjust targets and policies accordingly based on the changing needs of the people and the condition of public services.

In a blog post, J-PAL staff reflect on maintaining this long-term research partnership.

References

Kementrian Desa, Pembangunan Daerah Tertinggal, dan Transmigrasi. 2018. “Buku Saku Desa dalam Penanganan Stunting” Jakarta. http://pdtu.bindola.com/uploads/attachment/2018/03/1522313956.pdf.

Local Solutions to Poverty. 2015. PNPM Generasi: 2015 Annual Progress Report. Jakarta. https://localsolutionstopoverty.org/sites/default/files/2018-04/2015_Q4_W1-1_Generasi.pdf.

Olken, Benjamin, and Audrey Sacks. 2019. Indonesia - Long-Term Generasi Impact Evaluation (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/523031547706730571/Indonesia-Long-Term-Generasi-Impact-Evaluation.

Olken, Benjamin A., Junko Onishi, and Susan Wong. 2014. "Should Aid Reward Performance? Evidence from a Field Experiment on Health and Education in Indonesia." American Economic Journal: Applied Economics 6(4): 1-34. http://dx.doi.org/10.1257/app.6.4.1.

Olken, Benjamin, Audrey Sacks, Gregorius Kelik Agus Endarso, Ali Winoto Subandoro, and RobertWrobel. 2018. Indonesia - Long-term impact evaluation of Generasi (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/481161529900287499/Indonesia-Long-term-impact-evaluation-of-Generasi.

Sacks, Audrey, and Jesse Hession Grayman. 2018. Indonesia - Long-Term Generasi Qualitative Study (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/102101542808714703/Indonesia-Long-Term-Generasi-Qualitative-Study.

World Bank. 2018. Indonesia - Investing in Nutrition and Early Years Program for Results: Technical Assessment Summary (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/527251527686447818/Final-Technical-Assessment-Investing-in-Nutrition-and-Early-Years-P164686.docx.

World Bank. 2018. Indonesia - Investing in Nutrition and Early Years Program for Results: Program Appraisal Document (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/303221529811053670/pdf/180601-INEY-PAD-for-BOARD-06012018.pdf.