Building Resilient Health Systems: Community Monitoring and Nonfinancial Awards During Sierra Leone’s 2014 Ebola Outbreak

Researchers:
Darin Christensen
Johannes Haushofer
Bilal Siddiqi
Maarten Voors
Location:
Sierra Leone
Sample:
254 government-run primary health clinics
Timeline:
2011 - 2015
Target group:
  • Children
  • Health care providers
  • Mothers and pregnant women
  • Families and households
Outcome of interest:
  • Citizen satisfaction
  • Health outcomes
  • Maternal health
  • Service provider performance
Intervention type:
  • Health care delivery
  • Community monitoring
  • Non-monetary incentives
AEA RCT registration number:
AEARCTR-0002085

Partners:

Over eight million people in low- and middle-income countries die annually from treatable conditions, in part because patients doubt the quality of the health care system and do not utilize low-cost and potentially life-saving services. Researchers partnered with the Government of Sierra Leone to evaluate the impact of community monitoring and nonfinancial awards programs on health care utilization and health outcomes. Both programs improved clinic utilization, patient satisfaction, and symptom reporting during the 2014 Ebola crisis. Further, community monitoring improved child health and reduced mortality among Ebola patients.

Policy issue

Over eight million people die annually in low- and middle-income countries from treatable conditions. Potentially life-saving health services are often available at low cost but are frequently underutilized, in part because patients doubt the quality of health care. Under-utilization of health services can be particularly damaging during health crises such as epidemics, when complying with public health directives like testing and contact tracing can help contain their spread.

One approach to improving health care delivery and utilization is holding frontline health providers accountable through community monitoring, whereby health providers and community members are brought together to jointly address obstacles to adequate health care provision.1 An alternative approach involves introducing competition among health providers and giving status awards to high performers as a form of motivation. These nonfinancial social incentives differ from the use of pay-for-performance incentives, which have shown mixed effectiveness and may be too expensive for resource-constrained governments.234

Context of the evaluation

In 2010, the Government of Sierra Leone launched a free health care initiative in an effort to combat high rates of maternal and child mortality. The initiative removed user fees for pregnant and lactating women and children under age five, and increased health worker salaries. The government also wanted to institute additional programs to motivate frontline health workers in providing health services, and partnered with researchers to introduce community monitoring and nonfinancial awards programs. 

In 2014, about one year after these programs ended, the West Africa Ebola epidemic reached Sierra Leone. Fears of substandard care and a lack of confidence in health workers deterred patients from reporting to health clinics for treatment. By the end of the crisis in early 2016, Sierra Leone had roughly 14,000 cases of Ebola. The timing of the evaluation relative to the epidemic meant researchers could examine whether the community monitoring and nonfinancial awards programs contributed to the health system’s capacity to respond to crises.

Health care workers in Sierra Leone put on personal protective equipment during the 2014-16 Ebola outbreak.
Photo: Belen B. Massieu | Shutterstock.com

Details of the intervention

Researchers, working with Innovations for Poverty Action, partnered with the Government of Sierra Leone, the World Bank, Concern Worldwide, International Rescue Committee, and Plan International to evaluate the impact of two programs—community monitoring and nonfinancial awards—on health care utilization and health outcomes before and during the 2014 Ebola epidemic. They randomly assigned 254 government-run primary health clinics in four districts to either community monitoring, nonfinancial awards, or a comparison group.

Under the community monitoring program, facilitators shared clinic scorecards with staff and community members. The scorecards compared clinic performance to the district average on several maternal and child health indicators. Clinic performance was discussed during three facilitated meetings involving: only community members; only clinic staff; and interface meetings with both groups. During interface meetings, both groups shared complaints and formulated an action plan outlining activities they would take to improve services and the time frame for each step. They also held three follow-up meetings to revisit the plan and monitor progress.

The nonfinancial awards program facilitated competition among clinics. The highest performing and most-improved clinics in each district could win status awards, including letters of commendation from district health officials and a public award ceremony. Clinics were ranked on key measures such as worker absenteeism, staff attitude, and charging of illegal fees, though the criteria were not publicly revealed to avoid distorting staff effort.

To measure the impact of the programs, researchers surveyed 5080 households in 508 communities surrounding clinics about their health, health care utilization, and health care satisfaction. Clinics were also surveyed to collect data on clinic organization and services. Further, researchers surveyed community leaders on community relations with the clinics. To examine the interventions’ longer-run impacts on the Ebola epidemic, researchers matched government data on reported Ebola cases to the areas around 160 clinics in the study.5

Results and policy lessons

Community monitoring and nonfinancial awards for clinic staff improved clinic utilization, patient satisfaction, and symptom reporting during the 2014 Ebola crisis, while community monitoring also improved child health and reduced mortality among Ebola patients.

Both programs increased patient satisfaction and health care utilization. In the areas surrounding program clinics, patients’ satisfaction with their care increased by 0.10 standard deviations on average relative to those near clinics in the comparison group. General health care utilization also increased by 0.11 standard deviations on average. Increased utilization suggests that individuals perceived improvements in the quality of care, which encompassed changes in both actual quality as well as views on quality.  

Figure  1 Health care utilization increased across both interventions
Health care utilization increased across both interventions

Community monitoring also improved child health, leading to 38 percent fewer deaths of children under age five. Since utilization increased by similar levels under both programs, while health outcomes only improved under community monitoring, community monitoring may have improved the effectiveness of health services. 

Improvements in the perceived quality of care increased reporting of Ebola symptoms and willingness to seek treatment during the epidemic. During the 2014 Ebola crisis, areas with program clinics increased reporting of Ebola cases—including patients that tested both positive and negative for the virus—by 62 percent relative to comparison areas. These results likely stem from improvements in the perceived quality of care, which increased patients’ willingness to report their symptoms and seek treatment. The increases in symptom reporting supported containment efforts and do not suggest increased transmission.

Figure  2 Reporting of suspected Ebola cases increased relative to the comparison group
Reporting of suspected Ebola cases increased relative to the comparison group

Similar to the pre-Ebola period, only community monitoring improved health outcomes during the Ebola crisis. Community monitoring reduced Ebola-related deaths from one patient death for every four Ebola cases to about one in ten. This suggests that improvements in the quality of care administered by community monitoring clinics persisted during the crisis.

Figure  3 Community monitoring reduced Ebola deaths from 1 in 4 to about 1 in 10 cases
Monitoring reduced Ebola deaths from 1 in 4 to about 1 in 10 cases

Based on the project’s success within the health sector, the Government of Sierra Leone scaled up the community monitoring intervention to cover additional sectors, including water, education, waste management, and social services. The planned expansion was delayed by the Ebola outbreak, but scale-up efforts resumed in 2016. 

Christensen, Darin, Oeindrila Dube, Johannes Haushofer, Bilal Siddiqi, and Maarten Voors. 2021. “Building resilient health systems: Experimental evidence from Sierra Leone and the 2014 Ebola outbreak”. Quarterly Journal of Economics, Vol. 136, 1145-1198

1.
Björkman, Martina and Jakob Svensson. 2009. “Power to the People: Evidence from a Randomized Field Experiment of Community-Based Monitoring in Uganda.” Quarterly Journal of Economics 124(2): 735-769. https://doi.org/10.1162/qjec.2009.124.2.735 and https://www.povertyactionlab.org/evaluation/community-based-monitoring-primary-healthcare-providers-uganda 
2.
Finan, Frederico, Benjamin A. Olken, and Rohini Pande. 2017. “The Personnel Economics of the Developing State.” Handbook of Field Experiments II, 467–514. https://doi.org/10.1016/bs.hefe.2016.08.001
3.
Huillery, Elise and Juliette Seban. Forthcoming. “Financial Incentives, Efforts, and Performances in the Health Sector: Experimental Evidence from the Democratic Republic of Congo.” Economic Development and Cultural Change. https://doi.org/10.1086/703235
4.
Donato, Katherine, Grant Miller, Manoj Mohanan, Yulya Truskinovsky, and Marcos Vera-Hernández. 2017. "Personality Traits and Performance Contracts: Evidence from a Field Experiment among Maternity Care Providers in India." American Economic Review 107 (5): 506-10.  https://www.povertyactionlab.org/evaluation/designing-contracts-healthcare-providers-india
5.
This included data on confirmed, negative, suspected, and probable Ebola cases. Data on reported Ebola cases and deaths came from the Viral Hemorrhagic Fever (VHF) database, maintained by the Government of Sierra Leone and external partners, including the US Centers for Disease Control and Prevention (CDC).