We need evidence-based public policies in Brazil
This post was first published as an op-ed in Nexo Jornal and has been translated from the original Portuguese.
The main challenge when it comes to public policy is knowing what works and why. Brazilian policy debate too often is built on belief-based arguments and ideology and relies little on data or rigorous evidence.
The adoption of policies based on empirical evidence is growing worldwide. Unfortunately, Brazil still lies very far behind when it comes to evaluating its public policies and using evidence to support them. Several recent policy discussions on reducing the age of criminal responsibility, labor reform, subsidies to automobile manufacturers, and political campaign financing reform, have been carried out with limited attention to evidence and evaluation.
In the column I am heading for Nexo, I will discuss current issues related to public policies, focusing on economic aspects and empirical evidence from Brazil and the world. The column will tackle a variety of topics, such as poverty reduction, education quality, corruption control, business productivity, health care quality, and crime and violence reduction.
The main challenge when it comes to public policy is knowing what works and why. Brazilian debate too often is built on belief-based arguments and ideology and relies little on data or rigorous evidence.
What do I mean by rigorous evidence? When talking about public policy impacts, we need to talk about cause and effect. That is, we need to attribute some action to a policy that causes people to change behavior after an intervention. The main challenge of empirical work is to isolate this relationship between cause and effect. In controlled laboratory studies, this analysis is done using experiments. In the social sciences, on the other hand, isolating this cause-effect relationship is more difficult. How can we do that?
First, we need to bring the lab into the real world. A growing number of studies use randomized evaluations to identify the effects of public policies when some individuals (or units, such as schools or towns) are given access to a specific policy and others are not. The decision of who gets what is done randomly—i.e. similar to a lottery or draw. When we choose the beneficiaries through a lottery, we make sure that the people who are given the policy are similar—except in the policy they have received—to those who have not received it. Thus, we can assign any measured program effects to the policy in question.
The impact evaluation of Progresa (Education, Health, and Food Program) in Mexico provides an example of this methodology. Progresa was a program similar to the Brazilian social welfare program, Bolsa Família. It consisted of an income transfer to low-income families, conditional on their compliance with education and health requirements. However, unlike the Brazilian program, beginning with its implementation in 1997, it had an evaluation built in.
In 1998, 506 towns in seven states were selected through a lottery for the evaluation. Of these, 320 towns were selected to receive the program, while 186 towns served as the comparison group. Low-income households in the towns chosen to start the program received a monthly transfer of 90 Mexican Pesos (approximately US$4.50 or R$7). Researchers conducted a survey in 14,500 households with approximately 89,000 individuals to analyze the baseline conditions for these families. After that, the same information from the initial survey was collected every six months for two years. The surveys measured income, consumption, labor, and indicators of education, fertility, and health.
A few years later, data showed that the program was highly effective in reducing school dropout rates (especially among young people) and improving health indicators. Conditional income transfer programs are now present in most Latin American countries, in part, thanks to the Progresa impact evaluation. In addition, the evaluation served to persuade subsequent governments to continue the program, even when new parties came to power.
In other situations, the government itself uses lotteries to implement programs when there is high demand for a service, but the resources or vacancies of the program are limited.
In Brazil, for example, the Controladoria Geral da União (CGU)—a federal agency tasked with preventing and investigating the misuse of public funds—has been auditing city governments since 2003 and selects, through a lottery each year, 60 cities to be audited. This lottery creates a natural experiment that allows us to compare audited city governments with those not audited in order to evaluate if the audits are effective in reducing corruption in municipal governments.
The CGU case offers an example of what can be done when, despite its advantages, it is impossible to bring the laboratory to real life. Often, policies are implemented based on other motivations and randomly selecting beneficiaries is not feasible, either for ethical or political reasons. How then can we assess these policies?
Several real-world situations resemble laboratory experiments, where we have people or places that receive a policy while others do not. Economists call such situations natural experiments. In these cases, we can often use the units that did not adopt the policy as a comparison group. What is important is that there is a comparison group that allows us to measure the effects of the desired policy—rather than using before and after comparisons, which are routinely made by politicians or the media.
Another example of these ideal natural experiments occurred in Miami in 1980. Within a few months’ time, approximately 125,000 immigrants arrived in the city by boat, fleeing Cuba. Economist David Card, a professor at the University of California, Berkeley, used neighboring towns as a comparison group to account for what happened to the wages of local workers when there was a large influx of low-skilled workers from outside. To the surprise of many economists, and a controversy that exists to this day, he found no effect on local wages as a result of this large migratory movement.
Brazilian researchers are largely aware of the numerous methodologies at their disposal to evaluate public policies, and there is sufficient data available in Brazil to conduct high-quality studies.
However, most public policies implemented in the country are not evaluated, and, when they are, the results are not used. For example, a recent study in Rio de Janeiro found the Renda Melhor Jovem program demonstrated positive results for reducing school drop-outs. Regardless, the program was discontinued by the state government and the evaluation was never discussed.
It is time for Brazil to bring empirical evidence to the center of its political debates and for public policies not to be implemented or discontinued without society knowing their real impacts. I hope this column contributes to that discussion.
Authored By
Conditional cash transfer (CCT) programs have become a popular means of encouraging school enrollment, but some worry that when children attend school and the supply of child labor in a village decreases, child wages could rise. In Mexico, researchers evaluated the impact of PROGRESA, a national CCT program, on both school enrollment and child wages. Cash transfers successfully incentivized parents to enroll their children in school, despite the fact that local wages for child labor increased when the program was implemented.
Policy issue
Although most governments have eliminated fees for primary and secondary education, families may face other costs of sending their children to school. Many schools require pupils to wear uniforms or have basic supplies, and the cost of these materials can deter families from enrolling their children. Children who are attending school also have less time to work and earn a wage to help support their family. Conditional cash transfer (CCT) programs have become an increasingly popular means of encouraging school attendance, by providing families cash conditional upon their children attending school a minimum amount of time. However, questions about the secondary effects of CCT programs remain. When children attend school and the supply of child labor in the village decreases, will child wages subsequently increase, and will parents become less likely to keep their kids enrolled in school?
Context of the evaluation
In 1997, the Mexican government implemented a national CCT program, called PROGRESA, with the objective of improving children's nutrition, health, and education. The cash transfer benefit from PROGRESA comes in two forms. The first is a monthly fixed stipend of 90 pesos (approximately US$7) conditional on family members obtaining preventive medical care. The second type of transfer comes in the form of educational grants, which are given to families of children starting the third grade, conditional on children attending school a minimum of 85 percent of the time and not repeating a grade more than twice. The size of the education stipend is larger at higher grades and is also higher for girls, because the government wanted to encourage older girls, in particular, to stay in school. The rates vary from 60 pesos per month for children enrolled in third grade to 225 pesos per month for females enrolled in the third year of junior high school.
Details of the intervention
Census data from 1997 was used to identify communities eligible for PROGRESA on the basis of socioeconomic status. For budgetary and administrative reasons, all eligible communities could not be brought into the program at the same time. Instead, communities had to be phased into the program over a two-year period. In 1998, of the 50,000 communities deemed eligible, 506 were chosen to participate in the evaluation - 320 were assigned to the treatment group and would receive program benefits immediately, and 185 were assigned to the comparison group and would begin the program almost two years later. Households in the treatment group received the cash transfer every month, conditional on completing the required health and education components. Once enrolled, households received benefits for a minimum of three years, after which they were re-assessed for eligibility.
In addition to the baseline survey, follow-up surveys were conducted for all households living in the 506 evaluation communities every 6 months between November 1998 and April 2000. The household survey covered each child' s age, gender, education, current job status, earnings, school participation, and health status. The household survey was also supplemented by a locality questionnaire, which provided information on the prices of various commodities and average local agricultural wage.
Results and policy lessons
The results suggest that PROGRESA had a positive effect on the enrollment of children, especially after primary school. Although the program had no significant impact on enrollment of 10 year-old boys, it increased enrollment for 14 year-old boys by 14 percentage points. Program impact then declined for older boys. There was no impact on enrollment of 10 year olds, because almost all children below grade 6 already attended school. The program impact increased with age as the probability of dropping out increased, until around age 14, at which point the cash transfer was no longer available.
If PROGRESA was effective in increasing school participation, then this could result in a reduction of the supply of child labor available in villages, which could result in an increase in child wages. The results suggest that the program did indeed increase child wages by around 6 percent. If families valued the cash transfer and labor income equally, then such an increase in child wages could actually cause parents to pull their children out of school. However, the results suggest that families place more than double the value on the cash transfer than on potential labor income. In other words, the cash transfer has a stronger impact on enrollment than would an equivalent reduction in child wages.
In an attempt to further understand the impact of the cash transfer on enrollment, researchers simulated different variations of PROGRESA using a dynamic model. A simulated program, which increased the amount of the transfer for children above grade 6, while eliminating the transfer for primary school children, could almost double the impact on enrollment for older boys, at no additional cost. At the same time, this kind of program would not affect school enrollment of younger primary-age children, since almost all of them would attend school up to grade 6 even without the treatment. The Mexican government is now piloting this variation of the traditional PROGRESA program.
In J-PAL's comparative cost-effectiveness analysis, PROGRESA led to 0.01 additional years of education per $100 spent. For more information, see the full comparative cost-effectiveness analysis.
Conditional cash transfer (CCT) programs have proved effective at raising health and educational achievement, but policymakers remain concerned that households may become dependent on the transfers to maintain their living standards. Researchers conducted a randomized evaluation of the impact of the Oportunidades CCT program in Mexico on recipients’ investments decisions and long-run poverty levels. They found that the cash transfers promoted long-term increases in consumption through investment in productive activities, enabling beneficiary households to achieve higher living standards that can be sustained even after transitioning off the program.
Policy issue
Cash transfer programs are increasingly common across developing countries. These programs provide relief through income support to those living in extreme poverty, and in the case of conditional cash transfer (CCT) programs, provide incentives for parents to invest in the human capital of their children by making the transfers conditional on certain behaviors. Despite their established benefits in terms of raising health and educational achievement, many policymakers and development practitioners remain concerned about the extent to which households may become dependent on the cash transfers to maintain their living standards. It is not clear whether families will revert to pre-program poverty levels when the transfers are no longer provided, or whether the transfers enable more permanent changes in household finances.
Context of the evaluation
In 1997, the Mexican government implemented a national CCT program, called PROGRESA (later called Oportunidades), with the objective of improving children's health and nutritional status thereby breaking the intergenerational transmission of poverty. To this end, the program provides financial incentives to parents to invest in the health, nutrition, and education of their children. Oportunidades is one of the largest CCT interventions in the world - by 2010, the program had distributed approximately US$4.5 billion to 5.8 million beneficiary households.
The cash transfer benefit from Oportunidades comes in two forms. The first is a monthly fixed stipend of 90 pesos (approximately US$7) conditional on family members obtaining preventive medical care. The second type of transfer comes in the form of educational scholarships, which are given to families of children starting the third grade, conditional on children attending school a minimum of 85 percent of the time and not repeating a grade more than twice. The size of the education stipend is larger at higher grades and is also higher for girls because the government wanted to encourage older girls, in particular, to stay in school. The rates vary from 60 pesos per month for children enrolled in third grade to 225 pesos per month for females enrolled in the third year of junior high school.
Details of the intervention
This study evaluates the ability of the Oportunidades CCT program in Mexico to enable increased productive investments and therefore long-run impacts on poverty levels.
Census data from 1997 was used to identify eligible communities on the basis of socioeconomic status. For budgetary and administrative reasons, all eligible communities could not be brought into the program at the same time. Instead, communities had to be phased into the program over an 18-month period. In 1998, of the 50,000 communities deemed eligible, 506 were chosen to participate in the evaluation320 were assigned to the treatment group and would receive program benefits immediately, and 186 were assigned to the comparison group and would begin the program eighteen months later. Households in the treatment group received the cash transfer every month, conditional on completing the required health and education components. Once enrolled, households received benefits for a minimum of three years, after which they were re-assessed for eligibility.
In addition to the baseline survey, follow-up surveys were conducted for all households living in the 506 evaluation communities every 6 months between March 1998 and November 2000, and again in November 2003.
Results and policy lessons
Impact on agriculture activities: The results suggest that treatment households increased ownership of productive farm assets significantly faster than comparison households. Treatment households were 17.1 percent more likely to own draft animals and 5.1 percent more likely to own production animals compared to comparison households. Agricultural production in terms of both crops and animal products increased faster for treatment households than for comparison households, resulting in a 9.6 percent increase in agricultural income.
Impact on microenterprise activities: Participation in the Oportunidades program also led to a 3.3 percentage point increase in participation in nonagricultural microenterprises - mainly production of handcrafts for sale - which translates into a 67 percent increase in the number of households operating such business.
Impact on long-term living standards: Four years after households in the comparison group were incorporated into the program, consumption levels for the original treatment households were 5.6 percent higher than for the original comparison households. This suggests that returns on investments made by treatment households during the initial 18-month experimental period translated into improvements in their long-term living standards. If removed from the program, beneficiary households, therefore, would be unlikely to revert to pre-program poverty levels.
Each year, politicians embezzle billions of dollars in federal funds, which can breed citizen distrust in government and weaken democratic institutions. In a randomized evaluation, researchers leveraged data from Brazil’s longstanding anticorruption program to evaluate whether government audits can effectively combat corruption. Results indicate that audits can effectively reduce corruption in the long run by not only promoting electoral accountability but by also enhancing judicial punishment.
Policy issue
Billions of dollars in federal funds go missing each year. Corruption, through embezzlement, bribery, or other forms of resource misallocation, can foster citizen distrust in elected officials and weaken democracy. While there is widespread awareness of corruption’s potentially adverse consequences, there is limited consensus on how best to tackle this global challenge. Even less is known on how to effectively reduce corruption in the long run. Existing research suggests combatting corruption through one of two main channels: (1) strengthening political institutions, particularly through electoral accountability, to allow citizens to reward or punish corrupt behavior; or (2) building the capacity of judicial or prosecutorial institutions to discipline corrupt officials through legal action. Successful anti-corruption strategies will likely include reforms to strengthen both the political and judicial sectors, but the efficacy of these institutions ultimately depends on a government’s ability to detect corruption in the first place. Audit programs—aimed at uncovering the misuse of public resources—offer a potential solution. Can government audits of public resources reduce corruption and, if so, through which channel, political or judicial accountability?
Context of the evaluation
Corruption in Brazil primarily occurs through a combination of procurement fraud, diversion of funds, and over-invoicing of goods and services. Given the country’s highly decentralized political system, municipal-level corruption has become a major concern, as local officials can divert or misuse funds meant for local development. On average, municipalities receive R$15 million (US$4.7 million) in federal transfers per year.
In 2003, the Government of Brazil introduced a groundbreaking anticorruption program, implemented through the Controladoria Geral da União (CGU)—an autonomous federal agency tasked with preventing and investigating the misuse of public funds. The CGU randomly selects municipalities for audit via public lottery. All municipalities with a population of up to 500,000 inhabitants are eligible. Once audited, municipalities can only be audited again after several lotteries have elapsed. Overall, the likelihood of being audited is relatively low—only 1.3 percent in a given lottery. However, given the frequency of lotteries, the probability of being audited during a mayor’s political term can be high—from 8.6 to 26.4 percent depending on the state. Thus, it is difficult for mayors to predict whether they will be audited during their terms in office. Once chosen, trained audit teams investigate selected municipalities to match accounts with actual dollars spent and to verify the delivery of public services. After completing inspections, the agency publishes detailed reports describing any irregularities online and shares their findings with local media sources, along with local and federal judicial offices. As of 2015, the CGU had conducted 2,241 audits, covering 1,949 municipalities and R$22 billion (US$5.6 billion) worth of federal funds. To date, data generated from CGU audits has been widely used in political campaigns to inform voters and has been leveraged by federal police and prosecutors to build cases against corrupt politicians and civil servants.
Details of the intervention
Researchers leveraged the program’s built-in public lottery design to conduct a randomized evaluation to assess the impact of government audits on local-level political corruption over time. Using a combination of CGU, electoral, judicial, and municipal data, researchers compared corruption levels in municipalities undergoing their first audits to municipalities that had been audited more than once. Because municipalities faced the same probability of being audited, this comparison estimates the causal effects of a past audit on future corruption levels. In reviewing CGU data, researchers differentiated between irregularities due to acts of corruption versus mismanagement (e.g. misfiling of documents, improper storage of supplies).
The study focused on corruption occurring in the 2004-2008 and 2008-2012 electoral terms. Thus, the main sample consists of all audits conducted between July 2006 and March 2013. Out of the 1,949 municipalities audited in this period, 14 percent had been audited multiple times: 253 audited twice, 18 three times, and one municipality four times.
Researchers examined four potential mechanisms through which audits may deter corrupt behavior:
- Electoral Accountability – If audits increase the perceived future probability of being exposed to voters, then mayors who have re-election concerns may refrain from corruption.
- Judicial Accountability – If audits increase legal or reputational costs, mayors may refrain from corruption even when they face term limits.
- Political Selection – If audits allow voters to punish corrupt mayors and reward good incumbents through re-election decisions, better politicians may be elected in places where the incumbent was audited prior to the election.
-
Political Entry – If audits change the political environment, new, less corrupt, candidates may run for office.
Results and policy lessons
Results indicate that audits can be an effective policy instrument for reducing corruption. Through additional modeling, the researchers suggest that audits reduced corruption by primarily increasing the perceived legal costs of engaging in corruption, rather than through other possible channels. This suggests that a sustainable reduction in corruption may require policies aimed at improving the state’s capacity to detect and prosecute corrupt politicians.
Corruption and Mismanagement: Being audited in the past reduced the likelihood of future corruption. Municipalities that had experienced a previous audit saw 7.9 percent fewer acts of corruption compared to those that had not. However, researchers found no evidence that audits affected mismanagement. In municipalities in which mayors experienced multiple audits within the same term, the audits led to an even larger reduction in corruption—a 12.7 percent reduction in corruption relative to thecomparison group. The audits uncovered that, on average, 30 percent of funds were diverted, implying that audits reduced corruption by R$567,135 (US$178,000) per year per municipality.
Local Media: Local media played a crucial role in strengthening the anti-corruption effects of audits. Municipalities that were not audited, but that were located next to audited municipalities and had access to local media, also experienced a decline in corruption. Where AM radio was present, municipalities with neighbors who were audited saw a 7.5 percent reduction in corrupt acts relative to municipalities with neighbors in the comparison group. Municipalities with access to local television saw a 10.4 percent reduction in corrupt acts. Researchers found no evidence of similar effects in municipalities without a media presence. These findings highlight the importance of media in enhancing political accountability.
Judicial Accountability: Modeling suggests that reductions in corruption came mostly from audits increasing the perceived legal costs of engaging in corruption. Municipalities that had been audited in the past were 0.5 percentage points more likely to face legal action—such as through police investigations or mayoral convictions—relative to 3 percent of municipalities in the comparison group. This effect implies that audits led to an increase of approximately 30 legal actions from a base of 140 among comparison municipalities. Furthermore, for every 1 percent increase in corrupt acts, there was an 8.8 percent increase in the likelihood of legal action, suggesting that the legal costs of engaging in corruption are substantial. Researchers estimated that this legal disciplining effect accounted for nearly 72 percent of the total reduction in local corruption that resulted from the audits. However, this effect was largely concentrated in municipalities with a judiciary district, implying that audits may be less effective in environments where governments have less capacity for judicial enforcement.