Measuring social attitudes and norms in impact evaluations: Examples from India

Understanding social norms is important to reducing inequality and improving outcomes for marginalized groups. Broadly, social norms can be thought of as shared expectations or informal rules among a set of people. These norms define acceptable attitudes and behaviors for different identity categories—for example, those of gender, race, and caste.

Social norms also often persist despite economic growth, and can negatively impact development outcomes. Evaluating the impact of interventions targeting social norms can be useful to understand how they might be changed.

When social norms manifest as gender norms, they can lead to harmful—or helpful—gender-related practices. For example, research indicates that when gender norms within a society are supportive of female employment, it can result in delayed marriage, increased female work aspirations, and improved child health.

But among researchers, social and gender norms are often seen as nebulous constructs that cannot be objectively measured. These constructs may be difficult to quantify into measurable household-level indicators, as researchers would do for an academic study, because they also reflect structural issues embedded at the societal level. Or so it is widely perceived.

Here, we interrogate the notion that measurement of social norms is difficult and present an overview of tools J-PAL affiliated and invited researchers have used to measure such outcomes in impact evaluations.

We present examples of such research to demonstrate that researchers and practitioners who are concerned that asking direct questions about norms may yield biased results have solutions at hand: evaluating how norms change attitudes and assessing how norms alter behaviors. 

Researchers employ these measurement approaches in impact evaluations of development programs. These approaches reveal valuable insights into what works, under which circumstances, to improve outcomes. The two approaches have helped implementers establish the effectiveness of their programs, and policymakers identify and scale up effective programs.

Attitude Change

Development and behavioral economics often borrow tools from social psychological research in order to study attitudes (one’s belief of what is socially acceptable).

Let’s take the case of a J-PAL-funded evaluation of a school-based gender attitude change program implemented by the NGO Breakthrough in Haryana, India.

Researchers used one such tool, the Implicit Association Test (IAT). The IAT exposes participants to two concepts and detects how strongly individuals connect these concepts. One application of this test is in understanding how strongly people associate certain groups with certain stereotypes. In the case of gender, the test would measure how strongly people associate women with concepts stereotypically associated with women—for example, domesticity or docility. 

Practitioners and researchers often believe that for tools such as the IAT, field data collection in low-income settings is challenging. Certainly, such tools may require equipment, internet connectivity, and specially trained surveyors. However, the use of the IAT in low-income government schools in Haryana for the Breakthrough study demonstrates the applicability of these tools in development economics research. 

Asking individuals about their views towards certain social groups is another method of evaluating social attitudes, especially if IATs may not be possible,. For example, in the evaluation of Breakthrough’s gender attitude-change program, respondents were asked to indicate the extent to which they agreed with statements such as “A woman’s most important role is being a good homemaker.” 

One concern with these direct survey questions is that respondents may give a socially desirable response—a phenomenon known as “social desirability bias”—and therefore biasing the results. However, there are ways to statistically test for the existence of this bias.

For instance, the Marlowe-Crowne scale of social desirability assesses the extent to which a respondent shares their actual opinions, rather than misrepresenting their views to appear more socially acceptable. The scale gauges if respondents give answers that are “too good to be true,” such that no one would agree with these statements unless trying to give the “right answer.” 

Respondents in the Breakthrough evaluation were tested on this scale, and it was found that treatment effects were equally large among study participants with a strong tendency to provide honest responses.

Behavior Change

Changing attitudes may not be enough to change behaviors, as there may be other constraints. For practitioners interested in behaviors as a way to better understand the prevalent social norms in a given context, studying real and hypothetical decision-making is a novel tool to do so.

In the evaluation of Breakthrough’s program, one measure in the two-year follow-up survey assessed whether study participants would sign a public petition supporting gender equality, and another looked at take-up of a scholarship offered to girls.

Thus, if the school-based program is successful, deeply held gender attitudes change and gender-related behaviors will concomitantly change: all students will be more vocal in support of gender equality, and girls will be more motivated to pursue higher education. 

Similarly, in a J-PAL-funded study on the impact of recreational cricket leagues on inter-caste contact, researchers measured behavior change by asking participants to choose future teammates and people they would consider “friends” from a list. The researcher hypothesized that participants who were exposed to people of other castes would have more positive attitudes about other-caste individuals and therefore select more diverse friends and future teammates. 

Another study on caste discrimination used grades given on test papers that researchers randomly labelled with different caste names to understand how discriminatory school teachers were.

Common to all three studies is the approach of measuring people’s exhibited behaviors as a way to understand social norms and if, and how, they change. 

Looking ahead

It is an exciting time for researchers, donors, and implementers aiming to transform social norms hindering development and equality. The research community is continuously developing and refining new tools to measure outcomes that were once considered difficult to measure.

J-PAL affiliated researchers and staff have contributed to this endeavor by producing the Practical Guide to Measuring Women’s Empowerment, which draws on experiences from their research around the world and offers practical tips for how to measure women's and girls’ empowerment in impact evaluations. 

Although norms may seem intangible, the studies cited above illustrate that it is possible to measure norms as an outcome in impact evaluations. Stakeholders stand to benefit from drawing from the literature of impact evaluations of programs aimed at changing social norms, and applying these concepts in their own work.

This rich and growing literature can help implementers gain a more nuanced understanding of successful program components, and can also aid policymakers in scaling what works.