Rigorous testing of social policies in Europe: What are we learning?

This text is also available in the following languages: French, German, Italian, Romanian, and Spanish.

Researchers in the J-PAL network have conducted over 80 ongoing or completed randomized evaluations of social programs and policies across 20 European countries, with a particular focus on education and labor markets, and today we launch a summary of insights generated from this research. 

The complexity and uncertainty surrounding Europe’s Covid-19 pandemic response over the past year have drawn attention to the importance of rigorous evaluation in determining effective policy responses. 

Just as a series of vaccines have moved through clinical trials to prove their efficacy and to avoid harm, increased attention should be given to evaluating responses to the broader social and economic challenges the pandemic has drawn into focus. Across Europe, national education systems are struggling to deal with the learning loss that disproportionately affected students from poorer backgrounds, while unemployment rates have risen, particularly affecting those with lower levels of education. In many cases, the economic fallout of Covid-19 has exacerbated existing inequities. Careful—and increasingly nimble—evaluation of the potential responses to these challenges can play an important role in determining effective responses.  
 
For example, as school systems across the world turned to remote learning, J-PAL affiliates Michela Carlana and Eliana La Ferrara launched an evaluation of online tutoring for disadvantaged middle school students in Italy. Through a low-cost online homework tutoring program, school pupils from disadvantaged backgrounds improved their academic performance and wellbeing during the lockdown. In Turkey, where schools have often seen social segregation along ethnic lines, Sule Alan and co-authors have been evaluating an interactive classroom program aimed at fostering social cohesion by encouraging students to consider one another’s perspectives.

While J-PAL may be best known for its work in lower-income countries—measuring the impact of pricing on uptake of malaria bednets across Africa, for example; or evaluating the best ways to eliminate leakage in public spending programs in India or Indonesia—the network is bringing the same methods to answer unresolved questions here in Europe. How can we best reduce the impact of the learning gap that has widened between students of different economic backgrounds since the spring lockdowns? Do boarding schools offer an effective path to competitive third-level education for high-performing children who might otherwise be excluded? What kinds of job counseling are most cost-effective for helping jobseekers find work? 

Our new Evidence in Europe resource highlights some of the lessons that are emerging in response to these questions and others across the fields of education, labor, finance, governance, women’s empowerment, and social inclusion. 

Improving access to high-quality education

Students from disadvantaged backgrounds face multiple barriers to high-quality education. In some cases, the problem may be one of physical access: better schools are often located outside poorer neighborhoods. 

In France, there has been continued interest in whether boarding schools for high-achieving students from disadvantaged backgrounds play an important role in addressing unequal access to elite post-secondary academic tracks. Offering a seat to the country’s first boarding school of excellence to students who lacked supportive home environments had a long-term impact on students’ schooling choices, leading them to choose more selective baccalaureate tracks at the end of high school.

Sometimes barriers to advancement may exist within a single school. In Turkey, researchers have evaluated a program geared towards promoting inclusion of refugees within the classroom (Syrian refugees attend school alongside their Turkish-born counterparts). The “perspective-taking curriculum,” designed to encourage students to consider situations from the perspective of others, reduced violence in classrooms, increased friendships between the two groups, and led to increased Turkish-language abilities for the Syrian children. 

Aspirations and information about educational opportunities can also play a role in determining the academic path of students. Here, teacher expectations or prejudices can play a determinative role, as can the information and aspirations of a student’s parents. 

In Italy, research found that a majority of teachers exhibited some form of bias against immigrant students, potentially discouraging them in school and having a negative impact on their future careers. Informing teachers of their own bias reduced discrimination in grading and improved the grades teachers gave to immigrant students. Another Italian randomized evaluation found that a career counseling program helped close the education gap between immigrant and Italian-born boys, improving immigrants’ academic performance and increasing the likelihood of enrolling in more demanding high school tracks better aligned with their abilities.

We know that including parents can also be an important tool for improving student performance—particularly parents who may otherwise lack access to such information. In France, a study found that interventions that provided information to parents (in ten languages) and encouraged them to become more involved in their children’s education led to better educational outcomes. By increasing parental engagement, this series of interventions improved student behavior and reduced dropout rates.

These are just a few examples of evaluations across Europe that have influenced plans of the respective education ministries to scale up the programs once proven effective. In Turkey, the ministry is using results from these evaluations to inform new educational curricula aimed at improving students’ soft skills. In France, following the evaluations, the ministry has decided to make the parental involvement program available to all schools in the country.

Shaping minds to learn

Soft skills, such as patience, self-control, and perseverance, have been shown to play a key role in children’s and adolescents’ development. Fostering these skills in children can help improve academic performance and potentially level the playing field between different groups in the classroom. In Turkey, researchers partnered with the Ministry of National Education to test a series of innovative policies to improve educational outcomes through soft skills development. 

For example, an innovative curriculum focused on improving students’ grit through animated videos, case studies, and classroom activities helped students exert more effort in learning. The intervention also reduced the gender gap in competitiveness by encouraging girls who were likely to succeed to compete. In another study in Turkish schools, teaching students to be more patient increased patient decision-making and led students to receive higher behavior grades in the classroom.

Quality employment 

Labor markets are also an important forum for improving social inclusion, and improving outcomes here can pay dividends for individuals across their careers, as well as improve the welfare of their families. 

A priority for many state employment agencies is finding the best ways to connect jobseekers with quality, long-term employment. Many countries make extensive use of job search counseling, but this can be costly for employment agencies and jobseekers alike. A series of studies has examined how to most effectively provide this assistance. 

Research in Denmark, France, and Germany has shown the effectiveness of such efforts in certain situations, particularly when it is tailored and established early in any unemployment spell. Sometimes intensive counseling efforts (weekly meetings, for example) can be effective, but this is not true for everyone: One study in Denmark showed that increasing the intensity of job counseling for youth with limited job prospects can actually decrease employment rates, as time spent with a case worker may have replaced time spent working or seeking work. 

In a market where jobseekers may be competing for a limited number of jobs, investing in job search counseling for some may help them find jobs at the expense of other jobseekers, raising questions of the overall effectiveness of such programs. A landmark study in France showed that increased counseling for some jobseekers harmed the prospects of other jobseekers in the same area in the short term and that, after a year, those who had benefited appeared no more likely to have found long-term work. 

Helping the unemployed across Europe find work in the wake of the pandemic is going to require more than matching them to existing jobs. Another area of research has focused on promoting entrepreneurship, particularly among youth. These programs have not always been successful. But an evaluation of one entrepreneurship training program in the Paris suburbs that emphasized independent decision-making found that participants were no more likely to set up their own businesses, but that they were more likely to find stable employment two years after the start of the program and reported higher salaries, as well as greater confidence and optimism.   

These results suggest that providing information and business training may be insufficient to encourage business creation and that tackling other barriers, such as low self-confidence or a lack of entrepreneurial spirit, could offer a more effective way to support unemployed youth.

Applying evidence-based insights

Impact evaluation is most effective when we leverage the insights produced not only to consider the impact of a particular intervention but more broadly to guide future program and policy design investments by governments and civil society alike. 

The scale of the challenge that Europe faces in building back from the pandemic is clear. We are going to need innovative responses to tackling issues such as addressing widening learning gaps between students, or to matching people living in precarity to better jobs in a changed employment landscape. 

The solutions that emerge are unlikely to be as easily reproducible as a proven vaccine. But investing in more systematic evaluation of these solutions, paired with careful thinking about the range of factors behind their success and how they might need to be adapted to apply in other settings, will strengthen our ability to respond at scale.  

At J-PAL Europe, our aim is to build on the 80 randomized evaluations produced by affiliated researchers—along with rigorous evidence from across the continent—to produce lessons that can inform the design of new solutions, or support evidence-backed programs to move to scale. We partner with governments and organizations not only to launch new evaluations but also to apply what we have already learned. We hope that the lessons outlined in our new Evidence in Europe resource will be a useful tool in designing a more inclusive Europe for all.