J-PAL at 20: Promoting more inclusive education in Italy
How can we ensure that students from all backgrounds can achieve their academic potential? This blog post, the first in a series exploring J-PAL's research and activities across Europe, looks at a series of randomised evaluations in Italy that explore this question with a particular focus on students from immigrant families. As school systems across Europe welcome increasingly diverse populations, this research provides insights into pathways for transforming students’ educational aspirations and academic performance, as well as their well-being.
Schools play an integral role in promoting social inclusion. They aim to provide students with a range of skills that can equip them for success in further study and employment, as well as for broader participation in the economic and social life of their communities. Addressing exclusion in schools can therefore play at least two important roles, by removing barriers for students from disadvantaged backgrounds to realise their full academic potential and by establishing norms around equality and inclusion that shape students’ participation in their communities now and in the future.
Recent research by J-PAL affiliated professors in Italy has evaluated how discrete interventions can shift academic trajectories and well-being by either transforming the perceptions and practices of teachers or providing students with additional information and resources. These studies were led by J-PAL affiliated professors Michela Carlana (Harvard University), Eliana La Ferrara (Harvard University), and Paolo Pinotti (Bocconi University), along with coauthors including the late Alberto Alesina and Marinella Leone (University of Pavia).
This work has largely focused on the crucial middle school years, when students in Italy, as is common across Europe, must make important choices about whether to pursue academic or vocational high school tracks. This choice has significant consequences for employment outcomes later in life.
Identifying effective measures for tackling social exclusion is a primary focus of the J-PAL network’s research in Europe. This involves a focus not only on households in poverty but also those at risk of poverty or exclusion more broadly from the economic and social life of their communities. The European Social Inclusion Initiative was launched in 2019 to support more rigorous research in this field, and researchers in our network have sought in particular to identify ways in which educational interventions can help break the intergenerational transmission of social exclusion.
Documenting and addressing the impact of teacher bias in Italian schools
One strand of this work has documented the harmful role that teacher biases can play and showed that simply making teachers aware of these biases can contribute to removing barriers to achievement.
Alesina, Carlana, La Ferrara and Pinotti found that more than two-thirds of teachers in a sample of Italian schools exhibit moderate to severe bias against immigrants, as revealed by an implicit association test. This can affect student grading—when examining students of similar ability, as reported on national standardised exams, math teachers with higher levels of bias were more likely to grade immigrant students more harshly than Italian students. In separate work, Carlana found that female students assigned to math teachers with biased views of girls’ abilities not only received lower math scores but also chose not to pursue more demanding educational tracks and reported lower self-esteem.
To test the effect of providing teachers with information on their implicit bias, the researchers randomly varied the timing of this information: either before or after reporting end-of-semester grades. They found that receiving this information leads teachers to give immigrant students higher grades and that this shift appears to be driven by teachers who may have been unaware of their bias.
The potential of counselling and tutoring support
If addressing teacher bias appears to be one important element in addressing barriers for students, another part of the solution lies in providing students with additional out-of-classroom resources. Two evaluations of counselling and tutoring interventions show that these can shift students’ academic performance and aspirations.
Carlana, La Ferrara and Pinotti evaluated a programme in Italian middle schools that helped high-achieving immigrant boys better align their high-school track choices with their abilities. They found that male students from immigrant backgrounds were more likely than native-born Italian counterparts of similar abilities to enrol in vocational rather than technical or academic-track high schools (the same was not true for girls). They examined whether a programme that combined career counselling and additional tutoring could help address this gap. Drawing on social cognitive career theory, the counselling was designed to make students aware of their skills, available opportunities, and to develop career aspirations in line with those abilities. They also received additional tutoring if their performance was below a set threshold.
Boys who participated in the programme reported greater confidence and saw fewer barriers to academic success; they performed better on a standardised test in 8th grade and were as likely as native Italian students to select academic or technical high-school tracks. Participation in the programme also shifted teacher perceptions of these students, as they became more likely to recommend them for the higher track.
One key finding from the study is that the academic aspirations of high-achieving boys in middle school are malleable, as are their teachers’ perceptions of their academic potential. In considering how to potentially scale the intervention, the researchers have explored how to potentially reduce the costs. One path is to provide the counselling arm through in-school psychologists, but in many education systems, there is a limited pool of such staff and they face many existing demands on their time. An alternative path could be to explore whether volunteers with more limited professional training could deliver a version of this counselling and still generate similar impacts? This is an important question for further study.
A second study by Carlana and La Ferrara examined the impact of online tutoring when Italian schools first closed in response to the Covid-19 pandemic. The programme targeted middle school students from disadvantaged backgrounds who had been identified by principals as at risk of falling behind during online learning. Volunteer tutors drawn primarily from teaching training programmes provided these students with an average of three to six hours of individual tutoring per week online.
An average exposure of six weeks to the programme translated into significantly higher student effort and improved test scores. Tutored students also reported higher aspirations and participation improved their overall well-being: they reported a greater degree of control over events in their lives and fewer symptoms of depression. This improvement in well-being was driven largely by students from immigrant backgrounds.
The authors are now exploring in a series of follow-up evaluations how to most efficiently deliver impactful tutoring, as well as to investigate impacts on standardised test scores.
The classroom as a model for building inclusive environments
Tackling the exclusion of children in the classroom may be not only a way to improve academic outcomes for students from different backgrounds; it might also be a way to build more cohesive communities. Carlana has partnered with J-PAL affiliated researcher Sule Alan (European University Institute) and their coauthor Marinella Leone (University of Pavia) to explore how a teacher training programme focused on developing socio-emotional skills might foster both academic achievement and more inter-ethnic cohesion. The researchers have worked with a team of cognitive scientists at the University of Pavia to design a pilot intervention in primary schools where ethnic diversity is salient. This research is expected to provide further insight into ways to create inclusive classrooms where teaching practices can promote achievement for all, as well as give children the social skills that promote cohesive communities.
Directions for future research on education in Europe
School systems across Europe are seeking ways to ensure that classrooms with students from an increasingly diverse range of backgrounds can promote the academic success of all students and give children the skills for building strong and cohesive communities in the future. These studies provide valuable insights for designing more inclusive education across Europe (and beyond). Their findings join existing research from France, which has shown that providing students and families with additional information on schooling can help them align their aspirations with their abilities, as well as research from Turkey that has shown how teaching focused on promoting understanding can create more inclusive classrooms with better academic outcomes for non-native students. The findings on the positive effects of tutoring form part of a growing body of evidence on the potential for tutoring to improve academic outcomes in primary and secondary education.
To extend and leverage these findings, European educators may wish to adopt and adapt some of the building blocks identified here. Giving teachers information about their implicit biases is likely to be a relevant tool in many other settings in Europe, and there are likely ways to strengthen and complement the effect of this information. The success of the Equality of Opportunity Programme in helping immigrant students align their aspirations with their abilities invites further study of how this could be scaled at reasonable costs. It also invites further innovation into whether other interventions could achieve the same aims, and among different populations (such as for non-immigrant students from disadvantaged backgrounds). Finally, finding ways to provide more cost-effective tutoring remains an important area for research. Carlana and La Ferrara are continuing to explore these in Italy and further afield. At J-PAL Europe, we are also happy to help educators across the continent consider how to build upon and adapt the promising findings discussed above to evaluate new solutions.
Authored By
The COVID-19 outbreak forced governments worldwide to close schools for an extended period of time resulting in massive learning losses and adverse psychological effects for children, especially the most vulnerable and those from low socioeconomic backgrounds. Researchers designed a free online tutoring program delivered by volunteer university students to test whether it could mitigate these effects. They found that the program had a positive outcome on students' learning outcomes, socio-emotional skills, and psychological well-being.
الموضوع الأساسي
Over 190 countries closed schools at the peak of the COVID-19 outbreak.1 School closure has created massive learning losses for children, leading to less time spent receiving schooling,2 lower achievement, and worse test scores.3 The pandemic has also had adverse psychological and social effects for children and adolescents, leading to higher depression and lower development of socio-emotional skills.45 Students from lower-income families were disproportionately affected, as they often had limited access to technology, received less support from parents and lower quality of remote learning from schools. The combination of these effects risks having long term consequences on the human capital of the cohorts affected by school closures, as it hinders the full development of the skills needed to contribute to a productive and cohesive society.6
While many countries have tried to mitigate learning losses by switching to remote instruction and using different types of platforms, the implementation of these tools has varied substantially, even within the same country and largely based on income levels. Schools in wealthier areas have shown higher prevalence of learning through live interaction with teachers and online participation of students. 7 High-income students also tend to have access to better homeschooling inputs, including technology, help from parents,8 and online learning resources,91011 which exacerbates educational inequalities.
Policymakers have explored numerous avenues to address these challenges, including tutoring, a program often adopted for remedial education. Taking many forms, tutoring generally consists of supplemental one-on-one or small group instruction sessions aimed at improving students’ academic performance.11 Existing evidence12 has pointed toward tutoring as an effective strategy for addressing learning losses and helping students catch up, that has consistently led to large improvements in learning in certain contexts. Yet, little evidence exists on whether a remote delivery model, as it was made essential during a global pandemic, could achieve similar results.
سياق التقييم
In 2020, Italian schools were closed from the beginning of March until the summer – more than a third of the entire school year. To facilitate the implementation of remote learning, the government allocated 85 million euros for buying tablets that students could temporarily borrow, improving internet connection and online platforms of schools, and offering digital training to teachers. Yet, not all students in need received support in time due to bureaucratic delays. Additionally, the response of teachers to remote teaching varied greatly, with some taking longer to adjust to remote teaching, as prior to the pandemic, less than 50 percent of teachers had used digital tools in their daily lectures. By June 2020, more than 96 percent of teachers in the sample were providing live online classes, 85 percent also provided about an hour of recorded content per week, and almost all assigned some homework every week.
In response to the educational crisis, researchers partnered with pedagogical experts to rapidly design and implement an innovative online tutoring program: TOP (“Tutoring Online Program”). The program targeted middle school students (grade 6 to 8) from disadvantaged backgrounds in terms of socioeconomic status, linguistic barriers, or learning difficulties, who were identified by school principals among those lagging behind during distance learning. Thirty-two percent of selected students had a learning disability, 22 percent were immigrants, and 81 percent needed help in more than one subject. The majority of tutors were female (70 percent), born in Italy (98 percent), moved by a desire to help others when applying to TOP (83 percent), had previous experience as volunteers (83 percent), and were pursuing a STEM (34 percent) or economics/business major (28 percent).
معلومات تفصيلية عن التدخل
Researchers conducted a randomized evaluation to assess the impact of TOP on students’ learning outcomes, socio-emotional skills, aspirations, and psychological well-being. Shortly after the announcement of schools’ closure, researchers contacted all middle school principals in Italy asking them to identify up to three students per class who might benefit from additional support during the months of remote learning and ensure they had internet connection and a computer or tablet. In the meantime, they recruited volunteer tutors amongst the students enrolled in graduate and undergraduate programs at three large universities in Milan. Then, researchers invited the identified students and their parents to participate in the free program. A total of 1,059 students from 76 middle schools from all over the country applied to the program and researchers randomly assigned a tutor to 530 of them, with the remaining 529 students serving as a comparison group. Researchers processed applications on a rolling basis to ensure that students could start as soon as possible. Tutoring was offered fully online and consisted of help with the homework assigned by school teachers in three subjects: math, Italian, English. Tutoring activities began in April 2020 and lasted for five weeks, until the end of the school year in June.
The tutor training was the most expensive component of the program and limited the number of tutors recruited to 530, though 2,000 volunteers had applied. Tutors received an online self-training module, regular group meetings, and on-demand one-to-one sessions with expert educators. The majority of students received 3 hours of tutoring per week, with a subset of those needing help in more than one subject randomly assigned to receive 6 hours per week.
To measure the impact of the program, researchers administered a standardized test very close in format, though shorter, to the national standardized one to participating students and conducted surveys of parents, students, tutors and teachers to measure students’ academic performance, aspirations, socio-emotional skills, and psychological well-being. Additionally, the surveys enabled researchers to measure the impact of the program on tutors’ ability to empathize with students and views on the role of hard work and effort to achieve success in life. Researchers collected data in April, before the tutoring began, and in June.
النتائج والدروس المستفادة بشأن السياسات
Overall, researchers found that the online tutoring program improved students’ academic performance, socio-emotional skills, and psychological well-being.
Student effort, attendance, and behavior: students enrolled in TOP exerted more effort in their homework compared to their peers, according to reports by students, parents and teachers. Additionally, teachers reported that students in TOP were 9.4 percentage points more likely to attend class regularly, a 16 percent increase with respect to their peers in the comparison group. Students in TOP were also 8.1 percentage points (or 10 percent) less likely than their peers to report difficulty in following the classes online and using their school’s online platform. Students in TOP also behaved better during school hours, with the fraction of students for which teachers reported behavioral problems being 6.4 percentage points lower relative to the comparison group, where behavioral problems were reported for 83 students. Finally, students in TOP were 4.9 percentage points more likely to enjoy the subjects of math, literature or English relative to comparison students (amongst which only 28 percent reported enjoying the subjects).
Academic performance: For students in TOP, standardized test scores improved by 0.26 standard deviations relative to the comparison group. Students who had participated in the tutoring sessions, answered more questions correctly, a 9 percent increase relative to an average of 53 percent correct in the comparison group. The effects are particularly strong for math, the subject in which most students received tutoring. Teachers’ assessments of overall learning also improved for students in TOP compared to their peers, by 0.18 standard deviations.
Aspirations, socio-emotional skills and psychological well-being: The program did not have an impact on student or parent aspirations. Students in TOP reported a greater belief in their ability to control the outcome of events in their lives (0.19 standard deviations), fewer symptoms of depression (by 0.16 standard deviations), and parents reported higher happiness levels for their children (by 0.16 standard deviations), relative to the comparison group.
Different effects based on students’ characteristics: Children from lower socioeconomic background, whose parents both worked outside the home, saw a larger improvement in learning outcomes. Additionally, students with learning disorders seemed to benefit more from the program and saw their performance increasing by over 0.5 standard deviations. This is potentially because this is a group that might have struggled the most with the learning methods and materials that schools provided during distanced learning. Additionally, the improvement in wellbeing was entirely driven by immigrant students, for whom the effect was an increase of 0.77 standard deviations. Researchers hypothesized that this group might have a less dense network of friendship and so felt more isolated during the lockdown and benefitted from the regular interaction with a tutor. Turning to parental employment, having a tutor improved students’ performance significantly more for the children of blue-collar mothers compared to white collar mothers.
Impact on tutors: Researchers found that participating in TOP increased tutors’ empathy by 3.4 percentage points, 0.27 standard deviations increase compared to volunteers who were not assigned a student to supervise. Researchers did not detect a meaningful effect on tutors’ perceptions of the role of hard work to achieve success in life.
The researchers are currently piloting TOP in the Dominican Republic and having discussions with several other countries in Latin America to expand the program in the fall of 2021. In Italy, the program will be scaled to 3,000 children during the school year 2021-22 as part of a wider partnership with the Ministry of Education.
UNESCO. 2021. “One year into COVID-19 education disruption: Where do we stand?”. Accessed September 27, 2021. https://en.unesco.org/news/one-year-covid-19-education-disruption-where-do-we-stand
Azevedo, J. P., Koen Geven, Diana Goldemberg, Amer Hasan, and Syedah Aroob Iqbal. 2020. “Simulating the Potential Impacts of Covid-19 School Closures on Schooling and Learning Outcomes: A set of Global Estimates”. Policy Research Working Paper Series (9284), The World Bank.
De Witte, Kristoff and Joana Maldonado. 2020. “The Effect of School Closures on Standardised Student Test.” FEB Research Report Department of Economics.
Espada, José, Elisa Delvecchio, Claudia Mazzeschi, Alexandra Morales and Mireia Orgilés. 2020. “Immediate Psychological Effects of the Covid-19 Quarantine in Youth from Italy and Spain.” PsyArXiv. Doi: https://doi.org/10.3389/fpsyg.2020.579038
Golberstein, Ezra, Hefei Wen, and Benjamin F. Miller. 2020. “Coronavirus Disease 2019 (Covid-19) and Mental Health for Children and Adolescents.” JAMA pediatrics.
World Bank. 2020. The Human Capital Index 2020 Update : Human Capital in the Time of COVID-19. World Bank, Washington, DC. World Bank. https://openknowledge.worldbank.org/handle/10986/34432.
Malkus, Nat. 2020. “School districts’ remote-learning plans may widen student achievement gap”. Education Next, 20(3). Doi: https://www.educationnext.org/school-districts-remote-learning-plans-may-widen-student-achievement-gap-only-20-percent-meet-standards/
Agostinelli, Francesco, Matthias Doepke, Giuseppe Sorrenti, and Fabrizio Zilibotti. 2020. “When the Great Equalizer Shuts Down: Schools, Peers, and Parents in Pandemic Times.” IZA Discussion Paper No. 13965.
Engzell, Perr, Arun Frey, and Mark Verhagen. 2020. “Learning Inequality During the Covid-19 Pandemic.” SocArXiv. Doi: 10.31219/osf.io/ve4z7
Bacher-Hicks, Andre, Joshua Goodman, and Christine Mulhern, 2020. “Inequality in Household Adaptation to Schooling Shocks: Covid-Induced Online Learning Engagement in Real Time.” Journal of Public Economics, 193:104345.
Doyle, Orla. 2020. “Covid-19: Exacerbating Educational Inequalities?” Working Paper.
J-PAL Evidence Review. 2020. “The Transformative Potential of Tutoring for PreK-12 Learning Outcomes: Lessons from Randomized Evaluations.” Cambridge, MA: Abdul Latif Jameel Poverty Action Lab.
This blog discusses the conference hosted by J-PAL Europe and IDEE in November 2022, which aimed to bring together stakeholders—primarily policymakers, practitioners, and researchers—and increase understanding about how impact evaluations can be used in education settings.
“Globally, children are learning less in their school years than we would have expected,” says Esther Duflo, director and co-founder of J-PAL. “The system is not treating students with the quality it should be,” and, in France, there is a “rapid and disturbing decline in students' mathematical skills.”
To help address these and other challenges, J-PAL Europe, in collaboration with the French Ministry of Education (Direction de l'Évaluation, de la Prospective et de la Performance) and leading French universities, launched the Innovation, Data, and Experiments in Education (IDEE) programme in 2022.
This effort is motivated in part by growing demand for the rigorous scientific evaluation of education policies and programmes in France. The French Ministry of Education, and its statistical department in particular, has shown a strong commitment for greater and more rigorous evaluation. Still, while many innovative programmes have been developed to respond to challenges in the French education landscape, few among them have been rigorously evaluated.
In turn, this shortage of scientific evaluation limits the use of evidence in policymaking and teaching practices, and therefore the identification of the most effective education programmes. Furthermore, without close partnerships between policymakers, researchers, and practitioners, evidence generation and use becomes more challenging for all parties.
In an effort to reduce these barriers, IDEE has an eight year mandate (2022-2030) to foster evidence generation and use in education in France, with funding from the French National Research Agency. The programme helps education researchers conduct rigorous impact evaluations, including by facilitating access to administrative data and providing research resources and measurement tools. Furthermore, IDEE is well positioned to create enduring relationships between government, researchers, and practitioners by providing opportunities to collaborate and strengthen capacities for evidence generation and use.
Bringing stakeholders together
In November 2022, J-PAL Europe and IDEE hosted a conference to bring together stakeholders—primarily policymakers, practitioners, and researchers—and increase understanding about how impact evaluations can be used in education settings. The conference aimed to raise awareness about the added value of evidence, and impact evaluations in particular, in education. It also facilitated the exchange of ideas and research priorities at the crossroads of policy, practice, and research.
At the conference, IDEE’s scientific coordinator and J-PAL affiliated professor Marc Gurgand (Paris School of Economics) outlined the motivations of IDEE, saying:
"The programme provides a structural framework that seeks to align actors, actions, points of view, and objectives in the service of research that is itself in the service of the public education system, and that cannot be conducted without actors from across the entire French education ecosystem."
This mission led to numerous productive conversations. Alongside keynote addresses from Duflo and Edouard Geffray (French Ministry of Education), multiple panels featuring education experts and practitioners allowed participants to converge on research and policy priorities. With topics ranging from research on math card games to teenage sleep patterns, participants engaged in multidisciplinary discussions where they touched not only on the process of evidence generation but also on how it can be further incorporated into education policy decisions.
The cognitive neuroscientist Stanislas Dehaene (Collège de France) also joined the event and anchored an evidence-sharing panel discussing research results across different domains. Panelists also shared some of the constraints of running randomized evaluations. For example, J-PAL affiliated professor Élise Huillery (Université Paris-Dauphine) dove into an evaluation she conducted aimed at reducing the high-school dropout rate in France via a socio-emotional skills programme. The evaluation, which took place over a period of five years, found that in-class discussions focused on developing skills like motivation, perseverance, and self-discipline had numerous positive impacts: students, and girls in particular, had better learning outcomes and reported improved moods and set higher goals.
In another notable panel, Lou Aisenberg, IDEE’s strategic development manager, along with researchers, policymakers, and practitioners, discussed how to best co-construct a research agenda that both builds on existing evidence and meets the needs of policymakers and practitioners. To achieve this goal, panelists noted the importance of creating space and time to align partners’ priorities and account for both methodological and on-the-ground constraints.
Featured evaluation: Increasing math skills in France through card games
In 2021 and 2022, the IDEE team provided methodological support to develop two evaluations of math card games. The first programme, called ‘Well played!,’ invited approximately 2,500 first grade students from more than 100 classes in France to play various math games during the summer holidays of 2021.
To measure the impact of this programme, the research team used national tests to assess students' performance in math. Students' confidence in math and their cooperation with each other was also measured using a specific test. The study results showed no effects on children’s learning, which could be due to the fact that it was conducted during the summer holidays.
In turn, this informed the development of a new math card game titled ‘Counting birds’, which was distributed in November 2021 to more than 65,000 students, in first grade classes throughout France. These cards, developed by the French Ministry of Education, can be broken down into several games that leverage different fundamental math skills. The aim of the programme was to build interest and encourage students to practice their math skills through playing games at home. National assessments were used to test students’ progress in mathematics, and their anxiety toward math was also measured using a separate survey at the end of the school year. The results of this intervention are forthcoming.
Looking forward
As IDEE enters its second year, the programme is building on the momentum from November’s conference with regular research seminars, methodological support for research teams running (or interested in running) randomized evaluations, and trainings supporting education policymakers and practitioners across France to better understand and use evidence in their decisions.
To raise awareness and further promote evidence-use in policymaking and teaching practices, IDEE is also set to launch an online class titled “Evidence-informed education: an introduction to impact evaluations."
This multi-pronged approach of direct and indirect engagement is laying the foundation for a stronger research infrastructure and an evidence-informed revitalization of the education ecosystem in France. As Duflo said, “There is no magic wand that takes scientific knowledge and applies it to the whole world.” In fact, collaborations between researchers and policymakers allow for an iterative research process that examines both successes and failures. In turn, the goal of improving educational outcomes comes closer to being met.
Interested in learning more? Visit IDEE’s website or contact Lou Aisenberg. You can also dive deeper into the topics discussed at the conference.
Researchers in the J-PAL network have conducted over 80 ongoing or completed randomized evaluations of social programs and policies across 20 European countries, with a particular focus on education and labor markets. J-PAL Europe has launched a summary of insights generated from this research.
This text is also available in the following languages: French, German, Italian, Romanian, and Spanish.
Researchers in the J-PAL network have conducted over 80 ongoing or completed randomized evaluations of social programs and policies across 20 European countries, with a particular focus on education and labor markets, and today we launch a summary of insights generated from this research.
The complexity and uncertainty surrounding Europe’s Covid-19 pandemic response over the past year have drawn attention to the importance of rigorous evaluation in determining effective policy responses.
Just as a series of vaccines have moved through clinical trials to prove their efficacy and to avoid harm, increased attention should be given to evaluating responses to the broader social and economic challenges the pandemic has drawn into focus. Across Europe, national education systems are struggling to deal with the learning loss that disproportionately affected students from poorer backgrounds, while unemployment rates have risen, particularly affecting those with lower levels of education. In many cases, the economic fallout of Covid-19 has exacerbated existing inequities. Careful—and increasingly nimble—evaluation of the potential responses to these challenges can play an important role in determining effective responses.
For example, as school systems across the world turned to remote learning, J-PAL affiliates Michela Carlana and Eliana La Ferrara launched an evaluation of online tutoring for disadvantaged middle school students in Italy. Through a low-cost online homework tutoring program, school pupils from disadvantaged backgrounds improved their academic performance and wellbeing during the lockdown. In Turkey, where schools have often seen social segregation along ethnic lines, Sule Alan and co-authors have been evaluating an interactive classroom program aimed at fostering social cohesion by encouraging students to consider one another’s perspectives.
While J-PAL may be best known for its work in lower-income countries—measuring the impact of pricing on uptake of malaria bednets across Africa, for example; or evaluating the best ways to eliminate leakage in public spending programs in India or Indonesia—the network is bringing the same methods to answer unresolved questions here in Europe. How can we best reduce the impact of the learning gap that has widened between students of different economic backgrounds since the spring lockdowns? Do boarding schools offer an effective path to competitive third-level education for high-performing children who might otherwise be excluded? What kinds of job counseling are most cost-effective for helping jobseekers find work?
Our new Evidence in Europe resource highlights some of the lessons that are emerging in response to these questions and others across the fields of education, labor, finance, governance, women’s empowerment, and social inclusion.
Improving access to high-quality education
Students from disadvantaged backgrounds face multiple barriers to high-quality education. In some cases, the problem may be one of physical access: better schools are often located outside poorer neighborhoods.
In France, there has been continued interest in whether boarding schools for high-achieving students from disadvantaged backgrounds play an important role in addressing unequal access to elite post-secondary academic tracks. Offering a seat to the country’s first boarding school of excellence to students who lacked supportive home environments had a long-term impact on students’ schooling choices, leading them to choose more selective baccalaureate tracks at the end of high school.
Sometimes barriers to advancement may exist within a single school. In Turkey, researchers have evaluated a program geared towards promoting inclusion of refugees within the classroom (Syrian refugees attend school alongside their Turkish-born counterparts). The “perspective-taking curriculum,” designed to encourage students to consider situations from the perspective of others, reduced violence in classrooms, increased friendships between the two groups, and led to increased Turkish-language abilities for the Syrian children.
Aspirations and information about educational opportunities can also play a role in determining the academic path of students. Here, teacher expectations or prejudices can play a determinative role, as can the information and aspirations of a student’s parents.
In Italy, research found that a majority of teachers exhibited some form of bias against immigrant students, potentially discouraging them in school and having a negative impact on their future careers. Informing teachers of their own bias reduced discrimination in grading and improved the grades teachers gave to immigrant students. Another Italian randomized evaluation found that a career counseling program helped close the education gap between immigrant and Italian-born boys, improving immigrants’ academic performance and increasing the likelihood of enrolling in more demanding high school tracks better aligned with their abilities.
We know that including parents can also be an important tool for improving student performance—particularly parents who may otherwise lack access to such information. In France, a study found that interventions that provided information to parents (in ten languages) and encouraged them to become more involved in their children’s education led to better educational outcomes. By increasing parental engagement, this series of interventions improved student behavior and reduced dropout rates.
These are just a few examples of evaluations across Europe that have influenced plans of the respective education ministries to scale up the programs once proven effective. In Turkey, the ministry is using results from these evaluations to inform new educational curricula aimed at improving students’ soft skills. In France, following the evaluations, the ministry has decided to make the parental involvement program available to all schools in the country.
Shaping minds to learn
Soft skills, such as patience, self-control, and perseverance, have been shown to play a key role in children’s and adolescents’ development. Fostering these skills in children can help improve academic performance and potentially level the playing field between different groups in the classroom. In Turkey, researchers partnered with the Ministry of National Education to test a series of innovative policies to improve educational outcomes through soft skills development.
For example, an innovative curriculum focused on improving students’ grit through animated videos, case studies, and classroom activities helped students exert more effort in learning. The intervention also reduced the gender gap in competitiveness by encouraging girls who were likely to succeed to compete. In another study in Turkish schools, teaching students to be more patient increased patient decision-making and led students to receive higher behavior grades in the classroom.
Quality employment
Labor markets are also an important forum for improving social inclusion, and improving outcomes here can pay dividends for individuals across their careers, as well as improve the welfare of their families.
A priority for many state employment agencies is finding the best ways to connect jobseekers with quality, long-term employment. Many countries make extensive use of job search counseling, but this can be costly for employment agencies and jobseekers alike. A series of studies has examined how to most effectively provide this assistance.
Research in Denmark, France, and Germany has shown the effectiveness of such efforts in certain situations, particularly when it is tailored and established early in any unemployment spell. Sometimes intensive counseling efforts (weekly meetings, for example) can be effective, but this is not true for everyone: One study in Denmark showed that increasing the intensity of job counseling for youth with limited job prospects can actually decrease employment rates, as time spent with a case worker may have replaced time spent working or seeking work.
In a market where jobseekers may be competing for a limited number of jobs, investing in job search counseling for some may help them find jobs at the expense of other jobseekers, raising questions of the overall effectiveness of such programs. A landmark study in France showed that increased counseling for some jobseekers harmed the prospects of other jobseekers in the same area in the short term and that, after a year, those who had benefited appeared no more likely to have found long-term work.
Helping the unemployed across Europe find work in the wake of the pandemic is going to require more than matching them to existing jobs. Another area of research has focused on promoting entrepreneurship, particularly among youth. These programs have not always been successful. But an evaluation of one entrepreneurship training program in the Paris suburbs that emphasized independent decision-making found that participants were no more likely to set up their own businesses, but that they were more likely to find stable employment two years after the start of the program and reported higher salaries, as well as greater confidence and optimism.
These results suggest that providing information and business training may be insufficient to encourage business creation and that tackling other barriers, such as low self-confidence or a lack of entrepreneurial spirit, could offer a more effective way to support unemployed youth.
Applying evidence-based insights
Impact evaluation is most effective when we leverage the insights produced not only to consider the impact of a particular intervention but more broadly to guide future program and policy design investments by governments and civil society alike.
The scale of the challenge that Europe faces in building back from the pandemic is clear. We are going to need innovative responses to tackling issues such as addressing widening learning gaps between students, or to matching people living in precarity to better jobs in a changed employment landscape.
The solutions that emerge are unlikely to be as easily reproducible as a proven vaccine. But investing in more systematic evaluation of these solutions, paired with careful thinking about the range of factors behind their success and how they might need to be adapted to apply in other settings, will strengthen our ability to respond at scale.
At J-PAL Europe, our aim is to build on the 80 randomized evaluations produced by affiliated researchers—along with rigorous evidence from across the continent—to produce lessons that can inform the design of new solutions, or support evidence-backed programs to move to scale. We partner with governments and organizations not only to launch new evaluations but also to apply what we have already learned. We hope that the lessons outlined in our new Evidence in Europe resource will be a useful tool in designing a more inclusive Europe for all.