Reflecting on research during times of uncertainty: A recap from Puerto Rico’s EDUGESPRO graduation

In 2017, the Puerto Rico Department of Education (PRDE) faced multiple difficult policy decisions. A team of researchers led by J-PAL affiliated researcher Gustavo J. Bobonis worked with J-PAL North America to establish a collaboration to support the PRDE in applying an evidence-based approach to their policymaking process. Four years later, we reflect on the evolution of our research-practice partnership, how we pivoted in the face of national and international emergencies, and our efforts to improve academic achievement and reduce inequality in student learning in Puerto Rico. 

Forging a research partnership to inform education policy

What began as an exploratory collaboration with the PRDE has grown into a robust partnership that comprises a group of J-PAL affiliated researchers and other experts, including Marco Gonzalez-Navarro, Daniela Scur, Orlando Sotomayor, and Rubén Gaztambide-Fernández. Working with its local partner, the Universidad del Sagrado Corazón in Puerto Rico, the team recruited an Evaluation and Research Officer, Emily Goldman, to work with the PRDE to identify and support the implementation of high-priority projects where rigorous evaluation could help answer key policy questions.

From its earliest days, the research-practice partnership has been a joint effort focused on improving learning outcomes, reducing inequality in student academic achievement, and providing high-impact professional development tools to the PRDE’s leaders. At the same time, we recognized that the collaboration arrangement would need to be flexible enough to respond to the PRDE’s emerging needs.

For example, the research team has provided technical support with department-wide monitoring projects and with protocols to support online assessment during the Covid-19 pandemic. The events of the past years, including Hurricanes Irma and Maria, recurring earthquakes, and the Covid-19 pandemic, have tested this partnership’s relevance and durability. By all accounts, however, the partnership has thrived. One key aspect of its success has been its flexible nature, which has enabled it to quickly respond to emergencies and to the PRDE’s evolving priorities, while continuing its ongoing evaluation projects to inform policy decisions.

The most illustrative example of this partnership is the envisioning, implementation, and evaluation of a system-wide school principal leadership management training program entitled the Professional Development Academy for Leadership Management in Education (EDUGESPRO, for its Spanish initials). Early on, the PRDE identified professional development as a crucial lever that could improve students’ academic achievement. The team has been working ever since in close partnership with the PRDE’s Institute for Professional Development and its leader, Operations Manager Dr. Damarys Varela Vélez, to develop and evaluate state-of-the-art professional development programs for PRDE staff.

The team conducted a territory-wide survey of school principals’ leadership practices during the 2018-19 academic year and identified areas for improvement. In particular, we found that principals could benefit from strengthening strategies employed to support teachers’ data-driven planning, track their students’ and schools’ performance, and set challenging but informed goals.

Using this information, as well as best practices from the field of educational leadership, the partners designed the leadership training intervention. EDUGESPRO is a large-scale principal training program seeking to improve academic leadership and management practices in order to, in turn, improve students’ academic achievement. The program began in the summer of 2019 and will be offered to all school principals in the island’s public education system over a three-year period.

Pivoting in the face of a global pandemic

After conducting the first series of training sessions during the fall of 2019, a series of earthquakes and subsequent tremors resulted in the temporary closure of many PRDE school facilities. We quickly convened the EDUGESPRO team to chart a course forward and gave the participating principals a short recess from the training program as they worked to ensure educational continuity in their communities. We did not foresee, however, that on March 16, 2020, the PRDE would have to close all school facilities and announce that all learning would be remote until further notice due to the Covid-19 pandemic.

As these events unfolded, we realized that EDUGESPRO’s training program had the potential to support school principals during the pandemic. Therefore, while many other PRDE programs announced extended breaks, we worked with the training provider to get the entire program online in the span of three weeks. Training resumed during April 2020, and the first cohort of EDUGESPRO concluded training in the summer.

Participation rates in the program actually increased after the program became fully available online. We found that the online version eliminated the logistical challenge that attending day-long trainings posed for busy school principals, as they could log into the combined synchronous and asynchronous sessions from their workspaces. The higher participation rates assuaged our fears that principals might not attend the online programming due to the more limited opportunities to interact with peers and opened up a range of possibilities for future online professional development training initiatives.

Our EDUGESPRO team meets weekly to rapidly follow-up on program developments in order to ensure the project’s success. Not only did the program migrate online; it also became a source of emotional support for principals navigating a public health crisis. At the EDUGESPRO graduation ceremony, participating principals spoke of how this community of practice supported them as they navigated school leadership during the Covid-19 pandemic. In program-curated WhatsApp groups and discussion forums, principals shared best practices, advice, and the occasional motivational quote, video, or meme.

The importance of these communities in a time of extreme stress and limited peer support was exemplified by one participating principal’s graduation address, in which she shared that, after heart surgery, she had asked hospital nurses to set up a computer monitor so that she could participate in the online sessions from her hospital bed.

The Covid-19 pandemic increased the urgency and complexity of decisions that the PRDE must make, but the collaborative research partnership’s flexible working arrangement has helped the PRDE to navigate a number of these challenges. For example, we had scheduled a territory-wide survey of school principals as part of the EDUGESPRO evaluation for May 2020. At the PRDE’s request, we included a survey module that could provide the PRDE with rapid data on school-level responses during the Covid-19-related lockdowns.

The research collaboration is also expanding. Other PRDE units noticed the utility and positive results of the team’s work and have since requested and received technical support from the research team on a series of initiatives intended to limit learning loss stemming from Covid-19 school closures.

Leveraging preliminary results and charting the way forward

Despite the many unforeseen interruptions and adjustments, preliminary quantitative results from an evaluation of EDUGESPRO’s impact during the program’s first year show gains in many leadership and management outcomes among program participants, and the size and significance of these improvements will become clearer as more principals participate in the program. The second cohort of school principals will begin the year-long training in March 2021.

Looking ahead, the partners will identify other areas in which program impact evaluations could provide important public policy guidance. The partnership’s expanded scope has enabled us to work on additional professional development initiatives and to support the PRDE’s efforts to address their public policy priorities. Having endured multiple crises, the team remains more committed than ever to using rigorous evidence to inform policy decisions for the benefit of all students in Puerto Rico’s public education system.

Puerto Rico faces deeply rooted and complex education challenges. We formed a partnership between the Puerto Rico Department of Education (PRDE) and academic researchers to study these challenges and promote decision-making based on data, research, and evidence. Through a collaboration built on trust and a common agenda, we aim to transform the Puerto Rico education system to improve youth academic achievement.

An education system facing historical and deeply rooted challenges

Puerto Rico’s school system faces longstanding challenges that have resulted in historically poor student academic performance. In 2017, for example, National Assessment of Education Progress (NAEP) test scores showed that 85 percent of fourth graders in Puerto Rico did not demonstrate basic proficiency in mathematics, as compared to 21 percent of their peers in the nation.

These challenges were further exacerbated by the fiscal crisis and the devastation inflicted by Hurricane Maria in 2017. Owing more than $70 billion to creditors, Puerto Rico had filed for bankruptcy-like protection a few months before Maria hit. And when the hurricane made landfall in September 2017, it inflicted an estimated $90 billion worth of damage to the island.

Prior to 2017, student enrollment had been steadily decreasing due to the decade-long economic recession. Following Maria, this situation worsened as more families relocated to the US mainland. The Puerto Rico government now estimates that their school system has lost more than 125,000 students since 2014 and now serves about 307,000 students.

Addressing the education crisis

The PRDE is taking a multi-prong approach to address these challenges. In the summer of 2018, the PRDE closed over 250 schools to allow finite funds to continue to provide a full set of resources—including a full teaching staff—at every open school.

Six pillars of education reform were also introduced, including:

1. A new education bill, which passed in March 2018;
2. The creation of seven regional offices with a Superintendent leading each;
3. A clear definition of the resources that each school needs;
4. Programming priorities for academic and student services;
5. A transparent, restructured budget; and
6. A new pay scale and overall salary increase for principals and teachers.

Critically, the PRDE also turned towards an evidence-based approach to tackle important education policy questions. And an emerging partnership between the PRDE, J-PAL North America, and academic researchers served as the catalyst behind this new approach.

Foundations of a new partnership

This latest partnership stems from previous relationships with various segments of the Puerto Rican government developed over the past decade by J-PAL affiliated researchers. In 2013, Marco González Navarro (University of California, Berkeley), and I (Gustavo) first connected with the Puerto Rico Administration for Socioeconomic Development of the Family (ADSEF). This initial collaboration with ADSEF focused on identifying opportunities to evaluate programs to support individuals with low-income in finding and maintaining employment.

In November 2016, after a new governor was elected, researchers and staff from J-PAL’s Global, North America, and Latin America & the Caribbean offices visited Puerto Rico for an extensive series of meetings with the new administration.

During these meetings, the education sector was identified as a top priority in the government's agenda, with poor student outcomes and ineffective bureaucratic and management structures highlighted as major concerns.

Government leadership expressed interest in J-PAL’s evidence-based approach, and members of the administration signaled their openness to using data and evidence, as well as input and collaboration from external partners, as tools for overhauling the education system.

This initial step of listening to the new administration and understanding its priorities helped open the door to broader collaboration.

As an outcome of this engagement, I (Nicolás) was hired by J-PAL North America to be embedded as the Chief Evaluation Officer at the PRDE, where I was tasked with building evaluation capacity within the department. What was initially envisaged as a six-month position later turned into a two-year engagement.

Embedding this Chief Evaluation Officer position proved important for several reasons:

1. First, it signaled to the PRDE that we were genuinely interested in and committed to understanding the agency’s needs and priorities.
2. The position also played a key liaison function between the different parties involved in the partnership—providing timely and precise communication that could be adapted to the distinct agendas and schedules of both policymakers and researchers.
3. It was also critical to identifying opportunities to build evaluation into PRDE projects and initiatives.
4. And, finally, it set the foundation for building PRDE staff capacity in evidence-based decision making by providing guidance on how to incorporate randomized evaluations as a component of program design.

A promising research agenda

Embedding me (Nicolás) within the PRDE paved the way for better understanding the agency’s strategic goals—allowing for the development of a joint research agenda aimed at reducing inequality in student academic achievement. The research team eventually expanded to include researchers Daniela Scur (Cornell University) and Orlando Sotomayor (University of Puerto Rico Mayagüez).

Together, we identified high-priority projects where rigorous evaluation would be most useful in answering key policy questions for the PRDE. One priority is to understand the implications of recent school closures in order to build on any positive effects and identify policies to reduce potentially negative outcomes.

Another top priority is to evaluate management-level interventions seeking to improve student learning.

Management-level interventions have been shown to lead to improvements in school quality and students’ academic achievement, but the adoption of these practices in the public sector is often not sustained in the long run. The PRDE and its academic partners became interested in understanding how successful management-level interventions can be promoted sustainably and at scale.

Together with the PRDE, we designed an evaluation of EDUGESPRO—a large-scale principal training program seeking to improve managerial practices and student achievement. The program, which commenced in June 2019, will be implemented among all school principals in the island’s public education system over a three-year period.

Catalyzed by early learnings and successes, we began exploring opportunities to sustainably fund the embedded evaluation position and the growing research agenda. We were thrilled to receive an Institutional Challenge Grant Award from the William T. Grant Foundation and the Spencer Foundation in April 2019 to continue the partnership’s work.

Coupled with funding from the J-PAL State and Local Innovation Initiative and Social Science and Humanities Research Council grants from the Government of Canada, the partnership will begin implementing our research agenda in earnest in June 2019. And, critically, the Chief Evaluation Officer position at the PRDE will be sustained for another three years.

Lessons from the partnership

Among the partnership’s greatest successes are the institutional changes it forged. At the PRDE, there is now a commitment to develop and formalize internal processes to increase the use of research, and the department looks forward to creating an internal team that can lead this evaluation work in the future.

At my (Gustavo’s) institution, the University of Toronto, the Department of Economics was interested in facilitating partnership engagements between faculty members and public agencies. The academic partnership with the PRDE served as an impetus.

Today, the University is developing a policy research lab that formalizes a framework, based on the PRDE partnership, to engage faculty and public agencies and facilitate research-practice partnerships.

Reflecting back, we believe the partnership’s success hinged on: setting clear expectations; leveraging each partner’s unique expertise; and crafting a common agenda grounded on problems of practice.

This collaboration between policymakers and researchers was also the result of carefully cultivated mutual understanding—investments in building strong relationships, developing trust, identifying mutual interests, and finding a common ground led us to where we are today.

Most importantly, this partnership framework ensures that the research questions being addressed are pertinent to policymakers and, ultimately, to the people of Puerto Rico. Through this approach, we aim to ensure that evidence informs decisions and policymaking, and that those informed decisions contribute to improved education quality in Puerto Rico.