Introducing the Health Care Evaluation Toolkit: Resources for randomized evaluations in health care

Today, J-PAL North America released a new Health Care Evaluation Toolkit that provides guidance and advice for social scientists conducting randomized evaluations of health care delivery interventions, within health systems, or using health care data, leveraging case studies from the United States. The Health Care Evaluation Toolkit builds on and complements J-PAL’s research resources which provide practical advice for designing, implementing, and communicating about evaluations. In the past decade, J-PAL North America’s US Health Care Delivery Initiative has funded more than 50 randomized evaluations of health care delivery interventions to make health care delivery in the United States more efficient, effective, and equitable. Based on J-PAL North America’s extensive cumulative experience supporting randomized evaluations in the health care context, the Health Care Evaluation Toolkit contains five resources that address real-world problems that researchers face in health care contexts. Resources in the toolkit were developed collaboratively with J-PAL affiliated researchers, some of whom are highlighted in the descriptions below. 

Navigating hospital Institutional Review Boards (IRBs)

Social scientists seeking to partner with hospitals to conduct research on health care delivery may find it daunting working with a hospital IRB more accustomed to reviewing medical trials. “I’ve really benefited from hospital collaborations, but hospital IRBs can pose unique challenges. In one case, my collaborators and I wanted to develop and test e-mail alerts to clinicians. We initially struggled to figure out how to submit this work to the IRB, but a series of conversations with IRB officials helped us understand the appropriate way forward: some of our work was research subject to IRB review, and another piece was quality improvement with a different approval process. Ultimately, the project was a success, but our experience shows that budgeting extra time to learn about hospital IRBs can help researchers head off challenges later,” says Adam Sacarny (Columbia University). This resource provides guidance for how to carefully explain study procedures and impacts on important hospital constituencies like patients and providers, as well as guidance on how to approach IRB review when a project spans multiple institutions or subject areas. 

Assessing power and feasibility

Although designing well-powered studies is critical in all domains, health care interventions present particular challenges, particularly when it comes to measuring impacts on health. Health care delivery settings also present opportunities, in that implementing partners have important complementary expertise to address these challenges. “When evaluating the impact of providing fresh food and education on diabetes outcomes, we worked collaboratively with implementing partners to estimate minimal detectable effects, using data they had access to, leading to more accurate calculations,” says Marcella Alsan (Harvard University). “Partners shared data similar to the data we would use in the trial itself. We could then estimate power incorporating the impact of control variables in the same context as the study, ” says Joseph Doyle (MIT). This resource on power calculations emphasizes the importance of discussing power with partners early and often, discusses considerations for choosing health outcome measures, and provides guidance on how to think carefully about subgroup analysis and heterogeneous effects.

Designing intake and consent processes

Complex relationships between patients and providers, where patients are in potentially vulnerable positions while receiving or seeking care, create additional challenges to intake and consent in the health care context. “When evaluating the impact of a care management program on individuals with complex medical and social needs, the research team decided to obtain informed consent before randomization. While this maximized study power, it meant that enrollment staff were placed in the difficult position of having to inform vulnerable individuals that they would not be getting the intervention. In consultation with our implementing partner, we supported staff by fostering a collaborative environment in which enrollment specialists could celebrate milestones, share best practices, and support each other, and we prioritized hiring enrollment specialists from the community where the study took place,” says Amy Finkelstein (MIT). This resource uses experiences from two case studies to provide guidance for designing intake and consent processes that maximize statistical power, avoid bias, minimize health care disruption, and support enrollment staff while asking patients for consent.

Acquiring and using administrative data

Regulations from the Health Insurance Portability and Accountability Act (HIPAA) overlap with requirements for human subjects research and can be confusing to navigate when requesting data for research. This resource provides guidance on HIPAA from a research perspective, defines levels of identified health care data, and provides tips for effective data requests and understanding compliance with IRBs and DUAs.

Publishing in medical journals

Social scientists interested in publishing in medical journals may encounter unfamiliar requirements that require advanced planning. “Results from our workplace wellness evaluation were desk-rejected at one medical journal because we had pre-registered our trial in a registry that did not meet the journal's specific requirements. This experience highlighted the importance of our research team being aware of the unique publication requirements of medical journals.” says David Molitor (University of Illinois). This resource provides a checklist for publishing in a medical journal, with key steps before, during, and after study implementation. The checklist includes guidance on registering a trial in an International Committee of Medical Journal Editors (ICMJE) compliant registry, developing a trial protocol and statistical analysis plan, and writing a manuscript that follows the Consolidated Standards of Reporting Trials (CONSORT) guidelines. 

If you’ve explored the Health Care Evaluation Toolkit and have feedback to share, please reach out to us at [email protected].