Summary

This resource provides guidance on acquiring and working with administrative data for researchers working in US health care contexts, with a particular focus on how the Health Insurance Portability and Accountability Act (HIPAA) structures data access. It illustrates the concepts with examples drawn from J-PAL North America’s experience doing research in this area. This resource assumes some knowledge of administrative data, IRBs, and the Common Rule. Readers seeking a comprehensive overview of how to obtain and use nonpublic administrative data for randomized evaluations across multiple contexts should consult the resource on using administrative data for randomized evaluations. 

_{Disclaimer: This document is intended for informational purposes only. Any information related to the law contained herein is intended to convey a general understanding and not to provide specific legal advice. Use of this information does not create an attorney-client relationship between you and MIT. Any information provided in this document should not be used as a substitute for competent legal advice from a licensed professional attorney applied to your circumstances.}

Introduction

There are a number of advantages to using administrative data for research, including cost, reduced participant burden and logistical burden for researchers, near-universal coverage and long term availability, accuracy, and potentially reduced bias. For health research in particular, administrative data contains precise records of health care utilization, procedures, and their associated cost, which would be difficult or impossible to learn from surveying participants directly.¹ 

Despite these advantages, a key challenge for research in health care contexts in the United States is acquiring administrative data when researchers are outside the institution that generated the data. Protected Health Information (PHI) governed by the Health Insurance Portability and Accountability Act (HIPAA) makes health care data especially sensitive and challenging for researchers to acquire, which may cause lengthy negotiation processes and confusion about the appropriate level of protection for transferring and using the data. 

This resource covers:

• The relationships between HIPAA, human subjects research regulations, and researchers
• Levels of data defined by HIPAA 
• Important considerations for making a health care data request
• Compliance with IRBs and DUAs

What is HIPAA and how does it affect researchers?

In the United States, the Privacy Rule of the Health Insurance Portability and Accountability Act (HIPAA) regulates the sharing of health-related data generated or held by health care entities such as hospitals and insurance providers. HIPAA imposes strict data protection requirements with strict penalties and liability for non-compliance to the health care entities that it regulates (known as covered entities). These regulations allow, but do not require, sharing data for research. These regulations augment the more general research protections codified in the Common Rule—the US federal policy for the protection of human subjects that outlines the criteria and mechanisms for IRB review—sometimes in overlapping and confusing ways. 

Researchers (in most cases) are not covered entities, however researchers must understand HIPAA requirements in order to interact with data providers who are bound by them. Many of the obligations to protect data under HIPAA will be passed to the researcher and their institution through data use agreements (DUAs) executed with covered entities.² HIPAA imposes restrictions on what and how covered entities may disclose for research. HIPAA introduces distinct categories of data for health care research that researchers should understand, as they dictate much of the compliance environment a particular research project will fall under. In addition, researchers must understand how HIPAA requirements interact with requirements from the Common Rule. Note that HIPAA only applies in the United States and only to health care data; other topic areas and countries may have their own compliance standards.³

What kind of data does HIPAA cover?

The HIPAA Privacy Rule is a US federal law that covers disclosure of Protected Health Information (PHI) generated, received, or maintained by covered entities and their business associates. Covered entities are health care providers, health plans, and health care clearinghouses (data processors). Business associates include organizations that perform functions on behalf of a covered entity that may involve protected health information. In the research context, researchers may interact with business associates who work as data processors, data warehouses, or external analytics or data science teams. Data from covered entities used in research may include “claims” data from billing or insurance records, hospital discharge data, data from electronic medical records like diagnoses and doctors’ notes, or vital statistics like dates of birth. 

Protected health information (PHI) refers to identifiable health data maintained or received by a covered entity or its business associates. More specifically, the HIPAA Privacy Rule defines 18 identifiers and 3 levels of data — research identifiable data, also known as research identifiable files (RIF), limited data sets (LDS) and de-identified data — each with different requirements depending on their inclusion of these identifiers, detailed in Table 1 below. Data without any of the 18 identifiers is considered de-identified, does not contain PHI, and is therefore not restricted by HIPAA. The inclusion of any identifiers means that the data is subject to at least some restrictions. Notably, between fully identified and de-identified data, HIPAA defines a middle category — limited data — that contains some potentially-identifiable information and is subject to some but not all restrictions. As a result, the extent to which a data request must comply with HIPAA depends on the level of protected health information contained in the data to be disclosed. 

Not all health data is protected health information (PHI) governed by HIPAA. Health information gathered by researchers themselves during an evaluation is not PHI, though it would be personally identifiable information subject to human subject protections under the Common Rule.⁴ Not all secondary data providers are necessarily covered entities either. Student health records from a university-affiliated medical center, for example, are governed by the education privacy law FERPA rather than HIPAA, because FERPA offers stronger privacy protections. Researchers should confirm whether their implementing partner or data provider is a covered entity, as some organizations will incorrectly assume they are subject to HIPAA, invoking unnecessary review and regulation. For guidance ondetermining whether an organization is subject to the HIPAA Privacy Rule, the US Department of Health & Human Services (HHS) defines a covered entity and the Centers for Medicare & Medicaid Services (CMS) provides a flowchart tool.

Levels of data

Research identifiable data

Research identifiable data contain sufficient identifying information such that the data may be directly matched to a specific individual. Under HIPAA this means any identifiers (such as name, address, or record numbers) not including the dates and locations allowed in limited data sets (LDS). A list of variables that make a dataset identifiable under the HIPAA Privacy Rule can be found in Table 1. Identifiable data may only be shared for research purposes with individual authorization from each patient or a waiver of authorization approved by an institutional review board (IRB) or privacy board (a process similar to informed consent or waivers of consent as required by the Common Rule). This level of data is typically not necessary to conduct analysis for impact evaluations if a data provider or third party is able to perform data linkages on the researcher’s behalf. Researchers may need to acquire identified data to perform data linkages or because the data provider is not able or willing to remove identifiers prior to sharing the data.

Researchers receiving research identifiable files must take pains to protect data with strong data security measures and by removing and separately storing identifiers from analysis data after they are no longer needed (a J-PAL minimum must do). Researchers should expect extensive negotiations and review of their data request and to receive IRB approval and execute a DUA prior to receiving data. Protecting participant data confidentiality is a vital component of conducting ethical research. 

Limited data sets

The HIPAA Privacy Rule defines limited data sets as those that contain geographic identifiers smaller than a state (but less exact than street address) or dates related to an individual (including dates of medical service and birthdates for those younger than 90) but do not otherwise contain HIPAA identifiers. For research purposes, this geographic and time information (such as hospital admission and discharge dates) can be particularly useful for analysis. This makes limited data a particularly useful class of administrative health data that balances ease of access with ease of use. 

Limited data sets may be shared by a covered entity with a data use agreement (DUA).⁵ HIPAA does not require individual authorization or a waiver of authorization. Researchers should always seek IRB approval for research involving human subjects or their data, but it is possible that research using only a limited data set may be determined exempt or not human subjects research by an IRB if there is no other involvement of human subjects. Using a limited data set for randomized evaluations typically means that another party besides the researcher, such as the data provider or an intermediary, must link data to evaluation subjects. 

• In Health Care Hotspotting, J-PAL affiliated researchers partnered with the Camden Coalition of Healthcare Providers in New Jersey to evaluate the impact of a care transition program that provided assistance to high-cost, high-need patients with frequent hospitalizations and complex social needs, known as “super-utilizers.” Researchers executed multiple DUAs for limited data sets. As discussed above, working with hospital discharge data required matching to study participants on site in Camden, NJ before taking limited data back to MIT for analysis. For other datasets, including Medicaid data used in secondary analysis, the Camden Coalition submitted a finder file to the data provider, who matched study participants to their outcome data before delivering a dataset stripped of identifiers to the researchers. More options for data transfers where researchers cannot receive identifiers are discussed in the resource on using administrative data for randomized evaluations.

De-identified or publicly available data

De-identified data does not contain any of the 18 HIPAA identifiers and is therefore considered not to contain sufficient identifiers to link to specific individuals with certainty. HIPAA permits health care providers to share de-identified data for research purposes without further obligations (U.S. Department of Health & Human Services 2018). De-identified data is not PHI. 

Using de-identified data means that it is possible to obtain data through a much less onerous legal and contractual process, with less risk. However, this is balanced against the need for the data provider to conduct any linkages and to pre-process any information that cannot be included, such as turning dates of service into relative dates or exact locations into distances if such information is needed for analysis. 

• For the study “Clinical decision support for high-cost imaging: A randomized clinical trial,” researchers received only de-identified data from the implementing partner and data provider Aurora Health Care. This approach simplified legal compliance but necessitated additional work on the side of the data provider (and trust on the side of the researchers) to make the data usable for research, such as converting dates of service to days measured relative to the introduction of the intervention. This case is documented in significant detail in a chapter of J-PAL’s IDEA handbook
   

Considerations for making a data request

Picking a level of data

Researchers should understand what level of data is required to conduct their research. Using the least identifiable data possible will lower compliance costs and may increase the speed of negotiations and their likelihood of success. However, data without sufficient information can limit your research questions and additional data preparation may be required when working with de-identified data. Researchers should consider what is necessary to conduct their research and request what they need. Requesting limited data is often a good compromise as it balances these considerations.

The level of data is not always under researcher control. Data providers that frequently work with researchers may be able to provide essentially the same data in several formats. However data providers may also limit what is possible with each data type.

• For research using Medicare data, identifiable data is often necessary as using a unique beneficiary identifier is needed to track individuals across datasets and not all limited datasets include this information. Medicare also limits the dates allowed in limited datasets relative to the HIPAA definition. Researchers interested in working with CMS data should consult ResDAC’s documentation defining what is included in each file type.⁶
• For Health Care Hotspotting, researchers requested uniform billing data from the New Jersey Department of Health for all hospital discharges in the state related to an 800 patient trial in Camden, NJ. The health department preferred to send all data on discharges in the state, with identifiers. Researchers balanced the opportunity to conduct the match to study participants themselves with MIT’s desire not to receive identified data by having the implementing partner, the Camden Coalition of Healthcare Providers, receive the data, matching the data onsight in Camden, and only transferring the matched limited data set back to MIT. This process nonetheless required researchers to implement strong data security requirements and required a lengthy data request process.

Minimum necessary requirement

In addition to considering data in terms of the level of identification, HIPAA requires that data providers release only the minimum information “reasonably necessary to accomplish the purpose [of the research].” This restriction applies to data that contains PHI unless researchers have individual authorization from research subjects. Covered entities are required to develop their own protocols for determining what constitutes the minimum necessary, so the restrictiveness of this requirement varies significantly. Covered entities may also rely on IRB approval of a data request as evidence of meeting the minimum necessary standard. Researchers should consider the scope of a data request both in terms of the number of individuals and the number of variables requested. Minimizing the size of a request in either dimension can help smooth the data request process. Researchers should be prepared to justify the inclusion of particular variables as necessary to the research. To limit the number of individuals requested, it may be possible to request a random sample, particularly in cases where treatment is not assigned at the individual level or when requesting data for observational research.

Additional data provider requirements

Data providers may apply additional restrictions or conditions beyond what the law requires for the researcher to use their administrative data, especially for data considered “sensitive” for any reason. Practically speaking, researchers must follow whatever additional requirements data providers request. Data holders are typically under no obligation to share data. 

Researchers may be asked to demonstrate how their research will benefit the program from which they are requesting data. In some cases this may be to prioritize responding to requests for data. In others this may be legally required. Emphasizing the value of the research can be helpful in building trust and interest in any data request process. 

Researchers may be asked to submit their project for additional IRB review beyond what their own institution requests.

• In Health Care Hotspotting, the New Jersey Department of Health requires all data requests to be approved by the Rowan University IRB, which reviews projects on their behalf and which will not cede to an existing IRB.
• In an evaluation of the End Stage Renal Disease Treatment Choice model, a payment reform model to encourage greater use of home dialysis, CMS required a waiver of individual authorization issued by MIT’s IRB in order to process the request to access Medicare data via the Virtual Research Data Center (VRDC). Because MIT had initially determined the project exempt, researchers had to go back to the IRB and request a full review in order to be granted the waiver. CMS’s own Privacy Board also reviews requests for identifiable data.

Data use agreements frequently extend required protections for PHI to data elements that are not PHI, such as information about doctors or hospitals rather than patients. Researchers may also be asked to implement onerous data security requirements, whereas the Privacy Rule requires that researchers only "use appropriate safeguards," not the full list of administrative, technical, and physical safeguards the HIPAA Security Rule envisions for covered entities with complex businesses. In the experience of research staff at J-PAL North America, this has included setting up automated audit trails, intrusion detection, and vulnerability scans on research servers, which required close collaboration with IT staff and went significantly beyond standard operating procedures. 

Researchers are typically not in a strong negotiating position for reducing additional requirements. However it can be helpful to be knowledgeable about what HIPAA requires and what safeguards the research team and institution provide in order to build trust. Building trust and highlighting the value of the research to the data provider may increase the provider’s motivation to work with you. (Further consideration of each of these points is included in the resource on using administrative data). For example, particularly in cases where data providers do not frequently share data, providers may be unaware that outside researchers are not covered entities or unaware of the different requirements related to levels of identified data.⁷ Data providers who initially request security frameworks appropriate for handling health care data at enterprise scale may be satisfied with more limited measures already in place in a research computing environment, especially if researchers can speak knowledgeably about them and remain sensitive to the data provider’s concerns.⁸ 

How to receive data

How data will be received or accessed by researchers has implications for data security as well as cost. It may be possible to send data directly to a personal research server from a data provider via secure file transfer, or have the data provider send data to a data intermediary that maintains suitable infrastructure for research access. The National Bureau of Economic Research (NBER), for example, houses CMS data in use by multiple researchers. It may also be possible to access data remotely on a system maintained by the data provider instead of taking physical possession of the data.

• As an alternative to receiving actual copies of data, in an evaluation of the End Stage Renal Disease Treatment Choice model researchers chose to access Medicare data remotely, using CMS’s VRDC. This access method provides significant advantages in terms of speed of access (decreased lag time between when an event occurs and when it is reflected in the data), cost, and ease of use (responsibility for data security infrastructure remains with the data provider).

Compliance

Researchers should expect to need to receive IRB approval and execute a DUA prior to receiving data. Even de-identified data may be considered sensitive by the data provider and require a legal agreement. Researchers have their own obligation to seek IRB approval under the Common Rule and data providers may request evidence that someone external to the research team has signed off on a project and its use of data even if not required under HIPAA. In most instances, an IRB will also function as an institution’s HIPAA Privacy Board. 

IRB Approval and HIPAA Authorization

Researchers seeking access to identifiable data can do so with either individual authorization or a waiver of authorization. In either case, an IRB serves to document and approve the authorization form or the waiver. 

An authorization to release protected health information is a signed record of an individual’s permission to allow a HIPAA Covered Entity to use or disclose their Protected Health Information (PHI). The authorization must describe the information requested and the purpose, and must be written in plain language that is readily understood by the individual. This is similar to the concept of informed consent, and is often embedded within an informed-consent document. However, an authorization has a distinct set of criteria and may be a separate written agreement obtained outside of the informed-consent process. Researchers should consult their IRB for specific guidelines or templates. If possible it is also recommended to confirm with a data provider that a planned authorization will be sufficient to access data. 

The waiver standard is likewise quite similar to the waiver of individual consent under the Common Rule. Research must be no more than minimal risk, could not be accomplished without the protected health information being requested, and could not practically be accomplished without the waiver. In other words, it would not be possible to contact subjects to seek their consent, which may be the case in randomized evaluations where researchers are not involved in the delivery of the intervention, where the program does not interact with all research subjects — such as an informational intervention or encouragement design that does not contact the control group — or in observational research.

For researchers seeking a limited data set, HIPAA only requires a DUA. However, researchers (who are not covered entities and therefore bound by the Common Rule not the HIPAA Privacy Rule) must still seek IRB approval for human subjects research. In the case that research involves only limited data and not other contact with human subjects, it may be determined exempt under category 4 (ii) by an IRB because limited data is not “readily identifiable,” despite the inclusion of PHI in the form of dates and locations. Research involving only de-identified data is not considered human subjects research. Note that in the context of running randomized evaluations, there are likely elements of the project beyond the data set that would require human subjects compliance. When in doubt about what constitutes human subjects research, researchers should consult their institution’s IRB. IRB approval is a minimum must do for all J-PAL projects.

Data Use Agreement (DUA)

A data use agreement outlines the allowable uses of the data to be transferred and the requirements the receiving institution must adhere to for protecting the data. A DUA is a legal contract executed by the institutions sharing and receiving data. DUAs should be signed by a legal representative of the institution, not the researcher.

HIPAA lists specific requirements for what a data use agreement must include. In particular, recipients must agree to the following:

• "Not to use or disclose the information other than as permitted by the data use agreement or as otherwise required by law; 
• Use appropriate safeguards to prevent the use or disclosure of the information other than as provided for in the data use agreement;
• Report to the covered entity any use or disclosure of the information not provided for by the data use agreement of which the recipient becomes aware; 
• Ensure that any agents, including a subcontractor, to whom the recipient provides the limited data set agrees to the same restrictions and conditions that apply to the recipient with respect to the limited data set; and
• Not to identify the information or contact the individual.”

Data providers may impose additional restrictions or safeguards. However, the DUA should also make clear the freedom of the researcher to publish the results of their work without interference from the data provider, except for reasonable review of publications to ensure no accidental disclosures.

Rather than construct a DUA from scratch, researchers should make use of an existing DUA template already in use by one of the parties and approved by their legal office. Universities will have standardized forms and many data providers, particularly if they share data frequently, will as well. For example, CMS’s DUA templates are linked on the ResDAC website. Despite a large degree of standardization, researchers should be aware that DUA negotiations can often take significant time (months, if not more) and plan for these delays. 

For guidance on formulating a data request and executing a data use agreement, please see the resource on using administrative data for randomized evaluations.

Relationship between IRB and DUA

Simultaneously seeking IRB approval and executing a data use agreement often generates a circular problem where the IRB wants to approve a research protocol that includes a data use agreement and the data provider wants to execute a data use agreement with researchers that have already received IRB approval. IRBs want to review data use agreements, as they contain details about what information will be acquired about study participants and how that information will be handled. Data providers, in contrast, often request proof that IRB approval has been received, before beginning negotiations. 

It may be possible to get provisional approval from one party to move forward with approval from the other party and then iterating between the two, as noted in the resource on using administrative data for randomized evaluations. In practice, this has typically meant first seeking IRB approval, providing as much detail as possible on the data request process, using that approval to execute the DUA, and then submitting the DUA (and any additional details, as necessary) as an amendment to the IRB protocol. 

If an IRB does in fact require documentation from a data provider, it may be possible to submit the data request application materials, which will often detail the purpose of the data for the research, the specific variables requested for the study, specify how data will be linked and potentially de-identified, and explain the data security plan, which are the necessary elements that an IRB will consider when making a determination about a research project. If the IRB requests approval from a data provider that does not yet exist, it may be possible to produce a letter of intent or support from the data provider, indicating general support for providing data to the research project, with the specifics of the data transfer subject to further negotiation and the execution of the DUA. (An example of such a letter is included as an attachment). 
 

_{Acknowledgments: We are grateful to Catherine Darrow, Amy Finkelstein, Laura Ruiz, Lisa Turley Smith, and Fatima Vakil for their insight and advice. This resource is heavily indebted to the creators of J-PAL’s original and more expansive guide to using administrative data for randomized evaluations: Laura Feeney, Jason Bauman, Julia Chabrier, Geeti Mehra, and Michelle Woodford. Amanda Buechele copyedited this document. Creation of this resource was supported by the National Institute On Aging of the National Institutes of Health under Award Number P30AG064190. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health}

Summary

This resource provides guidance to avoid challenges when partnering with a hospital to implement a randomized evaluation and working with the hospital's institutional review board (IRB), which may be more familiar with medical research than social science evaluations. Conducting randomized evaluations in these settings may involve logistical considerations that researchers should be aware of before embarking on their experiment. Topics include how to write a protocol to avoid confusion, objections, and delay by taking pains to thoroughly explain a study design and demonstrate thoughtful approaches to randomization, risks and benefits, and how research impacts important hospital constituencies like patients, providers, and vulnerable groups. The resource also addresses how to structure IRB review when work involves multiple institutions or when research intersects with quality improvement (QI) in order to reduce concerns about which institution has responsibility for a study and to minimize administrative burden over the length of a study. For a more general introduction to human subjects research regulations and IRBs, including responding to common concerns of IRBs not addressed here, please first consult the resource on institutional review board proposals.

The challenge

Randomized evaluations, also known as randomized controlled trials (RCTs), are increasingly being used to study important questions in health care delivery, although the methodology is still not as commonly used in the social sciences as it is in studies of drugs and other medical interventions (Finkelstein 2020; Finkelstein and Taubman 2015). Economists and other social scientists may seek to partner with hospitals to conduct randomized evaluations of health care delivery interventions. However, compared to conducting research at their home institutions, it may seem as if some hospital IRBs may have a more lengthy process for reviewing social science RCTs. While all IRBs follow the guidelines set forth in the Common Rule, differences in norms and expectations between disciplines may complicate reviews and require clear communication strategies. 

Lack of familiarity with social science RCTs

Hospital IRBs, accustomed to reviewing randomized evaluations in the context of clinical trials, may review social science RCTs as if they were comparatively higher risk clinical trials, while researchers may feel that social science RCTs are generally less risky. These IRBs may be more accustomed to reviewing clinical trials, research involving drugs or devices, or more intensive behavioral interventions. Thus, a protocol that mentions randomization may prompt safety concerns and lead to review processes required for trials that demand more scrutiny and risk mitigation.

Randomization 

Randomization invites scrutiny on decisions about who receives an intervention, as well as scrutiny of the intervention itself. This can cause a disconnect for the researcher when trying to justify the research, or trigger assumptions that reviewers may have regarding ethics. Is the researcher denying someone something they should be able to receive? Unlike a drug or device that requires testing, it may not be clear to a hospital IRB why an intervention, for instance, to increase social support or to provide feedback on prescribing behavior to physicians cannot be applied to everyone. 

Researchers being external to the reviewing IRB organization

Although not a challenge inherent to the field of health care delivery, partnering with a hospital to conduct research may involve researchers interacting with a new IRB. This presents challenges when IRBs are unaware of, or unable to independently assess the expertise of researchers, may not understand the division of labor between investigators at the hospital and elsewhere, or may not be clear on where hospital responsibility for a project starts and ends. This may also pose logistical challenges if it limits who has access to IRB software systems. 

Despite these challenges, researchers affiliated with J-PAL North America have successfully partnered with hospitals to conduct randomized evaluations. 

• In Clinical Decision Support for High-cost Imaging: A Randomized Clinical Trial, J-PAL affiliated researchers partnered with Aurora Health Care, a large health care provider in Wisconsin and Illinois to conduct a large-scale randomized evaluation of a clinical decision support (CDS) system on the ordering of high-cost diagnostic imaging by healthcare providers. Providers were randomized to receive CDS in the form of an alert that would appear within the order entry system when they ordered scans that met certain criteria. Aurora Health Care’s IRB reviewed the study and approved a waiver of informed consent for patients, the details of which are well documented in the published paper. 
• In Health Care Hotspotting, J-PAL affiliated researchers partnered with the Camden Coalition of Healthcare Providers in New Jersey and several Camden area hospitals (each with their own IRB) to evaluate the impact of a care transition program that provided assistance to high-cost, high-need patients with frequent hospitalizations and complex social needs, known as “super-utilizers.” The study recruited and randomized patients while in the hospital, although the bulk of the intervention was delivered outside of the hospital after discharge. Each recruiting hospital IRB reviewed the study.
• In Prescribing Food as Medicine among Individuals Experiencing Diabetes and Food Insecurity, J-PAL affiliated researchers partnered with the Geisinger Health System in Pennsylvania to evaluate the impact of the Fresh Food Farmacy (FFF) “food-as-medicine” program on clinical outcomes and health care utilization for patients with diabetes and food insecurity. A reliance agreement was established between MIT and the Geisinger Clinic, with the Geisinger Clinic’s IRB serving as the reviewing IRB.
• In Encouraging Abstinence Behavior in a Drug Epidemic: Does Age Matter? J-PAL affiliated researchers partnered with DynamiCare Health and Advocate Aurora Health to evaluate whether an app-based abstinence incentive program through a mobile application is effective for older adults with opioid use disorders. Advocate Aurora Health reviewed the study, and a reliance agreement was not needed, as it was determined that the co-PIs at the University of California, Santa Cruz and the University of Chicago were not involved in human subjects research. 

The following considerations are important to ensure positive relationships, faster reviews, and increased likelihood of approval when working with hospital IRBs:

Considerations for the institutional arrangement

Partner with a PI based at the IRB's institution

RCTs generally involve partnerships between investigators and implementing partners. Hospitals differ from typical implementing partners in that they are research institutions with their own IRBs. Consequently, hospital partnerships often require partnership with a researcher based at that hospital. Therefore, if you want to work with a hospital to conduct your research, plan to partner with a PI based at that hospital. Beyond the substantive knowledge and expertise that a hospital researcher brings, there may be practical benefits for the IRB process. This includes both institutional knowledge and the mundane task of clicking submit on an internal IRB website. Procedures for approving external researchers vary from hospital to hospital and having an internal champion in this process can be invaluable. 

Building a relationship with a local PI as your co-investigator can be beneficial for a number of reasons: utilizing content expertise during the research design, addressing the alignment of the research goals with the mission, interest, and needs of the hospital, and also in navigating IRB logistics.  A local PI may already have knowledge of nuances to consider when designing a protocol and submitting for IRB review. 

• In the evaluation of clinical decision support, J-PAL and MIT-based researchers partnered with Aurora-based PI, Dr. Sarah Reimer, who played a key role in getting the study approved. Aurora requires all research activities to obtain Research Administrative Preauthorization (RAP) before submitting the proposal to the IRB for approval. Dr. Reimer ensured the research design made it through the formal steps of both reviews. She also helped with the approval process allowing the external J-PAL investigators to participate in the research, which included a memo sent to Aurora’s IRB explaining their qualifications, experience, and role on the project. 
• Challenges in collaboration are always a possibility. In Health Care Hotspotting, researchers initially partnered with Dr. Jeffrey Brenner, the founder of the Camden Coalition and a clinician at the local hospitals. When Dr. Brenner left the Coalition and the research project, the research team needed to find a hospital-based investigator in order to continue managing administrative tasks like submitting IRB amendments. 

Decide which IRB should review

Multi-site research studies (including those with one implementation site but researchers spread across institutions, if all researchers are engaged in human subjects research) can generally opt to have a single IRB review the study with the other IRBs relying on the main IRB. In fact, this is required for NIH-funded multi-site trials. Implementation outside of the NIH-required context is more varied. 

All else equal, it may be easier to submit and manage an IRB protocol at one’s home university, with partnering hospitals ceding review. However, not all institutions will agree to cede review. This seems to be the case particularly when most activities will take place at another institution (e.g., you are the lead PI based at a university but recruitment is taking place at a hospital and the hospital IRB does not want to cede review). Sometimes hospitals may require their own independent IRB review, which results in the same protocol being reviewed at two (or more) institutions. Understanding each investigator’s role and level of involvement in the research, along with the research activities happening at each site is crucial when thinking about a plan to submit to the IRB. Consult the IRBs and talk to your fellow investigators about how to proceed with submitting to the IRB and the options for single IRB review. For long-term studies, consider how study team turnover and the possibility of researchers changing institutions will affect these decisions.

• In Health Care Hotspotting, all four recruiting hospitals required their own IRB review. MIT, where most researchers were based, ceded review to Cooper University Hospital, the primary hospital system where most recruitment occurred. Cooper has since developed standard operating procedures for working with external researchers and determining who reviews. Other institutions may have similar resources. Maintaining separate approvals proved labor intensive for research and implementing partner staff.
• Reliance agreements can be easier with fewer institutions. In Prescribing Food as Medicine among Individuals Experiencing Diabetes and Food Insecurity, a reliance agreement was established between MIT (where the J-PAL affiliated researcher was based) and the Geisinger Health System. MIT ceded to Geisinger, since recruitment and the intervention took place at Geisinger affiliated clinics. 

We recommend single IRB review whenever possible, because a single IRB will drastically reduce the effort required for long term compliance, such as submitting amendments and continuing reviews. While setting up single IRB review initially may not be straightforward,  doing so will pay dividends over the length of the project. 

If a researcher at a university wants to work with a co-investigator at a hospital and have only one IRB oversee the research, a reliance agreement or institutional authorization agreement (IAA) will be required by the IRBs. Researchers select one IRB as the ‘reviewing IRB’, which provides all IRB oversight responsibilities. The other institutions sign the agreement stating that they agree to act as ‘relying institutions’ and their role is to provide any conflicts of interest, the qualifications of their study team members, and local context if applicable but otherwise cede review. 

SMART IRB is an online platform aimed at simplifying reliance agreements, funded through the NIH National Center for Advancing Translational Sciences. NIH now requires a single reviewing IRB for NIH-funded multisite research. Many institutions use SMART IRB, however some may use their own paper based systems or use a combination of both (e.g., SMART IRB for medical studies that require it, paper forms for social science experiments). An example of a reliance agreement form can be found here. The process of getting forms signed and sent back and forth between institutions can take time. If you are planning to collaborate with a researcher at another institution, plan ahead by looking into whether they have SMART IRB and talk to other researchers about their experiences with obtaining a reliance agreement via traditional forms. 

Anticipate roadblocks to reliance agreements to save yourself time in the future and avoid potential compliance issues. Explain your roles and responsibilities in the protocol, check to see if your institution(s) use SMART IRB, discuss with your co-investigator(s) which institution should be the relying institution, and talk to your IRB if you are unsure whether a reliance agreement will be needed. 

Understand who is “engaged” in research

It is fairly common for research teams to be spread across multiple universities; however, not all universities may be considered engaged in human subjects research. Considering which institutions must approve the research can cut down on the number of approvals needed, even in cases of single IRB review.

An institution is engaged in research if an investigator at that institution is substantively involved in human subjects research. Broadly, “engagement” involves activities that merit professional recognition (e.g., study design) or varying levels of interaction with human subjects, including direct interaction with subjects (e.g., consent, administering questionnaires) or interaction with identifiable private information (e.g., analysis of data provided by another institution). If you have co-investigators who are not involved in research activities with subjects and do not need access to the data, even if they are collaborators for authorship and publication purposes, reliance may not be necessary (See Engagement of Institutions in Human Subjects Research, Section B 11). To learn more about criteria for engagement, please review guidance from the Department of Health & Human Services. When in doubt, an IRB may issue a determination that an investigator is “not engaged” in human subjects research. 

• In Treatment Choice and Outcomes for End Stage Renal Disease, the study team was based at MIT, Harvard, and Stanford. Data access was restricted such that only the MIT-based team could access individual data. The study design, analysis, and paper writing were collaborative efforts, but the IRBs determined that only MIT was engaged in human subjects research. 
• One J-PAL affiliated researcher experienced a weeks-long exchange when trying to set up a reliance agreement between their academic institution and their co-PI’s hospital. As co-PIs they designed the research protocol together. The plan was for recruitment and enrollment of patients into the study to take place at the hospital where the co-PI was situated and for our affiliate to lead the data analyses. The affiliate initiated the reliance agreement, but because the study was taking place at the hospital and our affiliate would not actually be involved in any of the direct research activities (e.g., recruitment, enrollment), their IRB questioned whether their level of involvement even qualified as human subjects research and subsequently did not see the need for a reliance agreement. Our affiliate provided their IRB with justification of their level of involvement in the study design and planned data analyses. An academic affiliation was also set up for the J-PAL affiliate at the co-PI’s hospital for data access purposes. Because the study received federal funding, the investigators emphasized the need for compliance with the NIH’s single IRB policy. Ultimately, the reliance agreement was approved. 

Determine whether your study is Quality Improvement (QI)

Quality improvement (QI) is focused on evaluating and identifying ways to improve processes or procedures internally at an organization. Because the focus is more local, the goals of QI differ from research, which is defined as “a systematic investigation… to develop or contribute to generalizable knowledge” (45 CFR 46.102(2)). Instead, QI has purposes that are typically clinical or administrative. QI projects are not human subjects research and are not subject to IRB review. However, hospitals typically have an oversight process for QI work, including making a determination that a project is QI and thus not human subjects research. If you are doing QI, it is crucial to follow the local site’s procedures, so make sure to discuss your work with the IRB or, if separate, the office responsible for QI determinations.

Quality improvement efforts are common in health care. In some cases, researchers may be able to distinguish between the QI effort and the work to evaluate it. In such work, the QI would receive a determination of “not human subjects research” and would not be overseen by the IRB, whereas a non-QI evaluation would be considered research and would receive oversight from the IRB. If the evaluation only involved retrospective data analysis, it might be eligible for simplified IRB review processes, such as exemption or expedited review. 

Quality improvement projects are often published as academic articles. The intent to publish does not mean your QI project fits the regulatory definition of research (45 CFR 46.102(d)) and should not, on its own, hinder you from pursuing a QI project. If your QI project also has a research purpose, then regulations for the protection of human subjects in research may apply. In this case, you should discuss with the IRB and/or the office responsible for QI determinations. The OHRP has an FAQ page on Quality Improvement Activities that can help you get a better understanding of the differences between QI and research.  

Understanding QI matters because hospital-based partners may feel that a project does not require IRB review. While a project may be intended to evaluate and identify ways to improve processes or procedures, there may be research elements embedded in the design. If so, then your project must be reviewed by an IRB, per federal regulations. If you are unsure if your project can be classified as QI, human subjects research, or both, talk to your IRB and/or the office that makes QI determinations. 

Additionally, there may be funder requirements to be mindful of when it comes to IRB review. For example, J-PAL’s Research Protocols require PIs to obtain IRB approval, exemption, or a non-human subjects research determination. 

• The Rapid Randomized Controlled Trial Lab at NYU Langone, run by Dr. Leora Horwitz, does exclusively QI, because the lab prioritizes the speed of projects. By limiting the projects to just QI and certain types of evaluations, the lab is able to complete projects within a timespan of weeks. Researchers have investigated non-clinical improvements to the health system, such as improving post-discharge telephone follow-up.
• Lessons from Langone: QI projects may not need IRB review, but that doesn’t mean ethical considerations should not be taken seriously. Before getting started, the researchers discussed the approach with their IRB, who then determined that their projects qualified as QI. Two hallmarks of QI work include: no collection of personal identifiers, since this is generally unnecessary in evaluation of an effect, and the prioritization of oversubscribed interventions to avoid denying patients access to a potentially beneficial program (Horowitz, et al 2019).

Quality improvement should not be undertaken to avoid ethical review of research. As the researcher, it is your obligation to be honest and thorough in designing your protocol and assessing the roles human subjects will play and how individuals may be affected. 

Do not outsource your ethics! Talk to your IRB about your options for review if you think your protocol involves both QI and research activities. It is always possible to submit a protocol, which may lead to a determination that a project is not human subjects research, or exempt from review. 

Consider splitting up your protocol

A research study may involve several components, some of which may qualify for exemption or expedited review while others will not. For example, researchers may be interested in accessing historical data (retrospective chart review, defined below) to inform the design of a prospective evaluation. In this case, the research using historical data could be submitted to the IRB as one protocol, likely qualifying for an exemption, while the prospective evaluation could later be submitted as a separate protocol. Removing some elements of the project from the requirements of ongoing IRB review may be administratively beneficial and doing so allows some of the research to get approved and commence before all details of an evaluation are finalized. Similar strategies may be used when projects involve elements of QI. An IRB may also request that projects be split into separate components. 

Considerations for writing the protocol

The following sections contain guidance for how to write the research protocol to avoid confusion and maintain a positive relationship with an IRB, but these points also relate to important design considerations for how to design an ethical and feasible randomized evaluation that fits a particular context. For more guidance on these points from a research design perspective consult the resource on randomization and the related download on real-world challenges to randomization and their solutions. 

Use clear language and default to overexplaining

IRB members reviewing your protocol are likely not experts in your field. Explain your project so that someone with a different background can understand what you are proposing. If you have co-investigators who have previously submitted to hospital IRBs, consult with them on developing the language for your protocol. You should also consult implementing partners in order to sufficiently describe the details of your intervention. Because IRBs only know as much as you tell them, it is important to be diligent when explaining your research. By being thorough and anticipating questions in the protocol, you are leaving less ground for confusion and questions that may arise and elongate the process of review and approval. Questions from the IRB aren’t necessarily bad or indicative that you won’t receive approval, however providing ample explanation is a way to proactively answer questions that may come up during review, therefore potentially saving time when waiting to hear back about the determination. 

Be mindful of jargon and specific terminology

The IRB may imbue certain words with particular meaning. 

• “Preparatory work,” for example, may be interpreted as a pilot or feasibility study and the IRB may instruct you to modify your application and specify that your study is a pilot in the protocol and informed consent. If you are not actually conducting a pilot, this can lead to confusion and delay. 
• Similarly, mentioning “algorithms” may suggest an investigation of new algorithms to determine a patient’s treatment, prompting an IRB to review your work like it was designed to test a medical device. 
• In the medical world, IRBs often refer to research with historical data as retrospective chart review. Chart review is research requiring only administrative data — materials, such as medical records, collected for non research purposes — and no contact with subjects. Chart reviews can be retrospective or prospective, although many IRBs will grant exemptions only to retrospective chart reviews. Prospective research raises the question of whether subjects should be consented, and prospective data collection may require more identifying information to extract data. Proposals therefore require additional scrutiny. 
• If working only with data where the “identity of the human subjects cannot readily be ascertained” (including limited data sets subject to a DUA) these studies can be determined exempt under category 4(ii) (45 CFR 46.104). If chart review involves data that is identified, this still qualifies for expedited review (category #5) if minimal risk. In this case researchers are trading off a longer IRB process for greater freedom with how to code identifiers.

IRBs may provide glossaries or lists of definitions that are helpful to consult when writing a protocol. NIA also maintains a glossary of clinical research terms. 

Carefully consider and explain the randomization approach.

Typical justifications for randomization are oversubscription (i.e., resource constraints or scarcity), where demand exceeds supply, or equipoise. Randomization given oversubscription ensures a fair allocation of a limited resource. Equipoise is defined as a “state of genuine uncertainty on the relative value of two approaches being compared.” In this case, randomization for the sake of generating new knowledge is ethically sound. 

Careful explanation of the motivations behind randomization and the approach are crucial in making sure that both the IRB and participants understand they are not being deprived of care that they would normally be entitled to. 

• In Health Care Hotspotting, eligible patients were consented and randomized into an intervention group that was enrolled in the Camden Core Model and a control group that received the usual standard of care (a printed discharge plan). Great care was taken to explain to patients that no one would be denied treatment that they were otherwise entitled to as a result of participating or not participating in the study. Recruiters emphasized that participation in the study would not affect their care at the hospital or from their regular doctors. However, the only way to access the Camden program was through the study and randomization. More detail on this intake process is in the resource designing intake and consent processes in health care contexts.

Different design decisions may reduce concerns about withholding potentially valuable treatments. Strategies such as phasing in the intervention, so all units are eventually treated but in a random order; randomizing encouragement to take up the intervention rather than randomizing the intervention itself; and randomizing among the newly eligible when expanding eligibility can all be considered during study design. For more details on these strategies, refer to this resource on randomization. 

Explain risks and benefits

Behavioral interventions may not pose the risk of physical harm that drug or medical interventions might, however social scientists still must consider the risks participants face. Using data (whether administrative or collected by researchers) involves the risk of loss of privacy and confidentiality, including the possibility of reidentification if anonymized data is linked back to individuals. The consequences of these risks depend on the data content, but health data is generally considered sensitive. Risks beyond data-related ones do exist and must be considered. Are there physical, psychological, or financial risks that participants may be subjected to?

Do not minimize the appearance of risks. Studies are judged on whether risks are commensurate with the expected benefits. They need not be risk free. Being thoughtful in the consideration of risks is more effective than asserting their absence.

Don’t outsource your ethics; as the researcher, you have an obligation to consider risks and whether they are appropriate for your study. Researchers should ensure that they consider a study to be ethical before they attempt to make the case to someone else like an IRB. Many but not all behavioral studies will be minimal risk. Minimal risk does not mean no risk, but rather a level of risk comparable to what one would face in everyday life (45 CFR 46. 102(i)). Such risks, for example, may include taking medication that has the risk of side effects. Minimal risk studies may be eligible for expedited review, as detailed in this guidance set forth by the Office for Human Research Protections (OHRP). If you do not plan to use a medical drug or device intervention in your research, clearly distinguishing your study from medical research may be helpful in ensuring that your research is reviewed commensurate with risks. If you have a benign behavioral intervention* that is no more than minimal risk to participants, use those terms and justify your reasoning (*please see section 8. in Attachment B - Recommendations on Benign Behavioral Intervention for a more detailed definition of what constitutes a benign behavioral intervention). This will put your study in a shared framework of understanding. Check your IRB’s website for a definitions page and contact your IRB if you have any clarifying questions. 

• In Encouraging Abstinence Behavior in a Drug Epidemic: Does Age Matter?, the intervention is administered using an app for a mobile device. Through the app, participants will receive varying incentive amounts for drug negative saliva tests. Participants must have a smartphone with a data plan in order to run the application. There is a potential risk of participants incurring extra charges on their phone plan if they exceed their data limit while uploading videos to the app. To address this risk, researchers made sure that the option in the app settings to “Upload only over Wi-Fi” was known to participants so that they can upload their videos without using mobile data and risking extra charges. 

There is no requirement that your study must pose no more than minimal risk. Studies that are greater than minimal risk will face full institutional review board review. You must also explain the measures you plan to take to ensure that protections are put into place for the safety and best interests of the participants.

All studies must report adverse or unanticipated events or risks. An example of an adverse event reporting form can be found here (under “Protocol Event Reporting Form”). However, for studies that are greater than minimal risk, your IRB may ask you to propose additional safety monitoring measures. It is possible that hospital IRBs with a history of reviewing proposals for clinical trials are more accustomed to the need for safety monitoring plans and may expect them even from social science trials that do not appear risky. 

• In Health Care Hotspotting, where the only risk was a breach of data confidentiality, weekly project meetings involved a review of data management procedures with staff as well as discussion of any adverse events. The hospital IRB submission prompted researchers to detail these plans. 

Additionally, depending on the level of risk and funder requirements, you may encounter the need for a Data and Safety Monitoring Board (DSMB) to review your study. A DSMB is a committee of independent members responsible for reviewing study materials and data to ensure the safety of human subjects and validity and integrity of data. National Institutes of Health (NIH) policy requires DSMB review and possible oversight for multi-site clinical trials and Phase III clinical trials funded through the NIH and its Institutes and Centers. The policy states that monitoring must be commensurate with the risks, size, and complexity of the trial. For example, a DSMB may review your study if it involves a vulnerable population or if it has a particularly large sample size. The DSMB will make recommendations to ensure the safety of your subjects or possibly set forth a frequency for reporting on safety monitoring (e.g., submit a safety report every six months). Be sure to check your funder’s requirements and talk to your IRB about safety monitoring options if you think your study may be greater than minimal risk.

Address the inclusion of vulnerable populations

The Common Rule (45 CFR 46 Subparts B, C, and D) requires additional scrutiny of research on vulnerable populations, specifically pregnant people, prisoners, and children. This stems from concern around additional risks to the person and fetus (in the case of pregnancy) and the ability of the individual to voluntarily consent to research (in the case of those with diminished autonomy or comprehension).

The inclusion of vulnerable populations should not deter you from conducting research, whether they are the population of interest, a subgroup, or included incidentally. Although it may seem easier to exclude vulnerable individuals, relevant statuses such as pregnancy may be unknown to the researcher and unnecessary exclusions may limit the generalizability of findings. In addition to generalizability of findings, diversifying your research populations is important to avoid overburdening the same groups of participants. If you are considering the inclusion of vulnerable populations in your research, brainstorm safeguards that can be implemented to ensure vulnerable populations can safely participate in the research if there are additional risks. In some cases (including if funders require), you may need a Data Safety and Monitoring Board (DSMB) to review your research and provide safety recommendations.

• One J-PAL affiliated researcher proposed a study with pregnant women as the population of interest. Although the researcher was working with a vulnerable population, the study itself was a benign behavioral intervention involving text messages and reminders of upcoming appointments. Thorough explanation of the research activities and how the vulnerable population faced no more than minimal risk led to the study being approved without a lengthy review process. 

• In the evaluation of clinical decision support, researchers argued that it was important to understand the impact of CDS on advanced imaging ordering by providers for all patients, including vulnerable populations. Although not explicitly targeted for the study, vulnerable populations should not be excluded from having this benefit. Furthermore, the researchers noted that identifying vulnerable populations for the purposes of exclusion would result in an unnecessary loss of their privacy. 

Consider the role of patients even if they are not the research subjects

In a hospital setting, patient considerations will be paramount, even if they are not the direct subject of the intervention, unit of randomization, or unit of analysis. It therefore behooves researchers to address them directly, even if they are not research subjects. 

• In the evaluation of clinical decision support (CDS), healthcare providers were randomly assigned to receive the CDS. The provider was the unit of randomization and analysis, and the outcome of interest was the appropriateness of images ordered. The IRB considered potential risks to patients in terms of data use and patient welfare. The researchers proposed and the IRB approved a waiver of informed consent based on the following justification: the CDS could be overridden, ensuring providers maintained professional discretion in which images to order, which contributed to a determination of minimal risk to the patients. Further, obtaining consent from patients would not be practicable, and may have increased risk by requiring the collection of personally identifiable information that was not otherwise required for the study. You can learn more about activities requiring consent and waivers in this research resource.

These considerations boil down to communication of details and proper explanation in the protocol.

Conclusion

While social scientists may be initially hesitant, collaborating with hospitals to conduct randomized evaluations is a great way to increase the number of RCTs conducted in health care delivery research. Understanding levels of risk, writing a thorough protocol, strategizing for effective collaboration, and considering the role of quality improvement (QI) are important when planning for a submission to a hospital IRB. 

_{Acknowledgments: This resource was a collaborative effort that would not have been possible without the help of everyone involved. A huge amount of thanks is owed to several folks. Thank you to the contributors, Laura Feeney, Jesse Gubb, and Sarah Margolis; thank you to Adam Sacarny for taking the time to share your experiences and expertise; and thank you to everyone who reviewed this resource: Catherine Darrow, Laura Feeney, Amy Finkelstein, Jesse Gubb, Sarah Margolis, and Adam Sacarny.}  

_{Creation of this resource was supported by the National Institute On Aging of the National Institutes of Health under Award Number P30AG064190. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.} 

Summary

This resource is intended for researchers in the social sciences who are considering publishing their randomized evaluation in a medical journal. Publishing in a medical journal allows researchers to share their study with researchers and practitioners in different disciplines and expand the reach of their findings to those outside of their immediate professional network. This resource highlights journal guidelines and requirements that may be unfamiliar to those used to publishing in the social sciences. Most medical journals are members of the International Committee of Medical Journal Editors (ICMJE) and follow its recommendations and requirements for publishing.¹ Some of the requirements for publishing necessitate action before a study has even started. While most of the content in this checklist is relevant across all medical journals, certain elements may differ from journal to journal. Researchers are encouraged to consult requirements listed in this resource in conjunction with the requirements of their specific journal(s) of interest, as well as requirements from funders and universities that could influence the publication process.  

Key steps for publishing a randomized evaluation in a medical journal:

• Before participant enrollment begins, register a trial in an ICMJE compliant registry such as clinicaltrials.gov. Failure to do so may prevent publication in most medical journals. Note that the American Economic Association (AEA) registry is not an ICMJE compliant registry. 
• Upload a statistical analysis plan (SAP) to the trial registry prior to the start of the study. Although not strictly a requirement for most medical journals, publically archiving a SAP in advance will allow greater flexibility in presenting and interpreting causal results. In particular, ensure primary vs. secondary outcomes and analysis methods are prespecified in the SAP. 
• Draft a manuscript according to journal requirements. Medical journals require papers to contain a limited number of figures and tables and adhere to specific formats and short word counts. Consult the Consolidated Standards of Reporting Trials (CONSORT) guidelines used by most journals as well as individual journal requirements before drafting your manuscript. 
• Be aware of embargo and preprint policies. Some journals may de-prioritize study publication if study results are public or shared through a working paper or preprint. However, other journals publicly state they will not penalize such papers. Journals also impose restrictions about what can be shared after acceptance and prior to publication. Make sure to check the journal’s specific guidelines and adhere to these rules.

​​1. Before starting an RCT

Register the trial before enrollment begins

The ICMJE defines a clinical trial as, “any research study that prospectively assigns human participants or groups of humans to one or more health-related interventions to evaluate the effects on health outcomes.” If a randomized evaluation meets the definition of a clinical trial, researchers must register it in an ICMJE compliant registry before the first participant is enrolled. If researchers fail to meet this condition, they risk having their paper rejected by medical journals. Note that registration on the AEA’s RCT Registry (supported by J-PAL) does not meet this requirement. 

Not every randomized evaluation that researchers submit to a medical journal will meet the ICMJE definition of a clinical trial and require registration. However, if it is unclear whether a study meets the definition of a clinical trial, follow the registration guidelines for clinical trials as a cautionary measure. This way if a journal editor feels that a study is a clinical trial, researchers will have met the registration requirement. 

Some scenarios where registration prior to enrollment may be unnecessary (although different journals may reach different conclusions) include the following: 

1. Secondary analysis of a clinical trial does not need to be registered separately. Instead, researchers should reference the trial registration of the primary trial. 
   • Example: In Health Care Hotspotting, researchers conducted a randomized evaluation of a care transition program’s impact on hospital readmissions for those with complex needs. In a related secondary study, the same research team is analyzing the effect of the program on outpatient utilization. Researchers referenced the original study’s registration instead of separately registering the secondary analysis.  
2. Analysis of a clinical trial implemented by someone else without researcher involvement is also considered a secondary analysis and does not need to be registered separately. Researchers may reference the existing clinical trial record, if one exists. As a best practice, researchers may also register a secondary analysis separately.
   • Example: In The Spillover Effects of a Nationwide Medicare Bundled Payment Reform, independent researchers conducted a secondary analysis of a randomized evaluation implemented by the Centers for Medicare and Medicaid Services (CMS). This trial did not have an existing registration, so the researchers created one for their analysis. The trial was registered prior to analysis but after enrollment had begun.
3. Evaluations of health care providers where the study population is made up solely of health care providers and the outcomes of interest include only effects on providers (rather than patients or community members), do not need to be registered. In this case, researchers may still consider registering the trial to avoid any barriers to publication if a journal editor disagrees with their assessment. Regardless, if the trial evaluates the effects of the provider intervention on patient outcomes then the trial should be registered. 
   • Example: In a clinical trial evaluating an educational intervention's impact on physical therapists’ prescribing behavior, registration is not required because outcomes are measured at the provider level (see case study example three in Hinman 2015). 

More information on trial registration, including J-PAL requirements, can be found in this resource on trial registration. Information on labeling a journal submission that is not a clinical trial is discussed below. 

Develop a trial protocol

A trial protocol outlines the objectives, methodology, study procedures, statistical analysis plan (SAP), and ethical considerations of a study. This is the same protocol that researchers must submit to an institutional review board (IRB) prior to beginning human subjects research. Medical journals require the trial protocol to be submitted at the time of manuscript submission. Because IRBs may vary in what information they request, researchers should use the SPIRIT reporting guidelines when preparing protocols to ensure completeness. An example is the trial protocol (with an integrated SAP) for Health Care Hotspotting.

Researchers should plan to submit the original approved protocol along with all amendments and their rationale. Amendments should always be reviewed and approved by an IRB prior to taking effect. Significant changes to the original protocol should be described in any future publications. The trial protocol is typically included with a publication as an online supplement. 

Develop a statistical analysis plan (SAP)

A statistical analysis plan (also known as a pre-analysis plan) provides detailed information on how data will be coded and analyzed. A SAP can be integrated into a trial protocol or may exist as a separate document. Most medical journals require a SAP to be submitted along with the manuscript.

Unlike trial registration, journals differ on when a SAP must be completed. However, for some journals, completing a SAP prior to the launch of the intervention, or at least prior to receiving data and beginning analysis, can increase the credibility of findings and/or allow researchers greater freedom in using causal language when interpreting results. Defining analysis plans in advance also reduces concerns about specification searching (p-hacking), particularly when there is researcher discretion about what outcomes to consider and when and how they are measured. Most medical journals will require researchers to indicate which analyses are prespecified and which are post hoc. As with the trial protocol, the final SAP should be submitted to the journal, with any changes to original plans tracked and justified. 

J-PAL does not universally require SAPs, but many individual J-PAL initiatives and funders require that an analysis plan be completed prior to the launch of an evaluation. For example, all J-PAL North America funded projects must upload a SAP to the AEA RCT Registry. See this resource on pre-analysis plans for more detailed information and recommended practices. 

Guidelines for the Content of Statistical Analysis Plans in Clinical Trials provides a checklist that researchers can follow to help structure their SAPs. A detailed explanation of each section can be found in Appendix 2 of the article. Some important topics to cover in a SAP include:

2. While implementing an RCT

Medical journals typically require significant detail on how participants progress through an evaluation. During an evaluation’s implementation stage, researchers should keep track of participants’ study involvement, the mechanics of randomization, and fidelity to the study protocol.

Track participants throughout the evaluation

Researchers should track the flow of participants as they move through each stage of the trial and should report this in their paper. The Consolidated Standards of Reporting Trials’ (CONSORT) Statement, a minimum set of recommendations for reporting RCTs, recommends using this diagram (see Figure 2) to display participant flow. All trials published in medical journals include this CONSORT flow diagram, and one should always be included as a figure in your paper (typically the first exhibit). See an example of a CONSORT flow diagram below. 

At a minimum, researchers should track the number of individuals randomly assigned, the number who actually receive the intervention, and the number analyzed. Information on individuals assessed for eligibility prior to randomization can be included if available. Any additional information on exclusions or attrition (also known as loss to follow-up) should be included as well.

_{Source: Finkelstein et al. 2020}

Track the mechanics of randomization

Researchers should keep track of the randomization process and the mechanics of implementation, as these details must be reported in detail in a manuscript. This typically includes the type of randomization procedure — was randomization stratified, done on the spot or from a list —  as well as who generated random assignments, how the information was concealed or protected from tampering, and how information was revealed to participants. Items 8-10 of the CONSORT checklist cover reporting on randomization processes. See Financial Support to Medicaid-Eligible Mothers Increases Caregiving for Preterm Infants for an example of reporting these details in a manuscript. 

Monitor implementation fidelity

The CONSORT statement notes that “the simple way to deal with any protocol deviations is to ignore them.” This is consistent with the intention to treat approach to randomized evaluation. However, providing detail is helpful to contextualize and interpret results. See the resource on implementation monitoring and real-time monitoring and response plans for guidance. Researchers approach monitoring implementation and reporting on it in different ways. 

• In Health Care Hotspotting, researchers quantitatively tracked program metrics in the treatment group and reported them in a table. 
• In Effects of a Workplace Wellness Program on Employee Health, Health Beliefs, and Medical Use, researchers reported adherence by presenting completion rates for screenings and other elements of the intervention, as well as control group awareness of the intervention.

3. After completing an RCT 

Clinical trial reporting in medical journals follows a specific style characterized by consistent structure, short word counts, and a small number of tables and figures. The Consolidated Standards of Reporting Trials’ (CONSORT) Statement provides a set of standardized recommendations, in the form of a checklist and flow diagram, for reporting randomized evaluations in medical journals. Researchers should consult the CONSORT Statement, as well as particular journal requirements, in order to ensure a manuscript is properly formatted and structured. This section highlights elements of analysis and manuscript preparation that may be less familiar to social scientists.

Adhere to medical journal analysis norms

CONSORT guidelines state that researchers should not report statistical tests of balance in participant characteristics, and many medical journals follow this recommendation. This practice differs from a common norm in economics to report balance tests in summary statistics tables. Researchers can find more information on the logic behind this rule in the CONSORT Explanation and Elaboration resource. 

Results should be reported for all primary and secondary outcomes, regardless of their statistical significance. Detailed results should be reported for each outcome, typically mean levels for each group, treatment effect, confidence interval, and p-value. Confidence intervals are preferred to standard errors. For binary outcomes, both absolute and relative effects should be reported. Researchers should make clear which analyses are prespecified, particularly in the case of subgroup analyses. 

Below is an example of a results table with multiple outcomes from Health Care Hotspotting. Each row highlights one outcome, with mean levels for each group, an unadjusted group difference (from a regression without controls), an adjusted group difference (from a regression with controls), and corresponding confidence intervals. In this case, all analyses were prespecified and p-values were omitted. 

_{Source: Finkelstein et al. 2020}

Construct tables and figures 

Journals typically cap the total number of exhibits (tables and figures) at five. Figure 1 should typically be the participant flow diagram. A summary statistics table, typically Table 1, should always describe baseline demographic and clinical characteristics for each group. The remaining figures and tables display the results of the evaluation.

For studies with multiple treatment arms or subgroup analyses, it may be helpful to use an exhibit with multiple panels or a hybrid exhibit that combines a table and a figure to provide more detail. Figure 2 in Effectiveness of Behaviorally Informed Letters on Health Insurance Marketplace Enrollment gives an example of a hybrid exhibit. Note, however, that some journals may not view this as a single exhibit. Additional exhibits may be included in a supplemental appendix.

Prepare your manuscript with the correct structure

Before preparing a manuscript for a medical journal, you should consult the specific word and exhibit limits of that journal as well as its requirements on how the manuscript should be structured. Compared to social science publications, reports of clinical trials in medical journals are consistently structured and short (e.g., 3000 words) with binding limits on the number of words and exhibits. Medical journals are generally much more prescriptive about what each section in the paper must accomplish. Medical journals do allow for supplemental appendices where researchers can include details that did not fit in the main body of the paper. Most journals follow CONSORT reporting requirements. Consult the CONSORT checklist as you write and make sure to include all sections. Some journals require submission of the checklist to ensure that all elements have been addressed.

Some key requirements from the CONSORT checklist have been highlighted below:

Researchers should ensure their paper is properly pitched for a medical or public health audience rather than an economics audience. Consulting the stated aims and scope of a journal can help with framing study results and emphasizing its relevance to the field. Highlighting the clinical rationale and patient care implications of a study can be helpful as medical journal readership is often interested in these factors. If possible, researchers should have someone who has frequently published in medical journals review their manuscript before submission to make sure the manuscript is properly framed for the intended audience.

Submit your manuscript for review and publication 

There are some additional considerations to keep in mind at the time of submission:

4. Examples of published work from J-PAL researchers 

A selection of J-PAL affiliated-studies that have been published in medical journals has been highlighted below. 

• Health Care Hotspotting — A Randomized, Controlled Trial (NEJM)
• Mandatory Medicare Bundled Payment Program for Lower Extremity Joint Replacement and Discharge to Institutional Postacute Care (JAMA) 
• Financial Support to Medicaid-Eligible Mothers Increases Caregiving for Preterm Infants (Maternal and Child Health Journal) 
• Effect of a Workplace Wellness Program on Employee Health and Economic Outcomes: A Randomized Clinical Trial (JAMA)
• Effects of a Workplace Wellness Program on Employee Health, Health Beliefs, and Medical Use: A Randomized Clinical Trial (JAMA Internal Medicine)

_{Acknowledgments: Thanks to Marcella Alsan, Catherine Darrow, Joseph Doyle, Laura Feeney, Amy Finkelstein, Ray Kluender, David Molitor, Hannah Reuter, and Adam Sacarny for their thoughtful contributions. Amanda Buechele copy-edited this document. Creation of this resource was supported by the National Institute On Aging of the National Institutes of Health under Award Number P30AG064190. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.}