Checklist for social scientists publishing in medical journals: before, during, and after running an RCT

Summary

This resource is intended for researchers in the social sciences who are considering publishing their randomized evaluation in a medical journal. Publishing in a medical journal allows researchers to share their study with researchers and practitioners in different disciplines and expand the reach of their findings to those outside of their immediate professional network. This resource highlights journal guidelines and requirements that may be unfamiliar to those used to publishing in the social sciences. Most medical journals are members of the International Committee of Medical Journal Editors (ICMJE) and follow its recommendations and requirements for publishing.¹ Some of the requirements for publishing necessitate action before a study has even started. While most of the content in this checklist is relevant across all medical journals, certain elements may differ from journal to journal. Researchers are encouraged to consult requirements listed in this resource in conjunction with the requirements of their specific journal(s) of interest, as well as requirements from funders and universities that could influence the publication process.  

Key steps for publishing a randomized evaluation in a medical journal:

• Before participant enrollment begins, register a trial in an ICMJE compliant registry such as clinicaltrials.gov. Failure to do so may prevent publication in most medical journals. Note that the American Economic Association (AEA) registry is not an ICMJE compliant registry. 
• Upload a statistical analysis plan (SAP) to the trial registry prior to the start of the study. Although not strictly a requirement for most medical journals, publically archiving a SAP in advance will allow greater flexibility in presenting and interpreting causal results. In particular, ensure primary vs. secondary outcomes and analysis methods are prespecified in the SAP. 
• Draft a manuscript according to journal requirements. Medical journals require papers to contain a limited number of figures and tables and adhere to specific formats and short word counts. Consult the Consolidated Standards of Reporting Trials (CONSORT) guidelines used by most journals as well as individual journal requirements before drafting your manuscript. 
• Be aware of embargo and preprint policies. Some journals may de-prioritize study publication if study results are public or shared through a working paper or preprint. However, other journals publicly state they will not penalize such papers. Journals also impose restrictions about what can be shared after acceptance and prior to publication. Make sure to check the journal’s specific guidelines and adhere to these rules.

​​1. Before starting an RCT

Register the trial before enrollment begins

The ICMJE defines a clinical trial as, “any research study that prospectively assigns human participants or groups of humans to one or more health-related interventions to evaluate the effects on health outcomes.” If a randomized evaluation meets the definition of a clinical trial, researchers must register it in an ICMJE compliant registry before the first participant is enrolled. If researchers fail to meet this condition, they risk having their paper rejected by medical journals. Note that registration on the AEA’s RCT Registry (supported by J-PAL) does not meet this requirement. 

Not every randomized evaluation that researchers submit to a medical journal will meet the ICMJE definition of a clinical trial and require registration. However, if it is unclear whether a study meets the definition of a clinical trial, follow the registration guidelines for clinical trials as a cautionary measure. This way if a journal editor feels that a study is a clinical trial, researchers will have met the registration requirement. 

Some scenarios where registration prior to enrollment may be unnecessary (although different journals may reach different conclusions) include the following: 

1. Secondary analysis of a clinical trial does not need to be registered separately. Instead, researchers should reference the trial registration of the primary trial. 
   • Example: In Health Care Hotspotting, researchers conducted a randomized evaluation of a care transition program’s impact on hospital readmissions for those with complex needs. In a related secondary study, the same research team is analyzing the effect of the program on outpatient utilization. Researchers referenced the original study’s registration instead of separately registering the secondary analysis.  
2. Analysis of a clinical trial implemented by someone else without researcher involvement is also considered a secondary analysis and does not need to be registered separately. Researchers may reference the existing clinical trial record, if one exists. As a best practice, researchers may also register a secondary analysis separately.
   • Example: In The Spillover Effects of a Nationwide Medicare Bundled Payment Reform, independent researchers conducted a secondary analysis of a randomized evaluation implemented by the Centers for Medicare and Medicaid Services (CMS). This trial did not have an existing registration, so the researchers created one for their analysis. The trial was registered prior to analysis but after enrollment had begun.
3. Evaluations of health care providers where the study population is made up solely of health care providers and the outcomes of interest include only effects on providers (rather than patients or community members), do not need to be registered. In this case, researchers may still consider registering the trial to avoid any barriers to publication if a journal editor disagrees with their assessment. Regardless, if the trial evaluates the effects of the provider intervention on patient outcomes then the trial should be registered. 
   • Example: In a clinical trial evaluating an educational intervention's impact on physical therapists’ prescribing behavior, registration is not required because outcomes are measured at the provider level (see case study example three in Hinman 2015). 

More information on trial registration, including J-PAL requirements, can be found in this resource on trial registration. Information on labeling a journal submission that is not a clinical trial is discussed below. 

Develop a trial protocol

A trial protocol outlines the objectives, methodology, study procedures, statistical analysis plan (SAP), and ethical considerations of a study. This is the same protocol that researchers must submit to an institutional review board (IRB) prior to beginning human subjects research. Medical journals require the trial protocol to be submitted at the time of manuscript submission. Because IRBs may vary in what information they request, researchers should use the SPIRIT reporting guidelines when preparing protocols to ensure completeness. An example is the trial protocol (with an integrated SAP) for Health Care Hotspotting.

Researchers should plan to submit the original approved protocol along with all amendments and their rationale. Amendments should always be reviewed and approved by an IRB prior to taking effect. Significant changes to the original protocol should be described in any future publications. The trial protocol is typically included with a publication as an online supplement. 

Develop a statistical analysis plan (SAP)

A statistical analysis plan (also known as a pre-analysis plan) provides detailed information on how data will be coded and analyzed. A SAP can be integrated into a trial protocol or may exist as a separate document. Most medical journals require a SAP to be submitted along with the manuscript.

Unlike trial registration, journals differ on when a SAP must be completed. However, for some journals, completing a SAP prior to the launch of the intervention, or at least prior to receiving data and beginning analysis, can increase the credibility of findings and/or allow researchers greater freedom in using causal language when interpreting results. Defining analysis plans in advance also reduces concerns about specification searching (p-hacking), particularly when there is researcher discretion about what outcomes to consider and when and how they are measured. Most medical journals will require researchers to indicate which analyses are prespecified and which are post hoc. As with the trial protocol, the final SAP should be submitted to the journal, with any changes to original plans tracked and justified. 

J-PAL does not universally require SAPs, but many individual J-PAL initiatives and funders require that an analysis plan be completed prior to the launch of an evaluation. For example, all J-PAL North America funded projects must upload a SAP to the AEA RCT Registry. See this resource on pre-analysis plans for more detailed information and recommended practices. 

Guidelines for the Content of Statistical Analysis Plans in Clinical Trials provides a checklist that researchers can follow to help structure their SAPs. A detailed explanation of each section can be found in Appendix 2 of the article. Some important topics to cover in a SAP include:

2. While implementing an RCT

Medical journals typically require significant detail on how participants progress through an evaluation. During an evaluation’s implementation stage, researchers should keep track of participants’ study involvement, the mechanics of randomization, and fidelity to the study protocol.

Track participants throughout the evaluation

Researchers should track the flow of participants as they move through each stage of the trial and should report this in their paper. The Consolidated Standards of Reporting Trials’ (CONSORT) Statement, a minimum set of recommendations for reporting RCTs, recommends using this diagram (see Figure 2) to display participant flow. All trials published in medical journals include this CONSORT flow diagram, and one should always be included as a figure in your paper (typically the first exhibit). See an example of a CONSORT flow diagram below. 

At a minimum, researchers should track the number of individuals randomly assigned, the number who actually receive the intervention, and the number analyzed. Information on individuals assessed for eligibility prior to randomization can be included if available. Any additional information on exclusions or attrition (also known as loss to follow-up) should be included as well.

_{Source: Finkelstein et al. 2020}

Track the mechanics of randomization

Researchers should keep track of the randomization process and the mechanics of implementation, as these details must be reported in detail in a manuscript. This typically includes the type of randomization procedure — was randomization stratified, done on the spot or from a list —  as well as who generated random assignments, how the information was concealed or protected from tampering, and how information was revealed to participants. Items 8-10 of the CONSORT checklist cover reporting on randomization processes. See Financial Support to Medicaid-Eligible Mothers Increases Caregiving for Preterm Infants for an example of reporting these details in a manuscript. 

Monitor implementation fidelity

The CONSORT statement notes that “the simple way to deal with any protocol deviations is to ignore them.” This is consistent with the intention to treat approach to randomized evaluation. However, providing detail is helpful to contextualize and interpret results. See the resource on implementation monitoring and real-time monitoring and response plans for guidance. Researchers approach monitoring implementation and reporting on it in different ways. 

• In Health Care Hotspotting, researchers quantitatively tracked program metrics in the treatment group and reported them in a table. 
• In Effects of a Workplace Wellness Program on Employee Health, Health Beliefs, and Medical Use, researchers reported adherence by presenting completion rates for screenings and other elements of the intervention, as well as control group awareness of the intervention.

3. After completing an RCT 

Clinical trial reporting in medical journals follows a specific style characterized by consistent structure, short word counts, and a small number of tables and figures. The Consolidated Standards of Reporting Trials’ (CONSORT) Statement provides a set of standardized recommendations, in the form of a checklist and flow diagram, for reporting randomized evaluations in medical journals. Researchers should consult the CONSORT Statement, as well as particular journal requirements, in order to ensure a manuscript is properly formatted and structured. This section highlights elements of analysis and manuscript preparation that may be less familiar to social scientists.

Adhere to medical journal analysis norms

CONSORT guidelines state that researchers should not report statistical tests of balance in participant characteristics, and many medical journals follow this recommendation. This practice differs from a common norm in economics to report balance tests in summary statistics tables. Researchers can find more information on the logic behind this rule in the CONSORT Explanation and Elaboration resource. 

Results should be reported for all primary and secondary outcomes, regardless of their statistical significance. Detailed results should be reported for each outcome, typically mean levels for each group, treatment effect, confidence interval, and p-value. Confidence intervals are preferred to standard errors. For binary outcomes, both absolute and relative effects should be reported. Researchers should make clear which analyses are prespecified, particularly in the case of subgroup analyses. 

Below is an example of a results table with multiple outcomes from Health Care Hotspotting. Each row highlights one outcome, with mean levels for each group, an unadjusted group difference (from a regression without controls), an adjusted group difference (from a regression with controls), and corresponding confidence intervals. In this case, all analyses were prespecified and p-values were omitted. 

_{Source: Finkelstein et al. 2020}

Construct tables and figures 

Journals typically cap the total number of exhibits (tables and figures) at five. Figure 1 should typically be the participant flow diagram. A summary statistics table, typically Table 1, should always describe baseline demographic and clinical characteristics for each group. The remaining figures and tables display the results of the evaluation.

For studies with multiple treatment arms or subgroup analyses, it may be helpful to use an exhibit with multiple panels or a hybrid exhibit that combines a table and a figure to provide more detail. Figure 2 in Effectiveness of Behaviorally Informed Letters on Health Insurance Marketplace Enrollment gives an example of a hybrid exhibit. Note, however, that some journals may not view this as a single exhibit. Additional exhibits may be included in a supplemental appendix.

Prepare your manuscript with the correct structure

Before preparing a manuscript for a medical journal, you should consult the specific word and exhibit limits of that journal as well as its requirements on how the manuscript should be structured. Compared to social science publications, reports of clinical trials in medical journals are consistently structured and short (e.g., 3000 words) with binding limits on the number of words and exhibits. Medical journals are generally much more prescriptive about what each section in the paper must accomplish. Medical journals do allow for supplemental appendices where researchers can include details that did not fit in the main body of the paper. Most journals follow CONSORT reporting requirements. Consult the CONSORT checklist as you write and make sure to include all sections. Some journals require submission of the checklist to ensure that all elements have been addressed.

Some key requirements from the CONSORT checklist have been highlighted below:

Researchers should ensure their paper is properly pitched for a medical or public health audience rather than an economics audience. Consulting the stated aims and scope of a journal can help with framing study results and emphasizing its relevance to the field. Highlighting the clinical rationale and patient care implications of a study can be helpful as medical journal readership is often interested in these factors. If possible, researchers should have someone who has frequently published in medical journals review their manuscript before submission to make sure the manuscript is properly framed for the intended audience.

Submit your manuscript for review and publication 

There are some additional considerations to keep in mind at the time of submission:

4. Examples of published work from J-PAL researchers 

A selection of J-PAL affiliated-studies that have been published in medical journals has been highlighted below. 

• Health Care Hotspotting — A Randomized, Controlled Trial (NEJM)
• Mandatory Medicare Bundled Payment Program for Lower Extremity Joint Replacement and Discharge to Institutional Postacute Care (JAMA) 
• Financial Support to Medicaid-Eligible Mothers Increases Caregiving for Preterm Infants (Maternal and Child Health Journal) 
• Effect of a Workplace Wellness Program on Employee Health and Economic Outcomes: A Randomized Clinical Trial (JAMA)
• Effects of a Workplace Wellness Program on Employee Health, Health Beliefs, and Medical Use: A Randomized Clinical Trial (JAMA Internal Medicine)

 

_{Acknowledgments: Thanks to Marcella Alsan, Catherine Darrow, Joseph Doyle, Laura Feeney, Amy Finkelstein, Ray Kluender, David Molitor, Hannah Reuter, and Adam Sacarny for their thoughtful contributions. Amanda Buechele copy-edited this document. Creation of this resource was supported by the National Institute On Aging of the National Institutes of Health under Award Number P30AG064190. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.}