Research Resources

Resources for researchers new to randomized evaluations

Summary

This resource is a brief introduction to conducting field research that distills many years of implementation experience. It is geared towards new PIs (Principal Investigators), such as graduate students or post docs, but may be useful for more seasoned researchers and research staff as well. While the language in this section centers around working with J-PAL, the overarching guidance applies more broadly to projects implemented outside of the J-PAL network.

Project life cycle

A project's life cycle can be split into four primary phases: inception, planning, implementation, and wrap-up. Key activities to undertake at each stage are shown in the diagram below. 

Project life cycle image
Adapted from a J-PAL Project Life Cycle presentation at Research Staff Training, Tithee Mukhopadyay, 2018.

These activities are distilled into "minimum must dos" and best practices, specified in our Research Protocols, to ensure ethical conduct of research, data security, and data quality.1 All projects funded or carried out by J-PAL must adhere to these guidelines.

Working with J-PAL

Funding: If you receive funding from a J-PAL initiative, the J-PAL Global Finance team or regional initiative staff will be your main point of contact for grant deadlines and deliverables such as the initial IRB approval for the project. PI compliance with grant requirements helps J-PAL fulfill its obligations towards its own funders and its host universities.

Field implementation: PIs may work with J-PAL regional offices to carry out field projects. Each office is a research center within a university. All research staff are part of the reporting structure within J-PAL (from Associates to Managers to Associate Directors and Executive Director). The host university’s HR rules apply (for vacation, holidays, salary bands, work time etc.). A project may have one or more RAs and Senior RAs, as well as a Research Manager (RM) (who often splits time between several projects). Offices provide shared resources such as office space, survey tablets, and financial coordinators who help manage project funds. Policy & Training staff can support partnership and capacity building. All project needs should be discussed with the regional office and built into the budget.

J-PAL support for (new) PIs: Ask your regional office about what exactly J-PAL can offer.

  • J-PAL hosts regular trainings for new research staff, and PIs can re-use slides, exercises, etc. from these trainings. Many other resources were built over time, such as pre-programmed survey modules and guides, e.g., on data de-identification. Many are available on our website or on our internal Google drive for PIs (see the Information for Affiliates page)
  • J-PAL offices have many pre-existing policy and government partnerships, which may give researchers access to administrative data or the possibility to collaborate on impact evaluations. They can connect PIs with local academic researchers as well as specialists such as translators, programmers, videographers, etc.
  • Several regional offices will submit IRB applications on the PI’s behalf and handle permits and contracting with local providers or partners, such as data use agreements, sub-awards, or contracting out survey work.
  • PIs can draw on advice from experienced research staff (e.g. Associate Director of Research; Director of Research, Education, and Training). 

More information on J-PAL's important policies and services for affiliates can be found on our Information for Affiliates page.

High-priority tasks

The PIs need to keep sight of some high-priority tasks where issues can jeopardize months of work. Data may be lost, survey permits withdrawn, treatments not rolled out, partners alienated, funding rescinded, etc. In multi-PI teams, one PI can be assigned to sign off on each key task; this division of responsibilities should be shared with the research team. J-PAL research staff can provide support, especially on local compliance, but final responsibility lies with the PIs.

  What and why? What can go wrong?
Funding and budget
  • Projects must stay within budget even if e.g. exchange rate fluctuations change grant payments.
  • PIs are responsible for adhering to grant reporting deadlines and completing the project in the grant period.
  • PIs should regularly check that budget assumptions are met and watch their cash flow

J-PAL financial coordinators provide support and reminders, but ultimate responsibility is with the PI. 
  • Funders may withhold or revoke grant tranches when grant conditions are not met.
  • Remaining grant funds may be withdrawn when the original project deadline is reached and a “no-cost extension” was not filed.
  • Incorrect assumptions about exchange rates, surveying speed, or intervention costs can reduce sample sizes and statistical power.
  • Liquidity problems may impact crucial project deadlines.
Legal compliance
  • Local rules for conducting data collection, protecting subjects, or sharing data across borders (e.g. GDPR in Europe) may apply.
  • Projects need to comply with visa and tariff regulations, tax, employment, and substance use laws, procurement rules (in particular for projects with government) and binding contracts.
  • Official (and sometimes informal, custom based) approval processes should be respected.
  • Non-adherence to local laws may put subjects and staff at risk. Non-compliance e.g. with non-disclosure agreements, may shut down data collection or incur fines.
  • Local sentences may be harsher than in the PI’s country. Non-residents may have to leave the country.
Non-compliance may jeopardize multiple projects and crucial J-PAL partnerships
Ethics, consent, and IRB compliance
  • All projects must have IRB approval for the latest research design (before data collection starts).
  • All data collection requires informed consent from subjects (before individual survey start).
  • Researchers must protect subject privacy and personal data through encryption and de-identification (as soon as data is processed).
  • PIs should put procedures in place to handle adverse events if/when they occur, e.g. security, mental/physical health related, for research and survey staff and subjects (before data collection).
  • Careless data handling or unaddressed adverse events can put research subjects at serious risk. At a minimum they may bias survey responses or lead to sample attrition.
  • In case of violations, IRBs can impose remedies (such as destruction of data that was collected without consent, debriefing of subjects). Researchers can be banned from receiving federal funding in the US.
Current and future projects are at risk if PIs are not fully committed to protecting subjects and staff.
Randomization and treatment assignment
  • Implementing personnel at all levels need to be trained on the importance of adhering to the treatment assignment and avoiding spillovers.
  • Any randomization code should be fully replicable (e.g. use seed, stable sort to make code output deterministic).
  • Check the randomization output repeatedly for balance and correct stratification.
  • Monitor treatment receipt with administrative and survey data starting early in the experiment and, if necessary, make corrections.
  • Non-compliance with treatment assignment or implementation rules reduces statistical power and can create spillovers that bias the treatment effect estimate.
  • Typos in the randomization code can lead to unbalanced treatment assignment or even missing treatments.
  • Systematic deviations at the implementation stage from the randomized treatment assignment may introduce bias.
Survey design, translation, and coding
  • Pilot survey question phrasing, duration, and multiple-choice options extensively. Make sure key outcomes and treatment receipt are measured.
  • Use back translation to ensure accuracy.
  • Pilot surveys with test (non-study) subjects to verify skip patterns, flow, and that data is saved correctly.
  • Conduct initial and refresher trainings for surveyors which clarify what to say or not to say to help subjects interpret questions.
A PI should sign off on each final survey module.
  • Subject fatigue, question ambiguity, translation errors, or missing multiple-choice options lead to measurement and data entry error and missing data.
  • Programming errors or changes to the survey code after survey start can lead to significant data loss.
  • Significant surveyor effects can increase the noise in the data and invalidate power calculations or even make data unusable.
High frequency and back checks during data collection
  • Conduct checks as data come in: do summary statistics make sense? How many missing data points are there? Are all multiple-choice options used? Are there large discrepancies between surveyors or survey teams?
  • Check data at high frequency and conduct random back checks on surveyors (and intervention staff if applicable).
PIs should prepare for data quality assurance activities ahead of survey start and plan staff time for them.
  • When problems are not detected early, data issues (above) may occur throughout the entire survey. Programming data checks or putting back check procedures in place only after survey roll-out can cost valuable time.
  • Survey staff may make (systematic) mistakes, or (in rare cases) even falsify data, leading to missing data - or, worst case, undetected bias and data errors.

Managing a project

Some simple principles and tips can prevent errors and make sure that projects run smoothly.

Timeline management: A shared project calendar helps to keep track of team meetings and important deadlines, such as survey code freezes, surveyor training, PI field visits, and grant deadlines. Plan buffers for IRB turnaround and partner or government approvals. It’s also useful to include public holidays, trips, or training/onboarding of team members. 

Communication: Plan at least weekly hour-long team calls. RAs should be instructed to carefully document all relevant decisions, problems, etc. as they occur. A “running agenda” attached to the calendar invite, where the latest meeting is added to the top of a shared Google Doc or Dropbox document, serves as a record of discussions and decisions. Agree on only one “permanent” channel (typically email) for important communications such as task delegation, and one instant messaging channel, e.g. Slack. It is useful to set a norm for all team members of replying within, say, two business days. Avoid WhatsApp or text messaging linked to personal phones to retain a project record when staff leave the team.

In-person field visits: Many project tasks can be accomplished remotely, but there are key times to be on the ground. PIs should meet key partners in person. The best way to learn whether the survey flow and individual modules work as expected is to accompany a survey team on pilot visits. The period shortly before survey start, including final pilots and surveyor training, is extremely valuable for catching errors and motivating the research and field team.

Research team capacity: PIs are responsible for hiring, training, and evaluating research team members. Hiring the right person can be the single most significant contribution to the success of a research project. The best way for research teams to learn best practices, such as how to structure field teams, is to observe other projects or learn from experienced research staff. All new research staff should attend the week-long J-PAL/IPA Research Staff Training and receive onboarding from a senior research staff member. Explicitly plan for professional development for new tasks, such as staff attending the surveyor training on another project or watching the online “power and sample size” lecture from J-PAL 101x

Partner management: Plan regular project updates and in-depth conversations with implementing partners to convey the importance of randomization and treatment arm adherence and dispel any concerns (ethical, financial, logistical…). It builds trust and engagement when PIs attend key meetings and communicate project milestones directly.

Managing young research team members

Managing a field team may be one of the most complex management jobs you have had – PIs are often remote and in a different time zone, there are language and connectivity barriers, and projects are often resource constrained; especially during crunch times, stress levels can be high. We can here only give an incomplete overview of some points to consider.

Motivation and priorities: PIs are delegating many key components of the implementation of their research project to junior staff who are often quite junior. Spending some time explaining the logic of the research design and the underlying economic model, as well as the rationale for making changes later, e.g., after piloting, will help the team understand why what they do is important and what elements are key. Going over the priorities list above together will help the team focus. The PI’s attitude towards IRB and compliance informs how much responsibility research staff feel as well. 

Delegation: Staff may not always express when they did not hear or understand a task, or do not see its relevance. For important tasks it may help to let the staff member describe the steps they plan on taking, and how long they will take, back to the PI in their own words. All tasks should have a concrete timeline and a clear deliverable attached to them.

Feedback: The most conscientious staff will most take feedback to heart. PIs should remember to acknowledge when things went well, and think about how to respond if they go wrong. A constructive response focused on addressing the problem can promote learning and avoid future errors. It is useful to set up regular check-ins (once every 1-2 months) where the focus is the team’s working relationship and what might need changing. Some projects have a bi-annual feedback conversation between the RA and their J-PAL supervisor (e.g. a Senior RM). They discuss the PIs feedback with the RA, and then in turn debrief with the PI afterwards. This can be an invaluable mechanism to receive feedback.

Power differential: A joint research project can start a rewarding mentor-mentee relationship between PI and RA. RAs see PIs as role models. However, PIs, especially new ones, are often not aware of the significant power imbalance. RAs may depend on letters of reference or feel intimidated. In this situation, only the PI may feel like they are speaking as equals. Cultural, language, and technological communication barriers can compound the issue. 

Work-life balance: As a result, PIs cannot rely on RAs to tell them when they are overtaxed. Research teams should schedule regular meetings during the RA’s normal work hours, to avoid conflicts with family obligations or compromising their safety. Outside work hours, instant channels such as chat, cell phone, and messaging should be reserved for emergencies; weekend work should be compensated with off days. The PIs should signal their interest in the safety and well-being of field and research staff e.g. during survey work and travel and ensure there is a budget for safe transport.

Last updated August 2020.

These resources are a collaborative effort. If you notice a bug or have a suggestion for additional content, please fill out this form.

1.
Our partner organization Innovations for Poverty Action has a similar set of Research Protocols in place. They give a comprehensive overview on the steps to implement a high-quality field project.

In this resource