Resources for adapting to phone surveys during COVID-19

Posted on:
Authors:
Saurabh Bhajibhakare
Putul Gupta
Sarah Kopper
A close up image of a man holding and looking at a smartphone
Photo: Shutterstock.com

The safety of research participants and staff is of utmost importance to J-PAL. On March 17, J-PAL suspended all research activities that require in-person contact in order to limit the spread of COVID-19. Since the suspension of in-person research activities, J-PAL staff have worked to quickly adapt to phone and online surveys where possible to minimize disruption to ongoing research projects.

Over the past few months, we’ve developed new resources on transitioning to phone surveys, with guides that span from budgeting to remote trainings to quality assurance best practices. These resources, along with those developed by Innovations for Poverty Action (IPA), are listed below. They are intended to complement our crowd-sourced best practices by providing in-depth guides that address challenges specific to remote survey work. For tips on obtaining electronic signatures (e.g., for consent forms), see also J-PAL North America’s recent blog post. IPA’s RECOVR hub houses COVID-related questionnaires and will be updated as IPA’s technical resources are made public.

Planning the transition to CATI

  • Transitioning to CATI checklist (12 pages) Saurabh Bhajibhakare, Ambika Chopra, Putul Gupta, and Mustufa Patel, J-PAL South Asia

    • A starting point for projects transitioning to phone surveys. Provides a high-level overview of the steps involved so that key tasks are not forgotten. The appendix includes detailed guidance on remote monitoring of surveyors, from hiring to tracking productivity and conducting weekly debrief sessions.
  • Budgeting for phone surveys with remote monitoring (5 pages) Putul Gupta, J-PAL South Asia
    • This resource includes surveyor productivity estimates, a reference budget, sample team structures, and more. Please note that the costs and training scenarios are tailored to the J-PAL South Asia office and may need adapting to other contexts.
    • Key takeaways:
      • Budget for additional days of training even if the instruments are shorter and simpler, since projects will need to spend additional time training surveyors on protocols for conducting phone surveys (tracking calls and making appointments) as well as data security and transfer protocols. It is also important that each surveyor gets enough practice and attention, an element that becomes even more critical in the remote training setup.
      • Be conservative in productivity estimates, as a fair share of surveyors’ time is spent reaching out to potential respondents multiple times before success. For a 30 minute phone survey, we’ve found 4-8 surveys/person/day to be a realistic estimate; for shorter (10-15 minutes), we aim for 10-14 surveys/person/day.
      • Productivity estimate should take into account whether you’re trying to target a specific respondent (caregiver, household head) for the survey, which might mean more attempts to get that specific individual on the line.
      • Consider budgeting for one supervisor for 3-4 surveyors. The supervisor can listen in on calls, verify call logs, tally attempts and completed surveys at the end of each day, track surveyor start and end time, ensure appointments are being completed, etc.
      • Costs will change, particularly if surveyors are working from home. For example, travel allowances will not apply. If surveyors use their own devices, tablet rentals may not be needed, but surveyors will need number masking. Additionally, projects will need to budget for data packs for surveyors keeping in mind the nature of monitoring protocols (use of recordings) and training plan (platform, use of videos, etc.)

Training and quality assurance

  • Remote trainings for field staff (7 pages) Putul Gupta, J-PAL South Asia
    • Synthesizing learnings from six J-PAL South Asia projects that launched phone surveys during the COVID-19 lockdown, this resource provides tips on planning for remote trainings, guidance on training effectively, and possible mitigation strategies to deal with challenges unique to remote training.
    • Key takeaways:
      • Do your homework on potential enumerators and ensure that they have the required equipment for phone surveys before they come to the training,
      • Choose your training platform carefully. Get your training participants to familiarize themselves with the platform ahead of the actual training.
      • Break the training plan into a number of small sessions to facilitate better comprehension and retention. Build in enough time into the training schedule for mock sessions and team debriefs. To overcome logistical hurdles, budget a few breaks of 10-15 minutes through the day when devices can be charged.
      • Train a small team of senior field staff ahead of the actual training and leverage their support in building resources and planning for the training.
      • Share sharp and concise resources that help training participants understand and apply what they learn during the training. Avoid sharing all the resources at once. Ensure that the training resources are appropriately formatted for viewing on mobile devices.
      • Leverage technology (videos, screen sharing, breakouts) to keep the participants engaged.
      • Utilize quizzes and demo recordings to identify knowledge gaps at the enumerator level as well as systematic gaps in understanding within the team for specific questions.
      • PRACTICE, PRACTICE, PRACTICE!
  • Quality assurance for CATI (4 pages) Saurabh Bhajibhakare, J-PAL South Asia
    • This resource covers specific modifications to accompaniments, back checks, high frequency checks, audio audits, and daily supervision that can be made to accommodate phone surveys.
    • Key takeaways:
      • Closely monitor call status attempts, refusal rates (by time, day of the week), and the number of incorrect phone numbers in your sample from day 1. This information should actively feed into revising your productivity assumptions and budgets.
      • Accompaniments for phone surveys can be conducted through call conferencing (or a three-way call)—though note that IRB approvals are necessary if you plan to have a third party listen to calls for monitoring purposes. Be sure this information is included in the informed consent process. Staff might incur call or message costs and a reimbursement process should be put in place to cover this.
      • Audio recordings can function like accompaniments but done post hoc (though also require IRB approval). Options include SurveyCTO’s audio audit feature and call recording features from platforms such as Exotel. Recordings are enabled by default when using the web version of Exotel, but when using the Exotel field plug-in for SurveyCTO call recordings can be toggled on or off.
      • Call logs are a good place to check if the right phone numbers were called, the number of call attempts, and the duration of the calls. For SurveyCTO, use the early release versions of Android Collect 2.70.2+ to capture call logs using the phone-call-log() function. For Exotel, the call reports can be exported from the web platform.
      • Set up and test the SurveyCTO data flow for high frequency checks well in advance, including setting up import .do files and the code for running High Frequency Checks (HFCs). Check out IPA’s HFCs in Stata and J-PAL’s R code for HFCs. Also check out Stata utilities to work with SurveyCTO data (additional information on the sctoapi command here).

Technical solutions

Authored By

Research Methods

Previous

Next

Recommended content
Inglés
Summary

Since the spread of COVID-19, researchers across the globe have experienced the challenges of conducting research in a socially distant manner. This blog provides an overview of how to obtain signatures--for example, to document informed consent, or to execute a legal agreement—without physical contact.

Date
Image
Collecting electronic signatures
Body

Since the spread of COVID-19, researchers across the globe have experienced the challenges of conducting research in a socially distant manner. Many processes have shifted from in-person to virtual methods, including a shift to online and phone surveys (please see the related blog here about phone survey methods), and collecting consent electronically. This blog provides an overview of how to obtain signatures—for example, to document informed consent, or to execute a legal agreement--without physical contact. The most appropriate method will vary by project; factors to consider include technological capabilities, legal requirements, Institutional Review Board (IRB) requirements, and study design. This document provides a high-level overview of different methods.

Considerations for collecting electronic signatures

  • Confirm with partners that electronic signatures in general and the method selected in particular is acceptable and legal. For example, one project found signatures obtained via DocuSign (explained below) are not accepted in a particular state, but signatures collected by an alternative (hellosign.com) were legally recognized. If electronic signatures are not accepted altogether, consider ways of securely obtaining hard copies.
  • If obtaining signatures for consent, confirm with the IRB that electronic signatures are acceptable. Indicate in the original IRB materials if using electronic signatures. If changing to an electronic signature after obtaining IRB approval (for example, in response to COVID-19), contact the IRB prior to making this change to determine whether a formal amendment request is necessary. Confirm with the IRB that all aspects of collecting the signature (e.g., how you are sending and collecting the form and signature) meet appropriate levels of security. Many IRBs have updated guidance around both electronic and verbal consent. While this blog does not address verbal consent, this may be another viable option to reduce in-person contact with human subjects.

Methods of collecting electronic signatures

  • Scanned signature. In this method, the signatory signs a physical piece of paper and scans or uploads the document to return to the researchers via email. If the person signing does not have access to a scanner, there are phone apps that can take a picture and upload the document (e.g., CamScanner). Confirm the security of any apps used to scan or send documents.
  • Signature entered into a Word or PDF document. This could be a typed signature entered into a fillable form PDF, or an image of a signature that the signatory pastes into a document. Consider encrypting email attachments and providing instructions for return encryption if the form contains Personally Identifiable Information (PII).
  • Electronic signature software. There are a variety of software options that collect electronic signatures. The exact methods vary, but generally the recipient will receive a notification, such as via email, asking for a signature and providing instructions. Software options that work to primarily collect signatures include DocuSign and Hellosign.
    • DocuSign has a variety of relatively inexpensive options (ranging from $120 to $480 annually), and was noted as being particularly easy for those being asked to sign a document.
    • Hellosign.com is another relatively inexpensive software (ranging from $156 to $480 annually) that has been used in a recent J-PAL project. The team found it easy to incorporate into the research process, and also easy to upload the consent into the online system of their partner organization.
  • Integrate signature process into survey software: Many survey software systems have the ability to collect electronic signatures. This type of software is often adaptable, and the signature itself may involve checking a box/filling in a bubble to acknowledge consent, a typed signature, a drawn signature if using touchscreen technology, entering a Social Security number (or other PIN/password) for high security situations, or an image of a signature.
    • SurveyCTO, primarily an electronic survey platform, can also collect electronic signatures. Because it is intended for broader use than signature collection, it is more expensive than signature-only software (the lowest-priced advertised SurveyCTO plan is $2,376 annually). SurveyCTO also has a Community Contract that allows free use of the software for up to 200 submissions per month. This software was also noted as being easy for those providing signatures. 
    • Confirmit provides a range of technological solutions including survey design and sampling. Users can integrate electronic consent processes into larger survey design and collection activities.
    • Random Assignment, Participant Tracking, Enrollment and Reporting RAPTER® is a modular system that can scale to support projects ranging from a simple random assignment site to a full-fledged case management and participant tracking system. RAPTER® is designed with a completely secure and cloud-native infrastructure to scale from single-site evaluations to nationwide studies while keeping data safe. Electronic consent and assent can be built into the larger project system.

Choosing the appropriate method

  • Each method has a variety of pros and cons that should be considered for each study. Always consider ethical and legal context, as well as any regulations and recommendations set by the IRB. Broad suggestions about context and signature type are noted below.
  • Lower-tech methods such as collecting a scanned signature, or a signature entered via a Word or PDF document, might be commonplace in some circumstances. Entities that commonly sign legal documents such as contracts or data use agreements, or those who sign documents frequently (e.g., judges or lawyers) may have experience with various acceptable methods for providing electronic signatures. Populations with less experience providing signatures may need more guidance from study staff on exactly how to provide an acceptable signature.
  • Electronic signature software may be useful when collecting signatures from many people, and when populations may have limited knowledge of signing PDF or Word documents. In addition, they minimize troubleshooting with individual respondents and processing emails with attachments (as handling Word/PDF signatures may require).
  • Survey or data collection software solutions (such as SurveyCTO, Confirmit, RAPTER®) offer full integration with data collection. A research team can collect signatures for consent, deliver surveys, and manage data collection within a single system. This method may be particularly attractive for larger projects that have both survey and other data collection components.
Recommended content
Best practices for conducting phone surveys
Author(s)
Amanda Lee
Sarah Hughes
Shawn Marsh

Research Methods

Previous

Next

Inglés
Summary

The protection of staff and respondents who participate in our research is of highest priority for J-PAL. As of March 17, J-PAL has therefore suspended all research activities that contribute to in-person interactions in order to limit COVID-19 spread. Many other research organizations have done so or will do so soon. In order to disseminate expertise quickly and build on our combined networks, we are crowd-sourcing best practices on switching from in-person to surveying online or via phone. 

Date
Image
Mobile phone
Body

Thank you to everyone who attended the webinar with Tavneet Suri and the UBI Kenya team on Tuesday, March 24. We have posted a recording of the webinar, as well as Tavneet's slides, SurveyCTO code, and the modified questionnaire.

The protection of staff and respondents who participate in our research is of highest priority for J-PAL. As of March 17, J-PAL has therefore suspended all research activities that contribute to in-person interactions in order to limit COVID-19 spread. Many other research organizations have done so or will do so soon. In order to disseminate expertise quickly and build on our combined networks, we are crowd-sourcing best practices on switching from in-person to surveying online or via phone. 

This is a living document that aggregates crowd-sourced tips and factors to consider when conducting remote surveys (in particular phone surveys) while practicing social distancing. We will be updating this page regularly. Many thanks to those who have contributed to this collection thus far.

Do you have experience conducting surveys online or via phone? You can contribute via this form. 

Last updated: April 14, 2020

Resources

Four resources that cover many of the topics below are:

Resources from Tavneet Suri's webinar on phone survey use:

Other references (not reviewed):

Obtaining informed consent via phone

  • Check with the IRB of record about moving to verbal consent. If you have approval for written consent, you will need to submit an amendment. If you already have approval for (in-person) verbal consent, you may not need to submit an amendment. 
  • Keep the informed consent script short and use simple, clear language. Be sure it includes the purpose of the call, who is calling (organization and individual), confidentiality, and duration of the survey. Pilot it internally over the phone, to get a sense of the length and whether it is easily understood. You may need to make several revisions before implementation.
  • The survey protocol should log when verbal consent was given. The survey should not continue unless consent was explicitly confirmed by the enumerator. Consent should be given by clearly speaking a short phrase, such as “Yes, I agree”. (Poppy Widyasari, J-PAL Southeast Asia)

See also sample survey protocols for the UBI Kenya project (Tessie Lezcano, Eunice Kioko, and Debborah Muthoki, IPA)

Other IRB amendments: social distancing of enumerators

  • The risk of data loss might increase when enumerators survey from home. Is there a higher risk that devices are lost or stolen? Are enumerators able to upload data to a secure server (and off of their devices) in real-time? Depending on the answers to these questions, you may need to update the discussion of risks and benefits in an amended IRB application. (Ben Morse, J-PAL Global)

Strategies to ensure that participants answer the phone (and stay on the line)

Finding the respondent:

  • Collect multiple contact details for each respondent to maximize response rates over rounds. Herath et al. 2019 compiled tips from running phone surveys in urban centers of South Africa. They collected participants’ current number(s), a family member’s phone number, and a friend’s phone number. Because these details are so important, they recommend putting in place multiple measures to avoid data-entry errors (e.g., double-entry, length constraints, etc.). They also collected email addresses, but found that email wasn’t an effective channel for communication. 
  • You may also collect different contact channels. For example, a survey in Indonesia collected respondents’ WhatsApp number, Line, Facebook, WeChat, IMO, or Skype ID, and found that most respondents preferred WhatsApp or Skype calls.

Making sure the respondent picks up:

  • Sending a text message ahead of time improves response rates. Kasy & Sautmann (2020) in collaboration with PAD conducted a test of how to maximize response rates for phone enrollment to an extension service (see experimental results in this dashboard). Response rates varied between 13 and 20% (“cold-calling” farmers from a government curated list of phone numbers). The most successful was a morning call at 10am, preceded by a text message an hour in advance. Morse et al. (2016) found that sending text messages about 5 minutes before the first call attempt was helpful. The best time of day and the best interval between text message and call may vary, depending on the population. 
  • Use social media as an alternative to SMS to contact the respondents and explain the purpose of the call before calling (provided there is IRB approval/consent for these forms of contact) (Yuna Liang, IPA).
    • Depending on the size of the survey, radio programs are an alternative means of informing respondents about the study (Sarah Hughes, Mathematica)
  • Set up protocols for failed call attempts by making the call at different times of the day (e.g., in 3-hour intervals). This holds whether the calls are made via automated systems or by surveyors. Determine a number of attempts that must be made over several days before considering the respondent unreachable. (Tessie Lezcano, IPA and Grant Bridgman, Uliza).
    • The protocols should include a system for recording when the first attempt was made, including the time of day and day of the week. Tracking the timing of successful calls can tell you when future calls are most likely to be successful. Be sure to pre-test the system. (Sarah Hughes, Mathematica)
    • If response rates remain low, increase the number of attempts (e.g., from 9 to 12). (Tessie Lezcano, IPA)
  • In Indonesia, a 37% response rate was achieved by contacting respondents three times to set up an appointment and then calling at the appointed time. In Turkey, a 75% response rate was achieved just by placing more calls at different times of the day, with a large boost coming from placing calls in off-hours. (Ozler & Cuevas 2019)
  • Build overtime into the study to be able to call participants outside of their work hours. This helps to reach those who cannot speak on the phone during the day, and it ensures that the study isn’t biased by only capturing findings of specific groups. (Herath et al. 2019).
  • Brand your phone number if you can, and build trust and recognition around your brand (put up posters, mention it on the radio, tell node members of the community) (Grant Bridgman, Uliza).
  • In some contexts, obtaining and sending out an official letter, e.g., from the government, before survey start can increase response rates.
  • Run a de-duplication on the phone numbers in your sample so that respondents aren’t called multiple times (Sarah Hughes, Mathematica)

Sometimes the unexpected works:

  • Respondents may pick up when a new number calls. Let enumerators switch their call lists of unreached respondents; the respondent may pick up a new number. (Yuna Liang, IPA).

Completing the survey:

  • Delivery matters: enumerators should sound professional. A polished script can keep respondents on the line. (Grady Killeen, Precision Agriculture for Development (PAD)/HBS)
  • Build on established rapport between enumerator and subject. Morse et al. (2016) used phone surveys to track trends in health service usage during the Ebola crisis in Liberia. They had the advantage of surveying respondents in person at baseline, using phone surveys for follow-up. When possible, the same enumerator was assigned for both baseline and follow-up. 
  • Positive, unambiguous interactions are helpful for re-establishing contact. Be clear with participants at the outset on when and how communication will take place over time, and be open about how long a survey is likely to take. Also explain why the survey is being conducted over the phone, particularly if earlier contact was in person. (Herath et al. 2019 and Michael Kleinman, Orange Door Research).
  • Compensating respondents via mobile money or airtime for the time spent on the survey and the costs incurred from having to charge phones makes participation easier and more attractive for them.
    • In a survey in Ghana, we paid 3 cedis (about $1.30 US) per call and had 85% completion rate (and if a call was missed, the vast majority of the time, the person answered the next call: complete dropout was rare.)  (Rachel Heath, University of Washington).
    • If airtime can’t be sent electronically, surveyors can purchase airtime refill cards and read the code off to respondents after the survey is complete. (Ben Morse, J-PAL Global)
  • Including the duration of the survey in the consent procedure will reduce instances where the respondent runs out of time or phone battery and cuts the interview short.
  • Build in as many options as possible to do the survey at the respondent's convenience. For example, rescheduling to another time or day, calling on another phone number, or allowing the respondent to interrupt to take incoming calls.
    • Be sure that a refusal is in fact a real refusal, rather than a respondent who just doesn't have time right then. (Tessie Lezcano, IPA)
  • If you are going to be conducting phone surveys with the same respondents over time, ask them in the first survey what time is appropriate to call them during weekdays or weekends. This is particularly important for rural households who go to the field, have poor network connections, share phones or charge their phones irregularly. This kind of information can also help you plan your survey better when working on a different project in the same area or region. Knowing your targeted respondents patterns of life and phone usability is key. (Simon Rubangakene, PAD)
  • With participants with access to smart phones, email, and/or computers: use a combination of electronic notifications and provide multiple options for response (e.g. email, text messages, app pushes for notifications, and for responses, telephone call-in to recorded question/answer, link to website, texting back and forth, emailing, app questionnaire). Use telephone calls only for non-response follow up. Make the number of questions, and the questions themselves, short; if need be, survey frequently (e.g., every three weeks).  Compensate for completed surveys immediately. We have used these methods in the U.S. and have achieved response rates of over 85 percent (Jim Greiner, Harvard Law School).
  • Know particular contexts and the usability of SMS or online services. For example, in some places, SMS is not widely used and so may not be an effective means of communication. (Simon Rubangakene, PAD)

Data security and technical solutions for in-home call centers

  • Surveyors may have trouble finding quiet places to conduct surveys. Consider buying them noise cancelling headphones, altering working hours to take place during quiet times, and helping them set up a room to avoid sound disturbances (e.g., by filling it with soft furniture). (Sarah Hughes, Mathematica)
  • Surveyors may need equipment such as a 4G adapter, dongle, or SIM card. If possible, try to deliver to them directly so they do not have to go to the market to make purchases. Put enough money on the SIM card for the survey (or set up a system for sending surveyors phone credit), but strongly enforce the message that the SIM card is for the survey only. (Sarah Hughes, Mathematica)

Commercial or open source software or SAAS solutions

  • SurveyCTO is developing a starter kit to facilitate the transition from CAPI to CATI using the SurveyCTO Collect mobile app and will have a software update that includes a dialer plug-in and the ability to create field plug-ins at the end of March. More COVID-related resources and updates from SurveyCTO can be found here. (Chris Robert, SurveyCTO)
  • Twilio is a good platform for sending SMS and can also be incorporated into your customized application (available countries here). (recommended by several people)
  • Other commercial options include Telerivet (worldwide), Vonage APIs (formerly Nexmo), Africa’s Talking, Geopoll (CATI and SMS), Viamo (IVR), engageSPARK, and Survey Solutions (Grant Bridgman, Uliza; Sarah Hughes, Mathematica; and Michael Kleinman, Orange Door Research)

  • Uliza.org specializes in putting together interactive voice response (IVR) solutions in local languages. These are automated calls that allow respondents to answer in different ways. Uliza has also developed voice-based ID software using unique voice characteristics. This is useful when you have repeated engagements with someone over the phone to verify that you are talking to the person who gave the initial consent. Uliza also provides a (human mediated) service that allows researchers to receive open-ended voice questions/comments from their sample in local languages, and give answers that will be transmitted back in local-language voice (Grant Bridgman, Uliza).

  • If an internet connection is available, VOIP may be an option. VOIPStudio can be configured so that calls are coming from a real number, not an unidentified number. (Eduardo Vargas Sanchez, J-PAL LAC)

  • 60 Decibels specializes in designing and implementing large-scale voice, SMS and IVR surveys for the social sector, focused primarily on sub-Saharan Africa, South Asia and East Asia. 60 Decibels partners with aid organizations, social enterprises, impact investing funds, corporations, foundations, civil society organizations and donors. (Sasha Dichter and Ashley Speyer, 60 Decibels)
  • See pages 6-8 of the Remote Survey Toolkit from 60 Decibels for pros and cons of different technology solutions, as well as a list of remote survey providers. (Sasha Dichter and Ashley Speyer, 60 Decibels)
  • Though currently at capacity, Orange Door Research specializes in designing and implementing large-scale CATI, IVR, and SMS surveys for the development and humanitarian sectors, focused primarily on otherwise difficult-to-access geographies. Orange Door Research partners with UN Agencies, the World Bank, academics, civil society organizations, and donors. (Michael Kleinman, Orange Door Research)
  • Try working with local service providers: In Benin, we are working directly with the mobile service provider MTN to use StandardPro, a service that will permit recording the calls and that can help us with managing the phone dialing in a distributed fashion (Sarah Hughes, Mathematica)

Best practices for modifying survey protocols for phone surveying

  • Keep phone surveys short: at most 30 minutes. Figure out your primary outcomes and be sure those are included. You may need to aggregate questions, including some outcomes. If your survey is especially long, consider breaking it up into multiple phone calls. (Tavneet Suri, MIT)
  • Eliminate questions that are sensitive as far as possible (for example on stress, mental health, sex, domestic violence, etc) as these are harder to administer over the phone. Simplify or eliminate questions that require a lot of probing.
    • If you do need to ask sensitive questions and the respondent is surrounded by other people, ask if they can be called back later when they are alone. (Tessie Lezcano, IPA)
  • For new surveys: Before starting the main survey, conduct a check on a small sample of phone numbers to see how many are unreachable/ wrong numbers/ etc. This will give you an upper bound on the possible response rate, and you can adjust your sample size accordingly.
  • For new surveys, you may also need to adjust for lower response rates; 50% may be a good rule of thumb, though response are generally higher in the places we work than in the US. (Michael Kleinman, Orange Door Research)
  • Respondents may be charged for incoming phone calls in some locations. Consider a protocol that sends them some airtime credit before calling. 
  • Created drop down menus for question responses where possible; even if they are long, doing so can reduce entry errors and reduce the time enumerators spend typing (Grady Killeen, PAD/HBS and Michael Kleinman, Orange Door Research)
  • Random digit dialing and asking the village chief/elder for phone numbers are two options for sampling when the research team doesn't have access to a comprehensive list of phone numbers. Random digit dialing is just what it sounds like: set the relevant prefixes (e.g., country code), then randomly generate numbers to call. (Tavneet Suri, MIT)
  • The Busara Center has tips for adapting lab-in-the-field protocols to be done via phone; many of these tips also apply to field projects done outside the lab. Topics include specific suggestions for changing protocols based on the target sample and study protocols, as well as alternatives to focus group discussions. (Busara Center via Wim Louw, J-PAL Africa)
  • In countries where many languages are spoken, track languages on the tracking sheet so that respondents can be reassigned to a surveyor who speaks the language. (Tessie Lezcano, IPA)
  • See also sample survey protocols for the UBI Kenya project (Tessie Lezcano, Eunice Kioko, and Debborah Muthoki, IPA)
  • Though we are not qualified to provide medical advice, it may be useful for surveyors to have information about the nearest health clinics on hand, or to remind respondents about the importance of hand-washing and social distancing during the conversation (Tavneet Suri, MIT Sloan; Subha Mani, Fordham; Bidisha Barooah, 3ie)

Hiring, training, and data quality when enumerators work remotely

Hiring

  • Ensure fluency in the local languages represented in your sample. (Sarah Hughes, Mathematica)

  • Include in the scope of work and surveyor contract that the phone numbers for your sample may not be used for any other purpose than your survey. (Sarah Hughes, Mathematica)

Training

  • Find a way to conduct small group calls, so one enumerator can conduct the interview with a pilot respondent, and other enumerators can listen in. After the interview, discuss what went well and what could be improved. Don't forget to ask respondents’ consent to be on the phone with several enumerators. (Yuna Liang, IPA)
  • Record a test interview to listen to as an alternative to a group call.
  • You may need a higher trainer to trainee ratio and more time than previously budgeted for if you are doing a remotely based training (Grant Bridgman, Uliza).
  • Use mock scripts (not just role-play) for training and testing each interviewer's ability to dial the correct number, gain cooperation, administer each question correctly and enter each response. Create a confidentiality agreement that each interviewer must sign. (Sarah Hughes, Mathematica)
  • Depending on the size of the team and at-home internet access, training of new enumerators can be done remotely via Skype, Zoom, or similar (Tavneet Suri, MIT Sloan)

See also Herath et al.’s (2019) blog post for general tips on training and practicing.

Keeping data quality high:

  • Increase the frequency of back-checks and HFCs, as well as the amount of time the field manager and RA spend looking through data, e.g., at least 50% of the field manager’s time daily. One target for back checks is to do them on 20-25% of respondents (up from 10-15%). (Tessie Lezcano, IPA and Sarah Hughes, Mathematica)
    • Pay extra attention to filter questions (skip logic): questions for which answering "no" skips a module or set of questions. (Sarah Hughes, Mathematica)
  • Use back checks to confirm that consent was given. (Grady Killeen, PAD/HBS)
  • If it is hard to find back check questions that should not have changed from the original survey, take a different approach. Ask questions such as, "Was the interviewer polite?", "Were you offered an incentive?", and "How long did the survey take?" (Sarah Hughes, Mathematica)
    • Check the respondent's reported duration against the timestamped data.
  • In the endline survey, include questions that should not have changed from the baseline survey. (Tavneet Suri, MIT)
  • Supervising field staff should call each field team member daily for a quick check-in and to go through any questions they might have, as well as go over their tracking sheet. (Tessie Lezcano, IPA)
  • You may consider recording calls for quality control and to conduct spot checks as long you have IRB approval and the subject gives their consent. Another surveyor can fill the survey questionnaire from the audio recording in order to conduct data analysis (e.g. determine error rates) on the back checks.
  • Set up a Whatsapp or Facebook chat group for each enumerator team where they can share issues or ask questions, and report each successful interview (this is particularly motivating if the supervisor gives positive feedback and encouragement, or even disburses small rewards such as phone credit). Research team members should be in the group as well to answer any questions. (Tessie Lezcano, IPA and Yuna Liang, IPA)
  • Use the SurveyCTO function of exporting data to google sheet to make a live monitoring tool of survey progress. Set the SurveyCTO app to auto sending completed forms, and use the google sheet monitoring dashboard to auto-update with incoming data. (Yuna Liang, IPA)
  • Looking at call metadata can help spot unusual patterns, such as an enumerator making very short calls. (Rachel Heath, University of Washington).
    • SurveyCTO also has a sensor metadata function that can capture the volume level of the environment where the SurveyCTO form was filled. While there is no set standard of “reasonable values,” outliers on the lower end (e.g., 10 decibels) could flag surveys that require additional scrutiny. (Yuna Liang, IPA)
    • Add the start and end time to SurveyCTO to see call duration. Call logs can also be used to see if field staff are making calls during the correct timeframe. (Tessie Lezcano, IPA)
  • Use the same team that you trust whenever possible but also bring in new enumerators to check patterns in the data for recurring surveys. (Simon Rubangakene, PAD)

Evidence on data quality

Garlick et al. (2019) compared weekly in-person, and weekly phone surveys for a 12-week microenterprise survey panel. They report: “The results show few differences across the groups in measured means, distributions, and deviations of measured data from an objective data-quality standard provided by Benford’s Law. However, phone interviews generated higher within-enterprise variation through time in several variables and may be more sensitive to social desirability bias.”

Webinar Recording: Adaptions for phone surveys with Tavneet Suri

J-PAL office(s)
J-PAL Global
Author(s)
Sarah Kopper
Anja Sautmann

Research Methods

Previous

Next