Working with a third-party survey firm

Research teams may decide to contract a survey firm rather than run a survey in-house if the research organization has limited capacity or experience working with a given population, and there are established survey firms in the area. However, survey firms may not have experience working on randomized controlled trials (RCTs). This resource draws on the experience of J-PAL staff and researchers across regions on many different RCTs. It provides guidance on identifying and engaging a firm, as well as recommendations on how to ensure survey activities will suit the needs of the RCT. The resource assumes the research team is designing, reviewing, and approving the study and the survey. This guide limits coverage of survey or study design to elements most critical to review with the survey firm. Any examples are meant to be illustrative rather than prescriptive or comprehensive. 

The decision of whether to run a survey in-house or to outsource it depends on a number of factors: 

• Consider the capacity of the researcher’s organization to run a particular survey in-house. Survey firms may have existing infrastructure to handle complex or time-consuming processes, such as identifying and hiring the appropriate number and type of staff (both surveyors and supervisors) and the time associated with handling logistics and field team management (setting up transportation or accommodation, paying surveyors, etc.).
• Assess local knowledge, including experience working with the population of interest, languages spoken, modes and routes for transportation, and where to find accommodation. Consider whether a survey firm has more or less knowledge and experience than the research organization in working with the intended population. If there are no survey companies with experience conducting projects in the area or with the population of interest, consider hiring surveyors in-house. 
• Compare the level of experience working on RCTs. Survey firms may be more accustomed to running surveys outside of academic research or with non-RCT methods; these firms may not have experience doing RCT-specific tasks, such as randomizing or checking regularly for consistency in data collection procedures across treatment and control groups. When hiring a survey firm, it is critical to build systems into the contract to ensure high-quality work. Researchers with less experience in data collection who are hiring a firm for that firm’s experience must still be able to recognize good-quality work. Key considerations are outlined in the final section, Ensuring a Successful Study. 
• Weigh the cost. Running a survey is expensive whether conducted in-house or through a firm. Consider the full cost in either case. See the section on Negotiating a contract for specific items that may be negotiated. 

If time permits, an open-bidding process can be useful for comparing firms on metrics such as cost and value-add, and the different budget proposals will provide an idea of which items can be negotiated to improve cost-efficiency. If time is limited, single-source bidding (where a known or frequently used survey firm is solicited and then contracted) may be an option. Researchers should work closely with their organization’s procurement team to comply with any contracting requirements; universities and other organizations may require a competitive selection process for purchases or services priced over a certain threshold. They may also have specific requirements for the selection process, contracts, or terms¹. The process described below is typical of an open bidding process, though the timeline may vary based on the capacity of the team and availability of survey firms. 

Identifying survey firms

Local partners, other researchers who have worked in the study area, and desk research can be helpful for identifying a list of possible survey firms. For example, some J-PAL offices have a database detailing their previous work with survey firms. Consider the following factors when identifying and comparing potential survey firms:

• Survey firm’s expertise: Does the survey firm have experience conducting a similar study or a comprehensive understanding of both the study topic (e.g., education, health, energy) and location? Does the firm have particular knowledge about the study target population?
• Research experience: Does the firm have experience running a survey as part of an RCT or another type of impact evaluation? The expertise of high-level members such as advisors or directors can add useful information (e.g., those with a research background). 
• Network: Does the firm have an established network of relevant stakeholders or vendors (e.g., printing firms, address and phone number providers) that work in the geographic area or topic of your research? 
• Survey methods and technology: Does the firm have a specific set of skills, such as knowledge in survey methods (mail, phone, in-person) or technologies, that may improve the data collection process?
• Scale: Has the firm successfully conducted surveys at a similar scale and scope of the proposed survey?
• Organization structure: Does the firm structure fit the needs of a specific study? Surveys that require tailored training for surveyors or survey management by the research team may benefit from the structure of a smaller firm where employees are closely supervised. However, a large-scale study that covers a wide geographical area or has a substantial sample size may primarily require additional workforce, and large survey firms may have a stronger comparative advantage.  
• Initial information on cost: Is there information available about the cost of services? While specific budget estimates will be gained through the request for proposals (RFP) process, information about cost from previous engagements or general knowledge (sometimes specific firms are known to be particularly expensive) can help weed out firms whose costs are known to be too high for a particular engagement. 

After collecting and comparing information on an initial set of firms, the next step is to notify prospective firms. Organizations may have a particular system in place for these steps; if so, that process should be followed. The following section provides general information on what these steps may look like.

Developing Terms of Reference

A detailed description of the desired survey activities, as well as of the context and aims of the overall evaluation, is essential for potential survey firms to assess whether and how to bid for a survey contract. This description is detailed in the Terms of Reference (TOR). The set of activities a firm will be hired to perform, whether a single round of data collection or all survey-related activities, are outlined in a sub-section of this document: the Scope of Work (SOW).  While the Scope of Work for which the firm will be contracted may cover only a small subset of overall evaluation activities, the Terms of Reference should provide enough context that the firm understands how the contracted work fits into the project overall. 

Specific formats and processes of developing a TOR may vary depending on the project, organizational requirements, and local practices. A typical length is 5–10 pages, with appendices supplementing necessary information when appropriate, and the TOR is usually attached to the legal contract. Please see this generic TOR for a baseline survey as an example. Additional resources for writing TORs are available in the resources section. The following outlines the major components that may be included in a TOR, as well as brief descriptions and related considerations.

1. An executive summary consists of a brief overview of the project, including the research objective and outcome(s) of interest. 
2. An introduction or contextual information section briefly orients firms to the history and complexity of the research project. The context may relate to the area where the data is collected or social, regulatory, or political factors that may influence the project implementation as it relates to contracted activities. 
3. An evaluation background and purpose frames the project, which helps firms understand the extent of the full evaluation. This can include the theory of change², the full timeline and geographic area of the study, and a summary of existing or planned activities related to the evaluation³. 
4. A scope of work details the services the firm will be contracted to perform and the approach and methodology that the firm must take. This section includes key elements such as the sample size, process indicators to be measured, and availability of relevant data. Elements to include may vary with each project depending on the intervention, the process of data collection, and the activity to be subcontracted. This section can specify, or request firms to propose, any bonus or incentives schemes given to respondents and surveyors to ensure high-quality data and high response rates. In a competitive bidding process, assessing how firms interpret the approach and methodology of the data collection process is key to determining how much they understand. This section of firms’ proposals will allow candidates to differentiate themselves in terms of quality and how they plan to carry out the methodology proposed by the research team. Specifying the approach or providing an outline of activities can be the most challenging part of creating the SOW. A complete and detailed set of activities is not always possible, especially if contracting for the development of a process (for example, a pilot survey), when it can be difficult to project a budget, timeline, and potential activities. In these cases, it is important to highlight that the SOW is for a pilot and that a certain amount of exploration is required. Emphasize what process or materials need to be developed within the SOW but include as much detail of known parameters as possible. For example, identify whether there is a fixed budget, set timeline, or sample size. 
5. Integrate quality assurance, including any procedures, such as checks and monitoring, that firms must perform to ensure high-quality data collection, as well as any required data security procedures. These steps may include requirements to independently back translate (i.e., re-translate an already-translated questionnaire or other document back into the original language) and pilot questionnaires before data collection begins; regularly monitor and audit the activities of surveyors; track the number of non-responses and have a strategy to address issues such as attrition; conduct double-data entry if using paper data collection; and regularly check error rates. Finally, this section can require the firm to conduct field monitoring in coordination with the research team or to allow time for members of the research team to review all training materials and attend trainings. Specific considerations with respect to quality assurance are detailed below in the Ensuring a Successful Study section.
6. A section outlining the key personnel and qualifications required to carry out the evaluation includes the size of the team needed, the roles of each team member, the desired credentials or experience of each team member, and the role of team leader. Firms should submit curriculum vitae, as well as references or examples of past evaluation reports.
7. The timeline and deliverables are detailed, explicit, and reasonable. Evaluate whether deliverables are feasible, given the project budget and timeline. In addition to the SOW, this section requires careful consideration especially in the case of a pilot or other cases where ambiguity remains⁴. Clearly discussing expectations in the TOR can prevent misunderstandings later in the survey process. Guidance on deliverables will generally include the following information: 
   1. Type of product: This can include real-time or periodic transfers of raw and/or processed data, codebooks, translated questionnaires, written study protocols, data security plan, training materials, implementation and field monitoring reports, reports on response rates, reports on quality assurance (e.g., a summary of high frequency checks), and products required by the project funders. Ideally, each product will be listed separately to specify each requirement and the timeline of when the firm should submit the product. 
   2. Structure and format of the product: If a written product, include the length of the report, the main content, and in what language it should be delivered. If a data product, include the format, file type, and documentation that should be delivered. Specify whether live access to data, raw data extracts, or formatted reports are necessary.
   3. Timeframe for product: This includes milestones with deliverables and timelines clearly listed (e.g., in bullet points or in a Gantt chart). 
   4. Required meetings or coordination: This establishes points of contact between the firm and the research team (e.g., weekly email updates, regular phone calls, or a written summary of insights from surveyors on lessons learned from conducting the survey). 
8. The budgets and payment can be open-ended or prescriptive. The TOR can state the overall budget and ask firms to propose what they are able to achieve within that budget. Alternatively, the TOR can specify budget items that should be included and ask applicants to budget according to these categories. The former may be useful in early stages of the research design, while the latter approach can provide a clearer comparison of each activity’s cost across companies. This section explains payment details, including the process for invoicing, the timeline, and the means of payment. Firms should be asked to submit a financial proposal with a spreadsheet or table of the proposed budget and a budget narrative that both justifies the cost and demonstrates that the budget is adequate to complete the required activities listed in the TOR. Requiring transparency in cost calculations, such as asking firms to specify unit costs of labor and materials, can be helpful in evaluating trade-offs between survey firms. It can also be helpful to build flexibility in the contract, including in the budgets and payment section. For example, if contracting for a pilot, or project where uncertainty (say around response rates) could alter methods substantially, one option is to ask for costs on a range of possibilities and negotiate on a menu of prices. After piloting different methods, or once uncertainty resolves itself (e.g., response rate becomes known), the research team is then able to choose the appropriate methodology based on a negotiated and known set of costs.

The TOR should be reviewed by relevant teams (e.g., the finance or contracts personnel at the research team’s institution) to ensure it is consistent with internal contracting and procurement procedures. Once the TOR is finalized and released, the applicant firm may wish to discuss and clarify the TOR.

Reviewing proposals from firms 

After the submission of proposals, two or three firms should be shortlisted, though this number can depend on a research institution's policy or the quality of submitted proposals. The shortlisting process is necessary to strategically decide the survey firms with which to meet for clarification and negotiations before signing a contract. To weigh the strengths and weaknesses of each company objectively, it is advisable to create a set of parameters based on predetermined criteria as described above. This process should include an initial budgetary review to seek which proposal gives the best value and is likely to deliver quality work. Creating a table that summarizes how each company’s proposal addresses the set of criteria will be helpful (see J-PAL's Survey Firm Assessment template (.xls direct download) for an example). It is also important to conduct reference checks either before shortlisting a firm or before finalizing the contract. 

Meeting with shortlisted firms

Meetings with shortlisted firms can be used to confirm whether they can deliver on the proposal and to discuss further details of the project. This may be a good time to discuss more ambiguous items in the TOR (e.g., if contracting for a pilot, to determine if the research team and firm have a similar understanding of the activities a pilot may include). In addition, ask questions about the submitted SOW and at what point a contract modification would be required. For example, confirm if the firm’s understanding of a pilot allows for changes in design, such as additional mailings, surveys, or phone calls.

These meetings can also be a good time to discuss details of the firm’s data collection and quality assurance procedures to understand whether they are able to deliver the quality needed for your survey. It is common practice to state that the initial meetings are part of the bidding process and that the team is still in the process of considering which firm to contract. 

Negotiating and finalizing the contract

Negotiating the budget

Negotiating the budget can be a long and complicated process that typically runs in parallel with the contracting process. The goals are to maintain a good relationship with the firm and get the project running within budget and without sacrificing quality. Before entering a negotiation meeting be sure to determine the following: 

1. Assess whether the budget reasonably reflects all research activities. If available, compare this budget to those from similar projects or budgets that the firm has submitted for previous projects. It is advisable to coordinate with someone with experience and local knowledge in this area, such as a local partner, a research manager, or a member of the finance team.
2. Understand the budget limit and items that cannot be compromised. 
3. Note the items to negotiate. Based on our experience, key budget items that have the potential to be negotiated include the following:
   1. Questionnaire design (including programming and translation) may be done more cost-effectively by using resources available within the research team or organization. 
   2. Respondents' incentives (compensation for the time spent participating in the survey) depend on the type of respondents, the survey length, and the sample area. Survey firms may have experience that enables them to advise on suitable incentives for respondents (e.g., cash, vouchers, items). Consult the firm to identify an incentive that is within budget, logistically feasible to transport and distribute, and appropriately compensates the respondents for their time⁵. Buying in bulk may cut the costs of incentives. 
   3. Institutional fees (may also be called overhead or management fee) can vary by firm. If the shortlisted firm has a higher fee and the overall cost is a concern, see if this is negotiable.
   4. Miscellaneous items should be evaluated to determine how essential they are for the project and whether they can be done in-house or in an alternate way (e.g., in person vs. phone surveys).
   5. Confirm that the survey company has not missed any key activity in the data collection process (e.g., pilot, listing, conducting spot checks, and back checks). If there are any missing activities, consider revisiting the budget or SOW to ensure all key activities are included.

The negotiation process can be delicate. As such, it should be led by senior-level staff, such as the project researchers, managers, or finance personnel. Staff leading the negotiation should come prepared with an understanding of the available budget, the flexibility in both overall budget and specific line items, and how an overage would impact the rest of the project; additionally, staff should be aware of the necessity of maintaining good relations with the firm, particularly if it is the only viable candidate. 

Creating a contract

After the survey firm is selected, the research team should develop a contract in coordination with the relevant team within their organization (such as the finance team, procurement office, or the office of general counsel). Such teams may have template contracts available and will be familiar with legal provisions to ensure the research team is protected in case of breach of contract. 

The contract should address the rights and obligations of the parties involved. It should be specific and include the activities and requirements described in the TOR (often the TOR forms the basis for the contract or serves as an appendix), including any deliverables or requirements that have been further specified through the proposal and negotiation process. The contract should also include consequences that may apply if the firm fails to deliver quality work. The more specific the contract, the easier it is for the survey firm to know what is expected and for it to be legally enforced or terminated. In addition to items from the TOR, the contract may include the following elements: 

• Financial reporting requirements
  • For example, this could include report and projections of costs in a certain format following a specific timeline.  
• Timeline and deliverables
  • In addition to the timeline and deliverables as described under Developing Terms of Reference, however, the contract should update the TOR as needed with specific (vs. relative) dates, activities, and deliverables, and specify payments or incentives associated with meeting (or failing to meet) deliverables. 
• Data ownership terms
  • The research team typically owns all data collected by a third-party survey firm, including respondents’ answers and contact information. 
• Confidentiality agreements
  • This restricts firms and their employees from sharing data and requires that personally identifying information (PII) be encrypted. 
• Contract amendment process
  • Amendments may be needed to modify the timeline due to delays or unforeseen conditions or to modify the SOW, such as in response to findings from a pilot or requests from the project partners or donors. 

After drafting a contact, expect to negotiate and modify the draft to meet the needs of both the researchers and the survey firm. Once the research team and survey firm agree on the contract, it should be reviewed by the finance, legal, or contracts team at the researcher’s organization and signed by someone with institutional signatory authority. 

Conducting surveys as part of an RCT requires specific planning and coordination that may be new to otherwise highly experienced survey firms. For example, firms may have less experience ensuring balance across treatment arms and ensuring proper sequencing of survey activities with intervention activities and/or across treatment arms. Researchers can design deliverables, milestones, and payment methods to ensure that incentives to collect high-quality data are aligned and can set expectations that enable real-time insight into project activities. Research teams should clearly communicate RCT-specific considerations and data standards to the survey firm. Discussing roles and expectations around these items during the negotiation and contracting phase, as well as including those expectations within the contract, can help clarify the metrics that must be achieved, deliverables that are expected, and indicators that must be tracked. This section highlights topics specific to RCTs and study design, general study considerations, and project management recommendations to help ensure a successful study. Communicating about these topics in a way that acknowledges the survey firm’s expertise (the reason they were contracted) and identifies how certain activities or requirements feed into the overall study may allow for challenges to be identified early and for open conversations that can lead to resolution without blame. 

RCT-specific and study design considerations

Overall study design and effects on RCT procedures, power, and sample size

Survey firms may have expertise in the mechanics of delivering surveys and identifying representative samples, but do not necessarily have experience recruiting, randomizing, collecting informed consent, and enrolling participants into a study. Work closely with the survey firm to ensure their expertise in survey implementation is complemented with these critical considerations for an RCT: 

• The study intake, consent, and random assignment notification processes are designed to maximize response rates in the current and future rounds of the survey, maximize compliance with treatment assignments, and encourage accurate responses. The timing and implementation of these processes can have major implications for statistical power, bias, burden on respondents and/or surveyors, and cost. See the Define intake and consent process article for more information. 
• Expected response rates will be adequate to achieve the necessary statistical power, and the respondents will be representative of the intended sample. Based on the experience of the survey firm and/or desk research, identify expected response rates for the full sample and any critical sub-groups. Work with the firm to explore ways to maximize response rates and allocate survey efforts appropriately. For example, if the study covers multiple geographic locations, expected response rates may vary by region. Balance the number of individuals contacted in each region to account for this difference while maintaining an expected sample that is representative of the selected population and large enough to maintain statistical power. If balance along particular characteristics is important to the research design, ensure the survey contract specifies the composition of the sample, rather than just the overall size. See J-PAL's Sampling and Sample Size lecture slides.

If the survey firm is performing random assignment, emphasize that treatment assignment is not manipulable. The research team should specify which randomization checks (e.g., running summary statistics on a set of predetermined characteristics) the firm will run and what output is needed to confirm randomization was done correctly. This is especially important if the research team will not have access to the randomized list or raw data. The Evaluating technology-based interventions article provides additional guidance on how to check randomization procedure and sample when another entity is implementing the random assignment⁶.    

General study considerations

Considerations to protect the study population 

Appropriate survey mode and length 
Surveys can be delivered in-person, over the phone, via mail, or online. Each mode has trade-offs in terms of cost, accessibility, and time required for the participant, and some may be better suited to ensure accurate answers as well as the privacy and comfort of the respondents. While a firm may have a standard mode that works in most cases, review the survey and procedures with the specific survey questions, respondent needs, and respondent burdens in mind. Especially for surveys on sensitive topics, ensure the survey mode, conditions, and location will provide adequate privacy for the respondent. This is covered at greater length under Survey design.

Enumerator characteristics
Ideally, surveyors will have demonstrated sensitivity and familiarity in working with the community that comprises the study sample. This can facilitate understanding between surveyors and respondents and increase the respondents’ comfort levels, which can help to achieve dual aims of maximize response rates and accuracy while protecting and minimizing burdens on respondents. For example, for an in-person survey with young mothers, consider whether female enumerators would be preferable to respondents. If the research population comprises individuals who have experienced trauma or if the survey questions reference painful subjects, enumerators need to be equipped to interact with these individuals without inflicting further trauma and prepared to respond appropriately to signs of distress or requests for assistance related to the survey topics.  If the survey firm has previous experience working with the target population, they may have robust procedures and plans already in place. Verify that the survey firm will identify enumerators with the appropriate skills, experience, and characteristics. Ensure that enumerator trainings are appropriate for the survey topics and population and include training on how to be sensitive to the needs of the community. Surveyor training is covered at greater length in the corresponding resource.

The CITI Human Subjects Research training provides an introduction to considerations for special populations including minors, those with decisional impairment, and pregnant women/neonates. This is also covered further in Ethical conduct of randomized evaluations.

Survey monitoring

Data security measures
While a survey firm likely has established data security procedures, verify that all data security measures used, such as encrypting PII, using anti-virus software, and enabling remote data deletion capabilities, comply with your organization’s standards. See the Data security procedures for researchers article for an overview of data security measures to use when working with data. 

Data quality and cleaning
Include a requirement in the survey contract to share a data codebook or handbook, a description of the data storage plan, a description of data cleaning procedures, and other study protocols. If possible, work with the survey firm to receive access to raw data. Sometimes a firm will be unable to share raw data or provide access to the data directly. In this case, written protocols and plans can facilitate researchers’ understanding of how the firm plans to clean and manage data in order to verify that procedures align with the goals of the research study. See more under Data cleaning and management.

Data monitoring and management
Confirm with the survey firms what data quality procedures they use and identify whether additional actions are needed to ensure the resulting data will be suitable for the research study. Processes recommended by J-PAL include high-frequency checks (checks that illuminate problems in real-time related to data quality, enumerator performance, survey instrument, or the data flow), back checks (short, audit-style surveys of respondents who have already been surveyed), and spot checks (unanticipated visits by senior field staff to verify enumerators are following data collection protocols and surveying at the correct times and locations). A monitoring plan may randomly assign spot checks across enumerators or may prioritize visiting surveyors who were identified during training as needing extra support. In some cases, research staff can conduct some elements of data monitoring themselves. Having research staff directly conduct spot checks allows the researchers to directly observe the survey or data collection procedures, thus developing deeper insight into the data and allowing for a more nuanced analysis⁷. In coordination with the firm, plan who will be responsible for data monitoring activities, and confirm how the firm will notify the research team and address any discrepancies, errors, or challenges identified. See J-PAL’s Data quality checks resource for additional recommendations on data monitoring.  

Surveyor incentives
Compensation and incentive schemes for surveyors can have implications for survey response rates and data quality. Incentives should encourage enumerators to follow protocols and collect accurate data (for example, rewarding surveyors who collect high-quality data as verified by back checks and high-frequency checks) rather than focusing exclusively on the number of interviews. If tracking number of surveys, which may be helpful to ensure timelines are met, ensure that the targets are achievable within the specified timeframe. Depending on the firm, context, and project, researchers may have little to no control over surveyors and how they are paid. During the proposal review and interview stage, work to understand whether it is possible to design or adjust incentive structures to meet the study’s needs or to ensure established procedures do not pose a threat to data quality. Confirm the survey firm follows fair labor practices. This is discussed more under Surveyor hiring and training.

Surveyor training
Surveyor training—which should encompass training on the survey questions, survey protocols, as well as the survey instrument and related technology—offers a key opportunity for the research team to communicate directly with survey staff. Consider having research staff conduct, co-lead with the survey firm, or attend the surveyor training. Review training plans and protocols prior to the training to allow time to modify as needed. See J-PAL’s Surveyor hiring and training resource for information on how to develop enumerator trainings as well as strategies for retraining, as necessary. 

Surveyor performance
Discuss how the firm tracks individual enumerator performance and decide whether the procedures are sufficient for the study. Performance metrics may include per-surveyor information on number of surveys completed, results of back checks or other quality checks, and survey duration. It may also be useful to have a surveyor indicator included in the dataset (for example, to examine enumerator fixed effects). 

Surveyor support
A forum for surveyors to discuss difficult cases, troubleshoot, or ask questions can be helpful. Depending on the study’s aims and population, surveyors may be likely to experience distress or other emotional reactions. Support for surveyors is especially helpful in these cases. 

Project management recommendations

Using a third-party makes it more difficult to manage and monitor the study’s progress and budget. Below are management strategies to help ensure the project runs smoothly. 

Managing timeline

The survey(s) for which the firm is contracted may be only a subset of the overall project activities; for example, the research team or another entity may be responsible for implementing the intervention. To ensure the correct sequencing of survey and implementation activities, research teams may need real-time information on the survey activities, status, and timeline. The high-level timelines and due dates articulated in the TOR and/or contract likely will not include all interim steps required to complete all deliverables, and therefore may not be sufficient for use to track progress on the survey. The research team should consider asking for a detailed timeline or Gantt chart of activities and dependencies. Set expectations and create a plan with the firm to check in frequently on the timeline and to establish a system to notify the team if the firm anticipates delays. 

Managing budget

Being specific and thorough in creating a SOW at the RFP stage enables more accurate budget estimates. Nevertheless, project delays, project changes, or unforeseen circumstances can necessitate modifications to either the budget or the activities. Requiring detailed invoices or spending reports as a condition of the contract, reviewing those reports regularly (e.g., monthly), and discussing reasons for deviations from expected amounts, can allow the research team and survey firm to identify and mitigate challenges quickly. Mitigations may include adjusting the survey scope or methods or increasing the allowed budget. If there are indications the survey will go significantly over budget, refer to the financial terms in the contract and consider whether a contract modification is required. 

Last updated June 2020.

These resources are a collaborative effort. If you notice a bug or have a suggestion for additional content, please fill out this form.