Who Participates in Randomized Evaluations? 


The question of who participates in a randomized evaluations touches on some of the most sensitive issues faced by an evaluator. In answering this question an evaluator must consider what is ethical and fair. It would be unethical, for example, to deprive a household of a water treatment solution for the sake of an experiment if the household would have otherwise had access to the solution.

1.    Ethical Issues

So how can an evaluator conduct an experiment while still meeting fair and ethical standards?

Randomized evaluations can be appropriate in situations when there are resource constraints. Typically an organization does not have a large enough budget to provide everyone in a community or district or country with a program. Because of budget constraints an organization must decide who receives the program and who does not.  Even if they target the subgroup of people who particularly need the program, or those who would benefit most, they are unlikely to be able to cover everyone even in the target subgroups. This provides an evaluator with an opportunity to conduct a randomized evaluation. An evaluator can introduce an element of randomization into the decision of how to allocate scarce resources within the target subgroup.

An evaluator must not only ensure that an experiment is ethical, but that it is also fair. In randomly assigning participants to the control or experimental group, an evaluator should ensure that everyone has an equal chance of being in the experimental group and receiving the program.  Methods of fairly selecting participants include using a lottery, phasing in a program, and rotating participants through the program to ensure that everyone benefits. The selection process should also be transparent and appear fair to the community.

Typically evaluators are faced with the problem of allocating a program that is clearly beneficial, such as deworming drugs, or a water treatment solution. In other words, the ethical dilemma surfaces when creating a group of individuals who will be denied the program. Sometimes, however, the benefits have not been proven, meaning it is possible the program could potentially make participants worse off.  For example, drug companies often face this problem when testing new treatments on patients. In this case, an evaluator must put as much energy into ensuring that participants in the treatment group are not harmed.  If there is any potential risk in participating, then everyone involved must be informed of the risks and give their consent to participate. Even if there do not appear to be risks, any experiment should get the informed consent of all participants (in both treatment and comparison groups). Human subjects protocols have been developed by different nations and organizations and should be followed carefully. (See below)

2.    Research Subjects and the Institutional Review Board

An Institutional Review Board (IRB), also known as an independent ethics committee or a human subjects review board, is a group that has been formally designated by an institution (such as a university or non-profit) to approve, monitor, and review research involving humans as participants.   An IRB’s objective is to assure, both in advance of and by periodic review, that appropriate steps are taken to protect the rights and welfare of humans participating as subjects in a research study.

Because J-PAL studies involve human participants, J-PAL’s affiliates and their staff ensure that studies meet the guidelines of ethical research methods.  This includes:

  • Receiving institutional review board (IRB) approvals for each study before it begins,
  • All study personnel  completing an IRB training course,
  • Adhering to the IRB approved research protocol and guidelines throughout the course of the study.