Randomized Controlled Trials (RCTs)

0:00 This video covers randomized controlled trials, or RCTs, which are one option to consider when planning an impact evaluation. Before deciding to use RCTs, you will need to consider the building blocks of evaluation, research design, data collection and analysis and synthesis. RCTs measure the impact of an intervention on a particular outcome by conducting a carefully controlled experiment to compare at least two groups, one which has received the intervention and one which hasn't, known as the control group. For example, UNICEF recently funded a trial of early child development intervention in Pakistan. The trial aimed to evaluate the effectiveness of

0:46 early childhood interventions to strengthen and improve the health outcomes of young children. Hundreds of children were randomly selected at birth from a number of rural catchment areas to be part of the RCT experiment for a period of 24 months. The selected children were then randomly assigned to either the intervention groups or the control group. The process of random assignment to create these groups is a key feature of an RCT. Without it, it wouldn't be an RCT. In a simple RCT, a single intervention group is compared with a control group. The results of the evaluation are determined by comparing the progress of the intervention group against the control group. More complicated RCTs can have more than one intervention group. In the Pakistan example, there

1:36 were three intervention groups, each receiving a different set of treatments. The first intervention group received nutrition counselling, responsive feeding advice and nutrition supplements from 6 to 24 months of age. The second intervention group received stimulation and care for development advice, including coached practice, integrated in routine monthly home visits and group meetings. And the third intervention group received a combination of both the early childhood development and the enhanced nutrition interventions. They were all compared to each other, and also the control group, which had no new intervention. This design, in which a third intervention group receives both interventions whilst the

2:22 other two only receive one, is called a factorial design. The inclusion of this third group allows us to study the individual effect of all the interventions as well as their interaction with others. A factorial design can have a number of groups and variables and is not limited to the combination shown in our example. In this RCT, the children were sampled as individuals, but sometimes participants are randomly sampled as groups, for example by school or village. This is called a cluster RCT. Back to our example from Pakistan. The children's development was monitored against several different measures at different points until the children were 24 months of age. This study found that at

3:09 12 months of age, all three intervention groups had significantly greater cognitive, language, motor and social -emotional scores as compared with a control group. This continued to be the case at 24 months of age. Based on the results from this RCT, further action can then be taken to improve the developmental outcomes of other children in similar circumstances. Randomised control trials are not always the best method of evaluation. There are times when RCTs are more appropriate and others when they would not be as effective or may not be possible. For an RCT to be appropriate, there must be a large enough sample size to work with. A small sample size increases the risk of coming to the wrong conclusion. RCTs are applicable

3:59 when the intended impacts of the programme or policy can be readily agreed and measured, such as reductions in stunting or improvements in language development. They are not suitable when the intended impacts are varied, emergent and hard to measure, such as the difficulty of measuring all types of violence against children in a country. An RCT needs to be carefully planned from before the beginning of an intervention. RCTs cannot be undertaken retrospectively. Because data need to be gathered from the beginning of the programme or policy until impacts can be observed, RCTs are usually not suitable when quick answers are needed. Another situation when RCTs are not suitable is when the number of units of analysis is too low, for example capacity building in a single

4:48 organisation or advocacy campaigns at the national policy level. The low number of units won't allow random assignment and tests of statistical difference. Other evaluation methods will have to be considered. Random assignment makes it likely that the groups will be equivalent, but it cannot guarantee it. Simple assignment can sometimes result in groups with an unbalanced spread of characteristics, such as gender, ethnicity or age. There are things we can do to help ensure the treatment and control groups are similar, such as using stratified sampling, where you purposefully separate out a sample into strata based on a characteristic of interest, for example, different age groups.

5:34 You can then randomly assign each stratum at a time using a forced equal group size to make sure that the groups are evenly distributed. However, there is always a danger that there may be unobserved differences between groups which influence the results of the evaluation rather than the intervention, for example, differences in education attainment. Another way groups can become unbalanced is through attrition, individuals dropping out of the study. For example, a programme might work for some people and not for others. If the participants who are not getting good results drop out and are not followed up, their data will not be included and the programme will look more successful than it really is. Contamination is another cause of misleading

6:25 results. Imagine an evaluation where the results show that the control group has the same results as the treatment group. It is likely that people will conclude that the programme was ineffective. But what if individuals in the control group had actually received the intervention? What if they managed to access the treatment through other means, such as another programme funded by another donor? The programme might be highly effective, but the RCT would not show that. To guard against this, it is important to gather data on the treatments and services received by the control group and not to assume that they have not received any.

7:10 An ethical issue sometimes raised about RCTs is whether it is fair that one group will receive an intervention which may improve their lives while another will not. However, an RCT is only done when there is not sufficient knowledge about whether or not an intervention is beneficial. In some cases, it is possible to commit to rolling out the programme to everyone if it is found to be beneficial. Another ethical issue that must be addressed in RCTs, as in any evaluation, is the issue of informed consent. Participants must be adequately informed about the potential risks and benefits to them and not coerced or deceived into participating. It is also essential to have protocols

7:56 in place to stop the trial if there are early signs of harm. Results from a single RCT should be used carefully to inform future decisions. Just because an intervention was found to be effective in one context doesn't necessarily mean it can be scaled up effectively or transferred to another context. You also need to consider other issues when using the results of an RCT, such as whether there are sufficient funds to scale up, any equity issues that need to be addressed, and whether contextual factors will limit the feasibility or effectiveness of the programme in a new situation. You can find out more about research methodologies from the other videos and briefs in this series. For more detailed information on RCTs and other impact evaluation methods,

8:46 see the methodological briefs at www .unicef -irc .org