Focus on Alternative and Complementary Therapies
www.pharmpress.com/fact
Focus Alternat Complement Ther©2005 Pharmaceutical Press
Focus Altern Complement Ther 2007; 12: 153–4
The term ‘observational studies’ is somewhat confusing – after all, most research relies on observation, certainly clinical trials do. But a clinical trial is not an observational study. Typical observational studies are cohort studies and case-control studies. The proper definition of an observational study is ‘a method of assessment that is used by a researcher who, at best, identifies, observes, records, classifies and analyses relevant information in a study without interfering with the course of events’.1 There are two broad categories of studies: observational and experimental. In the context of clinical trials, ‘experimental’ means that an intervention is administered for the purpose of research. In an observational study, the very same intervention may be given – but not for the purpose of research; it is administered in the context of clinical practice. That usually means there is no control over a host of factors that might crucially influence the outcome.
Observational studies are wide open confounding factors. In other words their results can be significantly influenced by variables other than the experimental treatment. This makes it impossible to be sure about what actually caused a certain outcome. As we see, the crux seems to be causality. Observational studies can certainly generate valuable information, but are pretty useless whenever we need to know what caused an observed effect.
CAM (and indeed medicine as a whole) has a long tradition of disregarding this fact. Whenever CAM practitioners administer a treatment, they are likely to attribute any ensuing clinical change to the specific effects of their intervention. This means that definite conclusions about cause and effect are drawn on less than solid grounds.
Some conceptual clarity about what really is going on in such a situation seems to be in order. Figure 1 schematically depicts the results of a typical observational study in which a group of patients have received treatment x. Over time, the symptoms improve, and we therefore perceive a therapeutic effect. The assumption usually is that this ‘perceived therapeutic effect’ is due to the specific effect of the intervention.
Figure 1. Schematic drawing of typical observational study. As time passes, the primary outcome measure (e.g. pain) improves, as indicated by the declining line. Thus there is a perceived therapeutic effect (PTE) at the end of the treatment period, indicated by a before–after difference in the primary outcome measure. This difference is often wrongly attributed to be the sole result of treatment applied
The ‘perceived therapeutic effect’ could, however, be due to many other factors. Figure 2 shows schematically the range of factors which can be (and often are) involved. This schematic approach makes it easy to see that, even if the specific therapeutic effects were negative (i.e. the treatment leads to a worsening of the problem at hand), the total perceived therapeutic effect could still be positive (Figure 2). Ineffective and even harmful interventions can thus be falsely associated with overall improvement in observational studies. In other words, whenever we rely on such data we are at risk of drawing the wrong conclusions – even harmful interventions can appear to be beneficial.
Figure 2. Schematic differentiation of factors contributing to the perceived therapeutic effect.
As I stated above, observational studies can generate valuable information. They are therefore by no means useless. But if our research question relates to a therapy causing a certain outcome, they are far from reliable. Nevertheless, CAM researchers are often fond of observational studies and claim that they generate valuable information about cause and effect. Why? I’m not doubting that most CAM researchers are aware of the rather elementary relationships outlined above. I therefore suspect that they prefer observational studies because they frequently suggest positive results for treatments which, when tested in RCTs, are ineffective. Sadly these would then be a false-positive result – and such findings are a disservice to everyone, particularly our patients.
