Shang reviews of homeopathy

In 2005, The Lancet published the results of an innovative meta-analysis that became the single most cited study of homeopathy, arousing considerable media attention and a storm of protest from homeopaths.

In 1999, the Government of Switzerland, for a trial period of 5 years, allowed health costs for treatment with homeopathy and four other CAM modalities to be reimbursed under the country’s compulsory health insurance scheme, and set up a programme to evaluate the cost-effectiveness of these treatments (the Complementary Medicine Evaluation Programme (Programm Evaluation Komplementärmedizin, PEK). As part of this evaluation, a multi-disciplinary team, including a homeopath, set out to evaluate the evidence for the efficacy of homeopathy. The study, published by Shang et al., took a novel approach; whereas traditional meta-analyses combine all studies of a single, given condition, this was a "global" meta-analysis testing the hypothesis that all effects of homeopathy are placebo effects. If so, the authors reasoned, then the predominance of positive homeopathy trial reports reflects publication bias, and hence the magnitude of effects should diminish with sample size and study quality. They analyzed 110 placebo-controlled homoeopathy trials and 110 matched conventional-medicine trials. In both, effect size declined with improved study quality; however, some effect was still present in the largest and best conventional medicine trials, but not in the largest and best homeopathy trials. In the Shang et al. review, 21 homeopathic trials were judged of (relatively) “high quality”; these, overall, showed a benefit of homeopathic treatment. In the final analysis, the researchers included only the eight largest studies; these showed that homeopathic treatment was comparable with a placebo, while six similarly large conventional trials were not compatible with a placebo effect.

Thus the study exposed a problem with the reliability of small trials generally, but particularly small trials with methodological weaknesses; it's a problem that affects both conventional medical trials and trials of homeopathy equally in that trials that appear to produce a positive outcome are more likely to be published than trials which are inconclusive or produce a negative outcome. This problem, called "publication bias", reflects both the perceived unwillingness of academic journals to publish unexciting results, and the unwillingness of authors to write up 'uninteresting' results. It is compounded when weaknesses in trial design mean that statistical evaluations of outcomes are of doubtful validity.

The authors concluded that homeopathy was no better than placebo, and suggested that no further research on homeopathy is necessary. The article was accompanied by an unsigned editorial titled “The end of homeopathy" and another, signed, editorial.

Criticism from supporters of homeopathy

The Lancet subsequently published critical correspondence, and received an open letter from the Swiss Association of Homoeopathic Physicians (SVHA). which declared that none of the published trials of homeopathy had any value anyway:

"The meta-analysis may be statistically correct. But its validity and practical significance can be seen at a glance: not one single qualified homoeopath would ever treat one single patient in clinical practice as presented in any of the 110 analysed trials! The study cannot give the slightest evidence against homoeopathy because it does not measure real individual (classical) homoeopathy. It confounds real homoeopathic practice with distorted study forms violating even basic homeopathic rules.”

Other critics of the Shang et al. analysis noted that it involved subjective judgements of study quality. Several studies defined as "high quality" by Linde et al. (1997) were not defined by high quality by Shang et al. most of which showed a positive effect of homeopathic treatment.The Shang et al. analysis also excluded a relatively large study of chronic polyarthritis (N=176) because no matching trial could be found. The authors of an article in the Journal of Clinical Epidemiology say "This result can be interpreted differently. Following Shang's perspective it can be explained by small study bias (which includes publication bias). In contrast, one may hypothesize that Shang's result is falsely negative." The authors noted that four of the 21 best trials selected by Shang et al. dealt with preventing or treating muscle soreness—these consistently found no benefits to homeopathy, so if it is accepted that homeopathy is not useful in this condition, the remaining 17 trials show an overall significant effect, mainly determined by two trials on influenza-like diseases. Thus they argue that it is possible that homeopathy might be effective for some conditions and not others.