EDITORIAL

Data Frequently Fail To Support
Meta-Analysis Conclusions

Be careful about reading health books. You may die of a misprint.
— Mark Twain

he thrust of Mark Twain’s incisive comment can now be extended to meta-analyses—at least for antihypertensive drugs—according to an analysis just published in a respected medical journal.1 Although these studies of studies have a sacrosanct air about them—having a weighted value far beyond that of individual research papers—this new analysis reveals nothing as small as a misprint; it reveals repetitive misinterpretations of data that appear to be biased by—surprise, surprise—the invisible hand of sponsors that cater to their own financial interests and not those of the readers, whether physicians, patients, policymakers, or otherwise.

It is no secret that randomized, controlled trials have often been found to generate conclusions that tilt significantly in favor of the drug companies that sponsor them. However, it wasn’t clear to what extent these biases have infected meta-analyses, until now.

The Peer Review Process Is Flawed

Are we to assume that some professionals are allowing the biases of drug companies to influence their judgment and that the peer review process has sometimes failed to expose and correct these ethical lapses? The answer is an emphatic yes, according to Dr. Veronica Yank, of Stanford University, and colleagues, who wrote the harsh critique cited above. “Editors and peer reviewers, as well as policymakers, meta-analysts, and readers, should closely scrutinize the conclusions of meta-analyses to ensure that they are supported by the data,” the authors admonish.

Dr. Yank et al. investigated 124 meta-analyses in all, each of which involved antihypertensive drugs. These studies were published from 1983 through 2004, with the majority appearing after 1996. Forty percent of these studies (49 out of 124) had financial ties to one drug company (but not necessarily the same company in each case). Yet funding disclosure was minimal: only 9% in the meta-analysis, 10% in the sponsor’s supplemental literature, and 24% in previous papers by the contributing authors.

In their analysis, Dr. Yank et al. found that the number of favorable conclusions far outweighed the number of favorable results. Furthermore, with regard to sponsorship by one drug company, 92% (45 out of 49) had favorable conclusions while only 55% (27 out of 49) had shown favorable results. This gap of 37% dropped to 21% when two or more drug companies were involved, and it disappeared for studies done by nonprofit institutions alone, and even by nonprofits in conjunction with drug companies.

Spinning the Data

The authors concluded, in no uncertain terms, that meta-analyses are subject to the influence of systematic bias, as are other types of studies, where the results are often spun into conclusions that are not warranted by the data. These conclusions favor the financially vested sponsors, much to the detriment of physicians and their patients, for whom truthful conclusions are of great concern.

What Should Be Done?

In an accompanying editorial, Professor Richard A. Epstein of the University of Chicago Law School concurred that Dr. Yank and colleagues’ analysis offers added proof of drug company influence over conclusions, but not results, pointing out that the original data appear (in the studies in question) to be sound.2 Arguing against any legal or regulatory scheme in which drug companies are precluded from financing studies, Professor Epstein instead called for a rise in the quantity, distribution, and responsibility of editorial comment by unbiased experts. Encouraging this process could, in fact, “subtly induce original authors to soften their basic claims” and avoid a large reduction in the number of studies published, as would undoubtedly be the case if the incentive to publish were shifted to nonprofit entities. After all,2

The medical profession already has voluntary means to improve its performance. As long as the disagreements lie in the interpretation of data and not the collection of data, the solution is not state regulation; rather, doctors should be warned to be cautious in interpreting the conclusions of studies. Indeed, the largest problem for drug innovation does not lie with these studies, but in the ever greater time and money needed to bring new drugs to market, where the price of delay is too often measured in lives lost.

Ditto for the Supplement Industry

Recently, a growing number of meta-analyses for various supplements have been published that find weak, insufficient, or contradictory evidence for efficacy of the supplements. Many of these studies also have pharmaceutical sponsors, often drug companies that have competitive products. One of the problems is the severe restrictions on the admissible data in these studies, imposed because supplement studies frequently fail to meet the “gold standard” of analysis favored by the medical profession. But this standard is often exceedingly expensive to achieve, and that is a major reason why the cost of taking a drug to market has now ballooned to about $1.2 billion (and has slowed drug approval to a crawl).

But, as we have now learned, strong conclusions in the drug meta-analyses are often drawn from weak results, even though the data may be “gold standard.” Supplement meta-analyses were not always as stringent as they are today, a time when pharmaceutical sponsors are constantly hovering over the process.

Given that supplements are generally far safer than drugs, along with the fact that they can’t be patented and are thus not likely to make big bucks, why not give supplement studies the benefit of more readily admissible data? Then, per Richard Epstein’s suggestion, doctors should be encouraged to be more cautious and proactive in interpreting the conclusions of these studies and in making their recommendations.

References

  1. Yank V, Rennie D, Bero LA. Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ 2007 Nov 16. [Epub ahead of print]
  2. Epstein R. Influence of pharmaceutical funding on the conclusions of meta-analyses. BMJ 2007 Nov 16. [Epub ahead of print]

FREE Subscription

  • You're just getting started! We have published thousands of scientific health articles. Stay updated and maintain your health.

    It's free to your e-mail inbox and you can unsubscribe at any time.
    Loading Indicator