Book Review

Book Review

Reading Medical Literature
With an Eagle Eye

An expert teaches us how to navigate the minefields of bias, errors, and baloney
By Will Block

Interpreting the Medical Literature (4th ed.)
STEPHEN H. GEHLBACH, M.D., M.P.H.
McGraw-Hill, New York, 2002

o you have a pet? Everyone should have a pet. Perhaps your faithful dog is right now staring hopefully at you with a tennis ball in his mouth and a wag in his tail—or your pampered cat, secure in the knowledge that the world revolves around her, is walking helpfully across these pages as you’re trying to read them. How we love to indulge our pets, with petting and games and, of course, their big favorite—food. And what a fabulous “return on investment” we receive from them, in fun and companionship and pure, unadulterated love. And even in life—as in longer life.

That, at least, was the conclusion reached by researchers looking for factors that would predict survival rates of patients who were discharged from a coronary care unit after a heart attack or a bout of angina pectoris.1 They found that the relative risk of death among pet owners was 0.2 after one year, meaning that the chance of these folks dying during that interval was only 20% as great as that for non-pet owners. Put another way, the pet owners were five times more likely to survive that first year than those who were petless.

That’s something to bark about! It provides scientific evidence for the oft-repeated claims about the health benefits of pet ownership, especially for the elderly who live alone. We’re told that owning a pet is good for your health and longevity because it lowers your blood pressure, wards off depression, gives you a sense of purpose, and generally makes you a happier, more sociable person.

You Must Act Like a Ferret

The study just cited is used as an illustrative example of risk analysis in an excellent book, Interpreting the Medical Literature (4th ed.), by Stephen H. Gehlbach, M.D., M.P.H., dean of the School of Public Health and Health Sciences at the University of Massachusetts.2 Dr. Gehlbach is an authority on the proper methods of design and execution of medical research studies. Here he explains, in lucid, vivid prose, how health professionals should tackle the job of interpreting such publications with a critical eye for potential weaknesses—not to tear the work down (unless it actually deserves it), but to gain understanding and avoid being misled.*


*The first thing to look for is clarity in the writing. If it’s not clear, beware! Sloppy writing is a sure sign of sloppy thinking.


The book is richly endowed with examples taken from the medical literature, and among its most valuable features are discussions of the many forms of bias that can infiltrate and taint an otherwise credible study. Some are relatively benign and cause little harm, but others can wreak havoc; the reader must try relentlessly to ferret them out and judge the damage done. The same is true, of course, for sources of outright error in the measurements or calculations. Quoting Gehlbach regarding the aim of his book,

We will balance the strengths and flaws uncovered in an article and come up with an independent assessment of whether the author’s message rings true . . . The bottom line is validity. Are the results believable? Do they represent the truth? Are they applicable to our practice? Will the patients we see and treat respond in the same way as those described in the study? Does the paper really support its claims?
In truth, many papers do not support their claims, because the studies were poorly designed or poorly executed, or both—or, sometimes, because the authors’ conflict of interest (e.g., being employed by a company that funded the research) destroys their credibility. All the more reason to focus on those studies that do hold up to scrutiny, as the pet-ownership study apparently does. Let’s look at some of the questions a reader should ask (just as the researchers should ask in the first place), to make sure.

Always Ask, “What If . . . ?”

The correlation between pet ownership and longevity in coronary care patients is impressive, but does having a Fido or a Fluffy really cause you to live longer? The authors of the study suggested that it could, based on the hypothesis that social affiliation and companionship have important positive health benefits. Also, pet owners tend to get more exercise and fresh air, thanks to their furry friends, than those who are under no such compulsion.

But what if, Gehlbach asks, there is some other explanation? What if there are predisposing factors in pet owners that would give them an edge over the pet-challenged even if they didn’t own a pet? Perhaps people who are inclined to own pets are by nature more tolerant and easy-going, thus reducing their risk for future heart problems. Perhaps, because pets require care and attention, their owners are younger, on average, and more physically active. The authors of the study presumably considered such factors and corrected for them in their data analysis, because Gehlbach credits them with having asked the patients for “information on a broad range of personal topics, including pet ownership.”

In any case, it seems unlikely that a survival effect as large as the one observed in the coronary care patients could be due to some overlooked factor, and Gehlbach half-jokingly suggests that a prescription for a beagle might do more good than one for a beta-blocker.

Confound It!

Alternate explanations such as those mentioned above are based on what are called confounding factors. They’re the bane of researchers studying almost any aspect of health or disease, because they make it difficult to isolate the one factor of interest (such as pet ownership, or HDL-cholesterol levels, or supplement intake) and link it unequivocally to some particular health outcome, such as living longer.

Trailer Parks Cause Tornadoes

Remember that one? It’s funny precisely because, although the correlation between tornadoes and trailer parks is well known, the idea of a causal link is ridiculous. (There are, however, reasons why tornadoes seem to prefer trailer parks, and you can figure them out if you put your mind to it.) The joke lampoons the common blunder of seeing a cause where none exists.

Correlation Is Not Causation. Failure to recognize this simple but profound truth and to keep it always in mind is probably the greatest source of misunderstanding and misinterpretation in the field of medicine (and in other fields as well, such as politics and economics). It’s not just laymen who fall victim as they’re trying to make sense of things, but, surprisingly often, it’s the experts themselves who fail to heed the dictum and thus stumble into the inevitable traps. There’s a world of potential confusion and nonsense, always masquerading as The Truth, wrapped up in every instance of such failure.

Finding correlations is generally easy, but proving causation can be very hard, often requiring great scientific knowledge, judgment, and sophistication in a variety of disciplines, including advanced methods of statistical analysis. Occasionally causation can be established beyond a reasonable doubt. Smoking, e.g., is strongly correlated with a host of medical problems ranging from discomfort to disease to death, and it definitely causes some of them.* Most causal connections in medicine, however, are not nearly that definitive, and often the best one can do is to discern relative risks without knowing for certain what causes what.


*Just to complicate things, however, there is good evidence that smoking protects women from endometrial cancer, presumably by lowering estrogen secretions, which smoking is known to do.1 This one benefit of smoking, however, pales in comparison to the enormous damage it does—it is estimated to kill 30 times as many women annually as it saves.2


As an aid in trying to decide whether a correlation may in fact indicate a causation, Gehlbach cites a classic list of nine factors that the careful reader should take into account: (1) strength of the association; (2) consistency of the observed evidence; (3) specificity of the relationship; (4) temporality of the relationship; (5) biological gradient or dose response; (6) biological plausibility; (7) coherence of the evidence; (8) experimental confirmation; and (9) reasoning by analogy. If all or most of these factors hold up well to expert scrutiny, then perhaps you’re really on to something.

References

  1. Lesko SM et al. Cigarette smoking and the risk of endometrial cancer. N Engl J Med 1985;313:593-6.
  2. Weiss NS. Can not smoking be hazardous to your health? N Engl J Med 1985;313:632-3.

Typical confounding factors in a clinical trial of the safety or efficacy of some agent (such as a nutritional supplement) are age, gender, race, diet, exercise, smoking, drinking, use of drugs (in either sense of that word), personality type, socioeconomic status, educational level, general health, and specific health problems, i.e., the presence of diseases or disorders, such as high blood pressure or low thyroid. There are ways of controlling for such a welter of confounding factors, first through rigorous selection criteria for the individuals to be evaluated, and second through the mathematical technique of multivariate analysis (analysis involving multiple variables) on the data obtained. Both are fraught with pitfalls, and the careful reader must pay close attention to the details of how these problems were handled.

Does Sugar Cause Heart Disease?
Yes! No! Yes! . . . (Stay Tuned)

In 1964, a pair of researchers stirred up some controversy by showing that high sugar intake seemed to cause heart disease.3 When others took up the investigation, it became clear that there were other factors that might explain the correlation. For example, people who consumed a lot of sugar also tended to drink a lot of coffee and tea, and they smoked more. Aha! Smoking causes heart disease, and when this confounding factor was controlled for in the studies, the connection between sugar and heart disease disappeared—it had been an illusion.

Or had it been? Further, more refined studies and data analysis brought the illusion back to life by demonstrating that there was a connection, albeit a small one, between sugar intake and heart disease after all, just as there is between smoking and heart disease.4 (Bear in mind that smoking is far more harmful to your health than excess sugar, or than just about anything else, for that matter, with the exception of obesity.)

I Feel Like a Yo-Yo, Ma!

Does that story sound familiar? Don’t you wish you had a buck for every time you’ve heard that some food or drink (especially one that you really like) was bad for you, and then it was good for you, and then it was bad again, and so on? It’s like a yo-yo routine. Don’t blame the researchers (well, OK, blame them)—they’re doing the best they can at a wickedly difficult task. Reading Gehlbach’s book would give you a greater appreciation for how tough it is, and you would be better able to judge for yourself (assuming you have a solid background in science or medicine to begin with) how valid a given research study is likely to be.

Part of the problem is that any given study is fraught with potential sources of bias and error (no one is perfetc, you know). Furthermore, any two studies of the same subject are likely to be different in so many ways—some obvious, some subtle—as to make your head spin, and making meaningful comparisons between them is difficult to impossible. It’s little wonder that the results we hear are often contradictory. All the more reason to do our best to separate the wheat from the chaff by bringing expert judgment to bear on what we see in the medical literature.

Where’s the Wheat?

And what we see is a lot! Oceans of literature are available in medical libraries, and the Web too harbors vast amounts, with fantastic ease and speed of access. That’s a boon to scholars, whether medical professionals or amateur health enthusiasts, but just deciding what to look at, let alone try to digest, is daunting. That the cornucopia is so abundant is wonderful, but there’s a price to pay in terms of the chaff-to-wheat ratio, which grows ever larger. As Gehlbach says, “. . . developing skills to critically review and understand medical literature becomes even more important as the quantity of information (both good and bad) cascades upon us.”

Medical journals are proliferating like rabbits, and they cover a wide spectrum of scientific quality and integrity. As with most human enterprises, some are outstanding, most are OK, and some are terrible. In the last category are bottom-feeders that publish papers the better journals wouldn’t touch; some are not peer-reviewed or even copy-edited, let alone edited for factual accuracy. The material ranges from poor to outright incompetent, and it sometimes serves commercial interests more than it does the truth. Worse still, there are numerous online-only “journals” that, for a fee, will publish just about anything as long as it sounds scientific. Professionals should be able to see this material for what it usually is: trash.

Abstracts Are Often Error-Ridden

Absorbing the lessons of Gehlbach’s book will be helpful in developing one’s own radar for suspect material. One such lesson is that the abstract of a paper (which is all that most readers ever read) can be misleading for a variety of reasons. A study of six widely read medical journals, e.g., showed that serious deficiencies in the abstracts were commonplace, ranging from 18% to 68% and averaging about 30–40%.5 Reader beware!

Gehlbach also emphasizes that most readers pay far too little attention (or none at all) to a section of the paper that is extremely important but often rather boring: the section usually called Materials and Methods. This section is saturated with technical details regarding every aspect of the study, some of them perhaps very esoteric—and it’s in the details, as we all know, where the devil lurks. If something is wrong with the study, that’s probably where the clues will be found, by readers who are expert enough to understand the material and who have the professional experience to know what can go wrong, and how, and why.

Is There a Beast in Your Belly?

Gehlbach cites one example of a methodologically sound study (two related studies, actually), of a special type called nested case-control, that should be of special interest to readers of this magazine: it involved the search for a relationship between Helicobacter pylori infection (which is very common worldwide) and stomach cancer.6,7 The two research groups were able to show, using data that had been collected 20 years earlier, that there is indeed a positive link between H. pylori and stomach cancer. It is thus wise to avoid or eliminate such infections, as can be done with the herbal supplement called mastic.

We Want You to Be the Beneficiary

There is so much more to the book: extensive discussions of the various kinds of medical studies, including the all-important clinical trials and the pros and cons of their many different designs and protocols; the slippery nature of standards used for defining and categorizing and measuring things; the sources and dangers of random and systematic measurement errors; statistical methods of data analysis; the importance of the null hypothesis and the two major errors associated with it; the meanings of statistical significance and clinical importance and how they relate to each other (they don’t always); the different ways of defining and assessing risk; the value (and pitfalls) of review articles and meta-analyses; etc.

Gehlbach has done the health care profession a great service with this book. It’s an excellent resource for improving one’s ability to be a discerning consumer of the vast amount of medical literature now available, and we at Life Enhancement are grateful students of his work, which we hope will make our own work that much better and more valuable to you, our esteemed readers.

References

  1. Friedmann E et al. Animal companions and one-year survival after discharge from a coronary care unit. Public Health Rep 1980;95:307-12.
  2. Gehlbach SH. Interpreting the Medical Literature, 4th ed. McGraw-Hill, New York, 2002.
  3. Yudkin J, Roddy J. Levels of dietary sucrose in patients with occlusive atherosclerotic disease. Lancet 1964; 2:6-8.
  4. Elwood PC et al. Sucrose consumption and ischemic heart disease in the community. Lancet 1970 May 16; 1(7655):1014-6.
  5. Pitkin RM, Branagan MA, Burmeister LF. Accuracy of data in abstracts of published research articles. JAMA 1999;281(12):1110-1.
  6. Parsonnet J et al. Helicobacter pylori infection and the risk of gastric carcinoma. N Engl J Med 1991;325: 1127-31.
  7. Nomura A et al. Helicobacter pylori and gastric carcinoma among Japanese Americans in Hawaii. N Engl J Med 1991;325:1132-6.

  8. Will Block is the publisher and editorial director of Life Enhancement magazine.

FREE Subscription

  • You're just getting started! We have published thousands of scientific health articles. Stay updated and maintain your health.

    It's free to your e-mail inbox and you can unsubscribe at any time.
    Loading Indicator