August 2004

Different Trials, Different Results
How to Explain Them?

News reports about medical studies can get confusing. Have you ever thought that something was supposed to be good for your health, but then suddenly heard it was bad for you? How in the world can medical studies come to such different conclusions about the same thing?

Dr. Julie Buring, an epidemiologist at Brigham and Women's Hospital in Boston, recently gave some insight when she spoke at a cancer research symposium at the National Institutes of Health. Her keynote address explained why there are often discrepancies between the two major types of medical studies you often hear about, observational studies and randomized trials.

To understand why these two types of research studies can come up with different answers, you first need to understand what they are. In observational studies, researchers simply observe people, grouping them according to whether or not they have chosen to take a particular drug, live a particular lifestyle, or have been exposed to something of interest. The researchers then compare these groups of people to see their subsequent risk of disease.

In contrast, exposures in intervention studies are controlled by investigators, who assign people to two or more study groups and then control what treatment or exposure to something of interest each group receives. The best-known type of intervention study is the randomized clinical trial, considered to be the gold standard of clinical research studies. (See the side box for more details.)

Dr. Buring gave some examples that have been widely reported in the news to illustrate how observational studies and randomized clinical trials can have different outcomes. One example involves the discovery of a surprising relationship between beta-carotene and heightened lung cancer risk.

"In animal laboratory studies," she explained, "beta-carotene could block the carcinogenic process and inhibit specific tumor growth." There was also, she said, a large body of evidence associating the consumption of fruits and vegetables with a lower risk of cancer. Many researchers believed that these pieces of evidence, taken together, showed that beta-carotene helped people ward off cancer.

But in randomized clinical trials, beta-carotene showed no benefit in preventing cancer and actually increased the risk of lung cancer among heavy smokers.

There are many possible explanations for this discrepancy, Dr. Buring said. People may have made an incorrect leap to one specific factor. It could have been something besides beta-carotene, or there might have been other nutrients in the diet that worked with beta-carotene to prevent cancer.

"Many people prefer taking a pill to changing their diet," Dr. Buring said. But those consuming high levels of beta-carotene in the observational studies had also been eating lots of fruits and vegetables, and were likely unique in other ways — eating more fiber, exercising more, and perhaps differing in other habits researchers haven't even thought of.

Maybe the clinical trial simply lasted for too short a time to see an effect, Dr. Buring offered as another explanation. Or it might have been that the actual blood levels of beta-carotene in the clinical trial patients differed significantly from those in the observational studies.

As for the smokers, researchers have since been able to explain why their risk of lung cancer increased. It turns out that, when combined with changes inside the body brought about by smoking, high doses of beta-carotene actually create harmful compounds.

Dr. Buring also recounted the recent findings from the Women's Health Initiative studies on hormone replacement therapy (HRT). Many doctors had been giving their patients HRT based on their own observations, she explained, and some thought it was actually unethical to conduct a randomized trial, which would require denying the benefits of these hormones to control groups.

But it turned out that randomized clinical trials had to be stopped early because the risks of using HRT for extended periods outweighed their benefits.

There were many possible reasons for these unexpected results. One likely explanation is that women taking HRT in the observational studies may have been living healthier lifestyles and seeing their doctors more often.

This explanation highlights an inherent limitation in observational studies, Dr. Buring said: "People who select one way to live also select others." Unforeseen or "confounding factors" in an observational study can lead to a clinical trial that returns unexpected results.

Asking Different Questions

Many people get frustrated with these discrepancies, Dr. Buring acknowledged, particularly when the media hypes contradictory conclusions. But she argued that these discrepancies aren't necessarily a sign of bad studies. "Discrepancies may mean the system is working," she said. "This is exactly why we do trials. Apparent discrepancies are not necessarily contradictory or wrong; they may be looking at different questions."

So why do observational studies at all, if randomized clinical trials are the best way to answer a question? "Sometimes, an observational study is the only feasible or ethical approach," Dr. Buring explained. For instance, researchers cannot ethically expose people to a suspected carcinogen to find out whether or not they develop cancer.

Dr. Buring concluded that "Neither a reliance on randomized clinical trials nor observational studies is appropriate." Both are needed to answer a medical question.

So if you find yourself wondering how seriously to take the latest research report, ask yourself what kind of study it was. Is this really the final word on the topic, or do researchers need to explore the topic further? Those "contradictory" research results, once you put them in perspective, may suddenly not be as baffling as they seemed.u

— Written by Harrison Wein, Ph.D.

For more information on understanding news reports on medical risks, see Understanding Risk from the April 2004 issue of The NIH Word on Health at http://www.nih.gov/news/WordonHealth/apr2004/risk.htm.

You can also view or order a free copy of the National Institute on Aging publication Understanding Risk: What Do Those Headlines Really Mean? at http://www.niapublications.org/engagepages/risk.asp. Or, call 1-800-222-2225 (TTY: 1-800-222-4225).