Earlier this month, the Atlantic published an excellent article by David H. Freedman on the abundance of subtle bias present in biomedical research. It’s probably not a surprising revelation. After all, humans are at least to some degree, self-interested beings. But how does this unconscious bias affect research?
When people read scientific papers or see presentations from researchers, the information is laid out in a very chronological manner. One experiment leads to another and usually, some sort of pattern emerges. By the end of the paper or seminar, some point or new discovery is made. But research doesn’t actually happen that way.
Usually, a researcher or student starts with some sort of idea. And then, you try to support that idea through evidence gathered from tissue culture, from bioinformatics, and from animal studies. It could happen in any order and usually does.
At each step, you are trying to prove a certain idea. Of course you want to succeed. So in tumour studies you pick mice which are receptive to a certain cell line, or for vascularization studies you might do tissue culture with cells that are good at sprouting… But are these choices the best scientific choice, or do they represent the best choice for a “successful” experiment?
Dr. John Ioannidis is a meta-researcher – he studies other studies for patterns. Lots of other studies. His research team has shown that when applied to human health, much of what is published in biomedical research studies is exaggerated, misleading or sometimes, just plain wrong. And worse – many studies which have proven to be wrong are still cited and used by current researchers and doctors.
Ioannidis even developed a mathematical proof which shows that most of the time, researchers will come up with the wrong findings! From the article:
Simply put, if you’re attracted to ideas that have a good chance of being wrong, and if you’re motivated to prove them right, and if you have a little wiggle room in how you assemble the evidence, you’ll probably succeed in proving wrong theories right.
His study looked at 49 of the most-cited, most prestigious medical findings of the last 13 years. 45 of these papers claimed the discovery of some sort of effective treatment. Since publication, 34 of these claims were retested – 14 were found to be plain wrong or grossly exaggerated. 14/45 is 31% – Almost a third of the most cited medical research is wrong? This is scary business.
An observant lay person would notice the same patterns: Alcohol is bad. Oh but wine is good. But too much is still bad. But people who drink anything at all live longer than people who abstain. Substitute alcohol/wine/drink for “hormone replacement therapy” or “carbohydrates” or “vitamin C” and you will have reconstituted most of the major health headlines from the last 2 decades.
What’s a health conscious, scientifically literate person to do? Ioannidis advocates the publication of negative results in addition to positive results. He believes that if scientists were not punished (through grants, funding, publication etc.) for lack of positive results, then the tendency to spin out dynamic stories and dramatic “discoveries” would taper off. I tend to agree. If research is being done conscientiously and carefully, the results you get are the results you get. Most of it is probably dreadfully dull. But to try to spin it into a story, to make it “fit” into a bigger picture… that’s not science any more, that’s fiction.
And I should note that Ioannidis focuses on health-related research. I would venture that aside from conflicting egos and personalities, lesser funded areas of study such as plants, evolution or biogeography (to name a few of my interests, haha) suffer less from the need to constantly publish “breakthroughs”.
From Ioannidis:
“Science is a noble endeavor, but it’s also a low-yield endeavor,” he says. “I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact.”
Science is important and should certainly continue but the public and the publishers need to foster an environment for science to operate honestly. People need to realize that science is constantly evolving and each publication is a drop in the water. Each To continually expect “breakthroughs” or “potential cures” will only lead to disappointment.
And yes – Ioannidis admits that he may be similarly biased in his own research. 🙂
Read Full Post »
There is little more I hate than passive aggressiveness.
AlbinoMouse 