How I Ruined Science and Other Stories

Today I came across one of those hi-larious comic flowcharts, this one about alternative medicine. Now, it’s hardly new or innovative to make fun of ‘alternative therapies’ (though this is a fairly well-done piece of humour), but I want to draw your attention to one corner of it in particular. That is, the options for those wanting a “wholly ‘natural’ remedy” and who believe that “Yes, Big Pharma are the devil”. The choice is then based on the “Quantity of active ingredients required”. “Bugger all” leads to “Homeopathy”;* “An unknown, uncontrolled & untested amount” leads to “Herbal Medicine”.

This idea of testing has been at the centre of most of the more civil debates I’ve had or seen about herbal medicines, and it’s an important one. Many arguments are marred throughout by both sides’ tendency to argue as though more committed to being on a side than to striving towards Truth, no matter what they may claim. That is: typically, someone on the anti-herbal side will point out that little or no medical testing has been done for most herbal remedies. Then someone on the pro-herbal side will either bemoan the lack of funding for testing in most places — at which point arguments usually end because the opponents see that they are on the same ‘side’ really, the side of scientific testing, they are just coming into it with differing hypotheses — or else the pro-herbalist will question the validity of medical testing itself. And that is when it usually gets nasty.**

It’s this sort of oppositional attitude, I think, that leads people to ridiculously extreme positions of either disregarding all scientific research, or blindly accepting it all just because it’s *~*~science~*~* (though it’s worth noting that the latter view seems to be much more prominent among rationalistic non-scientists than practicing scientists or especially scientific researchers). The trouble, of course, is that a lot of scientific research, and — this excellent article in this month’s Atlantic magazine leads me to believe — medical research in particular, is often filled with methodological flaws. Some are the result of bias or fraud, but many are simply unavoidable, and probably many more are simply oversights. It is simply not healthy — literally or figuratively — to accept all research uncritically.

In the above-linked article, meta-researcher Dr. John Ioannidis claims, and has come up with a mathematical proof to demonstrate, that under normal conditions, most medical research turns out to be wrong. Moreover: “His model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted: 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials.” And yet, of course, it would be wrong to say that this is a reason to automatically distrust all medical research — though it certainly appears to be a reason only to trust randomized trials, and even to take those with a grain of salt. It is still less reason to think we should abandon the concept of medical research altogether. It just means that we need to work to make that research better.

An example from my own life has been niggling at my conscience for years now. St Andrews is a major centre for certain kinds of psychological research, as well as having a host of psychology grad students with their own research projects, and as such it is fairly common for students to earn bits of extra money by participating in experiments. Now, St Andrews is also a small town, and the university is small, and quite a lot of students know each other, and even more students, I would think, will know each other within the set of current students who participate in these experiments, because a lot of them find out about them through friends who’ve done them too — I mean, really, what student anywhere would give up the chance, or fail to pass on word of the chance, to earn almost minimum wage for pressing buttons for 45 minutes?

For most experiments, the fact that a lot of participants know each other is surely a non-issue. But one of the bigger labs within the department is one that researches perceptions of faces. This surely must be affected by the participant’s familiarity with the faces they view within the experiment. In the one I did, I was first given a basic colour-blindness test and then asked to rate how “healthy” various faces looked. There were fifteen or twenty faces in the cycle, and I knew close to a third of them. Two were close friends! I’m sure this must have made a difference, because I could tell where my friends’ faces had been digitally manipulated or stretched or discoloured, which I generally couldn’t with the strangers’.

I tried to tell someone this at the time, but they were all so busy and I was so shy that I didn’t work up the courage to demand one of their attention long enough to point out this potential (and potentially serious) methodological flaw. Then they took my picture to add to their database, gave me my handful of coins and sent me on my way. Ever since, I’ve been idly wondering whether or not I should email someone, but I don’t know who I would email, and the more time passes the more embarrassing it would seem to be, to initiate the discussion. But REALLY. It’s probably not something that most experimenters would need to think to control for, if they were in larger cities or had larger or older databases or whatever, but in that particular situation, it seems like a gross oversight — and one quite easily corrected within the experiment, with just a button or something the participant could click if the face generated was an acquaintance’s. Or by having a time lag of a good few years in between entering a participant’s photograph into the database and having it show up in experiments. Or something.

The good news, though, is that Dr. Ioannidis’ work has been exceptionally well-received by the medical community. Yet there is apparently controversy within the meta-research community for exactly the reasons described above: some fear that seeding public doubts about scientific research will simply drive people to seek “alternative” therapies or ignore the medical establishment, or their own health, altogether. I much prefer his proposed solution. To quote the Atlantic article: “We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right. That’s because being wrong in science is fine, and even necessary—as long as scientists recognize that they blew it, report their mistake openly instead of disguising it as a success, and then move on to the next thing, until they come up with the very occasional genuine breakthrough.”

* As well it should.

** Let us be clear: it also gets nasty because of the anti-herbal camp’s tendency to lump herbal remedies together with all other “alternative” therapies, like homeopathy and crystal healing and bullshit like that, and equivocate between them in their refutations; and by the tendency of many proponents of herbal remedies to also believe in bullshit like homeopathy and crystal healing.

Advertisement