Effect Measure Fail 101

Over the past day a number of my skeptical peers in Vancouver have been discussing this article from Effect Measure about over-reaching statements made in favour of the most easily accesible science journals in this article on the Science Based Medicine blog.

The Effect Measure article is essentially correct from all that I can glean.

My first read of the article left me feeling as though my epistemological senses had been kicked in the nads.

But the criticism fails on one key fact. Most of us non-scientists are not skilled enough to fully understand the content of journals about broad-disciplines, let alone sub-sub-disciplines of science. And more to the point, we don't have time. We spend most of our time doing OUR jobs, not scientists jobs.

What follows is a slightly edited response I made to a group of friends in a group email discussion. It touches on a few things I would have eventually got around to talking about anyhow. I'll probably revisit them in the future, but for now... why waste a solid bit of thinking on an email to three other folk?

---------------------------------

I was only half joking when I claimed to feeling lost after reading that article.

I am by any measure against any science, a lay-person. All I have is other people's authority. When you get right down to it, I am merely taking Fizeau & Foucault at their word when I accept that the speed of light is 186000 miles per second. But it seems relatively (pun intentional) consoling that 186000+ other scientists agree.

I am totally guilty of giving - or at least appearing to give - absolute faith in the writings I find in 'reputable' journals. But never without at least a sliver of wiggle room - that IS the nature of science. The design, such as it is, of logic and science results in an inevitable gravitation towards greater and great accuracy and truth. A necessary requirement of that is - as counter-intuitive as it may seem - that there be occassional faulted, outright wrong, misleading, and even fraudulent information in the mix.

(Okay, fuck... life is not making this easy today. I can't imagine being interrupted more, and the point I was originally making has gone the way of Kublah Khan's Pleasure Dome.... so now I descend to rambling and hoping I can drag myself back to where I was before.)

Uh yeah... so I'm a lay-person. Certainly a lay-person well ahead of the curve, but a lay person. I regularly act as though I have 100% confidence in the knowledge I have, but let's face it; when it all comes down to it, it's all a numbers game. But when the odds are solid enough I'm willing to behave as though there is no gamble. As a skeptic, I'm never going to take the long-odds bet, and I'm totally prepared to lose on a small number of near-even wagers... but I'm not going to invest a lot in those bets.

Are there psychic powers in the world? I'd put it at more than 1:1000000000 against. Homeopathic efficacy? Even less chance. Yeah, that's kind of arbitrary, but as we're talking the difference between a one in a billion chance and a one in a trillion chance it doesn't really affect how I live my life. My faith in the speed of light...? Time to pull out the scientific notation.

Less certain things?

Does my girlfriend love me? Ten Thousand to one in my favour... still enough that I can't be bothered to worry about it much.

9/11 was an inside job? Still pretty high. Similar to the GF's love.

The Bush Administration willfully turned a blind eye to dangers leading up to 9/11, knowing that it might lead to an event they could leverage into political gain? Okay... now it starts to get a bit interesting... say... 1 chance in 2500. Still unlikely, but it's starting to feel possible. I live assuming not, but will my world-view shatter if I'm wrong? No.

How about the study published just yesterday in the New England Medical Journal about the effects of Letrozole Therapy on women with Breast Cancer? (Admittedly I could have found something with a more immediate and obvious impact upon me personally, but I chose the most recent example that I could reasonably grok.
But let's assume that I'm thirty-years older and the previously mentioned girlfriend has breast cancer and is considering Letrozole therapy, and that this is the most recent information.) Well... it's a reputable journal... that's promising. I know diddly-squat about medicine. I would look into it further than this, but assuming this was all I had available, I'd be encouraging her to look for other options - options that put the numbers in our favour.
Looking at this study I'd arbitrarily start at one hundred to one against the value of Letrozole.

(I should emphasize that the 'odds' are really very arbitrary - subjective, even - simply for the purposes of illustrating my point.)

In the above example I have a faux-stake that would encourage me to look further. But in reality I haven't got any reasonable stake, so I'm perfectly happy to accept the NEMJ's published result. If it's wrong, what have I lost? Virtually nothing. I've told myself it's probably true... but only probably. I've only got one source and no weight of prior knowledge to rely upon. If it matters more to me later - if I find I have a greater stake in the bet - I'll run the odds with more care. But if I've got to put my chit down now, I'm going with the odds I have.
In the Reality Casino, just like everywhere else, in the long run the house wins... and the house is science.

Okay, to extend the gamble metaphor further is to break it... 'cause really what we are talking about is that there is more than one house, each telling us different odds.

I've been wrestling with this for the duration of my time involved with formalized skepticism. The error in the article seems to be, to me, that it is making an un-realistic generalization.

First, it assumes equal value on every subject to each individual. You couldn't squeeze enough 'care' out of me over Letrozole to fill a thimble, so I'm not looking further.
I am however, totally-fucking-furious-insane-mad-ass-kicking-angry about the anti-vax movement. I can't imagine there are a lot lay-people like me who have read the original Wakefield article in the Lancet, but I have. (Honest to "The thing that made the things for which there is no known maker and that causes and directs the events that we can't otherwise explain and which doesn't need to have been made and is the one thing from which you can ask for things that no human can give and without whom we can't be fully happy and is unlimited by all the laws of physics and never began and will never finish and is invisible but actually everywhere at once and who is so perfect that even if he killed millions of people including babies he'd still be perfect and who is so powerful and magical he could even make a virgin pregnant if he wanted to", I have.) I've read a bunch of other anti-vax information too, because I've invested myself in it. Weighing the wealth of material to the best of my abilities I have decided that I'm not putting my chit down with Wakefield.

It also seems to assume a percieved value in spending our infinite amount of time to peruse undigested scientific literature. I am far more likely than say, Citizen Sarah, to see value in reading all that science, and I have many better things to do with my finite time than that - and it's obvious that despite reading "all of them, everything that's in front of me", Sarah is even less prone to reading undigested scientific literature than I.

So where does that leave me? Unless there is specific motivation to better assess the odds, I'm going to rely on the most available reliable scientific journals as a short-cut - an heuristic.