Skip to main content

Bad science writer, bad! A major mea culpa

The authors found that increased neck pain is 25% more likely with SMT than if you did nothing or stuck to safe and neutral treatments. No. Wrong. Bzzz! Thank you for playing. This morning I received a note from Lisa Carlesso, PT, MSc, first author of the paper, letting me know that I got it wrong: although her data showed that 25% number, it was not a statistically significant number. And that’s significant. If there were any noteworthy increases in neck pain following this kind of neck treatment, presumably clearer data would have emerged. Thus the paper concluded that there is “strong evidence that neck manipulation or mobilization does not result in an increase in neck pain.” I’m not sure if I quite agree that a statistically insignificant number constitutes “strong evidence” so much as just generally low confidence in the results (and Carlesso acknowledges this in the paper as well, practically in the next sentence: I basically saw in the data what I wanted to see. Funny how that works. “However, the limitations of the Strunk study and the low GRADE rating remain, affecting confidence in the estimate.”) But I am guilty of doing something I’ve accused others of doing: emphasizing a statistically insignificant number to make my point. When the numbers lean your way but fail to reach statistical significance, it’s called a “trend.” A trend in favour of a therapy is often trotted out as if it were supportive evidence. I did the opposite, and so I am doing the walk of shame now. Bad science writer, bad! I erroneously thought the number was statistically significant, probably because I’m a debunker by nature, and I basically saw in the data what I wanted to see. Funny how that works. (Gosh, I wonder what system of knowledge-seeking could possibly compensate for that aspect of human nature? You get a gold star if you guessed “science.”) What does it mean? Not much, really. “Here be statistical dragons.” There are so many ways that all this stuff about treatment harms can be wrong that I can’t really walk away from this feeling like I’ve learned anything terribly important one way or the other. The hard statistical bottom line is that statistically insignificant means that no conclusion can actually be drawn — the data was no more than suggestive. Interestingly, Lisa Carlesso pointed out that she has written another paper about how difficult it is to study adverse effects.

Popular posts from this blog

PF-ROM Exercises

"Pain-free range of motion’ or early mobilization exercises can help you heal" PF-ROM is physical therapy talk for “pain free range of motion.” When a therapist evaluates an injury, he or she will be interested to see how far you can move affected joints without hurting. Sometimes, of course, you can’t move at all without pain. But in most injuries, even many serious ones, you will have at least some painless movement. And whatever you’ve got, you should use. When you are hurt, the pain-free range is your new best friend: that’s the range you’ll be exercising in for a while. Pain free range of motion exercises are also known as “early mobilization.” Use it or lose it “Use it or lose it,” they say. And it’s true. While many seemingly simple medical questions are controversial, this one appears to be straightforward: plenty of recent research demonstrates that early mobilization is A Very Good Thing. A 2006 study of people with surgically repaired achilles tendon ruptures sh...
Read more

Exercise

Two sets of US guidelines have revised the public health advice on physical activity to clarify that gentle exercise is not enough to improve health, The Guardian reported. Current guidelines suggest that 30 minutes of exercise a day is enough to offer health benefits, however the newspaper said that adults “need to add jogging and twice-weekly weight-training sessions if they want to cut their risk of heart disease and obesity.” The newspaper reports that “the lightest of activities such as dusting and the stroll to the car are being counted as exercise” and quotes the researchers as saying that people have “not accepted, and others have misinterpreted the original recommendation.” The expert panel process and the recommendations offered have clarified some of the more unclear recommendations that were published in 1995. The group of experts met in 2000 to prepare this update, and this was supplemented by panel members’ own searches of the literature. It is not possible to validate ...
Read more

A Short Story

A silent, anonymous place I was still a child the last time I’d seen it. The hospital was an anonymous place my parents had taken me to, somewhere in the big city, far from home. I didn’t know where I was at the time, but when I saw it again as an adult, there was a match with something stored in my skull: the outline of the building like a stack of building blocks, the primary colours, the silence that seems out of place around a place built for children. Medical science failed me in that hospital. I was diagnosed early with “slow growth syndrome,” and this was the first mistake that science made with me: giving it a name. Slow growth syndrome is no syndrome at all. Some people grow slowly. So what? I was small for my age. I didn’t have a pathology: I had a velocity. I was taking my time. Nevertheless, the doctor wanted to keep an eye on me, so my parents dutifully took me back every two years to be measured and the pace of my growth evaluated for more distressing signs of oddity. I...
Read more