Medical "Research"

[selenite0]

Reuters has an interesting news article up: Tetanus jab may curb multiple sclerosis risk but it turns out to be yet another example of how bad medical research is. The sample size they're looking at is so small they can't get any useful conclusions. And taking a harder look at it I think the conclusion they're pushing is false.

Since MS is an autoimmune disease (by leading theories) the article immediately caught my interest. One of the reasons we worry about the vaccine schedule for kids is that it affects only one part of the immune system, potentially causing an imbalance leading to autoimmune disorders (yes, this is speculation). The researchers have some analogous speculations:

The biologic mechanism by which the tetanus vaccination may protect against MS is unclear, according to the authors. They note, however, that vaccination with tetanus toxoid may shift the T helper cell immune response from a proinflammatory Th1 response to an anti-inflammatory Th2 response.

Then I caught another bit on my second read through the article:

Analyses centered on a total of 963 MS cases and 3126 controls. They found that a history of having been immunized against tetanus was associated with a 33 percent decrease in risk of MS.

As far as I can tell they're saying the controls, all having had tetanus shots, were less likely to get MS. [Or that the MS cases were less likely to have gotten tetanus, but that's unlikely to produce a nice round 33%] So what they're saying is "We got two cases when we should have expected three." A quick google showed that the incidence of MS in the USA is 0.91 per 1000 people.

Let's drop this into Excel. The study is asserting that for their group of tetanus vaccine recipients the odds of getting MS were lower than for the general population. I can simulate their study by putting a random function in 3126 cells, which each have a 0.091% probability of getting MS. So I ran this 100 times (ie, doing their experiment 100 times) and counted how many MS cases I got in each run. That gives me this graph:

The first thing to notice is that 100 trials isn't enough to produce a smooth distribution. The mean (average) came out a little low, 2.6 instead of the 2.8 it should be. But that's close enough to give us a feel for it. So note how tall that "2" column is. The prediction of three cases comes from multiplying the incidence by the number of people. But when you do that in real life you're going to get a few samples with lots of cases and many with fewer to balance them out. That's part of the problem with using a sample size so small. The expected result presses up against zero and gives you a distorted curve instead of a nice easily analyzed bell curve.

So what we're getting is that "two cases" is both the median (result in the middle) and the mode (most common result) of the distribution. So getting that in their sample isn't a "33% improvement", it's WHAT THEY SHOULD EXPECT. The result of the study doesn't show the probability of getting MS changed from getting tetanus vaccine. If it had gone down dramatically they should have seen one or no cases of MS in the group. But even if that had happened it wouldn't be proof. There's a 1 in 4 chance that would have happened even with the normal probability of MS.

So the conclusions to draw are:
1. Don't draw conclusions from small sample sizes.
2. Tetanus has no apparent impact on your odds of MS.

There's also a 1-in-4 chance that they'd have come up with results showing that tetanus vaccination increases the chance of getting MS. I wonder if they would have published that.

In defense of Dr. Hernan and his colleagues, they're using a larger sample size than the nine studies they're doing a meta-analysis of. And I'm working from a Reuters article, so this may have been a discussion of the innumeracy of science reporters, not medical researchers. But it's a good example of why I'm very wary of widely publicized medical research showing something is/isn't safe.

Comments

[tmc4242.livejournal.com]

Didn't you already post this ? I'm having flashbacks or something... :-)

[archangelbeth]

Me too! O:>

Maybe it's an artifact of LJ.

[selenite.livejournal.com]

Nope, me deciding it had to go back into the oven for a bit.

[celticdragonfly.livejournal.com]

I told him it needed to be unpacked more for clarity.

[selenite.livejournal.com]

I posted most of it, then took it down for a rewrite. I guess I didn't take it down fast enough for this crowd . ..

Repost of <lj user=btripp> comment

[selenite.livejournal.com]

[this was lost when I took down the original draft post]

"But it's a good example of why I'm very wary of widely publicized medical research showing something is/isn't safe."

With a very few exceptions, the MSM is filled with people who probably know so little about science that they can't figure out how to put batteries in right. As such, almost ALL science/medical reporting is based on whatever "emotional" content the reporter could glean from whatever he/she had been assigned to cover. It's the "Oh, that sounds terrible!" response, which more often than not is based on a total misunderstanding of the data. There have been products and industries irreparably harmed by this sort of reporting (think: Cyclamates) ... so I never trust anything that's reported in other than specifically "scientific/medical" forums (and even those twist shit around way too frequently)!

Re: Repost of <lj user=btripp> comment

[carbonelle.livejournal.com]

This is why I don't read the MSM: I read Science News. Decent source and one can always go back to the originals.

Re: Repost of <lj user=btripp> comment

[p-o-u-n-c-e-r.livejournal.com]

You'd like NumberWatch.com. Bragnell, the host, is big on these sorts of errors, but from a Brit perspective.

The MSM is filled with people who can't do ARITHMETIC, much less science. I've been working with a nice man who covers our suburb's politics. Me: "Last month the superintendent said 64 million. Today he says 72 million. Do you see a problem here?" "Well, that's close enough for an estimate, isn't it? We can use round numbers for both and say 70 million, right?"

Devil's Advocate

[carbonelle.livejournal.com]

One of the intersting thing I've read about is how a lack of exposure to various parasites, etc. can increase the likelihood of autoimmune disorders such as life threatening allergies. The concept is that modern man, who has spent "billions and billions of years" wallowing in a sink of pathogens, has a system unused to so much disease-free cleanliness.

If true, we could radically increase the *number* of vaccines children get with only good (barring the odd idiosyncratic reaction: Curse that genetic variability :-( consequences, so long as we took care to space them out.

[ndrosen.livejournal.com]

Hurray for Science News! Pity they didn't write this up.

It could be that what the researchers actually found was, let's say, that 90% of the controls but only 60% of the MS patients had been vaccinated against tetanus, which could be significant, That isn't what the snippet you quoted said, but, as has been pointed out, reporters are often innumerate.

[selenite.livejournal.com]

I'd lean toward that interpretation if they'd described a 29% or 38% benefit from the vaccinations. But 33% makes me suspicious.

Hopefully Science News--or some other reputable source--will write this up with more useful details. Then I'll get to update this post with more precision or apologies as required.

Links for later review

[selenite.livejournal.com]

From Raye Johnsen on LMB list:

These are all pages based in Australia, on Australian stats.

http://www.ratbags.com/rsoles/gl/vaccines1.htm
a list of pages about vaccines and vaccinations; I followed the links there for the following two links.

http://www.cdc.gov/nip/publications/fs/gen/WhatIfStop.htm
- no specific statistics but contains number comparisons of infections before and after the introduction of vaccines.

http://www.skeptics.com.au/journal/1997/1_immunise.htm
- a journal article responding to another article by a doctor asserting that vaccination is ineffective; about a third of the way in, there's a table giving the statistics in Japan of infection rate of and deaths due to whooping cough during the immunisation period (1970-1974) and after general vaccination was discontinued (1975-1979).

http://www.vaccination.inoz.com/vacprotect2.html -
gives death rates before and after introduction of vaccine, argues that death from common childhood diseases were already on the decline before the introduction of vaccines.