Review: The average cost of measles cases and adverse events following vaccination in industrialised countries
One of our readers posted a comment asking us: “Can you tease out some facts in this study?” We think we could probably write several papers discussing the issues surrounding this study: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC128813/ but we will provide a brief summary because a study such as this isn’t worth spending an inordinate amount of time on.
We are going to make a few opening caveats:
1) Some of us believe that measles might be the only vaccine that is justifiable on a large population basis and that is only because there is some evidence that measles can have a relatively high (still low on an absolute basis) rate of serious side effects in some populations.
2) Cost justification studies (such as this) are usually based on a house of cards, and are only as good as the data that underlies the layers of assumptions made in the model.
3) Cost justification studies that are used to support mass vaccination mandates almost invariably turn out to be wrong due to underestimating the cost of the vaccine program and side-effects and overestimating the effectiveness of the vaccines.
4) These studies are often misleading because they are usually sensitive to a few key assumptions and they normalize everything to a dollar value. Here is a sample problem with normalizing everything to a dollar value: let us imagine we have two different vaccines that we want to give to 1 million people. In the testing, 50% of the people suffered 3 days of mild illness causing missed work with no long term effects from vaccine A. 1 person died from vaccine B with no one else suffering any ill effects. The cost of vaccine A would be far higher in almost any financial model, and yet clearly we would much rather fall mildly ill for 3 days rather than risk a 1 in a million chance of dying.
This particular study is interesting in that it is not actually performing a cost comparison or justification. It is only trying to set the cost of a measles case and the cost of a measles vaccine reaction. In order to judge the likelihood of bias in a study, a quick check on the authors reveals that although there are no direct conflicts of interest declared, several of them work for organizations which were desperate to defend the MMR vaccine in the wake of the Wakefield papers from around that time period. There is nothing wrong with that, but it provides context for the timing, content, and potential bias’ of the study.
When evaluating any study, we always find it very important to look up the references and at least do a read. Since we’re quite familiar with the evidence available surrounding vaccines, we were interested in the references they used to support the background information. The study states early on
These cases of measles can lead to serious complications and deaths as recently demonstrated in outbreaks in Ireland , Germany  and the Netherlands [4,5].
Unfortunately, the references provided do not actually confirm the causes of death in the two cases from Ireland. Ironically the German reference #2 states:
No serious complications were observed, apart from a case of otitis media, resulting in an unusually low complication rate…
I stopped there because I’m already getting a sense of bias from that blatant misrepresentation of the reference. This reference is very important because most of the measles data from sources such as the CDC site are worldwide averages which are severely skewed by numbers from the developing world. Noting this modern German reference, it’s interesting that the low rate of complications surprised the authors as it differed from the data they were expecting. This issue really strikes at the heart of the matter of measles vaccination because it is these very estimates which form the basis of the cost calculation this paper is going to do.
If we fast forward to the references they use to create their model, we find a reference to a UK study from 1985 which itself looked at data ranging from 1957 to 1976. One could question whether the reporting/tracking quality in that time period is really applicable to the 1990′s. The US reference is even better (sarcasm). It is referencing another study that is doing a cost/benefit estimate and model. This is a secondary reference at best which is itself an estimate. Unraveling the layers of estimates and assumptions is a lot of work, and it’s a shame the authors didn’t actually list source data.
Validating the serious measles reactions data is even more tricky. The study from the UK in 1963 is actually pretty good even if it’s old (reference 10). Indeed, this first reference notes that
Deaths have indeed declined rapidly in recent years to about 2 per 10,000 notifications and a recent study has shown that about half the deaths occur in persons with serious chronic disease or disability (Report, 1963).
That is why we don’t accept press releases as references when it comes to judging deaths from measles or any illness for that matter. Even worse, when doing estimates on vaccine adverse reaction rates, any confounders (people with chronic illness etc) will be eliminated from the study and yet, here, when calculating data for measles costs, those people are included. In other words, they are comparing apples and oranges. The statistics on measles deaths and disability come from the full population including those that have chronic illness and underlying conditions. The statistics on adverse vaccine events come from clinical trial populations which are almost always perfectly health people because these studies go out of their way to exclude those with underlying health conditions. I think it’s obvious that the cost of illness data will be biased higher and the cost of adverse reaction biased lower. I don’t have the time to actually check to see if the references justify the adverse reaction rate they chose, but I can tell you that a weighted average of 5 studies can be very misleading.
Before we move on to the cost of adverse reactions, we would like to point to the conclusion of the Cochrane report on measles vaccination published first in 2005 and subsequently in 2006.
The design and reporting of safety outcomes in MMR vaccine studies, both pre- and post-marketing, are largely inadequate. The evidence of adverse events following immunisation with MMR cannot be separated from its role in preventing the target diseases.
If you read the actual report, they are pretty clear that the safety study and reporting of adverse reactions is abysmal for a vaccine that has been used for several decades now. Given this background, we can already assess that this study from 2002 is not working with any reliable data surrounding adverse events, and therefore the costs associated with them are essentially guesses.
However, for fun, let’s review what they did say. The first big assumption they make is that the vaccines didn’t cause any of the serious neurological outcomes including: GBS, Autism, and Chrohn’s disease. They quote three references, all of which were highly publicized following the Wakefield study (remember above we noted the likely bias). These same studies were included in the Cochrane review and yet, Cochrane did not make any definitive findings. We also know that MMR has been associated with Autism in at least one case (Hannah Poling). Clearly this assumption will have a huge impact on the outcome of this cost analysis. They also disregarded all long term issues with post-vaccination encephalitis or febrile seizures. This is incomprehensible given that the US VICP has been handing out large monetary compensation for long term damage occurring post encephalitis from MMR vaccination (including Hannah Poling). One of the two studies referenced to support this ridiculous assumption is a twin study (extremely limited data set) and the other study is from the Wakefield era (bias warning again, in fact, this was clearly produced by the CDC group post Wakefield which includes numerous conflicted individuals –we didn’t see a conflict of interest declaration on this paper…). This study excluded children who suffered non-febrile seizures. Additionally, if you look at the result which concludes that the risk of subsequent long term issues post febrile seizure from vaccines, the confidence interval is huge (0.07 – 4.2). That is pretty inconclusive and the risks could be as high as 4 times. Based on this data, they dismissed all possibility of long term damage from vaccine reactions to MMR.
Taken in totality, I can safely conclude that the cost analysis of the vaccine reactions is severely biased and flawed.
Why bother going further? To be honest, I couldn’t resist looking at the sensitivity analysis and this is where we find ourselves back to the misleading data I warned about in point #4 earlier.
The three most influential variables on costs for the average cost per measles case were the average number of work days lost by the mother for a non-hospitalized case, the proportion of cases not seeking medical attention and the proportion of encephalitis cases developing sequelae leading to residential care.
So the cost of measles is highly sensitive to the number of work days lost to mild cases and how many mild cases there are! Let’s look at the adverse event data:
The three most influential variables for the average cost per AEFI were the percentage of fever cases seen in a clinic, the rate of fever and the average period of absenteeism when a child had to visit a physician.
Of course they didn’t mention it is likely highly sensitive to the complete omission of any possibility of encephalitis cases developing sequelae leading to residential care.
We encourage you to read the references. As always, the devil is in the details.