Efficaciously Efficacious

I’ve been trying to find a relatively simple way of understanding the effectiveness of vaccines. I work best with squiggles - an equation makes more sense to me than spreadsheets full of data.

The TL;DR here is just the following (which we all already know)

Excluding the Limbo group (jabbed, but not yet considered vaccinated) really makes your efficacy look better than it really is

What’s the issue I’m trying to understand?

Outline of problem

Vaccine effectiveness is defined in terms of a ratio of two things. It compares two populations. But how do we find a sensible measure of effectiveness when we have three things to compare?

Interestingly, there’s a similar problem in some of the quantum mechanics research stuff I’ve done. In quantum mechanics there’s a possibility for two objects to be more strongly correlated with one another than is possible in classical physics. This is a result of something called entanglement, which is an entirely quantum property. Two quantum-correlated objects can easily be characterised. But how do we characterise things when 3 (or more) objects are quantum-correlated? There isn’t an easy answer to this question.

In order to try to get some sort of insight with vaccines I restricted things to looking at just a single vaccine shot. We start off with everyone in the unvaccinated state. We start vaccinating and for a period of time (14 or 21 days, for example) we have two populations - the unvaccinated, and those in something I have termed vaccine limbo.

After this time (14 or 21 days, for example) those who have survived vaccine limbo for this period get moved into a new population group - the vaccinated. Thus we have 3 populations to compare as the situation unfolds. It’s like we have 3 tanks - the first one is initially full and then the tap is opened and it starts to fill a second tank. Some time later the tap on this second tank is opened and a third tank starts to fill up.

But vaccine effectiveness is only defined in terms of a comparison between two populations. So what’s the way to proceed here?

Potential for survivorship bias

The first thing to note is there is a potential for something called survivorship bias. Let’s imagine that the vaccines do nothing in terms of preventing death from covid. Absolutely zilch. Let’s further suppose that the vaccines do harm such that those who would have died from covid are killed by the jab. The effect of this, then, would be to filter out the vulnerable - the only people who end up in the vaccinated group would be those who don’t die from covid, the survivors, the invulnerable, so to speak. The people remaining in the unvaccinated group would be a mix of the vulnerable and the invulnerable.

If we compared the vaccinated group with just the unvaccinated group we’d find 100% vaccine efficacy - there would be no deaths in the vaccinated group, even though the vaccine was doing nothing to prevent death from infection.

That’s obviously an extreme limit case - but it illustrates the problems both with survivorship bias and with a single comparative measure.

Here’s another, less extreme, way something like this could play a role. Let’s imagine we split our population into strong (S), frail (F), and very frail (VF). The very frail do not get vaccinated because it is considered to be too risky for them. The unvaccinated group would consist of a mix of S, F and VF people. The vaccinated group consists only of S and F people. Thus, when you do an efficacy comparison between these two groups, you’re not properly comparing like with like - there is a bias in favour of the vaccinated group (they are slightly more healthy on average initially). In the calculation you have to take out the VF population from the unvaccinated group to get a fair comparison.

The definition of vaccine efficacy

When you hear statements like “the vaccine is 90% effective” what does that actually mean? In a vaccine trial you will have 2 populations, V and P - the vaccinated group and the placebo group. If you are looking for a measure of “efficacy against death” what would we do?

If there were the same number of people in both vaccine and placebo groups we would look at number of deaths in V and the number of deaths in P, and work out the difference in terms of a percentage. We would look at:

number of placebo deaths minus number of vaccinated deaths : D(P) - D(V)
then divide this difference by number of placebo deaths and multiply by 100

But we might not have the same number of people in both groups. So, more completely, we have to work in terms of rates. We take the same construction as above - but this time in terms of rates, not just raw numbers.

Here’s what it looks like (I’ve dropped the multiplication by 100 here - so we’re expressing vaccine efficacy as a fraction rather than a percentage).

What do we compare with what?

We have three groups - U, L and V (the unvaccinated, the limbo and the vaccinated) - but we only have a measure of efficacy that compares two groups at a time. We could compare U and V and exclude L altogether. We could lump U and L together so that we would compare (U,L) with V. We could lump L and V together so that we compare U with (L,V). These are just some of the possibilities. But which is “correct” and how do we understand the consequences of choosing one grouping over another?

There’s no good answer to which of these is “correct”. However, if you’re going to use the comparison (U,L) vs V, then you’re mixing up those who have had an intervention with those who haven’t. The argument for using this grouping is that people are not considered to be vaccinated until after a certain period when the protection kicks in - so in order to properly figure out whether the vaccine is working you have to compare those whose protection has kicked in with everyone else (not to mention the fact that if the jabs make things temporarily worse, then you’re effectively shifting this effect into the categorization “unvaccinated” - which obviously is going to make things appear better for the vaccine).

I don’t like this, personally. I would rather compare those who have had an intervention vs those who haven’t. This is the grouping U vs (L,V). The reason why this makes more sense for these vaccines, as mentioned above, is that the data strongly suggests there is a period after vaccination where you are more likely to catch covid. The official charts from Alberta suggest something like this (these charts were removed from the official website after people began to notice them)

It very much matters which grouping we use. Let’s just compare 2 of the different efficacies we can compute.

Exclusion efficacy : in this one we will compare the groups U and V and exclude L

Intervention efficacy : in this one we compare those who have with those who haven’t had an intervention - so it’s the comparison of U with (L,V)

I’m only going to consider efficacy against infection here. This is purely for simplification purposes. What I’m interested in is the consequence of choosing one grouping over another - the conclusions would still hold if we considered efficacy against death although the calculations wouldn’t be as straightforward.

Comparison of efficacies

In the UK the uptake of the vaccines can be reasonably modelled as a sigmoid curve - especially for the older cohorts. As an example, the 2nd dose data looks like this

There is an initial sharp uptake followed by a steady state. The time between the start and the steady state is only a matter of a few weeks in the older cohorts. I’m going to consider an overall time period which begins with the start of the vaccination and goes some way into the steady state.

If you look at the 75 to 80 group here, for example, we see the vaccine rollout really begin to take off around week 10 to 11 and the steady state is essentially reached around week 17.

Everyone who ends up in the vaccinated group has passed through vaccine limbo. This allows us to make a nice simplification. Over the time period we count the numbers of infections in each group and we can work out some rate of infection (infections per population) to give us an effective rate when we’ve reached the steady state.

We would have an effective R(U), the rate at which the unvaccinated are infected. We can write R(L) = qR(U) where q is going to be some factor that allows us to express the rate in limbo in terms of the rate in the unvaccinated. This is not necessarily greater than 1 because limbo exists only for a short period even though being in limbo might increase your chances of infection on a given day. And we have R(V), the rate of infection when people have passed through vaccine limbo.

The exclusion efficacy is just 1 - R(V)/R(U). We label this as e(E). This measure ignores any of the vaccine limbo data. If the vaccines expose someone to more risk for a short period then this efficacy measure will favour the vaccines - but by how much?

To work out the intervention efficacy, e(I), we have to know R(L,V) - the effective rate in the Limbo and Vaccinated populations combined. The nice simplification here is that if we’re looking at the data when we’ve reached the steady state, then the population here is just the number in the vaccinated group, and this is also the number of those who have passed through limbo. This means that we can write

R(L,V) = R(L) + R(V)

We can understand this in terms of probabilities. In the steady state your chance of having had an infection if you’re in the vaccinated group is just going to be the chance you were infected in Limbo plus the chance you were infected when you passed through Limbo.

When you plug this into the expression for efficacy we find that

e(I) = e(E) - q

In other words when you work out the steady-state efficacy by comparing intervention vs no intervention you find a lower vaccine efficacy by this amount q than when you calculate an efficacy based on excluding the Limbo group.

It’s no wonder that those who want to promote the vaccine will want to exclude the Limbo group when calculating efficacy - it leads to a higher value.

TL;DR - full of technical mumbo-jumbo

The message here is the same one many people have been saying for quite a while. Excluding the Limbo group (jabbed, but not yet considered vaccinated) really makes your efficacy look better than it really is.

I’ve only considered just one shot of a vaccine here - it’s a simplification for the purposes of getting some insight into the effect of what I can only see as mis-categorization.

I’ve played with the other possible groupings but I’m getting a bit of mathematical symbol salad coming out for some - I am probably being dumb and not seeing something obvious. I will keep you posted if my brain ever starts working properly again.

---