Death by Stats

I’ve looked at the issue of ‘framing’ before. As have many others (who have generally done it better and more insightfully). It’s a very important thing.

If, for example, you work in a frame in which The Orange Don represents an existential threat to democracy, it’s not going matter what any of his specific policy proposals are. I’m the same, but the other way round. I think the Democrat party, based on current evidence and their record in office, represent the clearest threat to the institutions and laws that allow democracy to flourish in the US.

Our initial ‘framing’ tends to colour everything we subsequently think.

In my last piece I wanted to highlight the issue of an innocent vs guilty framing for Lucy Letby, a nurse in the UK who was convicted of the murder of 7 babies in a high-dependency neonatal baby unit.

The framing of this as an issue of innocence vs guilt already implies a crime has been committed.

But it’s far from clear in the Letby case whether any crime was committed at all.

Babies are a very emotive issue. As if by magic, once they are outside of their mother’s wombs they cease to be mere clumps of cells and become little human bundles of joy. And woe betide anyone who would harm them (provided they’re on the outside, that is).

So the suspicion that a serial killer of children was operating in one of the UK’s hospitals hits those heart strings very hard.

We demand justice and no little retribution for such heinous crimes.

The problem, as I outlined, is that in one specific baby unit in the UK there was a spike in deaths in the years 2015 and 2016. It would seem that such an apparently unusual pattern cries out for explanation. Maybe it does, but maybe it doesn’t. And this is where statistics comes into play in helping us get a better idea of what we may (or may not be) dealing with.

The first thing to do, then, is to recognise that babies in these kinds of units are not well at all. That’s why they’re there; they need considerable levels of extra care and attention. Sadly, even despite that extra care and attention, a number do still die.

But just looking at the raw numbers isn’t quite enough. One needs to look at the ‘per capita’ data to see whether anything unusual is happening. 8 deaths may sound a lot in one year, but is that 8 deaths out of 100 births or 1,000 births?

One may then compare rates with other hospitals in the UK to see whether an apparently unusual pattern is, indeed, unusual. You can see some comparator data in this post.

If we just compare with one, for example, by looking at The Countess of Chester Hospital (where Letby worked) and Sherwood Forest Hospitals we find the following crude rates per 1,000 births by year listed from 2014 to 2017

CoCH : 1.32 2.96 2.62 1.03
SFH : 3.77 1.77 2.04 1.15

Look at that rate in Sherwood Forest for 2014. It’s 3.77. Must have been a serial killer operating, no?

Well, no, obviously. You need to delve a bit into the detail and try to see if there was any systematic problem which may be a causal factor. And there may not have been. It may just be the vagaries of statistics and that a rate of 3.77 falls within expected statistical fluctuation.

The problem is is that once you have convinced yourself that any pattern is sufficiently unusual so that the most likely explanation is foul play you can end up letting your imagination run wild and come up with any number of possible scenarios as explanation. Air embolisms, insulin injections, maybe even some witchcraft.

The question to answer is whether it is more likely the pattern was caused by foul play, or just the sad fact of life that in any unit where the babies are very vulnerable and unwell a number are going to die whatever you do.

If you do decide the pattern is sufficiently unusual to warrant a detailed investigation then there are other explanations that one might, more reasonably, look to before leaping onto the serial killer bandwagon. Understaffing? Protocols not properly adhered to? Hygiene problems on the unit? Did the hospital upgrade to a Level 2 unit before it was sufficiently ready to do so?

Any of these factors could introduce a bias that would tend to lead to a rise in death rate without any necessity of leaping to the conclusion of foul play.

But just how good are we at recognising an unusual pattern in the first place?

Not very good at all, I would suggest.

It may all seem like a bit of statistical frippery, some arcane dark craft only accessible to those who have waded through hundreds of textbooks (and understood them), but it turns out to be quite important.

Look at the role bad statistics has played in all of our lives in the years since 2020. The words “safe and effective” should send the mother of all chills down our spines by now.

The thing is that in any sequence of ‘random’ data there are probably going to be sub-sequences that don’t ‘look’ random.

If I asked you to write a 12-digit random sequence of 1’s and 0’s where 1 and 0 were equally likely, most people would not write something like this

101010101010

Why not? Because to a human eye this does not ‘look’ random. Yet the occurrence of this specific 12-digit sequence is just as likely as

011000101101

if truly chosen (uniformly) at random. Our pattern recognition circuits interpret the second sequence here as being, somehow, ‘more random’.

Here’s how our ‘intuitive’ feeling for random vs non random can trip us up a bit.

Suppose I asked you to flip a (fair) coin 7 times. What’s the probability that you’re going to end up with 7 heads? It’s fairly simple to work out what this probability is.

1/2 probability of a head on first flip, 1/2 on the second, 1/2 on the third, and so on. To get the final probability we just multiply these together so that the probability of getting 7 heads in a row is

1/2 x 1/2 x 1/2 x 1/2 x 1/2 x 1/2 x 1/2 = 1/128 = 0.0078

But what if I asked the probability of seeing a sequence of 7 heads in 40 flips of a coin? We might still think is a low probability. If getting 7 heads in a row is rare for 7 flips, then getting 7 heads in a row anywhere is going to be similarly rare, might be our initial thinking. After all, couldn’t we just view the point at which we get 7 heads as being like our initial problem where we had only 7 flips to begin with?

Not quite. Because we’re looking for this ‘order’, this pattern, anywhere in the whole sequence of 40 flips.

Working out what this probability is is not a trivial calculation to do (you can see one way to do this here). The answer turns out to be about 0.13 which is about 16 times more likely than getting 7 heads in 7 flips.

This would not be judged to be extremely unlikely happening, as it does, in more than 1 in 10 times you flip a coin 40 times.

The point here is in a 40 flip sequence you’ve got a decent probability (although still low-ish) of seeing something like the following

stuff that ‘looks’ random —> appearance of order —> more random ‘looking’ stuff

In other words, we have a sequence that ‘looks’ like

usual —> unusual —> usual

But, statistically speaking, seeing this kind of sequence in data is not at all unusual.

This is why we have to extremely careful, particularly if you’re going to accuse someone of something as heinous as murdering babies, to make sure we are actually dealing with something for which the most likely explanation is the most unusual one.

Are we sufficiently certain that the entire framing of the 7 deaths (out of 17) that were attributed to Letby as being murders is the correct framing? That framing colours everything. That framing did not originate from any forensic assessment by a pathologist but was driven by the association of what was claimed to be an unusual pattern of deaths with working shifts - and that association has been demonstrated to have been very questionable in itself.