Testing, Testing, 1,2,3, . . .

Today, I want to revisit an old topic, and talk about something most of you probably already know about. I’m going to try to explain why mass testing of asymptomatic individuals is dumber than a Brussels sprout.

It’s especially important right now because almost everyone in the media appears to be having a real panic-fest over “cases” - as do our politicians. I’ve never properly understood (other than as a method to generate panic) why there has been such a focus on cases. It might be a useful thing to know, certainly, but really the most important thing is the number of people who are actually seriously ill.

The Great Ballbag of Doom, Professor Neil “I’ve got to get it right sometime soon, dammit” Ferguson, said we could see 5,000 Omicron deaths a DAY in the UK if we don’t cower in extreme terror and hide behind our sofas, . . . or something.

Cuddly Neil, happy to double and triple jab his mistress during lockdown whilst telling the rest of us to isolate, goes in more wrong directions than a supermarket trolley.

Unlike Neil, however, a supermarket trolley does serve a useful purpose.

With Ferguson’s track record I would have been much more worried if he had said we’ll see only 10 deaths a day. Given his ability to be orders of magnitude out from reality, this would have been worrying indeed. I do sometimes wonder whether he’s got a sticky zero key on his computer.

I’m kind of ashamed to say it these days, but I actually got my PhD in Theoretical Physics from Imperial.

So what’s the deal with testing? Why is there so much controversy and argument over it? Why do so many people have faith in the results? Unfortunately, it’s one of those things that does require a teeny bit of understanding of probability - no way round it - but I’ll try and make it as painless as I can.

You go to the docs - well these days you can’t actually do that unless they’re sticking something into you. Cancer, madam? Bugger that - have a covid booster jab instead, there’s a really bad cold going round.

But suppose in some medical wonderland, some utopian alternate universe, you actually could get to see your doc and he/she/it decides you need a test for something. Heeshit (my new catchy word to encompass he/she/it and be inclusive of all genders and identities) tells you the test is 99% accurate. It comes back positive. At this point most people would believe there’s a 99% chance they actually have the disease they’ve been tested for.

Unfortunately, this is not correct. More unfortunately still, the reason why it’s incorrect is quite difficult to appreciate.

The best way to get a bit of insight here is to think of a somewhat crazy example. Dr Loon invents a test for a virus that doesn’t even exist. It’s a great test - only 1% false positive rate - which means that if you test 100 people who definitely don’t have the disease you’ll get 1 positive result back (on average). So on a per test basis it’s pretty good and accurate.

Dr Loon proceeds to test a million people. NONE of these people actually have the disease, of course. What do we find? Well, we’re going to find that 10,000 people return a positive result (on average). Every single one of these positive results is a false positive.

This is how you can have a very good test (on a per test basis), but which, at a population level, gives you meaningless results.

So let’s suppose that Dr Loon is somewhat vindicated and it turns out that the virus really does exist, after all. It’s not much of a virus, though, and only infects 1 person in a million.

What happens when we test our million people now? Well, we’re still going to see about 10,000 false positives on average - and we’ll also probably pick up the ONE person who actually is infected.

You get a positive result here. What’s the chance you’re that ONE person who is actually infected?

This, in a nutshell, is the problem. When you hear numbers about test “accuracy” these are answers to a different question than the one you’re interested in.

The question YOU are interested in when you take a test is the following:

Patient Question

Oh no - I got a positive test result. What’s my chance of actually having the disease?

The thing is, when we characterize a medical test, this is NOT the question we ask. We ask the following:

Medical Test Question 1

If I test people who definitely have the disease, what’s the chance my test will confirm that?

You can see, I hope, just how different these questions are. Actually, there are two questions that get asked when characterizing a medical test. We’ve already seen a version of it, but let’s repeat it for completeness:

Medical Test Question 2

If I test people who definitely do not have the disease, what’s the chance my test will confirm that?

The answer to the first medical test question here gets called the sensitivity. The answer to the second medical test question here gets called the specificity.

The specificity is what’s telling us about the rate of false positives on a per test basis.

What we really need to do is to get an answer to the patient’s question. The good news is that we can do this if we know 3 things - the test sensitivity, the test specificity, and the prevalence of the disease in the tested population.

The bad news, of course, is that the way we connect these things involves some maths and probability. We end up with:

answer to patient question = stuff involving sensitivity, specificity and prevalence

The prevalence turns out to be a very critical parameter - and you can see why from the Dr Loon examples above. I won’t write down the formula here, but I’ll illustrate things with an example. Let’s suppose we have a good test with sensitivity and specificity at 99%

This means that if I test 100 people who definitely have the disease I’m going to pick up 99 of them with my test, on average (sensitivity)

And if I test 100 people who definitely don’t have the disease I’m going to get a negative result for 99, on average (specificity)

Let’s suppose that we have 1 in 50 people infected in the general population and we test everyone, regardless of symptoms. What we find is that the answer to the patient’s question “I have a positive test. Do I actually have the disease?” is

You are 67% likely to have the disease.

This means that, at a population level, one third of the people who tested positive have a false positive result.

This is why, when performing medical tests for diseases, it is strongly recommended you test only those with suspected symptoms, or those whom you suspect of having the disease for some good reason. By doing this, you’re selecting a smaller population with a higher chance of being infected - in effect, you’re looking at a sample with a higher prevalence than exists in the general population.

This is the general (well-established, entirely non-controversial) background to the issues involved in medical testing - but we make even more of a mess than this when it comes to covid.

The PCR test is a brilliant technique for detecting trace amounts of genetic material. Because it is so sensitive it has to be performed under very carefully controlled conditions. When you have a test this sensitive, cross-contamination becomes a major concern. When you hear about the accuracy of the PCR test (high sensitivity and specificity) you have to bear in mind that these figures are telling you about lab results on these things. In other words, results from very-carefully controlled environments with highly-trained and skilled personnel.

Unfortunately, we’ve ramped up PCR testing to an industrial scale. We shouldn’t expect quite the same performance figures. But even this is not really the issue.

The issue is that the PCR test only detects the presence of genetic material. It is not a test for active infection. Of course, there’s going to be a strong correlation between positive tests and active infections - but the fact that the PCR test can pick up expired infections alters things.

Black Box Test 1

I have a black box with some test inside. I input the sample to be tested. This black box is answering the question : have I detected the genetic material?

When you see the per-test false positive rate being reported - it is a false positive in relation to this question

Black Box Test 2

Unfortunately, when we do a covid test this is not the question that gets asked. Here we have the same black box, same test, same sample - but we’re using it to answer the different question: have I detected a case?

The (per-test) false-positive rate here is going to be higher. The (per-test) false-positive rate is dependent on the probability question that we’re trying to answer.

This is something that is often overlooked. People quote the (per-test) false-positive rate in relation to Black Box 1, when really we need the (per-test) false-positive rate in relation to Black Box 2. In fact, I don’t think we actually know the (per-test) false-positive rate for Black Box 2 - but it’s (obviously) higher than for Black Box 1.

---

When you’re encouraging people to test themselves several times a week of course you’re going to get crazy results. If you want to get some (still imperfect) measure of how things are really changing you need to look at the share of the tests that are positive - and not just the raw numbers.

But we have been (needlessly) frightened into an obsession with case numbers. They (whoever they are) clearly want us to be fearful - it’s figuring out why they want us to be this fearful that is not so easy.

Overall, my response to cases and testing these days can be summed up as follows: