Home » Blog »

The hidden side of clinical trials

Watch the AllTrials TEDx talk on YouTube

Learn more

Evidence matters to the public

Join us on 1st November at Parliament to make the case

Learn more

Plant Science Panel

Insecticides, biofuels, GMOs …

Learn more

'The Ugly Truth'

by Tracey Brown, director of Sense About Science

Learn more

Blog


 

July 2014

Who will take responsibility?

The Government has known that the official statistics are wrong since the Royal Statistical Society raised the issue in the wake of the 2009 H1N1 pandemic. Since then it’s been bounced around departments until we and charities who use and fund research that is affected by this had to write to the Prime Minister calling for action.

I have been trying to track down a response since we sent the letter a month ago. In that time, I’ve repeatedly called Number 10, Department of Health, Ministry of Justice and Home Office, each of whom have sent it to someone else for a response at least once. Number 10 has most recently tasked the Ministry of Justice with providing a response and told us Number 10 won’t be able to help us anymore. Today, the Ministry of Justice told me that a draft response has just been sent to the Minister’s office for sign off and we could possibly expect a response by early next week.

Of course, I’ve now been told several times that we could expect a response very soon but one has yet to materialise. I’ll believe it when I see it and in the meantime, the official statistics are still wrong.


The "accuracy" of screening tests

This is a guest post by Professor David Colquhoun, FRS

Anything about Alzheimer’s disease is front line news in the media. No doubt that had not escaped the notice of Kings College London when they issued a press release about a recent study of a test for development of dementia based on blood tests. It was widely hailed in the media as a breakthrough in dementia research. For example, the far from accurate BBC report. The main reason for the inaccuracies are, as so often, the press release. It said:

"They identified a combination of 10 proteins capable of predicting whether individuals with MCI would develop Alzheimer’s disease within a year, with an accuracy of 87 percent"

The original paper says

"Sixteen proteins correlated with disease severity and cognitive decline. Strongest associations were in the MCI group with a panel of 10 proteins predicting progression to AD (accuracy 87%, sensitivity 85% and specificity 88%)."

What matters to the patient is the probability that, if they come out positive when tested, they will actually get dementia. The Guardian quoted Dr James Pickett, head of research at the Alzheimer’s Society, as saying

"These 10 proteins can predict conversion to dementia with less than 90% accuracy, meaning one in 10 people would get an incorrect result."

That statement simply isn't right (or, at least, it's very misleading). The proper way to work out the relevant number has been explained in many places - I did it recently on my blog.

The easiest way to work it out is to make a tree diagram. The diagram is like that on my blog, but with a sensitivity of 85% and a specificity of 88%, as specified in the paper.

In order to work out the number we need, we have to specify the true prevalence of people who will develop dementia, in the population being tested. In the tree diagram, this has been taken as 10%. The diagram shows that, out of 1000 people tested, there are 85 + 108 = 193 with a positive test result. Out ot this 193, rather more than half (108) are false positives, so if you test positive there is a 56% chance that it's a false alarm (108/193 = 0.56). A false discovery rate of 56% is far too high for a good test. This figure of 56% seems to be the basis for a rather good post by NHS Choices with the title “Blood test for Alzheimer’s ‘no better than coin toss’ 

If the prevalence were taken as 5% (a value that's been given for the over-60 age group) that fraction of false alarms would rise to a disastrous 73%.

How are these numbers related to the claim that the test is "87% accurate"? That claim was parroted in most of the media reports, and it is why Dr Pickett said "one in 10 people would get an incorrect result.

The paper itself didn't define "accuracy" anywhere, and I wasn't familiar with the term in this context (though Stephen Senn pointed out that it is mentioned briefly in the Wiikipedia entry for Sensitivity and Specificity). The senior author confirmed that "accuracy" means the total fraction of tests, positive or negative, that give the right result. We see from the tree diagram that, out of 1000 tests, there are 85 correct positive tests and 792 correct negative tests, so the accuracy (with a prevalence of 0.1) is (85 + 792)/1000 = 88%, close to the value that's cited in the paper.

Accuracy, defined in this way, seems to me not to be a useful measure at all. It conflates positive and negative results and they need to be kept separate to understand the problem. Inspection of the tree diagram shows that it can be expressed algebraically as

accuracy = (sensitivity × prevalence) + (specificity × (1 − prevalence))

It is therefore merely a weighted mean of sensitivity and specificity (weighted by the prevalence). With the numbers in this case, it varies from 0.88 (when prevalence = 0) to 0.85 (when prevalence = 1). Thus it will inevitably give a much more flattering view of the test than the false discovery rate.

No doubt, it is too much to expect that a hard-pressed journalist would have time to figure this out, though it isn't clear that they wouldn't have time to contact someone who understands it. It is clear that it should have been explained in the press release. It wasn't.

In fact, reading the paper shows that the test was not being proposed as a screening test for dementia at all. It was proposed as a way to select patients for entry into clinical trials. The population that was being tested was very different from the general population of old people, being patients who come to memory clinics in trials centres (the potential trials population)

How best to select patients for entry into clinical trials is a matter of great interest to people who are running trials. It is of very little interest to the public. So all this confusion could have been avoided if Kings had refrained from issuing a press release at all, for a paper like this.

I guess universities think that PR is more important than accuracy. That's a bad mistake in an age when pretentions get quickly punctured on the web.

There is more discussion of this work in the follow-up on my blog


Hightable with Tracey Brown: Asking for Evidence to make Sense About Science

Hephzi Tagoe, Science Writer at Oxbridge Biotech Roundtable, and past volunteer at Sense About Science, interviewed Tracey Brown about her job and changing public discussion about science.

You have been director of Sense About Science for over 10 years, what is the secret to your success? 

A large part of it is probably that I am, and we are, hard on ourselves, always asking whether we’re really making a difference. I hate complacency. It means I tend to see the mountain in front of us rather than being pleased with what we’ve climbed, which probably drives the team crazy at times, but it gives us a self-critical culture that drives us on.

Continue reading >