A brain scan that detects autism in adults could mean much more straightforward diagnosis of the condition, scientists say. Reported the BBC, Sky the Guardian and many more.

I had great difficulty getting hold of this paper, it wasn’t published online at the time of the press release. I managed to get a copy via Ben Goldacre at Bad Science and Evidence Matters who sent me the full text. Given this problem in getting the paper, it is highly likely no one who released the story has actually read the paper.

The news all report the headline ‘The researchers detected autism with over 90% accuracy, the Journal of Neuroscience reports.’

Sounds impressive, but this is one of the most obvious mistakes to make in interpreting a diagnostic test result. Never mind this is not the correct study type.

What has happened is the sensitivity has been taken for the positive predictive value, which is what you want to know: if I have a positive test do I have the disease?

Sensitivity: The proportion of people with disease who have a positive test.
Positive predictive value (+PV): The proportion of people with a positive test who have disease.

So, for a prevalence of 1% the actual positive predictive value is 4.5%. That is about 5 in every 100 with a positive test would have autism. Even at a prevalence of 2%, only 8.5% would be correctly identified.

Suddenly, not that great a test. This has to be one of the worst examples of misinterpreting diagnostic test results in the media I’ve ever seen.

When I started in evidence-based medicine, it was a big shock that probably the most under-researched area of health is how health practitioners should diagnose illness; i.e. “diagnostic strategies”. Individual studies and systematic reviews have focused on drugs and interventions, but it is now recognised that such reviews are also necessary to evaluate diagnostic tests. In children, clinical signs (e.g. a raised temperature) or test results are even more important than in adults, because the child may not be able to describe his or her symptoms, particularly when they are very unwell.

In the Lancet “online first” this week, a systematic review studied the value of rapid breathing, poor peripheral circulation and other factors in confirming or excluding the possibility of serious infection in children presenting to general practice or other outpatient care. After looking through nearly 2000 potentially relevant articles, the authors included 30 studies in their analysis. They collated information from the individual studies to calculate “likelihood ratios” for each factor or test; i.e. how likely a disease is after a test result. A likelihood ratio of >1 indicates the test result is associated with the disease. A likelihood ratio <1 indicates that the result is associated with absence of the disease. Clinical features with a positive likelihood ratio of more than 5 were termed “red flags” (i.e. warning signs for serious infection).

Cyanosis, or a blue colouration of the skin, led to a positive likelihood ratio range of 3–52, whilst the same ranges for rapid breathing and petechial rash (the non-blanching rash parents are taught to look for if we are worried about meningitis) were 1–10 and 6–84 respectively. Put another way, a petechial rash increases the odds of serious infection by between 6-fold and 84-fold. The likelihood of the disease is influenced by the prevalence of the disease in the population. For example, in a setting where meningitis or infection is very rare, the likelihood of meningitis if you have a petechial rash is increased. No single clinical sign could rule-out serious infection but some combinations could be used to exclude the possibility of serious infection. For example, pneumonia is very unlikely if the child is not short of breath and there is no parental concern.

It is amazing that only one study was set in primary care, where such information would change practice most since GPs often have to make decisions based on simple clinical signs. That study highlighted two red flags that are rarely evaluated: parental concern and doctor’s instinct. Parental concern gave a positive likelihood ratio of 14 and clinician instinct increased the likelihood of serious infection by 24 times. This study only looked at developed country research, and not research from developing countries, but the authors concluded, “Most of the red flags already recommended by WHO for use in developing countries can be used in the initial assessment of children presenting to ambulatory care settings in developed countries.” The next step is “to identify the level of risk at which clinical action should be taken”, based on these red flag tests.

So much of modern medicine is about tests and making diagnoses on the basis of the results, that old school doctors often lament the death of the stethoscope and the traditional clinical skills of the physician. Not only are patients entering hospitals and general practices immediately hit by a battery of X-rays, blood tests, scans and other specialised tests; many tests are available for home use by the patients themselves, e.g. home glucose monitoring, home ultrasound probes for antenatal scans and electronic blood pressure meters. Both patients and doctors often make the mistake of assuming that a test is 100% trustworthy and accurate, but we should always ask how good a test is at picking up OR ruling out what it is meant to. The result from a test is only as good as the test itself, and the person using the test. There have been warnings against the use of home foetal heart monitors this week because the inexperience of parents makes the test less reliable and unsafe.

A positive test result can label somebody with diabetes, cancer and all kinds of other illnesses which have many implications of a person’s life. Therefore, we need to know how good a test is at picking up the people with the disease. The “sensitivity” of a test looks at the proportion of diseased individuals that will have a positive test. That is to say, there will be some people with disease who will get a “false negative”. A negative test result can give somebody reassurance that they do not have a disease, but if the test is unreliable, this may be false reassurance and may lead to the psychological trauma and adverse health effects of a later diagnosis. The “specificity” of a test looks at the proportion of individuals without a disease that will have a negative test. In a test for screening (for example, for colorectal cancer), we want to be confident that we are ruling out the disease; i.e. the test must be very specific. In other settings, picking up a positive might be more important, such as the simple urine dipstick test, which is 90% sensitive for urinary tract infection, but has a specificity of only 60%.

Once we have a positive result, how likely is the patient to have the disease in question? This is called the “positive predictive value” or PPV and tells us what proportion of people with a positive test have the disease (do not confuse with sensitivity). Unfortunately, the PPV is affected by how common a disease is in the population. If there is a high prevalence in the population, then the predictive value will be high, but if the disease is uncommon, the PPV will be low.

If you were paying attention during lesson 2, you will realise that neither sensitivity nor specificity of the monofilament are exact values; they will lie within ranges of values. The monofilament is a special tool, used to test whether diabetics have lost sensation in their feet. A review of all relevant studies showed that sensitivity ranged from 41% to 93% and specificity ranged from 68% to 100%. So next time you ask what the diagnosis is, ask how good the test is first.

Lesson 4