Type 1 & 2 errors have been hammered home since our days of high school statistics — Type 1 being a false positive, or incorrectly rejecting the null hypothesis, for example, convicting an innocent person or diagnosing cancer in someone who doesn’t have it and Type 2 being a false negative, or incorrectly failing to reject the null hypothesis, for example, exonerating a guilty person or failing to diagnose cancer in a person that has it. We regularly consider and weigh them.
But, it’s Type 3 error — the right answer to the wrong question — that deserves more spotlight, though it often plays third fiddle. It has alternative definitions, including rejecting the wrong hypothesis, correctly rejecting the null hypothesis for the wrong reason, inferring the incorrect alternative hypothesis, and being wrong about the direction of the difference between two groups (resolved with a 2-tailed test). Here, we’ll use the general definition of the right answer to the wrong question, for example, prescribing an effective drug for the wrong disease, efficiently downsizing when staffing isn’t the real problem, or generating an effective marketing campaign when the real problem lies in product-market fit. Type 3 errors lead us down rabbit holes, wasting time, money, and resources.
In the case of covid diagnostics, they can also be deadly. Michael Mina et al published a great article titled ‘Rethinking Covid-19 Test Sensitivity– a Strategy for Containment’ in the NEJM. In the U.S., covid has been incorrectly framed as a clinical problem. The FDA, CDC, and NIH have focused on a test’s analytic sensitivity, or lower limit of detection. By doing so, they’ve been asking the wrong question: how can we detect SARS-CoV-2 in a sample to accurately diagnose covid?
Covid isn’t a clinical problem. It’s a public health problem. The right questions are how can infections be detected in a population? How can we limit the spread of covid? As Mina states, focusing on the right problems would help create a ‘covid filter’, identifying and isolating infected, especially asymptomatic, people. Clinicians and labs are considering tests outside of their context of who’s being tested, when in the course of an infection they’re tested, how often they’re tested, and when they receive results. The metric that matters isn’t test sensitivity, but the sensitivity of a testing regimen. This is nothing new to clinicians– they weigh the effectiveness of a treatment regimen, not just a single dose.
High-sensitivity PCR tests are successful at providing a more definitive clinical diagnosis in a single test for symptomatic patients. But, when it comes to returning quick results repeatedly in order to reduce asymptomatic spread, they fail. A key problem arises from SARS-CoV-2’s long tail of RNA positivity and the fact that transmission occurs at peak viral load a few days after exposure.
A covid test should be used at the beginning of infection to limit spread. However, PCR results can often be delayed by 1-3 days, with testing requiring central lab transport and trained operators (costly). A high-sensitivity test result is moot if a positive patient spreads the infection for days before they know. And, due to the long tail of RNA positivity, most people who do test positive are no longer infectious. Moreover, they’re being sent into 10-day quarantines after they’ve passed the transmissible stage and may have already spread the infection.
Defining the problem as population spread with the aim of limiting spread rather than focusing on clinical diagnosis, we arrive at a better solution to complement current PCR testing. A national surveillance regimen of high-frequency (multiple times per week), lower-sensitivity, low-cost rapid lateral-flow antigen testing (e.g., Abbott’s BinaxNOW, SD Biosensor’s Standard Q) similar to that of South Korea (and ideally at-home testing) would be more effective at stopping transmission chains. While any single rapid test will miss an infection due to its lower sensitivity (no amplification step), many of them used nationally at high-frequency would catch more infections in the population when it matters.
The U.S.’s failed covid testing strategy underscores the importance of Type 3 error. We should weigh and avoid it. Framing the right problems. Identifying the right questions. Thinking through how our models and products are used, who’s using them, why they’re using them, when they’re using them, how often they’re using them, their limitations and measures of success, and their 2nd and 3rd order consequences. It would save time, money, resources, and lives.
References
https://www.nejm.org/doi/full/10.1056/NEJMp2025631
http://onbiostatistics.blogspot.com/2016/10/type-iii-or-type-3-error.html
https://www.statisticshowto.com/type-iii-error-in-statistical-tests
