www.lesswrong.com/posts/GbWwesBnetyiomxEH/many-portions-of-magnifica-humanitas-a...
2 corrections found
Jabarian and Imas indicate that it achieves a false positive rate of approximately 0, while having a true negative rate of under 5%.
This misstates the paper’s metrics. Jabarian and Imas report Pangram had near-zero false positive and false negative rates; a near-zero false positive rate implies a near-100% true negative rate, not under 5%.
Full reasoning
In binary classification, the true negative rate (TNR) is the share of human-written text correctly classified as human, and it equals 1 − false positive rate (FPR). So if a detector’s FPR is approximately zero, its TNR is approximately 100%, not “under 5%.”
The cited Jabarian–Imas paper explicitly says Pangram achieved “a near zero FPR and FNR” on their stimuli. That means the post appears to have confused false negative rate (FNR) with true negative rate (TNR), and then stated the wrong quantity.
So the problem here is not a subtle interpretation dispute: the post names the wrong metric, and the named metric is incompatible with the paper’s reported false-positive result.
2 sources
- Artificial Writing and Automated Detection (Jabarian & Imas, BFI Working Paper 2025-116)
The paper states that ‘Pangram in particular [achieved] a near zero FPR and FNR within our set of stimuli.’ If FPR is near zero, the true negative rate is near 1, not under 5%.
- Artificial Writing and Automated Detection (NBER Working Paper 34223)
The NBER version reports Pangram ‘achieves a zero FPR on longer passages and essentially a zero FPR on medium passages,’ and then separately discusses false negative rate. That contradicts the post’s claim of a ‘true negative rate of under 5%.’
They found that Pangram performed quite well, achieving a literally 0% false positive rate and a true negative rate of around 28%.
This misreads the reported metric. In the cited paper, Pangram’s false positive rate is 0.0000 and its true positive rate is 0.7791; a 0% false positive rate implies a true negative rate of 100%, not 28%.
Full reasoning
The EVALITA 2026 paper reports Pangram’s results in a table with Accuracy, TPR, and FPR. For Pangram, the table gives TPR = 0.7791 and FPR = 0.0000.
That means the post’s “true negative rate of around 28%” is not what the paper reports. If anything, the table’s closest relevant quantity is the true positive rate of 77.91%. And because true negative rate = 1 − false positive rate, a 0.0000 FPR means Pangram’s true negative rate is 100% on that test set, not 28%.
So this sentence appears to confuse the paper’s metrics and substantially misstate the result.
1 source
- DeSegMa-IT at EVALITA 2026: Overview of the "Detection and Segmentation of Machine Generated Text in Italian" Task
Table 3 reports Pangram with Accuracy 0.8893, TPR 0.7791, and FPR 0.0000. The paper also says Pangram was 'the only system with 0 FPR,' indicating it never attributed human-written texts to generative AI on this test set.