www.lesswrong.com/posts/4SsFMGaBx59wBA3Ej/can-we-hold-intellectuals-to-similar-p...
1 correction found
Expert Political Judgement provided a fair amount of evidence for just how poorly calibrated all famous and well esteemed intellectuals seem to be.
Tetlock's book did not test "all famous and well esteemed intellectuals." It studied political and economic forecasters specifically, and even within that group it found meaningful variation rather than universal poor calibration.
Full reasoning
This overstates what Expert Political Judgment actually showed.
Tetlock's study was about political/economic forecasting experts and pundits, not about all famous and well‑esteemed intellectuals across domains. A contemporaneous summary of the book says Tetlock selected 284 people who made their living "commenting or offering advice on political and economic trends" and scored their forecasts over time. Tetlock himself later summarized the book's main result as: "the major finding in Expert Political Judgment was that political experts are pretty seriously overconfident."
That is a much narrower claim than the post makes. The evidence concerns a specific class of forecasters in politics/economics, not all public intellectuals in general.
The book also did not show that this entire set was uniformly bad in the same way. Tetlock distinguished better and worse thinkers within the sample: the book's own chapter heading contrasts foxes and hedgehogs, and Tetlock said fox-like experts were better calibrated than hedgehog-like ones. So even within the studied population, the result was not that all such people were equally poorly calibrated.
A more accurate summary would be that the book found many political experts/pundits were overconfident and often weak forecasters, especially some styles of thinker—not that it established poor calibration for all famous intellectuals.
2 sources
- How to Avoid the Perils of Political Forecasting - Knowledge at Wharton
Philip Tetlock: "The major finding in Expert Political Judgment was that political experts are pretty seriously overconfident." He also says political experts are good at some things, but "they're not very good at forecasting."
- Everybody's an Expert | The New Yorker
Menand's summary of Tetlock's book says Tetlock chose 284 people who made their living "commenting or offering advice on political and economic trends" and scored 82,361 forecasts. It also reports that high scorers looked like foxes while low scorers looked like hedgehogs.