x.com/iosif_lazaridis/status/2022754624042631579
1 correction found
Nick Bostrom argues that we should choose superintelligence, as the upside (it gives us radical life extension) overwhelms the downside (it potentially kills us).
This overstates Bostrom's position. His paper explicitly analyzes a narrow person-affecting framework, says other important factors were left out, and he later said he does not endorse that framework all-things-considered.
Full reasoning
Bostrom's paper does not straightforwardly argue, without qualification, that "we should choose superintelligence" because life extension outweighs extinction risk.
What the paper actually says is narrower:
- The abstract says the analysis is conducted "from a person-affecting stance" and sets aside other considerations.
- In the LessWrong version, Bostrom adds a prefatory note saying "other important factors, not covered here, ... would need to be taken into account if one wishes to determine which timeline would be best all things considered."
- Bostrom then clarified in discussion that he "would not argue that this is the correct stance or the one I would all-things-considered endorse."
So the post's unqualified attribution is misleading in two ways:
- It presents a conditional, framework-limited analysis as Bostrom's all-things-considered recommendation.
- It treats life extension as a definite payoff ("it gives us radical life extension"), whereas the paper describes that as a conditional possibility if aligned superintelligence goes well. The paper says superintelligence "could" accelerate medicine and that "if aligned superintelligence would enable major life extension" then high catastrophe probabilities can be worth accepting in that model.
A more accurate summary would be that Bostrom argues that within a specific person-affecting model, the possible life-extension benefits of aligned superintelligence can justify accepting high risks — not that he simply argues, full stop, that we should choose superintelligence because it gives us radical life extension.
3 sources
- Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Sorry about the lengthiness of this post... other important factors, not covered here, that would need to be taken into account if one wishes to determine which timeline would be best all things considered. ... We examine optimal timing from a person-affecting stance ...
- Nick Bostrom comments on Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Yes the post explicitly considers things only from a mundane person-affecting stance, and I would not argue that this is the correct stance or the one I would all-things-considered endorse.
- Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Superintelligence would be able to enormously accelerate advances in biology and medicine ... developing powerful anti-aging and rejuvenation therapies ... A simple go/no-go model illustrated how, if aligned superintelligence would enable major life extension and quality-of-life improvements, then even very high levels of [risk] can be worth incurring.