x.com/liron/status/2098871402799448211
1 correction found
we should only build AI when it's on track to Coherent Extrapolated Volition
This overstates Yudkowsky's post-2005 position. In his own later writing, he says a CEV-based autonomous AGI is probably not what we should try first, and instead discusses a first, task-directed AGI and other minimal approaches.
Full reasoning
Interpreting "Yud" here as Eliezer Yudkowsky, this claim is not an accurate summary of his post-2005 position.
Yudkowsky did propose Coherent Extrapolated Volition (CEV) as an alignment target for a fully autonomous superintelligence. But in later writing he explicitly says that this is not what we should try to build first.
Two especially direct examples:
-
In the LessWrong page "Autonomous AGI" (edited by Yudkowsky, June 7, 2016), he writes that although CEV is a proposed target for an autonomous superintelligence, it is "probably not something we should attempt to do on our first try," and that we should "first target a Task AGI instead." That directly contradicts the blanket claim that he was "very clear" we should only build AI when it is on track to CEV.
-
In MIRI's 2022 post "Six Dimensions of Operational Adequacy in AGI Projects" (a lightly edited copy of a 2017 Yudkowsky document), he describes a model where "the first task-directed AGI" comes first, and later mentions "building a cleaner and more stable AGI to run a CEV" as a subsequent possibility subject to veto. Again, that is not "only build AI when it's on track to CEV"; it is a sequence where a task-directed AGI may precede any CEV system.
A third supporting piece: in "Yudkowsky on AGI ethics" (2017), Rob Bensinger summarizes Yudkowsky's view as: "The first AGI ever built should therefore execute the least dangerous plan" and argues that if a system required developers to solve "meta-ethics or value aggregation" just to safely perform a modest task, that would be a design mistake. That is also at odds with the post's claim, because CEV is exactly a very ambitious value-aggregation target.
So the issue is not that Yudkowsky rejected CEV altogether. It's that the post turns his view into a much stronger claim than he actually made. His later writing distinguishes between:
- CEV as a possible target for a highly trusted autonomous superintelligence, and
- what to build first, where he repeatedly discusses task-directed/minimal AGI instead.
That makes the quoted sentence materially misleading as a statement of his position.
3 sources
- Autonomous AGI — LessWrong
Edited by Eliezer Yudkowsky last updated 7th Jun 2016: "Nonetheless, being really really really that sure ... seems like a large enough problem that perhaps we shouldn't be trying to build this class of AI for our first try, and should first target a Task AGI instead... Coherent extrapolated volition is a proposed alignment target for an autonomous superintelligence, but again, probably not something we should attempt to do on our first try."
- Six Dimensions of Operational Adequacy in AGI Projects — Machine Intelligence Research Institute
MIRI notes this is a lightly edited copy of a 2017 Yudkowsky document. It says: "Adequacy on common good commitment is based on my model wherein the first task-directed AGI..." and later: "any use of AGI powers along the lines of ... building a cleaner and more stable AGI to run a CEV, would be subject to the explicit veto of the command structure operating the first task-directed AGI."
- Yudkowsky on AGI ethics — LessWrong
Rob Bensinger summarizes Yudkowsky's view: "The first AGI ever built should therefore execute the least dangerous plan for preventing immediately following AGIs from destroying the world six months later." The post also says that if a safe system required "a full understanding of meta-ethics or value aggregation," that would be "some kind of catastrophic design mistake."