thezvi.substack.com/p/gpt-6-astra-can-do-ambitious-things?open=false#%C2%A7our-p...
2 corrections found
It scores 66% on ARC-AGI-3 using our standard harness
ARC Prize’s published Astra results put its standard-harness ARC-AGI-3 score at 62.7%, not 66%.
Full reasoning
ARC Prize’s own published Astra results contradict the 66% figure.
On the official ARC Prize Astra results page, the best observed ARC-AGI-3 Semi-Private result with the Standard harness is listed as 62.7%, with the verified max-setting score shown more precisely as 62.71%. ARC Prize’s accompanying blog post says the same thing: “With our Standard harness ... Astra (max) scores 62.7% on ARC-AGI-3 Semi-Private.”
So the claim that Astra scores 66% on ARC-AGI-3 using ARC Prize’s standard harness does not match ARC Prize’s own published benchmark result.
2 sources
- GPT-6 Astra - ARC-AGI Results
GPT-6 Astra's best observed ARC-AGI-3 Semi-Private result with the Standard harness ... was 62.7% ... Verified scores ... ARC-AGI-3 (Standard harness) ... Max ... 62.71%.
- OpenAI's GPT-6 Astra on ARC-AGI-3
With our Standard harness, OpenAI's Astra (max) scores 62.7% on ARC-AGI-3 Semi-Private for $26K.
Fable 5.1 was below trend and scored 163, same as Fable 5.
Epoch AI currently lists Claude Fable 5.1 at 164 ECI, while Claude Fable 5 is 163.
Full reasoning
Epoch AI’s published model data contradicts this comparison.
On Epoch AI’s Models index, Claude Fable 5.1 is listed as ECI #2, 164, while Claude Fable 5 is listed separately as ECI #3, 163. Epoch AI’s GPT-6 Astra comparison table also shows Claude Fable 5.1 with an ECI score of 164.
So Fable 5.1 is not listed by Epoch AI as scoring 163, and it is not the same ECI score as Fable 5 on those pages.
2 sources
- Models | Epoch AI
Claude Fable 5.1 Anthropic Sep. 1, 2026 Closed weights ECI #2 164 ... Claude Fable 5 Anthropic Jun. 9, 2026 Closed weights ECI #3 163.
- GPT-6 Astra | Epoch AI
GPT-6 Astra vs select alternatives ... ECI score Best score 167 | 167 | 162 | 164 | 158 ... Claude Fable 5.1 Anthropic Sep. 1, 2026 Closed weights.