www.lesswrong.com/posts/wt4kk6vFPEhkXvF8Q/how-good-are-slop-vestigators
1 correction found
We evaluate twelve models with time budgets of 10 minutes, 30 minutes and 2 hours, and three independent runs for each time budget.
The published results are not a complete 12×3 matrix with three runs in every cell. The repo’s own result files show at least one model missing a 120-minute cell entirely, and two 120-minute cells with only two runs.
Full reasoning
The sentence states that every evaluated model had results at 10, 30, and 120 minutes, with three independent runs for each time budget. But the benchmark’s published result files show otherwise.
In benchmark/figures/combined_score.csv, Haiku 4.5 has entries only for 10 and 30 minutes; there is no 120-minute row for that model. The same file also shows Kimi K3 at 120 minutes with n_runs = 2, and Sonnet 5 at 120 minutes with n_runs = 2, not 3.
The repository README also explicitly warns that the figures only generally show three runs per model/budget and that missing samples are recorded in the figure data. So the published evaluation is not actually "three independent runs for each time budget" across all twelve models.
This does not undermine the whole benchmark, but the sentence overstates the completeness of the reported run matrix.
2 sources
- messageboardauditbench/benchmark/figures/combined_score.csv
"Kimi K3",moonshot,157.62,True,120,2,0,... ; "Sonnet 5",anthropic,156.16,True,120,2,0,... ; "Haiku 4.5",anthropic,142.41,True,10,3,0,... ; "Haiku 4.5",anthropic,142.41,True,30,3,0,... (no 120-minute Haiku row appears)
- messageboardauditbench README
The figures generally show three runs per model and budget. Missing samples and model fallbacks are recorded in the figure data and report indexes; do not treat all archived runs as headline samples or an Opus fallback as a single-model result.