x.com/jyangballin/status/2051677509486547023/photo/1
1 correction found
Improving performance on ProgramBench means that models can increasingly build software from requirements alone.
That overstates what ProgramBench measures. In ProgramBench, agents do not work from requirements alone: they are given the compiled executable plus documentation, and can probe the executable’s behavior during the task.
Full reasoning
ProgramBench is not a benchmark where models build software from only a requirements document or natural-language spec.
According to the official ProgramBench materials:
- the task gives the agent a compiled binary and its documentation;
- the agent must re-implement the given executable;
- and the agent can run the given program with any input and observe what it does.
That means better ProgramBench scores show improved ability to reconstruct software from an executable + documentation + behavioral probing, not from requirements alone.
This distinction matters because having access to the reference executable is much richer than having only requirements: the executable acts as an oracle for expected behavior, and the benchmark website explicitly says there are no product requirement documents in the task setup.
So the quoted sentence is incorrect because it attributes a stronger implication to ProgramBench than the benchmark actually supports.
3 sources
- ProgramBench
In each task, the agent receives an executable and its documentation, and it must re-implement the given executable.
- ProgramBench
The agent can run the given program with any input and observe exactly what it does, so there's nothing hidden that can't be discovered through experimentation.
- GitHub - facebookresearch/ProgramBench: Can Language Models Rebuild Programs From Scratch?
Given only a compiled binary and its documentation, AI agents must architect and implement a complete codebase that reproduces the original program's behavior.