x.com/jyangballin/status/2051677497562210552
1 correction found
Given just an executable and no starter code or internet access.
ProgramBench does not give models only an executable. Its official description says each task includes the executable *and documentation/usage docs*; the benchmark removes source code and internet access, but not all documentation.
Full reasoning
The phrase "Given just an executable" is contradicted by ProgramBench's own description.
- The official ProgramBench homepage says: "Given only a compiled binary and its documentation" and later, "In each task, the agent receives an executable and its documentation."
- BenchLM's ProgramBench summary likewise says agents receive "only a compiled executable and documentation" and that each task starts from "an execute-only binary and usage documentation".
So the benchmark is indeed no starter code and no internet access, but it is not "just an executable"; models also get documentation. That matters, because documentation provides behavioral/interface information beyond the binary alone.
2 sources
- ProgramBench
Given only a compiled binary and its documentation, agents must architect and implement a complete codebase... In each task, the agent receives an executable and its documentation... it cannot use the internet.
- ProgramBench Benchmark 2026: 9 model averages | BenchLM.ai
A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation... Each task starts from an execute-only binary and usage documentation, with no source code, internet, decompilation, or prescribed skeleton.