ramblingafter.substack.com/p/can-you-make-chatgpt-follow-instructions
1 correction found
They’re not rewarded for process
This blanket claim is too broad: major LLM developers explicitly train some models on intermediate reasoning or self-critique processes, not just final outputs.
Full reasoning
The sentence states, categorically, that LLMs are not rewarded for process. That is contradicted by public documentation from major model developers.
OpenAI has explicitly described training models with process supervision, i.e. rewarding intermediate reasoning steps rather than only the final answer. In OpenAI's own words, it trained a model by "rewarding each correct step of reasoning" instead of only rewarding the outcome.
Anthropic has likewise publicly described training Claude with a structured internal process: during one phase, the model is trained to critique and revise its own responses using constitutional principles, and in a later phase it is further trained via reinforcement learning using AI-generated feedback. That is a form of process-oriented training, not merely scoring the final surface text.
Because at least some prominent LLMs are in fact trained with explicit process-oriented signals, the article's blanket statement that LLMs "are not rewarded for process" is factually incorrect as written. A narrower claim such as "many deployments still optimize mostly for outputs" would be more defensible, but the absolute statement is contradicted by the cited sources.
3 sources
- Improving mathematical reasoning with process supervision | OpenAI
We’ve trained a model to achieve a new state-of-the-art in mathematical problem solving by rewarding each correct step of reasoning (“process supervision”) instead of simply rewarding the correct final answer (“outcome supervision”).
- Learning to reason with LLMs | OpenAI
Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process.
- Claude's constitution | Anthropic
During the first phase, the model is trained to critique and revise its own responses using the set of principles and a few examples of the process. During the second phase, a model is trained via reinforcement learning ... using AI-generated feedback based on the set of principles.