x.com/paradite_/status/2068371685166977448
1 correction found
the whole point of training is to reduce loss, gain rewards so that the model adhere to rules and ground truth.
This misstates what common LLM training objectives do. Standard language-model loss is not the same as optimizing for objective ground truth, and RLHF rewards human preferences rather than truth itself.
Full reasoning
This claim conflates several different training objectives and treats them as if they all directly enforce "rules and ground truth." The technical literature says otherwise.
- In pretraining, the usual objective is next-token / maximum-likelihood loss. A 2023 Stanford paper on factuality fine-tuning states that "maximum likelihood ... does not always encourage factual predictions". The authors give a concrete example where a factually wrong answer can still receive lower loss depending on the training distribution.
- In RLHF, the reward is not objective ground truth; it is a learned signal based on human preference rankings. The original InstructGPT paper says it collects ranked model outputs and uses those rankings as the reward signal to fine-tune the model. That means the objective is alignment to labeler or user preferences, not truth in any strict mathematical sense.
- There are also papers that explicitly use SFT + RL to optimize novelty in idea generation. So it is not true that training is only about making a model adhere to fixed ground truth; researchers can and do optimize dimensions such as novelty, feasibility, and effectiveness.
Because of this, saying the "whole point of training" is to make a model adhere to rules and ground truth is technically inaccurate. Common LLM training objectives are distribution-matching and preference optimization objectives, and they do not reduce to a direct "ground truth" objective.
3 sources
- Fine-tuning Language Models for Factuality
The paper states that "maximum likelihood, the most common objective for pre-training language models, does not always encourage factual predictions" and that DPO enables fine-tuning "on objectives other than supervised imitation."
- Training language models to follow instructions with human feedback
The InstructGPT paper says it "collect[s] a dataset of rankings of model outputs" and uses reinforcement learning from human feedback; the reward signal comes from human preferences and aims to align the model with "user intent," not objective ground truth.
- LDC: Learning to Generate Research Idea with Dynamic Control
This paper proposes a two-stage approach combining Supervised Fine-Tuning (SFT) and controllable Reinforcement Learning (RL), where reward models "evaluate and optimize the model across key dimensions" including novelty, feasibility, and effectiveness.