www.lesswrong.com/posts/GRmvZsHXH4vaijPMv/four-llm-loss-functions-four-flavors-o...
1 correction found
The Bing-Sydney chatbot from 2023 was trained by pure imitative learning (pretraining + SFT, with no RL at all).
This overstates Sydney’s training setup. Microsoft said Bing ran on GPT-4, and OpenAI says GPT-4 was aligned using RLHF, so it was not trained with “no RL at all.”
Full reasoning
Microsoft publicly confirmed that the new Bing was running on GPT-4. OpenAI’s GPT-4 documentation, in turn, says GPT-4’s behavior was fine-tuned using reinforcement learning with human feedback (RLHF).
So even if Microsoft also added search-specific customization or supervised fine-tuning, the blanket claim that Bing-Sydney was trained with “no RL at all” is contradicted by the official descriptions of the model stack:
- Microsoft: Bing was running on GPT-4.
- OpenAI: GPT-4 was aligned via RLHF.
That means Sydney was not purely a pretraining+SFT system with zero RL anywhere in the model’s training pipeline.
2 sources
- Confirmed: the new Bing runs on OpenAI's GPT-4 | Bing Search Blog
"We are happy to confirm that the new Bing is running on GPT-4, which we've customized for search."
- GPT-4 | OpenAI
"To align it with the user’s intent within guardrails, we fine-tune the model’s behavior using reinforcement learning with human feedback (RLHF)."