www.lesswrong.com/posts/TbrmHshmcQMobveD3/how-can-llm-rl-work-despite-informatio...
2 corrections found
Atari-style agents do not care about on-policyness at all. You can have a replay buffer with traces from whenever and the agent just chugs through them and learns.
Classical Atari RL was not indifferent to off-policy or stale data. Papers on Atari and distributed RL explicitly report that stale replay can hurt performance and that actor/learner lag needs off-policy correction for stable learning.
Full reasoning
This overstates how tolerant “old-school” RL was to off-policy data.
Two concrete counterexamples:
-
Replay staleness can hurt Atari agents. In A Deeper Look at Experience Replay, Zhang and Sutton report that replay-buffer size “needs carefully tuning” and that “a large replay buffer can significantly hurt the performance,” including on Atari games. That directly contradicts the idea that Atari agents can simply learn from “traces from whenever” without caring.
-
Distributed deep RL needed explicit off-policy correction because policy lag mattered. In the IMPALA paper, Espeholt et al. say they achieve stable learning by combining decoupled acting/learning with a novel off-policy correction method called V-trace, and they evaluate on Atari-57. If old-style RL truly “did not care about on-policyness at all,” such correction for stale actor trajectories would not be necessary for stable learning.
So the accurate picture is not that classical RL was completely unconcerned with on/off-policyness, but that some successful older methods were more tolerant of off-policy data than current LLM RL. That is a much weaker claim than the sentence in the post.
2 sources
- A Deeper Look at Experience Replay
"We showcase that a large replay buffer can significantly hurt the performance. ... We showcase its utility in both simple grid world and challenging domains like Atari games."
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
"We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace. ... We demonstrate the effectiveness of IMPALA ... on ... Atari-57."
Technically, this off-policy bias can theoretically be completely corrected by importance sampling, which is what we do in practice, but this creates other issues such as extreme variance and numerical issues from extremely large or small importance weights, as well as mitigations to these such as clipping or dropping tokens with extreme importance weights reintroduces bias.
It is not accurate to say exact importance-sampling correction is simply “what we do in practice” for off-policy RL. Standard off-policy methods such as Q-learning do not use importance sampling in the one-step case, and many practical algorithms explicitly depart from exact IS to control variance.
Full reasoning
The sentence overgeneralizes practical off-policy RL.
- Sutton and Barto’s RL textbook explicitly notes an entire class of off-policy methods without importance sampling: “Q-learning and Expected Sarsa from Chapter 6 do this for the one-step case.” So exact importance-sampling correction is not simply what practitioners do across off-policy RL.
- Modern off-policy RL also often departs from exact IS on purpose. Graves and Ghiassian write that “most off-policy algorithms in the literature depart from per-decision importance sampling in a subtle way,” specifically to reduce variance.
So the first clause is too broad. A more accurate version would be: importance sampling is one important way to correct off-policy bias, but practical off-policy RL also includes major families of algorithms that do not use exact IS, or only use approximate / modified corrections because exact IS is often too high-variance.
2 sources
- Reinforcement Learning: An Introduction (draft)
"Off-policy Learning Without Importance Sampling ... Is off-policy learning possible without importance sampling? Q-learning and Expected Sarsa from Chapter 6 do this for the one-step case..."
- Importance Sampling Placement in Off-Policy Temporal-Difference Methods
"However, the update rules for most off-policy algorithms in the literature depart from per-decision importance sampling in a subtle way; they correct the entire TD error instead of just the TD target."