x.com/HowToAI_/status/2054611399792644386
2 corrections found
NL proves that a model is actually a set of thousands of smaller, "nested" optimization problems running in parallel.
The paper presents Nested Learning as a new paradigm or perspective, not a proof that all models literally are thousands of parallel optimizers.
Full reasoning
The official sources use much more qualified language than this post.
- Google's blog says Nested Learning is "a new approach to machine learning that views models as a set of smaller, nested optimization problems" and later says "We argue" that architecture and optimization are the same concepts at different levels.
- The paper abstract likewise says "we present a new learning paradigm" and that NL "coherently represents" a model as nested, multi-level, and/or parallel optimization problems.
So the post overstates the paper in two ways:
- "proves" is wrong. The authors frame NL as a paradigm, lens, or argument about how to interpret models—not as a proof of a literal fact about all models.
- "thousands ... running in parallel" is invented. The paper does not state that models consist of "thousands" of sub-problems, and its own wording is broader: nested, multi-level, and/or parallel.
That distinction matters because readers could come away thinking the paper formally established a universal structural fact, when the authors actually present a theoretical framing and a proposed design philosophy.
2 sources
- Introducing Nested Learning: A new ML paradigm for continual learning
"We introduce Nested Learning, a new approach to machine learning that views models as a set of smaller, nested optimization problems..." The post also says, "We argue that the model's architecture and the rules used to train it ... are fundamentally the same concepts."
- Nested Learning: The Illusion of Deep Learning Architectures
The abstract says, "we present a new learning paradigm, called Nested Learning (NL), that coherently represents a machine learning model with a set of nested, multi-level, and/or parallel optimization problems."
The new architecture (HOPE) demonstrated 100% stability in long-context memory and "post-training adaptation" that was previously impossible.
The paper does not report perfect long-context stability, and it explicitly says existing LLMs already had an adaptable in-context-learning mechanism before Hope.
Full reasoning
This claim contains two errors.
1) The paper does not show "100% stability in long-context memory"
The official Google blog says Hope shows "better long-context memory management" and the paper says Hope shows "promising results". Neither source claims perfect stability.
More importantly, the paper's own results are clearly below 100% on long-context tasks. In the PDF, the authors write that long-context results are in Table 1, where Hope is best among attention-free models—but not perfect. For example, the table includes harder 16K Needle-in-a-Haystack settings with Hope scores such as 78.2 and 24.8, which directly contradict a claim of 100% stability.
2) "post-training adaptation" was not previously impossible
The same paper explicitly says existing LLMs already had an adaptable component: in-context learning. In the introduction, the authors write that "The only adaptable component of LLMs is their in-context learning ability ... [which] enables fast adaption to the context and so perform zero- or few-shot tasks."
So Hope may improve adaptation and continual learning, but it is false to say adaptation after pretraining was previously impossible. The authors' own framing is that prior models already adapted in-context; Hope is proposed as a way to improve continual learning beyond that.
3 sources
- Introducing Nested Learning: A new ML paradigm for continual learning
Google says Hope "demonstrates better long-context memory management than existing state-of-the-art models" and later that it "showcases superior memory management"—not 100% stability.
- Nested Learning: The Illusion of Deep Learning Architecture (PDF)
In the long-context section the authors say, "The results are reported in Table 1. Comparing with other attention-free models, Hope achieves the best performance..." Table 1 includes Hope scores below 100 on hard 16K NIAH tasks, including values such as 78.2 and 24.8.
- Nested Learning: The Illusion of Deep Learning Architecture (PDF)
In the introduction, the paper says: "The only adaptable component of LLMs is their in-context learning ability ... that enables fast adaption to the context and so perform zero- or few-shot tasks."