x.com/ihtesham2005/status/2051296304740069635
1 correction found
Modern transformer models use self-attention, which is technically a mechanism where a network attends to its own internal states across layers.
This misdescribes self-attention. In the original Transformer paper, self-attention relates different positions in a sequence, not a network’s internal states across layers.
Full reasoning
The original Transformer paper defines self-attention as "an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence." It further explains that in an encoder self-attention layer, the queries, keys, and values all come from the same place—the output of the previous layer—and that each position attends to all positions in that sequence.
So the key operation is within-sequence / across positions (tokens), not "across layers." Layers are stacked in a Transformer, and one layer's output becomes the next layer's input, but that is different from saying self-attention itself is a mechanism for attending across layers. The paper distinguishes self-attention from encoder-decoder attention, where queries come from the decoder and keys/values come from the encoder.
In short: transformers do use self-attention, but the technical description here is wrong. Self-attention is primarily about computing relationships among positions in a sequence representation, not about a network attending to its internal states across layers.
1 source
- Attention Is All You Need
"Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence." The paper also states: "The encoder contains self-attention layers. In a self-attention layer all of the keys, values and queries come from the same place ... Each position in the encoder can attend to all positions in the previous layer of the encoder."