en.wikipedia.org/wiki/Rectified_linear_unit
3 corrections found
Early Transformer scaling giants like GPT-3 (2020) and Falcon-180B (2023) relied on the rectified linear unit function explicitly
This is incorrect: GPT-3 and Falcon use GELU-family activations in their feed-forward layers, not plain ReLU.
Full reasoning
The claim says GPT-3 and Falcon-180B used ReLU explicitly. The available technical sources point the other way.
- The GPT-3 paper states that GPT-3 uses the same model and architecture as GPT-2.
- OpenAI's GPT-2 implementation defines a
gelufunction and applies it inside the MLP block (h = gelu(...)). So if GPT-3 keeps GPT-2's architecture, its feed-forward activation is GELU rather than plain ReLU. - For Falcon, the released Falcon configuration in the Transformers implementation specifies the feed-forward
activationas"gelu", not ReLU.
So the statement is wrong for both examples it names: GPT-3 and Falcon-180B are not examples of transformer scaling models that "relied on the rectified linear unit function explicitly."
3 sources
- Language Models are Few-Shot Learners
We use the same model and architecture as GPT-2 [RWC+19], including the modified initialization, pre-normalization, ... last being the model we call GPT-3.
- OpenAI GPT-2 model.py
def gelu(x): return 0.5*x*(1+tf.tanh(np.sqrt(2/np.pi)*(x+0.044715*tf.pow(x, 3)))) ... def mlp(x, scope, n_state, *, hparams): ... h = gelu(conv1d(x, 'c_fc', n_state))
- Falcon configuration in Transformers
activation (`str`, *optional*, defaults to `"gelu"`): The activation function used in the feedforward layer. ... activation: str | None = "gelu"
These variants were used to improve training stability while fundamentally preserving the rectified principle of zeroing low responses.
This overstates what smooth activations like GELU and Swish preserve. They do not keep ReLU’s hard-zeroing behavior for low or negative inputs.
Full reasoning
The sentence says smoother variants such as GELU or SwiGLU still preserve the ReLU-style principle of zeroing low responses. That is not how these activations are defined.
- The GELU paper defines GELU as
x Φ(x)and explicitly contrasts it with ReLU by saying GELU weights inputs by their value, rather than gates inputs by their sign as in ReLUs. - The Swish paper defines Swish as
x · sigmoid(βx). Becausesigmoid(βx)is strictly positive for finitex, Swish does not hard-zero negative or small inputs the way ReLU does.
So smooth activations like GELU/Swish are not just ReLU with nicer optimization properties; mathematically, they replace ReLU's exact hard-thresholding with smooth weighting. Saying they "fundamentally preserve" zeroing low responses is inaccurate.
2 sources
- Gaussian Error Linear Units (GELUs)
The GELU activation function is xΦ(x) ... The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs (x1_{x>0}).
- Searching for Activation Functions
Our experiments show that the best discovered activation function, f(x) = x · sigmoid(βx), which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets.
Parametric ReLU (PReLU, 2016) takes this idea further by making α a learnable parameter along with the other network parameters.
The year is incorrect. PReLU was introduced in a 2015 paper, not 2016.
Full reasoning
PReLU was introduced by He, Zhang, Ren, and Sun in 2015 in the ICCV paper "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." The paper explicitly says it proposes a Parametric Rectified Linear Unit (PReLU), and the proceedings entry is dated 2015.
So the article's "PReLU, 2016" label misdates the method.
1 source
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015 ... First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit.