www.lesswrong.com/posts/ZvQfcLbcNHYqmvWyo/the-paper-that-killed-deep-learning-th...
1 correction found
The authors show that on both ImageNet and CIFAR-10, explicit regularization methods such as data augmentation or weight decay do not seem to affect the test accuracy of the algorithms very much.
Zhang et al. did not find regularization had only a tiny effect on test accuracy. They reported a large 18-point top-1 ImageNet drop when regularizers were removed, and explicitly said regularization improved CIFAR-10 generalization too.
Full reasoning
This sentence overstates what the paper found.
Zhang et al. do argue that deep nets can still generalize without explicit regularization, but they do not show that regularization barely affects test accuracy.
In the ImageNet results they report a large change: with regularization, Inception v3 reaches 77.84% top-1 test accuracy, while with all regularizers off it falls to 59.80%. That is an 18.04 percentage-point drop, which is not reasonably described as affecting test accuracy only "very much".
For CIFAR-10, the follow-up CACM version summarizing the same experiments states that "Both regularization techniques help to improve the generalization performance". So the paper's point was more limited: explicit regularization is helpful, but it is not the whole explanation for why deep nets generalize at all.
So the inaccurate part is the claim that the paper showed regularization did not affect test accuracy very much on both datasets. The paper instead showed that models can still generalize without explicit regularization, while also showing that regularization materially improves accuracy, especially on ImageNet.
2 sources
- Understanding Deep Learning (Still) Requires Rethinking Generalization – Communications of the ACM
Both regularization techniques help to improve the generalization performance... A 18% top-1 accuracy drop is observed when we turn off all the regularizers. Specifically, the top-1 accuracy without regularization is 59.80%, while random guessing only achieves 0.1% top-1 accuracy on ImageNet. More strikingly, with data augmentation on but other explicit regularizers off, Inception is able to achieve a top-1 accuracy of 72.95%.
- Understanding deep learning requires rethinking generalization (course mirror PDF)
Table 2: The top-1 and top-5 accuracy (in percentage) of the Inception v3 model on the ImageNet ... yes yes yes 92.18 99.21 77.84 93.92 ... no no no 99.53 100.0 59.80 (63.16) 80.38 (84.49).