en.wikipedia.org/wiki/Alex_Krizhevsky
1 correction found
Previously neural networks were trained on CPUs.
This is incorrect because neural networks were being trained on GPUs before AlexNet. Published papers from 2009 and 2011 describe training deep belief networks and deep neural nets on graphics processors.
Full reasoning
The sentence says that before AlexNet, neural networks were trained on CPUs. That is contradicted by published research before 2012 showing GPU-based neural-network training.
- In 2009, Raina, Madhavan, and Ng published Large-scale Deep Unsupervised Learning using Graphics Processors, which explicitly says they developed massively parallel methods for deep unsupervised learning on graphics processors and that their DBN implementation was much faster than a CPU implementation.
- In 2011, Cireșan et al. published Handwritten Digit Recognition with a Committee of Deep Neural Nets on GPUs, whose abstract says they used graphics cards to greatly speed up training of deep neural nets.
So while CPU training was common, it is false to say neural networks were previously trained on CPUs full stop. GPU training was already in use before AlexNet.
2 sources
- Large-scale Deep Unsupervised Learning using Graphics Processors
In this paper, we suggest massively parallel methods to help resolve these problems... We argue that modern graphics processors far surpass the computational capabilities of multicore CPUs... Our implementation of DBN learning is up to 70 times faster than a dual-core CPU implementation.
- Handwritten Digit Recognition with a Committee of Deep Neural Nets on GPUs
Submitted on 23 Mar 2011... Abstract: Recently we were able to significantly improve this result, using graphics cards to greatly speed up training of simple but deep MLPs, which achieved 0.35%.