In the past few years, we have seen an explosion in the use of ‘Deep Learning’ as its software platforms and the supporting hardware mature, especially as GPUs with larger memories become widely available. Even though this is a recent development, ‘Deep Learning’ has entrenched historical roots, tracing back all the way to the sixties or possibly earlier.

By reading-up on its history, we get a better understanding of the current state of the art of ‘Deep Learning algorithms’ and the ‘Neural Networks’ that you build with them.

There is a broad set of papers to read if we want to dive deep into the history. It would take us multiple weekends. Instead, we will be reading an excellent overview paper from 2014 by Jürgen Schmidhuber. Jürgen evaluates the current state of the art in ‘Deep Learning’ by tracing it back to its roots. Ergo, we get excellent historical context.

Enjoy!

“Deep Learning in Neural Networks: An Overview.” Jürgen Schmidhuber, in Neural Networks, Volume 61, January 2015, Pages 85-117 (DOI: 10.1016/j.neunet.2014.09.003)