Downloaded 35 times
![Layer-wise organization
Most networks are fully-connected
Not counting Input layer (3-layered on picture)
Output layer - no activation function
4 + 4 + 1 = 9 neurons
[3 x 4] + [4 x 4] + [4 x 1]
= 12 + 16 + 4 = 32 weights
4 + 4 + 1 = 9 biases
∑ = 41 learnable params
Modern NN ~100 million parameters with ~10-20 layers
Example: Visual Geometry Group Network (Oxford) have 19 layers and 138 Millions parameters to learn](https://image.slidesharecdn.com/mldeep-161020093742/75/Introduction-to-Deep-Learning-6-2048.jpg)
More Related Content
Similar to Introduction to Deep Learning
Introduction to Deep Learning
- 1. Introduction to Deep Learning (c)Oleg Mygryn 2016
- 2. Neuron Model 1899 -Discovered. Santiago Ramón Neuron Input 1 Input 2 Input 3 weight 2weight 1 weight 3 Activation Function Output 1958 - Perceptron. Frank Rosenblatt 1984 - Backpropagation optimization 2010 - Recurrent and Deep FF nets 2012+ ….
- 3. ImageNet Challenge 150,000 imagesand 1000 object classes top 5 suggestions, error rate, % Deep Learning and last 10 years - AlexNet (2012) exploded industry. 5 layers - ZF Net (2013) - 5 layers improved AlexNet - GoogLeNet (2015) - 22 layers - VGG Net (2014) - Oxford 19, layers In 2016: - NVIDIA DGX-1 system. ~170TFlops!!! - Intel® Xeon Phi™ 7210. ~ 3 TFlops
- 4. Top AI scientists GeoffreyE. Hinton University of Toronto, AlexNet curator, researcher Andrew NgYann LeCun Facebook, AI research group, working on AI since 1998 Chief Scientist of Baidu, Co-Founder Coursera, Professor at Stanford University
- 5. Modeling Neuron Human body: ~86 billion neurones ~ 100 trillons synapses Activation functions: - sigmoid; - tanh; - ReLU;
- 6. Layer-wise organization Most networksare fully-connected Not counting Input layer (3-layered on picture) Output layer - no activation function 4 + 4 + 1 = 9 neurons [3 x 4] + [4 x 4] + [4 x 1] = 12 + 16 + 4 = 32 weights 4 + 4 + 1 = 9 biases ∑ = 41 learnable params Modern NN ~100 million parameters with ~10-20 layers Example: Visual Geometry Group Network (Oxford) have 19 layers and 138 Millions parameters to learn
- 7.
- 8. ReLU Rocks ! ReLUs(solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line) by Alex Krizhevsky
- 9.
- 10.
- 11.
- 12.
- 13.
- 14. AlexNet. Image recognitionsamples
- 15. https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf AlexNet (2012): Illustrationof the Architecture “Train time on GTX 580 3GB GPUs 5-6 days. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available” (c) Alex Krizhevsky
- 17. GoogleNet (2015): Inception 9Inception modules in the whole architecture, with over 100 layers in total
- 18. GoogleNet (2015): Inception 9Inception modules in the whole architecture, with over 100 layers in total
- 19.
- 20.
- 22. @ Google tensorflow.org launchedin Nov 2015. - most popular ML library - GitHub: 35,000 stars 15,000 forks - 350 contributors
- 23. TensorFlow - Python API -C++ API (poorly documented) - Java API ??? (TBA in 201X) API Features - CPU or multiple CPU, GPU or multiple GPU; - Async computation with lazy loading of execution graph; - Many of algorithm have already implemented; - TensorBoard: graph execution visualisation + debugging; import tensorflow as tf hello = tf.constant('Hello, TensorWorld!') sess = tf.Session() print sess.run(hello)
- 24. Thank YOU 1. www.cs231n.github.io 2.www.tensorflow.org 3. www.playground.tensorflow.org 4. www.yann.lecun.com 5. www.kdnuggets.com 6. www.devblogs.nvidia.com