BigDL Deep Learning in Apache Spark - AWS re:invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Large Scale Deep Learning
with BigDL

Large Scale Deep Learning
with BigDL
T i m F o x | B i g D a t a a n d M a c h i n e L e a r n i n g C o n s u l t a n t | E l e p h a n t S c a l e

ABOUT ME
Ti m F o x ,
P r i n c i p a l @ E l e p h a n t S c a l e
P r a c t i t i o n e r a n d Tr a i n e r i n D a t a E n g i n e e r i n g
a n d D a t a S c i e n c e
Author of “Data Science in Python” on LinkedIn Learning
t i m @ e l e p h a n t s c a l e . c o m
L i n k e d i n : t i m - f o x - 0 0 6 3 5 4 1

ABOUT Elephant Scale
• Tr a i n i n g i n B i g D a t a a n d A I t e c h n o l o g i e s
• B i g D a t a : S p a r k , H a d o o p , C l o u d , N o S Q L , S t r e a m i n g
• A I : M a c h i n e L e a r n i n g , D e e p L e a r n i n g , B i g D L ,
Te n s o r f l o w
• B i g D L t r a i n i n g a v a i l a b l e !
• P u b l i c a n d P r i v a t e t r a i n i n g s a v a i l a b l e
• B i g D L S a n d b o x : e l e p h a n t s c a l e . c o m / s a n d b o x
E l e p h a n t s c a l e . c o m
i n f o @ e l e p h a n t s c a l e . c o m

Quick Roundup of AI / Machine
Learning / Deep Learning

AI / MACHINE LEARNING / DEEP LEARNING
Artificial Intelligence (AI):
Broader concept of machines being able to carry
out 'smart' tasks
Machine Learning:
A type of AI that allows software to learn from
data without explicitly programmed
Deep Learning:
Using Neural Networks to solve some hard
problems
A r t i f i c i a l
I n t e l l i g e n c e
M a c h i n e
L e a r n i n g
D e e p
L e a r n i n g

DEEP LEARNING APPLICATIONS
S e l f D r i v i n g C a r s
• ML system using image recognition
• Where the edge of the road / road sign / car in front
F a c e r e c o g n i t i o n
• Facebook images
• System learns from images manually tagged and then automatically detects faces in
uploaded photos

DEEP LEARNING HISTORY
E a r l y a t t e m p t s a t D e e p L e a r n i n g d i d n o t s u c c e e d .
• Compute Power was insufficient for the time.
• Training Datasets were insufficiently sized for good results.
• We lacked the ability to parallelize our work.
I n t h e m o d e r n e r a , D e e p L e a r n i n g h a s b e e n s u c c e s s f u l .
• 'Big Data' – now we have so much data to train our models
• 'Big Data ecosystem' – excellent big data platforms (Hadoop, Spark, NoSQL)
are available as open source
• 'Big Compute' - cloud platforms significantly lowered the barrier to massive
compute power
• $1 buys you 16 core + 128 G + 10 Gigabit machine for 1 hr on AWS!
• So running a 100 node cluster for 5 hrs  $500

AI SOFTWARE ECO SYSTEM
M a c h i n e L e a r n i n g D e e p L e a r n i n g
Java
- Weka
- Mahout
- DeepLearning4J
Python
- SciKit
- Tensorflow
- Theano
- Caffe
R
- Many libraries - Deepnet
- Darch
Distributed
- H20
- Spark
- H20
- Spark
- BigDL
Cloud - AWS - AWS

MACHINE LEARNING AND BIG DATA
Until recently most of the machine learning is done on “single computer” (with lots of
memory–100s of GBs)
Most R/Python/Java libraries are “single node based”
Now Big Data tools make it possible to run machine learning algorithms at massive scale –
distributed across a cluster

MODERN DEEP LEARNING FRAMEWORKS

TOOLS FOR SCALABLE MACHINE LEARNING
A p a c h e S p a r k M L
• Runs on top of popular Spark framework
• Massively scalable
• Can use memory (caching) effectively for
iterative algorithms
• Language support: Scala, Java, Python, R
B i g D L
• Built for Apache Spark and Optimized for Intel Xeon
• Language Support: Scala, Java, Python
Te n s o r F l o w
• Based on “data flow graphs”
• Language support: Python, C++
• https://www.tensorflow.org/

TOOLS FOR SCALABLE CLOUD
MACHINE LEARNING
A m a z o n M a c h i n e L e a r n i n g
• Ready to go algorithms
• Visualization tools
• Wizards to guide
• Scalable on Amazon Cloud

BigDL

WHAT IS BIGDL
A d i s t r i b u t e d d e e p l e a r n i n g l i b r a r y f o r A p a c h e S p a r k
F e a t u r e p a r i t y wi t h p o p u l a r d e e p l e a r n i n g f r a m e wo r k s
• Caffe, Torch, Tensorflow
H i g h P e r f o r m a n c e
• Powered by Intel Math Kernel Library (MKL) and multi threaded programming
C a n s c a l e t o h u g e d a t a s e t s
• Using Apache Spark for scale
O p e n s o u r c e ! ( D e c 2 0 1 6 )
A c t i v e D e v e l o p m e n t

PRODUCTION ML/DL SYSTEMS ARE COMPLEX!
A c t u a l M L / D L i s o n l y s m a l l p o r t i o n o f m a s s i v e p r o d u c t i o n
s y s t e m
B i g D L r u n n i n g o n a s c a l a b l e p l a t f o r m l i k e S p a r k h e l p s
s i m p l i f y t h e c o m p l e x i t y

BIGDL FILLS THE 'GAP' IN BIG DATA +
DEEP LEARNING
F o l l o ws p r o v e n d e s i g n p a t t e r n s f o r d e a l i n g wi t h B i g D a t a
S e n d s ' c o m p u t e t o d a t a ' r a t h e r t h a n r e a d i n g m a s s i v e d a t a
o v e r n e t wo r k .
U s e s ' d a t a l o c a l i t y ' o f H D F S ( H a d o o p F i l e S y s t e m
U t i l i z e s ' c l u s t e r m a n a g e r s ' l i k e YA R N / M E S O S
• Automatically handles hardware/software failures
• Elasticity and resource sharing in a cluster

BIGDL & SPARK
R u n B i g D L a p p l i c a t i o n s a s S p a r k a p p l i c a t i o n s
S c a l a , J a v a , a n d P y t h o n s u p p o r t
U s e o t h e r S p a r k ' s f e a t u r e s
• In memory compute
• Integrate with Spark ML and Streaming
E a s y d e v e l o p m e n t wi t h J u p y t e r N o t e b o o k

BIGDL VS TENSORFLOW
B i g D L Te n s o r f l o w
Runtime Scala Engine with Python
front-end
C++ Engine with Python front-
end
Hadoop compatibility Can run natively on Spark &
Hadoop
Accesses Hadoop data as a
client only
Distributed Operation Scalable with Apache Spark
for massive scale out of the
box
Does not support massive
distribution out of the box
Runs Tensorflow Models Yes Yes
Acceleration CPU w/MKL CPU/GPU
Summary Excellent for distributing
deep-learning models to
massive scale on big-data.
Great TCO value.
Excellent library for small-
medium scale data, although
GPU hardware costs can be
significant.

BIGDL: BIG COMPUTE PLUS BIG DATA
B i g D L h e l p s u s i n b a l a n c i n g o u r n e e d s
• Big Compute: Fast Linear Algebra, Intel MKL library
• Optimized for Intel Xeon
• Big Data: I/O parallelized to run on many CPUs
B i g D L A l l o ws M a s s i v e S c a l a b i l i t y
• Natively Designed to run on Spark
• Works with Hadoop eco system (via Spark)
Hadoop is THE Big Data platform for on-premise deployments
P l a y s n i c e l y wi t h o t h e r B i g D L f r a m e wo r k s
• Use existing Tensorflow or Caffe at scale in BigDL
• Train new models based on existing TF / Caffe models

BIGDL USE CASES
F r a u d d e t e c t i o n
S e n t i m e n t a n a l y s i s
I m a g e r e c o g n i t i o n
Find more at: https://github.com/intel-analytics/analytics-zoo/

GPUs and CPUs

GPUS (GRAPHICS PROCESSING UNITS)
G P U s h a v e a d d r e s s e d p a s t i s s u e s i n t r a i n i n g p e r f o r m a n c e
• Example: Tensorflow - optimized to run well on GPUs.
C P U i n p a s t n o t v e c t o r i z e d f o r p a r a l l e l c o m p u t e
• Meant that GPUs were much faster for deep learning
M o d e r n I n t e l X e o n C P U s h a v e v e c t o r i z e d l i n e a r a l g e b r a
• Properly optimized, approaches speed of GPUs
• CPUs are now a credible alternative to running on GPUs
• Cost Advantage and Scalability

INTEL MATH KERNEL LIBRARY (MKL)
F e a t u r e s h i g h l y o p t i m i z e d , t h r e a d e d , a n d v e c t o r i z e d m a t h
f u n c t i o n s t h a t m a x i m i z e p e r f o r m a n c e o n e a c h p r o c e s s o r f a m i l y .
U t i l i z e s i n d u s t r y - s t a n d a r d C a n d F o r t r a n A P I s f o r c o m p a t i b i l i t y
w i t h p o p u l a r B L A S , L A PA C K , a n d F F T W f u n c t i o n s — n o c o d e
c h a n g e s r e q u i r e d .
D i s p a t c h e s o p t i m i z e d c o d e f o r e a c h p r o c e s s o r a u t o m a t i c a l l y
w i t h o u t t h e n e e d t o b r a n c h c o d e .
P r o v i d e s p r i o r i t y s u p p o r t , c o n n e c t i n g y o u d i r e c t l y t o I n t e l
e n g i n e e r s f o r c o n f i d e n t i a l a n s w e r s t o t e c h n i c a l q u e s t i o n s

INTEL MKL PERFORMANCE

CPU VERSUS GPU FOR BIG DATA
C P U o f f e r s h i g h e r s c a l a b i l i t y a t l o w e r c o s t v e r s u s G P U
O p t i m i z e d S o f t w a r e a n d l i b r a r i e s o n C P U a l l o w s i n g l e - n o d e
p e r f o r m a n c e t o a p p r o a c h G P U p e r f o r m a n c e .
G P U p l u s C P U a r c h i t e c t u r e s c a n b e e f f e c t i v e f o r s m a l l e r n u m b e r
o f n o d e s , w h e n c o s t i s n o t a c o n c e r n .
” B i g C o m p u t e ” v e r s u s “ B i g D a t a ”

Running BigDL

RUNNING BIGDL
D e ve l o p i n g :
U s e t h e f o l l o wi n g t o d e v e l o p
y o u r B i g D L a p p s e ff o r t l e s s l y
• Docker
• VM Sandbox
D e p l o y i n g :
C l o u d r e a d y d e p l o y m e n t
• Amazon AMI

DEMO: GETTING STARTED WITH BIGDL
We wi l l p r o v i d e :
• Docker
• Sandbox VM
• AWS Marketplace AMI

BigDL Summary
B i g D L o f f e r s o u t s t a n d i n g s c a l a b i l i t y a n d p e r f o r m a n c e
B i g D L o p t i m i z e s T C O b y r u n n i n g b e i n g t u n e d a n d o p t i m i z e d f o r
I n t e l X e o n P r o c e s s o r s
B i g D L b r i n g s d e e p l e a r n i n g t o S p a r k C l u s t e r s a n d H a d o o p
D a t a s e t s
B i g D L c a n b e u s e d t o d e p l o y Te n s o r f l o w a n d C a f f e m o d e l s t o b i g
d a t a .

IMAGE RECOGNITION WITH
APACHE SPARK AND BIGDL
A l e x K a l i n i n | V P, A I / M a c h i n e L e a r n i n g | S i z m e k

ABOUT ME
A l e x K a l i n i n
V P, A I / M a c h i n e L e a r n i n g | S i z m e k
a l e x . k a l i n i n @ s i z m e k . c o m
L i n k e d i n : l i n k e d i n . c o m / i n / a l e x k a l i n i n /

https://www.linkedin.com/in/alexkalinin/
alex.kalinin@sizmek.com
1 0 0 , 0 0 0 , 0 0 0 , 0 0 0 r e q u e s t s p e r d a y
P B s o f t r a i n i n g d a t a
AI-POWERED MARKETING AND OPTIMIZATION
7 0 , 0 0 0 , 0 0 0 / m i n u t e
1 , 2 0 0 , 0 0 0 / s e c

FEED-FORWARD NETWORK

43
37
45
40
𝑦 = (𝑤𝑖 ∗ 𝑥𝑖)
?
?
?
?
?
?

-0.53
0.01
-0.17
0.70
0.51
?
?
𝑦 = (𝑤𝑖 ∗ 𝑥𝑖)
43
37
45
40
?
?
?
?

-0.53
0.01
-0.17
0.70
0.51
?
?
𝑦 = (𝑤𝑖 ∗ 𝑥𝑖)
43
37
45
40
-1.56
?
?
?

-0.53
0.01
-0.17
0.70
0.51
?
?
𝑦 = 𝑅𝑒𝐿𝑈( 𝑤𝑖 ∗ 𝑥𝑖 )
43
37
45
40
-1.56
?
?
?

-0.53
0.01
-0.17
0.70
0.51
?
?
43
37
45
40
?
?
?
0

43
37
45
40
-0.12
0.13
0.21
-0.07
-0.05
?
?
?
?
0
11.9

?
?
43
37
45
40
-.11
?
0
11.9

?
?
43
37
45
40 ?
0
11.9
0

?
?
43
37
45
40 0.15
0
11.9
0

-0.67
?
43
37
45
40 0.15
0
11.9
0

0
?
43
37
45
40 0.15
0
11.9
0

0
0.52
43
37
45
40 0.15
0
11.9
0

FULLY CONNECTED

FULLY CONNECTED
Input Size:
Connections:
40,000
1,600,000,000
200
200
200
200
10 layers: 16 billion

INVENTION OF CONVOLUTIONAL
NEURAL NETWORKS
• L e N e t - 5 n e t wo r k d e v e l o p e d i n 1 9 9 8 b y Ya n n L e C u n
• To r s t e n H u b e l a n d D a v i d W i e s e l

HUBEL & WIESEL
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1363130/

HIERARCHICAL & LOCAL VISUAL CORTEX

HIERARCHICAL & LOCAL VISUAL CORTEX
Lines,
Dots
Orientation,
Movement
High-Level
Shapes

KEY FEATURES OF CONVOLUTIONAL
NETWORK
• C o n v o l u t i o n
• P o o l i n g

CONVOLUTION

CONVOLUTION
O n l y f o u r we i g h t s

CONVOLUTION
0.10 -0.06
0.24 0.17
Filter

POOLING
Source: https://cs231n.github.io/convolutional-networks/

CONVOLUTIONAL NETWORK
0
0
0
0
1
0
0
0
0
0
0
1
2
3
4
5
6
7
8
9
Convolution Pooling Convolution PoolingInput FC FC
Source: https://www.clarifai.com /technology

http://scs.ryerson.ca/~aharley/vis/conv/flat.html

DEMO
G i t H u b : h t t p s : / / g i t h u b . c o m / a l e x - k a l i n i n / l e n e t - b i g d l

Getting Started With BigDL

ABOUT ME
S u j e e M a n i y a m
F o u n d e r / P r i n c i p a l @ E l e p h a n t S c a l e
P r a c t i t i o n e r a n d Tr a i n e r i n D a t a E n g i n e e r i n g
a n d D a t a S c i e n c e
Author
- "Hadoop and Spark" video training on O'Reilly Media
- "HBase Design Patterns"
- "Hadoop illuminated"
s u j e e @ e l e p h a n t s c a l e . c o m
L i n k e d i n : l i n k e d i n . c o m / i n / s u j e e m a n i y a m

RUNNING BIGDL
D e ve l o p i n g :
U s e t h e f o l l o wi n g t o d e v e l o p
y o u r B i g D L a p p s e ff o r t l e s s l y
• Docker
• VM Sandbox
• Amazon AMI
D e p l o y i n g :
C l o u d r e a d y d e p l o y m e n t
• Amazon AMI

GETTING STARTED WITH BIGDL
We wi l l d e m o n s t r a t e
• Docker
• Sandbox VM
• AWS Marketplace AMI

Docker
Step 1 : Install Docker on your laptop
Step 2 : get docker image
docker pull elephantscale/bigdl-sandbox
Step 3 : download tutorials
git clone https://github.com/elephantscale/bigdl-tutorials
Step 4 : Launch docker
cd bigdl-tutorials
./run-bigdl-docker.sh elephantscale/bigdl-sandbox
Step 5 : Go to Jupyter notebook

VM-Sandbox
Step 1 : Install VMware Player or VirtualBox on your laptop
Step 2 : Download BigDL-Sandbox image
http://elephantscale.com/sandbox/
Step 3 : (In Sandbox) download tutorials
Step 4 : (In Sandbox) Run BigDL natively
cd bigdl-tutorials
./run-bigdl-native.sh
Step 5 : (In Sandbox) Go to Jupyter notebook

Docker on AWS
Step 1 : Spin up an AMI (Ubuntu recommended)
Step 2 : Install Docker on the instance
Step 3 : get docker image
docker pull elephantscale/bigdl-sandbox
Step 5 : Launch docker
cd bigdl-tutorials
./run-bigdl-docker.sh elephantscale/bigdl-sandbox

AMI on AWS
Step 1 : Spin up BigDL AMI
Step 3 : Run BigDL
cd bigdl-tutorials
./run-bigdl-native.sh

QUESTIONS
G i t H u b : h t t p s : / / g i t h u b . c o m / a l e x - k a l i n i n / l e n e t - b i g d l
L i n k e d I n : h t t p s : / / ww w. l i n k e d i n . c o m / i n / a l e x k a l i n i n /

Notebooks and Resources
BigDL: software.intel.com/bigdl
Tutorials: github.com/dnielsen/bigdl-resources
Sandbox: elephantscale.com/sandbox
BigDL AMI: aws.amazon.com/marketplace/
Training: elephantscale.com
Slides: slideshare.net/dcnielsen/
Tim Fox Sujee Maniyam Alex Kalinin Dave Nielsen
Elephant Scale Elephant Scale Sizmek Intel Software

THANK YOU!

BigDL Deep Learning in Apache Spark - AWS re:invent 2017

More Related Content

Similar to BigDL Deep Learning in Apache Spark - AWS re:invent 2017

More from Dave Nielsen

Recently uploaded

BigDL Deep Learning in Apache Spark - AWS re:invent 2017

Editor's Notes