Taming Big Data With Modern Software Architecture

Software Architecture
Taming Big Data
With
Modern Software Architecture
Big Data User Group
Stuttgart, 26.09.2013
Thomas Widmann
Managing Director, WidasConcepts
Vinoth Kannan
Senior Consultant, WidasConcepts

About the Speakers
2

Thomas Widmann
Vinoth Kannan
!   Senior Consultant in WidasConcepts
!   Working primarily around Big Data ecosystems of Hadoop and Storm
!   Experienced in IT industries of Europe, North America and Asia
!   Masters degrees from Politecnico di Torino, Italy & Grenoble INP, France
! Contact: vinoth.kannan@widas.de
!   Managing Director of WidasConcepts
!   Expert in modern Sales- and Service-Solutions
!   More than two decades of consulting experience in Web X.0
concepts, continuous delivery and IT-strategies
!   Big Data Evangelist
! Contact: thomas.widmann@widas.de

Who we are
!   Innovative IT-Consulting Company
founded in 1997
! Headquaters in Wimsheim with
offices in Munich and Essen
! Specialized in Software for Modern
Business in the areas of
!   Strategic IT Consulting
!   Next Business Intelligence
!   Software Development
!   35 expert consultants specializing in
!   Data Science
!   Software Architecture
!   Mobile & Web Development
!   Business Consulting
WidasConcepts GmbH
3

Agenda
4

How Big Data can help
business
Big Data applied in
CarbookPlus Platform
New Software
Architectures for Big Data
Q & A

How big data can help business
5
Big challenges of todays business
eCommerce Financial Services Telecommunication
HealthCare Marketing National Security
o  Increase in Sales
o  Determine pricing
o  Avoid Shop cart abandoners
o  Manage logistics and inventory
o  Fraud detection
o  Real Time analytics on
financial data
o  Managing mobile payments
o  Manage network traffic
overload
o  Hardware cost optimisation
based on traffic data
o  Real time monitoring of
patients vitals for predictive
intervention and care.
o  Cost savings on patient care
o  Mood prediction for market
research
o  Real time bidding
o  Location based marketing
o  Monitor real time feed from
CCTV, traffic data, mobile data
o  Find patterns/cluster for
people, places, products, things
Building sector, all manufacturers, energy, media and many more

Getting the Right Data
to the Right People
at the Right Time
6

Mobile Payment
Money Cards / POS
Banking Portal
Adviser / Agency
Banking
Operations
Event
Database
Business
Activity
Monitoring
CEP
Making Use of Events
Real Time Analysis
Cross-Up-Selling
Fraud Detection
Product Innovation

Impact of Big Data on Business
Results on Europe-wide survey by TU-Berlin, Sept 2013
Source: European Smart Data Survey, Sept 2013
7

Need for Big Data
4Vs concept of Big Data
Volume
To handle the huge
amount of data genereted
by businesses
Velocity
To store, analyse and
retrive huge dataset in
good speed
Variety
To process data from
multiple sources, majorly
unstructured data
Source: Oracle
Value
To ask the right questions
to generate maximum
value
8

What makes data so big?
Combination of structured, unstructured data
created by sensors, actors, humans
9

New Software Architecture
Need of the hour for modern architecture
analyse

huge

volume
of

data

fast

consolidate

data
from

mul;ple

sources

Intelligent

predic;ve

10

Agenda
11

business
Big Data applied in
New Software
architectures for Big Data
Q & A

Datawarehouse – Classic BI
Online System
Database n
Classic Software Architectures
Main goal: keep business records
<Client>
Browser
Presentation
Business Logic
Database 2
StagingETL ETL
BI
Reporting
Database 1
Batch
Processing
Main goal: Reporting
12

Need for real time system
for enterprise
13

The Approach
Impact of analytics, event processing and decision management in building a real
time big data system for enterprise
Acquire Data:
•  Business Trx./Events
•  Observational Events
from anywhere
Observe
Orient
Decide
Act

Systems:
•  Various systems
•  Preferrable: MQ
•  put in context
•  Evaluate data, predict
•  determine actions
•  Initiate actions
•  manage data
Systems:
•  complex event processing,
rule management
•  realtime analysis with new
big data technologies
•  select action, if
necessary
•  Perform action
(manually or
automated)
Systems:
•  Business software
•  Decision support
software
•  Reporting
(John Boyd) OODA-Loop ist ein Informationsstrategiekonzept aus dem militärischen Bereich

Phases of realising
real time system
Step by step approach for the architecture
• Create structured data from raw data
• Extract, merge, filter etcData Extraction
• Aggregate and select relevant data
• Speed and flexibilityModel development
• Testing the model
• Comparing the result
Evaluation &
deployment
• Triggered by action in the decision layer
• Score to validate outputReal-time Scoring
• Need to refresh the ever changing data
• Real time data viewModel Refresh
14

Big Data Architecture
Service Layer CQRS
Distributed DB
Distributed Messaging System
Recommendations
Realtime-Processing Batch-Processing
Lambda architecture
Business Transactional Data
Business Analytics Data
Recommendation Data
Real Time Event Alert Data
Big Data Eco System
Distributed Search
Engine
Business
Data Formats
Contextual Data
Business Transactions
Reporting
Notifications
Notification SystemData Retrieval
In-Memory Technology
15

Distributed DBNoSQL DB

Basic idea behind
Lambda Architecture
query = function(all data)
Nathan Marz
Big Data - principles and best practices
of scalable realtime data systems
16

Basic Idea behind Lambda
𝑓( 𝑎↓0 … 𝑎↓𝑚 )
Perform some function from real-time data “0“ to the history data “n“
Real Time Big Data
𝑓( 𝑎↓0 .. 𝑎↓𝑛 … 𝑎↓𝑚 ) = 𝑓( 𝑎↓0 .. 𝑎↓𝑛 ) + 𝑓( 𝑎↓𝑛+1 … 𝑎↓𝑚 )
Lambda Architecture
Hadoop ProcessStorm
Process
Real Time Big
Data
}
}
}
Letting the History data processed by Hadoop makes process faster
17

Agenda
18

business
Big Data applied in
New Software architectures
for Big Data
Q & A

What
is
Carbookplus?

Change
the
way
you
drive.

19

Why
Carbookplus?

Beneﬁts
of
being
a
Carbookplus
member

Track trips
Maintain Logbook
Get Notified about best gas stations & snipers
Manage and compare vehicle cost
Fleet management
Social platform connecting drivers

Big
Data
in
Carbookplus

Where
do
we
use
it

Trip
Transac;on
Data

Recommend
possible
trips

Create
trips

Update
logbook
entry

No;fy
user
for
discrepancy
in
the

logbook
entry

No;fy
user
for
best
gas
sta;on

Update
vehicle
mileage

Generate
analy;c
reports

Real Time !!!

Big
Data
in
Carbookplus

Event
Driven
Model

Use Case : Track trips on Real Time
Start
address

Intermediate
address
1

End
address

Event 1
Event 2
Event n Intermediate
address
n

…..

Event n+1
Object Model
Event …

Big
Data
in
Carbookplus

How
do
we
use
it

Trip
Transac;on

Data

Real Time Storm Cluster
In
Memory
Data
Processing

MongoDB

Trip
Data
Hadoop
Cluster

Map
&
Reduce

Analy;cal
Data

Decision
Management

Recommender

Engine

Rule

Engine

KaYa

ETL

Output
Mobile
Web
BI

User
Data
ETL

Vehicle
Data
ETL

Historical Data
Real Time Data

Conclusion

Ø  Big Data clearly improves business efficiency and answers lot of the
questions which were previously unanswerable in this data driven world.
Ø  Need to adapt our IT architecture to remain competitive and see the big
picture
Ø  Flipside : No standardised implementation architecture. Need to be
tailor-made for individual business needs.
Ø  Road Ahead : Need for phased integration into Big Data ecosystem
Ø  We are hiring !!
§  Data Scientists
§  Big Data Engineers
§  Interns
Visit us : www.widas.de
Email : contact@widas.de

Agenda
25

How BigData can help
business
BigData applied in
New Software architectures
for
Big Data
Q & A

Taming Big Data With Modern Software Architecture

Taming Big Data With Modern Software Architecture

More Related Content

What's hot

Viewers also liked

Similar to Taming Big Data With Modern Software Architecture

Recently uploaded

Taming Big Data With Modern Software Architecture