Big Data Tech Stack

Big Data
Tech Stack
Big Data 2015
by Abdullah Cetin CAVDAR

Ex Entrepreneur
I had 3 start-ups

Senior Software
Engineer
@Udemy

Founder and Organizer of
meetup.com/ankara-big-data-meetup

What's Big Data
Big data is data that exceeds the processing capacity
of conventional database systems.

What's Big Data
Big data is when the data itself becomes part of the
problem.

Multitude of Data
Types
Structured
Semi-structured
Unstructured

What We Need?
Store
Join
Index
Analytics
Aggregate
Visualize

Challenge
The challenge in big data analytics is to
dig deeply
quickly (real time?)
and widely

"ilities" or NFR?
Availability
Scalability
Security
Performance
...

What're essential
components?

Multiple internal
& external
data sources

Different
Volume, Variety,
Velocity

Aim is to create
a funnel after
proper validation
and cleaning

separate the
noise from
relevant info

It has capability to
Validate
Cleanse
Transform
Reduce
Integrate

Fault tolerance
Parallelization

HDFS
massively scalable distributed
file system

Non-relational,
distributed data?

CAP theorem
Consistency, Availability,
Partition Tolerance

Ingestion to DFS
Sqoop, Flume, MapReduce, ETL

Infrastructure &
Platform Layer

MapReduce
is the main computation paradigm

One cluster,
distributed storage,
distributed scheduler,
many types of applications.

Blueprints
NoSQL with HBase
Stream Processing with Storm/Spark
Graph Processing with Giraph
SQL on Hadoop with Impala
Columnar Data Formats

Data need to be protected
Meet compliance requirements
Individual's privacy

Proper
authorization and
authentication
needed

What can we do?
Authentication protocol like Kerberos
Enable file layer encryption
Use SSL, certificates and trusted keys
Provision with Chef, Puppet or Ansible like tools
Log all the communication for detecting anomalies
Monitor whole system

Get a complete
picture
of our Big Data tech stack

Satisfy SLAs with
min downtime

Co-Existence
with Traditional
BI
Data warehouse in the traditional way
Distributed MR processing on big data stores

Mediate data in either direction
i.e use Hive/HBase with Sqoop

Real-time analysis can leverage
low-latency NoSQL stores
i.e Cassandra, Vertica, ...

R may be used for complex
statistical algorithms

Huge volume and
variety of data

Need blazing fast search
mechanism
to index and search for big data
analytics

Gain insight faster
Look at different aspects of
data visually

There is no
"One Size Fits All"
solution

We need
Continuous
Development

Big Data Tech Stack

More Related Content

What's hot

Similar to Big Data Tech Stack

More from Abdullah Çetin ÇAVDAR

Recently uploaded

Big Data Tech Stack