Download - Big data technology by Data Sciences Thailand ในงาน THE FIRST NIDA BUSINESS ANALYTICS AND DATA SCIENCES CONTEST/CONFERENCE จัดโดย คณะสถิติประยุกต์และ

Big Data Technologies

The First NIDA Business Analytics and Data Sciences Contest/Conference

วันที่ 1-2 กันยายน 2559 ณ อาคารนวมินทราธิราช สถาบันบัณฑิตพัฒนบริหารศาสตร์

https://businessanalyticsnida.wordpress.com

https://www.facebook.com/BusinessAnalyticsNIDA/

Big Data คืออะไร

สถาปัตยกรรมสําหรับข้อมลูขนาดใหญ่เป็นเช่นไร

จะจัดการกับข้อมูลขนาดใหญ่ได้อย่างไร

การประมวลผลข้อมลูขนาดใหญ่จะทาํเช่นไร

Unstructured data ต่างจาก Relational Database Management System หรือไม่

เทคโนโลยีล่าสุดของข้อมลูขนาดใหญ่มีอะไรบ้าง

ทีมงาน Data Science Thailand

นวมินทราธิราช 3001 วันที่ 1 กันยายน 2559 15.15-16.30 น.

The First NIDA Business Analytics & Data Sciences Conference

Big Data Technology

1-2 September 2016

The First NIDA Business Analytics & Data Sciences Conference : 1-2 September 2016

People think this [big data] is a tech revolution. But it is really a business revolution enabled by technology.

– Steven Messer

Let's define “Big Data”


3Vs : Volume Velocity Variety

5Vs : Veracity Values

Let's define “Big Data”


Let's talk about Scalability


Scale-UP (vertical)


Scale-OUT (horizontal)


What are the challenges?


Even with the best hardware, frequent failure is the norm.

Challenges:Handling failure


Divide & Conquer strategies

Challenges:Parallelization


Challenges:Barrier to entry

“The IBM Blue Gene/P supercomputer installation at the Argonne Leadership Angela Yang Computing Facility located in the Argonne National Laboratory.”https://en.wikipedia.org/wiki/File:IBM_Blue_Gene_P_supercomputer.jpg




“The Borg, a beowulf cluster used by the McGill University pulsar group to search for binary pulsars (among other things).”https://en.wikipedia.org/wiki/File:Beowulf-cluster-the-borg.jpg



Let's talk about the revolution


Hadoop : A brief history

Cutting & Cafarella


Hadoop : Abstraction Layers

Storage Layer : HDFS (GFS)

Compute Layer : YARN + MapReduce


Hadoop

HDFS(Hadoop Distributed File System)


Hadoop : HDFS

FILE_001 A B C

NODE1 NODE2 NODE3

A2

A1

A3

B1

B2 B3

C1

C2C3


Hadoop : HDFShttps://ha doop.apach e.org/doc s/r1.2.1/h dfs_desig n.html


Hadoop

YARN(Yet Another Resource Negotiator)


Hadoop : YARN

A B C D

DATA1 DATA2 DATA1 DATA2


Hadoop : YARN

A B

DATA1 DATA2Network is expensive


Hadoop : YARN

A B C D



Hadoop : YARN

A B C D


App Master


Hadoop

MapReduce


Hadoop : MapReduce Paradigm

Consists of 2 primary functionsMap & Reduce

Both functions transform data



https://twitter.com/steveluscher/status/741089564329054208



https://twitter.com/steveluscher/status/741089564329054208

Parallelism achieved!


Hadoop :The Big Data enabler

●Fault-tolerant storage●Fault-tolerant computation●Parallel processing paradigm for the rest of us ?????


Hadoop : Not everyone codes


Hive : A High-level toolfor the rest of us

●Developed by Facebook●HiveQL (looks just like SQL)●Schema-on-Read●Data warehousing at scale



HiveQL(SQL) MapReduce job(s)→



DDLCREATE TABLE inventory(sku int, product_name string, manufacturer string, num_instock int) ROW FORMAT DELIMITEDFIELDS TERMINATED BY ','STORED AS TEXTFILETBLPROPERTIES("skip.header.line.count"="1");



DMLLOAD DATA INPATH '/home/user/products.csv' OVERWRITE INTO TABLE inventory;

SELECT SUM(num_instock) as instock_countFROM inventory;


What else?


Other selected tools

● Hbase● Kafka● Spark (SQL, GraphX, MLLib, Streaming)● Docker● Kubernetes● Mesos

Impacts of Hadoop(and other “Big Data” tools)




Democratization of high-volume, high-velocity data processing



We support Hadoop



We provide an alternative

Adopting “Big Data” technologies




Big Data is not a technology. It’s about answering business questions and delivering value.

– Teresa de Onis



It's not about building a Hadoop clusteror other “Big Data” solutions for that matter



http://mat tturck.co m/2016/02 /01/big-d ata-lands cape/



Big Data tools = building blocksBuild whatever you want



https://gi thub.com/ fluxcapac itor/pipe line



It requires a team effort… and support from the management


If You Want To Succeed With Big Data,

Start Small

– Doug Cutting


Thank you for your time

Q&A

[email protected]

Download - Big data technology by Data Sciences Thailand ในงาน THE FIRST NIDA BUSINESS ANALYTICS AND DATA SCIENCES CONTEST/CONFERENCE จัดโดย คณะสถิติประยุกต์และ

Top Related