big data taiwan 2014 keynote 4: monetize enterprise data – big data...
DESCRIPTION
講者：Etu 資深協理 | 陳育杰 簡介：過去這兩年內，Big Data 在企業的應用架構已逐漸形塑出來，我們看到，不同的產業，陸續開始運用 Hadoop 來解決不同的問題，而背後的 IT 架構，其實都具有一些共通性。我們將透過這些共通性的架構來探索 Big Data / Hadoop 具體展現的企業應用。TRANSCRIPT
Monetize Enterprise Data Big Data 在台灣的經典應用與行動�
陳育杰 Eric Chen� Senior AVP, Etu Business Development� [email protected] �
的特性
2
傳統平行運算架構
The Old Way: Bringing Data to Compute�
Hadoop 架構 = 平行運算 + 分佈式儲存
The New Way: Bringing Compute to Data�
運算�
儲存�
MapReduce�
HDFS�
Big Data 新應用架構
3
RDB
Business Intelligence
ETL
Business Analy>cs
Voice file Video file Image file�
Doc file Txt file XML file�
Web Logs Click event�
Social network
Associated map News
Feeds
Sensor Embedded RFID Tags
Geographic GPS
Event Others
MapReduce�
HDFS�
HBase� HIVE� Impala�
Mahout� Pig�
Big Data 新應用架構 Hadoop as a “Data Store”
4
RDB
Business Intelligence
ETL
Business Analy>cs
Voice file Video file Image file�
Doc file Txt file XML file�
Web Logs Click event�
Social network
Associated map News
Feeds
Sensor Embedded RFID Tags
Geographic GPS
Event Others
MapReduce�
HDFS�
HBase� HIVE� Impala�
Mahout� Pig�
Big Data 新應用架構 Hadoop as a “Data Pre-processing Platform”
5
RDB
Business Intelligence
ETL
Business Analy>cs
Voice file Video file Image file�
Doc file Txt file XML file�
Web Logs Click event�
Social network
Associated map News
Feeds
Sensor Embedded RFID Tags
Geographic GPS
Event Others
HDFS�
HBase� HIVE� Impala�
Mahout�
MapReduce�
Pig� HIVE QL�
Join, Aggrega,on, Filter, Sor,ng, Correla,on ……..
Data Integra4on
Big Data 新應用架構 Hadoop as a “DB”
6
RDB
BI
ETL
Business Analy>cs
Voice file Video file Image file�
Doc file Txt file XML file�
Web Logs Click event�
Social network
Associated map News
Feeds
Sensor Embedded RFID Tags
Geographic GPS
Event Others
MapReduce�
HDFS�
HBase� HIVE� Impala�
Mahout� Pig� ODBC
API
Big Data 新應用架構 Hadoop as a “Data Analytics
Engine”
7
RDB
Business Intelligence
ETL
Business Analy>cs
Voice file Video file Image file�
Doc file Txt file XML file�
Web Logs Click event�
Social network
Associated map News
Feeds
Sensor Embedded RFID Tags
Geographic GPS
Event Others
MapReduce�
HDFS�
HBase� HIVE� Impala�
Mahout� Pig�
Use Case & Reference Architecture
l Real-time Query (電信)
l Customer Services (電信)
l Yield Rate Improvement (電子製造)
l Recommendation & Behavior Analysis (電
商、媒體、零售)
l Smart Retailing (Internet of Things)
8
Real-time Query (電信) Use Case & Reference Architecture
9
A10/ F5
Etu Hadoop Cluster
HDFS�
HBase
Syslog / TCP
Plug-‐in Etu Data Flow
MSISDN IP / Time
IP by MSISDN
IP by MSISDN
MSISDN by IP
Real-‐>me Query End-‐to-‐End < 5 sec
Historical Query
Internet
Customer Services (電信) Use Case & Reference Architecture
10
DPI
Syslog UDP
Internet
Border Router
Etu Log Collector 200K
EPS
HDFS�
FTP
1 TB / Day
0.7 TB / Day
Table A Table B Table C
Etu Hadoop Cluster for Correla4on (Regional)
HDFS�
HBase
Etu Hadoop Cluster for Query (180 days)
(Central)
11
Etu TQuery 電信巨量資料多樣查詢最佳化解決方案�
Yield Rate Improvement (電子製造) Use Case & Reference Architecture
12
組裝包產線�
SMT產線�
製程資料�
組裝包產線�
SMT產線�
SMT產線�
SPC SFCS�
SMT SFCS SPI�
1. 生產問題及時發現: 資料處理與計算時間大幅縮短，可以提升品質判斷速度，減少產線損失 �
2. 運算效能佳: 採用平行運算與分散式檔案系統，減少過多Temp Files與資料轉換，生成統計表提供查詢�
3. 容量擴充成本低: 因應產線擴充，機台測試資料增加與保存時間延長可線性擴充 (Scale out)�
Etu IProspects� 統計分析加以確認�
統計分析�
製作圖表�
良率低主因素� RDB�
N mins�
MPP DB�
資料探勘�
特徵規則 (平行運算)�
HDFS�
No SQL�
SPC� SMT� SFCS�
SMT Data Files�
及時�
算得快�
擴充成本低�
Etu Recommender
Etu Insight
Recommendation & Behavior Analysis (電商、媒體、零售) ���Use Case & Reference Architecture
13
Consumer Discovery Consumer Connect Customer Profile Discovery
Data Converter
Customer Behavior
Data Warehouse
HIVE JDBC/ODBC
Driver
Event Collector Customer Behavior
Data Store
Analytics core
Etu Hadoop Platform HDFS� HBase� HIVE�
User Tracking Dashboard
Recommendation Recommendation
List
商品推薦� 內容推薦� 廣告推薦� EDM 整合�
CRM 整合�
第三方 分析工具
連結�
Recommendation & Behavior Analysis (電商、媒體、零售) ���Use Case & Reference Architecture
14
Etu Recommender
商品� 內容� 廣告�
Consumer Connect
Customer Behavior
Recommenda4on List
Etu Recommender
商品� 內容� 廣告�
Consumer Connect
Customer Behavior
Recommenda4on List
Consumer Discovery
DW CRM
推薦運算叢集�
3600 Customer View
推薦運算叢集� 客戶行為分析叢集�
Customer Profile Discovery Data
Converter Customer Behavior Data Warehouse
HIVE JDBC /ODBCDriver
Event Collector Customer Behavior
Data Store
Analytics core Event Collector Customer Behavior
Data Store
Analytics core
Smart Retailing (IoT) Use Case & Reference Architecture (Proof-of-Concept)
15
Etu Recommender
商品� 內容� 廣告�
Consumer Connect
Customer On-‐line Behavior
Recommenda4on List
Consumer Discovery
DW CRM
3600 Customer View
Customer Profile Discovery Data
Converter Customer Behavior Data Warehouse
HIVE JDBC /ODBC Driver
Event Collector Customer Behavior
Data Store
Analytics core
Customer Off-‐line Behavior
318, Rueiguang Rd., Taipei 114, Taiwan T: +886 2 7720 1888 [email protected] www.etusolu4on.com
Thank you