Apache Pig

Agenda

• What is Apache Pig • How to Setup• Tutorial Examples

PIG Introduction

• Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs

• Pig generates and compiles a Map/Reduce program(s) on the fly

PIGParse

Compile

Optimize

Plan

Pig Latin Scripts

有 Pig 後 Map-Reduce 簡單了！ ?

• Apache Pig 用來處理大規模資料的高級查詢語言• 適合操作大型半結構化數據集• 比使用 Java ， C++ 等語言編寫大規模資料處理程式

的難度要小 16 倍，實現同樣的效果的代碼量也小 20倍。

• Pig 元件– Pig Shell (Grunt)– Pig Language (Latin)– Libraries (Piggy Bank)– UDF: 使用者定義功能

4figure Source : http://www.slideshare.net/ydn/hadoop-yahoo-internet-scale-data-processing

大象遇到豬 ( setup )

• 解壓縮

• 修改 ~/.bashrc

• 啟動 pig shell

export JAVA_HOME=/usr/lib/jvm/java-7-oracleexport HADOOP_HOME=/home/hadoop/hadoopexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport PIG_HOME=/home/hadoop/pigexport PATH=$PATH:$PIG_HOME/bin

cd /home/hadoopwget http://archive.cloudera.com/cdh5/cdh/5/pig-0.12.0-cdh5.3.2.tar.gztar –zxvf pig-0.12.0-cdh5.3.2.tar.gzmv pig-0.12.0-cdh5.3.2 pig

$ pig grunt> grunt> ls /hdfs://master:9000/hadoop <dir>hdfs://master:9000/tmp <dir>hdfs://master:9000/user <dir>grunt>

豬也會的程式設計

6

豬BigData

pigLatin功能 指令讀取 LOAD

儲存 STORE

資料處理

REGEX_EXTRACT, FILTER, FOREACH, GROUP, JOIN, UNION, SPLIT, …

彙總運算

AVG, COUNT, MAX, MIN, SIZE, …

數學運算

ABS, RANDOM, ROUND, …

字串處理

INDEXOF, SUBSTRING, REGEX EXTRACT, …

Debug DUMP, DESCRIBE, EXPLAIN, ILLUSTRATE

HDFS cat, ls, cp, mkdir, …

$ pig –x grunt> A = LOAD ‘file1’ AS (x, y, z);grunt> B = FILTER A by y > 10000;grunt> STORE B INTO ‘output’;

用 shell 硬把程式兜出來，放棄用 hadoop 了 使用 PIG 發憤圖強，廢寢忘食的研究…

練習一 :

• 場景 :– 老闆要我統計組織內所有員工的平均工時。於是我取

得了全台灣的打卡紀錄檔 ( 打卡鐘的 log 檔 ) ，還跟人事部門拿到了員工 id 對應表。這些資料量又多且大，我想到要餵進去 Hadoop 的 HDFS, .. 然後

• 問題 :– 為了寫 MapReduce ，開始學 Java, 物件導向 ,

hadoop API, … @@• 解法 :

7

整型前的 mapreduce code

8

nm dp Id Id dt hr

劉 北 A1 A1 7/7 13

李 中 B1 A1 7/8 12

王 中 B2 A1 7/9 4

Java Code

Map-Reduce

A1 劉 北 7/8 13

A1 劉 北 7/9 12

A1 劉 北 Jul 12.5

用 pig 整形後

9

北 A1 劉 12.5

LOADLOAD

FILTER

JOIN

GROUP

FOREACH

STORE

(nm, dp, id)

(nm, dp, id)(id, dt, hr)

(nm, dp, id, id, dt, hr)

(group, {(nm, dp, id, id, dt, hr)})

(group, …., AVG(hr))

(dp,group, nm, hr)

Logical PlanPig LatinA = LOAD 'file1.txt' using PigStorage(',') AS (nm, dp, id) ;B = LOAD ‘file2.txt' using PigStorage(',') AS (id, dt, hr) ;C = FILTER B by hr > 8;D = JOIN C BY id, A BY id;E = GROUP D BY A::id;F = FOREACH E GENERATE $1.dp,group,$1.nm, AVG($1.hr);STORE F INTO '/tmp/pig_output/';

nm dp Id Id dt hr

劉 北 A1 A1 7/7 13

李 中 B1 A1 7/8 12

王 中 B2 A1 7/9 4

Tips : 先用小量資料於 pig –x local 模式驗證；每行先配合 dump or illustrate 看是否正確

練習一 : 實作 cd ~

git clone https://github.com/waue0920/hadoop_example.git

cd ~/hadoop_example/pig/ex1 pig -x local -f exc1.pig cat /tmp/pig_output/part-r-00000

練習 : 執行 pig –x mapreduce ，將 exc1.pig 每一行單獨執行，並搭配 dump , illustrate 來看結果，如 :

Grunt> A = LOAD 'file1.txt' using PigStorage(',') AS (nm, dp, id);Grunt> Dump AGrunt> Illustrate A

Q : result 是否有改進空間 ?Q : 如何改進 ?

進階Simple Types Description Exampleint Signed 32-bit integer 10long　

Signed 64-bit integer　

Data:     10L or 10lDisplay: 10L

float　

32-bit floating point　

Data:     10.5F or 10.5f or 10.5e2f or 10.5E2FDisplay: 10.5F or 1050.0F

double　

64-bit floating point　

Data:     10.5 or 10.5e2 or 10.5E2Display: 10.5 or 1050.0

chararray Character array (string) in Unicode UTF-8 format hello world

bytearray Byte array (blob) 　boolean boolean true/false (case insensitive)

datetime datetime 1970-01-01T00:00:00.000+00:00

biginteger Java BigInteger 2E+11bigdecimal Java BigDecimal 33.45678332Complex Types Description ExampleFields A piece of data Johntuple An ordered set of fields. (19,2)bag An collection of tuples. {(19,2), (18,1)}map A set of key value pairs. [open#apache]

進階

• cat data;

• A = LOAD 'data' AS ( t1:tuple(t1a:int,t1b:int,t1c:int), t2:tuple(t2a:int,t2b:int,t2c:int) );

• X = FOREACH A GENERATE t1.t1a,t2.$0;

(3,8,9) (4,5,6)(1,4,7) (3,7,5) (2,5,8) (9,5,8)

((3,8,9),(4,5,6))((1,4,7),(3,7,5))((2,5,8),(9,5,8))

(3,4)(1,3) (2,9)

進階

進階

進階Data Types and

More Relational Operators

Complex Types ASSERT MAPREDUCEBags COGROUP ORDER BYTuples CROSS RANKFields CUBE SAMPLEMap DEFINE SPLIT

Simple Types DISTINCT STOREint FILTER STREAMlong FOREACH UNIONfloat GROUP 　double IMPORT 　chararray JOIN (inner) 　bytearray JOIN (outer) 　boolean LIMIT 　datetime LOAD 　biginteger UDF Statementsbigdecimal DEFINE REGISTER

練習二

• 說明 : 從數字陣列中，觀察 pig 的語法，以及結果的變化

• 使用技術 : filter .. by, foreach .. by, group .. by, foreach .. generate, cogroup

• Input/ output

See : https://wiki.apache.org/pig/PigLatin (last edited 2010)

myfile.txt B.txt

練習二 cd ~/hadoop_example/pig/ex2 hadoop fs -put myfile.txt B.txt ./ pig -x mapred> A = LOAD 'myfile.txt' USING PigStorage('\t') AS (f1,f2,f3);> B = LOAD 'B.txt' ; dump A; dump B;> Y = FILTER A BY f1 == '8'; dump Y;> Y = FILTER A BY (f1 == '8') OR (NOT (f2+f3 > f1)); dump Y;> X = GROUP A BY f1; dump X;> X = FOREACH A GENERATE f1, f2; dump X;> X = FOREACH A GENERATE f1+f2 as sumf1f2; dump X;> Y = FILTER X by sumf1f2 > 5.0; dump Y;> C = COGROUP A BY $0, B BY $0; dump C;> C = COGROUP A BY $0 INNER, B BY $0 INNER; dump C;

練習三

• 說明 : 從 <userid, time, query_term> 的記錄檔，做出使用者喜愛的關鍵字分析

• 使用技術 : UDF, DISTINCT, FLATTEN, ORDER

• Source pigtutorial.tar.gz• Input / output

See : https://cwiki.apache.org/confluence/display/PIG/PigTutorial

練習三 cd ~/hadoop_example/pig/ex3 pig -x local> REGISTER ./tutorial.jar;> raw = LOAD 'excite-small.log' USING PigStorage('\t') AS (user, time, query);> clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);> clean2 = FOREACH clean1 GENERATE user, time, org.apache.pig.tutorial.ToLower(query) as

query;> houred = FOREACH clean2 GENERATE user, org.apache.pig.tutorial.ExtractHour(time) as

hour, query;> ngramed1 = FOREACH houred GENERATE user, hour,

flatten(org.apache.pig.tutorial.NGramGenerator(query)) as ngram;> ngramed2 = DISTINCT ngramed1;> hour_frequency1 = GROUP ngramed2 BY (ngram, hour);> hour_frequency2 = FOREACH hour_frequency1 GENERATE flatten($0), COUNT($1) as

count;> uniq_frequency1 = GROUP hour_frequency2 BY group::ngram;> uniq_frequency2 = FOREACH uniq_frequency1 GENERATE flatten($0),

flatten(org.apache.pig.tutorial.ScoreGenerator($1));> uniq_frequency3 = FOREACH uniq_frequency2 GENERATE $1 as hour, $0 as ngram, $2 as

score, $3 as count, $4 as mean;> filtered_uniq_frequency = FILTER uniq_frequency3 BY score > 2.0;> ordered_uniq_frequency = ORDER filtered_uniq_frequency BY hour, score;> STORE ordered_uniq_frequency INTO 'result' USING PigStorage();

pig -x local -f script1-local.pig cat result/part-r-00000

UDF

package myudfs;import java.io.IOException;import org.apache.pig.EvalFunc;import org.apache.pig.data.Tuple;import org.apache.pig.impl.util.WrappedIOException;

public class UPPER extends EvalFunc<String>{ public String exec(Tuple input) throws IOException { if (input == null || input.size() == 0) return null; try{ String str = (String)input.get(0); return str.toUpperCase(); }catch(Exception e){ throw WrappedIOException.wrap("Caught exception processing input row ", e);}}}

grunt> register myudfs.jargrunt> A = load 'student_data' using PigStorage(',') as (name:chararray, age:int,gpa:double);grunt> B = FOREACH A GENERATE myudfs.UPPER(name);grunt> dump B;

Reference

• Pig 說明– http://pig.apache.org/docs/r0.12.0/basic.html

• Pig 參考投影片– http://www.slideshare.net/ydn/hadoop-yahoo-

internet-scale-data-processing• Pig 範例參考

– https://cwiki.apache.org/confluence/display/PIG/PigTutorial