big data insight v1.7_140617

1. 2014.06.17 Big data TFT

2. Contents I. Summary II. DW AWS() DW DW simulation III. End-user Big Data Basic Mock up Big Data Insight Mock up IV. SBD

3. I. Summary TFT TF // 1.0 M/M Web UI/ 0.5 M/M // 1.0 M/M Big Data Data DW - Step 1 : Big data Basic / - Step 2 : Big data insight / - Step 3 : Big data advanced // TFT Timeline Big data TFT Kickoff 2014-02-28 K-BEC 2014 2014-03-12 2014-02-28 / 2014-03-20 AWS 2014-04-01~22 TFT sponsor TFT DW 2014-05-01 Data / 2014-05-12 2014-06-14 Modeling / ETL 2014-05-19 Big data basic Open 3 DW 2014-04-23

4. II. DW 4 ()

5. AWS Data Size Data Analysis AWS DW Data Analysis Data BigData AWS DW : AWS , DATA AWS vs DW Process DW AWS Step 1 DB Step 2 Data //Modeling Step 3 DB ETL, Query Step 4 Web & visualization S3 data load Redshift EC2/Jaspersoft Data , () AWS vs DW COST DW AWS Step 1 0.5 M/M X 0.5 M/M O ( ) Step 2 0.5 M/M X 0.5 M/M X Step 3 0.5 M/M X 0.5 M/M X Step 4 2.5 M/M O (DW, ) 1 M/M O (/ ) 3.0 M/M : 1,400,000 (, M/M ) 2.5 M/M : $1,000 (, 1T ) II. DW 5

6. II. DW DW Big data Basic - - 1 Big data insight - - 2 Big data Advanced - - 6 / /

7. II. DW Data Data Data Data *ETL Data Warehouse /// : Data *DSS, EIS : CEO, Data Data Data Data Data Data 7 OLAP / : Tool : *ETL : Extraction(), Transformation(), Loading() *DSS : decision support system , *EIS : executive information system ()

8. II. DW / Data Data DW , Data Data seq_noINT IDENTITY(1,1)NOT -- ,date_numINTNOT -- ,year_numINTNOT -- ,quarter_numINTNOT -- ,month_numINTNOT -- ,week_numINTNOT -- ,weekday_numINTNOT -- ,serialINTNOT --SM_SLAE_TB ,orderidVARCHAR(30)NOT -- ,adidINTNOT -- ,payidVARCHAR(50)--() ,pay_type_nmVARCHAR(30)NOT -- ,sub_section_cdVARCHAR(10)NOT -- ,account_nmVARCHAR(50)-- ,content_type_nmVARCHAR(10)-- ,main_product_nmVARCHAR(200)-- ,option_product_nmVARCHAR(200)-- ,pay_amtINTNOT -- ,pay_dtDATETIMENOT --() ,last_pay_dtDATETIME-- ,pay_durationINT--(Insight data) ,useridVARCHAR(40)NOT -- ,user_join_dtDATETIME-- ,recruit_start_dtCHAR(12)-- ,recruit_end_dtCHAR(12)-- ,job_class1VARCHAR(50)--1 1 ,job_class2VARCHAR(50)--1 2 ,job_class3VARCHAR(50)--1 3 ,job_sub_class1VARCHAR(50)--2 1 ,job_sub_class2VARCHAR(50)--2 2 ,job_sub_class3VARCHAR(50)--2 3 ,job_work_area_classVARCHAR(30)-- (,) ,job_work_area_sub_classVARCHAR(30)-- (,) ,job_agelimitminTINYINT--() ,job_agelimitmaxTINYINT--() ,job_paycdCHAR(3)-- ,job_payINT--( ) ,company_nameVARCHAR(50)-- ,company_reg_noVARCHAR(12)-- ,infrowtypecdCHAR(5)-- ,job_genderVARCHAR(10)-- ,com_type_nmVARCHAR(50)-- ,user_type_nmVARCHAR(30)-- ,user_grade_cdVARCHAR(10)-- 8 , , , , .

9. simulation II. DW DW IBM X3650 M3 Xeon X5680 6C,3.33GHz,12M L3 * 2Ea 4G*8Ea(PC3-10600R) 146G(15K,SAS2.5)*2Ea RAID 0 300G(10K,SAS2.5)*4Ea RAID 5 ( 900GB) Define Data : 415GB, : 5GB(daily) Basic Data 1.5GB, : 0.5GB(daily) Basic Data : 2.5GB (daily), 17.5GB (weekly), 75GB (monthly) Insight 9 , , 14 8 DW : 7,000,000 * IDC , AWS : 12,768,000 * 10TB,

10. II. DW ID / PW Heaven Analytics VPN Y N N Y DSS, EIS CEO, , DW DSS, EIS , OLAP ()

11. II. DW 5/01~5/09 , Data Re-Define/, Data Data Size DW , Data Loading Method Test & Confirm 5/12~5/16 Data / (1 Big data Basic ) , DW , DW DB Define Data Loading Define Data loading () Define Data loading 5/19~5/23 Data ,, (ETL) Basic Data Modeling, Basic Data ETL Process Basic Data ETL Process Data Basic Data Re-Modeling, Data 5/26~5/30 Basic Data Modeling, Basic Data ETL Process Basic Data ETL Process Data Basic Data Re-Modeling, Data Data Basic Data Re-Modeling, Data 6/02~6/09 Basic Data / Basic Data ETL Process 6/10 Insight Data , Data / Data 6/10~6/27 Visualization & End-User Visualization // QA 6/30 Big Data Basic Data 11

12. III. End-User Step 1 - Big Data Basic Mock up(Dashboard) CI 71,272,602 1,379 3% 43% ECOMM 34,959,890 1,277 3% 3% MCOMM 834,650 101 13% 13% AGENT 0 0 3% 3% 4,900 1 3% 3% Dashboard ECOMM AGENT X X X X X - 500,000,000 1,000,000,000 1,500,000,000 2,000,000,000 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 2013 2014 ECOMM MCOMM AGENT SMTRO X IT.. X . X .. X X 2014 6 : ~ 2014-06-12 2014-06-12 : 2013.01.01~2014.06.09 EXECEL PDF

13. Step 2 - Big Data Basic Mock up ( ) CI Dashboard TOP 1 2 3 4 5 6 7 2014 5,624,185,784 2,759,266,054 832,418,928 249,115,955 13,662,000 2,268,300 281,700 9,481,198,721 729,930,458 376,198,460 150,315,594 32,726,585 1,204,500 337,900 26,000 1,290,739,497 TOP 1,175,439,506 551,949,624 152,486,246 53,207,000 2,494,800 625,380 20,150 1,936,222,706 1,262,285,300 628,569,670 151,184,875 56,772,960 3,393,500 514,590 113,750 2,102,834,645 M 1,236,910,501 638,833,560 277,272,447 52,581,870 3,241,700 377,200 36,900 2,209,254,178 1,219,620,019 563,714,740 101,159,766 53,827,540 3,327,500 413,230 84,900 1,942,147,695 5,624,185,784 2,759,266,054 832,418,928 249,115,955 13,662,000 2,268,300 281,700 9,481,198,721 0 500,000,000 1,000,000,000 1,500,000,000 1 2 3 4 5 2014 2013.01.01~2014.06.09 DASHBOARD MVC X III. End-User

14. Step 2 - Big Data Insight Mock up ( ) CI - - TOP 2013.01.01~2014.06.09 1-7 15-308-14 31-60 61-90 1-7 8-14 15-20 21-30 4.758 1,987 958 325 8,028 + 4,758 1,987 958 325 8,028 4,758 1,987 958 325 8,028 III. End-User

15. Step 2 - Big Data Insight Mock up ( ) CI - - TOP 2014.06.09 ID 12.3 2014.05.22 2014.06.03 2014.06.13 5,500 0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 0 20,000 40,000 60,000 80,000 100,000 120,000 III. End-User

16. UI _SBD

17. Big Data Analytics (BASIC) Description. 17 > 2014.06.13 1 PC 1024*768px Mobile 320px viewport 2 BI 3 4 5 ID 6 7 *3,5,6 . 8 p.15 Dev. : dw.alba.co.kr/ 1 1 Analytics ID Login X X 4 4 76 5 3 8 2014 ArbeitChunkook, Inc SSL-VPN 9 2 .

18. Big Data Analytics (BASIC) Description. 18 > 2014.06.13 1 2 2-1 3 4 ID () 5 5-1 6 * 6-1 *null, 6-2 Dev : dw.alba.co.kr/ Analytics ID Login X X 4 4 86 5 3 7 2014 ArbeitChunkook, Inc SSL-VPN 9 ID 3 . (T.010-8806-8163 | E. [email protected]) 1 3 4 5 6 . X5-1 2 2-1

19. Big Data Analytics (BASIC) Description. 19 >>ECOMM D.>1 2014.06.13 1 BI 2 GNB 3 LNB 4 5 LNB / 6 DB 7 DB 8 DB total 4, 9 / . p.24 10 null p. 24 4 3 2 Analytics 1 000, . ID Total : 0,000 6 7 5 No. ID (email) () ( ) 1 20140 211 [email protected] (2014.06.30) 2014.06.30 (1) 2 20140 211 [email protected] 2014.06.30 2014.06.30 (2) 3 20140 211 [email protected] 2014.06.30 2014.06.30 (2) 4 20140 211 [email protected] 2014.06.30 2014.06.30 (2) 5 20140 211 [email protected] 2014.06.30 2014.06.30 (2) 6 20140 211 [email protected] 2014.06.30 2014.06.30 (2) 7 20140 211 [email protected] 2014.06.30 2014.06.30 (2) 8 20140 211 [email protected] 2014.06.30 2014.06.30 (2) 9 20140 211 [email protected] 2014.06.30 2014.06.30 (2) 8 9 10

20. Big Data Analytics (BASIC) Description. 20 >>ECOMM D.>1 2014.06.13 1 2 * 0000 3 , , 3 4 5 ( ) 6 1~6 / * 7 * Analytics 000, . * * * ID * * * 1 20140211 [email protected] ************************* 2014.06.30 >ECOMM PDF 2 2014.06.30 >ECOMM PDF 2 2014.06.30 >ECOMM PDF 2 2014.06.30 >ECOMM PDF 2 2014.06.30 >ECOMM PDF 2 3 2 4 5 6 7

21. Big Data Analytics (BASIC) Description. 21 >>ECOMM D.>1 2014.06.13 1 2 3 * 4 4 4-1 4-2 GNB ( GNB ) * 2 GNB 5 5-1 5-2 5-3 * 5-1 * 1 Analytics 000, . GNB 00 01 02 5 3 1 X X X X X 2 X 5-3 5-2 5-1 GNB X 4-2 4-1 4

22. Big Data Analytics (BASIC) Description. 22 2014.05.01~2014.05.31 ECOMM ECOMM MVC >>ECOMM D.>1() 2014.06.13 1 GNB 2 *PDF, 2 3 *1 ECOMM *2 4 4-1 4-2 * * (, , :-) 5 (ECOMM , MVC, X) - 6 * ( ) 7 8 : : : 9 , , (dim) 10 4 * 1 2 4 5 8 9 7 ( ) ( ) () MVC 961,476,524 9,447 101,776 14.3% 6.8% 8.0% X 980,671,171 42,744 22,943 0.0% 1.0% 1.0% 10 - 200,000,000 400,000,000 600,000,000 800,000,000 1,000,000,000 1,200,000,000 1,400,000,000 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 MVC X Analytics X ECOMM 3 000, . 6 X X X X IT.. X . X .. X X 2014 6 : ~ 2014-06-12 2014-06-12 4-1 4-2

23. Big Data Analytics (BASIC) Description. 23 ECOMM >>ECOMM D.>2 2014.06.13 1 *2 2 2-1 (MVC ) * 3 MVC X , / *2 MVC ( ) - 200,000,000 400,000,000 600,000,000 800,000,000 1,000,000,000 1,200,000,000 1,400,000,000 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 MVC X Analytics ECOMM 1 2 3 MVC () 2013 980,988,633 1,789,874,956 2014 997,933,933 1,723,928,386 3456 1234 % 2% 4% X 2013 953,564,564 1,756,654,252 2014 987,546,695 1,875,869,652 2 MVC S 2-1 B A C ECOMM MVC X 2014.02.01~2014.03.31 000, . 3

24. Big Data Analytics (BASIC) Description. 24 ECOMM >>..>1 2014.06.13 1 *1 : *2 : 2 2-1 ( ) * : : , , : : * 3 * (dim) 4 10 5 * 100~500 ( 10 ) Analytics 1 () () 315,259,534 22,117 14.3% 6.8% TOP 288,048,329 3,040 0.0% 1.0% 260,412,194 5,850 14.3% 6.8% 190,602,050 826 0.0% 1.0% 148,746,580 153 14.3% 6.8% 110,249,800 2,696 0.0% 1.0% 87,904,300 656 14.3% 6.8% 76,117,000 97 0.0% 1.0% TOP - 50,000,000 100,000,000 150,000,000 200,000,000 250,000,000 300,000,000 350,000,000 5 2014 TOP TOP 10 000, . 2 3 4 : 5 2 2014.05.01~2014.05.31

25. Big Data Analytics (BASIC) Description. 25 ECOMM >>..>2 2014.06.13 1 *2 : 2 ( , ) 2 * : MVC, X * : , HR, , , 3 / 4 / * 3 Analytics 1 TOP MVC X MVC X 23,650,700 23,650,700 23,650,700 70,494,600 70,494,600 70,494,600 124,202,694 124,202,694 124,202,694 129,966,830 129,966,830 129,966,830 28,793,000 28,793,000 28,793,000 131,537,100 131,537,100 131,537,100 TOP 134,616,365 134,616,365 134,616,365 56,262,200 56,262,200 56,262,200 27,969,304 27,969,304 27,969,304 41,520,200 41,520,200 41,520,200 769,012,993 769,012,993 769,012,993 - 50,000,000 100,000,000 150,000,000 200,000,000 250,000,000 300,000,000 350,000,000 MVC - 2014 - 5 X - 2014 - 5 2014.05.01~2014.05.31 000, . 2 4 10 :

26. Big Data Analytics (BASIC) Description. 26 ECOMM >> .>1 2014.06.13 1 *1 : *2 : 2 ( ) 3 * (dim) * : , : 4 10 : : , , (/) 5 * 100~500 ( 10 ) Analytics 1 () () (, ) 315,259,534 22,117 14.3% 6.8% 288,048,329 3,040 0.0% 1.0% 260,412,194 5,850 14.3% 6.8% 190,602,050 826 0.0% 1.0% 148,746,580 153 14.3% 6.8% 110,249,800 2,696 0.0% 1.0% 87,904,300 656 14.3% 6.8% 76,117,000 97 0.0% 1.0% TOP 10 000, . 2 3 4 : 5 0 200,000,000 400,000,000 600,000,000 800,000,000 1,000,000,000 1,200,000,000 1,400,000,000 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 2013 2014 (, ) IT 2014.02.01~2014.03.31

27. Big Data Analytics (BASIC) Description. 27 ECOMM >>..>2 2014.06.13 1 *2 : 2 ( , ) 2 * : MVC, X * : , HR, , , 3 2 4 / * 3 Analytics 1 TOP MVC X MVC X (, ) 1,072,515,487 678,267,448 1,750,782,935 2,580,776 27,322,527 29,903,303 68,326,136 279,941,863 348,267,999 40,772,544 66,368,420 107,140,964 100,981,677 98,767,827 199,749,504 106,126,597 208,007,284 314,133,881 32,817,815 80,308,465 113,126,280 109,217,835 525,044,703 634,262,538 IT 1,541,909 14,894,029 16,435,938 1,534,880,776 1,978,922,566 3,513,803,342 2014.05.01~2014.05.31 000, . 2 4 10 : 0 200,000,000 400,000,000 600,000,000 800,000,000 1,000,000,000 1,200,000,000 (, ) IT

28. Big Data Analytics (BASIC) Description. 28 ECOMM >>..>2 2014.06.13 2 Analytics 1 TOP 2014.05.01~2014.05.31 000, . 2 10 : HR HR (, ) 278,169,070 161,522,870 200,023,410 395,141,523 367,965,676 728,831,386 2,131,653,935 681,600 5,893,300 1,519,600 14,587,050 7,010,450 541,700 30,233,700 17,018,100 35,111,078 20,074,380 192,739,990 60,038,200 29,498,700 354,480,448 2,658,500 14,977,300 6,077,500 27,257,150 41,272,400 38,293,200 130,536,050 5,316,380 26,617,362 62,478,294 51,202,306 58,559,861 99,631,556 303,805,759 10,231,300 26,269,950 6,809,800 97,219,446 99,582,850 78,350,530 318,463,876 3,122,941 18,040,300 7,999,980 40,142,108 27,274,668 18,104,800 114,684,797 23,114,100 53,428,172 15,976,000 496,339,233 43,204,874 10,151,200 642,213,579 IT 394,600 1,919,800 146,300 6,259,249 7,624,800 360,300 16,705,049 340,706,591 343,780,132 321,105,2641,320,888,055 712,533,779 1,003,763,372 4,042,777,193 0 100,000,000 200,000,000 300,000,000 400,000,000 500,000,000 600,000,000 700,000,000 800,000,000 (, ) IT

big data insight v1.7_140617

Internet