managing eight-terabyte telco sas warehouse and make use ... · managing eight-terabyte telco sas...

33
Corporate Communications Turkcell Genel Sunum Ing 280206 28.02.2006 Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram TURKCELL The Premier GSM Operator in Turkey

Upload: truongthuan

Post on 25-Dec-2018

224 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Managing eight-terabyte TelcoSAS Warehouse and make useof SPD Server for performance

enhancements

Esra BayramTURKCELL

The Premier GSM Operator in Turkey

Page 2: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Table of Contents

• About Turkcell• Turkcell Data Mining Environment• Turkcell SAS Warehouse Architecture• Turkcell Data Mining Projects• Turkcell SAS Warehouse ETL• SPDS Server Configuration• SPDS Server Security Model• SPDS Server Enhancements• SAS EM4.3 Performance Tips

Page 3: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

About Turkcell

1. As of 31 March 2006

Leading GSM operator with 28.7 million1

Among the top five mobile operators in Europe in terms of number of subscribersCapital expenditures of US$4.7 billion only in Turkey as of December 2005Covers almost 100% of Turkey with over 10,000 base stationsAmong the leaders in Europe with respect to GPRS and MMS services offered to customersGlobal operator with international operations and worldwide roaming including international, GPRS and Camel

Page 4: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

About TurkcellDirectly and indirectly creates employment for more than 20,000 people in Turkey aloneThe only Turkish company listed on the New York Stock Exchange (NYSE) Represented on the Board (21 members) and Executive Management Committee of the GSM Association (13 members)Promotes ISO9001, EU and US regulatory, financial and corporate governance standards throughout its operationsMassive tax contribution to the Turkish governmentSocially responsible with extensive and internationally renowned educational, cultural and sports sponsorships

Page 5: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

As of end of March 2006, Turkcell covers 96.58% of Turkish population and 78.48% of the country’s geography.

Coverage Area

Page 6: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

As April 2006;International Roaming: 501 operators in 191 countries

GPRS Roaming: 250 operators in 98 countriesActive Camel Roaming: 112 operators in 68 countries

Passive Camel Roaming: 119 operators in 68 countriesInternational and International GPRS Roaming on Oceans

INTERNATIONAL ROAMING

Turkcell is among the top ten operators for international roaming and is the world’s #1 for GPRS roamingOceans are also included in Turkcell roaming and GPRS roaming coverage through an agreement reached with MCP Norway in 2005

Page 7: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Mobile PaymentMobile e-mailRing Back ToneMobile MusicMobile Video

MessengerVisual RadioPush to TalkWAP Blog

Turkcell Services

Turkcell offers products and services concurrently with global operators

Page 8: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

BlackberryTeleconferencem-municipalitym-government

Messaging servicesMobile marketing

IVR Services

Corporate SolutionsBy leveraging advancements in GSM technology, Turkcell is making mobile solutions an essential tool for business and government

Page 9: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Turkcell Data Mining Environment - 1

OPERATIONAL DATACustomer Billing Call

EXTRACT / TRANSFORM / LOAD

TURKCELLDATAWAREHOUSE

(20TB)

OLAP TOOL

Data MiningETL Data Mart

(Miner)ETL

Data MiningEUL Data Mart

(Viper)SAS EM5.3SAS EG 3.0SAS/ACCESS Interface to PC FilesSAS/ETS SoftwareSAS/STAT Software

Page 10: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Data Mining ETL Data Mart(Miner)• Sun Fire 4800• Hostname : Miner• 12X1200MHz CPU (1200 MHz), • 24 GB Memory• 11.5 TB Disk Space (8.5 TB Full )• Development and Production(ETL) Environment

• SAS Base 9.1.3 ,• SAS/Share,• SAS/Stat, • SAS/Access to Oracle, • SAS/Access to Sybase, • SAS/Warehouse Administrator, • SAS/Connect, • SAS/Enterprise Miner, • SPDS 4.3 (Dev/Prod)• SAS Integration Technologies, • SAS/GRAPH, • SAS/ETS, • SAS/QC

Turkcell Data Mining Environment - 2

Page 11: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Data Mining EUL Data Mart(Viper)• Sun V490• Hostname : Viper , • 4X1500MHz CPU • 16 GB Memory• 1.6 TB Disk Space (1.1 TB Full )• Server Environment

– SAS Enterprise Miner– SAS Integration Technologies– Enterprise Miner Server– Enterprise Miner Thin Client– SAS/ACCESS Interface to ORACLE

• User Environment – Windows– SAS EM5.2– SAS EG 3.0– SAS/ACCESS Interface to PC Files– SAS/ETS Software– SAS/STAT Software

Turkcell Data Mining Environment - 3

Page 12: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Turkcell SAS Warehouse Architecture-1Development Environment

• On Miner Server• Different SPDS4.3 Server• Different Libraries (tmp_dev, odd_dev, dm_dev, dev_user libraries)

libname=tmp_dev pathname=/cadm18/SPDS4/meta/tmp_dev owner=sasspdsroptions="metapath=('/cadm18/SPDS4/meta/tmp_dev')datapath=('/cadm17/SPDS4/data/tmp_dev' '/cadm18/SPDS4/data/tmp_dev'

'/cadm19/SPDS4/data/tmp_dev')indexpath=('/cadm19/SPDS4/index/tmp_dev')

• Different file systems for SPDS libraries• Different Unix Users• Logically separated from Production Environment• Read permission on Production Libraries• Different SPDS and SAS Work

Page 13: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Turkcell SAS Warehouse Architecture-1

Production Environment

• On Miner Server• Different SPDS4.3 Server• Different Libraries (tmptcell, oddtcell, dmtcell)

libname=tmptcell pathname=/cadm13/SPDS4/meta/tmptcell owner=sasspdsroptions="metapath=('/cadm13/SPDS4/meta/tmptcell')datapath=('/cadm24/SPDS4/data/tmptcell' '/cadm29/SPDS4/data/tmptcell'

'/cadm35/SPDS4/data/tmptcell' '/cadm04/SPDS4/data/tmptcell' '/cadm12/SPDS4/data/tmptcell' '/cadm33/SPDS4/data/tmptcell')

indexpath=('/cadm24/SPDS4/index/tmptcell' ‘‘/cadm29/SPDS4/index/tmptcell')"

• Different file systems for SPDS libraries• Different Unix User• Logically separated from Development Environment• Read Permission on Development Libraries• Different SPDS and SAS Work

Page 14: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Turkcell SAS Warehouse Architecture-2End User Environment

• On Viper Server, Windows• SPDS4.3 Server is installed• Each End user and EM Projects have their own libraries

– 29 end users, 20 EM projects– libnames.parm

libname=usr_a pathname=/cadm01/spdsmeta/usr_a owner=sasspdsroptions="metapath=('/cadm01/spdsmeta/usr_a')datapath=('/cadm02/spdsdata/usr_a' '/cadm03/spdsdata/usr_a'

'/cadm04/spdsdata/usr_a' )indexpath=('/cadm01/spdsindex/usr_a')“

libname=prj_a pathname=/cadm01/spdsmeta/prj_a owner=sasspdsroptions="metapath=('/cadm01/spdsmeta/prj_a')datapath=('/cadm02/spdsdata/prj_a' '/cadm03/spdsdata/prj_a'

'/cadm04/spdsdata/prj_a' )indexpath=('/cadm01/spdsindex/prj_a')“

• Read permission on Production and Development Libraries

Page 15: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

• Payment Behavior Segmentation (Credit Scoring)

• Application Scoring

• Customer Value Score (CVS)

• Churn Prediction

• Needs Based Segmentation

• Credit Limit

Turkcell Data Mining Projects

Page 16: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Turkcell SAS Warehouse – ETL

• LSF ( Screen shot)

• How is the projects scheduled ?

• How many processes are running ?

• How is the new projects are added ?

Page 17: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

LSF and SAS/Warehouse Administrator is used for job scheduling

Page 18: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Appropriate Calendar is created in LSFCodes are copied from Development Env to Production EnvCodes are defined in a SAS catalogJobs are defined for SAS catalog programs in SAS/WA Process EditorUsing LSF Scheduler Add-Ins jobs are sent to LSF

Page 19: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

Calendars used :

Project Name Calendar Name Calendar DefinitionAppMDL miner_AppMDL Application Scoring Model DevelopmentCredit Scoring miner_billing after 3rd billing date of each monthCreditScoringUPS miner_billing after 3rd billing date of each monthCreditScoringCPS miner_billing after 3rd billing date of each monthAppDay miner_daily daily at 01:00 pmLoyalty miner_Loyalty Loyalty Run TimeChurn miner_monthly 5-8 th day of each monthYTS miner_monthly 5-8 th day of each monthCoprCVS miner_monthly 5-8 th day of each monthCVS miner_monthly 5-8 th day of each monthCBS miner_monthly 5-8 th day of each monthAppMon miner_reporting 1-5 th day of each monthCredit Scoring Reporting miner_reporting 1-5 th day of each month

PeriodUpdated CalendarName Update the autoexec yearmonth

Miner_Monthly miner_monthly YES

miner_billmonth

miner_billrepmonth

Miner_Billing miner_billing YES

miner_billreport

miner_billmonth

miner_billrepmonth

Miner_Reporting miner_reporting YES

miner_billreport

miner_billrepmonth

# of jobs : 357 LSF jobs

ETL Period : 20 days

Concurrent ETL Sessions: 8

Extraction from Oracle:

104jobs, 130Hrs

Transformation in SAS:

161 jobs, 120Hrs

DataMart in SAS:

68jobs, 120 Hrs

Upload from SAS:

24jobs, 20Hrs

Page 20: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Configuration-1• There are three SPD Servers for Development, Production and End User Layer

Miner(ETL machine)Development SPDS4.3spdsna 5180/tcp #SPDS name server port for SAS9spdssn 5181/tcp #SPDS Snet server port for SAS9

Production SPDS4.3spdsna43 5200/tcp #SPDS name server port for SAS9 SPDS4.3spdssn43 5201/tcp #SPDS Snet server port for SAS9 SPDS4.3

Viper (EUL machine)EUL SPDS4.3spdsna43 5200/tcp #SPDS name server port for SAS9 SPDS4.3spdssn43 5201/tcp #SPDS Snet server port for SAS9 SPDS4.3

Page 21: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Configuration-2Production spdsserv.parmSORTSIZE=2048M;INDEX_SORTSIZE=512M;GRPBYROWCACHE=256M; BINBUFSIZE=32K;INDEX_MAXMEMORY=30M;WORKPATH="('/cadm03/SpdsW

ork4.3‘)”;NOCOREFILE;SEQIOBUFMIN=64K;RANIOBUFMIN=4K;MAXWHTHREADS=32;MAXSEGRATIO=75;WHERECOSTING;RANDOMPLACEDPF;MINPARTSIZE=2048M;

Development spdsserv.parmSORTSIZE=2048M;INDEX_SORTSIZE=128M;GRPBYROWCACHE=128M; BINBUFSIZE=32K;INDEX_MAXMEMORY=30M;WORKPATH="('/cadm38/SpdsWo

rkDev4.3')";NOCOREFILE;SEQIOBUFMIN=64K;RANIOBUFMIN=4K;MAXWHTHREADS=32;MAXSEGRATIO=75;WHERECOSTING;RANDOMPLACEDPF;MINPARTSIZE=2048M;

EUL spdsserv.parmSORTSIZE=2048M;INDEX_SORTSIZE=512M;GRPBYROWCACHE=256M; BINBUFSIZE=32K;INDEX_MAXMEMORY=30M;WORKPATH="('/cadm01/Spds

Work4.3' )";NOCOREFILE;SEQIOBUFMIN=64K;RANIOBUFMIN=4K;MAXWHTHREADS=16;MAXSEGRATIO=75;WHERECOSTING;RANDOMPLACEDPF;MINPARTSIZE=512M;

Page 22: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Security Model-1• Unix user is created first.

– useradd –u userid username

• This user must be added to the SPDS User list. – Psmgr

• add yyucel SASpw9 SASpw9 3 - - SUPPORT - -• chgpass yyucel SASpw9 SASpw2 SASpw2

• A macro is written to assign security an easy way it accepts two parameters, library to be secured and users who have write access all the rest will have read access

• Users are login with their Unix users password, and assign the library with thespds user created the same as unix user.

libname tmp_dev sasspds 'tmp_dev' server=miner.spdsna43 user="&SYSUSERID" passwd="SASpw2" IP=YES;

Page 23: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Security Model-2%macro libsecure(_libname=, _users=);...........................................................

proc SPDO lib=&_libname;set ACLTYPE DATA;set acluser &user;delete acl _all_;%if &user=sasadm %then

add acl/libname persist;add acl /generic persist;modify acl _all_/read nowrite noalter nocontrol;..........................................................%do ind2=1 %to &numofusers;

%let userpass = %sysfunc(scan(&_users,&ind2,' '));&userpass=(y,y,y,y)

%end; /*do loop*/quit;

......................................................%mend;

libname tmp_dev sasspds 'tmp_dev' server=miner.spdsprod user="sasadm" passwd="SASpw2" unixdomain=yes netcomp=no aclspecial=yes;

%libsecure(_libname=tmp_dev, _users=ebayram sasprod churnusr sasdev hsari ukaradag);

Page 24: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Security Model-3

Page 25: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Enhancements-1• MINPARTSIZE=2GB

Most of the tables are bigger than 2GB. MINPARTSIZE was 128M before.Increasing to 2GB, result in less data partitions for a table and decrease thenumber of I/O requests.Test: Reading from a table having 22 partitions was 100% faster than readingfrom the same table having only 351 partitions.

• SORTSIZEIncreased from 48MB to 2048MBFew MERGE jobs would benefit from the extra memory allocated for SORT. The machine has 24GB of RAM.

• The MAXWHTHREAD server parameter was set to 64 threads. During scalability tests, it was determined that 32 threads were adequate. By cutting the number by half, This would prevent the server from overthreading and further reduce thread contention during queries.

Page 26: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Enhancements-2126 data &perm_tmp_lib..Corporate_Indicators30 ;127 merge128 &perm_tmp_lib..Corporate_Indicators(in=in1)129 &perm_tmp_lib..SegmentIndicators_&RefPeriod130 &perm_tmp_lib..ScoreIndicators_&RefPeriod131 &perm_tmp_lib..NumDiffBnoIndicators_&RefPeriod132 ;133 by Party_id;134 135 if in1;136 137 run;

NOTE: There were 170660 observations read from the data set TMPTCELL.CORPORATE_INDICATORS.NOTE: There were 35093410 observations read from the data set TMPTCELL.SEGMENTINDICATORS_200508.NOTE: There were 27819995 observations read from the data set TMPTCELL.SCOREINDICATORS_200508.NOTE: There were 38038783 observations read from the data set TMPTCELL.NUMDIFFBNOINDICATORS_200508.NOTE: The data set TMPTCELL.CORPORATE_INDICATORS30 has 170660 observations and 1412 variables.NOTE: Compressing data set TMPTCELL.CORPORATE_INDICATORS30 decreased size by 63.23 percent.NOTE: DATA statement used (Total process time):

real time 1:19:12.66cpu time 56:52.57

Page 27: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Enhancements-3140 proc sql;141 create table &perm_tmp_lib..Corporate_Indicators31 as select a.*, b.*,c.* ,d.*142 from &perm_tmp_lib..Corporate_Indicators a143 left join &perm_tmp_lib..SegmentIndicators_&RefPeriod b on a.party_id=b.party_id144 left join &perm_tmp_lib..ScoreIndicators_&RefPeriod c on a.party_id=c.party_id145 left join &perm_tmp_lib..NumDiffBnoIndicators_&RefPeriod d on a.party_id=d.party_id146 ;NOTE 49-169: The meaning of an identifier after a quoted string may change in a future SAS release. Inserting

white space betweena quoted string and the succeeding identifier is recommended.

NOTE 49-169: The meaning of an identifier after a quoted string may change in a future SAS release. Inserting white space between

a quoted string and the succeeding identifier is recommended.NOTE: Physical Name: TMPTCELLNOTE: Compressing data set TMPTCELL.CORPORATE_INDICATORS31 decreased size by 63.23 percent.NOTE: Table TMPTCELL.CORPORATE_INDICATORS31 created, with 170660 rows and 1412 columns.

147 quit;NOTE: PROCEDURE SQL used (Total process time):

real time 25:30.96cpu time 55.61 seconds

Page 28: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Enhancements-4177 proc sql;178 create table &perm_tmp_lib..Corporate_Indicators2 as select a.*, b.*,c.*

,d.*,e.*,f.*,g.*,h.*,j.*,k.*,l.*,m.*,n.*179 from &perm_odd_lib..Corporate_Indicators a180 left join &perm_tmp_lib..SegmentIndicators_&RefPeriod b on a.party_id=b.party_id181 left join &perm_tmp_lib..ScoreIndicators_&RefPeriod c on a.party_id=c.party_id182 left join &perm_tmp_lib..NumDiffBnoIndicators_&RefPeriod d on a.party_id=d.party_id183 left join &perm_tmp_lib..SekreterCellIndicators_&RefPeriod e on a.party_id=e.party_id184 left join &perm_tmp_lib..ServiceIndicators_&RefPeriod f on a.party_id=f.party_id185 left join &perm_tmp_lib..TelephoneNumHistInd_&RefPeriod g on a.party_id=g.party_id186 left join &perm_tmp_lib..AccountStatsIndicators_&RefPeriod h on a.party_id=h.party_id187 left join &perm_tmp_lib..Postpaidpayment_&RefPeriod j on a.party_id=j.party_id188 left join &perm_tmp_lib..Freeminuteindicators_&RefPeriod k on a.party_id=k.party_id189 left join &perm_tmp_lib..Freeminuteindicators2_&RefPeriod l on a.party_id=l.party_id190 left join &perm_tmp_lib..MergeCallStatsIndicators_&RefPeriod m on a.party_id=m.party_id191 left join &perm_tmp_lib..Loyaltyindicators_&RefPeriod n on a.party_id=n.party_id192 ;NOTE: PROCEDURE SQL used (Total process time):

real time 1:16:48.51cpu time 1:15.06

Page 29: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SPD Server Enhancements-5• Adding option IP=YES to all SPD Server libname statements.

The option would enable implicit SQL pass-through

Counting the rows of a table with proc sql is very fast now. In SAS EM 4.3 also, when you choose a table from Input Data Source, allthe rows are counted. For a 4GB table, it lasts 2 minutes to countthe rows before, but 2 seconds after the option is set. This option willincrease the speed of choosing a table from Input Data Sourcenode in case of counting the rows.

Page 30: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SAS EM4.3 Performance Tips-1• Duration of opening an EM4.3

project is too long.• Duration of listing the libraries in

IDS is too long.

We decreased the number of libraries from 67 to 26

Numberof Libraries

Duration toopen theEM4.3 project (sec)

Duratio to listthe librariesin IDS (sec)

3 42 9

10 50-70 17

15 50-70 18

25 100-120 42

35 100-130 41

45 100-130 46

55 100-130 54

67 100-130 63

Page 31: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SAS EM4.3 Performance Tips-2

• Duration of listing the tables in IDS is too long.

#of Libraries

TMPTCELL(769 tables)

EERDAL(42 tables)

3 120 5

10 131 5

15 87 6

25 99 4

35 92 5.5

45 86 6

55 96 5.5

67 96 6

Page 32: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

SAS EM4.3 Performance Tips-3

# of Records # of Columns Table Name Size of Table(MB)Duration toopen a tablein IDS(secs)

100,000 564 ODDTCELL.TEST100bin 402 0.3 200,000 564 ODDTCELL.TEST200bin 803 0.6 300,000 564 ODDTCELL.TEST300bin 1,205 1.0 400,000 564 ODDTCELL.TEST400bin 1,606 1.3 500,000 564 ODDTCELL.TEST500bin 2,008 2.0 600,000 564 ODDTCELL.TEST600bin 2,410 2.7 700,000 564 ODDTCELL.TEST700bin 2,811 3.1 800,000 564 ODDTCELL.TEST800bin 3,213 2.6 900,000 564 ODDTCELL.TEST900bin 3,614 3.2

1,000,000 564 ODDTCELL.TEST1000bin 4,016 3.6 1,200,000 564 ODDTCELL.TEST1200bin 4,819 4.1 1,400,000 564 ODDTCELL.TEST1400bin 5,622 5.2 1,600,000 564 ODDTCELL.TEST1600bin 6,425 5.3 1,800,000 564 ODDTCELL.TEST1800bin 7,229 6.0 2,000,000 564 ODDTCELL.TEST2000bin 8,032 7.3 2,159,786 50 ODDTCELL.NTEST50 788 0.5 2,159,786 100 ODDTCELL.NTEST100 1,606 1.1 2,159,786 150 ODDTCELL.NTEST150 2,430 1.9 2,159,786 200 ODDTCELL.NTEST200 3,279 2.7 2,159,786 250 ODDTCELL.NTEST250 4,185 6.4 2,159,786 300 ODDTCELL.NTEST300 4,663 2.8 2,159,786 350 ODDTCELL.NTEST350 5,487 6.0 2,159,786 400 ODDTCELL.NTEST400 6,265 4.6 2,159,786 564 CKOYLU.NEEDS_FINAL 8,673 7.1

• Duration of opening a tablein IDS is too long.

Page 33: Managing eight-terabyte Telco SAS Warehouse and make use ... · Managing eight-terabyte Telco SAS Warehouse and make use of SPD Server for performance enhancements Esra Bayram

Corporate Communications Turkcell Genel Sunum Ing 28020628.02.2006

THANK YOU