7078734 siebel eim best practise1

8/3/2019 7078734 Siebel Eim Best Practise1

1/36

Designing EIM Interfaces

from a Performance

Perspective

DRAFT 1

Published by BT Retail T&T

IN CONFIDENCE

Siebel EIM Best Practice

IN CONFIDENCE


2/36

Siebel EIM Best PracticeIN CONFIDENCE

About this document...

Author

The author of this document may be contacted at:

Paul Duncan (MEH4)

Alexander Bain House

2nd Floor (pp 2DAQ)

15 York Street

GLASGOWG2 8LA

Tel:

Mobile:

Email:

0141 220 8704

07939 215 541

[email protected]

Distribution List

BT Siebel EIM Interface Development Teams:

High Level Designers

Low Level Designers Development Teams

BT Siebel Performance Testing and Management

Intended Audience

This document contains detailed technical information relating to Siebel Enterprise IntegrationManager (EIM) and Oracle. A level of understanding of these technologies is assumed.

Filing

The filing reference for this document is: TBD

Page 2 of 36

IN CONFIDENCE


3/36


History

Issue Date Author Reason

0.1 08-AUG-2004 Paul Duncan Initial Creation

Page 3 of 36

IN CONFIDENCE


4/36


Contents

1 OVERVIEW .......................................................................................................................................................................5

1.1 PURPOSE..........................................................................................................................................................................5

1.2 WHATSNOTINCLUDED......................................................................................................................................................5

1.3 DOCUMENT CONVENTIONS..................................................................................................................................................6

2 EIM BEST PRACTICE ...................................................................................................................................... ........ ......7

2.1 INTRODUCTION..................................................................................................................................................................7

2.2 EIM CONFIGURATION........................................................................................................................................................7

2.2.1 Overview ................................................................................................................................................................72.2.2 IFB Generic Parameters ........................................................................................................................... ......... ...9

2.2.2.1 Only Base Tables ............................................................................................. ................................................. ..... ..... .....9

2.2.2.2 Ignore Base Tables .................................................................................................. ................................................... ......92.2.2.3 Ignore Base Columns ............................................................................................................ ................................... ..... .102.2.2.4 Default Columns ........................................................................................................ ............................................. ..... ..10

2.2.2.5 Fixed Columns ............................................................................................................. ............................................... ...102.2.2.6 Use Index Hints, Use Essential Index Hints .............................................................................................. ............ ..... ....11

2.2.2.7 Session SQL .................................................................................................. ............................................. ..... ..... ..... ....122.2.2.8 Using Synonyms ........................................................................................................ ............................................. ..... ..12

2.2.2.9 Rollback On Error, Commit Each Pass, Commit Each Table ............................................................................ ..... ..... ...132.2.2.10 Update After Failed Insert ............................................................................................................. ......................... ..... .14

2.2.3 IFB Import Process Specific Parameters ........................................................................................................... .152.2.3.1 Only Base Columns ...................................................................................................... .................................. ..... ..... .....15

2.2.3.2 Insert Rows, Update Rows ....................................................................................................... ................................. .....152.2.3.3 Net Change ........................................................................................................ ............................................... ..... ..... ...16

2.2.3.4 Oracle Insert Append Mode ....................................................................................................... ...................... ..... ..... ....17

2.2.3.5 Trim Spaces ..................................................................................................... .............................................. ...... ..... .....182.2.4 EIM Delete Process Configuration ....................................................................................................................19

2.2.4.1 Delete All Rows ............................................................................................. ......................................................... ..... ..19

2.2.4.2 Delete Exact ............................................................................................................ ..................................................... ..192.2.4.3 Delete Matches .............................................................................................. ............................................... ..... ..... ..... ..20

2.2.5 EIM Merge Process Configuration ........................................................................................................... ........ ..22

2.2.6 Other Parameters ........................................................................................................................................... .....24

3 RUNNING EIM .................................................................................................................................................... ........ ...26

3.1 DATA.............................................................................................................................................................................26

3.2 IFB FILEAND EIM PROCESSES........................................................................................................................................27

3.3 SCHEDULING...................................................................................................................................................................28

3.3.1 Degree of Parallelism/Layout .................................................................................................................... .........28

3.3.2 Batchsize ..............................................................................................................................................................283.3.3 EIM Streaming Configurations .................................................................................................................. .........28

3.3.4 Control-M Scheduling ............................................................................................................................ ........ .....29

3.4 EIM TABLE MAINTENANCE.............................................................................................................................................31

4 VOLUME 4 LESSONS LEARNED ......................................................................................................................... ......32

4.1 ENHANCEMENTSTO EIM PROCESSES.................................................................................................................................32

4.1.1 Redundant Updates .............................................................................................................................. ......... ......324.1.2 Splitting Updates ......................................................................................................................................... ........33

4.1.3 Hints and Optimiser Mode ...................................................................................................................... ......... ...35

4.2 DATABASE CONFIGURATION...............................................................................................................................................35

4.2.1 Indexes .................................................................................................................................................................35

4.3 ODM ADAPTER FUNCTIONALITY......................................................................................................................................36

Page 4 of 36

IN CONFIDENCE


5/36


1 Overview

1.1 Purpose

This document is intended to be used as a guide for the design and deployment of EIMinterfaces to support the delivery of Siebel based projects at BT. It will be of particular relevanceto the delivery of OneSiebel (aka Account Managed Platform, AMP) and OneView (aka VolumeSiebel Release 6).

This document is split into two main sections as follows:

1. Siebel EIM Best Practice This section details techniques to be adopted to ensure that EIM interfaces and their

configurations adhere to Siebel and BT Best Practices.2. OneView Release 4 data load Lessons Learned

This section details the key lessons learned from the OneView (aka Volume Siebel, BigData, CCC) bulk data load of Billing Accounts, Assets and Consumer Updates into theOneView Siebel instance.

This document has been assembled from experience from BTs eVolve and Volume platforms.These applications are fed data via ODM controlled data loads. Where processes other thanODM will drive EIM interfaces (e.g. Converge to OneSiebel migration jobs) then some ODMrelated content will still be relevant and so all sections should be read.

1.2 Whats not included

This document is intended to share Best Practice and Lessons Learned and is neither a

High Level nor a Low Level Design. This document should be used to feed into the abovementioned designs.

Details of Oracle-Level tuning is not included although some techniques may be

mentioned in the context of other EIM specific parameters or configurations. This is not a One Stop Shop from which to learn EIM. Instead, this document is

intended to communicate EIM Best Practice.

Page 5 of 36

IN CONFIDENCE


6/36


1.3 Document Conventions

Key points within this document have been included in call-out boxes as follows:

This indicates a convention that should be followed by Development Teams withinBT.

A warning sign indicates that specific care should be taken, that a piece of work isrequired to determine the best configuration or that a convention or parametershould not be followed.

An information symbol is used to indicate additional points of interest, notnecessarily relevant to specific implementations at BT.

These Document Conventions appear for most key point (in the EIM Best Practice section) andso the documents recommendations could be summarised by reading just the illustrated points.

Page 6 of 36

IN CONFIDENCE


7/36


2 EIM Best Practice

2.1 Introduction

Siebels Enterprise Integration Manager (EIM) operation is determined by a configuration file.This file is called the IFB file and, coupled with loading data into a Siebel EIM/Interface table (astaging table that Siebel uses so that it can build the logical data relationships as defined withinSiebels Repository) determines what operations EIM carries out.

As a generic, out-of-the-box interface mechanism, EIMs performance can be streamlined byreducing the amount of work it has to do. This is done by imposing strict IFB guidelines.

Not all parameters are discussed in detail.

A single Interface Table maps to multiple Base Tables (to closely follow the nature of Siebel

Business Objects and Business Components).

All tables and related foreign keys are processed by default unless parameters are specified inthe ifb file.

This document describes how to alter EIMs default operation by specifying parameters in the ifbfile. Some of these changes make functional impacts (i.e. change the way in which howEIMprocesses the data) whereas others can improve performance (normally by limiting the amountofworkEIM and the underlying database has to do).

2.2 EIMConfiguration

2.2.1Overview

EIM supports the following 4 modes of operation:

o Import

o Delete

o Merge

o Export

The EIM Import job is actually an Upsert operation and so this document will refer to bothImport (Insert) and Update EIM operations.

There are a number of generic ifb parameters that are valid for all modes of operation as well asthose that are operation specific.

This document will explore Import, Delete and Merge jobs in more detail. EIM Exports will notbe covered, as they are not (generally) used within BT instead, PL/SQL (often controlled by AbInitio graphs) is used to query the Siebel data tables directly in order to export information.

Page 7 of 36

IN CONFIDENCE


8/36


So that some of the terminology within this document is clear, heres an example IFB file (indicates lines removed for clarity):

[Siebel Interface Manager]LOG TRANSACTIONS TO FILE = FALSEUSING SYNONYMS = FALSE

[Update_Volume_Bill_Accnt]

TYPE = IMPORTBATCH = $BatchRangeTABLE = EIM_ACCOUNTONLY BASE TABLES = S_PARTY, S_ORG_EXTINSERT ROWS = S_PARTY, FALSEUPDATE ROWS = S_PARTY, FALSEINSERT ROWS = S_ORG_EXT, FALSENET CHANGE = FALSEONLY BASE COLUMNS = S_PARTY.PARTY_UID, \

S_PARTY.PARTY_TYPE_CD, \S_ORG_EXT.NAME, \S_ORG_EXT.LOC, \S_ORG_EXT.BU_ID, \

Note: a backslash (\) should be used to escape new lines for readability.

Many parameters can be specified in the Header section, the Process section or both. The rulesof inheritance are as follows:

If a parameter is set in the Header section, it is included in all Process sections.

If a parameter is used in a Shell process, it is included in all Shell Sub processes

(detailed later). If a parameter is in the Header section and also in the Process section, then the value in

the Process section over-rides that of the Header section. The same is true for Shellprocess and Shell Sub process.

If a parameter exists only in a Process section and not in the Header section, then it is

only used in that Process.

Page 8 of 36

IN CONFIDENCE

headersection

Processsection


9/36


2.2.2IFB Generic Parameters

The parameters here apply to all EIM job types, with process specific (i.e. Import, Delete orMerge) parameters and techniques being specified in subsequent sections.

2.2.2.1 Only Base Tables

This is a Process section parameter.

Usage:

ONLY BASE TABLES = BASE_TABLE1, BASE_TABLE2

where BASE_TABLE1 & BASE_TABLE2 are tables mapped to by the EIM table in question.

By only including the Base Table relationships that you need to resolve, EIMs work is reducedand so Performance can be improved.

Example: In OneView, a Consumer Update job incorrectly carried out updates to S_PARTY,S_ORG_EXT, S_ACCNT_POSTN and S_ORG_BU despite the fact that the data inS_ACCNT_POSTN and S_ORG_BU was being defaulted by the Consumer Adapter (Ab Initio)each time. By removing these tables from the ONLY BASE TABLES parameter, the performanceof the Volume Siebel Release 4 Consumer Update job was greatly improved. So thatperformance gains such as these can be made, detailed functional analysis of the EIM interfacemust be carried out in order to fully understand the data source.

This parameter should be used ONLY BASE TABLES

2.2.2.2 Ignore Base Tables

This is a Process section parameter.Usage:

INGORE BASE TABLES = BASE_TABLE1, BASE_TABLE2

where BASE_TABLE1 & BASE_TABLE2 are tables mapped to by the EIM table in question.

As an alternative to ONLY BASE TABLES, this parameter can be used to detail specific tablesthat should be ignored by the EIM process.

For clarity, the only instance in which this parameter would be used instead of ONLY BASETABLES is where the majority of the tables must be included. By using this parameter, it will beclear which tables will not be used.

This parameter will generally not be used IGNORE BASE TABLES

Page 9 of 36

IN CONFIDENCE


10/36


2.2.2.3 Ignore Base Columns


Usage:

IGNORE BASE COLUMNS = BASE_TABLE1.BASE_COLUMN1,\BASE_TABLE1.BASE_COLUMN2,\BASE_TABLE2.BASE_COLUMN1

where BASE_TABLE1 and BASE_TABLE2 are mapped to by the EIM table in question andBASE_COLUMN1 & BASE_COLUMN2 are columns on each Base Table.

This parameter can be used to instruct EIM to ignore the processing of columns specified. Thisis most useful for ignoring unused foreign key. This would result in EIM not attempting toresolve the foreign key in question.

This parameter would generally not be used on OneSiebel with the use of ONLY BASE COLUMNS

being preferred.

This parameter will generally not be used IGNORE BASE COLUMNS

Not supported for Merge processes.

2.2.2.4 Default Columns

There are a number of EIM columns where it may be desirable to set a default value for acolumn, if no value is specified.

There is an option to specify these via the ifb file (DEFAULT COLUMN = COLUMN_NAME, value)however this gives EIM an additional SQL statement to execute.

In tuning EIM, the objective is to reduce the work that EIM has to do, therefore fixed & defaultcolumns should be set by the external interface architecture i.e. Ab Initio.

This parameter should not be used DEFAULT COLUMN

2.2.2.5 Fixed Columns

There are a number of EIM columns where it may be desirable to set a fixed value for a column.

There is an option to specify these via the ifb file (FIXED COLUMN = COLUMN_NAME, value)however this gives EIM an additional SQL statement to execute.

In tuning EIM, the objective is to reduce the work that EIM has to do, therefore fixed & defaultcolumns should be set by the external interface architecture i.e. Ab Initio.

This parameter should not be used DEFAULT COLUMN

Page 10 of 36

IN CONFIDENCE


11/36


2.2.2.6 Use Index Hints, Use Essential Index Hints

These are Process section ifb file parameter.

Usage:

USE INDEX HINTS = TRUE|FALSEUSE ESSENTIAL INDEX HINTS = TRUE|FALSE

The default values of these parameters are indicated in bold, meaning that EIM adds indexhints to certain SQL statements. There are two levels of granularity available. By specifying USEINDEX HINTS = TRUE, then additional index hints are used from the default. If no hints aredesired, then set both parameters to FALSE.

EIM does not always suggest the most appropriate execution path and so to determine whetherthe (hard coded) hints improve performance or not, EIM jobs should be performance tested withand without this set to True in order to gauge performance characteristics. Within Volume mostinterfaces (but not all) have been found to perform better with no index hints in place when the

Oracle Cost Based Optimizer (CBO) is used.

The actual index hints themselves cannot be specified just an ability to turn on|off index hintsis provided.

In Volume Release 4 this parameter was used in conjunction with the SESSION SQL = "Altersession set optimizer_mode = all_rows" although as OneSiebel have chosen the Siebel 7.7version, then the database will already be running in Cost Based mode. It is important thatOneSiebel performance testing explores this area.

As Siebel 7.7s now supports (in fact, mandates) the Oracle Cost Based Optimizer, then

the USE INDEX HINTS, USE ESSENTIAL INDEX HINTS must be thoroughly examined byPerformance Testing.

On Volume it has been observed that Siebel often adds inappropriate index hints suchas hinting the use of S_ORG_EXT_U2 to resolve foreign keys to S_ORG_EXT. As thisindex is led by BU_ID which is non-selective in the Volume application (all Accountsbelong to the BT Volume organisation), this is an inappropriate index to use. Byturning hints off, the more appropriate S_ORG_EXT_U1 index (or better still a customindex) may be chosen by Oracle.

Page 11 of 36

IN CONFIDENCE


12/36


2.2.2.7 Session SQL

This parameter has already been introduced in the section on USE INDEX HINTS, USEESSENTIAL INDEX HINTS.

Usage:

SESSION SQL = sql statement to be executed

Note that no terminating semi-colon (;) should be used eg.

SESSION SQL = "Alter session set optimizer_mode = all_rows"

This parameter can be used to execute custom SQL at the beginning of EIMs session (EIM is asingle threaded, single database connection component). Only a single statement can be issued.

Session SQL should NOT be used to modify data in the Siebel schema directly as Siebel doesnot support direct SQL (except for querying).

Session SQL should NOT be used to modify data in the Siebel schema directly as Siebeldoes not support direct SQL (except for querying).

As Siebel 7.7s now supports (in fact, mandates) the Oracle Cost Based Optimizer, thensetting of optimiser mode via SESSION SQL may no longer be appropriate. ThePerformance Team will investigate this area.

2.2.2.8 Using Synonyms

This is an EIM Process section parameter.

Usage:

USING SYNONYMS = TRUE|FALSE

EIMs default operation is to look up Account Synonyms in order to resolve foreign keys toS_ORG_EXT. This requires that EIM does an additional lookup and so setting this parameter toFalse can reduce overall run time. The default is True.

Although OneSiebel may have a requirement to support Synonyms (Trading As names), ODMwill supply Siebel User Keys of the Account as defined in S_ORG_EXT and so setting thisparameter to FALSE will not impact this functionality.

This parameter should be used USING SYNONYMS = FALSE

Page 12 of 36

IN CONFIDENCE


13/36


2.2.2.9 Rollback On Error, Commit Each Pass, Commit Each Table

These parameters determine what happens when a low level EIM error occurs (such as anOracle error).

By default, EIM commits each table and each pass, meaning that if an error occurs, the result

may be partially imported data.

Example The Volume Consumer entity consists of an entry in S_PARTY, S_ORG_EXT,S_ACCNT_POSTN, S_ORG_BU. If a Low Leveloccurs during EIM Import, an entry may beinserted into S_PARTY, S_ORG_EXT but not into other tables, making this record logically invalidwithin OneView.

In order to properly rollback transactions in the event of a Low Levelerror, then this parametershould be used in conjunction with COMMIT EACH PASS = FALSE, COMMIT EACH TABLE =FALSE.

Usage:

ROLLBACK ON ERROR = TRUE|FALSECOMMIT EACH PASS = TRUE|FALSECOMMIT EACH TABLE = TRUE|FALSE

The defaults are indicated in bold.

As per Siebel Alert 290, these settings should be set to perform a single commit (ROLLBACK ONERROR = TRUE, COMMIT EACH PASS = FALSE, COMMIT EACH TABLE = FALSE) for all Mergeand Delete Operations.

Whilst it may immediately appear beneficial to use these settings for Insert and Update

operations, there is a high risk that this would impact competing processes (like SiebelAnalytics) by causing an Oracle Snapshot too old error. This combination of parameters should,therefore, only be used for EIM Merge and Delete processes.

As it does not make sense to use ROLLBACK ON ERROR without using both COMMIT EACH PASS= FALSE and COMMIT EACH TABLE = FALSE, then this parameter should not be included in theheader section of the ifb file (as is the case currently on Volume).

Page 13 of 36

IN CONFIDENCE


14/36


2.2.2.10Update After Failed Insert

If single commit is not implemented (and it should generally not be used for EIM Importprocesses) in OneView, ODMs TrueDiff process should check all tables in a multi-table entity toensure that the Update after failed Insertoperation performs properly.

EIM Import jobs intended to Insert-only into the Base Tables are not generally configured withUPDATE ROWS = FALSE, which may otherwise be expected (to avoid a Data Quality Issuewhere an anomaly has occurred). This is so that if the initial insert fails, then sending the recordagain will result in an update to a partial record. In Volume at present, this is not implementedproperly because the TrueDiff process (the ODM process that identifies which records to insert,update or delete from Siebel based on comparing the data within Siebel with that within ODMsdatastores) does not compare columns in all tables for every entity.

As per Siebel Alert 290, these settings should be set to perform a single commit(ROLLBACK ON ERROR = TRUE, COMMIT EACH PASS = FALSE, COMMIT EACH TABLE =FALSE) for all Merge and Delete Operations.

If single commit is not used (and it should not be), ODMs TrueDiff process should becapable of including all columns in all Base tables involved in each EIM process.

Page 14 of 36

IN CONFIDENCE


15/36


2.2.3IFB Import Process Specific Parameters

This section details those parameters that are specific to EIM jobs of type Import (both Insertand Update operations).

As well as the parameters detailed within this section, those parameters in the Generic sectioncan also be applied.

2.2.3.1 Only Base Columns


Usage:

IGNORE BASE COLUMNS = BASE_TABLE1.BASE_COLUMN1,\BASE_TABLE1.BASE_COLUMN2,\BASE_TABLE2.BASE_COLUMN1

where BASE_TABLE1 and BASE_TABLE2 are mapped to by the EIM table in question andBASE_COUMN1 & BASE_COLUMN2 are columns on each Base Table.

Whereas the generic parameter Ignore Base Columns instructs EIM to ignore specific columns,Only Base Columns instructs EIM to only process those columns specifically listed.

It does not make sense to use both Ignore Base Columns and Only Base Columns in the samejob. Either specify one or the other.

ONLY BASE COLUMNS should be used for every interface to achieve best performance

You must include Foreign Key columns (Insert and Update), required fields (Insert operation)

and any data columns to be loaded.

ONLY BASE COLUMNS should be used for every interface to achieve best performance.

2.2.3.2 Insert Rows, Update Rows

These two Process section parameters impact upon both Data Quality and Performance.

Usage:

INSERT ROWS = BASE_TABLEN, TRUE|FALSEUPDATE ROWS = BASE_TABLEN, TRUE|FALSE

As has already been stated, an EIM table maps to multiple base tables. The use of the aboveparameters controls what type of operation is allowed on each specified table. The default forboth parameters is True.

As an EIM Import job is actually an Upsertoperation. i.e. if the User Keys populated in the EIMtable matches those of an existing row in the Base Tables, then the operation is an Update. Ifthe User Keys populated in the EIM table do not match those of an existing row in the BaseTables, then the operation is an Insert operation.

BT has invested heavily in the ODM layers capability of creating separate Insert and Updatetransactions, based around the unloadfunctionality which is necessary due to multi-mastering.

As such, ODM determines whether the operation shouldbe an insert or an update to Siebelsdata. Therefore, the author believes that every EIM job should specifically use INSERT ROWS

Page 15 of 36

IN CONFIDENCE


16/36


and UPDATE ROWS to specify the operations that EIM is allowedto perform. This will preventany rows from being updated (where User Keys match in error) or inserted (when User Keys arenot located) unintentionally.

Performance can be improved by preventing redundant updates.

INSERT ROWS and UPDATE ROWS should be used to tighten Data Quality and improveperformance.

If single commit is not used, ODMs TrueDiff process should be capable of checking alltables involved in an EIM interface. This would mean, optimally, that an Insert interfacecould specify UPDATE ROWS = FALSE on base tables so that no erroneous updatewould be allowed. If TrueDiff is not fully aware of each Siebel entity, then Insert jobswould need to be set to allow updates although this wouldnt always fix records asintended (self-healing capability).

Although INSERT ROWS = TRUE is the default, it is a good idea to state this specificallyso that the interfaces intention is not ambiguous.

Full use of INSERT ROWS and UPDATE ROWS will mean that any occurrences ofPARTIALLY_IMPORTED should be treated as rightful errors.

2.2.3.3 Net Change

This parameter is used to specify whether columns can be updated to NULL.

Usage:

NET CHANGE = TRUE|FALSE

The default value is shown in bold.

When set to FALSE, Base Table columns will be updated to NULL if the corresponding EIM tablecolumn is NULL and ONLY BASE COLUMNS is used. This is the only way to update a column toNULL.

EIMs default behaviour is to skip processing of NULL columns in the EIM table (except whereONLY BASE COLUMNS is used).

Because of the way in which ODM adapters are designed and owing to the fact thatONLY BASE COLUMNS is used throughput, NET CHANGE = FALSE is used for all Updateprocesses.

Whilst the underlying SQL generated by NET CHANGE = FALSE implements this by useof a large NVL based statement, this has not been seen to cause a significantprocessing overhead.

Page 16 of 36

IN CONFIDENCE


17/36


2.2.3.4 Oracle Insert Append Mode

This is a new feature in Siebel 7.5x.

Usage:

ORACLE INSERT APPEND MODE = TRUE|FALSE

The default is shown in bold.

Using this means that when inserting data into the Base tables, Oracles Insert Append Mode isused. This inserts data into new blocks above the High Water Mark.

Whilst this may improve general insert performance, use of this parameter will lead to increasedfragmentation and increased storage requirements. The Volume Production DBA team do notsupport use of this parameter.

Do not use this parameter as it can lead to increased Base Table fragmentation as wellas increased storage requirements.

Page 17 of 36

IN CONFIDENCE


18/36


2.2.3.5 Trim Spaces

This is an EIM Process section parameter.

Usage:

TRIM SPACES = TRUE|FALSE

EIMs default operation is to right trim white space around data fields:

UPDATE SIEBEL.EIM_ACCOUNTSET IF_ROW_STAT_NUM = 0,

IF_ROW_STAT = 'IN_PROGRESS',T_ORG_EXT__STA = 0,T_ORG_EXT__RID = NULL,T_ORG_EXT__EXS = 'N',T_ORG_EXT__UNQ = 'N',T_ORG_EXT_BU_ID = NULL,T_ORG_EXT_CURRPR = NULL,......

END_DT = DECODE(END_DT,NULL,TO_DATE(NULL),TO_DATE(TO_CHAR(END_DT,'YYYY-MM-DD')||' 00:00:00','YYYY-MM-DD HH24:MI:SS')),

START_DT = DECODE(START_DT,NULL,TO_DATE(NULL),TO_DATE(TO_CHAR(START_DT,'YYYY-MM-DD')||' 00:00:00','YYYY-MM-DD HH24:MI:SS')),

NAME = RTRIM(NAME),LOC = DECODE(LOC,NULL,NULL,RTRIM(LOC)),ALIAS_NAME = DECODE(ALIAS_NAME,NULL,NULL,RTRIM(ALIAS_NAME)),ASGN_PH_AC = DECODE(ASGN_PH_AC,NULL,NULL,RTRIM(ASGN_PH_AC)),ASGN_PH_CC = DECODE(ASGN_PH_CC,NULL,NULL,RTRIM(ASGN_PH_CC)),BASE_CURCY_CD = DECODE(BASE_CURCY_CD,NULL,NULL,RTRIM(BASE_CURCY_CD)),......URL = DECODE(URL,NULL,NULL,RTRIM(URL)),VAT_REGN_NUM = DECODE(VAT_REGN_NUM,NULL,NULL,RTRIM(VAT_REGN_NUM)),ROW_STATUS = DECODE(ROW_STATUS,NULL,NULL,RTRIM(ROW_STATUS)),COMMENTS = DECODE(COMMENTS,NULL,NULL,RTRIM(COMMENTS)),STATUS = DECODE(STATUS,NULL,NULL,RTRIM(STATUS))

WHERE (IF_ROW_BATCH_NUM = 1)

Not all columns are trimmed but this cannot be controlled.

By setting this parameter to False, then this operation is not performed during the InitializeInterface Tables EIM pass. Whilst this theoretically slims down EIM processing, the gains to byhad by setting TRIM SPACES = FALSE will be negligible when compared to the impact of otherEIM parameters.

Additionally, both the eVolve and Volume development teams have expressed that they preferto retain the default of TRUE.

TRIM SPACES should be left at the default value of TRUE. It adds no value to specifythis explicitly in the ifb.

Page 18 of 36

IN CONFIDENCE


19/36


2.2.4EIM Delete Process Configuration

This section details parameters and techniques as well as important Technical Information fromSiebel Support, which is specific to EIM jobs of type Delete and Merge.

This section is structured differently than other like sections within this document as this walksthe reader through each type of Delete job, namely:

o Delete All Rows

o Delete Exact

o Delete Matches

2.2.4.1 Delete All Rows

This Process section parameter is used to specify that all rows mapped to by the interface tableshould be deleted. This parameter should therefore be used with extreme caution!

Usage:

DELETE ALL ROWS = TRUE|FALSE

The default value is False.

No other parameters are really be valid as all data in the specified Base Table will be deleted.

Once run, the User Keys of records deleted by the process will be loaded into the EIM tables andcan be archived or deleted.

This parameter should be used with extreme caution! It is not particularly relevant toProduction Processes.

2.2.4.2 Delete Exact

This Process section parameter is used to specify that EIM should delete rows in the Base Tableswhose User Keys match those of records loaded into the Interface Tables.

Usage:

DELETE EXACT = TRUE|FALSE

As EIM only needs the User Keys, then only the User Keys should be loaded.

This operation is primarily concerned with deleting rows from the primary Base Tables mappedto by the given EIM table. Deletion of Child Rows can be controlled by use of ONLY BASETABLES.

DELETE ROWS can be used to specify from which tables deletes are allowed. The default valueis True.

Usage:

DELETE ROWS = BASE_TABLEN, TRUE|FALSE

where BASE_TABLEN is a Base Table mapped to by the specific EIM table.

UPDATE ROWS is also valid for this job.

Page 19 of 36

IN CONFIDENCE


20/36


Setting DELETE ROWS and/or UPDATE ROWS to FALSE should be done with extreme caution asit may result in dangling foreign keys i.e. foreign keys that no longer point to a valid record.

Once run, no additional data is populated into the interface tables and so the original rows thatwere loaded into the EIM table may be archived or deleted.

Setting DELETE ROWS and/or UPDATE ROWS to FALSE should be done with extremecaution as it may result in dangling foreign keys.

DELETE EXACT is the safest method of deleting data in bulk from Siebel and so is theadopted delete method at BT.

2.2.4.3 Delete Matches

This Process section parameter is used to specify that EIM should delete data from the Base

Tables that matches the where clause specified in the ifb.

e.g.

DELETE MATCHES = EIM_ACCOUNT, (CUST_STAT_CD = Inactive)

In the example above, all rows whose CUST_STAT_CD = Inactive would be deleted.

Once run, this EIM job will have written the User Keys of the records deleted into the EIMtables. This data can then be either archived or deleted.

There is no concept of Batchsize in a Delete Matches job all rows that meet the criteria are

processed in a single run.

There is no concept of batchsize in a Delete Matches job. Therefore, several jobs maybe required to delete the intended records, perhaps grouping them by a data range ofrange of Unique Identifiers.

Page 20 of 36

IN CONFIDENCE


21/36


2.2.4.3.1Important Information from Siebel Support

There are two important Siebel Alerts need to be kept in mind for all Delete and Merge jobs.TheseAlerts should be implemented (subject to confirmation/approval from Siebel) in all Deleteand Merge jobs.

o There is a Siebel Alert (Alert 306: Specify $NUM_IFTABLE_LOAD_CUTOFF = 0 for

EIM merge and delete) that states that you must specify a parameter($NUM_IFTABLE_LOAD_CUTOFF=0) otherwise EIM will be blindto some metadatarelating to foreign keys. This could clearly cause DQ issues as it may leave danglingforeign keys. Whilst this parameter should be able to be specified in the IFB Headersection as detailed within the Alert, this was found to repeatedly cause a core dump onthe Solaris Application Server. A Service Request was logged on Siebel SupportWeb andthe workaround of specifying this via Extended Parameters on the command line wasfound.

o Orphaned records may be created in the current configuration in the event that an

Oracle error (or other similarly severe error) occurs whilst the EIM job is running. This isas a result of another Siebel Alert (Alert 209: EIM Merge and Delete processesshould use only one commit per batch to avoid data integrity issues) and, again,may cause DQ issues. To prevent orphans from being created the following settingsshould be specified in the Process section:

COMMIT EACH TABLE = FALSECOMMIT EACH PASS = FALSEROLLBACK ON ERROR = TRUE

(If a severe error does occurs, the entire batch is rolled back but records are left in theEIM table with an IF_ROW_STAT of IN_PROGRESS because Siebel does a commit after

preparing the interface tables (see Trim Spaces section within this document for anexample EIM log file showing this). Therefore, if records are left with an IF_ROW_STAT =

IN_PROGRESS after all EIM jobs have completed, then it is clear that a low-level errorhas occurred.)

Use the following parameters for all Delete and Merge jobs:

COMMIT EACH TABLE = FALSECOMMIT EACH PASS = FALSEROLLBACK ON ERROR = TRUE

$NUM_IFTABLE_LOAD_CUTOFF = 0 should be specified for all Delete and Merge jobs.ODMs GSL (or any other calling process) must be able to specify this parameter and, infact, it would be wise that the GSL could build a flexible Server Manager command linemode call so that Extended Parameters and other specialised attributes can beincluded. This would require that the current ODM EIM_DETAIL table and surroundingfunctionality be enhanced.

Including $NUM_IFTABLE_LOAD_CUTOFF = 0 within the IFB files Header section maycause core dumps on the Solaris Application Server. To workaround this, specify theparameter via Extended Parameters on the command line.

Page 21 of 36

IN CONFIDENCE


22/36


2.2.5EIM Merge Process Configuration

In a Merge job, victim and survivorrecords are specified in the EIM tables. TheIF_ROW_MERGE_ID on the victim record is populated with the row_id (in the Interface Table) ofthe survivorrecord e.g.

IF_ROW_BATCH_NUM ROW_ID NAME IF_ROW_MERGE_ID

100 1 OneSiebel

100 2 OneView

100 3 Volume 2

100 4 Account Managed Platform 1

100 5 Big Data 2

In the example above:

OneSiebeland OneVieware survivor records (no IF_ROW_MERGE_ID specified).

Volume and Big Data are merged into OneView. Account Managed Platform is merged into OneSiebel.

A Merge job does not have any specific parameters, however there are a number ofrecommendations and observations:

o A Merge job only needs the User Keys and IF_ROW_MERGE_ID on the victim record to

be specified.o Siebel Alerts 306 and 209 (as detailed in the section entitled Important Information from

Siebel Support) should be implemented.o UPDATE ROWS is the only other relevant setting. This controls whether foreign keys in a

child table are updated to point towards the survivoror not. Setting this parameter to

FALSE can cause dangling foreign keys and is not recommended.o The batchsize used should be considerably smaller than the batchsize usually adopted

e.g. on Volume the standard EIM batchsize is 10,000 but a batchsize of just 2000 is usedfor Address Merge operations.

o Whilst EIM Merge can theoretically be run in parallel if records are victim records are

mutually exclusive across all batches, this is not recommended Oracle locks occurredduring Performance Testing on a number of occasions and as database resource isconsiderable even for a single EIM stream, then running EIM merge in parallel is likely tosaturate IO resources very quickly and result in an own goal.

o As in the example above if more than 1 victim is to be merged into survivor, this should

be done in the same batch (if mutual exclusivity can be guaranteed). Merge operationsshould NOT be split into batches of 2 containing just a survivor and a victim (as was the

case on Volume at one point).

Page 22 of 36

IN CONFIDENCE


23/36


o Volume is currently experiencing difficulties with EIM Merge jobs due to Performance and

Data Quality Issues.o Performance

Due to the large number of relationships within the Siebel database, notall relevant foreign key columns on the database have been indexed, outof the box. This invokes Full Table Scans which are not viable againstOneViews largest tables.

Siebel Expert services are currently engaged to resolve the issues and aresolution is forthcoming.

o Data Quality

A Siebel documented issue whereby duplicate columns are left behind inintersection tables was recently experienced by the Volume team. Whilstthis issue is reportedly resolved in 7.7, it merits that due care be taken.

Use the following parameters for all Delete and Merge jobs:

COMMIT EACH TABLE = FALSE

COMMIT EACH PASS = FALSEROLLBACK ON ERROR = TRUE

$NUM_IFTABLE_LOAD_CUTOFF = 0 should be specified for all Delete and Merge jobs.ODMs GSL must be able to specify this parameter and, in fact, it would be wise thatthe GSL could build a flexible Server Manager command line mode call so thatExtended Parameters and other specialised attributes can be included. This wouldrequire that the current Volume EIM_DETAIL table and surrounding functionality beenhanced.

The Volume team encountered Data quality issues due to an out-of-the-box Siebelissue. Whilst this has been reportedly fixed in 7.7, due care needs to be taken.

Page 23 of 36

IN CONFIDENCE


24/36


2.2.6Other Parameters

The following table includes details on other EIM parameters for completeness. Parametersrequiring attention include description text in bold.

N.B. This guide was prepared with reference to both Siebel 7.5.2 and Siebel 7.7.

Parameter Description

Header Section Only ParametersOther parameters can generally be specified in either the header or the process section.

CONNECT ODBC data source not required as this is set using dataretrieved from the Gateway Server

LOG TRANSACTIONS TOFILE

A Siebel Remote related parameter this should be set toFALSE in the header section of all IFB files whilst SiebelRemote is not being used. All Siebel Remote relatedparameters will need to be reviewed if Siebel Remote is to bedeployed.

USERNAME This parameter should not be set. Rather this information isretrieved from the Gateway.

PASSWORD This parameter should not be set. Rather this information isretrieved from the Gateway.

TABLEOWNER This parameter should not be set. Rather this information isretrieved from the Gateway.

PROCESS This defines the process to invoke when EIM is started withthe IFB file. This should not be used as it is preferred thatthe process name be specified externally i.e. specified on thecommand line when invoking EIM.

Generic Process Parameters

BATCH The batch or batch range to be processed.

INCLUDE Used in Shell Processes to automate the running of a numberof processes automatically. Shell Processes should generallynot be used, as scheduling of jobs is generally mastered out-with the EIM environment and specific EIM processes toinvoke are specified on the Server Manager command line.Default.ifb, which can be found within the Application Serversadmin directory, contains examples of calling Shell Processes.

LOG TRANSACTIONS A Siebel Remote related parameter. This parameter is onlyrelevant ifDocking:Transaction Logging System Preference isset to True. Not relevant to either OneSiebel r1.0 or OneViewr6.0.

SKIP BU_ID DEFAULT The default value of False should be retained to specify that

EIM uses the default value defined within the Repositorywhen setting virtual null keys.

TABLE This parameter specifies which EIM table is to be processedby the EIM job.

TRANSACTION SQL The optional parameter can be used (similarly to SESSIONSQL) to run ad-hoc SQL after each EIM SQL statement iscommitted. To date no specific use has been found for thisparameter.

TYPE Specifies whether the EIM process is of type IMPORT(includes BT Update and Import processes), EXPORT, DELETE,MERGE or SHELL. Shell processes should not generally beused. Default.ifb, which can be found within the Application

Servers admin directory, contains examples of calling ShellProcesses.

Page 24 of 36

IN CONFIDENCE


25/36


Parameter Description

Header Section Only ParametersOther parameters can generally be specified in either the header or the process section.

UPDATE STATISTICS As this parameter only applies to DB2, it should not beused within BT (BT uses Oracle).

Import Process Parameters

COMMIT OPERATIONS A Siebel Remote related parameter. Not relevant to eitherOneSiebel r1.0 or OneView r6.0.

FILTER QUERY This option can be used to specify which rows in the EIMtable are to be processed e.g. FILTER QUERY =(CUST_STAT_CD = Active). As BT has invested heavily inexternal preparation of data for EIM (Ab Initio graphs), thenthere should be no need to use a parameter like this.

MISC SQL This parameter should be used for initial loads only theauthor will review its applicability for OneSiebel at a laterdate.

ATTACHMENT DIRECTORY This parameter is used to specify the location of attachmentsto be loaded into Siebel. EIM must have both read and writeaccess to this directory. (Default = SIEBEL_HOME/INPUT)

Export Process ParametersNote: The EIM Export process is not generally used within BT but

CLEAR INTERFACE TABLE The default for this parameter is True meaning that the EIMexport process deletes all existing rows in the interface tableprior to starting the export process. Use of this parameter isonly possible where no other users/processes are sharing thespecific EIM table to be used.

EXPORT ALL ROWS Synonymous with DELETE ALL ROWS

EXPORT MATCHES Synonymous with DELETE MATCHES

Delete Process Parameters

CASCADE DELETE ONLY Used to specify whether child records are deleted (TRUE) ortheir FK is updated to NULL (TRUE). The default is False.

DELETE SKIP PRIMARY This parameter specifies whether EIM should perform acascade update to the primary child column. The Defaultvalue is TRUE.

Merge Process Parameters

SET BASED LOGGING A Siebel Remote related parameter. This parameter is onlyrelevant ifDocking:Transaction Logging System Preference isset to True. Not relevant to either OneSiebel r1.0 or OneViewr6.0.

Page 25 of 36

IN CONFIDENCE


26/36


3 Running EIM

3.1 Data

Reviews of several Siebel implementations at BT has revealed that there has, on occasion, beensome confusion as to what data needs to be populated for each different job type. This sectionclarifies what data is required for each type of EIM job.

Import (Insert) To insert a new record (into Primary or Child Base Tables), User

Keys, Required Fields, Data columns and System Columns need to be populated. Import (Update) To update an existing record just the User Keys, specific Data

Columns to update and System Columns need to be populated. Delete To Delete a record using Delete Exact, just the User Keys and System Columns

need to be populated. Merge To Merge records, just User Keys and System fields need be populated.

The following definitions apply to the above:

User Keys the combination of columns used by EIM to uniquely identify a row

(synonymous with Compound Keys). Required Fields those fields that must be populated when creating a new record.

Data Columns regular columns in the EIM table that map to columns in the Base

Tables. System Columns CREATED, CREATED_BY, LAST_UPD, LAST_UPD_BY, ROW_ID,

IF_ROW_BATCH_NUM, IF_ROW_MERGE_ID, IF_ROW_STAT, MODIFICATION_NUM,CONFLICT_ID.

The following are mandatory System Columns:

IF_ROW_BATCH_NUM controls the Batch to which the record belongs

ROW_ID used to uniquely identify records within a batch. The combination of

IF_ROW_BATCH_NUM and ROW_ID must be unique. IF_ROW_MERGE_ID (for EIM Merge processes only) set this to the ROW_ID within

the batch that the record will be merged into. IF_ROW_STAT set with a meaningful status, such as FOR_IMPORT, FOR_UPDATE.

This will be set to IN_PROGRESS by EIM during processing.

The other System Columns are defaulted on a table level (Oracle default value) in someinstances not all instances. It is, however, good practice to set these fields explicitly so thatrecords in EIM tables can be easily identified. Note that the values populated in these otherSystem Columns will not influence the corresponding Base table records these columns areset by the EIM process.

Only load the data required by the interface.

Page 26 of 36

IN CONFIDENCE


27/36


3.2 IFB File and EIM Processes

This section details guidelines that should be adopted by BT Siebel interface development teamsrelating to how to logically collate multiple EIM processes into ifb files.

In general, there should be a single ifb file per entity, with multiple processes being maintainedwithin that ifb file e.g.:

A file called onesiebel_customer.ifb should contain all Customer related EIM processes forOneSiebel, such as:

Import_OneSiebel_Customer

Update_OneSiebel_Customer

Cease_OneSiebel_Customer

Delete_OneSiebel_Customer

Merge_OneSiebel_Customer

Note: the Cease process is a logical delete of the Customer whilst the Delete process is aphysical delete.

By including multiple EIM processes for a given entity within a single IFB file, then configurationmanagement can be controlled at an appropriate level.

The calling process (Ab Initio adapter, script etc) should specify the process to run (and otherparameters) via the command line e.g.:

$SIEBEL_ROOT/bin/srvrmgr -g $GATEWAY:$GPORT -e $ENTERPRISE -s $APPSERVER -u$SUSER -p $SPASS -c "run task for component eim with config="$IFB", traceflags=3,errorflags=1, SQLFlags=8, ExtendedParams="BatchRange=$BATCH", Process="$PNAME""

(UNIX escape sequences have been omitted for clarity).

$SIEBEL_ROOT the path where the Siebel Application Server is installed

$GATEWAY the IP address (or hostname) of the Siebel Gateway Server

$GPORT the port on which the Gateway Server is running

$ENTERPRISE the name of the (Logical) Siebel Enterprise Server

$APPSERVER the name of the (Logical) Siebel Application Server

$SUSER - the Siebel application user to be used to run the EIM process under

$SPASS the password of the aforementioned user

$IFB the full path and file name of the IFB file to be used

$BATCH the batch or batch range to be processed by this EIM process (the use ofbatch ranges minimizes the overheads associated with starting an EIM process)

$PNAME the name of the EIM Process within the IFB file to invoke

As EIM name-value parameter pairs and extended parameters may need to be controlled on anad-hoc basis, then the calling process should be configurable in the way in which the commandline is structured and sent to Server Manager.

By including multiple EIM processes for a given entity within a single IFB file, thenconfiguration management can be controlled at an appropriate level. This guidelineshould be adopted.

Page 27 of 36

IN CONFIDENCE


28/36


3.3 Scheduling

This section discusses some factors effecting the running of EIM as follows:

As EIM operates on batches of data, then use of database server resources (in terms of

redo, archiving and undo) can be optimised by use of different batch sizes. EIM runs as a single threaded process against the database server, therefore it can be run in

parallel to increase throughput. Different scheduling options.

3.3.1Degree of Parallelism/Layout

The GSL supports the running of EIM batches in parallel. This means that concurrent EIMstreams are fired off at the same time, working against different ranges of batchessimultaneously.

During Volume 4 Performance Testing determined the optimal degree of parallelism by varyingthe number of concurrent streams and analysing throughput. A limit comes about when thecompetition for I/O results in a degraded throughout as more and more streams are added. If a

job is more CPU bound than IO bound, then parallelism can generally be increased (e.g. to 8-way).

By determining the optimal degree of parallelism, then better EIM throughput can be achievedalthough competing processes also need to be catered for.

Performance Testing should vary the Layout to achieve the highest throughput.

3.3.2Batchsize

Similarly to Layout, EIMs best throughput can be found by varying batchsize during testing.

For Volume, the optimal batchsize is generally 10,000 records per batch although smallerbatches must be used for Merge and Delete jobs to reduce the risk of causing competingprocess to fail (e.g. Snapshot too old errors). Such a large optimal batchsize is possible whenthe buffer cache is large.

Performance Testing should vary the Batchsize to achieve the highest throughput.

3.3.3EIM Streaming Configurations

Whilst Converge determined that when spawning more than 5 EIM stream this needed to bespread across multiple Siebel Application Servers, such behaviour has yet to be exhibited onVolume.

Volume runs all EIM processes from a single Siebel Application Server.

Page 28 of 36

IN CONFIDENCE


29/36


Volume 4s Performance was consistent in determining that I/O on the database server was thesingle biggest bottleneck (for loads into tables not populated, the single biggest bottleneck isgenerally with CPU). This therefore did not give rise to performing testing the spawning of EIMfrom multiple Siebel Application Servers.

Performance Testing should determine whether increasing the number of physicalSiebel Application Servers enhances EIM performance. If this behaviour is not verified,the multi-server EIM streaming configuration should not be adopted.

3.3.4Control-M Scheduling

In Volume, the current schedule (maintained and operated from Control-M) is based upon 3streams and looks something like this:

Process 1 Process 4

UnloadEntity 1

Process 2

Process 3Unload

Entity 2

Unload

Entity nProcess n

Process 5

Process n

Stream 1 Stream 2 Stream 3

In this schedule, EIM table loads happen at the tail end of the on-line day (8pm onwards) thishas not yet caused any performance impact upon the on-line processes.

In the excerpt above, multiple streams run in the schedule:

Stream 1 runs EIM processes (each of which is running in parallel). There are dependencies

between the processes between the streams Stream 2 runs EIM processes as in Stream 1.

Stream 3 runs unloads with dependencies with the EIM processes in the other two streams

a table isnt unloaded until all EIM processing against that table is complete (S_ORG_EXT is,therefore, unloaded last in Stream 3 as it is accessed by mostEIM jobs).

In this approach jobs that process different EIM and Base tables can be run simultaneously, withunloads going on in parallel dependencies between EIM streams and unloads must beenforced. This maximises use of the batch window, as long as the database server can copewith the demands put upon it.

Page 29 of 36

IN CONFIDENCE


30/36


Where a process is determined to run in isolation with maximum throughput at a certain degreeof parallelism during testing, this may need to be reduced if run in parallel with other EIMprocesses and alongside unloads.

Page 30 of 36

IN CONFIDENCE


31/36


3.4 EIM Table Maintenance

Due to the number of operations executed against the EIM tables, then the following table

maintenance is recommended:

Truncate EIM tables regularly This has many advantages including releasing unused

storage and removing fragmentation. Analyse EIM tables after each table load As EIM executes many SQL statements in

Cost Based Mode, then table (and index) statistics assist Oracle in choosing anappropriate execution path. Regular statistics gathering on the base tables should alsobe done for the same reasons.

Storage Parameters given that Row Chaining can occur on some EIM tables, then

the project DBAs should consider setting a high PCTFree value.

Page 31 of 36

IN CONFIDENCE


32/36


4 Volume 4 Lessons Learned

This section outlines some of the key lessons learned during Performance Test and Deployment

of Volume 4 as well as from an exercise to overhaul all EIM interfaces in Production on Volume.

Changes made to the ODM Ab Initio components have already been integrated into Volume 6and AMP High Level Designs and so are only mentioned here for completeness.

4.1 Enhancements to EIM Processes

This section contains some of the enhancements made to EIM Processes that should be adoptedby subsequent BT Siebel projects.

4.1.1Redundant Updates

Removal of redundant operations within the EIM process is the single biggest opportunity (withpossible exception of ensuring best possible sorting) to make performance improvements

This is best illustrated by example.

The original Update_Volume_Consumer process looked similar to that shown below:

[Update_Volume_Consumer]TYPE = IMPORTBATCH = $BatchRangeTABLE = EIM_ACCOUNTONLY BASE TABLES = S_PARTY, S_ORG_EXT, S_ORG_BU, S_ACCNT_POSTNINSERT ROWS = S_PARTY, FALSE

INSERT ROWS = S_ORG_EXT, FALSEINSERT ROWS = S_ORG_BU, FALSEINSERT ROWS = S_ACCNT_POSTN, FALSENET CHANGE = FALSEONLY BASE COLUMNS = S_PARTY.PARTY_UID, \

S_PARTY.PARTY_TYPE_CD, \S_PARTY.NAME, \S_PARTY.ROOT_PARTY_FLG, \S_PARTY.GROUP_TYPE_CD, \S_ORG_EXT.NAME, \S_ORG_EXT.LOC, \S_ORG_EXT.BU_ID, \S_ORG_EXT.PAR_ROW_ID, \S_ORG_EXT.ACCNT_FLG, \S_ORG_EXT.ACTIVE_FLG, \...S_ORG_EXT.ACCNT_TYPE_CD, \

S_ORG_EXT.OU_TYPE_CD, \S_ORG_EXT.INTEGRATION_ID, \S_ORG_EXT.OU_NUM, \S_ORG_EXT.MASTER_OU_ID, \S_ORG_EXT.BASE_CURCY_CD, \S_ORG_BU.ORG_ID, \S_ORG_BU.ORG_LOC, \S_ORG_BU.ORG_NAME, \S_ORG_BU.BU_ID, \S_ORG_EXT.CURR_PRI_LST_ID, \S_ACCNT_POSTN.OU_EXT_ID, \S_ACCNT_POSTN.POSITION_ID, \S_ACCNT_POSTN.ROW_STATUS, \S_ORG_EXT.PR_POSTN_ID, \S_ORG_EXT.X_BT_FOLLOW_UP_RATING, \

Page 32 of 36

IN CONFIDENCE


33/36


The following redundant updates were removed:

S_ORG_BU, S_ACCNT_POSTN these tables are populated with default relationships by the

adapter. As this data never changes, then redundant updates to these large tables wereremoved, a large improvement was made. This was done by removing the aforementionedtables from the ONLY BASE TABLES parameter.

S_ORG_EXT.MASTER_OU_ID as the Consumers Master account is itself (a self-reference),

and this can never change, then the Master_Ou_Id column was able to be removed. Being aForeign Key to S_ORG_EXT, gains in the end to end EIM runtime forUpdate_Volume_Consumer will be able to be almost halved (this change is yet to beimplemented in Production). This was done by removing S_ORG_EXT.MASTER_OU_ID fromthe ONLY BASE COLUMNS parameter.

The following index changes were made:

Siebel was found to hint the inappropriate S_ORG_EXT_U2 index (led by BU_ID which is

non-selective in Volume) to resolve relationships to S_ORG_EXT. By removing index hints(and invoking CBO), Oracle was able to start using the more appropriate S_ORG_EXT_U1index (led by name). A marginal gain was realised through this change. A new index hasbeen designed that will further improve this interface.

Two indexes were also identified to reduce IO. Detailed Oracle configuration is not included

within this document.

Once all these changes have been implemented in live, then this processs typical throughputfor bulk update operations will have increased several folds.

Ensure that update processes do not update tables or resolve foreign keysunnecessarily.

4.1.2Splitting Updates

Another way to yield big gains in to split update records out into multiple update processes asopposed to a single update process.

All (previous) ODM EIM update jobs simply detected that something within the record hadchanged and provided a full update record to EIM. It is inefficient to resolve foreign keyscolumns that have not changed redundant updates.

Changes to the Update_Volume_Asset are currently in progress on Volume and this serves as a

good way to illustrate the concept of split updates:

The original Asset Update process looked like this:

[Update_Volume_Asset]TYPE = IMPORTTABLE = EIM_ASSETBATCH = $BatchRangeUSE INDEX HINTS = FALSEUSE ESSENTIAL INDEX HINTS = FALSESESSION SQL = "Alter session set optimizer_mode = all_rows"ONLY BASE TABLES = S_ASSETINSERT ROWS = S_ASSET, FALSENET CHANGE = FALSEONLY BASE COLUMNS = S_ASSET.ASSET_NUM, \

Page 33 of 36

IN CONFIDENCE


34/36


S_ASSET.PROD_ID, \S_ASSET.BU_ID, \S_ASSET.AUTO_SR_FLG, \S_ASSET.COMP_UND_WRNTY_FLG, \S_ASSET.PER_ADDR_ID, \S_ASSET.PAR_ASSET_ID, \S_ASSET.ROOT_ASSET_ID, \S_ASSET.BILL_ACCNT_ID, \S_ASSET.X_BT_HIERARCHY_CD

Examination reveals that there are 4 FKs to heavily populated tables that are resolved by thisupdate process, namely, PER_ADDR_ID (FK to S_ADDR_PER), PAR_ASSET_ID andROOT_ASSET_ID (FKs to S_ASSET) and BILL_ACCNT_ID (FK to S_ORG_EXT).

Functional analysis determined that the Asset Hierarchy would neverchange and that Bill

Account and Address would only ever rarelychange. By drilling down in the ODM TrueDiffprocess to produce separate individual outputs in the event that one of these FKs actually doeschange, then performance can be dramatically improved.

The result is that a single Asset Update process is being split to produce the following EIMupdate processes:

[Update_Volume_Asset] this process is used where only attributes have changed. The

4 FKs were removed from this process. [Update_Volume_Asset_Address] specifically designed for Volume 5 where a large

number of Assets addresses would change, this process only updates Asset Attributes andAsset Address. The Asset Hierarchy and Bill Account FKs have been removed from this

process. [Update_Volume_Asset_All] as it was determined that Asset Hierarchy should never

change and that Bill Account would only change rarely, these changes were groupedtogether. Due to the way in which the TrueDiff code works with split updates, then thisprocess would update Asset Attributes and allFKs (read on).

The Ab Initio code for the TrueDiff Update splits works as follows:

1. If there are changes to Asset Attributes, then the recordis flagged for[Update_Volume_Asset].

2. If there is a change to the Asset Address, then the recordis flagged for[Update_Volume_Asset_Address]. By flagging the record for this process, any flag that wasset previously is over-written.

3. If there is a change in the Asset Hierarchy or the Bill Account, then the recordis flagged forthe [Update_Volume_Asset_All] process. Any previous flag that was set is over-written. Inthis case all attributes and FKs are updated.

In general EIM update performance can be improved by only updating FKs that actually need tobe updated. This is particularly prevalent when the foreign key is to a heavily loaded Base tablesuch as S_ORG_EXT.

Split out update processes into separate processes so that foreign keys are not resolvedunnecessarily.

Page 34 of 36

IN CONFIDENCE


35/36


4.1.3Hints and Optimiser Mode

Loads into the already heavily populated S_ORG_EXT table were improved by turning of indexhints and altering the sessions database optimiser to Cost Based mode (from the default of

Rules based):

USE INDEX HINTS = FALSEUSE ESSENTIAL INDEX HINTS = FALSE

SESSION SQL = "Alter session set optimizer_mode = all_rows"

This technique was used so that Oracle would pick a custom index (described in the nextsection) using RBO to improve throughput.

As OneSiebel has opted to use Siebel 7.7, then execution path optimisation will work differently.The Performance Test team will investigate this.

Analyse EIM performance using different Oracle optimizer modes with and withoutSiebel generated index hints.

4.2 Database Configuration

4.2.1Indexes

Most of the User Key indexes in Siebel do not contain the ROW_ID column meaning that 2 I/Osare often required where just one could suffice. As I/O was consistently the major contributingfactor to less-than-desired throughput, then this technique can be used to reduce I/O intensive

operations.

e.g.

UPDATE siebel.EIM_ACCOUNT ITSET...T_ORG_EXT_MASTER =(SELECTMIN(BT.ROW_ID)

FROM siebel.S_ORG_EXT BTWHERE (BT.BU_ID = IT.MASTER_ACCNT_BI AND

BT.NAME = IT.MASTER_ACCNT_NAME AND

BT.LOC = IT.MASTER_ACCNT_LOC)),...

WHERE (IF_ROW_BATCH_NUM = 21 ANDIF_ROW_STAT_NUM = 0 ANDT_ORG_EXT__STA = 0);

In the SQL above, the S_ORG_EXT_U1 index (S_ORG_EXT(NAME, LOC, BU_ID, CONFLICT_ID))

is chosen by the Cost Based Optimizer. This is an improvement on the default Siebel executionpath where the S_ORG_EXT_U2 index (S_ORG_EXT(BU_ID, NAME, LOC, BU_ID, CONFLICT_ID))

is hinted, but still not good enough.

In this situation an IO must be made to the S_ORG_EXT_U1 index in order to locate the

matching row in Oracle. The index returns the internal Oracle RowId of the row in question, but

Page 35 of 36

IN CONFIDENCE


36/36


then has to do another IO on the S_ORG_EXT table to get the Siebel Row_id required by thestatement.

By creating a custom index on S_ORG_EXT(NAME, LOC, BU_ID, ROW_ID), then the query is

optimised as it need only make a single IO on S_ORG_EXT and its indexes.

Also, as this new index simply overlays an existing Siebel Uxindex (but replaces theunnecessary CONFLICT_ID with the more useful ROW_ID) the risk to abnormal GUI operation isvery low.

Analyse long running EIM SQL statements and add/remove indexes as appropriate.Experienced DBAs are required to assist with this task.

Full regression testing is required for any changes to indexes. Indexes should only bealtered after verification by experienced DBAs.

4.3 ODM Adapter Functionality

ODM enhancements have already been incorporated into OneView 6 and OneSiebel ODM HighLevel Designs and are not discussed in detail here.

Several performance enhancements were made to the ODM Ab Initio adapters during Release 4deployment. These changes were as follows:

Sorting by sorting the data at the input of the adapter (before any throttle

functionality) as well as within the GSL itself, then the data was able to be sorted as best

as possible. Data sorting was the single biggest win (with the exception of removingredundant updates) during Release 4.

Serial Batch Calculation although creating batches within the adapter serially as

opposed to in parallel increases processing time in the Ab Initio environment, the gainsobserved within EIM are worth the sacrifice. These gains are a result the improvedsorting that serial calculation provides.

Filter before throttling the filter functionality was used so that throttled data one

night would continue where it left offthe previous night. This meant that the data wassorted more tightly.

< End of Document >

7078734 siebel eim best practise1

Documents