enabling an application for tivoli enterprise data...

Enabling an Application forTivoli Enterprise Data WarehouseVersion 1 Release 1

GC32-0745-00

Enabling an Application for Tivoli Enterprise Data Warehouse

Copyright Notice

© Copyright IBM Corporation 2002. All rights reserved. May only be used pursuant to a TivoliSystems Software License Agreement, an IBM Software License Agreement, or Addendum for TivoliProducts to IBM Customer or License Agreement. No part of this publication may be reproduced,transmitted, transcribed, stored in a retrieval system, or translated into any computer language, in anyform or by any means, electronic, mechanical, magnetic, optical, chemical, manual, or otherwise,without prior written permission of IBM Corporation. IBM Corporation grants you limited permissionto make hardcopy or other reproductions of any machine-readable documentation for your own use,provided that each such reproduction shall carry the IBM Corporation copyright notice. No otherrights under copyright are granted without prior written permission of IBM Corporation. Thedocument is not intended for production and is furnished “as is” without warranty of any kind. Allwarranties on this document are hereby disclaimed, including the warranties of merchantabilityand fitness for a particular purpose.

U.S. Government Users Restricted Rights—Use, duplication or disclosure restricted by GSA ADPSchedule Contract with IBM Corporation.

Trademarks

IBM, the IBM logo, Tivoli, the Tivoli logo, AIX, Tivoli Enterprise, Tivoli Enterprise Console, and TMEare trademarks or registered trademarks of International Business Machines Corporation in the UnitedStates, other countries, or both.

Lotus® is a registered trademark of Lotus Development Corporation and/or IBM Corporation.

Microsoft, Windows, and Windows NT, and the Windows logo are trademarks of MicrosoftCorporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of SunMicrosystems, Inc. in the United States and other countries.

Other company, product, and service names may be trademarks or service marks of others.Notices

References in this publication to Tivoli Systems or IBM products, programs, or services do not implythat they will be available in all countries in which Tivoli Systems or IBM operates. Any reference tothese products, programs, or services is not intended to imply that only Tivoli Systems or IBMproducts, programs, or services can be used. Subject to valid intellectual property or other legallyprotectable right of Tivoli Systems or IBM, any functionally equivalent product, program, or servicecan be used instead of the referenced product, program, or service. The evaluation and verification ofoperation in conjunction with other products, except those expressly designated by Tivoli Systems orIBM, are the responsibility of the user. Tivoli Systems or IBM may have patents or pending patentapplications covering subject matter in this document. The furnishing of this document does not giveyou any license to these patents. You can send license inquiries, in writing, to the IBM Director ofLicensing, IBM Corporation, North Castle Drive, Armonk, New York 10504-1785, U.S.A.

Contents

Preface . . . . . . . . . . . . . . . vWho should read this guide . . . . . . . . . vWhat this guide contains . . . . . . . . . . vPublications . . . . . . . . . . . . . . vi

Tivoli Enterprise Data Warehouse library . . . viRelated publications . . . . . . . . . . viAccessing publications online . . . . . . . viiOrdering publications . . . . . . . . . . viiProviding feedback about publications . . . . viii

Contacting customer support . . . . . . . . viiiConventions used in this book. . . . . . . . viii

Chapter 1. Overview of Tivoli EnterpriseData Warehouse . . . . . . . . . . . 1Central data warehouse schema overview . . . . 3

Chapter 2. Planning for the extract,transform, and load process . . . . . . 5Filling out the enablement template . . . . . . 7

Component types . . . . . . . . . . . . 7Components . . . . . . . . . . . . . 8Relationship Type, Relationship Rule, andComponent Relationship . . . . . . . . . 8Component attributes . . . . . . . . . . 9Measurement Types . . . . . . . . . . . 9Component Measurement Rule . . . . . . . 10Measurement Group . . . . . . . . . . 10Measurement Group Member . . . . . . . 11Time Summary . . . . . . . . . . . . 11Measurement . . . . . . . . . . . . . 12

Common tasks . . . . . . . . . . . . . 13Common component types and attributes . . . 14Time values in UTC . . . . . . . . . . 14Parent child relationships . . . . . . . . . 15Static data . . . . . . . . . . . . . . 15Measurements . . . . . . . . . . . . 17

Chapter 3. Implementing the extract,transform, and load process . . . . . 19Using the Data Warehouse Center . . . . . . . 19

Using the SQL execution engine . . . . . . 22Incremental extracts using extract control . . . . 31

Run-time example of extract control . . . . . 33SQL script examples of incremental extract . . . 34

Recoverability in case of failure. . . . . . . . 38Generic rules . . . . . . . . . . . . . 41

Component attributes that vary over time . . . . 42Pruning old data . . . . . . . . . . . . 42Initial load of data into the central data warehouse 44Aggregation and rollup . . . . . . . . . . 44Customers and centers . . . . . . . . . . 45

Multicustomer and multicenter support . . . . 46Handling data exceptions. . . . . . . . . . 52Creating a star schema or multidimensional cube. . 52

Rule 1: metric dimension table . . . . . . . 53Rule 2: component dimension tables . . . . . 54Rule 3: fact table. . . . . . . . . . . . 54Star schema example . . . . . . . . . . 55Performance considerations . . . . . . . . 57Report scheduling . . . . . . . . . . . 57

ETL guidelines . . . . . . . . . . . . . 58

Chapter 4. Migrating Tivoli DecisionSupport function . . . . . . . . . . 59Tivoli Decision Support cube queries . . . . . . 61Tivoli Decision Support calculated columns. . . . 62Collection of distributed data by the application . . 63Tivoli Enterprise Data Warehouse functionality . . 63

Chapter 5. Naming conventions . . . . 65Objects created using the Data Warehouse Center . 65

Subject areas . . . . . . . . . . . . . 65Warehouse sources . . . . . . . . . . . 66Warehouse targets . . . . . . . . . . . 66Processes . . . . . . . . . . . . . . 66Steps . . . . . . . . . . . . . . . 66Warehouse schemas . . . . . . . . . . 67

Application-specific objects in data mart databases 67Dimension tables based on components . . . . 67Dimension tables based on measurement types 67Fact Tables . . . . . . . . . . . . . 67

Application-specific objects in the central datawarehouse. . . . . . . . . . . . . . . 68

Staging tables. . . . . . . . . . . . . 68Application-specific data in the generic central datawarehouse tables . . . . . . . . . . . . 68

Component type codes . . . . . . . . . 68Attribute type codes . . . . . . . . . . 68Measurement type names. . . . . . . . . 69

Objects displayed using the report interface . . . 69Report names. . . . . . . . . . . . . 69Data mart names . . . . . . . . . . . 69Star schema names . . . . . . . . . . . 69Metric names . . . . . . . . . . . . . 69

Chapter 6. Creating reports . . . . . . 71Extreme case reports . . . . . . . . . . . 71Health check reports . . . . . . . . . . . 74Summary reports . . . . . . . . . . . . 78Report metadata . . . . . . . . . . . . . 80Report formatting . . . . . . . . . . . . 85

Chapter 7. Internationalizing reportsand application data . . . . . . . . . 87National language support process . . . . . . 87

Chapter 8. Assembling the warehouseenablement pack . . . . . . . . . . 93

iii

Top-level directory . . . . . . . . . . . . 95Product code directory . . . . . . . . . . 95Application installation properties . . . . . . . 95Application components . . . . . . . . . . 96Required files. . . . . . . . . . . . . . 98

Appendix A. Central data warehousedata model . . . . . . . . . . . . . 99Attribute Domain . . . . . . . . . . . . 106Attribute Rule . . . . . . . . . . . . . 106Attribute Type . . . . . . . . . . . . . 107Center . . . . . . . . . . . . . . . . 107Component . . . . . . . . . . . . . . 107Component Attribute . . . . . . . . . . . 109Component Change . . . . . . . . . . . 110Component Relationship . . . . . . . . . 110Component Type . . . . . . . . . . . . 111Customer. . . . . . . . . . . . . . . 112Extract Control . . . . . . . . . . . . . 112Extract Log . . . . . . . . . . . . . . 112Geographic Area . . . . . . . . . . . . 113Geographic Type . . . . . . . . . . . . 113Measurement . . . . . . . . . . . . . 113Measurement Group . . . . . . . . . . . 114Measurement Group Member . . . . . . . . 115Measurement Group Type . . . . . . . . . 115Measurement Rule. . . . . . . . . . . . 115Measurement Source . . . . . . . . . . . 116Measurement Type . . . . . . . . . . . 116Measurement Unit. . . . . . . . . . . . 117

Measurement Unit Category . . . . . . . . 117Measurement Unit Conversion . . . . . . . 117Measurement Unit Subunit . . . . . . . . . 118Prune Measurement Control . . . . . . . . 118Prune Measurement Log. . . . . . . . . . 118Relationship Rule . . . . . . . . . . . . 118Relationship Type . . . . . . . . . . . . 119Schema Version. . . . . . . . . . . . . 119Time Summary . . . . . . . . . . . . . 119Time Zone . . . . . . . . . . . . . . 120Time Zone Install . . . . . . . . . . . . 120Translated Term . . . . . . . . . . . . 121

Appendix B. Central data warehouseexamples . . . . . . . . . . . . . 123

Appendix C. Central data warehousetriggers . . . . . . . . . . . . . . 129

Appendix D. SQL execution enginedata transfer support . . . . . . . . 135

Appendix E. Sample Perl script. . . . 139

Glossary . . . . . . . . . . . . . 141

Index . . . . . . . . . . . . . . . 145

iv Enabling an Application for Tivoli Enterprise Data Warehouse

Preface

Tivoli Enterprise™ Data Warehouse enables you to access the underlying dataabout your network devices and connections, desktops, software, and the activitiesin managing your infrastructure. Enabling an Application for Tivoli Enterprise DataWarehouse provides information describing the tasks related to connecting anapplication into the Tivoli Enterprise Data Warehouse.

Who should read this guideThis guide is for the following audience:v Application programmers who want to use Tivoli Enterprise Data Warehouse to

store and report on their application’s datav Data warehousing experts who want to import Tivoli Enterprise Data Warehouse

data into business intelligence applicationsv Customers who want to use their local data in the warehouse or migrate Tivoli

Decision Support guide customizations to the Tivoli Enterprise Data Warehouseformat

Readers should be familiar with the following:v IBM® DB2® and relational database management system (RDBMS) design

conceptsv Structured Query Language (SQL)v Tivoli® software applicationsv Data warehouse information and design, extract, transform, and load (ETL)

processes, and online analytical processing (OLAP)

What this guide containsThis book contains the following sections:v Chapter 1, “Overview of Tivoli Enterprise Data Warehouse” on page 1 provides

an introduction to Tivoli Enterprise Data Warehouse.v Chapter 2, “Planning for the extract, transform, and load process” on page 5

describes the planning required for creating extract, transform, and load (ETL)programs.

v Chapter 3, “Implementing the extract, transform, and load process” on page 19describes how to implement extract, transform, and load (ETL) programs,including common tasks and ETL guidelines.

v Chapter 4, “Migrating Tivoli Decision Support function” on page 59 describeshow to migrate the data sources from which Tivoli Decision Support data wascreated into Tivoli Enterprise Data Warehouse information.

v Chapter 6, “Creating reports” on page 71 provides examples of the TivoliEnterprise Data Warehouse report types and discusses the report formatting andmetadata used by the Tivoli Enterprise Data Warehouse report interface.

v Chapter 7, “Internationalizing reports and application data” on page 87 detailsthe national language support process and the steps required to internationalizeapplication data.

v Chapter 8, “Assembling the warehouse enablement pack” on page 93 providesthe structure required for creating an installable warehouse pack.

v

v Appendix A, “Central data warehouse data model” on page 99 describes each ofthe tables that compose the central data warehouse generic schema.

v Appendix B, “Central data warehouse examples” on page 123 provides sampledata showing how to save basic management data into the central datawarehouse.

v Appendix C, “Central data warehouse triggers” on page 129 lists the databasetriggers provided by Tivoli Enterprise Data Warehouse.

v Appendix E, “Sample Perl script” on page 139 provides a sample Perl script thatcan be used in conjunction with the SQL execution engine.

PublicationsThis section lists publications in the Tivoli Enterprise Data Warehouse library and anyother related documents. It also describes how to access Tivoli publications online,how to order Tivoli publications, and how to make comments on Tivolipublications.

Tivoli Enterprise Data Warehouse libraryThe following documents are available in the Tivoli Enterprise Data Warehouselibrary:v Installing and Configuring Tivoli Enterprise Data Warehouse, GC32-0744

Describes how Tivoli Enterprise Data Warehouse fits into your enterprise,explains how to plan for its deployment, and gives instructions for installingand configuring it. It provides an introduction to the built-in program forcreating and running reports, and contains maintenance procedures andtroubleshooting information.

v Tivoli Enterprise Data Warehouse Release Notes, GI11-0857Provides late-breaking information about Tivoli Enterprise Data Warehouse.

For information about creating and using groups to control access to data marts;creating, customizing, and running reports; and viewing and saving the output ofreports, consult the help topics in the IBM Console. Information about gettingstarted with the IBM Console is provided in Installing and Configuring TivoliEnterprise Data Warehouse.

Related publicationsThe DB2 library contains important information about the database and datawarehousing technology provided by IBM DB2, DB2 Data Warehouse Center, andDB2 Warehouse Manager. Refer to the DB2 library for help in installing,configuring, administering, and troubleshooting DB2. The DB2 library is availableonline, as described in “Accessing publications online” on page vii. After installingDB2, its library is also available on your system.

The following DB2 documents are particularly relevant for people working withTivoli Enterprise Data Warehouse:v IBM DB2 Universal Database for Windows Quick Beginnings, GC09-2971

Guides you through the planning, installation, migration (if necessary), andsetup of a partitioned database system using the IBM DB2 product on MicrosoftWindows.

v IBM DB2 Universal Database for UNIX Quick Beginnings, GC09-2970Guides you through the planning, installation, migration (if necessary), andsetup of a partitioned database system using the IBM DB2 product on UNIX.

Preface

vi Enabling an Application for Tivoli Enterprise Data Warehouse

v IBM DB2 Universal Database Administration Guide: Implementation, SC09-2944Covers the details of implementing your database design. Topics includecreating and altering a database, database security, database recovery, andadministration using the Control Center, a DB2 graphical user interface.

v IBM DB2 Universal Database Data Warehouse Center Administration Guide,SC26-9993Provides information on how to build and maintain a data warehouse using theData Warehouse Center.

v IBM DB2 Warehouse Manager Installation Guide, GC26-9998Provides the information to install the following Warehouse Managercomponents: Information Catalog Manager, warehouse agents, and warehousetransformers.

v IBM DB2 Universal Database and DB2 Connect Installation and ConfigurationSupplement, GC09-2957Provides advanced installation considerations and guides you through theplanning, installation, migration (if necessary), and set up a platform-specificDB2 client. Once the DB2 client is installed, you then configure communicationsfor both the client and server, using the DB2 GUI tools or the Command LineProcessor. This supplement also contains information on binding, setting upcommunications on the server, the DB2 GUI tools, DRDA AS, distributedinstallation, the configuration of distributed requests, and accessingheterogeneous data sources.

v IBM DB2 Universal Database Message Reference Volume 1, GC09-2978 and IBM DB2Universal Database Message Reference Volume 2, GC09-2979Lists the messages and codes issued by DB2, the Information Catalog Manager,and the Data Warehouse Center, and describes the actions you should take.

Accessing publications onlineYou can access many Tivoli publications online at the Tivoli Customer SupportWeb site:

http://www.tivoli.com/support/documents/

These publications are available in PDF or HTML format, or both. Translateddocuments are also available for some products.

Ordering publicationsYou can order many Tivoli publications online at the following Web site:

http://www.elink.ibmlink.ibm.com/public/applications/publications/cgibin/pbi.cgi

You can also order by telephone by calling one of these numbers:v In the United States: 800-879-2755v In Canada: 800-426-4968v In other countries, for a list of telephone numbers, see the following Web site:

http://www.tivoli.com/inside/store/lit_order.html

Preface

Preface vii

http://www.tivoli.com/support/documents/



http://www.tivoli.com/inside/store/lit_order.html

Providing feedback about publicationsWe are very interested in hearing about your experience with Tivoli products anddocumentation, and we welcome your suggestions for improvements. If you havecomments or suggestions about our products and documentation, contact us in oneof the following ways:v Send an e-mail to [email protected] Complete our customer feedback survey at the following Web site:

http://www.tivoli.com/support/survey/

Contacting customer supportIf you have a problem with any Tivoli product, you can contact Tivoli CustomerSupport. See the Tivoli Customer Support Handbook at the following Web site:

http://www.tivoli.com/support/handbook/

The handbook provides information about how to contact Tivoli CustomerSupport, depending on the severity of your problem, and the followinginformation:v Registration and eligibilityv Telephone numbers and e-mail addresses, depending on the country you are inv What information you should gather before contacting support

Conventions used in this bookThe following typeface conventions are used in this book:

Bold Lowercase and mixed-case commands, command options, andflags that appear within text appear like this, in bold type.

Graphical user interface elements (except for titles of windows anddialogs) and names of keys also appear like this, in bold type.

Italic Variables, values you must provide, new terms, and words andphrases that are emphasized appear like this, in italic type.

Monospace Code examples, output, and message text appear like this, inmonospace type.

Text strings you must type, when they appear within text, namesof Java™ methods and classes, and HTML and XML tags alsoappear like this, in monospace type.

Preface

viii Enabling an Application for Tivoli Enterprise Data Warehouse

http://www.tivoli.com/support/survey/

http://www.tivoli.com/support/handbook/

Chapter 1. Overview of Tivoli Enterprise Data Warehouse

Tivoli Enterprise Data Warehouse enables you to access the underlying data aboutyour network devices and connections, desktops, applications and software, andthe activities in managing your infrastructure. Having all this information in a datawarehouse enables you to look at your IT costs, performance, and other trends ofinterest across your enterprise. Tivoli Enterprise Data Warehouse can be used toshow the value and payback of Tivoli and IBM software, and it can be used toidentify areas where you can be more effective.

Tivoli Enterprise Data Warehouse provides the infrastructure for the following:v Schema generation of the central data warehousev Extract, transform, and load (ETL) processes through the IBM DB2 Data

Warehouse Center toolv Report interfaces

It is also flexible and extensible enough to allow you to integrate application dataof your own.

As shown in Figure 1 on page 2, Tivoli Enterprise Data Warehouse consists of acentral data warehouse where historical data from management applications isaggregated and correlated for use by reporting and third-party online analyticalprocessing (OLAP) tools as well as planning, trending, analysis, accounting, anddata mining tools.

1

A data mart is a subset of a data warehouse that contains data tailored andoptimized for the specific analysis needs of a department or team. Both the initialextraction of the data from the source applications into the central data warehouse

Report interface

Data marts

Data mart ETLs

TEC DB

Tivoli Management Framework

TEC DM monitorsTAPM

TWSM TBSM

Central data warehouse ETLs

Central data warehouse

Tivoli Enterprise Data Warehouse

Warehouse packs

Any data analysis tools

Data mining

OLAP

Businessintelligence

Figure 1. Tivoli Enterprise Data Warehouse overview

2 Enabling an Application for Tivoli Enterprise Data Warehouse

and the extraction of the data from the central data warehouse into the specificdata marts is handled by extract, transform, and load (ETL) processes. The ETLprocesses gather data from selected external sources, alter it as necessary tointegrate with other sources, and mass load it into a central data warehouse, andthen into specific data marts for end-user analysis or consuming applications.

The central data warehouse ETL reads the data from the operational data stores ofthe application that collects it, verifies the data, makes the data conform to theschema, and places the data into the central data warehouse. The data mart ETLextracts a subset of data from the central data warehouse, transforms it, and loadsit into one or more star schemas, which can be included in data marts to answerspecific business questions. The data extracted by the data mart ETL can also beloaded into an application database to be used for application-specific purposes.

The central data warehouse contains the fully integrated set of historical data forthe data warehouse environment, while each data mart contains a subset of thatdata transformed into usable information for a specific business need. The centraldata warehouse uses a generic schema. As new components or new applicationsare added, more data is added to the database; however, no new tables or columnsare added in the schema.

The Tivoli Enterprise Data Warehouse report interface is the common reportinginfrastructure for generating simple Web-based historical reports. This interfaceallows for static report output. Note that other business intelligence tools can beused for customized OLAP reporting. Also, other Tivoli software applications cantake the information from Tivoli Enterprise Data Warehouse and use it for theirown specific purposes.

Central data warehouse schema overviewThe central data warehouse is optimized for the efficient storage of large amountsof data and has a documented format that makes the data accessible to manyanalysis solutions. The central data warehouse schema is a generic reporting datamodel that is designed to be scalable and flexible.

Note: The Tivoli Enterprise Data Warehouse installation program creates thecentral data warehouse tables. Applications insert data into these tables.

See Appendix A, “Central data warehouse data model” on page 99 for a detaileddescription of the tables that compose the central data warehouse generic schema.

At the simplest level, the warehouse schema consists of two primaryitems—components and measurements. Components are what you are interested inand measurements are the nature of your interest in the component. Experience sofar indicates components can be of two distinct varieties:v Objects that represent concrete things like applications, pieces of hardware or

softwarev Objects that represent events generated by applications

Components are defined by records stored in the Comp table. Measurements arenumerical values recorded against components. For components that representmore concrete concepts, the measurements can be CPU speeds, disk usage, or anyother type of numerical value that makes sense. For event components,

Chapter 1. Overview of Tivoli Enterprise Data Warehouse 3

measurements tend to be counts of the number of times the event occurred duringa certain time frame. Measurements are defined by records stored in the Msmttable.

Components are static objects. Component attributes are characteristics ofcomponents that might change frequently when compared to the life of acomponent. The CompAttr table stores component attribute data that describes acomponent at a particular instance in time. In most cases, component attributes donot affect the type of measurements collected for a component (although they canin some cases). A computer attribute might define the specific operating systemversion loaded on a computer but does not change the fact that there is acomputer you want to collect data on and might not change the fact that you wantto collect disk usage information for the computer.

Correlation and grouping of data is an important warehouse concept. To groupmeasurements or group components to provide correlation, individual componentsand individual measurements need to be categorized. The type tables in thewarehouse schema provide a mechanism for categorization. The CompTyp tablehelps to categorize individual components. For example, there might be hundredsor thousands of computers at a company (this potentially represents hundreds orthousands of components), but they will fall into relatively few categories such asservers/workstations, Unix/Windows, Remote/Local, and so on. Reports can bewritten that report on data for all Unix servers, for example, instead of generatingindividual reports for thousands of machines. The MsmtTyp table helps tocategorize individual measurements. For example, there are many different typesof measurements that can be collected on a computer but they fall into generalcategories, such as disk usage measurements, CPU speeds, usage counts, and soon. One report can be generated for components with a CPU speed measurement,for example, instead of generating an individual report for every computer withdata in the warehouse.

Many of the other tables in the warehouse schema exist to enforce common senserules. Collecting CPU usage measurements for event components makes littlesense. Illogical data is prevented from entering the warehouse databaseaccidentally through a series of check tables. Data in the AttrRul, RelnRul, andMsmtRul tables, along with relational constraints enforce checks that preventillogical data from entering the database. Other tables in the warehouse schemasupport automated data management facilities. The Prune_Msmt_Control tablecontains entries that determine how long particular types of data should beretained in the database before being automatically deleted. The Extract_Controltable contains entries indicating the next set of data to be pulled from a particularsource database table so that incremental chunks of data can be loaded rather thanreloading all the data from an application every time.

It can be overwhelming to consider all of the warehouse database tables in onesitting. This is especially true when attempting to design a brand new central datawarehouse ETL process. Common questions arise around what definitions to usefor components and what groupings might make sense. Unfortunately, there is nota simple solution explaining how to map an application’s data into the warehouseschema. Focus on the basic concepts of theschema—component/measurement/groupings for reports to make your task moremanageable. Examine existing types and rules to see how they were defined and ifthey make sense to reuse for your application. Most of all, focus on the end use ofthe data to determine what components and measurements make sense.


Chapter 2. Planning for the extract, transform, and loadprocess

The applications and services that monitor and manage an IT enterprise generatedata in a variety of formats, including relational databases, spreadsheet data, logfiles, and others. One set of ETL programs takes the data from these sources andplaces it in the central data warehouse. Another set of ETL programs extracts fromthe central data warehouse a subset of historical data that contains data tailored toand optimized for a specific reporting or analysis task. This subset of data is usedto create one or more data marts, a subset of the historical data that satisfies theneeds of a specific department, team, or customer. A star schema is optimized forinteractive reporting and data analysis. The format of a star schema is specific tothe reporting or analysis tool you plan to use. (Unless you are writing your ownwarehouse application or your own reports, you do not need to deal with theformat of a star schema. An application that provides a data mart ETL must createits data marts in the appropriate format. See “Creating a star schema ormultidimensional cube” on page 52.) Customers can then use an analysis programto analyze a specific aspect of their enterprise using the data in one or more starschemas. When developing applications for use with Tivoli Enterprise DataWarehouse, consider how the application data will be analyzed to determine theextract, transform, and load (ETL) programs you need to provide.

Tivoli Enterprise Data Warehouse uses an open architecture allowing you tointegrate non-Tivoli data in Tivoli Enterprise Data Warehouse.

The ETL process requires a user with knowledge of the source databases involvedand good SQL skills. No interface is provided that allows users unfamiliar with thedata and with SQL to simply move data from any application’s data store to thedata warehouse. Instead, application developers must provide a set of scripts andprocesses that move data from Tivoli application data stores to the central datawarehouse.

Use the following steps to enable your application for Tivoli Enterprise DataWarehouse:1. Define the data you need. This can be done by determining reporting needs or

determining the set of data that needs to be placed in the central datawarehouse to satisfy other analysis tasks. If your application has a TivoliDecision Support guide, that is an appropriate place to start. The data thatwas collected and reported in Tivoli Decision Support will most likely beneeded in the central data warehouse also. See Chapter 4, “Migrating TivoliDecision Support function” on page 59 for more information and generalguidelines for determining data requirements.In addition to the Tivoli Decision Support requirements, applications shouldevaluate the data collected by the application and determine what data mightbe useful to other analysis tasks or products that the customer owns. Ifapplications collect any of the following data, they should save the data in theTivoli Enterprise Data Warehouse:v Availability datav Response time data

This includes both end user response time data such as probes and internalapplication response time.

5

v Best practicesMost monitoring applications monitor specific resources (such as servers) orspecific properties (CPU utilization, available disk space, and so on) as adefault.

If an attribute is critical enough to monitor as a default, then that informationshould be saved in the Tivoli Enterprise Data Warehouse.

2. Study the central data warehouse generic schema and existing applications.Familiarize yourself with the set of static data that is common to allapplications.

3. Fill out the application enablement template. Define your data in terms of thecentral data warehouse. Use common static data as much as possible as thisfacilitates cross application reporting and analysis. (See “Filling out theenablement template” on page 7.)

4. Review naming conventions (See Chapter 5, “Naming conventions” onpage 65.)

5. Perform a Tivoli Enterprise Data Warehouse installation and install at leastone application. Review the installation directory structure. Name your scriptsusing the correct naming standards and place them in the correct location forcreating the warehouse enablement pack.

6. Create the scripts to insert static data. Use the common static script:twh_cdw_data.sql, and create product_code_cdw_data.sql. (See “Static data” onpage 15.)

7. Review extract control design and determine which columns can be used forincremental extracts. If no columns are available, then a modification to thesource application data might be necessary. See “Incremental extracts usingextract control” on page 31 to assist you with determining tables other thanthe source that need to be processed with extract control.

8. Review timestamps. Timestamps must be stored in Universal TimeCoordinated (UTC) in the central data warehouse. If time zone information isnot available, then a modification to the source application data might benecessary. (See “Time values in UTC” on page 14 and “Multicenter support”on page 51.)

9. Review “Common tasks” on page 13 and apply to your transformationprocessing as needed.

10. Code the central data warehouse ETL. Develop standalone SQL scripts thatquery your source data and populate the Comp, CompAttr (optional),CompReln (optional), and Msmt tables. Run the scripts from the commandline using the SQL execution engine. (See “Manually testing the customscript” on page 28.) Once they are running from the command line, create aprocess in the Data Warehouse Center that accomplishes this task.

11. Based on reporting and analysis needs, define star schemas.12. Code the data mart ETL for each star schema. If you are utilizing the report

interface (RPI), a specific structure must be followed. (See “Creating a starschema or multidimensional cube” on page 52.)

13. Provide the required translation strings for internationalization. (SeeChapter 7, “Internationalizing reports and application data” on page 87.)

14. Use the appropriate naming and packaging conventions for your applicationcomponents and create the warehouse enablement pack. (See Chapter 8,“Assembling the warehouse enablement pack” on page 93.)


Filling out the enablement templateAfter an application identifies the data that should be saved in the TivoliEnterprise Data Warehouse, one of the next steps is to determine how theapplication data will be mapped into the central data warehouse schema. Thissection describes how application data should be mapped into the key central datawarehouse tables. (For a complete description of each of the tables that composethe central data warehouse generic schema, see Appendix A, “Central datawarehouse data model” on page 99.)

One of the key benefits of the Tivoli Enterprise Data Warehouse is thatcross-application reports and higher-level applications can process data frommultiple applications. For cross-application correlation to work correctly, allapplications must follow specific conventions when saving data in the TivoliEnterprise Data Warehouse.

Component typesComponent types represent the type or class of the resource and not the resourceitself. For example, a server could be a component type and 9.67.241.43 would bean instance (component) of a server. Also notice that the Component Type(CompTyp) table has a Parent column that allows component types to bestructured in a hierarchy. Component types should be shared as much as possiblebetween source applications to foster correlation. For example, if the resource is aWeb server, the component type should be an IP_HOST and not something likeAPPNAME_WEBSRVR. By using IP_HOST as a standard component type,measurement data from a number of different applications can be correlated to asingle resource.

The following rules are associated with component types:v A source application should use the IP_HOST component type if the managed

resource can be identified by a fully qualified host name. Information about theIP_HOST, such as the LAST_KNOWN_IPADDRESS should be saved ascomponent attributes. (See “Component attributes” on page 9 for details).

v A source application should use IP_INTERFACE as the component type if the IPresource does not have a fully qualified host name. Source applications shouldalways attempt to create IP resources with the IP_HOST component type. Thefully qualified host name should be used as the component name. If for somereason the fully qualified host name is unavailable, then the IP_INTERFACEcomponent type should be used. Correlation between IP resources is moredifficult using IP Addresses, so source applications should use IP_INTERFACEonly if the fully qualified host name is unavailable.

v Source applications should create application-specific component types if thecomponent type is unique to the application. For example, if application ’X’creates a view that represents a service, then information about that view willonly be known by application ’X’. Source applications should not useapplication-specific types if there is a type already defined that fits their need.

v Source applications should make the changes necessary to use standardcomponent types. For example, the fully qualified host name can almost alwaysbe available, but some source applications currently use either the shorthistamine or the IP Address to identify the resource. In this case, the sourceapplication should be changed to collect the fully qualified histamine so that theIP_HOST component type can be used.

Chapter 2. Planning for the extract, transform, and load process 7

ComponentsComponents are the actual resource such as workstations, bridges, routers, hubs,applications, servers, and so on. It is also common for components to havesubcomponents (or children). For example, a router typically has multiple physicalinterfaces, such as an ATM interface. The ATM interface might actually havechildren such as ATM PVCs that run over the physical interface. A sourceapplication that reports router ATM PVC measurement data would createcomponents for the router, the interfaces, and all the ATM PVCs.

The Component (Comp) table itself does not allow for components to be groupedin a hierarchy. However, component hierarchies can be created using relationshipsbetween components. One important relationship in which every component takespart is the naming or PCHILD relationship.

The name for the component (entered in the Comp_Nm column) is important.Component names should uniquely identify the component within the context ofthe parent. For example, the actual ATM interface on a router is typically identifiedby a number, such as the number 13. So 13 should be used as the component namefor the ATM interface.

The following rules are associated with the Component table:v Composite names should never be used in the Comp_Nm column. In the router

example above, applications should use 13 as the Comp_Nm and notterps.maryland.cp.edu-13.

v The component name should be displayable strings that make sense to the enduser. The component name should not be a key or some unique identifier that isnot generally known to the user. The name of a component should make it easyto relate the information in the central data warehouse back to information inthe source application’s user interface.

v In the central data warehouse, components are not deleted once entered. If acomponent is no longer valid, then the Comp_End_DtTm column should beused to indicate when the resource was deleted or changed. Setting theComp_End_DtTm value marks the component as deleted while still allowingapplications to report on historical data for that component.

Relationship Type, Relationship Rule, and ComponentRelationship

Source applications can use the Relationship Type (RelnTyp), Relationship Rule(RelnRul), and Component Relationship (CompReln) tables to define any uniquerelationship between components.

The following relationship rules only apply to source applications that need tomodel components as a hierarchy. For example, if you have a router with interfacesand various protocols running over the physical interfaces, then the hierarchybetween the router, interfaces, and protocols needs to be modeled usingrelationships.v All components that have subcomponents must use the PCHILD relationship

type (RelnTyp_Cd = PCHILD in the RelnTyp table) to model the namingrelationship between the components. Child components are allowed to havemultiple parents. In other words, components can have multiple names (aliases).

v Component Relationship Rule - For the PCHILD relationship, sourceapplications must define which types of components are the parents and whichtypes of components are the children. In the router example, this would basically


define that an ATM interface is a child of a router. This is done by adding a rowto the Component Relationship table where the Parent component type (router)is specified in the CompTyp_Source_Cd and the Child Component type (ATMinterface) is specified in the CompTyp_Target_Cd. This defines the patterns fornames in the central data warehouse. In other words, the name of an ATMinterface will always be in terms of a router.

v Component relationship. Any component that is not named directly under root(has more than one token in its name) must have an entry for the PCHILDrelationship in the Component Relationship table. For example, if a router has 13interfaces, then each interface has a row in the Component Relationship tablewhere the Component ID of the router is specified as the Comp_Source_ID.

Component attributesComponent attributes, which are recorded in the Component Attribute (CompAttr)table, can be used to store additional information that helps describe a particularcomponent. For example, a fully qualified host name identifies an IP_HOST andthe attribute LAST_IP_ADDRESS is a good way to save the last known IP Addressfor that IP_HOST.

The following guidelines are associated with component attributes:v (Optional) For IP_HOST components the LAST_KNOWN_IPADDRESS

(AttrTyp_Cd = LAST_IP_ADDRESS) should be used as the attribute to save theIP_ADDRESS.

v (Optional) If an IP Address changes for a particular IP_HOST, the sourceapplication should update the LAST_KNOWN_IPADDRESS attribute. This isoptional because in some source applications, the IP Address might not beavailable. If the IP Address is available to the source application, then the sourceapplication should always update the LAST_KNOWN_IPADDRESS. The reasonfor this is that with DHCP the IP Address can change for a given fully qualifiedhost name. When a source application updates the warehouse, the assumption isthat the source application typically has more recent data than the warehouse.So if the warehouse LAST_IP_ADDRESS does not match the IP Address in thesource application, it is likely that DHCP assigned a new IP Address.

v (Optional) Source applications should not save component name information asattributes. For example, an IP_HOST component is identified (named) by itsfully qualified host name. Because the fully qualified host name is already usedin the component name, there is no additional value in creating an attribute forthe fully qualified host name.

Measurement TypesMeasurement types are the types of data that source applications manage.Measurement types are things such as Response Time and Memory Usage. Formost types of measurements (Response Time, Utilization, and so on), sourceapplications need to create a single row in the Measurement Type (MsmtTyp) tablefor each measurement that the application supports.

The exception to this rule is when a source application needs to report status,availability, or state information. For availability, status, and state measurements,source applications need to create a row in the MsmtTyp table for each of thepossible states. For example, assume that an IP_HOST can be UP or DOWN. Inthis case, the source application needs to create three rows in the MsmtTyp table torepresent the UP, DOWN, and UNKNOWN states.

The following rules are associated with measurement types:


v State, Status, and Availability data must be modeled in the Tivoli EnterpriseData Warehouse as percentage in state measurement types.

Note: This means that the MUnit_Cd='PRC' in the Measurement Type table.v For State, Status, and Availability data, the UNKNOWN state should be used

when the source application cannot determine the state of the component. Forexample, if an IP_HOST can be UP or DOWN, a source application should onlyreport UP and DOWN measurements when data is available and the componentis known to be in the UP or DOWN state. If data is missing for any reason, thenthe UNKNOWN state should be used to report that the status of the resource isnot known.

v Both the MsmtTyp_Nm and MsmtTyp_Ds columns in the Measurement Typetable are displayable strings that will appear on reports and the end user screenif using the report interface. These columns should not contain abbreviationsbecause application-specific abbreviations lose some of their context when thedata is used by cross application reporting or higher-level applications.

Component Measurement RuleThe Measurement Rule (MsmtRul) table defines what measurements are valid foreach component. For most measurement types there is a single row in this tablethat associates a measurement type, such as packet loss to a particular componenttype (for example, router interface). State, Status, and Availability measurementtypes require a row for each possible state.

The following rules are associated with the Measurement Rule table:v Source applications need to create at least one row in this table that associates a

particular measurement type to a given component type.v For State, Status, and Availability Measurement types each component requires

rows in the MsmtRul table for each of the possible states. For example, for anIP_HOST that can be UP, DOWN, or UNKNOWN, there should be three rows inthe MsmtRul table that associate the UP, DOWN, and UNKNOWN measurementtypes with the IP_HOST.

Measurement GroupSource applications can create measurement groups to help organize themeasurement types. Measurements can be in multiple groups and sourceapplications can create as many measurement groups as necessary to help organizethe measurement types. For example, a source application might have some BestPractices measurement types that are always monitored as a default. Ameasurement group could be created that groups all the measurement typesincluded in the Best Practices profile.

The following measurement groups are critical to cross-application reporting andhigher-level applications:v AVG_E (Average value exists)v MIN_E (Minimum value exists)v MAX_E (Maximum value exists)v TOT_E (Total value exists)v STATE (Percentage State measurements)

All source applications are required to use these standard measurement groups sothat applications performing higher-level correlation know what data is to beexpected from the source application. These measurement groups are created in the


central data warehouse when the warehouse enablement pack is installed in aTivoli Enterprise Data Warehouse environment.

The following rules are associated with the Measurement Group (MGrp) table:v MIN_E, MAX_E, and AVG_E must be used if values are reported in the

Msmt_Min_Val, Msmt_Max_Val, and Msmt_Avg_Val columns in theMeasurement table. Typically, when management data is summarized, theinformation that is produced is the minimum, maximum, and average values forthe interval. For example, an application might have the minimum, maximum,and average response time values over the duration of hour. Another examplewould be that the minimum, maximum, and average number of connected userswould be summarized over the duration of an hour. If a source application savesminimum, maximum, and average measurements in the Measurement table,then the MIN_E, MAX_E, and AVG_E measurement groups must be used.

v For some types of management data, the minimum and maximum values arenot needed. Source applications should use the TOT_E measurement group ifthe measurement data that is reported is a single value in the Msmt_Tot_Valcolumn.

v For Availability, Status, and State data, the STATE measurement group must beused.

Measurement Group MemberThe Measurement Group Member (MGrpMbr) table associates a particularmeasurement type to measurement group.

The following rules are associated with the Measurement Group Member table:v For each measurement type, source applications must create a row in the

Measurement Group Member table with the MGrp_Cd = MIN_E if theMsmt_Min_Val contains a measurement value.

v For each measurement type, source applications must create a row in theMeasurement Group Member table with the MGrp_Cd = MAX_E if theMsmt_Max_Val contains a measurement value.

v For each measurement type, source applications must create a row in theMeasurement Group Member table with the MGrp_Cd = TOT_E if theMsmt_Tot_Val contains a measurement value.

v For each measurement type, source applications must create a row in theMeasurement Group Member table with the MGrp_Cd = AVG_E if theMsmt_Avg_Val contains a measurement value.

v For Availability, Status, and State measurements, there must be a row in theMeasurement Group Member table with the MGrp_Cd = STATE for each of thepossible states. For example, if UP, DOWN, and UNKNOWN are the validstates, then there are three rows in the Measurement table for the measurementtype associated with the percentage in state availability.

v For Availability, Status, and State measurements, the TOT_E measurement groupmust be specified and the percentage in state measurements must be reported inthe Msmt_Tot_Val column in the Measurement table.

Time SummaryThe Time Summary (TmSum) table contains the codes that describe the intervalsthat are used to summarize the measurement data. It is recommended that allmeasurement data be summarized on an hourly basis for the following reasons:v Summarizing data on an hourly basis reduces data and disk space requirements.


v The Tivoli Enterprise Data Warehouse report interface uses hourly data.v Rollup triggers are based on hourly data.

MeasurementThe following rules are associated with the Measurement (Msmt) table:v All measurement data must be summarized on even time summary values.

Hourly data is summarized on the top of the hour, daily data at midnight,weekly data at the end of the week (note that this changes for localities that useMonday as the first day of the week), monthly at midnight on the last day of themonth, and so on. For example, hourly data should be summarized starting atthe top of the hour (1 a.m., 2 a.m., and so on) and not something like 2:43 a.m.,3:43 a.m., and so on.

v All measurement data must be summarized. Measurement data points that aremore frequent than one hour should not be saved in the central data warehouse.

v Put a measurement in the ″total″ column if you add values together from hourlydata to make daily data, and use the min/max/avg columns if you averagehourly values to get values for a day. For example, a counter for the totalnumber of users that logged on to TSO should be reported under theMsmt_Tot_Val. The value would represent the total for that hour (5 users loggedon to TSO), but the hourly measurements could be totaled for a day to producea daily total. Data such as percentages and average response times should not bereported in the Msmt_Tot_Val column. For example, totaling percentagesproduces values that are not valid. The percentage data should be reported inthe Msmt_Avg_Val column so that all the hourly percentages are averagedtogether.

v Measurements must include values for Msmt_Smpl_Cnt and Msmt_Err_Cnt ifthis data is available to the source application. If a source application does notsupport Sample Count (Msmt_Smpl_Cnt) and Error Count (Msmt_Err_Cnt), thena value of NULL should be reported. Zero values indicate that there were nosamples or no errors for the interval. NULL values indicate that the sourceapplication does not support these columns.

v Response time measurements must include values for Msmt_Min_Val,Msmt_Max_Val, Msmt_Avg_Val, Msmt_Smpl_Cnt, and Msmt_Err_Cnt, assumingMsmt_Smpl_Cnt and Msmt_Err_Cnt data is available. All data must besummarized into hourly data before saving the data in the central datawarehouse. When summarizing response time data, you should havemeasurement minimum, maximum, average, sample count, and error countvalues.

Note: If data is only available in larger time granularities than an hour, thenwhat’s available is used. If data is available in smaller granularities, itneeds to be summarized to hourly.

v For State, Status, and Availability data, the measurements for a Time Summaryinterval must add up to 100%. For example, if a router is up 100% of the timebetween 3 a.m. and 4 a.m., then only the UP measurement with a value of 100needs to be reported for the 3 a.m. to 4 a.m. time period. If the router was downfor 45 minutes between 4 a.m. and 5 a.m., then two rows are required in themeasurement table. The first row is the UP row with a value of 25 and thesecond row is the DOWN row with a value of 75. If the router was down for 30minutes between 4 a.m. and 5 a.m., and the router could not be reached for 15minutes between 4 a.m. and 5 a.m., then three rows are required in themeasurement table. The first row is the UP row with a value of 15. The secondrow is the DOWN row with a value of 30. The third row is the UNKNOWN rowwith a value of 15.


v Measurements for Availability, State, and Status data must be reported for eachinterval. For example, there should be a minimum of 24 hourly measurementsfor hourly percentage in state data. If there are missing values, the ETL must fillin rows for missing data with UNKNOWN.

v Measurements for Response Time and Utilization data only needs to be reportedif there is measurement data for that interval. For example, if a probe monitorse-mail response time from 8 a.m. to 4 p.m., then only measurement data from 8a.m. to 4 p.m. needs to be saved in the measurement table. Source applicationsdo not have to report measurements with zero values and zero sample countsfrom midnight to 8 a.m. and from 4 p.m. to midnight.

The following optional conventions are associated with the Measurement table:v The measurement data should only be saved with a single time summary code.

For example, if the data is summarized as hourly measurements, then dailymeasurements should not be saved in the central data warehouse. If daily datais needed, then the hourly measurements can be used to determine the dailydata. Saving the same data as hourly, daily, monthly, and so on, takes upadditional disk space without adding value.

v Response Time measurements should only include successful data points andnot include response data for failed or canceled data points.

Common tasksThis section discusses the common tasks for applications inserting data into thecentral data warehouse.

Notes:

1. The central data warehouse tables, indexes, and triggers are created as TWG.x,where TWG stands for Tivoli Warehouse Generic. Applications must use TWGto refer to the tables in the generic schema only, for example:insert into TWG.comp ...

Otherwise, applications must use their product code for staging tables, forexample:insert into CTO.stage_x ...

2. When creating triggers with a trigger action that is more than one SQLstatement, enclose the statements, or trigger body, in a block of code as follows:BEGIN ATOMIC

stmt;stmt;

END

The statements in the trigger body are enclosed between BEGIN ATOMIC and END,and are separated by semicolons. Use the number sign (#), rather than asemicolon, as the statement termination character for all other SQL statementsin the SQL files. For example:BEGIN ATOMICIF LOCATE(’.’,N.Comp_Nm)>0 THEN

IF LOCATE(’.’,N.Comp_Nm,LOCATE(’.’,N.Comp_Nm)+1)>0 THENSET N.CompTyp_Cd = ’IP_HOST’;

ELSESET N.CompTyp_Cd = ’DM_HOST’;

END IF;ELSE


SET N.CompTyp_Cd = ’DM_HOST’;END IF;END#

Common component types and attributesTo address host correlation issues, all ETLs must follow the guidelines for usingstandard component types and attributes described in “Component types” onpage 7 and “Component attributes” on page 9.

It is important for all source applications to attempt to resolve the host to a fullyqualified host name. To verify whether a host name is a fully qualified host nameor not, check if at least one period mark (.) is contained in the field value. Themost efficient way to do this in SQL is to use a CASE clause and the LOCATEfunction in DB2. The following example shows the insert into TWG.Comp usingthis logic. The variable t.name is the input string that is searched for a period. If aperiod cannot be located in t.name, AppName_HOST is assigned as the componenttype code, where AppName is TWSM in this example. If a period is located int.name, the name is assumed to be a fully qualified host name and IP_HOST isassigned as the component type code.insert into TWG.COMPSELECT

...case

when LOCATE(’.’,t.name) = 0 then ’TWSM_HOST’else ’IP_HOST’

end as comptyp_cd,...

FROMBWM.stage_ept_data t,

The following component types and attribute types are created in the central datawarehouse during a warehouse enablement pack installation.

Table 1. Component Type (CompTyp table)

CompTyp_Cd CompTyp_Parent_Cd CompTyp_Nm CompTyp_Strt_DtTm CompTyp_End_DtTm

'IP_HOST' NULL 'IP Host' current timestamp -current timezone

'9999-01-01-12.00.00.000000'

'IP_INTERFACE' NULL 'IP Interface' current timestamp -current timezone

'9999-01-01-12.00.00.000000'

'TME_ENDPOINT' NULL 'TME Endpoint' current timestamp -current timezone

'9999-01-01-12.00.00.000000'

Table 2. Attribute Types (AttrTyp table)

AttrTyp_Cd AttrTyp_Nm

'LAST_IP_ADDRESS' 'Last IP Address'

'TME_OBJECT_ID' 'TME Object ID'

Time values in UTCAll time values inserted into the central data warehouse must be in Universal TimeCoordinated (UTC). The DB2 function current timezone gives you the offset, socurrent timestamp - current timezone converts local time to UTC. For example:


insert into twg.comptyp values ('HOST', NULL, 'Host',current timestamp - current timezone, '9999-01-01-12.00.00.000000');

insert into twg.comp values (0, 'HOST', 'CDW', 0, NULL,'No Parent Entry', 'No Parent Entry',current timestamp - current timezone, '9999-01-01-12.00.00.000000',NULL );

This is required because data can be accessed from different time zones (and thetime zone is not saved with the time). With UTC, you can adjust the times to yourlocale when running reports, when using the data, and so on.

Parent child relationshipsUsing a relationship defined in the CompReln table (rather than a field in theComp table), allows components to have multiple names (parents). Every namemust resolve to a unique component. The relationship has to be defined before it isused.

Use the PCHILD Component Relationship Type to identify componentrelationships.

Static dataTivoli Enterprise Data Warehouse provides a script named twh_cdw_data.sql,which inserts common static data into the central data warehouse database. Thetwh_cdw_data.sql script is located in the enablement directory on the TivoliEnterprise Data Warehouse installation CD. Applications should copy this scriptinto the tedw_apps_etl\product_code\pkg\vversion\cdw\dml directory of theirwarehouse enablement pack. During a warehouse pack installation, the installationprogram runs the twh_cdw_static.sql first, followed by any application-specificdata insertion script that is present in this directory. (See Chapter 8, “Assemblingthe warehouse enablement pack” on page 93 for more information.) This ensuresthe common data is entered appropriately if your application’s warehouse pack isthe first or only warehouse pack installed in the Tivoli Enterprise Data Warehouseenvironment.

The data insertion script written by applications, which is used for inserting datainto the central data warehouse tables, must use where not exists SQL logic to checkif the value already exists. This allows the script to be rerun if it fails beforecompleting successfully. For example://Create temp tabledeclare global temporary table munit (

MUnit_Cd CHAR(6) NOT NULL,MUnitCat_Cd CHAR(6),MUnit_Nm VARCHAR(120) NOT NULL

)with replace on commit preserve rows not logged;

//Insert data into temp tableinsert into session.munit values (’MB’, ’QTY’, ’Megabytes’);insert into session.munit values (’QSC’, ’RT’, ’Quantity per sec’);insert into session.munit values (’QTY’, ’QTY’, ’Quantity’);insert into session.munit values (’KBS’, ’RT’, ’Kilobytes per sec’);insert into session.munit values (’BYS’, ’RT’, ’Bytes per sec’);insert into session.munit values (’PRC’, ’PRC’, ’Percentage’);

//Insert data from temp table into central data warehouse table//using where not exists logicinsert into twg.munit


select * from session.munitcat swhere not exists (

select 1 fromtwg.munit t wheret.munit_Cd = s.munit_Cd

);

All applications should use the following values in the Time Summary,Measurement Unit Category, Measurement Unit, Measurement Group Type,Measurement Group, and Measurement Source tables, which are created by thetwh_cdw_data.sql script during an application’s warehouse enablement packinstallation. (Check the twh_cdw_data.sql file for the latest information.) Note thatthe central data warehouse is empty after the base Tivoli Enterprise DataWarehouse installation. The following values are populated only after at least onewarehouse pack installation is completed.

Table 3. Time Summary (table TmSum)

TmSum_Cd TmSum_Nm

'H' 'Hourly'

'D' 'Daily'

'W' 'Weekly'

'M' 'Monthly'

'Q' 'Quarterly'

'Y' 'Yearly'

Table 4. Measurement Unit Category (table MUnitCat)

MUnitCat_Cd MUnitCat_Nm

'TM' 'Time Duration'

'QTY' 'Quantity'

'PRC' 'Percentage'

'RT' 'Rate'

Table 5. Measurement Unit (table MUnit)

MUnit_Cd MUnitCat_Cd MUnit_Nm

'PRC' 'PRC' 'Percentage'

'Bps' 'RT' 'Bytes per second'

'MBps' 'RT' 'Megabytes per second'

'KBps' 'RT' 'Kilobytes per second'

'Rps' 'RT' 'Requests per second'

'Qps' 'RT' 'Quantity per second'

'Qpm' 'RT' 'Quantity per minute'

'QTY' 'QTY' 'Quantity'

'GB' 'QTY' 'gigabytes'

'KB' 'QTY' 'kilobytes'

'MB' 'QTY' 'megabytes'

'B' 'QTY' 'bytes'

'MSec' 'TM' 'milliseconds'


Table 5. Measurement Unit (table MUnit) (continued)


'Sec' 'TM' 'seconds'

'Min' 'TM' 'minutes'

'Hr' 'TM' 'hours'

'Day' 'TM' 'days'

'HSc' 'TM' 'Hundredths of a second'

Table 6. Measurement Group Type (table MGrpTyp)

MGrpTyp_Cd MGrpTyp_Nm

'CATEG' 'Category'

'GROUP' 'Aggregate Types or Group Functions'

Table 7. Measurement Group (table MGrp)

MGrpTyp_Cd MGrp_Cd MGrp_Parent_Cd MGrp_Nm

'PERF' 'CATEG' NULL 'Performance'

'UTIL' 'CATEG' NULL 'Utilization'

'AVL' 'CATEG' NULL 'Availability'

'STATE' 'CATEG' NULL 'Percentage Statemeasurements'

'STORAG' 'CATEG' NULL 'Storage'

'AVG_E ' 'GROUP' NULL 'Average value exists'

'MIN_E' 'GROUP' NULL 'Minimum value exists'

'MAX_E' 'GROUP' NULL 'Maximum value exists'

'TOT_E' 'GROUP' NULL 'Total value exists'

Table 8. Measurement Source (table MSrc)

MSrc_Cd MSrc_Parent_Cd MSrc_Nm

'Tivoli' NULL 'Tivoli Application'

MeasurementsAll measurements are application-specific. The measurement source code(MSrc_Cd) uniquely identifies the application using its product code. For Tivolisoftware and other products from IBM, this is the three-character prefix assignedby IBM to a product. Third-party applications use a three-character code thatcontains at least one number. This code is used in log messages and other placeswhere an application must be uniquely identified. For example, in Table 9, themeasurement source code of CTO represents Tivoli Enterprise Console.

Table 9. Measurement Type (table MsmtTyp)

MsmtTyp_ID MSrc_Cd MUnit_Cd MsmtTyp_Nm MsmtTyp_Ds

0 'CTO' 'PRC' 'avgcpubusy' 'CPU percent busy'

Note: The MsmtTyp_ID column is a sequence number, which is populated by atrigger.


You define measurement groups in the Measurement Group (MGrp) table, asshown in Table 7 on page 17, but you associate which types of measurements areassociated with the group in the Measurement Group Member (MGrpMbr) table asshown in the following example:insert into twg.mgrpmbrselect ’MIN_E’,’GROUP’, mt.MsmtTyp_ID from TWG.MsmtTyp mtwhere MsmtTyp_Nm in ( ’avgcpubusy’);

insert into twg.mgrpmbrselect ’MAX_E’,’GROUP’, mt.MsmtTyp_ID from TWG.MsmtTyp mtwhere MsmtTyp_Nm in ( ’avgcpubusy’);

insert into twg.mgrpmbrselect ’AVG_E’,’GROUP’, mt.MsmtTyp_ID from TWG.MsmtTyp mtwhere MsmtTyp_Nm in ( ’avgcpubusy’);

insert into twg.mgrpmbrselect ’TOT_E’,’GROUP’, mt.MsmtTyp_ID from TWG.MsmtTyp mtwhere MsmtTyp_Nm in ( ’numErrors’ );

The following defines which types of measurements are appropriate for whichtypes of components:insert into twg.msmtrulselect ’HOST’, mt.MsmtTyp_ID from twg.msmttyp mtwhere MsmtTyp_Nm in (’avgcpubusy’)

;


Chapter 3. Implementing the extract, transform, and loadprocess

This chapter describes how to implement extract, transform, and load (ETL)processes for Tivoli Enterprise Data Warehouse. It provides information describingthe tools used to create ETLs, common tasks for ETL developers, and ETLguidelines.

Using the Data Warehouse CenterIBM DB2 Universal Database provides the Data Warehouse Center, a componentthat automates data warehouse processing. The Data Warehouse Center allows youto define a data extraction and transformation process that puts data into awarehouse. It supports the extraction of data from flat files and most vendordatabases with Open Database Connectivity (ODBC) drivers, provides a GUI tobuild SQL that transforms the data to the warehouse schema, and populates theDB2 warehouse. There is also a scheduling facility that automates the running ofthe extraction and the transformation process.

Note: If you are extracting data from flat files, each file must be defined as anODBC data source.

The extraction and transformation process is modeled as a series of steps. Toidentify and group processes that relate to a logical area of the business, you firstdefine a subject area. To define how data is to be moved and transformed for thedata warehouse, you define a process, which contains a series of steps in thetransformation and movement process, within the subject area. Each step defines atransformation of data from a source format to a target format. Figure 2 on page 20shows the process model for a sample process namedSPP_c05_Extract_Inventory_Data_Process. This process consists of a step namedSPP_c05_s010_extractInvData.

19

With large applications, the number of SQL steps gets very large andunmanageable. One or more sources are input to an SQL step or user-definedprogram and one or more targets are output. A user-defined program is a programthat you create or a program that is created by a vendor. You define a user-definedprogram to the Data Warehouse Center so that one or more steps can use theprogram for processing.

Tivoli Enterprise Data Warehouse provides a user-defined program namedSQLScript, which reduces the number of steps and allows you to develop and testa transformation process outside of the Data Warehouse Center. The SQLScriptprogram takes an SQL script as input and runs the statements on the source,target, or source and target. When integrated into the Data Warehouse Center, aprocess has at least one source table as input to the user-defined program, which issimply running an SQL script, and optionally one target table for output. SeeFigure 3 on page 21. The wrapper script, sqlscript.sh, calls the execsql.exe. Theexecsql.exe program, or SQL execution engine, can read or write to tables that arein the same databases as the input table or the output table. Note that the inputand output tables can reside in different databases. This is accomplished with a

Figure 2. A sample process model in the Data Warehouse Center


few directives like ″execute at target″ or ″execute at source″. See “Using the SQLexecution engine” on page 22 for more information.

Once an ETL process is built, the Data Warehouse Center provides an exportmetadata facility that allows you to export your ETL processes to an ETLinterchange file (*.tag), which can be imported at another Data Warehouse Center.You might want to import sample data into the warehouse or you might want toimport data if you are creating a prototype. This facility can also be helpful formoving from a development environment to a production environment.1. From the Data Warehouse Center window, right-click Warehouse and select

Export Metadata → Interchange File. The Export Metadata window opens.2. Under Available objects, expand the items under Subject Areas by clicking the

plus sign (+).3. Select the subject area you want to export as shown in Figure 4 on page 22 and

click the right arrow to add the subject area to the Selected objects.

sqlscript.sh

execsql.exe

script_name

Figure 3. Process flow

Chapter 3. Implementing the extract, transform, and load process 21

4. Under Available objects, expand the items under Warehouse Schemas byclicking the plus sign (+).

5. Select the warehouse schemas you want to export and click the right arrow toadd the schemas to the Selected objects.

6. Keep the Export depended source properties and Include process schedulescheck boxes selected.

7. Click OK to begin the export process.

Note: Do not select Warehouse Sources or Warehouse Targets when exportingmetadata.

For more information on the Data Warehouse Center, see IBM DB2 UniversalDatabase Data Warehouse Center Administration Guide.

Using the SQL execution engineAn extract, transform, and load process that reads data from a source applicationand places the data into the central data warehouse database is called a central datawarehouse ETL. A central data warehouse ETL is developed within, and scheduledand run by, the IBM DB2 Data Warehouse Center.

Due to the distributed nature of Tivoli software applications and the TivoliEnterprise Data Warehouse, some features required for central data warehouse ETLprocesses are not supported by the Data Warehouse Center. Specifically, a centraldata warehouse ETL must update the source database (application database) withinformation on which records are already loaded into the warehouse. SQL steps inthe Data Warehouse Center are not designed to offer the facility to update thesource database during the ETL process.

Tivoli Enterprise Data Warehouse provides a tool called the SQL execution engine,which augments the DB2 Data Warehouse Center functionality. It consists primarilyof a C program (platform dependent) that allows the caller to pass in a series ofSQL statements and have them run on any ODBC data source (for example, a

Figure 4. Exporting metadata using the Data Warehouse Center


database). A wrapper script is provided with the C program to make it easilycallable from a Data Warehouse Center ETL process step.

The SQL execution engine provides the following benefits to the ETL developer:v Allows updates to source databasesv Significantly improves the run-time performance of SQL statementsv Reduces the time required to install an application’s ETL scripts on the Data

Warehouse Center machinev Makes ETL processes defined in the Data Warehouse Center simpler

You can also use the SQL execution engine when working on data mart ETLprocesses (moving data from the central data warehouse to a data mart). Thissection discusses central data warehouse ETL exclusively because a majority ofissues encountered during data mart ETL development are similar to the centraldata warehouse ETL process.

ETL developers should run as much central data warehouse ETL (and data martETL) SQL as possible using the SQL execution engine instead of using the DB2Data Warehouse Center SQL steps. The rest of this section describes how to set upthe SQL execution engine in DB2 Data Warehouse Center and how to use it in acentral data warehouse ETL process.

SQL execution engine overviewThe SQL execution engine and its wrapper script are installed by the TivoliEnterprise Data Warehouse installation software on the machine that hosts theTivoli Enterprise Data Warehouse control server. If the Tivoli Enterprise DataWarehouse central data warehouse or Tivoli Enterprise Data Warehouse data martsare installed on other systems, then the SQL execution engine and its wrapperscript are installed on those systems also.

The execsql.exe and sqlscript.sh files are placed in the TWH_TOPDIR\tools\bindirectory along with bash and other executable programs required by TivoliEnterprise Data Warehouse, where TWH_TOPDIR is the environment variable thatrepresents the base installation directory for Tivoli Enterprise Data Warehouse.)The version of the bash shell provided with Tivoli Enterprise Data Warehouse isassumed to be the only version of bash to be used with ETL scripts. Theinstallation program extends the PATH environment variable to include theTWH_TOPDIR\tools\bin directory.

During a Tivoli Enterprise Data Warehouse installation, the DB2 Data WarehouseCenter is automatically modified to include SQLScript as a user-defined programor transformer. This is accomplished by importing a user-defined programdescription into the Data Warehouse Center. To view the user-defined program:1. From the Data Warehouse Center, expand the Administration folder.2. Expand the Programs and Transformers folder.3. Expand the User-Defined Program and Transformers folder.4. Expand the Tivoli folder. The SQLScript program is listed along with rollup

and runReport. For information on rollup and runReport, see “Aggregation androllup” on page 44 and “Report scheduling” on page 57.

The SQL execution engine handles any type of SQL statement. The SQL can be runin one of three places:v SQL to be run on the source data source (application database)v SQL to be run on the target data source (central data warehouse database)


v SQL that runs on both the source and target data sources using thecross-data-source INSERT INTO SQL construct.

To run SQL statements, the SQL execution engine makes ODBC calls on ODBCdrivers. ODBC data sources that correspond to source and target databases mustbe defined on the machine where the SQL execution engine runs.

SQL execution engine setupThe execsql.exe and sqlscript.sh files that compose the SQL execution engine mustbe placed in a path that can be located by the DB2 Data Warehouse Center. In atypical Tivoli Enterprise Data Warehouse installation, the sqlscript.sh, execsql.exe,and bash programs are located in the TWH_TOPDIR\tools\bin directory. After theinstallation of Tivoli Enterprise Data Warehouse, this directory is placed in thePATH environment variable.

The ETL developer must write one or more custom scripts that contain one ormore SQL statements to be run by the SQL execution engine. The developer canuse a text editor of choice to develop the custom script, or scripts. The SQL to berun must fit one of the three types described in “SQL execution engine overview”on page 23.

Note: The SQL execution engine does not work with network mounted drives.SQL scripts run by the SQL execution engine must be on a local drive.

After the custom scripts are developed, they must be put in a particular directorypath to work with the DB2 Data Warehouse Center and the SQL execution engine.The following directory path simulates the path to an application’s ETLinstallation. The application ETL scripts are installed by the Tivoli Enterprise DataWarehouse installation program using application-supplied media that contains therelevant scripts. The scripts are installed in the following location:

TWH_TOPDIR\apps\product_code\vversion\etl\sql\yourCustomScript

where:

TWH_TOPDIRIs the base installation directory for Tivoli Enterprise Data Warehouse.

product_codeIs the code that uniquely identifies an application. IBM and Tivoli softwareapplications must use the three-character code assigned by IBM to aproduct. Third-party applications must also use a three-character code thatincludes at least one number. This code is used in log messages and otherplaces where an application must be uniquely identified.

version Is an integer value representing the version of the ETL scripts (for example,110).

yourCustomScript

Is the script name. The script name must use the following namingconvention, which is also defined in Chapter 5, “Naming conventions” onpage 65.

part1_part2_part3_part4.part5

where:

part1 Is the product code for the application.

part2 Is the process ID, where cnn indicates a process where the target is


the central data warehouse and mnn indicates a process where thetarget is a data mart. The two-digit number represented by nn is aunique process number assigned by the application for its ETLprocesses. The first process should be numbered starting with 05.Subsequent processes should add 5 to the initial number (05, 10,15, and so on). Lower to higher numbers in part 2 of the scriptname indicates the order in which the script runs.

part 3 Is the step name where snnn represents a particular step in theprocess. The first step in a process should start with 010 andsubsequent steps should increment the step count by 10.

part4 Is the step description. It is recommended that the table namebeing populated be supplied as the description. Component tablesare represented by a name like X _Comp. Measurement tables areX_Msmt. Dimension and fact tables are Dim_tablename andFact_tablename, respectively. With the execsql.exe program you canpopulate many tables at the same time, so an extension namebased on a target table might not always make sense.

part5 Is a suffix that describes the database vendor type to be used. Thepossible suffix values are:

.db2 DB2

.oracle Oracle

.sybase Sybase

.informix Informix

.mssql Microsoft® SQL Server

.tsm Tivoli Storage Manager ODBC driver

When the SQL execution engine runs, part5 of the name (including the period) isexcluded from the custom script name passed as a parameter to the program. Theexecsql.exe program automatically generates a suffix based on the ODBC datasource definition and appends this value to the custom script name at run time.

If no matches to the suffix list are found, a log message is generated by theexecsql.exe program and the execsql.exe program attempts to use the custom filewithout any extension.

Note: When setting up warehouse sources in the Data Warehouse Center, specifyGeneric ODBC in the Warehouse source type list. This enables you to have asingle step that can read from any ODBC data source; for example, DB2,Oracle, SQL Server, Sybase, Informix. The execsql.exe program checks thevendor type of the ODBC data source and looks for a script named asfollows:

scriptname.db2for DB2

scriptname.oraclefor Oracle

scriptname.mssqlfor Microsoft SQL Server

scriptname.sybasefor Sybase


scriptname.informixfor Informix

By default, the SQL statements are run against the source data source. Thefollowing control statements can precede any SQL statement in the custom scriptthat you want run:

--#INSERT_INTO_TARGETRequires the words INSERT INTO table (followed by a SELECT statement) inthe SQL statement, where table is the name of a table that exists on thetarget data source. Otherwise, this command is ignored and the SQLstatement is run entirely on the source data source.

If the table name is not qualified with the schema name (for example,eco.mytable), the value for user will determine the schema name used inOracle, DB2, and Informix. For Sybase and MSSQL, the SQL executionengine uses the default table accessible by the user.

--#INSERT_INTO_SOURCEEvaluates the following INSERT INTO table SQL statement, pulling datafrom the TARGET data source and inserting it into the specified table in theSOURCE data source.

--#EXECUTE_AT_TARGETRuns the following SQL statement at the target data source instead of atthe source data source (default).

--#IGNORE_ERRORIgnores any database error messages returned by the following SQLstatement. The error is still logged but it does not prevent subsequent SQLstatements in your custom script from running. For example:--#IGNORE_ERRORdrop table cto.stage_example;

create table cto.stage_example like cto.template_stage_example;

Note that system errors such as running out of memory cannot be ignored.

--#EDIT_USING perl convert.plUsed in conjunction with the --#INSERT_INTO_* directives. Allows datafrom the SQL SELECT to be written into a temporary file in CSV format.The Perl script provided after the directive is called. A ″base″ temporaryfile name is provided in the following format:

VWS_LOGGING/execsqlTMP.data_source.nn

where:

VWS_LOGGINGIs the DB2 logging directory

data_sourceIs the source ODBC data source name

nn Is an integer used to separate multiple --#EDIT_USING directives ina script (starts at 0 and increments per directive)

The script is then required to append ″.preedit″ to the temporary file nameto determine the actual file name to read from. The script is also requiredto append ″.postedit″ to the base temporary file name it outputs the altered


data to. The *.postedit file must also be in CSV format. To provide forhandling of double quotes, an escape character ″\″ is inserted immediatelyprior to the embedded double quote on both *.preedit and *.postedit files.Failure to account for this in the script could result in damaged data. Notethat double quotation marks are used to contain SQL_CHAR andSQL_VARCHAR column data. Only SQL_CHAR and SQL_VARCHARcolumn types are supported by --#EDIT_USING.

--#OVERRIDE_TERMINATORChanges the default SQL statement terminator from a semicolon (;) towhatever the first non-whitespace character is following the directive. Thechange is only applied to the current statement. The statement followingthe altered terminator is again ended with a semicolon. For example, thefollowing sets the terminator to a $ instead of a semicolon on the currentstatement only:--#OVERRIDE_TERMINATOR $CREATE TRIGGER DMN.CompTyp_Cd_Trig NO CASCADEBEFORE INSERT ON DMN.stage_compREFERENCING NEW AS NFOR EACH ROWMODE DB2SQLBEGIN ATOMICIF LOCATE(’.’,N.Comp_Nm)>0 THEN

IF LOCATE(’.’,N.Comp_Nm,LOCATE(’.’,N.Comp_Nm)+1)>0 THENSET N.CompTyp_Cd = ’IP_HOST’;

ELSESET N.CompTyp_Cd = ’DMN_HOST’;

END IF;ELSE

SET N.CompTyp_Cd = ’DMN_HOST’;END IF;END$

Notes:

1. Comments are signified by lines that start with two dashes and a characterother than the # symbol.

2. You can have two comments for a single SQL statement. For example:--#IGNORE_ERROR--#EXECUTE_AT_TARGETdrop table DM.stage_dm_metrics;

3. SQL statements in the script must be ended by a semicolon (;). For example:INSERT INTO TSM.ADMINS (SELECT ADMIN_NAME FROM ADSM.ADMINS);

4. If there is an imbalanced quote condition (failure to close the quote on a stringliteral), the SQL execution engine logs an error and stops.

5. If there are non-whitespace characters (excluding comments) following the finalsemicolon (for example, an SQL command at the end that is not ended by asemicolon), a warning message is logged and the SQL statement is ignored.

6. The following directives fail if they are used with parenthetical lists. Do not useparenthetical lists in the INSERT INTO statement when using these directives.

--#INSERT_INTO_SOURCE

--#INSERT_INTO_TARGET

--#EDIT_USING

For example, assume the following:v Table tgttable has four columns: tgtcol1, tgtcol2, tgtcol3, and tgtcol4v Table srctable has three columns: srccol1, srccol2, and srccol3


The following SQL statement, which excludes tgtcol4, fails in execsql whenusing one of the directives listed above.insert into tgttable (tgtcol1, tgtcol3, tgtcol2)select srccol1, srccol2, srccol3 from srctable

In this example, using (tgtcol1, tgtcol3, tgtcol2) returns an error becausethe SQL execution engine attempts to run ″(tgtcol1, tgtcol3, tgtcol2)select srccol1, srccol2, srccol3 from srctable″ as if it were a proper selectstatement, which results in an SQL error. If you do not want to insert data intocolumn tgtcol4, use NULL in the select list ordinal position that matches thetarget column in the target table. For example:insert into tgttable select srccol1, srccol3, srccol2, NULL from srctable

This inserts the data from srccol1 into tgtcol1, srccol3 into tgtcol2, srccol2into tgtcol3 and NULL into tgtcol4, achieving the same result as the originalinsert statement.

The following shows an example using the --#INSERT_INTO_TARGET controlstatement:-- Fills in the twg.compattr table with transaction components attribute: TX_DETAIL--#INSERT_INTO_TARGETinsert into twg.compattrselect0,a.comp_id,c.attrtyp_cd,current timestamp as compattr_strt_dttm,’9999-01-01-00.00.00.000000’ as compattr_end_dttm,casewhen b.tx_detail is null then ’’else b.tx_detailend as compattr_valfromtwg.comp a,apf.stage_tx_attr_copy b,twg.attrrul cwherea.comptyp_cd = ’TAPM_TX’and a.comp_nm = b.tx_nameand a.comptyp_cd = c.comptyp_cdand c.attrtyp_cd = ’TX_DETAIL’;

Manually testing the custom scriptBefore attempting to run your custom script from the Data Warehouse Center, runthe script manually to validate the script.

To run the script manually, start bash from a Microsoft Windows® commandprompt. The bash program is installed on your system when the Tivoli EnterpriseData Warehouse software is installed. Be sure you are using this copy of bash foryour testing.

Enter the following information, where script_name is the name of your customscript; for example, SPP_c05_s010_extractInvData. (For details on how to nameyour scripts, see Chapter 5, “Naming conventions” on page 65.)

sqlscript.sh product_code script_name source_db source_uid source_pwd target_dbtarget_uid target_pw


The SQL in your custom script should run and you should see the results in yoursource or target database.

Setting up the DB2 Data Warehouse CenterTo use the SQL execution engine in the Data Warehouse Center, the program andscript must be configured. The Tivoli Enterprise Data Warehouse installationprogram automatically installs sqlscript.sh as a user-defined program. However,this section describes the steps necessary to configure the SQL execution engine inthe Data Warehouse Center from scratch so that a developer can understand theprocess.1. Start the Data Warehouse Center:

a. Click Start → Programs → IBM DB2 → Control Center.b. In the DB2 Control Center window, click Tools → Data Warehouse Center.

The Data Warehouse Center Logon window opens.c. Supply login information for the control database of the Data Warehouse

Center. After login, close the Data Warehouse Center Launch Pad if itappears.

2. In the left pane of the Data Warehouse Center window, expand the itemsunder the Administration Selection by clicking the plus sign (+). Continue toexpand the tree as follows: Program and Transformers → User-DefinedProgram and Transformers.

3. Right-click the User-Defined Program and Transformers folder and selectDefine Group. The Program Group dialog opens.

4. Type Tivoli in the Name field and supply an appropriate description and anyrelevant notes. Leave the Administrator field as Default DWC User and clickOK. A new entry, Tivoli, appears under the User-Defined Programs andTransformers folder in the left pane of the window.

5. Right-click the Tivoli Program Group and select Define Program. The DefineProgram window opens.

6. Specify information for your program:v In the Name field, type sqlscript.v In the Program name field, type bash.v Leave the Administrator field as Default DWC User.

7. Click the Parameters tab. Click Add and select Step Name from the SelectParameters window. Leave the type as System Parameters. Click OK.A new step is added in the Define Program window. Do this two more timesto create a total of three steps (the information in the steps is edited later inthis process).

8. Click Add and select the Warehouse source database name selection in theSelect Parameters window.

9. Click Add again and select the Warehouse source user id selection in theSelect Parameters window and click OK.

10. Click Add again and select the Warehouse source password selection in theSelect Parameters window and click OK.

11. Click the Add button three more times to create parameters for Warehousetarget database name, Warehouse target user id, and Warehouse targetpassword.

Nine parameters are now defined in the Parameters tab of the Properties -SQLScript window.12. Change the name of the first parameter to sql execution script. Edit the

Value field of the first parameter and change it to sqlscript.sh


13. Change the name of the next parameter to product code. Edit the Value fieldof the second parameter and change it to PRODUCT_CODE.

14. Change the name of the next parameter to custom script name. Edit the Valuefield of the third parameter and change it to scriptname.

15. Leave the Type setting on these three parameters as Character. Click ShowCommand and you should see the following:

sqlscript.sh PRODUCT_CODE scriptname &SDB &SUID &SPWD &TDB &TUID &TPWD

16. Click Close to close the Show Command window.17. In the Properties - SQLScript window, click the Agent sites tab. Select the

Default DWC Site entry in the Available agent sites pane and move it to theSelected agent sites pane. Click OK and the Properties-SQLScript windowcloses and your changes are saved.

To test the SQL execution engine setup, define a sample process with a samplesource and target tables:1. From the Data Warehouse Center tree, right-click the Subject Areas folder and

select Define. The Subject Area Properties notebook opens.2. In the Name field, type TestSubject. Leave the Administrator as Default

DWC User. Click OK to create the TestSubject subject area in the DataWarehouse Center tree.

3. Expand the entries under the TestSubject object by clicking the plus sign (+)in the left pane.

4. Right-click Processes and select Define. The Define Process notebook opens.5. In the Name field, type TestProcess. Leave the Administrator as Default

DWC User. Click OK. The new process is displayed when you expand theProcesses folder.

6. Open TestProcess by double-clicking its entry in the left pane. The ProcessModel dialog opens.

7. Create a source table in the process. Create a target table in the process. Notethat in many cases a target table is not required by the Data WarehouseCenter when running a user-defined program. However, the execsql.exeprogram requires a source table.

8. Select the User-Defined Programs and Transformers icon (the only one with apicture of a person) and select Tivoli → SQLScript.

9. Connect the TestProcess-SQLScript Transformer to the source table and to thetarget table using the Data Warehouse Center GUI.

10. Right-click the TestProcess-SQLScript Transformer step and select Properties.Select the Parameters tab on the Properties - TestProcess - SQLScript dialog.

11. Replace PRODUCT_CODE with the application’s product code and replacescriptname with the name of the custom script created above.

12. Click OK and put the TestProcess-SQLScript step in Test mode.13. Run the process.

Return codes from execsql.exeReturn codes from execsql.exe:// return codes:// 0 : success// 8085 : destination table does not exist// 8086 : wrong number of args// 8087 : check for SQL error details in logfile// 8088 : unable to open log file for writing


// 8089 : unable to open sql file for reading// 8090 : a single sql statement exceeds buffer// 8091 : unable to allocate memory// 8092 : # dest columns != # src columns// 8093 : creating the insert statement failed// 8094 : Target datasource was used, but not provided in command line// 8095 : unbalanced quote in script// 8100 : write unsupported chartype to CSV file// 8101 : A column > 4000 was attempted to be displayed or written

Debugging suggestionsThe SQL execution engine creates a log file that can be helpful in debuggingproblems with the ETL. This log file, yourCustomScriptName.log, is placed in theDB2 logging directory, which is specified by the VWS_LOGGING environmentvariable.

See Appendix D, “SQL execution engine data transfer support” on page 135 fordetails describing the data type transfers supported by the SQL execution engine.

Problems frequently occur when path information is not available to the DataWarehouse Center or path information is not available for the sqlscript.sh scriptthat is run from the Data Warehouse Center. If you encounter problems where theSQLScript program fails to run at all from the Data Warehouse Center, tryproviding explicit paths in the step definition as follows:1. In the Data Warehouse Center, right-click the step and select Properties.2. In the Properties window, select the Parameters tab.3. Double-click the Parameter value field for the sql execution script parameter

and type the new parameter value. For example,C:\TWH\tools\bin\sqlscript.sh

You might encounter problems if you are using a different version of bash or lsthan the versions included with Tivoli Enterprise Data Warehouse. The sqlscript.shscript relies on the ls command to work properly and it is known that certainversions of bash do not work properly in the script environment. Use the typecommand in a bash shell to determine the location of the ls program on yourmachine and to verify its version.

It might be helpful to add echo statements to the sqlscript.sh script (and send theoutput to a file) to determine if it is being called at all by the Data WarehouseCenter, and if it is being called, where the failures are occurring.

Incremental extracts using extract controlExtract control is a method to enable data to be incrementally extracted from asource database by ETL processes. Typically, ETL processes are run once in a 24hour period. Each ETL run should extract only the data that has appeared in thesource database since the previous successful ETL run. Extracting all themeasurement data from each source database every ETL run would take too muchtime and too many resources. Also, because the central data warehouse is apermanent record of component insert, update, and delete activity, changes to thesource database that are required to be copied to the central data warehouse musthappen exactly once.

The following are required to perform extract control:v A way to identify when or what order measurements are generated


1. Typically, measurement data has some kind of timestamp that relates to thetime that the measurement arrived in the source database. This timestampcan be used as the field on which to base the extracts.

2. Another alternative is that the source database assigns an ascending integeror binary sequence number to each measurement.

3. If neither 1 nor 2 is true, an extra column might need to be added to thesource table. This field can be a timestamp, an ascending sequence number,or any other type that can successfully identify whether data records havealready been placed in the data warehouse. The column can be populated bya trigger or by a function in the source application. This extra column can beadded to an existing table or you can create a new table that includes all ofthe primary key columns of an existing table along with a sequence ortimestamp column.

v Use of the SQL execution engine (execsql.exe)The extract control design requires that data in temporary tables in theapplication specific schema of the central data warehouse be written to thesource database, which can be a non-DB2 database. A record containing the lastsource record to be extracted (identified by sequence number of timestamp) issaved in a temporary table in the source database so it can be joined in an SQLquery when selecting new records. The SQL execution engine enables data to becopied from one database to another by way of ODBC. (See Appendix D, “SQLexecution engine data transfer support” on page 135 for details describing thedata transfer supported by the SQL execution engine.)

The following two central data warehouse tables are used in extract control, whereTWG stands for Tivoli Warehouse Generic:v TWG.Extract_Control - 1 row per source table to target table combination

This table is used to control each extract.v TWG.Extract_Log - 1 row per execution of each source table to target table

combinationThis table is used as a log for all extracts.

These tables are created in the central data warehouse database during the TivoliEnterprise Data Warehouse installation using the following table creationstatements:CREATE TABLE TWG.Extract_Control (

ExtCtl_Source VARCHAR(120) NOT NULL,ExtCtl_Target VARCHAR(120) NOT NULL,ExtCtl_From_RawSeq CHAR(10) FOR BIT DATA,ExtCtl_To_RawSeq CHAR(10) FOR BIT DATA,ExtCtl_From_IntSeq BIGINT,ExtCtl_To_IntSeq BIGINT,ExtCtl_From_DtTm TIMESTAMP NOT NULL,ExtCtl_To_DtTm TIMESTAMP NOT NULL,CONSTRAINT XPKExtract_Control

PRIMARY KEY (ExtCtl_Source, ExtCtl_Target));

CREATE TABLE TWG.Extract_Log (ExtLog_Source VARCHAR(120) NOT NULL,ExtLog_Target VARCHAR(120) NOT NULL,ExtLog_Done_DtTm TIMESTAMP NOT NULL,ExtLog_From_RawSeq CHAR(10) FOR BIT DATA,ExtLog_To_RawSeq CHAR(10) FOR BIT DATA,ExtLog_From_IntSeq BIGINT,ExtLog_To_IntSeq BIGINT,


ExtLog_From_DtTm TIMESTAMP NOT NULL,ExtLog_To_DtTm TIMESTAMP NOT NULL,CONSTRAINT XPKExtract_Log

PRIMARY KEY (ExtLog_Source, ExtLog_Target,ExtLog_Done_DtTm)

);

Each source table to target table combination is identified by the following in theTWG.Extract_Control table:

ExtCtl_SourceThe source table name

ExtCtl_TargetThe target table name

Each source table to target table combination can choose to select data to beextracted based on the following:v A timestamp fieldv An integer fieldv A binary field (10 bytes)

For a timestamp field, the ExtCtl_From_DtTm and ExtCtl_To_DtTm columns areused. For an integer field, the ExtCtl_From_IntSeq and ExtCtl_To_IntSeq columnsare used. For a binary field, the ExtCtl_From_RawSeq and ExtCtl_To_RawSeqcolumns are used. An initial row is inserted into the TWG.Extract_Control table foreach by the product_code_cdw_data.sql script. This sets up the from and to valuesfor the first extract. The from and to values make up the extract window, or timeperiod of the extract.

Run-time example of extract controlIn the following example, a column named insert_dttm exists in the table(src_table) in the source database, which holds the measurement data.


Source Database Central Data Warehouse Database

1. drop/create temp table to hold extract window(temp_extract_control)

2. b. insert a row in temp_extract_controlwith the "from" value <======== a. get the "from" value

(used for SQL join operations) for the source table totarget table combination(get all date/times largerthan this value)

3. update temp_extract_control set the"to" value = max(insert_dttm) from src_table10 seconds - so we do not try to grab recordsthe application is just placing in the database

4. ..................................................... drop/create stage table

5. a. select from src_table, temp_extract_controlwhereinsert_dttm > extctl_from_dttm andinsert_dttm <= extctl_to_dttm ==========> b. insert into stage table

6. .................................................. drop/create table to hold"to" value (temp_to)

7. select "to" value from temp_extract_control ======> insert into temp_to

8. ................................................. update TWG.extract_controlset to_dttm = select "to"value from temp_to

9. ................................................. insert into TWG.extract_logwith the extract window,current timestamp

10. ................................................ trigger runs that updatesextract_control set"from" = "to"

SQL script examples of incremental extractThe following is an example of incremental extract, which uses a binary sequencenumber to base extracts on. This example is a subset of what goes into yourcustom SQL script, or scripts. As you read the SQL script example, follow alongwith the run-time example. See “Run-time example of extract control” on page 33.

Note: The following table is part of the Tivoli Distributed Monitoring applicationdatabase and was added to support central data warehouse extract control.

CREATE TABLE DM.TWH_DM_METRICS (COLLECTION_DATE DATE NOT NULL,HOSTNAME VARCHAR(32) NOT NULL,ENDPOINT VARCHAR(32) NOT NULL,PROFILE_COLLECTION VARCHAR(64) NOT NULL,PROBE_COLLECTION VARCHAR(32) NOT NULL,PROBE VARCHAR(32) NOT NULL,PROBE_ARG VARCHAR(32) NOT NULL,TWH_INSERT_SEQ CHAR(10) FOR BIT DATA NOT NULL,TWH_GMTOFFSET_MIN SMALLINT NOT NULL

);

-- ////////////////////////////////////////////////////-- DO INCREMENTAL EXTRACT - source database is DB2-- Src=spr_rim Tgt=twh_cdw


-- ////////////////////////////////////////////////////---- //////////////// drop table to hold the extract window-- //////////////// at spr_rim----#IGNORE_ERRORdrop table temp_extract_control;-- //////////////// create table to hold the extract window-- //////////////// at spr_rim--create table temp_extract_control (

extctl_from_rawseq char(10) for bit data not null,extctl_to_rawseq char(10) for bit data not null)

;-- //////////////// get the from_rawseq from the extract_control in the CDW-- //////////////// from twh_cdw to spr_rim----#INSERT_INTO_SOURCEinsert into temp_extract_controlselectextctl_from_rawseq,x’00000000000000000000’fromtwg.extract_controlwhereextctl_source=’DM_METRICS’ andextctl_target=’DM.STAGE_DM_METRICS’;-- //////////////// update with the max twh_insert_seq-- //////////////// at spr_rim-- //////////////// twh_dm_metrics is new table create for extract control

--update temp_extract_controlset extctl_to_rawseq = (select max(twh_insert_seq) from twh_dm_metrics);-- //////////////// drop the stage table in twh_cdw-- //////////////// at twh_cdw----#EXECUTE_AT_TARGET--#IGNORE_ERRORdrop table DM.stage_dm_metrics;-- //////////////// create the stage table in twh_cdw-- //////////////// at twh_cdw

--#EXECUTE_AT_TARGETCREATE TABLE DM.stage_dm_metrics like DM.stage_dm_metrics_template;;-- //////////////// do the extract-- //////////////// from spr_rim to twh_cdw

--#INSERT_INTO_TARGETinsert into DM.stage_dm_metricsselectd.COLLECTION_DATE, d.DT_STAMP, d.HOSTNAME, d.ENDPOINT, d.PROFILE_COLLECTION,d.PROBE_COLLECTION, d.PROBE, d.PROBE_DESC, d.PROBE_ARG, d.MIN_VALUE_00,d.MAX_VALUE_00, d.AVG_VALUE_00, t.TWH_INSERT_SEQ, t.TWH_GMTOFFSET_MINfromDM_METRICS d,TWH_DM_METRICS t,temp_extract_control ecwhered.COLLECTION_DATE=t.COLLECTION_DATE andd.HOSTNAME=t.HOSTNAME andd.ENDPOINT=t.ENDPOINT and


d.PROFILE_COLLECTION=t.PROFILE_COLLECTION andd.PROBE_COLLECTION=t.PROBE_COLLECTION andd.PROBE=t.PROBE andd.PROBE_ARG=t.PROBE_ARG andt.TWH_INSERT_SEQ > ec.extctl_from_rawseq andt.TWH_INSERT_SEQ <= ec.extctl_to_rawseq;-- //////////////// drop to_rawseq temp table in CDW-- //////////////// at twh_cdw----#EXECUTE_AT_TARGET--#IGNORE_ERRORdrop table DM.temp_to_rawseq;-- //////////////// create to_rawseq temp table in CDW-- //////////////// at twh_cdw----#EXECUTE_AT_TARGETcreate table DM.temp_to_rawseq (

extctl_to_rawseq char(10) for bit data not null);-- //////////////// copy to_rawseq value into temp table in CDW-- //////////////// from spr_rim to twh_cdw----#INSERT_INTO_TARGETinsert into DM.temp_to_rawseqselect extctl_to_rawseq from temp_extract_control;-- //////////////// update extract control with to_rawseq value-- //////////////// at twh_cdw----#EXECUTE_AT_TARGETupdate TWG.extract_controlset extctl_to_rawseq = (select extctl_to_rawseq from DM.temp_to_rawseq)whereextctl_source=’DM_METRICS’ andextctl_target=’DM.STAGE_DM_METRICS’;-- //////////////// write extract info to extract log-- //////////////// this runs the trigger that updates extract control to-- //////////////// close the window----#EXECUTE_AT_TARGETinsert into TWG.extract_logselectextctl_source, extctl_target,current timestamp - current timezone,extctl_from_rawseq, extctl_to_rawseq,extctl_from_intseq, extctl_to_intseq,extctl_from_dttm, extctl_to_dttmfromTWG.extract_control ecwhereec.extctl_source=’DM_METRICS’ andec.extctl_target=’DM.STAGE_DM_METRICS’;

In the following example, the source database does not contain a timestamp orascending sequence number in the source database measurement table. Theexample creates a separate table that contains a timestamp and the keys from theoriginal source table so that extract control can be performed. The execsql.exedirectives in front of each SQL statement are purposely omitted.

Note: Additional extract control logic is required once the data is in the stagingtables in the central data warehouse, which is not shown in this example.


--% sample source table--% This application has a metric name--% and then has an associated measurement.--% There is no information in this table to--% tell if this is already in the warehouse or not.create table sample_source (

metric_name varchar(120) not null,metric_value bigint

);

--% statement defines the primary key of the sample tablealter table sample_source create primary key (metric_name);

--% define the NEW additional table required to perform--% the extract control logic--% The table must contain the primary key of the source--% table plus the additional timestamp field for extract--% controlcreate table req_extctrl (

metric_name varchar(120) not null,timereported timestamp

);

--%trigger to tie the source able with--%the new extract control tablecreate trigger extctrl_trackingafter insert on sample_sourcereferencing new as nfor each row mode db2sqlinsert into req_extctrl(metric_name, timereported)values (n.metric_name, current timestamp);

--% created in the source database--% required to create this extra--% table because joins cannot be--% done across databases. This--% is used to control what actually--% gets selected by the etl logic--% during a particular run.drop temp_extract_control;create table temp_extract_control (

extctl_source varchar(120),extctl_from_dttm timestamp,extctl_to_dttm timestamp

);

--% insert a record into the temp extract control table--% that was created in the source database for this ETL--% runinsert into temp_extract_controlselect

extctl_source,extctl_from_dttm,extctl_to_dttm

fromtwg.extract_control

whereextctl_source = ’sample_source’ andextctl_target = ’myapp.cdwdata’

;

--% make the to time 10 seconds in the past so--% we don’t end up grabbing data that is just


--% coming in. We will get that data next timeupdate temp_extract_controlset extctl_to_dttm = current_timestamp – 10 seconds;

--% actually does the selection of the data--% this is inserting data into the app specific--% staging table in the cdw database prior to--% putting the data in actual cdw tablesinsert into myapp.cdwdataselect ss.metric_name,

ss.metric_valuefrom sample_source ss, req_extctrl rxc, temp_extract_control xcwhere ss.metric_name = rxc.metric_name and

rxc.timereported >= xc.extctl_from_dttm andrxc.timereported <= xc.extctl_to_dttm;

Recoverability in case of failureETL processes must be written so that if a failure occurs (for a variety of reasons),the process can be rerun without duplicating or corrupting data already in thecentral data warehouse database. In some cases, manual intervention by a databaseadministrator might be required to correct a problem prior to running the ETL.Because ETLs are typically scheduled to run automatically, the ETL logs should bechecked after each run to assure that manual intervention is not required to correctproblems before the next automated ETL run.

This section describes the following:v How to code an ETL process to be recoverable in certain failure scenariosv What customers would have to do to ensure recoverability in case of failure

For example:– If they can fix the problem (due to lack of space) before the next ETL cycle,

then they can just rerun the ETL.– If they cannot or do not fix the problem before the next ETL cycle, then they

must rely on database backups.

The extract control method should cover all the failure cases.


Table 10 on page 40 describes each of the failure cases depicted in Figure 5 andtheir recovery implementation.

RIMdatabase

Central datawarehousedatabase

Data martdatabase

Fact tables

Rim_table

Stage_orig_table

Stage_table(after exception process)

Comp, CompAttr,CompReln, Msmt tables

Stage_cdw_dimensiontables

Dimension tables

1

2

3

4

5

6

}}Data mart

ETL

Central datawarehouse

ETL

Figure 5. Failure cases


Table 10. Failure and recovery scenarios

FailureCase

Description Recovery Implementation

Central data warehouse ETL failure cases

1 Failure during the transfer of data from sourcetable in the RDBMS Interface Module (RIM)database to a staging table in the central datawarehouse database. This transfer is crossdatabase. The target table is a staging table (thatis persistent across connections) whose contentscan be overwritten.

Use the SQL execution engine (cross database)

1. Target is dropped automatically (happens in script).

2. Extract window is set to the number of rows totransfer. (This must be manually checked.)

3. Incremental transfer is started.

a. If failure, rerun the process.

b. If success, extract window is moved forward byscript (extract log is updated).

2 Failure during the transfer of data fromStage_orig table to another staging table, after theexception process. This transfer is within thecentral data warehouse database. The target table(staging table) is a permanent table whosecontents can be overwritten.

Target is emptied automatically by script.

Extract window is set to the number of rows totransfer. (This must be manually checked.)

Incremental transfer is started.

1. If failure, rerun the process.

2. If success, extract window is moved forward byscript (extract log is updated).

3 Failure during the insertion of data from thestaging table to the central data warehouse tables(different schema). The Comp, CompReln, andCompAttr target tables are permanent tableswhose contents are always added to. The objectsof these tables are unique and can be identifiedby a unique key. The Msmt table is a permanenttable whose content is always added to. The rowsof the Msmt tables are unique bytimestamp/metric_id/comp_id only.

For the Comp, CompAttr, and CompReln tables, usethe where not exists SQL statement to be able to rerunthe process in case of failure. For the measurementtable, use the extract window principle, such as:

1. Extract window is set to the number of rows totransfer.

2. Transfer is started.


b. If success, window is moved forward by script(extract log is updated).

Data mart ETL failure cases

4 Failure during the transfer of data between thecomponent tables to the shadow dimension tablein the same central data warehouse database(different schema). The target database is apermanent table whose contents is always addedto.Note: Stage_cdw_dimension tables (shadowdimension tables) are necessary as soon as youneed more than one unique insert to update thedimensions tables. For example, for a databasedimension that might have a state date (adatabase might have been upgraded to a newversion at some time), be sure to insert only newstate date in the dimension, and calculate whichstate dates already exist. If more than one insertis needed, it is easier to create a shadowdimension in the central data warehouse andthen do the transfer across another database tosimplify the failure cases.

1. Target is emptied.




b. If success, window is moved forward (extractlog is updated).


Table 10. Failure and recovery scenarios (continued)

FailureCase

Description Recovery Implementation

5 Failure during the transfer of data from thestage_cdw_dimension to the Mart dimensiontables. This transfer is cross database. The targettable is a permanent table whose contents isalways added to.

Use the SQL execution engine (cross database).





6 Failure during the transfer of data from the Msmttable to the Mart fact table. This transfer is crossdatabase. The target table is a permanent tablewhose contents is always added to.

Use the SQL execution engine (cross database).


2. Incremental transfer is run.



Generic rulesTo rerun a transfer in case of failure, tables that can be overwritten are required forcross database transfers. A cross database transfer should be composed of a simpleinsert.

Note: When a step fails, the user should be able to run that step over again aftertaking appropriate action, which includes checking extract control, resolvingreason for failure, and checking execsql logs. Because inserts intoTWG.Comp, TWG.CompAttr, and TWG.CompReln should always include awhere not exists clause, this is typically not an issue for SQL script steps thatcontain these SQL statements. However, it can become an issue when SQLscript steps have multiple SQL statements that insert into the TWG.Msmttable. If the first Msmt insert succeeds but the second fails, then the stepcannot be run a second time because the rerunning of the first Msmt insertwill attempt to reinsert the same Msmt data and fail with a uniqueconstraint error. There are two possible solutions:v Separate the Msmt inserts into multiple SQLScript steps. This is

recommended for applications that have up to four Msmt inserts.v Use extract control for each insert of data from a stage table to the Msmt

table. This is a little more complex but is recommended for applicationsthat have more than four Msmt inserts.

Component attributes that vary over timeA central data warehouse trigger, CompAttr_Dttm_Trg, is used to handlecomponent attributes that can vary over time (for example, operating systemversion). This trigger enters an end date for the component attribute by filling inthe value for the CompAttr_End_Dttm when the component attribute valuechanges. The end date is populated when a new attribute is added to replace theold one. This trigger also exists to fill in an end date way into the future for a newCompAttr entry. For example, if an ending timestamp is not provided for a newCompAttr entry, then it is set to an unexpired date (1/1/9999).


This trigger only works if you are changing an existing attribute. If you decide tostop using an attribute your code must set the end date.

Extract control handles the late data cases. Late data means that some part of thedelivery system for the measurement data failed and was restarted. This stillassumes that the data arrives in the order in which it is collected by a monitoringfunction for a given resource. This is important because for a given component (forexample, a host) all the measurements are processed in the same order as the orderin which it is collected. If the following data is received, the data arriving on Day20 should be recorded as an exception. The data is not valid because the monitorfiring date is not in the right order. (It is earlier than what was recorded on Day10.)

Insertion in CentralData Warehouse

Host Comp OS version Compattr Monitor FiringDttm

Day 3 host1 5.6 Day 2



To prevent error conditions, source applications should ensure the following:v Measurement data is inserted in the central data warehouse in the same order

(per component) as it was originally collected by some monitoring function.v Data received out of order is trapped and an exception recorded. See “Handling

data exceptions” on page 52.

Pruning old dataRemoving old data from the Msmt table is implemented with a combination oftriggers and warehouse processes as follows:1. Schedule the CDW_c05_PurgeMsmt_Process process.

CDW_c05_PurgeMsmt_Process is a process in theCDW_TivoliEnterpriseDataWarehouse_v1.1.0_Subject_Area subject area, whichis installed with Tivoli Enterprise Data Warehouse. You schedule when youwant to run the pruning process. It can run separately or as a linked process inthe ETL processes.

2. The process runs and performs a simple insert into the Prune_Msmt_Log table.3. A trigger named Prune_Msmt_Trg runs because of the insert into the

Prune_Msmt_Log table. The trigger deletes matching entries from the Msmttable.

Note: Log messages for the process go to the Data Warehouse Center logs. Logmessages for the trigger go to the DB2 logging directory, which is specifiedby the VWS_LOGGING environment variable.

Which data gets pruned is controlled by the TWG.Prune_Msmt_Control table,shown in Table 11.

Table 11. Prune_Msmt_Control table

Column name Data type Null option Attribute name anddescription

MSrc_Cd CHAR(6) NOT NULL Measurement Source Code

TmSum_Cd CHAR NOT NULL Time Summary Code


Table 11. Prune_Msmt_Control table (continued)

Column name Data type Null option Attribute name anddescription

PMsmtC_Age_In_Days DECIMAL(8, 0) NOT NULL Date duration yyyymmdd

A sample SQL statement to add data to the Prune_Msmt_Control table looks likethe following:UPDATE TWG.Prune_Msmt_Control

SET PMsmtC_Age_In_Days = 300WHERE TmSum_Cd = ’H’AND Msrc_Cd = ’INV’

;

Sample data in the Prune_Msmt_Control table looks like the following:

MSrc_Cd TmSum_Cd PMsmtC_Age_In_Days

'INV' 'H' 300.

'INV' 'W' 10000.

'INV' 'M' 10000.

Although the PMsmtC_Age_In_Days column name includes ″Age_In_Days″, thecolumn represents a date duration whose format is yyyymmdd. The values shownin the preceding table, 300. and 10000., represent 3 months and 1 year, respectively.Preceding zeros are not included in the date duration value. For example, a valueof 108. is interpreted as follows and represents 0 years, 1 month, and 8 days.

Years Months Days

0000 01 08

When the process runs, it runs the trigger that prunes matching MSrc_Cd andTmSum_Cd data that is older than what is specified in the PMsmtC_Age_In_Dayscolumn.

Applications must implement similar triggers for data mart fact tables.

Initial load of data into the central data warehouseApplications that are already installed prior to installing Tivoli Enterprise DataWarehouse might need to copy existing data from a source database to the centraldata warehouse; for example, when migrating Tivoli Decision Support data toTivoli Enterprise Data Warehouse. The following steps outline the recommendedmethod for copying existing data in the source application database into thecentral data warehouse for an initial load. Typically, this is a once-onlyinitialization ETL process.1. The source application should provide an ETL process to copy existing data

from the source application database. The limit of how many days of datashould be copied over is determined by the values in theTWG.Prune_Msmt_Control table, which is described in “Pruning old data” onpage 42. The following sample data record is designed to control the pruning ofold data from the Msmt table. This record is placed in the central datawarehouse during the application’s warehouse enablement pack installation.


MSrc_Cd TmSum_Cd PMsmtC_Age_In_Days

'DMN' 'H' 600.

The values shown in this example cause Hourly Msmt data for the TivoliDistributed Monitoring application to be deleted as soon as it is approximately6 months old. Because there is no value in copying the data over only to haveit pruned when the prune process runs (however, there is no harm in doing so),only the last 6 months of data in this example should be copied over in theinitialization ETL process.

2. The extract control and extract log tables should be used in the initializationETL process. This provides a record of the extract. The extract source and targetshould be set to the table names plus INIT; for example, DM_METRICS_INIT.

3. When values in incoming data are unknown, reasonable defaults should beassigned. For example, if the GMT offset was never recorded for the existingdata, the GMT offset of the source application database should be used.

4. If manual procedures are required to set up the initialization ETL, theprocedures should be included in the source application’s warehouseenablement pack documentation.

5. During testing of the source application, test the initialization ETL with n daysof data where n is the value set for PMsmtC_Age_in_Days in theTWG.Prune_Msmt_Control table for the MSrc_Cd. The initialization ETL hasmuch greater requirements for database log space than the regular ETL becauseof the amount of data transferred.

Aggregation and rollupWhen the data mart ETL process runs, it populates the data mart F_<>_HOURtable with data. To support rollup or aggregation to the F_<>_DAY, F_<>_WEEK,F_<>_MONTH tables, the user-defined program named rollup should be run. Therollup program is installed with Tivoli Enterprise Data Warehouse. It requires atable named STAGE_F_<>_HOUR be created and populated with today’s data inthe application-specific schema in the central data warehouse. It then populatesF_<>_DAY, F_<>_WEEK, F_<>_MONTH tables in the data mart based on the datain STAGE_F_HOUR. It also populates RPI.SSUpdated to enable report scheduling.The process flow looks like the following:twg.msmt →execsql→STAGE_F_HOUR and F_HOUR

|→rollup→RPI.SSUpdated [optionally →runReport]

This assumes that the central data warehouse ETL is providing data summarizedon an hourly basis; otherwise, the rollup.sh script does not work. In other words, ifyour application happens to report daily data only, then the rollup.sh script willnot work for you. Also, you must place your star schema in the TWH_MARTdatabase or create the DAY_AGGREG table in your own data mart database.

The rollup program updates any of the aggregate tables as appropriate. It handlesthe updating of any aggregate tables if data out of order is detected. For any of theaggregate tables that are updated, it determines which star schemas have beenupdated and places an entry in the RPI.SSUpdated table. The report gets rerunwhen the runReport user-defined program is run if the following are true:v The RPI.SSUpdated table has an entry for the star schema indicating that the

data is new.v When the user created a report in the report interface GUI, they selected the

option to schedule reports.


The parameters passed to rollup.sh are:v Schema name or product codev Name of hourly fact tablev Name of hourly stage tablev &SDBv &SUIDv &SPWDv &TDBv &TUIDv &TPWDv &STEPNAME

Note: The first two parameters are used to construct the temporary and log filenames.

Most of the parameters need to be modified by users as they configure the ETL fortheir environment. (Initially, the parameters are modified by ETL developers, butusers need to modify the parameters to fit their environment.) The others are setup using Data Warehouse Center variables and are filled in by the Data WarehouseCenter. For example, &SDB, which is the warehouse source database name. In thereport interface case, this is the data mart database.

Customers and centersTo assign components to customers and centers, Tivoli Enterprise Data Warehouseuses two lookup tables per application, product_code.cust_lookup orproduct_code.centr_lookup, or both. These two tables serve to map a value from thesource application data to a customer or center. Define the tables as follows, whereproduct_code represents the identifier that distinguishes a particular application’smatching table from other application’s matching tables:create table product_code.cust_lookup (

Cust_ID INTEGERValue VARCHAR(120)

);create table product_code.centr_lookup (

Centr_Cd CHAR(6),Value VARCHAR(120)

);

Note: You can also use a customer account code (Cust_Acct_Cd) method to map avalue from the source application data to a customer. If this method is used,two lookups need to occur—one to get the Cust_Acct_Cd and another to getthe Cust_ID required by the TWG tables. The benefit of this approach is thatCust_Acct_Cd values can be human readable values while the Cust_ID isjust a number.create table product_code.cust_lookup (

Cust_Acct_Cd CHAR(10),Value VARCHAR(120)

);

Typically, the host name in the source application data is compared against theValue column in these lookup tables to determine the Cust_ID value or Centr_Cdvalue.


An application should be shipped in single-customer, single-center mode bydefault. This means that all components are mapped to one (user-defined)customer and a center named 'CDW' by default.

In a single customer environment, product_code.cust_lookup is populated with asingle value that indicates this is a single customer environment. The single row ofdata entered in the Value field of the product_code.cust_lookup table contains anampersand (@). It is entered as a wild card symbol. The ampersand is coded in thecentral data warehouse ETL so that if it is present in the product_code.cust_lookuptable, all data extracted from the particular source application is assigned a singlecustomer Cust_ID or Cust_Acct_Cd. The Cust_ID or Cust_Acct_Cd is the same forall incoming data, no matter what the Value field is. Any Cust_ID or Cust_Acct_Cdcan be assigned, but by default the application should use 1 to represent thedefault Cust_ID and 'DEF' to represent the default Cust_Acct_Cd.

The following shows a sample insert of data into the lookup table to set up asingle customer environment using the Cust_ID method:insert into product_code.cust_lookup values (1, ’@’);

Or if using the Cust_Acct_Cd method:insert into product_code.cust_lookup values (’DEF’, ’@’);

For single center, a single row exists in the centr_lookup table withCentr_Cd='CDW', Value='@'. This row signifies that all components are mapped toCentr_Cd='CDW'. The following shows a sample insert of data into the lookuptable to set up a single center environment:insert into product_code.centr_lookup values ('CDW', ’@’);

Multicustomer and multicenter supportThe Tivoli Enterprise Data Warehouse multicustomer and multicenter designmakes the installation and management of a single customer or center environmentas simple as possible while still providing features that enable a multicustomer ormulticenter environment when it is required.

To facilitate multicustomer and multicenter installations, some added complexity isrequired in the design and installation process that impacts the single customer orcenter environment.

As part of the multicustomer support, two different application scenarios areprovided:v Application data sources that are input sources to the central data warehouse

that contain information valid for one customer only.v Application data sources that are input sources to the central data warehouse

that contain data for multiple customers.

A single customer installation can smoothly migrate to a multicustomerenvironment. However, if any application data from a single customer environmentis imported into the central data warehouse so that it cannot be identified with aparticular customer, the central data warehouse cannot be used to support amulticustomer environment. In this scenario, a customer’s recourse is to migratetheir old central data warehouse data into the new environment to support themulticustomer environment.


During the reporting phase of the warehouse life cycle, customers are onlypermitted to see and work with data and reports based on their data and not anyother customer’s data.

Separate applications can use different data fields to distinguish betweencustomers, but within applications, the data fields should be consistent. Anapplication team must determine which field in their data source is applicable foridentifying multiple customer data. An application might decide that a single datafield determines customer uniqueness. Other applications might decide that two ormore data fields are needed to distinguish customer data. Either case is permittedas long as the data fields used are consistent within a single application.

Data from applications is assigned valid customer account codes by matchingcertain data fields in the incoming data with pre-identified values in a matchingdatabase table. (See tables in “Customers and centers” on page 45.) An assumptionin the design is that most applications do not have an existing customer accountcode field in their data fields, and even if they do, their customer account codesmight not match the values or format used by the central data warehouse. Becauseeach application can use different fields and different number of fields to identifycustomers, each application has its own matching table that it uses during thecentral data warehouse ETL process.

Before a central data warehouse ETL can be run for the multicustomerenvironment, the product_code.cust_lookup and TWG.Cust tables must bepopulated with data. Designers of the central data warehouse ETL process shouldresist the temptation to try to populate these tables during the central datawarehouse ETL process itself. Auto-population during the central data warehouseETL process can cause auto-generation of customers that have not been explicitlyconfigured in the warehouse by an administrator. This is considered highlyproblematic and could lead to the mixing of customer data during the reportingprocess.

The customer should populate the TWG.Cust and product_code.cust_lookup tablesas part of the planning and installation of the central data warehouse. An outsideprocess or script can be used to populate the tables but it should be run as adistinct step scheduled separately from the central data warehouse ETL process.The data contents of the TWG.Cust and product_code.cust_lookup tables should befully understood by an administrator to avoid data mingling between customers.

If the user wants to implement either multicustomer or multicenter functionality,they can do this by changing the contents of the lookup tables, cust_lookup orcentr_lookup, and populating the TWG.Cust and TWG.Centr tables in the centraldata warehouse. For example, for multicustomer, the user must:v Delete the single row in the product_code.cust_lookup table. This row has

Cust_Acct_Cd='DEF', Value='@' or Cust_ID=1, Value='@'.v Populate the product_code.cust_lookup table. The data in this table is static and

should not be populated as part of an ETL process. The data might be based onpart of the fully qualified domain name. For example, all hosts in theford.myserviceprovider.com domain belong to 'Ford'.

v Populate the TWG.Cust table with valid values. If using the customer accountcode method, the Cust_Acct_Cd field in the TWG.Cust table must match theCust_Acct_Cd field in the product_code.cust_lookup table. If using the customerID, the Cust_ID field in the TWG.Cust table must match the Cust_ID field in theproduct_code.cust_lookup table


For example, a customer has a Tivoli Enterprise Console application running attheir site in a multicustomer environment. They deem that host name is the correctsource field to distinguish customers. Correspondingly, the host name field in theirTivoli Enterprise Console data source has multiple entries. The administrator addsentries to the product_code.cust_lookup and TWG.Cust tables in the central datawarehouse. If there are no entries in these tables for the corresponding data in thesource tables, the data is not extracted into the central data warehouse and isplaced into an exception table. When properly configured, the central datawarehouse ETL runs and populates the valid data into the central data warehouse.

Implementing multicustomer designFirst, define the customers that are needed to support your installation.insert into TWG.Cust

(Cust_ID, Cust_Parent_ID, GeoArea_Cd, TmZon_ID, Cust_Acct_Cd, Cust_Nm)values (0, NULL,’US’, 8, ’KSCorp’, ’Kansas City Corporation’);

insert into TWG.Cust(Cust_ID, Cust_Parent_ID, GeoArea_Cd, TmZon_ID, Cust_Acct_Cd, Cust_Nm)values (0, NULL,’US’, 5, ’SFCorp’, ’San Francisco Corporation’);

Note: The Cust_ID column is populated with a sequence by a trigger; 0 is replacedby the next value in the sequence.

Populate the lookup tables. As mentioned previously, this data is probablygenerated from another application or database that describes the hosts in acustomer environment. For example, if using the customer ID:insert into inv.cust_lookup values (1,’a123.dev.kccorp.com’)insert into inv.cust_lookup values (1,’b123.sales.kccorp.com’)insert into inv.cust_lookup values (2,’s123.dev.sfcorp.com’)insert into inv.cust_lookup values (2,’t123.sales.sfcorp.com’)

Or if using the customer account code:insert into inv.cust_lookup values (’KSCorp’,’a123.dev.kccorp.com’)insert into inv.cust_lookup values (’KSCorp’,’b123.sales.kccorp.com’)insert into inv.cust_lookup values (’SFCorp’,’s123.dev.sfcorp.com’)insert into inv.cust_lookup values (’SFCorp’,’t123.sales.sfcorp.com’)

Central data warehouse ETLSQL performing the matching operations between source data and theproduct_code.cust_lookup table should have the following general format. Note thatthis example uses a match on Cust_ID to assign data to a customer.--% By default, every ETL should assign a default cust and center to each--% comp row created. The following is an example of assigning default--% cust and center codes. Most of this logic will reside in your--% product_code_cdw_data.sql script.

--% Note: A session table is available only during the duration of--% the connection. When the connection to the database ends, the session--% table and all of its contents are dropped automatically by DB2.declare global temporary table product_code_cust_lookup (

cust_id integer,value varchar(120))

with replace on commit preserve rows not logged;

--% The following is a wild card indicator, which means all data is--% assigned the same customer ID. If you need to assign separate customer--% IDs, then replace the insert statement below with a list of--% all the customer IDs that are required. Make sure the--% product_code.cust_lookup table entry containing the wild card--% information is deleted. The reason that a session table is used initially,--% and then the data is moved to the "permanent"


--% product_code.cust_lookup table, is to allow this script to run--% more than once and to allow it to run successfully even if stopped in--% the middle of the insert statements.insert into session.product_code_cust_lookup values (1, ’@’);

--% Move the data to a working table in an application specific schema--% in the central data warehouse.INSERT INTO product_code.cust_lookupSELECT * FROM session.product_code_cust_lookup sWHERE NOT EXISTS (

SELECT 1 FROMproduct_code.cust_lookup t WHEREt.cust_id = s.cust_id

);

declare global temporary table product_code_centr_lookup (centr_cd char(10),value char(120)

)with replace on commit preserve rows not logged;

insert into session.product_code_centr_lookup values (’CDW’, ’@’);

--% Move the data to a working table in an application specific schema--% in the central data warehouse.INSERT INTO product_code.centr_lookupSELECT * FROM session.product_code_centr_lookup sWHERE NOT EXISTS (

SELECT 1 FROMproduct_code.centr_lookup t WHEREt.centr_cd = s.centr_cd

);

--% Make an insertion into the twg.comp table.--% Note the logic that assigns IP_HOST if dots are--% detected in the name and TWSM_HOST if the name appears--% to be a non-qualified name without dots.--#EXECUTE_AT_TARGETinsert into TWG.COMPSELECT

0 as Comp_ID,case

when LOCATE(’.’,t.ipaddress) = 0 then ’TWSM_HOST’else ’IP_HOST’

end as comptyp_cd,centr.centr_cd as centr_cd,cust.cust_id cust_id,0 as Comp_Corr_Id,ipaddress AS Comp_Nm,ENDPOINTID AS Comp_Corr_Val,ENDPOINTDATE AS Comp_Strt_DtTm,’9999-01-01-00.00.00.000000’ AS Comp_End_Dttm,DESCRIPTION AS Comp_Ds

FROMproduct_code.stage_ept_data t,product_code.cust_lookup cust,product_code.centr_lookup centr

WHERE(cust.value = t.ipaddress or cust.value = ’@’) and(centr.value = t.ipaddress or centr.value = ’@’) andnot exists (select 1 from

session.temp_cdw_ep ewhere


e.name=t.ipaddress);

For the multicustomer setup, records not matching should be rejected intoexception tables. For some applications it makes sense to perform the customer IDassignment as the very first step in the central data warehouse ETL process. Forapplication sources such as Tivoli Distributed Monitoring and Tivoli Inventory,significant amounts of merging must take place in the source data prior to havinga single source data table. That is, bits and pieces of data from multiple source datatables must be assembled into a single table through multiple ETL steps to achievea single table of input data.

In these cases, the customer-checking logic shown in the previous example mighttake place later in the central data warehouse ETL process, perhaps as late asduring population of the components into the component table. Customer checkingshould be done as early as practical during the central data warehouse ETL processto discard incorrect customer data as soon as possible. The logic to check for validcustomer identification is the same regardless of when it occurs in the central datawarehouse ETL process.

Reporting data mart ETLAt least one table in the star schema will contain the Cust_ID field. Other tables inthe star schema must be able to filter their data by joining with the table that hasthe Cust_ID.

For multicustomer environments, a view is created of each table in a star schema.The exception might be the D_METRIC table because the same metrics are likely toapply to all customers. The administrator at the customer site is responsible forcreating views. ETL designers provide template insert statements such as thefollowing in thetedw_apps_etl/product_code/pkg/vnnn/mart/ddl/product_code_mart_schema.sqlscript:#Run this set of inserts to create views for each customer.#You must replace <cust_id> with your valid customer ID,#and code inserts for each customer.Create view SPP.F_V_DM_<cust_id>_hour as (select * from SPP.F_DM_hourwhere host_id in (select host_id from d_dm_host where cust_id = <cust_id>));

Create view SPP.D_V_DM_host_<cust_id> as (select * from SPP.D_DM_hostwhere cust_id = <cust_id>);

There is an important security aspect to consider for star schemas in themulticustomer environment. Tivoli Enterprise Data Warehouse requires customersto create database users that include specific grants to only their applicable views.When views are created for specific customers, the users should be given access tothe views by database ID and their access to views should be compartmentalizedbetween different customer users.

For single-customer environments, no views are created and only tables are used.(There is nothing that prevents views from being used in a single customerenvironment.)

A star schema’s metadata is retrieved from the Data Warehouse Center table,IWH.StarSchema. With a multicustomer implementation, applications must also


create star schemas made up of views in the Data Warehouse Center. This meansthat the IWH.StarSchema table contains both view star schemas and table starschemas.

An example of star schemas in the Data Warehouse Center for all customers and aspecific customer follows:

SPP.F_hourSPP.d_host_stateSPP.d_metric

SPP.F_v_Ford_hourSPP.F_v_Ford_host_stateSPP.d_metric

The first star schema contains tables and applies to all customers. The second starschema contains views and applies to a specific customer, which is Ford in thisexample.

Using a Web browser and the IBM Console graphical user interface, you can defineuser groups (for example, cust1 and cust2), assign users to those groups, andassign data marts to the user groups using the report interface.

Multicenter supportMultiple centers are implemented using a similar design as multicustomers.Centers provide a way to partition your data for the different requirements anapplication or customer has. For example, centers could be used as a grouping tobuild data marts if you want to track performance based on a datacenter ratherthan the entire company.

A common problem is that there might not be any time zone information carriedwith a timestamp in source data. Because all timestamps in the central datawarehouse must be in UTC, centers can be used to associate time zones with dataand convert incoming data to UTC. A center has a time zone associated and theUTC offset for that time zone can be looked up in the TWG.TmZon table. Thisvalue can be subtracted from the incoming timestamp to convert it to a UTCtimestamp.

To implement multicenter support, first define the centers that are needed tosupport your installation.insert into TWG.centr(Centr_Cd, Centr_Parent_Cd, GeoArea_Cd, TmZon_ID, Centr_Nm)values (’KS’, NULL,’US’, 8, ’Kansas City Center’);insert into TWG.centr(Centr_Cd, Centr_Parent_Cd, GeoArea_Cd, TmZon_ID, Centr_Nm)values (’SF’, NULL,’US’, 5, ’San Francisco Center’);

Next, populate the product_code.centr_lookup table with values that relate a host ordata to a center:insert into inv.centr_lookup values (’KS’,’a123.dev.kccorp.com’)insert into inv.centr_lookup values (’KS’,’b123.sales.kccorp.com’)insert into inv.centr_lookup values (’SF’,’s123.dev.sfcorp.com’)insert into inv.centr_lookup values (’SF’,’t123.sales.sfcorp.com’)

During the central data warehouse ETL, appropriate center codes are assigned toall data using the lookup table.


Handling data exceptionsDuring the ETL process a variety of error conditions can occur in incoming datathat cause exceptions. The following describes two different exception scenariosand how to handle the exceptions when creating ETLs for Tivoli Enterprise DataWarehouse:v Some part of the data is not valid; however, not all data is unusable.

For example, a Tivoli Application Performance Management host name that is ashort name is an exception and goes in the Comp table with a CompTyp_Cdvalue of TAPM_HOST. A component is created and the measurements (Msmtdata) are kept. This is because the Msmt data is still useful even thoughcorrelation by host name is broken.

v Data is unusable.Cases that are more serious require you to create an explicit exception table andput any bogus measurements in that table—the same is true for bogus Comp orCompAttr data. The application needs to document the table information so thatthe customer knows where to look for exception data. This type of exceptiontable typically contains all the columns of the real table plus two additionalcolumns (for example, Insert_DtTm and Msg) to record the insertion timestampand a message to describe the exception.

Creating a star schema or multidimensional cubeWhen building the historical reporting aspects of your application, the data mart’sstar schema serves as a subject-oriented reporting and analysis database. The datamart ETL transforms data into business information and provides transparentaccess for reporting and analysis tools. The star schema consists of the following:v One metric dimension tablev One or many component dimension tablesv One fact table

The inherent flexibility of the central data warehouse makes it inefficient to buildqueries that run directly off of the warehouse schema spanning differentcomponents and types of measurements. The purpose of the star schemas is totransform raw business data into business information that can be easily reportedand analyzed. A data mart contains data tailored for the specific needs of usergroups. Each data mart targets a specific business audience and a particular dataanalysis problem domain. For example:v Single customer analysis for performance engineersv Infrastructure analysis for network analystsv Summarized, overall customer health for USF management

Data marts are designed to be customizable by the customer. Security settings canbe applied at the data mart level as a user’s access can be limited to only thosedata marts that the group to which the user belongs can access. For the reportinterface (RPI) supplied with Tivoli Enterprise Data Warehouse, data marts have toconform to a set of rules. A data mart that conforms to these rules is referred to asan RPI-ready data mart.

A data mart intended for use by the Tivoli Enterprise Data Warehouse reportinterface must be published as a collection of schemas. This is done by creating astar schema in the IBM DB2 Data Warehouse Center. A star schema is a set of threeor more tables used to satisfy a single SQL query generated by the report interface.


This enables the report interface to determine which tables are in a star schemaand the joining columns between the tables.

Note that the organization of star schemas in report interface data marts and thesecurity settings for report interface data marts are both configured using thereport interface itself. The schema name for tables in RPI-ready data marts mustmatch the application product code. For example, a product code of TWR wouldresult in table names of TWR.D_HOST. Each warehouse schema within RPI-readydata marts must conform to the following rules.

Rule 1: metric dimension tableMetrics are the items about which you collect data. Sample metrics include theamount of RAM on a PC, the number of help desk calls made by a customer, theaverage CPU busy time for a server, and mean time to failure for a hardwaredevice. A dimension is a single piece of descriptive information used to identify facttable records. Dimensions are typically string information used to constrain thenumerical information stored in the fact table. Dimensions are used to constrainthe facts (numerical data) selected by SQL SELECT statements when pullinginformation from a fact table for reporting.

A single dimension table contains metric information with a name that must end inMETRIC. For example, D_x_METRIC, where x is optional and represents themetric group description. Specifying a unique value for x ensures that namingcollisions don’t occur between applications sharing the same data mart. Thecolumns must include those shown in Table 12.

Table 12. Metric dimension table columns

Column Data type Null option Description

metric_id INTEGER NOT NULL A unique integer that identifies thismetric record. This number can begenerated by a sequence numbergenerator in the database or someother mechanism. It is a foreign keyused to link the fact table of a starschema to a particular metric record.

met_category VARCHAR(254) NULL RPI requires that this column exist inthe table; however, the value stored inthe column is not used. Populate thefield with a value such as ″not used″.

met_desc VARCHAR(254) NULL A description of the metric.

met_name VARCHAR(254) NOT NULL A description of what the data pointactually measures; for example, %CPU busy or % disk utilization.

met_units VARCHAR(254) NULL A unit of measure such as megabytesper second, total megabytes free, MHz,and so on.

min_exists CHAR(1) (Y orN)

NOT NULL Indicates whether this particular factrecord contains data for minimum.

max_exists CHAR(1) (Y orN)

NOT NULL Indicates whether this particular factrecord contains data for maximum.

avg_exists CHAR(1) (Y orN)

NOT NULL Indicates whether this particular factrecord contains data for average.

total_exists CHAR(1) (Y orN)

NOT NULL Indicates whether this particular factrecord contains data for total.


Table 12. Metric dimension table columns (continued)

Column Data type Null option Description

msrc_nm VARCHAR(254) NULL This field is displayed by the RPI GUIto let you know what application starschema you are working with. It is nottranslated. For combined star schemasthis might have some other stringother than the product name in it.

Each D_x_METRIC table contains metrics that are valid for that warehouseschema.

Rule 2: component dimension tablesOne or many dimension tables contain component information. The name startswith D_ to signify a dimension table; however, component dimension tables thatare not the metric dimension table must not end in _METRIC.

Component dimension tables contain component attributes (columnname=component attribute). These attributes are displayed in the report interfaceso that users can filter on them. If there are more than 27 values for a particularcomponent attribute, they are not displayed in the report interface for filtering.

If you want to add a column to a dimension table that is not a component attributeto be displayed in the report interface, prefix the columns with XQ.

Note: Any columns that join to the fact table, such as host_id or state_strt_dttm,are not considered component attributes and are not displayed in the reportinterface for filtering. Also, if you prefix the component attribute, or columnname, with XF, the report interface adds it to the filtering GUI, but thefiltering option reads no filtering allowed. Users can see the componentattribute, but cannot filter on it.

Rule 3: fact tableA single fact table contains measurement information (for example, F_DM_DAY).The name of the table starts with F_ to signify a fact table. The last part of thename is a period such as HOUR, DAY, WEEK, MONTH. The middle part of thename is free format and is used to distinguish one fact table from another facttable in the same data mart. The columns in a fact table must include thefollowing:v Foreign key columns to the component dimension tables

This is typically just a single column that includes an integer ID. However, insome cases another column, such as a date, might be needed. It is possible touse three or more columns to join into a dimension table although single integerIDs are more common.

v metric_idSingle integer key into the D_x_METRIC table.

v meas_date (day/week/month)This value should represent the start of the week, start of the day, start of themonth and is used when running a report that says something like return thelast 2 weeks of data.

v meas_hour (This is required for F_x_HOUR tables.)


Contains summarized timestamps on hourly boundaries. The report interfacerequires meas_hour for hourly fact tables and meas_date for all other fact tables.

v min_valuev max_valuev avg_valuev total_valuev sample_count

min_value, max_value, avg_value, total_value are only required if they are referredto in the metric table. If min_value, max_value, avg_value, total_value are in thetable but are not used in a particular record, their value should be set to 0.

Star schema exampleThe following SQL example sets up a star schema, inserts minimal data, and thendemonstrates a sample report query.-- run this script using db2 -tf createstar.sql-- from a db2cmd command prompt

-------------------------------------------------- data mart database where this info is held------------------------------------------------CONNECT TO TWH_DM;

DROP TABLE TSM.D_FILESYSTEM;DROP TABLE TSM.D_HOST;DROP TABLE TSM.D_METRIC;DROP TABLE TSM.F_TSM_CAP_DAYS;COMMIT WORK;

-------------------------------------------------- DDL Statements for table TSM.D_FILESYSTEM------------------------------------------------

CREATE TABLE "TSM"."D_FILESYSTEM" ("FILESYS_ID" INTEGER NOT NULL ,

"FILESYS_NAME" VARCHAR(64) )IN "USERSPACE1" ;

-- DDL Statements for primary key on Table "TSM"."D_FILESYSTEM"

ALTER TABLE "TSM"."D_FILESYSTEM"ADD PRIMARY KEY ("FILESYS_ID");

-------------------------------------------------- DDL Statements for table TSM.D_HOST------------------------------------------------

CREATE TABLE "TSM"."D_HOST" ("HOST_ID" INTEGER NOT NULL ,"TSM_NODE" VARCHAR(64) ,"TSM_SERVER" VARCHAR(64) ,"TSM_POLICY_DOMAIN" VARCHAR(64) ,"OS_LEVEL" VARCHAR(64) ,"TSM_CLIENT_VERSION" VARCHAR(64) )

IN "USERSPACE1" ;

-- DDL Statements for primary key on Table "TSM"."D_HOST"

ALTER TABLE "TSM"."D_HOST"ADD PRIMARY KEY ("HOST_ID");


-------------------------------------------------- DDL Statements for table TSM.D_METRIC------------------------------------------------

CREATE TABLE "TSM"."D_METRIC" ("METRIC_ID" INTEGER NOT NULL ,"METRIC_NAME" VARCHAR(64) ,"METRIC_UNITS" VARCHAR(64) ,"METRIC_CATEGORY" VARCHAR(64) ,"MIN_EXISTS" CHAR(1) ,"MAX_EXISTS" CHAR(1) ,"AVG_EXISTS" CHAR(1) ,"TOTAL_EXISTS" CHAR(1) )IN "USERSPACE1" ;

-- DDL Statements for primary key on Table "TSM"."D_METRIC"

ALTER TABLE "TSM"."D_METRIC"ADD PRIMARY KEY ("METRIC_ID");

-------------------------------------------------- DDL Statements for table TSM.F_TSM_CAP_DAYS------------------------------------------------

-- NOTE THAT THIS STAR SCHEMA ASSUMES NON_VARYING-- COMPONENT ATTRIBUTESCREATE TABLE "TSM"."F_TSM_CAP_DAYS" ("HOST_ID" INTEGER NOT NULL ,"FILESYS_ID" INTEGER NOT NULL ,"METRIC_ID" INTEGER NOT NULL ,"TOTAL_VALUE" DECIMAL(5,0) NOT NULL )IN "USERSPACE1" ;

-- DDL Statements for primary key on Table "TSM"."F_TSM_CAP_DAYS"

ALTER TABLE "TSM"."F_TSM_CAP_DAYS"ADD PRIMARY KEY ("HOST_ID", "FILESYS_ID", "METRIC_ID");

-- DDL Statements for foreign keys on Table "TSM"."F_TSM_CAP_DAYS"

ALTER TABLE "TSM"."F_TSM_CAP_DAYS"ADD CONSTRAINT "SQL011106114546440" FOREIGN KEY("HOST_ID")REFERENCES "TSM"."D_HOST"("HOST_ID")ON DELETE NO ACTIONON UPDATE NO ACTION;

ALTER TABLE "TSM"."F_TSM_CAP_DAYS"ADD CONSTRAINT "SQL011106114546480" FOREIGN KEY("FILESYS_ID")REFERENCES "TSM"."D_FILESYSTEM"("FILESYS_ID")ON DELETE NO ACTIONON UPDATE NO ACTION;

ALTER TABLE "TSM"."F_TSM_CAP_DAYS"ADD CONSTRAINT "SQL011106114546490" FOREIGN KEY("METRIC_ID")REFERENCES "TSM"."D_METRIC"


("METRIC_ID")ON DELETE NO ACTIONON UPDATE NO ACTION;

COMMIT WORK;

-- now add minimal amounts of bogus datainsert into tsm.d_host values (1, ’node1’, ’server1’, ’domain1’, ’os1’, ’version1’);

insert into tsm.d_filesystem values (1, ’filesystem1’);

insert into tsm.d_metric values (1, ’CAPACITY’, ’MBYTES’, ’DISKSPACE’, ’N’,’N’, ’N’, ’Y’);insert into tsm.d_metric values (2, ’USED’, ’MBYTES’, ’DISKSPACE’, ’N’,’N’, ’N’, ’Y’);insert into tsm.d_metric values (3, ’FREE’, ’MBYTES’, ’DISKSPACE’, ’N’,’N’, ’N’, ’Y’);

insert into tsm.f_tsm_cap_days values (1, 1, 1, 200.0);insert into tsm.f_tsm_cap_days values (1, 1, 2, 150.0);insert into tsm.f_tsm_cap_days values (1, 1, 3, 050.0);

-- VERY SIMPLE SAMPLE REPORT QUERY-- db2 select b.metric_name, a.total_value from tsm.f_tsm_cap_days a,-- tsm.d_metric b, tsm.d_host c, tsm.d_filesystem d where-- a.host_id = c.host_id and a.metric_id = b.metric_id and-- a.filesys_id = d.filesys_id and d.filesys_name=’filesystem1’ and-- c.tsm_node=’node1’ and (b.metric_name=’CAPACITY’ OR-- b.metric_name=’USED’ OR b.metric_name=’FREE’) and b.total_exists=’Y’

CONNECT RESET;

TERMINATE;

Performance considerationsBy adding indexes such as the following, you can improve the run-timeperformance for data mart queries. The indexes decrease the number of scannedpages and sort time by reducing the number of rows to look up in the fact table.db2 -v "create index f_x1 on dm.f_day (meas_date)"db2 -v "create index f_x2 on dm.f_day (metric_id)"db2 -v "create index f_x3 on dm.f_day (host_id, host_state_start_dttm)"

You also need to generate statistical information. For example:db2 -v runstats on table dm.f_day with distribution and indexes alldb2 -v runstats on table dm.d_metric with distribution and indexes alldb2 -v runstats on table dm.d_host_state with distribution and indexes all

Report schedulingFor report scheduling purposes, the last step of the data mart ETL, or buildMartprocess, should run a user-defined program named runReport.sh, which isinstalled by Tivoli Enterprise Data Warehouse. When calling runReport.sh, set thesource table for the step to PRODUCT_CODE.STAGE_F_HOUR and the targettable to RPI.SSUpdated. The runReport.sh program inserts a record into theRPI.SSUpdated table in the TWH_MD database. The record that is inserted tells thereport execution engine when a star schema has been updated and the reportexecution engine runs all scheduled reports that have been created from that star


schema. The step that calls the runReport.sh user-defined program can be removedfrom the data mart ETL by the customer if they do not want scheduled reports tobe run.

ETL guidelinesBecause the source applications are responsible for the source schema, central datawarehouse ETL, and data mart ETL, they ultimately determine which versions oftheir database schemas are compatible with their ETL versions. The followingguidelines should be noted:v If the source application supplies database version files, implement the ETL so

that it is compatible with all supported versions of the database.v When possible, source applications should make database schema changes that

require no updating of the central data warehouse ETL. However, if any of thedatabase changes affect data that must be accessible in Tivoli Enterprise DataWarehouse, changes to the ETL are necessary.

v When updating an existing central data warehouse ETL, make changes so thatthe ETL remains compatible with the previous, supported versions of the sourcedatabase.

v When updating an existing data mart ETL, make changes so that the ETLremains compatible with any existing, supported versions of the data marts.

v Source applications are required to use the MSrc_Cd (measurement source code)field for their schema for the staging or work tables in the central datawarehouse database. For third-party applications, the MSrc_Cd value must be athree-character code that contains one or more numbers; for example, ED1. ForTivoli and IBM applications, the MSrc_Cd value must match the application’sthree-letter component prefix code, or product code. For example, the productcode for Tivoli Manager for Oracle is CTO. All staging or work tables in thecentral data warehouse database that relate to Tivoli Manager for Oracle wouldbe named CTO.STAGE_TableDescription. For more information, see Chapter 5,“Naming conventions” on page 65.

v Central data warehouse ETLs should not use warehouse transformers becausethey only work with DB2 targets.

v When the warehouse source database is set up in the central data warehouse,the user must specify a user and password. The execsql.exe program makes atwo-part name based on the user defined for the warehouse source. This enablescustomers to use a user of their choice.When referencing Oracle tables (and other objects) in SQL scripts, always use thesingle-part name only. For example:select name from myoracletable

Only use the two-part name when referencing DB2. For example:select name from scott.myoracletable

v To prevent problems with wild card queries, report formatting, and so on, donot use trailing blank pads for component names and attributes. You can use aDB2 command to detect blank padding. For example:db2 select distinct replace(compattr_val,' ','x') from twg.compattr

The TWG.CompAttr table contains a column called CompAttr_Val. In thisexample, the DB2 command detects all distinct blanks contained in columnCompAttr_Val column and replaces the blanks with 'x'.


Chapter 4. Migrating Tivoli Decision Support function

The data sources from which Tivoli Decision Support data was created can bemigrated into Tivoli Enterprise Data Warehouse information. The first step in thisprocess is to understand how Tivoli Decision Support elements map to TivoliEnterprise Data Warehouse elements.

Table 13. Comparison of Tivoli Decision Support and Tivoli Enterprise Data Warehouseelements

Tivoli Decision Support Tivoli Enterprise Data Warehouse

Tivoli Decision Support DiscoveryAdministrator

Tivoli Enterprise Data Warehouse ETL byway of Data Warehouse Center

Cube Data mart

Dimension Central data warehouse component, datamart dimension

Measure Measurement type

Calculated column Derived component attribute

Query SQL step in a process

Event procedure SQL/external step

Tivoli Decision Support Discovery Interface Tivoli Enterprise Data Warehouse reportinterface

View Report

Topic map Manage Reports and Report Output task(within the IBM Console)

Once you have basic understanding of how the elements and terminology map toone another, use the following tasks as a guideline for creating Tivoli EnterpriseData Warehouse information.1. Study the views in the Tivoli Decision Support Discovery Interface.

It is recommended that you obtain cubes with some sample data. Validate theusefulness of each view to determine if it is a requirement to implement anequivalent Tivoli Enterprise Data Warehouse report. Based on the viewsavailable for a cube, decide what two dimensional reports are the mostproductive for customers. The reporting requirements define exactly what dataneeds to be extracted from the source application as well as how often and atwhat level of detail.

2. Understand the different purposes of Tivoli Decision Support and TivoliEnterprise Data Warehouse reports.Significant functional differences exist between Tivoli Decision Support viewsand Tivoli Enterprise Data Warehouse reports. Tivoli Decision Support views intheir initial state can display two, three, or four dimensions of data. (A stackedbar chart with a layer over the view is really four dimensions.) Tivoli DecisionSupport views can often be explored; for example, using drill-up or drill-downoperations. In contrast, Tivoli Enterprise Data Warehouse reports offer atwo-dimensional graph or report. This means that a Tivoli Enterprise DataWarehouse report is typically designed to answer a specific decision supportquestion quickly.

3. Study the Tivoli Enterprise Data Warehouse central data warehouse schema.

59

Understand the data model, especially the components, component attributes,and measurement types. See Appendix A, “Central data warehouse datamodel” on page 99. Once you have this information, you can map entities inthe source application to the central data warehouse schema.

4. Implement Tivoli Decision Support cubes as equivalent Tivoli Enterprise DataWarehouse data marts. To do this, study the transformer model for the TDScube. Also look at the calculated columns and queries that are input to the CSVfiles that comprise cube information in TDS. The columns might be transferredinto data mart ETL steps as listed in Table 13 on page 59.The transformer model for the cube can help to determine the Tivoli EnterpriseData Warehouse components, component attributes, and measurement types. Toaccess the dimension map for a cube, go into Tivoli Decision Support Admin,right click on a cube, and select Open transformer model. Look at themeasures. Measures can relate to either some kind of probe data, or to events.Within a dimension map, look at the lowest level of every dimension. A TivoliDecision Support dimension can be one of the following in Tivoli EnterpriseData Warehouse:v A msmt date or timev A componentv A base component attributev A derived component attributev A measurement type

Here are some hints and tips:v Any date or time dimension typically maps to the meas_date and meas_time

in a msmt table.v Look for dimensions that refer to the same logical entity more than once. For

example, host is often repeated in multiple dimensions.v A component is a logical entity such as host, transaction, user, application. It

can also be an event attribute such as status, duration_range, class, source.v A base component attribute is an attribute of a logical entity such as the IP

address of a host.v A derived component attribute is calculated or derived from a base

component attribute.v Tivoli Enterprise Data Warehouse measurement types can be:

– The list of Tivoli Decision Support measures.– A Tivoli Decision Support dimension (such as System Metric). In this case,

the Tivoli Decision Support measures might be avg, min, max.

The following are examples from the Tivoli Decision Support Guides for sourceapplications that have been analyzed:v Tivoli Distributed Monitoring/SPP Tivoli Decision Support Guide:

– Components: host– Component Attributes: O/S Name, O/S Ver, O/S sub version, Phys Memory

KB, NW Address, Num of CPUs, CPU Model, CPU speed, system purpose– Measurement Types: avgcpusys, avgcpuusr, netcollirate, netinerrate, netinrate,

netouterrate, netoutrate, pageinrate, pageoutrate, pagescanrate, and so onv Event Management Tivoli Decision Support Guide:

– Components: host, event class, event severity, event source/subsource, eventduration range


– Component Attributes: parent event class, grandparent event class– Measurement Types: number of events, number of events per system, number

of times met SLA, and so onv Tivoli Application Performance Management Tivoli Decision Support Guide:

– Components: host, user, transaction, application– Component Attributes: IP address– Measurement Types: avg response time, min response time, max response

time, number of transactions succeeded, and so on

Refer to the Tivoli Manager for Oracle Tivoli Enterprise Data Warehouseexploitation for a detailed example.

Tivoli Decision Support cube queriesWhen extracting data from source applications, a simple ODBC query should beused that typically involves one or two tables in the source database. These initialextraction queries can be copied directly from the Tivoli Decision Support Guide inmany cases. If any outer join functionality is required, then only two tables shouldbe joined at maximum in a single query. The output from the ODBC query isplaced in a staging table (in DB2) that can then be joined with other staging tables(with as many outer joins as required) in subsequent steps. An alternative to usestaging tables is to make database views that include outer joins in the sourcedatabase. Range Parameters in Tivoli Decision Support are used by cube queries torestrict the rows that are selected from the RDBMS. In Tivoli Enterprise DataWarehouse this is implemented as rows in the TWG.Extract_Control table in thecentral data warehouse.

The following shows an example query using extract control:insert into BWM.STAGE_QOSSELECT

e.ROUNDTRIPTIME,e.SERVICETIME,e.PAGERENDERTIME,e.EAARECORDID,e.TASKSEQ,e.HTTPSTATUSSEQ,t.TIMEOFDAY,d.FULLDATE,k.TASKID ,’N’ as VALID

FROMTDDATE d,TDTIME t,TFEAA e,TDTASK k,temp_extract_control ec

WHEREe.DWCREATEDATE > ec.extctl_from_DtTm ande.DWCREATEDATE <= ec.extctl_to_DtTm ande.ENDPOINTDATESEQ = d.DATESEQ ANDe.ENDPOINTTIMESEQ = t.TIMESEQ ANDe.TASKSEQ = k.TASKSEQ ANDe.HTTPSTATUSSEQ >= 14

;

Chapter 4. Migrating Tivoli Decision Support function 61

Tivoli Decision Support calculated columnsIn TDS, after data has been selected from the source database using a cube query,extra data can be appended to each line of the result set by the use of calculatedcolumns. These are evaluated using event procedures implemented in Visual BasicScript. These event procedures have functions that can be classified like this:v Functions that only require input from the data coming from the cube query

For example: derive the domain name from the fully qualified host name.v Functions that require access to Categorization/Terminology Parameters

In Tivoli Decision Support, Categorization Parameters are used by calculatedcolumn event procedures to find and replace values based on ranges of values.For example, when columnA is <70 and >80 then substitute with valueY. InTivoli Decision Support, Terminology Parameters are used by a calculatedcolumn event procedure as a translation table. For example, when columnA isvalueX then substitute with valueY.

v Functions that require access to the source database of the cube queryFor example, compute a forecasted value for a measurement based on the last 30days values.

The options available for implementing similar functions in Tivoli Enterprise DataWarehouse are:v SQL Steps that use these RDBMS facilities:

scalar functionsWhen supported across DB2 and Oracle via DataJoiner or DB2 v8

views Can hide DB2 and Oracle differences

create table defaultsCan be used for default dates for example

temporary tablesCan be used as a form of local storage triggers. These triggers can beused to implement function that differs in DB2 and Oracle.

stored proceduresRDBMS-internal stored procedures (SQL stored procedures in DB2,PL/SQL in Oracle)

Note that external stored procedures cannot be used in Java. This includes allthe warehouse transformers because they are not supported on a non-DB2 targetdatabase.

v External StepsPerl is the recommended scripting language as it offers the best combination ofperformance and function. External steps should not be written in Visual Basicbecause UNIX support is required.

For performance and scalability reasons, exploit the native RDBMS functionality toevaluate derived component attributes, which are calculated columns in TivoliDecision Support.

Tivoli Decision Support calculated columns are implemented in Tivoli EnterpriseData Warehouse data marts as columns in the dimension tables and are not storedas component attributes. They are known as derived attributes. For example, for theDM/SPP Guide, the derived attributes are:


IP_A_Network, IP_B_Network, IP_C_Network, network_domain,network_subdomain, and so on

Complex trending functions are not recommended in Tivoli Enterprise DataWarehouse because there is a requirement for a single methodology for trendingacross all Tivoli applications. The solution for this should be a specializedconsumer application such as Capacity Planner. Currently in Tivoli DecisionSupport Guides, each application has implemented their own trending function.

Collection of distributed data by the applicationSimilar to Tivoli Decision Support, Tivoli Enterprise Data Warehouse requires thatthe source data belonging to a single application must reside in a single database(or a small number of databases). This means that for applications like TivoliDistributed Monitoring that can potentially be collecting probe data from a largenumber of systems, an initial data collection must be performed by the applicationto place the data into a single database. Ideally, this application-implemented initialdata collection will summarize the data to hourly″maximum/minimum/average/total/number of samples″ values. In this case, thescheduling of the Data Warehouse Center ETL processes can be signalled by theexternal program that performs the data collection. Aggregation of rawmeasurement data to the hour level is the recommended approach, sinceprocessing all the measurement data without any time-based aggregation raises therisk of flooding the system with too much data. In other words, if a monitor runsevery five minutes, instead of inserting 12 measurements per hour with the actualprobe results, only the hourly ″maximum/minimum/average/total/number ofsamples″ values are kept in the central data warehouse.

Tivoli Enterprise Data Warehouse functionalityThe following is Tivoli Enterprise Data Warehouse functionality that is notprovided in Tivoli Decision Support:v Component correlation

When two or more applications collect data about the same component(typically a machine), matching rules should be established to determine anexact match. For example, App A might use a short host name (myhost),whereas App B uses a fully qualified domain name (myhost.dev.tivoli.com). Thematching rules should be in a form so that the customer can fully understandthem, and if necessary make controlled modifications to suit their requirements.

v Exception handling for suspect dataThis enables suspect data to be recorded for analysis. If data from sourceapplications is missing or not valid, then exception data should be written toexception tables. These tables are similar to the central data warehouse tableswith the addition of a timestamp and message column. A summary report of theexceptions should be made available at the end of ETL processing.

v Temporal historyStart and end timestamps are maintained for both components and componentattributes, which enables temporal history to be maintained. This means that theexact state of a component when a measurement was taken can be establishedeven if the component has changed its attributes over time. For example, areport covering last month for all machines with Microsoft Windows NT ServicePack 5 should correctly handle the machines that have moved from Service Pack5 to Service Pack 6 in the middle of the month.

v Maintenance processes

Chapter 4. Migrating Tivoli Decision Support function 63

Maintenance processes should be provided to purge data, reorganizetables/indexes and generate statistics on tables/indexes. For example, a purgeprocess could automatically aggregate hourly data to the day level when thedata ages past a certain number of days (perhaps 6 months).

v Access control including multicustomer supportThe Tivoli Enterprise Data Warehouse will support multicustomer environments.

v Cross-application reportsApplications should aim to write cross-application reports to leverage the datastored in Tivoli Enterprise Data Warehouse. For example, Tivoli ApplicationPerformance Management could plot CPU load metrics on the same graphs astheir own response time metrics.

v Globalization supportTivoli Enterprise Data Warehouse enables the display of components, componentattributes, measurement types in a translated form by way of the TivoliEnterprise Data Warehouse report interface. It is the responsibility ofapplications exploiting Tivoli Enterprise Data Warehouse to provide thesetranslation strings.


Chapter 5. Naming conventions

To prevent object name conflicts, you must use a product code to uniquely identifyan application.

For third-party applications:The product code must consist of three characters, which includes at leastone number. For example, ET2.

For IBM and Tivoli software applications:The product code, sometimes referred to as the AVA code, consists of threealphabetic characters. This product code is also part of the Tivoli MessageStandard, which is described at:http://w3.tivoli.com/customer_support/serviceability/msgstandard.html

The Tivoli message ID format is XXXYY####Z, where XXX is thecomponent prefix, or product code. The product code is assigned throughthe IBM product registry. Application teams can request componentprefixes from IBM by completing the form at:http://w3sms.boulder.ibm.com/element.nsf/WPrefixRequest?OpenForm.

The Tivoli Enterprise Data Warehouse installation program uses the followingdatabase names:

TWH_CDWThe central data warehouse database

TWH_MDThe control database for metadata

TWH_MARTThe data mart database for star schemas

Use these database names to allow your ETL interchange files (*.tag) to beimported into the Data Warehouse Center without changes.

Objects created using the Data Warehouse CenterThe following objects are created using the Data Warehouse Center and are storedin the Tivoli Enterprise Data Warehouse control database, which is namedTWH_MD. Only the characters A through Z, a through z, 0 through 9, underscore(_), comma (,), and parentheses (()) are allowed in names stored in the controldatabase metadata tables.

Subject areasA subject area identifies and groups the processes that relate to a logical area of thebusiness.

<PRODUCT_CODE>_<full name of delivery application>_v<full version>_Subject_Area

Note: <full name of delivery application> should use mixed case.

Examples: CTO_Tivoli_Manager_for_Oracle_v2.1.0_Subject_Area

65

http://w3.tivoli.com/customer_support/serviceability/msgstandard.html

http://w3sms.boulder.ibm.com/element.nsf/WPrefixRequest?OpenForm

Warehouse sourcesWarehouse sources identify the source that will provide data to your warehouse.

<PRODUCT_CODE>_<Source DB Alias>_Source

Note: <Source DB Alias> should be uppercase only.

Example: CTO_TWH_CDW_Source

Warehouse targetsWarehouse targets are the databases that contain data that has been transformed.

<PRODUCT_CODE>_<Target DB Alias>_Target

Notes:

1. <Target DB Alias> should be uppercase only.2. TWH_CDW is used for <Target DB Alias> when the central data warehouse

database is a target.3. TWH_MART is used for <Target DB Alias> when the data mart database is a

target.

Examples:v CTO_TWH_CDW_Targetv CTO_TWH_MART_Target

ProcessesA process contains a series of steps that perform a transformation and movementof data for a specific warehouse use.

<PRODUCT_CODE>_<Process ID>_<Process Description>_Process

Notes:

1. <Process ID> is c<nn> or m<nn>, where:v c indicates target is central data warehouse.v m indicates target is data mart.v <nn> is a two digit number starting at 05 and incrementing by 5.

2. <Process Description> is a description of the process.

Examples:v CTO_c05_Exceptions_Processv CTO_c05_Initialize_Processv CTO_m05_Mart_Process

StepsA step is the definition of a single operation within the warehouse.

<PRODUCT_CODE>_<Process ID>_s<nnn>_<Step Description>.

Notes:

1. For Process ID, see “Processes”.


2. nnn is the order number of the step. Steps are numbered starting with 010 andincremented by 10 so that if a step needs to be added later, there is room toadd it in between existing steps and maintain the sequential numberingscheme.

3. <Step Description> can be any preferred description of the step.

Examples:v CTO_c05_s010_Extractv CTO_c05_s020_Transformv APF_c05_s010_init

Warehouse schemas<PRODUCT_CODE> <by time granularity> <Warehouse Schema Description> <StarSchema>

Notes:

1. <by time granularity> should be one of Hourly, Daily, Weekly, Monthly,Quarterly, Yearly

2. Spaces, rather than underscores, should be used as this name is displayed inthe report interface

Example: CTO Hourly Oracle Database Star Schema

Application-specific objects in data mart databasesOnly the characters A through Z, 0 through 9, and underscore (_) are allowed intable names in the data mart databases. All tables created should start with<PRODUCT_CODE>.

Dimension tables based on components<PRODUCT_CODE>.D_<Description of dimension table>

Note: Singular should be used instead of plural; for example, DATABASE, ratherthan DATABASES.

Examples:v CTO.D_ORA_DATABASEv CTO.D_ORA_DB_COMPv CTO.D_ORA_DB_SUB_COMP

Dimension tables based on measurement types<PRODUCT_CODE>.D_<Metric group description>_METRIC

Example: CTO.D_ORA_METRIC

Fact Tables<PRODUCT_CODE>.F_<Fact description>_<time granularity>

Note: <time granularity> should be one of HOUR, DAY, WEEK, MONTH

Example: CTO.F_ORA_DB_HOUR

Chapter 5. Naming conventions 67

Application-specific objects in the central data warehouseOnly the characters A through Z, 0 through 9, and underscore (_) are allowed innames for application-specific objects in the central data warehouse database. Thefirst part of the name for all objects should be <PRODUCT_CODE>.

Note: Applications must create their own schema and should never create objectsin the generic schema of the central data warehouse.

Staging tables<PRODUCT_CODE>.<STAGE>_<Description of staging table>

Example: CTO.STAGE_ORA

Application-specific data in the generic central data warehouse tablesDo not abbreviate product or service names. Use approved short names only.Creating other versions of a name (by abbreviating it or changing it in any otherway) can cause confusion and confusion about names can cause legal andtranslation problems.

Note: Use the following capitalization guidelines for data in the component typenames, measurement type names, and measurement type descriptions:v If there are multiple words, use sentence-style capitalization. For example:

Siebel serverv If product names are used, follow the capitalization rules for product

names:– Use initial caps for a full official name or for a simplified official name.– Use lowercase when the shortened name is a generic name.– Use capital letters when the shortened name is an all-caps abbreviation.

v For items with abbreviated names, such as PrcTotCpuTimeAvg, capitalizethe first letter and other letters as necessary so that the user can quicklydiscern the meaning of the name.

Component type codesComponent type codes are entered in the CompTyp_Cd column of theTWG.CompTyp table.

<short name for measurement source>_<short name for component type>

Notes:

1. <short name for measurement source> can include [A-Z], [0-9], underscore.2. <short name for component type> can include [A-Z], [0-9], underscore.3. Maximum length is 17 characters.

Examples:v CTO_DATABASEv CTO_SEGMENT

Attribute type codesAttribute type codes are entered in the AttrTyp_Cd column of the TWG.AttrTyptable.


<short name for attribute type>

Notes:

1. <short name for attribute type> can include [A-Z], [0-9], underscore.2. Maximum length is 17 characters.

Examples:v DUMPSPACE_TYPEv SERVICE_TYPE

Measurement type namesMeasurement type names are entered in the MsmtTyp_Nm column of theTWG.MsmtType table.

<measurement name>

Examples:v PhysicalReadsv PrcTotCpuTimeAvg

Objects displayed using the report interfaceAll names specific to applications must be named in the most meaningful waypossible so that end users can understand them. No codes should be used. Theuniqueness of these names is validated against a master central data warehousedatabase.

Report names<Name of Report>

Example: Least Free Space for Oracle Databases - Daily

Data mart namesData mart names represent the logical grouping of star schemas for access control,rather than the physical data mart databases.

<Name of Data Mart> Data Mart

Example: Tivoli Manager for Oracle Data Mart

Star schema names<Name of Warehouse Schema>

The name of the warehouse schema (minus the product code) is used. See“Warehouse schemas” on page 67.

Metric names<Metric Name>

Example: Freespace in Oracle Tablespaces

Chapter 5. Naming conventions 69

Chapter 6. Creating reports

Tivoli Enterprise Data Warehouse provides a report interface that creates outputcontaining a two-dimensional view of your data. The reports are designed toquickly answer common operational questions asked by customers about theirenvironment. The reports do not have the user interactivity that is typical of anOLAP product. You can use other tools to perform OLAP analysis, businessintelligence reporting, or data mining. The report interface uses the Web version ofthe IBM Console. The IBM Console is a distributed, device-independent, andplatform-independent presentation layer for Tivoli products. It is installed whenTivoli Enterprise Data Warehouse is installed. Web browsers are used to interactwith the Tivoli Enterprise Data Warehouse report interface. Installation ofadditional software is not required.

Tivoli Enterprise Data Warehouse provides the following types of reports:v Extreme casev Health checkv Summary

This chapter provides examples of report output for each report type and the SQLquery used to generate the report. Some knowledge of SQL is required tounderstand the information in this chapter.

Extreme case reportsA typical use of a ″One Measurement Versus Many Components″ class of report isto display the worst n cases (where n is a number such as 10). Extreme case reportsshow the extreme cases, such as the top 10 servers having the most critical events.Figure 6 on page 72 shows an example of extreme case report output.

71

The following items need to be gathered for creation of an extreme case report:

Group by attributes:Component attributes by which grouping is done. This is applied on theX-axis. For example, group all machines with common CPU speed andphysical memory; chosen by user during report creation.

Metric:Answers the question, ″What is our measurement type?″; chosen by userduring report creation. For example: average percent total CPU time busy

Aggregate function:Calculation done within the SQL to output the highest/lowest n values.Can be: average, total, minimum, maximum; chosen by user during reportcreation. For example: average percent total CPU time busy

Figure 6. Extreme case report output example


Star schema tables:Collection of fact and dimension tables that we are pulling informationfrom; Defined within the data mart in the RPI GUI. For example, SPPDaily DMInv Star Schema

Star schema joins:The relationship of the star schema tables to each other.

Time interval:Defined by which fact table is used (hour/day and so on); chosen by userduring report creation.

Start time:Beginning time frame for data in the report; chosen by user during reportcreation, but can be changed at report run time.

End time:Ending time frame for data in the report; chosen by user during reportcreation, but can be changed at report run time.

Order by:This is always the output of the aggregation function in a Top n report.

Order type:Ascending or descending; chosen by user during report creation.Determines order of data on X axis.

Component restrictions (optional):Limitations or filters set on the component; chosen by user during reportcreation

Based on the gathered information, an SQL query such as the following is createdby the report interface:select avg(avg_value), //Aggregation value chosen was average.

//avg_value gives the actual metric: PrcTotCpuTimeDM.D_HOST_STATE.PHYSICAL_MEMORY, //groupby attribute that will go on XaxisDM.D_HOST_STATE.CPU_SPEED //groupby attribute that will go on Xaxis

fromDM.D_HOST_STATE, //Star schema dimension table as defined in data martDM.D_METRIC, //Star schema dimension table as defined in data martDM.F_DAY //Star schema fact table as defined in data martwhereDM.F_DAY.HOST_ID = DM.D_HOST_STATE.HOST_ID AND //Star schema table joinsDM.F_DAY.METRIC_ID = DM.D_METRIC.METRIC_ID ANDDM.D_HOST_STATE.STATE_STRT_DTTM = DM.F_DAY.HOST_STATE_STRT_DTTM ANDDM.D_HOST_STATE.HOST_ID = DM.D_HOST_STATE.HOST_ID ANDDM.F_DAY.HOST_ID = DM.F_DAY.HOST_ID ANDDM.F_DAY.HOST_STATE_STRT_DTTM = DM.D_HOST_STATE.STATE_STRT_DTTM AND

//Metric = PrcTotCpuTimeDM.D_METRIC.met_name = ’PrcTotCpuTime’

//Time restriction column between chosen time frameAND DM.F_DAY.meas_date between (timestamp(current date - 30 days, ’00.00.00’)) and

(timestamp(current date, ’00.00.00’))group by DM.D_HOST_STATE.PHYSICAL_MEMORY, //groupby attribute that will go

//on X axisDM.D_HOST_STATE.CPU_SPEED //groupby attribute that will go

//on X axisorder by1 //order by will always be the output of

//aggregate function in an extreme//case report

desc //Order type as chosen by the//user (descending)

Chapter 6. Creating reports 73

Health check reportsA typical use of a ″Many Measurements Against Many Components Versus Time″class of report is a health check report. This type of report depicts a givenmeasurement, or measurements, against a specified time frame. The X-axis alwaysshows a specified time frame. The Y-axis (or axes) shows the measures you wouldlike to see, within the specified X-axis time frame. You can illustrate one to fivemeasurements on the Y-axis. Figure 7 shows an example of health check reportoutput.

The following items need to be gathered for creation of a health check report:

Group by attribute:Time unit by which grouping is done; chosen by user during reportcreation.

Aggregate function:Calculation done within the SQL. Can be such things as average, total,minimum, maximum; chosen by user during report creation.

Figure 7. Health check report output example


Metric:Answers the question, ″What are our measurements?″; chosen by userduring report creation.

Star schema tables:Collection of fact and dimension tables that information is pulled from;defined within the data mart

Star schema joins:The relationship of the star schema tables as defined in the data mart

Start time:Beginning time frame; chosen by user during report creation, but can bechanged at report run time

End time:Ending time frame; chosen by user during report creation, but can bechanged at report run time

Time interval:Defined by which fact table is used (hour/day and so on); chosen by userduring report creation

Order by:This is always the aggregate attribute in a health check report

Order type:Ascending or descending; always ascending for health check reports


With a health check report, each metric chosen represents a line on the graph. Fiveis the maximum number of metrics, so there are no more than 5 lines on a graph.In the following example, five metrics are chosen. Each metric corresponds to onecomplete SQL statement.

Metric A: Percent Total CPU Time Average

Metric B: Percent Total CPU Time Minimum

Metric C: Percent Total CPU Time Maximum

Metric D: Percent Total Privilege Time Average

Metric E: Percent Total Privilege Time Minimum

Based on Metric A, an SQL query such as the following is created by the reportinterface:select avg(avg_value), //Aggregation chosen was average.

//avg_value gives the actual metric: PrcTotCpuTime averagemeas_date //Time interval, defined by the fact table will be shown

//on the X-axis.from DM.D_HOST_STATE, //Star schema dimension table as defined in the data mart.DM.D_METRIC, //Star schema dimension table as defined in the data mart.DM.F_DAY //Star schema fact table as defined in the data mart.

//Note that this is a daily fact table, so this gives//us the time interval.

where DM.F_DAY.HOST_ID = DM.D_HOST_STATE.HOST_ID //Star schematable joinsAND DM.F_DAY.METRIC_ID = DM.D_METRIC.METRIC_ID


AND DM.D_HOST_STATE.STATE_STRT_DTTM = DM.F_DAY.HOST_STATE_STRT_DTTMAND DM.D_HOST_STATE.HOST_ID = DM.D_HOST_STATE.HOST_IDAND DM.F_DAY.HOST_ID = DM.F_DAY.HOST_IDAND DM.F_DAY.HOST_STATE_STRT_DTTM = DM.D_HOST_STATE.STATE_STRT_DTTM

//Metric chosen is PrcTotCpuTimeAND DM.D_METRIC.met_name = ’PrcTotCpuTime’

//Time restriction between start and end timeAND DM.F_DAY.meas_date between ’2001-11-01-00.00.00.000000’and ’2001-11-30-00.00.00.000000’group by meas_date //For health check, always group by the time field.order by meas_date asc //For health check, always order by the time field.

//Ascending or Descending as chosen by user.

Based on Metric B, an SQL query such as the following is created by the reportinterface:select min(min_value), //Aggregation chosen was minimum.

//min_value gives the actual metric: PrcTotCpuTime minimummeas_date //Time interval, defined by the fact table will be shown

//on the X-axis.from DM.D_HOST_STATE, //Star schema dimension table as defined in the data mart.DM.D_METRIC, //Star schema dimension table as defined in the data mart.DM.F_DAY //Star schema fact table as defined in the data mart.

//Note that this is a daily fact table, so this gives//us the time interval.

where DM.F_DAY.HOST_ID = DM.D_HOST_STATE.HOST_ID AND //star schematable joinsDM.F_DAY.METRIC_ID = DM.D_METRIC.METRIC_IDAND DM.D_HOST_STATE.STATE_STRT_DTTM = DM.F_DAY.HOST_STATE_STRT_DTTMAND DM.D_HOST_STATE.HOST_ID = DM.D_HOST_STATE.HOST_IDAND DM.F_DAY.HOST_ID = DM.F_DAY.HOST_IDAND DM.F_DAY.HOST_STATE_STRT_DTTM = DM.D_HOST_STATE.STATE_STRT_DTTM


//Time restriction between start and end timeAND DM.F_DAY.meas_date between ’2001-11-01-00.00.00.000000’and ’2001-11-30-00.00.00.000000’

group by meas_date //For health check, always group by the time field.order by meas_date asc //For health check, always order by the time field.

//Ascending or descending as chosen by user

Based on Metric C, an SQL query such as the following is created by the reportinterface:select max(max_value), //Aggregation chosen was maximum.

//max_value gives the actual metric://PrcTotCpuTime maximum

meas_date //Time interval, defined by the fact table will//be shown on the X-axis

from DM.D_HOST_STATE, //Star schema dimension table as defined in//the data mart.

DM.D_METRIC, //Star schema dimension table as defined in//the data mart.

DM.F_DAY //Star schema fact table as defined in the data mart.//Note that this is a daily fact table, so//this gives us the time interval.

where DM.F_DAY.HOST_ID = DM.D_HOST_STATE.HOST_ID AND //Star schematable joinsDM.F_DAY.METRIC_ID = DM.D_METRIC.METRIC_IDAND DM.D_HOST_STATE.STATE_STRT_DTTM = DM.F_DAY.HOST_STATE_STRT_DTTMAND DM.D_HOST_STATE.HOST_ID = DM.D_HOST_STATE.HOST_IDAND DM.F_DAY.HOST_ID = DM.F_DAY.HOST_IDAND DM.F_DAY.HOST_STATE_STRT_DTTM = DM.D_HOST_STATE.STATE_STRT_DTTM






Based on Metric D, an SQL query such as the following is created by the reportinterface:select avg(avg_value), //Aggregation chosen was average.

//avg_value gives the actual metric://PrcTotPrivTime average

meas_date //Time interval, defined by the fact table,//will be shown on the X-axis

from DM.D_HOST_STATE, //Star schema dimension table as defined in//the data mart.

DM.D_METRIC, //Star Schema dimension table as defined in//the data mart.

DM.F_DAY //Star schema fact table as defined in the//data mart. Note that this is a daily fact//table, so this gives us the time interval.

where DM.F_DAY.HOST_ID = DM.D_HOST_STATE.HOST_ID AND //star schematable joinsDM.F_DAY.METRIC_ID = DM.D_METRIC.METRIC_IDAND DM.D_HOST_STATE.STATE_STRT_DTTM = DM.F_DAY.HOST_STATE_STRT_DTTMAND DM.D_HOST_STATE.HOST_ID = DM.D_HOST_STATE.HOST_IDAND DM.F_DAY.HOST_ID = DM.F_DAY.HOST_IDAND DM.F_DAY.HOST_STATE_STRT_DTTM = DM.D_HOST_STATE.STATE_STRT_DTTM

//Metric chosen is PrcTotPrivTimeAND DM.D_METRIC.met_name = ’PrcTotPrivTime’




Based on Metric E, an SQL query such as the following is created by the reportinterface:select min(min_value), //Aggregation chosen was minimum

//min_value gives the actual metric://PrcTotPrivTime minimum

meas_date //Time interval, defined by the fact table,//will be shown on the X-axis

from DM.D_HOST_STATE, //Star schema dimension table as defined//in the data mart.

DM.D_METRIC, //Star schema dimension table as defined//in the data mart.

DM.F_DAY //Star schema fact table as defined in the data mart.//Note that this is a daily fact table, so this gives//us the time interval.


//Metric chosen is PrcTotPrivTimeAND DM.D_METRIC.met_name = ’PrcTotPrivTime’





Summary reportsA typical use of a ″Many Measurements Versus Many Components″ class of reportis a summary report. This type of report is useful for showing subtotals for aparticular grouping and grand totals for the entire report. The rows representcomponents, or more often component groupings. The columns contain typicallybetween 10-15 measurements. Figure 8 shows an example of summary reportoutput.

The following items need to be gathered for creation of a summary report:

Group by attributes:Component attributes by which grouping is done; chosen by user duringreport creation

Aggregate function:Calculation done within the SQL. Can be: average, total, minimum,maximum; chosen by user during report creation

Metric:Answers the question, ″What are the measurements?″; chosen by userduring report creation

Figure 8. Summary report output example


Star schema tables:Collection of fact and dimension tables that information is pulled from;defined within the data mart

Star schema joins:The relationship of the star schema tables as defined in the data mart

Start time:Beginning time frame; chosen by user during report creation, but can bechanged at report run time

End time:Ending time frame; chosen by user during report creation, but can bechanged at report run time

Time interval:Defined by which fact table is used; chosen by user during report creation

Order by attributes:Component attributes by which ordering is done; chosen by user duringreport

Order type:Ascending or descending; chosen by user during report creation


Based on the gathered information, an SQL query such as the following is createdby the report interface:select avg(avg_value) as avgofavg_value, //Aggregation chosen was average

//avg_value gives the actual metric://PrcTotCpuTime average

max(max_value) as maxofmax_value, //Aggregation chosen was maximum//max_value gives the actual metric://PrcTotPrivTime maximum

DM.D_HOST_STATE.PHYSICAL_MEMORY, //group by as chosen by userDM.D_HOST_STATE.CPU_SPEED, //group by as chosen by userDM.D_METRIC.met_name //Always choose the metric name for

//summary reports.from DM.D_HOST_STATE, //Star schema dimension table as defined

//in the data mart.DM.D_METRIC, //Star schema dimension table as defined

//in the data mart.DM.F_DAY //Star schema fact table as defined

//in the data mart.//Note that this is a daily fact table,//so this gives us the time interval.


//Metrics chosen were PrcTotCpuTime and PrcTotPrivTimeAND DM.D_METRIC.met_name in (’PrcTotCpuTime’, ’PrcTotPrivTime’)


group by DM.D_HOST_STATE.PHYSICAL_MEMORY, //group by as chosen by user//DM.D_HOST_STATE.CPU_SPEED,


//group by as chosen by userDM.D_METRIC.met_name //Always group by the metric name for

//a summary reportorder by DM.D_HOST_STATE.PHYSICAL_MEMORY asc, //order by as chosen by userDM.D_HOST_STATE.CPU_SPEED asc, //order by as chosen by userDM.D_METRIC.met_name //Always order by the metric name for

//a summary report.

Report metadataMetadata is data about data. To implement reporting functionality, the reportinterface needs to store report metadata such as the report type, the SQL queryused to generate the report, the report name, data marts, star schemas, usergroups, and so on. The Tivoli Enterprise Data Warehouse control database is usedto store this report metadata.

Use an SQL script similar to the following to export application-specificinformation from the report interface (RPI) database to four comma-separatedvalue (CSV) files.

-- This csv contains the Datamart and Reports

export to rpi_export1.csv of del select d.datamart_name, d.datamart_desc,r.report_name, r.report_desc, r.report_type, r.selections, r.report_creator,r.isPublic, r.isScheduled from rpi.datamart d, rpi.report rwhere r.datamart_id = d.datamart_id;

-- This csv contains the Reports and Queries

export to rpi_export2.csv of del select d.datamart_name, r.report_name,q.sql from rpi.datamart d, rpi.report r, rpi.query qwhere r.datamart_id = d.datamart_id AND q.report_id = r.report_id;

-- This csv contains the star schema name for getting the IWH.starschema.iwhid-- for insertion into rpi.relationship where REL_TYPE = ’SchemaToDatamart’

export to rpi_export3.csv of del select i.name, d.datamart_namefrom iwh.starschema i, rpi.datamart d, rpi.relationship rwhere r.source_id = i.iwhid AND r.target_id = d.datamart_id ANDr.rel_type = ’SchemaToDatamart’;

-- This csv contains the star schema name for getting the-- IWH.starschema.iwhid for insertion into rpi. relationship-- where REL_TYPE = ’SchemaToResourceBundle’

export to rpi_export4.csv of del select i.name,r.target_id from iwh.starschema i,rpi.relationship r where r.rel_type = ’SchemaToResourceBundle’AND i.iwhid = r.source_id;

Prior to running this script, be sure that your RPI database (TWH_MD) does notcontain any other application’s data. In other words, it should only contain reportsyou want to include with your product. After running the script, add yourapplication’s product code to the name of each resulting CSV file. For example, ifyour product code is spp, you would rename the CSV files spp_rpi_export1.csv,spp_rpi_export2.csv, and so on. The four CSV files must be put into the followingdirectory in your warehouse enablement pack:tedw_apps_etl\product_code\pkg\vversion\report. The Tivoli Enterprise Data


Warehouse installation uses the CSV files to input the appropriate database recordsinto the customer’s environment. (See Chapter 8, “Assembling the warehouseenablement pack” on page 93 for more information.) The customer can then use theTivoli Enterprise Data Warehouse report interface to view your data marts, reports,and so on.

The following tables are used for RPI metadata.

Note: Do not manually update or delete the content of these tables. Unique binarydata is used throughout the tables and the RPI has a dependency on thedata relationships.

Table 14. RPI.Datamart. This table contains information about all RPI-ready data marts.Population occurs when a data mart is created through the RPI GUI.

Column name Column type Column description Key?

Datamart_Id CHAR FOR BITDATA

Unique numberidentifying each query

Primary key

Datamart_Name VARCHAR Unique name of datamart

No

Datamart_Desc VARCHAR Description of data mart No

DM_Alias VARCHAR The DSN name set upon the control server

No

DM_User VARCHAR User ID for DMdatabase

No

DM_Schema VARCHAR Database schema name No

DM_URL VARCHAR The JDBC driver to beused. It is alwaysdb2:jdbc:<alias_name>

No

DM_Pwd_Encrypt VARCHAR Database password No

DM_Driver VARCHAR Not currently used No

Last_Changed TIMESTAMP Date and time of lastmodification of this datamart

No

Table 15. Table RPI.Report. This table contains information about all Tivoli Enterprise DataWarehouse reports. Population of the table occurs when a new report is created from theRPI GUI.


Report_Id CHAR FOR BITDATA

Unique numberidentifying reports

Primary key

Datamart_Id CHAR FOR BITDATA

ID of data mart Foreign key

Report_Name VARCHAR Name of the report asinput by the user

No

Report_Desc VARCHAR Description of report No

Report_Type VARCHAR One of three possiblereport types

No

Report_Creator VARCHAR User ID of the personthat created thereport.

No


Table 15. Table RPI.Report (continued). This table contains information about all TivoliEnterprise Data Warehouse reports. Population of the table occurs when a new report iscreated from the RPI GUI.


Is_Public CHAR Designates whetherthe report is public orpersonal

No

Is_Scheduled CHAR Designates if thereport is manuallyrun or automaticallygenerated

No

Last_Changed TIMESTAMP Identifies whenreport was lastmodified.

No

Table 16. RPI.Query. This table contains information about SQL queries used to createreports. Population of the table occurs when a report is created from the RPI GUI.


Query_Id CHAR FOR BITDATA

Unique numberidentifying eachquery

Primary key


Unique numberidentifying reports

Foreign key

SQL VARCHAR SQL statement usedto generate the report

No

Table 17. RPI.ReportView. This table contains information about the running of reports. Itincludes the information needed to display report output. Population of the table occurswhen a report output is saved from the RPI GUI.


View_Id CHAR FOR BITDATA

Unique numberidentifying reportoutput

Primary key


Unique numberidentifying sourcereport ID

Foreign key

View_Name VARCHAR Unique name ofreport output

No

View_Desc VARCHAR Description of reportoutput

No

Time_Selections VARCHAR Stores theuser-selected timevalues for this reportoutput, such as dateranges and filtering.

No


Table 17. RPI.ReportView (continued). This table contains information about the running ofreports. It includes the information needed to display report output. Population of the tableoccurs when a report output is saved from the RPI GUI.


Time_Gran INTEGER Integer value statingthe time_granularityof the star schemaused for this reportoutput.

0 = hourly

1 = daily

2 = weekly

3 = monthly

No

Timestamp TIMESTAMP Date and time of lastmodification of thisreport output

No

Output BINARY The output of thisquery with theseparameters in theform of one or manyJPEG files.

No

Last_Changed TIMESTAMP Not used No

Table 18. RPI.TimeFilters. This table contains filter names and values that can be usedwhen creating reports. The value is added to the SQL query when a report is created. Thistable is pre-populated during RPI installation. New filters and values can be added, butproper SQL statements must be entered for the values.


Time_Filter_Name VARCHAR Filter name as itappears in RPI GUI

Primary key

Time_Filter_Values VARCHAR SQL fragment thatgets appended towhere clause

No

Table 19. RPI.UserGroup. This table contains information about users that are grouped foraccess control to data marts.


UGroup_Id CHAR FOR BITDATA

Unique numberidentifying usergroup

Primary key

UGroup_Name VARCHAR Unique nameidentifying usergroup

No

UGroup_Descr VARCHAR Describes usergroup

No

Last_Changed TIMESTAMP Time of update No

Table 20. RPI.SchemaVersion. This table lists which version of the RPI schema is in use.


Version CHAR Version number Primary key


Table 20. RPI.SchemaVersion (continued). This table lists which version of the RPI schemais in use.


Migrate_Script VARCHAR Name of the scriptused to update theversion of themetadata database

No

Table 21. RPI.Strings. This table contains information on the star schemas such as joinsbetween tables, display columns, and table names. It is updated by triggers on DB2 tableswhen the Data Warehouse Center is updated and by scripts during an applicationwarehouse pack installation, assuming the warehouse pack contains pre-developed reports.


Name VARCHAR Star schema name No

Type CHAR One of thefollowing:

v Qjoincol

v Qfromline

v Qdispcol

No

Seqno INTEGER Counter Primary key

Text VARCHAR Lists the actual joinor column name

Table 22. RPI.SSUpdated. This table contains the ID of star schemas that have beenupdated with new data. It is populated by a user-defined program in ETL and is used by theRPI to run scheduled reports once the star schema has been updated. The RPI deletesentries from this table when the scheduled reports have been run and saved.


Id CHAR Star schema ID Primary key

Table 23. RPI.Relationship. This table contains information about the various relationshipsand dependencies across RPI metadata. (Note that much of this data is binary.)


Rel_Type VARCHAR Text that tells thesource and targetfor the relationship

No

Source_Id VARCHAR ID of source. Variesper relationshiptype

No

Target_Id VARCHAR ID of target. Variesper relationshiptype

No

Last_Changed TIMESTAMP Time of update No


The following table shows the valid relationship types for Table 23 on page 84.

Table 24. Relationship types

Relationship type Description Source Target

UgroupToUser Usermembershipin the group

UGroup_Id fromRPI.UserGrouptable–group identifier

User ID as it is used byTivoli PresentationServices

UgroupToDatamart Group accessto data mart

UGroup_Id fromRPI.UserGrouptable–group identifier

Datamart_Id fromRPI.Datamarttable–data martidentifier

SchemaToDatamart Star schema,or schemas,belonging todata mart

IWHID fromIWH.StarSchematable–star schemaidentifier

Datamart_Id fromRPI.Datamarttable–data martidentifier

SchemaToResourceBundle Tells whichresourcebundle a starschema uses

IWHID fromIWH.StarSchema

Resource bundle filename

ReportToStarSchema Tells whichstar schema,or schemas, areport uses.

Report_Id fromRPI.Reporttable–report identifier

IWHID fromIWH.StarSchema tablestar schema identifier

Report formattingThe Tivoli Enterprise Data Warehouse report interface uses JClass Chartcomponents for its reporting needs. JClass Chart provides multiple chart types fordata presentation. JClass Chart supports many different types of data sourcesincluding files, sockets, and real-time streams. The Tivoli Enterprise DataWarehouse report interface provides the following report formatting features:v Uses JClass Chart controls to display graphs for extreme case and health check

report typesv Displays values on X and Y axes as they best fit the output screen. The user

cannot change the appearance of the axis’ labels. The Tivoli Enterprise DataWarehouse report interface displays multiple Y axes where appropriate. Forexample, a health check report can be displayed with the 2nd Y axis displayinga scale more appropriate for a certain measurement.

v Displays a legend describing the chart contents. The legend is displayed belowthe chart and justified to the report output center. The legend can be put to theright of the chart if space allows.

Tivoli Presentation Services is used to display report results. Information isdisplayed using HTML that is compatible with all standard Web browsers. TheTivoli Enterprise Data Warehouse report interface saves the output of JClass Chartin JPEG format when a report is run and saved. The JPEG image is displayedwhen viewing the report output with a browser. After the user finishes workingwith the report output and saves it, no changes of chart properties are allowed.The report output contents are stored as an HTML page with references to theappropriate images. The report output is stored in the Tivoli Enterprise DataWarehouse control database (TWH_MD).


Chapter 7. Internationalizing reports and application data

Tivoli Enterprise Data Warehouse enables the display of the following objects in atranslated form by using the report interface. Source applications must extract thefollowing values and deliver corresponding Java resource files:v Central data warehouse metadata from the central data warehouse database

(TWH_CDW):– Component type names (CompTyp.CompTyp_Nm)– Component relationship type names (RelnTyp.RelnTyp_Nm)– Attribute type names (AttrTyp.AttrTyp_Nm)– Measurement group type names (MGrpTyp.MGrpTyp_Nm)– Measurement group names (MGrp.MGrp_Nm)– Measurement unit category names (MUnitCat.MUnitCat_Nm)– Measurement unit names (MUnit.MUnit_Nm)– Time summary names (TmSum.TmSum_Nm)– Measurement source names (MSrc.MSrc_Nm)– Measurement type descriptions (MsmtTyp.MsmtTyp_Ds)

v Metric metadata from the data mart database (TWH_MART):– Metric names– Metric descriptions– Metric units– Metric categories

v Star schema metadata from the control database (TWH_MD):– Displayable dimension column names– Star schema descriptions

Tivoli Enterprise Data Warehouse displays the names as they were inserted for thefollowing objects:v Report namesv Data mart names

National language support processTranslated terms are provided in two formats:v Java resource files provided by ETL developersv A database table (Translated_Term) within the central data warehouse, which is

populated with entries from the Java resource files during a warehouseenablement pack installation

The report interface uses the Java resource files to look up translated terms.Applications can use the Java resource files or the Translated_Term table to look uptranslated terms.

87

The following steps describe the process illustrated in Figure 9.1. Read from the Tivoli Enterprise Data Warehouse metadata and generate

application-specific resource bundle files that contain all terms requiringtranslation. Include your application’s product code in the file names asfollows, where PRODUCT_CODE must be uppercase:

PRODUCT_CODEnls.javaPRODUCT_CODEnls_en.java

Edit both Java files and place the appropriate package statement containing thefollowing information into the file, where product_code must be lowercase:package com.ibm.twh.product_code.nls

The contents of the two files are identical except for the the Java class names.The following shows a portion of a sample Java file.package com.ibm.twh.dmn.nls;

import java.util.ListResourceBundle;

public class DMNnls extends ListResourceBundle {

public Object[][] getContents() {return contents_;

}

private static final Object[][] contents_ = {

//----------------------------------------

App1

Tivoli EnterpriseData Warehousereport interface

Consumer application

App1 warehousebundle

(English only)

Source application

1

3

6

.

.

Bundles

Installation media54

Tivoli EnterpriseData Warehouse

Translated_Term tableTranslation

Control databasemetadata

Report interface

TWHResources

App4

App1 warehousebundles

(all languages)

App1 applicationbundles




Applicationresource tool

Central datawarehouse metadata

Data mart databasemetadata

Control databasemetadata

2

Figure 9. Understanding the national language support process


// Component Type Name (from the CDW DB)//----------------------------------------{"CompTyp.CompTyp_Nm.TME_ENDPOINT", "TME endpoint"},{"CompTyp.CompTyp_Nm.UNIX_IP_HOST", "Unix IP host"},{"CompTyp.CompTyp_Nm.UNIX_AMW_HOST", "Unix AMW host"},{"CompTyp.CompTyp_Nm.UNIX_DMN_HOST", "Unix DMN host"},{"CompTyp.CompTyp_Nm.WIN_AMW_HOST", "Windows AMW host"},{"CompTyp.CompTyp_Nm.IP_INTERFACE", "IP interface"},{"CompTyp.CompTyp_Nm.WIN_DMN_HOST", "Windows DMN host"},{"CompTyp.CompTyp_Nm.AMW_HOST", "AMW host"},{"CompTyp.CompTyp_Nm.WIN_IP_HOST", "Windows IP host"},{"CompTyp.CompTyp_Nm.IP_HOST", "IP host"},{"CompTyp.CompTyp_Nm.DMN_HOST", "DMN host"},

//-----------------------------------------------------// Component Relationship Type Name (from the CDW DB)//-----------------------------------------------------{"RelnTyp.RelnTyp_Nm.PCHILD", "Parent Child Relation"},

...

};}

Note: When compiling the resource bundles, be sure to use the UTF8 characterencoding option. For example:javac -encoding UTF8 *.java

2. Translate the default list resource bundle (PRODUCT_CODEnls.java) into thesupported languages for your application. The resulting set of Java resourcefiles should contain the translated text.Place the resulting Java files into your production build. The build processshould produce two national language support (NLS) JAR files. One JAR filecontains the PRODUCT_CODEnls.class and the PRODUCT_CODEnls_en.classfiles. (Note that in the class file names, the product code must be uppercase.)The other JAR file contains all of the other translated bundle classes. Class filesinside the JAR file must be in the package structure/com/ibm/twh/avacode/nls/...A version must be supplied in the JAR file name to facilitate tracking ofversions; for example, 1.1.0. Using the following format for the JAR file namesgenerated by the application build process:

[email protected][email protected]

Note: The version must correspond with the value of the APP_REL_VERSIONproperty, which is specified in the twh_install_props.cfg file. (See“Application installation properties” on page 95.)

These NLS bundle JAR files are packaged on the application installation mediain the tedw_apps_etl\product_code\pkg\vversion\i18n subdirectory. SeeChapter 8, “Assembling the warehouse enablement pack” on page 93 for moreinformation.

3. The customer uses the Tivoli Enterprise Data Warehouse installation program toinstall the application warehouse enablement pack. During installation, the Javaresource files are placed on the same machine as the Tivoli Enterprise DataWarehouse report interface server.

Chapter 7. Internationalizing reports and application data 89

Note: For the translated strings to be displayed using the report interface, anentry must be made to the control database metadata that indicates thespecific resource files to be used for each star schema. This is apost-installation step users must perform manually. See Installing andConfiguring Tivoli Enterprise Data Warehouse for instructions.

4. During installation, the application-specific Java resource files are parsed andan entry is made to the Translated_Term table for every key and locale.

5. The report interface uses the Java resource files to look up translated terms foreach star schema that it uses for reporting.

6. Consumer applications that do not use the report interface can use the Javaresource files or a customized Java class for accessing the translated stringsstored in the Translated_Term table of the central data warehouse.

The Translated_Term table in the central data warehouse stores applicationresource translation strings. The format of this table and sample data for Englishand French is shown in Table 25:

Table 25. Translated_Term table

Term_ID Term_Key Term_Locale Term_Value

1 CompTyp.CompTyp_Nm.HOST en_US '484F5657'

2 CompTyp.CompTyp_Nm.IP_ADDRESS fr_FR '515A56723452376'

... ... ... ...

An entry for each application resource translation is made at installation time.

Term_IDIs populated by an ascending database sequence and can be used to detectwhen new entries appear in the table.

Term_KeyIs composed of TableName.ColumnName.CodeValue, where CodeValue is thevalue of the code column in the table. For example,MGrp.MGrpTyp.TOT_E, rather than MGrp.MGrpTyp.Total value exists.

Term_LocaleUses the Java locale format ″Language_Country_Variant″

Term_ValueIs the translated version of the Term_Key in Unicode format. As thecolumn is binary (for bit data), no code page applies to the column.

If you are mixing two or more application’s data in the same star schema andlanguage bundles already exist for those applications, or if you are pulling two ormore application’s data for your own application, you can read source applicationresource translations out of the central data warehouse database and use themdirectly or create a consolidated resource bundle on any target system. Thisaddresses the following:v Translated resources do not have to be identified in advance.v Entries in the table are unique, which addresses consistency and efficiency

concerns.v Sharing of resource bundles is done by way of the central data warehouse using

database mechanisms. No new distribution mechanism needs to be created.


If you are developing new star schemas from an existing warehouse pack from asingle application, use the bundle included with the warehouse pack. If you aredeveloping new star schemas with your own application data, simply use yourown bundle.

Chapter 7. Internationalizing reports and application data 91

Chapter 8. Assembling the warehouse enablement pack

A warehouse enablement pack, or warehouse pack, is a separately-installable partof a Tivoli software product that provides Tivoli Enterprise Data Warehousefunctionality. To create a warehouse pack for an application, you must use thenaming conventions and specific file layout structure that is described in thischapter.

Figure 10 on page 94 shows the layout of a warehouse pack.

93

\tedw_apps_etl\twh_app_install_list.cfg

\pkg\twh_install_props.cfg

\vversion

\cdw

product_code_cdw_drop_schema.sql

product_code_cdw_drop_triggers.sql

product_code_cdw_schema.sql

product_code_cdw_triggers.sql

product_code_cdw_data.sql

\mart

product_code_cdw_drop_schema.sql

product_code_cdw_drop_triggers.sql

product_code_cdw_schema.sql

product_code_cdw_triggers.sql

product_code_mart_data.sql

\i18n

\report

\doc

\etl

\ddl

\dml

\ddl

\dml

product_code.nls@ .jarversion

product_code.nls_intl@ .jarversion

product_code_rpi_export1.csv




product_code_dwc_data.tag

\misc

product_code_after.sh

\ \twh_app_install_list.cfgproduct_code

twh_cdw_data.sql

\sql

Figure 10. Warehouse pack layout


Top-level directoryThe top-level directory, tedw_apps_etl, must contain a file namedtwh_app_install_list.cfg. This file lists the product code of each of the warehousepacks to be installed by the Tivoli Enterprise Data Warehouse installation program.For example:############################################################################# This file includes the list of directories that equate to the application# warehouse enablement packs to be installed by the Tivoli Enterprise# Data Warehouse installation program.## List the product codes that equate to the directories for each# application under the application media top directory.# Use lowercase values.## Comment out any application you do not want the installation program to install# with a ’#’.##############################################################################cto/pkg#spp/pkgapf/pkg

Product code directoryThe product code directory, tedw_apps_etl\product_code, must also contain a filenamed twh_app_install_list.cfg. This file simply contains the keyword pkg tosupport single warehouse enablement pack installations. For example:############################################################################# This file includes the keyword ’pkg’ to enable a single application warehouse# enablement pack installation by the Tivoli Enterprise Data Warehouse# installation program.#############################################################################pkg

Application installation propertiesThe tedw_apps_etl\product_code\pkg directory must contain a file namedtwh_install_props.cfg, which includes the application installation propertiesrequired to complete a warehouse pack installation. All fields in this file must havea supplied value. For example:## Display Name (use the following format)# Product name, Version n.n.n, Warehouse Enablement Pack, Version n.n.nAPP_DISPLAY_NAME=Tivoli Application Performance Management, Version 2.1.0,Warehouse Enablement Pack, Version 1.1.0## Product code of applicationAVA_CODE=apf## Required version of Tivoli Enterprise Data Warehouse Core Tools# Must already be installedTWH_CORE_PREREQ_VERSION=1.1.0## Release version of application warehouse packAPP_REL_VERSION=1.1.0## Patch Version of this applicationPATCH_REL_VERSION=0## Custom application after script run at end of warehouse pack installation.

Chapter 8. Assembling the warehouse enablement pack 95

# Set value to 'null' (without single quotes) if no after script is needed.AFTER_SCRIPT=null## Should try to use product code for data mart schema 2 part name.APP_DATAMART_SCHEMA=apf## values either true or false - will skip report processing if falseREPORTS_INCLUDED=false

Application componentsThe application top-level directory is:

tedw_apps_etl\product_code\pkg\vfull version of warehouse pack (no periods)\

For example: tedw_apps_etl\cto\pkg\v210\

Note: The version number that the directory requires must be identical to thevalue of the APP_REL_VERSION property in the twh_install_props.cfg file,except the periods need to be removed in the directory name. (See“Application installation properties” on page 95.) This version numbershould be three period-separated numerical fields, which represent majorrelease, minor release, and patch or service level, respectively. For example,APP_REL_VERSION=1.1.0 (from the twh_install_props.cfg file) shouldequate to the following:v v110 (directory under the product_code directory in the warehouse pack)v [email protected] and

[email protected] (in the i18n directory in thewarehouse pack).

The version number identifies each application’s warehouse packdeployment, not the version of the application itself. Subsequent versions ofthe warehouse pack must have a numerically higher value than the versionit replaces. Otherwise, you are free to manage the number within yourorganization.

Within this directory, use the following directories to store the various applicationcomponents. Note that all directory and file names should be in lowercase.

cdw Contains the following subdirectories:

ddl\ Contains create scripts (data definition language files) for stagingtables intended for the central data warehouse database. Files inthis directory must be named as follows:v product_code_cdw_drop_schema.sqlv product_code_cdw_drop_triggers.sqlv product_code_cdw_schema.sqlv product_code_cdw_triggers.sql

dml\ Contains insert data scripts (data manipulation language files) fortables in the central data warehouse database. The files in thisdirectory must be named as follows:v product_code_cdw_data.sqlv twh_cdw_data.sql


The twh_cdw_data.sql script is located in the enablement directory on theTivoli Enterprise Data Warehouse installation CD. Copy the script from theCD to this directory in your warehouse pack.

mart Contains the following subdirectories:

ddl\ Contains the create scripts (DDL) for star schema tables intendedfor the data mart database. Files in this directory must be namedas follows:v product_code_mart_drop_schema.sqlv product_code_mart_drop_triggers.sqlv product_code_mart_schema.sqlv product_code_mart_triggers.sql

dml\ Contains the insert data scripts (DML) for star schema tablesintended for the data mart database. The file in this directory mustbe named as follows:v product_code_mart_data.sql

i18n Java resource bundle files for internationalization translated strings. Files inthis directory must be named as follows, where version represents theversion of the warehouse enablement pack; for example, 1.1.0:v [email protected] [email protected]

report Report metadata in comma-separated value (CSV) format for theapplication’s canned information that is specific to the report interface(RPI). Files in this directory must be named as follows:v product_code_rpi_export1.csvv product_code_rpi_export2.csvv product_code_rpi_export3.csvv product_code_rpi_export4.csv

doc The application’s installation, usage, and interface guide for TivoliEnterprise Data Warehouse (in PDF format).

etl ETL interchange files (*.tag) for the DB2 Data Warehouse Center. At leastone *.tag file is required, which must be named as follows:v product_code_dwc_data.tag

If multiple *.tag files are provided, use the following naming convention:v product_code_yourchoice_dwc_data.tag

where yourchoice represents whatever nomenclature you prefer. If you arealso providing *.inp files, the file names should correlate with the *.tag filenames.

This directory also contains the following subdirectory:

sql Contains SQL scripts that are defined in ETL processes to be runby the SQL execution engine. (See “SQL execution engine setup”on page 24.)

misc Custom installation after scripts and supporting pieces. Initial script mustbe named as follows:v product_code_after.sh

Chapter 8. Assembling the warehouse enablement pack 97

Note: An after script should perform relatively small tasks that cannot beperformed during the warehouse pack installation. Exercise cautionwhen using after scripts.

Required filesTable 26 lists each of the files required in the warehouse pack. The installationprogram requires that all of the files listed must be provided regardless of whetheryou are providing a central data warehouse ETL, a data mart ETL, or both.However, depending on which type of ETL you are providing, the file can beempty. A check mark (U) indicates that if you are providing that type of ETL, thefile must not be empty and must contain valid content.

Table 26. Files required by the Tivoli Enterprise Data Warehouse installation program

File nameCentral datawarehouse ETL

Data mart ETL

product_code_cdw_drop_schema.sql U

product_code_cdw_drop_triggers.sql U

product_code_cdw_schema.sql U

product_code_cdw_triggers.sql U

product_code_cdw_data.sql U

twh_cdw_data.sql (supplied with Tivoli EnterpriseData Warehouse)

U

product_code_mart_drop_schema.sql U

product_code_mart_drop_triggers.sql U

product_code_mart_schema.sql U

product_code_mart_triggers.sql U

product_code_mart_data.sql U

[email protected] U

[email protected] U

product_code_rpi_export1.csv U




product_code_dwc_data.tag U

product_code_yourchoice_dwc_data.tag U

product_code_after.sh


Appendix A. Central data warehouse data model

This section describes each of the tables that compose the central data warehousegeneric schema. The following details are included:v The physical datatype for the table column; for example, CHAR, VARCHAR,

INTEGER, FLOATv Whether a value must be entered for the table column or whether it can be

NULLv Whether the table column comprises (part of) the primary key (PK) for the tablev Whether the table column comprises (part of) a foreign key (FK) to another table

Note: The Tivoli Enterprise Data Warehouse installation program creates thecentral data warehouse tables. Applications insert data into these tablesusing either an ETL process or an insert script for common static data. Thefollowing tables are populated by ETL processes:v “Component” on page 107v “Component Attribute” on page 109v “Component Relationship” on page 110v “Measurement” on page 113v “Extract Control” on page 112v “Extract Log” on page 112v “Prune Measurement Log” on page 118

Figure 11 on page 100 shows the logical database design of the entire central datawarehouse database.

99

Figure 13 on page 102, Figure 14 on page 103, Figure 15 on page 104, Figure 16 onpage 105, and Figure 17 on page 105 show the physical database designs by subjectarea.

Figure 11. Central data warehouse data model


Figure 12. Component—Physical Database Design Diagram

Appendix A. Central data warehouse data model 101

Figure 13. Component Configuration—Physical Database Design Diagram


Figure 14. Component Measurement—Physical Database Design Diagram


Figure 15. Component Measurement Context—Physical Database Design Diagram


Figure 16. Geographic Area—Physical Database Design Diagram

Figure 17. Administration—Physical Database Design Diagram


Attribute DomainOne instance of a set of valid values for a Component Attribute Type. For example,AIX is one instance of a valid value for an OS attribute that can be associated witha host. It is only used when the set of valid values is predefined.

Domains are enabled or disabled using the start and end dates and times. Thesedates and times reflect the effective period of the latest enablement, but do nottrack history of enablement or disablement. This means that these dates and timesare only important when inserting new attributes, that is, existing attributes shouldnot be affected when a Domain is disabled.

Table 27. Attribute Domain (table AttrDom)

Column name Data type Nulloption

Is PK Is FK Attribute name and description

AttrDom_ID INTEGER NOTNULL

Yes No Attribute Domain ID

CompTyp_Cd CHAR(17) NOTNULL

No Yes Component Type Code

AttrTyp_Cd CHAR(17) NOTNULL

No Yes Attribute Type Code

AttrDom_Strt_DtTm TIMESTAMP NOTNULL

No No Attribute Domain Start DateTime

AttrDom_End_DtTm TIMESTAMP NOTNULL

No No Attribute Domain End DateTime

AttrDom_Val VARCHAR(254) NOTNULL

No No Attribute Domain Value

AttrDom_Ds VARCHAR(254) NULL No No Attribute Domain Description

Attribute RuleDefines which types of attributes can be tracked for which types of components.For example, OS can be tracked for host, and tablespace can be tracked for adatabase.

Rules are enabled or disabled using the start and end dates and times. These datesand times reflect the effective period of the latest enablement, but do not trackhistory of enablement or disablement. This means that these dates and times areonly important when inserting new attributes; that is, existing attributes should notbe affected when a Rule is disabled.

Table 28. Attribute Rule (table AttrRul)




Yes Yes Component Type Code


Yes Yes Attribute Type Code

AttrRul_Strt_DtTm TIMESTAMP NOTNULL

No No Attribute Rule Start DateTime

AttrRul_End_DtTm TIMESTAMP NOTNULL

No No Attribute Rule End DateTime


Table 28. Attribute Rule (table AttrRul) (continued)



AttrRul_Dom_Ind CHAR NOTNULL

No No Attribute Rule Domain Indicator

Attribute TypeDefines the types of characteristics or properties that can be captured about acomponent managed by an application. Examples are Manufacturer, Machine Type,Model, OS, OS Version, and Microcode Level.

Table 29. Attribute Type (table AttrTyp)




Yes No Attribute Type Code

AttrTyp_Nm VARCHAR(120) NOTNULL

No No Attribute Type Name

CenterA facility or location that delivers services to customers. For example,Tivoli-Austin, Tivoli-Raleigh

Table 30. Center (table Centr)



Centr_Cd CHAR(6) NOTNULL

Yes No Center Code

Centr_Parent_Cd CHAR(6) NULL No Yes Center Parent Code

GeoArea_Cd CHAR(6) NULL Yes Geographic Area Code

TmZon_ID INTEGER NOTNULL

No Yes Time Zone ID

Centr_Nm VARCHAR(120) NOTNULL

No No Center Name

ComponentA resource managed by an application, such as a particular server, switch, router,firewall, storage server, gateway, or hub. This also includes subcomponents such asCPUs, disks, adapters, ports, and so on.

Components are instances of component types. The CompTyp table (“ComponentType” on page 111) is populated statically and the Comp table is populated in realtime based on measurement data. For example:

CompTyp_Cd is 'HOST' and CompName is 'jdoe.dev.tivoli.com' in the Comptable.


Table 31. Component (table Comp)



Comp_ID INTEGER NOTNULL

Yes No Component ID

This is a sequence number and is generatedby a trigger when you insert a row. The indexis defined once on the table and DB2maintains that to access data in the tablefaster.


No Yes Component Type Code

This is the link to the CompTyp table.

Centr_Cd CHAR(6) NULL No Yes Center Code

The default is 'CDW'. You can create centersas a way of interpreting your data. If youwere a service provider, you might have acenter in New York and one in San Francisco.You would look up centers based on somedata; the typical example is host name.

Cust_ID INTEGER NULL No Yes Customer ID

The default is 0, which uses the default 'CDW'center. Use the Customer ID in combinationwith centers to filter data. For example, saythe NewYork center has Customers Ford andIBM.

Comp_Corr_ID INTEGER NULL Component Correlation ID

Comp_Corr_Id is a correlator field that isused by ETLs to move data into the centraldata warehouse. Typically, this field isreserved for central data warehouse ETLcorrelation purposes.

Comp_Nm VARCHAR(254) NOTNULL

No No Component Name

The Comp_Nm must be a displayable orreadable string. The name should uniquelyidentify the component within the context ofthe component’s parent.

For standard Component Types, theComp_Nm must follow the namingconvention for that CompTyp_Cd. Forexample, the Comp_Nm for all componentswith a CompTyp_Cd = ″IP_HOST″ must beset to the fully qualified host name.

A component name should not be acomposite name. For example, a componentshould not be named Router XYZ - Port 23.There should be unique components for bothRouter XYZ and for Port 23.


Table 31. Component (table Comp) (continued)



Comp_Corr_Val VARCHAR(254)

NULL No No Component Correlation Value

The Comp_Corr_Val is reserved for use by thecentral data warehouse ETLs. This columncan be used by central data warehouse ETLsto help in correlation when loading data intothe central data warehouse. Data mart ETLsshould not use this column.

Comp_Strt_DtTm TIMESTAMP NOTNULL

No No Component Start DateTime

The start date is when the component getsplaced into service (the first time that data isinserted).

Comp_End_DtTm TIMESTAMP NOTNULL

No No Component End DateTime

The end date is when the component isplaced out of service; for example, when a PCbecomes obsolete.

The Comp_End_DtTm should not be set forcomponents that use a standard componenttype. The standard component types areshared components and applications shouldnot set out of service dates for sharedcomponents

Comp_Ds VARCHAR(254) NULL No No Component Description

This is an optional description, which is adisplayable and readable string that can bedisplayed on printed reports; for example,'John Henry's primary PC'

Component AttributeDefines a characteristic or property that further describes a Component. The validattributes for a component are defined by its type, which is the main purpose ofthe Component Type. For example, a server can have an attribute to describe itsoperating system or version whereas a switch blade or port can have an attributeto describe its speed. (See “Attribute Type” on page 107 for examples.)

Table 32. Component Attribute (table CompAttr)



CompAttr_ID INTEGER NOTNULL

Yes No Component Attribute ID


Yes Yes Component ID


Yes Yes Attribute Type Code

This is a key to the AttrTyp table. Itdefines what type of attribute is beingcollected; for example, INNCPU for thenumber of processors on the box.


Table 32. Component Attribute (table CompAttr) (continued)



CompAttr_Strt_DtTm TIMESTAMP NOTNULL

Yes No Component Attribute Start DateTime

CompAttr_End_DtTm TIMESTAMP NOTNULL

No No Component Attribute End DateTime

CompAttr_Val VARCHAR(254) NOTNULL

No No Component Attribute Value

Component ChangeEnables ETL processes to detect that a component has an end date assigned to it. Atrigger is created that runs on updates to the Comp table. This trigger populatesthe Comp_Change table with Comp_ID from the Comp table plus two additionalcolumns: Change_ID and Change_DtTm. Change_ID is set by way of a sequenceand Change_DtTm is the time stamp of the update of the Comp row.

Table 33. Component Change (table Comp_Change)



Change_ID INTEGER NOTNULL

Yes No Change ID

Change_DtTm TIMESTAMP NOTNULL

No No Component Change DateTime. Representsthe timestamp of the update of the Comprow.


No Yes Component ID

Component RelationshipA connection, logical or physical, between two separate components such as aserver IP card and a switch IP port, a host FC port and a switch FC port.

Table 34. Component Relationship (table CompReln)



CompReln_ID INTEGER NOTNULL

Yes No Component Relationship ID

Comp_Source_ID INTEGER NULL No Yes Component Source ID

Comp_Target_ID INTEGER NOTNULL

No Yes Component Target ID

RelnTyp_Cd CHAR(6) NOTNULL

No Yes Relationship Type Code

CompReln_Strt_DtTm TIMESTAMP NOTNULL

No No Component Relationship Start DateTime

CompReln_End_DtTm TIMESTAMP NOTNULL

No No Component Relationship End DateTime


Component TypeDefines the types of components and subcomponents that are managed by anapplication. For example: Host, database, event.

Types are enabled or disabled using the start and end dates and times. These datesand times reflect the effective period of the latest enablement, but do not trackhistory of enablement or disablement. This means that these dates and times areonly important when inserting new components; that is, existing componentsshould not be affected when a Type is disabled.

Table 35. Component Type (table CompTyp)

Column name Data type Null option Is PK Is FK Attribute name and description

CompTyp_Cd CHAR(17) NOT NULL Yes No Component Type Code

This is a name that is referencedby a component to define its type.It will have values like 'IP_HOST','DATABASE', and so on.

CompTyp_Parent_Cd CHAR(17) NULL No Yes Component Type Parent Code

This allows you to have ahierarchy of types. If there is noparent, then this field is NULL.For a 'HOST' compTyp this fieldwould be NULL. For a 'TABLE'compTyp, this field might be'DATABASE'. The link tocomponent is the CompTyp_Cdfield.

If the CompTyp_Parent_Cd is notNULL, there must be a row in theComponent Type table for theparent.

CompTyp_Nm VARCHAR(120) NOT NULL No No Component Type Name

This is a longer description of theCompTyp_Cd. It is often the sameas CompTyp_Cd.

CompTyp_Strt_DtTm TIMESTAMP NOT NULL No No Component Type Start DateTime

When a component type iscreated, a start date is assigned(usually the current time stamp).The end date is set to 9999-99-99indicating it is a valid componenttype.

CompTyp_End_DtTm TIMESTAMP NOT NULL No No Component Type End DateTime

If a component type is no longervalid, update this row and set theend date to the date it was nolonger valid.


CustomerExamples: Lucent, Sears, Sedar, Air Canada, IBM Internal.

Table 36. Customer (table Cust)


Cust_ID INTEGER NOT NULL Yes No Customer ID

Cust_Parent_ID INTEGER NULL No Yes Customer Parent ID

GeoArea_Cd CHAR(6) NULL No Yes Geographic Area Code

TmZon_ID INTEGER NULL No Yes Time Zone ID

Cust_Acct_Cd CHAR(10) NULL No No Customer Account Code

Cust_Nm VARCHAR(120) NOT NULL No No Customer Name

Extract ControlControls each extract.

Table 37. Extract Control (table Extract_Control)


ExtCtl_Source VARCHAR(120) NOT NULL Yes No Extract Control Source

ExtCtl_Target VARCHAR(120) NOT NULL Yes No Extract Control Target

ExtCtl_From_RawSeq CHAR(10) NULL No No Extract Control From Raw Sequence

ExtCtl_To_RawSeq CHAR(10) NULL No No Extract Control To Raw Sequence

ExtCtl_From_IntSeq BIGINT NULL No No Extract Control From Integer Sequence

ExtCtl_To_IntSeq BIGINT NULL No No Extract Control To Integer Sequence

ExtCtl_From_DtTm TIMESTAMP NOT NULL No No Extract Control From Date and Time

ExtCtl_To_DtTm TIMESTAMP NOT NULL No No Extract Control To Date and Time

Extract LogUsed as a log for all extracts.

Table 38. Extract Log (table Extract_Log)


ExtLog_Source VARCHAR(120) NOT NULL Yes No Extract Log Source

ExtLog_Target VARCHAR(120) NOT NULL Yes No Extract Log Target

ExtLog_Done_DtTm TIMESTAMP NOT NULL Yes No Extract Log Complete Date andTime

ExtLog_From_RawSeq CHAR(10) NULL No No Extract Log From Sequence

ExtLog_To_RawSeq CHAR(10) NULL No No Extract Log To Sequence

ExtLog_From_IntSeq BIGINT NULL No No Extract Log From Sequence

ExtLog_To_IntSeq BIGINT NULL No No Extract Log To Sequence

ExtLog_From_DtTm TIMESTAMP NOT NULL No No Extract Log From Date and Time

ExtLog_To_DtTm TIMESTAMP NOT NULL No No Extract Log To Date and Time


Geographic AreaA geographic unit such as a country/region, state/province or city. Also includeslarger areas such as North America (NA), Europe, Middle East and Africa (EMEA),and so on, plus smaller divisions such as US West.

These can be actual geo-political units, or IBM organizational units.

Table 39. Geographic Area (table GeoArea)


GeoArea_Cd CHAR(6) NOT NULL Yes No Geographic Area Code

GeoArea_Parent_Cd CHAR(6) NULL No Yes Geographic Area Parent Code

GeoTyp_ID INTEGER NULL No Yes Geographic Type ID

TmZon_ID INTEGER NULL No Yes Time Zone ID

GeoArea_Nm VARCHAR(120) NOT NULL No No Geographic Area Name

Geographic TypeA standard type for Geographic Areas; for example, Region, Country,State/Province, City.

Table 40. Geographic Type (table GeoTyp)



GeoTyp_ID INTEGER NOTNULL

Yes No Geographic Type ID

GeoTyp_Nm VARCHAR(120) NOTNULL

No No Geographic Type Name

MeasurementA numeric observation regarding capacity, utilization, availability, performance,and so on, of a component. Each measurement is taken at a specific time, andwhen summarized, is represented as an average, maximum, or total value overthat period in time.

The Measurement Rule table defines which measurements are valid for whichcomponents.

Table 41. Measurement (table Msmt)



Msmt_ID BIGINT NOTNULL

Yes No Measurement ID


No Yes Component ID

This is the link to the Comp table. This isthe sequence number.

MsmtTyp_ID INTEGER NOTNULL

No Yes Measurement Type ID


Table 41. Measurement (table Msmt) (continued)



TmSum_Cd CHAR NOTNULL

No Yes Time Summary Code

Msmt_Strt_Dt DATE NOTNULL

No No Measurement Start Date

Msmt_Strt_Tm TIME NOTNULL

No No Measurement Start Time

Msmt_Min_Val FLOAT NULL No No Measurement Minimum Value

Msmt_Max_Val FLOAT NULL No No Measurement Maximum Value

Msmt_Avg_Val FLOAT NULL No No Measurement Average Value

Msmt_Tot_Val FLOAT NULL No No Measurement Total Value

This is the total. Most data will be eitheravg/min/max or total. It is very rare forboth to make sense. In the MGrpMbr table,you indicate what Measurements haveavg/min/max versus total.

Msmt_Smpl_Cnt INTEGER NULL No No Measurement Sample Count

Msmt_Smpl_Cnt should include successfuldata points only. If Response time data isbeing reported, then the response timeshould include the response time forsuccessful data points only.

Msmt_Err_Cnt INTEGER NULL No No Measurement Error Count

Msmt_Err_Cnt should be the number offailed and canceled data points.(Attempted data points = Msmt_Smpl_Cnt+ Msmt_Err_Cnt)

Measurement GroupMeasurements can be grouped in various ways. For example, they can be groupedinto categories such as:v Capacity/configuration - how components are configured; for example, size,

capacityv Availability - availability or uptime for a componentv Utilization - component consumption such as CPU, disk space, bandwidth, and

so onv Errors - error rates such as downtime, packet loss, collisions, and so onv Performance - how well the component is able to perform its function

They can also be grouped into monitoring collections; that is, measurements that acollected together as part of a particular monitoring activity.





MGrp_Cd CHAR(6) NOTNULL

Yes No Measurement Group Code

MGrpTyp_Cd CHAR(6) NOTNULL

Yes Yes Measurement Group Type Code

MGrp_Parent_Cd CHAR(6) NULL No Yes Measurement Group Parent Code

MGrp_Nm VARCHAR(120) NOTNULL

No No Measurement Group Name

Measurement Group MemberDefines a type of measurement within a particular grouping (Measurement Group).A Measurement Type can be a member of several Measurement Groups.

Table 43. Measurement Group Member (table MGrpMbr)



MGrp_Cd CHAR(6) NOTNULL

Yes Yes Measurement Group Code


Yes Yes Measurement Group Type Code


Yes Yes Measurement Type ID

Measurement Group TypeA high-level classification for a Measurement Group. Used to distinguish betweenMeasurement Groups set up for different reasons; for example, categorizationversus monitoring collection.





Yes No Measurement Group Type Code

MGrpTyp_Nm CHAR(120) NOTNULL

No No Measurement Group Type Name

Measurement RuleDefines which types of measurements are appropriate for which types ofcomponents. For example, server CPU utilization is tracked for server CPUs;however, it can also be a summarized measurement for a server.


Table 45. Measurement Rule (table MsmtRul)




Yes Yes Component Type Code


Yes Yes Measurement Type ID

Measurement SourceAn application that provides data into the Central Data Warehouse.




MSrc_Cd CHAR(6) NOTNULL

Yes No Measurement Source Code

For IBM and Tivoli software applications thismust match the product code. For moreinformation, see Chapter 5, “Namingconventions” on page 65.

Third-party applications use a three-characterproduct code, which contains at least onenumber.

MSrc_Parent_Cd CHAR(6) NULL No Yes Measurement Source Parent Code

If the MSrc_Parent_Cd is not NULL, theremust be a corresponding row in the MSrctable for that parent.

MSrc_Nm VARCHAR(120) NOTNULL

No No Measurement Source Name

Measurement TypeA type of measurement that can be collected for a component (based on itscomponent type). For example, CPU Utilization can be collected for a server CPU,whereas Bytes Transmitted or Received can be collected for a switch port.





Yes No Measurement Type ID

A sequence number that is created by atrigger when you insert a row of data.

MUnit_Cd CHAR(6) NOTNULL

No Yes Measurement Unit Code

MSrc_Cd CHAR(6) NOTNULL

No Yes Measurement Source Code

MsmtTyp_Nm VARCHAR(120) NOTNULL

No No Measurement Type Name

MsmtTyp_Ds VARCHAR(254) NULL No No Measurement Type Description


Measurement UnitThe unit of measure associated with a measurement; for example, KBs,milliseconds.




MUnit_Cd CHAR(6) NOTNULL

Yes No Measurement Unit Code

MUnitCat_Cd CHAR(6) NULL No Yes Measurement Unit Category Code

MUnit_Nm VARCHAR(120) NOTNULL

No No Measurement Unit Name

Measurement Unit CategoryA high level classification of measurements as follows:v Time Duration - the amount of time to performed an activityv Percentage - a ratio comparing one time or quantity against anotherv Quantity - a unit count associated with an activity; for example, bytes

transferredv Rate - a quantity over a fixed time period




MUnitCat_Cd CHAR(6) NOTNULL

Yes No Measurement Unit Category Code

MUnitCat_Nm VARCHAR(120) NOTNULL

No No Measurement Unit Category Name

Measurement Unit ConversionThe mathematical conversion between two compatible units of measure. Forexample, the conversion factor to convert from KB to MB is 1000.

Table 50. Measurement Unit Conversion (table MUnitCnv)



MUnit_Source_Cd CHAR(6) NOTNULL

Yes Yes Measurement Unit Source Code

MUnit_Target_Cd CHAR(6) NOTNULL

Yes Yes Measurement Unit Target Code

MUnitCnv_Factor FLOAT NOTNULL

No No Measurement Unit Conversion Factor


Measurement Unit SubunitA decomposition of a Measurement Unit into component units. Can be used toconvert measurements into corresponding measurement units for correlation. Forexample: Bps (Bytes per Second) breaks into Bytes and Seconds.

Table 51. Measurement Unit Subunit (table MUnitSub)



MUnit_Parent_Cd CHAR(6) NOTNULL

Yes Yes Measurement Unit Parent Code

MUnit_Sub_Cd CHAR(6) NOTNULL

Yes Yes Measurement Unit Subunit code

Prune Measurement ControlDefines when data is deleted from the Msmt table.

Table 52. (table Prune_Msmt_Control)

Column name Data type Null option Is PK Is FK Attribute name anddescription

MSrc_Cd CHAR(6) NOT NULL Yes Yes Measurement Source Code

TmSum_Cd CHAR NOT NULL Yes Yes Time Summary Code

PMsmtC_Age_In_Days DECIMAL(8, 0) NOT NULL No No Date duration yyyymmdd

Prune Measurement LogRecords when data was removed from the Msmt table.

Table 53. (table Prune_Msmt_Log)



PMsmtL_Run_DtTm TIMESTAMP NOTNULL

Yes No Timestamp of pruning action

TmSum_Cd CHAR NULL No Yes Time Summary Code

MSrc_Cd CHAR(6) NULL No Yes Msmt Source Code

PMsmtL_Age_In_Days DECIMAL(8,0)

NOTNULL

No No Date Duration

Relationship RuleDefines which types of components can participate in which types of relationships.For example, an IP (Internet Protocol) connection associates server networkinterface card to a switch IP port, whereas an FC (Fibre Channel) connection canassociate a host FC port to a switch FC port.

Rules are enabled and disabled using the start and end dates and times. Thesedates and times reflect the effective period of the latest enablement, but do nottrack history of enablement and disablement. This means that these dates andtimes are only important when inserting new relationships; that is, existingrelationships should not be affected when a Rule is disabled.


Table 54. Relationship Rule (table RelnRul)



CompTyp_Source_Cd CHAR(17) NOTNULL

Yes Yes Component Type Source Code

CompTyp_Target_Cd CHAR(17) NOTNULL

Yes Yes Component Type Target Code


Yes Yes Relationship Type Code

RelnRul_Strt_DtTm TIMESTAMP NOTNULL

Yes No Relationship Rule Start DateTime

RelnRul_End_DtTm TIMESTAMP NOTNULL

No No Relationship Rule End DateTime

Relationship TypeDefines the types of relationships that can be captured about two independentcomponents; for example, physical connections such as IP connections and FCconnections, plus logical relationships such as network dispatchers associated withWeb servers.

Table 55. Relationship Type (table RelnTyp)




Yes No Relationship Type Code

RelnTyp_Nm VARCHAR(120) NOTNULL

No No Relationship Type Name

Schema VersionVersion number of the central data warehouse schema.

Table 56. Schema Version (table Schema_Version)



Version_Cd CHAR(17) NOTNULL

Yes No Schema Version Code

Version_Nm VARCHAR(120) NOTNULL

No No Schema Version Name

Time SummaryThe period over which a measurement can be summarized; for example, hourly,daily, weekly, and so on.





TmSum_Cd CHAR NOTNULL

Yes No Time Summary Code

TmSum_Nm VARCHAR(120) NOTNULL

No No Time Summary Name

Time ZoneA geographical area of the world subscribing to the same time; for example,Eastern Standard Time (EST). Described by a (+/-) offset from Universal TimeCoordinated (UTC), or Greenwich Mean Time (GMT). This offset is captured as atime duration, hhmmss, where hh represents hours (00 to 12), mm representsminutes (00 or 30), and ss represents seconds (always 00 in this case). This can bepositive or negative; that is, it can be added or subtracted to convert between GMTand the designated time zone. A time zone can be related to a whole country orregion, a state or province or a group of cities. Note that there can be more thanone time zone name associated with the same GMT offset. For example, CentralTime (US & Canada) and Mexico City Time both have offset -06:00. Also, there areseveral time zones with half hour offsets; for example, Newfoundland (-03:30) andIndia (+05:30).

If there is no requirement to report the name of a time zone, then one entry perGMT offset would suffice. However, for customer enablement and reporting, atime zone name is probably required.

Table 58. Time Zone (table TmZon)



TmZon_ID INTEGER NOTNULL

Yes No Time Zone ID

TmZon_Nm VARCHAR(120) NOTNULL

No No Time Zone Name

TmZon_GMT_Offset DECIMAL(6,0) NOTNULL

No No Time Zone GMT Offset Duration

TmZon_CDW_Diff DECIMAL(6,0) NULL No No Time Zone CDW Difference

Time Zone InstallHelper table that has entries for the data when daylight saving time (DST) startsand ends for the central data warehouse. The installation uses this whendetermining the time zone of the central data warehouse.

Table 59. Time Zone Install (table TmZon_Install)



TInstall_Begin_DST DATE NOTNULL

No No Time Zone Install Begin DST Date

TInstall_End_DST DATE NOTNULL

No No Time Zone Install End DST Date


Translated TermContains the translated resources for all source applications.

Table 60. Translated Term (table Translated_Term)



Term_ID INTEGER NOTNULL

Yes No Term ID

Populated by an ascending database sequence.

Term_Key VARCHAR(254) NOTNULL

No No Term Key

Composed of: table.column.code_value, wherecode_value is the code column in the table, forexample, CompTyp_Cd.

Term_Locale VARCHAR(20) NOTNULL

No No Term Locale

Is the locale; for example, en_US, fr_FR, and soon.

Term_Value VARCHAR(768) NOTNULL

No No Term Value

Is the translated version of the value oftable.column referred to by Term_Key inUnicode format. As the column is binary (forbit data), no code page applies to the column.


Appendix B. Central data warehouse examples

Note to reviewersMinor changes will be made to this example to match up with current TivoliDistributed Monitoring ETL.

Typically, most of the systems management data falls into one of three categories:1. Availability

This category includes both status (normal, marginal, and so on) and state(initializing, starting, running, terminated, and so on) data. The Tivoli BusinessSystems Manager examples in the following tables show how to saveavailability type data. The Tivoli Business Systems Manager data is shown likethis, in underlined type.

2. Response timeThis category includes measurement data that has a time duration. Responsetime data can be probe based response time, or internal application-specificmeasurements. The Tivoli Application Performance Management examplesshow how to save response time data in the central data warehouse. The TivoliApplication Performance Management data is shown like this, in bold type.To ensure consistent response time reporting across all applications, thefollowing guidelines must be followed when saving response time data:v The response time measurements should only include measurements for

successful data points.v The sample count should (if available) be the number of successful data

points.v The error count (if available) should be the number of failed and aborted

data points.v The total number of attempted data points should = sample count + error

count.3. Threshold data (CPU utilization, packet loss, number of connected users, and

so on)Most of the data from Tivoli Distributed Monitoring, SNMP, and the IBM TivoliMonitoring for x products fall into this category. Threshold data is shown likethis, in bold underlined type.

Consider the following when modeling data after the examples in this section:v Some of the data in the following tables comes from more than one application.

For example, the IP_HOST component is shared between Tivoli ApplicationPerformance Management and Tivoli Distributed Monitoring.

v The examples are intended to show how to save the basic management data intothe central data warehouse. The examples are not intended to show all possiblemeasurement groups, attributes, centers, or measurement types.

Table 61. Component Type (table CompTyp)

CompTyp_Cd CompTyp_Parent_Cd CompTyp_Nm Comp_Typ_Strt_DtTm Comp_Typ_End_DtTm

TBSM_LOB NULL TBSM line of business 1/1/2002 12:00:00 PM '9999-01-01-00.00.0.000000'

IP_HOST NULL IP host name 1/2/2002 12:00:00 PM '9999-01-01-00.00.00.000000'

123

Tabl

e62

.C

ompo

nent

(tab

leC

omp)

Com

p_I

DC

omp

Typ

_Cd

Cen

tr_C

dC

ust

_ID

Com

p_C

orr_

IdC

omp

_Nm

Com

p_C

orr_

Val

Com

p_S

trt_

DtT

mC

omp

_En

d_D

tTm

Com

p_D

s

1T

BSM

_LO

BE

aste

rnSa

les

TB

SM_L

OB

_ID

’999

9-01

-01-

00.0

0.00

.000

000’

Dir

ect

Sale

sfo

rE

aste

rnU

SA

2T

BSM

_LO

BC

ICS

Eas

tern

TB

SM_L

OB

_ID

’999

9-01

-01-

00.0

0.00

.000

000’

TB

SML

OB

des

crip

tion

3T

BSM

_LO

BW

este

rnSa

les

TB

SM_L

OB

_ID

’999

9-01

-01-

00.0

0.00

.000

000’

Dir

ect

Sale

sfo

rW

este

rnU

SA

4T

BSM

_LO

BC

ICS

Wes

tern

TB

SM_L

OB

_ID

’999

9-01

-01-

00.0

0.00

.000

000’

5T

BSM

_LO

BSa

les

TB

SM_L

OB

_ID

’999

9-01

-01-

00.0

0.00

.000

000’

Dir

ect

Sale

sfo

rU

SA

6IP

_HO

STTe

rps.

mar

ylan

d.T

ivol

i.com

’999

9-01

-01-

00.0

0.00

.000

000’

7TA

PM

_TX

Dow

nlo

ad’9

999-

01-0

1-00

.00.

00.0

0000

0’

8TA

PM

_AP

PL

Net

scap

e’9

999-

01-0

1-00

.00.

00.0

0000

0’

9TA

PM

_US

ER

Ad

min

istr

ator

’999

9-01

-01-

00.0

0.00

.000

000’


Table 63. Component Relationship Type (table RelnTyp)

RelnTyp_Cd RelnTyp_Nm

PCHILD Parent – child relationship

Table 64. Component Relationship Rule (table RelnRul)

CompTyp_Source_Cd CompTyp_Target_Cd RelnTyp_Cd RelnRul_Strt_DtTm RelnRul_End_DtTm

TBSM_LOB TBSM_LOB PCHILD

TAPM_USER IP_HOST PCHILD

TAPM_APPL TAPM_USER PCHILD

TAPM_TX TAPM_APPL PCHILD

Table 65. Component Relationship (table CompReln)

CompReln_ID Comp_Source_ID Comp_Target_ID RelnTyp_Cd CompReln_Strt_DtTm CompReln_End_DtTm

1 5 1 PCHILD

2 5 3 PCHILD

3 1 2 PCHILD

4 3 4 PCHILD

5 7 6 PCHILD

6 9 8 PCHILD

7 8 7 PCHILD

8 6 9 PCHILD

Table 66. Attribute Type (table AttrTyp)

AttrTyp_Cd AttrTyp_Nm

TBSMNM TBSM component name

LAST_IP_ADDRESS Last known IP address

TME_OBJECT_LABEL TME endpoint object label

TX_DETAIL TAPM transaction detail

Table 67. Attribute Rule (table AttrRul)

CompTyp_Cd AttrTyp_Cd AttrRul_Strt_DtTm AttrRul_End_DtTm AttrRul_Dom_Ind

TBSM_LOB TBSMNM N

IP_HOST LAST_IP_ADDRESS N

IP_HOST TME_OBJECT_LABEL N

TAPM_TX TX_DETAIL N

Table 68. Component Attribute (table CompAttr)

CompAttr_ID Comp_ID AttrTyp_Cd CompAttr_Strt_DtTm CompAttr_End_DtTm CompAttr_Val

1 6 LAST_IP_ADDRESS ’9999-01-01-00.00.00.000000’

9.67.214.123

2 6 TME_OBJECT_LABEL ’9999-01-01-00.00.00.000000’

Cafaron_ep

3 7 TX_DETAIL ’9999-01-01-00.00.00.000000’

HTML Page download



STATE State

Appendix B. Central data warehouse examples 125

Table 69. Measurement Group Type (table MGrpTyp) (continued)


GROUP Group


MGrp_Cd MGrpTyp_Cd MGrp_Parent_Cd MGrp_Nm

LOBSTA STATE NULL Line of business status

TOT_E GROUP NULL Total value exists

MIN_E GROUP NULL Minimum value exists

MAX_E GROUP NULL Maximum value exists

AVG_E GROUP NULL Average value exists

Table 71. Measurement Group Member (table MGrpMbr)

MGrp_Cd MGrpTyp_Cd MsmtTyp_Id

LOBSTA STATE 1

LOBSTA STATE 2

LOBSTA STATE 3

LOBSTA STATE 4

AVG_E GROUP 1

AVG_E GROUP 2

AVG_E GROUP 3

AVG_E GROUP 4

MIN_E GROUP 5

MAX_E GROUP 5

AVG_E GROUP 5

MAX_E GROUP 6

MIN_E GROUP 6

AVG_E GROUP 6


MUnitCat_Cd MUnitCat_Nm

PRC Percentage

QTY Quantity

TM Time duration



PRC PRC Percentage

MB QTY Megabytes

MS TM Milliseconds


TmSum_Cd TmSum_Nm

H Hourly



MSrc_Cd MSrc_Parent_Cd MSrc_Nm

GTM Tivoli Business System Manager

Tivoli NULL Tivoli Applications

DMN Tivoli Distributed Monitoring

AFP Tivoli Application Performance Management


MsmtTyp_Id MUnit_Cd MSrc_Cd MsmtTyp_Nm MmtTyp_Ds

1 PRC GTM LOB state unknown Line of business stateunknown

2 PRC GTM LOB state green Line of business stategreen

3 PRC GTM LOB state yellow Line of business stateyellow

4 PRC GTM LOB state red Line of business statered

5 MB DMN Swap space Available swap space

6 TM APF Transaction responsetime

Transaction responsetime

Table 77. Measurement Rule (table MsmtRul)

CompTyp_Cd MsmtTyp_ID

TBSM_LOB 1

TBSM_LOB 2

TBSM_LOB 3

TBSM_LOB 4

IP_HOST 5

TAPM_TX 6

Appendix B. Central data warehouse examples 127

Tabl

e78

.M

easu

rem

ent

(tab

leM

smt)

Msm

t_ID

Com

p_I

DM

smtT

yp_I

DT

mS

um

_Cd

Msm

t_S

trt_

Dt

Msm

t_S

trt_

Tm

Msm

t_M

in_

Val

Msm

t_M

ax_

Val

Msm

t_A

vg_

Val

Msm

t_To

t_V

alM

smt_

Sam

pl_

Cn

tM

smt_

Err

_C

nt

15

2H

2001

/12

/25

15:0

0:00

100

25

2H

2001

/12

/25

16:0

0:00

50

35

1H

2001

/12

/25

16:0

0:00

50

45

2H

2001

/12

/25

17:0

0:00

100

51

2H

2001

/12

/25

15:0

0:00

50

61

3H

2001

/12

/25

15:0

0:00

50

71

2H

2001

/12

/25

16:0

0:00

100

81

2H

2001

/12

/25

17:0

0:00

100

93

4H

2001

/12

/25

15:0

0:00

100

103

3H

2001

/12

/25

16:0

0:00

25

113

4H

2001

/12

/25

16:0

0:00

75

123

2H

2001

/12

/25

17:0

0:00

100

133

5H

2001

/12/

2515

:00:

0023

4530

143

5H

2001

/12/

2516

:00:

005

118

153

5H

2001

/12/

2517

:00:

002

9814

167

6H

2001

/12/

2515

:00:

002

43

21

177

6H

2001

/12/

2517

:00:

004

86

295

0


Appendix C. Central data warehouse triggers

The section describes the triggers and sequences defined on the tables in thecentral data warehouse generic schema.create trigger TWG.compattr_end_stmtafter insert on TWG.compattrreferencing new as nfor each row mode db2sqlupdate

TWG.compattr cset

c.compattr_end_dttm = n.compattr_strt_dttmwhere

c.comp_id=n.comp_id andc.attrtyp_cd=n.attrtyp_cd andc.compattr_strt_dttm =

( selectmax(c2.compattr_strt_dttm)

from twg.compattr c2wherec2.comp_id=n.comp_id andc2.attrtyp_cd=n.attrtyp_cd andc2.compattr_strt_dttm < n.compattr_strt_dttm)

;

-- a view of current comp typescreate view TWG.cur_comptyp asselect *from TWG.comptypwhere current timestamp - current timezonebetween CompTyp_Strt_DtTm and CompTyp_End_DtTm;

-- a view of current compscreate view TWG.cur_comp asselect *from TWG.compwhere current timestamp - current timezonebetween Comp_Strt_DtTm and Comp_End_DtTm;

-- a view of current compattrscreate view TWG.cur_compattr asselect *from TWG.compattrwhere current timestamp - current timezonebetween CompAttr_Strt_DtTm and CompAttr_End_DtTm;

-- a view of current comprelncreate view TWG.cur_compreln asselect *from TWG.comprelnwhere current timestamp - current timezonebetween CompReln_Strt_DtTm and CompReln_End_DtTm;

CREATE SEQUENCE TWG.comp_id_seqSTART WITH 0INCREMENT BY 1NO MAXVALUENO CYCLE

129

CACHE 24;

CREATE TRIGGER TWG.COMP_ID_TRIGNO CASCADEBEFORE INSERT ON TWG.COMPREFERENCING NEW AS NFOR EACH ROWMODE DB2SQLSET N.COMP_ID = NEXTVAL for TWG.comp_id_seq;

CREATE SEQUENCE TWG.msmttyp_id_seqSTART WITH 1INCREMENT BY 1NO MAXVALUENO CYCLECACHE 24;

CREATE TRIGGER TWG.MSMTTYP_ID_TRIGNO CASCADEBEFORE INSERT ON TWG.MSMTTYPREFERENCING NEW AS NFOR EACH ROWMODE DB2SQLSET N.MSMTTYP_ID = NEXTVAL for TWG.msmttyp_id_seq;

CREATE SEQUENCE TWG.msmt_id_seqSTART WITH 1INCREMENT BY 1NO MAXVALUENO CYCLECACHE 24;

CREATE TRIGGER TWG.MSMT_ID_TRIGNO CASCADEBEFORE INSERT ON TWG.MSMTREFERENCING NEW AS NFOR EACH ROWMODE DB2SQLSET N.MSMT_ID = NEXTVAL for TWG.msmt_id_seq;

CREATE SEQUENCE TWG.compattr_id_seqSTART WITH 1INCREMENT BY 1NO MAXVALUENO CYCLECACHE 24;

CREATE TRIGGER TWG.COMPATTR_ID_TRIGNO CASCADEBEFORE INSERT ON TWG.COMPATTRREFERENCING NEW AS NFOR EACH ROWMODE DB2SQLSET N.COMPATTR_ID = NEXTVAL for TWG.compattr_id_seq;


CREATE SEQUENCE TWG.compreln_id_seqSTART WITH 1INCREMENT BY 1NO MAXVALUENO CYCLECACHE 24;

CREATE TRIGGER TWG.COMPRELN_ID_TRIGNO CASCADEBEFORE INSERT ON TWG.COMPRELNREFERENCING NEW AS NFOR EACH ROWMODE DB2SQLSET N.COMPRELN_ID = NEXTVAL for TWG.compreln_id_seq;

--Provide a default END DTTM if none was specifiedcreate trigger twg.Comp_Dttm_Trg

no cascadebefore insert on twg.Compreferencing new as nfor each rowmode db2sql

set n.Comp_End_Dttm =case

when n.Comp_End_Dttm is NULLthen ’9999-01-01-00.00.00.000000’

elsen.Comp_End_Dttm

end;

create trigger twg.CompAttr_Dttm_Trgno cascadebefore insert on twg.CompAttrreferencing new as nfor each rowmode db2sql

set n.CompAttr_End_Dttm =case

when n.CompAttr_End_Dttm is NULLthen ’9999-01-01-00.00.00.000000’

elsen.CompAttr_End_Dttm

end;

--Trigger to prune data from the Msmt table

create trigger twg.Prune_Msmt_Trgafter insert on twg.Prune_Msmt_Logreferencing new as nfor each rowmode db2sql

delete from twg.Msmt mwhere

(date(current timestamp - current timezone)

Appendix C. Central data warehouse triggers 131

- m.Msmt_Strt_Dt) > n.PMsmtL_Age_In_Days ANDm.TmSum_Cd = n.TmSum_Cd ANDn.MSrc_Cd = (select mtyp.MSrc_Cd from twg.MsmtTyp mtyp

where mtyp.MsmtTyp_ID = m.MsmtTyp_ID);

-- Detected End Dates

CREATE SEQUENCE TWG.comp_chg_id_seqSTART WITH 1INCREMENT BY 1NO MAXVALUENO CYCLECACHE 24;

CREATE TRIGGER TWG.CHANGE_ID_TRIGNO CASCADEBEFORE INSERT ON TWG.COMP_CHANGEREFERENCING NEW AS NFOR EACH ROWMODE DB2SQLSET N.CHANGE_ID = NEXTVAL for TWG.comp_chg_id_seq;

CREATE TRIGGER TWG.COMP_CHG_TRIGAFTER UPDATE ON TWG.COMPREFERENCING NEW AS NFOR EACH ROWMODE DB2SQLINSERT INTO TWG.COMP_CHANGE(Change_DtTm, Comp_ID)Values(current timestamp - current timezone, n.Comp_ID);

-- Extract Window Forward

create trigger twg.extract_ctrl_updafter insert on twg.extract_logreferencing new as nfor each row mode db2sqlupdate twg.extract_controlsetextctl_from_rawseq = n.extlog_to_rawseq ,extctl_from_intseq = n.extlog_to_intseq ,extctl_from_dttm = n.extlog_to_dttmwhereextctl_source = n.extlog_source andextctl_target = n.extlog_target;

CREATE SEQUENCE TWG.term_id_seqSTART WITH 1INCREMENT BY 1NO MAXVALUENO CYCLECACHE 24;

CREATE TRIGGER TWG.TERM_ID_TRIGNO CASCADEBEFORE INSERT ON TWG.TRANSLATED_TERMREFERENCING NEW AS NFOR EACH ROW


MODE DB2SQLSET N.TERM_ID = NEXTVAL for TWG.term_id_seq;

CREATE SEQUENCE TWG.cust_id_seqSTART WITH 1INCREMENT BY 1NO MAXVALUENO CYCLECACHE 24;

CREATE TRIGGER TWG.CUST_ID_TRIGNO CASCADEBEFORE INSERT ON TWG.CUSTREFERENCING NEW AS NFOR EACH ROWMODE DB2SQLSET N.CUST_ID = NEXTVAL for TWG.cust_id_seq;

Appendix C. Central data warehouse triggers 133

Appendix D. SQL execution engine data transfer support

The following tables detail the data transfer supported by the SQL executionengine.

Note: Data transfers between non-DB2 databases are not supported.

Table 79. Sybase to DB2 data transfer

Sybase

DB2

big int char forbit data

char timestamp decimal float int real smallint varcharfor bitdata

varchar

binary

char 3 3 3 3 3 3 3

datetime 3 2 3

decimal 1 3 1 1 1 1 3

float 1 3 1 1 1 1 3

int 1 3 1 1 1 1 3

real 1 3 1 1 1 1 3

smallint 1 3 1 1 1 1 3

varbinary

varchar 3 3 3 3 3 3 3

Unshaded cells indicate the action is supported if data is in compliance with vendor-specific ODBC driver restrictions.Shaded cells indicate the action is not supported.

Notes:

1. Significant digits/characters could be truncated

2. Potential loss of timestamp data (for example, milliseconds)

3. Supported if data is appropriate for destination column

Table 80. Oracle to DB2 data transfer

Oracle

DB2



varchar

number 1 3 1 1 1 1 1 3

char 3 3 3 3 3 3 3

date 3 2 3

decimal 1 3 1 1 1 1 3

float 1 3 1 1 1 1 3

int 1 3 1 1 1 1 3

real 1 3 1 1 1 1 3

smallint 1 3 1 1 1 1 3

raw

varchar2 3 3 3 3 3 3 3

135

Table 80. Oracle to DB2 data transfer (continued)


Notes:




Table 81. MSSQL to DB2 data transfer

MSSQL

DB2



varchar

binary

char 3 3 3 3 3 3 3

datetime 3 2 3

decimal 1 3 1 1 1 1 3

float 1 3 1 1 1 1 3

int 1 3 1 1 1 1 3

real 1 3 1 1 1 1 3

smallint 1 3 1 1 1 1 3

varbinary

varchar 3 3 3 3 3 3 3


Notes:




Table 82. Informix to DB2 data transfer

Informix

DB2



varchar

Int8 3 1 1 1 1 1 3

char 3 3 3 3 3 3 3

datetime 3 3

decimal 1 3 1 1 1 1 3

float 1 3 1 1 1 1 3

integer 1 3 1 1 1 1 3

smallfloat 1 3 1 1 1 1 3

smallint 1 3 1 1 1 1 3

byte

varchar 3 3 3 3 3 3 3


Table 82. Informix to DB2 data transfer (continued)


Notes:




Table 83. DB2 to DB2 data transfer

DB2

DB2

bigint char forbit data


varchar

big int 3 1 1 1 1 1 3

char forbit data

char 3 3 3 3 3 3 3

timestamp 3 3

decimal 1 3 1 1 1 1 3

float 1 3 1 1 1 1 3

int 1 3 1 1 1 1 3

real 1 3 1 1 1 1 3

smallint 1 3 1 1 1 1 3

varcharfor bitdata

varchar 3 3 3 3 3 3 3


Notes:




Appendix D. SQL execution engine data transfer support 137

Appendix E. Sample Perl script

The following example illustrates the type of functionality that can be implementedusing a Perl script called by a Data Warehouse Center External Step. Perl scriptscan be used with the SQL execution engine. For more information, see “Using theSQL execution engine” on page 22.if (! @ARGV) {

$log = ’evalhosttype.log’;if (! open(LOG, ">$log")) {

print "EvalHostType Error: Unable to open the specified log filefor writing:\n";

print "\n \"$log\".\n\n";print "$!\n";exit(1);

}print LOG "\nEvalHostType Error: a file name is required.\n\n";usage(1);

}

$log = $ARGV[0].’.log’;if (! open(LOG, ">$log")) {

print "\nEvalHostType Error: Unable to open the specified log filefor writing:\n";

print "\n \"$log\".\n\n";print "$!\n";exit(1);}

print LOG "Logfile $log opened.\n";

$hfile = $ARGV[0].’.preedit’;if (! open(HOSTS, "<$hfile")) {

print LOG "\nEvalHostType Error: unable to open the specified hostnamefile for reading:\n";

print LOG "\n \"$hfile\".\n\n";print LOG "$!\n";close(LOG);exit(1);

}

$out = $ARGV[0].’.postedit’;if (! open(OUT, ">$out")) {

print LOG "\nEvalHostType Error: Unable to open the specified outputfile for writing:\n";

print LOG "\n \"$out\".\n\n";print LOG "$!\n";close(LOG);exit(1);

}

while ($inline = <HOSTS>) {@inlist = parse_csv($inline);@outline = ();$outline[0] = $inlist[0];$outline[1] = $inlist[1];$outline[6] = $inlist[2];

if ($inlist[0] =~ m/^[a-zA-Z0-9]+([\w\-]*[a-zA-Z0-9]+)*\.([a-zA-Z0-9]+([\w\-]*[a-zA-Z0-9]+)*\.)+[a-zA-Z0-9]+([\w\-]*[a-zA-Z0-9]+)*$/ &&$inlist[0] =~ /[a-zA-Z_]+/){$outline[2] = "\"IP_HOST\"";$outline[3] = $inlist[0];

139

if ($inlist[1] =~ m/^([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])$/ && $inlist[1] ne "0.0.0.0")

{$outline[4] = $inlist[1];}

}elsif ($inlist[1] =~ m/^([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|

25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])$/ && $inlist[1] ne "0.0.0.0")

{$outline[2] = "\"IP_INTERFACE\"";$outline[3] = $inlist[1];$outline[5] = $inlist[0];}elsif ($inlist[0] =~ m/^([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|

25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])$/ && $inlist[0] ne "0.0.0.0")

{$outline[2] = "\"IP_INTERFACE\"";$outline[3] = $inlist[0];$outline[5] = $inlist[0];}else{$outline[2] = "\"TEC_HOST\"";$outline[3] = $inlist[0];}

print LOG "\nProcessing Input: \"$inlist[0]\",\"$inlist[1]\",\"$inlist[2]\"";print OUT "\"$outline[0]\",\"$outline[1]\",$outline[2],\"$outline[3]\",

\"$outline[4]\",\"$outline[5]\",\"$outline[6]\"\n";print LOG "\nOutput: \"$outline[0]\",\"$outline[1]\",$outline[2],\"$outline[3]\",

\"$outline[4]\",\"$outline[5]\",\"$outline[6]\"\n";}

close(HOSTS);close(OUT);close(LOG);

exit(0);

sub usage {$rc = @_;print LOG <<USAGE;

evalhosttype.pl <HOSTS_FILE>\nwhere HOSTS_FILE.preedit is a csv file with ’hostname’,’origin’,’unique_hndl’

USAGEclose(LOG);exit($rc);

}

sub parse_csv {my $text = shift;my @new = ();push(@new, $+) while $text =~ m{"([^\"\\]*(?:\\.[^\"\\]*)*)",?

| ([^,]+),?| ,

}gx;push(@new, undef) if substr($text, -1,1) eq ’,’;return @new;

}


Glossary

advanced report author. A Tivoli Enterprise DataWarehouse role that gives a user the ability to create,modify, run, display, and save both personal and publicreports. Contrast with report author. See also reportreader.

attribute. Data associated with a component. Forexample, a server component might have attributessuch as host name, IP address, operating system type,operating system version, number of CPUs, CPU speed,type of RAM, number of hard drives, network domain,and so on.

banner area. The area that is located below the titlebar and can be customized by a system administratorto include relevant information for a particularorganization. For example, in the banner area, anorganization might want to include the role descriptorfor the particular user, the company logo, and links toInternet and intranet sites.

base table. A table created with the CREATE TABLEstatement. Such a table has both its description anddata physically stored in the database. Contrast withview.

central data warehouse ETL. In Tivoli Enterprise DataWarehouse, the extract, transform, and load (ETL)process that reads the data from the operational datastores of the application that collects it (for example, alog file, a Tivoli Inventory repository, or a TivoliEnterprise Console database), verifies the data, makesthe data conform to the Tivoli Enterprise DataWarehouse schema, and places the data into the centraldata warehouse. See also extract, transform, and load(ETL). Contrast with data mart ETL.

central data warehouse server. The machine wherethe central data warehouse component of TivoliEnterprise Data Warehouse is installed.

central data warehouse. The component of TivoliEnterprise Data Warehouse that contains the cleansedhistorical data. Data in the central data warehouse isderived from operational data, although operationaldata is not stored directly in the central datawarehouse.

cleanse. The process of manipulating the dataextracted from operational systems so as to make itusable by the data warehouse.

component. An entity about which measurements arecollected for reporting purposes. Sample componentsinclude a specific network storage device; the Webaddress http://www.ibm.com; and a person withwhom you have a customer relationship. Each

component type in the data model has a set of metricsand attributes that apply to all components of thattype.

context menu. A menu that presents the actions thatare relevant for a particular item and that is shownonly when a user requests it.

consumer application. An application that uses thedata in the central data warehouse for a specificbusiness need. Consumer applications utilize reportingand third-party online analytical processing (OLAP)tools as well as planning, trend-tracking, analysis,accounting, and data mining tools. See also sourceapplication.

control database. The component of Tivoli EnterpriseData Warehouse that contains the metadata thatdescribes the data in the warehouse, including thesource of the data, how the data was transformedbefore being placed in the warehouse, when the datawas collected, and the formats used to publish the data(for example, the star schemas used to create TivoliEnterprise Data Warehouse data marts).

control server. (1) In Tivoli Enterprise DataWarehouse, the machine where the control databasecomponent of Tivoli Enterprise Data Warehouse isinstalled. (2) In DB2 replication, the database location ofthe applicable subscription definitions and Applyprogram control tables.

data definition language (DDL). A language fordescribing data and its relationships in a database.Synonym for data description language.

data description language. Synonym for data definitionlanguage (DDL).

data mart ETL. In Tivoli Enterprise Data Warehouse,the extract, transform, and load (ETL) process thatextracts a subset of data from the central datawarehouse, transforms it, and loads it into one or morestar schemas. These schemas then can be included indata marts to answer specific business questions. Seealso extract, transform, and load (ETL). Contrast withcentral data warehouse ETL.

data mart. A subset of a data warehouse that containsdata tailored and optimized for the specific reportingneeds of a department or team. A data mart can be asubset of a warehouse for an entire organization, suchas data contained in online analytical processing(OLAP) tools.

141

data mart server. A machine that hosts one or moredata marts. Multiple data mart servers can draw datafrom the same central data warehouse.

Data Warehouse Center. The component of DB2Universal Database that provides the graphicalinterface and the software behind it that enables you towork with the components of the warehouse. You canuse the Data Warehouse Center to define and managethe warehouse data and the processes that create thedata in the warehouse; to define the processes thatmove and transform data for the warehouse; and toschedule, maintain, and monitor these processes.

DDL. See data definition language (DDL)

ETL. See extract, transform, and load (ETL).

extract, transform, and load (ETL). The process ofcollecting data from one or more sources, cleansing andtransforming it, and then loading it into a database.

extreme case report. A Tivoli Enterprise DataWarehouse report that shows the instances of acomponent that have either the highest or the lowestvalues (but not both) for a specified metric. Typically,an extreme case report shows the best or worst n cases(where n is a number), such as the 10 servers havingthe most critical events. See also health check report andsummary report.

health check report. A Tivoli Enterprise DataWarehouse report that shows the values over time ofone or more metrics, which can be selected from one ormore star schemas, for one or more components.Typically, a health check report shows atime-delineated, diagnostic report showing thefluctuation of key indicators. See also extreme casereport and summary report.

IBM Console. The role-based user interface forperforming tasks using Tivoli management software.

metadata. Data that describes the characteristics ofstored data; descriptive data. For example, themetadata for a database table might include the nameof the table, the name of the database that contains thetable, the names of the columns in the table, and thecolumn descriptions, either in technical terms orbusiness terms.

metric. A measurement type. Each resource that canbe monitored for performance, availability, reliability,and other attributes has one or more metrics aboutwhich you collect data. Sample metrics include theamount of RAM on a PC, the number of help desk callsmade by a customer, the average CPU busy time for aserver, and mean time to failure for a hardware device.

OLAP. See online analytical processing (OLAP).

online analytical processing (OLAP). The process ofcollecting data from one or many sources; transforming

and analyzing the consolidated data quickly andinteractively; and examining the results across differentdimensions of the data by looking for patterns, trends,and exceptions within complex relationships of thatdata.

operational data store. The place where operationaldata resides, such as a database or a log file.

operational data. Data collected by an applicationduring its operation. An application can store itsoperational data in many formats, such as relationaldatabases, log files, and spreadsheet files. It is ″live″data, as opposed to the historical data in the centraldata warehouse. See also operational data store.

personal report. A Tivoli Warehouse report that isavailable only to its author. Contrast with public report.

portfolio. In the IBM Console, the primary way thatyour work is organized within the IBM Console. Theportfolio is a container for the tasks that apply to eachrole that you have been assigned. It is titled “MyWork.” When the portfolio is closed, it is indicated bythe down arrow on the portfolio title bar. When theportfolio is open, it displays within the IBM Console tothe left of the work area.

public report. A Tivoli Warehouse report that isavailable to all users with access to the data mart fromwhich the report extracts its data. Contrast withpersonal report.

RDBMS. See relational database management system

relational database. A database that can be perceivedas a set of tables and manipulated in accordance withthe relational model of data.

relational database management system (RDBMS). Acollection of hardware and software that organizes andprovides access to a relational database.

report author. A Tivoli Enterprise Data Warehouserole that gives a user the ability to run, display, andsave the output of both personal and public reportsand to create and modify personal reports. Contrastwith advanced report author. See also report reader.

report interface (RPI). The component of TivoliEnterprise Data Warehouse that provides historicalreporting capabilities. Using the report interface, userscan create and run reports; create and manage datamarts that have the format required by the TivoliEnterprise Data Warehouse report-generating tools; andcreate and manage the groups that control access tothose data marts.

report reader. A Tivoli Enterprise Data Warehouse rolethat gives a user the ability to run, display, and savethe output of public reports. See also advanced reportauthor and report author.


resource. A hardware or software entity that is trackedand managed by Tivoli management software

role. A job function, such as software distributor, thatidentifies the tasks that a user can perform and theresources to which a user has access. A user can beassigned one or more roles.

report server. The machine where the report interfacecomponent of Tivoli Enterprise Data Warehouse isinstalled. See also report interface (RPI).

RPI. See report interface (RPI).

source application. An application whose data iscollected from its operational data stores and placedinto the central data warehouse using an extract,transform, and load (ETL) process. See also consumerapplication.

star schema. A type of relational database schemamade up of a fact table and a set of dimension tables.In Tivoli Enterprise Data Warehouse, the fact tableholds the values of the component’s metrics, and thedimension tables hold the values of the attributes of acomponent or a metric.

summary report. A Tivoli Enterprise Data Warehousereport that shows the values of many metrics for manycomponents from only one star schema. Typically, asummary report examines large numbers of metrics,often showing subtotals for a particular grouping andgrand totals for the entire report. Data in a summaryreport is typically displayed as a text table, rather thanin a graphical format. See also extreme case report andhealth check report.

task. An activity that has business value, is initiatedby a user, and is performed by software.

taskbar. In the IBM Console, the bar that is locatedbelow the work area and contains a button for eachtask window. The taskbar also includes a button foreach of these actions:

v Resetting your Web browser if you have displayproblems

v Signing off of the IBM Console

Task Assistant. In the IBM Console, the place to gofor answers to your questions. The Task Assistant isrepresented by the question mark in the upper right ofthe IBM Console. When it is open, the Task Assistantdisplays within the IBM Console to the right of thework area.

The Task Assistant provides help for the tasks that youare performing at any given moment. This includeshelp for all Tivoli management software that isinstalled at your location.

For detailed information about the IBM Console, referto the Overview section within the Table of Contentsfor the Task Assistant.

task button. In the IBM Console, the button on thetaskbar that represents a task window. A task can havemultiple task windows. When you click a task button,the associated task window opens in the work area.Each task button also includes a small icon thatindicates the status of the task.

task driver. In Tivoli Presentation Services, thefunction that interacts with the appropriate Tivolimanagement software to perform a task. Also, if therespective task has a user interface, the task driverprovides that interface.

task group. In the IBM Console, a method fororganizing tasks into logical categories.

Tivoli Presentation Services. The Tivoli user interfacearchitecture. It is a distributed, device-independent, andplatform-independent presentation layer for all Tivoliproducts. The IBM Console is an integral part of thisarchitecture.

trigger. In a database, a set of Structured QueryLanguage (SQL) statements that automatically initatean action when a specific operation, such as changingdata in a table, occurs. A trigger consists of an event(an INSERT, DELETE, or UPDATE statement issuedagainst an associated table) and an action (the relatedprocedure). Triggers are used to preserve data integrityby checking on or changing data in a consistentmanner.

user. A person who uses Tivoli management softwareand is assigned one or more roles.

view. A logical table that consists of data that isgenerated by a query. A view is based on anunderlying set of base tables, and the data in a view isdetermined by a SELECT statement that is run on thebase tables. Contrast with base table.

warehouse agent. Software that manages the flow ofdata between one or more data sources and one ormore target warehouses. Warehouse agents use OpenDatabase Connectivity (ODBC) drivers or the DB2command line interface (CLI) to communicate withdifferent databases.

warehouse enablement pack. A separately-installablepart of a Tivoli software product that provide TivoliEnterprise Data Warehouse functionality by providingscripts; extract, transform, and load (ETL) programs tocollect data from the product’s operational data stores,place data in the central data warehouse, and extractsubset of the data to create data marts; and providecustomizable reports for the IBM Console to answerspecific busines questions.

warehouse pack. See warehouse enablement pack.

warehouse security administrator. A Tivoli EnterpriseData Warehouse role that gives a user the ability to

Glossary 143

control access to Tivoli Enterprise Data Warehouse databy creating and managing user groups and data marts.

work area. In the IBM Console, the area in which taskwindows are displayed. This area does not include theportfolio and the Task Assistant.


Index

Aabbreviations 10attribute type codes

naming conventions 68AVA code

See product codeavailability measurements 11

Bbooks

feedback vionline viordering vii

business intelligence reporting 71

Ccapitalization guidelines 68center, single 46centers

multiple 46, 51time zone 51using 51

central data warehousetables 99

central data warehouse databaselogical database design 99physical design by subject area 100

central data warehouse ETL 3central data warehouse schema

mapping data into 7overview 3

common component types and attributesIP_HOST 7IP_INTERFACE 7LAST_KNOWN_IPADDRESS 7

component attributesguidelines for creating 9

component dimension tables 54component relationship rules 8component type codes

naming conventions 68component types, rules 7components, rules 8composite names 8control database 65correlation of data 4customer data, distinguishing

uniqueness 47Customer Support viiicustomer, single 46customers, multiple 46

Ddata mart

definition 2RPI-ready 52, 81

data mart database 65data mart ETL 3data mart names

naming conventions 69data mining 71Data Warehouse Center

using 19data, correlation 4database names, Tivoli Enterprise Data

Warehouse 65dimension tables

based on components 67based on measurement types 67

Ee-mail contact viiiETL

See extract, transform, and load (ETL)exceptions, handling 52execsql.exe

See SQL execution engineexport metadata facility 21extract control 31extract, transform, and load (ETL)

guidelines 58overview 3

extreme case reports 71

Ffact tables

description of columns 54naming conventions 67

feedback about publications viii

Hhealth check reports 74host names 14

IIBM Console 71incremental extracts 31indexes

names 13internationalization 87IP addresses 9IP_HOST 7IP_INTERFACE 7

LLAST_KNOWN_IPADDRESS 7log files

Data Warehouse Center 42SQL execution engine 31

Mmanuals


measurement data, summarizing 11measurement groups

AVG_E 11MAX_E 11MIN_E 11standard 10STATE 11TOT_E 11

measurement type namesnaming conventions 69

measurement typespercentage in state 10rules 9

metadataexporting to a tag language file 21report 80

metric dimension tables 53metric names

naming conventions 69

Nnaming conventions

attribute type codes 68component type codes 68data in the central data warehouse

tables 68data mart names 69measurement type names 69metric names 69processes 66report names 69staging tables 68star schema names 69steps 66subject areas 65warehouse schemas 67warehouse sources 66warehouse targets 66

OODBC data sources 19, 23, 24, 25OLAP tools 1, 3, 71online publications viiordering publications vii

Pparent child relationships 15PCHILD relationship type 8performance

improving data mart queries 57tracking 1, 51, 113, 114

145

Perl scriptsample 139using with SQL execution engine 26

processesdefining 19naming conventions 66

product codeIBM and Tivoli software

applications 65third-party applications 65

pruning data 42publications


Rrelationship rules 8report

formatting 85scheduling 44

report interface (RPI)report formatting features 85tables for storing metadata 81

report namesnaming conventions 69

report scheduling 57reports

creating 71extreme case 71health check 74summary 78

resource bundlescompiling 89sharing 90

response time measurements 12response time, measurements 13rollup.sh script 44RPI

See report interface (RPI)RPI.SSUpdated table 44, 57, 84runReport.sh script 57

Sschemas

central data warehouse genericschema 3

star schema 52scripts, manually testing 28SQL execution engine

data transfer support 135debugging suggestions 31log files 31overview 23return codes 30setup 24using 22

SQL scriptcontrol statements 26manually testing 28running from the Data Warehouse

Center 20SQLScript 20, 23sqlscript.sh 20, 23, 24, 28, 29, 31

staging tables 13naming conventions 68

star schema namesnaming conventions 69

star schemascreating 52example 55

state measurements 11static data

inserting 15twh_cdw_data.sql 15

status measurements 11steps

defining 19naming conventions 66

subject areasdefining 19naming conventions 65

summarizing measurement data 11summary reports 78

Ttables

names 13time values 14Tivoli Customer Support viiiTivoli Decision Support

comparison with Tivoli EnterpriseData Warehouse 59

function mapping 59migrating function 59

trailing blanks 58translated terms 87triggers

central data warehouse tables 129creating 13names 13

TWH_CDW 65twh_cdw_data.sql 15TWH_MART 65TWH_MD 65

UUniversal Time Coordinated (UTC) 14user-defined programs 20

rollup 23runReport 23, 57SQLScript 23viewing in Data Warehouse

Center 23UTC

See Universal Time Coordinated(UTC)

utilization data, measurements 13

Wwarehouse enablement packs

creating 93files required 98layout 94

warehouse packsSee warehouse enablement packs

warehouse schemasnaming conventions 67

warehouse sourcesnaming conventions 66

warehouse targetsnaming conventions 66


Printed in the United States of Americaon recycled paper containing 10%recovered post-consumer fiber.

GC32-0745-00

enabling an application for tivoli enterprise data...

Documents