Download - CH02.ppt_2206

Chapter 2Chapter 2

Data WarehousingData Warehousing

Learning ObjectivesLearning Objectives Understand the basic definitions and concepts of Understand the basic definitions and concepts of

data warehousesdata warehouses Describe data warehouse architectures (high Describe data warehouse architectures (high

level). level). Describe the processes used in developing and Describe the processes used in developing and

managing data warehousesmanaging data warehouses Explain data warehousing operationsExplain data warehousing operations Explain the role of data warehouses in decision Explain the role of data warehouses in decision

supportsupport

Learning ObjectivesLearning Objectives Explain data integration and the extraction, Explain data integration and the extraction,

transformation, and load (ETL) processestransformation, and load (ETL) processes Describe real-time (active) data Describe real-time (active) data

warehousingwarehousing Understand data warehouse Understand data warehouse

administration and security issues administration and security issues

Continental Airlines CaseContinental Airlines Case 1.1. Describe the benefits of implementing the Continental Describe the benefits of implementing the Continental

Go Forward strategy.Go Forward strategy. 2. 2. Explain why it is important for an airline to use a real-Explain why it is important for an airline to use a real-

time data warehouse.time data warehouse. 3. Examine the sample system output screen at 3. Examine the sample system output screen at

http://www.teradata.com/td/page/139245. Describe how it . Describe how it can assist the user in identifying problems and can assist the user in identifying problems and opportunities.opportunities.

4. Identify the major differences between the traditional 4. Identify the major differences between the traditional data warehouse and a real-time data ware house, as was data warehouse and a real-time data ware house, as was implemented at Continental.implemented at Continental.

5. What strategic advantage can Continental derive from 5. What strategic advantage can Continental derive from the real-time system as opposed a traditional information the real-time system as opposed a traditional information system?system?

http://www.teradata.com/td/page/139245

Data Warehousing Data Warehousing Definitions and ConceptsDefinitions and Concepts

Data warehouseData warehouseA physical repository where relational data A physical repository where relational data are specially organized to provide are specially organized to provide enterprise-wide, cleansed data in a enterprise-wide, cleansed data in a standardized format standardized format


Characteristics of data warehousing Characteristics of data warehousing Subject oriented (sales, products, customers) Subject oriented (sales, products, customers) Integrated (consistent format)Integrated (consistent format) Time variant (time series)Time variant (time series) Nonvolatile (can’t change/update data)Nonvolatile (can’t change/update data) Web based Web based Relational/multidimensional structureRelational/multidimensional structure Client/server (for easy end user access)Client/server (for easy end user access) Real-time Real-time Include metadata (data about data) Include metadata (data about data)


Data martData mart A A departmentaldepartmental data warehouse that stores data warehouse that stores only relevant data only relevant data

Dependent data martDependent data mart A subset that is created directly from a data A subset that is created directly from a data warehouse warehouse

Independent data martIndependent data martA small data warehouse designed for a A small data warehouse designed for a strategic business unit or a department strategic business unit or a department


Operational data stores (ODS)Operational data stores (ODS)A type of database often used as an A type of database often used as an interim interim areaarea for a data warehouse, especially for for a data warehouse, especially for customer information files customer information files Updated during the course of business Updated during the course of business

operationsoperations Used for short term decisions Used for short term decisions


Enterprise data warehouse (EDWEnterprise data warehouse (EDW))A technology thatA technology that provides a vehicle for provides a vehicle for pushing data from source systems into a pushing data from source systems into a data warehouse data warehouse Large-scale, integration of data from many Large-scale, integration of data from many

sources, standard formalsources, standard formal

MetadataMetadata Data about data. In a data warehouse, Data about data. In a data warehouse,

metadata describe the contents of a data metadata describe the contents of a data warehouse and the manner of its use warehouse and the manner of its use Document data about data elements or attributes, Document data about data elements or attributes,

(name, size, data type, etc) and data about records or (name, size, data type, etc) and data about records or data structures (length, fields, columns, etc) and data data structures (length, fields, columns, etc) and data about data (where it is located, how it is associated, about data (where it is located, how it is associated, ownership, etc.).ownership, etc.).

May include descriptive information about the context, May include descriptive information about the context, quality and condition, or characteristics of the data.quality and condition, or characteristics of the data.

Data Warehousing Data Warehousing Process OverviewProcess Overview

Organizations continuously collect data, Organizations continuously collect data, information, and knowledge at an information, and knowledge at an increasingly accelerated rate and store increasingly accelerated rate and store them in computerized systemsthem in computerized systems

The number of users needing to access the The number of users needing to access the information continues to increase as a information continues to increase as a result of improved reliability and availability result of improved reliability and availability of network access, especially the Internet of network access, especially the Internet


The major components of a data The major components of a data warehousing process warehousing process Data sources (legacy systems, ERP, OTLP, Data sources (legacy systems, ERP, OTLP,

web logs)web logs) Data extraction Data extraction Data loading Data loading Comprehensive database (EDW)Comprehensive database (EDW) Metadata Metadata Middleware tools (enable access to the DW) Middleware tools (enable access to the DW)

Data Warehousing ArchitecturesData Warehousing Architectures Three parts of the data warehouseThree parts of the data warehouse

The data warehouse that contains the data and The data warehouse that contains the data and associated softwareassociated software

Data acquisition (back-end) software that Data acquisition (back-end) software that extracts data from legacy systems and external extracts data from legacy systems and external sources, consolidates and summarizes them, sources, consolidates and summarizes them, and loads them into the data warehouseand loads them into the data warehouse

Client (front-end) software that allows users to Client (front-end) software that allows users to access and analyze data from the warehouseaccess and analyze data from the warehouse

Data Warehousing ArchitecturesData Warehousing Architectures

Separates functions of the DW, eliminates resource constraints, can create data marts

Data Warehousing ArchitecturesData Warehousing ArchitecturesDSS engine runs on same platform as the DW, so more economical


Web server manages the inflow and outflow of information between client and servers. This approach offers ease of access, platform independence, and lower cost


1.1. Information Information interdependence interdependence between organizational between organizational unitsunits

2.2. Upper management’s Upper management’s information needsinformation needs

3.3. Urgency of need for a Urgency of need for a data warehousedata warehouse

4.4. Nature of end-user tasksNature of end-user tasks

5.5. Constraints on resources Constraints on resources 6.6. Strategic view of the data Strategic view of the data

warehouse prior to warehouse prior to implementationimplementation

7.7. Compatibility with existing Compatibility with existing systemssystems

8.8. Perceived ability of the in-Perceived ability of the in-house IT staffhouse IT staff

9.9. Technical issuesTechnical issues10.10. Social/political factorsSocial/political factors

Ten factors that potentially affect the architecture selection decision:

Data Integration and the Data Integration and the Extraction, Transformation, Extraction, Transformation, and Load (ETL) Processand Load (ETL) Process

Data integrationData integration Integration that comprises three major Integration that comprises three major processes: data access, data federation, processes: data access, data federation, and change capture. When these three and change capture. When these three processes are correctly implemented, data processes are correctly implemented, data can be accessed and made accessible to can be accessed and made accessible to an array of ETL and analysis tools and data an array of ETL and analysis tools and data warehousing environmentswarehousing environments Bank of America Ex. (App. Case 2.2)Bank of America Ex. (App. Case 2.2)


Extraction, transformation, and load (ETL)Extraction, transformation, and load (ETL)A data warehousing process that consists of A data warehousing process that consists of extraction (i.e., reading data from a extraction (i.e., reading data from a database), transformation (i.e., converting database), transformation (i.e., converting the extracted data from its previous form into the extracted data from its previous form into the form in which it needs to be so that it can the form in which it needs to be so that it can be placed into a data warehouse or simply be placed into a data warehouse or simply another database), and load (i.e., putting the another database), and load (i.e., putting the data into the data warehouse)data into the data warehouse)


Issues affect whether an organization will Issues affect whether an organization will purchase data transformation tools or build purchase data transformation tools or build the transformation process itself the transformation process itself Data transformation tools are expensiveData transformation tools are expensive Data transformation tools may have a long Data transformation tools may have a long

learning curvelearning curve It is difficult to measure how the IT organization It is difficult to measure how the IT organization

is doing until it has learned to use the data is doing until it has learned to use the data transformation tools transformation tools


Important criteria in selecting an ETL toolImportant criteria in selecting an ETL tool Ability to read from and write to an unlimited Ability to read from and write to an unlimited

number of data source architecturesnumber of data source architectures Automatic capturing and delivery of metadataAutomatic capturing and delivery of metadata A history of conforming to open standardsA history of conforming to open standards An easy-to-use interface for the developer and An easy-to-use interface for the developer and

the functional user the functional user

Data Warehouse DevelopmentData Warehouse Development Direct benefits of a data warehouseDirect benefits of a data warehouse

Allows end users to perform extensive analysis Allows end users to perform extensive analysis Allows a consolidated view of corporate data Allows a consolidated view of corporate data

(single version of the truth) (single version of the truth) Better and more timely informationBetter and more timely information Enhanced system performance Enhanced system performance Simplification of data access Simplification of data access

Data Warehouse DevelopmentData Warehouse Development

Indirect benefits result from end users using Indirect benefits result from end users using these direct benefits these direct benefits Enhance business knowledgeEnhance business knowledge Present competitive advantage (see Coke App. Present competitive advantage (see Coke App.

Case 2.3)Case 2.3) Enhance customer service and satisfactionEnhance customer service and satisfaction Facilitate decision makingFacilitate decision making Help in reforming business processesHelp in reforming business processes

Data Warehouse DevelopmentData Warehouse Development Data warehouse vendors (Table 2.1 list)Data warehouse vendors (Table 2.1 list)

Six guidelines to considered when developing a Six guidelines to considered when developing a vendor list:vendor list:1.1. Financial strengthFinancial strength2.2. ERP linkagesERP linkages3.3. Qualified consultantsQualified consultants4.4. Market shareMarket share5.5. Industry experienceIndustry experience6.6. Established partnerships Established partnerships

Data Warehouse DevelopmentData Warehouse Development Data warehouse development approachesData warehouse development approaches

Inmon Model: EDW approach Inmon Model: EDW approach • Top-down, relational DB toolsTop-down, relational DB tools

Kimball Model: Data mart approach Kimball Model: Data mart approach • Bottom-up, dimensional modeling; plan big, build Bottom-up, dimensional modeling; plan big, build

smallsmall Which model is best?Which model is best?

There is no one-size-fits-all strategy to data There is no one-size-fits-all strategy to data warehousing (Table 2.3 does a comparison)warehousing (Table 2.3 does a comparison)

Could outsource: hosted warehouseCould outsource: hosted warehouse HP consolidated DMs into EDW (App. 2.4)HP consolidated DMs into EDW (App. 2.4)

Data Warehouse DevelopmentData Warehouse Development Data warehouse structure: The Star Schema Data warehouse structure: The Star Schema

Dimensional modelingDimensional modeling A retrieval-based system that supports high-A retrieval-based system that supports high-volume query access volume query access

Dimension tablesDimension tablesA table that address A table that address howhow data will be analyzed data will be analyzed (exs.: location, time, and size are dimensions that (exs.: location, time, and size are dimensions that would support central fact rows)would support central fact rows)

Fast query-response time, Fast query-response time, simplicity, ease of simplicity, ease of maintenance for read-only DB structuresmaintenance for read-only DB structures

Star Schema Example

Data Warehouse DevelopmentData Warehouse Development Grain Grain

A definition of the highest level of detail that A definition of the highest level of detail that is supported in a data warehouse is supported in a data warehouse

Drill-downDrill-downThe process of probing beyond a The process of probing beyond a summarized value to investigate each of summarized value to investigate each of the detail transactions that comprise the the detail transactions that comprise the summary summary

Data Warehouse DevelopmentData Warehouse Development Data warehousing implementation issues Data warehousing implementation issues

Implementing a data warehouse is generally a Implementing a data warehouse is generally a massive effort that must be planned and massive effort that must be planned and executed according to established methodsexecuted according to established methods

There are many facets to the project lifecycle, There are many facets to the project lifecycle, and no single person can be an expert in each and no single person can be an expert in each area area


1.1. Establishment of Establishment of service-level service-level agreements and data-agreements and data-refresh requirementsrefresh requirements

2.2. Identification of data Identification of data sources and their sources and their governance policiesgovernance policies

3.3. Data quality planningData quality planning4.4. Data model designData model design5.5. ETL tool selectionETL tool selection

6.6. Relational database Relational database software and platform software and platform selectionselection

7.7. Data transportData transport8.8. Data conversionData conversion9.9. Reconciliation processReconciliation process10.10. Purge and archive Purge and archive

planningplanning11.11. End-user supportEnd-user support

Eleven major tasks that could be performed in parallel for successful implementation of a data warehouse (Solomon, 2005) :

Data Warehouse DevelopmentData Warehouse Development Some best practices for implementing a Some best practices for implementing a

data warehouse data warehouse (Weir, 2002):(Weir, 2002): Project must fit with corporate strategy and Project must fit with corporate strategy and

business objectivesbusiness objectives There must be complete buy-in to the project There must be complete buy-in to the project

by executives, managers, and usersby executives, managers, and users It is important to manage user expectations It is important to manage user expectations

about the completed projectabout the completed project The data warehouse must be built The data warehouse must be built

incrementallyincrementally Build in adaptability Build in adaptability

Data Warehouse DevelopmentData Warehouse Development Some best practices for implementing a Some best practices for implementing a

data warehouse data warehouse (Weir, 2002):(Weir, 2002): The project must be managed by both IT and The project must be managed by both IT and

business professionalsbusiness professionals Develop a business/supplier relationshipDevelop a business/supplier relationship Only load data that have been cleansed and Only load data that have been cleansed and

are of a quality understood by the organizationare of a quality understood by the organization Do not overlook training requirementsDo not overlook training requirements Be politically aware Be politically aware

Data Warehouse DevelopmentData Warehouse Development Failure factors in data warehouse projects:Failure factors in data warehouse projects:

Cultural issues being ignoredCultural issues being ignored Inappropriate architectureInappropriate architecture Unclear business objectivesUnclear business objectives Missing informationMissing information Unrealistic expectationsUnrealistic expectations Low levels of data summarizationLow levels of data summarization Low data quality Low data quality

Data Warehouse DevelopmentData Warehouse Development Issues to consider to build a successful Issues to consider to build a successful

data warehouse:data warehouse: Starting with the wrong sponsorship chainStarting with the wrong sponsorship chain Setting expectations that you cannot meet and Setting expectations that you cannot meet and

frustrating executives at the moment of truthfrustrating executives at the moment of truth Engaging in politically naive behaviorEngaging in politically naive behavior Loading the warehouse with information just Loading the warehouse with information just

because it is availablebecause it is available


data warehouse:data warehouse: Believing that data warehousing database Believing that data warehousing database

design is the same as transactional database design is the same as transactional database designdesign

Choosing a data warehouse manager who is Choosing a data warehouse manager who is technology oriented rather than user orientedtechnology oriented rather than user oriented

Focusing on traditional internal record-oriented Focusing on traditional internal record-oriented data and ignoring the value of external data data and ignoring the value of external data and of text, images, and, perhaps, sound and and of text, images, and, perhaps, sound and videovideo


data warehouse:data warehouse: Delivering data with overlapping and confusing Delivering data with overlapping and confusing

definitionsdefinitions Believing promises of performance, capacity, Believing promises of performance, capacity,

and scalabilityand scalability Believing that your problems are over when the Believing that your problems are over when the

data warehouse is up and runningdata warehouse is up and running Focusing on ad hoc data mining and periodic Focusing on ad hoc data mining and periodic

reporting instead of alertsreporting instead of alerts

Data Warehouse DevelopmentData Warehouse Development Implementation factors that can be Implementation factors that can be

categorized into three criteriacategorized into three criteria Organizational issuesOrganizational issues Project issuesProject issues Technical issues Technical issues

User participation in the development of User participation in the development of data and access modeling is a critical data and access modeling is a critical success factor in data warehouse success factor in data warehouse development development

Data Warehouse DevelopmentData Warehouse Development Massive data warehouses and scalability Massive data warehouses and scalability

The main issues pertaining to scalability:The main issues pertaining to scalability:• The amount of data in the warehouseThe amount of data in the warehouse• How quickly the warehouse is expected to growHow quickly the warehouse is expected to grow• The number of concurrent usersThe number of concurrent users• The complexity of user queries The complexity of user queries

Good scalability means that queries and other Good scalability means that queries and other data-access functions will grow linearly with the data-access functions will grow linearly with the size of the warehouse size of the warehouse

Real-Time Data WarehousingReal-Time Data Warehousing Real-time (active) data warehousing Real-time (active) data warehousing

(called RDW or ADW)(called RDW or ADW)The process of loading and providing data The process of loading and providing data via a data warehouse as they become via a data warehouse as they become available available

Supports operational decision-makingSupports operational decision-making

Real-Time Data WarehousingReal-Time Data Warehousing Levels of data warehouses:Levels of data warehouses:

1.1. Reports what happenedReports what happened2.2. Some analysis occursSome analysis occurs3.3. Provides prediction capabilities,Provides prediction capabilities,4.4. OperationalizationOperationalization5.5. Becomes capable of making events happenBecomes capable of making events happen

Real-Time Data WarehousingReal-Time Data Warehousing

Real-Time Data WarehousingReal-Time Data Warehousing The need for real-time dataThe need for real-time data

A business often cannot afford to wait a whole day for A business often cannot afford to wait a whole day for its operational data to load into the data warehouse for its operational data to load into the data warehouse for analysisanalysis

Provides incremental real-time data showing every state Provides incremental real-time data showing every state change and almost analogous patterns over timechange and almost analogous patterns over time

Maintaining metadata in sync is possibleMaintaining metadata in sync is possible Less costly to develop, maintain, and secure one huge Less costly to develop, maintain, and secure one huge

data warehouse so that data are centralized for BI/BA data warehouse so that data are centralized for BI/BA toolstools

An EAI (Enterprise Application Integration) with real-An EAI (Enterprise Application Integration) with real-time data collection can reduce or eliminate the nightly time data collection can reduce or eliminate the nightly batch processes batch processes

Data Warehouse Data Warehouse Administration and Security IssuesAdministration and Security Issues

Data warehouse administrator (DWA)Data warehouse administrator (DWA)A person responsible for the administration A person responsible for the administration and management of a data warehouse and management of a data warehouse

Data Warehouse Data Warehouse Administration and Security IssuesAdministration and Security Issues

Effective security in a data warehouse Effective security in a data warehouse should focus on four main areas:should focus on four main areas:

Establishing effective corporate and security Establishing effective corporate and security policies and procedurespolicies and procedures

Implementing logical security procedures and Implementing logical security procedures and techniques to restrict accesstechniques to restrict access

Limiting physical access to the data center Limiting physical access to the data center environmentenvironment

Establishing an effective internal control review Establishing an effective internal control review process with an emphasis on security and process with an emphasis on security and privacy privacy

Download - CH02.ppt_2206

Top Related