monitoring of aggregation levels in distributed component based data production systems

27. February 2003GfK Group Monitoring of Aggregation Levels. Anja SchanzenbergerNon-Food Tracking

Monitoring of Aggregation Levels Monitoring of Aggregation Levels in Distributed Component Based in Distributed Component Based Data Production SystemsData Production Systems

BTW 2003, Leipzig, 27.02.2003

Anja SchanzenbergerGfK Marketing Services, NürnbergUniversity of Middlesex, London

Colin Tully, Dave LawrenceUniversity of Middlesex, London


2 Application

3 Monitoring of Aggregation Levels

1 Application Area

Agenda

The General Business of GfK Marketing Services The Basic Idea of Data Production System

The Planning, Controlling and Monitoring System

Single Record Tracking The Tubing System Reconstructing Aggregation Levels


Application Area1

4


The GfK Group: Key Features

Total revenue

Anticipated EUR 568 million in 2002; previous year: EUR 506 million

Increase on the previous year: +12%

Employees More than 4,800 full-time staff 70% of which abroad

Network

Over 130 subsidiaries, branches and participations in 50 countries on five continents

Services

Integrated systems using standardised instruments throughout Europe and beyond

5


Four Complementary Business Divisions

Consumer Tracking

Consumer and retail panel based Business Information Solutions for manufacturers and retailers for consumer packaged goods and service companies

In interview and panel based audience and readership measurement and consumer response testing for TV, print, radio and Internet

Media

Non-Food Tracking

Retail panelbased marketing

information for manufacturers and

retailers in consumer technology industries

Interview and test market based support

information for new product development

and brand management across a

wide range of industries

Ad Hoc Research

6


GfK Business Divisions

Consumer Tracking16.4%

12.1%Media

Non-Food Tracking23.6%

41.5%Ad Hoc Research

Other6.4%

Share of total

performance

7


Non-Food Tracking: Key Services

Information services in 44 countries on marketing, sales, logistics in retail and industry for companies operating in consumer technology markets.

Direct access to databases and/or transmission of standardized analyses to support, monitor and manage short, medium and long term decisions on product and pricing policy, advertising, distribution, sales and logistics.

Key services

The advantage for clients

Consumer Tracking Media

Ad Hoc Research

Non-Food Tracking

Non-Food Tracking

Retail panel

periodical monitoring

periodical monitoring

Market leader in the regions Europe and Asia and Pacific as well as in the Arab countries; together with partner NPD Intelect, market leader in North America.

Positioning

Information services on consumer durables, in particular for the consumer electronics, photographic, information technology, telecommunications, software, domestic appliances and equipment markets


Application2

9


StarTrack Working Areas

Data Warehouse(Extrapolation, Reports)

Data - IN Data - Preparation

DWH

Retailers Clients

Creating value through knowledge

IDAS

MDM

10


Data Production System

General InterfaceManager (GIM)

Data receipt

Separation

Identification(WebTAS)

Central IDASoutput pool

Local Output

Planning – Controlling – Monitoring System

Local client

Local server Central server

DWH Projectionsystem

Mainframe

11


PCMS Dimensions

current state • predefined process steps

• manual state checking

• manual error tracking

envisioned state

• dynamic production process configuration

• production planning and monitoring

• proactive error handling

Data Data Production Production

SystemSystem

PLANNING

MONITORING CONTROLING


Monitoring of Aggregation Levels3

13


Definitions

aggregation

separation

(disaggregation)

aggregation levels

input

many data sets

output

many data sets

output

one data set

input

one data set

instruction

aggregate functions

multiple groupings

14


Aspects to the Monitoring of Aggregation Levels

Summaries after significant process steps summaries of operating figures

Single Record Tracking tracking of single retailer items up to the

customer report simulation of planned production cycles

(ETL-Tools)

15


Example - Single Record Tracking

component X

Item AR: Vobis – DP: CW 04/2002-sales volume: 6Item BR: Vobis – DP: CW 04/2002-sales volume: 9

Item AR: Vobis – DP: CW 05/2002-sales volume: 4

Item AR: Vobis – RP: Jan 2002-sales volume: 10

Item BR: Vobis – RP: CW 04/2002-sales volume: 9

R: retailer CW: calendar weekDP: delivery periodRP: reporting period

pool A pool B

16


Strategies of Tracing Aggregation Levels

Tubing SystemTubing System

the complete workflow cycle error situations

17


Characteristics of Monitoring

TYPE CHARACTERISTIC

amount of data static / non static

aggregation policies known / unknown

job parameters known / unknown

storage requirements

minimal to maximal

18


Possibilities to reconstruct Aggregation Levels (1)

component X

Static Volumes of Data

-all items

-all retailers

-all delivery

periods

instruction:

SELECT...WHEREDP1=CW 6, DP2=CW 7DP3=CW 8, DP4=CW 9GROUP BY Vobis, item

pool A pool B

item,retailer: Vobisreporting period: Feb/2002

DP: delivery periodCW: calendar week

job parameters:itemretailerreporting period

19


(1) Static Volumes of Data

Advantages no additional storage required historically stored data allows stepwise tracking

possibilities

Disadvantages historically stored requires increased storage

facilities this approach is only significant for a small (historical

stored) quantity of data all job parameters are required increasing the quantity of data in storage slows

down the control system as well as the controlled system

requires additional administration effort

20


Possibility (2)

component X

-all items

-all retailers

-all delivery periods

pool A pool B

item,retailer: Vobisreporting period: Feb/2002

job parameters:job_iditemretailerreporting period

Single Record Logging

log:

timestampjob parameters records of A: -item -retailer -delivery period -facts -price

21


(2) Single Record Logging

Advantages no policies needed no static volumes of data

Disadvantages additional job parameters are needed at least twice the storage requirement additional administration effort slowdown of systems

22


Possibility (3)

Advantages storage requirement (approach 3) <

storage requirement (approach 2) no policies needed no static data volumes

Disadvantages no deleting of records, but new attribute values

for the same records additional administration effort slowdown of systems

Primary Key Logging most important attributes job parameters needed logging: item, retailer, delivery period reduction at GfK: ~1/5

23


Possibility (4)

component X

Data Evaluation

pool A1 pool B1

pool A2 pool B2

processing time

tracking time

all

all

item,retailer: Vobisreporting-period: Feb/2002

item,retailer: Vobisreporting-period: Feb/2002

instruction

instruction

job parameters:itemretailerreporting period

24


(4) Data Evaluation

Advantages no additional logging no additional storage required alterations of records allowed no static data volumes

Disadvantages policies are needed program extension job parameters are needed only an imprecise estimate

(processing time <> tracking time) double execution time of component

25


Conclusion (I)

1. Static Volumes of Data environments: (historical) static data volumes least logging effort best approach, but often not applicable

2. Single Record Logging environments: min. 2*storage required and slowdown

acceptable suitable when gathered amount of data >> processed

amount of data (e.g. ad-hoc reports)

3. Primary Key Logging environments: less manipulations acceptable deleting of records is not allowed additional logging effort

26


Conclusion (II)

… more [email protected]

4. Data Evaluation environments: level of impreciseness acceptable no additional logging effort no additional implementation work no additional storage required system load increases -> recommended for slack times

monitoring of aggregation levels in distributed component based data production systems

Documents