Theory and Application of Benchmarking in Business Surveys

Susie Fortier and Benoit QuennevilleStatistics Canada -TSRAC

ICES – June 2007

2

Content

Introduction NotationBenchmarking methodsTimeliness issue

Implied forecasts and annual growth ratesOther uses:

Seasonally adjusted dataLinking problem

Conclusions

3

Introduction

Main references

Dagum, E.B. and Cholette, P. (2006) Benchmarking, Temporal Distribution, and Reconciliation Methods for Time Series, New York: Springer-Verlag, Lecture Notes in Statistics 186.

Bloem, A. M., R. J. Dippelsman, and N. Ø. Mæhel (2001): Quarterly National Accounts Manual, Concepts, Data Sources and Compilation. International Monetary Fund, Washington DC.

4

Introduction

Benchmarking :

Combining a series of high-frequency data with a series of less frequent data into a consistent time series.

Monthly/Quarterly Annual

Explicit information about the short-term movement

Reliable information on the overall level and long-term movement

The “indicator” series The “benchmarks”

5

Introduction

Issues in Benchmarking :

Preserve period to period movement of the indicator (monthly/quarterly) series while simultaneously attaining the level of the benchmarks (annual).

Consider the timeliness of the benchmarks.

6

IntroductionExample of a quarterly series

7

IntroductionA quarterly series with its auxiliary source

8

IntroductionTimeliness issue

9

IntroductionBenchmarked series

10

Notation

Methodological details :

Indicator (monthly/quarterly) Benchmarks (annual)

DATA mySeries;

INPUT @01 year 4.

@06 period 1.

@08 value;

CARDS;

2000 1 1851

2000 2 2436

2000 3 3115

2000 4 2205

2001 1 1987

…

;

RUN;

DATA myBenchmarks;

INPUT @01 startYear 4.

@06 startPeriod 1.

@08 endYear 4.

@13 endPeriod 1.

@15 value;

CARDS;

2000 1 2000 4 10324

2001 1 2001 4 10200

…

;

RUN;

Ttss t ,,1),( Mmaa m ,,1),(

11

Notation

Methodological details :

With binding benchmarking, the benchmarked series is

such that

Ttt ,,1),ˆ(ˆ

Mmammt

t ,,1,ˆ

12

Notation

A bias parameter can be estimated and used to pre-adjust the indicator series:

A bias corrected series is obtained as:

m mt

m mtt

mm sa

b1

bss tt *

13

Notation

Alternatively, the bias can be expressed in terms of a ratio:

The bias corrected series is then:

bss tt *

m mtt

mm

s

ab

14

Notation

Bias correction is a preliminary adjustment to reduce, on average, the discrepancies between the two sources of data.

Useful for periods not covered by benchmarks.

15

Notation

Effect of the Bias Correction (ratio)

16

Methods : Pro-rating

A simple way to respect the constraints

is to use

This is the well-known formula for pro-rating.

mt for s

as

mt

mt

t

t

,ˆ*

*

Mmammt

t ,,1,

17

Methods : Pro-ratingBenchmarked series with pro-rating

18

Methods : Pro-ratingBI ratio with pro-rating

19

Methods : Pro-ratingGrowth rates with pro-rating

20

Methods : Pro-ratingGrowth rates with pro-rating

DATEIndicator

SeriesBenchmarked

SeriesGrowth Rate in

Indicator Series (%)Growth Rate in

Benchmarked Series (%)

2000-01

1851 1989.15 . .

2000-02

2436 2617.81 31.60 31.60

2000-03

3115 3347.48 27.87 27.87

2000-04

2205 2369.57 -29.21 -29.21

2001-01

1987 1945.42 -9.89 -17.90

2001-02

2635 2579.86 32.61 32.61

2001-03

3435 3363.12 30.36 30.36

2001-04

2361 2311.60 -31.27 -31.27

2002-01

2183 2059.05 -7.54 -10.93

21

Methods : Proportional DentonBenchmarked series with Prop. Denton

22

Methods : Proportional DentonBI ratio with Prop. Denton

23

Methods : Proportional DentonGrowth rates with Prop. Denton

24

Methods : Proportional DentonGrowth rates with Prop. Denton

DATEIndicator

SeriesBenchmarked

SeriesGrowth Rate in

Indicator Series (%)Growth Rate in

Benchmarked Series (%)

2000-01

1851 1989.15 . .

2000-02

2436 2617.81 31.60 30.86

2000-03

3115 3347.48 27.87 26.18

2000-04

2205 2369.57 -29.21 -30.86

2001-01

1987 1945.42 -9.89 -12.66

2001-02

2635 2579.86 32.61 29.04

2001-03

3435 3363.12 30.36 27.62

2001-04

2361 2311.60 -31.27 -32.14

2002-01

2183 2059.05 -7.54 -8.16

.

31.60

27.87

-29.21

-17.90

32.61

30.36

-31.27

-10.93

Pro-rating

25

Based on Dagum and Cholette (2006).

Generalization of many well-known methods:Pro-rating

Denton (and proportional Denton)

Implemented at Statistics Canada with a user-defined SAS procedure: PROC BENCHMARKING

Project ForillonForillon

Software Demo

Main method

26

Main method : Formula

The benchmarked series can be obtained as the solution of a minimization problem.

For given parameters and find the values that minimize the following function of :

subject to

Ttt ,,1,ˆ

T

tt

tt

t

tt

s

s

s

s

s

s

2

2

*1

1*

1

*

*2

*1

1*12 )1(

Mmammt

t ,,1,

27

*1*ˆ JsaVJVs de

Solution when :

Solution:

Main method : Formula

1

“Regression-based”model from

Dagum & Cholette

JJVV

CCV

sC

mtjjJ

ed

ee

Tjiji

e

t

tmtm

,...,1,

*

,,

)(diag

else 0

if1

28

Solution when :

Solution:

where W is the T x M upper-right corner matrix from :

Main method : Formula

W

WI

IJ

CC

J

JCC T

M 0

0

0

111111

**ˆ JsaWs

mult.) (Lagrange W

matrixIdentity 1,

MM

MMIM

TTji

1

whereelse0

1when 1

when 1

, ij

ij

ji

29

Main method : FormulaWe can obtain pro-rating with the general formula with and :minimise

under

gives

Mmammt

t ,,1,

T

tt

tt

t

tt

s

s

s

s

s

s

2

2

*1

1*

1

*

*2

*1

1*12 )1(

21 0

T

tt

tt

s

s

1

2

*

*

T

tt

tt

s

s

1

2

*

*

mts

as

mt

mt

t

t

for ,ˆ*

*

30

Main method : Effect of

Consider the case where and .

The function to be minimized under the constraints

which aims at preserving the period-to-period change in the original series.

Modified DentonModified Denton method

T

tt

tt

t

tt

s

s

s

s

s

sf

2

2

*1

1*

1

*

*2

*1

1*12 )1(

Mmammt t ,,1,

T

ttttt ssf

2

2

1*

1*

0 1

31

Main method : Effect of

Consider the case where and .

The function to be minimized under the constraints

which seeks to minimize the change in the ratios (not to preserve the growth rates but a fairly close

approx). Variant of Proportional Proportional DentonDenton method

T

tt

tt

t

tt

s

s

s

s

s

sf

2

2

*1

1*

1

*

*2

*1

1*12 )1(

Mmammt t ,,1,

T

t t

t

t

t

ssf

2

2

*1

1*

1 1

with positive data!

32

Main method : Effect of

3 parameters at play: : model adjustment parameter : “smoothing” parameter bias (implied with )

subject to

T

tt

tt

t

tt

s

s

s

s

s

s

2

2

*1

1*

1

*

*2

*1

1*12 )1(

Mmammt

t ,,1,

*ts

33

Main method : Effect of

Illustration of the effect of the rho parameter(BI ratios)

0.80

0.85

0.90

0.95

1.00

1.05

1.10

1.15

2000 2001 2002 2003 2004 2005 2006 2007

Rho=0 Rho=0.2 3̂ Rho=0.4 3̂ Rho=0.6 3̂ Rho=0.8 3̂

Rho=0.9 3̂ Rho=0.99 3̂ Rho=1 Bias=0.964

34

Main method : Effect of bias

Benchmarking without bias ( )39.0,1

35

Main method : Effect of bias

Benchmarking with bias ( )39.0,1

36

Main method : Effect of bias

Benchmarking without bias ( )39.0,1

37

Main method : Effect of bias

Benchmarking with bias ( )39.0,1

38

Timeliness issues

Adjustments for periods without benchmarks:

Benchmarked series give an implicit forecast for the unknown annual values.

The better the forecast, the lesser the revision!

Proportional Denton (ρ=1, λ=1) Benchmarking with bias (ρ=0.93, λ=1)

39

Timeliness issues

2 implicit forecasts for 2006:

Enhanced benchmarking method with explicitexplicit forecasts

Year Benchmark Indicator Benchmarked

(bias)

Benchmarked

(prop Denton)

2004 11,582

4.37%

11,891

1.98%

11,582

4.37%

11,582

4.37%

2005 11,092

-4.23%

12,399

4.27%

11,092

-4.23%

11,092

-4.23%

2006 n/a

n/a

12,196

-1.64%

11,352

2.35%

10,689

-3.64%

40

Timeliness issues

One possibility for explicit forecast:Use the annual growth rate from the indicator series on the last known benchmark.

Year Benchmark Indicator Benchmarked

(bias)

Benchmarked

(prop Denton)

2004 11,582

4.37%

11,891

1.98%

11,582

4.37%

11,582

4.37%

2005 11,092

-4.23%

12,399

4.27%

11,092

-4.23%

11,092

-4.23%

2006 10,910

-1.64%

12,196

-1.64%

11,352

2.35%

10,689

-3.64%

41

Timeliness issues

With explicit forecast ( )1,1

42

Timeliness issues

With explicit forecast ( )1,1

43

Timeliness issues

With ″recent″ bias( , bias=0.94)39.0,1

44

Timeliness issues

With ″recent″ bias( , bias=0.94)39.0,1

45

Timeliness issues

Minimize revision?

Bias Explicit forecast

(based on indicator)

Will change annual growth rate of indicator series

Preserve annual growth rate of indicator when nothing else is available

Could be ″infected″ with non-representative historical data

Annual discrepancies based only on one year

46

Methods : Summary so far!

Summary of methods presented:Pro-rating

Denton (and proportional Denton)

Regression-based (Dagum and Cholette)with or without bias correction

Denton with explicit forecast

Results from all of the above can be obtained by PROC BENCHMARKING.

47

Methods

Other methodsOther numerical methods revolve around different minimisation functions.

Statistical model-based approaches

See annex 6.1 in Bloem, Dippelsman, and Mæhel (2001) for variants and references

See also Chen and Wu (2006) for link between numerical, regression based and signal extraction methods.

Future version of PROC benchmarking?

48

Syntax : PROC Benchmarking

PLEASE SEE SOFTWARE DEMO !!PLEASE SEE SOFTWARE DEMO !!

PROC BENCHMARKING

BENCHMARKS=myBenchmarks

SERIES=mySeries

OUTBENCHMARKS=outBenchmarks

OUTSERIES=outSeries

OUTGRAPHTABLE=outGraph

RHO=0.729 LAMBDA=1 BIASOPTION=3;

RUN;

49

In SAS Enterprise Guide®(Demo)

50

Other uses : Seasonal adjustment

Seasonally adjusted series can be required to ″match″ given annual totals :

System of National Accounts (typical cases)

X-12-ARIMA version 0.3+FORCE spec (table D11 A)With argument Type=regress : same methodology as PROC BENCHMARKING

51

Other uses : Seasonal adjustment

X-12-ARIMA V0.3

Bias parameter option is replaced with argument target, which specifies which series is used as the target for forcing the totals of the seasonally adjusted series. The choices are:

Original

Caladjust (Calendar adjusted series)

Permprioradj (Original series adjusted for permanent prior adjustment factors)

Both (Original series adjusted for calendar and permanent prior adjustment factors)

52

X-12-ARIMA V0.3

By default, the FORCE spec implies that the calendar year totals in the SA = calendar year totals of the target series.

Alternative starting period for the annual total can be specified with start argument.

Annual total starting at any other period other than start may not be equal.

Other uses : Seasonal adjustment

53

Other uses : Seasonal adjustment

X-12-ARIMA V0.3 : example specseries{… save = A18}

transform{function=log}

regression{ variables=(TD easter[8])}

outlier{ …}

arima{…}

forecast{…}

x11{… save = D11}

force{ type=regress

lambda=1

rho=0.9

target=calendaradj

save=SAA }

54

Other uses : Seasonal adjustment

Canadian Department Stores Sales SA (D11) and SA with forced annual totals (D11 A)

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Jan-

91

Jan-

92

Jan-

93

Jan-

94

Jan-

95

Jan-

96

Jan-

97

Jan-

98

Jan-

99

Jan-

00

Jan-

01

Jan-

02

Jan-

03

Jan-

04

Mill

ions

D11 D11A (λ=1,ρ=0.9)

55

Other uses : Seasonal adjustment

Canadian Department Stores Sales

Differences between D11 and D11A

-15,000

-10,000

-5,000

0

5,000

10,000

Jan-

91

Jan-

92

Jan-

93

Jan-

94

Jan-

95

Jan-

96

Jan-

97

Jan-

98

Jan-

99

Jan-

00

Jan-

01

Jan-

02

Jan-

03

Jan-

04

D11-D11A (λ=1,ρ=0.9)

56

Other uses : Seasonal adjustment

Canadian Department Stores Sales

Growth rates

-2.0%

-1.0%

0.0%

1.0%

2.0%

3.0%

4.0%

Jan- 95 A

pr-

95 Jul-

95

Oct

-95 Ja

n- 96 Apr

-96 Ju

l-96

Oct

-96

Growth rate in D11 Growth rate in D11A

57

Other uses : Seasonal adjustment

Annual total starting at any other period other than start may not be equal.

Differences between the sum of 12 consecutive months computed on D11A and on A18

-40-20

020406080

100120140160

Jan-

91

Jan-

92

Jan-

93

Jan-

94

Jan-

95

Jan-

96

Jan-

97

Jan-

98

Jan-

99

Jan-

00

Jan-

01

Jan-

02

Th

ou

san

ds

SumD11A-SumA18

58

Other uses : Linking (bridging)

Linking segments of time series with different levels or ranges.

Used to minimize breaks caused by survey redesign, reclassification, change in concept…

Challenges:Estimation of the potential break (parallel run, forecasting, backcasting, …)

Preserve data coherence.

59

Other uses : Linking (bridging)

Can usually be achieved with PROC BENCHMARKING:

If the two segments overlap (if not, use a model to extend one of the two segments)

With proper identification of “anchor” points as benchmarks

The smoothing parameter can gradually “bridge” the gap between the two levels

60

Other uses : Linking (bridging)

Two segments of a series

61

Other uses : Linking (bridging)

Adjusted as a level shift (λ=1, ρ=0.9, bias)

62

Other uses : Linking (bridging)

Adjusted as a level shift (λ=1, ρ=0.9, bias)

63

Other uses : Linking (bridging)

Adjusted as a gradual level shift (λ=1, ρ=0.9, no bias)

64

Other uses : Linking (bridging)

BI ratio for a gradual level shift (λ=1, ρ=0.9, no bias)

65

Conclusions

Summary :Many numerical methods can be achieved through PROC BENCHMARKINGDifferent uses of benchmarking

Future developments in PROC BENCHMARKINGSimplify the use of explicit forecastsImprove bias estimationEnhance batch processing (VAR and BY statements)Include more options provided in Dagum and Cholette (2006): more generalised autocorrelation structure of the residuals, measurement errors in the input series, variance estimation of the results.

66

For more information please contact

Pour plus d’information veuillez contacter

Susie.Fortier@Statcan.ca

Benoit.Quenneville@Statcan.ca