Model Smoothing - Evaluation

Supporting Document: DESC_Model Smoothing Review09_101109.pdf

DESC 10th November 2009

Model Smoothing Model Smoothing -- EvaluationEvaluation

Supporting Document: Supporting Document: DESC_ModelDESC_Model Smoothing Review09_101109.pdfSmoothing Review09_101109.pdf

DESC 10DESC 10thth November 2009November 2009

Model Smoothing

� Model smoothing: Averaging of 3 years models (including the current /

most recent data sets) to derive new parameters

� First undertaken 1999/00 and applied to all subsequent years based on similar methodology

� This current analysis is the first full assessment of model smoothing results since Autumn 2007 and has been carried out along the same lines

� Inform decisions on approach and application of model smoothing for

Spring 2010

� Supporting document gives full commentary and detailed analysis

Principles of Model Smoothing

� Introduced to address year on year volatility in EUC models

� Averaging 3 years models

� Greater stability and removal of year on year volatility rather than improving model predictability (‘accuracy’)

� Two objectives of stability and accuracy are not necessarily consistent

� Underlying change in customer behaviour

� May lead to changes in model characteristics

� BUT: May then be stabilised by model smoothing

� BUT: Underlying change or trend of single year? (Modelling annual trends)

� Consider: predictive, volatility and trend analysis - single to smoothed year comparisons to assess appropriateness

Principles of Model Smoothing Evaluation

� Analysis compares actual 2008 / 09 consumption data with:

� 2007/08 Models – Single Year model that would have been applied

� Smoothed models for 2005/06, 2006/07 and 2007/08 – Smoothed Models that

were applied

� Using (Consumption Band & WAR Bands)

� CWV Intercepts – Point the line crosses the x-axis (weather sensitivity): predictive ability of single or smoothed compared to actual

� Root Mean Squared (RMS) - Variance of smoothed or single year from actual

� Year on Year Volatility - Change in CWV intercept values each year in single year

and smoothed models

� Trend Analysis - Identification of trends in single year CWV intercepts and change

in load factor values

Analysis 1: Predictive Analysis - Purpose

� Compares CWV intercept values for:

� Most recent actual data set 2008/09 gas year against

� Single year model from 2007/08 that would have applied to 2008/09

� Smoothed model (05/06, 06/07, 07/08) that was applied

� Compare variance of actual intercept to single year model and smoothed year

model

� Includes RMS value for smoothed & single model

� Highlights the variance of the model from the actual

Predictive Analysis: Consumption Bands – Small NDMCWV Intercepts: Predictive Accuracy - Smoothed and Single Year Models

FIGURE 1: SMALL NDM (<2,196,000) CONSUMPTION BAND EUCs PREDICTIVE ABILITY:

Actual Consumption Model Intercept - Single or Smoothed Year Model Intercept

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

<-1.75 -1.75 to -1.25 -1.25 to -0.75 -0.75 to -0.25 -0.25 to 0.25 0.25 to 0.75 0.75 to 1.25 1.25 to 1.75 >1.75

CWV INTERCEPT DIFFERENCE

FR

EQ

UE

NC

Y

08/09-07/08 (RMS=0.6) 08/09-SMOOTHED (05/06-07/08) (RMS=0.5)

Smoothed Model

Similar CWV

Intercept Differences

Lower RMS Values

Similar Predictive

Ability

Predictive Analysis: Consumption Bands – Large NDMCWV Intercepts: Predictive Accuracy - Smoothed and Single Year Models

FIGURE 2: LARGE NDM (>2,196,000)CONSUMPTION BAND EUCs - PREDICTIVE ABILITY:

Actual Consumption Model Intercept - Single or Smoothed Year Model Intercept

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

< -7 -7 to -5 -5 to -3 -3 to -1 -1 to 1 1 to 3 3 to 5 5 to 7 >7

CWV INTERCEPT DIFFERENCE

FR

EQ

UE

NC

Y

08/09-07/08 (RMS=6.5, 6.6 excl. 09B) 08/09-SMOOTHED (05/06-07/08) (RMS=5.3, 5.9 excl. 09B)

Smoothed Model

Better ‘spread’ of

CWV Intercept

Differences

Lower RMS Values

Better Predictive

Ability

Predictive Analysis: All EUC Bands – Small NDMCWV Intercepts: Predictive Accuracy - Smoothed and Single Year Models

FIGURE 3: SMALL NDM EUCs PREDICTIVE ABILITY:

Actual Consumption Model Intercept - Single or Smoothed Year Model Intercept

0

10

20

30

40

50

60

70

80

90

100

110

120

<-7 -7 to -5 -5 to -3 -3 to -1 -1 to 1 1 to 3 3 to 5 5 to 7 >7

CWV INTERCEPT DIFFERENCE

FR

EQ

UE

NC

Y

08/09-07/08 (RMS=2.7) 08/09-SMOOTHED (05/06-07/08) (RMS=2.1)

Smoothed Model

Similar CWV

Intercept Differences

Lower RMS Values

Better Predictive

Ability

Predictive Analysis: All EUC Bands – Large NDMCWV Intercepts: Predictive Accuracy - Smoothed and Single Year Models

FIGURE 4: LARGE NDM EUCs PREDICTIVE ABILITY:

Actual Consumption Model Intercept - Single or Smoothed Year Model Intercept

0

20

40

60

80

100

120

140

160

180

200

< -14 -14 to -10 -10 to -6 -6 to -2 -2 to 2 2 to 6 6 to 10 10 to 14 >14

CWV INTERCEPT DIFFERENCE

FR

EQ

UE

NC

Y

08/09-07/08 (RMS=7.1, 7.2 excl. 09B) 08/09-SMOOTHED (05/06-07/08) (RMS=6.1, 6.3 excl. 09B)

Smoothed Model

Similar CWV

Intercept Differences

Lower RMS Values

Better Predictive

Ability

Analysis 1: Predictive Analysis - Conclusions

� Comparing single & smoothed year model CWV intercepts to actual

consumption models:

� Consumption band analysis highlights smoothed model is slightly better in

predictive ability overall to the single year model

� Consumption + WAR band analysis (all EUCs) highlights smoothed model

has an improved spread of CWV intercepts compared to the single year

model and RMS values are lower for smoothed model

� Overall, comparisons are not overwhelming evidence as to the superior predictive capability of smoothed models. However there is a small

improvement

� Main driver for using a smoothed model is the mitigation of year on year

volatility rather than predictive capability.

� Further analysis required to confirm model smoothing appropriateness…..

Analysis 2: Year on Year Volatility Analysis - Purpose

� Compares year on year volatility reduction of each model type (smoothed

and single year)

� AIM: To reduce differences in between each year

� Compare 09/10 applied smoothed model (06/07, 07/08, 08/09)

� TO

� Applied smoothed model for 08/09 (05/06, 06/07, 07/08)

� Compare 08/09 single year model (that would have been applied to 09/10)

� TO

� Single year model for 07/08 (that would have been applied to 08/09)

� Using variations in CWV intercept and RMS values to identify level of

volatility between model types and years

FIGURE 7: SMALL NDM CONSUMPTION BAND EUCs YEAR ON YEAR VOLATILITY:

08/09 - 07/08 Single Year Model COMPARED TO 09/10 - 08/09 Smoothed Model

0

5

10

15

20

25

30

35

40

<-1.75 -1.75 to -1.25 -1.25 to -0.75 -0.75 to -0.25 -0.25 to 0.25 0.25 to 0.75 0.75 to 1.25 1.25 to 1.75 >1.75

CWV INTERCEPT DIFFERENCE

FR

EQ

UE

NC

Y

08/09-07/08 (RMS=0.6) SMOOTHED (06/07-08/09) - SMOOTHED (05/06-07/08) (RMS=0.2)

Volatility Analysis: Consumption Bands – Small NDM08/09 - 07/08 Single Year Model : 09/10 - 08/09 Smoothed Model

Smoothed Model

Smaller CWV

Intercept Differences

Lower RMS Values

Less Volatility

FIGURE 8: LARGE NDM CONSUMPTION BAND EUCs YEAR ON YEAR VOLATILITY:

08/09 - 07/08 Single Year Model COMPARED TO 09/10 - 08/09 Smoothed Model

0

5

10

15

20

25

30

35

40

< -7 -7 to -5 -5 to -3 -3 to -1 -1 to 1 1 to 3 3 to 5 5 to 7 >7

CWV INTERCEPT DIFFERENCE

FR

EQ

UE

NC

Y

08/09-07/08 (RMS=6.5, 6.6 excl. 09B) SMOOTHED (06/07-08/09) - SMOOTHED (05/06-07/08) (RMS=1.6, 1.7 excl. 09B)

Volatility Analysis: Consumption Bands – Large NDM08/09 - 07/08 Single Year Model : 09/10 - 08/09 Smoothed Model

Smoothed Model

Smaller CWV

Intercept Differences

Lower RMS Values

Less Volatility

Volatility Analysis: All EUC Bands – Small NDM08/09 - 07/08 Single Year Model : 09/10 - 07/08 Smoothed Model

FIGURE 5: SMALL NDM EUCs YEAR ON YEAR VOLATILITY:

08/09 - 07/08 Single Year Model COMPARED TO 09/10 - 08/09 Smoothed Model

0

20

40

60

80

100

120

140

160

<-7 -7 to -5 -5 to -3 -3 to -1 -1 to 1 1 to 3 3 to 5 5 to 7 >7

CWV INTERCEPT DIFFERENCE

FR

EQ

UE

NC

Y

08/09-07/08 (RMS=2.7) SMOOTHED (06/07-08/09) - SMOOTHED (05/06-07/08) (RMS=0.6)

Smoothed Model

Smaller CWV

Intercept Differences

Lower RMS Values

Less Volatility

Volatility Analysis: All EUC Bands – Large NDM08/09 - 07/08 Single Year Model : 09/10 - 07/08 Smoothed Model

FIGURE 6: LARGE NDM EUCs YEAR ON YEAR VOLATILITY:

08/09 - 07/08 Single Year Model COMPARED TO 09/10 - 08/09 Smoothed Model

0

25

50

75

100

125

150

175

200

225

250

< -14 -14 to -10 -10 to -6 -6 to -2 -2 to 2 2 to 6 6 to 10 10 to 14 >14

CWV INTERCEPT DIFFERENCE

FR

EQ

UE

NC

Y

08/09-07/08 (RMS=7.1, 7.2 excl. 09B) SMOOTHED (06/07-08/09) - SMOOTHED (05/06-07/08) (RMS=1.7, 1.8 excl. 09B)

Smoothed Model

Smaller CWV

Intercept Differences

Lower RMS Values

Less Volatility

Analysis 2: Volatility Assessment - Conclusions

� Results as expected

� Smoothed models show less year on year volatility

� Smaller CWV intercept differences between smoothed models

� Better RMS values for all Small and Large NDM EUCs

� Analysis continues to support principles of model smoothing

� 3rd aspect of model smoothing appropriateness needs to be considered...

Analysis 3: Trend Analysis - Purpose

� Identification of trends occurring

� Appropriate: Model averaging or annual model extrapolation

� Extrapolation of annual trend (no smoothing) has been deemed as only

appropriate if a clear trend emerging over recent years could be detected

� Applying single year models without such evidence = higher levels of

volatility in ALP, DAF and load factors each year

� Analysis: Compares ‘trends’ in CWV intercept value for the 3 single year

models constituting the 09/10 smoothed model

� 2006 / 07

� 2007 / 08

� 2009 / 10

� 5 possible outcomes…

Trend Analysis - Outcomes

2008/092007/08

2006/07

UP/UP (UU)

2008/09

2007/08

2006/07

UP/DOWN (UD)

2006/07

2005/062004/05

DOWN/DOWN (DD)

2008/09

2007/082006/07

DOWN/DOWN (DD)

2006/072004/05

FLAT (F)

2005/06

2008/092006/07

FLAT (F)

2007/08

2006/07

2005/06

2004/05

DOWN/UP (DU)

2008/09

2007/08

2006/07

DOWN/UP (DU)

2008/092007/08

2006/07

UP/UP (UU)

2008/092007/08

2006/07

UP/UP (UU)

2008/09

2007/08

2006/07

UP/DOWN (UD)

2008/09

2007/08

2006/07

UP/DOWN (UD)

2006/07

2005/062004/05

DOWN/DOWN (DD)

2008/09

2007/082006/07

DOWN/DOWN (DD)

2006/07

2005/062004/05

DOWN/DOWN (DD)

2008/09

2007/082006/07

DOWN/DOWN (DD)

2006/072004/05

FLAT (F)

2005/06

2008/092006/07

FLAT (F)

2007/08

2006/072004/05

FLAT (F)

2005/062005/06

2008/092006/07

FLAT (F)

2007/08

2006/07

2005/06

2004/05

DOWN/UP (DU)

2008/09

2007/08

2006/07

DOWN/UP (DU)

2006/07

2005/06

2004/05

DOWN/UP (DU)

2008/09

2007/08

2006/07

DOWN/UP (DU)

Trend Analysis – 3 Year Analysis Results

� Over the three years there are limited number of instances of specific EUCs and LDZs where a trends occur to a notable extent

� Supporting document Table 1, 2 and 3 highlight full results

� Most frequently observed pattern: UP / UP (129 of 429: 30%)

� DOWN / DOWN: 37 of 429 (9%)

� DOWN / UP & UP / DOWN: 224 of 429 (52%)

� Trends this year are not consistent with previous years

� UP / UP instances 11% in Autumn 2008

� DOWN / DOWN instances higher at 21% in Autumn 2008

� No obvious trend developing, further investigation supports this……..

Trend Analysis – 4 Year Analysis Results

� Inclusion of 4th year (05/06) to extend analysis

� 356 of 429 cases indicate no consistent trend (83%)

� Similar to previous years analysis (352 in 2008 and 353 in 2007)

� EUCs indicating a consistent trend are small

� 18 of 429 instances indicate a decreasing CWV intercept trend

� 16 of 429 instances indicate an increasing CWV intercept trend

� Worst case is EUC xxE0905W01 which shows upward trend for 4 years

(in 6 of 13 LDZs) – LFs show upward trend in 3 of them (representing

0.19% of aggregate NDM load).

� Conclusion is further supported when observing the single year model

load factors

� Overriding evidence shows the predominant effect is one of no consistent

trend (see document)

Model Smoothing - Conclusion

� Principles of model smoothing

� Reduce year on year volatility (remove modelling just annual trends)

� Not necessarily improve model prediction

� Necessary to review and assess if emerging trends are identified

� Current analysis consistent with results from previous analysis

� Model smoothing overall does reduce year on year volatility

� Model smoothing highlights slightly better level of predictability

� No signs of emerging trends of sufficient clarity have been identified

� Transporters view current methodology to use model smoothing over 3 years to be appropriate and fit for purpose

� Recommend model smoothing is applied for 2010/11 analysis