a modeller’s dilemma: overfitting or underperforming
TRANSCRIPT
![Page 1: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/1.jpg)
A Modeller’s Dilemma: Overfitting or Underperforming?
Tracey Pershouse & Yun Bu (AECOM)
![Page 2: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/2.jpg)
About this presentation
Requirements for transport modelling
Three case studies
o Trip generation
o Trip distribution
o Trip distribution and mode choice
Summary
Page 2
![Page 3: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/3.jpg)
Requirements for Transport Modelling
![Page 4: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/4.jpg)
Requirements for Transport Modelling
Replicating base year situations: In practice, it is used as key measurement of model quality and is often the focus of model development
Forecasting future year implications: In practice, it receives less attention although forecasting is the ultimate use of models
![Page 5: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/5.jpg)
Case Study – Trip Generation
![Page 6: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/6.jpg)
Case Study – Trip Generation
Alternative B: Introduce retail job density as an additional explanatory factor for the number of attractions of shopping trips
Presentation Title Page 6May 1, 2023
Modelling trip attraction to major shopping centres:
HBS trip attraction = a * number of retail jobs + b * Ln(retail job density, in jobs / hectare)
Alternative A: Flags a number of shopping centers as ‘special generators’ and apply adjustment factors derived from observed data
HBS trip attraction = [a * number of retail jobs] * adjustment factors
![Page 7: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/7.jpg)
Validation Performance
Presentation Title Page 7May 1, 2023
The Alternative B produces considerably worse validation results for the largest centres (A and B) but better outcomes for relatively smaller shopping centres (C to H).
![Page 8: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/8.jpg)
Forecasting Performance
Page 8
Alternative A: growth in jobs and growth in trip attractions are similar at all the shopping centres
Alternative B, faster growth rate for trip attractions at the shopping centres with relatively lower density, and slower growth rate at D where the density reaches a very high level
![Page 9: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/9.jpg)
Case Study – Trip Distribution
![Page 10: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/10.jpg)
Case Study – Trip Distribution
May 1, 2023
Calibrate gravity model for HBW trip distribution:
Alternative A:
- Use observed trip productions and attractions as inputs for model calibration, even though it is modelled trip ends are that input to the gravity model for model application
- Only include the OD movements with observed trips > 0, excluding the ODs with zero trips, even though some of the zeros may reflect travellers’ choice of not travelling
![Page 11: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/11.jpg)
Case Study – Trip Distribution
Presentation Title Page 11May 1, 2023
Calibrating gravity model for HBW trip distribution:
Alternative B:
- Use modelled trip productions and attractions as inputs for calibrating the gravity model parameters
- Include all the o-d pairs including those with zero trips. This requires aggregating the movements to sector (SA2) level to remove the zeros that are due to the small sample size in HTS
![Page 12: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/12.jpg)
Base Year Performance – for model calibration
Page 12May 1, 2023
Observed number of trip productions and attractions are used to produce the trip distribution curves
![Page 13: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/13.jpg)
Base Year Performance – for model validation
Presentation Title Page 13May 1, 2023
Modelled number of trip productions and attractions are used to produce the trip distribution curves
![Page 14: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/14.jpg)
Forecasting Performance
Presentation Title Page 14May 1, 2023
Modelled elasticity of changes in trip length in response to cost increase
![Page 15: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/15.jpg)
Case Study – Trip Distribution and Mode Choice
![Page 16: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/16.jpg)
Case Study – Trip Distribution and Mode Choice
Presentation Title Page 16May 1, 2023
Alternative A: mode choice occurs after the trip distribution (or destination choice)
This ordering is computationally convenient: OD trip tables from distribution provide inputs to the mode choice
Arguably, this order also fits in a modelers’ experience of real life in so far as on many occasions travelers’ decide on their destinations first and then choose between transport modes
Develop destination and mode choice model for HBW trips:
![Page 17: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/17.jpg)
Case Study – Trip Distribution and Mode Choice
Presentation Title Page 17May 1, 2023
Develop destination and mode choice model for HBW trips:
Alternative B: mode choice occurs before the trip distribution (or destination choice).
Less computationally straightforward: the generalized cost input for mode choice must be aggregated at trip production level, instead of at trip (origin-destination) level.
Due to the aggregation of cost inputs, the resultant base year validation outcomes tend to be less desirable than the destination choice first approach.
![Page 18: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/18.jpg)
Validation Performance - Overall mode shares
Presentation Title Page 18May 1, 2023
Similar performance between the two alternatives
![Page 19: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/19.jpg)
Validation Performance – Mode share for SA2 to SA2 movements
Presentation Title Page 19May 1, 2023
Alternative A:
![Page 20: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/20.jpg)
Validation Performance – Mode share for SA2 to SA2 movements
Presentation Title Page 20May 1, 2023
Alternative B – similar performance to Alternative A
![Page 21: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/21.jpg)
Forecasting performance
May 1, 2023 Page 21
Car GC = 30 mins
PT GC = 70 min
Car GC = 30 mins
PT GC = 70 min
D1
D2
Base case assumptions:
1000 trips from origin to destination D1, D2, via Car or PT
Destination and Mode choice results in base case (for both alternative):
D1, Car = 478 trips
D1, PT = 22 trips
D2, Car = 478 trips
D2, PT = 22 trips
![Page 22: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/22.jpg)
Forecasting performance
May 1, 2023 Page 22
Car GC = 40 mins
PT GC = 70 min
Car GC = 30 mins
PT GC = 70 min
D1
D2
Scenario testing:
Car travel cost to D1 increases from 30 to 40 minutes
Destination and Mode choice results , Alternative A (double constrained)
D1, Car = 432 trips (decrease from 478 in Base)
D1, PT = 68 trips (increase from 22 trips in Base)
D2, Car = 478 trips (unchanged from Base)
D2, PT = 22 trips (unchanged from Base)
![Page 23: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/23.jpg)
Forecasting performance
May 1, 2023 Page 23
Car GC = 40 mins
PT GC = 70 min
Car GC = 30 mins
PT GC = 70 min
D1
D2
Scenario testing:
Car travel cost to D1 increases from 30 to 40 minutes
Destination and Mode choice results , Alternative B
D1, Car = 394 trips (more significant decrease)
D1, PT = 106 trips (more significant increase)
D2, Car = 480 trips (nearly unchanged)
D2, PT = 20 trips (nearly unchanged)
![Page 24: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/24.jpg)
Summary
![Page 25: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/25.jpg)
Implications
Overfitting: Use synthetic measures for fitting base year outcomes to observed data, which may affect forecasting performance adversely.
Underperforming: Accommodate underperformance (compared with overfitting option) in base year validation, in exchange of improved forecasting performance
Another option:
Common practice?
![Page 26: A modeller’s dilemma: overfitting or underperforming](https://reader036.vdocuments.net/reader036/viewer/2022062503/587894001a28ab375f8b636b/html5/thumbnails/26.jpg)
Thank You