![Page 1: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/1.jpg)
L. Mentaschi, G. Besio, F. Cassola, A. MazzinoDipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA),
Università degli studi di Genova
![Page 2: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/2.jpg)
Validation of wave model in the Mediterranean sea
Model Wavewatch IIIArdhuin et al. (2008) source terms.Tolman and Chalikov (1996) source terms, not tuned
to the Mediterranean sea conditions.
O
OSNBI
)( Normalized Bias (NBI)
Statistical indicators used for validation
Normalized Root MeanSquare Error (NRMSE) 2
2)(
O
OSNRMSE
Correlation coeff. (ρ)OS
ii
N
OOSS
)()(
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 3: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/3.jpg)
Statistics on 17 storms and 23 buoys
ACC350 T & C
NBI 2.1% -11.2%
ρ 0.889 0.883
NRMSE 0.2864 0.2798 (-2.3%)
February 1990 storm
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 4: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/4.jpg)
In what sense NRMSE is not completely reliable?A numerical example
ρ=0.614 for both of the simulations
NBI = -12% for the red simulation
One would say the best simulation is the blue one.
But …
NRMSE(blue) = 0.384
NRMSE(red) = 0.356 (~ -7.2%)
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 5: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/5.jpg)
Geometrical decomposition of NRMSE
2
22
22
2
2)]()[(NBI
O
ONRMSE
O
OOSSSI
Oi
ii
NBIO
OBC
O
NRMSE 22
2
222NRMSEBCSINRMSE
Bias component
Scatter component (sometimes called Scatter Index)
O
OSNBI
where
Are SI and BC independent?
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 6: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/6.jpg)
Example of set of simulations with constant ratio :
In general and are not independent.
.~ constSS
SS /
iiNBI SNBIS 0)1(
where is the unbiased simulation.iS0
S S
Let’s assume this relation:
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
Holds for amplifications
Amplification factor: NBI1
![Page 7: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/7.jpg)
Relation SI – NBI
22
22
2
2
2 2)]()[(
O
OSOS
i
ii
OO
OOSSSI
)1( NBIOS
)1(./ 0 NBIconstS SSS •
22
2200
20
2202 )(2
O
OSOSSS
OSI
•
OS ~0 0~)1(2
~ 2022
2
0
2
SIONBI
SI
O
O
NBI
SI grows linearly in NBI around NBI=0
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 8: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/8.jpg)
NRMSE
O BCO
OSISI
2
22
0 2
11~
Fits well real world data
Significant wave height Mean period
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 9: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/9.jpg)
NBISISI
2
11~ 0
The simulation with the minimum value of NRMSE underestimates the average value
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 10: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/10.jpg)
How to overcome this problem?
Hanna and Heinold (1985) indicator:
SO
OS
OS
OSHH
ii
ii22 )()(
Demonstration that with ρ constantHH is minimum when bias is null
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 11: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/11.jpg)
2)( 22222
2
OS
OS
ii
ii
OS
OS
OS
OSHH
OS ~0•
0~0
2
NBINBI
HH
02~22
22
0
2
22
O
O
NBIO
O
NBI
HH
HH has a minimumfor null bias
)1( NBIOS
)1(./ 0 NBIconstS SSS •
•
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 12: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/12.jpg)
Wavewatch III validation on the Mediterranean sea.
ACC350 T & C
NBI 0.021 -0.112
ρ 0.889 0.883
NRMSE 0.2864 0.2798 (-2.3%)
HH 0.3459 0.3634 (+4.8%)
HH has a minimum for null bias
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 13: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/13.jpg)
Future development
• NRMSE tends to assume smaller values for simulations that underestimate observations, and this is mostly evident when relation holds.
• HH indicator overcomes this problem introducing a different normalization of the root mean square error.
.~/ constSS
Conclusions
• Usage of HH indicator to validate Wavewatch III model on the Mediterranean sea.
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
![Page 14: L. Mentaschi, G. Besio, F. Cassola, A. Mazzino Dipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA), Università degli studi di Genova](https://reader033.vdocuments.net/reader033/viewer/2022061306/55149360550346d36e8b533d/html5/thumbnails/14.jpg)
Thank you!
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
L. Mentaschi, G. Besio, F. Cassola, A. MazzinoDipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA),
Università degli studi di Genova