assessment of agreement between skybit predictions and on-site measurements henry k. ngugi, phd....
TRANSCRIPT
![Page 1: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/1.jpg)
Assessment of Agreement between SkyBit Predictions and
On-site Measurements
Henry K. Ngugi, PhD.
Penn State Fruit Research & Extension Center, Biglerville, PA.
![Page 2: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/2.jpg)
Quantitative Epidemiology and Commercial Fruit Production
Scientist
Industry Orchardist
Adjudicate between the grower and industry to maintain a commercially viable tree fruit industry in PA
![Page 3: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/3.jpg)
Diseases of concern in the mid-Atlantic region
![Page 4: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/4.jpg)
![Page 5: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/5.jpg)
• SkyBit---Marketed by ZedX Inc.
• Growers pay ~$60 per month for pest management information
• Fire blight and apple scab models developed by Dr. James W. Travis (Penn State Univ.)
• Models were never validated
• How do SkyBit forecasts perform?
Assessment of Agreement Between Forecasts
Adjudicating: The SkyBit Example
![Page 6: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/6.jpg)
How do SkyBit weather data predictions compare with data
collected on-site?
Gold standard = Campbell Scientific and Spectrum Technologies weather stations at PSU-FREC, Biglerville, PA
![Page 7: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/7.jpg)
• How do SkyBit data predictions compare with data collected on-site?
• Perform statistical agreement tests between data collected on-site and SkyBit predictions
• Lin’s concordance analysis for continuous variables
• Limits of agreement statistics for continuous variables
• Concordance tests for categorical variables
Reliability of SkyBit Disease Forecasts
![Page 8: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/8.jpg)
Agreement in weather data
I. Temperature
SkyBit mean temperature (oC)
0 5 10 15 20 25 30S
pect
rum
mea
n te
mpe
ratu
re (
oC
)0
5
10
15
20
25
30
SkyBit temperature (oC)
0 5 10 15 20 25 30
Cam
pbel
l Sci
. te
mpe
ratu
re(o
C)
0
5
10
15
20
25
30
r = 0.988ρ = 0.987
r = 0.992ρ = 0.991
Highly significant agreement between SkyBit data and on-site daily mean temperature measurements
![Page 9: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/9.jpg)
Shapiro-Wilk test W = 0.986, P = 0.223 i.e., cannot reject normality hypothesis
Differences must be normally distributed to compute limits of
agreement
![Page 10: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/10.jpg)
Confidence limits agreement
Mean of SkyBit and Spectrum (oC)
5 10 15 20 25
Diff
eren
ce (
Spe
ctru
m-S
kyB
it; o
C)
-3
-2
-1
0
1
2
3Over a 4-month period, only in 4 out of 152 days did the SkyBit measurements significantly differ from on-site temp. measurements
95CL = d 1.96SD, in this case = -1.82 to 1.65
![Page 11: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/11.jpg)
Rainfall data: April to June 2009
SkyBit (mm)
0 10 20 30 40
Nat
iona
l Wea
ther
Ser
vice
(m
m)
0
10
20
30
40
B
Good agreement with on-site data from the Spectrum Tech. weather station but SkyBit underestimates the rainfall amounts
SkyBit (mm)
0 10 20 30 40
Spe
ctru
m (
mm
)
0
10
20
30
40 A
No agreement with data from National Weather Service
r = 0.882ρ = 0.875
r = 0.405ρ = 0.403
![Page 12: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/12.jpg)
Spectrum Tech. data Lin’s concordance coefficient, ρ = 0.860; r = 0.876
Campbell Sci. data Lin’s concordance coefficient, ρ = 0.746; r = 0.788
SkyBit (wetness hours)
0 5 10 15 20 25
Spe
ctru
m (
wet
ness
hou
rs)
0
5
10
15
20
25
SkyBit (wetness hours)
0 5 10 15 20 25
Cam
pbel
l Sci
. (w
etne
ss h
ours
)
0
5
10
15
20
25
Wetness hours: April to June 2009
![Page 13: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/13.jpg)
Wetness confidence limits analysis
Mean Cambell & SkyBit (hrs)
0 5 10 15 20 25
Diff
eren
ce (
Cam
pb.
- S
kyB
it; h
rs)
-15
-10
-5
0
5
10
15
Generally good agreement but SkyBit over-estimates in wet days and underestimates in dry
Good agreement between the ‘gold standards’
Campbell Sci. (wetness hours)
0 5 10 15 20 25
Spe
ctru
m (
wet
ness
hou
rs)
0
5
10
15
20
25
r = 0.929ρ = 0.919
![Page 14: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/14.jpg)
Test for agreement between fire blight predictions (MaryBlyt and
SkyBit)Concordance coefficients and McNemar's test statistics for agreement between SkyBit and MaryBlyt fire blight forecasts for Biglerville, PA April and May, 2009 Coefficient Estimate SEx RemarkKendall’s tau-B 0.607 0.103 ModerateStuart’s tau-C 0.482 0.112 No concordanceCohen's Kappa 0.570 0.115 ModeratePearson’s correlation 0.607 0.115 Weak
McNemar's testy χ2 = 6.40 P = 0.014; df = 1; n = 55xAsymptotic standard erroryFor the null hypothesis that the two data sets disagree
![Page 15: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/15.jpg)
SkyBit predictions are more cautious
Concordance coefficients and McNemar's test statistics for agreement between SkyBit and MaryBlyt fire blight forecasts for Biglerville, PA April and May, 2009 Coefficient Estimate SEx RemarkKendall’s tau-B 0.884 0.065 High Concordance Stuart's tau-C 0.826 0.086 High concordanceCohen's Kappa 0.883 0.065 High concordancePearson’s correlation 0.884 0.064 Strong
McNemar's testy χ2 = 0.333 P = 0.564; df = 1; n = 55xAsymptotic standard erroryFor the null hypothesis that the two data sets are disagree
When MaryBlight “H” is counted as “I”
![Page 16: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/16.jpg)
Disease forecast assessments• No agreement between SkyBit and MaryBlyt
forecasts (χ2 = 6.4; P < 0.011; McNemar’s test) for infection events
• Agreement only when MaryBlyt ‘H’ is counted as = ‘I’ (χ2 = 0.333; P = 0.564)
• Good agreement in apple scab forecast (χ2 = 2.0; P = 0.15 and χ2 = 3.6; P = 0.058 for Spectrum and Mill table models, respectively)
![Page 17: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/17.jpg)
Summary of Results
• SkyBit delivers reliable data to growers in the mid-Atlantic region Temperature measurements highly reliable Underestimates rainfall amounts Wetness measurements unreliable in very dry
or wet conditions
• SkyBit forecasts for apple scab are as reliable as Mill’s Table or Spectrum model
• SkyBit fire blight predictions are conservative relative to those of MaryBlyt
![Page 18: Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville,](https://reader036.vdocuments.net/reader036/viewer/2022062805/5697c0231a28abf838cd43d8/html5/thumbnails/18.jpg)
Acknowledgements
People in Ngugi Lab
$$$ FOR Ngugi Lab
USDA –CSREES
SHAP, PSU-CASDrs. Jim Travis & N.O. HalbrendtDrs. Jim Travis & N.O. Halbrendt