you get what you measure why metrics are important - usenix › sites › default › files ›...
TRANSCRIPT
![Page 1: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/1.jpg)
Kumar SrinivasamurthyBing & Cortana Engineering@00kumars
You get what you measure—why metrics are important
![Page 2: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/2.jpg)
Examples
![Page 3: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/3.jpg)
Metric definition, education and reliability of monitors
https://mars.jpl.nasa.gov/msp98/news/mco990930.html https://www.nytimes.com/1983/07/30/us/jet-s-fuel-ran-out-after-metric-conversion-errors.html
![Page 4: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/4.jpg)
Availability & Downtime
https://en.wikipedia.org/wiki/High_availability
Simplified formula:
𝐴𝑣𝑎𝑖𝑙𝑎𝑏𝑖𝑙𝑖𝑡𝑦 =(𝑇𝑜𝑡𝑎𝑙 𝑅𝑒𝑞𝑢𝑒𝑠𝑡𝑠 −𝐹𝑎𝑖𝑙𝑒𝑑 𝑅𝑒𝑞𝑢𝑒𝑠𝑡𝑠)
𝑇𝑜𝑡𝑎𝑙 𝑅𝑒𝑞𝑢𝑒𝑠𝑡𝑠%
𝐴 =𝑇−𝐹
𝑇%
![Page 5: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/5.jpg)
Hitting the target and missing the point 𝐴 =𝑇−𝐹
𝑇%
100% On-time delivery
![Page 6: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/6.jpg)
http://www.milliondollarhomepage.com/
![Page 7: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/7.jpg)
How and where you measure matters 𝐴 =𝑇−𝐹
𝑇%
![Page 8: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/8.jpg)
Gaps in measurement
https://twitter.com/dannysullivan/status/460899863715405825
![Page 9: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/9.jpg)
• Periodic update of metrics• Above fold vs below fold
measurement
![Page 10: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/10.jpg)
The great Hanoi rat massacre
More: http://freakonomics.com/media/vannrathunt.pdf
https://www.atlasobscura.com/articles/hanoi-rat-massacre-1902
https://flic.kr/p/ovusTD
![Page 11: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/11.jpg)
https://en.wikipedia.org/wiki/Window_tax#/media/File:Window_Tax.jpg https://en.wikipedia.org/wiki/Window_tax#/media/File:Windows_in_Brighton_Street,_Edinburgh.jpg
Unintended consequences - Window tax
![Page 12: You get what you measure why metrics are important - USENIX › sites › default › files › conference › ... · 2019-12-18 · Kumar Srinivasamurthy Bing & Cortana Engineering](https://reader036.vdocuments.net/reader036/viewer/2022081400/5f1b8ea5ebd9e038c656754e/html5/thumbnails/12.jpg)
• High data quality
• Correctness, completeness and cleanliness
• Periodic metrics update
• Metrics do get “gamed”
• Wrong incentives – unintended consequences
• Metrics overload
• Percentiles can hide problems
• Some metrics do flat-line. Important to let them not regress