smart monitoring system for anomaly detection on business … · 2019. 12. 18. · alibaba group...
TRANSCRIPT
![Page 1: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/1.jpg)
Smart Monitoring System for Anomaly Detection on Business
Trends in Alibaba
Zhaogang [email protected]
1
![Page 2: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/2.jpg)
About me
• Senior Specialist of GOC(Global Operation Center) Team in Alibaba Group• Business trend monitoring
• Business fault diagnosis and root cause analysis
• Data warehouse for infrastructure and operation data
• Before I joined Alibaba• Senior Engineer of SRE Team in Baidu
2
![Page 3: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/3.jpg)
Introduction to Alibaba Group
3
![Page 4: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/4.jpg)
About business trends monitoring in Alibaba
• Business faults management• Mapping business functions to business trends
• Faults Priority Definitions• Orders per minute on Taobao decreased by XX% or above => P1 Fault• Transactions per minute on Alipay decreased by X% to XX% => P2 Fault
• Business trends monitoring• Business faults can be found by anomaly detection on business trends
Business Functions
Faults Priority Definitions
Business TrendTime Series
Business Units
4
![Page 5: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/5.jpg)
Features of businesses trends
Cyclicity
Holiday Effect
Noise and interference
5
![Page 6: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/6.jpg)
Challenges of anomaly detection on business trends
• How to adopt the characteristics of different business trends?
• How to meet the artificial standards of faults?
• How to get all the configurations in automation?
6
![Page 7: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/7.jpg)
Summary of anomaly detection approaches
• Local trend based• Static threshold• Dynamic threshold• Local regression
• Historical trend based• Trend prediction
• Segment average of historical data• Time series decomposition
• Holt-winters• STL (Seasonal Trend LOESS)
• Machine Learning• Deep Learning(LSTM)
Anomaly Detection
Prediction
7
![Page 8: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/8.jpg)
Our choice
• Our choice• STL (Seasonal Trend on LOESS)
• Advantages of STL on business trends time series• Suitable for cyclical data
• Suitable for data with drifting trend
• Robust to local noises and interference
https://quantdare.com/wp-content/uploads/2014/09/decomp-example.png
8
![Page 9: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/9.jpg)
How to get a good “prediction”
• A good “prediction”• Accurately fits business trends
• Smooth and stable
original value
predicted value
9
![Page 10: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/10.jpg)
Using STL directly on original data…
• Drawbacks• Effected by noise
• Not smooth or stable
• Not enough sensitive to recent trends
• Solutions• Customized data preprocessing
original value
predicted value
10
![Page 11: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/11.jpg)
Customized data preprocessingRemove history noises
Smooth the data
Complete the “future” data. Smooth the data again:
Use recent trends to adjust the outline of historical data 11
![Page 12: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/12.jpg)
A better “prediction” is born
original value
predicted value
12
![Page 13: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/13.jpg)
Anomaly detection based on predicted curve• The traditional N-sigma law
• Anomaly point : residence > N * sigma
• N == 3?• Sigma varies with the time
segment• Sigma varies with the business
trend
• We need• Different N for each time
segment and each business trend
13
![Page 14: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/14.jpg)
How to determine the “N”s
• Divide the time segments by residence for each business trend
• Initialize the N for each time segment
• Adjust the N according to manual feedback
14
![Page 15: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/15.jpg)
Manual feedback loop
• About the label data• Label data from the operators’ team
• Effectiveness of the anomaly points
• Quantity of the label data
• How to utilize the label data• Adjust the N parameter according to
the label data
• Tolerant the errors in the label data
15
![Page 16: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/16.jpg)
Evaluation
• Anomaly detection• Precision: 80%
• Recall: 80%
• Configuration cost• Auto parameter initialization
• Auto parameter adjustment• When the business trend changes
16
![Page 17: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/17.jpg)
Future work
• Lightweight anomaly detection for system metrics
• Early warning for business faults
• Fault diagnosis and root cause analysis
17
![Page 18: Smart Monitoring System for Anomaly Detection on Business … · 2019. 12. 18. · Alibaba Group •Business trend monitoring •Business fault diagnosis and root cause analysis •Data](https://reader036.vdocuments.net/reader036/viewer/2022071417/6114384a82d993538e38d527/html5/thumbnails/18.jpg)
Q & A
18