almaz monitoring
TRANSCRIPT
Adaptive machine learning real-time data
quality monitoring of corporate reporting,
business and technological processes
No active monitoring of data quality
Data
Warehouse
5% of data becomes null
Incorrect financial reports
Incorrect operational
reports
Anti-target
marketing
Why revenue goes
down while traffic is
unchanged?
I can’t trust
these
reports!I don’t need these!
Example of streaming data
Normal behavior
Abnormality detected: no peaks
Almaz Monitoring
Abnormal behavior is detected!
System administrator
There was not enough lubricant!
Old lubricant worked just a month
and probably was defected.
Production Manager
Good job, guys!
If the turbine broke, It could cost us
$250 000
13:21
13:22
13:23
Business needs
Financial and operational reports
must have trusted data
Corporate data layer must be
verified before it is used by other
systems of data scientists
Operational KPI’es must be
monitored in real-time and there
should be immediate reaction to
any abnormal behavior before it
leads to substantial damage
Tasks Solution Benefits
Constant autonomous data quality
monitoring of corporate reporting,
business and technological
processes
Instant alarms on occur of any
statistically significant deviation.
Notifications to mobile devices
Self learning system that can
adapt to the data and user
behavior. The system that does
not need any expert or data
scientist and start working right
out of the box.
Trusted data
Revenue growth by minimizing
downtimes
No more important business
decisions on inaccurate data
Objective data quality control by
machine. Eliminated “human
factor”.
Prediction of potential
mulfuntions
Machine Learning Quality Control
Spikes Gaps
Outliers Symmetric
deviations
Change Points
Instant detection of significant
abnormality of real-world data flows
Seasonality trends
Weekend or holidays
Daily intervals
Outliers
Non-cleansed data
Other particular business specifics
Visual
Identification of abrupt changes in the
generative parameters of sequential data.
Major algorithms are adopted to enterprise
data
• Moving average
• Bayesian
• Autocorrelation
• Regressions
• CUSUM
• Shewhart
Adaptive Machine Learning
Accurate changepoint detection
Adaptive Machine Learning
Data Quality Monitoring at
Enterprise scale
Financial reports
Analytical aggregates and views
Real time data flows
Operational reporting
Big Data streams and storage
Enterprise scale
High Level Schema
Data connectors Control and
Visualization
module (web ui)
Notification
Engine
Trello
SMS
… and many
more
… and many
more
Integration
layer
Adaptive Machine
Learning Module
A high-
throughput
distributed
messaging
system
MLlib
User Interface
Create new monitoring
System allows to drag&drop KPI’es and ‘group by’ fields
System scans the data and automatically detects the type of fields and their specifics (discrete/
continuous, numeric / string, dictionary lookups etc.)
Visualize Abnormal Behavior
Trello: Notifications and Issue resolution workflow
Convenient issue tracking and resolution workflow
Push notifications on mobile devices
Competitors
New Generation of Data Quality
Adaptive machine learning real-time data quality monitoring of corporate reporting, business and technological processes
Artificial Intelligence and Adaptive machine learning
Complete autonomy
Big Data Real-time monitoring and notifications
Classical Data Quality Monitoring
Convenient tools of data visualization and data
analysis for data scientists
Rules templates. Experts create validation rules and
refresh them after data trends changes
Monitoring and notifications
Informatica Proactive Monitoring for Data Quality
SAS® Data Quality
InfoSphere Information Server for Data Quality
Trillium Software Data Quality
SAP Data Quality Management
Thank you!
Vladimir BakhovCell-phone: +7 (905) 716 54 46E-mail: [email protected]
Resident of Skolkovo Innovation Center