adaptive presentation on machine learning at dama nyc · 18/10/2018 · machine learning vs...

Adaptive Presentation onMachine Learning

atDAMA NYC

October 18, 2018James Cerrato

Chief Product [email protected]

1Copyright © 2018 Adaptive, Inc. All Rights Reserved.

Jeff Goins

[email protected]

Confidentiality & Disclosure Agreement

Copyright © 2018 Adaptive, Inc. All Rights Reserved. 2

This is an unpublished work, the copyright in which vests in Adaptive, Inc. (“Adaptive”). All rights reserved.

The information contained herein is confidential and the property of Adaptive, Inc. and is supplied without liability for errors or commissions. No part may be reproduced, disclosed or used, except as authorized by contract or other written permission. The copyright and the foregoing restriction on reproduction and use extend to all media in which the information may be embodied.

• All product names used herein are for identification purposes only and may be trademarks of their respective companies.

What is Machine Learning?

• Machine Learning is a branch of artificial intelligence based on the idea that systems can learn from data, identify patterns and make decisions with minimal human intervention

• The Pattern: Understand, learn, predict, adapt… and repeat

• When systems start acting autonomously they can produce surprising resultso Google is using deep learning algorithms to produce better,

translations between languages. This led to the system creating its own “language” to serve as a canonical representation of the meaning of word. This machine language is used as an intermediate translation between human languages.


Why has Machine Learning become hot?

• Produces analytical outcomes and insights not possible using traditional development

• Significantly reduces labor costs• Leverages the growing volume and velocity of data from

e-transactions, sensors, Internet of Things, social media…

• Create new predictive models from raw data• Ever increasing power of computing and miniaturization

of computing devices is making it more feasible• Availability of skilled resources is increasing

4Copyright © 2018 Adaptive, Inc. All Rights Reserved.

Cont.

• “The biggest reason is because it works… Take the simple problem of recognizing traffic signs in different weather and lighting conditions… The best deep learning models (committees of convolutional neural networks) can now achieve about 99.5% average accuracy, compared to about 99% for the best humans, and a bit lower than 99% for average humans.”

Matthew Lai, Research Engineer @ Google DeepMind

• “Machine Learning provides insights and solutions for business process optimization and operational enhancements. It helps sorting through vast amount of data by analysing and identifying patterns that can help almost every business.”

Nishant Poojary


What are examples of it being used successfully?


Natural Language Processing

Recommendation Systems

Algorithmic Trading

Source: Iliya Valchanov

Cont.

• Predict if an employee will stay with your company or leave.

• Decide if a customer is worth your time, if they are likely to buy from a competitor, or not buy at all. You can optimize processes, predict sales, and discover hidden opportunities.

• And many more…

Discover the business value hidden in your data!

Copyright © 2018 Adaptive, Inc. All Rights Reserved. 7Source: Iliya Valchanov

Cont.

It’s all around us…

• Many of our day-to-day activities are powered by machine learning algorithmso Fraud detection, Web search results, Text-based sentiment analysis, Credit

scoring and next-best offers, Prediction of equipment failures, Email spam filtering, license plate readers, facial recognition etc.

• Machine Learning can be applied in more complex applications o Self-parking cars, Guiding robots, Airplane navigation systems (manned

and unmanned), Space exploration, Medicine etc


Machine Learning vs Traditional Application Development

• Software engineers use their ingenuity to come up with a solution and formulate it as a precise program a computer can executeo Some problems require creative problem solving

• Machine learning systems, collect input data and desired target values. They instruct a computer to build a program or model that computes an optimal output for each input value.


Machine

Learning

Outcomes

Data

Quality

Enterprise

Architecture

Data Governance

OntologiesData

Mapping

Rules

Management

Big Data

Algorithms

Predictive Models

Objective Function

OptimizationAlgorithm

OptimizedModels

Metadata

Machine Learning is a catalyst for many IT domains

How does Machine Learning apply to other IT domains?

Enterprise Architecture o Process Automation o Sales Optimization o Customer Service o Security

Metadata Managemento Data gathering o Modeling & Indexing o Determining dependencies o Automating mapping o Data Lineage o Data retention policies


Big Data o Data visualization o Data integration o Dashboards and BI o Automated, data-driven decisions

Data Quality o Cleaning and Maintaining data o Data Selection o Feature Extraction o Creating Analytical datasets


Cont.

Ontologieso Automate mapping between ontologieso Automate mapping from technical data elements to ontologyo Make ontologies executable


Cont.

Data Governance is still required

Data Governance teams are needed to perform these tasks• Identify sources for training and validation data sets• The Machine Learning models created need to be cataloged and governed

o Track where they are deployedo Record dependencies (e.g. the platform such as TensorFlow, Watson)o Assign stewards, stakeholders, consumers…

• Verify the quality of the data sets (e.g. size/variability)• Perform change management for models• Treat the models as part of the lineage for any output data• Map the data sets to well specified semantic data models• Map the operational data the ML models will be used on to data with the

same meaningo Don’t do training on one set of data and apply it to operational data with a different

meaning!


What are the critical success factors for a Machine Learning project?

• Find new team members with the right skills• Train existing team members in the new skills• More data, more data, MORE DATA!• Understand what the key features are in the data

o What are the characteristics of good training data?

• The quality of the data, and that it is representativeo Data Quality is still important!

• Stay focused on the business problem to be solved• Find the right parameters of the model• Define the best Objective Function for measuring the

outcomes


What are the risks to be aware of?

• The design of the algorithms is not as transparent• Algorithms are created “inside the black box”, which can

lead to intentional or unintentional biaseso Why is Facebook putting this in my News Feed?

• If the design is not apparent, monitoring is more difficulto For example, forensic analysis of how a result was achieved

becomes difficult or impossible

• The data collected for Machine Learning algorithms by companies may violate the privacy of users


Machine Learning Algorithms


Source: Analytics vidhya

Supervised Learning

• First train the model with lots of training data

• Then we apply the model to new data to form predictions

• This process is called Supervised Learning which is really fast and accurate.


Source: Madhu Sanjeevi

• In Unsupervised Learning the training data does not include Targets, we don’t tell the system where to go , the system has to understand itself from the data we give. The training data is not structured.



Unsupervised Learning

• Reinforcement learning is really powerful and complex to apply for problems.

• It is a type of Machine Learning technique that enablesan agent to learn in an interactive environment by trialand error using feedback from its own actions andexperiences.



Basic explanation of how Machine Learning works

Datao Prepare a certain amount of data to train on. Usually, this is

historical data, which is often readily available.

Modelo The simplest model to train on is a linear model. A linear

model is just the tip of the iceberg, though it lets us create complicated non-linear models. They usually fit the data much better than a simple linear relationship.



Objective functiono The third ingredient is the objective function. After feeding the

data to the model, we want to obtain an output that is as close to reality as possible. That’s where the objective function comes in.

o The objective function allows you to estimate how accurate the model’s outputs are, on average.



Optimization algorithmo The final ingredient is the optimization algorithm, or the mechanics

through which we vary the parameters of the model in order to optimize the objective function.

o For each set of parameters, we would calculate the objective function.

o Then choose the model with the highest predictive power — the one with an optimal objective function.



Tools and Technologies


Cont.


adaptive presentation on machine learning at dama nyc · 18/10/2018 · machine learning vs...

Documents