Practical Issues in Machine Learning－how to create an effective ML model?

Prof. Shou-de Lin (林守德)

CSIE, National Taiwan University

sdlin@csie.ntu.edu.tw

Diagnose a Machine Learning Solution-Why My ML Model Doesn’t Work?

Prof. Shou-de Lin (林守德)

CSIE, National Taiwan University

sdlin@csie.ntu.edu.tw

Machine Discovery and Social Network Mining Lab,

CSIE, NTU

• PI: Shou-de Lin

– B.S. in NTUEE– M.S. in EECS, UM– M.S. in Computational Linguistics, USC– Ph.D. in CS, USC– Postdoc in Los Alamos National Lab

• Courses:– Machine Learning and Data Mining-

Theory and Practice– Machine Discovery– Social network Analysis– Technical Writing and Research

Method– Probabilistic Graphical Model

• Awards:– All-time ACM KDD Cup Champion

(2008, 2010, 2011, 2012, 2013)– Google Research Award 2008– Microsoft Research Award 2009– Best Paper Award WI2003, TAAI 2010,

ASONAM 2011, TAAI 2014– US Areospace AROAD Research Grant

Award 2011, 2013, 2014, 2015, 2016

Machine Learning with Big

Data

MachineDiscovery

Learning IoTData

Applications inNLP&SNA&

Recommender

PracticalIssues inLearning

3

Talk Materials Based on Hands-On Experiencein Solving ML Tasks, Including

Participating ACM KDD Cup for 6 years

>20 Industrial collaboration

Visiting Scholar in Microsoft from 2015~2016

Team NTU’s Performance on ACM KDD CupKDD Cups 2008 2009 2010 2011 2012 2013

Organizer Siemens OrangePSLC

DatashopYahoo! Tencent Microsoft

TopicBreast Cancer

PredictionUser Behavior

Prediction

Learner Performance

PredictionRecommendation

Internetadvertising

(track 2)

Author-paper & Author name Identification

Data Type Medical Telcom Education MusicSearch Engine

LogAcademic Search

Data

ChallengeImbalance

DataHeterogeneous

Data

Time-dependent instances

Large Scale Temporal +

Taxonomy Info

Click through rate prediction

Alias in names

# of records

0.2M 0.1M 30M 300M 155M250K Authors, 2.5M papers

# of teams >200 >400 >100 >1000 >170 >700

Our Record Champion 3rd place Champion Champion Champion Champion

5

Industrial Collaboration企業產學合作

異質探勘技術 Google 2009

結合社群模型與遊戲激勵機制

IBM 2015

空氣品質自動偵測 Microsoft 2014

感測網路內容分析 INTEL 2011-2015

分散式學習 US Airforce 2011-2017

收視率預測以及交通狀況預測

資策會 2011-2015

音樂歌詞情緒辨識 中華電信 2014

銀行履歷推薦分析 104人力銀行 2011-2013

客服自動系統 趨勢科技 2016-

大數據徵信及生命週期預估

KPMG安侯建業 2015

資料探勘顧問 台達電子 2015

技術轉移與技術服務

ranked-based matrix factorization 工研院資通所 2013

Multi-task Learning Models 工研院巨資中心 2015

rankNet 工研院資通所 2013

Multi-label Classification Tools

工研院巨資中心 2013

社群網路技術服務MobiApp 2012

商品推薦服務技術合作i-True 2012

What is Machine Learning?

General Def of Machine Learning (ML)

• ML tries to optimize a performance criterion using example data or past experience.

• Mathematically speaking: given data X, we want to learn a function mapping f(X) for certain purpose, e.g.– f(x)= a label y classification– f(x)= a set Y in X clustering– f(x)=p(x) distribution estimation– …

• ML techniques tell us how to produce high quality f(x), given certain objective and evaluation metrics

8

Why My Machine Learning Models Fail (meaning prediction accuracy is low)?

A series of analyses are required to understand why

The ML Diagnose Tree

10

Is it ML?

NoTry othersolutions

Correct MLScenario?

NoTry Other Scenario

Have you identified a

suitable model?’

Yes Do you have enough

data?

Yes Is your model too

complicated?

Right Feature?

Yes Performed feature

engineering?

Suitable Evaluation Metrics?

Proper validation set?

Diagnose 1: Is it an ML task?

• Are you sure Machine Learning is the best solution for your task?

To ML or not to ML, that is the question !!

11

• X come from a close set with limited variation– simply memorize all possible XY mappings

– E.g. Word translation using dictionary

• F(x) can be easily derived by writing rules– E.g. compression/de-compression

• X is (sort of) independent of Y– E.g. X<ID, name, wealth>, YHeight

• f(x) is not smooth, or f(x+X) y+ y

Tasks Doubtful for ML

Too hard!!

Too easy!!

12

OK, my task belongs to ML-solvable, but I still failed

The ML Diagnose Tree

Is it ML?

NoTry othersolutions

Correct MLScenario?

NoTry Other Scenario

Have you identified a

suitable model?’

Yes Do you have enough

data?

Yes Is your model too

complicated?

Right Feature?

Yes Performed feature

engineering?

Suitable Evaluation Metrics?

Proper validation set?

14

Diagnose 2: Which ML Scenario?

• Have you modeled your task into the right ML scenario?

– ML Classification/Regression SVM, DNN, DT

• Which ML toolbox should you choose?

Machine Learning

Supervised Learning

Classification

SVM

15

Let’s Talk About….. Understanding Machine Learning in 10 Mins

What

Can Do

A variety of ML Scenarios

Supervised Learning

Classification & Regression

Multi-label learning

Multi-instance learning

Cost-sensitive leering

Semi-supervised learning

Active learning

Unsupervised Learning

Clustering

Learning Data Distribution

Pattern Learning

Reinforcement learning

Variations

Transfer learning

OnlineLearning

Distributed Learning

17

Supervised Learning

• Given: a set of <input X, output Y> pairs• Goal: given an unseen input, predict the corresponding

output• There are two kinds of outputs

– Categorical: classification problem• Binary classification vs. Multiclass classification

– Real values: regression problem

• Example:1. Binary classification: the sensor information, output:

whether such sensor is broken (INTEL) 2. Multi-class classification: Lyric of a song, output: the mood:

happy/sad/surprise/angry (CHT)3. Regression: the weather/traffic/air condition, output: PM

2.5 value (Microsoft Research Asia)18

Cost-sensitive Learning• A classification task with non-uniform cost for

different types of classification error.

• Goal: To predict the class C* that minimizes the expected cost rather than the misclassification rate

• An example cost matrix L : medical diagnosis

LjkActual Cancer Actual Normal

Predict Cancer 0 1

Predict Normal 10000 0

19

A variety of ML Scenarios

Supervised Learning

Classification & Regression

Multi-label learning

Multi-instance learning

Cost-sensitive leering

Semi-supervised learning

Active learning

Unsupervised Learning

Clustering

Learning Data Distribution

Pattern Learning

Reinforcement learning

Variations

Transfer learning

OnlineLearning

Distributed Learning

20

Unsupervised Learning• Learning without teachers (presumably harder than

supervised learning)– Learning “what normally happens”

– E.g. babies learn their first language (unsupervised) vs. how people learn their 2nd language (supervised).

• Given: a bunch of input X (there is no output Y)

• Goal: depending on the tasks, for example– Estimate P(X) then we can find augmax P(X) Bayesian

– Finding P(X2|X1) we can know the dependency between inputs Association Rule, Probabilistic Graphical Model

– Finding Sim(X1,X2) then we can group similar X’s clustering

21

A variety of ML Scenarios

Supervised Learning

Classification & Regression

Multi-label learning

Multi-instance learning

Cost-sensitive leering

Semi-supervised learning

Active learning

Unsupervised Learning

Clustering

Learning Data Distribution

Pattern Learning

Reinforcement learning

Variations

Transfer learning

OnlineLearning

Distributed Learning

22

Reinforcement Learning (RL)• RL is a “decision

making” process.– How an agent should

make decision to maximize the long-term rewards

• RL is associated with a sequence of states X and actions Y (i.e. think about Markov Decision Process) with certain “rewards”.

• It’s goal is to find an optimal policy to guide the decision.

Figure from Mark Chang

23

• 1st Stage:天下棋手為我師 (Supervised Learning)– Data: 過去棋譜– Learning: f(X)=Y, X: 盤面, Y:next move– Results: AI can play Go now, but not an expert

• 2nd Stage:超越過去的自己(Reinforcement Learning)– Data: generating from playing with 1st Stage AI– Learning: Observation盤面, rewardif win,

action next move

AlphaGo: SL+RL

24

Key: Finding which ML Scenario Best Suits the Current Task

• CTR: the ratio that users click into a displayed advertisement

• It looks like a regression task, but indeed it is not.– CTR= #click/#view 3/10 =? 300/1000

– Better solution: treating it as a binary classification, transfer 3/10 to 3 positive cases and 7 negative cases

Case Study: Click Through Rate Prediction (KDD Cup 2012)

26

FAQ: I have a lot of Data, but they are not labelled

• This is by far the most common question I have been asked.

• My honest answer: try to get them labelled because you anyway need the ground truth for evaluation.

• In several cases, labelling are too costly

– Semi-supervised solution

– Transfer learning (using labelled data in other domains)

– Active learning (query a subset of labels)

28

A variety of ML Scenarios

Supervised Learning

Classification & Regression

Multi-label learning

Multi-instance learning

Cost-sensitive leering

Semi-supervised learning

Active learning

Unsupervised Learning

Clustering

Learning Data Distribution

Pattern Learning

Reinforcement learning

Variations

Transfer learning

OnlineLearning

Distributed Learning

29

Semi-supervised Learning

• Only a small portion of data are annotated (usually due to high annotation cost)

• Leverage the unlabeled data for better performance

[Zhu

20

08

]

30

Transfer Learning

Learning Process of

Traditional ML

Learning Process of

Transfer Learning

training items

Learning System Learning System

training items

Learning System

Learning SystemKnowledge

Improving a learning task via incorporating knowledge from learning tasks in other domains (with different feature space and data distribution).

More labeled data

Fewer labeled data

31Pan et al.

A Label for that Example

Request for the Label of another Example

A Label for that Example

Request for the Label of an Example

Active Learning

Unlabeled Data

. . .

Algorithm outputs a classifier

Learning Algorithm

Expert / Oracle

• Achieves better learning with fewer labeled training data via actively selecting a subset of unlabeled data to be annotated

32

OK, I have identified a suitable ML scenario, but my model still doesn’t work

The ML Diagnose Tree

Is it ML?

NoTry othersolutions

Correct ML Scenario?

NoTry Other Scenario

Have you identified a

suitable model?’

Yes Do you have enough

data?

Yes Is your model too

complicated?

Right Feature?

Yes Performed feature

engineering?

Suitable Evaluation Metrics?

Proper validation set?

34

Did you choose a proper model?

• A proper model considers – The size of data

• Small data linear (or simpler) model • Large data linear model or non-linear model

– The sparsity of data• Sparse data more tricks to perform better and faster • Dense data requires light algorithm that consumes less

memory

– The balance condition of data• Imbalanced data special treatment for minority class

– The quality of data (whether there are noise, missing values, etc)• Some loss function (e.g. 0/1 loss or L2) are more robust to

noise than others (e.g. hinge loss or exponential loss)35

Do you have enough data to train the given model?

• Draw the learning curve to understand whether your data is sufficient

Data Size

Accuracy Enough Data

Need more data!!

36

Is your model too complicated?

• Increasing the model complexity and check for the training/testing performance

– E.g. increasing the latent dimension k in matrix factorization

Model Complexity37

FAQ: How to avoid overfitting?

– Occam's Razor: simpler model first

– Regularization: a way to constraint the complexity of the model

– Carefully design your experiment (see later)

– Being aware of the drifting phenomenon

Overfitting: the Biggest Enemy of ML !!

38

I still cannot do well given a reasonable model is identified

The ML Diagnose Tree

Is it ML?

NoTry othersolutions

Correct ML Scenario?

NoTry Other Scenario

Have you identified a

suitable model?’

Yes Do you have enough

data?

Yes Is your model too

complicated?

Right Feature?

Yes Performed feature

engineering?

Suitable Evaluation Metrics?

Proper validation set?

40

• Have you identified all potentially useful features (X)?

Diagnose : Quality of Features X

41

• Use domain knowledge and human judgement to determine which features to obtain for training.

• The rule of thumb: If you don’t know whether a feature is useful, then try it!!

– Human judgement can be misleading

Which Information shall be extracted as Features?

Let ML algorithms (e.g. feature selection) determine whether certain information is useful !!

42

Feature Engineering

• Feature engineering is arguably the best strategy to improve the performance.

• The goal is to explicitly reveal important information to the model

– domain knowledge might or might not be useful

• Original features different encoding of the features combined features

43

FAQ: What shall I do if I absolutely need to boost the accuracy

• Combining a variety of models:

– If the models are diverse enough with similar performance, there is a high chance the performance will be boosted.

• Additional validation data are needed to learn the ensemble weights.

44

I am really happy that my ML model FINALLY works so well!!

The ML Diagnose Tree

Is it ML?

NoTry othersolutions

Correct ML Scenario?

NoTry Other Scenario

Have you identified a

suitable model?’

Yes Do you have enough

data?

Yes Is your model too

complicated?

Right Feature?

Yes Performed feature

engineering?

Suitable Evaluation Metrics?

Proper validation set?

46

Is your model evaluated correctly?

• Accuracy is NOT always the best way to evaluate a machine learning model

47

Case Study: An 99.999% accurate system in Detecting Malicious Personnel

• Randomly picked person not likely a terrorist.

• Thus, a model that always guess ‘non-terrorist’ will achieve very high accuracy

– But it is useless !!

• “Area under ROC Curve” (or AUC) is generally used to evaluate such system.

48

Have the Right Evaluation Data Been Used?

• We normally divide labeled data (or ground truth data) into three parts: training, validation (or heldout), and testing

• Performance on training data is obviously biasedsince the model was constructed to fit this data.

– Accuracy must be evaluated on an independent (usually disjoint) test set.

– Cannot peak the test set labels!!

• Use validation set to adjust hyper-parameters

How the validation set is chosen can affect the performance of the model!!

Training Validation Testing

49

Case Study: KDD Cup 2008 (identify potential cancer patients)

• Training set: a set of positive and negative instances (each contains 118 features)– Each positive patient contain a set of negative

instances (i.e. an ROI in the X-ray) and at least one positive instances.

– ALL instances in a negative patient are negative– It’s a multi-instance classification problem.

• Random division for CV: – training: 90%, testing: 72% peeking the testing

• Patient-based CV:– training: 80%, testing: 77%

50

1. Determine whether it shall be model as an ML task

2. Model the task into a suitable ML scenario

3. Given the known scenario, determine a suitable model

4. Feature Engineering

5. Parameter learning

6. Evaluation (e.g. cross validation)

7. Apply to unseen data

Take Home Point: How to Solve My Task Using ML?

51

Final Remark: The (weak) AI Era Has Arrived

Big Data

Knowledge Representation

Machine Learning

AI

科學人2015

52

So, What’s Next for ML & AI?-My three bold predictions about the future

Halle Tecco

On AI Revolution

Machine will take over some non-trivial jobs that are used to be conducted by human beings

– Those jobs (unfortunately?) include data analytics

54

On Decision Making

• The decision makers will gradually shift from humans to machines

55科學人2015

On the Next of Machine Learning

• The evolution of human intelligence:

memorizing

learning

discovering

• Machine Discovery (Shou-De Lin, Fall 2016, NTU CSIE)

Machine

Machine

Machine

56

Thank you for your attention !!