active learning and human-in-the-loop
TRANSCRIPT
![Page 1: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/1.jpg)
Lukas Biewald
![Page 2: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/2.jpg)
The Effect of Better Algorithms
Naïve Bayes Maximum Entropy SVM0%
5%
10%
15%
20%
25%
Classifier Error Rate
Active Semi-Supervised Learning for Improving Word Alignment(Vamshi ACL ’10)
Real World Data
2
![Page 3: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/3.jpg)
The Effect of Better Features
Unigrams Bigrams Unigrams+Bigrams0%
5%
10%
15%
20%
25%
30%
Classifier Error Rate
3
![Page 4: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/4.jpg)
The Effect of More Data
Active Semi-Supervised Learning for Improving Word Alignment(Vamshi ACL ’10)
Real World Data
N 2N 4N0%
2%
4%
6%
8%
10%
12%
14%
Classifier Error Rate
4
![Page 5: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/5.jpg)
The Effect of Cleaner Data
90% Accurate Data 95% Accurate Data 100% Accurate Data0%
2%
4%
6%
8%
10%
12%
14%
Classifier Error Rate
5
![Page 6: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/6.jpg)
6
Where Do Data Scientists Spend Their Time?
Source: CrowdFlower Data Science Report 2015
![Page 7: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/7.jpg)
7
CrowdFlower Data Enrichment Platform
![Page 8: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/8.jpg)
8
Color Data
![Page 9: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/9.jpg)
9
![Page 10: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/10.jpg)
10
![Page 11: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/11.jpg)
11
![Page 12: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/12.jpg)
12
![Page 13: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/13.jpg)
13
![Page 14: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/14.jpg)
14
![Page 15: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/15.jpg)
Apple Watch
15
![Page 16: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/16.jpg)
Apple Watch
16
![Page 17: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/17.jpg)
Apple Watch
17
![Page 18: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/18.jpg)
Apple Watch
18
![Page 19: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/19.jpg)
Collecting the Same Data Over and Over
19
![Page 20: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/20.jpg)
20
Open Data
![Page 21: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/21.jpg)
21
Make Your Data Public Setting
![Page 22: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/22.jpg)
22
Data for Everyone
![Page 23: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/23.jpg)
23
Data For Everyone Library
![Page 24: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/24.jpg)
24
Data for Everyone
![Page 25: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/25.jpg)
25
Data For Everyone
![Page 26: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/26.jpg)
Open Data API
26
![Page 27: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/27.jpg)
URL Categorization
27
![Page 28: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/28.jpg)
Categorize URLs
28
![Page 29: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/29.jpg)
Record Data
29
![Page 30: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/30.jpg)
Extracting Names and Titles
30
![Page 31: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/31.jpg)
Summarization
31
![Page 32: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/32.jpg)
Is an Image Funny?
32
![Page 33: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/33.jpg)
Classifying Medical Images
33
![Page 34: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/34.jpg)
Attributes of People
34
![Page 35: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/35.jpg)
35
![Page 36: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/36.jpg)
Kaggle accuracy
Baseline 12-May 13-May 14-May 15-May0%
10%
20%
30%
40%
50%
60%
70%
Accuracy of Best Performing Model
Accu
racy
36
![Page 37: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/37.jpg)
Kaggle accuracy over time
12-M
ay
14-M
ay
16-M
ay
18-M
ay
20-M
ay
22-M
ay
24-M
ay
26-M
ay
28-M
ay
30-M
ay1-J
un3-J
un5-J
un7-J
un9-J
un
11-Ju
n
13-Ju
n
15-Ju
n
17-Ju
n
19-Ju
n
21-Ju
n
23-Ju
n
25-Ju
n
27-Ju
n
29-Ju
n1-J
ul3-J
ul5-J
ul0%
10%
20%
30%
40%
50%
60%
70%
80%
Accuracy of the Best Performing Model
Accu
racy
37
![Page 38: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/38.jpg)
Kaggle Participation
12-M
ay
14-M
ay
16-M
ay
18-M
ay
20-M
ay
22-M
ay
24-M
ay
26-M
ay
28-M
ay
30-M
ay1-J
un3-J
un5-J
un7-J
un9-J
un
11-Ju
n
13-Ju
n
15-Ju
n
17-Ju
n
19-Ju
n
21-Ju
n
23-Ju
n
25-Ju
n
27-Ju
n
29-Ju
n0
200
400
600
800
1000
1200
1400
Number of Participating Teams
38
![Page 39: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/39.jpg)
AIClassifier OutputConfident
Human in the Loop
39
![Page 40: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/40.jpg)
Human in the Loop
Confident OutputAIClassifier Not Confident Human
Annotation
40
![Page 41: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/41.jpg)
Human in the Loop
Confident OutputAIClassifier
Active Learning
Not Confident Human Annotation
41
![Page 42: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/42.jpg)
Human in the Loop
Confident OutputAIClassifier
Active Learning
Not Confident Human Annotation
42
![Page 43: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/43.jpg)
Active Learning
From hunch.net active learning tutorial ICML ‘0943
![Page 44: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/44.jpg)
Active Learning Accuracy Improvement
44
![Page 45: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/45.jpg)
Google Cars Miles Per Disengage
45
![Page 46: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/46.jpg)
Adaptive Cruise Control
Image source: ExtremeTech
46
![Page 47: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/47.jpg)
Advanced Chess
Image source: Computer Chess
47
![Page 48: Active Learning and Human-in-the-Loop](https://reader036.vdocuments.net/reader036/viewer/2022070514/587f30211a28ab121d8b5fe3/html5/thumbnails/48.jpg)
AlphaGo
48