active learning from theory to practice · interactive machine-human data collection and...
Post on 06-Apr-2020
7 Views
Preview:
TRANSCRIPT
Active Learning from Theory to Practice
Robert Nowak UW-Madisonrdnowak@wisc.edu
Steve Hanneke Toyota Technological Institute at Chicagosteve.hanneke@gmail.com
Tutorial Outline
Part 1: Introduction to Active Learning (Rob)
Part 2: Theory of Active Learning (Steve)
Part 3: Advanced Topics and Open Problems (Steve)
Part 4: Nonparametric Active Learning (Rob)
slides: http://nowak.ece.wisc.edu/ActiveML.html
Conventional (Passive) Machine Learning
unlabeled raw data
human labeling
machine learning
predictive model
labeled data
dog
boat
…
millions of labeled images1000’s of human hours trained on more texts than a
human could read in a lifetime
learned by playing more games than even my teenage soncould stomach !
?
Can we train machines with less labeled data and less human supervision?
Active Machine Learning
data selection algorithm
unlabeled raw data
human labeling
machine learning
predictive model
labeled data
Goal: machine automatically and adaptively selects most informative data for labeling
Motivating Application
provides labels to machine learner (several minutes / EHR)
EHR feature 1
EH
R fe
atur
e 2
best linear classifier
Non-adaptive strategy: Label a random sample
Active strategy: Label a sample near best decision boundary based on labels seen so far
# labels
error rate ✏
active learning finds optimal classifier with much less human supervision!
Active Learning
11000 patient records 8000 positive 3000 negative
6182 Numerical Features icd9 codes lab tests patient data
Classification task: cataracts or healthy
less than half as many labeled examples needed by active learning
active learning passive learning
Active Logistic Regression
nextml.org
Active learning to optimize crowdsourcing and rating in New Yorker Cartoon Caption Contest
Actively learning user’s beer preferences
Principles of Active Learning
What and Where Information
x<latexit sha1_base64="Whvr4b/LFT0obOEEGu5nSiC2KTo=">AAAEO3icbZNNb9MwGMezlJdRKHRw5GLRIiEOVTMO7IK0qZvUQzW1Zd06NWVyHGeN6tiR7UAjK9+LC1+CGxcuHECIK3ec1gnLNkvO89fveXP84sUkFLLb/bZl1+7cvXd/+0H94aPG4yfNnaengiUc4QlihPGpBwUmIcUTGUqCpzHHMPIIPvOWvdx/9hFzETJ6ItMYzyN4ScMgRFBqdLFjjxoNl+JPcoEZx5GKOYszNdRfJsI8JKu4CY4imKnB2lQ8xmbqxIiKFzHOCIE8zVSvlJUIHwch3XRUh/91JUZPyJeZGm9so153dRpwHeVGUC68ADg6YY2ODPLAUYGGJRpmJnFUolERNS7RuEB9pVwECehnBTkw5KAkPUN6JZkaMi3JuSHn2S2VzYomhkxKMjBkUBIvUa7HiC/SSBuVlA60UOjD66J4FKleoY/Vsd6ufC8R00dHfV3zfXa1jHuIiYRli1WlxSpvsUjjBaYxppDI9J3T1UMfAGUh9TGVwBUBaLdX7fZFs9XtdNcD3BSOES3LjOFF86vrM5REugoiUIiZ043lXEEuQ0Sw7pwIHEO0hJd4piWFERZztb77GXipiQ8CxvXUq1jTqxkKRiL/Cx2Zn6u47svhbb5ZIoO9uQppnEhM0aZRkBAgGcgfEvBDjpEkqRYQcX1bEUALyCGS+rnV9SY413/5pjjd7ThvOs5ot7W/Z7Zj23puvbBeWY711tq3+tbQmljI/mx/t3/av2pfaj9qv2t/NqH2lsl5ZlVG7e8/HPJrjA==</latexit>
p(y|x)<latexit sha1_base64="9s4p/1EEotGoHDdhpiNeEwbD0aU=">AAAEQHicbZNNb9MwGMezlJdRKOvgyMWiRRocpmYc2AVpUzeph2pqy7p1WsrkOM4S1bEj24FGJh+NCx+BG2cuHECIKyec1g3LNkvO89fvefNL7CUkErLT+bZm1+7cvXd//UH94aPG443m5pMTwVKO8BgxwvjEgwKTiOKxjCTBk4RjGHsEn3qzbuE//YC5iBg9llmCpzG8pFEQISg1uti0TxoNl+KPMsSM41glnCW5GugvE1ERklfcBMcxzFV/YSoeY3N1bETFixhnhECe5apbykqEj4OILjuqg/+6EqMn5LNcjZa2Ua+7Og24jnJjKEMvAI5OWKBDgzxwuEKDEg1ykzgs0XAVNSrRaIV6SrkIEtDLV2TfkP2SdA3plmRiyKQkZ4ac5bdUNisaGzIuSd+Qfkm8VLkeI77IYm1UWjpQqND7V6vicay6K32kjvRxFWeJmL466uua7/KrZdwDTCQsW8wrLeZFizBLQkwTTCGR2Vuno4e+AMoi6mMqgSsC0G4nW9mn+ct2+6LZ6mx3FgPcFI4RLcuMwUXzq+szlMa6FCJQiHOnk8ipglxGiGDdPhU4gWgGL/G5lhTGWEzV4gHk4IUmPggY11MvZUGvZigYi2IrOrK4XHHdV8DbfOepDHanKqJJKjFFy0ZBSoBkoHhNwI84RpJkWkDE9S+LAAohh0jqN1fXh+Bc3/JNcbKz7bzedoY7rb1dcxzr1jPrubVlOdYba8/qWQNrbCH7s/3d/mn/qn2p/aj9rv1ZhtprJuepVRm1v/8AyrttdA==</latexit>
Density estimation: What is p(y|x)?Classification: Where is p(y|x) > 0?
<latexit sha1_base64="6/jpTwqMPj9rFfECacA215Xkksk=">AAAEn3icbZNvT9QwGMAHdyqenoK+1BdVMEFfkA1fSEwU8CCSeCEHcnKEnaTrOtbQtUvb6S11H8sv4ju/jd2uN29Ak+558nv+dU+fBiklUrnu34XFVvvO3XtL9zsPHnYfPV5eefJN8kwgPESccjEKoMSUMDxURFE8SgWGSUDxaXDVK+2nP7CQhLMTlad4nMBLRiKCoDLoYmXxd7frM/xTxZgLnOhU8LTQA/PlkpQuRcNMcZLAQvcr0bBYWegTqzSsiAtOKRR5oXu12vAIcUTYtKLe+683fMyG4qrQx1PZ7XR8EwZ8T/sJVHEQAc8EVGjfogDsz9CgRoPCBh7V6GjmdVyj4xk60NpHkIKDYkZ2LdmtSc+SXk1GloxqcmbJWXFLZnuioSXDmvQt6dckyLQfcBrKPDFCZ7UBxRp9fzNLniS6N9MP9aFpV9lLxM3VsdDk/FrMp/H3MFWwLjFplJiUJeI8jTFLMYNU5R881yxzAYwTFmKmgC8jsIeZmZocYKlIUk3Ye3AaQwWIBGvpev5r8nptGwDfn49TeKKqQTb3G5r5oFDKekLLeCzwXIKP7tp2cbG86m641QI3Fc8qq45dg4vlP37IUZaYiqgscO65qRprKBRBFJu/yyROIbqCl/jcqAwmWI51dawCvDIkBBEXZpsTV3Q+QsNElp0ynuXsyOu2Et5mO89UtDXWhKWZwgxNC0UZBYqD8rGCkAiMFM2NApEwLwIBFEMBkTJPumOa4F3/5ZvKt80N7+2Gd7S5urNl27HkPHNeOuuO57xzdpwDZ+AMHdR63vrU+tLqt1+0P7cP24Op6+KCjXnqNFb77B/mhYr5</latexit>
x<latexit sha1_base64="Whvr4b/LFT0obOEEGu5nSiC2KTo=">AAAEO3icbZNNb9MwGMezlJdRKHRw5GLRIiEOVTMO7IK0qZvUQzW1Zd06NWVyHGeN6tiR7UAjK9+LC1+CGxcuHECIK3ec1gnLNkvO89fveXP84sUkFLLb/bZl1+7cvXd/+0H94aPG4yfNnaengiUc4QlihPGpBwUmIcUTGUqCpzHHMPIIPvOWvdx/9hFzETJ6ItMYzyN4ScMgRFBqdLFjjxoNl+JPcoEZx5GKOYszNdRfJsI8JKu4CY4imKnB2lQ8xmbqxIiKFzHOCIE8zVSvlJUIHwch3XRUh/91JUZPyJeZGm9so153dRpwHeVGUC68ADg6YY2ODPLAUYGGJRpmJnFUolERNS7RuEB9pVwECehnBTkw5KAkPUN6JZkaMi3JuSHn2S2VzYomhkxKMjBkUBIvUa7HiC/SSBuVlA60UOjD66J4FKleoY/Vsd6ufC8R00dHfV3zfXa1jHuIiYRli1WlxSpvsUjjBaYxppDI9J3T1UMfAGUh9TGVwBUBaLdX7fZFs9XtdNcD3BSOES3LjOFF86vrM5REugoiUIiZ043lXEEuQ0Sw7pwIHEO0hJd4piWFERZztb77GXipiQ8CxvXUq1jTqxkKRiL/Cx2Zn6u47svhbb5ZIoO9uQppnEhM0aZRkBAgGcgfEvBDjpEkqRYQcX1bEUALyCGS+rnV9SY413/5pjjd7ThvOs5ot7W/Z7Zj23puvbBeWY711tq3+tbQmljI/mx/t3/av2pfaj9qv2t/NqH2lsl5ZlVG7e8/HPJrjA==</latexit>
p(x)<latexit sha1_base64="/13CxrgjkO8jDL8OQDbBnD1TWw4=">AAAEPnicbZNNb9MwGMezlJdRKHRw5GLRIg0OUzMO7IK0qZvUQzW1pd06NaVyHGeN6tiR7UArK5+MC5+BG0cuHECIK0ec1gnLNkvO89fveXP84sUkFLLV+rZlV+7cvXd/+0H14aPa4yf1nadngiUc4RFihPGxBwUmIcUjGUqCxzHHMPIIPvcW7cx//hFzETI6lKsYTyN4ScMgRFBqNNuxh7WaS/EnOceM40jFnMWp6ukvE2EWkpbcBEcRTFV3bUoeY1M1NKLkRYwzQiBfpapdyFKEj4OQbjqq4/+6FKMn5ItUDTa2Vq26Og24jnIjKOdeABydsEYnBnngJEe9AvVSk9gvUD+PGhRokKOOUi6CBHTSnBwZclSQtiHtgowNGRfkwpCL9JbKZkUjQ0YF6RrSLYiXKNdjxBerSBuVFA40V+jD67x4FKl2rk/Vqd6ubC8R00dHfV3zfXq1jHuMiYRFi2WpxTJrMV/Fc0xjTCGRq3dOSw99AJSF1MdUAlcEoNmMd5evms1ZvdHaa60HuCkcIxqWGb1Z/avrM5REuhAiUIiJ04rlVEEuQ0Swbp4IHEO0gJd4oiWFERZTtb7+KXipiQ8CxvXUC1nTqxkKRiL7ER2ZHa247svgbb5JIoODqQppnEhM0aZRkBAgGcjeEvBDjpEkKy0g4vrCIoDmkEMk9Yur6k1wrv/yTXG2v+e82XP6+43DA7Md29Zz64W1aznWW+vQ6lg9a2Qh+7P93f5p/6p8qfyo/K782YTaWybnmVUalb//AEy9bGs=</latexit>Density estimation: What is p(x)?
Clustering: Where is p(x) > ✏ ?<latexit sha1_base64="tkVbhFrcHREGGL/VYApkj5cW6m0=">AAAEoHicbZPxTxMxFMcPNhWnU9AfjUkDmKAxZIc/SEwUyCDZDxM3ZDDkBun13riGXntpe7qluX/LP8Tf/G/sbd3JgCa9fvN57/X1Xl/DlFGlG42/C4uV6oOHj5Ye1548rT97vrzy4kSJTBLoEcGE7IdYAaMceppqBv1UAk5CBqfhdbOwn/4Eqajgx3qcwiDBV5wOKcHaosuVxd/1esDhl45BSEhMKkWam479CkULl3zOzCBJcG7ak2XO4tbcHDsxZyVCCsawHOemWco5jwiGlE8zmv3/es7HTiyvc3M0Xeu1WmDDUOCbIME6DofItwETdOBQiA5mqFOiTu4CuyXqzryOSnQ0Qy1jAoIZauUzsufIXkmajjRL0nekX5IzR87ye3Z2J+o50itJ25F2ScLMBKFgkRondjFZaSCxIRfvZpsniWnO9KE5tOUqakmEvToe2T2/5ze3CfaBaVymGM2lGBUp4nEaA0+BY6bHn/2GHfYCuKA8Aq5RoIZoH7jtmjECpWky6bBP6DTGGlGF1tON0dv1HYSC4GaUhpGetLG93ch2B8uUBkn5VREJEsrQLwGkijLB14P3O/nl8lpjszEZ6K7wnVjz3OhcLv8JIkGyxCYlDCt17jdSPTBYakoY2N/LFKSYXOMrOLeS4wTUwExOlqM3lkRoKKSd9tATejPC4EQVpbKeRfOo27YC3mc7z/Rwe2AoTzMNnEwTDTOGtEDFa0URlUA0G1uBibRPgiASY4mJrZGq2SL4t3/5rjjZ2vQ/bPrdrbXdbVeOJe+Vt+pteL730dv1Wl7H63mk8rrSrLQrX6ur1Vb1W7U7dV1ccDEvvblR/fEPTTqLOw==</latexit>
E[y|x]<latexit sha1_base64="iJnzcTJQm4dWqYlWoJGCWInHwls=">AAAEQXicbZNNb9MwGMezhJdRKHRw5GKxIiEOU7Md2AVpU1eph2nqxrp1asrkOM4S1bEj24FFJl+NC9+AG3cuHECIKxec1gnLNkvO89fvefNL7KckFrLX+7ZiO3fu3ru/+qD18FH78ZPO2tMTwTKO8BgxwvjEhwKTmOKxjCXBk5RjmPgEn/rzfuk//YC5iBk9lnmKZwm8oHEYIyg1Ol+zT9ttj+KPMsKM40SlnKWFGukvE3EZUjTcBCcJLNT+wjQ8xhbq2IiGFzHOCIE8L1S/lo2IAIcxXXZUe/91I0ZPyOeFOlradqvl6TTgucpLoIz8ELg6YYEGBvlgUKFRjUaFSTys0WEVdVSjowoNlfIQJGBYVGTXkN2a9A3p12RiyKQmZ4acFbdUNisaGzKuyb4h+zXxM+X5jAQiT7RRWe1AkULvX1fFk0T1K32gDvRxlWeJmL46Guia74qrZbw9TCSsW1w2WlyWLaI8jTBNMYVE5m/dnh76AiiLaYCpBJ4IQbfrDab5p8tZt3veWe9t9BYD3BSuEeuWGaPzzlcvYChLdC1EoBBTt5fKmYJcxohg3T8TOIVoDi/wVEsKEyxmavECCvBSkwCEjOup17KgVzMUTES5Fx1Z3q647ivhbb5pJsPtmYppmklM0bJRmBEgGSifEwhijpEkuRYQcf3PIoAiyCGS+tG19CG417d8U5xsbrhbG+7h5vrOtjmOVeu59cJ6ZbnWG2vHGloja2wh+7P93f5p/3K+OD+c386fZai9YnKeWY3h/P0HtwluFg==</latexit>
x<latexit sha1_base64="Whvr4b/LFT0obOEEGu5nSiC2KTo=">AAAEO3icbZNNb9MwGMezlJdRKHRw5GLRIiEOVTMO7IK0qZvUQzW1Zd06NWVyHGeN6tiR7UAjK9+LC1+CGxcuHECIK3ec1gnLNkvO89fveXP84sUkFLLb/bZl1+7cvXd/+0H94aPG4yfNnaengiUc4QlihPGpBwUmIcUTGUqCpzHHMPIIPvOWvdx/9hFzETJ6ItMYzyN4ScMgRFBqdLFjjxoNl+JPcoEZx5GKOYszNdRfJsI8JKu4CY4imKnB2lQ8xmbqxIiKFzHOCIE8zVSvlJUIHwch3XRUh/91JUZPyJeZGm9so153dRpwHeVGUC68ADg6YY2ODPLAUYGGJRpmJnFUolERNS7RuEB9pVwECehnBTkw5KAkPUN6JZkaMi3JuSHn2S2VzYomhkxKMjBkUBIvUa7HiC/SSBuVlA60UOjD66J4FKleoY/Vsd6ufC8R00dHfV3zfXa1jHuIiYRli1WlxSpvsUjjBaYxppDI9J3T1UMfAGUh9TGVwBUBaLdX7fZFs9XtdNcD3BSOES3LjOFF86vrM5REugoiUIiZ043lXEEuQ0Sw7pwIHEO0hJd4piWFERZztb77GXipiQ8CxvXUq1jTqxkKRiL/Cx2Zn6u47svhbb5ZIoO9uQppnEhM0aZRkBAgGcgfEvBDjpEkqRYQcX1bEUALyCGS+rnV9SY413/5pjjd7ThvOs5ot7W/Z7Zj23puvbBeWY711tq3+tbQmljI/mx/t3/av2pfaj9qv2t/NqH2lsl5ZlVG7e8/HPJrjA==</latexit>
Function estimation: What is E[y|x]?Bandit optimization: Where is maxx E[y|x]?
<latexit sha1_base64="uzqlJL2aZvP3o4CtSisy/1l5kxc=">AAAErXicbZNvT9QwHMcHdyqenoA+9EkjmBge4A0fSEyM4IHhASEHcnB4Oy5d13EN/bO0nd6se3W+A5/5bux2vcmAJl2/+fz+db+2YUKJ0p3O34XFRvPBw0dLj1tPnrafLa+sPj9TIpUI95GgQg5CqDAlHPc10RQPEokhCyk+D6+7hf38O5aKCH6qswSPGLziJCYIaovGq4u/2+2A4x96goXEzCRSJLnp2a9QpHDJa2aKGYO5OSyXmsWtuTl1omZFQgpKocxy061kzSPCMeGzimbvv6752AnldW5OZmu71QpsGAh8EzCoJ2EMfBtQon2HQrA/R70K9XIXeFyh47nXSYVO5ujAmABBCg7yOdl1ZLciXUe6FRk4MqjIhSMX+T2Z3Y76jvQrcujIYUXC1AShoJHKmF1MWhnQxKDLjXlyxkx3ro/MkW1X0Usk7NHxyOb8mt9ME+xhqmFVYlorMS1KTLJkgnmCOaQ6++h37LAHwAXhEeYaBCoGX1KOimMDWGnCyiv2AZxPoAZEgfVgf5j9mo7WPwEQBDcjNZ7q8irbE45y89nuj2ggEpuD/KyyYIlnaRicjqegypa3xitrnc1OOcBd4Tux5rnRG6/8CSKBUmarIwqVGvqdRI8MlJogiu2/pgonEF3DKzy0kkOG1ciUW8zBa0siEAtpp919SW9GGMhU0TfrWdwkddtWwPtsw1TH2yNDeJJqzNGsUJxSoAUoni6IiMRI08wKiKR9HwigCZQQafvAiyb4t3/5rjjb2vTfbfrHW2s7264dS95L75X3xvO9996Od+D1vL6HGhuNXuOi8a35ttlvBs3Lmevigot54dVG8+ofEeuRPQ==</latexit>
Active learning is more efficient than passive learning for localized “where” information
Version-Space (VS) Active Learning
initialize VS: H = all models/hypotheses
while (stopping-criterion) not met
1. sample at random from available dataset
2. label only those samples that distinguish models in H
3. reduce H by removing all models inconsistent with labels
output: best model in final H
Model Space
Select examples to label
Labeled Data
1. Integrating data from disparate sources
2. Optimizing human resources in the data collection and processing chain
3. Reasoning and planning in highly uncertain environments
This proposal outlines a research agenda for developing the mathematical and computational foundationsof large-scale interactive data analytics. Our aim is to focus on two interconnected components necessary tooptimize information processing systems:
• A1. Sequential and adaptive sensing, data-collection and decision-making.
• A2. Interactive Machine-Human Data Collection and Co-Processing.
• A3. Cloud-based platforms for large-scale interactive data analytics.
These challenges will be studied and will influence the development of new mathematical theory andcomputational tools.
The State of Interactive Machine Learning Most supervised machine learning algorithms designed us-ing a pre-selected set of “training data.” This approach is referred to as passive learning. Gathering andannotating training datasets can be costly, placing limits on their size and complexity. For example, in natu-ral language processing or scene recognition tasks, human experts are usually required to label the trainingdata. Active learning integrates the processes of training data collection and algorithm design. Rather thanworking with a pre-selected pool of training data, the selection of data for labeling or annotation is guided bythe machine as it learns the task at hand, focusing data gathering toward examples that are most ambiguousgiven previously collected training data.
model space
questions/queries
“training data”
Human Judgementsanalysis, labeling,
annotation, comparison
active learning
Figure 2: Active learning closes the loop. The machine learner only requests human assistance/labeling for data thatdoes not fit its current learned model. As the machines prediction performance improves, it requires less and lessassistance from human experts.
Active learning addresses an inherently practical problem: train a machine to perform a particular taskwhile using as little input from human experts as possible. Unfortunately, the most impressive accomplish-ments in this field are theoretical in nature and almost useless in practice. Most existing active learningmethods suffer from at least one of the following problems that have limited success in practice:
2
machine
1. Integrating data from disparate sources
2. Optimizing human resources in the data collection and processing chain
3. Reasoning and planning in highly uncertain environments
This proposal outlines a research agenda for developing the mathematical and computational foundationsof large-scale interactive data analytics. Our aim is to focus on two interconnected components necessary tooptimize information processing systems:
• A1. Sequential and adaptive sensing, data-collection and decision-making.
• A2. Interactive Machine-Human Data Collection and Co-Processing.
• A3. Cloud-based platforms for large-scale interactive data analytics.
These challenges will be studied and will influence the development of new mathematical theory andcomputational tools.
The State of Interactive Machine Learning Most supervised machine learning algorithms designed us-ing a pre-selected set of “training data.” This approach is referred to as passive learning. Gathering andannotating training datasets can be costly, placing limits on their size and complexity. For example, in natu-ral language processing or scene recognition tasks, human experts are usually required to label the trainingdata. Active learning integrates the processes of training data collection and algorithm design. Rather thanworking with a pre-selected pool of training data, the selection of data for labeling or annotation is guided bythe machine as it learns the task at hand, focusing data gathering toward examples that are most ambiguousgiven previously collected training data.
model space
questions/queries
“training data”
Human Judgementsanalysis, labeling,
annotation, comparison
active learning
Figure 2: Active learning closes the loop. The machine learner only requests human assistance/labeling for data thatdoes not fit its current learned model. As the machines prediction performance improves, it requires less and lessassistance from human experts.
Active learning addresses an inherently practical problem: train a machine to perform a particular taskwhile using as little input from human experts as possible. Unfortunately, the most impressive accomplish-ments in this field are theoretical in nature and almost useless in practice. Most existing active learningmethods suffer from at least one of the following problems that have limited success in practice:
2
machine
active learning
questions& tests
answers & decisions
models data data
human judgments
models
questions& tests
answers & decisions
human judgments
Figure 2: Active learning closes the loop. The conventional, passive learning approach is shown on the left. A humanexpert supplies a set of labeled training data to the machine without any interaction, and the machine selects the bestmodel based on these data. In the active learning model, on the right, the learning process is continually refined as themachine’s prediction performance improves. The machine uses information gleaned from previous human inputs torule out incompatible models, and then queries the human for information about only the most ambiguous data. Thisreduces the burden on human experts and accelerates the learning process.
and many of its relaxations computationally intractable (the first problem identified above). In the contextof Figure 3, each node may have a very large number of possible sensing actions (children) associated withit, corresponding to the multitude not only of available sensors, but also their various operating modes.Similarly, the space of possible target identities and locations is generally enormous. Furthermore, MDPformulations assume critical information about the cost or reward function is known a priori, touching onthe second issue above, while in active learning such prior knowledge is unavailable.
. . .
. . .
. . .
...
�
sensing space:
target identificationand localization
Figure 3: Optimal solutions to active learning and adaptive sensing can be represented in terms of a search tree. Inlarge-scale, complex tasks, the size of the search tree grows exponentially in terms of data dimensionality. New theoryand methods are needed to handle this challenge.
Development of a general framework that encompasses all active learning models and avoids the pitfallsjust discussed is not realistic given current understanding. Instead, we will address specific problems, build-ing on our long track record of research in the field of active learning [30,8,43,38,18,27,13,21,35,19]. Wewill identify several machine learning techniques that have demonstrated good performance in real-worldapplications, and develop new algorithms that are provably optimal or near-optimal over a large range ofmodel assumptions and noise conditions. By considering specific hypothesis classes and slightly relaxedoptimality conditions we believe that we can devise novel algorithms that avoid the problems identifiedabove. Not surprisingly, the kinds of models, algorithms and tasks that have gained popularity due to their
3
1 2
3
Meta-Algorithm for Active Learning
random selection (passive)
binary search (active)
Learning a 1-D Classifier
labeled data
binary search quickly finds decision boundary
passive : ✏ ⇠ n�1
active : ✏ ⇠ e�cn
errerr 2�n
<latexit sha1_base64="LF0MCNfhuOTCFY7Wls1cSmk+ymE=">AAAB8XicbVA9T8MwED3zWcpXgZHFokFioUrKAGMFC2OR6IdoQ+W4TmvVcSLbQaqi/gsWBhBi5d+w8W9w2wzQ8qSTnt670929IBFcG9f9Riura+sbm4Wt4vbO7t5+6eCwqeNUUdagsYhVOyCaCS5Zw3AjWDtRjESBYK1gdDP1W09MaR7LezNOmB+RgeQhp8RY6cFxqo/ZuZw4Tq9UdivuDHiZeDkpQ456r/TV7cc0jZg0VBCtO56bGD8jynAq2KTYTTVLCB2RAetYKknEtJ/NLp7gU6v0cRgrW9Lgmfp7IiOR1uMosJ0RMUO96E3F/7xOasIrP+MySQ2TdL4oTAU2MZ6+j/tcMWrE2BJCFbe3YjokilBjQyraELzFl5dJs1rxLireXbVcu87jKMAxnMAZeHAJNbiFOjSAgoRneIU3pNELekcf89YVlM8cwR+gzx+gEY+S</latexit>
Consider a possibly infinite set of hypotheses F with finite VC dimension dand for each f 2 F define the risk (error rate):
R(f) := P(f(x) 6= y)<latexit sha1_base64="IHQlOfu2sYOkwvQOa+oB6zAaTts=">AAAEPHicbZPPb9MwFMezlh+jULbBkcsTC1LLYWp2YZqEtCkM7YCmUu1Hp6VMjvOyWk3sYLuMysofxoU/ghsnLhxAiCtn3DTN6IYl29983nt2/J4dZglTutP5ulSr37p95+7yvcb9B82HK6trj46VGEuKR1QkQvZDojBhHI800wn2M4kkDRM8CUf+1H7yAaVigh/qSYaDlFxwFjNKtEXna7VesxlwvNRDFBJTk0mR5aZrR6HY1CVfMCeYpiQ3b4ppwVLOuTksxYKVCimShMhJbvxKLnhEGDM+29G8utILPrYTOcpNbzY3G43AhkHgmSAlehjG4NmAAu2VKIS9OepWqDtHvQr15mjfmICSBPbzOdktyW5F/JL4FemXpF+R05KcVoQODX33fP6Vpsaf6wNzUBxGWc0F4xFyDb7gikUogYCthWJhMgHGi8QgKNQgYhhOMmGTo1CBO9vude7CJdNDMAGmUHof+xCxFPn0FuTgRi4EARAeQSwkIKFDcOOAcbhaoqgGgl0bJFMjaKGU1lcSje3thuv2WnEbAth+aYeg24pbH9u2UO9h0nbdxvnqemejUzS4KbxSrDtl656vfgkiQcf2DzVNiFJnXifTA0OkZjTBvBGMFWaEjsgFnlnJSYpqYIrLn8MzS2YniYVNW0H/jTAkVWqShtZzWmt13TaF/7OdjXW8NTCMZ2ONnM42iscJaAHTl2RTKpFqW5WIESrtdaVAh0QSqu17mybBu37km+J4c8PrbHhvN9d3tsp0LDtPnKdOy/GcF86Os+90nSOH1j7VvtV+1H7WP9e/13/Vf89ca0tlzGNnodX//AX/ZWVT</latexit><latexit sha1_base64="IHQlOfu2sYOkwvQOa+oB6zAaTts=">AAAEPHicbZPPb9MwFMezlh+jULbBkcsTC1LLYWp2YZqEtCkM7YCmUu1Hp6VMjvOyWk3sYLuMysofxoU/ghsnLhxAiCtn3DTN6IYl29983nt2/J4dZglTutP5ulSr37p95+7yvcb9B82HK6trj46VGEuKR1QkQvZDojBhHI800wn2M4kkDRM8CUf+1H7yAaVigh/qSYaDlFxwFjNKtEXna7VesxlwvNRDFBJTk0mR5aZrR6HY1CVfMCeYpiQ3b4ppwVLOuTksxYKVCimShMhJbvxKLnhEGDM+29G8utILPrYTOcpNbzY3G43AhkHgmSAlehjG4NmAAu2VKIS9OepWqDtHvQr15mjfmICSBPbzOdktyW5F/JL4FemXpF+R05KcVoQODX33fP6Vpsaf6wNzUBxGWc0F4xFyDb7gikUogYCthWJhMgHGi8QgKNQgYhhOMmGTo1CBO9vude7CJdNDMAGmUHof+xCxFPn0FuTgRi4EARAeQSwkIKFDcOOAcbhaoqgGgl0bJFMjaKGU1lcSje3thuv2WnEbAth+aYeg24pbH9u2UO9h0nbdxvnqemejUzS4KbxSrDtl656vfgkiQcf2DzVNiFJnXifTA0OkZjTBvBGMFWaEjsgFnlnJSYpqYIrLn8MzS2YniYVNW0H/jTAkVWqShtZzWmt13TaF/7OdjXW8NTCMZ2ONnM42iscJaAHTl2RTKpFqW5WIESrtdaVAh0QSqu17mybBu37km+J4c8PrbHhvN9d3tsp0LDtPnKdOy/GcF86Os+90nSOH1j7VvtV+1H7WP9e/13/Vf89ca0tlzGNnodX//AX/ZWVT</latexit><latexit sha1_base64="IHQlOfu2sYOkwvQOa+oB6zAaTts=">AAAEPHicbZPPb9MwFMezlh+jULbBkcsTC1LLYWp2YZqEtCkM7YCmUu1Hp6VMjvOyWk3sYLuMysofxoU/ghsnLhxAiCtn3DTN6IYl29983nt2/J4dZglTutP5ulSr37p95+7yvcb9B82HK6trj46VGEuKR1QkQvZDojBhHI800wn2M4kkDRM8CUf+1H7yAaVigh/qSYaDlFxwFjNKtEXna7VesxlwvNRDFBJTk0mR5aZrR6HY1CVfMCeYpiQ3b4ppwVLOuTksxYKVCimShMhJbvxKLnhEGDM+29G8utILPrYTOcpNbzY3G43AhkHgmSAlehjG4NmAAu2VKIS9OepWqDtHvQr15mjfmICSBPbzOdktyW5F/JL4FemXpF+R05KcVoQODX33fP6Vpsaf6wNzUBxGWc0F4xFyDb7gikUogYCthWJhMgHGi8QgKNQgYhhOMmGTo1CBO9vude7CJdNDMAGmUHof+xCxFPn0FuTgRi4EARAeQSwkIKFDcOOAcbhaoqgGgl0bJFMjaKGU1lcSje3thuv2WnEbAth+aYeg24pbH9u2UO9h0nbdxvnqemejUzS4KbxSrDtl656vfgkiQcf2DzVNiFJnXifTA0OkZjTBvBGMFWaEjsgFnlnJSYpqYIrLn8MzS2YniYVNW0H/jTAkVWqShtZzWmt13TaF/7OdjXW8NTCMZ2ONnM42iscJaAHTl2RTKpFqW5WIESrtdaVAh0QSqu17mybBu37km+J4c8PrbHhvN9d3tsp0LDtPnKdOy/GcF86Os+90nSOH1j7VvtV+1H7WP9e/13/Vf89ca0tlzGNnodX//AX/ZWVT</latexit><latexit sha1_base64="IHQlOfu2sYOkwvQOa+oB6zAaTts=">AAAEPHicbZPPb9MwFMezlh+jULbBkcsTC1LLYWp2YZqEtCkM7YCmUu1Hp6VMjvOyWk3sYLuMysofxoU/ghsnLhxAiCtn3DTN6IYl29983nt2/J4dZglTutP5ulSr37p95+7yvcb9B82HK6trj46VGEuKR1QkQvZDojBhHI800wn2M4kkDRM8CUf+1H7yAaVigh/qSYaDlFxwFjNKtEXna7VesxlwvNRDFBJTk0mR5aZrR6HY1CVfMCeYpiQ3b4ppwVLOuTksxYKVCimShMhJbvxKLnhEGDM+29G8utILPrYTOcpNbzY3G43AhkHgmSAlehjG4NmAAu2VKIS9OepWqDtHvQr15mjfmICSBPbzOdktyW5F/JL4FemXpF+R05KcVoQODX33fP6Vpsaf6wNzUBxGWc0F4xFyDb7gikUogYCthWJhMgHGi8QgKNQgYhhOMmGTo1CBO9vude7CJdNDMAGmUHof+xCxFPn0FuTgRi4EARAeQSwkIKFDcOOAcbhaoqgGgl0bJFMjaKGU1lcSje3thuv2WnEbAth+aYeg24pbH9u2UO9h0nbdxvnqemejUzS4KbxSrDtl656vfgkiQcf2DzVNiFJnXifTA0OkZjTBvBGMFWaEjsgFnlnJSYpqYIrLn8MzS2YniYVNW0H/jTAkVWqShtZzWmt13TaF/7OdjXW8NTCMZ2ONnM42iscJaAHTl2RTKpFqW5WIESrtdaVAh0QSqu17mybBu37km+J4c8PrbHhvN9d3tsp0LDtPnKdOy/GcF86Os+90nSOH1j7VvtV+1H7WP9e/13/Vf89ca0tlzGNnodX//AX/ZWVT</latexit>
VC bound:<latexit sha1_base64="ggE1XIRxSUc+FWOFqoAe0Sk4VaA=">AAADnHicbZJdb9MwFIbThI9RKOvgCiEhiwoJcVE1u2Hihk3ZpCJgKtXadWpK5TjOatUfke0AlZU/xU/hjn+DmyaBbFhy/OZ5z4lz7BOllCg9GPxuud6du/fu7z1oP3zUebzfPXgyVSKTCE+QoELOIqgwJRxPNNEUz1KJIYsovozWwda//IalIoJf6E2KFwxec5IQBLVFy4PWz04n5Pi7XmEhMTOpFGluRvYpFNmG5A2bYsZgbj4VS8Mp19xclKLhIiEFpVBuchPUshER44Tw3Y7m9K9uxNgJ5To3493aabdDmwZC34QM6lWUAN8mFOisRBE4q9CoRqMKjWs0rtDQmBBBCoZ5RU5KclKToCRBTWYlmdXkqiRXNUErg76+qd4YM0Glz815UYyymgvCY8w1CDX+oYsLtnXHuZkGIBIZj9/ly25v0B8UA9wWfil6TjlGy+6vMBYoY/aziEKl5v4g1QsDpSaI4rwdZgqnEK3hNZ5bySHDamGKvXPwypIYJELaaX+roP9mGMiU2rDIRm7PUt30tvB/3jzTydHCEJ5mGnO02yjJKNACbDsVxERipOnGCoikbQcE0ApKiLTt57Y9BP9mybfF9LDvD/r+l8Pe8VF5HHvOc+el89rxnbfOsTN0Rs7EQe4z9707dD94L7xT76P3eRfqtsqcp05jeNM/bDAtBQ==</latexit><latexit sha1_base64="ggE1XIRxSUc+FWOFqoAe0Sk4VaA=">AAADnHicbZJdb9MwFIbThI9RKOvgCiEhiwoJcVE1u2Hihk3ZpCJgKtXadWpK5TjOatUfke0AlZU/xU/hjn+DmyaBbFhy/OZ5z4lz7BOllCg9GPxuud6du/fu7z1oP3zUebzfPXgyVSKTCE+QoELOIqgwJRxPNNEUz1KJIYsovozWwda//IalIoJf6E2KFwxec5IQBLVFy4PWz04n5Pi7XmEhMTOpFGluRvYpFNmG5A2bYsZgbj4VS8Mp19xclKLhIiEFpVBuchPUshER44Tw3Y7m9K9uxNgJ5To3493aabdDmwZC34QM6lWUAN8mFOisRBE4q9CoRqMKjWs0rtDQmBBBCoZ5RU5KclKToCRBTWYlmdXkqiRXNUErg76+qd4YM0Glz815UYyymgvCY8w1CDX+oYsLtnXHuZkGIBIZj9/ly25v0B8UA9wWfil6TjlGy+6vMBYoY/aziEKl5v4g1QsDpSaI4rwdZgqnEK3hNZ5bySHDamGKvXPwypIYJELaaX+roP9mGMiU2rDIRm7PUt30tvB/3jzTydHCEJ5mGnO02yjJKNACbDsVxERipOnGCoikbQcE0ApKiLTt57Y9BP9mybfF9LDvD/r+l8Pe8VF5HHvOc+el89rxnbfOsTN0Rs7EQe4z9707dD94L7xT76P3eRfqtsqcp05jeNM/bDAtBQ==</latexit><latexit sha1_base64="ggE1XIRxSUc+FWOFqoAe0Sk4VaA=">AAADnHicbZJdb9MwFIbThI9RKOvgCiEhiwoJcVE1u2Hihk3ZpCJgKtXadWpK5TjOatUfke0AlZU/xU/hjn+DmyaBbFhy/OZ5z4lz7BOllCg9GPxuud6du/fu7z1oP3zUebzfPXgyVSKTCE+QoELOIqgwJRxPNNEUz1KJIYsovozWwda//IalIoJf6E2KFwxec5IQBLVFy4PWz04n5Pi7XmEhMTOpFGluRvYpFNmG5A2bYsZgbj4VS8Mp19xclKLhIiEFpVBuchPUshER44Tw3Y7m9K9uxNgJ5To3493aabdDmwZC34QM6lWUAN8mFOisRBE4q9CoRqMKjWs0rtDQmBBBCoZ5RU5KclKToCRBTWYlmdXkqiRXNUErg76+qd4YM0Glz815UYyymgvCY8w1CDX+oYsLtnXHuZkGIBIZj9/ly25v0B8UA9wWfil6TjlGy+6vMBYoY/aziEKl5v4g1QsDpSaI4rwdZgqnEK3hNZ5bySHDamGKvXPwypIYJELaaX+roP9mGMiU2rDIRm7PUt30tvB/3jzTydHCEJ5mGnO02yjJKNACbDsVxERipOnGCoikbQcE0ApKiLTt57Y9BP9mybfF9LDvD/r+l8Pe8VF5HHvOc+el89rxnbfOsTN0Rs7EQe4z9707dD94L7xT76P3eRfqtsqcp05jeNM/bDAtBQ==</latexit><latexit sha1_base64="ggE1XIRxSUc+FWOFqoAe0Sk4VaA=">AAADnHicbZJdb9MwFIbThI9RKOvgCiEhiwoJcVE1u2Hihk3ZpCJgKtXadWpK5TjOatUfke0AlZU/xU/hjn+DmyaBbFhy/OZ5z4lz7BOllCg9GPxuud6du/fu7z1oP3zUebzfPXgyVSKTCE+QoELOIqgwJRxPNNEUz1KJIYsovozWwda//IalIoJf6E2KFwxec5IQBLVFy4PWz04n5Pi7XmEhMTOpFGluRvYpFNmG5A2bYsZgbj4VS8Mp19xclKLhIiEFpVBuchPUshER44Tw3Y7m9K9uxNgJ5To3493aabdDmwZC34QM6lWUAN8mFOisRBE4q9CoRqMKjWs0rtDQmBBBCoZ5RU5KclKToCRBTWYlmdXkqiRXNUErg76+qd4YM0Glz815UYyymgvCY8w1CDX+oYsLtnXHuZkGIBIZj9/ly25v0B8UA9wWfil6TjlGy+6vMBYoY/aziEKl5v4g1QsDpSaI4rwdZgqnEK3hNZ5bySHDamGKvXPwypIYJELaaX+roP9mGMiU2rDIRm7PUt30tvB/3jzTydHCEJ5mGnO02yjJKNACbDsVxERipOnGCoikbQcE0ApKiLTt57Y9BP9mybfF9LDvD/r+l8Pe8VF5HHvOc+el89rxnbfOsTN0Rs7EQe4z9707dD94L7xT76P3eRfqtsqcp05jeNM/bDAtBQ==</latexit>
w.p. � 1� �<latexit sha1_base64="vMjZktiD39IFMDIfusr5xcdEuNU=">AAADmXicbZLfbtMwFMbdhj8jUNaBxM1uLDokhESV7IZdcLGqFFUITaWsW6emTI7rtFbtONgOU2XlmXgX7ngb3DQJZMOSc778znfi+OiECaNKe97vRtO5d//Bw71H7uMnraf77YNnF0qkEpMJFkzIaYgUYTQmE001I9NEEsRDRi7DdX+bv/xBpKIiPtebhMw5WsY0ohhpi64PGj9brSAmN3pFhCTcJFIkmRnZp1B0a8lqaUY4R5n5nIdapoiZOS9ELYuFFIwhuclMv5I1x4JENN6daD781TWP3UiuMzPexZbrBrYMBr4JONKrMIK+LcjRoEAhHJRoVKFRicYVGpdoaEyAEYPDrCS9gvQq0i9IvyLTgkwrclWQq4rglcHf3pRvnJt+qc/MWX4ZZXUsaLwgsYbuTTfpwqNgSb7DAPpvrZVpdHTd7nhdL1/wrvAL0QHFGl23fwULgVNuv4kZUmrme4meGyQ1xYxkbpAqkiC8RksyszJGnKi5yScrg68sWcBISLvtP+X03wqDuFIbHlrntpHqdm4L/5ebpTo6mRsaJ6kmMd4dFKUMagG3YwoXVBKs2cYKhKWdBQzxCkmEtR1m1zbBv33lu+LiuOt7Xf/Lcef0pGjHHjgEL8Fr4IN34BQMwQhMAG6+aL5vDpofnUOn5wydTztrs1HUPAe15Xz9AzDWKR8=</latexit><latexit sha1_base64="vMjZktiD39IFMDIfusr5xcdEuNU=">AAADmXicbZLfbtMwFMbdhj8jUNaBxM1uLDokhESV7IZdcLGqFFUITaWsW6emTI7rtFbtONgOU2XlmXgX7ngb3DQJZMOSc778znfi+OiECaNKe97vRtO5d//Bw71H7uMnraf77YNnF0qkEpMJFkzIaYgUYTQmE001I9NEEsRDRi7DdX+bv/xBpKIiPtebhMw5WsY0ohhpi64PGj9brSAmN3pFhCTcJFIkmRnZp1B0a8lqaUY4R5n5nIdapoiZOS9ELYuFFIwhuclMv5I1x4JENN6daD781TWP3UiuMzPexZbrBrYMBr4JONKrMIK+LcjRoEAhHJRoVKFRicYVGpdoaEyAEYPDrCS9gvQq0i9IvyLTgkwrclWQq4rglcHf3pRvnJt+qc/MWX4ZZXUsaLwgsYbuTTfpwqNgSb7DAPpvrZVpdHTd7nhdL1/wrvAL0QHFGl23fwULgVNuv4kZUmrme4meGyQ1xYxkbpAqkiC8RksyszJGnKi5yScrg68sWcBISLvtP+X03wqDuFIbHlrntpHqdm4L/5ebpTo6mRsaJ6kmMd4dFKUMagG3YwoXVBKs2cYKhKWdBQzxCkmEtR1m1zbBv33lu+LiuOt7Xf/Lcef0pGjHHjgEL8Fr4IN34BQMwQhMAG6+aL5vDpofnUOn5wydTztrs1HUPAe15Xz9AzDWKR8=</latexit><latexit sha1_base64="vMjZktiD39IFMDIfusr5xcdEuNU=">AAADmXicbZLfbtMwFMbdhj8jUNaBxM1uLDokhESV7IZdcLGqFFUITaWsW6emTI7rtFbtONgOU2XlmXgX7ngb3DQJZMOSc778znfi+OiECaNKe97vRtO5d//Bw71H7uMnraf77YNnF0qkEpMJFkzIaYgUYTQmE001I9NEEsRDRi7DdX+bv/xBpKIiPtebhMw5WsY0ohhpi64PGj9brSAmN3pFhCTcJFIkmRnZp1B0a8lqaUY4R5n5nIdapoiZOS9ELYuFFIwhuclMv5I1x4JENN6daD781TWP3UiuMzPexZbrBrYMBr4JONKrMIK+LcjRoEAhHJRoVKFRicYVGpdoaEyAEYPDrCS9gvQq0i9IvyLTgkwrclWQq4rglcHf3pRvnJt+qc/MWX4ZZXUsaLwgsYbuTTfpwqNgSb7DAPpvrZVpdHTd7nhdL1/wrvAL0QHFGl23fwULgVNuv4kZUmrme4meGyQ1xYxkbpAqkiC8RksyszJGnKi5yScrg68sWcBISLvtP+X03wqDuFIbHlrntpHqdm4L/5ebpTo6mRsaJ6kmMd4dFKUMagG3YwoXVBKs2cYKhKWdBQzxCkmEtR1m1zbBv33lu+LiuOt7Xf/Lcef0pGjHHjgEL8Fr4IN34BQMwQhMAG6+aL5vDpofnUOn5wydTztrs1HUPAe15Xz9AzDWKR8=</latexit><latexit sha1_base64="vMjZktiD39IFMDIfusr5xcdEuNU=">AAADmXicbZLfbtMwFMbdhj8jUNaBxM1uLDokhESV7IZdcLGqFFUITaWsW6emTI7rtFbtONgOU2XlmXgX7ngb3DQJZMOSc778znfi+OiECaNKe97vRtO5d//Bw71H7uMnraf77YNnF0qkEpMJFkzIaYgUYTQmE001I9NEEsRDRi7DdX+bv/xBpKIiPtebhMw5WsY0ohhpi64PGj9brSAmN3pFhCTcJFIkmRnZp1B0a8lqaUY4R5n5nIdapoiZOS9ELYuFFIwhuclMv5I1x4JENN6daD781TWP3UiuMzPexZbrBrYMBr4JONKrMIK+LcjRoEAhHJRoVKFRicYVGpdoaEyAEYPDrCS9gvQq0i9IvyLTgkwrclWQq4rglcHf3pRvnJt+qc/MWX4ZZXUsaLwgsYbuTTfpwqNgSb7DAPpvrZVpdHTd7nhdL1/wrvAL0QHFGl23fwULgVNuv4kZUmrme4meGyQ1xYxkbpAqkiC8RksyszJGnKi5yScrg68sWcBISLvtP+X03wqDuFIbHlrntpHqdm4L/5ebpTo6mRsaJ6kmMd4dFKUMagG3YwoXVBKs2cYKhKWdBQzxCkmEtR1m1zbBv33lu+LiuOt7Xf/Lcef0pGjHHjgEL8Fr4IN34BQMwQhMAG6+aL5vDpofnUOn5wydTztrs1HUPAe15Xz9AzDWKR8=</latexit>
supf2F
|R(f)� bR(f)| 6
rd log(n/�)
n<latexit sha1_base64="ohtS4E8AeM7NGQ90lvwpjf4Me6g=">AAAD23icbZPPb9MwFMfTlh+jUOjgyMWiReqQKM2QgAOHTWWoBzSVat061aVyHKe16tiZ7TBVnrlwACGu/GPc+DP4D3DbNNANS877+vPey4vtlyBhVOlW61ehWLp2/cbNrVvl23cqd+9Vt+8fK5FKTPpYMCEHAVKEUU76mmpGBokkKA4YOQlm7YX/5CORigp+pOcJGcVowmlEMdIOjbcLvysVyMm5nhIhSWwSKRJruu4pFF2E2A03I3GMrHm3NBuezFpzlIkNLxZSMIbk3Jp2LjciQhJRvqpo3vzVGzFuIjmzpreylXIZujQAfQNjpKdBBHyXsEQHGQrAwRp1c9Rdo16OemvUMQZixEDHrsl+RvZz0s5IOyeDjAxycpqR05zgqcEfnqxXcWzaa31oDpebUU5zQXlIuAbleh2qNBmbCFK+etdbay96jWjnKTynIZkiDRarCwABZOTMmRcAqjOpDYwkwiaETExAgz9zVZhGO9Zwa+v1cbXWaraWA1wVfiZqXja64+pPGAqcxu6rMENKDf1WokcGSU0xI7YMU0UShGdoQoZOchQTNTLL3rTgsSMhiIR00+1qSf/NMChWah4HLnJxFeqybwH/5xumOno1MpQnqSYcrwpFKQNagEWjg5BKgjWbO4GwdN2EAZ4idy7a/Q5ldwj+5S1fFce7Tf9503+/W9t7nR3HlvfQe+Q1PN976e15Ha/r9T1cHBQ/Fb8Uv5ZGpc+lb6Xvq9BiIct54G2M0o8/KctD4w==</latexit>
Given training data {(xj , yj)}nj=1, learn a function f to predict y from x<latexit sha1_base64="DY5fekva7qVFNgVRXpdsaLjf+JI=">AAAD23icbZPPb9MwFMfTlh+jUOjgyMWiQRpoqppxgANIm8pED2gq1bp1arrKcZzWrX9EtjNWWeHCAYS48o9x48/gP8Bp00A3LDnvm897T857eQ5iSpRutX6VypUbN2/d3rpTvXuvdv9BffvhiRKJRLiPBBVyEECFKeG4r4mmeBBLDFlA8Wkwb2f+0wssFRH8WC9iPGJwwklEENQWjbdLv2s1n+OPeoqFxMzEUsSp6dqnUCQLSTfcFDMGU/N+aTY8uU3NcS42vEhIQSmUi9S0C7kREeKI8NWJ5u1fvRFjN5Tz1PRWtlat+jYN+J7xGdTTIAKeTViiwxwF4HCNugXqrlGvQL016hjjI0hBJ12Tg5wcFKSdk3ZBBjkZFOQsJ2cFQVODzp+v3xgz7bU+MkfLYpTVXBAeYq7BO3KBOdAS2lbwCQihhsD1zc7leLa7GM+e+enYzN546Tl3dwHFUHIAQZRwlPUNuJELtAB2FkKCNHAXLoikYMC9dMf1RqvZWi5wXXi5aDj56o7rP/1QoITZr0IUKjX0WrEeGSg1QRSnVT9ROIZoDid4aCWHDKuRWc5mCp5aEoJISLttVUv6b4aBTKkFC2xk9ivUVV8G/+cbJjp6NTKEx4nGHK0OihKaVZ0NOgiJxEjThRUQSTtNCKAplBBpex2qtgne1ZKvi5O9pvei6X3Ya+y/ztux5Tx2njg7jue8dPadjtN1+g4qD8qfyl/KXyujyufKt8r3VWi5lOc8cjZW5ccfJGdDJA==</latexit>
error rate ontraining data:
<latexit sha1_base64="WkOtKxzuaQlW7C7HIKPFlNbJUZk=">AAADsnicbZJdb9MwFIbdho9RKHRwyY1FhYS4KMmExMTVpjKpF2gq1bp1NF3lOE5r1R+R7QCVlf/HNXf8G9w0CWTjSInfPO85ss+Jo5RRbXz/d6vt3bv/4OHBo87jJ92nz3qHzy+1zBQmUyyZVLMIacKoIFNDDSOzVBHEI0auos1w5199I0pTKS7MNiULjlaCJhQj49DysPWz2w0F+W7WRCrCbapkmtuxe0tNdyl5w2aEc5Tbz8XScMo1txelaLhYKskYUtvcDmvZyIhJQsV+R/vpr27kuAepTW4n+7Xb6YSuDIaBDTky6yiBgSso0FmJInhWoXGNxhWa1GhSoZG1IUYMjvKKnJbktCbDkgxrMivJrCbXJbmuCV5bfPO2+uLcDit9bs+LZrTTQlIRE2FgaMgPU/xg13ecW6KUVFAhQ6AUMAyhUchNSaxgjAz6mC97fX/gFwHviqAUfVDGeNn7FcYSZ9xthhnSeh74qVlYpAzFjOSdMNMkRXiDVmTupECc6IUtTpTD147EMHEnSqQ7bEH/rbCIa73lkcvcTVjf9nbwf948M8nxwlKRZoYIvN8oyRg0Eu7uL4ypItiwrRMIK3dJMMRrpBA27pZ33BCC2y3fFZdHg8AfBF+O+ifH5TgOwEvwCrwBAfgATsAIjMEU4Pa79rR90156772vHvLwPrXdKmtegEZ47A+HyTYR</latexit><latexit sha1_base64="WkOtKxzuaQlW7C7HIKPFlNbJUZk=">AAADsnicbZJdb9MwFIbdho9RKHRwyY1FhYS4KMmExMTVpjKpF2gq1bp1NF3lOE5r1R+R7QCVlf/HNXf8G9w0CWTjSInfPO85ss+Jo5RRbXz/d6vt3bv/4OHBo87jJ92nz3qHzy+1zBQmUyyZVLMIacKoIFNDDSOzVBHEI0auos1w5199I0pTKS7MNiULjlaCJhQj49DysPWz2w0F+W7WRCrCbapkmtuxe0tNdyl5w2aEc5Tbz8XScMo1txelaLhYKskYUtvcDmvZyIhJQsV+R/vpr27kuAepTW4n+7Xb6YSuDIaBDTky6yiBgSso0FmJInhWoXGNxhWa1GhSoZG1IUYMjvKKnJbktCbDkgxrMivJrCbXJbmuCV5bfPO2+uLcDit9bs+LZrTTQlIRE2FgaMgPU/xg13ecW6KUVFAhQ6AUMAyhUchNSaxgjAz6mC97fX/gFwHviqAUfVDGeNn7FcYSZ9xthhnSeh74qVlYpAzFjOSdMNMkRXiDVmTupECc6IUtTpTD147EMHEnSqQ7bEH/rbCIa73lkcvcTVjf9nbwf948M8nxwlKRZoYIvN8oyRg0Eu7uL4ypItiwrRMIK3dJMMRrpBA27pZ33BCC2y3fFZdHg8AfBF+O+ifH5TgOwEvwCrwBAfgATsAIjMEU4Pa79rR90156772vHvLwPrXdKmtegEZ47A+HyTYR</latexit><latexit sha1_base64="WkOtKxzuaQlW7C7HIKPFlNbJUZk=">AAADsnicbZJdb9MwFIbdho9RKHRwyY1FhYS4KMmExMTVpjKpF2gq1bp1NF3lOE5r1R+R7QCVlf/HNXf8G9w0CWTjSInfPO85ss+Jo5RRbXz/d6vt3bv/4OHBo87jJ92nz3qHzy+1zBQmUyyZVLMIacKoIFNDDSOzVBHEI0auos1w5199I0pTKS7MNiULjlaCJhQj49DysPWz2w0F+W7WRCrCbapkmtuxe0tNdyl5w2aEc5Tbz8XScMo1txelaLhYKskYUtvcDmvZyIhJQsV+R/vpr27kuAepTW4n+7Xb6YSuDIaBDTky6yiBgSso0FmJInhWoXGNxhWa1GhSoZG1IUYMjvKKnJbktCbDkgxrMivJrCbXJbmuCV5bfPO2+uLcDit9bs+LZrTTQlIRE2FgaMgPU/xg13ecW6KUVFAhQ6AUMAyhUchNSaxgjAz6mC97fX/gFwHviqAUfVDGeNn7FcYSZ9xthhnSeh74qVlYpAzFjOSdMNMkRXiDVmTupECc6IUtTpTD147EMHEnSqQ7bEH/rbCIa73lkcvcTVjf9nbwf948M8nxwlKRZoYIvN8oyRg0Eu7uL4ypItiwrRMIK3dJMMRrpBA27pZ33BCC2y3fFZdHg8AfBF+O+ifH5TgOwEvwCrwBAfgATsAIjMEU4Pa79rR90156772vHvLwPrXdKmtegEZ47A+HyTYR</latexit><latexit sha1_base64="WkOtKxzuaQlW7C7HIKPFlNbJUZk=">AAADsnicbZJdb9MwFIbdho9RKHRwyY1FhYS4KMmExMTVpjKpF2gq1bp1NF3lOE5r1R+R7QCVlf/HNXf8G9w0CWTjSInfPO85ss+Jo5RRbXz/d6vt3bv/4OHBo87jJ92nz3qHzy+1zBQmUyyZVLMIacKoIFNDDSOzVBHEI0auos1w5199I0pTKS7MNiULjlaCJhQj49DysPWz2w0F+W7WRCrCbapkmtuxe0tNdyl5w2aEc5Tbz8XScMo1txelaLhYKskYUtvcDmvZyIhJQsV+R/vpr27kuAepTW4n+7Xb6YSuDIaBDTky6yiBgSso0FmJInhWoXGNxhWa1GhSoZG1IUYMjvKKnJbktCbDkgxrMivJrCbXJbmuCV5bfPO2+uLcDit9bs+LZrTTQlIRE2FgaMgPU/xg13ecW6KUVFAhQ6AUMAyhUchNSaxgjAz6mC97fX/gFwHviqAUfVDGeNn7FcYSZ9xthhnSeh74qVlYpAzFjOSdMNMkRXiDVmTupECc6IUtTpTD147EMHEnSqQ7bEH/rbCIa73lkcvcTVjf9nbwf948M8nxwlKRZoYIvN8oyRg0Eu7uL4ypItiwrRMIK3dJMMRrpBA27pZ33BCC2y3fFZdHg8AfBF+O+ifH5TgOwEvwCrwBAfgATsAIjMEU4Pa79rR90156772vHvLwPrXdKmtegEZ47A+HyTYR</latexit>
“empirical risk”<latexit sha1_base64="F41JY1o7g9kvevePn/f/upmVwWY=">AAADpXicbZLdbtMwFMfdhI9RKHRwyY3VCoG4mJLdbJebyqRewNRVbdepKZ3jOqtVO45sB6isPBlvwR1vg5MmgWwcyTn//M45/jxhwqjSnve75biPHj95evCs/fxF5+Wr7uHrmRKpxGSKBRNyHiJFGI3JVFPNyDyRBPGQketwO8jj19+IVFTEE71LyJKju5hGFCNt0eqw9bPTCWLyXW+IkISbRIokMyP7FYrmKVkjzAjnKDOfC9eIlD4zk1I0olhIwRiSu8wMatnIWJOIxvsVzae/upFjB5LbzIz3vtNuB7YMBr4JONKbMIK+LSjQRYlCeFGhUY1GFRrXaFyhoTEBRgwOs4qcl+S8JoOSDGoyL8m8JjcluakJ3hj89WP1x7kZVPrSXBaHUVbHgsZrEmsYaPJDFw9sQpaSzNzeEp5QSfNZJVXbXrbq9r0jrzD4UPil6IPSRqvur2AtcMrt/JghpRa+l+ilQVJTzEjWDlJFEoS36I4srIwRJ2ppik1k8J0laxgJaYfdX0H/rTCIK7Xjoc3ML1Xdj+Xwf7FFqqPTpaFxkmoS4/1CUcqgFjBvWbimkmDNdlYgLG1fYIg3SCKsbWO37SX494/8UMyOj3zvyL867p+dltdxAN6CHvgAfHACzsAQjMAUYKfnDJ0rZ+y+d7+4E3e2T3VaZc0b0DB39QdlfTEy</latexit><latexit sha1_base64="F41JY1o7g9kvevePn/f/upmVwWY=">AAADpXicbZLdbtMwFMfdhI9RKHRwyY3VCoG4mJLdbJebyqRewNRVbdepKZ3jOqtVO45sB6isPBlvwR1vg5MmgWwcyTn//M45/jxhwqjSnve75biPHj95evCs/fxF5+Wr7uHrmRKpxGSKBRNyHiJFGI3JVFPNyDyRBPGQketwO8jj19+IVFTEE71LyJKju5hGFCNt0eqw9bPTCWLyXW+IkISbRIokMyP7FYrmKVkjzAjnKDOfC9eIlD4zk1I0olhIwRiSu8wMatnIWJOIxvsVzae/upFjB5LbzIz3vtNuB7YMBr4JONKbMIK+LSjQRYlCeFGhUY1GFRrXaFyhoTEBRgwOs4qcl+S8JoOSDGoyL8m8JjcluakJ3hj89WP1x7kZVPrSXBaHUVbHgsZrEmsYaPJDFw9sQpaSzNzeEp5QSfNZJVXbXrbq9r0jrzD4UPil6IPSRqvur2AtcMrt/JghpRa+l+ilQVJTzEjWDlJFEoS36I4srIwRJ2ppik1k8J0laxgJaYfdX0H/rTCIK7Xjoc3ML1Xdj+Xwf7FFqqPTpaFxkmoS4/1CUcqgFjBvWbimkmDNdlYgLG1fYIg3SCKsbWO37SX494/8UMyOj3zvyL867p+dltdxAN6CHvgAfHACzsAQjMAUYKfnDJ0rZ+y+d7+4E3e2T3VaZc0b0DB39QdlfTEy</latexit><latexit sha1_base64="F41JY1o7g9kvevePn/f/upmVwWY=">AAADpXicbZLdbtMwFMfdhI9RKHRwyY3VCoG4mJLdbJebyqRewNRVbdepKZ3jOqtVO45sB6isPBlvwR1vg5MmgWwcyTn//M45/jxhwqjSnve75biPHj95evCs/fxF5+Wr7uHrmRKpxGSKBRNyHiJFGI3JVFPNyDyRBPGQketwO8jj19+IVFTEE71LyJKju5hGFCNt0eqw9bPTCWLyXW+IkISbRIokMyP7FYrmKVkjzAjnKDOfC9eIlD4zk1I0olhIwRiSu8wMatnIWJOIxvsVzae/upFjB5LbzIz3vtNuB7YMBr4JONKbMIK+LSjQRYlCeFGhUY1GFRrXaFyhoTEBRgwOs4qcl+S8JoOSDGoyL8m8JjcluakJ3hj89WP1x7kZVPrSXBaHUVbHgsZrEmsYaPJDFw9sQpaSzNzeEp5QSfNZJVXbXrbq9r0jrzD4UPil6IPSRqvur2AtcMrt/JghpRa+l+ilQVJTzEjWDlJFEoS36I4srIwRJ2ppik1k8J0laxgJaYfdX0H/rTCIK7Xjoc3ML1Xdj+Xwf7FFqqPTpaFxkmoS4/1CUcqgFjBvWbimkmDNdlYgLG1fYIg3SCKsbWO37SX494/8UMyOj3zvyL867p+dltdxAN6CHvgAfHACzsAQjMAUYKfnDJ0rZ+y+d7+4E3e2T3VaZc0b0DB39QdlfTEy</latexit><latexit sha1_base64="F41JY1o7g9kvevePn/f/upmVwWY=">AAADpXicbZLdbtMwFMfdhI9RKHRwyY3VCoG4mJLdbJebyqRewNRVbdepKZ3jOqtVO45sB6isPBlvwR1vg5MmgWwcyTn//M45/jxhwqjSnve75biPHj95evCs/fxF5+Wr7uHrmRKpxGSKBRNyHiJFGI3JVFPNyDyRBPGQketwO8jj19+IVFTEE71LyJKju5hGFCNt0eqw9bPTCWLyXW+IkISbRIokMyP7FYrmKVkjzAjnKDOfC9eIlD4zk1I0olhIwRiSu8wMatnIWJOIxvsVzae/upFjB5LbzIz3vtNuB7YMBr4JONKbMIK+LSjQRYlCeFGhUY1GFRrXaFyhoTEBRgwOs4qcl+S8JoOSDGoyL8m8JjcluakJ3hj89WP1x7kZVPrSXBaHUVbHgsZrEmsYaPJDFw9sQpaSzNzeEp5QSfNZJVXbXrbq9r0jrzD4UPil6IPSRqvur2AtcMrt/JghpRa+l+ilQVJTzEjWDlJFEoS36I4srIwRJ2ppik1k8J0laxgJaYfdX0H/rTCIK7Xjoc3ML1Xdj+Xwf7FFqqPTpaFxkmoS4/1CUcqgFjBvWbimkmDNdlYgLG1fYIg3SCKsbWO37SX494/8UMyOj3zvyL867p+dltdxAN6CHvgAfHACzsAQjMAUYKfnDJ0rZ+y+d7+4E3e2T3VaZc0b0DB39QdlfTEy</latexit>
This is the guarantee for the test at just one point. If we test a k points during the binary search
process, then the probability that one or more of the tests fails is bounded by k�. Thus, if we replace
� with �/k the bound on m above, then the probability that all k tests are correct is at least 1� �. So
we will perform k steps of the binary search and collect
m = O
✓2 log(2k/�)
�2
◆
samples in each step. Taking k = log(1/✏) yields an ✏-accurate solution with probability at least
1� � and the total number of (noisy) function evaluations is
O
0
BBB@log(1/✏)
2 log(2 log(1/✏)/�)
�2| {z }price paid due to noise
1
CCCA.
4 Binary Classification
The binary search problems above assume one can request the label at any chosen point x, but this is
not the case in most machine learning (ML) settings. Rather, in ML applications have access to a
pool of examples of x distributed according to some distribution and a subset of these examples can
be labeled (usually at a cost). The goal is to learn a good classifier using as few labeled examples as
possible.
Consider the finite set of hypotheses fi, i = 1, . . . , k. Each hypothesis has a probability of error
R(fi) := P(fi(x) 6= y)
where the probability is computed over the distribution on x and the binary distribution of y given x.
Empirical risk minimization (ERM) is a standard ML approach which proceeds as follows. Select the
hypothesis that makes the fewest prediction errors on a set of iid labeled examples {(xi, yi)}mi=1. The
basic ingredient in ERM is an estimate of the error rate of each hypothesis
bR(fi) =1
m
MX
i=1
1(f(xi) 6= yi) ,
which is usually referred to as the empirical risk. Note that the terms in this average are iid binary
random variables, and using the Chernoff bound confidence intervals we have
|R(fi)� bR(fi)| r
log(2/�)
2m
with probability at least 1� �. In order to have the confidence intervals hold simultaneously for all
hypotheses, we apply the union bound (replace � with �/k) to have
|R(fi)� bR(fi)| r
log(2k/�)
2mfor i = 1, . . . , k
Now suppose we want to select a hypothesis with true error rate within � > 0 of mini R(fi). Define
true and empirical risk minimizers
f? = arg min
f2{fi}R(f)
bf = arg minf2{fi}
bR(f)
To determine how many samples will be sufficient to guarantee that R( bf) R(f?) + �, consider the
confidence intervals for these two hypotheses. The lower confidence bound for f?
is
R(f?) � bR(f?)�r
log(2k/�)
2m
3
bR(f) =1
n
nX
i=1
⇣f(xi) 6= yi
⌘
<latexit sha1_base64="uLm4hnIYWO8m0dqspPt8xRONBQg=">AAADzXicbZPPb9MwFMfThB+jUNbBkYtFi9RyqJpxgAOTNsqkCqFRqnXr1HSV4zqtVf8ItsMoJlz5/7hx5h/BadNAN57k+JvP9z0578kJY0qUbrd/lVzv1u07d3fule8/qDzcre49OlMikQgPkKBCDkOoMCUcDzTRFA9jiSELKT4PF53MP/+MpSKCn+pljMcMzjiJCILaosle6XelEnB8pedYSMxMLEWcmp59CkWylHTLppgxmJr3q23LyffUnOZiy0VCCkqhXKamU8itjCmOCF+faN7+1Vs5dkG5SE1/vVfK5cCWgcA3AYN6HkbAtwUrdJyjEBxvUK9AvQ3qF6i/QV1jAgQp6KYbcpSTo4J0ctIpyDAnw4Jc5OSiIGhu0OXzzRtjprPRJ+Zk1YyymgvCp5hrUK7XgysyxXOoQb8RNUEADuwKIgmR8VPD00AlbGLIgZ9e8sx5Q2aNqPFlQpp2aJ/AckIy1KzXJ9Vau9VeBbgp/FzUnDx6k+rPYCpQwux3IAqVGvntWI8NlJogitNykCgcQ7SAMzyykkOG1disbmMKnlkyBZGQdtk+VvTfCgOZUksW2sxs+Oq6l8H/eaNER6/GhvA40Zij9UFRQoEWILvaYEokRpourYBI2vuDAJpDOy5tf4CyHYJ/veWb4my/5b9o+R/3a4ev83HsOE+cp07D8Z2XzqHTdXrOwEHuOzd2l+5X74OXeN+87+tUt5TXPHa2wvvxBzFCPO0=</latexit>
Vapnik-Chervonenkis (VC) Theory
Empirical Risk Minimization (ERM)
bR( bf)<latexit sha1_base64="anfNbe5yQlK0MQq0cHBm0IafXYA=">AAADmnicbZLfbtMwFMbdhj+jUOjgggu4sOiQBhdVsht2g7RRBkWgqVTt1qnpKsdxVqt2HNkOU2XlnXgW7ngb3DQJZMOSc778vnPi+OgECaNKu+7vRtO5c/fe/Z0HrYeP2o+fdHafnimRSkwmWDAhpwFShNGYTDTVjEwTSRAPGDkPVv2Nf/6DSEVFPNbrhMw5uoppRDHSFi12Gz/bbT8m13pJhCTcJFIkmRnap1B0k5LVbEY4R5n5loeaU8TMjAtRc7GQgjEk15npV7KWEZKIxtsTzce/upZjN5KrzIy2sd1q+bYM+p7xOdLLIIKeLcjRSYECeFKiYYWGJRpVaFSigTE+RgwOspIcF+S4Iv2C9CsyLci0IhcFuagIXhp8+bZ849z0S31qTvPLKKtjQeOQxBru+dc0JEuk4Wi/ktGbvUWn6/bcfMHbwitEFxRruOj88kOBU24/ihlSaua5iZ4bJDXFjGQtP1UkQXiFrsjMyhhxouYmH60MvrYkhJGQdtufyum/FQZxpdY8sJmbTqqb3gb+z5ulOjqcGxonqSYx3h4UpQxqATdzCkMqCdZsbQXC0g4DhniJJMLaTnPLNsG7eeXb4uyg57k97/tB9+iwaMcOeAFegX3ggXfgCAzAEEwAbj5vvm9+an52XjofnC/O121qs1HUPAO15Yz/ACx5KpA=</latexit><latexit sha1_base64="anfNbe5yQlK0MQq0cHBm0IafXYA=">AAADmnicbZLfbtMwFMbdhj+jUOjgggu4sOiQBhdVsht2g7RRBkWgqVTt1qnpKsdxVqt2HNkOU2XlnXgW7ngb3DQJZMOSc778vnPi+OgECaNKu+7vRtO5c/fe/Z0HrYeP2o+fdHafnimRSkwmWDAhpwFShNGYTDTVjEwTSRAPGDkPVv2Nf/6DSEVFPNbrhMw5uoppRDHSFi12Gz/bbT8m13pJhCTcJFIkmRnap1B0k5LVbEY4R5n5loeaU8TMjAtRc7GQgjEk15npV7KWEZKIxtsTzce/upZjN5KrzIy2sd1q+bYM+p7xOdLLIIKeLcjRSYECeFKiYYWGJRpVaFSigTE+RgwOspIcF+S4Iv2C9CsyLci0IhcFuagIXhp8+bZ849z0S31qTvPLKKtjQeOQxBru+dc0JEuk4Wi/ktGbvUWn6/bcfMHbwitEFxRruOj88kOBU24/ihlSaua5iZ4bJDXFjGQtP1UkQXiFrsjMyhhxouYmH60MvrYkhJGQdtufyum/FQZxpdY8sJmbTqqb3gb+z5ulOjqcGxonqSYx3h4UpQxqATdzCkMqCdZsbQXC0g4DhniJJMLaTnPLNsG7eeXb4uyg57k97/tB9+iwaMcOeAFegX3ggXfgCAzAEEwAbj5vvm9+an52XjofnC/O121qs1HUPAO15Yz/ACx5KpA=</latexit><latexit sha1_base64="anfNbe5yQlK0MQq0cHBm0IafXYA=">AAADmnicbZLfbtMwFMbdhj+jUOjgggu4sOiQBhdVsht2g7RRBkWgqVTt1qnpKsdxVqt2HNkOU2XlnXgW7ngb3DQJZMOSc778vnPi+OgECaNKu+7vRtO5c/fe/Z0HrYeP2o+fdHafnimRSkwmWDAhpwFShNGYTDTVjEwTSRAPGDkPVv2Nf/6DSEVFPNbrhMw5uoppRDHSFi12Gz/bbT8m13pJhCTcJFIkmRnap1B0k5LVbEY4R5n5loeaU8TMjAtRc7GQgjEk15npV7KWEZKIxtsTzce/upZjN5KrzIy2sd1q+bYM+p7xOdLLIIKeLcjRSYECeFKiYYWGJRpVaFSigTE+RgwOspIcF+S4Iv2C9CsyLci0IhcFuagIXhp8+bZ849z0S31qTvPLKKtjQeOQxBru+dc0JEuk4Wi/ktGbvUWn6/bcfMHbwitEFxRruOj88kOBU24/ihlSaua5iZ4bJDXFjGQtP1UkQXiFrsjMyhhxouYmH60MvrYkhJGQdtufyum/FQZxpdY8sJmbTqqb3gb+z5ulOjqcGxonqSYx3h4UpQxqATdzCkMqCdZsbQXC0g4DhniJJMLaTnPLNsG7eeXb4uyg57k97/tB9+iwaMcOeAFegX3ggXfgCAzAEEwAbj5vvm9+an52XjofnC/O121qs1HUPAO15Yz/ACx5KpA=</latexit><latexit sha1_base64="anfNbe5yQlK0MQq0cHBm0IafXYA=">AAADmnicbZLfbtMwFMbdhj+jUOjgggu4sOiQBhdVsht2g7RRBkWgqVTt1qnpKsdxVqt2HNkOU2XlnXgW7ngb3DQJZMOSc778vnPi+OgECaNKu+7vRtO5c/fe/Z0HrYeP2o+fdHafnimRSkwmWDAhpwFShNGYTDTVjEwTSRAPGDkPVv2Nf/6DSEVFPNbrhMw5uoppRDHSFi12Gz/bbT8m13pJhCTcJFIkmRnap1B0k5LVbEY4R5n5loeaU8TMjAtRc7GQgjEk15npV7KWEZKIxtsTzce/upZjN5KrzIy2sd1q+bYM+p7xOdLLIIKeLcjRSYECeFKiYYWGJRpVaFSigTE+RgwOspIcF+S4Iv2C9CsyLci0IhcFuagIXhp8+bZ849z0S31qTvPLKKtjQeOQxBru+dc0JEuk4Wi/ktGbvUWn6/bcfMHbwitEFxRruOj88kOBU24/ihlSaua5iZ4bJDXFjGQtP1UkQXiFrsjMyhhxouYmH60MvrYkhJGQdtufyum/FQZxpdY8sJmbTqqb3gb+z5ulOjqcGxonqSYx3h4UpQxqATdzCkMqCdZsbQXC0g4DhniJJMLaTnPLNsG7eeXb4uyg57k97/tB9+iwaMcOeAFegX3ggXfgCAzAEEwAbj5vvm9+an52XjofnC/O121qs1HUPAO15Yz/ACx5KpA=</latexit>
bR(f?)<latexit sha1_base64="uMwFHPQEDuDGFAvCjiN8ZN9Fiwk=">AAADl3icbZLfbtMwFMbdhj+jUNYBN4gbiw5pcFElu2Hihk3ZUC/Q1FXr1qnpJsdxVqt2HNkOU2XlkXgZ7ngb3DQJZMOSc778vnPi+OiEKaNKu+7vVtt59PjJ061nnecvui+3ezuvLpTIJCYTLJiQ0xApwmhCJppqRqapJIiHjFyGS3/tX/4gUlGRnOtVSuYc3SY0phhpi252Wj+73SAhd3pBhCTcpFKkuRnZp1B0nZI3bEY4R7n5XoSGU8bcnJei4WIhBWNIrnLj17KREZGYJpsTzfFf3cixG8llbsab2O10AlsGA88EHOlFGEPPFhTopEQhPKnQqEajCo1rNK7Q0JgAIwaHeUWOSnJUE78kfk2mJZnW5KokVzXBC4OvP1VvnBu/0qfmtLiMsjoRNIlIouFucEcjskAajvfi60BpJD/u3vT67sAtFnwovFL0QblGN71fQSRwxu0XMUNKzTw31XODpKaYkbwTZIqkCC/RLZlZmSBO1NwUc5XDD5ZEMBbSbvtHBf23wiCu1IqHNnPdRnXfW8P/ebNMxwdzQ5M00yTBm4PijEEt4HpIYUQlwZqtrEBY2knAEC+QRFjbUe7YJnj3r/xQXOwPPHfgne33Dw/KdmyBd+A92AMe+AwOwRCMwATg9pv2l7bfPnbeOl+db85wk9pulTWvQWM5Z38AbnkphA==</latexit><latexit sha1_base64="uMwFHPQEDuDGFAvCjiN8ZN9Fiwk=">AAADl3icbZLfbtMwFMbdhj+jUNYBN4gbiw5pcFElu2Hihk3ZUC/Q1FXr1qnpJsdxVqt2HNkOU2XlkXgZ7ngb3DQJZMOSc778vnPi+OiEKaNKu+7vVtt59PjJ061nnecvui+3ezuvLpTIJCYTLJiQ0xApwmhCJppqRqapJIiHjFyGS3/tX/4gUlGRnOtVSuYc3SY0phhpi252Wj+73SAhd3pBhCTcpFKkuRnZp1B0nZI3bEY4R7n5XoSGU8bcnJei4WIhBWNIrnLj17KREZGYJpsTzfFf3cixG8llbsab2O10AlsGA88EHOlFGEPPFhTopEQhPKnQqEajCo1rNK7Q0JgAIwaHeUWOSnJUE78kfk2mJZnW5KokVzXBC4OvP1VvnBu/0qfmtLiMsjoRNIlIouFucEcjskAajvfi60BpJD/u3vT67sAtFnwovFL0QblGN71fQSRwxu0XMUNKzTw31XODpKaYkbwTZIqkCC/RLZlZmSBO1NwUc5XDD5ZEMBbSbvtHBf23wiCu1IqHNnPdRnXfW8P/ebNMxwdzQ5M00yTBm4PijEEt4HpIYUQlwZqtrEBY2knAEC+QRFjbUe7YJnj3r/xQXOwPPHfgne33Dw/KdmyBd+A92AMe+AwOwRCMwATg9pv2l7bfPnbeOl+db85wk9pulTWvQWM5Z38AbnkphA==</latexit><latexit sha1_base64="uMwFHPQEDuDGFAvCjiN8ZN9Fiwk=">AAADl3icbZLfbtMwFMbdhj+jUNYBN4gbiw5pcFElu2Hihk3ZUC/Q1FXr1qnpJsdxVqt2HNkOU2XlkXgZ7ngb3DQJZMOSc778vnPi+OiEKaNKu+7vVtt59PjJ061nnecvui+3ezuvLpTIJCYTLJiQ0xApwmhCJppqRqapJIiHjFyGS3/tX/4gUlGRnOtVSuYc3SY0phhpi252Wj+73SAhd3pBhCTcpFKkuRnZp1B0nZI3bEY4R7n5XoSGU8bcnJei4WIhBWNIrnLj17KREZGYJpsTzfFf3cixG8llbsab2O10AlsGA88EHOlFGEPPFhTopEQhPKnQqEajCo1rNK7Q0JgAIwaHeUWOSnJUE78kfk2mJZnW5KokVzXBC4OvP1VvnBu/0qfmtLiMsjoRNIlIouFucEcjskAajvfi60BpJD/u3vT67sAtFnwovFL0QblGN71fQSRwxu0XMUNKzTw31XODpKaYkbwTZIqkCC/RLZlZmSBO1NwUc5XDD5ZEMBbSbvtHBf23wiCu1IqHNnPdRnXfW8P/ebNMxwdzQ5M00yTBm4PijEEt4HpIYUQlwZqtrEBY2knAEC+QRFjbUe7YJnj3r/xQXOwPPHfgne33Dw/KdmyBd+A92AMe+AwOwRCMwATg9pv2l7bfPnbeOl+db85wk9pulTWvQWM5Z38AbnkphA==</latexit><latexit sha1_base64="uMwFHPQEDuDGFAvCjiN8ZN9Fiwk=">AAADl3icbZLfbtMwFMbdhj+jUNYBN4gbiw5pcFElu2Hihk3ZUC/Q1FXr1qnpJsdxVqt2HNkOU2XlkXgZ7ngb3DQJZMOSc778vnPi+OiEKaNKu+7vVtt59PjJ061nnecvui+3ezuvLpTIJCYTLJiQ0xApwmhCJppqRqapJIiHjFyGS3/tX/4gUlGRnOtVSuYc3SY0phhpi252Wj+73SAhd3pBhCTcpFKkuRnZp1B0nZI3bEY4R7n5XoSGU8bcnJei4WIhBWNIrnLj17KREZGYJpsTzfFf3cixG8llbsab2O10AlsGA88EHOlFGEPPFhTopEQhPKnQqEajCo1rNK7Q0JgAIwaHeUWOSnJUE78kfk2mJZnW5KokVzXBC4OvP1VvnBu/0qfmtLiMsjoRNIlIouFucEcjskAajvfi60BpJD/u3vT67sAtFnwovFL0QblGN71fQSRwxu0XMUNKzTw31XODpKaYkbwTZIqkCC/RLZlZmSBO1NwUc5XDD5ZEMBbSbvtHBf23wiCu1IqHNnPdRnXfW8P/ebNMxwdzQ5M00yTBm4PijEEt4HpIYUQlwZqtrEBY2knAEC+QRFjbUe7YJnj3r/xQXOwPPHfgne33Dw/KdmyBd+A92AMe+AwOwRCMwATg9pv2l7bfPnbeOl+db85wk9pulTWvQWM5Z38AbnkphA==</latexit>
“empirical risk minimizer”<latexit sha1_base64="Lh7SF6DJ2t8iQyBnXkS/mkKeOmY=">AAADr3icbZLfb9MwEMfdhB+jUOjgkRdrFRJCqEr2sj1uKpP6gKZSrWunpmsd11mt2nFkO0Cx8ufxD/DGf4OTJoFsnOTcN5+7y9kXhwmjSnve75bjPnr85OnBs/bzF52Xr7qHr6+VSCUmEyyYkLMQKcJoTCaaakZmiSSIh4xMw+0gj0+/EqmoiK/0LiELju5iGlGMtEXLw9bPTieIyTe9IUISbhIpksyM7FMomqdkjTAjnKPMfC5cI1L6zFyVohHFQgrGkNxlZlDLRsaaRDTedzSf/upGjl1IbjMz3vtOux3YMhj4JuBIb8II+ragQBclCuFFhUY1GlVoXKNxhYbGBBgxOMwqcl6S85oMSjKoyawks5rclOSmJnhj8O2H6o1zM6j0pbksDqOsjgWN1yTWMNDkuy5+sAlZSjKzWhGeUEnzr0qqtpDbIXH6g8ijbNnteX2vMPhQ+KXogdJGy+6vYC1wym0nzJBSc99L9MIgqSlmJGsHqSIJwlt0R+ZWxogTtTDFdjL4zpI1jIS0y+60oP9WGMSV2vHQZubjVfdjOfxfbJ7q6HRhaJykmsR43yhKGdQC5pcXrqkkWLOdFQhLe0MwxBskEdb2irftEPz7R34oro/7vtf3vxz3zk7LcRyAt+AIvAc+OAFnYAhGYAKw89EZO3MncH136t66q32q0ypr3oCGufQPfi81ig==</latexit><latexit sha1_base64="Lh7SF6DJ2t8iQyBnXkS/mkKeOmY=">AAADr3icbZLfb9MwEMfdhB+jUOjgkRdrFRJCqEr2sj1uKpP6gKZSrWunpmsd11mt2nFkO0Cx8ufxD/DGf4OTJoFsnOTcN5+7y9kXhwmjSnve75bjPnr85OnBs/bzF52Xr7qHr6+VSCUmEyyYkLMQKcJoTCaaakZmiSSIh4xMw+0gj0+/EqmoiK/0LiELju5iGlGMtEXLw9bPTieIyTe9IUISbhIpksyM7FMomqdkjTAjnKPMfC5cI1L6zFyVohHFQgrGkNxlZlDLRsaaRDTedzSf/upGjl1IbjMz3vtOux3YMhj4JuBIb8II+ragQBclCuFFhUY1GlVoXKNxhYbGBBgxOMwqcl6S85oMSjKoyawks5rclOSmJnhj8O2H6o1zM6j0pbksDqOsjgWN1yTWMNDkuy5+sAlZSjKzWhGeUEnzr0qqtpDbIXH6g8ijbNnteX2vMPhQ+KXogdJGy+6vYC1wym0nzJBSc99L9MIgqSlmJGsHqSIJwlt0R+ZWxogTtTDFdjL4zpI1jIS0y+60oP9WGMSV2vHQZubjVfdjOfxfbJ7q6HRhaJykmsR43yhKGdQC5pcXrqkkWLOdFQhLe0MwxBskEdb2irftEPz7R34oro/7vtf3vxz3zk7LcRyAt+AIvAc+OAFnYAhGYAKw89EZO3MncH136t66q32q0ypr3oCGufQPfi81ig==</latexit><latexit sha1_base64="Lh7SF6DJ2t8iQyBnXkS/mkKeOmY=">AAADr3icbZLfb9MwEMfdhB+jUOjgkRdrFRJCqEr2sj1uKpP6gKZSrWunpmsd11mt2nFkO0Cx8ufxD/DGf4OTJoFsnOTcN5+7y9kXhwmjSnve75bjPnr85OnBs/bzF52Xr7qHr6+VSCUmEyyYkLMQKcJoTCaaakZmiSSIh4xMw+0gj0+/EqmoiK/0LiELju5iGlGMtEXLw9bPTieIyTe9IUISbhIpksyM7FMomqdkjTAjnKPMfC5cI1L6zFyVohHFQgrGkNxlZlDLRsaaRDTedzSf/upGjl1IbjMz3vtOux3YMhj4JuBIb8II+ragQBclCuFFhUY1GlVoXKNxhYbGBBgxOMwqcl6S85oMSjKoyawks5rclOSmJnhj8O2H6o1zM6j0pbksDqOsjgWN1yTWMNDkuy5+sAlZSjKzWhGeUEnzr0qqtpDbIXH6g8ijbNnteX2vMPhQ+KXogdJGy+6vYC1wym0nzJBSc99L9MIgqSlmJGsHqSIJwlt0R+ZWxogTtTDFdjL4zpI1jIS0y+60oP9WGMSV2vHQZubjVfdjOfxfbJ7q6HRhaJykmsR43yhKGdQC5pcXrqkkWLOdFQhLe0MwxBskEdb2irftEPz7R34oro/7vtf3vxz3zk7LcRyAt+AIvAc+OAFnYAhGYAKw89EZO3MncH136t66q32q0ypr3oCGufQPfi81ig==</latexit><latexit sha1_base64="Lh7SF6DJ2t8iQyBnXkS/mkKeOmY=">AAADr3icbZLfb9MwEMfdhB+jUOjgkRdrFRJCqEr2sj1uKpP6gKZSrWunpmsd11mt2nFkO0Cx8ufxD/DGf4OTJoFsnOTcN5+7y9kXhwmjSnve75bjPnr85OnBs/bzF52Xr7qHr6+VSCUmEyyYkLMQKcJoTCaaakZmiSSIh4xMw+0gj0+/EqmoiK/0LiELju5iGlGMtEXLw9bPTieIyTe9IUISbhIpksyM7FMomqdkjTAjnKPMfC5cI1L6zFyVohHFQgrGkNxlZlDLRsaaRDTedzSf/upGjl1IbjMz3vtOux3YMhj4JuBIb8II+ragQBclCuFFhUY1GlVoXKNxhYbGBBgxOMwqcl6S85oMSjKoyawks5rclOSmJnhj8O2H6o1zM6j0pbksDqOsjgWN1yTWMNDkuy5+sAlZSjKzWhGeUEnzr0qqtpDbIXH6g8ijbNnteX2vMPhQ+KXogdJGy+6vYC1wym0nzJBSc99L9MIgqSlmJGsHqSIJwlt0R+ZWxogTtTDFdjL4zpI1jIS0y+60oP9WGMSV2vHQZubjVfdjOfxfbJ7q6HRhaJykmsR43yhKGdQC5pcXrqkkWLOdFQhLe0MwxBskEdb2irftEPz7R34oro/7vtf3vxz3zk7LcRyAt+AIvAc+OAFnYAhGYAKw89EZO3MncH136t66q32q0ypr3oCGufQPfi81ig==</latexit>
true risk minimizer<latexit sha1_base64="bpumDvr+zEjXz9/NWhJUJNc4pMA=">AAADp3icbZLfb9MwEMfThB+jUNbBIy8WBQl4qJq9sMdNZVKF0Cil7To1pXLcy2rVjiPbAYqVP41/gjf+G9w0CWTDknNff+5OZ18uTBhVutf73XC9O3fv3T940Hz4qPX4sH30ZKpEKglMiGBCzkKsgNEYJppqBrNEAuYhg8tw09/5L7+CVFTEY71NYMHxdUwjSrC2aHnU+NlqBTF802sQErhJpEgyM7RfoeguJKu5GXCOM/MhNzVPYTMzLkTNS4QUjGG5zUy/krWIFUQ03lc07/7qWozdWG4yM9rbVrMZ2DQU+CbgWK/DCPk2IUfnBQrReYmGFRqWaFShUYkGxgQEMzTISnJWkLOK9AvSr8isILOKXBXkqiJkbciXN+WJc9Mv9YW5yB+jrI4FjVcQaxRo+K7zH2xClkJmtEwBSao2iNvucPoDZLZsd3rdXr7QbeEXouMUa7hs/wpWgqTcViAMKzX3e4leGCw1JQyyZpAqSDDZ4GuYWxljDmph8mtk6KUlKxQJabe9YU7/zTCYK7XloY3ctVXd9O3g/3zzVEcnC0PjJNUQk32hKGVIC7QbWrSiEohmWyswkXYyCCJrLDHRdrSbtgn+zSffFtPjrt/r+p+OO6cnRTsOnGfOc+eV4ztvnVNn4AydiUPcF+5797M79l57H72pN9uHuo0i56lTWx7+A6+IMmI=</latexit><latexit sha1_base64="bpumDvr+zEjXz9/NWhJUJNc4pMA=">AAADp3icbZLfb9MwEMfThB+jUNbBIy8WBQl4qJq9sMdNZVKF0Cil7To1pXLcy2rVjiPbAYqVP41/gjf+G9w0CWTDknNff+5OZ18uTBhVutf73XC9O3fv3T940Hz4qPX4sH30ZKpEKglMiGBCzkKsgNEYJppqBrNEAuYhg8tw09/5L7+CVFTEY71NYMHxdUwjSrC2aHnU+NlqBTF802sQErhJpEgyM7RfoeguJKu5GXCOM/MhNzVPYTMzLkTNS4QUjGG5zUy/krWIFUQ03lc07/7qWozdWG4yM9rbVrMZ2DQU+CbgWK/DCPk2IUfnBQrReYmGFRqWaFShUYkGxgQEMzTISnJWkLOK9AvSr8isILOKXBXkqiJkbciXN+WJc9Mv9YW5yB+jrI4FjVcQaxRo+K7zH2xClkJmtEwBSao2iNvucPoDZLZsd3rdXr7QbeEXouMUa7hs/wpWgqTcViAMKzX3e4leGCw1JQyyZpAqSDDZ4GuYWxljDmph8mtk6KUlKxQJabe9YU7/zTCYK7XloY3ctVXd9O3g/3zzVEcnC0PjJNUQk32hKGVIC7QbWrSiEohmWyswkXYyCCJrLDHRdrSbtgn+zSffFtPjrt/r+p+OO6cnRTsOnGfOc+eV4ztvnVNn4AydiUPcF+5797M79l57H72pN9uHuo0i56lTWx7+A6+IMmI=</latexit><latexit sha1_base64="bpumDvr+zEjXz9/NWhJUJNc4pMA=">AAADp3icbZLfb9MwEMfThB+jUNbBIy8WBQl4qJq9sMdNZVKF0Cil7To1pXLcy2rVjiPbAYqVP41/gjf+G9w0CWTDknNff+5OZ18uTBhVutf73XC9O3fv3T940Hz4qPX4sH30ZKpEKglMiGBCzkKsgNEYJppqBrNEAuYhg8tw09/5L7+CVFTEY71NYMHxdUwjSrC2aHnU+NlqBTF802sQErhJpEgyM7RfoeguJKu5GXCOM/MhNzVPYTMzLkTNS4QUjGG5zUy/krWIFUQ03lc07/7qWozdWG4yM9rbVrMZ2DQU+CbgWK/DCPk2IUfnBQrReYmGFRqWaFShUYkGxgQEMzTISnJWkLOK9AvSr8isILOKXBXkqiJkbciXN+WJc9Mv9YW5yB+jrI4FjVcQaxRo+K7zH2xClkJmtEwBSao2iNvucPoDZLZsd3rdXr7QbeEXouMUa7hs/wpWgqTcViAMKzX3e4leGCw1JQyyZpAqSDDZ4GuYWxljDmph8mtk6KUlKxQJabe9YU7/zTCYK7XloY3ctVXd9O3g/3zzVEcnC0PjJNUQk32hKGVIC7QbWrSiEohmWyswkXYyCCJrLDHRdrSbtgn+zSffFtPjrt/r+p+OO6cnRTsOnGfOc+eV4ztvnVNn4AydiUPcF+5797M79l57H72pN9uHuo0i56lTWx7+A6+IMmI=</latexit><latexit sha1_base64="bpumDvr+zEjXz9/NWhJUJNc4pMA=">AAADp3icbZLfb9MwEMfThB+jUNbBIy8WBQl4qJq9sMdNZVKF0Cil7To1pXLcy2rVjiPbAYqVP41/gjf+G9w0CWTDknNff+5OZ18uTBhVutf73XC9O3fv3T940Hz4qPX4sH30ZKpEKglMiGBCzkKsgNEYJppqBrNEAuYhg8tw09/5L7+CVFTEY71NYMHxdUwjSrC2aHnU+NlqBTF802sQErhJpEgyM7RfoeguJKu5GXCOM/MhNzVPYTMzLkTNS4QUjGG5zUy/krWIFUQ03lc07/7qWozdWG4yM9rbVrMZ2DQU+CbgWK/DCPk2IUfnBQrReYmGFRqWaFShUYkGxgQEMzTISnJWkLOK9AvSr8isILOKXBXkqiJkbciXN+WJc9Mv9YW5yB+jrI4FjVcQaxRo+K7zH2xClkJmtEwBSao2iNvucPoDZLZsd3rdXr7QbeEXouMUa7hs/wpWgqTcViAMKzX3e4leGCw1JQyyZpAqSDDZ4GuYWxljDmph8mtk6KUlKxQJabe9YU7/zTCYK7XloY3ctVXd9O3g/3zzVEcnC0PjJNUQk32hKGVIC7QbWrSiEohmWyswkXYyCCJrLDHRdrSbtgn+zSffFtPjrt/r+p+OO6cnRTsOnGfOc+eV4ztvnVNn4AydiUPcF+5797M79l57H72pN9uHuo0i56lTWx7+A6+IMmI=</latexit>
f⇤ = argminf2F
R(f)
bf = argminf2F
bR(f)<latexit sha1_base64="4tCcmg1tPzZut6bYSupsgXZXOec=">AAAEwHichZPfT9swEMcD7TaWrRtsj3uxhjYxHlDDHjZNQoIVJh4QKh2FItxVjuM0Fv4RHGeQWfkn9zBp/82c1g0UkObI8Vefu/PlznGYMprpdvvvwmKj+ejxk6Wn/rPnrRcvl1denWQyV5j0sWRSDUKUEUYF6WuqGRmkiiAeMnIaXnQq++lPojIqxbEuUjLkaCxoTDHSFo1WFv+0WlCQK50QqQg3qZJpabr2LTNauZRzZkY4R6U5mCxzFreW5tiJOSuWSjKGVFGaTi3nPCISUzHNaHZv9JyPnUhdlKY3XVu+D20YgIGBHOkkjEFgAyZoz6EQ7M1Qt0bd0gUe1eho5tWrUW+G9o2BGDGwX87IjiM7Nek40qnJwJFBTc4cOSsf2Nl9Ud+Rfk0OHDmoSZgbGEoWZQW3i8lrA04M/rE+25xz05npQ3No21X1Ekt7dCKye34vb28DdwnTqE5xPZfiukqRFGlCREoEYrrYCtp22AMQkoqICA1gFgMfhmRMhSGXAimFivXSj3+sg/dgy06I1BhyKkYmhlSAaVXfyhL01uIPAEJgH+jDKxqRBGkQ/y+s9qzifUhsUTd5R8ur7Y32ZID7InBi1XOjO1r+DSOJc25LwQxl2XnQTvXQIKUpZsSWn2ckRfgCjcm5lQJxkg3N5AKW4J0lEYilstO2YkJvRxjEs6qV1rP6ubK7tgo+ZDvPdfx5aKhIc00EniaKcwa0BNVtBhFVBGtWWIGwslcGA5wghbC2d963TQjulnxfnGxuBB83gqPN1e0vrh1L3hvvrbfmBd4nb9vb97pe38ONrQZusAZvfm0mTdm8nLouLriY197caP76B+BplO8=</latexit>
R( bf) bR( bf) + 6
rd log(n/�)
n<latexit sha1_base64="ypzaxYPsuuEjjz1gpDBpuu5Lr7c=">AAAEpnicbZPdTtswGIYD7TbWrRtshzuxVk0qTGINkzZOJoEKEpMQKoVCEe7AcZzGwrGD7QwqK3e2q9jZ7mZO6wYCWHL86vl+8yUOUkaV7nT+LSzW6s+ev1h62Xj1uvnm7fLKuxMlMonJAAsm5DBAijDKyUBTzcgwlQQlASOnwVW3sJ/+JlJRwY/1JCWjBI05jShG2qKLlcU/zSbk5EbHREiSmFSKNDc9+xSKFi55xcxIkqDc7E+PisWduTl2omLFQgrGkJzkplvKikdIIspnFc3Ona742I3kVW76s7PZaEAbBqBvYIJ0HETAtwFTtOtQAHbnqFeiXu4CD0t0OPfql6g/R3vGQIwY2MvnZNuR7ZJ0HemWZOjIsCRnjpzlT2R2HQ0cGZRk35H9kgSZgYFgoZok9jBZacCxwb/W5smTxHTn+sAc2HEVs8TCfjoe2pxH+f00cIcwjcoSt5USt0WJeJLGhKeEI6YnP/yOXfYDcEF5SLgGUEWgAQMyptyQa46kRJO1vNFvwxsakhhpEK0CyMg1KEHV9hl8g+paagMjibAJIRPjNv9iG7KNreaGF00Q2/pd9ovlVme9M13gsfCdaHlu9S6W/8JQ4CyxDWOGlDr3O6keGSQ1xYzY/JkiKcJXaEzOreQoIWpkptcsB58sCUEkpN32haf0foRBiSoGZj2LX0g9tBXwKdt5pqPNkaE8zTTheFYoyhjQAhR3FoRUEqzZxAqEpb0YGOAY2SFpe7Mbdgj+w1d+LE421v2v6/7hRmtr041jyfvgffTanu9997a8Pa/nDTxca9V+1vq1o3q7flAf1E9nrosLLua9V1n1y/+XMY5Y</latexit>
R(f⇤) � bR(f⇤)� 6
rd log(n/�)
n<latexit sha1_base64="uv9aAuHxfnOlEhEZ/ObRFHpmsCo=">AAAEmHicbZNvT9QwHMcHdyqenoLGJ/qkkZgAiecNEyUxJpCDiAkhx8nBEXqQruvuGrp2tJ2wNHtLvhif+W7s7nqDAU26fvP5/d1va5AwqnS7/W9uvlZ/9PjJwtPGs+fNFy8Xl14dKZFKTPpYMCEHAVKEUU76mmpGBokkKA4YOQ4uOoX9+DeRigp+qLOEDGM04jSiGGmLzpfm/zSbkJMrPSZCktgkUiS56dqnULRwyStmRuIY5WZvclQs7szNoRMVKxZSMIZklptOKSseIYkon1Y02ze64mM3khe56U3PZqMBbRiAvoEx0uMgAr4NmKAdhwKwM0PdEnVzF3hQooOZV69EvRnaNQZixMBuPiNbjmyVpONIpyQDRwYlOXHkJH8gs+uo70i/JHuO7JUkSA0MBAtVFtvDpKUBjw0+W5slj2PTmel9s2/HVcwSC/vpeGhz/spvp4HbhGlUlriulLguSoyzZEx4QjhiOvvut+2yH4ALykPCNYAqAg0YkBHlhlxyJCXK1vJGbyU6W1sFcEQuAbyiIRkjDRz8CL5AdSm1gZFE2ISQidEK/2RbsK2s5oYXZYlt9ibf+eJyu9WeLHBf+E4se251zxf/wlDgNLYtYoaUOvXbiR4aJDXFjNj8qSIJwhdoRE6t5CgmamgmFysHHywJQSSk3fYVJ/R2hEGxKkZkPYufRt21FfAh22mqo42hoTxJNeF4WihKGdACFLcUhFQSrFlmBcLSXgUM8BjZIWl7lxt2CP7dV74vjtZb/ueWf7C+vLnhxrHgvfPeeyue7331Nr1dr+v1PVx7U/tW267t1N/WN+s/6j+nrvNzLua1V1n13n+KH4gV</latexit>
12
rd log(n/�)
n<latexit sha1_base64="GnDK2yP8/w+2bbKd3jgK5tWNSdk=">AAAElnicbZNNT9swGMcD7TbWrRuMy6RdrKEhxoE17DB22AQraBwQKoxCEekqx3nSRjh2sJ2NyvI32qfZbd9mTusGAlhy/NfvebOf2GFGE6larX9z87X6o8dPFp42nj1vvni5uPTqVPJcEOgSTrnohVgCTRh0VaIo9DIBOA0pnIWX7cJ+9guETDg7UeMM+ikesiROCFYWDZbm/zSbAYPfagRcQKozwTOjO/bLZVK4mIqZQppiow8mS8XiVqNPnKhYCRecUizGRrdLWfGIIE7YtKLevdEVHzuxuDT6eLo2G43AhqHA10GK1SiMkW8DJmjPoRDtzVCnRB3jAo9KdDTzOi7R8Qztax0QTNG+mZEdR3ZK0nakXZKeI72SnDtybh7I7HbUdaRbkgNHDkoS5joIOY3kOLWLzksDGWnyc32WPE11e6YP9aFtV9FLwu2vY5HN+cPcThPsAlW4LHFdKXFdlBiNsxGwDBimavzFb9lhfwDjCYuAKRRIG6fgWk0upQ5pDkY3ghCGCdNwxbAQeLxuGqsooHCFVpG/GcgroXQQC0x0FFA+XGMfbH27j/dGs6Im2J3exJrB4kprozUZ6L7wnVjx3OgMFv8GESd5ajdIKJbywm9lqq+xUAmhYAvkEjJMLvEQLqxkOAXZ15MTGPTOkgjFXNhpDzihtyM0TmXRIOtZXBl511bAh2wXuYq3+jphWa6AkWmhOKdIcVS8URQlAoiiYyswEfYhEERG2HZJ2ZfcsE3w7x75vjjd3PA/bvhHmyvbW64dC94b76235vneJ2/b2/c6XtcjteXa59q3Wrv+uv61vlf/PnWdn3Mxy15l1Dv/ATG2iYY=</latexit>
Goal: select hypothesis with true error rate within ✏ > 0 of minf2F R(f)<latexit sha1_base64="aMQNp2TlCGApiYyduI5Id4Vt+xk=">AAAD4XicbZNNb9MwGMfThpdRKHRw5GLRIg0OVTMOTBzQpjLoAU2lWrdOTVc5rrNY9UtkO0yVlRsXDiDElW/FjU/CFTdNAtmwlPjv3/95ZPuxHcSUKN3r/arV3Rs3b93eutO4e695/0Fr++GJEolEeIwEFXISQIUp4XisiaZ4EksMWUDxabDsr/3Tj1gqIvixXsV4xuAFJyFBUFs03679bjZ9ji91hIXEzMRSxKkZ2r9QZB2SVmyKGYOpeZ91FSfvU3Oci4qLhBSUQrlKTb+UlYgFDgnfzGje/NWVGPtBuUzNaNM3Gw3fpgHfMz6DOgpC4NmEDB3mKACHBRqWaFigUYlGBRoY4yNIwSAtyEFODkrSz0m/JJOcTEpylpOzkqDIoPPnxYgx0y/0kTnKNqOs5oLwBeYavBOQvgL2ZDHSIFrFwlZBEQUuiY6AlgkGWEohgYQaZ5Bw0PFxrAgV/HWvA0Rox4zwuQl9622W8zZNwWgnfNaZt9q9bi9r4LrwctF28jact376C4ESZteGKFRq6vViPTNQaoIoTht+onAM0RJe4KmVHDKsZia7oSl4askChHa5obB7y+i/GQYypVYssJHrA1FXvTX8nzdNdLg3M4THicYcbSYKEwq0AOvrDhZE2vrRlRUQSXunEEARlBBp+ygatgje1S1fFye7Xe9F1/uw297fy8ux5Tx2njg7jue8dPadgTN0xg6qn9c/1b/Uv7rI/ex+c79vQuu1POeRU2nujz8GVkbi</latexit>
su�cient numberof training examples:
<latexit sha1_base64="1LtGrTG8rs9j7ZksoQ9EjmLxiLY=">AAADvXicbZJdb9MwFIbTlo9RKHRwyY1FhYS4qJLdMO2GTWVSL9BUqnXr1JThuCetqT8i22GrrPxJuOLf4KZJIBuWHL953nP8cewoYVQb3//daLYePHz0eO9J++mzzvMX3f2XF1qmisCESCbVNMIaGBUwMdQwmCYKMI8YXEbrwda//AFKUynOzSaBOcdLQWNKsHHoer/xq9MJBdyYFUgF3CZKJpkdua/UdBuS1WwGnOPMfs6HmlOMmT0vRM0lUknGsNpkdlDJWsQCYip2K9pPf3UtxnWs1pkd78ZOux26NBQGNuTYrKIYBS4hR6cFitBpiUYVGpVoXKFxiYbWhgQzNMxKclKQk4oMCjKoyLQg04pcFeSqImRlydf35R/ndlDqM3uWH0Y7LSQVCxAGhQZuTX7B7tyLzOo0dtdGt5ZIeQQKhSGSMTIKu2KJJYJbzBMG+ii77vb8vp83dF8Eheh5RRtdd3+GC0lS7iYnDGs9C/zEzC1WhhIGWTtMNSSYrPESZk4KzEHPbb65DL11ZIFiqVx3m8vpvxkWc603PHKR22Lru94W/s+bpSY+nFsqktSAILuF4pQhI9H2KaMFVUAM2ziBiXLvhSCywgoT4x582xUhuHvk++LioB/4/eDLQe/4sCjHnvfae+O98wLvg3fsDb2RN/FI86j5rUmb31sfW9BiLbELbTaKnFderbVu/gCy6jsf</latexit><latexit sha1_base64="1LtGrTG8rs9j7ZksoQ9EjmLxiLY=">AAADvXicbZJdb9MwFIbTlo9RKHRwyY1FhYS4qJLdMO2GTWVSL9BUqnXr1JThuCetqT8i22GrrPxJuOLf4KZJIBuWHL953nP8cewoYVQb3//daLYePHz0eO9J++mzzvMX3f2XF1qmisCESCbVNMIaGBUwMdQwmCYKMI8YXEbrwda//AFKUynOzSaBOcdLQWNKsHHoer/xq9MJBdyYFUgF3CZKJpkdua/UdBuS1WwGnOPMfs6HmlOMmT0vRM0lUknGsNpkdlDJWsQCYip2K9pPf3UtxnWs1pkd78ZOux26NBQGNuTYrKIYBS4hR6cFitBpiUYVGpVoXKFxiYbWhgQzNMxKclKQk4oMCjKoyLQg04pcFeSqImRlydf35R/ndlDqM3uWH0Y7LSQVCxAGhQZuTX7B7tyLzOo0dtdGt5ZIeQQKhSGSMTIKu2KJJYJbzBMG+ii77vb8vp83dF8Eheh5RRtdd3+GC0lS7iYnDGs9C/zEzC1WhhIGWTtMNSSYrPESZk4KzEHPbb65DL11ZIFiqVx3m8vpvxkWc603PHKR22Lru94W/s+bpSY+nFsqktSAILuF4pQhI9H2KaMFVUAM2ziBiXLvhSCywgoT4x582xUhuHvk++LioB/4/eDLQe/4sCjHnvfae+O98wLvg3fsDb2RN/FI86j5rUmb31sfW9BiLbELbTaKnFderbVu/gCy6jsf</latexit><latexit sha1_base64="1LtGrTG8rs9j7ZksoQ9EjmLxiLY=">AAADvXicbZJdb9MwFIbTlo9RKHRwyY1FhYS4qJLdMO2GTWVSL9BUqnXr1JThuCetqT8i22GrrPxJuOLf4KZJIBuWHL953nP8cewoYVQb3//daLYePHz0eO9J++mzzvMX3f2XF1qmisCESCbVNMIaGBUwMdQwmCYKMI8YXEbrwda//AFKUynOzSaBOcdLQWNKsHHoer/xq9MJBdyYFUgF3CZKJpkdua/UdBuS1WwGnOPMfs6HmlOMmT0vRM0lUknGsNpkdlDJWsQCYip2K9pPf3UtxnWs1pkd78ZOux26NBQGNuTYrKIYBS4hR6cFitBpiUYVGpVoXKFxiYbWhgQzNMxKclKQk4oMCjKoyLQg04pcFeSqImRlydf35R/ndlDqM3uWH0Y7LSQVCxAGhQZuTX7B7tyLzOo0dtdGt5ZIeQQKhSGSMTIKu2KJJYJbzBMG+ii77vb8vp83dF8Eheh5RRtdd3+GC0lS7iYnDGs9C/zEzC1WhhIGWTtMNSSYrPESZk4KzEHPbb65DL11ZIFiqVx3m8vpvxkWc603PHKR22Lru94W/s+bpSY+nFsqktSAILuF4pQhI9H2KaMFVUAM2ziBiXLvhSCywgoT4x582xUhuHvk++LioB/4/eDLQe/4sCjHnvfae+O98wLvg3fsDb2RN/FI86j5rUmb31sfW9BiLbELbTaKnFderbVu/gCy6jsf</latexit><latexit sha1_base64="1LtGrTG8rs9j7ZksoQ9EjmLxiLY=">AAADvXicbZJdb9MwFIbTlo9RKHRwyY1FhYS4qJLdMO2GTWVSL9BUqnXr1JThuCetqT8i22GrrPxJuOLf4KZJIBuWHL953nP8cewoYVQb3//daLYePHz0eO9J++mzzvMX3f2XF1qmisCESCbVNMIaGBUwMdQwmCYKMI8YXEbrwda//AFKUynOzSaBOcdLQWNKsHHoer/xq9MJBdyYFUgF3CZKJpkdua/UdBuS1WwGnOPMfs6HmlOMmT0vRM0lUknGsNpkdlDJWsQCYip2K9pPf3UtxnWs1pkd78ZOux26NBQGNuTYrKIYBS4hR6cFitBpiUYVGpVoXKFxiYbWhgQzNMxKclKQk4oMCjKoyLQg04pcFeSqImRlydf35R/ndlDqM3uWH0Y7LSQVCxAGhQZuTX7B7tyLzOo0dtdGt5ZIeQQKhSGSMTIKu2KJJYJbzBMG+ii77vb8vp83dF8Eheh5RRtdd3+GC0lS7iYnDGs9C/zEzC1WhhIGWTtMNSSYrPESZk4KzEHPbb65DL11ZIFiqVx3m8vpvxkWc603PHKR22Lru94W/s+bpSY+nFsqktSAILuF4pQhI9H2KaMFVUAM2ziBiXLvhSCywgoT4x582xUhuHvk++LioB/4/eDLQe/4sCjHnvfae+O98wLvg3fsDb2RN/FI86j5rUmb31sfW9BiLbELbTaKnFderbVu/gCy6jsf</latexit>
12
rd log(n/�)
n ✏
<latexit sha1_base64="k6b3p7ASnxTVHUysyUh++Y3RPyw=">AAAEoHicbZNPT9swFMAD7TaWrRtsx2lSBJrEOLCGHcZlEqgg9dCxllEowx1ynJc2wrGD7WxUlr/WPshu+zZzWjdQmKXkPf3eXz/bUU5TqZrNv0vLtfqjx09WnvrPnjdevFxde3UqeSEI9AmnXAwiLIGmDPoqVRQGuQCcRRTOoqtWaT/7CUKmnJ2oSQ7DDI9YmqQEK4su15Z/NxqIwS81Bi4g07ngudFd++cyLV3MgplClmGjO1OxYHHS6BOnLFgJF5xSLCZGtyp1wSOGJGWzivrgVl/wsR8WV0Yfz2TD95ENC1CoUYbVOEqC0AZM0aFDUXA4R90KdY0L7FWoN/c6rtDxHLW1RgTToG3mZN+R/Yq0HGlVZODIoCLnjpyb/2R2HfUd6Vek40inIlGhUcRpLCeZFbqoDGSsyY+tefIs0625fqSP7LjKWRJuj47FNuc3czcNOgCqcFXiZqHETVliPMnHwHJgmKrJ57Bplz0AxlMWA1MBkjZOwY2aXkodUUzsUfkoglHKNFwzLASebBk/3EHyWiiNEoGJjhHlo032wda19d8bzYwJUIAoXJcCcplSznwEtufbJOZydaO53Zyu4KESOmXDc6t7ufoHxZwUmW2VUCzlRdjM1VBjoVJCwW6vkJDblvEILqzKcAZyqKd7McE7S+Ig4cJ+dqtTejdC40yWo7Ke5eWR920l/J/tolDJ7lCnLC8UMDIrlBQ0UDwoX2sQpwKIohOrYCLskyABGWM7N2XftG+HEN7f8kPldGc7/Lgd9nY29nbdOFa8N966t+mF3idvz2t7Xa/vkdrbWqvWqX2pr9fb9a/13sx1ecnFvPYWVv37P18GjiE=</latexit>
n = eO⇣d log(1/�)
✏2
⌘
<latexit sha1_base64="SJR9nsso93nJDyhS0fq2vYlYG+g=">AAAEq3icbZPdTtswFIAD7TbWrRtsl7uxhiYVpLGGXYwbJFhB4gKx8lMowwU5zmlr4diZ4wwqKw+3V9jd3mYnbRooYCk5R9/59bEdxFIkttn8NzdfqT57/mLhZe3V6/qbt4tL704TnRoOHa6lNt2AJSCFgo4VVkI3NsCiQMJZcN3K7We/wSRCqxM7iqEXsYESfcGZRXS1NP+nXqcKbuwQtIHIxUbHmWvjXycid8lmzBKiiGVufyxmLIXM3EmhzFi5NlpKZkaZa5XqjEcIfaEmFd3OnT7jgx8z15k7msh6rUYxjFDf0YjZYdAnPgaM0W6BArI7Re0StbMi8LBEh1OvoxIdTdGec5QzSfayKdkuyHZJWgVplaRbkG5Jzgtynj2RueioU5BOSfYLsl+SIHU00DJMRhEKl5YGPnT8cnWaPIpca6ofuAMcVz5LrvHoVIg5j7P7aegOSMvKErczJW7zEsNRPAQVg2LSjjb9Ji48AKWFCkFZQhOMs3Brx5fSBZJxPKoaDWAglINfihnDRqtZTRGySQi9ESFYIUMgP+h3MWjQvmHchYRKPWj4X7AP7GcFe4Q4EVKry/Us91upUcD27/JlV4vLzbXmeJHHil8oy16x2leLf2moeRph11yyJLnwm7HtOWas4BJwp2kCMXbPBnCBqmIRJD033lZGPiEJSV8b/HDXY3o/wrEoyaeGnvk9Sh7acviU7SK1/Y2eEypOLSg+KdRPJbGa5A+XhMIAt3KECuMGXwcnfMhwZhafdw2H4D/c8mPldH3N/7rmH64vb20U41jwPngfvYbne9+8LW/Pa3sdj1calYPKWaVb/Vw9rv6s0onr/FwR896bWVX4D8cIkRs=</latexit>
bf minimizes empirical risk:
bR( bf) bR(f⇤)<latexit sha1_base64="JFfuybFBfbjU2tcBHdGGn2WlglU=">AAACXnicbVFBT9swGHUCY5DC6MYFaRdrDRLboUrawyZOCC47AqKA1HSV43yhn2o7me0MdVH/5G7TLvspc0ukjsInWX5673v25+e0FGhsFP32/I3NV1uvt3eC1u7em/3223c3pqg0hwEvRKHvUmZAoIKBRSvgrtTAZCrgNp2eL/TbH6ANFurazkoYSXavMEfOrKPG7SpRBaoMlKWJyWmYPGAGE2ZpHlKJCiX+BENBlqidR1CNZnpCw1Xf1fHK8pEmNBHwfbGt9PzbJydMTMk41H1U8zActztRN1oWfQ7iBnRIUxfj9q8kK3gl3ZxcMGOGcVTaUc20RS5gHiSVAXf+lN3D0EHFJJhRvYxnTo8ck9G80G65dy7Z/x01k8bMZOo6JbMTs64tyJe0YWXzL6MaVVlZUPzxorwS1BZ0kTXNUAO3YuYA4xrdrJRPmGbcuh8JXAjx+pOfg5teN+5348te5/SsiWObvCcfyDGJyWdySr6SCzIgnPzxPC/wWt5ff8vf8/cfW32v8RyQJ+Uf/gM+bbKb</latexit>
error
hypotheses (ordered according to empirical risks)
· · ·
1 2 3 k-1 k
Empirical Risks and Confidence Intervals
hypotheses (ordered according to empirical risks)
· · ·
1 2 3 k-1 k
Empirical Risks and Confidence Intervals
more training data ) smaller confidence intervals<latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit><latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit><latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit><latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit>
hypotheses (ordered according to empirical risks)
· · ·
1 2 3 k-1 k
Empirical Risks and Confidence Intervals
more training data ) smaller confidence intervals<latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit><latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit><latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit><latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit>
hypotheses (ordered according to empirical risks)
· · ·
1 2 3 k-1 k
bR(f3)<latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit>
ERM is Wasting Labeled Examples
hypotheses (ordered according to empirical risks)1 2 3 k-1 k
ERM is Wasting Labeled Examples
· · ·bR(f3)<latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit>
at this point we can safely removef3 from further consideration
<latexit sha1_base64="vmSDb2u8HBmAr9vmd20H28xA4gY=">AAADyHicdZLfb9MwEMezlh+jUOjgkZcTGxLioUoKhYqnTaXShNBUqnXr1JTKcZzVqn9EtrOpsvLCn8gbL/wtuG0SKIKTnPv6c3eyfbkoZVQb3/+xV6vfuXvv/v6DxsNHzcdPWgdPL7TMFCZjLJlUkwhpwqggY0MNI5NUEcQjRi6jZX8dv7whSlMpzs0qJTOOrgVNKEbGofnB3s9mMxTk1iyIVITbVMk0t0P3lZquU/KdMCOco9x+3ridSOFze16InSiWSjKG1Cq3/UruZMQkoWJ7ov34W+/kuIXUMrejrW82GqErgzCwIUdmESUQuIINGhQogkGJhhUalmhUoVGJTq0NMWJwmpfkpCAnFekXpF+RSUEmFbkqyFVF8MLir6/LHee2X+oze7Z5jJBUxEQYCHUCyIBZUA2pgwZuCWAkQKOEsBW4RsgbAmEIR8n8zREkSnJIMuUapQBLoWlM1PYPtw79tv+u+zbogd/u+kEv6DrR6Qa+34Gg7W/s0CtsOG99D2OJM+6ugRnSehr4qZlZpAzFjOSNMNMkRXiJrsnUSYE40TO7GcQcXjoSQyKVW+7SG/pnhUVc6xWPXOa67/rv2Br+KzbNTNKbWSrSzBCBtwclGQMjYT3VEFNFsHGdiSnCyo0OBrxACmHjZr/hmlC+FP4vLjrtwG8HXzqHxx+Kdux7z70X3isv8N57x96pN/TGHq4NasuaqWX1T/W0fltfbVNre0XNM2/H6t9+AQH0PdQ=</latexit><latexit sha1_base64="vmSDb2u8HBmAr9vmd20H28xA4gY=">AAADyHicdZLfb9MwEMezlh+jUOjgkZcTGxLioUoKhYqnTaXShNBUqnXr1JTKcZzVqn9EtrOpsvLCn8gbL/wtuG0SKIKTnPv6c3eyfbkoZVQb3/+xV6vfuXvv/v6DxsNHzcdPWgdPL7TMFCZjLJlUkwhpwqggY0MNI5NUEcQjRi6jZX8dv7whSlMpzs0qJTOOrgVNKEbGofnB3s9mMxTk1iyIVITbVMk0t0P3lZquU/KdMCOco9x+3ridSOFze16InSiWSjKG1Cq3/UruZMQkoWJ7ov34W+/kuIXUMrejrW82GqErgzCwIUdmESUQuIINGhQogkGJhhUalmhUoVGJTq0NMWJwmpfkpCAnFekXpF+RSUEmFbkqyFVF8MLir6/LHee2X+oze7Z5jJBUxEQYCHUCyIBZUA2pgwZuCWAkQKOEsBW4RsgbAmEIR8n8zREkSnJIMuUapQBLoWlM1PYPtw79tv+u+zbogd/u+kEv6DrR6Qa+34Gg7W/s0CtsOG99D2OJM+6ugRnSehr4qZlZpAzFjOSNMNMkRXiJrsnUSYE40TO7GcQcXjoSQyKVW+7SG/pnhUVc6xWPXOa67/rv2Br+KzbNTNKbWSrSzBCBtwclGQMjYT3VEFNFsHGdiSnCyo0OBrxACmHjZr/hmlC+FP4vLjrtwG8HXzqHxx+Kdux7z70X3isv8N57x96pN/TGHq4NasuaqWX1T/W0fltfbVNre0XNM2/H6t9+AQH0PdQ=</latexit><latexit sha1_base64="vmSDb2u8HBmAr9vmd20H28xA4gY=">AAADyHicdZLfb9MwEMezlh+jUOjgkZcTGxLioUoKhYqnTaXShNBUqnXr1JTKcZzVqn9EtrOpsvLCn8gbL/wtuG0SKIKTnPv6c3eyfbkoZVQb3/+xV6vfuXvv/v6DxsNHzcdPWgdPL7TMFCZjLJlUkwhpwqggY0MNI5NUEcQjRi6jZX8dv7whSlMpzs0qJTOOrgVNKEbGofnB3s9mMxTk1iyIVITbVMk0t0P3lZquU/KdMCOco9x+3ridSOFze16InSiWSjKG1Cq3/UruZMQkoWJ7ov34W+/kuIXUMrejrW82GqErgzCwIUdmESUQuIINGhQogkGJhhUalmhUoVGJTq0NMWJwmpfkpCAnFekXpF+RSUEmFbkqyFVF8MLir6/LHee2X+oze7Z5jJBUxEQYCHUCyIBZUA2pgwZuCWAkQKOEsBW4RsgbAmEIR8n8zREkSnJIMuUapQBLoWlM1PYPtw79tv+u+zbogd/u+kEv6DrR6Qa+34Gg7W/s0CtsOG99D2OJM+6ugRnSehr4qZlZpAzFjOSNMNMkRXiJrsnUSYE40TO7GcQcXjoSQyKVW+7SG/pnhUVc6xWPXOa67/rv2Br+KzbNTNKbWSrSzBCBtwclGQMjYT3VEFNFsHGdiSnCyo0OBrxACmHjZr/hmlC+FP4vLjrtwG8HXzqHxx+Kdux7z70X3isv8N57x96pN/TGHq4NasuaqWX1T/W0fltfbVNre0XNM2/H6t9+AQH0PdQ=</latexit><latexit sha1_base64="vmSDb2u8HBmAr9vmd20H28xA4gY=">AAADyHicdZLfb9MwEMezlh+jUOjgkZcTGxLioUoKhYqnTaXShNBUqnXr1JTKcZzVqn9EtrOpsvLCn8gbL/wtuG0SKIKTnPv6c3eyfbkoZVQb3/+xV6vfuXvv/v6DxsNHzcdPWgdPL7TMFCZjLJlUkwhpwqggY0MNI5NUEcQjRi6jZX8dv7whSlMpzs0qJTOOrgVNKEbGofnB3s9mMxTk1iyIVITbVMk0t0P3lZquU/KdMCOco9x+3ridSOFze16InSiWSjKG1Cq3/UruZMQkoWJ7ov34W+/kuIXUMrejrW82GqErgzCwIUdmESUQuIINGhQogkGJhhUalmhUoVGJTq0NMWJwmpfkpCAnFekXpF+RSUEmFbkqyFVF8MLir6/LHee2X+oze7Z5jJBUxEQYCHUCyIBZUA2pgwZuCWAkQKOEsBW4RsgbAmEIR8n8zREkSnJIMuUapQBLoWlM1PYPtw79tv+u+zbogd/u+kEv6DrR6Qa+34Gg7W/s0CtsOG99D2OJM+6ugRnSehr4qZlZpAzFjOSNMNMkRXiJrsnUSYE40TO7GcQcXjoSQyKVW+7SG/pnhUVc6xWPXOa67/rv2Br+KzbNTNKbWSrSzBCBtwclGQMjYT3VEFNFsHGdiSnCyo0OBrxACmHjZr/hmlC+FP4vLjrtwG8HXzqHxx+Kdux7z70X3isv8N57x96pN/TGHq4NasuaqWX1T/W0fltfbVNre0XNM2/H6t9+AQH0PdQ=</latexit>
and we probably could have removedother hypotheses even sooner
<latexit sha1_base64="S7G0oKTPHCcf5R7g3HVkTMxHLao=">AAADxHicdZLfb9MwEMfTlh+jUNjgkZcTFRLioUoqChNPm8rQHtBUpnXrtJTKcS6rNccOttOpssIfyRv8NbhtEiiCk5z75nN3cu5yUcaZNr7/o9Fs3bl77/7Og/bDR53HT3b3np5rmSuKYyq5VJOIaORM4Ngww3GSKSRpxPEiuhmu4hcLVJpJcWaWGU5Tci1YwigxDs32Gj87nVDgrZmjVJjaTMmssCP3lJqtUoqtMMc0JYX9tHZbkdIX9qwUW1EqleScqGVhh7XcyogxYWJzo/3wW2/luEPUTWFPN77TboeuDMLAhikx8yiBwBWs0VGJIjiq0KhGowqd1ui0QsfWhpRwOC4qcliSw5oMSzKsyaQkk5pcluSyJnRu6ZfX1Vua2mGlT+zJuhkhmYhRGAh1AkTEcIvgfkhEIr4EKnMew5wsENwc5AJjCEOQbjYK5stsJTRqwAUK0FIKVLPdrt/z3w7eBPvg9wZ+sB8MnOgPAt/vQ9Dz19b1ShvNdr+HsaR56j6BcqL1VeBnZmqJMoxyLNphrjEj9IZc45WTgqSop3a9hAW8dCSGRCp3XAtr+meFJanWyzRymauZ679jK/iv2FVukv2pZSLLDQq6uSjJORgJq42GmCmkxk0oZoQqtzYU6JwoQo3b+7YbQtUp/F+c93uB3ws+97sH78tx7HjPvRfeKy/w3nkH3rE38sYebR42r5tZ82vrY4u3dCvfpDYbZc0zb8ta334B4bk9GA==</latexit><latexit sha1_base64="S7G0oKTPHCcf5R7g3HVkTMxHLao=">AAADxHicdZLfb9MwEMfTlh+jUNjgkZcTFRLioUoqChNPm8rQHtBUpnXrtJTKcS6rNccOttOpssIfyRv8NbhtEiiCk5z75nN3cu5yUcaZNr7/o9Fs3bl77/7Og/bDR53HT3b3np5rmSuKYyq5VJOIaORM4Ngww3GSKSRpxPEiuhmu4hcLVJpJcWaWGU5Tci1YwigxDs32Gj87nVDgrZmjVJjaTMmssCP3lJqtUoqtMMc0JYX9tHZbkdIX9qwUW1EqleScqGVhh7XcyogxYWJzo/3wW2/luEPUTWFPN77TboeuDMLAhikx8yiBwBWs0VGJIjiq0KhGowqd1ui0QsfWhpRwOC4qcliSw5oMSzKsyaQkk5pcluSyJnRu6ZfX1Vua2mGlT+zJuhkhmYhRGAh1AkTEcIvgfkhEIr4EKnMew5wsENwc5AJjCEOQbjYK5stsJTRqwAUK0FIKVLPdrt/z3w7eBPvg9wZ+sB8MnOgPAt/vQ9Dz19b1ShvNdr+HsaR56j6BcqL1VeBnZmqJMoxyLNphrjEj9IZc45WTgqSop3a9hAW8dCSGRCp3XAtr+meFJanWyzRymauZ679jK/iv2FVukv2pZSLLDQq6uSjJORgJq42GmCmkxk0oZoQqtzYU6JwoQo3b+7YbQtUp/F+c93uB3ws+97sH78tx7HjPvRfeKy/w3nkH3rE38sYebR42r5tZ82vrY4u3dCvfpDYbZc0zb8ta334B4bk9GA==</latexit><latexit sha1_base64="S7G0oKTPHCcf5R7g3HVkTMxHLao=">AAADxHicdZLfb9MwEMfTlh+jUNjgkZcTFRLioUoqChNPm8rQHtBUpnXrtJTKcS6rNccOttOpssIfyRv8NbhtEiiCk5z75nN3cu5yUcaZNr7/o9Fs3bl77/7Og/bDR53HT3b3np5rmSuKYyq5VJOIaORM4Ngww3GSKSRpxPEiuhmu4hcLVJpJcWaWGU5Tci1YwigxDs32Gj87nVDgrZmjVJjaTMmssCP3lJqtUoqtMMc0JYX9tHZbkdIX9qwUW1EqleScqGVhh7XcyogxYWJzo/3wW2/luEPUTWFPN77TboeuDMLAhikx8yiBwBWs0VGJIjiq0KhGowqd1ui0QsfWhpRwOC4qcliSw5oMSzKsyaQkk5pcluSyJnRu6ZfX1Vua2mGlT+zJuhkhmYhRGAh1AkTEcIvgfkhEIr4EKnMew5wsENwc5AJjCEOQbjYK5stsJTRqwAUK0FIKVLPdrt/z3w7eBPvg9wZ+sB8MnOgPAt/vQ9Dz19b1ShvNdr+HsaR56j6BcqL1VeBnZmqJMoxyLNphrjEj9IZc45WTgqSop3a9hAW8dCSGRCp3XAtr+meFJanWyzRymauZ679jK/iv2FVukv2pZSLLDQq6uSjJORgJq42GmCmkxk0oZoQqtzYU6JwoQo3b+7YbQtUp/F+c93uB3ws+97sH78tx7HjPvRfeKy/w3nkH3rE38sYebR42r5tZ82vrY4u3dCvfpDYbZc0zb8ta334B4bk9GA==</latexit><latexit sha1_base64="S7G0oKTPHCcf5R7g3HVkTMxHLao=">AAADxHicdZLfb9MwEMfTlh+jUNjgkZcTFRLioUoqChNPm8rQHtBUpnXrtJTKcS6rNccOttOpssIfyRv8NbhtEiiCk5z75nN3cu5yUcaZNr7/o9Fs3bl77/7Og/bDR53HT3b3np5rmSuKYyq5VJOIaORM4Ngww3GSKSRpxPEiuhmu4hcLVJpJcWaWGU5Tci1YwigxDs32Gj87nVDgrZmjVJjaTMmssCP3lJqtUoqtMMc0JYX9tHZbkdIX9qwUW1EqleScqGVhh7XcyogxYWJzo/3wW2/luEPUTWFPN77TboeuDMLAhikx8yiBwBWs0VGJIjiq0KhGowqd1ui0QsfWhpRwOC4qcliSw5oMSzKsyaQkk5pcluSyJnRu6ZfX1Vua2mGlT+zJuhkhmYhRGAh1AkTEcIvgfkhEIr4EKnMew5wsENwc5AJjCEOQbjYK5stsJTRqwAUK0FIKVLPdrt/z3w7eBPvg9wZ+sB8MnOgPAt/vQ9Dz19b1ShvNdr+HsaR56j6BcqL1VeBnZmqJMoxyLNphrjEj9IZc45WTgqSop3a9hAW8dCSGRCp3XAtr+meFJanWyzRymauZ679jK/iv2FVukv2pZSLLDQq6uSjJORgJq42GmCmkxk0oZoQqtzYU6JwoQo3b+7YbQtUp/F+c93uB3ws+97sH78tx7HjPvRfeKy/w3nkH3rE38sYebR42r5tZ82vrY4u3dCvfpDYbZc0zb8ta334B4bk9GA==</latexit>
only require labels for examples that hypotheses 1 and 2 label differently (i.e., examples where they disagree)
Disagreement-Based Active Learning
consider points uniform on unit ball andlinear classifiers passing through origin
<latexit sha1_base64="7RzfEZM/sGdvL6/YfQwlqXI87hY=">AAAD2HicbZLPb9MwFMfTlh+jUOjgyOWJCglxqJpdhjhtKpN6QFOp1q2jKZPjuI1V/4hsB1RZkTiAEFf+NG78E/wNOGkayMaTHH/zee/Zfn4OE0a1GQx+NZqtW7fv3N27177/oPPwUXf/8bmWqcJkiiWTahYiTRgVZGqoYWSWKIJ4yMhFuB7m/ouPRGkqxZnZJGTB0UrQJcXIOHS13/jd6QSCfDIxkYpwmyiZZHbsvlLTPCSruRnhHGX2bTHVPOWc2bNS1LxYKskYUpvMDitZi4jIkortjvbNX12LcQOpdWYn27nTbgcuDQLfBhyZOFyC7xIKdFKiEE52aFyh8Q5NKjTZoZG1AUYMRtmOHJfkuCLDkgwrMivJrCKXJbmsCI4t/vBy98e5He70qT0tihGSiogIA4FeApZC04goSBw1GlLXNqk4SJFLAyFiDJCIIAgg7z5SAJghrV13XcMhyaVYgYmVTFcxSEVXVFx1e4P+oDC4KfxS9LzSxlfdn0EkccrdsYrV5/4gMQuLlKGYkawdpJokCK/RisydFIgTvbDFw8zguSMRuGO74coq6L8ZFnGtNzx0kXkf9HVfDv/nm6dm+WphqUhSQwTebrRMGRgJ+SuHiCqCDds4gbByTwkDjpFC2LirabtL8K+XfFOcH/T9Qd9/d9A7el1ex5731HvmvfB879A78kbe2Jt6uDlt2uaX5tfW+9bn1rfW921os1HmPPFq1vrxBzY9RTE=</latexit><latexit sha1_base64="7RzfEZM/sGdvL6/YfQwlqXI87hY=">AAAD2HicbZLPb9MwFMfTlh+jUOjgyOWJCglxqJpdhjhtKpN6QFOp1q2jKZPjuI1V/4hsB1RZkTiAEFf+NG78E/wNOGkayMaTHH/zee/Zfn4OE0a1GQx+NZqtW7fv3N27177/oPPwUXf/8bmWqcJkiiWTahYiTRgVZGqoYWSWKIJ4yMhFuB7m/ouPRGkqxZnZJGTB0UrQJcXIOHS13/jd6QSCfDIxkYpwmyiZZHbsvlLTPCSruRnhHGX2bTHVPOWc2bNS1LxYKskYUpvMDitZi4jIkortjvbNX12LcQOpdWYn27nTbgcuDQLfBhyZOFyC7xIKdFKiEE52aFyh8Q5NKjTZoZG1AUYMRtmOHJfkuCLDkgwrMivJrCKXJbmsCI4t/vBy98e5He70qT0tihGSiogIA4FeApZC04goSBw1GlLXNqk4SJFLAyFiDJCIIAgg7z5SAJghrV13XcMhyaVYgYmVTFcxSEVXVFx1e4P+oDC4KfxS9LzSxlfdn0EkccrdsYrV5/4gMQuLlKGYkawdpJokCK/RisydFIgTvbDFw8zguSMRuGO74coq6L8ZFnGtNzx0kXkf9HVfDv/nm6dm+WphqUhSQwTebrRMGRgJ+SuHiCqCDds4gbByTwkDjpFC2LirabtL8K+XfFOcH/T9Qd9/d9A7el1ex5731HvmvfB879A78kbe2Jt6uDlt2uaX5tfW+9bn1rfW921os1HmPPFq1vrxBzY9RTE=</latexit><latexit sha1_base64="7RzfEZM/sGdvL6/YfQwlqXI87hY=">AAAD2HicbZLPb9MwFMfTlh+jUOjgyOWJCglxqJpdhjhtKpN6QFOp1q2jKZPjuI1V/4hsB1RZkTiAEFf+NG78E/wNOGkayMaTHH/zee/Zfn4OE0a1GQx+NZqtW7fv3N27177/oPPwUXf/8bmWqcJkiiWTahYiTRgVZGqoYWSWKIJ4yMhFuB7m/ouPRGkqxZnZJGTB0UrQJcXIOHS13/jd6QSCfDIxkYpwmyiZZHbsvlLTPCSruRnhHGX2bTHVPOWc2bNS1LxYKskYUpvMDitZi4jIkortjvbNX12LcQOpdWYn27nTbgcuDQLfBhyZOFyC7xIKdFKiEE52aFyh8Q5NKjTZoZG1AUYMRtmOHJfkuCLDkgwrMivJrCKXJbmsCI4t/vBy98e5He70qT0tihGSiogIA4FeApZC04goSBw1GlLXNqk4SJFLAyFiDJCIIAgg7z5SAJghrV13XcMhyaVYgYmVTFcxSEVXVFx1e4P+oDC4KfxS9LzSxlfdn0EkccrdsYrV5/4gMQuLlKGYkawdpJokCK/RisydFIgTvbDFw8zguSMRuGO74coq6L8ZFnGtNzx0kXkf9HVfDv/nm6dm+WphqUhSQwTebrRMGRgJ+SuHiCqCDds4gbByTwkDjpFC2LirabtL8K+XfFOcH/T9Qd9/d9A7el1ex5731HvmvfB879A78kbe2Jt6uDlt2uaX5tfW+9bn1rfW921os1HmPPFq1vrxBzY9RTE=</latexit><latexit sha1_base64="7RzfEZM/sGdvL6/YfQwlqXI87hY=">AAAD2HicbZLPb9MwFMfTlh+jUOjgyOWJCglxqJpdhjhtKpN6QFOp1q2jKZPjuI1V/4hsB1RZkTiAEFf+NG78E/wNOGkayMaTHH/zee/Zfn4OE0a1GQx+NZqtW7fv3N27177/oPPwUXf/8bmWqcJkiiWTahYiTRgVZGqoYWSWKIJ4yMhFuB7m/ouPRGkqxZnZJGTB0UrQJcXIOHS13/jd6QSCfDIxkYpwmyiZZHbsvlLTPCSruRnhHGX2bTHVPOWc2bNS1LxYKskYUpvMDitZi4jIkortjvbNX12LcQOpdWYn27nTbgcuDQLfBhyZOFyC7xIKdFKiEE52aFyh8Q5NKjTZoZG1AUYMRtmOHJfkuCLDkgwrMivJrCKXJbmsCI4t/vBy98e5He70qT0tihGSiogIA4FeApZC04goSBw1GlLXNqk4SJFLAyFiDJCIIAgg7z5SAJghrV13XcMhyaVYgYmVTFcxSEVXVFx1e4P+oDC4KfxS9LzSxlfdn0EkccrdsYrV5/4gMQuLlKGYkawdpJokCK/RisydFIgTvbDFw8zguSMRuGO74coq6L8ZFnGtNzx0kXkf9HVfDv/nm6dm+WphqUhSQwTebrRMGRgJ+SuHiCqCDds4gbByTwkDjpFC2LirabtL8K+XfFOcH/T9Qd9/d9A7el1ex5731HvmvfB879A78kbe2Jt6uDlt2uaX5tfW+9bn1rfW921os1HmPPFq1vrxBzY9RTE=</latexit>
only label points in theregion of disagreement D
<latexit sha1_base64="ct3Eyv5JAlSrQdQBIHQLRnZdcDg=">AAADw3icbZJLb9NAEMcdm0cJBFI4clmRIiEOkd1LEadUaaQcUBWiPlLFIVqv18kq+7B216Bo5S/Jrd+GtWMb3DLSev7+zcw+RhOllCjt+/cd13vy9Nnzoxfdl696r9/0j9/eKJFJhK+RoEIuIqgwJRxfa6IpXqQSQxZRfBvtxkX89ieWigh+pfcpXjG44SQhCGqL1sed+14v5PiX3mIhMTOpFGluZvYrFClS8laYYsZgbr6VrhWpfG6uKtGKIiEFpVDuczNuZCsjxgnhhxPNxV/dyrELyl1u5gff63ZDWwbCwIQM6m2UgMAWlGhSoQhMajRr0KxG8wbNazQ1JkSQgmlek/OKnDdkXJFxQxYVWTTkriJ3DUFbg358rv8YM+NaX5rL8jFcEB5jrkGoEiA43QMKI0xBarlWgHBgOwHCEEi8sc0BIgExUXAjMWZF2Un5mkTCnbnIT9b9gT/0SwOPRVCJgVPZbN3/HcYCZcVWiEKlloGf6pWBUhNEcd4NM4VTiHZwg5dWcsiwWplyBnPw0ZIYJELaZa9S0n8rDGRK7VlkM4tLqoexAv4vtsx08mVlCE8zjTk6HJRkFGgBioG2HZAYadurmEAk7dQggLZQQqTt2HdtE4KHT34sbk6HgT8Mvp8ORl+rdhw5750PzicncM6ckTN1Zs61g9yRm7jCTb2Jt/Okpw+pbqeqeee0zMv/AEUCO+U=</latexit><latexit sha1_base64="ct3Eyv5JAlSrQdQBIHQLRnZdcDg=">AAADw3icbZJLb9NAEMcdm0cJBFI4clmRIiEOkd1LEadUaaQcUBWiPlLFIVqv18kq+7B216Bo5S/Jrd+GtWMb3DLSev7+zcw+RhOllCjt+/cd13vy9Nnzoxfdl696r9/0j9/eKJFJhK+RoEIuIqgwJRxfa6IpXqQSQxZRfBvtxkX89ieWigh+pfcpXjG44SQhCGqL1sed+14v5PiX3mIhMTOpFGluZvYrFClS8laYYsZgbr6VrhWpfG6uKtGKIiEFpVDuczNuZCsjxgnhhxPNxV/dyrELyl1u5gff63ZDWwbCwIQM6m2UgMAWlGhSoQhMajRr0KxG8wbNazQ1JkSQgmlek/OKnDdkXJFxQxYVWTTkriJ3DUFbg358rv8YM+NaX5rL8jFcEB5jrkGoEiA43QMKI0xBarlWgHBgOwHCEEi8sc0BIgExUXAjMWZF2Un5mkTCnbnIT9b9gT/0SwOPRVCJgVPZbN3/HcYCZcVWiEKlloGf6pWBUhNEcd4NM4VTiHZwg5dWcsiwWplyBnPw0ZIYJELaZa9S0n8rDGRK7VlkM4tLqoexAv4vtsx08mVlCE8zjTk6HJRkFGgBioG2HZAYadurmEAk7dQggLZQQqTt2HdtE4KHT34sbk6HgT8Mvp8ORl+rdhw5750PzicncM6ckTN1Zs61g9yRm7jCTb2Jt/Okpw+pbqeqeee0zMv/AEUCO+U=</latexit><latexit sha1_base64="ct3Eyv5JAlSrQdQBIHQLRnZdcDg=">AAADw3icbZJLb9NAEMcdm0cJBFI4clmRIiEOkd1LEadUaaQcUBWiPlLFIVqv18kq+7B216Bo5S/Jrd+GtWMb3DLSev7+zcw+RhOllCjt+/cd13vy9Nnzoxfdl696r9/0j9/eKJFJhK+RoEIuIqgwJRxfa6IpXqQSQxZRfBvtxkX89ieWigh+pfcpXjG44SQhCGqL1sed+14v5PiX3mIhMTOpFGluZvYrFClS8laYYsZgbr6VrhWpfG6uKtGKIiEFpVDuczNuZCsjxgnhhxPNxV/dyrELyl1u5gff63ZDWwbCwIQM6m2UgMAWlGhSoQhMajRr0KxG8wbNazQ1JkSQgmlek/OKnDdkXJFxQxYVWTTkriJ3DUFbg358rv8YM+NaX5rL8jFcEB5jrkGoEiA43QMKI0xBarlWgHBgOwHCEEi8sc0BIgExUXAjMWZF2Un5mkTCnbnIT9b9gT/0SwOPRVCJgVPZbN3/HcYCZcVWiEKlloGf6pWBUhNEcd4NM4VTiHZwg5dWcsiwWplyBnPw0ZIYJELaZa9S0n8rDGRK7VlkM4tLqoexAv4vtsx08mVlCE8zjTk6HJRkFGgBioG2HZAYadurmEAk7dQggLZQQqTt2HdtE4KHT34sbk6HgT8Mvp8ORl+rdhw5750PzicncM6ckTN1Zs61g9yRm7jCTb2Jt/Okpw+pbqeqeee0zMv/AEUCO+U=</latexit><latexit sha1_base64="ct3Eyv5JAlSrQdQBIHQLRnZdcDg=">AAADw3icbZJLb9NAEMcdm0cJBFI4clmRIiEOkd1LEadUaaQcUBWiPlLFIVqv18kq+7B216Bo5S/Jrd+GtWMb3DLSev7+zcw+RhOllCjt+/cd13vy9Nnzoxfdl696r9/0j9/eKJFJhK+RoEIuIqgwJRxfa6IpXqQSQxZRfBvtxkX89ieWigh+pfcpXjG44SQhCGqL1sed+14v5PiX3mIhMTOpFGluZvYrFClS8laYYsZgbr6VrhWpfG6uKtGKIiEFpVDuczNuZCsjxgnhhxPNxV/dyrELyl1u5gff63ZDWwbCwIQM6m2UgMAWlGhSoQhMajRr0KxG8wbNazQ1JkSQgmlek/OKnDdkXJFxQxYVWTTkriJ3DUFbg358rv8YM+NaX5rL8jFcEB5jrkGoEiA43QMKI0xBarlWgHBgOwHCEEi8sc0BIgExUXAjMWZF2Un5mkTCnbnIT9b9gT/0SwOPRVCJgVPZbN3/HcYCZcVWiEKlloGf6pWBUhNEcd4NM4VTiHZwg5dWcsiwWplyBnPw0ZIYJELaZa9S0n8rDGRK7VlkM4tLqoexAv4vtsx08mVlCE8zjTk6HJRkFGgBioG2HZAYadurmEAk7dQggLZQQqTt2HdtE4KHT34sbk6HgT8Mvp8ORl+rdhw5750PzicncM6ckTN1Zs61g9yRm7jCTb2Jt/Okpw+pbqeqeee0zMv/AEUCO+U=</latexit>
Active Binary Classification
parametric rate
exponential speed-up
# labels
Bayes error rate
passive
active
passive: ✏ ⇠ d
n
active: ✏ ⇠ exp⇣� c
n
d
⌘
<latexit sha1_base64="82Q4UYsIF/dj5C7yNeqNYQNT/N4=">AAAE8nicfZRRb9MwEMeztcAoFDp45MViAo1JTM14YEJC2ugm7WGaurFuneYyOc6ltebYwXFGK8sfgxceQIhXPg1vfBucNs3WbcJS6tPv7v6XOzsNEs5S3Wz+nZuvVO/cvbdwv/bgYf3R48bik6NUZopCh0ouVTcgKXAmoKOZ5tBNFJA44HAcnLdy//EFqJRJcahHCfRi0hcsYpRoh84WKwv1OhbwRQ9AKohNomRiTdv9ypTlIXbGzSGOiTW7423GU+zWHBbGjJdKJTknamRNqzRnIkKImJhUNFuX9kyMe4g6t+ZgstdrNezSEPYNjokeBBHyXcIYbRcoQNtT1C5R2xaJ+yXan0YdlOhginaMwZRwtGOnZLMgmyVpFaRVkm5BuiU5KciJvUW5eKNOQTol2S3IbkmCzOBA8jAdxW4zWemgA0M/rUzF49i0pvae2XPjymdJpTs6ETrNj/aqDN4CrklZYjhTYpiXGIySAYgEBOF69N5vuuUOQEgmQhAa4TRCNRxAnwkDnwVRioxWXJoTcGIahnp8W90hhtYkJE3ZBbyzFr1EGJKUcSlyM2UxwghHilDj4oRFGN/UCHgG1hCq/6cBwwR/YP3l13Qq6K5WaHP2aqwKbgyXb3rWWGquNscL3TT8wljyitU+a/zBoaRZ7Jqn3PVz6jcT3TNEaUY5uM6zFBJCz0kfTp0pSAxpz4wbsOiFIyGKpHKPG96YXs0wJE7z4bvI/Dqm1305vM13mulovWeYSDINgk4KRRlHWqL8+0chU0A1HzmDUOU+MorogLjpaPcvUXND8K+3fNM4Wlv136z6+2tLG+vFOBa8Z95zb9nzvbfehrfjtb2ORyuy8rXyvfKjqqvfqj+rvyah83NFzlNvZlV//wNX0q3o</latexit>
active
passive: ✏ ⇠ d
n
active: ✏ ⇠ exp⇣� c
n
d
⌘
<latexit sha1_base64="82Q4UYsIF/dj5C7yNeqNYQNT/N4=">AAAE8nicfZRRb9MwEMeztcAoFDp45MViAo1JTM14YEJC2ugm7WGaurFuneYyOc6ltebYwXFGK8sfgxceQIhXPg1vfBucNs3WbcJS6tPv7v6XOzsNEs5S3Wz+nZuvVO/cvbdwv/bgYf3R48bik6NUZopCh0ouVTcgKXAmoKOZ5tBNFJA44HAcnLdy//EFqJRJcahHCfRi0hcsYpRoh84WKwv1OhbwRQ9AKohNomRiTdv9ypTlIXbGzSGOiTW7423GU+zWHBbGjJdKJTknamRNqzRnIkKImJhUNFuX9kyMe4g6t+ZgstdrNezSEPYNjokeBBHyXcIYbRcoQNtT1C5R2xaJ+yXan0YdlOhginaMwZRwtGOnZLMgmyVpFaRVkm5BuiU5KciJvUW5eKNOQTol2S3IbkmCzOBA8jAdxW4zWemgA0M/rUzF49i0pvae2XPjymdJpTs6ETrNj/aqDN4CrklZYjhTYpiXGIySAYgEBOF69N5vuuUOQEgmQhAa4TRCNRxAnwkDnwVRioxWXJoTcGIahnp8W90hhtYkJE3ZBbyzFr1EGJKUcSlyM2UxwghHilDj4oRFGN/UCHgG1hCq/6cBwwR/YP3l13Qq6K5WaHP2aqwKbgyXb3rWWGquNscL3TT8wljyitU+a/zBoaRZ7Jqn3PVz6jcT3TNEaUY5uM6zFBJCz0kfTp0pSAxpz4wbsOiFIyGKpHKPG96YXs0wJE7z4bvI/Dqm1305vM13mulovWeYSDINgk4KRRlHWqL8+0chU0A1HzmDUOU+MorogLjpaPcvUXND8K+3fNM4Wlv136z6+2tLG+vFOBa8Z95zb9nzvbfehrfjtb2ORyuy8rXyvfKjqqvfqj+rvyah83NFzlNvZlV//wNX0q3o</latexit>
passive
✏ = R( bf)�R(f⇤)<latexit sha1_base64="pvQUBf6y7D81KHp+zc7wo3bkWXg=">AAAEfXicbZNBT9swFMdD222sWzfYjrtYQ0xQbVXDDnBBAhUkDgiVjkIR6ZDjvBALx84cZxBZ+RT7Zrvtq+yyOa0bKGDJ8T+/956f/Wz7CaOp6nb/LNTqjWfPXyy+bL563Xrzdmn53WkqMklgSAQTcuTjFBjlMFRUMRglEnDsMzjzr3ul/ewnyJQKfqLyBMYxvuI0pAQrgy6Xa79aLY/DjYpASIh1IkVS6L75ipSWLsWcmUEc40IfToY5ix0LfWLFnJUIKRjDMi90r5JzHgGElE8z6r07PedjOpbXhR5Mx1az6Zkw5Lnai7GK/BC5JmCC9i3y0f4M9SvUL2zgcYWOZ16DCg1m6EBrj2CGDooZ2bVktyI9S3oVGVkyqsi5JefFEzPbFQ0tGVbk0JLDiviZ9nzBgjSPzaCzykAiTb63Z5PHse7N9JE+MuUqa0mEOToemDm/Ffen8faAKVyluJ1LcVumiPIkAp4Ax0zl227XNHMAXFAeAFfIS0PU9Hy4olzDD46lxHnbhEGSUiY4+oS2TR+seTc0gAgrFK6jL+Y//N5e98Cs6C7ocmml2+lOGnosXCtWHNv6l0u/vUCQLDbrIAyn6YXbTdRYY6koYWAWkaWQYHKNr+DCSI5jSMd68noKtGpIgEIhTTf7mND7ERrHaVkH41nejPShrYRP2S4yFW6NNeVJpoCTaaIwY0gJVD5FFFAJRLHcCEykue8EkQhLTJR5sE1TBPfhlh+L042O+7XjHm+s7GzZciw6H5yPzprjOpvOjnPg9J2hQ2p/66i+Xm/X/zVWG58bnalrbcHGvHfmWmPzP7VTffk=</latexit>
✏ = R( bf)�R(f⇤)<latexit sha1_base64="pvQUBf6y7D81KHp+zc7wo3bkWXg=">AAAEfXicbZNBT9swFMdD222sWzfYjrtYQ0xQbVXDDnBBAhUkDgiVjkIR6ZDjvBALx84cZxBZ+RT7Zrvtq+yyOa0bKGDJ8T+/956f/Wz7CaOp6nb/LNTqjWfPXyy+bL563Xrzdmn53WkqMklgSAQTcuTjFBjlMFRUMRglEnDsMzjzr3ul/ewnyJQKfqLyBMYxvuI0pAQrgy6Xa79aLY/DjYpASIh1IkVS6L75ipSWLsWcmUEc40IfToY5ix0LfWLFnJUIKRjDMi90r5JzHgGElE8z6r07PedjOpbXhR5Mx1az6Zkw5Lnai7GK/BC5JmCC9i3y0f4M9SvUL2zgcYWOZ16DCg1m6EBrj2CGDooZ2bVktyI9S3oVGVkyqsi5JefFEzPbFQ0tGVbk0JLDiviZ9nzBgjSPzaCzykAiTb63Z5PHse7N9JE+MuUqa0mEOToemDm/Ffen8faAKVyluJ1LcVumiPIkAp4Ax0zl227XNHMAXFAeAFfIS0PU9Hy4olzDD46lxHnbhEGSUiY4+oS2TR+seTc0gAgrFK6jL+Y//N5e98Cs6C7ocmml2+lOGnosXCtWHNv6l0u/vUCQLDbrIAyn6YXbTdRYY6koYWAWkaWQYHKNr+DCSI5jSMd68noKtGpIgEIhTTf7mND7ERrHaVkH41nejPShrYRP2S4yFW6NNeVJpoCTaaIwY0gJVD5FFFAJRLHcCEykue8EkQhLTJR5sE1TBPfhlh+L042O+7XjHm+s7GzZciw6H5yPzprjOpvOjnPg9J2hQ2p/66i+Xm/X/zVWG58bnalrbcHGvHfmWmPzP7VTffk=</latexit>
✏ = R( bf)�R(f⇤)<latexit sha1_base64="pvQUBf6y7D81KHp+zc7wo3bkWXg=">AAAEfXicbZNBT9swFMdD222sWzfYjrtYQ0xQbVXDDnBBAhUkDgiVjkIR6ZDjvBALx84cZxBZ+RT7Zrvtq+yyOa0bKGDJ8T+/956f/Wz7CaOp6nb/LNTqjWfPXyy+bL563Xrzdmn53WkqMklgSAQTcuTjFBjlMFRUMRglEnDsMzjzr3ul/ewnyJQKfqLyBMYxvuI0pAQrgy6Xa79aLY/DjYpASIh1IkVS6L75ipSWLsWcmUEc40IfToY5ix0LfWLFnJUIKRjDMi90r5JzHgGElE8z6r07PedjOpbXhR5Mx1az6Zkw5Lnai7GK/BC5JmCC9i3y0f4M9SvUL2zgcYWOZ16DCg1m6EBrj2CGDooZ2bVktyI9S3oVGVkyqsi5JefFEzPbFQ0tGVbk0JLDiviZ9nzBgjSPzaCzykAiTb63Z5PHse7N9JE+MuUqa0mEOToemDm/Ffen8faAKVyluJ1LcVumiPIkAp4Ax0zl227XNHMAXFAeAFfIS0PU9Hy4olzDD46lxHnbhEGSUiY4+oS2TR+seTc0gAgrFK6jL+Y//N5e98Cs6C7ocmml2+lOGnosXCtWHNv6l0u/vUCQLDbrIAyn6YXbTdRYY6koYWAWkaWQYHKNr+DCSI5jSMd68noKtGpIgEIhTTf7mND7ERrHaVkH41nejPShrYRP2S4yFW6NNeVJpoCTaaIwY0gJVD5FFFAJRLHcCEykue8EkQhLTJR5sE1TBPfhlh+L042O+7XjHm+s7GzZciw6H5yPzprjOpvOjnPg9J2hQ2p/66i+Xm/X/zVWG58bnalrbcHGvHfmWmPzP7VTffk=</latexit>
Assuming optimal Bayes classifer f⇤ in VC class with dimension dand “nice” distributions (e.g., bounded label noise)
<latexit sha1_base64="lssTZsHpGLEgnSo2BGjEW8D/EzQ=">AAAEtHicbZNRT9swEMcD7TbWrRtsj3uxoJMYmqqGSRsvk2AFiQeECqNQRAo4zqW1cOzIdgaRlS+4x73t28xp3YwClhz/9bs73/lihymjSnc6fxcWa/Vnz18svWy8et1883Z55d2pEpkk0CeCCTkIsQJGOfQ11QwGqQSchAzOwptuaT/7BVJRwU90nsIwwSNOY0qwtuhqZfF3sxlwuNVjEBISk0qRFqZnv0LR0qWYMzNIElyYg8kyZ3FrYU6cmLMSIQVjWOaF6VZyziOCmPJpRrP7X8/52InlTWGOp2uz0QhsGAp8EyRYj8MY+TZggvYcCtHeDPUq1Ctc4FGFjmZexxU6nqF9YwKCGdovZmTHkZ2KdB3pVmTgyKAi546cF0/s7CrqO9KvyIEjBxUJMxOEgkUqT+xisspAxoZcbsw2TxLTnelDc2jbVfaSCPvreGT3/Fnc3ybYBaZxleJuLsVdmWKcp2PgKXDMdP7d79hhfwAXlEfANQpUjHaUyhLKR0ikmia26B84B4UIw0rRGCRqxZcbLUQ5Ou1OKbqleowimgAv7yhqRS0UBMhWiK6vOSWwao1KSxpm5X1QaB3ao/ZnFIrMpo0QwyEwZItQ8Olqea3T7kwGeix8J9Y8N3pXy3+CSJDM5taTYi78TqqHBktNCQN75ExBiskNHsGFlRwnoIZm8ugK9NGSCMVC2mmPP6H3IwxOVNk+61leKPXQVsKnbBeZjreGhvI008DJNFGcMaQFKl+w7YcEolluBSbSPhOCyBhLTLR95w3bBP/hkR+L0822/6XtH22ubW+5dix5H7xVb93zvW/etrfv9by+R2p+7ax2XcP1r/WgTuowdV1ccDHvvblR5/8ApXqSAQ==</latexit>
Tutorial Outline
Part 1: Introduction to Active Learning (Rob)
Part 2: Theory of Active Learning (Steve)
Part 3: Advanced Topics and Open Problems (Steve)
Part 4: Nonparametric Active Learning (Rob)
slides: http://nowak.ece.wisc.edu/ActiveML.html
Recommended Reading (Foundations of Active Learning)
Dasgupta, Sanjoy, Daniel J. Hsu, and Claire Monteleoni. "A general agnostic active learning algorithm." Advances in neural information processing systems. 2008.
Balcan, Maria-Florina, Alina Beygelzimer, and John Langford. "Agnostic active learning." Journal of Computer and System Sciences 75.1 (2009): 78-89.
Dasgupta, Sanjoy. "Two faces of active learning." Theoretical computer science 412.19 (2011): 1767-1781.
Zhu, Xiaojin, John Lafferty, and Zoubin Ghahramani. "Combining active learning and semi-supervised learning using gaussian fields and harmonic functions." ICML 2003 workshop. Vol. 3. 2003.
Cohn, David, Les Atlas, and Richard Ladner. "Improving generalization with active learning." Machine learning 15.2 (1994): 201-221.
Settles, Burr. "Active learning." Synthesis Lectures on Artificial Intelligence and Machine Learning 6.1 (2012): 1-114.
Nowak, Robert D. "The geometry of generalized binary search." IEEE Transactions on Information Theory 57, no. 12 (2011): 7893-7906.
Hanneke, Steve. "Theory of active learning." Foundations and Trends in Machine Learning 7, no. 2-3 (2014).
Castro, Rui M., and Robert D. Nowak. "Minimax bounds for active learning." IEEE Transactions on Information Theory 54, no. 5 (2008): 2339-2353.
top related