active learning from theory to practice · interactive machine-human data collection and...

Active Learning from Theory to Practice

Robert Nowak [email protected]

Steve Hanneke Toyota Technological Institute at [email protected]

Tutorial Outline

Part 1: Introduction to Active Learning (Rob)

Part 2: Theory of Active Learning (Steve)

Part 3: Advanced Topics and Open Problems (Steve)

Part 4: Nonparametric Active Learning (Rob)

slides: http://nowak.ece.wisc.edu/ActiveML.html

Conventional (Passive) Machine Learning

unlabeled raw data

human labeling

machine learning

predictive model

labeled data

dog

boat

…

millions of labeled images1000’s of human hours trained on more texts than a

human could read in a lifetime

learned by playing more games than even my teenage soncould stomach !

?

Can we train machines with less labeled data and less human supervision?

Active Machine Learning

data selection algorithm

unlabeled raw data

human labeling

machine learning

predictive model

labeled data

Goal: machine automatically and adaptively selects most informative data for labeling

Motivating Application

provides labels to machine learner (several minutes / EHR)

EHR feature 1

EH

R fe

atur

e 2

best linear classifier

Non-adaptive strategy: Label a random sample

Active strategy: Label a sample near best decision boundary based on labels seen so far

# labels

error rate ✏

active learning finds optimal classifier with much less human supervision!

Active Learning

11000 patient records 8000 positive 3000 negative

6182 Numerical Features icd9 codes lab tests patient data

Classification task: cataracts or healthy

less than half as many labeled examples needed by active learning

active learning passive learning

Active Logistic Regression

nextml.org

Active learning to optimize crowdsourcing and rating in New Yorker Cartoon Caption Contest

Actively learning user’s beer preferences

Principles of Active Learning

What and Where Information

x<latexit sha1_base64="Whvr4b/LFT0obOEEGu5nSiC2KTo=">AAAEO3icbZNNb9MwGMezlJdRKHRw5GLRIiEOVTMO7IK0qZvUQzW1Zd06NWVyHGeN6tiR7UAjK9+LC1+CGxcuHECIK3ec1gnLNkvO89fveXP84sUkFLLb/bZl1+7cvXd/+0H94aPG4yfNnaengiUc4QlihPGpBwUmIcUTGUqCpzHHMPIIPvOWvdx/9hFzETJ6ItMYzyN4ScMgRFBqdLFjjxoNl+JPcoEZx5GKOYszNdRfJsI8JKu4CY4imKnB2lQ8xmbqxIiKFzHOCIE8zVSvlJUIHwch3XRUh/91JUZPyJeZGm9so153dRpwHeVGUC68ADg6YY2ODPLAUYGGJRpmJnFUolERNS7RuEB9pVwECehnBTkw5KAkPUN6JZkaMi3JuSHn2S2VzYomhkxKMjBkUBIvUa7HiC/SSBuVlA60UOjD66J4FKleoY/Vsd6ufC8R00dHfV3zfXa1jHuIiYRli1WlxSpvsUjjBaYxppDI9J3T1UMfAGUh9TGVwBUBaLdX7fZFs9XtdNcD3BSOES3LjOFF86vrM5REugoiUIiZ043lXEEuQ0Sw7pwIHEO0hJd4piWFERZztb77GXipiQ8CxvXUq1jTqxkKRiL/Cx2Zn6u47svhbb5ZIoO9uQppnEhM0aZRkBAgGcgfEvBDjpEkqRYQcX1bEUALyCGS+rnV9SY413/5pjjd7ThvOs5ot7W/Z7Zj23puvbBeWY711tq3+tbQmljI/mx/t3/av2pfaj9qv2t/NqH2lsl5ZlVG7e8/HPJrjA==</latexit>

p(y|x)<latexit sha1_base64="9s4p/1EEotGoHDdhpiNeEwbD0aU=">AAAEQHicbZNNb9MwGMezlJdRKOvgyMWiRRocpmYc2AVpUzeph2pqy7p1WsrkOM4S1bEj24FGJh+NCx+BG2cuHECIKyec1g3LNkvO89fvefNL7CUkErLT+bZm1+7cvXd//UH94aPG443m5pMTwVKO8BgxwvjEgwKTiOKxjCTBk4RjGHsEn3qzbuE//YC5iBg9llmCpzG8pFEQISg1uti0TxoNl+KPMsSM41glnCW5GugvE1ERklfcBMcxzFV/YSoeY3N1bETFixhnhECe5apbykqEj4OILjuqg/+6EqMn5LNcjZa2Ua+7Og24jnJjKEMvAI5OWKBDgzxwuEKDEg1ykzgs0XAVNSrRaIV6SrkIEtDLV2TfkP2SdA3plmRiyKQkZ4ac5bdUNisaGzIuSd+Qfkm8VLkeI77IYm1UWjpQqND7V6vicay6K32kjvRxFWeJmL466uua7/KrZdwDTCQsW8wrLeZFizBLQkwTTCGR2Vuno4e+AMoi6mMqgSsC0G4nW9mn+ct2+6LZ6mx3FgPcFI4RLcuMwUXzq+szlMa6FCJQiHOnk8ipglxGiGDdPhU4gWgGL/G5lhTGWEzV4gHk4IUmPggY11MvZUGvZigYi2IrOrK4XHHdV8DbfOepDHanKqJJKjFFy0ZBSoBkoHhNwI84RpJkWkDE9S+LAAohh0jqN1fXh+Bc3/JNcbKz7bzedoY7rb1dcxzr1jPrubVlOdYba8/qWQNrbCH7s/3d/mn/qn2p/aj9rv1ZhtprJuepVRm1v/8AyrttdA==</latexit>

Density estimation: What is p(y|x)?Classification: Where is p(y|x) > 0?

<latexit sha1_base64="6/jpTwqMPj9rFfECacA215Xkksk=">AAAEn3icbZNvT9QwGMAHdyqenoK+1BdVMEFfkA1fSEwU8CCSeCEHcnKEnaTrOtbQtUvb6S11H8sv4ju/jd2uN29Ak+558nv+dU+fBiklUrnu34XFVvvO3XtL9zsPHnYfPV5eefJN8kwgPESccjEKoMSUMDxURFE8SgWGSUDxaXDVK+2nP7CQhLMTlad4nMBLRiKCoDLoYmXxd7frM/xTxZgLnOhU8LTQA/PlkpQuRcNMcZLAQvcr0bBYWegTqzSsiAtOKRR5oXu12vAIcUTYtKLe+683fMyG4qrQx1PZ7XR8EwZ8T/sJVHEQAc8EVGjfogDsz9CgRoPCBh7V6GjmdVyj4xk60NpHkIKDYkZ2LdmtSc+SXk1GloxqcmbJWXFLZnuioSXDmvQt6dckyLQfcBrKPDFCZ7UBxRp9fzNLniS6N9MP9aFpV9lLxM3VsdDk/FrMp/H3MFWwLjFplJiUJeI8jTFLMYNU5R881yxzAYwTFmKmgC8jsIeZmZocYKlIUk3Ye3AaQwWIBGvpev5r8nptGwDfn49TeKKqQTb3G5r5oFDKekLLeCzwXIKP7tp2cbG86m641QI3Fc8qq45dg4vlP37IUZaYiqgscO65qRprKBRBFJu/yyROIbqCl/jcqAwmWI51dawCvDIkBBEXZpsTV3Q+QsNElp0ynuXsyOu2Et5mO89UtDXWhKWZwgxNC0UZBYqD8rGCkAiMFM2NApEwLwIBFEMBkTJPumOa4F3/5ZvKt80N7+2Gd7S5urNl27HkPHNeOuuO57xzdpwDZ+AMHdR63vrU+tLqt1+0P7cP24Op6+KCjXnqNFb77B/mhYr5</latexit>


p(x)<latexit sha1_base64="/13CxrgjkO8jDL8OQDbBnD1TWw4=">AAAEPnicbZNNb9MwGMezlJdRKHRw5GLRIg0OUzMO7IK0qZvUQzW1pd06NaVyHGeN6tiR7UArK5+MC5+BG0cuHECIK0ec1gnLNkvO89fveXP84sUkFLLV+rZlV+7cvXd/+0H14aPa4yf1nadngiUc4RFihPGxBwUmIcUjGUqCxzHHMPIIPvcW7cx//hFzETI6lKsYTyN4ScMgRFBqNNuxh7WaS/EnOceM40jFnMWp6ukvE2EWkpbcBEcRTFV3bUoeY1M1NKLkRYwzQiBfpapdyFKEj4OQbjqq4/+6FKMn5ItUDTa2Vq26Og24jnIjKOdeABydsEYnBnngJEe9AvVSk9gvUD+PGhRokKOOUi6CBHTSnBwZclSQtiHtgowNGRfkwpCL9JbKZkUjQ0YF6RrSLYiXKNdjxBerSBuVFA40V+jD67x4FKl2rk/Vqd6ubC8R00dHfV3zfXq1jHuMiYRFi2WpxTJrMV/Fc0xjTCGRq3dOSw99AJSF1MdUAlcEoNmMd5evms1ZvdHaa60HuCkcIxqWGb1Z/avrM5REuhAiUIiJ04rlVEEuQ0Swbp4IHEO0gJd4oiWFERZTtb7+KXipiQ8CxvXUC1nTqxkKRiL7ER2ZHa247svgbb5JIoODqQppnEhM0aZRkBAgGcjeEvBDjpEkKy0g4vrCIoDmkEMk9Yur6k1wrv/yTXG2v+e82XP6+43DA7Md29Zz64W1aznWW+vQ6lg9a2Qh+7P93f5p/6p8qfyo/K782YTaWybnmVUalb//AEy9bGs=</latexit>Density estimation: What is p(x)?

Clustering: Where is p(x) > ✏ ?<latexit sha1_base64="tkVbhFrcHREGGL/VYApkj5cW6m0=">AAAEoHicbZPxTxMxFMcPNhWnU9AfjUkDmKAxZIc/SEwUyCDZDxM3ZDDkBun13riGXntpe7qluX/LP8Tf/G/sbd3JgCa9fvN57/X1Xl/DlFGlG42/C4uV6oOHj5Ye1548rT97vrzy4kSJTBLoEcGE7IdYAaMceppqBv1UAk5CBqfhdbOwn/4Eqajgx3qcwiDBV5wOKcHaosuVxd/1esDhl45BSEhMKkWam479CkULl3zOzCBJcG7ak2XO4tbcHDsxZyVCCsawHOemWco5jwiGlE8zmv3/es7HTiyvc3M0Xeu1WmDDUOCbIME6DofItwETdOBQiA5mqFOiTu4CuyXqzryOSnQ0Qy1jAoIZauUzsufIXkmajjRL0nekX5IzR87ye3Z2J+o50itJ25F2ScLMBKFgkRondjFZaSCxIRfvZpsniWnO9KE5tOUqakmEvToe2T2/5ze3CfaBaVymGM2lGBUp4nEaA0+BY6bHn/2GHfYCuKA8Aq5RoIZoH7jtmjECpWky6bBP6DTGGlGF1tON0dv1HYSC4GaUhpGetLG93ch2B8uUBkn5VREJEsrQLwGkijLB14P3O/nl8lpjszEZ6K7wnVjz3OhcLv8JIkGyxCYlDCt17jdSPTBYakoY2N/LFKSYXOMrOLeS4wTUwExOlqM3lkRoKKSd9tATejPC4EQVpbKeRfOo27YC3mc7z/Rwe2AoTzMNnEwTDTOGtEDFa0URlUA0G1uBibRPgiASY4mJrZGq2SL4t3/5rjjZ2vQ/bPrdrbXdbVeOJe+Vt+pteL730dv1Wl7H63mk8rrSrLQrX6ur1Vb1W7U7dV1ccDEvvblR/fEPTTqLOw==</latexit>

E[y|x]<latexit sha1_base64="iJnzcTJQm4dWqYlWoJGCWInHwls=">AAAEQXicbZNNb9MwGMezhJdRKHRw5GKxIiEOU7Md2AVpU1eph2nqxrp1asrkOM4S1bEj24FFJl+NC9+AG3cuHECIKxec1gnLNkvO89fvefNL7KckFrLX+7ZiO3fu3ru/+qD18FH78ZPO2tMTwTKO8BgxwvjEhwKTmOKxjCXBk5RjmPgEn/rzfuk//YC5iBk9lnmKZwm8oHEYIyg1Ol+zT9ttj+KPMsKM40SlnKWFGukvE3EZUjTcBCcJLNT+wjQ8xhbq2IiGFzHOCIE8L1S/lo2IAIcxXXZUe/91I0ZPyOeFOlradqvl6TTgucpLoIz8ELg6YYEGBvlgUKFRjUaFSTys0WEVdVSjowoNlfIQJGBYVGTXkN2a9A3p12RiyKQmZ4acFbdUNisaGzKuyb4h+zXxM+X5jAQiT7RRWe1AkULvX1fFk0T1K32gDvRxlWeJmL46Guia74qrZbw9TCSsW1w2WlyWLaI8jTBNMYVE5m/dnh76AiiLaYCpBJ4IQbfrDab5p8tZt3veWe9t9BYD3BSuEeuWGaPzzlcvYChLdC1EoBBTt5fKmYJcxohg3T8TOIVoDi/wVEsKEyxmavECCvBSkwCEjOup17KgVzMUTES5Fx1Z3q647ivhbb5pJsPtmYppmklM0bJRmBEgGSifEwhijpEkuRYQcf3PIoAiyCGS+tG19CG417d8U5xsbrhbG+7h5vrOtjmOVeu59cJ6ZbnWG2vHGloja2wh+7P93f5p/3K+OD+c386fZai9YnKeWY3h/P0HtwluFg==</latexit>


Function estimation: What is E[y|x]?Bandit optimization: Where is maxx E[y|x]?

<latexit sha1_base64="uzqlJL2aZvP3o4CtSisy/1l5kxc=">AAAErXicbZNvT9QwHMcHdyqenoA+9EkjmBge4A0fSEyM4IHhASEHcnB4Oy5d13EN/bO0nd6se3W+A5/5bux2vcmAJl2/+fz+db+2YUKJ0p3O34XFRvPBw0dLj1tPnrafLa+sPj9TIpUI95GgQg5CqDAlHPc10RQPEokhCyk+D6+7hf38O5aKCH6qswSPGLziJCYIaovGq4u/2+2A4x96goXEzCRSJLnp2a9QpHDJa2aKGYO5OSyXmsWtuTl1omZFQgpKocxy061kzSPCMeGzimbvv6752AnldW5OZmu71QpsGAh8EzCoJ2EMfBtQon2HQrA/R70K9XIXeFyh47nXSYVO5ujAmABBCg7yOdl1ZLciXUe6FRk4MqjIhSMX+T2Z3Y76jvQrcujIYUXC1AShoJHKmF1MWhnQxKDLjXlyxkx3ro/MkW1X0Usk7NHxyOb8mt9ME+xhqmFVYlorMS1KTLJkgnmCOaQ6++h37LAHwAXhEeYaBCoGX1KOimMDWGnCyiv2AZxPoAZEgfVgf5j9mo7WPwEQBDcjNZ7q8irbE45y89nuj2ggEpuD/KyyYIlnaRicjqegypa3xitrnc1OOcBd4Tux5rnRG6/8CSKBUmarIwqVGvqdRI8MlJogiu2/pgonEF3DKzy0kkOG1ciUW8zBa0siEAtpp919SW9GGMhU0TfrWdwkddtWwPtsw1TH2yNDeJJqzNGsUJxSoAUoni6IiMRI08wKiKR9HwigCZQQafvAiyb4t3/5rjjb2vTfbfrHW2s7264dS95L75X3xvO9996Od+D1vL6HGhuNXuOi8a35ttlvBs3Lmevigot54dVG8+ofEeuRPQ==</latexit>

Active learning is more efficient than passive learning for localized “where” information

Version-Space (VS) Active Learning

initialize VS: H = all models/hypotheses

while (stopping-criterion) not met

1. sample at random from available dataset

2. label only those samples that distinguish models in H

3. reduce H by removing all models inconsistent with labels

output: best model in final H

Model Space

Select examples to label

Labeled Data

1. Integrating data from disparate sources

2. Optimizing human resources in the data collection and processing chain

3. Reasoning and planning in highly uncertain environments

This proposal outlines a research agenda for developing the mathematical and computational foundationsof large-scale interactive data analytics. Our aim is to focus on two interconnected components necessary tooptimize information processing systems:

• A1. Sequential and adaptive sensing, data-collection and decision-making.

• A2. Interactive Machine-Human Data Collection and Co-Processing.

• A3. Cloud-based platforms for large-scale interactive data analytics.

These challenges will be studied and will influence the development of new mathematical theory andcomputational tools.

The State of Interactive Machine Learning Most supervised machine learning algorithms designed us-ing a pre-selected set of “training data.” This approach is referred to as passive learning. Gathering andannotating training datasets can be costly, placing limits on their size and complexity. For example, in natu-ral language processing or scene recognition tasks, human experts are usually required to label the trainingdata. Active learning integrates the processes of training data collection and algorithm design. Rather thanworking with a pre-selected pool of training data, the selection of data for labeling or annotation is guided bythe machine as it learns the task at hand, focusing data gathering toward examples that are most ambiguousgiven previously collected training data.

model space

questions/queries

“training data”

Human Judgementsanalysis, labeling,

annotation, comparison

active learning

Figure 2: Active learning closes the loop. The machine learner only requests human assistance/labeling for data thatdoes not fit its current learned model. As the machines prediction performance improves, it requires less and lessassistance from human experts.

Active learning addresses an inherently practical problem: train a machine to perform a particular taskwhile using as little input from human experts as possible. Unfortunately, the most impressive accomplish-ments in this field are theoretical in nature and almost useless in practice. Most existing active learningmethods suffer from at least one of the following problems that have limited success in practice:

2

machine

1. Integrating data from disparate sources

2. Optimizing human resources in the data collection and processing chain

3. Reasoning and planning in highly uncertain environments

This proposal outlines a research agenda for developing the mathematical and computational foundationsof large-scale interactive data analytics. Our aim is to focus on two interconnected components necessary tooptimize information processing systems:

• A1. Sequential and adaptive sensing, data-collection and decision-making.

• A2. Interactive Machine-Human Data Collection and Co-Processing.

• A3. Cloud-based platforms for large-scale interactive data analytics.

These challenges will be studied and will influence the development of new mathematical theory andcomputational tools.

The State of Interactive Machine Learning Most supervised machine learning algorithms designed us-ing a pre-selected set of “training data.” This approach is referred to as passive learning. Gathering andannotating training datasets can be costly, placing limits on their size and complexity. For example, in natu-ral language processing or scene recognition tasks, human experts are usually required to label the trainingdata. Active learning integrates the processes of training data collection and algorithm design. Rather thanworking with a pre-selected pool of training data, the selection of data for labeling or annotation is guided bythe machine as it learns the task at hand, focusing data gathering toward examples that are most ambiguousgiven previously collected training data.

model space

questions/queries

“training data”

Human Judgementsanalysis, labeling,

annotation, comparison

active learning

Figure 2: Active learning closes the loop. The machine learner only requests human assistance/labeling for data thatdoes not fit its current learned model. As the machines prediction performance improves, it requires less and lessassistance from human experts.

Active learning addresses an inherently practical problem: train a machine to perform a particular taskwhile using as little input from human experts as possible. Unfortunately, the most impressive accomplish-ments in this field are theoretical in nature and almost useless in practice. Most existing active learningmethods suffer from at least one of the following problems that have limited success in practice:

2

machine

active learning

questions& tests

answers & decisions

models data data

human judgments

models

questions& tests

answers & decisions

human judgments

Figure 2: Active learning closes the loop. The conventional, passive learning approach is shown on the left. A humanexpert supplies a set of labeled training data to the machine without any interaction, and the machine selects the bestmodel based on these data. In the active learning model, on the right, the learning process is continually refined as themachine’s prediction performance improves. The machine uses information gleaned from previous human inputs torule out incompatible models, and then queries the human for information about only the most ambiguous data. Thisreduces the burden on human experts and accelerates the learning process.

and many of its relaxations computationally intractable (the first problem identified above). In the contextof Figure 3, each node may have a very large number of possible sensing actions (children) associated withit, corresponding to the multitude not only of available sensors, but also their various operating modes.Similarly, the space of possible target identities and locations is generally enormous. Furthermore, MDPformulations assume critical information about the cost or reward function is known a priori, touching onthe second issue above, while in active learning such prior knowledge is unavailable.

. . .

. . .

. . .

...

�

sensing space:

target identificationand localization

Figure 3: Optimal solutions to active learning and adaptive sensing can be represented in terms of a search tree. Inlarge-scale, complex tasks, the size of the search tree grows exponentially in terms of data dimensionality. New theoryand methods are needed to handle this challenge.

Development of a general framework that encompasses all active learning models and avoids the pitfallsjust discussed is not realistic given current understanding. Instead, we will address specific problems, build-ing on our long track record of research in the field of active learning [30,8,43,38,18,27,13,21,35,19]. Wewill identify several machine learning techniques that have demonstrated good performance in real-worldapplications, and develop new algorithms that are provably optimal or near-optimal over a large range ofmodel assumptions and noise conditions. By considering specific hypothesis classes and slightly relaxedoptimality conditions we believe that we can devise novel algorithms that avoid the problems identifiedabove. Not surprisingly, the kinds of models, algorithms and tasks that have gained popularity due to their

3

1 2

3

Meta-Algorithm for Active Learning

random selection (passive)

binary search (active)

Learning a 1-D Classifier

labeled data

binary search quickly finds decision boundary

passive : ✏ ⇠ n�1

active : ✏ ⇠ e�cn

errerr 2�n

<latexit sha1_base64="LF0MCNfhuOTCFY7Wls1cSmk+ymE=">AAAB8XicbVA9T8MwED3zWcpXgZHFokFioUrKAGMFC2OR6IdoQ+W4TmvVcSLbQaqi/gsWBhBi5d+w8W9w2wzQ8qSTnt670929IBFcG9f9Riura+sbm4Wt4vbO7t5+6eCwqeNUUdagsYhVOyCaCS5Zw3AjWDtRjESBYK1gdDP1W09MaR7LezNOmB+RgeQhp8RY6cFxqo/ZuZw4Tq9UdivuDHiZeDkpQ456r/TV7cc0jZg0VBCtO56bGD8jynAq2KTYTTVLCB2RAetYKknEtJ/NLp7gU6v0cRgrW9Lgmfp7IiOR1uMosJ0RMUO96E3F/7xOasIrP+MySQ2TdL4oTAU2MZ6+j/tcMWrE2BJCFbe3YjokilBjQyraELzFl5dJs1rxLireXbVcu87jKMAxnMAZeHAJNbiFOjSAgoRneIU3pNELekcf89YVlM8cwR+gzx+gEY+S</latexit>

Consider a possibly infinite set of hypotheses F with finite VC dimension dand for each f 2 F define the risk (error rate):

R(f) := P(f(x) 6= y)<latexit sha1_base64="IHQlOfu2sYOkwvQOa+oB6zAaTts=">AAAEPHicbZPPb9MwFMezlh+jULbBkcsTC1LLYWp2YZqEtCkM7YCmUu1Hp6VMjvOyWk3sYLuMysofxoU/ghsnLhxAiCtn3DTN6IYl29983nt2/J4dZglTutP5ulSr37p95+7yvcb9B82HK6trj46VGEuKR1QkQvZDojBhHI800wn2M4kkDRM8CUf+1H7yAaVigh/qSYaDlFxwFjNKtEXna7VesxlwvNRDFBJTk0mR5aZrR6HY1CVfMCeYpiQ3b4ppwVLOuTksxYKVCimShMhJbvxKLnhEGDM+29G8utILPrYTOcpNbzY3G43AhkHgmSAlehjG4NmAAu2VKIS9OepWqDtHvQr15mjfmICSBPbzOdktyW5F/JL4FemXpF+R05KcVoQODX33fP6Vpsaf6wNzUBxGWc0F4xFyDb7gikUogYCthWJhMgHGi8QgKNQgYhhOMmGTo1CBO9vude7CJdNDMAGmUHof+xCxFPn0FuTgRi4EARAeQSwkIKFDcOOAcbhaoqgGgl0bJFMjaKGU1lcSje3thuv2WnEbAth+aYeg24pbH9u2UO9h0nbdxvnqemejUzS4KbxSrDtl656vfgkiQcf2DzVNiFJnXifTA0OkZjTBvBGMFWaEjsgFnlnJSYpqYIrLn8MzS2YniYVNW0H/jTAkVWqShtZzWmt13TaF/7OdjXW8NTCMZ2ONnM42iscJaAHTl2RTKpFqW5WIESrtdaVAh0QSqu17mybBu37km+J4c8PrbHhvN9d3tsp0LDtPnKdOy/GcF86Os+90nSOH1j7VvtV+1H7WP9e/13/Vf89ca0tlzGNnodX//AX/ZWVT</latexit><latexit sha1_base64="IHQlOfu2sYOkwvQOa+oB6zAaTts=">AAAEPHicbZPPb9MwFMezlh+jULbBkcsTC1LLYWp2YZqEtCkM7YCmUu1Hp6VMjvOyWk3sYLuMysofxoU/ghsnLhxAiCtn3DTN6IYl29983nt2/J4dZglTutP5ulSr37p95+7yvcb9B82HK6trj46VGEuKR1QkQvZDojBhHI800wn2M4kkDRM8CUf+1H7yAaVigh/qSYaDlFxwFjNKtEXna7VesxlwvNRDFBJTk0mR5aZrR6HY1CVfMCeYpiQ3b4ppwVLOuTksxYKVCimShMhJbvxKLnhEGDM+29G8utILPrYTOcpNbzY3G43AhkHgmSAlehjG4NmAAu2VKIS9OepWqDtHvQr15mjfmICSBPbzOdktyW5F/JL4FemXpF+R05KcVoQODX33fP6Vpsaf6wNzUBxGWc0F4xFyDb7gikUogYCthWJhMgHGi8QgKNQgYhhOMmGTo1CBO9vude7CJdNDMAGmUHof+xCxFPn0FuTgRi4EARAeQSwkIKFDcOOAcbhaoqgGgl0bJFMjaKGU1lcSje3thuv2WnEbAth+aYeg24pbH9u2UO9h0nbdxvnqemejUzS4KbxSrDtl656vfgkiQcf2DzVNiFJnXifTA0OkZjTBvBGMFWaEjsgFnlnJSYpqYIrLn8MzS2YniYVNW0H/jTAkVWqShtZzWmt13TaF/7OdjXW8NTCMZ2ONnM42iscJaAHTl2RTKpFqW5WIESrtdaVAh0QSqu17mybBu37km+J4c8PrbHhvN9d3tsp0LDtPnKdOy/GcF86Os+90nSOH1j7VvtV+1H7WP9e/13/Vf89ca0tlzGNnodX//AX/ZWVT</latexit><latexit sha1_base64="IHQlOfu2sYOkwvQOa+oB6zAaTts=">AAAEPHicbZPPb9MwFMezlh+jULbBkcsTC1LLYWp2YZqEtCkM7YCmUu1Hp6VMjvOyWk3sYLuMysofxoU/ghsnLhxAiCtn3DTN6IYl29983nt2/J4dZglTutP5ulSr37p95+7yvcb9B82HK6trj46VGEuKR1QkQvZDojBhHI800wn2M4kkDRM8CUf+1H7yAaVigh/qSYaDlFxwFjNKtEXna7VesxlwvNRDFBJTk0mR5aZrR6HY1CVfMCeYpiQ3b4ppwVLOuTksxYKVCimShMhJbvxKLnhEGDM+29G8utILPrYTOcpNbzY3G43AhkHgmSAlehjG4NmAAu2VKIS9OepWqDtHvQr15mjfmICSBPbzOdktyW5F/JL4FemXpF+R05KcVoQODX33fP6Vpsaf6wNzUBxGWc0F4xFyDb7gikUogYCthWJhMgHGi8QgKNQgYhhOMmGTo1CBO9vude7CJdNDMAGmUHof+xCxFPn0FuTgRi4EARAeQSwkIKFDcOOAcbhaoqgGgl0bJFMjaKGU1lcSje3thuv2WnEbAth+aYeg24pbH9u2UO9h0nbdxvnqemejUzS4KbxSrDtl656vfgkiQcf2DzVNiFJnXifTA0OkZjTBvBGMFWaEjsgFnlnJSYpqYIrLn8MzS2YniYVNW0H/jTAkVWqShtZzWmt13TaF/7OdjXW8NTCMZ2ONnM42iscJaAHTl2RTKpFqW5WIESrtdaVAh0QSqu17mybBu37km+J4c8PrbHhvN9d3tsp0LDtPnKdOy/GcF86Os+90nSOH1j7VvtV+1H7WP9e/13/Vf89ca0tlzGNnodX//AX/ZWVT</latexit><latexit sha1_base64="IHQlOfu2sYOkwvQOa+oB6zAaTts=">AAAEPHicbZPPb9MwFMezlh+jULbBkcsTC1LLYWp2YZqEtCkM7YCmUu1Hp6VMjvOyWk3sYLuMysofxoU/ghsnLhxAiCtn3DTN6IYl29983nt2/J4dZglTutP5ulSr37p95+7yvcb9B82HK6trj46VGEuKR1QkQvZDojBhHI800wn2M4kkDRM8CUf+1H7yAaVigh/qSYaDlFxwFjNKtEXna7VesxlwvNRDFBJTk0mR5aZrR6HY1CVfMCeYpiQ3b4ppwVLOuTksxYKVCimShMhJbvxKLnhEGDM+29G8utILPrYTOcpNbzY3G43AhkHgmSAlehjG4NmAAu2VKIS9OepWqDtHvQr15mjfmICSBPbzOdktyW5F/JL4FemXpF+R05KcVoQODX33fP6Vpsaf6wNzUBxGWc0F4xFyDb7gikUogYCthWJhMgHGi8QgKNQgYhhOMmGTo1CBO9vude7CJdNDMAGmUHof+xCxFPn0FuTgRi4EARAeQSwkIKFDcOOAcbhaoqgGgl0bJFMjaKGU1lcSje3thuv2WnEbAth+aYeg24pbH9u2UO9h0nbdxvnqemejUzS4KbxSrDtl656vfgkiQcf2DzVNiFJnXifTA0OkZjTBvBGMFWaEjsgFnlnJSYpqYIrLn8MzS2YniYVNW0H/jTAkVWqShtZzWmt13TaF/7OdjXW8NTCMZ2ONnM42iscJaAHTl2RTKpFqW5WIESrtdaVAh0QSqu17mybBu37km+J4c8PrbHhvN9d3tsp0LDtPnKdOy/GcF86Os+90nSOH1j7VvtV+1H7WP9e/13/Vf89ca0tlzGNnodX//AX/ZWVT</latexit>

VC bound:<latexit sha1_base64="ggE1XIRxSUc+FWOFqoAe0Sk4VaA=">AAADnHicbZJdb9MwFIbThI9RKOvgCiEhiwoJcVE1u2Hihk3ZpCJgKtXadWpK5TjOatUfke0AlZU/xU/hjn+DmyaBbFhy/OZ5z4lz7BOllCg9GPxuud6du/fu7z1oP3zUebzfPXgyVSKTCE+QoELOIqgwJRxPNNEUz1KJIYsovozWwda//IalIoJf6E2KFwxec5IQBLVFy4PWz04n5Pi7XmEhMTOpFGluRvYpFNmG5A2bYsZgbj4VS8Mp19xclKLhIiEFpVBuchPUshER44Tw3Y7m9K9uxNgJ5To3493aabdDmwZC34QM6lWUAN8mFOisRBE4q9CoRqMKjWs0rtDQmBBBCoZ5RU5KclKToCRBTWYlmdXkqiRXNUErg76+qd4YM0Glz815UYyymgvCY8w1CDX+oYsLtnXHuZkGIBIZj9/ly25v0B8UA9wWfil6TjlGy+6vMBYoY/aziEKl5v4g1QsDpSaI4rwdZgqnEK3hNZ5bySHDamGKvXPwypIYJELaaX+roP9mGMiU2rDIRm7PUt30tvB/3jzTydHCEJ5mGnO02yjJKNACbDsVxERipOnGCoikbQcE0ApKiLTt57Y9BP9mybfF9LDvD/r+l8Pe8VF5HHvOc+el89rxnbfOsTN0Rs7EQe4z9707dD94L7xT76P3eRfqtsqcp05jeNM/bDAtBQ==</latexit><latexit sha1_base64="ggE1XIRxSUc+FWOFqoAe0Sk4VaA=">AAADnHicbZJdb9MwFIbThI9RKOvgCiEhiwoJcVE1u2Hihk3ZpCJgKtXadWpK5TjOatUfke0AlZU/xU/hjn+DmyaBbFhy/OZ5z4lz7BOllCg9GPxuud6du/fu7z1oP3zUebzfPXgyVSKTCE+QoELOIqgwJRxPNNEUz1KJIYsovozWwda//IalIoJf6E2KFwxec5IQBLVFy4PWz04n5Pi7XmEhMTOpFGluRvYpFNmG5A2bYsZgbj4VS8Mp19xclKLhIiEFpVBuchPUshER44Tw3Y7m9K9uxNgJ5To3493aabdDmwZC34QM6lWUAN8mFOisRBE4q9CoRqMKjWs0rtDQmBBBCoZ5RU5KclKToCRBTWYlmdXkqiRXNUErg76+qd4YM0Glz815UYyymgvCY8w1CDX+oYsLtnXHuZkGIBIZj9/ly25v0B8UA9wWfil6TjlGy+6vMBYoY/aziEKl5v4g1QsDpSaI4rwdZgqnEK3hNZ5bySHDamGKvXPwypIYJELaaX+roP9mGMiU2rDIRm7PUt30tvB/3jzTydHCEJ5mGnO02yjJKNACbDsVxERipOnGCoikbQcE0ApKiLTt57Y9BP9mybfF9LDvD/r+l8Pe8VF5HHvOc+el89rxnbfOsTN0Rs7EQe4z9707dD94L7xT76P3eRfqtsqcp05jeNM/bDAtBQ==</latexit><latexit sha1_base64="ggE1XIRxSUc+FWOFqoAe0Sk4VaA=">AAADnHicbZJdb9MwFIbThI9RKOvgCiEhiwoJcVE1u2Hihk3ZpCJgKtXadWpK5TjOatUfke0AlZU/xU/hjn+DmyaBbFhy/OZ5z4lz7BOllCg9GPxuud6du/fu7z1oP3zUebzfPXgyVSKTCE+QoELOIqgwJRxPNNEUz1KJIYsovozWwda//IalIoJf6E2KFwxec5IQBLVFy4PWz04n5Pi7XmEhMTOpFGluRvYpFNmG5A2bYsZgbj4VS8Mp19xclKLhIiEFpVBuchPUshER44Tw3Y7m9K9uxNgJ5To3493aabdDmwZC34QM6lWUAN8mFOisRBE4q9CoRqMKjWs0rtDQmBBBCoZ5RU5KclKToCRBTWYlmdXkqiRXNUErg76+qd4YM0Glz815UYyymgvCY8w1CDX+oYsLtnXHuZkGIBIZj9/ly25v0B8UA9wWfil6TjlGy+6vMBYoY/aziEKl5v4g1QsDpSaI4rwdZgqnEK3hNZ5bySHDamGKvXPwypIYJELaaX+roP9mGMiU2rDIRm7PUt30tvB/3jzTydHCEJ5mGnO02yjJKNACbDsVxERipOnGCoikbQcE0ApKiLTt57Y9BP9mybfF9LDvD/r+l8Pe8VF5HHvOc+el89rxnbfOsTN0Rs7EQe4z9707dD94L7xT76P3eRfqtsqcp05jeNM/bDAtBQ==</latexit><latexit sha1_base64="ggE1XIRxSUc+FWOFqoAe0Sk4VaA=">AAADnHicbZJdb9MwFIbThI9RKOvgCiEhiwoJcVE1u2Hihk3ZpCJgKtXadWpK5TjOatUfke0AlZU/xU/hjn+DmyaBbFhy/OZ5z4lz7BOllCg9GPxuud6du/fu7z1oP3zUebzfPXgyVSKTCE+QoELOIqgwJRxPNNEUz1KJIYsovozWwda//IalIoJf6E2KFwxec5IQBLVFy4PWz04n5Pi7XmEhMTOpFGluRvYpFNmG5A2bYsZgbj4VS8Mp19xclKLhIiEFpVBuchPUshER44Tw3Y7m9K9uxNgJ5To3493aabdDmwZC34QM6lWUAN8mFOisRBE4q9CoRqMKjWs0rtDQmBBBCoZ5RU5KclKToCRBTWYlmdXkqiRXNUErg76+qd4YM0Glz815UYyymgvCY8w1CDX+oYsLtnXHuZkGIBIZj9/ly25v0B8UA9wWfil6TjlGy+6vMBYoY/aziEKl5v4g1QsDpSaI4rwdZgqnEK3hNZ5bySHDamGKvXPwypIYJELaaX+roP9mGMiU2rDIRm7PUt30tvB/3jzTydHCEJ5mGnO02yjJKNACbDsVxERipOnGCoikbQcE0ApKiLTt57Y9BP9mybfF9LDvD/r+l8Pe8VF5HHvOc+el89rxnbfOsTN0Rs7EQe4z9707dD94L7xT76P3eRfqtsqcp05jeNM/bDAtBQ==</latexit>

w.p. � 1� �<latexit sha1_base64="vMjZktiD39IFMDIfusr5xcdEuNU=">AAADmXicbZLfbtMwFMbdhj8jUNaBxM1uLDokhESV7IZdcLGqFFUITaWsW6emTI7rtFbtONgOU2XlmXgX7ngb3DQJZMOSc778znfi+OiECaNKe97vRtO5d//Bw71H7uMnraf77YNnF0qkEpMJFkzIaYgUYTQmE001I9NEEsRDRi7DdX+bv/xBpKIiPtebhMw5WsY0ohhpi64PGj9brSAmN3pFhCTcJFIkmRnZp1B0a8lqaUY4R5n5nIdapoiZOS9ELYuFFIwhuclMv5I1x4JENN6daD781TWP3UiuMzPexZbrBrYMBr4JONKrMIK+LcjRoEAhHJRoVKFRicYVGpdoaEyAEYPDrCS9gvQq0i9IvyLTgkwrclWQq4rglcHf3pRvnJt+qc/MWX4ZZXUsaLwgsYbuTTfpwqNgSb7DAPpvrZVpdHTd7nhdL1/wrvAL0QHFGl23fwULgVNuv4kZUmrme4meGyQ1xYxkbpAqkiC8RksyszJGnKi5yScrg68sWcBISLvtP+X03wqDuFIbHlrntpHqdm4L/5ebpTo6mRsaJ6kmMd4dFKUMagG3YwoXVBKs2cYKhKWdBQzxCkmEtR1m1zbBv33lu+LiuOt7Xf/Lcef0pGjHHjgEL8Fr4IN34BQMwQhMAG6+aL5vDpofnUOn5wydTztrs1HUPAe15Xz9AzDWKR8=</latexit><latexit sha1_base64="vMjZktiD39IFMDIfusr5xcdEuNU=">AAADmXicbZLfbtMwFMbdhj8jUNaBxM1uLDokhESV7IZdcLGqFFUITaWsW6emTI7rtFbtONgOU2XlmXgX7ngb3DQJZMOSc778znfi+OiECaNKe97vRtO5d//Bw71H7uMnraf77YNnF0qkEpMJFkzIaYgUYTQmE001I9NEEsRDRi7DdX+bv/xBpKIiPtebhMw5WsY0ohhpi64PGj9brSAmN3pFhCTcJFIkmRnZp1B0a8lqaUY4R5n5nIdapoiZOS9ELYuFFIwhuclMv5I1x4JENN6daD781TWP3UiuMzPexZbrBrYMBr4JONKrMIK+LcjRoEAhHJRoVKFRicYVGpdoaEyAEYPDrCS9gvQq0i9IvyLTgkwrclWQq4rglcHf3pRvnJt+qc/MWX4ZZXUsaLwgsYbuTTfpwqNgSb7DAPpvrZVpdHTd7nhdL1/wrvAL0QHFGl23fwULgVNuv4kZUmrme4meGyQ1xYxkbpAqkiC8RksyszJGnKi5yScrg68sWcBISLvtP+X03wqDuFIbHlrntpHqdm4L/5ebpTo6mRsaJ6kmMd4dFKUMagG3YwoXVBKs2cYKhKWdBQzxCkmEtR1m1zbBv33lu+LiuOt7Xf/Lcef0pGjHHjgEL8Fr4IN34BQMwQhMAG6+aL5vDpofnUOn5wydTztrs1HUPAe15Xz9AzDWKR8=</latexit><latexit sha1_base64="vMjZktiD39IFMDIfusr5xcdEuNU=">AAADmXicbZLfbtMwFMbdhj8jUNaBxM1uLDokhESV7IZdcLGqFFUITaWsW6emTI7rtFbtONgOU2XlmXgX7ngb3DQJZMOSc778znfi+OiECaNKe97vRtO5d//Bw71H7uMnraf77YNnF0qkEpMJFkzIaYgUYTQmE001I9NEEsRDRi7DdX+bv/xBpKIiPtebhMw5WsY0ohhpi64PGj9brSAmN3pFhCTcJFIkmRnZp1B0a8lqaUY4R5n5nIdapoiZOS9ELYuFFIwhuclMv5I1x4JENN6daD781TWP3UiuMzPexZbrBrYMBr4JONKrMIK+LcjRoEAhHJRoVKFRicYVGpdoaEyAEYPDrCS9gvQq0i9IvyLTgkwrclWQq4rglcHf3pRvnJt+qc/MWX4ZZXUsaLwgsYbuTTfpwqNgSb7DAPpvrZVpdHTd7nhdL1/wrvAL0QHFGl23fwULgVNuv4kZUmrme4meGyQ1xYxkbpAqkiC8RksyszJGnKi5yScrg68sWcBISLvtP+X03wqDuFIbHlrntpHqdm4L/5ebpTo6mRsaJ6kmMd4dFKUMagG3YwoXVBKs2cYKhKWdBQzxCkmEtR1m1zbBv33lu+LiuOt7Xf/Lcef0pGjHHjgEL8Fr4IN34BQMwQhMAG6+aL5vDpofnUOn5wydTztrs1HUPAe15Xz9AzDWKR8=</latexit><latexit sha1_base64="vMjZktiD39IFMDIfusr5xcdEuNU=">AAADmXicbZLfbtMwFMbdhj8jUNaBxM1uLDokhESV7IZdcLGqFFUITaWsW6emTI7rtFbtONgOU2XlmXgX7ngb3DQJZMOSc778znfi+OiECaNKe97vRtO5d//Bw71H7uMnraf77YNnF0qkEpMJFkzIaYgUYTQmE001I9NEEsRDRi7DdX+bv/xBpKIiPtebhMw5WsY0ohhpi64PGj9brSAmN3pFhCTcJFIkmRnZp1B0a8lqaUY4R5n5nIdapoiZOS9ELYuFFIwhuclMv5I1x4JENN6daD781TWP3UiuMzPexZbrBrYMBr4JONKrMIK+LcjRoEAhHJRoVKFRicYVGpdoaEyAEYPDrCS9gvQq0i9IvyLTgkwrclWQq4rglcHf3pRvnJt+qc/MWX4ZZXUsaLwgsYbuTTfpwqNgSb7DAPpvrZVpdHTd7nhdL1/wrvAL0QHFGl23fwULgVNuv4kZUmrme4meGyQ1xYxkbpAqkiC8RksyszJGnKi5yScrg68sWcBISLvtP+X03wqDuFIbHlrntpHqdm4L/5ebpTo6mRsaJ6kmMd4dFKUMagG3YwoXVBKs2cYKhKWdBQzxCkmEtR1m1zbBv33lu+LiuOt7Xf/Lcef0pGjHHjgEL8Fr4IN34BQMwQhMAG6+aL5vDpofnUOn5wydTztrs1HUPAe15Xz9AzDWKR8=</latexit>

supf2F

|R(f)� bR(f)| 6

rd log(n/�)

n<latexit sha1_base64="ohtS4E8AeM7NGQ90lvwpjf4Me6g=">AAAD23icbZPPb9MwFMfTlh+jUOjgyMWiReqQKM2QgAOHTWWoBzSVat061aVyHKe16tiZ7TBVnrlwACGu/GPc+DP4D3DbNNANS877+vPey4vtlyBhVOlW61ehWLp2/cbNrVvl23cqd+9Vt+8fK5FKTPpYMCEHAVKEUU76mmpGBokkKA4YOQlm7YX/5CORigp+pOcJGcVowmlEMdIOjbcLvysVyMm5nhIhSWwSKRJruu4pFF2E2A03I3GMrHm3NBuezFpzlIkNLxZSMIbk3Jp2LjciQhJRvqpo3vzVGzFuIjmzpreylXIZujQAfQNjpKdBBHyXsEQHGQrAwRp1c9Rdo16OemvUMQZixEDHrsl+RvZz0s5IOyeDjAxycpqR05zgqcEfnqxXcWzaa31oDpebUU5zQXlIuAbleh2qNBmbCFK+etdbay96jWjnKTynIZkiDRarCwABZOTMmRcAqjOpDYwkwiaETExAgz9zVZhGO9Zwa+v1cbXWaraWA1wVfiZqXja64+pPGAqcxu6rMENKDf1WokcGSU0xI7YMU0UShGdoQoZOchQTNTLL3rTgsSMhiIR00+1qSf/NMChWah4HLnJxFeqybwH/5xumOno1MpQnqSYcrwpFKQNagEWjg5BKgjWbO4GwdN2EAZ4idy7a/Q5ldwj+5S1fFce7Tf9503+/W9t7nR3HlvfQe+Q1PN976e15Ha/r9T1cHBQ/Fb8Uv5ZGpc+lb6Xvq9BiIct54G2M0o8/KctD4w==</latexit>

Given training data {(xj , yj)}nj=1, learn a function f to predict y from x<latexit sha1_base64="DY5fekva7qVFNgVRXpdsaLjf+JI=">AAAD23icbZPPb9MwFMfTlh+jUOjgyMWiQRpoqppxgANIm8pED2gq1bp1arrKcZzWrX9EtjNWWeHCAYS48o9x48/gP8Bp00A3LDnvm897T857eQ5iSpRutX6VypUbN2/d3rpTvXuvdv9BffvhiRKJRLiPBBVyEECFKeG4r4mmeBBLDFlA8Wkwb2f+0wssFRH8WC9iPGJwwklEENQWjbdLv2s1n+OPeoqFxMzEUsSp6dqnUCQLSTfcFDMGU/N+aTY8uU3NcS42vEhIQSmUi9S0C7kREeKI8NWJ5u1fvRFjN5Tz1PRWtlat+jYN+J7xGdTTIAKeTViiwxwF4HCNugXqrlGvQL016hjjI0hBJ12Tg5wcFKSdk3ZBBjkZFOQsJ2cFQVODzp+v3xgz7bU+MkfLYpTVXBAeYq7BO3KBOdAS2lbwCQihhsD1zc7leLa7GM+e+enYzN546Tl3dwHFUHIAQZRwlPUNuJELtAB2FkKCNHAXLoikYMC9dMf1RqvZWi5wXXi5aDj56o7rP/1QoITZr0IUKjX0WrEeGSg1QRSnVT9ROIZoDid4aCWHDKuRWc5mCp5aEoJISLttVUv6b4aBTKkFC2xk9ivUVV8G/+cbJjp6NTKEx4nGHK0OihKaVZ0NOgiJxEjThRUQSTtNCKAplBBpex2qtgne1ZKvi5O9pvei6X3Ya+y/ztux5Tx2njg7jue8dPadjtN1+g4qD8qfyl/KXyujyufKt8r3VWi5lOc8cjZW5ccfJGdDJA==</latexit>

error rate ontraining data:

<latexit sha1_base64="WkOtKxzuaQlW7C7HIKPFlNbJUZk=">AAADsnicbZJdb9MwFIbdho9RKHRwyY1FhYS4KMmExMTVpjKpF2gq1bp1NF3lOE5r1R+R7QCVlf/HNXf8G9w0CWTjSInfPO85ss+Jo5RRbXz/d6vt3bv/4OHBo87jJ92nz3qHzy+1zBQmUyyZVLMIacKoIFNDDSOzVBHEI0auos1w5199I0pTKS7MNiULjlaCJhQj49DysPWz2w0F+W7WRCrCbapkmtuxe0tNdyl5w2aEc5Tbz8XScMo1txelaLhYKskYUtvcDmvZyIhJQsV+R/vpr27kuAepTW4n+7Xb6YSuDIaBDTky6yiBgSso0FmJInhWoXGNxhWa1GhSoZG1IUYMjvKKnJbktCbDkgxrMivJrCbXJbmuCV5bfPO2+uLcDit9bs+LZrTTQlIRE2FgaMgPU/xg13ecW6KUVFAhQ6AUMAyhUchNSaxgjAz6mC97fX/gFwHviqAUfVDGeNn7FcYSZ9xthhnSeh74qVlYpAzFjOSdMNMkRXiDVmTupECc6IUtTpTD147EMHEnSqQ7bEH/rbCIa73lkcvcTVjf9nbwf948M8nxwlKRZoYIvN8oyRg0Eu7uL4ypItiwrRMIK3dJMMRrpBA27pZ33BCC2y3fFZdHg8AfBF+O+ifH5TgOwEvwCrwBAfgATsAIjMEU4Pa79rR90156772vHvLwPrXdKmtegEZ47A+HyTYR</latexit><latexit sha1_base64="WkOtKxzuaQlW7C7HIKPFlNbJUZk=">AAADsnicbZJdb9MwFIbdho9RKHRwyY1FhYS4KMmExMTVpjKpF2gq1bp1NF3lOE5r1R+R7QCVlf/HNXf8G9w0CWTjSInfPO85ss+Jo5RRbXz/d6vt3bv/4OHBo87jJ92nz3qHzy+1zBQmUyyZVLMIacKoIFNDDSOzVBHEI0auos1w5199I0pTKS7MNiULjlaCJhQj49DysPWz2w0F+W7WRCrCbapkmtuxe0tNdyl5w2aEc5Tbz8XScMo1txelaLhYKskYUtvcDmvZyIhJQsV+R/vpr27kuAepTW4n+7Xb6YSuDIaBDTky6yiBgSso0FmJInhWoXGNxhWa1GhSoZG1IUYMjvKKnJbktCbDkgxrMivJrCbXJbmuCV5bfPO2+uLcDit9bs+LZrTTQlIRE2FgaMgPU/xg13ecW6KUVFAhQ6AUMAyhUchNSaxgjAz6mC97fX/gFwHviqAUfVDGeNn7FcYSZ9xthhnSeh74qVlYpAzFjOSdMNMkRXiDVmTupECc6IUtTpTD147EMHEnSqQ7bEH/rbCIa73lkcvcTVjf9nbwf948M8nxwlKRZoYIvN8oyRg0Eu7uL4ypItiwrRMIK3dJMMRrpBA27pZ33BCC2y3fFZdHg8AfBF+O+ifH5TgOwEvwCrwBAfgATsAIjMEU4Pa79rR90156772vHvLwPrXdKmtegEZ47A+HyTYR</latexit><latexit sha1_base64="WkOtKxzuaQlW7C7HIKPFlNbJUZk=">AAADsnicbZJdb9MwFIbdho9RKHRwyY1FhYS4KMmExMTVpjKpF2gq1bp1NF3lOE5r1R+R7QCVlf/HNXf8G9w0CWTjSInfPO85ss+Jo5RRbXz/d6vt3bv/4OHBo87jJ92nz3qHzy+1zBQmUyyZVLMIacKoIFNDDSOzVBHEI0auos1w5199I0pTKS7MNiULjlaCJhQj49DysPWz2w0F+W7WRCrCbapkmtuxe0tNdyl5w2aEc5Tbz8XScMo1txelaLhYKskYUtvcDmvZyIhJQsV+R/vpr27kuAepTW4n+7Xb6YSuDIaBDTky6yiBgSso0FmJInhWoXGNxhWa1GhSoZG1IUYMjvKKnJbktCbDkgxrMivJrCbXJbmuCV5bfPO2+uLcDit9bs+LZrTTQlIRE2FgaMgPU/xg13ecW6KUVFAhQ6AUMAyhUchNSaxgjAz6mC97fX/gFwHviqAUfVDGeNn7FcYSZ9xthhnSeh74qVlYpAzFjOSdMNMkRXiDVmTupECc6IUtTpTD147EMHEnSqQ7bEH/rbCIa73lkcvcTVjf9nbwf948M8nxwlKRZoYIvN8oyRg0Eu7uL4ypItiwrRMIK3dJMMRrpBA27pZ33BCC2y3fFZdHg8AfBF+O+ifH5TgOwEvwCrwBAfgATsAIjMEU4Pa79rR90156772vHvLwPrXdKmtegEZ47A+HyTYR</latexit><latexit sha1_base64="WkOtKxzuaQlW7C7HIKPFlNbJUZk=">AAADsnicbZJdb9MwFIbdho9RKHRwyY1FhYS4KMmExMTVpjKpF2gq1bp1NF3lOE5r1R+R7QCVlf/HNXf8G9w0CWTjSInfPO85ss+Jo5RRbXz/d6vt3bv/4OHBo87jJ92nz3qHzy+1zBQmUyyZVLMIacKoIFNDDSOzVBHEI0auos1w5199I0pTKS7MNiULjlaCJhQj49DysPWz2w0F+W7WRCrCbapkmtuxe0tNdyl5w2aEc5Tbz8XScMo1txelaLhYKskYUtvcDmvZyIhJQsV+R/vpr27kuAepTW4n+7Xb6YSuDIaBDTky6yiBgSso0FmJInhWoXGNxhWa1GhSoZG1IUYMjvKKnJbktCbDkgxrMivJrCbXJbmuCV5bfPO2+uLcDit9bs+LZrTTQlIRE2FgaMgPU/xg13ecW6KUVFAhQ6AUMAyhUchNSaxgjAz6mC97fX/gFwHviqAUfVDGeNn7FcYSZ9xthhnSeh74qVlYpAzFjOSdMNMkRXiDVmTupECc6IUtTpTD147EMHEnSqQ7bEH/rbCIa73lkcvcTVjf9nbwf948M8nxwlKRZoYIvN8oyRg0Eu7uL4ypItiwrRMIK3dJMMRrpBA27pZ33BCC2y3fFZdHg8AfBF+O+ifH5TgOwEvwCrwBAfgATsAIjMEU4Pa79rR90156772vHvLwPrXdKmtegEZ47A+HyTYR</latexit>

“empirical risk”<latexit sha1_base64="F41JY1o7g9kvevePn/f/upmVwWY=">AAADpXicbZLdbtMwFMfdhI9RKHRwyY3VCoG4mJLdbJebyqRewNRVbdepKZ3jOqtVO45sB6isPBlvwR1vg5MmgWwcyTn//M45/jxhwqjSnve75biPHj95evCs/fxF5+Wr7uHrmRKpxGSKBRNyHiJFGI3JVFPNyDyRBPGQketwO8jj19+IVFTEE71LyJKju5hGFCNt0eqw9bPTCWLyXW+IkISbRIokMyP7FYrmKVkjzAjnKDOfC9eIlD4zk1I0olhIwRiSu8wMatnIWJOIxvsVzae/upFjB5LbzIz3vtNuB7YMBr4JONKbMIK+LSjQRYlCeFGhUY1GFRrXaFyhoTEBRgwOs4qcl+S8JoOSDGoyL8m8JjcluakJ3hj89WP1x7kZVPrSXBaHUVbHgsZrEmsYaPJDFw9sQpaSzNzeEp5QSfNZJVXbXrbq9r0jrzD4UPil6IPSRqvur2AtcMrt/JghpRa+l+ilQVJTzEjWDlJFEoS36I4srIwRJ2ppik1k8J0laxgJaYfdX0H/rTCIK7Xjoc3ML1Xdj+Xwf7FFqqPTpaFxkmoS4/1CUcqgFjBvWbimkmDNdlYgLG1fYIg3SCKsbWO37SX494/8UMyOj3zvyL867p+dltdxAN6CHvgAfHACzsAQjMAUYKfnDJ0rZ+y+d7+4E3e2T3VaZc0b0DB39QdlfTEy</latexit><latexit sha1_base64="F41JY1o7g9kvevePn/f/upmVwWY=">AAADpXicbZLdbtMwFMfdhI9RKHRwyY3VCoG4mJLdbJebyqRewNRVbdepKZ3jOqtVO45sB6isPBlvwR1vg5MmgWwcyTn//M45/jxhwqjSnve75biPHj95evCs/fxF5+Wr7uHrmRKpxGSKBRNyHiJFGI3JVFPNyDyRBPGQketwO8jj19+IVFTEE71LyJKju5hGFCNt0eqw9bPTCWLyXW+IkISbRIokMyP7FYrmKVkjzAjnKDOfC9eIlD4zk1I0olhIwRiSu8wMatnIWJOIxvsVzae/upFjB5LbzIz3vtNuB7YMBr4JONKbMIK+LSjQRYlCeFGhUY1GFRrXaFyhoTEBRgwOs4qcl+S8JoOSDGoyL8m8JjcluakJ3hj89WP1x7kZVPrSXBaHUVbHgsZrEmsYaPJDFw9sQpaSzNzeEp5QSfNZJVXbXrbq9r0jrzD4UPil6IPSRqvur2AtcMrt/JghpRa+l+ilQVJTzEjWDlJFEoS36I4srIwRJ2ppik1k8J0laxgJaYfdX0H/rTCIK7Xjoc3ML1Xdj+Xwf7FFqqPTpaFxkmoS4/1CUcqgFjBvWbimkmDNdlYgLG1fYIg3SCKsbWO37SX494/8UMyOj3zvyL867p+dltdxAN6CHvgAfHACzsAQjMAUYKfnDJ0rZ+y+d7+4E3e2T3VaZc0b0DB39QdlfTEy</latexit><latexit sha1_base64="F41JY1o7g9kvevePn/f/upmVwWY=">AAADpXicbZLdbtMwFMfdhI9RKHRwyY3VCoG4mJLdbJebyqRewNRVbdepKZ3jOqtVO45sB6isPBlvwR1vg5MmgWwcyTn//M45/jxhwqjSnve75biPHj95evCs/fxF5+Wr7uHrmRKpxGSKBRNyHiJFGI3JVFPNyDyRBPGQketwO8jj19+IVFTEE71LyJKju5hGFCNt0eqw9bPTCWLyXW+IkISbRIokMyP7FYrmKVkjzAjnKDOfC9eIlD4zk1I0olhIwRiSu8wMatnIWJOIxvsVzae/upFjB5LbzIz3vtNuB7YMBr4JONKbMIK+LSjQRYlCeFGhUY1GFRrXaFyhoTEBRgwOs4qcl+S8JoOSDGoyL8m8JjcluakJ3hj89WP1x7kZVPrSXBaHUVbHgsZrEmsYaPJDFw9sQpaSzNzeEp5QSfNZJVXbXrbq9r0jrzD4UPil6IPSRqvur2AtcMrt/JghpRa+l+ilQVJTzEjWDlJFEoS36I4srIwRJ2ppik1k8J0laxgJaYfdX0H/rTCIK7Xjoc3ML1Xdj+Xwf7FFqqPTpaFxkmoS4/1CUcqgFjBvWbimkmDNdlYgLG1fYIg3SCKsbWO37SX494/8UMyOj3zvyL867p+dltdxAN6CHvgAfHACzsAQjMAUYKfnDJ0rZ+y+d7+4E3e2T3VaZc0b0DB39QdlfTEy</latexit><latexit sha1_base64="F41JY1o7g9kvevePn/f/upmVwWY=">AAADpXicbZLdbtMwFMfdhI9RKHRwyY3VCoG4mJLdbJebyqRewNRVbdepKZ3jOqtVO45sB6isPBlvwR1vg5MmgWwcyTn//M45/jxhwqjSnve75biPHj95evCs/fxF5+Wr7uHrmRKpxGSKBRNyHiJFGI3JVFPNyDyRBPGQketwO8jj19+IVFTEE71LyJKju5hGFCNt0eqw9bPTCWLyXW+IkISbRIokMyP7FYrmKVkjzAjnKDOfC9eIlD4zk1I0olhIwRiSu8wMatnIWJOIxvsVzae/upFjB5LbzIz3vtNuB7YMBr4JONKbMIK+LSjQRYlCeFGhUY1GFRrXaFyhoTEBRgwOs4qcl+S8JoOSDGoyL8m8JjcluakJ3hj89WP1x7kZVPrSXBaHUVbHgsZrEmsYaPJDFw9sQpaSzNzeEp5QSfNZJVXbXrbq9r0jrzD4UPil6IPSRqvur2AtcMrt/JghpRa+l+ilQVJTzEjWDlJFEoS36I4srIwRJ2ppik1k8J0laxgJaYfdX0H/rTCIK7Xjoc3ML1Xdj+Xwf7FFqqPTpaFxkmoS4/1CUcqgFjBvWbimkmDNdlYgLG1fYIg3SCKsbWO37SX494/8UMyOj3zvyL867p+dltdxAN6CHvgAfHACzsAQjMAUYKfnDJ0rZ+y+d7+4E3e2T3VaZc0b0DB39QdlfTEy</latexit>

This is the guarantee for the test at just one point. If we test a k points during the binary search

process, then the probability that one or more of the tests fails is bounded by k�. Thus, if we replace

� with �/k the bound on m above, then the probability that all k tests are correct is at least 1� �. So

we will perform k steps of the binary search and collect

m = O

✓2 log(2k/�)

�2

◆

samples in each step. Taking k = log(1/✏) yields an ✏-accurate solution with probability at least

1� � and the total number of (noisy) function evaluations is

O

0

BBB@log(1/✏)

2 log(2 log(1/✏)/�)

�2| {z }price paid due to noise

1

CCCA.

4 Binary Classification

The binary search problems above assume one can request the label at any chosen point x, but this is

not the case in most machine learning (ML) settings. Rather, in ML applications have access to a

pool of examples of x distributed according to some distribution and a subset of these examples can

be labeled (usually at a cost). The goal is to learn a good classifier using as few labeled examples as

possible.

Consider the finite set of hypotheses fi, i = 1, . . . , k. Each hypothesis has a probability of error

R(fi) := P(fi(x) 6= y)

where the probability is computed over the distribution on x and the binary distribution of y given x.

Empirical risk minimization (ERM) is a standard ML approach which proceeds as follows. Select the

hypothesis that makes the fewest prediction errors on a set of iid labeled examples {(xi, yi)}mi=1. The

basic ingredient in ERM is an estimate of the error rate of each hypothesis

bR(fi) =1

m

MX

i=1

1(f(xi) 6= yi) ,

which is usually referred to as the empirical risk. Note that the terms in this average are iid binary

random variables, and using the Chernoff bound confidence intervals we have

|R(fi)� bR(fi)| r

log(2/�)

2m

with probability at least 1� �. In order to have the confidence intervals hold simultaneously for all

hypotheses, we apply the union bound (replace � with �/k) to have

|R(fi)� bR(fi)| r

log(2k/�)

2mfor i = 1, . . . , k

Now suppose we want to select a hypothesis with true error rate within � > 0 of mini R(fi). Define

true and empirical risk minimizers

f? = arg min

f2{fi}R(f)

bf = arg minf2{fi}

bR(f)

To determine how many samples will be sufficient to guarantee that R( bf) R(f?) + �, consider the

confidence intervals for these two hypotheses. The lower confidence bound for f?

is

R(f?) � bR(f?)�r

log(2k/�)

2m

3

bR(f) =1

n

nX

i=1

⇣f(xi) 6= yi

⌘

<latexit sha1_base64="uLm4hnIYWO8m0dqspPt8xRONBQg=">AAADzXicbZPPb9MwFMfThB+jUNbBkYtFi9RyqJpxgAOTNsqkCqFRqnXr1HSV4zqtVf8ItsMoJlz5/7hx5h/BadNAN57k+JvP9z0578kJY0qUbrd/lVzv1u07d3fule8/qDzcre49OlMikQgPkKBCDkOoMCUcDzTRFA9jiSELKT4PF53MP/+MpSKCn+pljMcMzjiJCILaosle6XelEnB8pedYSMxMLEWcmp59CkWylHTLppgxmJr3q23LyffUnOZiy0VCCkqhXKamU8itjCmOCF+faN7+1Vs5dkG5SE1/vVfK5cCWgcA3AYN6HkbAtwUrdJyjEBxvUK9AvQ3qF6i/QV1jAgQp6KYbcpSTo4J0ctIpyDAnw4Jc5OSiIGhu0OXzzRtjprPRJ+Zk1YyymgvCp5hrUK7XgysyxXOoQb8RNUEADuwKIgmR8VPD00AlbGLIgZ9e8sx5Q2aNqPFlQpp2aJ/AckIy1KzXJ9Vau9VeBbgp/FzUnDx6k+rPYCpQwux3IAqVGvntWI8NlJogitNykCgcQ7SAMzyykkOG1disbmMKnlkyBZGQdtk+VvTfCgOZUksW2sxs+Oq6l8H/eaNER6/GhvA40Zij9UFRQoEWILvaYEokRpourYBI2vuDAJpDOy5tf4CyHYJ/veWb4my/5b9o+R/3a4ev83HsOE+cp07D8Z2XzqHTdXrOwEHuOzd2l+5X74OXeN+87+tUt5TXPHa2wvvxBzFCPO0=</latexit>

Vapnik-Chervonenkis (VC) Theory

Empirical Risk Minimization (ERM)

bR( bf)<latexit sha1_base64="anfNbe5yQlK0MQq0cHBm0IafXYA=">AAADmnicbZLfbtMwFMbdhj+jUOjgggu4sOiQBhdVsht2g7RRBkWgqVTt1qnpKsdxVqt2HNkOU2XlnXgW7ngb3DQJZMOSc778vnPi+OgECaNKu+7vRtO5c/fe/Z0HrYeP2o+fdHafnimRSkwmWDAhpwFShNGYTDTVjEwTSRAPGDkPVv2Nf/6DSEVFPNbrhMw5uoppRDHSFi12Gz/bbT8m13pJhCTcJFIkmRnap1B0k5LVbEY4R5n5loeaU8TMjAtRc7GQgjEk15npV7KWEZKIxtsTzce/upZjN5KrzIy2sd1q+bYM+p7xOdLLIIKeLcjRSYECeFKiYYWGJRpVaFSigTE+RgwOspIcF+S4Iv2C9CsyLci0IhcFuagIXhp8+bZ849z0S31qTvPLKKtjQeOQxBru+dc0JEuk4Wi/ktGbvUWn6/bcfMHbwitEFxRruOj88kOBU24/ihlSaua5iZ4bJDXFjGQtP1UkQXiFrsjMyhhxouYmH60MvrYkhJGQdtufyum/FQZxpdY8sJmbTqqb3gb+z5ulOjqcGxonqSYx3h4UpQxqATdzCkMqCdZsbQXC0g4DhniJJMLaTnPLNsG7eeXb4uyg57k97/tB9+iwaMcOeAFegX3ggXfgCAzAEEwAbj5vvm9+an52XjofnC/O121qs1HUPAO15Yz/ACx5KpA=</latexit><latexit sha1_base64="anfNbe5yQlK0MQq0cHBm0IafXYA=">AAADmnicbZLfbtMwFMbdhj+jUOjgggu4sOiQBhdVsht2g7RRBkWgqVTt1qnpKsdxVqt2HNkOU2XlnXgW7ngb3DQJZMOSc778vnPi+OgECaNKu+7vRtO5c/fe/Z0HrYeP2o+fdHafnimRSkwmWDAhpwFShNGYTDTVjEwTSRAPGDkPVv2Nf/6DSEVFPNbrhMw5uoppRDHSFi12Gz/bbT8m13pJhCTcJFIkmRnap1B0k5LVbEY4R5n5loeaU8TMjAtRc7GQgjEk15npV7KWEZKIxtsTzce/upZjN5KrzIy2sd1q+bYM+p7xOdLLIIKeLcjRSYECeFKiYYWGJRpVaFSigTE+RgwOspIcF+S4Iv2C9CsyLci0IhcFuagIXhp8+bZ849z0S31qTvPLKKtjQeOQxBru+dc0JEuk4Wi/ktGbvUWn6/bcfMHbwitEFxRruOj88kOBU24/ihlSaua5iZ4bJDXFjGQtP1UkQXiFrsjMyhhxouYmH60MvrYkhJGQdtufyum/FQZxpdY8sJmbTqqb3gb+z5ulOjqcGxonqSYx3h4UpQxqATdzCkMqCdZsbQXC0g4DhniJJMLaTnPLNsG7eeXb4uyg57k97/tB9+iwaMcOeAFegX3ggXfgCAzAEEwAbj5vvm9+an52XjofnC/O121qs1HUPAO15Yz/ACx5KpA=</latexit><latexit sha1_base64="anfNbe5yQlK0MQq0cHBm0IafXYA=">AAADmnicbZLfbtMwFMbdhj+jUOjgggu4sOiQBhdVsht2g7RRBkWgqVTt1qnpKsdxVqt2HNkOU2XlnXgW7ngb3DQJZMOSc778vnPi+OgECaNKu+7vRtO5c/fe/Z0HrYeP2o+fdHafnimRSkwmWDAhpwFShNGYTDTVjEwTSRAPGDkPVv2Nf/6DSEVFPNbrhMw5uoppRDHSFi12Gz/bbT8m13pJhCTcJFIkmRnap1B0k5LVbEY4R5n5loeaU8TMjAtRc7GQgjEk15npV7KWEZKIxtsTzce/upZjN5KrzIy2sd1q+bYM+p7xOdLLIIKeLcjRSYECeFKiYYWGJRpVaFSigTE+RgwOspIcF+S4Iv2C9CsyLci0IhcFuagIXhp8+bZ849z0S31qTvPLKKtjQeOQxBru+dc0JEuk4Wi/ktGbvUWn6/bcfMHbwitEFxRruOj88kOBU24/ihlSaua5iZ4bJDXFjGQtP1UkQXiFrsjMyhhxouYmH60MvrYkhJGQdtufyum/FQZxpdY8sJmbTqqb3gb+z5ulOjqcGxonqSYx3h4UpQxqATdzCkMqCdZsbQXC0g4DhniJJMLaTnPLNsG7eeXb4uyg57k97/tB9+iwaMcOeAFegX3ggXfgCAzAEEwAbj5vvm9+an52XjofnC/O121qs1HUPAO15Yz/ACx5KpA=</latexit><latexit sha1_base64="anfNbe5yQlK0MQq0cHBm0IafXYA=">AAADmnicbZLfbtMwFMbdhj+jUOjgggu4sOiQBhdVsht2g7RRBkWgqVTt1qnpKsdxVqt2HNkOU2XlnXgW7ngb3DQJZMOSc778vnPi+OgECaNKu+7vRtO5c/fe/Z0HrYeP2o+fdHafnimRSkwmWDAhpwFShNGYTDTVjEwTSRAPGDkPVv2Nf/6DSEVFPNbrhMw5uoppRDHSFi12Gz/bbT8m13pJhCTcJFIkmRnap1B0k5LVbEY4R5n5loeaU8TMjAtRc7GQgjEk15npV7KWEZKIxtsTzce/upZjN5KrzIy2sd1q+bYM+p7xOdLLIIKeLcjRSYECeFKiYYWGJRpVaFSigTE+RgwOspIcF+S4Iv2C9CsyLci0IhcFuagIXhp8+bZ849z0S31qTvPLKKtjQeOQxBru+dc0JEuk4Wi/ktGbvUWn6/bcfMHbwitEFxRruOj88kOBU24/ihlSaua5iZ4bJDXFjGQtP1UkQXiFrsjMyhhxouYmH60MvrYkhJGQdtufyum/FQZxpdY8sJmbTqqb3gb+z5ulOjqcGxonqSYx3h4UpQxqATdzCkMqCdZsbQXC0g4DhniJJMLaTnPLNsG7eeXb4uyg57k97/tB9+iwaMcOeAFegX3ggXfgCAzAEEwAbj5vvm9+an52XjofnC/O121qs1HUPAO15Yz/ACx5KpA=</latexit>

bR(f?)<latexit sha1_base64="uMwFHPQEDuDGFAvCjiN8ZN9Fiwk=">AAADl3icbZLfbtMwFMbdhj+jUNYBN4gbiw5pcFElu2Hihk3ZUC/Q1FXr1qnpJsdxVqt2HNkOU2XlkXgZ7ngb3DQJZMOSc778vnPi+OiEKaNKu+7vVtt59PjJ061nnecvui+3ezuvLpTIJCYTLJiQ0xApwmhCJppqRqapJIiHjFyGS3/tX/4gUlGRnOtVSuYc3SY0phhpi252Wj+73SAhd3pBhCTcpFKkuRnZp1B0nZI3bEY4R7n5XoSGU8bcnJei4WIhBWNIrnLj17KREZGYJpsTzfFf3cixG8llbsab2O10AlsGA88EHOlFGEPPFhTopEQhPKnQqEajCo1rNK7Q0JgAIwaHeUWOSnJUE78kfk2mJZnW5KokVzXBC4OvP1VvnBu/0qfmtLiMsjoRNIlIouFucEcjskAajvfi60BpJD/u3vT67sAtFnwovFL0QblGN71fQSRwxu0XMUNKzTw31XODpKaYkbwTZIqkCC/RLZlZmSBO1NwUc5XDD5ZEMBbSbvtHBf23wiCu1IqHNnPdRnXfW8P/ebNMxwdzQ5M00yTBm4PijEEt4HpIYUQlwZqtrEBY2knAEC+QRFjbUe7YJnj3r/xQXOwPPHfgne33Dw/KdmyBd+A92AMe+AwOwRCMwATg9pv2l7bfPnbeOl+db85wk9pulTWvQWM5Z38AbnkphA==</latexit><latexit sha1_base64="uMwFHPQEDuDGFAvCjiN8ZN9Fiwk=">AAADl3icbZLfbtMwFMbdhj+jUNYBN4gbiw5pcFElu2Hihk3ZUC/Q1FXr1qnpJsdxVqt2HNkOU2XlkXgZ7ngb3DQJZMOSc778vnPi+OiEKaNKu+7vVtt59PjJ061nnecvui+3ezuvLpTIJCYTLJiQ0xApwmhCJppqRqapJIiHjFyGS3/tX/4gUlGRnOtVSuYc3SY0phhpi252Wj+73SAhd3pBhCTcpFKkuRnZp1B0nZI3bEY4R7n5XoSGU8bcnJei4WIhBWNIrnLj17KREZGYJpsTzfFf3cixG8llbsab2O10AlsGA88EHOlFGEPPFhTopEQhPKnQqEajCo1rNK7Q0JgAIwaHeUWOSnJUE78kfk2mJZnW5KokVzXBC4OvP1VvnBu/0qfmtLiMsjoRNIlIouFucEcjskAajvfi60BpJD/u3vT67sAtFnwovFL0QblGN71fQSRwxu0XMUNKzTw31XODpKaYkbwTZIqkCC/RLZlZmSBO1NwUc5XDD5ZEMBbSbvtHBf23wiCu1IqHNnPdRnXfW8P/ebNMxwdzQ5M00yTBm4PijEEt4HpIYUQlwZqtrEBY2knAEC+QRFjbUe7YJnj3r/xQXOwPPHfgne33Dw/KdmyBd+A92AMe+AwOwRCMwATg9pv2l7bfPnbeOl+db85wk9pulTWvQWM5Z38AbnkphA==</latexit><latexit sha1_base64="uMwFHPQEDuDGFAvCjiN8ZN9Fiwk=">AAADl3icbZLfbtMwFMbdhj+jUNYBN4gbiw5pcFElu2Hihk3ZUC/Q1FXr1qnpJsdxVqt2HNkOU2XlkXgZ7ngb3DQJZMOSc778vnPi+OiEKaNKu+7vVtt59PjJ061nnecvui+3ezuvLpTIJCYTLJiQ0xApwmhCJppqRqapJIiHjFyGS3/tX/4gUlGRnOtVSuYc3SY0phhpi252Wj+73SAhd3pBhCTcpFKkuRnZp1B0nZI3bEY4R7n5XoSGU8bcnJei4WIhBWNIrnLj17KREZGYJpsTzfFf3cixG8llbsab2O10AlsGA88EHOlFGEPPFhTopEQhPKnQqEajCo1rNK7Q0JgAIwaHeUWOSnJUE78kfk2mJZnW5KokVzXBC4OvP1VvnBu/0qfmtLiMsjoRNIlIouFucEcjskAajvfi60BpJD/u3vT67sAtFnwovFL0QblGN71fQSRwxu0XMUNKzTw31XODpKaYkbwTZIqkCC/RLZlZmSBO1NwUc5XDD5ZEMBbSbvtHBf23wiCu1IqHNnPdRnXfW8P/ebNMxwdzQ5M00yTBm4PijEEt4HpIYUQlwZqtrEBY2knAEC+QRFjbUe7YJnj3r/xQXOwPPHfgne33Dw/KdmyBd+A92AMe+AwOwRCMwATg9pv2l7bfPnbeOl+db85wk9pulTWvQWM5Z38AbnkphA==</latexit><latexit sha1_base64="uMwFHPQEDuDGFAvCjiN8ZN9Fiwk=">AAADl3icbZLfbtMwFMbdhj+jUNYBN4gbiw5pcFElu2Hihk3ZUC/Q1FXr1qnpJsdxVqt2HNkOU2XlkXgZ7ngb3DQJZMOSc778vnPi+OiEKaNKu+7vVtt59PjJ061nnecvui+3ezuvLpTIJCYTLJiQ0xApwmhCJppqRqapJIiHjFyGS3/tX/4gUlGRnOtVSuYc3SY0phhpi252Wj+73SAhd3pBhCTcpFKkuRnZp1B0nZI3bEY4R7n5XoSGU8bcnJei4WIhBWNIrnLj17KREZGYJpsTzfFf3cixG8llbsab2O10AlsGA88EHOlFGEPPFhTopEQhPKnQqEajCo1rNK7Q0JgAIwaHeUWOSnJUE78kfk2mJZnW5KokVzXBC4OvP1VvnBu/0qfmtLiMsjoRNIlIouFucEcjskAajvfi60BpJD/u3vT67sAtFnwovFL0QblGN71fQSRwxu0XMUNKzTw31XODpKaYkbwTZIqkCC/RLZlZmSBO1NwUc5XDD5ZEMBbSbvtHBf23wiCu1IqHNnPdRnXfW8P/ebNMxwdzQ5M00yTBm4PijEEt4HpIYUQlwZqtrEBY2knAEC+QRFjbUe7YJnj3r/xQXOwPPHfgne33Dw/KdmyBd+A92AMe+AwOwRCMwATg9pv2l7bfPnbeOl+db85wk9pulTWvQWM5Z38AbnkphA==</latexit>

“empirical risk minimizer”<latexit sha1_base64="Lh7SF6DJ2t8iQyBnXkS/mkKeOmY=">AAADr3icbZLfb9MwEMfdhB+jUOjgkRdrFRJCqEr2sj1uKpP6gKZSrWunpmsd11mt2nFkO0Cx8ufxD/DGf4OTJoFsnOTcN5+7y9kXhwmjSnve75bjPnr85OnBs/bzF52Xr7qHr6+VSCUmEyyYkLMQKcJoTCaaakZmiSSIh4xMw+0gj0+/EqmoiK/0LiELju5iGlGMtEXLw9bPTieIyTe9IUISbhIpksyM7FMomqdkjTAjnKPMfC5cI1L6zFyVohHFQgrGkNxlZlDLRsaaRDTedzSf/upGjl1IbjMz3vtOux3YMhj4JuBIb8II+ragQBclCuFFhUY1GlVoXKNxhYbGBBgxOMwqcl6S85oMSjKoyawks5rclOSmJnhj8O2H6o1zM6j0pbksDqOsjgWN1yTWMNDkuy5+sAlZSjKzWhGeUEnzr0qqtpDbIXH6g8ijbNnteX2vMPhQ+KXogdJGy+6vYC1wym0nzJBSc99L9MIgqSlmJGsHqSIJwlt0R+ZWxogTtTDFdjL4zpI1jIS0y+60oP9WGMSV2vHQZubjVfdjOfxfbJ7q6HRhaJykmsR43yhKGdQC5pcXrqkkWLOdFQhLe0MwxBskEdb2irftEPz7R34oro/7vtf3vxz3zk7LcRyAt+AIvAc+OAFnYAhGYAKw89EZO3MncH136t66q32q0ypr3oCGufQPfi81ig==</latexit><latexit sha1_base64="Lh7SF6DJ2t8iQyBnXkS/mkKeOmY=">AAADr3icbZLfb9MwEMfdhB+jUOjgkRdrFRJCqEr2sj1uKpP6gKZSrWunpmsd11mt2nFkO0Cx8ufxD/DGf4OTJoFsnOTcN5+7y9kXhwmjSnve75bjPnr85OnBs/bzF52Xr7qHr6+VSCUmEyyYkLMQKcJoTCaaakZmiSSIh4xMw+0gj0+/EqmoiK/0LiELju5iGlGMtEXLw9bPTieIyTe9IUISbhIpksyM7FMomqdkjTAjnKPMfC5cI1L6zFyVohHFQgrGkNxlZlDLRsaaRDTedzSf/upGjl1IbjMz3vtOux3YMhj4JuBIb8II+ragQBclCuFFhUY1GlVoXKNxhYbGBBgxOMwqcl6S85oMSjKoyawks5rclOSmJnhj8O2H6o1zM6j0pbksDqOsjgWN1yTWMNDkuy5+sAlZSjKzWhGeUEnzr0qqtpDbIXH6g8ijbNnteX2vMPhQ+KXogdJGy+6vYC1wym0nzJBSc99L9MIgqSlmJGsHqSIJwlt0R+ZWxogTtTDFdjL4zpI1jIS0y+60oP9WGMSV2vHQZubjVfdjOfxfbJ7q6HRhaJykmsR43yhKGdQC5pcXrqkkWLOdFQhLe0MwxBskEdb2irftEPz7R34oro/7vtf3vxz3zk7LcRyAt+AIvAc+OAFnYAhGYAKw89EZO3MncH136t66q32q0ypr3oCGufQPfi81ig==</latexit><latexit sha1_base64="Lh7SF6DJ2t8iQyBnXkS/mkKeOmY=">AAADr3icbZLfb9MwEMfdhB+jUOjgkRdrFRJCqEr2sj1uKpP6gKZSrWunpmsd11mt2nFkO0Cx8ufxD/DGf4OTJoFsnOTcN5+7y9kXhwmjSnve75bjPnr85OnBs/bzF52Xr7qHr6+VSCUmEyyYkLMQKcJoTCaaakZmiSSIh4xMw+0gj0+/EqmoiK/0LiELju5iGlGMtEXLw9bPTieIyTe9IUISbhIpksyM7FMomqdkjTAjnKPMfC5cI1L6zFyVohHFQgrGkNxlZlDLRsaaRDTedzSf/upGjl1IbjMz3vtOux3YMhj4JuBIb8II+ragQBclCuFFhUY1GlVoXKNxhYbGBBgxOMwqcl6S85oMSjKoyawks5rclOSmJnhj8O2H6o1zM6j0pbksDqOsjgWN1yTWMNDkuy5+sAlZSjKzWhGeUEnzr0qqtpDbIXH6g8ijbNnteX2vMPhQ+KXogdJGy+6vYC1wym0nzJBSc99L9MIgqSlmJGsHqSIJwlt0R+ZWxogTtTDFdjL4zpI1jIS0y+60oP9WGMSV2vHQZubjVfdjOfxfbJ7q6HRhaJykmsR43yhKGdQC5pcXrqkkWLOdFQhLe0MwxBskEdb2irftEPz7R34oro/7vtf3vxz3zk7LcRyAt+AIvAc+OAFnYAhGYAKw89EZO3MncH136t66q32q0ypr3oCGufQPfi81ig==</latexit><latexit sha1_base64="Lh7SF6DJ2t8iQyBnXkS/mkKeOmY=">AAADr3icbZLfb9MwEMfdhB+jUOjgkRdrFRJCqEr2sj1uKpP6gKZSrWunpmsd11mt2nFkO0Cx8ufxD/DGf4OTJoFsnOTcN5+7y9kXhwmjSnve75bjPnr85OnBs/bzF52Xr7qHr6+VSCUmEyyYkLMQKcJoTCaaakZmiSSIh4xMw+0gj0+/EqmoiK/0LiELju5iGlGMtEXLw9bPTieIyTe9IUISbhIpksyM7FMomqdkjTAjnKPMfC5cI1L6zFyVohHFQgrGkNxlZlDLRsaaRDTedzSf/upGjl1IbjMz3vtOux3YMhj4JuBIb8II+ragQBclCuFFhUY1GlVoXKNxhYbGBBgxOMwqcl6S85oMSjKoyawks5rclOSmJnhj8O2H6o1zM6j0pbksDqOsjgWN1yTWMNDkuy5+sAlZSjKzWhGeUEnzr0qqtpDbIXH6g8ijbNnteX2vMPhQ+KXogdJGy+6vYC1wym0nzJBSc99L9MIgqSlmJGsHqSIJwlt0R+ZWxogTtTDFdjL4zpI1jIS0y+60oP9WGMSV2vHQZubjVfdjOfxfbJ7q6HRhaJykmsR43yhKGdQC5pcXrqkkWLOdFQhLe0MwxBskEdb2irftEPz7R34oro/7vtf3vxz3zk7LcRyAt+AIvAc+OAFnYAhGYAKw89EZO3MncH136t66q32q0ypr3oCGufQPfi81ig==</latexit>

true risk minimizer<latexit sha1_base64="bpumDvr+zEjXz9/NWhJUJNc4pMA=">AAADp3icbZLfb9MwEMfThB+jUNbBIy8WBQl4qJq9sMdNZVKF0Cil7To1pXLcy2rVjiPbAYqVP41/gjf+G9w0CWTDknNff+5OZ18uTBhVutf73XC9O3fv3T940Hz4qPX4sH30ZKpEKglMiGBCzkKsgNEYJppqBrNEAuYhg8tw09/5L7+CVFTEY71NYMHxdUwjSrC2aHnU+NlqBTF802sQErhJpEgyM7RfoeguJKu5GXCOM/MhNzVPYTMzLkTNS4QUjGG5zUy/krWIFUQ03lc07/7qWozdWG4yM9rbVrMZ2DQU+CbgWK/DCPk2IUfnBQrReYmGFRqWaFShUYkGxgQEMzTISnJWkLOK9AvSr8isILOKXBXkqiJkbciXN+WJc9Mv9YW5yB+jrI4FjVcQaxRo+K7zH2xClkJmtEwBSao2iNvucPoDZLZsd3rdXr7QbeEXouMUa7hs/wpWgqTcViAMKzX3e4leGCw1JQyyZpAqSDDZ4GuYWxljDmph8mtk6KUlKxQJabe9YU7/zTCYK7XloY3ctVXd9O3g/3zzVEcnC0PjJNUQk32hKGVIC7QbWrSiEohmWyswkXYyCCJrLDHRdrSbtgn+zSffFtPjrt/r+p+OO6cnRTsOnGfOc+eV4ztvnVNn4AydiUPcF+5797M79l57H72pN9uHuo0i56lTWx7+A6+IMmI=</latexit><latexit sha1_base64="bpumDvr+zEjXz9/NWhJUJNc4pMA=">AAADp3icbZLfb9MwEMfThB+jUNbBIy8WBQl4qJq9sMdNZVKF0Cil7To1pXLcy2rVjiPbAYqVP41/gjf+G9w0CWTDknNff+5OZ18uTBhVutf73XC9O3fv3T940Hz4qPX4sH30ZKpEKglMiGBCzkKsgNEYJppqBrNEAuYhg8tw09/5L7+CVFTEY71NYMHxdUwjSrC2aHnU+NlqBTF802sQErhJpEgyM7RfoeguJKu5GXCOM/MhNzVPYTMzLkTNS4QUjGG5zUy/krWIFUQ03lc07/7qWozdWG4yM9rbVrMZ2DQU+CbgWK/DCPk2IUfnBQrReYmGFRqWaFShUYkGxgQEMzTISnJWkLOK9AvSr8isILOKXBXkqiJkbciXN+WJc9Mv9YW5yB+jrI4FjVcQaxRo+K7zH2xClkJmtEwBSao2iNvucPoDZLZsd3rdXr7QbeEXouMUa7hs/wpWgqTcViAMKzX3e4leGCw1JQyyZpAqSDDZ4GuYWxljDmph8mtk6KUlKxQJabe9YU7/zTCYK7XloY3ctVXd9O3g/3zzVEcnC0PjJNUQk32hKGVIC7QbWrSiEohmWyswkXYyCCJrLDHRdrSbtgn+zSffFtPjrt/r+p+OO6cnRTsOnGfOc+eV4ztvnVNn4AydiUPcF+5797M79l57H72pN9uHuo0i56lTWx7+A6+IMmI=</latexit><latexit sha1_base64="bpumDvr+zEjXz9/NWhJUJNc4pMA=">AAADp3icbZLfb9MwEMfThB+jUNbBIy8WBQl4qJq9sMdNZVKF0Cil7To1pXLcy2rVjiPbAYqVP41/gjf+G9w0CWTDknNff+5OZ18uTBhVutf73XC9O3fv3T940Hz4qPX4sH30ZKpEKglMiGBCzkKsgNEYJppqBrNEAuYhg8tw09/5L7+CVFTEY71NYMHxdUwjSrC2aHnU+NlqBTF802sQErhJpEgyM7RfoeguJKu5GXCOM/MhNzVPYTMzLkTNS4QUjGG5zUy/krWIFUQ03lc07/7qWozdWG4yM9rbVrMZ2DQU+CbgWK/DCPk2IUfnBQrReYmGFRqWaFShUYkGxgQEMzTISnJWkLOK9AvSr8isILOKXBXkqiJkbciXN+WJc9Mv9YW5yB+jrI4FjVcQaxRo+K7zH2xClkJmtEwBSao2iNvucPoDZLZsd3rdXr7QbeEXouMUa7hs/wpWgqTcViAMKzX3e4leGCw1JQyyZpAqSDDZ4GuYWxljDmph8mtk6KUlKxQJabe9YU7/zTCYK7XloY3ctVXd9O3g/3zzVEcnC0PjJNUQk32hKGVIC7QbWrSiEohmWyswkXYyCCJrLDHRdrSbtgn+zSffFtPjrt/r+p+OO6cnRTsOnGfOc+eV4ztvnVNn4AydiUPcF+5797M79l57H72pN9uHuo0i56lTWx7+A6+IMmI=</latexit><latexit sha1_base64="bpumDvr+zEjXz9/NWhJUJNc4pMA=">AAADp3icbZLfb9MwEMfThB+jUNbBIy8WBQl4qJq9sMdNZVKF0Cil7To1pXLcy2rVjiPbAYqVP41/gjf+G9w0CWTDknNff+5OZ18uTBhVutf73XC9O3fv3T940Hz4qPX4sH30ZKpEKglMiGBCzkKsgNEYJppqBrNEAuYhg8tw09/5L7+CVFTEY71NYMHxdUwjSrC2aHnU+NlqBTF802sQErhJpEgyM7RfoeguJKu5GXCOM/MhNzVPYTMzLkTNS4QUjGG5zUy/krWIFUQ03lc07/7qWozdWG4yM9rbVrMZ2DQU+CbgWK/DCPk2IUfnBQrReYmGFRqWaFShUYkGxgQEMzTISnJWkLOK9AvSr8isILOKXBXkqiJkbciXN+WJc9Mv9YW5yB+jrI4FjVcQaxRo+K7zH2xClkJmtEwBSao2iNvucPoDZLZsd3rdXr7QbeEXouMUa7hs/wpWgqTcViAMKzX3e4leGCw1JQyyZpAqSDDZ4GuYWxljDmph8mtk6KUlKxQJabe9YU7/zTCYK7XloY3ctVXd9O3g/3zzVEcnC0PjJNUQk32hKGVIC7QbWrSiEohmWyswkXYyCCJrLDHRdrSbtgn+zSffFtPjrt/r+p+OO6cnRTsOnGfOc+eV4ztvnVNn4AydiUPcF+5797M79l57H72pN9uHuo0i56lTWx7+A6+IMmI=</latexit>

f⇤ = argminf2F

R(f)

bf = argminf2F

bR(f)<latexit sha1_base64="4tCcmg1tPzZut6bYSupsgXZXOec=">AAAEwHichZPfT9swEMcD7TaWrRtsj3uxhjYxHlDDHjZNQoIVJh4QKh2FItxVjuM0Fv4RHGeQWfkn9zBp/82c1g0UkObI8Vefu/PlznGYMprpdvvvwmKj+ejxk6Wn/rPnrRcvl1denWQyV5j0sWRSDUKUEUYF6WuqGRmkiiAeMnIaXnQq++lPojIqxbEuUjLkaCxoTDHSFo1WFv+0WlCQK50QqQg3qZJpabr2LTNauZRzZkY4R6U5mCxzFreW5tiJOSuWSjKGVFGaTi3nPCISUzHNaHZv9JyPnUhdlKY3XVu+D20YgIGBHOkkjEFgAyZoz6EQ7M1Qt0bd0gUe1eho5tWrUW+G9o2BGDGwX87IjiM7Nek40qnJwJFBTc4cOSsf2Nl9Ud+Rfk0OHDmoSZgbGEoWZQW3i8lrA04M/rE+25xz05npQ3No21X1Ekt7dCKye34vb28DdwnTqE5xPZfiukqRFGlCREoEYrrYCtp22AMQkoqICA1gFgMfhmRMhSGXAimFivXSj3+sg/dgy06I1BhyKkYmhlSAaVXfyhL01uIPAEJgH+jDKxqRBGkQ/y+s9qzifUhsUTd5R8ur7Y32ZID7InBi1XOjO1r+DSOJc25LwQxl2XnQTvXQIKUpZsSWn2ckRfgCjcm5lQJxkg3N5AKW4J0lEYilstO2YkJvRxjEs6qV1rP6ubK7tgo+ZDvPdfx5aKhIc00EniaKcwa0BNVtBhFVBGtWWIGwslcGA5wghbC2d963TQjulnxfnGxuBB83gqPN1e0vrh1L3hvvrbfmBd4nb9vb97pe38ONrQZusAZvfm0mTdm8nLouLriY197caP76B+BplO8=</latexit>

R( bf) bR( bf) + 6

rd log(n/�)

n<latexit sha1_base64="ypzaxYPsuuEjjz1gpDBpuu5Lr7c=">AAAEpnicbZPdTtswGIYD7TbWrRtshzuxVk0qTGINkzZOJoEKEpMQKoVCEe7AcZzGwrGD7QwqK3e2q9jZ7mZO6wYCWHL86vl+8yUOUkaV7nT+LSzW6s+ev1h62Xj1uvnm7fLKuxMlMonJAAsm5DBAijDKyUBTzcgwlQQlASOnwVW3sJ/+JlJRwY/1JCWjBI05jShG2qKLlcU/zSbk5EbHREiSmFSKNDc9+xSKFi55xcxIkqDc7E+PisWduTl2omLFQgrGkJzkplvKikdIIspnFc3Ona742I3kVW76s7PZaEAbBqBvYIJ0HETAtwFTtOtQAHbnqFeiXu4CD0t0OPfql6g/R3vGQIwY2MvnZNuR7ZJ0HemWZOjIsCRnjpzlT2R2HQ0cGZRk35H9kgSZgYFgoZok9jBZacCxwb/W5smTxHTn+sAc2HEVs8TCfjoe2pxH+f00cIcwjcoSt5USt0WJeJLGhKeEI6YnP/yOXfYDcEF5SLgGUEWgAQMyptyQa46kRJO1vNFvwxsakhhpEK0CyMg1KEHV9hl8g+paagMjibAJIRPjNv9iG7KNreaGF00Q2/pd9ovlVme9M13gsfCdaHlu9S6W/8JQ4CyxDWOGlDr3O6keGSQ1xYzY/JkiKcJXaEzOreQoIWpkptcsB58sCUEkpN32haf0foRBiSoGZj2LX0g9tBXwKdt5pqPNkaE8zTTheFYoyhjQAhR3FoRUEqzZxAqEpb0YGOAY2SFpe7Mbdgj+w1d+LE421v2v6/7hRmtr041jyfvgffTanu9997a8Pa/nDTxca9V+1vq1o3q7flAf1E9nrosLLua9V1n1y/+XMY5Y</latexit>

R(f⇤) � bR(f⇤)� 6

rd log(n/�)

n<latexit sha1_base64="uv9aAuHxfnOlEhEZ/ObRFHpmsCo=">AAAEmHicbZNvT9QwHMcHdyqenoLGJ/qkkZgAiecNEyUxJpCDiAkhx8nBEXqQruvuGrp2tJ2wNHtLvhif+W7s7nqDAU26fvP5/d1va5AwqnS7/W9uvlZ/9PjJwtPGs+fNFy8Xl14dKZFKTPpYMCEHAVKEUU76mmpGBokkKA4YOQ4uOoX9+DeRigp+qLOEDGM04jSiGGmLzpfm/zSbkJMrPSZCktgkUiS56dqnULRwyStmRuIY5WZvclQs7szNoRMVKxZSMIZklptOKSseIYkon1Y02ze64mM3khe56U3PZqMBbRiAvoEx0uMgAr4NmKAdhwKwM0PdEnVzF3hQooOZV69EvRnaNQZixMBuPiNbjmyVpONIpyQDRwYlOXHkJH8gs+uo70i/JHuO7JUkSA0MBAtVFtvDpKUBjw0+W5slj2PTmel9s2/HVcwSC/vpeGhz/spvp4HbhGlUlriulLguSoyzZEx4QjhiOvvut+2yH4ALykPCNYAqAg0YkBHlhlxyJCXK1vJGbyU6W1sFcEQuAbyiIRkjDRz8CL5AdSm1gZFE2ISQidEK/2RbsK2s5oYXZYlt9ibf+eJyu9WeLHBf+E4se251zxf/wlDgNLYtYoaUOvXbiR4aJDXFjNj8qSIJwhdoRE6t5CgmamgmFysHHywJQSSk3fYVJ/R2hEGxKkZkPYufRt21FfAh22mqo42hoTxJNeF4WihKGdACFLcUhFQSrFlmBcLSXgUM8BjZIWl7lxt2CP7dV74vjtZb/ueWf7C+vLnhxrHgvfPeeyue7331Nr1dr+v1PVx7U/tW267t1N/WN+s/6j+nrvNzLua1V1n13n+KH4gV</latexit>

12

rd log(n/�)

n<latexit sha1_base64="GnDK2yP8/w+2bbKd3jgK5tWNSdk=">AAAElnicbZNNT9swGMcD7TbWrRuMy6RdrKEhxoE17DB22AQraBwQKoxCEekqx3nSRjh2sJ2NyvI32qfZbd9mTusGAlhy/NfvebOf2GFGE6larX9z87X6o8dPFp42nj1vvni5uPTqVPJcEOgSTrnohVgCTRh0VaIo9DIBOA0pnIWX7cJ+9guETDg7UeMM+ikesiROCFYWDZbm/zSbAYPfagRcQKozwTOjO/bLZVK4mIqZQppiow8mS8XiVqNPnKhYCRecUizGRrdLWfGIIE7YtKLevdEVHzuxuDT6eLo2G43AhqHA10GK1SiMkW8DJmjPoRDtzVCnRB3jAo9KdDTzOi7R8Qztax0QTNG+mZEdR3ZK0nakXZKeI72SnDtybh7I7HbUdaRbkgNHDkoS5joIOY3kOLWLzksDGWnyc32WPE11e6YP9aFtV9FLwu2vY5HN+cPcThPsAlW4LHFdKXFdlBiNsxGwDBimavzFb9lhfwDjCYuAKRRIG6fgWk0upQ5pDkY3ghCGCdNwxbAQeLxuGqsooHCFVpG/GcgroXQQC0x0FFA+XGMfbH27j/dGs6Im2J3exJrB4kprozUZ6L7wnVjx3OgMFv8GESd5ajdIKJbywm9lqq+xUAmhYAvkEjJMLvEQLqxkOAXZ15MTGPTOkgjFXNhpDzihtyM0TmXRIOtZXBl511bAh2wXuYq3+jphWa6AkWmhOKdIcVS8URQlAoiiYyswEfYhEERG2HZJ2ZfcsE3w7x75vjjd3PA/bvhHmyvbW64dC94b76235vneJ2/b2/c6XtcjteXa59q3Wrv+uv61vlf/PnWdn3Mxy15l1Dv/ATG2iYY=</latexit>

Goal: select hypothesis with true error rate within ✏ > 0 of minf2F R(f)<latexit sha1_base64="aMQNp2TlCGApiYyduI5Id4Vt+xk=">AAAD4XicbZNNb9MwGMfThpdRKHRw5GLRIg0OVTMOTBzQpjLoAU2lWrdOTVc5rrNY9UtkO0yVlRsXDiDElW/FjU/CFTdNAtmwlPjv3/95ZPuxHcSUKN3r/arV3Rs3b93eutO4e695/0Fr++GJEolEeIwEFXISQIUp4XisiaZ4EksMWUDxabDsr/3Tj1gqIvixXsV4xuAFJyFBUFs03679bjZ9ji91hIXEzMRSxKkZ2r9QZB2SVmyKGYOpeZ91FSfvU3Oci4qLhBSUQrlKTb+UlYgFDgnfzGje/NWVGPtBuUzNaNM3Gw3fpgHfMz6DOgpC4NmEDB3mKACHBRqWaFigUYlGBRoY4yNIwSAtyEFODkrSz0m/JJOcTEpylpOzkqDIoPPnxYgx0y/0kTnKNqOs5oLwBeYavBOQvgL2ZDHSIFrFwlZBEQUuiY6AlgkGWEohgYQaZ5Bw0PFxrAgV/HWvA0Rox4zwuQl9622W8zZNwWgnfNaZt9q9bi9r4LrwctF28jact376C4ESZteGKFRq6vViPTNQaoIoTht+onAM0RJe4KmVHDKsZia7oSl4askChHa5obB7y+i/GQYypVYssJHrA1FXvTX8nzdNdLg3M4THicYcbSYKEwq0AOvrDhZE2vrRlRUQSXunEEARlBBp+ygatgje1S1fFye7Xe9F1/uw297fy8ux5Tx2njg7jue8dPadgTN0xg6qn9c/1b/Uv7rI/ex+c79vQuu1POeRU2nujz8GVkbi</latexit>

su�cient numberof training examples:

<latexit sha1_base64="1LtGrTG8rs9j7ZksoQ9EjmLxiLY=">AAADvXicbZJdb9MwFIbTlo9RKHRwyY1FhYS4qJLdMO2GTWVSL9BUqnXr1JThuCetqT8i22GrrPxJuOLf4KZJIBuWHL953nP8cewoYVQb3//daLYePHz0eO9J++mzzvMX3f2XF1qmisCESCbVNMIaGBUwMdQwmCYKMI8YXEbrwda//AFKUynOzSaBOcdLQWNKsHHoer/xq9MJBdyYFUgF3CZKJpkdua/UdBuS1WwGnOPMfs6HmlOMmT0vRM0lUknGsNpkdlDJWsQCYip2K9pPf3UtxnWs1pkd78ZOux26NBQGNuTYrKIYBS4hR6cFitBpiUYVGpVoXKFxiYbWhgQzNMxKclKQk4oMCjKoyLQg04pcFeSqImRlydf35R/ndlDqM3uWH0Y7LSQVCxAGhQZuTX7B7tyLzOo0dtdGt5ZIeQQKhSGSMTIKu2KJJYJbzBMG+ii77vb8vp83dF8Eheh5RRtdd3+GC0lS7iYnDGs9C/zEzC1WhhIGWTtMNSSYrPESZk4KzEHPbb65DL11ZIFiqVx3m8vpvxkWc603PHKR22Lru94W/s+bpSY+nFsqktSAILuF4pQhI9H2KaMFVUAM2ziBiXLvhSCywgoT4x582xUhuHvk++LioB/4/eDLQe/4sCjHnvfae+O98wLvg3fsDb2RN/FI86j5rUmb31sfW9BiLbELbTaKnFderbVu/gCy6jsf</latexit><latexit sha1_base64="1LtGrTG8rs9j7ZksoQ9EjmLxiLY=">AAADvXicbZJdb9MwFIbTlo9RKHRwyY1FhYS4qJLdMO2GTWVSL9BUqnXr1JThuCetqT8i22GrrPxJuOLf4KZJIBuWHL953nP8cewoYVQb3//daLYePHz0eO9J++mzzvMX3f2XF1qmisCESCbVNMIaGBUwMdQwmCYKMI8YXEbrwda//AFKUynOzSaBOcdLQWNKsHHoer/xq9MJBdyYFUgF3CZKJpkdua/UdBuS1WwGnOPMfs6HmlOMmT0vRM0lUknGsNpkdlDJWsQCYip2K9pPf3UtxnWs1pkd78ZOux26NBQGNuTYrKIYBS4hR6cFitBpiUYVGpVoXKFxiYbWhgQzNMxKclKQk4oMCjKoyLQg04pcFeSqImRlydf35R/ndlDqM3uWH0Y7LSQVCxAGhQZuTX7B7tyLzOo0dtdGt5ZIeQQKhSGSMTIKu2KJJYJbzBMG+ii77vb8vp83dF8Eheh5RRtdd3+GC0lS7iYnDGs9C/zEzC1WhhIGWTtMNSSYrPESZk4KzEHPbb65DL11ZIFiqVx3m8vpvxkWc603PHKR22Lru94W/s+bpSY+nFsqktSAILuF4pQhI9H2KaMFVUAM2ziBiXLvhSCywgoT4x582xUhuHvk++LioB/4/eDLQe/4sCjHnvfae+O98wLvg3fsDb2RN/FI86j5rUmb31sfW9BiLbELbTaKnFderbVu/gCy6jsf</latexit><latexit sha1_base64="1LtGrTG8rs9j7ZksoQ9EjmLxiLY=">AAADvXicbZJdb9MwFIbTlo9RKHRwyY1FhYS4qJLdMO2GTWVSL9BUqnXr1JThuCetqT8i22GrrPxJuOLf4KZJIBuWHL953nP8cewoYVQb3//daLYePHz0eO9J++mzzvMX3f2XF1qmisCESCbVNMIaGBUwMdQwmCYKMI8YXEbrwda//AFKUynOzSaBOcdLQWNKsHHoer/xq9MJBdyYFUgF3CZKJpkdua/UdBuS1WwGnOPMfs6HmlOMmT0vRM0lUknGsNpkdlDJWsQCYip2K9pPf3UtxnWs1pkd78ZOux26NBQGNuTYrKIYBS4hR6cFitBpiUYVGpVoXKFxiYbWhgQzNMxKclKQk4oMCjKoyLQg04pcFeSqImRlydf35R/ndlDqM3uWH0Y7LSQVCxAGhQZuTX7B7tyLzOo0dtdGt5ZIeQQKhSGSMTIKu2KJJYJbzBMG+ii77vb8vp83dF8Eheh5RRtdd3+GC0lS7iYnDGs9C/zEzC1WhhIGWTtMNSSYrPESZk4KzEHPbb65DL11ZIFiqVx3m8vpvxkWc603PHKR22Lru94W/s+bpSY+nFsqktSAILuF4pQhI9H2KaMFVUAM2ziBiXLvhSCywgoT4x582xUhuHvk++LioB/4/eDLQe/4sCjHnvfae+O98wLvg3fsDb2RN/FI86j5rUmb31sfW9BiLbELbTaKnFderbVu/gCy6jsf</latexit><latexit sha1_base64="1LtGrTG8rs9j7ZksoQ9EjmLxiLY=">AAADvXicbZJdb9MwFIbTlo9RKHRwyY1FhYS4qJLdMO2GTWVSL9BUqnXr1JThuCetqT8i22GrrPxJuOLf4KZJIBuWHL953nP8cewoYVQb3//daLYePHz0eO9J++mzzvMX3f2XF1qmisCESCbVNMIaGBUwMdQwmCYKMI8YXEbrwda//AFKUynOzSaBOcdLQWNKsHHoer/xq9MJBdyYFUgF3CZKJpkdua/UdBuS1WwGnOPMfs6HmlOMmT0vRM0lUknGsNpkdlDJWsQCYip2K9pPf3UtxnWs1pkd78ZOux26NBQGNuTYrKIYBS4hR6cFitBpiUYVGpVoXKFxiYbWhgQzNMxKclKQk4oMCjKoyLQg04pcFeSqImRlydf35R/ndlDqM3uWH0Y7LSQVCxAGhQZuTX7B7tyLzOo0dtdGt5ZIeQQKhSGSMTIKu2KJJYJbzBMG+ii77vb8vp83dF8Eheh5RRtdd3+GC0lS7iYnDGs9C/zEzC1WhhIGWTtMNSSYrPESZk4KzEHPbb65DL11ZIFiqVx3m8vpvxkWc603PHKR22Lru94W/s+bpSY+nFsqktSAILuF4pQhI9H2KaMFVUAM2ziBiXLvhSCywgoT4x582xUhuHvk++LioB/4/eDLQe/4sCjHnvfae+O98wLvg3fsDb2RN/FI86j5rUmb31sfW9BiLbELbTaKnFderbVu/gCy6jsf</latexit>

12

rd log(n/�)

n ✏

<latexit sha1_base64="k6b3p7ASnxTVHUysyUh++Y3RPyw=">AAAEoHicbZNPT9swFMAD7TaWrRtsx2lSBJrEOLCGHcZlEqgg9dCxllEowx1ynJc2wrGD7WxUlr/WPshu+zZzWjdQmKXkPf3eXz/bUU5TqZrNv0vLtfqjx09WnvrPnjdevFxde3UqeSEI9AmnXAwiLIGmDPoqVRQGuQCcRRTOoqtWaT/7CUKmnJ2oSQ7DDI9YmqQEK4su15Z/NxqIwS81Bi4g07ngudFd++cyLV3MgplClmGjO1OxYHHS6BOnLFgJF5xSLCZGtyp1wSOGJGWzivrgVl/wsR8WV0Yfz2TD95ENC1CoUYbVOEqC0AZM0aFDUXA4R90KdY0L7FWoN/c6rtDxHLW1RgTToG3mZN+R/Yq0HGlVZODIoCLnjpyb/2R2HfUd6Vek40inIlGhUcRpLCeZFbqoDGSsyY+tefIs0625fqSP7LjKWRJuj47FNuc3czcNOgCqcFXiZqHETVliPMnHwHJgmKrJ57Bplz0AxlMWA1MBkjZOwY2aXkodUUzsUfkoglHKNFwzLASebBk/3EHyWiiNEoGJjhHlo032wda19d8bzYwJUIAoXJcCcplSznwEtufbJOZydaO53Zyu4KESOmXDc6t7ufoHxZwUmW2VUCzlRdjM1VBjoVJCwW6vkJDblvEILqzKcAZyqKd7McE7S+Ig4cJ+dqtTejdC40yWo7Ke5eWR920l/J/tolDJ7lCnLC8UMDIrlBQ0UDwoX2sQpwKIohOrYCLskyABGWM7N2XftG+HEN7f8kPldGc7/Lgd9nY29nbdOFa8N966t+mF3idvz2t7Xa/vkdrbWqvWqX2pr9fb9a/13sx1ecnFvPYWVv37P18GjiE=</latexit>

n = eO⇣d log(1/�)

✏2

⌘

<latexit sha1_base64="SJR9nsso93nJDyhS0fq2vYlYG+g=">AAAEq3icbZPdTtswFIAD7TbWrRtsl7uxhiYVpLGGXYwbJFhB4gKx8lMowwU5zmlr4diZ4wwqKw+3V9jd3mYnbRooYCk5R9/59bEdxFIkttn8NzdfqT57/mLhZe3V6/qbt4tL704TnRoOHa6lNt2AJSCFgo4VVkI3NsCiQMJZcN3K7We/wSRCqxM7iqEXsYESfcGZRXS1NP+nXqcKbuwQtIHIxUbHmWvjXycid8lmzBKiiGVufyxmLIXM3EmhzFi5NlpKZkaZa5XqjEcIfaEmFd3OnT7jgx8z15k7msh6rUYxjFDf0YjZYdAnPgaM0W6BArI7Re0StbMi8LBEh1OvoxIdTdGec5QzSfayKdkuyHZJWgVplaRbkG5Jzgtynj2RueioU5BOSfYLsl+SIHU00DJMRhEKl5YGPnT8cnWaPIpca6ofuAMcVz5LrvHoVIg5j7P7aegOSMvKErczJW7zEsNRPAQVg2LSjjb9Ji48AKWFCkFZQhOMs3Brx5fSBZJxPKoaDWAglINfihnDRqtZTRGySQi9ESFYIUMgP+h3MWjQvmHchYRKPWj4X7AP7GcFe4Q4EVKry/Us91upUcD27/JlV4vLzbXmeJHHil8oy16x2leLf2moeRph11yyJLnwm7HtOWas4BJwp2kCMXbPBnCBqmIRJD033lZGPiEJSV8b/HDXY3o/wrEoyaeGnvk9Sh7acviU7SK1/Y2eEypOLSg+KdRPJbGa5A+XhMIAt3KECuMGXwcnfMhwZhafdw2H4D/c8mPldH3N/7rmH64vb20U41jwPngfvYbne9+8LW/Pa3sdj1calYPKWaVb/Vw9rv6s0onr/FwR896bWVX4D8cIkRs=</latexit>

bf minimizes empirical risk:

bR( bf) bR(f⇤)<latexit sha1_base64="JFfuybFBfbjU2tcBHdGGn2WlglU=">AAACXnicbVFBT9swGHUCY5DC6MYFaRdrDRLboUrawyZOCC47AqKA1HSV43yhn2o7me0MdVH/5G7TLvspc0ukjsInWX5673v25+e0FGhsFP32/I3NV1uvt3eC1u7em/3223c3pqg0hwEvRKHvUmZAoIKBRSvgrtTAZCrgNp2eL/TbH6ANFurazkoYSXavMEfOrKPG7SpRBaoMlKWJyWmYPGAGE2ZpHlKJCiX+BENBlqidR1CNZnpCw1Xf1fHK8pEmNBHwfbGt9PzbJydMTMk41H1U8zActztRN1oWfQ7iBnRIUxfj9q8kK3gl3ZxcMGOGcVTaUc20RS5gHiSVAXf+lN3D0EHFJJhRvYxnTo8ck9G80G65dy7Z/x01k8bMZOo6JbMTs64tyJe0YWXzL6MaVVlZUPzxorwS1BZ0kTXNUAO3YuYA4xrdrJRPmGbcuh8JXAjx+pOfg5teN+5348te5/SsiWObvCcfyDGJyWdySr6SCzIgnPzxPC/wWt5ff8vf8/cfW32v8RyQJ+Uf/gM+bbKb</latexit>

error

hypotheses (ordered according to empirical risks)

· · ·

1 2 3 k-1 k

Empirical Risks and Confidence Intervals


· · ·

1 2 3 k-1 k

Empirical Risks and Confidence Intervals

more training data ) smaller confidence intervals<latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit><latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit><latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit><latexit sha1_base64="4KQPPiJfbzQ6Js5ya2E0VMUAD88=">AAAD0HicbZLPb9MwFMfThB+jUOjgyMWiQ0IcpmSXIU6byqQeYCrVunVqSuU4TmvVPyLb2VaZCHHlz+PGP8DfgZsmgWw8yXnffN57sd+Lo5QSpX3/V8v17t1/8HDnUfvxk87TZ93d5+dKZBLhMRJUyEkEFaaE47EmmuJJKjFkEcUX0aq/iV9cYamI4Gd6neIZgwtOEoKgtmi+2/rd6YQcX+slFhIzk0qR5mZon0KRTUreCFPMGMzNx8I1IqXPzVkpGlEkpKAUynVu+rVsZMQ4IXy7o/nwVzdy7IJylZvR1nfa7dCWgTAwIYN6GSUgsAUFOilRBE4qNKzRsEKjGo0qNDAmRJCCQV6R45Ic16Rfkn5NJiWZ1OSyJJc1QUuDvryt3hgz/UqfmtOiGS4IjzHXIFSWanyjix9s+45zw+wIgJbQToYvQAw1BHvhiCyWGkoprveAYpBSLAESPCH2MwgDwjWWV5CqfN7t+ft+YeCuCErRc0obzrs/w1igjNnjIAqVmgZ+qmcGSk0QxXk7zBROIVrBBZ5aySHDamaK8+bgtSUxSIS0y7ZT0H8rDGRKrVlkMzfzV7djG/i/2DTTybuZITzNtG1wu1GSUaAF2NxuEBOJkaZrKyCS9gohgJZQQmTnoNp2CMHtlu+K84P9wN8PPh/0jt6X49hxXjqvnDdO4Bw6R87AGTpjB7mfXOV+dXNv5N1437zv21S3Vda8cBrm/fgDv9BDEA==</latexit>


· · ·

1 2 3 k-1 k

bR(f3)<latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit>

ERM is Wasting Labeled Examples

hypotheses (ordered according to empirical risks)1 2 3 k-1 k

ERM is Wasting Labeled Examples

· · ·bR(f3)<latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit><latexit sha1_base64="DfD7G/1NGq2jDWAuNm5nZxdkLA0=">AAADpXicbZLfb9MwEMfdhh+jUNbBIy/WOsTgoUrGA4inTWVSH2Dqqrbr1JTKcZzVqh1HtsOorPxl/Be88d/gpkkgG5ac+/pzdzr7ckHCqNKu+7vRdB48fPR470nr6bP28/3OwYupEqnEZIIFE3IWIEUYjclEU83ILJEE8YCRq2Dd3/qvvhOpqIjHepOQBUc3MY0oRtqi5UHjZ7vtx+RWr4iQhJtEiiQzQ/sVim5DspqbEc5RZr7kpuYpbGbGhah5sZCCMSQ3melXshYRkojGu4rm819di7EbyXVmRjvbbrV8mwZ9z/gc6VUQQc8m5Oi8QAE8L9GwQsMSjSo0KtHAGB8jBgdZSc4KclaRfkH6FZkVZFaR64JcVwSvDP72rjxxbvqlvjAX+WNiQeOQxBr6ylJNfuj8B5uApSQzR/4tDckKaTg6jpbv3x5ly07X7bn5gveFV4guKNZw2fnlhwKn3NbADCk199xELwySmmJGspafKpIgvEY3ZG5ljDhRC5NfIoOvLQlhJKTd9o45/TfDIK7Uhgc2cttUdde3hf/zzVMdfVwYGiepJjHeFYpSBrWA25GFIZUEa7axAmFp5wJDvEISYW0Hu2Wb4N198n0xPel5bs+7POmefirasQdegUNwDDzwAZyCARiCCcDNw+agedkcOW+cr87Yme5Cm40i5yWoLWf5B5jYMAg=</latexit>

at this point we can safely removef3 from further consideration

<latexit sha1_base64="vmSDb2u8HBmAr9vmd20H28xA4gY=">AAADyHicdZLfb9MwEMezlh+jUOjgkZcTGxLioUoKhYqnTaXShNBUqnXr1JTKcZzVqn9EtrOpsvLCn8gbL/wtuG0SKIKTnPv6c3eyfbkoZVQb3/+xV6vfuXvv/v6DxsNHzcdPWgdPL7TMFCZjLJlUkwhpwqggY0MNI5NUEcQjRi6jZX8dv7whSlMpzs0qJTOOrgVNKEbGofnB3s9mMxTk1iyIVITbVMk0t0P3lZquU/KdMCOco9x+3ridSOFze16InSiWSjKG1Cq3/UruZMQkoWJ7ov34W+/kuIXUMrejrW82GqErgzCwIUdmESUQuIINGhQogkGJhhUalmhUoVGJTq0NMWJwmpfkpCAnFekXpF+RSUEmFbkqyFVF8MLir6/LHee2X+oze7Z5jJBUxEQYCHUCyIBZUA2pgwZuCWAkQKOEsBW4RsgbAmEIR8n8zREkSnJIMuUapQBLoWlM1PYPtw79tv+u+zbogd/u+kEv6DrR6Qa+34Gg7W/s0CtsOG99D2OJM+6ugRnSehr4qZlZpAzFjOSNMNMkRXiJrsnUSYE40TO7GcQcXjoSQyKVW+7SG/pnhUVc6xWPXOa67/rv2Br+KzbNTNKbWSrSzBCBtwclGQMjYT3VEFNFsHGdiSnCyo0OBrxACmHjZr/hmlC+FP4vLjrtwG8HXzqHxx+Kdux7z70X3isv8N57x96pN/TGHq4NasuaqWX1T/W0fltfbVNre0XNM2/H6t9+AQH0PdQ=</latexit><latexit sha1_base64="vmSDb2u8HBmAr9vmd20H28xA4gY=">AAADyHicdZLfb9MwEMezlh+jUOjgkZcTGxLioUoKhYqnTaXShNBUqnXr1JTKcZzVqn9EtrOpsvLCn8gbL/wtuG0SKIKTnPv6c3eyfbkoZVQb3/+xV6vfuXvv/v6DxsNHzcdPWgdPL7TMFCZjLJlUkwhpwqggY0MNI5NUEcQjRi6jZX8dv7whSlMpzs0qJTOOrgVNKEbGofnB3s9mMxTk1iyIVITbVMk0t0P3lZquU/KdMCOco9x+3ridSOFze16InSiWSjKG1Cq3/UruZMQkoWJ7ov34W+/kuIXUMrejrW82GqErgzCwIUdmESUQuIINGhQogkGJhhUalmhUoVGJTq0NMWJwmpfkpCAnFekXpF+RSUEmFbkqyFVF8MLir6/LHee2X+oze7Z5jJBUxEQYCHUCyIBZUA2pgwZuCWAkQKOEsBW4RsgbAmEIR8n8zREkSnJIMuUapQBLoWlM1PYPtw79tv+u+zbogd/u+kEv6DrR6Qa+34Gg7W/s0CtsOG99D2OJM+6ugRnSehr4qZlZpAzFjOSNMNMkRXiJrsnUSYE40TO7GcQcXjoSQyKVW+7SG/pnhUVc6xWPXOa67/rv2Br+KzbNTNKbWSrSzBCBtwclGQMjYT3VEFNFsHGdiSnCyo0OBrxACmHjZr/hmlC+FP4vLjrtwG8HXzqHxx+Kdux7z70X3isv8N57x96pN/TGHq4NasuaqWX1T/W0fltfbVNre0XNM2/H6t9+AQH0PdQ=</latexit><latexit sha1_base64="vmSDb2u8HBmAr9vmd20H28xA4gY=">AAADyHicdZLfb9MwEMezlh+jUOjgkZcTGxLioUoKhYqnTaXShNBUqnXr1JTKcZzVqn9EtrOpsvLCn8gbL/wtuG0SKIKTnPv6c3eyfbkoZVQb3/+xV6vfuXvv/v6DxsNHzcdPWgdPL7TMFCZjLJlUkwhpwqggY0MNI5NUEcQjRi6jZX8dv7whSlMpzs0qJTOOrgVNKEbGofnB3s9mMxTk1iyIVITbVMk0t0P3lZquU/KdMCOco9x+3ridSOFze16InSiWSjKG1Cq3/UruZMQkoWJ7ov34W+/kuIXUMrejrW82GqErgzCwIUdmESUQuIINGhQogkGJhhUalmhUoVGJTq0NMWJwmpfkpCAnFekXpF+RSUEmFbkqyFVF8MLir6/LHee2X+oze7Z5jJBUxEQYCHUCyIBZUA2pgwZuCWAkQKOEsBW4RsgbAmEIR8n8zREkSnJIMuUapQBLoWlM1PYPtw79tv+u+zbogd/u+kEv6DrR6Qa+34Gg7W/s0CtsOG99D2OJM+6ugRnSehr4qZlZpAzFjOSNMNMkRXiJrsnUSYE40TO7GcQcXjoSQyKVW+7SG/pnhUVc6xWPXOa67/rv2Br+KzbNTNKbWSrSzBCBtwclGQMjYT3VEFNFsHGdiSnCyo0OBrxACmHjZr/hmlC+FP4vLjrtwG8HXzqHxx+Kdux7z70X3isv8N57x96pN/TGHq4NasuaqWX1T/W0fltfbVNre0XNM2/H6t9+AQH0PdQ=</latexit><latexit sha1_base64="vmSDb2u8HBmAr9vmd20H28xA4gY=">AAADyHicdZLfb9MwEMezlh+jUOjgkZcTGxLioUoKhYqnTaXShNBUqnXr1JTKcZzVqn9EtrOpsvLCn8gbL/wtuG0SKIKTnPv6c3eyfbkoZVQb3/+xV6vfuXvv/v6DxsNHzcdPWgdPL7TMFCZjLJlUkwhpwqggY0MNI5NUEcQjRi6jZX8dv7whSlMpzs0qJTOOrgVNKEbGofnB3s9mMxTk1iyIVITbVMk0t0P3lZquU/KdMCOco9x+3ridSOFze16InSiWSjKG1Cq3/UruZMQkoWJ7ov34W+/kuIXUMrejrW82GqErgzCwIUdmESUQuIINGhQogkGJhhUalmhUoVGJTq0NMWJwmpfkpCAnFekXpF+RSUEmFbkqyFVF8MLir6/LHee2X+oze7Z5jJBUxEQYCHUCyIBZUA2pgwZuCWAkQKOEsBW4RsgbAmEIR8n8zREkSnJIMuUapQBLoWlM1PYPtw79tv+u+zbogd/u+kEv6DrR6Qa+34Gg7W/s0CtsOG99D2OJM+6ugRnSehr4qZlZpAzFjOSNMNMkRXiJrsnUSYE40TO7GcQcXjoSQyKVW+7SG/pnhUVc6xWPXOa67/rv2Br+KzbNTNKbWSrSzBCBtwclGQMjYT3VEFNFsHGdiSnCyo0OBrxACmHjZr/hmlC+FP4vLjrtwG8HXzqHxx+Kdux7z70X3isv8N57x96pN/TGHq4NasuaqWX1T/W0fltfbVNre0XNM2/H6t9+AQH0PdQ=</latexit>

and we probably could have removedother hypotheses even sooner

<latexit sha1_base64="S7G0oKTPHCcf5R7g3HVkTMxHLao=">AAADxHicdZLfb9MwEMfTlh+jUNjgkZcTFRLioUoqChNPm8rQHtBUpnXrtJTKcS6rNccOttOpssIfyRv8NbhtEiiCk5z75nN3cu5yUcaZNr7/o9Fs3bl77/7Og/bDR53HT3b3np5rmSuKYyq5VJOIaORM4Ngww3GSKSRpxPEiuhmu4hcLVJpJcWaWGU5Tci1YwigxDs32Gj87nVDgrZmjVJjaTMmssCP3lJqtUoqtMMc0JYX9tHZbkdIX9qwUW1EqleScqGVhh7XcyogxYWJzo/3wW2/luEPUTWFPN77TboeuDMLAhikx8yiBwBWs0VGJIjiq0KhGowqd1ui0QsfWhpRwOC4qcliSw5oMSzKsyaQkk5pcluSyJnRu6ZfX1Vua2mGlT+zJuhkhmYhRGAh1AkTEcIvgfkhEIr4EKnMew5wsENwc5AJjCEOQbjYK5stsJTRqwAUK0FIKVLPdrt/z3w7eBPvg9wZ+sB8MnOgPAt/vQ9Dz19b1ShvNdr+HsaR56j6BcqL1VeBnZmqJMoxyLNphrjEj9IZc45WTgqSop3a9hAW8dCSGRCp3XAtr+meFJanWyzRymauZ679jK/iv2FVukv2pZSLLDQq6uSjJORgJq42GmCmkxk0oZoQqtzYU6JwoQo3b+7YbQtUp/F+c93uB3ws+97sH78tx7HjPvRfeKy/w3nkH3rE38sYebR42r5tZ82vrY4u3dCvfpDYbZc0zb8ta334B4bk9GA==</latexit><latexit sha1_base64="S7G0oKTPHCcf5R7g3HVkTMxHLao=">AAADxHicdZLfb9MwEMfTlh+jUNjgkZcTFRLioUoqChNPm8rQHtBUpnXrtJTKcS6rNccOttOpssIfyRv8NbhtEiiCk5z75nN3cu5yUcaZNr7/o9Fs3bl77/7Og/bDR53HT3b3np5rmSuKYyq5VJOIaORM4Ngww3GSKSRpxPEiuhmu4hcLVJpJcWaWGU5Tci1YwigxDs32Gj87nVDgrZmjVJjaTMmssCP3lJqtUoqtMMc0JYX9tHZbkdIX9qwUW1EqleScqGVhh7XcyogxYWJzo/3wW2/luEPUTWFPN77TboeuDMLAhikx8yiBwBWs0VGJIjiq0KhGowqd1ui0QsfWhpRwOC4qcliSw5oMSzKsyaQkk5pcluSyJnRu6ZfX1Vua2mGlT+zJuhkhmYhRGAh1AkTEcIvgfkhEIr4EKnMew5wsENwc5AJjCEOQbjYK5stsJTRqwAUK0FIKVLPdrt/z3w7eBPvg9wZ+sB8MnOgPAt/vQ9Dz19b1ShvNdr+HsaR56j6BcqL1VeBnZmqJMoxyLNphrjEj9IZc45WTgqSop3a9hAW8dCSGRCp3XAtr+meFJanWyzRymauZ679jK/iv2FVukv2pZSLLDQq6uSjJORgJq42GmCmkxk0oZoQqtzYU6JwoQo3b+7YbQtUp/F+c93uB3ws+97sH78tx7HjPvRfeKy/w3nkH3rE38sYebR42r5tZ82vrY4u3dCvfpDYbZc0zb8ta334B4bk9GA==</latexit><latexit sha1_base64="S7G0oKTPHCcf5R7g3HVkTMxHLao=">AAADxHicdZLfb9MwEMfTlh+jUNjgkZcTFRLioUoqChNPm8rQHtBUpnXrtJTKcS6rNccOttOpssIfyRv8NbhtEiiCk5z75nN3cu5yUcaZNr7/o9Fs3bl77/7Og/bDR53HT3b3np5rmSuKYyq5VJOIaORM4Ngww3GSKSRpxPEiuhmu4hcLVJpJcWaWGU5Tci1YwigxDs32Gj87nVDgrZmjVJjaTMmssCP3lJqtUoqtMMc0JYX9tHZbkdIX9qwUW1EqleScqGVhh7XcyogxYWJzo/3wW2/luEPUTWFPN77TboeuDMLAhikx8yiBwBWs0VGJIjiq0KhGowqd1ui0QsfWhpRwOC4qcliSw5oMSzKsyaQkk5pcluSyJnRu6ZfX1Vua2mGlT+zJuhkhmYhRGAh1AkTEcIvgfkhEIr4EKnMew5wsENwc5AJjCEOQbjYK5stsJTRqwAUK0FIKVLPdrt/z3w7eBPvg9wZ+sB8MnOgPAt/vQ9Dz19b1ShvNdr+HsaR56j6BcqL1VeBnZmqJMoxyLNphrjEj9IZc45WTgqSop3a9hAW8dCSGRCp3XAtr+meFJanWyzRymauZ679jK/iv2FVukv2pZSLLDQq6uSjJORgJq42GmCmkxk0oZoQqtzYU6JwoQo3b+7YbQtUp/F+c93uB3ws+97sH78tx7HjPvRfeKy/w3nkH3rE38sYebR42r5tZ82vrY4u3dCvfpDYbZc0zb8ta334B4bk9GA==</latexit><latexit sha1_base64="S7G0oKTPHCcf5R7g3HVkTMxHLao=">AAADxHicdZLfb9MwEMfTlh+jUNjgkZcTFRLioUoqChNPm8rQHtBUpnXrtJTKcS6rNccOttOpssIfyRv8NbhtEiiCk5z75nN3cu5yUcaZNr7/o9Fs3bl77/7Og/bDR53HT3b3np5rmSuKYyq5VJOIaORM4Ngww3GSKSRpxPEiuhmu4hcLVJpJcWaWGU5Tci1YwigxDs32Gj87nVDgrZmjVJjaTMmssCP3lJqtUoqtMMc0JYX9tHZbkdIX9qwUW1EqleScqGVhh7XcyogxYWJzo/3wW2/luEPUTWFPN77TboeuDMLAhikx8yiBwBWs0VGJIjiq0KhGowqd1ui0QsfWhpRwOC4qcliSw5oMSzKsyaQkk5pcluSyJnRu6ZfX1Vua2mGlT+zJuhkhmYhRGAh1AkTEcIvgfkhEIr4EKnMew5wsENwc5AJjCEOQbjYK5stsJTRqwAUK0FIKVLPdrt/z3w7eBPvg9wZ+sB8MnOgPAt/vQ9Dz19b1ShvNdr+HsaR56j6BcqL1VeBnZmqJMoxyLNphrjEj9IZc45WTgqSop3a9hAW8dCSGRCp3XAtr+meFJanWyzRymauZ679jK/iv2FVukv2pZSLLDQq6uSjJORgJq42GmCmkxk0oZoQqtzYU6JwoQo3b+7YbQtUp/F+c93uB3ws+97sH78tx7HjPvRfeKy/w3nkH3rE38sYebR42r5tZ82vrY4u3dCvfpDYbZc0zb8ta334B4bk9GA==</latexit>

only require labels for examples that hypotheses 1 and 2 label differently (i.e., examples where they disagree)

Disagreement-Based Active Learning

consider points uniform on unit ball andlinear classifiers passing through origin

<latexit sha1_base64="7RzfEZM/sGdvL6/YfQwlqXI87hY=">AAAD2HicbZLPb9MwFMfTlh+jUOjgyOWJCglxqJpdhjhtKpN6QFOp1q2jKZPjuI1V/4hsB1RZkTiAEFf+NG78E/wNOGkayMaTHH/zee/Zfn4OE0a1GQx+NZqtW7fv3N27177/oPPwUXf/8bmWqcJkiiWTahYiTRgVZGqoYWSWKIJ4yMhFuB7m/ouPRGkqxZnZJGTB0UrQJcXIOHS13/jd6QSCfDIxkYpwmyiZZHbsvlLTPCSruRnhHGX2bTHVPOWc2bNS1LxYKskYUpvMDitZi4jIkortjvbNX12LcQOpdWYn27nTbgcuDQLfBhyZOFyC7xIKdFKiEE52aFyh8Q5NKjTZoZG1AUYMRtmOHJfkuCLDkgwrMivJrCKXJbmsCI4t/vBy98e5He70qT0tihGSiogIA4FeApZC04goSBw1GlLXNqk4SJFLAyFiDJCIIAgg7z5SAJghrV13XcMhyaVYgYmVTFcxSEVXVFx1e4P+oDC4KfxS9LzSxlfdn0EkccrdsYrV5/4gMQuLlKGYkawdpJokCK/RisydFIgTvbDFw8zguSMRuGO74coq6L8ZFnGtNzx0kXkf9HVfDv/nm6dm+WphqUhSQwTebrRMGRgJ+SuHiCqCDds4gbByTwkDjpFC2LirabtL8K+XfFOcH/T9Qd9/d9A7el1ex5731HvmvfB879A78kbe2Jt6uDlt2uaX5tfW+9bn1rfW921os1HmPPFq1vrxBzY9RTE=</latexit><latexit sha1_base64="7RzfEZM/sGdvL6/YfQwlqXI87hY=">AAAD2HicbZLPb9MwFMfTlh+jUOjgyOWJCglxqJpdhjhtKpN6QFOp1q2jKZPjuI1V/4hsB1RZkTiAEFf+NG78E/wNOGkayMaTHH/zee/Zfn4OE0a1GQx+NZqtW7fv3N27177/oPPwUXf/8bmWqcJkiiWTahYiTRgVZGqoYWSWKIJ4yMhFuB7m/ouPRGkqxZnZJGTB0UrQJcXIOHS13/jd6QSCfDIxkYpwmyiZZHbsvlLTPCSruRnhHGX2bTHVPOWc2bNS1LxYKskYUpvMDitZi4jIkortjvbNX12LcQOpdWYn27nTbgcuDQLfBhyZOFyC7xIKdFKiEE52aFyh8Q5NKjTZoZG1AUYMRtmOHJfkuCLDkgwrMivJrCKXJbmsCI4t/vBy98e5He70qT0tihGSiogIA4FeApZC04goSBw1GlLXNqk4SJFLAyFiDJCIIAgg7z5SAJghrV13XcMhyaVYgYmVTFcxSEVXVFx1e4P+oDC4KfxS9LzSxlfdn0EkccrdsYrV5/4gMQuLlKGYkawdpJokCK/RisydFIgTvbDFw8zguSMRuGO74coq6L8ZFnGtNzx0kXkf9HVfDv/nm6dm+WphqUhSQwTebrRMGRgJ+SuHiCqCDds4gbByTwkDjpFC2LirabtL8K+XfFOcH/T9Qd9/d9A7el1ex5731HvmvfB879A78kbe2Jt6uDlt2uaX5tfW+9bn1rfW921os1HmPPFq1vrxBzY9RTE=</latexit><latexit sha1_base64="7RzfEZM/sGdvL6/YfQwlqXI87hY=">AAAD2HicbZLPb9MwFMfTlh+jUOjgyOWJCglxqJpdhjhtKpN6QFOp1q2jKZPjuI1V/4hsB1RZkTiAEFf+NG78E/wNOGkayMaTHH/zee/Zfn4OE0a1GQx+NZqtW7fv3N27177/oPPwUXf/8bmWqcJkiiWTahYiTRgVZGqoYWSWKIJ4yMhFuB7m/ouPRGkqxZnZJGTB0UrQJcXIOHS13/jd6QSCfDIxkYpwmyiZZHbsvlLTPCSruRnhHGX2bTHVPOWc2bNS1LxYKskYUpvMDitZi4jIkortjvbNX12LcQOpdWYn27nTbgcuDQLfBhyZOFyC7xIKdFKiEE52aFyh8Q5NKjTZoZG1AUYMRtmOHJfkuCLDkgwrMivJrCKXJbmsCI4t/vBy98e5He70qT0tihGSiogIA4FeApZC04goSBw1GlLXNqk4SJFLAyFiDJCIIAgg7z5SAJghrV13XcMhyaVYgYmVTFcxSEVXVFx1e4P+oDC4KfxS9LzSxlfdn0EkccrdsYrV5/4gMQuLlKGYkawdpJokCK/RisydFIgTvbDFw8zguSMRuGO74coq6L8ZFnGtNzx0kXkf9HVfDv/nm6dm+WphqUhSQwTebrRMGRgJ+SuHiCqCDds4gbByTwkDjpFC2LirabtL8K+XfFOcH/T9Qd9/d9A7el1ex5731HvmvfB879A78kbe2Jt6uDlt2uaX5tfW+9bn1rfW921os1HmPPFq1vrxBzY9RTE=</latexit><latexit sha1_base64="7RzfEZM/sGdvL6/YfQwlqXI87hY=">AAAD2HicbZLPb9MwFMfTlh+jUOjgyOWJCglxqJpdhjhtKpN6QFOp1q2jKZPjuI1V/4hsB1RZkTiAEFf+NG78E/wNOGkayMaTHH/zee/Zfn4OE0a1GQx+NZqtW7fv3N27177/oPPwUXf/8bmWqcJkiiWTahYiTRgVZGqoYWSWKIJ4yMhFuB7m/ouPRGkqxZnZJGTB0UrQJcXIOHS13/jd6QSCfDIxkYpwmyiZZHbsvlLTPCSruRnhHGX2bTHVPOWc2bNS1LxYKskYUpvMDitZi4jIkortjvbNX12LcQOpdWYn27nTbgcuDQLfBhyZOFyC7xIKdFKiEE52aFyh8Q5NKjTZoZG1AUYMRtmOHJfkuCLDkgwrMivJrCKXJbmsCI4t/vBy98e5He70qT0tihGSiogIA4FeApZC04goSBw1GlLXNqk4SJFLAyFiDJCIIAgg7z5SAJghrV13XcMhyaVYgYmVTFcxSEVXVFx1e4P+oDC4KfxS9LzSxlfdn0EkccrdsYrV5/4gMQuLlKGYkawdpJokCK/RisydFIgTvbDFw8zguSMRuGO74coq6L8ZFnGtNzx0kXkf9HVfDv/nm6dm+WphqUhSQwTebrRMGRgJ+SuHiCqCDds4gbByTwkDjpFC2LirabtL8K+XfFOcH/T9Qd9/d9A7el1ex5731HvmvfB879A78kbe2Jt6uDlt2uaX5tfW+9bn1rfW921os1HmPPFq1vrxBzY9RTE=</latexit>

only label points in theregion of disagreement D

<latexit sha1_base64="ct3Eyv5JAlSrQdQBIHQLRnZdcDg=">AAADw3icbZJLb9NAEMcdm0cJBFI4clmRIiEOkd1LEadUaaQcUBWiPlLFIVqv18kq+7B216Bo5S/Jrd+GtWMb3DLSev7+zcw+RhOllCjt+/cd13vy9Nnzoxfdl696r9/0j9/eKJFJhK+RoEIuIqgwJRxfa6IpXqQSQxZRfBvtxkX89ieWigh+pfcpXjG44SQhCGqL1sed+14v5PiX3mIhMTOpFGluZvYrFClS8laYYsZgbr6VrhWpfG6uKtGKIiEFpVDuczNuZCsjxgnhhxPNxV/dyrELyl1u5gff63ZDWwbCwIQM6m2UgMAWlGhSoQhMajRr0KxG8wbNazQ1JkSQgmlek/OKnDdkXJFxQxYVWTTkriJ3DUFbg358rv8YM+NaX5rL8jFcEB5jrkGoEiA43QMKI0xBarlWgHBgOwHCEEi8sc0BIgExUXAjMWZF2Un5mkTCnbnIT9b9gT/0SwOPRVCJgVPZbN3/HcYCZcVWiEKlloGf6pWBUhNEcd4NM4VTiHZwg5dWcsiwWplyBnPw0ZIYJELaZa9S0n8rDGRK7VlkM4tLqoexAv4vtsx08mVlCE8zjTk6HJRkFGgBioG2HZAYadurmEAk7dQggLZQQqTt2HdtE4KHT34sbk6HgT8Mvp8ORl+rdhw5750PzicncM6ckTN1Zs61g9yRm7jCTb2Jt/Okpw+pbqeqeee0zMv/AEUCO+U=</latexit><latexit sha1_base64="ct3Eyv5JAlSrQdQBIHQLRnZdcDg=">AAADw3icbZJLb9NAEMcdm0cJBFI4clmRIiEOkd1LEadUaaQcUBWiPlLFIVqv18kq+7B216Bo5S/Jrd+GtWMb3DLSev7+zcw+RhOllCjt+/cd13vy9Nnzoxfdl696r9/0j9/eKJFJhK+RoEIuIqgwJRxfa6IpXqQSQxZRfBvtxkX89ieWigh+pfcpXjG44SQhCGqL1sed+14v5PiX3mIhMTOpFGluZvYrFClS8laYYsZgbr6VrhWpfG6uKtGKIiEFpVDuczNuZCsjxgnhhxPNxV/dyrELyl1u5gff63ZDWwbCwIQM6m2UgMAWlGhSoQhMajRr0KxG8wbNazQ1JkSQgmlek/OKnDdkXJFxQxYVWTTkriJ3DUFbg358rv8YM+NaX5rL8jFcEB5jrkGoEiA43QMKI0xBarlWgHBgOwHCEEi8sc0BIgExUXAjMWZF2Un5mkTCnbnIT9b9gT/0SwOPRVCJgVPZbN3/HcYCZcVWiEKlloGf6pWBUhNEcd4NM4VTiHZwg5dWcsiwWplyBnPw0ZIYJELaZa9S0n8rDGRK7VlkM4tLqoexAv4vtsx08mVlCE8zjTk6HJRkFGgBioG2HZAYadurmEAk7dQggLZQQqTt2HdtE4KHT34sbk6HgT8Mvp8ORl+rdhw5750PzicncM6ckTN1Zs61g9yRm7jCTb2Jt/Okpw+pbqeqeee0zMv/AEUCO+U=</latexit><latexit sha1_base64="ct3Eyv5JAlSrQdQBIHQLRnZdcDg=">AAADw3icbZJLb9NAEMcdm0cJBFI4clmRIiEOkd1LEadUaaQcUBWiPlLFIVqv18kq+7B216Bo5S/Jrd+GtWMb3DLSev7+zcw+RhOllCjt+/cd13vy9Nnzoxfdl696r9/0j9/eKJFJhK+RoEIuIqgwJRxfa6IpXqQSQxZRfBvtxkX89ieWigh+pfcpXjG44SQhCGqL1sed+14v5PiX3mIhMTOpFGluZvYrFClS8laYYsZgbr6VrhWpfG6uKtGKIiEFpVDuczNuZCsjxgnhhxPNxV/dyrELyl1u5gff63ZDWwbCwIQM6m2UgMAWlGhSoQhMajRr0KxG8wbNazQ1JkSQgmlek/OKnDdkXJFxQxYVWTTkriJ3DUFbg358rv8YM+NaX5rL8jFcEB5jrkGoEiA43QMKI0xBarlWgHBgOwHCEEi8sc0BIgExUXAjMWZF2Un5mkTCnbnIT9b9gT/0SwOPRVCJgVPZbN3/HcYCZcVWiEKlloGf6pWBUhNEcd4NM4VTiHZwg5dWcsiwWplyBnPw0ZIYJELaZa9S0n8rDGRK7VlkM4tLqoexAv4vtsx08mVlCE8zjTk6HJRkFGgBioG2HZAYadurmEAk7dQggLZQQqTt2HdtE4KHT34sbk6HgT8Mvp8ORl+rdhw5750PzicncM6ckTN1Zs61g9yRm7jCTb2Jt/Okpw+pbqeqeee0zMv/AEUCO+U=</latexit><latexit sha1_base64="ct3Eyv5JAlSrQdQBIHQLRnZdcDg=">AAADw3icbZJLb9NAEMcdm0cJBFI4clmRIiEOkd1LEadUaaQcUBWiPlLFIVqv18kq+7B216Bo5S/Jrd+GtWMb3DLSev7+zcw+RhOllCjt+/cd13vy9Nnzoxfdl696r9/0j9/eKJFJhK+RoEIuIqgwJRxfa6IpXqQSQxZRfBvtxkX89ieWigh+pfcpXjG44SQhCGqL1sed+14v5PiX3mIhMTOpFGluZvYrFClS8laYYsZgbr6VrhWpfG6uKtGKIiEFpVDuczNuZCsjxgnhhxPNxV/dyrELyl1u5gff63ZDWwbCwIQM6m2UgMAWlGhSoQhMajRr0KxG8wbNazQ1JkSQgmlek/OKnDdkXJFxQxYVWTTkriJ3DUFbg358rv8YM+NaX5rL8jFcEB5jrkGoEiA43QMKI0xBarlWgHBgOwHCEEi8sc0BIgExUXAjMWZF2Un5mkTCnbnIT9b9gT/0SwOPRVCJgVPZbN3/HcYCZcVWiEKlloGf6pWBUhNEcd4NM4VTiHZwg5dWcsiwWplyBnPw0ZIYJELaZa9S0n8rDGRK7VlkM4tLqoexAv4vtsx08mVlCE8zjTk6HJRkFGgBioG2HZAYadurmEAk7dQggLZQQqTt2HdtE4KHT34sbk6HgT8Mvp8ORl+rdhw5750PzicncM6ckTN1Zs61g9yRm7jCTb2Jt/Okpw+pbqeqeee0zMv/AEUCO+U=</latexit>

Active Binary Classification

parametric rate

exponential speed-up

# labels

Bayes error rate

passive

active

passive: ✏ ⇠ d

n

active: ✏ ⇠ exp⇣� c

n

d

⌘

<latexit sha1_base64="82Q4UYsIF/dj5C7yNeqNYQNT/N4=">AAAE8nicfZRRb9MwEMeztcAoFDp45MViAo1JTM14YEJC2ugm7WGaurFuneYyOc6ltebYwXFGK8sfgxceQIhXPg1vfBucNs3WbcJS6tPv7v6XOzsNEs5S3Wz+nZuvVO/cvbdwv/bgYf3R48bik6NUZopCh0ouVTcgKXAmoKOZ5tBNFJA44HAcnLdy//EFqJRJcahHCfRi0hcsYpRoh84WKwv1OhbwRQ9AKohNomRiTdv9ypTlIXbGzSGOiTW7423GU+zWHBbGjJdKJTknamRNqzRnIkKImJhUNFuX9kyMe4g6t+ZgstdrNezSEPYNjokeBBHyXcIYbRcoQNtT1C5R2xaJ+yXan0YdlOhginaMwZRwtGOnZLMgmyVpFaRVkm5BuiU5KciJvUW5eKNOQTol2S3IbkmCzOBA8jAdxW4zWemgA0M/rUzF49i0pvae2XPjymdJpTs6ETrNj/aqDN4CrklZYjhTYpiXGIySAYgEBOF69N5vuuUOQEgmQhAa4TRCNRxAnwkDnwVRioxWXJoTcGIahnp8W90hhtYkJE3ZBbyzFr1EGJKUcSlyM2UxwghHilDj4oRFGN/UCHgG1hCq/6cBwwR/YP3l13Qq6K5WaHP2aqwKbgyXb3rWWGquNscL3TT8wljyitU+a/zBoaRZ7Jqn3PVz6jcT3TNEaUY5uM6zFBJCz0kfTp0pSAxpz4wbsOiFIyGKpHKPG96YXs0wJE7z4bvI/Dqm1305vM13mulovWeYSDINgk4KRRlHWqL8+0chU0A1HzmDUOU+MorogLjpaPcvUXND8K+3fNM4Wlv136z6+2tLG+vFOBa8Z95zb9nzvbfehrfjtb2ORyuy8rXyvfKjqqvfqj+rvyah83NFzlNvZlV//wNX0q3o</latexit>

active

passive: ✏ ⇠ d

n

active: ✏ ⇠ exp⇣� c

n

d

⌘

<latexit sha1_base64="82Q4UYsIF/dj5C7yNeqNYQNT/N4=">AAAE8nicfZRRb9MwEMeztcAoFDp45MViAo1JTM14YEJC2ugm7WGaurFuneYyOc6ltebYwXFGK8sfgxceQIhXPg1vfBucNs3WbcJS6tPv7v6XOzsNEs5S3Wz+nZuvVO/cvbdwv/bgYf3R48bik6NUZopCh0ouVTcgKXAmoKOZ5tBNFJA44HAcnLdy//EFqJRJcahHCfRi0hcsYpRoh84WKwv1OhbwRQ9AKohNomRiTdv9ypTlIXbGzSGOiTW7423GU+zWHBbGjJdKJTknamRNqzRnIkKImJhUNFuX9kyMe4g6t+ZgstdrNezSEPYNjokeBBHyXcIYbRcoQNtT1C5R2xaJ+yXan0YdlOhginaMwZRwtGOnZLMgmyVpFaRVkm5BuiU5KciJvUW5eKNOQTol2S3IbkmCzOBA8jAdxW4zWemgA0M/rUzF49i0pvae2XPjymdJpTs6ETrNj/aqDN4CrklZYjhTYpiXGIySAYgEBOF69N5vuuUOQEgmQhAa4TRCNRxAnwkDnwVRioxWXJoTcGIahnp8W90hhtYkJE3ZBbyzFr1EGJKUcSlyM2UxwghHilDj4oRFGN/UCHgG1hCq/6cBwwR/YP3l13Qq6K5WaHP2aqwKbgyXb3rWWGquNscL3TT8wljyitU+a/zBoaRZ7Jqn3PVz6jcT3TNEaUY5uM6zFBJCz0kfTp0pSAxpz4wbsOiFIyGKpHKPG96YXs0wJE7z4bvI/Dqm1305vM13mulovWeYSDINgk4KRRlHWqL8+0chU0A1HzmDUOU+MorogLjpaPcvUXND8K+3fNM4Wlv136z6+2tLG+vFOBa8Z95zb9nzvbfehrfjtb2ORyuy8rXyvfKjqqvfqj+rvyah83NFzlNvZlV//wNX0q3o</latexit>

passive

✏ = R( bf)�R(f⇤)<latexit sha1_base64="pvQUBf6y7D81KHp+zc7wo3bkWXg=">AAAEfXicbZNBT9swFMdD222sWzfYjrtYQ0xQbVXDDnBBAhUkDgiVjkIR6ZDjvBALx84cZxBZ+RT7Zrvtq+yyOa0bKGDJ8T+/956f/Wz7CaOp6nb/LNTqjWfPXyy+bL563Xrzdmn53WkqMklgSAQTcuTjFBjlMFRUMRglEnDsMzjzr3ul/ewnyJQKfqLyBMYxvuI0pAQrgy6Xa79aLY/DjYpASIh1IkVS6L75ipSWLsWcmUEc40IfToY5ix0LfWLFnJUIKRjDMi90r5JzHgGElE8z6r07PedjOpbXhR5Mx1az6Zkw5Lnai7GK/BC5JmCC9i3y0f4M9SvUL2zgcYWOZ16DCg1m6EBrj2CGDooZ2bVktyI9S3oVGVkyqsi5JefFEzPbFQ0tGVbk0JLDiviZ9nzBgjSPzaCzykAiTb63Z5PHse7N9JE+MuUqa0mEOToemDm/Ffen8faAKVyluJ1LcVumiPIkAp4Ax0zl227XNHMAXFAeAFfIS0PU9Hy4olzDD46lxHnbhEGSUiY4+oS2TR+seTc0gAgrFK6jL+Y//N5e98Cs6C7ocmml2+lOGnosXCtWHNv6l0u/vUCQLDbrIAyn6YXbTdRYY6koYWAWkaWQYHKNr+DCSI5jSMd68noKtGpIgEIhTTf7mND7ERrHaVkH41nejPShrYRP2S4yFW6NNeVJpoCTaaIwY0gJVD5FFFAJRLHcCEykue8EkQhLTJR5sE1TBPfhlh+L042O+7XjHm+s7GzZciw6H5yPzprjOpvOjnPg9J2hQ2p/66i+Xm/X/zVWG58bnalrbcHGvHfmWmPzP7VTffk=</latexit>



Assuming optimal Bayes classifer f⇤ in VC class with dimension dand “nice” distributions (e.g., bounded label noise)

<latexit sha1_base64="lssTZsHpGLEgnSo2BGjEW8D/EzQ=">AAAEtHicbZNRT9swEMcD7TbWrRtsj3uxoJMYmqqGSRsvk2AFiQeECqNQRAo4zqW1cOzIdgaRlS+4x73t28xp3YwClhz/9bs73/lihymjSnc6fxcWa/Vnz18svWy8et1883Z55d2pEpkk0CeCCTkIsQJGOfQ11QwGqQSchAzOwptuaT/7BVJRwU90nsIwwSNOY0qwtuhqZfF3sxlwuNVjEBISk0qRFqZnv0LR0qWYMzNIElyYg8kyZ3FrYU6cmLMSIQVjWOaF6VZyziOCmPJpRrP7X8/52InlTWGOp2uz0QhsGAp8EyRYj8MY+TZggvYcCtHeDPUq1Ctc4FGFjmZexxU6nqF9YwKCGdovZmTHkZ2KdB3pVmTgyKAi546cF0/s7CrqO9KvyIEjBxUJMxOEgkUqT+xisspAxoZcbsw2TxLTnelDc2jbVfaSCPvreGT3/Fnc3ybYBaZxleJuLsVdmWKcp2PgKXDMdP7d79hhfwAXlEfANQpUjHaUyhLKR0ikmia26B84B4UIw0rRGCRqxZcbLUQ5Ou1OKbqleowimgAv7yhqRS0UBMhWiK6vOSWwao1KSxpm5X1QaB3ao/ZnFIrMpo0QwyEwZItQ8Olqea3T7kwGeix8J9Y8N3pXy3+CSJDM5taTYi78TqqHBktNCQN75ExBiskNHsGFlRwnoIZm8ugK9NGSCMVC2mmPP6H3IwxOVNk+61leKPXQVsKnbBeZjreGhvI008DJNFGcMaQFKl+w7YcEolluBSbSPhOCyBhLTLR95w3bBP/hkR+L0822/6XtH22ubW+5dix5H7xVb93zvW/etrfv9by+R2p+7ax2XcP1r/WgTuowdV1ccDHvvblR5/8ApXqSAQ==</latexit>

Tutorial Outline

Part 1: Introduction to Active Learning (Rob)

Part 2: Theory of Active Learning (Steve)

Part 3: Advanced Topics and Open Problems (Steve)

Part 4: Nonparametric Active Learning (Rob)

slides: http://nowak.ece.wisc.edu/ActiveML.html

Recommended Reading (Foundations of Active Learning)

Dasgupta, Sanjoy, Daniel J. Hsu, and Claire Monteleoni. "A general agnostic active learning algorithm." Advances in neural information processing systems. 2008.

Balcan, Maria-Florina, Alina Beygelzimer, and John Langford. "Agnostic active learning." Journal of Computer and System Sciences 75.1 (2009): 78-89.

Dasgupta, Sanjoy. "Two faces of active learning." Theoretical computer science 412.19 (2011): 1767-1781.

Zhu, Xiaojin, John Lafferty, and Zoubin Ghahramani. "Combining active learning and semi-supervised learning using gaussian fields and harmonic functions." ICML 2003 workshop. Vol. 3. 2003.

Cohn, David, Les Atlas, and Richard Ladner. "Improving generalization with active learning." Machine learning 15.2 (1994): 201-221.

Settles, Burr. "Active learning." Synthesis Lectures on Artificial Intelligence and Machine Learning 6.1 (2012): 1-114.

Nowak, Robert D. "The geometry of generalized binary search." IEEE Transactions on Information Theory 57, no. 12 (2011): 7893-7906.

Hanneke, Steve. "Theory of active learning." Foundations and Trends in Machine Learning 7, no. 2-3 (2014).

Castro, Rui M., and Robert D. Nowak. "Minimax bounds for active learning." IEEE Transactions on Information Theory 54, no. 5 (2008): 2339-2353.

active learning from theory to practice · interactive machine-human data collection and...

Documents