Machine Learning via

Advice Taking

Jude Shavlik

Thanks To ...

Rich MaclinLisa TorreyTrevor Walker

Prof. Olvi MangasarianGlenn FungTed Wild

DARPA

Quote (2002) from DARPA

Sometimes an assistant will merely watch you and draw conclusions.

Sometimes you have to tell a new person, 'Please don't do it this way' or 'From now on when I say X, you do Y.'

It's a combination of learning by example and by being guided.

Widening the “Communication Pipeline” between Humans and Machine Learners

Teacher

Pupil

Machine Learner

Our Approach to Building Better Machine Learners

• Human partner expresses advice “naturally” and w/o knowledge of ML agent’s internals

• Agent incorporates advice directly into the function it is learning

• Additional feedback (rewards, I/O pairs, inferred labels, more advice) used to refine learner continually

“Standard” Machine Learning vs. Theory Refinement

• Positive Examples (“should see doctor”) temp = 102.1, age = 21, sex = F, …

temp = 101.7, age = 37, sex = M, …

• Negative Examples (“take two aspirins”) temp = 99.1, age = 43, sex = M, …

temp = 99.6, age = 24, sex = F, …

• Approximate Domain Knowledge if temp = high and age = young … then neg example

Related work by labs of Mooney, Pazzani, Cohen, Giles, etc

Rich Maclin’s PhD (1995)

IF a Bee is (Near and West) & an Ice is (Near and North)Then Begin Move East Move North END

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

0

10

00

20

00

30

00

40

00

Number of Training Episodes

Re

info

rce

me

nt

on

Te

sts

et

Sample Results

Without advice

With advice

Our Motto

Give advice

rather than commands

to your computer

Outline

Prior Knowledge and Support Vector Machines Intro to SVM’s Linear Separation Non-Linear Separation Function Fitting (“Regression”) Advice-Taking Reinforcement Learning Transfer Learning via Advice Taking

Support Vector MachinesMaximizing the Margin between Bounding Planes

x0w = í + 1

x0w = í à 1

A+

A-

jjwjj22

Support Vectors?

Margin

Linear Algebra for SVM’s

• Given p points in n dimensional space• Represent by p-by-n matrix A of reals

• More succinctly

D(Awà eí )=e;where e is vector of ones

• Separate by two bounding planes

A iw=í + 1; for D i i = + 1;

A iw5 í à 1; for D i i = à 1:

• Each Ai in class +1 or -1

“Slack” VariablesDealing with Data that is not Linearly Separable

A+

A-

y

Support Vectors

Support Vector Machines Quadratic Programming Formulation

• Solve this quadratic program

D(Awà eí ) >e;y > 0;w; í e0ymin

s.t. + y÷ + 2

1jjwjj22

• Maximize margin by minimizing21kwk2

2jjwjj22

• Minimize sum of slack vars with wgt

÷e0y

Support Vector MachinesLinear Programming Formulation

Use 1-norm instead of 2-norm(typically runs faster; better feature selection;might generalize better, NIPS ‘03)

÷e0y+ kwk1y > 0;w; í

D(Awà eí ) + y > e

min

s.t.

Knowledge-Based SVM’sGeneralizing “Example” from POINT to REGION

A+

A-

Incorporating “Knowledge Sets”

Into the SVM Linear Program

This implication equivalent to set of constraints (proof in NIPS ’02 paper)

• Suppose that knowledge set belongs to class A+

Hence must lie in half space

èx??Bx 6 b

é

èxjx0w>í + 1

é

Bx6b ) x0w>í + 1

• We therefore have the implication

Resulting LP for KBSVM’s

We get this linear program (LP)

Ranges over # regions

KBSVM with Slack Variables

Was 0

SVMs and Non-Linear Separating Surfaces

f1

f2 +

+

_

_

h(f1, f2)

g(f1, f2) +

+

_

_

Non-linearly map to new space

Linearly separate in new space (using kernels)

Result is non-linear separator in original space

Fung et al. (2003) presents knowledge-

based non-linear SVMs

Support Vector Regression(aka Kernel Regression)

Linearly approximating a function, given array A of inputs and vector y of (numeric) outputs

f(x) ≈ x’w + b

Find weights such that

Aw + be ≈ y

In dual space, w = A’, so get

(A A’) + be ≈ y

Kernel’izing (to get non-linear approx)

K(A,A’) + be ≈ y

y

x

What to Optimize?

Linear program to optimize

• 1st term () is “regularizer” that minimizes model complexity

• 2nd term is approximation error, weighted by parameter C

• Classical “least squares” fit if quadratic version and first term ignored

Predicting Y for New X

y = K(x’, A’) + b

• Use Kernel to compute “distance” to each training point (ie, row in A)

• Weight by i (hopefully many i are zero), Sum

• Add b (a scalar)

Knowledge-Based SVRMangasarian, Shavlik, & Wild, JMLR ‘04

Add soft constraints to linear program (so need only follow advice approximately)

minimize ||w||1 + C ||s||1

+ penalty for violating advice

such that y - s Aw + b y + s “slacked” match to advice

Advice: In this region, y should exceed 4

S

y

4

Testbeds: Subtasks of RoboCup

Keep ball from opponents

[Stone & Sutton, ICML 2001]

Mobile KeepAway

Score goal

[Maclin et al., AAAI 2005]

BreakAway

Reinforcement Learning Overview

Take an actionReceive a state

Receive a reward

Policy: choose the action with the highest Q-value in the current state

Use the rewards to

estimate the Q-values of actions in

states

Described by a set of features

Incorporating Advice in KBKR

Advice format Bx ≤ d f(x) ≥ hx +

TeammatedistanceTo

shotAngle

GoaldistanceTo

is x

0 1 0

0 0 1

-

bwx

30

10x

9.0

If distanceToGoal ≤ 10 and

shotAngle ≥ 30

Then Q(shoot) ≥ 0.9

Giving Advice About Relative Values of Multiple Functions

Maclin et al, AAAI ’05

When the input satisfies

preconditions(input)

Then

f1(input) > f2(input)

Sample Advice-Taking Results

if distanceToGoal 10

and shotAngle 30

then prefer shoot over all other actions

0.0

0.2

0.4

0.6

0.8

1.0

0 5000 10000 15000 20000 25000

Games Played

Pro

b(S

core

Go

al)

advice

std RL2 vs 1 BreakAway, rewards +1, -1

Q(shoot) > Q(pass)Q(shoot) > Q(move)

Transfer Learning

Agent discovers how tasks are related

We use a user

mappingto tell the agent this

Agent learns Task A

Agent encounters related Task B

Agent uses knowledge from Task A to learn Task B faster

Task A is the

source Task B is

the target

Transfer Learning:The Goal for the Target Task

perf

orm

ance

training

with transfer

without transfer

better start

faster rise better asymptote

Our Transfer Algorithm

Observe source task games to learn skills

Use ILP to create advice for the target task

Learn target taskwith KBKR

Translate learned skills

into transfer advice

If there is user advice, add it

in

Learning Skills By Observation

• Source-task games are sequences: (state, action)• Learning skills is like learning to classify states

by their correct actions• ILP = Inductive Logic Programming

State 1distBetween(me,teammate2) = 15distBetween(me,teammate1) = 10distBetween(me,opponent1) = 5...action = pass(teammate2)outcome = caught(teammate2)

ILP: Searching for First-Order Rules

P :- true

P :- Q P :- R P :- S

P :- R, Q P :- R, S

P :- R, S, V, W, XWe also use a

random-sampling approach

Advantages of ILP

• Can produce first-order rules for skills• Capture only the essential aspects of the skill• We expect these aspects to transfer better

• Can incorporate background knowledge

pass(Teammate)

pass(teammate1)

pass(teammateN)

vs....

Example of a Skill Learned by ILP from KeepAway

pass(Teammate) :- distBetween(me, Teammate) > 14, passAngle(Teammate) > 30, passAngle(Teammate) < 150, distBetween(me, Opponent) < 7.

Also gave “human” advice about shooting, since that is new skill in BreakAway

TL Level 7: KA to BA Raw Curves

TL Level 7: KA to BA Averaged Curves

TL Level 7: StatisticsTL Metrics Average Reward

Type Name KA to BA MD to BA

Score P Value Score P Value

I Jump start 0.05 0.0312 0.08 0.0086

Jump start smoothed 0.08 0.0002 0.06 0.0014

II Transfer ratio 1.82 0.0034 1.86 0.0004

Transfer ratio (truncated) 1.82 0.0032 1.86 0.0004

Average relative reduction (narrow)

0.58 0.0042 0.54 0.0004

Average relative reduction (wide) 0.70 0.0018 0.71 0.0008

Ratio (of area under the curves) 1.37 0.0056 1.41 0.0012

Transfer difference 503.57 0.0046 561.27

0.0008

Transfer difference (scaled) 1017.00

0.0040 1091.2

0.0016

III Asymptotic advantage 0.09 0.0086 0.11 0.0040

Asymptotic advantage smoothed 0.08 0.0116 0.10 0.0030

Boldface indicates a significant difference was found

Conclusion

• Can use much more than I/O pairs in ML

• Give advice to computers; theyautomatically refine it based on feedback from user or environment

• Advice an appealing mechanism for transferring learned knowledgecomputer-to-computer

Some Papers (on-line, use Google :-)

Creating Advice-Taking Reinforcement Learners, Maclin & Shavlik, Machine Learning 1996

Knowledge-Based Support Vector Machine Classifiers, Fung, Mangasarian, & Shavlik, NIPS 2002

Knowledge-Based Nonlinear Kernel Classifiers, Fung, Mangasarian, & Shavlik, COLT 2003

Knowledge-Based Kernel Approximation, Mangasarian, Shavlik, & Wild, JAIR 2004

Giving Advice about Preferred Actions to Reinforcement Learners Via Knowledge-Based Kernel Regression, Maclin, Shavlik, Torrey, Walker, & Wild, AAAI 2005

Skill Acquisition via Transfer Learning and Advice Taking, Torrey, Shavlik, Walker, & Maclin, ECML 2006

Backups

Breakdown of Results

0

0.1

0.2

0.3

0.4

0.5

0.6

0 1000 2000 3000 4000 5000

Games Played

Pro

bab

ilit

y o

f G

oal

all advice

transfer advice onlyuser advice only

no advice

What if User Advice is Bad?

0

0.1

0.2

0.3

0.4

0.5

0.6

0 1000 2000 3000 4000 5000

Games Played

Pro

bab

ility

of

Go

al

Transfer with good advice

Transfer with bad adviceBad advice only

No advice

Related Work on Transfer

• Q-function transfer in RoboCup• Taylor & Stone (AAMAS 2005, AAAI 2005)

• Transfer via policy reuse• Fernandez & Veloso (AAMAS 2006, ICML workshop

2006)• Madden & Howley (AI Review 2004)• Torrey et al. (ECML 2005)

• Transfer via relational RL• Driessens et al. (ICML workshop 2006)