cpsc540 - computer science at ubcnando/540-2013/lectures/l8.pdf · february, 2013 university of...
TRANSCRIPT
![Page 1: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/1.jpg)
CPSC540
Nando de FreitasFebruary, 2013University of British Columbia
Decision trees
![Page 2: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/2.jpg)
Outline of the lecture
This lecture provides an introduction to decision trees. It discusses:
� Decision trees� Using reduction in entropy as a criterion for constructing decision trees.� The application of decision trees to classificationThe application of decision trees to classification
![Page 3: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/3.jpg)
Motivation example 1:object detection
![Page 4: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/4.jpg)
Motivation example 2: Kinect
![Page 5: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/5.jpg)
Image classification example
[MSR Tutorial on decision forests by Criminisi et al, 2011]
![Page 6: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/6.jpg)
Classification tree
[Criminisi et al, 2011]
![Page 7: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/7.jpg)
Another commerce example
Simafore.com
![Page 8: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/8.jpg)
From a spreadsheet to a decision node
[AI book of Stuart Russell and Peter Norvig]
![Page 9: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/9.jpg)
A learned decision tree
[AI book of Stuart Russell and Peter Norvig]
![Page 10: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/10.jpg)
How do we construct the tree ?i.e., how to pick attribute (nodes)?
np
n
np
n
np
p
np
p
np
n
np
pH
++−
++−=
++ 22 loglog),(
For a training set containing p positive examples and n negative examples, we have:
![Page 11: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/11.jpg)
� A chosen attribute A, with K distinct values, divides the training set E into subsets E1, … , EK.
� The Expected Entropy (EH) remaining after trying attribute A(with branches i=1,2,…,K) is
∑ ++++=
Kiiii
np
n
np
pH
np
npAEH ),()(
How to pick nodes?
� Information gain (I) or reduction in entropy for this attribute is:
� Choose the attribute with the largest I
∑= +++
=i iiii npnp
Hnp
AEH1
),()(
)(),()( AEHnp
n
np
pHAI −
++=
[Hwee Tou Ng & Stuart Russell]
![Page 12: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/12.jpg)
� Convention: For the training set, p = n = 6, H(6/12, 6/12) = 1bit
� Consider the attributes Patronsand Type(and others too):
bits 0)]4
2,
4
2(
12
4)
4
2,
4
2(
12
4)
2
1,
2
1(
12
2)
2
1,
2
1(
12
2[1)(
bits 0541.)]6
4,
6
2(
12
6)0,1(
12
4)1,0(
12
2[1)(
=+++−=
=++−=
HHHHTypeI
HHHPatronsI
Example
4412441222122212
[Hwee Tou Ng & Stuart Russell]
![Page 13: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/13.jpg)
Classification tree
[Criminisi et al, 2011]
![Page 14: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/14.jpg)
Use information gain to decide splits
[Criminisi et al, 2011]
![Page 15: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/15.jpg)
Advanced: Gaussian information gain to decide splits
[Criminisi et al, 2011]
![Page 16: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/16.jpg)
[Criminisi et al, 2011]
![Page 17: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/17.jpg)
Alternative node decisions
[Criminisi et al, 2011]
![Page 18: CPSC540 - Computer Science at UBCnando/540-2013/lectures/l8.pdf · February, 2013 University of British Columbia Decision trees. Outline of the lecture This lecture provides an introduction](https://reader036.vdocuments.net/reader036/viewer/2022071004/5fc1b9f5897923049a3f9394/html5/thumbnails/18.jpg)
Next lecture
The next lecture introduces random forests.