coms0018: practical-lab4 · 2019-11-11 · comsm0018: practical - lab4 - 2018/2019. debugging deep...
TRANSCRIPT
![Page 1: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/1.jpg)
COMS0018: PRACTICAL-Lab4
Dima [email protected]
Bristol University, Department of Computer ScienceBristol BS8 1UB, UK
October 26, 2018
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 2: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/2.jpg)
What else can we do?
I Data AugmentationI Assessing Dataset SizeI Debugging Strategies
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 3: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/3.jpg)
Data Augmentation
I Data augmentation is making the most of the training samples byintroducing variations of these samples to accommodate for requiredinvariances
I Why Data Augmentation?I Because it’s all about the size of your data –> More data for trainingI More importantly... to accommodate invariances
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 4: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/4.jpg)
Invariances in data
I A problem is invariant to a property when the problem remainsunchanged when transformations of a certain type are applied.
Problem Invariant to...Object Recognition translation, rotation, scaling,viewpointNumber plate recognition translation, scalingAction Recognition translation, rotation, scaling, viewpoint, speed
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 5: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/5.jpg)
Invariances in data
Problem Invariant to...Object Recognition translation, rotation, scaling, viewpoint
I How to provide invariance? → artificially augment for:I Translation: shifts – luckily CNNs do that for usI Rotation: rotationsI Scaling: croppingsI Viewpoint: Minor - affine transformations, otherwise :-( collect more data!
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 6: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/6.jpg)
Invariances in data
I Other invariances:I invariance to random noiseI invariance to occlusionI invariance to lighting conditionsI invariance to time of year!? — Generative!
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 7: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/7.jpg)
Data Augmentation
I Data augmentation for invariances existed before deep learning
Ozuysal et al (2010). Fast Keypoint Recognition Using Random Ferns. TPAMI.
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 8: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/8.jpg)
Data Augmentation
I Why can’t current deep learning methods do that automatically for us?
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 9: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/9.jpg)
But... Do I need more Data?
I There is a balance between the expense of collecting labelled dataand refining the method
I How are you performing on your training data?I poorly → your algorithm needs work, you are not making the most of the
data you haveI quite well → How are you performing on your test data?
I poorly → there is little you can do, collect more dataI quite well → you’re done! Find a more interesting problem
I I collected more data but things did not improve?I Think about the quality of your dataI Think about the quality of your labelsI → fix and start over
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 10: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/10.jpg)
Debugging Deep Learning Algorithms
I When a general machine learning code performs poorly, includingdeep learning code, it is very tricky to decide whether that is a bug inthe code or a problem in the algorithm
I Compiling correctly and getting numbers out is not an indication ofcorrectness
I We do not know what the "correct" implementation will give in terms ofaccuracy, that is in fact what we wish to discover
I Careful debugging is thus a must
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 11: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/11.jpg)
What is your baseline performance?
I Remember: you cannot perform worse than the baseline!I What can an algorithm that makes decisions by “chance” do?I For a binary classifier, your baseline is 50%I For a classifier into N balanced classes, your baseline is 1
N%
I For a classifier with unbalanced classes??
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 12: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/12.jpg)
Ex. of Baselines - EPIC-Kitchens 2018
I https://epic-kitchens.github.io
I https://youtu.be/Dj6Y3H0ubDw
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 13: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/13.jpg)
Ex. of Baselines - EPIC-Kitchens 2018
D Damen, H Doughty, GM Farinella, S Fidler, A Furnari, E Kazakos, D Moltisanti, J Munro, T Perrett, W Price, M Wray (2018). Scaling EgocentricVision: The EPIC-KITCHENS Dataset. European Conference on Computer Vision (ECCV)
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 14: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/14.jpg)
Ex. of Baselines - EPIC-Kitchens 2018
D Damen, H Doughty, GM Farinella, S Fidler, A Furnari, E Kazakos, D Moltisanti, J Munro, T Perrett, W Price, M Wray (2018). Scaling EgocentricVision: The EPIC-KITCHENS Dataset. European Conference on Computer Vision (ECCV)
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 15: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/15.jpg)
Debugging Strategies
I Mickey Mouse Examples - test your solution on small tests that youknow the outcome for
I Evaluate the performance of your building blocks in isolationI Monitor the model in actionI Look at failure cases (qualitative assessment)
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 16: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/16.jpg)
Debugging Strategies
“Directly observing the machine learning model performing its task willhelp to determine whether the quantitative performance numbers itachieves seem reasonable”
Goodfellow et al, p432
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 17: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/17.jpg)
Debugging Strategies
“Evaluation bugs can be some of the most devastating bugs because theycan mislead you into believing your system is performing well when it isnot”
Goodfellow et al, p432
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019
![Page 18: COMS0018: PRACTICAL-Lab4 · 2019-11-11 · COMSM0018: PRACTICAL - Lab4 - 2018/2019. Debugging Deep Learning Algorithms I When a general machine learning code performs poorly, including](https://reader033.vdocuments.net/reader033/viewer/2022041823/5e5f9f8c976ed328b24b506b/html5/thumbnails/18.jpg)
And now....
READY....
STEADY....
GO...
Dima [email protected]
COMSM0018: PRACTICAL - Lab4 - 2018/2019