hyperparameter optimization 101
TRANSCRIPT
Hyperparameter Optimization 101Alexandra Johnson
Software Engineer, SigOpt
What are Hyperparameters?
Hyperparameters affect model performance
How Do I Find The Best Hyperparameters?
Step 1: Pick an Objective Metric
Classification models Accuracy
Regression models Root MSE
Caveat: Cross Validate to Prevent Overfitting
Cross Validation
4 5 60 1 2 3 7 8 9
4 5 6 7 8 90 1 2 3data
train validate metric
Cross Validation
4 5 60 1 2 3 7 8 9
4 5 6 7 8 90 1 2 3data
train
6 7 91 2 4 5 0 3 8train
7 8 90 2 3 6 1 4 5train
metric
metric
metric
K ti
mes validate
validate
validate
Grid Search Random Search Bayesian Optimization
Step 2: Pick an Optimization Strategy
Step 3: Evaluate N Times
N Times
What is the Best Hyperparameter Optimization Strategy?
Primary Consideration: How Good are the “Best” Hyperparameters?
“Best Found Value” Distributionsex
perim
ents
accuracy
Secondary Consideration: How Much Time Do You Have?
Number of Evaluations Required
Grid Search Random Search Bayesian Optimization
2 parameters 100 ?? 20-40
3 parameters 1,000 ?? 30-60
4 parameters 10,000 ?? 40-80
5 parameters 100,000 ?? 50-100
SigOptEasy-to-use REST API, R, Java, Python Clients
Ensemble of Bayesian optimization techniques
Free trial, academic discount, we’re hiring!
SigOpt Tutorial VideosVersus untuned models:
+315.2% accuracy with TensorFlow CNN
+49.2% accuracy with Xgboost + unsupervised features
