semi-supervised knowledge transfer for deep learning from ...semi-supervised knowledge transfer for...

Semi-Supervised Knowledge Transfer For Deep LearningFrom Private Training Data

Nicolas Papernot1 Martn Abadi2 lfar Erlingsson2 IanGoodfellow2 Kunal Talwar 2

1Pennsylvania State University

2Google Brain

ICLR, 2017Presenter: Xueying Bai

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 1 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 2 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 3 /

30

Challenge of Learning from Private Data

Learning algorithm should protect the privacy of user’s private trainingdata (eg. private photographs).However, there are some problems revealed when learning from privatedata using traditional machine learning methods.

Some examples are implicitly memorized in the model.Experiments on deep neural network show this phenomenon.

Attacks can recover the sensitive training data from models.For example, Fredrikson used output probabilities of a computervision classifier to reveal individual faces from the training data.

The goal of this paper is to preserve the privacy of training data whenlearning classifiers.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 4 /

30

Attack

Two types of attacks:

Black-box Adversary: Recover private training data through modelquering, without knowing the parameters and the structure of themodel.

White-box Adversary: Know the model structure and parameters.

Strong attack assumption in this paper: Adversary can makeunbounded number of queries (black-box); adversary can access tomodel internals (white box).

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 5 /

30

Privacy

Strategy which has privacy guarantee:

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 6 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 7 /

30

Overview

Approach. First, an ensemble of teacher models is trained on disjointsubsets of the sensitive data. Then, using auxiliary, unlabelednon-sensitive data, a student model is trained on the aggregateoutput of the ensemble.

Privacy analysis of the approach.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 8 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 9 /

30

Data Partitioning and Teachers

Partition the data into n disjoint sets, then train a model separately oneach set.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 10 /

30

Aggregation

If most teachers agree on the label, it does not depend on specificpartitions, so the privacy cost is small.

If two majority classes have close vote counts, the disagreement mayreveal private information.

Add random noise to the vote counts to introduce ambiguity.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 11 /

30

Aggregation

j ∈ [m] is the class label, i ∈ [n] is the ith teacher, −→x is the input.

Count votes: nj(−→x ) = |{i : i ∈ [n], fi (

−→x ) = j}|Laplacian noise: Lap( 1

γ )

Output label: f (x) = arg maxj{nj(−→x ) + Lap( 1

γ )}

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 12 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 13 /

30

Reason to Use the Student Model

There are threats in teacher ensemble:

Each prediction increases privacy cost. For each query to theteacher, the aggregated result will reveal information of the data. Soneed the student model.

Inspection of internals may reveal private data. Later analysiswill prove the privacy guarantee.

The student model is the one deployed, in lieu of the teacher ensemble.

Student finishes training after fixed number of queries to the teacher.Then no matter how many queries the user gives to the student, theprivacy loss is fixed.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 14 /

30

The Student Model

The student model is trained on unlabeled public data. To dosemi-supervised learning, part of the data are labeled using the teacheraggregation result.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 15 /

30

Train the Student with GANs

The discriminator is extended to multi-class classifier with m + 1classes (m classes plus a generated class).Only trained discriminator will be used after training.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 16 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 17 /

30

Differential Privacy Preliminaries

Differential privacy is a strong standard. It’s defined using pairs of adjacentdatabases: d and d ′, which only differ by only 1 training example.

Definition 1. A randomized mechanism M with domain D and rangeR satisfies (ε, δ)-differential privacy if for any two adjacent inputd , d ′ ∈ D and for any subset of outputs S ⊂ R it holds that:

Pr[M(d) ∈ S ] ≤ eε Pr[M(d ′) ∈ S ] + δ

Smaller ε, stronger privacy guarantee. δ is the tolerant bias.

Definition 2. Let M : D → R be a randomized mechanism and d , d ′

a pair of adjacent databases. Let aux denote an auxiliary input. Foran outcome o ∈ R, the privacy loss at o is defined as:

c(o;M, aux , d , d ′)∆= log

Pr[M(aux , d) = o]

Pr[M(aux , d ′) = o]

The privacy random variable is defined as:

C (M, aux , d , d ′)∆= c(M(d);M, aux , d , d ′)

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 18 /

30

The Moments Accountant

A natural way to bound the approach’s privacy loss is to first bound theprivacy cost of each label queried by the student, and then use thecomposition theorem to derive the total cost of training the student. Soit’s better to track each step’s privacy cost.

Definition 3. Let M : D → R be a randomized mechanism and d , d ′

a pair of adjacent databases. Let aux denote the auxiliary input. Themoments accountant is defined as:

αM(λ)∆= max

aux ,d ,d ′αM(λ; aux , d , d ′)

αM(λ; aux , d , d ′)∆= log E [exp(λC (M, aux , d , d ′))]

αM(λ; aux , d , d ′) is a moment generating function of the privacy lossrandom variable.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 19 /

30

Composability and Tail bound

This theorem enables adding bounds over continuous steps, andderive an (ε, δ) guarantee for the final α.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 20 /

30

Privacy Analysis of PATE

These theorems enable the bound of specific moments.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 21 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 22 /

30

Settimgs

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 23 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 24 /

30

Training an Ensemble of Teachers and Privacy

Correct label: 1γ should be small.

Strong privacy: γ should be small.

Large gap will result to strong privacy. When increasing teachers, thegap will increase.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 25 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 26 /

30

Semi-supervised Training of the Student

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 27 /

30

Outline

1 IntroductionMotivationOverview

2 Model (PATE)Train the Ensemble of TeachersSemi-supervised Knowledge Transfer from an Ensemble to a Stuednt

3 Privacy Analysis of the ApproachPrivacy Analysis of the Approach

4 EvaluationSettimgsTraining an Ensemble of Teachers and PrivacySemi-supervised Training of the StudentComparison with Other Methods of Learning with Privacy

5 Conclusions

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 28 /

30

Comparison

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 29 /

30

Conclusions

Main contributions in this paper:

Combine semi-supervised learning with precise, data-dependentprivacy analysis.

Establish a precise guarantee of training data privacy.

The model is independent of the learning algorithm for either teachersor students, very generic.

Experiments show it can achieve comparable result with thestate-of-the-art under the privacy guarantee.

Nicolas Papernot, Martn Abadi, lfar Erlingsson, Ian Goodfellow, Kunal Talwar (Pennsylvania State University)Semi-Supervised Knowledge Transfer For Deep Learning From Private Training DataICLR, 2017 Presenter: Xueying Bai 30 /

30