language’technologies’for …...2015/12/15 · template’ideas’ ’ “as an internet user...
TRANSCRIPT
![Page 1: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/1.jpg)
Language Technologies for Understanding Law, Politics,
and Public Policy
Thesis Defense
December 15, 2015 William P. Li
1
![Page 2: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/2.jpg)
Language Technologies for…
2
Understanding Government Processes
Understanding Public Speech
![Page 3: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/3.jpg)
open government datasets
3
Research Approach
![Page 4: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/4.jpg)
open government datasets
analyses & visualizations to infer
origins of text
4
Research Approach
![Page 5: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/5.jpg)
open government datasets
analyses & visualizations to infer
origins of text
5
Research Approach
better understanding of government
![Page 6: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/6.jpg)
Research Projects U.S. Supreme Court Authorship Attribution (W. Li, P. Azar, D. Larochelle, P. Hill, J. Cox, R. Berwick,
A. Lo, Stanford Technology Law Review 2013)
Probabilistic Text Reuse (in preparation)
Law is Code (W. Li, P. Azar, D. Larochelle, P. Hill, A. Lo, Journal of Business and Technology Law 2014)
6
![Page 7: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/7.jpg)
Research Projects U.S. Supreme Court Authorship Attribution (W. Li, P. Azar, D. Larochelle, P. Hill, J. Cox, R. Berwick,
A. Lo, Stanford Technology Law Review 2013)
Probabilistic Text Reuse (in preparation)
Law is Code (W. Li, P. Azar, D. Larochelle, P. Hill, A. Lo, Journal of Business and Technology Law 2014)
7
![Page 8: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/8.jpg)
Thesis Contributions
8
Inferring hidden origins of text à New insights into law and policy
Supreme Court authorship attribution (supervised learning)
Probabilistic Text Reuse for understanding large document collections (unsupervised learning)
![Page 9: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/9.jpg)
• Motivation
• Data and Model
• Classification Performance
• Insights on Supreme Court Authorship
9
Supreme Court Authorship Attribution
![Page 10: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/10.jpg)
Motivation: Judicial Accountability
10
Who is writing Supreme Court opinions?
![Page 11: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/11.jpg)
Supreme Court Authorship
11
![Page 12: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/12.jpg)
Supreme Court Authorship
Did the Chief Justice write both the majority and dissenting opinions?
12
Chief Justice John Roberts
![Page 13: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/13.jpg)
Unsigned Per Curiam Opinions • Definition: “by decision of the Court in unanimous
agreement”
• In practice: not always unanimous! – Furman v. Georgia (1972) (5 concurrences, 4 dissents)
– Buckley v. Valeo (1976) (5 concurrences/dissents)
– Bush v. Gore (2000) (1 concurrence, 4 dissents)
– Roberts Court (since 2005): 65 unsigned opinions
• 39 states elect their judges – Texas Supreme Court: 41% of all opinions were unsigned
13
![Page 14: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/14.jpg)
Unsigned Per Curiam Opinions
14
“The Judges holding their offices for life are under two responsibilities only: 1. Impeachment. 2. Individual reputation. But this practice [per curiam opinions] completely withdraws them from both. For nobody knows what opinion any individual member gave in any case… The practice is certainly convenient for the lazy, the modest & the incompetent.”
- Thomas Jefferson, June 22, 1822
![Page 15: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/15.jpg)
Motivation
15
Can we determine the authorship:
• Cases with disputed authorship (Obamacare)?
• Unsigned per curiam (“by the court”) opinions?
![Page 16: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/16.jpg)
Data and Model
16
Authorship Attribution
![Page 17: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/17.jpg)
Data and Model
17
• 568 signed opinions from 2005-‐2011
• Choose unigrams, bigrams, and trigrams by information gain – e.g. “Furthermore,”, “troubling”, “undisturbed”,
“evidently”, “That would”, “assert that”, “the premise that”, “is apparent that”
• Train a nine-‐class MaxEnt classifier (Ratnaparkhi 1997)
![Page 18: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/18.jpg)
18
Highly Weighted N-‐Grams
![Page 19: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/19.jpg)
Results
• Train models on 451 opinions (80%)
• Test on 117 opinions (20%)
19
![Page 20: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/20.jpg)
Results
• Train models on 451 opinions (80%)
• Test on 117 opinions (20%)
20
95/117 (81.2%)
![Page 21: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/21.jpg)
Results ● Prediction Accuracy: 81.2%
(95/117 held-‐out test opinions)
21
max(probability) = 0.97
max(probability)
![Page 22: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/22.jpg)
Results ● Prediction Accuracy: 81.2%
(95/117 held-‐out test opinions)
22
threshold = 0.5
![Page 23: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/23.jpg)
Results ● Prediction Accuracy: 81.2%
(95/117 held-‐out test opinions)
23
threshold = 0.5
![Page 24: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/24.jpg)
Results ● Should we trust the prediction?
24
threshold = 0.5
![Page 25: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/25.jpg)
Results
● Obamacare:
Majority Dissent
25
![Page 26: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/26.jpg)
Results: Per Curiam Opinions
• 65 per curiam (“by the court”) opinions since 2005
26
![Page 27: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/27.jpg)
Results: Per Curiam Opinions
• 65 per curiam (“by the court”) opinions since 2005
27
7/65 written by liberal wing (4/9 justices)
![Page 28: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/28.jpg)
Results: What about the clerks?
28
● Clerks serve one-‐year terms
Partition Accuracy
Random
Year-‐based
![Page 29: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/29.jpg)
Results: What about the clerks?
29
● Clerks serve one-‐year terms
Partition Accuracy
Random 78.9% (325/412)
Year-‐based
![Page 30: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/30.jpg)
Results: What about the clerks?
30
● Clerks serve one-‐year terms
Partition Accuracy
Random 78.9% (325/412)
Year-‐based 74.8% (308/412)
![Page 31: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/31.jpg)
Results: June 2013 Predictions
• Performance on June 2013 opinions (DOMA, Proposition 8, Voting Rights Act)
31
7/9 (77.8%)
![Page 32: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/32.jpg)
Insights
• Supreme Court Authorship: – Prediction: Scalia wrote the Obamacare dissent
– Conservative justices appear to be writing more per curiam opinions
32
![Page 33: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/33.jpg)
Thesis Contributions
33
Inferring hidden structures in government text à new insights into law and policy
Supreme Court authorship attribution (supervised learning)
Probabilistic Text Reuse for understanding large document collections (unsupervised learning)
![Page 34: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/34.jpg)
Probabilistic Text Reuse (PTR)
Outline
• Motivation: Government text datasets
• Model
• Inference • Results: 800,000 FCC net neutrality comments
34
![Page 35: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/35.jpg)
PTR: Motivation #1 Repeated text occurs widely
in government documents
– Legislative bills (Wilkerson, Stramp, Smith 2015)
– 19th century American newspapers (Smith, Cordell, Dillon 2013)
– Public statements from Members of Congress (Lin, Margolin, Lazer 2015)
35
![Page 36: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/36.jpg)
Text Reuse in Congress
Enacted October 3, 2008
36
Troubled Asset Relief Program
![Page 37: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/37.jpg)
Text Reuse in Congress
Enacted October 3, 2008
37
Failed House Vote (Dow Jones
drops 778 points) September 29, 2008
Troubled Asset Relief Program
![Page 38: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/38.jpg)
Text Reuse in Congress
Enacted October 3, 2008
38
Failed House Vote (Dow Jones
drops 778 points) September 29, 2008
Troubled Asset Relief Program
![Page 39: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/39.jpg)
Text Reuse in Congress
Enacted October 3, 2008
39
Failed House Vote (Dow Jones
drops 778 points) September 29, 2008
Troubled Asset Relief Program
![Page 40: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/40.jpg)
Text Reuse in Congress
Enacted October 3, 2008
40
Failed House Vote (Dow Jones
drops 778 points) September 29, 2008
Troubled Asset Relief Program
![Page 41: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/41.jpg)
Text Reuse in Congress
Enacted October 3, 2008
41
Troubled Asset Relief Program
Failed House Vote (Dow Jones
drops 778 points) September 29, 2008
Tax Extenders and Alternative Minimum Tax Relief Act of 2008
Reauthorization of the Secure Rural Schools and Community Self-Determination Act of 2000
Energy Improvement and Extension Act of 2008
![Page 42: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/42.jpg)
Text Reuse in Public Comments
• Proposed net neutrality regulations
• 800,000 public comments (May to July 2014)
– 101 million words, 126 words/comment
42
![Page 43: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/43.jpg)
Template Ideas “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC to reclassify broadband Internet access as a telecommunications service, and save Net Neutrality. In addition, the FCC should reject the proposed rules that would allow Internet service providers to divide the Internet into fast lanes for wealthy corporations and slow lanes for the rest of us.”
(93,711 comments)
43
![Page 44: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/44.jpg)
• What are people saying? – Most common ideas and comments (templates)
– Variations of ideas (different wording)
– Less-‐common voices
44
Text Reuse in Public Comments
![Page 45: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/45.jpg)
PTR: Motivation #2
• Current models of text corpora fail to capture text reuse – Model mismatch
– Limitations on interpretability and usefulness
45
![Page 46: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/46.jpg)
topic mixtures
Existing Work: Probabilistic Topic Modeling
46
raw documents
bag of words
document clusters
key phrases
• Text passages à bags of words à text passages
Probabilistic Text Reuse (PTR)
PLSA (Hofmann 1999) LDA (Blei 2003)
![Page 47: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/47.jpg)
Probabilistic Text Reuse (PTR)
Represent documents with
reused text sequences
+
Probabilistic modeling
47
![Page 48: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/48.jpg)
PTR: Intuition Corpus:
• D documents (sequences of words)
Text generators:
• Background text model, π
• K ideas (text sequence generators)
48
aaa bbb ccc ddd eee
π
fff ggg hhh
iii jjj kkk lll
Generative process: 1. Choose # of
partitions, Z
![Page 49: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/49.jpg)
PTR: Intuition Corpus:
• D documents (sequences of words)
Text generators:
• Background text model, π
• K ideas (text sequence generators)
49
aaa bbb ccc ddd eee
π
fff ggg hhh
iii jjj kkk lll
Generative process: 1. Choose # of
partitions, Z 2. Assign each
partition to one of K+1 text generators
![Page 50: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/50.jpg)
PTR: Intuition Corpus:
• D documents (sequences of words)
Text generators:
• Background text model, π
• K ideas (text sequence generators)
50
aaa bbb ccc ddd eee
π
fff ggg hhh
iii jjj kkk lll
Generative process: 1. Choose # of
partitions, Z 2. Assign each
partition to one of K+1 text generators
3. Generate word
sequence from text generator
iii jjj kkk lll
π
fff zzz hhh
aaa ccc ddd eee yyy
π π π π π
![Page 51: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/51.jpg)
PTR: Intuition Corpus:
• D documents (sequences of words)
Text generators:
• Background text model, π
• K ideas (text sequence generators)
51
aaa bbb ccc ddd eee
π
fff ggg hhh
iii jjj kkk lll
Generative process: 1. Choose # of
partitions, Z 2. Assign each
partition to one of K+1 text generators
3. Generate word
sequence from text generator
fff zzz hhh
iii jjj kkk lll
π
aaa ccc ddd eee
yyy π π π π π
![Page 52: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/52.jpg)
PTR: Model
52
Documents, D
Ideas, K
Partitions, Z Assignments, A
Objective Function:
PTR
![Page 53: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/53.jpg)
PTR: Model
53
Objective Function (document n, partition m):
idea model
partition model
assignment model
text passage model
![Page 54: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/54.jpg)
Idea Model: P(K)
54
For each idea, k: 1. Choose length, L, from uniform distribution 2. Choose words, w, from background text model, π
![Page 55: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/55.jpg)
Partition Model: P(Z)
55
For each document, n: 1. Choose number of partitions, m, from uniform
distribution
![Page 56: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/56.jpg)
Assignment Model: P(A)
56
For each partition, z: 1. Choose an assignment, a, from multinomial
distribution over ideas • Learn θ during inference
![Page 57: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/57.jpg)
Text Passage Model:
• Background text: unigram language model
• Ideas: Probabilistic finite state transducers (PFST)
57
Example PFST for idea “w0 w1 w2”:
![Page 58: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/58.jpg)
PTR: Model Corpus:
• D documents (sequences of words)
Text generators:
• Background text model, π
• P(K)
58
aaa bbb ccc ddd eee
π
fff ggg hhh
iii jjj kkk lll
Generative process: 1. P(Z) 2. P(A) 3. P(dz|ka)
![Page 59: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/59.jpg)
PTR: Model
59
fff zzz hhh
iii jjj kkk lll
π
aaa ccc ddd eee
yyy
π π π π π π π π
π π π π π
aaa bbb ccc ddd eee
π
fff ggg hhh
iii jjj kkk lll
Corpus:
• D documents (sequences of words)
Text generators:
• Background text model, π
• P(K)
Generative process: 1. P(Z) 2. P(A) 3. P(dz|ka)
![Page 60: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/60.jpg)
Parameter Estimation
• Goal: maximize P(K,Z,A|D) • Coordinate descent: Iteratively optimize P(d|k), P(A), P(K), P(Z)
1. P(d|k) and P(A): assign pre-‐partitioned passages to best idea (or background)
2. P(K): re-‐estimate ideas given assigned partitions 3. P(Z): Propose concatenations and merges of ideas
60
![Page 61: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/61.jpg)
Parameter Estimation: Assignments
For each passage:
1. Find highest-‐ranked candidate ideas
2. Compute PFST probabilities for top candidates (forward algorithm)
3. Choose assignment that maximizes probability
61
fff zzz hhh
aaa bbb ccc ddd eee
π
fff ggg hhh
iii jjj kkk lll
mmm nnn ooo
![Page 62: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/62.jpg)
Parameter Estimation: Assignments
For each passage:
1. Find highest-‐ranked candidate ideas
2. Compute PFST probabilities for top candidates (forward algorithm)
3. Choose assignment that maximizes probability
62
fff zzz hhh
aaa bbb ccc ddd eee
π
fff ggg hhh
iii jjj kkk lll
mmm nnn ooo
![Page 63: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/63.jpg)
Parameter Estimation: Ideas
• Find idea that maximizes sum of transduction probabilities (Steiner consensus string) – An NP-‐hard problem
• Approximation: Find best idea among current passages
63
fff zzz hhh
fff ggg hhh
fff zzz yyy hhh
![Page 64: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/64.jpg)
Parameter Estimation: Partitions
• Proposals: – Concatenate ideas if they are frequently adjacent to each other
– Merge most similar ideas
• Check if proposal increases log-‐likelihood
64
![Page 65: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/65.jpg)
Evaluation
Dataset:
• FCC Net Neutrality comments
• 800,000 public comments (June-‐August 2014)
– # unique comments: 650,300
– # words/comment: mean = 131
65
![Page 66: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/66.jpg)
Evaluation • Use PTR to find:
– Ideas with variations in wording
– Template ideas
– Document structure
• Infer partitions and assignments on held-‐out test set (160K documents)
66
![Page 67: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/67.jpg)
Results: Word Variations
67
244 comments, “keep the internet a level playing field”
90 variations:keep the internet an even playing fieldkeep the internet an open playing fieldkeep it a level playing fieldkeep net neutrality keep a level playing fieldkeep the net a level playing fieldkeep the internet a level playing field that it iskeep the internet a fair playing fieldkeep the net on a level playing fieldkeep the internet open as a fair playing field keep the media landscape a level playing field…
![Page 68: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/68.jpg)
Results: Word Variations
68
168 comments, “the internet should be open”
21 variations:the internet should be publicly ownedthe internet should never be regulatedthe internet should be dividedthe internet should be open and neutralthe internet should be open to everyone equallythe internet should be an open platformthe internet should be equal opportunitythe internet should be taxedthe internet should be fairthe internet should absolutely be open…
![Page 69: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/69.jpg)
Results: Top Ideas “Net neutrality is the First Amendment of the Internet, the principle that Internet service providers (ISPs) treat all data equally. As an Internet user, net neutrality is vitally important to me. The FCC should use its Title II authority to protect it…”
(89,989 comments)
69
![Page 70: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/70.jpg)
Results: Top Ideas “As an American citizen, I wanted to voice my opposition to the FCC’s crippling new regulations that would put federal bureaucrats in charge of internet freedom, and urge you to stop these regulations before they’re enacted...Please stop the FCC’s dangerous new regulations, and protect the future of internet freedom here in America.”
(9331 comments)
70
![Page 71: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/71.jpg)
Results: Document Structure
71
![Page 72: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/72.jpg)
Results: Document Structure
72
“Dear FCC,My name is Steve Roberts and I live in West Lafayette, IN. Net neutrality, the principle that Internet service providers (ISPs) treat all data that travels over their networks equally, is important to me because without it…
“…ISPs could have too much power to determine my Internet experience by providing better access to some services but not others.” (41,524 comments)
“…users may have fewer options and a less diverse Internet.” (28,173 comments)
![Page 73: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/73.jpg)
Results: Document Structure
73
“A pay-to-play Internet worries me because...“…new, innovative services that can't afford expensive fees for better service will be less likely to succeed.” (36,509 comments)
“…ISPs could act as the gatekeepers to their subscribers.” (32,252 comments)
![Page 74: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/74.jpg)
Results: Idea Text Variations
74
The Internet is important to me because, as a IT professional, I need to know…
The Internet is important to me because, as a videographer & editor, I need to know…
The Internet is important to me because, as an artist and musician, I need to know…
The Internet is important to me because, as a business manager, I need to know…
![Page 75: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/75.jpg)
Results: Document Structure
• Total EFF template comments: ~68,000
75
“less diverse Internet”
“ISPs too much power”
“unaffordable expensive fees”
11,107 24,317
“pay-‐to-‐play Internet” 16,283 15,648
• Comments with >50 additional words: ~25,000
![Page 76: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/76.jpg)
Results: Document Structure
76
Alex Jones, Manhattan, NY:The internet's future is important to me because I am a Science student. By the time I enter the Science industry I hope to be able to use the internet to collaborate with out Scientists and teams around the world. If providers are allowed to choosewhich data is more important it may lead to the internet being used much less. This would remove any hopes of collaboration in the Science industry.
Frank Murphy, Colonia, NJ:Look.. Long-story-short, if it ain't broke, don't fix it. If you want to improve anything why don't we start by upgrading infrastructure, giving lower-income and impoverished areas access to free Internet so that they may learn ways to improvetheir situation. Instead of catering to the powerful why don't we grant all citizens equal access to the pursuit of happiness as the founding American diplomats and their families intended.
Kathleen Chisholm, Oakland, CA:This country was founded on the idea of civil liberty, and the internet as it exists now is a great representation of how that ideal is fostered today. We have enough regulation in our everyday practices. The internet should remain exactly like it is; an open forum where everybody has an equal opportunity to have their voice heard.
![Page 77: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/77.jpg)
Comparison: Topic Modeling
77
Top words from top 10 topics: coherent but difficult to interpret • isps, use, slow, able, destroy• isps, important, new, services, better• communications, corporations, rules, reclassify,
federal• free, access, equal, corporations, open• common, carriers, reclassify, carrier, broadband• companies, small, businesses, business, innovation• people, corporations, right, corporate, government• pay, content, companies, access, speed• wealthy, save, user, addition, believes• comcast, like, verizon, cable, time
![Page 78: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/78.jpg)
78
Comparison: Topic Modeling
![Page 79: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/79.jpg)
topic mixtures
Existing Work: Probabilistic Topic Modeling
79
raw documents
bag of words
document clusters
key phrases
• Text passages à bags of words à text passages
Probabilistic Text Reuse (PTR)
PLSA (Hofmann 1999) LDA (Blei 2003)
![Page 80: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/80.jpg)
PTR: Quantitative Evaluation
80
log-‐likelihood/token (higher is better)
model training set test set
baseline unigram -‐6.88 -‐14.01
100-‐topic LDA -‐5.40 -‐12.44
PTR
![Page 81: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/81.jpg)
PTR: Quantitative Evaluation
• Percentage of comments with substantial non-‐idea text (>20 words): 37.4% – (Knight Foundation 2014 Report: 40%)
81
log-‐likelihood/token (higher is better)
model training set test set
baseline unigram -‐6.88 -‐14.01
100-‐topic LDA -‐5.40 -‐12.44
PTR -‐3.26 -‐10.24
![Page 82: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/82.jpg)
PTR: Summary • A novel probabilistic model for text reuse
– Latent partitions, assignments, and ideas (PFSTs)
• Useful for understanding large document collections (e.g. public speech) – Identifying common viewpoints, less-‐popular voices, and variations of ideas
82
![Page 83: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/83.jpg)
Measuring Public Speech
83
petitions
polls
public comments
elections
more diverse, harder to analyze
(numbers)
(text)
less diverse, easier to analyze
![Page 84: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/84.jpg)
Contributions Supreme Court authorship attribution (Supervised Learning) Probabilistic Text Reuse for understanding large document collections (Unsupervised Learning) Inferring hidden structures of government text à new insights into law and policy
84
![Page 85: Language’Technologies’for …...2015/12/15 · Template’Ideas’ ’ “As an Internet user who believes strongly in the importance of a free and open Internet, I urge the FCC](https://reader033.vdocuments.net/reader033/viewer/2022042210/5eaed4b89794aa74d7206f15/html5/thumbnails/85.jpg)
Language Technologies for Understanding Law, Politics,
and Public Policy
William P. Li
85