user input and interactions on microsoft research esl assistant
DESCRIPTION
User Input and Interactions on Microsoft Research ESL Assistant. Claudia Leacock, Butler Hill Group Michael Gamon , Microsoft Research Chris Brockett, Microsoft Research. ... and William B. Dolan, Jianfeng Gao , Dmitriy Belenko , Lucy Vanderwende (Microsoft Research) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/1.jpg)
User Input and Interactions on Microsoft Research ESL Assistant
Claudia Leacock, Butler Hill GroupMichael Gamon, Microsoft ResearchChris Brockett, Microsoft Research
![Page 2: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/2.jpg)
... andWilliam B. Dolan, Jianfeng Gao, Dmitriy
Belenko, Lucy Vanderwende (Microsoft Research)
Alexandre Klementiev (University of Illinois at Urbana Champaign)
![Page 3: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/3.jpg)
Outline
• Who is using it and how often• How users are interacting with the system• Does it help the users to improve their writing?
![Page 4: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/4.jpg)
Most frequent errors made by East Asian non-native speakers
Noun Related: Articles (inclusion & choice), Noun Number, Noun of Noun• I think it’s *a/the best way to resolve issues like this.• Conversion always takes a lot of *efforts/effort.• Please send the *feedback of customer/customer feedback to me by mail.
Preposition Related: inclusion & choice• It seems ok and I did not pay much attention *on/to it. • I should *to ask/ask a rhetorical question.
Verb Related: Gerund/Infinitive Confusion, Auxiliary Verb Error, Verb Formation Errors (6), Cognate/ Verb confusion, Irregular Verbs• On Saturday, I with my classmate went *eating/to eat.• Hope you will *happy/be happy in Taiwan.• I *teached/taught him all the things I know.
Adjective Related: Adjective Confusion (4), Adjective Order• She is very *interesting/interested in the problem.• So *Korea/Korean Government is intensely fostering trade.
4
![Page 5: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/5.jpg)
User Interface Deployed 6/2008
5
![Page 6: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/6.jpg)
Page Views per Week
7/20/08
8/ 3/08
8/17/0
8
8/31/0
8
9/14/0
8
9/28/0
8
10/12/08
10/26/08
11/ 9/08
11/23/08
12/ 7/08
12/21/08
1/ 4/09
1/18/0
9
2/ 1/09
2/15/0
9
3/ 1/09
3/15/0
9
3/29/0
9
4/12/0
9
4/26/0
9
5/10/0
90
5,000
10,000
15,000
20,000
25,000
Page Views per Week
![Page 7: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/7.jpg)
User Locations
1. China 59,276 17.9% 10. Japan 5,941 1.8%2. United States 55,104 16.6% 11. Spain 5,924 1.8%
3. Taiwan 47,159 14.2% 12. United Kingdom 5,828 1.8%4. Korea - South 18,730 5.6% 13. Russian Federation 5,454 1.6%
5. Hong Kong 14,259 4.3% 14. France 3,971 1.2%6. Brazil 8,444 2.5% 15. Saudi Arabia 3,893 1.2%
7. Germany 8,219 2.5% 16. Mexico 3,878 1.2%
8. Canada 7,634 2.3% 17. Netherlands 3,330 1.0%
9. Italy 6,880 2.1% 18. Thailand 3,207 1.0%
![Page 8: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/8.jpg)
Repeat Users
once only 2 times or more 3 times or more 4 times or more 5 times or more0
10
20
30
40
50
60
70
80
90
100
Return frequencype
rcen
tage
of t
otal
visi
ts
8
![Page 9: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/9.jpg)
Frequent Users (4/21/09).
9
adj2%
verb10%
prep27%
noun61%
Frequent Users 854
Sessions 8,339
Session-Unique Sentences 66,765
Grammatical Error Flags 22,542
![Page 10: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/10.jpg)
Collected Data (4/21/09)
Email49%
Non-technical25%
Technical17%
Other4%
Unrelated5%
Writing Domains: By Number of SentencesWriting Domains: By Number of Sentences
10
![Page 11: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/11.jpg)
User Interaction 1:Responses to “Tell us what you think!”
Some users wrote: Other users wrote:
“This is awesome! It works really well.” “It didn’t work at all.”
“I found the tool very useful.” “I hate it.”
“Great tool in general – thank you!!!!!!!” “Terrible job.”
“I love the feature where it looks for a phrase in web pages.”
“The microsoft search results below confuses me.”
Bug reports: “When I first opened it, it wouldn’t let me type in any characters at all.”
“What wearies me is the message ‘Server is temporarily unavailable’.”
Suggestions: “There should be some indication that the check is done.”
“I would like a filter for business and personal use.”
![Page 12: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/12.jpg)
Users Examine 83% of SuggestionsAccept
42%
Trigger web search but don't accept
28%
Look at suggestion but not trigger web
search31%
12
Conclusion: A significant number of users are inspecting the suggested rewrites and making a deliberate choice to accept it or not accept it.
Inspect >18.3K Flags to Accept 7.6K
![Page 13: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/13.jpg)
Do users make the right choices?
Evaluated ~900 complete user sessions: 6K flags1. Calculate system performance for ALL
suggestions.2. Calculate performance for ONLY
suggestions that were accepted.3. Compare ratios of good and bad flags.
13
![Page 14: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/14.jpg)
Evaluation CategoriesEvaluation SubEval Description
Good Correct Flag The correction fixes a problem in the user input.
Neutral
Both Good The suggestion is a legitimate alternative of a well-formed original input. Ex: I like working/to work.
MisdiagnosisThe original input contained an error but the suggested rewrite neither improves nor further degrades the user input. Ex: If you have fail machine on hand.
Non-ascii A non-ascii or text processing mark-up character is in the immediate context. (Only applies to user data)
Bad False Flag The suggestion resulted in an error or would otherwise lead to a degradation over the original user input.
14
![Page 15: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/15.jpg)
Are users accepting good suggestions?All significant in the Wilcoxin’s signed-ranks test.
Noun-related Prep-related Verb-related Adj-related
15
good56%neut
28%
bad
16%
All Suggestions
good63%
neut
26%
bad11%
Accepted
good37%
neut
39%
bad24%
All Suggestions
good45%
neut
42%
bad13%
Accepted
good62%
neut32%
bad6%
All Suggestions
good72%
neut25%
bad3%
Accepted
good45%
neut 32%
bad23%
All Suggestions
good63%
neut28%
bad9%
Accepted
![Page 16: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/16.jpg)
By Domain: All significant in the Wilcoxin’s signed-ranks test.
Email Non-technical Technical
16
good53%neutral
32%
bad15%
Suggestions
good62%
neutral28%
bad9%
Accepted
good56%
neutral32%
bad12%
Suggestions
good56%
neutral34%
bad10%
Accepted
good38%
neutral28%
bad34%
Suggestions
good51%
neutral29%
bad19%
Accepted
![Page 17: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/17.jpg)
What do users do with neutral flags?
Both OK15%
Misdiagnosis78%
non-ascii7%
Neutral Categories
![Page 18: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/18.jpg)
• I don't know that you knew or not , this early morning i got a from head office ... – suggestion: delete “from”I don't know that you knew or not , this early morning I heard from the head
office ...• Please play with the software and Friday I will be by to work with any questions
you may regarding it.– suggestion: regardingregardPlease play with the software and Friday I will be by to work with any questions
you may have regarding it.
18
Neutral Flags not accepted but sentence edited to produce no flag
From 1,349 sentences with neutral flags found 215 subsequently submitted “similar” strings with no error flag.Users not accept suggestion but did something ELSE to make the flag go away.
![Page 19: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/19.jpg)
Users improve 40% of the time
19
Typed in suggestion44%
Revise and improve40%
Revise and not improve16%
Not Accept Suggestion but Revise Sentence
Identifying the location of an error can help the user.
![Page 20: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/20.jpg)
Conclusions
• Traffic: There is an interest in ESL proofing tools• Even current state-of-the-art error correction can be
useful for ELLs:Users do not accept proposed corrections blindly – they are
selective in their behaviorUsers make informed choices – they can distinguish correct
suggestions from incorrect ones Sometimes just identifying the location of an error enables
the users to repair the problem themselves
20
![Page 21: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/21.jpg)
New Interface
![Page 22: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/22.jpg)
![Page 23: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/23.jpg)
![Page 24: User Input and Interactions on Microsoft Research ESL Assistant](https://reader035.vdocuments.net/reader035/viewer/2022062815/568168b0550346895ddf7168/html5/thumbnails/24.jpg)
www.eslassistant.com