semeval-2017 task 11: end-user development using natural language

NLP & Semantic Computing Group

N L P

SemEval-2017 Task 11: End-User Development using Natural Language

Semeval, August 2017

Juliano Efson Sales, Siegfried Handschuh, Andre Freitas


Programming is a complex task

User effort associated with finding, understanding and integrating multiple

interfaces within a software development task.


https://medium.com/@keskkyla/https-medium-com-keskkyla-what-is-this-github-you-speak-of-dd457a29771

six thousand NEW people pushing code to github every day


The future of programming


and the role of NLP on it


KB of APIs Sem

an

tic

Pa

rsin

g

From this:

To this:

“Get the latest financial news about Google,

translate it to German and send it by email to

all stakeholders.”


Applications

• End-user programming user natural language.

• Semantic matching of software APIs and services.

Smart IDEs

Programming automation

Web service orchestration


Similar initiatives

• Robot commands (Pane and Myers, 2006), (Stenmark and Nugues, 2013), (Dukes, 2014)

• Operating system’s functions (Manaris and Dominick, 1993)

• Web services orchestration (Englmeier et al., 2006)

• Mobile programming by voice (Amos Azaria, 2016)

• Domain-specific natural language programming (Pane and Myers, 2006)


In comparison

• Open domain

• Large vocabulary

• Large KB of APIs (real world Web APIs)

• High lexical and semantic heterogeneity


From NL to n-ary predicate-argument structures

if (excitation_frequency > 20 & excitation_frequency < 20000) then simulation = valid


“Please convert US$ 475 to the Japanese currency and send this value to John Smith by SMS.”

SEQUENCE { ACTION: [ convert currency ] PARAMS: [US$ 475] - [(to) Japanese currency] ACTION: [ send sms ] PARAMS: [this value] - [(to) John Smith]

}

ACTION ENDPOINT: [action id] PARAMS:

from: “USD” to: “JPY” from amount: 475


…

KB Excerpt


Action KB • 3800 micro-services from Mashape (mashape.com)

• 1900 actions and triggers from the ifttt.com platform.


Test Collection Creation

• High-level task descriptions:

“You are arranging a meeting with some people in Andre’s office. Adamantios is coming for that meeting, but he does not know how to drive in Passau. Additionally, you do not know where the office is.”

1



• Convert into NL commands using crowd-sourcing:

Ask Andre for the address of his office;

Make a map from the university to it;

Send the map to Adamantios including driving directions.

2



• Data Curation

Many data quality issues

Correction of misspellings

3


Numbers about the Test Collection


Interpretation Challenges

• Lexical/conceptual approximation

“If someone reports a problem in GitHub, send the problem’s headline by Skype to John.”

any new issue (…)

issue title (…)


Syntactic and Ordering Variation

“If Manchester United wins, call me.”

“Get ready to call me in the case of victory of Manchester United.”


Co-reference and metonymy resolution

“If an issue is created, send its content to the Tech Manager.”

its -> return value of the issue

Tech Manager -> [email protected] according to the user KB


Evaluation

• 1. Resolved individual actions ignoring parameter values;

• 2. Resolved individual actions considering parameter values;

• 3. Resolved scenarios ignoring parameter values;

• 4. Resolved scenarios considering parameter values.

precision, recall

number of resolved scenarios


Participants

• Kubis et al. (2017) EUDAMU


Challenging examples

• Description of commands split into two sentences.

“Get the price of the book The Intelligent Investor. If it costs less than 25 Euros, buy it.”

where “25 Euros” is the parameter value of the action defined in the first sentence.



• Capturing actions with more specific/ fine-grained semantics.

“Once I have bet my running distance target of the week, set my current weight as 100 Kg in Fitbit.”

Selection of “Daily step goal achieved” instead of “Weekly distance goal reached”



“Pay the invoice that I will receive from [email protected] with my credit card”

Implicit trigger (receive email) into the "pay" action.

“Send to my mother‘s email the music Bachianas N.5 of Villa-Lobos from Youtube.” Implicit "search video on YouTube" action.

mailto:[email protected]




Post-mortem Analysis

• Task attracted a lot of attention but received one submissions.

Probable explanation: Does not fit easily as a machine learning problem.

• Parameter matching is significantly more difficult than discovering the action.

semeval-2017 task 11: end-user development using natural language

Data & Analytics