iso project 24617-2 semantic annotation framework, part 2: dialogue acts editorial group first...

ISO Project 24617-2 Semantic Annotation Framework,

Part 2: Dialogue Acts

Editorial Group first meeting Pisa, 29 - 30 September 2008

TC 37/SC 4/WG 2Kiyong Lee, convenor

Draft agenda 1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and results of

voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group6. Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS

results behind it7. Comments on WD N442 rev 00 8. Planning of future meetings -- next meeting: Tilburg, January 20099. Actions and procedures for October 2008 -- January 200910. Any other business and wrap-up (lunch time Tuesday?)

Project status

- Launched as ISO project 24617-2 at SC 4 meeting in Marrakech, 25 May 2008

- Documents: New Work Item Proposal ISO/TC 37/SC 4 N442; Working Draft WD ISO/TC 37/SC 4 N442 rev 00; Results of voting on new work item proposal ISO/TC 37/ SC 4 N458;

- Editorial group:- Jan Alexandersson (Germany)- Harry Bunt (Netherlands/Belgium) (PL)- Jean Carletta (UK)- Alex Chengyu Fang (China/HK)- Jae-Woong Choe (Korea)- Koiti Hasida (Japan)- Olga Petukhova (Netherlands)- Andrei Popescu-Belis (Switzerland)- Claudia Soria (Italy)- David Traum (USA)

Background in ISO

Lisbon, May 2004: formation of ISO TC 37/SC 4/TDG 3: Thematic Domain Group on Semantic Content

Objective: “To prepare activities for possibly developing international

standards and guidelines for semantic annotation”

eContent project LIRICS (2005-2007): To explore the needs, requirements, and possibilities of

international standards for semantic annotation; to define a set of preliminary concepts for semantic annotation, certified by ISO TC 37/SC 4/TDG 3, in the form if entries in the ISO Data Category Registry.

ISO/LIRICS Data categories

(Joint work by Tilburg U, U of Pisa, DFKI Saarbruecken, UPF Barcelona)

Data categories for:

- semantic role annotation- dialogue act annotation- reference annotation

- temporal annotation certified by ISO TC 37/SC 4 Thematic Domain Group 3(Semantic content)

LIRICS WP 4 Deliverables

D4.1”Methodological aspects of semantic annotation and representation” (Harry Bunt & Amanda Schiffrin). Methodological foundations for metamodeling; comparative analysis of semantic annotation efforts.

D4.3 “Documented compilation of semantic data categories ”. (Harry Bunt & Amanda Schiffrin)Set of data categories for the annotation of temporal information, reference, semantic roles and communicative functions.

D4.4 “Multilingual test suites for semantically annotated data.” (Harry Bunt, Olga Petukhova & Amanda Schiffrin) Description of application of data categories from D4.3 and their evaluation for Dutch, English, Italian, Spanish, German, approved by TDG 3.

Draft agenda 1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and

results of voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group

6. Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it

7. Comments on WD N442 rev 008. Planning of future meetings -- next meeting: Tilburg, January

20099. Actions and procedures for October 2008 -- January 200910. Any other business and wrap-up

Draft agenda 1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and

results of voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group

6. Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it

7. Comments on WD N442 rev 008. Planning of future meetings -- next meeting: Tilburg, January

20099. Actions and procedures for October 2008 -- January 200910. Any other business and wrap-up

Proposal NWIP ISO/... N442

Scope:

“provide well-defined concepts • for identifying dimensions of interaction that dialogue acts

may address;• for functional dialogue segmentation in multiple

dimensions;• for the definition of communicative functions.

The standard will specify data categories for a range of core communicative functions, starting from proposals made jointly by the EU LIRICS project and the ISO TC 37/SC 4 Thematic Domain Group TDG 3 on Semantic Content.”

Proposal NWIP ISO/... N442

Purpose: “... provide annotation guidelines and examples. (...) The

theoretical foundation of the LIRICS data categories (..) provides a basis for segmenting dialogue in multiple dimensions and allowing markables to be discontinuous and to overlap. The project will provide guidelines for how to effectively perform such segmentation.

While it seems feasible, given the current state of the art, to develop standard annotation concepts for a range of core dialogue acts, researchers and applications designers should also be supported in adding their own concepts for specific domains or purposes. The standard will provide general principles and guidelines for extending its core concepts.”

N458: Results of voting on N442

• Positive votes from 7 countries; no negative votes• UK and Korea: Working Draft acceptable as Committee

Draft• Nominated experts: Pavel Smrz (CZ), Thierry Declerck

(D), Aesun Yoon (Korea), UK to be nominated later• Specified time schedule

Draft agenda 1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and

results of voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group

6. Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it

7. Comments on WD 4 N442 rev 008. Planning of future meetings -- next meeting: Tilburg, January

20099. Actions and procedures for October 2008 -- January 200910. Any other business and wrap-up

Project time schedule

Submission of standards proposal in phases with maximal deadlines:• as CD: 31 September 2009• as DIS: 31 March 2011• as FDIS: 31 October 2011• for publication as IS: 15 June 2012

Hopefully we can deliver much faster than that!

Draft agenda 1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and

results of voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group

6. Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it

7. Comments on WD N442 rev 008. Planning of future meetings -- next meeting: Tilburg, January

20099. Actions and procedures for October 2008 -- January 200910. Any other business and wrap-up

Resonance group

Purpose:1. to accomodate nominated national experts2. to obtain input from a wider circle of researchers

For 1: Pavel Smrz, Thierry Declerck, Aesun Yoon,..For 2: expressions of interest from various people, including

James Allen, Laurent Romary, Gilles Francopoulo

Proposal: • email list• invitation to participate in meetings/workshops, such as

January 2009 meeting

Draft agenda 1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and

results of voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group

6. Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it

7. Comments on WD N442 rev 008. Planning of future meetings -- next meeting: Tilburg, January

20099. Actions and procedures for October 2008 -- January 200910. Any other business and wrap-up

LIRICS Results

Evaluation of data categories for dialogue acts

Inter-annotator agreement measurements for English and Dutch;

2 trained annotators working on raw text/audio

Results: almost perfect agreement (Rietveld & van Hout, 1993: kappa ≥ 0.80)

LIRICS Results

Function class English Dutch average

Information-seeking 0.96 0.98 0.97

Assistance-providing 0.98 0.99 0.98

Feedback 0.98 0.99 0.99

Interaction management

0.92 0.96 0.94

Social obligations management

0.94 0.94 0.94

Draft agenda

1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and

results of voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group

6. Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it

7. Comments on WD N442 rev 008. Planning of future meetings -- next meeting: Tilburg, January

20099. Actions and procedures for October 2008 -- January 200910. Any other business and wrap-up

Discussion items/WD comments

• At what level of generality should we aim to standardize?• How should the standard relate to existing annotation

schemas (DAMSL, SWBD-DAMSL, LIRICS, DIT++, AMI, Coconut, Verbmobil,...)?

• Specific technical comments• Specific textual/editorial comments and suggestions• Multimodality

Level of generality

• Principles for annotation schema design, illustrated with an instantiation;

• DiaML as an interlingua for relating annotation schemas;• Core dialogue acts and their use.

Level of generality

• Principles for annotation schema design, illustrated with an instantiation (possibly the LIRICS schema or a version of that);

• DiaML as an interlingua for relating annotation schemas;• Core dialogue acts and their use.

Level of generality

• Principles for annotation schema design, illustrated with an instantiation (possibly the LIRICS schema or a version of that);

• DiaML as a resource (an “interlingua”) for relating existing annotation schemas;

• Core dialogue acts and their use: a (multidimensional) taxonomy of dialogue acts which are present in many schemas in some form or another, and under some name or another, defined as ISO data categories (preferably with a specification of their semantics), and with principles for extending this set/ taxonomy with additional dialogue acts.

Relation of standard to existing annotation schemes

• Standard should not include specific annotation schema, but provide mappings between existing schemas;

• Standard should specify “interlingua” schema with standard concepts (as data categories), explaining their relation to existing schemas;

• Standard should include new annotation schema, based on the preliminary ISO/LIRICS work, as an improvement of DAMSL (etc.); this schema should be open, and allow the use of only a subset of its dialogue act types.

Specific technical comments

• Indirect speech acts as indirect forms of direct dialogue acts, or as different types of dialogue acts (David Traum)?

• Semantic content allowed in determining communicative function (David Traum, Jean Carletta)?

• Multidimensional annotation schemas versus flat lists of tags (Jean);• Purpose/intention/consciousness (Jean)• Acknowledge Bales’ Interaction Process Analysis (Jean)• Aims and purposes (DIN comment)• `Functional segment’ is not in the metamodel; what about things that

aren’t on a dimension (Jean)

Specific textual comments/suggestions

Section Purpose and justification not clear enough (Claudia, Jean); move material from beginning Section 5 and from Section 7 there (Claudia); Section 9 should also be place earlier (Claudia);

Definition of “dialogue act” missing (Claudia, Jean)Figure 1 would be clearer if it had words (Jean); a worked example

would helpSections 6 and 7 are somewhat confusing; the document should lead

the reader from background to theoretical justification to core model to possible extensions (Claudia)

Section 8 is very nice; DiaML could be a nice operationalization of the meta-model (Claudia)

A concrete example would help to understand the XML example on p. 16 (Claudia).

Draft agenda 1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and

results of voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group

6. Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it

7. Comments on WD N442 rev 008. Planning of future meetings -- next meeting: Tilburg, January

20099. Actions and procedures for October 2008 -- January 200910. Any other business and wrap-up

Meetings in 2009

• January 5-6, 2009 in Tilburg, the Netherlands, preceding IWCS-8 (7-9 January)

• May/June 2009, Boston or Colorado• October 2009, Berlin• January 2010, ICGL-2 Hong Kong

Draft agenda 1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and

results of voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group6. Comments on WD N442 rev 007. Planning of future meetings -- next meeting: Tilburg, January

20098. Actions and procedures for October 2008 -- January 20099. Any other business and wrap-up

Draft agenda 1. Background of the project2. ISO formalities and procedures (Kiyong Lee)3. Project (“work item”) proposal ISO/TC 37/SC 4 N 442 and

results of voting (ISO/TC 37/SC 4 N 458)4. Project time schedule5. Editorial group and resonance group

6. Working Draft ISO/TC 37/SC 4 N442 rev 00 and the ISO/LIRICS results behind it

7. Comments on WD N442 rev 008. Planning of future meetings -- next meeting: Tilburg, January

20099. Actions and procedures for October 2008 -- January 200910. Any other business and wrap-up