knowledge extraction from technical documents knowledge extraction from technical documents *with...

© Generative Software Technologies Corp. 1

Knowledge Extraction from Technical Documents

*With first class-support for Feature Modeling

Rehan Rauf, Michal Antkiewicz, and Krzysztof Czarnecki

Generative Software Technologies Corp. Waterloo, Canada

http://gensoftech.com

http://gensoftech.com/


The Idea


Specification Documents

Spec DocHeadingtext text text text text text texttext text text text text text - text text text text text text text text text text text text text text text text text text

text text text text text text text text text text text text texttext text text text text text text text text text text text text text text text text text text text text

Text Text Text Text Text Text

text text Text Text text text

text text text text text text

Section

Table

Paragraph

Physical structures

Functional Reqs

Business Rules

Use Case

Logical structures(specification elements)


Recognize and extract specification elements

based on physical document

structure


ET – Extraction Toolsearches for template instances

Spec Doctext text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text

Text Text Text

text text text

text text text

text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text

Text Text Text

text Text text

text text

UC Template UC 1

UC 2


Precondition:Documents have been authored with some

template in mind


Application scenarios


Import to Requirements Mgmt Tools

Spec DocHeadingtext text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text





DoorsHP Quality CenterRequisite Pro…

Functional Reqs

Business Rules

Use Case

Functional Reqs

Business Rules

Use Case

ET




QT



Structured Query




All use cases with actor = ‘customer’

Use Case



Functional Reqs

Use CaseUse Case

Business Rules


Spec Doc


Headingtext text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text






Tracing

Business Rules

Use Case

Use Case


Spec Doc


Headingtext text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text text




Template Conformance Checking

Use Case

Use Case


Main Challenge:Logical and Physical

Variation


Challenge – Variation

Instances of Use Case



Instances of Use Case Logical components Component Identifiers


Variation Types

Designed Accidental

Logical

Physical


Designed Logical Variation

Optional component


Designed Logical Alternatives

Deeper decomposition

Different methodologies lead to logical variation


Designed Physical Variation

Different formatting


Accidental Variation

LogicalMissing components, e.g., actor

PhysicalSpelling mistakes, e.g., “Actar”Style inconsistency, e.g., italics instead of bold


Solution


ET – Extraction Tool

Docs PSE

Physical componentsSections, lists, table cells

LSE

UC Template

Logical componentsActor, flow, extensions

Accidental variationvia match threshold

Designed variation

via template


UC Template

Metamodel

UC

Name : String Flow

Action : String

*

1 1

SectionHeading

List

Paragraph

Mapping


Example Template


Logical Structure


Mapping


Regular Expressions


Lists


Component Nesting


Optional Components


Physical Alternatives


Templates with Tables


Logical Alternatives


ET – Extraction Tool

Docs PSE

Physical components

Basic: Paragraph, cell, graphic

Composite: Sections, lists, tables, …

LSE

UC Template



Physical Structure Extraction

Docs PSE

Physical components

Basic: Paragraph, cell, graphic

Composite: Sections, lists, tables, …

LSE

UC Template


Only part dependent on

document-format


Performance


Can we extract logical structures from real-world documents?


Document Set

43 documents24 from 3 companies11 from public sources6 student projects2,000 to 23,000 words

ContentUse CasesData ObjectsBusiness RulesFunctional ReqsNon-Functional Reqs…

Docs


ET2) Verify extraction

Template Development

UC1

UC Template

UC Template

1) Write template manually

UC2

??

3) Refine template


Results

36 logical structuresUse cases, data objects, business rules, … Template sizes from 3 to 52 LOCTotal 942 instances

Nearly all instances perfectly recognized100% recall for 33 templates; over 80% for remaining 3100% precision for 35 templates; 87% for remaining 1

Error causesSevere formatting problems, e.g., manual line breaksForgotten ids


Other Questions

Amount & kind of template change in refinement 1% – 25% LOC affected during refinement81% changes concern optionality (add ‘?’ or component)

Amount of iterations1 instance (11 cases) to 50% of all instances (6 cases)

e.g., 10 out of 20 (2 cases); mostly simple edits, add `?’

ImplicationStart with few examples, then edit the template based on expert knowledge (e.g., add `?’)


Related Work

Import to Req Mgmt ToolsTools prescribe document structureManual markup for fine-grained extraction

Wrapper inductionMachine generated docs (web pages)Induced Regex not human readable (no modeling language)

Natural language processingCan benefit from structure-induced semantic tags


Future: Template by Example

UC1

UC Template

UC2

3) Refine template

1) Mark up sample document

UC Template

TE 2) Extract template

3) Verify extraction

ET


Summary


ET – Design

48

Functional Reqs

B. Rules

Use Case

B. Rules

Use Case

Use Case

PSE

Physical components

Spec Doc

Spec Doc

Spec Doc

UC Template

LSE

Logical components

Spec Doc

Spec Doc

Use CaseQT

Query

Functional Reqs

B. Rules

Use Case

ET

Import

Tracing

Conformance

Application scenarios Template development

Evaluation results

Nearly all instancesperfectly recognized

43 real-world documents


Technology available athttp://gensoftech.com/IntelligentET

http://gensoftech.com/IntelligentET