improving requirements glossary construction via clustering
DESCRIPTION
TRANSCRIPT
Improving Requirements Glossary Construction via Clustering:
Approach and Industrial Case Studies!
Chetan Arora1 Mehrdad Sabetzadeh1 Lionel Briand1 Frank Zimmer2
1 University of Luxembourg, Luxembourg!2 SES TechCom, Luxembourg
.lusoftware verification & validationVVS
NL Requirements• NL requirements can be vague and ambiguous
• Multiple interpretations of same words - scale
• Inconsistent terminology
• Multiple terms for same concepts
• element / component / object!
• Multiple representations of same keywords
• status of Ground Station Interface component!
• Ground Station Interface component’s status!
• Interface component status
2
• Glossaries help mitigate ambiguities
• consistent terminology
• improves communication - especially in scenarios with multiple stakeholders
Requirement Glossaries
3
Glossary Building Process• Identify / Search for relevant glossary terms
!
• Define these terms, along with related terms
!
• Ideally, glossary should be built along with requirements specification, but reality?
4
Automated Term Extraction
JATE
TextRank
TOPIA
TermRaider
5
“Wait, my colleagues have used so many
variations of the same terms”
“Even I’ve used different variations myself. Let me
correct them all in the document.”
“Let’s identify the glossary terms!!!”
Story Behind
“How about we cluster all of them together”
6
Identification of
Candidate Terms
Similarity Calculation Clustering
Candidate Terms
2
6666664
s11 · · · s1n... . . . ...sn1 · · · snn
3
7777775
Similarity Matrix
Identification of
Candidate Terms
Similarity Calculation Clustering
Candidate Terms
2
6666664
s11 · · · s1n... . . . ...sn1 · · · snn
3
7777775
Similarity Matrix
Identification of
Candidate Terms
Similarity Calculation Clustering
Candidate Terms
2
6666664
s11 · · · s1n... . . . ...sn1 · · · snn
3
7777775
Similarity Matrix
Identification of
Candidate Terms
Similarity Calculation Clustering
Candidate Terms
2
6666664
s11 · · · s1n... . . . ...sn1 · · · snn
3
7777775
Similarity Matrix
Approach
Similarity Measure
ClusteringParameter(s)
NL Requirements
Combination and Filtering Heuristics
Clusters
7
Identification of Candidate Terms Similarity Calculation ClusteringIdentification of Candidate Terms Similarity Calculation ClusteringIdentification of Candidate Terms Similarity Calculation Clustering
R1 - STS shall supply GSI monitoring information (GSI input parameters and GSI output parameters) to the STS subcontractor.
R2 - When GSI component’s status changes, STS shall update the progress of development activities.
R1 - STS shall supply GSI monitoring information (GSI input parameters and GSI output parameters) to the STS subcontractor.
R2 - When GSI component’s status changes, STS shall update the progress of development activities.• STS • STS Subcontractor
• GSI • GSI input parameter • GSI output parameter
• GSI component • GSI component’s status • GSI monitoring information
• development activity • progress of development activity
0.85
8
Identification of Candidate Terms Similarity Calculation Clustering
Empirical Evaluation
9
Case StudiesCas
e-A
Case-B
380 Requirements
110 Requirements
10
RQ1 - How accurate is our approach at identifying glossary terms?
Term Extraction Accuracy
11
JATE
TextRank
TOPIA
TermRaider
Clustering Accuracy!
RQ2 - Which similarity measure for candidate terms yields the most accurate clusters?
12
Tuning the Clustering Algorithm!
RQ3 - How can one systematically specify the number of clusters for clustering?
13
Clustering Effectiveness!
• RQ4 - Is clustering effective at grouping clustering terms?
• Comparison of generated clusters against Ideal clusters
14
Inferring Ideal Clusters• Difficult to objectively define ideal clusters
directly
• Systematic inference of ideal clusters using domain models
Initial Modelby Domain Expert
Model Completion by Researchers
Validation by Domain Expert
15
Var(status):- status- status of GSI component- GSI component status
Var(availability):- availability- availability of GSI component
Ground Station Interface
(GSI) statusavailability
GSI Component
GSI Monitoring Information
GSI Anomaly
GSI Output Parameter
GSI Input Parameter
Var(status):- status- status of GSI component- GSI component status
Var(availability):- availability- availability of GSI component
Ground Station Interface
(GSI) statusavailability
GSI Component
GSI Monitoring Information
GSI Anomaly
GSI Output Parameter
GSI Input Parameter
Var(status):- status- status of GSI component- GSI component status
Var(availability):- availability- availability of GSI component
Ground Station Interface
(GSI) statusavailability
GSI Component
GSI Monitoring Information
GSI Anomaly
GSI Output Parameter
GSI Input Parameter
From Domain Models to ClustersVar(status):- status- status of GSI component- GSI component status
Var(availability):- availability- availability of GSI component
Ground Station Interface
(GSI) statusavailability
GSI Component
GSI Monitoring Information
GSI Anomaly
GSI Output Parameter
GSI Input Parameter
Specialization Relation
Aggregation Association
Attributes
16
Overlapping
Var(status):- status- status of GSI component- GSI component status
Var(availability):- availability- availability of GSI component
Ground Station Interface
(GSI) statusavailability
GSI Component
GSI Monitoring Information
GSI Anomaly
GSI Output Parameter
GSI Input Parameter
Var(status):- status- status of GSI component- GSI component status
Var(availability):- availability- availability of GSI component
Ground Station Interface
(GSI) statusavailability
GSI Component
GSI Monitoring Information
GSI Anomaly
GSI Output Parameter
GSI Input Parameter
Var(status):- status- status of GSI component- GSI component status
Var(availability):- availability- availability of GSI component
Ground Station Interface
(GSI) statusavailability
GSI Component
GSI Monitoring Information
GSI Anomaly
GSI Output Parameter
GSI Input Parameter
Accuracy Metrics for Clusters
Precision = |Common Elements| ___________________ |Generated Cluster|
Recall = |Common Elements| ___________________ |Ideal Cluster|
Precision = 4 / 9 !
Recall = 4 / 412 3
4
5 6 7
8
9
17
Ideal Clusters - Only for Evaluation
Scalability!
RQ5 - Does our approach run in reasonable time?
19
Results
20
How accurate is our approach at identifying glossary terms? (RQ1)
16.6
22.7
12.5 16.1
20.2
30
49
20.8 26
48
72 70.7 73.6
70.7
92.9
49 49
70.6
53
90.2
Case A Case A Case A Case A Case A Case B Case B Case B Case B Case B
JATE TextRank TermRaider Topia Our Approach JATE TextRank TermRaider Topia Our Approach
Precision Recall F-measure
Case-A Case-B
21
Which similarity measure yields the most accurate clusters? (RQ2)
22
SyntacticSoftTFIDFMonge ElkanLevensteinRandom
Fmeasure vs. Num_Clusters
Num_Clusters0 5 10 20 30 40 50
Fmea
sure
0.100.20
0.30
0.40
0.50
0.60
Graph Builder
SyntaticSoftTFIDFMonge ElkanLevensteinRandom
F-measure vs. Number of Clusters
Number of Clusters0 50 100 150 200 250 300
F-m
easu
re
0.05
0.15
0.25
0.35
0.45
Where(100 rows excluded)
Graph Builder
Case-B
Number of ClustersNumber of Clusters
Case-A
Syntactic Similarity MeasureSoftTFIDFMonge ElkanLevensteinRandom
Fmeasure vs. Num_Clusters
Number of Clusters0 5 10 15 20 25 30 35 40 45 50
F-m
easu
re
0.100.150.200.250.300.350.400.450.500.550.600.65
Graph Builder
Syntactic Similarity MeasureSoftTFIDFMonge ElkanLevensteinRandom
Fmeasure vs. Num_Clusters
Number of Clusters0 5 10 15 20 25 30 35 40 45 50
F-m
easu
re
0.100.150.200.250.300.350.400.450.500.550.600.65
Graph Builder
Syntactic measure:SoftTFIDFMonge ElkanLevensteinRandom
Syntactic Similarity MeasureSoftTFIDFMonge ElkanLevensteinRandom
Fmeasure vs. Num_Clusters
Number of Clusters0 5 10 15 20 25 30 35 40 45 50
F-m
easu
re
0.100.150.200.250.300.350.400.450.500.550.600.65
Graph Builder
Syntactic Similarity MeasureSoftTFIDFMonge ElkanLevensteinRandom
Fmeasure vs. Num_Clusters
Number of Clusters0 5 10 15 20 25 30 35 40 45 50
F-m
easu
re
0.100.150.200.250.300.350.400.450.500.550.600.65
Graph Builder
Syntactic measure:SoftTFIDFMonge ElkanLevensteinRandom
F-m
easu
re(%
)
F-m
easu
re(%
)
051525
35
45
10203040
6050
How can one systematically specify the number of clusters for clustering? (RQ3)
23
7000
8000
9000
10000
11000
12000
13000
BIC
0 5 10 20 30 40 50Num_Clusters
Overlay Plot
600000
700000
800000
900000
1000000
1100000
BIC
0 50 100 150 200 250 300Num_Clusters
Overlay PlotCase-A Case-B
Number of Clusters
BIC
Number of Clusters
Chosen # of clusters = 120 Chosen # of clusters = 23
5% marginfrom max BIC
5% marginfrom max BIC
Is clustering effective at grouping candidate terms? (RQ4)
Case-A Case-B90.8
65.5
89.1
66.8 76.2
38.9
70.7
52.5
82.9
48.8
78.8
58.8
Upperbound Actual Uppderbound Actual
Precision Recall F-measure
24
UpperboundUpperbound Actual Actual
RQ5 - Does our approach run in reasonable time?
!
• Case - A : 380 requirements ~ 26 minutes
• Case-B : 110 requirements: < 2 minutes
25
Future Work• Further enhance the accuracy using semantic
analysis
• More empirical studies
!
!
• Identifying relationships between different development artefacts
26
Conclusion
27
16.6
22.7
12.5 16.1
20.2
30
49
20.8 26
48
72 70.7 73.6
70.7
92.9
49 49
70.6
53
90.2
Case A Case A Case A Case A Case A Case B Case B Case B Case B Case B
JATE TextRank TermRaider Topia Our Approach JATE TextRank TermRaider Topia Our Approach
Precision Recall F-measure
Case-A
Case-B
380 Requirements 110 Requirements
7000
8000
9000
10000
11000
12000
13000
BIC
0 5 10 20 30 40 50Num_Clusters
Overlay Plot
600000
700000
800000
900000
1000000
1100000
BIC
0 50 100 150 200 250 300Num_Clusters
Overlay Plot
Case-A Case-B
Number of Clusters
BIC
Number of Clusters
Chosen # of clusters = 120 Chosen # of clusters = 23
5% marginfrom max BIC
5% marginfrom max BIC