tacit knowledge extraction through affective computing · 2019-10-09 · tacit knowledge extraction...

Tacit Knowledge Extraction through Affective Computing

Tacit Knowledge Extractionthrough Affective Computing

2nd International Workshop on Affective Computing for Requirements Engineering

RE2019, September 23, 2019, Jeju, South Korea

Jan Ole [email protected]

Technical University of Munich Department of InformaticsGarching b. München, Germany

Invited Talk

Tacit Knowledge Extraction through Affective Computing 2

What is Tacit Knowledge ?

Michael Polanyi

Donald A. Schön

Gerd Gigerenzer

know more than we

“Wecan tell ” “Gut feeling”

“the workday life of the professional depends on

tacit knowing-in-action”


Definitions of Tacit Knowledge ?


Persistence Communication

Improvement

4

UnderstandingAcceptance

Needs Performance

Compliance

Why Tacit Knowledge Extraction?


Tacit Knowledge in Software Engineering

5

Knowledge about how to apply pinch gesture to change content size.

Screenshots © Google Maps on iOS

2x

Requirement: Enable change of zoom level of maps view.

Requirement: Simplify zoom level change when on the go.

Knowledge about the need to change content size

with one hand only.


Tacit Knowledge Extraction ProblemSoftware users carry tacit knowledge which is not shared with software developers.

Tacit Knowledge Integration ProblemTacit knowledge that resides within users is not systematically integrated into frequent and rapid software development processes.

Affective Computing as a means to gain access to user feedback.

Continuous Software Engineering as the process foundation.


Investigation Treatment Validation

Interview study with 24 practitioners from 17 companiesto investigate the state-of-the practice.

Investigation


Investigation Research Questions & Results

How do practitioners apply CSE during

software evolution?

How do practitioners involve users during

CSE?

How can usage knowledge support practitioners during

CSE?

How do Practitioners Capture and Utilize User Feedback during

Continuous Software Engineering?

Jan Ole Johanssen, Anja Kleebaum, Bernd Bruegge and Barbara Paech

Session 10: CrowdRESeptember 26, 201911:15 ~ 11:30 [R14]

• Practitioners exhibit five different perspectives on the definition of CSE

• They perceive users as one of the three most important CSE categories

• 19 positive, 56 neutral, and 17 negative experiences were reported; only four of them address users

• For the future, practitioners rely on three strategies: enhancement, expansion, and on-demand adaption

• Practitioners welcome a proposal to integrate knowledge into CSE

• Neutral feedback dominates the overall feasibility of the proposal

• Collection of practitioners’ answers on benefits and obstacles, as well as extension and addition, contributed to an improved proposal



: Continuous User Understanding in Software EvolutionCuuSE


Treatment The CuuSE Framework

Cuu Workflow

Cuu Workbench

Cuu Kits

Feature Crumbs

FeatureKit InteractionKit BehaviorKit EmotionKit NoteKit PersonaKit ThinkingAloudKit

CuuSE


Treatment Feature Crumbs

User Feedback

content

«enumeration»Type

ExplicitImplicit

Context

Raw Usage Data

process (Expertise)

Collector

Usage Information

relateTo (Feature)

Expertise

Usage Knowledge

fuse ( )

Feature


A lightweight, code-based concept to specify a feature's run-time characteristics.

12

Treatment

Multiple feature crumbs can be concatenated through the

Cuu workbench.

@IBAction func didTapTryFeatureKitButton(_ sender: UIButton) { // Notify the FeatureKit component of CUU that a step of a feature was triggered. CUU.seed(name: "Step 1: Select Item" ) }

The concatenation of feature crumbsis called a feature path, which is similarto a scenario.

Feature Crumbs


Treatment

Workbench

solutionscontrolDatafeatureData

inspectupdateconfigure (Feature)read: Feature

Controller

releaseactivate (Kit, Feature)

Blackboard

Kit

updateWorkbenchexecConditionexecActionmonitorextractintegrate

KnowledgeSource

DashboardDeveloper

Cuu Workbench


Workbench

solutionscontrolDatafeatureData

inspectupdateconfigure (Feature)read: Feature

Controller

releaseactivate (Kit, Feature)

Blackboard

Kit

updateWorkbenchexecConditionexecActionmonitorextractintegrate

KnowledgeSource

14

Treatment

DashboardDeveloper

Cuu Workbench


Treatment Cuu Kits

Facial Expression

User Sample Application

Emotion Observer

Feature Crumb

EmotionKit

«maps»

«observes»

«uses»

EmotionKitEmotionKit


Treatment Cuu Kits EmotionKit

Facial Expression

User Sample Application

Emotion Observer

Feature Crumb

EmotionKit

«maps»

«observes»

«uses»

User 1 User 2 User 3

Q1 Q2 Q3 Q4

Happiness

Surprise

Fear

Contempt

Sadness

Anger

Crumb

A: Map View Did Appear D: Round-circled Button Pressed

EmotionKit Spot Widget


Treatment Cuu Kits ThinkingAloudKit

ThinkingAloudKitDeveloperSetup Users Developer

Analysis

Classified Sentences per

Crumb

Yes

NoNo

Yes

No

Yes

Crumb is First Crumbof Activated

Feature?

CrumbBelongs toActivatedFeature?

Activate Featurefor CTA

Trigger Crumb

Fetch ActivatedFeature for CTA

Record Speech and Transform

to Text

ClassifySentences

Think Aloud while Using the Application

Identify Problemand Work on new Feature Increment

PrepareFeature under Development

TriggerNext Crumb

TranscribedText perCrumb

ActivatedFeature

has NextCrumb?

CTAResults

Inspect CTA Results in CuuSE

DashboardBundle and Store Results

Define Feature Path in CuuSE Dashboard


Treatment Cuu Kits

ThinkingAloudKitUsers DeveloperAnalysis


Crumb

Trigger Crumb


to Text

ClassifySentences


TriggerNext Crumb

TranscribedText perCrumb Inspect CTA

Results in CuuSE DashboardBundle and

Store Results

ThinkingAloudKit


ThinkingAloudKitUsers DeveloperAnalysis


Crumb

Trigger Crumb


to Text

ClassifySentences


TriggerNext Crumb

TranscribedText perCrumb Inspect CTA

Results in CuuSE DashboardBundle and

Store Results

19

Treatment Cuu Kits ThinkingAloudKit

Insecure Neutral Positive Negative

A: Map View Did Appear

D: Round-circled Button Pressed

0 1 3 4Number of Sentences

ThinkingAloud Spot Widget


Treatment Cuu Kits PersonaKit

User

name: String

Real User

realName: String

Persona

proxyName: String



User

name: String

Real User

realName: String

Persona

proxyName: String

User

name: String

Real User

realName: String

Persona

proxyName: String

Runtime Persona

Trait Description Characteristic

Session

duration: Number

PersonaKit

process ( )manageSession ( )

Application

start ( )terminate ( )

«derives»

«generates»

«uses»



User

name: String

Real User

realName: String

Persona

proxyName: String

Runtime Persona

Trait Description Characteristic

Session

duration: Number

PersonaKit

process ( )manageSession ( )

Application

start ( )terminate ( )

«derives»

«generates»

«uses»

Manual ConfigurationOverview Description

Hasty

Fearless

Jan

Name

Use application to find out about latest traffic reports.

Goals

← Previous Persona Next Persona →

PersonaKit Spot Widget

Jan is very hasty while using the app. He mostly opens MapViewController. He is a novice user of the app with an average of 3 sessions per day. But his sessions are quite short and only last about 10 seconds. He mostly uses the app on the go. He uses an iPhone and has installed iOS 12.1.3.


Kit y

Kit y

Kit x

Kit x

23

Treatment Cuu Kits

EmotionKitprocess()

EmotionKitrelateTo(Feature)

Kit 2 fuse()

Usage Data

Usage Information

Usage KnowledgeLevel 2

Usage KnowledgeLevel 3

Level 1

Level 0Core competence

of Cuu Kits

Tacit knowledge extraction, performed by developers.


Treatment

Cuu Dashboard

🔍

Cuu Workflow

Screenshots © Google Maps on iOS


Treatment Cuu Workflow

FeatureKit: Path Observations

CompletedStarted

Canceled

Novice Expert

Smartphone ExperienceTotal Users Feature Users

5 3BehaviorKit Spot Widget

Gain overview of feature observations.




CompletedStarted

Canceled

Feature Kit: Crumb Observation



Observations

6

3

Novice Expert



Gain more specific knowledge with

respect to the user.




CompletedStarted

Canceled




Observations

6

3


Q1 Q2 Q3 Q4

Happiness

Surprise

Fear

Contempt

Sadness

Anger

Crumb



Novice Expert



Start investigation on feature particular that

causes an issue.




CompletedStarted

Canceled




Observations

6

3







Q1 Q2 Q3 Q4

Happiness

Surprise

Fear

Contempt

Sadness

Anger

Crumb



Novice Expert



Investigate users’ response to potential

issue.






Observations

6

3






Timestamp

NoteKit Spot Widget

I am looking for the options to change the map layer Sept 1, 2019 10:11:00 AM

The button color is too bright Sept 1, 2019 10:12:00 AM

I like this Sept 1, 2019 10:15:00 AM

Timestamp


Q1 Q2 Q3 Q4

Happiness

Surprise

Fear

Contempt

Sadness

Anger

Crumb



Find more evidence.






Observations

6

3







Q1 Q2 Q3 Q4

Happiness

Surprise

Fear

Contempt

Sadness

Anger

Crumb



Timestamp

NoteKit Spot Widget

I am looking for the options to change the map layer Sept 1, 2019 10:11:00 AM

The button color is too bright Sept 1, 2019 10:12:00 AM

I like this Sept 1, 2019 10:15:00 AM

TimestampUnderstand reasons for issue.


Validation The CuuSE Framework

Cuu Workflow

Cuu Kits

Feature Crumbs

FeatureKit InteractionKit BehaviorKit EmotionKit NoteKit PersonaKit ThinkingAloudKit

Cuu Workbench

Applicability and Feasibility

Cuu Workbench Usefulness of Cuu Widgets


Validation EmotionKit

• Laboratory experiment with 12 subjects• Instruction: “Read and follow six screens”• Multiple usability issues seeded into the application

- Performance- Inconsistency- Missing feedback

• Manual observation ofreactions as the basisfor validation


Validation EmotionKitPr

obab

ility

of E

mot

ions

in %

0.4

0.2

0.0

00:10 00:20 00:30 00:40 00:50 01:00 01:10 01:20 01:30 01:40 01:50 02:00 02:10 02:20 02:30

0.6

Time

Emotions Happiness Surprise Fear Contempt Sadness Anger Disgust

ShowU1 Screen

ShowU2 Screen

Show Text ShowU3 Screen

ShowU4 Screen

ShowU5 Screen

ShowU6 Screen

Time series data for results retrieved from EmotionKit


Validation EmotionKitSu

m o

f Em

otio

n Pr

opab

ilitie

s in

%

1.0

0.5

0.0

00:10 00:20 00:30 00:40 00:50 01:00 01:10 01:20 01:30 01:40 01:50 02:00 02:10 02:20 02:30

1.5

Time

C

B

A

Emotional Response

ShowU1 Screen

ShowU2 Screen


ShowU4 Screen

ShowU5 Screen

ShowU6 Screen


Validation EmotionKitSu

m o

f Em

otio

n Pr

opab

ilitie

s in

%

1.0

0.5

0.0

00:10 00:20 00:30 00:40 00:50 01:00 01:10 01:20 01:30 01:40 01:50 02:00 02:10 02:20 02:30

1.5

Time

C

B

A

Emotional Response

ShowU1 Screen

ShowU2 Screen


ShowU4 Screen

ShowU5 Screen

ShowU6 Screen

# U1-C U1-I U2-C U2-I U3-C U3-I U4-C U4-I U5-C U5-I U6-C U6-I

1 TP TP TN FP TP TP FP TN ERR TP FP ERR2 TP TN TN TN TP TP TN TP FP TP FP TP3 FP TN TN TN ERR FP TP TP FP TP TN FP

4 TN TP TN FP TN TP TN FP TN TP TN ERR5 TN FP FP FP TN TP TP TP FP FP TN TP6 TN FP TN FP TN TP TN FP TN FP TN TN

7 TN TP FP TN TN TN TP TP TN TP TN TN8 TN FP TN FP TP FP FP TP FP TP FP TP9 TP TP TN TP TP TP TN TP TN TP TP TP10 TP TN TN ERR TP FN TP TN FP ERR FP ERR

11 ERR FP TP TN ERR TP TN TN TN TP TP TP12 TN FP TN FP TN FP TN FP TN TP TN TP

Combined Content Interaction

Sensitivity 0.980 1.0 0.9706

Specificity 0.600 0.7308 0.3939

Accuracy 0.7407 0.7941 0.6866

Comparison with manual observations


Validation Cuu Widgets


Hasty

Fearless

Jan

Name


Goals




Strongly Agree Agree Neutral Disagree Strongly Disagree”“

• Survey with 10 developers• Received introduction to CuuSE framework• Applied CuuSE framework over the period of four months

Visualizing Runtime Personas for a release is useful to better fit the feature under development to users’ requirements.


Validation


CompletedStarted

Canceled







Q1 Q2 Q3 Q4

Happiness

Surprise

Fear

Contempt

Sadness

Anger

Crumb




Hasty

Fearless

Jan

Name


Goals




Strongly Agree Agree Neutral Disagree Strongly Disagree

FeatureKit EmotionKitPersonaKit ThinkingAloudKit

Cuu Widgets

“The visualization is useful.”


Summary

40

• Practitioners’ usage of user feedback is sparse • They welcome the integration of knowledge into CSE

• CuuSE is a framework for continuous user understanding• Knowledge sources provide different affective states from users• Feature crumbs enable their allocation to particulars of a feature• A dashboard visualizes knowledge and allows its combination• Tacit knowledge is extracted by following the Cuu workflow

• Facial expression-based emotions allow usability issue detection• Knowledge visualization is key for extracting tacit knowledge


References

41

Michael Polanyi. The Tacit Dimension. University of Chicago Press, 2009.Gerd Gigerenzer. Bauchentscheidungen: Die Intelligenz des Unbewussten und die Macht der Intuition. Goldmann Verlag, 2008.Donald A. Schön. The Reflective Practitioner: How Professionals Think In Action. Basic Books, 1983.Jan Ole Johanssen. Continuous User Understanding in Software Evolution. Dissertation, Technical University of Munich, to appear.Jan Ole Johanssen, Anja Kleebaum, Barbara Paech, and Bernd Bruegge. Practitioners’ eye on continuous software engineering: An interview study. In Proceedings of the 2018 International Conference on Software and System Process, ICSSP ’18, pages 41–50, 2018. ACM.Jan Ole Johanssen, Anja Kleebaum, Barbara Paech, and Bernd Bruegge. Continuous software engineering and its support by usage and decision knowledge: An interview study with practitioners. Journal of Software: Evolution and Process, 31(5):e2169, 2019. e2169 smr.2169.Jan Ole Johanssen, Anja Kleebaum, Bernd Bruegge, and Barbara Paech. Feature crumbs: Adapting usage monitoring to continuous software engineering. In Marco Kuhrmann, Kurt Schneider, Dietmar Pfahl, Sousuke Amasaki, Marcus Ciolkowski, Regina Hebig, Paolo Tell, Jil Klünder, and Steffen Küpper, editors, Product-Focused Software Process Improvement, pages 263–271, Cham, 2018. Springer International Publishing.Jan Ole Johanssen, Anja Kleebaum, Bernd Bruegge, and Barbara Paech. Towards the visualization of usage and decision knowledge in continuous software engineering. In 2017 IEEE Working Conference on Software Visualization (VISSOFT), VISSOFT 2017, pages 104–108, September 2017.Jan Ole Johanssen, Jan Philip Bernius, and Bernd Bruegge. Toward usability problem identification based on user emotions derived from facial expressions. In Proceedings of the 4th International Workshop on Emotion Awareness in Software Engineering, SEmotion ’19, 2019.,Jan Ole Johanssen, Lara Marie Reimer, and Bernd Bruegge. Continuous thinking aloud. In Proceedings of the Joint 4th International Workshop on Rapid Continuous Software Engineering and 1st International Workshop on Data-Driven Decisions, Experimentation and Evolution, RCoSE-DDrEE ’19, pages 12–15, Piscataway, NJ, USA, 2019. IEEE Press.

(in order of appearance)

tacit knowledge extraction through affective computing · 2019-10-09 · tacit knowledge extraction...

Documents