jorge munoz-gama advisor: josep carmona december 2014 conformance checking and diagnosis in process...

Jorge Munoz-GamaAdvisor: Josep Carmona

December 2014

CONFORMANCE CHECKING

AND DIAGNOSIS IN PROCESS MINING

PRECISION DECOMPOSITION

CONCLUSIONS

Conformance Checking in a Nutshell

MODEL REALITY

PROCESSDOMAINEXPERTS

Biased Vision

MODEL REALITY

PROCESS

DOMAINEXPERTS

MODEL REALITY

PROCESS

MODEL REALITY

PROCESS

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

Structure and Outline

• Structure of the Presentation

Problem – Context – Contributions

• Outline of the Presentation• Precision

• Precision based on the Log• Qualitative Analysis of Precision Checking• Precision based on Alignments

• Fitness Decomposition• Decomposed Conformance Checking• Topological Conformance Diagnosis• Data-aware Decomposed Conformance Checking• Event-based Real-time Decomposed Conformance Checking

CONCLUSIONS

• Precision• Precision based on the Log• Qualitative Analysis of Precision Checking• Precision based on Alignments

• Fitness Decomposition• Decomposed Conformance Checking• Topological Conformance Diagnosis• Data-aware Decomposed Conformance Checking• Event-based Real-time Decomposed Conformance

Checking

Problem“Low Criticality Diagnosis” Process

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

Hospital Process-aware

Information System

HospitalStaff

“Low Criticality Diagnosis”Process Model

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

Initial ExaminationAllergy TestBlood Test

Radiology TestDiagnosis

Home Care

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

Initial ExaminationRadiology TestRadiology TestRadiology TestRadiology Test

ContextThe Importance of Precision

A good model must be fitting but also be precise

ContextEfficient and Comprehensive

• Approach to measure precision

• Based on potential points of improvement

• Not require an exhaustive model state-space exploration

• Previous works require model exploration/simulation

• Identify precision problems with a fine granularity

• Results for analysis and process improvement

ContributionsPrecision based on Escaping Arcs

MODEL BEHAVIOR

LOG BEHAVIOR

Exploration of the model’s behavior: costly, possibly infinite, or require simulation.

LOG BEHAVIOR

Model behavior traversal restricted by the log behavior.

Escaping arcs: points where the model allows more behavior than the one observed in the log.

ESCAPING ARC

ComputePrecision

ModeledBehavior

ObservedBehavior

Minimal Imprecise Traces

ETC Precision (etcp)

ContributionsOutline of Precision based on Escaping Arcs

a, b, d, g, ia, c, d, e, f, h, ia, c, e, d, f, h, ia, c, e, f, d, h, i

a, c, fa, c, d, fa, c, d, e, ea, c, e, d, ea, c, e, e

ComputePrecision

ModeledBehavior

ObservedBehavior

ContributionsObserved Behavior

1 g 1d

1 11 1

1 1 1 1 1

1 11 1

1 1111

1 1 1 1

ComputePrecision

ModeledBehavior

ObservedBehavior

ContributionsModeled Behavior

ComputePrecision

ModeledBehavior

ObservedBehavior

ContributionsCompute Precision

• For each state of the automaton we take into account the weight, the observed arcs and the allowed arcs:

observed states

weight escaping arcs

allowed arcs

ContributionsComputing Precision

1 1 1 1

1111… + 4 · 0 +…

… + 4 · 2 +… 1 -

ContributionsComputing Precision

1 1 1 1

1111… + 1 · 1 +…

… + 1 · 2 +… 1 -

ContributionsChallenges Addressed

• The precision based on escaping arcs does not require a complete exploration of the model behavior.

• Instead, the model exploration is restricted by the behavior observed in the log.

• Escaping arcs pinpoint the situations that need to be fixed to achieve a completely precise system.

• Collect imprecisions in terms of event log - Minimal Imprecise Log

• Precision• Precision based on the Log

• Qualitative Analysis of Precision Checking

• Precision based on Alignments

Checking

ProblemThe Effects of Exceptional Behavior

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

ProblemThe Effects of Exceptional Behavior

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

Initial Examination

Allergy TestBlood Test

Home Care

ProblemVariability of Precision in the Future

ETC Precision

CurrentMoment

CloseFuture

FarFuture

futurepresent

ProblemLimited Resources and Imprecision Points

Hospital Process

Imprecision Points

Limited Analysts and

Resources

ContextRobustness, Confidence and Severity

• Precision based on Escaping Arcs more robust to exceptional behavior.

• Estimate the possible variability of the metric in the future.

• Asses the severity of imprecision points and compare them.

ContributionsRobustness on Escaping Arcs

3199 3199

1435 1435 1435 1435

946 946

946 946 9460

764 764 764 764

54 54 54 54

3200 3200

1435 1435 1435 1435

947 947

946 946 9460

764 764 764 764

54 54 54 54

ContributionsRobustness on Escaping Arcs• Threshold parameter to cut exceptional behavior.

• Parametric threshold• High cut factor for main behavior • Low cut factor for extreme cases

• Local-context cut, not global-context cut

300500

3200 3200

1435 1435 1435 1435

947 947

946 946 9460

764 764 764 764

54 54 54 54

1 1 11ihf

Low Confidence High Confidence

ContributionsConfidence on Escaping Arcs Metric

ContributionsUpper Estimation of Precision

3200 3200

1435 1435 1435 1435

947 947

946 946 9460

764 764 764 764

54 54 54 54

• Best scenario = covering escaping arcs

ContributionsUpper Estimation of Precision

• Problem of optimization.

• Cover escaping arcs with the given k to maximize the metric.

• Cost of covering a escaping arc: the number of traces to overpass the threshold.

• Gain of covering a escaping arc: the weight of the state.

BIP Formulation

Upper Estimation

ContributionsLower Estimation of Precision

3200 3200

1435 1435 1435 1435

947 947

946 946 9460

764 764 764 764

54 54 54 54

• Worst scenario = new escaping arcs

1 1 1 1 1

Lower Estimation

A-1 A-1 A-1 A-1 A-1

• Subjective and multifactor

• Weight, Alternation, Stability, Criticality

C D G H F A946

1 1 1 1

D H F A764

D F A54545454

GD H F A

D F A54545454

C D G H F A946

1 1 1 1

D H F A764

D F A54545454

GD H F A

D F A54545454

All imprecisions equally important?

ContributionsSeverity of the Escaping Arcs

• Escaping arcs in parts with more weight more sever

3sever sever

ContributionsWeight of an Escaping Arc

• More chances to make a mistake more sever

sever sever

ContributionsAlternation of an Escaping Arc

• Apply perturbation • increase the number of instances in that point• proportional to the current occurrence number

• Measure how easy is to overpass the threshold

• Imprecision stable to perturbation more sever

3000sever sever

ContributionsStability of an Escaping Arc

• Importance of the task involved in the escaping arc

sever sever

CheckDateFormat

Bank Transfer

ContributionsCriticality of an Escaping Arc

• Robustness on the Precision based on Escaping Arcs.

• Confidence interval on the Precision metric.

• Severity assessment on the precision problems.

• Precision• Precision based on the Log• Qualitative Analysis of Precision Checking

• Precision based on Alignments

Checking

ProblemPrecision on Unfitting Scenarios

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

Home CarePerfect fitness is uncommon in real life

ContributionsUnfitting Observed Behavior

Log Trace

Model Behavior

ProblemFitness effects on Precision based on Log

a, a, b, b, d

What state reaches the model when the trace does not fit?

baa1 1 1 1 ???

• Option: Not considering the unfitting part.• The position of the fitting problem influences the precision.

ContextPrecision Independent of Fitness

• Unfitting scenarios are common in real-life

• Precision independent from Fitness

• A precision not based directly on the log but on a pre-alignment between the observed behavior and the modeled behavior.

ContextAligning Observed and Modeled Behavior

Log Trace

Model Behavior

ContextAligning Observed and Modeled Behavior

• Find the closest model trace in the model behavior for a given log trace

• From a global perspective• Able to deal with unfitting behavior• Optimal guaranteed

• Time-consuming problem based on A* search algorithms

* Adriansyah, A.: Aligning Observed and Modeled Behavior. PhD Thesis. Eindhoven University of Technology. 2014

ComputePrecision

ModeledBehavior

ObservedBehavior

Alignments

a d ba a b

ContributionsPrecision based on Alignments

ContributionsAligning Observed and Modeled Behavior

a ad ba a db

Log Trace

Alignment

Process Model

Log Trace adab aabd

a ad ba a db

Alignment

Process Model

Log Moves

Model Moves

Deviation

Fitting trace, closest to the original

Log Trace ad abd/acd

a da d

Alignment 1

Process Model

ba da d

Alignment 2c

Both alignments are optimal

ComputePrecision

ModeledBehavior

ObservedBehavior

Alignments

a d ba a b

New weight function

ContributionsObserved Behavior from 1-Alignment

a, a, b, da, b, da, d, a, ba, d

Event Log

a bdProcess Model

a, a, b, da, b, da, a, b, da, b, d

Fitting Traces

a, c, d

ac 2 2 2

ContributionsObserved Behavior from All-Alignment

a, a, b, da, b, da, d, a, ba, d

Event Log

a bdProcess Model

a, a, b, da, b, da, a, b, da, b, d

Fitting Traces

a, c, d

ac 2 2 2

1.5 1.5

ComputePrecision

ModeledBehavior

ObservedBehavior

Alignments

a d ba a b

New weight function

• Precision based on alignments.

• Precision for unfitting cases.

• Precision independent of fitness.

• Precision based on 1-alignment or All-alignments.

ContributionsExtensions to Precision based on Alignments

• Extensions to represent the modeled behavior.• Use of Representative-alignments.• Multi-sets to represent automaton states.

• Backwards use of the alignments.

a, c, d, eb, c, d, e edc

CONCLUSIONS

Checking

ProblemFitness in Large Models

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

ProblemFitness in Large Models

ContextFast, Comprehensible and Guaranteed

• Decompose the Fitness checking problem.

• Comprehensible decomposition and understandable diagnosis results.

• Formal guarantees.

• There is a fitness problem on the original net iff there is a fitness problem in one or more of the components.

• Fast compared to the monolithic approach.

The decomposition preserves the fitness.

ContributionsAlignment Fitness Checking

Log Trace

Model Behavior

ContributionsDecomposing Alignment Fitness Checking

Log Trace

Model Behavior

ContributionsDecomposition based on Graphs

• Based on Graph Decomposition

• Decomposition based on:• Single-Entry Single-Exit Components (SESE)• Refined Process Structure Tree (RPST)

* Artem Polyvyanyy: Structuring Process Models. PhD Thesis. University of Potsdam (Germany), January 2012

* Hopcroft, J., Tarjan, R.E.: Dividing a graph into triconnected components. SIAM J. Com- put. 2(3), 1973

ContributionsInterior, Boundary, Entry, and Exit nodes

• Entry node: boundary where • no incoming edge• or all outgoing edges

• Exit node: boundary where • no outgoing edge• or all incoming edges

Example of SESE and RPST

SESE: set of edges which graph has a Single Entry node and a Single Exit node

Refined Process Structure Tree (RPST) containing non overlapping SESEs

• Unique• Modular• Linear Time

ContributionsSESE and RPST

• Why SESE? • Only one entry; only one exit• Represent subprocesses within the process• Intuitive for conformance diagnosis

• Why RPST?• Partitioning over the RPST• Any cut is a partitioning• Algorithm to partitioning by size (k)

ContributionsSESE and RPST

• Why SESE? • Only one entry; only one exit• Represent subprocesses within the process• Intuitive for conformance diagnosis

• Why RPST?• Partitioning over the RPST• Any cut is a partitioning• Algorithm to partitioning by size (k)

• A decomposition based on SESEs preserves the fitness?

• Fitness Preservation: A model/log is perfectly fitting if and only if all the components are perfectly fitting

ContributionsPreserving the Fitness

• SESEs (per se) do not preserve fitness.

ContributionsSESE Decomposition does not Preserve Fitness

• 0 tokens in p abcdef S2 is blocked

• 0 tokens in p abcdef S2 is blocked• 1 token in p abcdef fits S but not S2

• 0 tokens in p abcdef S2 is blocked• 1 token in p abcdef fits S but not S2• 2 tokens in p abdecf fits S1 and S2 but not S

• The problem is in the shared places• No reflection on the log, therefore no synchronization.

• Valid Decomposition: a partition where only transitions are shared among components. No places neither arcs.

• There is a fitness problem on the original net iff there is a fitness problem in one or more of the components.

ContributionsValid Decomposition

Theorem: Valid Decomposition preserves the fitness.

* W.M.P. van der Aalst : Decomposing Petri nets for process mining: A generic approach. Distributed and Parallel Databases, 2013

• Create a ‘bridge’ for each shared place

ContributionsBridging a SESE Decomposition

S1’ S2’

Notice that not a SESE anymore

Theorem: SESE decomposition with Bridging post-processing preserves the fitness.

ContributionsSESE + Bridging Theorem

SESE decomposition with Bridging is a valid decomposition.

Monolithic 1h 15min

ContributionsDecomposition Fitness Results

Decomposition(7) 2min

ContributionsDecomposition Fitness Results

• Fitness Decomposition• Decomposed Conformance Checking

• Topological Conformance Diagnosis• Data-aware Decomposed Conformance Checking• Event-based Real-time Decomposed Conformance

Checking

ProblemLocate Fitness Problems in Large Models

ContextProblematic Components

• More than just report the list of model components with fitness problems.

• Provide a structure among the components.

• Visualize the structure of the decomposition.

• Use the structure to detect conflictive components highly related.

ContributionsTopological Fitness Checking

• Non-Fitting (Weakly) Connected Components

• Non-Fitting Subnet

ContributionsTopological Fitness Checking

• Fitness Decomposition• Decomposed Conformance Checking• Topological and Multi-level Conformance Diagnosis

• Data-aware Decomposed Conformance Checking

• Event-based Real-time Decomposed Conformance Checking

ProblemFitness in Data-aware Models

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

Initial ExaminationAllergy Test - FAIL Blood Test - PASS

Radiology Test - PASS Diagnosis - HOME

Home Care

tests diagnosis

Large MedicalData-aware Models

ContextData-aware Conformance Checking

• Existing techniques for data-aware fitness checking are time-consuming based on

A* (control-flow) + ILP (data)

• Decompose the data-aware fitness problem.

• Meaningful decomposition and diagnostic results.

• Formal guarantees on the fitness correctness.

• Fast compared with the monolithic approach.

ContributionsValid Decomposition of Data-aware Models

No Synchronization

• Shared places can be out of synchronization during the fitness checking.

• Valid Decompositions (no places or arcs shared) preserve the fitness.

Theorem: Valid Decomposition of Petri nets with data (no shared places, arcs, or data variables) preserves the

fitness.

No Synchronization

Details in the thesis

• Petri nets with Data are graphs.

• Decomposition based on SESEs for comprehensive results.

ContributionsValid Decomposition of Data-aware Models• Improve in the control flow + improve in the data

5 7 9 11 13 15 17 19 21 23 250

DecompNo Decomp

Average number of events per event-log trace

Real case: Dutch municipality From 52891 seconds to 52 seconds (99%)

• Fitness Decomposition• Decomposed Conformance Checking• Topological and Multi-level Conformance Diagnosis• Data-aware Decomposed Conformance Checking

• Event-based Real-time Decomposed Conformance Checking

ProblemReal-life Monitoring of Hospital Processes

Hospital Processesrunning

Large Process Model

Process-awareMonitoring System

ConformanceReports

ConformanceAlarms

ContextEvent-based, Fast, and Comprehensible

• Fitness real-life monitoring architecture for large process models.

• Based on events, not in complete traces.

• Real-time requires time efficiency

• Comprehensive results as part of the monitoring procedure.

ContributionsEvent-based Real-time Decomposed Fitness

DecomposedModel

Stream of Events

ContributionsDecomposition based on SESE

ContributionsEvent-based Real-time Decomposed Fitness

• Heuristic Replay

• Faster compared with alignments.

• Consequences of bad decisions are limited to the fragment.

• Event based.

• Not optimal, but heuristic.

a c fa c f

Log Trace

Replay

Look-aheadHeuristic

ContributionsExample of Real-time Decomposed Fitness

CONCLUSIONS

Contributions of the ThesisContribution

Precision Approach to quantify and analyze the precision between a log and a model based on escaping arcs.

Robustness and confidence interval for precision based on escaping arcs.

Severity assessment of the imprecision point detected.

Precision checking based on aligning observed and modeled behavior.

Abstraction and directionality in precision based on alignments.

Fitness Decomposition

Decomposed conformance checking based on SESE components.

Hierarchical and topological decomposition based on SESE components for conformance diagnosis.

Decomposed conformance checking for data-aware models.

Decomposed conformance checking for real-time scenarios.

Publications of the Thesis (Precision)Jorge Munoz-Gama, Josep Carmona

A Fresh Look at Precision in Process Conformance BPM 2010 – pp. 211 - 226

Jorge Munoz-Gama, Josep Carmona Enhancing precision in Process Conformance: Stability, confidence

and severity. CIDM 2011 – pp. 184-191

Jorge Munoz-Gama, Josep CarmonaA General Framework for Precision Checking

Journal of Innovative Computing, Information and Control – vol.8 no.7B

Arya Adriansyah, Jorge Munoz-Gama, Josep Carmona, Boudewijn F. van Dongen, Wil M. P. van der Aalst

Alignment Based Precision Checking BPM Workshops 2012 – pp. 137-149

Arya Adriansyah, Jorge Munoz-Gama, Josep Carmona, Boudewijn F. van Dongen, Wil M. P. van der Aalst

Measuring precision of modeled behaviorInformation Systems and e-Business Management

Publications of the Thesis (Decomposition)

Jorge Munoz-Gama, Josep Carmona, Wil M. P. van der AalstConformance Checking in the Large: Partitioning and Topology

BPM 2013 – pp. 130-145 – Best Student Paper Award

Jorge Munoz-Gama, Josep Carmona, Wil M. P. van der Aalst Hierarchical Conformance Checking of Process Models Based

on Event LogsPetri Nets 2013 – pp. 291-310

Jorge Munoz-Gama, Josep Carmona, Wil M. P. van der AalstSingle-Entry Single-Exit Decomposed Conformance Checking

Information Systems – vol.46 pp. 102-122

Massimiliano de Leoni, Jorge Munoz-Gama, Josep Carmona and Wil M.P. van der Aalst

Decomposing Conformance Checking on Petri Nets with DataCoopIS 2014 – pp. 3-20

Seppe K.L.M. vanden Broucke, Jorge Munoz-Gama, Josep Carmona, Bart Baesens and Jan Vanthienen

Event-based Real-time Decomposed Conformance AnalysisCoopIS 2014 – pp. 345-363

Impact of the Thesis

• Published in international journals and international conferences

• Best Student Paper Award in BPM 2013(Acceptance Rate 14%)

• Extensively used in the field• 150 citations • Used for:

measure precision and fitness in models evaluate discovery algorithms guide discovery techniques based on genetic algorithms CoBeFra framework recommender systems trainning

Directions for Future Work

• New metrics, new dimensions

• Decomposed alignment of observed and modeled behavior

• Decomposed conformance for other dimensions

• Visualization and diagnosis

• Model repair

Thesis and Acknowledgements

• More details in:

• … and to all the people that made this work possible, THANKS! 122

Jorge Munoz-GamaAdvisor: Josep Carmona

December 2014

AND DIAGNOSIS IN PROCESS MINING

Backup Slides

Escaping arcs: points where the model allows more behavior than the one observed in the log.

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

ProcessSimulation

Software

SimulationResults

ProcessSimulation

Software

SimulationResults

Initial Examination

Allergy Test

Blood Test

Radiology Test

Diagnosis

Hospital Treatment

Home Care

jorge munoz-gama advisor: josep carmona december 2014 conformance checking and diagnosis in process...

contributions precision

process mining slide

measure precision

precision problems

importance of precision

precision etcp

escaping arc slide

biased vision slide

Documents

roberto munoz

bonsai d’interieur origine chine · 2019-11-15 · bonsai...

carmona carmona juana_cev_tarea 1.2 (2)

elena munoz i_iforo_andece

munoz et al_2011_investigaciondeoperaciones.pdf

munoz booklet

gabriel munoz portfolio

decomposed conformance jorge munoz-gama, josep carmona and...

andres munoz portfolio

ruth munoz realidad_aumentada

event-based real-time decomposed conformance analysis seppe...

munoz liturgia curso

munoz inmobiliaria

munoz disidentifications

bonsai d’interieur origine chine - … · bonsai...

emily munoz

julia munoz

"decomposing alignment- based conformance checking of...

justin munoz portfolio

st. stephen s catholic church · anita morin, joe munoz...