the state of the nation for ontology development
DESCRIPTION
Invited talk at European Ontology Network (EUON) 2014 Ontologies are now quite big, both literally and metaphorically. They have become central resources in disciplines such as biology, medicine, healthcare and others. Such developments rely on people, tools and methods to deliver ontologies that do the desired job, on-time and on-budget. In this talk I wil ask the question of whether the tools and methods we have are capable of doing what is necessary to deliver robust and maintainable ontologies. To explore this question I will borro from the Capability Maturity Model used to assess the capabilities of institutions to deliver software projects. Instead of institutional assessment, I will bend the CCM to the discipline of ontology engineering. The levels of the CMM range from the ad hoc to one where metrics are used to monitor and adjust ontology development. In this talk I will use some audience participation to gather views on ontology engineering maturity level and then deliver my own view of that maturity.TRANSCRIPT
http://www.cs.manchester.ac.uk/our-research/
Robert StevensSchool of Computer Science
University of ManchesterOxford RoadManchester
United KingdomM13 9PL
The state of the Nation for Ontology Development
What’s the state of ontology development?
Ontologies are fairly well established as supports to information systems
We have KR languages like OWL that are widely used
We have at least one well established OWL API
We have varieties of tools that are research outcomes
We have lots of opinions on development processes
How mature are we as an engineering discipline and what does it say about where to go next?
What is maturity?
The term "maturity" relates to the degree of formality and optimization of processes, from ad hoc practices, to formally defined steps, to managed result metrics, to active optimization of the processes
http://en.wikipedia.org/wiki/Capability_Maturity_Model
What is maturity?
Immaturity: Ad hoc, firefighting, improvisation, lack of rigorous process management; lack of objective measures of product quality
Maturity: Organisation wide processes for managing software development and processes; processes are defined and accord with how work is done; there is communication and training; processes are monitored, analysed and updated in response
True maturity is when we no longer have heroics and value judgement and work from an evidence base
Moving from oral to fixed history
Oral history
Fixed history
The Capability Maturity Model
The Capability Maturity Model (CMM – now CMMI) is used to assess the capabilities of institutions that deliver (or not) software products
Developed by the Software Engineering Institute
Can we assess the maturity of ontology development as a discipline rather than as institutions?
A spectrum of maturity
Ad hoc and heroics
Repeatable processes
Defined processes and training
Quantitatively managed
Optimised processes
1
2
3
45
Bits of the CMM
Maturity levels – From ad hoc to metric, analysis and managed change
Key process areas – A set of related activities that when performed together achieve a goal
Goals – the features of a key process area that must exist for that area to have been implemented in an effective and persistent manner
Common features – Commitment and ability to perform, activities performed, measurement, analysis and verification
Key practices – the infrastructure and practices that contribute to the implementation of the area of activity
Software maturity levels and their key process areas
Initial
• None
Repeatable
• Requirements Management
• Software Project Planning
• Software Project Tracking and Oversight
• Software Subcontract Management
• Software Quality Assurance
• Software Configuration Management
Defined
• Organization Process Focus
• Organization Process Definition
• Training Program
• Integrated Software Management
• Software Product Engineering
• Intergroup Coordination
• Peer Reviews
Managed
• Quantitative Process Management
• Software Quality Management
Optimizing
• Defect Prevention
• Technology Change Management
• Process Change Management
A level 3 key process area
Software Product
Engineering
Analysis
Design
Coding
Testing
Documentation
Metrics for key process areas
Requirements Management
KPA
Requirement status
Requirements management
Requirements stability
has metrics for
has metric
s for
has metrics for
Can We Apply this to a Discipline?
The CMM looks at maturity in an organisation
It looks at the processes and their management (in its broadest sense)
A key is to have common practices in that organisation
There has to be something to manage
For a discipline we don’t need one common practice to be in place
…, but we do need common, replicable practices to exist
The question is, are we capable of being mature?
What maturity doesn’t mean
Everyone in the discipline does the same things the same way We don’t all need to use the same methods
But we do need to make those methods repeatable
This can be done for as many styles of ontology development as we see fit
Being level one doesn’t mean a decent ontology cannot be made
How to make a set of ontology development CMM process areas
Initial
• None
Repeatable
• Requirements Management
• Ontology Project Planning
• Ontology Project Tracking and Oversight
• Ontology Subcontract Management
• Ontology Quality Assurance
• Ontology Configuration Management
Defined
• Organization Process Focus
• Organization Process Definition
• Training Program
• Integrated Ontology Management
• Ontology Product Engineering
• Intergroup Coordination
• Peer Reviews
Managed
• Quantitative Process Management
• Ontology Quality Management
Optimizing
• Defect Prevention
• Technology Change Management
• Process Change ManagementWell, it's not really that easy
A level 3 key process area for ontology development
Ontology Product
Engineering
Analysis
Design
Coding
Testing
Documentation
A collection of ontology development practices
Knowledge gathering and requirements
• Specification • Knowledge
Acquisition • Elicitation
Modelling and implementatio
n
• Conceptualis. • Formalization • Enrichment • Update • Repair • Reuse • Integration
Testing
• Evaluation • Quality
Assurance • Assessment • Verification
Development process
management
• Configuration Management control
• Versioning • Documentation
Do we know how to do these, let alone manage them?
How mature is the ontology development discipline?
Still too much ad hoc and heroics in the discipline
Can still produce good ontologies, but…
We have parts of the infrastructure (technology readiness level?)
We have some of the defined processes
Not much in the way of metrics
We’ve come a long way, but…
What should happen next?
We can transfer a lot from software engineering
But we need to know more about what needs to be done and how it is done – and then make the tools to support it
Then we can manage it
Increased sociotechnical understanding of the development processes for ontologies
Tool support for those processes
The ability to measure more and to depend on value judgements less
Acknowledgements
Nico Matentzoglu did the pictures
Misconceptions of the capability Maturity Model, Karl E. Wiegers at www.processimpact.com
Capability Maturity Model for Software, Version 1.1. Mark C. Paulk, Bill. Curtis, Mary Beth Chrissis, Charles V. Weber
Towards a glossary of activities in the ontology engineering field MC Suárez-Figueroa, A Gómez-Pérez