defining an open source software trustworthiness model
DESCRIPTION
This presentation show the results of my PhD thesis.Modern society depends on large-scale software systems of astonishing complexity. Because the consequences of their possible failure are so high, it is vital that software systems should exhibit a trustworthy behavior.Trustworthiness is a major issue when people and organizations are faced with the selection and the adoption of new software. Although some ad-hoc methods have been proposed (see for instance OpenBQR, OpenBRR and QSOS), there is not yet general agreement about the software characteristics contributing to its trustworthiness.Therefore, this work focuses on defining an adequate notion of trustworthiness of Open Source Software products and artifacts and identifying a number of factors that influence it to provide both developers and users with an instrument that guides them when deciding whether a given program (or library or other piece of software) is “good enough” and can be trusted in order to be used in an industrial or professional context.More details on www.taibi.itTRANSCRIPT
Davide Taibi
Università degli Studi dell’Insubria
Defining an Open Source Software Trustworthiness Model
Defining an Open Source Software Trustworthiness Model
Advisor: Prof. Sandro MorascaReviewer: Prof. Alberto SIllitti
Outline
• Motivations and Research Goals• What Trustworthiness is• How to measure trustworthiness• The Approach
– Trustworthiness Factors– Tool Definition and Building– Model Building
• Conclusions
15-09-2010 Defining an Open Source Software Trustworthiness Model
Motivation
Who is behind Open Source?
Why to be confident in OSS?
How can I make users confident in my software?
Open Source Trustworthiness ModelOpen Source Trustworthiness Model15-09-2010 Defining an Open Source Software
Trustworthiness Model
Research Goals
• OSS Trustworthiness Evaluation– Evidence-based approach
• experiments• static and dynamic measures• testing• tools• validated models
• Tools for evaluating OSS trustworthiness
15-09-2010 Defining an Open Source Software Trustworthiness Model
What Trustworthiness is
15-09-2010 Defining an Open Source Software Trustworthiness Model
http://www.hwupgrade.it/articoli/stampa/portatili/1160/peso.jpg
How to measure trustworthiness
TRUSTWORTHINESS
15-09-2010 Defining an Open Source Software Trustworthiness Model
The Approach
15-09-2010 Defining an Open Source Software Trustworthiness Model
Trustworthiness Factors identification
• Factors Identification 151 Interviews
• Interviews to understand– The confidence parameters of trustworthiness– The roles of the involved individuals– The problem domains– The correlations between the first 3 aspects
15-09-2010 Defining an Open Source Software Trustworthiness Model
Trustworthiness Factors
Interviews - Roles
35.00%
21.70%28.30%
5.00%13.30%
25.00%
0.00%5.00%
10.00%15.00%20.00%25.00%30.00%35.00%40.00%
15-09-2010 Defining an Open Source Software Trustworthiness Model
Trustworthiness Factors• Economics
– ROI– TCO
• Quality– functional requirements– reliability– performance– usability– maintainability– portability– size– complexity– modularity– standard architecture– patterns– standard compliance– self containedness– interoperability– localization
• Development– type of licenses– tools– best practices– documentation– environment– training / guidelines– user community– maintainer organization– short term support– reputation of vendor– distribution channel– language uniformity– user community that witness quality– benchmarks / test suites
• Customer– customer satisfaction– interoperability issues– law conformance– standard imposed
15-09-2010 Defining an Open Source Software Trustworthiness Model
Analysis of relevant projects
• Objectives:• finding what kind of information is out there
to help “users” choose• finding what kind of information is missing• checking if there is a gap between “demand”
and “supply”
15-09-2010 Defining an Open Source Software Trustworthiness Model
Analysis of relevant projects
• Results:• Some factors are not directly assessable
• Proxy-measures defined
• Some factors need some tools to be developed
• Other factors can not assessed unless developers provide the information (e.g., the number of downloads)
15-09-2010 Defining an Open Source Software Trustworthiness Model
Analysis of relevant projects
• Example: The degree to which an OSS product satisfies / covers functional requirements
15-09-2010 Defining an Open Source Software Trustworthiness Model
Analysis of relevant projects
• Results:
Open Product Portal Assessment Model
www.op2a.tk
• 44 Portals analyzed• Apache Tomcat Portal refactoring (proposal)
15-09-2010 Defining an Open Source Software Trustworthiness Model
Model Building
•Objectives:• Definition of measures, starting from the factors we identified• Reuse/define sensible measures
An initial set of measures has been defined, to capture these dimensions from different viewpoints in a quantitative wayAn initial set of measures has been defined, to capture these dimensions from different viewpoints in a quantitative way
Use of a goal-oriented approach: Goal/Question/Metric paradigmUse of a goal-oriented approach: Goal/Question/Metric paradigm
15-09-2010 Defining an Open Source Software Trustworthiness Model
Model Building• Product related factors
15-09-2010 Defining an Open Source Software Trustworthiness Model
Model Building• Process related factors
15-09-2010 Defining an Open Source Software Trustworthiness Model
Tools identification and Building
• Objectives:• Definition and building of the tools required for the assessment of GQM metrics
• Steps:• Check existing OSS tools• Build new tools
15-09-2010 Defining an Open Source Software Trustworthiness Model
Toos Identification and Building
• Developed– MacXim (static code analysis tool)
– qualipso.dscpi.uninsubria.it • Reused
– Spago4Q– STATsvn– FOSSology– Junit
15-09-2010 Defining an Open Source Software Trustworthiness Model
Experimentation
• Objective:– build OSS trustworthiness models that are
• goal-oriented• evidence-based• customizable
• Steps– a specific measure repository was designed– a suitable tool for statistical analysis was selected and
suitable scripts were coded– a questionnaire for collecting users’ opinions on OSS
trustworthiness and other qualities•44 OSS projects (22 Java and 22 C++ projects)15-09-2010 Defining an Open Source Software
Trustworthiness Model
Experimentation
• Experiments:– 565 questionnaires were collected– 3750 product evaluations collected with the questionnaire
(6,63 evaluations per questionnaire)– Correlations between objective and subjective measures
• Results:– a set of statistically significant models (MOSST: Model of Open Source Software
Trustworthiness)• between measurable code attributes (the X's) and the evaluated
trustworthiness of OSS products (the Y), evaluated reliability (the Y)• a few correlational models between the measurable internal
characteristics of OSS .
15-09-2010 Defining an Open Source Software Trustworthiness Model
Experimentation
0.5
0.6
0.7
0.8
0.9
1.0
GoodGood
NotNotgoodgood
AcceptAcceptableable
15-09-2010 Defining an Open Source Software Trustworthiness Model
Experimentation
6 7 8 9 10 11
0.5
0.6
0.7
0.8
0.9
1.0
x
rtru
stTr
ustw
orth
ines
s
0.50.6
0.70.8
0.91.0
15-09-2010 Defining an Open Source Software Trustworthiness Model
Experimentation
15-09-2010 Defining an Open Source Software Trustworthiness Model
Experimentation Subjective qualities vs. measures
Subjective quality Objective measure Outcome
Reliability CBO Reliability LCOM Reliability McCabe (class average) Reliability Size (total eLOC) Reliability Total num. methods Reliability Total num. classes Trustworthiness CBO Trustworthiness LCOM Trustworthiness McCabe (class average) Trustworthiness Size (total eLOC) Trustworthiness Total num. methods Trustworthiness Total num. classes Trustworthiness Size (total) & McCabe (class average) Trustworthiness Num methods & McCabe Trustworthiness Num classes & McCabe
15-09-2010 Defining an Open Source Software Trustworthiness Model
Subjective quality Subjective quality Outcome
Trustworthiness Reliability (logistic)
Trustworthiness Reliability (linear)
Trustworthiness Reliability (non-parametric)
Trustworthiness ReliabilityGood, ReliabilityBad (linear)
TrustworthinessGood ReliabilityGood (non-parametric)
Trustworthiness Reusability (linear)
Trustworthiness Interoperability (linear)
Trustworthiness Efficiency (linear)
Trustworthiness Documentation (linear)
Trustworthiness Usability Trustworthiness Portability Trustworthiness Functionality Trustworthiness Security Trustworthiness Efficiency Trustworthiness Community support
Experimentation Correlations between subjective qualities
15-09-2010 Defining an Open Source Software Trustworthiness Model
Experimentation Correlations between measures
Objective var Objective var Outcome
Size (total eLOC) Total num. methods (log-log)
Size (total eLOC) Total num. classes (log-log)
Size (total eLOC) Total num. classes & methods
Total num. methods Total num. classes (linear)
15-09-2010 Defining an Open Source Software Trustworthiness Model
Experimentation Trustworthiness vs popularity
15-09-2010 Defining an Open Source Software Trustworthiness Model
Conclusions
- MOSST: Model for Open Source Trustworthiness- Macxim: Static code analysis tool- OP2A: OSS Product Portal Assessment model
15-09-2010 Defining an Open Source Software Trustworthiness Model
Thanks
15-09-2010 Defining an Open Source Software Trustworthiness Model