Process and Open Source Software

Walt ScacchiInstitute for Software Research

UC Irvine

http://www.ics.uci.edu/~wscacchi/Presentations/Process/Informatix-Process-Lecture.ppt

Overview

Core concerns Software process and process modeling

Backstory/case study History of software process Current challenges in software process Open source software processes Best practices

Core concerns Process(es)

What to do and how to do it Tasks, inputs, outputs, performers, tools

Performers enact tasks using tools to transform inputs into outputs

Tasks are partially ordered Tasks, inputs, performers, tools may have pre-conditions Tasks, outputs, performers may have post-conditions

Organize and coordinate who does what, (with what resources), when, where, how, and why

Formal modeling of processes enables systematic querying and reasoning about complex processes

Core concerns Process modeling

Understanding, analyzing, and (re)designing software development processes

Models may be either Prescriptive—what should be done and how to do it Proscriptive—what might be done and how Descriptive—what was done and how (historic record)

Models may be either Narrative Semi-structured (form-based; hypermedia-based) Graphic (attributed flow graphs) Formal (language-based process programs or specifications)

Families of models inter-related through meta-model Meta-model provides model ontology and process epistemology

Backstory/case study Major TelCo wants to develop broadband multi-media

telecommunications system Anticipates $1B software development, up to 1500

software developers working 2-3 years Seeks industrial/IT partners to provide supporting

infrastructure to reduce risk IT partner wants to showcase new “process support

technology” products as sales lead IT partner brings in academic research team to analyze

and advise TelCo on “process issues”

Backstory

Academic team, IT partner, and major TelCo jointly elicit, capture, codify (model) and inter-relate TO-BE system development process.

Academic team employs IT partner’s products to present results of their “process analysis”

IT partner considers its process-centered product major success to help large sales engagements

Academic team view of their effort -- a major success for publication, but...

P.K. Garg, P. Mi, T. Pham, W. Scacchi, and G. Thunquest, The SMART Approach to Software Process Engineering, Proc. 16th. Intern. Conf. Software Engineering, IEEE Computer Society, Sorrento, Italy, 341-350, May 1994.

A complex organizational process (and model): a decomposition-precedence relationship view

(19 levels of decomposition, 400+ tasks)

W. Scacchi, Experience with Software Process Simulation and Modeling, J. Systems and Software, 46(2/3):183-192,1999.

Backstory

Academic team suggests overall process will not succeed -- too complex, too much delegation, problematic hand-offs (“throwing it over the wall”)

TelCo and IT partner dismiss academic team Less than one year later, IT vendor abandons process

technology product Two years later, business press reports TelCo

experiences major project failure and losses greater than $200M, and no system.

Software process model history

Process flow charts Software life cycle (“waterfall” diagram) Incremental development Spiral model Process programs (formal specification) Capability maturity models and industrial/military

standards Not process models--only focus on what to do, but not

how to do it.

Software Development Process Model, c. 1955.

Software Development Process Model, c. 1960.

Software Development Process Model, c. 1970.

Recent software process challenges

Distributed, decentralized, and/or global software development

Process improvement Process design optimization or redesign Continuous process improvement (learning)

Articulating software evolution processes Open source software development processes Process discovery

(semi-)automated extraction of process events, conditions, timestamps, and other meta-data from software development artifacts

J. Noll and W. Scacchi, Supporting Software Development in Virtual Enterprises, Journal of Digital Information 1(4), February 1999.

Free/Open Source Software Development Processes and

Practices

What is free/open source software development?

Free (as in “freedom”) vs. open source Freedom to access, browse/view, study, modify and

redistribute the source code Free is always open, but open is not always free

F/OSSD is not “software engineering” Different: F/OSSD can be faster, better, and cheaper than

SE in some circumstances F/OSSD involves more software development tools, Web

resources, and personal computing resources

OSS Development Models

Free Software (GPL) Open Source (BSD/MIT, Mozilla, Apache) Corporate Source (Hewlett-Packard) Consortium/Alliance (OSDL, SugarCRM) Corporate-Sponsored (IBM-Eclipse, Sun-Netbeans,

Sun-OpenOffice, HP-Gelato) Community Source (Sakai, Westwood)

OSSD Project Characteristics

OSS Developers are always users of what they build, while OSS users (>1%) are also OSS developers

Requires “critical mass” of contributors and OSS components connected through socio-technical interaction networks

OSSD projects emerge/evolve via bricolage Unanticipated architectural (de)compositions Multi-project component integrations

OSSD teams use 10-50 OSSD tools to support their development work

OSSD Project Characteristics

Operational code early and often--actively improved and continuously adapted

Post-facto software system requirements and design OSSD is not Software Engineering OSSD has its own “-ilities” which differ from those for

SE Caution: the vast majority of OSSD projects fail to

grow or to produce a beta release.

SourceForge.net

Google Summer of Code 2007

OSS Processes for Requirements or Design

OSS Requirements/Designs not explicit not formal

OSS Requirements/Designs are embedded within “informalisms”

Example OSS informalisms to follow (as screenshot displays)

OSS Requirements/Design processes are different from their SE counterparts.

SE vs. OSS processes for Requirements• Elicitation• Analysis • Specification and

modeling• Validation

• Communicating and managing

• Post-hoc assertion• Reading, sense-

making, accountability• Continually emerging

webs of discourse• Condensing and

hardening discourse• Global access to

discourse

Evolutionary redevelopment, reinvention, and redistribution in OSS One evolutionary dynamic of OSSD is reinvention

Reinvention enables continuous improvement OSS evolve through minor mutations

Expressed, recombined, redistributed via incremental releases OSS systems co-evolve with their development

community Success of one depends on the success of the other

Closed legacy systems may be revitalized via opening and redistribution of their source When enthusiastic user-developers want their cultural experience

with such systems to be maintained.

Configuration management and work coordination in OSSD

Use CM to coordinate and control who gets to update what part of the system/online artifacts. Many OSSD projects use CVS (single centralized code

repository with update locks) and frequent releases (daily releases on active projects)

Linux Kernel: Git (multiple parallel builds and release repositories)

Collab.Net and Tigris.org: Subversion (CVS++) Apache: Single major release, with frequent “patch”

releases (e.g., “A patchy server”) GNU arch seeks to develop Free CM unification

Project management and career development in OSSD

OSSD projects self-organize as a meritocractic role-hierarchy and virtual project management Meritocracies embrace incremental innovations over radical

innovations VPM requires people to act in leadership roles based on skill,

availability, and belief in project community

OSS developers value freedom of choice and expression. want to learn about new stuff (tools, techniques, skills, etc.), have fun building software, exercise their technical skill (e.g., developing reusable code), try out new kinds of systems to develop, and/or interconnect multiple F/OSSD projects

(images from A.J. Kim, Community Building on the Web, 2000)

A meritocractic role hierarchy for OSSD

------------------------------------------------

Socio-technical and cultural evolution processes in OSSD

New kinds of OSSD processes under study Joining and contributing to a project in progress Role-task migration: from project periphery to center Alliance formation and community development

Independent and autonomous project communities can interlink via social networks that manipulate objects of interaction Enables possible exponential growth of interacting and

interdependent community as socio-technical interaction network

Discovering the what and how of OSSD processes

Practitioner reflections and anecdotes Surveys Ethnographic field work (virtual ethnography) Mining OSSD project repositories Multi-modal modeling, analysis, and validation of

OSSD processes

Sun Microsystems

The BoardRelease Manager

Maintainer Developers/ Contributors

Website

Mailing Lists

Users

Contribute to community

, meet time constraints for the release

Maintain a project/ module, manage a group of developers

Ensure that the netbeans

community is being run in a fair and open manner

Start new release phase, propose schedule/plan

CVS

Funds, support, Promote Java/Open source

IssueZilla

QA Team

Produce Q- builds and ensure quality of the software

release proposal, release updates,

branch for current release, release post mortem,

review release candidates (2) & decide final release

download development builds and test

, release Q-builds

download new release

report bugs

select feature to develop

, bug to fix, download netbeans,

commit code

decide features for the project and merge patches/bug fixes

, create module web page

make decisions for the community, on high level

grant CVS commit privilege to developers

Link to all Use Cases Links to all Agents

CVS Manager

Configure and maintain CVS

grant access

Site Administrator

Manage website

deploy builds

SourceCast

Tools

Link to Tools

Download and use free software

Community Manager

Share knowledge and ensure all community issues are addressedrespond to tech issu

es, unanswered questions

Rich Picture

NetBeans

Formal model of an OSSD process coded in PML (excerpt) ... sequence Test { action Execute automatic test scripts { requires { Test scripts, release binaries } provides { Test results } tool { Automated test suite (xtest, others) } agent { Sun ONE Studio QA team } script { /* Executed off-site */ } } action Execute manual test scripts { requires { Release binaries } provides { Test results } tool { NetBeans IDE } agent { users, developers, Sun ONE Studio QA team, Sun ONE Studio developers } script { /* Executed off-site */ } } iteration Update Issuezilla { action Report issues to Issuezilla { requires { Test results } provides { Issuezilla entry } tool { Web browser } agent { users, developers, Sun ONE Studio QA team, Sun ONE Studio developers } script { <br><a href="http://www.netbeans.org/issues/">Navigate to Issuezilla </a> <br><a href="http://www.netbeans.org/issues/query.cgi">Query Issuezilla </a> <br><a href="http://www.netbeans.org/issues/enter_bug.cgi">Enter issue </a> } } …

PML validation analysis

Summary of analysis for netbeans_req_release.pmlModel size (source lines): 307Actions: 36Resources: 72Actions neither requiring no r providing resources: 1Resources required but not provid ed (potential inputs): 0Resources provided but not required (potential outputs): 0Miracles: 2Black holes: 6Transfo rmations: 30

Best practices

Processes with explicit process models are easier to manage, analyze, improve, distribute, and reuse

New/ unfamiliar software tools and techniques are best candidates for software process support

Process meta-modeling enables process life cycle engineering and formal reasoning about processes

OSSD is a community building process not just a technical development process F/OSS peer review creates a community of peers

OSSD processes often iterate daily versus infrequent singular (milestone) Software Life Cycle Engineering events OSSD: frequent, rapid cycle time (easier to improve) vs. SLC: infrequent, slow cycle time (harder to improve)

Best practices

Easiest to improve a process that is formally modeled

Process management and improvement have been one of the most enduring practices in Software Engineering for improving productivity and quality, and to reducing cost and risks.

Research opportunities

Research opportunities

FOSSD is poised to alter the calculus of empirical SE Software process discovery, modeling, and

simulation Repository mining can support software

visualization, refactoring/redesign studies Comparison of SE versus FOSSD approaches to

software inspection and peer review

Research opportunities

Based on results from individual motivation, participation, role migration, and turnover in FOSSD projects, SE world would benefit from empirical studies that examine similar patterns in conventional software development projects Is FOSSD more fun, interesting, and rewarding than

SE?

Research opportunities

Conventional software cost estimation techniques (e.g., “total cost of operation”) slight/ignore social capital and socio-technical resources Miscalculation of total resources and capabilities

that affect predicted/actual costs of software development or FOSSD

Research opportunities

Results from study of cooperation, coordination and control in FOSSD Virtual project management and role migration

processes can provide a lightweight approach to SE project management

Unclear whether proprietary software projects willing to embrace VPM

Research opportunities

Alliance formation and social networking results suggest SE projects operate at a disadvantage compared to FOSSD projects SE projects tend to produce systems whose

growth/evolution is limited FOSSD projects can produce systems capable of

sustained exponential growth/evolution of both software and developer-user community

Research opportunities

How best to encourage the emergence of a social movement that combines best practices of FOSSD and SE Consider participation or study of open source

software engineering (OSSE) projects at Tigris.org OSSE seeks to combine SE and FOSSD tools,

techniques, and concepts

Detailed study report available

W. Scacchi, Free/Open Source Software Development: Recent Research Results and Methods, in M. Zelkowitz (ed.), Advances in Computers, Vol. 69, 243-295, 2007.

• http://www.ics.uci.edu/~wscacchi/Papers/New/Draft_Chapter_Scacchi.pdf

Acknowledgements

Project collaborators: Mark Ackerman, UMichigan, Ann Arbor Les Gasser, UIllinois, Urbana-Champaign John Noll, Santa Clara University Margaret Ellliot, Chris Jensen, UCI-ISR and others ar ISR

Funding support: National Science Foundation: #0083075, #0205679, #0205724,

#0350754 and #0534771. Digital Industry Promotion (DIP) Agency, Global R&D Collaboration

Center, Daegu, South Korea No endorsement implied.