critical success factors in software process improvement: research

Critical Success Factors in SPI: Research Methods and Organizational Paradigms 1

Copyright © 2005, David J. Williamson

Critical Success Factors in Software Process Improvement: Research Methods and Organizational Paradigms

David J. Williamson

Capella University

June 19, 2005

Abstract

This study reviews 30 representative contributions to the software process improvement (SPI) litera-ture, first assessing history, broad themes, and research methods, then categorizing each article by its stated or implied organizational paradigm. In addition to the traditional rational, natural, and open systems views, the emerging paradigm of complex adaptive systems from chaos theory is applied. Based on this perspective, the factors influencing SPI success and failure are grouped, analyzed, and summarized by the organizational paradigms from which they were observed. Factors are then sub-categorized through affinity analysis into those related to management, organizations, processes, projects, teams, and users. The overall purpose of this classification scheme is to determine whether factors found to influence success and failure of SPI vary by the researcher's organizational para-digm. Significant differences are found between the factors emphasized by researchers applying dif-ferent paradigms. Factors vary by distribution among the subcategories and by content within the subcategories. Implications for practioners and researchers include the necessity of understanding the operative organizational paradigm before undertaking SPI or SPI research. Keywords: Software Process Improvement, Research Methods, Organizational Paradigms, Complex Adaptive Systems

oftware Process Improvement (SPI) is difficult (Caputo, 1998), complex (Gra-dy, 1997; Kautz & Nielson, 2004), and

failure-prone (McConnell, 2001). Organiza-tions frequently experience mixed results (Rainer & Hall, in press), with failure rates approaching 70% (Statz, Oxley, & O'Toole, 1997). However, effective SPI has the poten-tial to make significant contributions to orga-nizational efficiency and profitability (Con-strux, 2002; Herbsleb, Carleton, Rozum, Seie-gel, & Zubrow, 1994; Paulk, Weber, Garcia, Chrisis, & Bush, 1993).

Scholarly and practitioner literature sug-gests a confusing array of factors influencing the success and failure of SPI. It is difficult to know where to focus efforts. Depending on the sources consulted, would-be SPI practi-tioners are advised to increase process formal-ity, or decrease process formality. They are

told that management commitment is the most important factor, or they are told that management should permit the effort to or-ganize itself. They are advised that SPI projects must be planned, controlled, tracked, measured, and reviewed, or that projects should use evolutionary methods and minim-ize interdependencies.

To distill some order from this confusion, a sample of 30 SPI studies were analyzed (see Table 1) to gain insights into the research me-thods and organizational paradigms applied by the researchers. Studies were organized first by the research approach described, then by use of a screening model developed for the paper to determine organizational paradigm. In addition to the traditional rational, natural, and open systems views, the emerging para-digm of complex adaptive systems from chaos

S



theory was applied (Lorenz, 1963; Peculis, 2005; Rogers, 1993; Swanson, 1994).

Specifically, the paper addresses the fol-lowing research questions:

RQ1: What research methods are applied to SPI research?

RQ2: What organizational paradigms are applied to SPI research?

RQ3: What are the critical factors affect-ing SPI success and failure?

RQ4: Do the factors suggested differ by the organizational paradigm of the researcher?

The remainder of the paper is organized as follows: the Overview of SPI Literature describes SPI history, broad themes, seminal sources, and research approaches. The Me-thods section describes the selection of ar-ticles, the development and application of the screening model used to determine researcher organizational paradigm, and the grouping of factors with affinity analysis. The Results sec-tion summarizes the findings related to the research questions. The Discussion section describes the apparent influence of organiza-tional paradigm on the critical SPI factors de-tected by researchers, as well as some limita-tions of the study. The Conclusions section summarizes the findings and suggests implica-tions for practitioners and researchers, and questions for future research. Following the References section, the Appendix contains five tables listing the studies reviewed, re-search methods applied, article attributes used to determine organizational paradigm, catego-rization of studies by organizational paradigm, and critical SPI factors by organizational pa-radigm and affinity group.

Overview of the SPI Literature

The SPI literature is relatively new, but ex-tensive. One recent literature survey (Hansen, Rose, & Tjornhoj, 2004) investigated 322 ar-ticles. An EndNote library available on the internet (Rose, 2004) contains 448 references; all but seven are from 1981 or later. Swanson and Ramiller included several SPI-related top-

ics in their 1993 taxonomy of information sys-tems research topics.

Leading journals include those from IEEE and ACM, Communications of the ACM, MIS Quarterly, the Database for Ad-vances in Information Systems, the Journal of Systems and Software, Journal of Systems Ar-chitecture, Information and Software Tech-nology, American Programmer, Journal of Knowledge Management, Journal of Software Process – Improvement and Practice, and Software Quality Journal (Hansen, Rose, & Tjornhoj). Leading organizations include the Software Engineering Institute (SEI) at Car-negie-Mellon University, the Institute of Elec-trical and Electronics Engineers (IEEE) Computer Society, the Association for Com-puting Machinery (ACM), the Society for In-formation Management (SIM), and the Asso-ciation for Information Systems (AIS).

Building on this brief overview of the re-search community, the following section con-tains a brief history of SPI and a summary of its broad themes.

SPI History

Lehman (1997) suggested that the concept of software process improvement found its origin in a 1951 book by Wilkes, Wheeler, and Gill entitled "The Preparation of Programs for an Electronic Digital Computer" as a dis-cussion of the need to manage subroutines. Until recently, research continued to focus on the artifacts of software development, rather than the process itself.

In 1986, a keynote address by Osterweil (1986) at the Third International Process Workshop created interest in modeling soft-ware development processes for better under-standing. The formation of the Software En-gineering Institute at Carnegie-Mellon Univer-sity in 1984 was also followed in 1986 by its first efforts to develop the Capability Maturity Model. The growth of software process im-provement then accelerated rapidly with the 1991 publishing of SEI's Capability Maturity Model for Software, Version 1.0 (SEI, 2004).



Despite its brief history, some seminal sources and broad themes regarding SPI have emerged.

Broad Themes, Seminal Sources

In its 15 to 20 year history, the SPI com-munity has witnessed the development of several broad themes, including process frameworks, process assessments, process improvement, and most recently, complex adaptive systems.

The Capability Maturity Model for Soft-ware (CMM-SW), Software Process Im-provement and Capability dEtermination (SPICE) model, and ISO 9001 are prominent examples of SPI frameworks. CMM and SPICE are specifically directed toward soft-ware processes, while ISO 9001 addresses quality process documentation in general. Conradi (1997) compared these three ap-proaches with Total Quality Management (TQM), the Quality Improvement Paradigm (QIP), and the European BOOTSTRAP and AMI methods. He concluded that frameworks are essential descriptive, diagnostic, and pre-scriptive tools for software process improve-ment.

As such, frameworks are often the basis for software process assessment. CMM is the world's most popular SPI assessment model, with some 30,000 people trained in its use and more than 2,400 organizations undergoing assessment (SEI, 2004).

Software process assessment, except when used solely for certification, usually leads to process improvement. SPI is a growing field of scholarly and practitioner focus, with an expanding literature base and a well-established practitioner network. Prominent among practitioner SPI organizations is the Software Engineering Institute's Software Process Improvement Network (SEI-SPIN). SPIN Chapters exist in nearly every US state and in most industrialized nations (SEI, 2005).

Recently in the SPI literature, the concept of complex adaptive systems has appeared. Cusumano and Selby's 1995 best-seller "Mi-

crosoft Secrets" is still required reading at many software companies. In it, they revealed the methods used by Microsoft, the world's largest and most successful software compa-ny, for managing its software development projects, processes, and people. These me-thods, including hiring smart people who know the technology and the business; im-proving through continuous self-critiquing, feedback, and sharing; and focusing creativity by evolving software features, correlate well with recommendations from chaos theory. As early as 1987, Tom Peters advocated a new approach to management, not just of software and technology, but of business in general, based on the principles of chaos theory.

From this broad perspective of frame-works, assessment, improvement, and com-plexity, it is possible to extract a high level overview of the research methods used to study SPI.

SPI Research Methods

SPI research utilizes a variety of research methods. Arisholm, Anda, Jorgensen, and Sjoberg (1999) discussed the challenges of IS research, particularly in comparison with the natural sciences. Rather than studying natural phenomena, IS research studies the artificial interface between a man-made artifact and its environment. Controlled experiments are very difficult to construct in IS, and most empirical research consists of surveys and case studies. Further sharing of effective case study re-search methods in SPI is needed.

Alavi and Carlson (1992), Lee (1999), Mingers (2001), Orlikowski and Baroudi (1991), and Vessey, Ramesh, and Glass (2002) have all advocated diversity of IS research me-thods. Alavi and Carlson examined 908 ar-ticles published between 1968 and 1988 in eight MIS journals, classifying them by re-search strategy. They found the IS literature almost evenly weighted between empirical and non-empirical approaches, with field studies and description making up the majority of empirical articles and illustration constituting



more than half of the non-empirical articles. This study found a different proportion of empirical and non-empirical articles in the SPI literature, as well as different approaches with-in these two large categories (see Table 2).

Rainer and Hall (in press) strongly sup-ported the multiple strategy approach. Using such an approach themselves, they investi-gated the factors affecting SPI, combining qualitative and quantitative analysis of case studies, then comparing the case study results to survey data. Their results indicated differ-ent methods of investigating SPI yield differ-ent critical factors for success.

The following section describes the me-thods used in this study to select and analyze the sample of SPI literature.

Methods

Webster and Watson (2002) emphasized the importance of literature reviews in IS re-search, suggesting the IS field benefits from analysis and synthesis of its brief theoretical legacy. Their recommendations for an effec-tive literature review included the use of a conceptual model or analytical framework for surveying and synthesizing previous research.

Researchers have used many such ap-proaches to analyze SPI literature. For exam-ple, Hansen, Rose, & Tjornhoj (2004) applied a three-part framework to the primary goals of 322 articles, assessing whether each article's intent and contribution was prescriptive, de-scriptive, or reflective. Rainer and Hall (in press) assessed research strategies in their SPI literature review. Stelzer and Mellis (1999) re-viewed studies from the perspective of orga-nizational change. In this paper, organization-al paradigms and factor affinity analysis are the selected methods.

Article Selection

The 30 articles reviewed for this study were selected as representative of over 50 ar-ticles identified by searching ABI/Inform Global, Academic Search Premier, Business Source Premier, CiteSeer, Google Scholar,

IEEE's Online Library, AIS ISWorld, and the internet in general. Search terms included var-ious combinations of "software process im-provement," "factors," "success," "failure," and "critical." Additional articles were identi-fied by reviewing these articles' citations. Webster and Watson (2002) recommended going backward by reviewing citations in this manner. They also recommended going for-ward by using the Web of Science, the online version of the Social Sciences Citation index.

Organizational Paradigms

The three traditional organizational para-digms are the rational systems view, the natu-ral systems view, and the open systems view (Scott, 2003). In this paper, these three tradi-tional views are applied along with an emerg-ing fourth view, complex adaptive systems from chaos theory.

Chaos theory has been applied to organi-zational research for over 30 years--albeit sparsely at first. Cohen, March, and Olsen's (1972) classic article on "A Garbage Can Model of Organizational Choice" is a succinct description of complex adaptive systems, or chaos theory, in action within "organized anarchies" (p. 1). More recently, Thietart and Forgues (1995) and Fioretti and Visser (2004) interpreted chaos theory as "the existence of continuous processes of convergence and di-vergence, stability and instability, evolution and revolution [embedded] in every organiza-tion" (Thietart & Forgues, p. 19), and "intri-cate patterns emerging from the interaction of the constituent parts of an organization that themselves follow relatively simple behavioral views...[and not] a set of rules to solve a par-ticular set of problems, but rather a perspec-tive that may provide a new understanding to problems" (Fioretti and Visser, p. 12).

This study's classification model for orga-nizational paradigm was developed in four steps (see Table 3). First, the organizational characteristics emphasized by each of the four paradigms were summarized from Scott (2003) and Thietart and Forgues (1995). Then,



representative keywords were identified by scanning organizational descriptions. Third, Scott's summary table 5-1 (p. 108) on domi-nant organization theory models and repre-sentative theorists was culled for examples of authors representing the rational, natural, and open systems views, and chaos theory litera-ture was examined for representative refer-ences. Fourth, some well-known software de-velopment authors were associated with their apparent organizational paradigm or para-digms where they wrote from multiple views.

The resulting model (Table 3) was used to classify each article reviewed under one or more organizational paradigms. Some articles stated their paradigms explicitly, while others required more extensive application of the filtering process enabled by the model. The model was piloted on a small sample of four articles, then adjusted for greater clarity by adding additional keywords and authors. All 30 studies were then investigated using the model, and categorized according to their stated or implied organizational paradigm (see Table 4). A similar investigation was previous-ly performed on data warehousing literature (Wixom & Watson, 2001).

Factor Affinity Analysis

Once the articles were categorized by or-ganizational paradigm, they were examined for factors leading to SPI success or failure. Factors were first listed by organizational pa-radigm and then grouped using affinity analy-sis (Pande, et al., 2000). Six resulting groups of factors emerged: management, organization, process, project, team, and user factors (see Table 5). Further normalization was per-formed on the factor names, rewording them in noun-attribute sequence and combining synonyms. Factors appearing in multiple ar-ticles were given a numerical suffix in paren-theses indicating the number of times the fac-tor appeared. The normalization process was not piloted, and was highly subjective. Since the study is qualitative rather than quantitative

in nature, exact counts and totals were deemed not critical.

The results of the analysis are described in the following section.

Results

Results of the study indicate quantitative and qualitative, empirical and non-empirical research methods were all used. Organization-al paradigm was heavily weighted toward the rational systems view, but the complex adap-tive systems view is emerging. Factors found to be critical to SPI success were numerous and often contradictory, but became more comprehensible when categorized by re-searcher organizational paradigm, grouped by affinity, and normalized to noun-attribute form. In the following subsections, each re-search question is addressed individually.

RQ1: What research methods are applied to SPI re-search?

Given that most SPI literature is more re-cent than the period studied by Alavi and Carlson (1992), it is not surprising that the distribution of methods differed significantly. Approximately 83% (25 out of 30) articles applied qualitative, interpretivist methods, while 33% (10 out of 30) applied quantitative, positivist methods. The sum of these percen-tages is greater than 100% because four of the articles employed multiple, mixed methods.

Within the qualitative articles, 40% (10 out of 25) applied empirical methods and 60% (15 out of 30) applied non-empirical methods. The most popular qualitative empirical me-thods were surveys and case studies, while the most popular non-empirical methods were conceptual frameworks and models.

Within the quantitative group, only one article applied an experimental method, con-firming Arisholm, Anda, Jorgensen, and Sjo-berg's (1999) thesis that experiments are diffi-cult in IS. The other nine quantitative articles applied survey methods, with one using a cross-sectional approach.



The trend toward mixed methods in SPI research appears to be accelerating. Of the four articles in this study using mixed me-thods, all were less than five years old; two were from 2000, one was from 2003, and one has yet to be published.

RQ2: What organizational paradigms are applied to SPI research?

Not surprisingly, since the background of most IS researchers is technical and scientific, the rational systems view dominated the litera-ture sampled (see Table 4). Over half (16 out of 30) of the articles applied the rational sys-tems view, often unconsciously, as if it were a foregone conclusion that organizations and software processes could be highly forma-lized, measured, and disciplined. The CMM is the ultimate example of a rational systems view of software processes. The rational sys-tems view appeared in the literature sample as early as 1991, and as recently as 2004, so there was every indication it will continue to be ap-plied.

The next most frequent organizational pa-radigm was the open systems view, appearing in approximately one third (9 out of 30) of the articles.

Another third of the articles were nearly evenly divided between the natural systems view and the complex adaptive systems (CAS) view (4 out of 30, and 5 out of 30, respective-ly). The CAS view articles were not necessarily the more recent, with the first appearing in 1997 while the first example of the natural systems view appeared in 2000. Interestingly, none of the articles that applied multiple pa-radigms selected the CAS view, but 50% (2 out of 4) of the articles published in 2005 ap-plied the CAS view. It is possible that the CAS view is becoming more prevalent in SPI research.

To reiterate, the total number of paradigm observations exceeds the number of articles because three articles applied two paradigms (Conradi & Fugetta, 2002; Halvorsen & Con-radi, 2002), and one article used three (Kautz

& Nielsen, 2004). There did not appear to be any correlation between the application of multiple organizational paradigms and the use of multiple research methods.

RQ3: What are the critical factors affecting SPI suc-cess and failure?

Critical factors described as affecting the success and failure of software process im-provement were numerous and seemingly contradictory (see Table 5). A total of 63 fac-tors were mentioned in the 16 studies apply-ing the rational systems view, 12 factors were identified in the 4 natural systems studies, 57 factors were identified in the 9 open systems studies, and 26 factors were identified in the 5 CAS studies, for a grand total of 158 factors. Some factors like management commitment, realistic management expectations, external competition, and organizational skills and in-telligence appeared under more than one pa-radigm, but no factor appeared under every paradigm.

Overall, the most frequently cited factor was management commitment, appearing in at least 13 of the 30 studies. It was not men-tioned, however, in any study applying the natural systems or complex adaptive systems views. The second most common factor was team SPI expertise, cited in 9 of the 30 stu-dies, but again not in any study applying the natural systems or complex adaptive systems views. User involvement was the third most common factor, appearing 5 times, yet again not in any study applying the natural systems or complex adaptive systems views.

RQ4: Do the factors suggested differ by the organiza-tional paradigm of the researcher?

Yes, the most common factors, the distri-bution of factors among the affinity groups, and the content of the factors within the af-finity groups all differed significantly by re-searcher organizational paradigm.

Strikingly, studies performed from the natural systems (NS) view made almost no mention of management, except to say that



financial commitment was necessary. Studies from the other views all addressed manage-ment factors, with the rational systems (RS) view focusing primarily on buy-in, commit-ment, expectations, and leadership, and the open systems (OS) view focusing on com-mitment, measurement, expectations, and software experience. The CAS view takes an entirely different perspective on management factors, emphasizing instead management's love of change, self-organization, and shared values.

Studies from all four paradigms addressed the organization, with the OS view representing the broadest range of organiza-tion factors and the CAS view emphasizing organization flexibility, communication, and partnerships instead of more formal characte-ristics.

Studies from all four paradigms also in-cluded process factors, with the heaviest em-phasis from the RS and OS views, moderate emphasis from the CAS view, and almost no emphasis from the NS view.

As would be expected from the trends displayed in previous factor groups, the RS view showed the highest emphasis on project factors as well. The other three views detected project factors much less frequently, with the NS view barely mentioning them at all.

Again strikingly, the NS view articles made no mention of another group of factors, those related to the team. The RS and OS views mentioned them about equally, and the CAS view emphasized co-location, appropri-ate size, collaboration, knowledge, learning, and experience over the more traditional RS and OS view project factors .

Finally, only articles from the RS and OS views made any mention of user factors, in each case emphasizing user involvement in, and knowledge of, the SPI effort.

In the Discussion section, these findings are interpreted in light of the analytical model and research questions, and some of the study's limitations are addressed.

Discussion

In this study, the rational, natural, open, and complex adaptive systems views led to significantly different results when applied to the analysis of factors influencing software process improvement success and failure. The difficulty, complexity, and uncertainty of posi-tive SPI outcomes merit a better approach utilizing an organizational paradigm appropri-ate for the host organization. The potential leverage of SPI in cost reduction and profita-bility increase also motivates better results. This study is a step in that direction.

In addition, chaos theory and its applica-tion to complex, adaptive organizations is an important new field of study. It has significant implications for organizations and manage-ment (Jantsch, 1980).

Limitations of the Study

Although it is treated separately in the lite-rature, it is unclear if SPI is ultimately differ-ent from other major software or organiza-tional change efforts. It is, in essence, a com-plex change to a complex process of produc-ing a complex, changing product in a com-plex, changing environment. Osterweil's (1986) treatment of software process as meta-software supports this view.

This study did not address whether SPI li-terature and software project literature could be combined for a broader perspective. It also did not consider the rich and extensive body of literature on organizational change. How-ever, in order to expand the article selection and SPI factors addressed, some articles about other software topics were included when they were similar to SPI (Esteves, Pastor, & Casanovas, 2002; Little, 2005; Wixom & Wat-son, 2001).

The article classification model for deter-mining organizational paradigm was briefly piloted, but was not externally calibrated. The article sample was small and not likely repre-sentative; it is more of a convenience sample than a representative sample of SPI literature.



In addition, no factor analysis was per-formed on the factors identified. Factors were grouped by affinity analysis, using ad-hoc me-thods that are likely not fully repeatable.

In spite of these limitations, some signifi-cant conclusions may be drawn from this study. They are presented, along with implica-tions, in the next section.

Conclusions

Critical factors identified for SPI success depended heavily on the organizational para-digm of the researcher. Whether or not cer-tain areas such as user influence or project team structure were even investigated also depended on the paradigm applied. Further-more, the subject and content of success fac-tors within the same group varied across stu-dies from different organizational paradigms.

Implications for Practitioners

Practitioners need to understand their or-ganizations and their shared organizational paradigms before proceeding with software process improvement. Their own biases and mental models of how organizations work may have a greater impact on SPI success and failure than any other factor considered.

Organizational culture strongly influences how we perceive the challenges, opportuni-ties, and problems facing organizations (Schein, 1985). Organizational paradigm is a major contributor to culture.

Chaos theory suggests there is little man-agement can do to actually control a complex adaptive organization. Much of what passes for planning is just an exercise to make man-agers comfortable with uncertainty, and for-mal decision-making processes are often used to justify decisions after the fact (Thietart & Forgues, 1995). Managers need to overcome the "illusion of managing" (p. 24) and learn about chaos theory and organizations.

Implications for Researchers

The view of SPI success and failure clearly depends on the researcher's organizational paradigm. Researchers should recognize their own paradigms, consciously choose which paradigm to apply in SPI and IS research, and identify the paradigm in the article.

It is helpful to organize literature review results by stated or implied organizational pa-radigm. This approach has the potential to reduce contradictory results and clarify re-search outcomes.

Researchers and practitioners can benefit from increased application of the complex adaptive systems paradigm. This emerging view from the field of chaos theory has the potential to redefine many current IS research topics.

Questions for Further Research

Several questions for further research are apparent, particularly regarding the interac-tions of software processes and organizations as viewed from the complex adaptive systems paradigm, the business value of SPI, and the future of SPI practice and research.

Is chaos theory just another paradigm, or is it cor-rect that managers have little influence? The revolu-tionary prospect that management might have very little influence on the actual success of organizations is both frightening and inspir-ing. It could lead to new theories and ap-proaches to running organizations.

Is the effort to apply structured, rational processes counterproductive? Some firms find success with highly structured, rational methods, but the majority do not. Is this because the con-straints imposed by structured methods tip the balance of complex systems toward chaos?

Is SPI significantly different from other large or complex software projects or organizational change efforts? Is SPI the "mother of all software projects?" Osterweil's 1986 declaration that "software processes are software too" sug-gested that SPI may just be a very complex software (or "meta-software") project. As



such, do the factors for success in software projects apply directly to SPI?

Does software process maturity contribute to busi-ness success? Conradi & Fuggetta (2002) found the following CMM levels for three cellular telephone manufacturers:

Motorola: CMM Level 5 Ericsson: CMM Level 3 Nokia: CMM Level 1

A quick glance at the stock pages and the phones on people's hips shows business suc-cess in the opposite order.

Does SPI really help? Demarco and Lister, in their classic work Peopleware (1987), found the influence of office environment on soft-ware productivity and quality to be 2.6 times greater than software process maturity. Should we be researching office furniture and layout instead (Alexander, 1979)?

Could the application of a few well-chosen tech-niques have an even greater contribution than full-blown SPI? Chaos theory suggests this is true, with its counteracting forces and dynamic equilibrium.

Quality improvement theory comes from manufacturing, a highly rational field. Does it even apply to large scale software development, a highly chaotic field?

SPI assesses the engineering aspects of software: project management, requirements management, configuration management, etc. It does not assess the success of the software development company, e.g. profitability, competitiveness, market strategy, time to market, or user satisfaction (Conradi & Fug-getta). Is it worth the time and effort?

Do software process automation and software process technology (SPT) improve business success? Automating unstable processes makes them fail faster. Automating stable processes makes them harder to change. Should we even be trying to automate software processes?

Are chaos theory and its view of complex adaptive systems a better approach for studying organizations? Is it time to retire the rational, natural, and open systems views in favor of an emerging paradigm that could more accurately model and predict the behavior of complex organiza-

tions? Is chaos theory the natural extension of the contingency theory model (Lawrence & Lorsch, 1967)?

Information systems and organization re-searchers have some interesting and challeng-ing times ahead of them.

References

Aaen, I., & Damsgaard, J. (1998). Software process improvement: What management tends to forget. IFIP WG8.2 & WG8.6 Joint Working Conference on Information Systems: Current Issues and Fu-ture Changes Helsinki, Finland, December 10-13, 1998.

Abrahamsson, P., & Jokela, T. (2000). Development of management commitment to software process improvement. Proc. 23rd IRIS Conference, Keruu, Finland, August 2000.

Alavi, M., & Carlson, P. (1992). A review of MIS research and disciplinary development. Journal of Management Information Systems, 8(4), 45-62.

Alexander, C. (1979). The timeless way of building. New York: Oxford University Press.

Arisholm, E., Anda, B., Jørgensen, M., and Sjøberg, D. (1999). Guidelines on Conducting Software Process Improvement Studies in Industry, Proc. 22nd IRIS Conference, Keruu, Finland, 7-10 August 1999.

Auer, T. (1995). Information systems related organizational maturity: A conceptual framework and an assessment me-thod. Unpublished Doctoral Dissertation: Turku School of Economics and Business Administra-tion.

Baddoo, N., Hall, T., & Wilson, D. (2000). Implementing a people focused SPI pro-gramme. In Maxwell K et al (eds) Project Control: The human factor, Shaker Publishing, pp 373-383 Procs of ESCOM conference, Munich April 18-20

Benbasat, I., & Zmud, R. W. (1999). Empirical research in information systems: The practice of relevance. MIS Quarterly, 23(1), 3-16.

Berander, P., & Wohlin, C. (2003). Identification of key factors in software process management - a case study. Pa-per presented at the 2003 International Sympo-sium in Empirical Software Engineering (ISESE'03).

Caputo, K. (1998). CMM implementation guide: Choreo-graphing software process improvement. Reading, MA: Addison-Wesley.



Ceschi, M., Sillitti, A., Succi, G., & De Panfilis, S. (2005). Project management in plan-based and agile companies. IEEE Software, Vo-lume(May/June), 21-27.

Clegg, C. W., Waterson, P. E., & Axtell, C. M. (1996, July). Software development: knowledge-intensive work organizations. Behaviour & Information Tech-nology, 15(4), 237-249.

Cohen, M. D., March, J. G., & Olsen, J. P. (1972). A garbage can model of organizational choice. Ad-ministrative Science Quarterly, 17(1), 1-25.

Conradi, R. (1997). SPI frameworks: TQM, CMM, SPICE, ISO 9001, QIP experiences and trends, Norwegian SPIQ project. PROMO-TER/RENOIR Summer School on Software Process Technology, June 30 – July 4, 1997, Gre-noble, France.

Conradi, R., & Fuggetta, A. (2002). Improving software process improvement. IEEE Software, July/August, 2-9.

Construx. (2002). The business case for better software practices [Presentation]: Construx Soft-ware Builders.

Cusumano, M. A., & Selby, R. W. (1995). Microsoft secrets; how the world's most powerful software company creates technology, shapes markets, and manages people. New York: Touchstone.

DeMarco, T., & Lister, T. (1987). Peopleware: Produc-tive projects and teams. New York: Dorset House.

Diaz, M., & Sligo, J. (1997). How software process improvement helped Motorola. IEEE Soft-ware(September/October), 75-81.

El Emam, K., Goldenson, D., McCurley, J., & Herbsleb, J. (1998). Success or failure? Modeling the likelihood of software process improvement. (No. ISERN-98-15): International Software Engineering Re-search Network.

Esteves, J., Pastor, J. A., & Casanovas, J. (2002). Using the partial least squares (PLS) method to establish critical success factors interdependence in ERP implementa-tion projects. Paper presented at the Ninth Americas Conference on Information Systems.

Fioretti, G., & Visser, B. (2004). A cognitive inter-pretation of organizational complexity. Emergence: Complexity and Organization, 6(1-2), 11-23.

Georgiadou, E. (2003). Software process and prod-uct improvement: A historical perspective. Cyber-netics and Systems Analysis, 39(1), 125-142.

Goldenson, D. R., & Herbsleb, J. D. (1995). After the appraisal: A systematic survey of process improvement, its benefits, and factors that influence success. (No.

CMU/SEI-95-TR-009). Pittsburgh, PA: Software Engineering Institute.

Grady, R. B. (1997). Successful software process improve-ment. Upper Saddle River, NJ: Prentice-Hall.

Halvorsen, C.P., & Conradi, R. (2002, June). A taxonomy to compare software process improve-ment frameworks. Software Quality Management, 4(3).

Hansen, B., Rose, J., & Tjornehoj, G. (2004). Pre-scription, description, reflection: the shape of the software process improvement field. International Journal of Information Management, 24, 457-472.

Herbsleb, J., Carleton, A., Rozum, J., Seiegel, J., & Zubrow, D. (1994). Benefits of CMM-based software process improvement: Initial results (No. CMU/SEI-94-TR-013). Pittsburgh, PA: Software Engineering Institute.

Humphrey, W., Snyder W.S., & Willis, T.R. (1991). Software process improvement at Hughes Air-craft. IEEE Software 8(4), 11-23.

Iverson, J., Nielsen, P. A., & Norberg, J. (1998, December 10-13). Problem diagnosis in software process improvement. Paper presented at the IFIP WG8.2 & WG8.6 Joint Working Conference on Information Systems: Current Issues and Future Changes, Hel-sinki, Finland.

Jalote, P. (n.d.). The success of the SPI efforts in india: Department of Computer Science and En-gineering, Indian Institute of Technology, Kanpur.

Jantsch, E. (1980). The self-organizing universe. New York: Pergamon.

Kautz, K., & Nielsen, P. A. (2004). Understanding the implementation of software process improve-ment innovations in software organizations. Infor-mation Systems Journal, 14(1), 3-22.

Kendra, K.A., & Taplin, A.J. (2004, Winter). Change agent competencies for information technology project managers. Consulting Psychology Journal: Prac-tical Research, 56(1), 20-34.

Lawrence, P., & Lorsch, J. (1967). Organization and environment: Managing differentiation and integration. Boston: Graduate School of Business Administra-tion, Harvard University.

Lee, A. (1999). Rigor and relevance in MIS research: Beyond the approach of positivism alone. MIS Quarterly, 23, 29-33.

Lehman, M. M. (1997). Process improvement - the way forward: Imperial College, London.

Lehman, M. M. (2000). Approach to a theory of software evolution and software process: Imperial College, London.



Lindstrom, L., & Jeffries, R. (2004). Extreme pro-gramming and agile software development me-thodologies. Information Systems Management, 21, 41-52.

Little, T. (2005). Context-adaptive agility: Managing complexity and uncertainty. IEEE Software, 2005(May/June), 28-35.

Lorenz, E. N. (1963). Deterministic nonperiodic flow. Journal of the Atmospheric Sciences, 20(2), 130-141.

Markus, M. L., & Benjamin, R. I. (1996). Change agentry—the next IS frontier. MIS Quarterly, 20(2), 385-407.

McConnell, S. C. (2001). Quantifying soft factors. IEEE Software November/December. Retrieved June 6, 2005, from http://www.stevemcconnell.com/ieeesoftware/eic14.htm

McFeeley, M. (1996). Ideal: A user's guide for software process improvement. (No. CMU/SEI-96-HB-001): Software Engineering Institute.

Mingers, J. (2001). Combining IS research methods: Towards a pluralist methodology. Information Sys-tems Research, 12, 240-259.

Myers, M. D. (2005). Qualitative research in infor-mation systems. MISQ Discovery Retrieved April 14, 2005, from http://www.qual.auckland.ac.nz.

Orlikowski, W. J., & Barley, S. R. (2001). Technolo-gy and institutions: What can research on informa-tion technology and research on organizations learn from each other? MIS Quarterly, 25(2), 145-165.

Orlikowski, W. J., & Baroudi, J. J. (1991). Studying information technology in organizations: Research approaches and assumptions. Information Sys-tems Research, 2, 1-28.

Orlikowski, W. J., & Hofman, J. D. (1997). An improvisational model for change management: The case of groupware technologies. Sloan Man-agement Review(Winter), 11-21.

Osterweil, L. (1986). Software processes are software too: Iteration is the software process. Paper presented at the 3rd International Process Workshop, Brecke-nridge, CO.

Pande, P. S., Neuman, R. P., & Cavanagh, R. R. (2000). The Six Sigma way: How GE, Motorola, and other top companies are honing their performance. New York: McGraw Hill.

Paulk, M. C. (1999, March 8-11). Practices of high maturity organizations. Paper presented at the 1999 SEPG Conference, Atlanta, GA.

Paulk, M. C., Curtis, B., Chrisis, M. B., & Weber, C. V. (1993). Capability Maturity Model for software, ver-sion 1.1. Pittsburgh, PA: Software Engineering Institute.

Paulk, M. C., Weber, C. V., Garcia, S. M., Chrisis, M. B., & Bush, M. (1993). Key practices of the Capability Maturity Model, version 1.1 (No. CMU/SEI-93-TR-025). Pittsburgh, PA: Software Engineering Insti-tute.

Peculis, R. (2005). Who dares, wins: A chaos man-agement approach for software development. Retrieved June 7, 2005, from http://www.matra.com.au/~rpeculis/wdw.htm

Peters, T. J. (1987). Thriving on chaos: Handbook for a management revolution. New York: Harper & Row.

Pourkomeylian, P. (2000). Knowledge creation in improv-ing a software organization. Paper presented at the IRIS 23 Laboratorium for Interaction Technology, University of Trollhättan, Uddevalla.

Rainer, A., & Hall, T. (in press). A quantitative and qualitative analysis of factors affecting software processes. Journal of Systems and Software.

Rogers, E. M. (1993). Diffusion of Innovations (3rd ed.). New York: Free Press.

Rose, J. (2004). Software process improvement: Endnote library. Retrieved June 4, 2005, from http://www.isworld.org/endnote/endnote.htm#libraries

Schein, E. H. (1985). Organizational culture and leader-ship: A dynamic view. San Francisco: Jossey-Bass.

Software Engineering Institute. (2004). Annual report FY 2004: 20/20 impact/vision. Retrieved June 6, 2005, from http://www.sei.cmu.edu/annual-report/files/2004_SEI_AnnualReport.pdf

Software Engineering Institute. (2005). Software process improvement network (SPINs). Re-trieved June 6, 2005, from http://www.sei.cmu.edu/collaborating/spins/spins.html

Statz, J., Oxley, D., & O'Toole, P. (1997). Identify-ing and managing risks for software process im-provement. Crosstalk, 1997(4).

Stelzer, D. & Mellis, W. (1999). Success Factors of Organizational Change in Software Process Im-provement. Software Process Improvement and Practice, 4(4).

Swanson, E. B. (1994). Information systems innova-tion among organizations. Management Science, 40(9), 1069-1091.



Swanson, E.B., & Ramiller, N. (1993). Information systems research thematics: Submissions to a new journal. Information Systems Research, 4, 299-330.

Thietart, R. A., & Forgues, B. (1995). Chaos theory and organization. Organization Science, 6(1), 19-31.

Thomsett, R. (2002). Extreme project management. Agile Project Management Executive Report, 2(2).

Thomson, H. E., & Mayhew, P. J. (1997). Ap-proaches to software process improvement. Soft-ware Process: Improvement and Practice, 3(1), 3-17.

Vessey, I., Ramesh, V., & Glass, R. L. (2002). Re-search in information systems: An empirical study of diversity in the discipline and its journals. Jour-nal of Management Information Systems, 19, 129-174.

Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2), xiii-xxiii.

Wiegers, K. E. (1996a). Creating a software engineering culture. New York: Dorset House.

Wiegers, K.E. (1996b). Software process improve-ment: Ten traps to avoid. Software Development, May 1996.

Wiegers, K. E. (1999). Software process improve-ment in web time. IEEE Software(July/August).

Wilkes, M. V., Wheeler, D. J., & Gill, S. (1951). The preparation of programs for an electronic digital computer: Addison-Wesley.

Wixom, B. H., & Watson, H. J. (2001). An empirical investigation of the factors affecting data ware-housing success. MIS Quarterly, 25, 17-41.

Woodruff, W.D. (2003). Introduction of test process improvements and the impact on the organization. Software Quality Professional, 5(4), 24-32.



Appendix

Table 1

Studies Assessed for Research Methods, Organizational Paradigms, and Factors Influencing SPI Success and Failure

Aaen & Damsgaard, 1998

Abrahamsson & Jokela, 2000

Baddoo, Hall, & Wilson, 2000

Berander & Wohlin, 2003

Ceschi, et al., 2005

Conradi, 1997

Conradi & Fuggetta, 2002

Diaz & Sligo, 1997

Dyba, 2005

Esteves, Pastor, & Casanovas, 2002

Goldenson & Herbsleb, 1995

Halvorsen & Conradi, 2002

Humphrey, Snyder, & Willis, 1991

Iverson, Nielsen, & Norberg, 1998

Jalote, n.d.

Kautz & Nielsen, 2004

Lehman, 1997

Lehman, 2000

Little, 2005

McConnell, 2001

Paulk, 1999

Peculis, 2005

Pourkomeylian, 2000

Rainer & Hall, in press

Statz, Oxley, & O'Toole, 1997

Stelzer & Mellis, 1999

Thomsett, 2002

Wiegers, 1996b

Wiegers, 1999

Wixom & Watson, 2001



Table 2

SPI Studies: Summary of Research Methods Used (Alavi & Carlson, 1992; Benbaset & Zmud, 1999;Lee, 1999; Mingers, 2001; Myers, 2005; Vessey, Ramesh, & Glass, 2002)

Quantitative (n=10) Qualitative (n=25)

Experimental (n=1) Non-Experimental (n=9) Empirical (n=10) Non-Empirical (n=15)

Pourkomeylian, 2000* Berander & Wohlin, 2003* Ceschi, et al., 2005 Dyba, 2005 Esteves, Pastor, & Casanovas, 2002 Lehman, 1997 Paulk, 1999 Pourkomeylian, 2000* Rainer & Hall, in press* Wixom & Watson, 2001

Abrahamsson & Jokela, 2000* Baddoo, Hall, & Wilson, 2000 Berander & Wohlin, 2003* Diaz & Sligo, 1997 Humphrey, Snyder, & Willis, 1991 Kautz & Nielsen, 2004 Pourkomeylian, 2000* Rainer & Hall, in press* Statz, Oxley, & O'Toole, 1997 Stelzer & Mellis, 1999

Aaen & Damsgaard, 1998 Abrahamsson & Jokela, 2000* Conradi, 1997 Conradi & Fuggetta, 2002 Goldenson & Herbsleb, 1995 Halvorsen & Conradi, 2002 Iverson, Nielsen, & Norberg, 1998 Jalote, n.d. Lehman, 2000 Little, 2005 McConnell, 2001 Peculis, 2005 Thomsett, 2002 Wiegers, 1996b Wiegers, 1999

*Mixed methods: study appears in more than one column.



Table 3

SPI Studies: Summary of Article Attributes Used to Determine Organizational Paradigms (Scott, 2003; Thietart & Forgues, 1995)

Organizational Paradigms

Article Attributes Rational Systems Natural Systems Open Systems Complex Adaptive Systems

Organizational Characte-ristics

High goal specificity High formalization

Multiple interests Informal structure

Environmental influence Interdependent activities

Interdependent parts Interdependent on environment Unstable relationships

Keywords

Administrative Bureaucratic CMM Detailed Discipline Formal ISO 9000 Quantitative Scientific Six Sigma

Conflict Consensus Cooperative Organizing

Competition Contingency Ecology Institutional Processes Resource Synchronization Transaction

Adaptive, Agile Chaos, Complexity Dynamic Equilibrium Irreversibility Nonlinearity Nonreplicability Patterns Prediction impossibility Self-Organizing Unstructured

Organizational Theory Authors Cited

Blau & Scott, 1962 Fayol, 1919 Simon, 1945 Taylor, 1911 Weber, 1968

Barnard, 1938 Gouldner, 1954 Mayo, 1945 Whyte, 1950

Hannan & Freeman, 1977 Lawrence & Lorsch, 1967 March & Simon, 1958 Meyer & Rowan, 1977 Miller & Rice, 1967 Pfeffer & Salancik, 1978 Selznick, 1949 Weick, 1969 Williamson, 1975

Fombrun, 1986 Jauch & Kraft, 1986 Levy, 1994 March, 1981 Nystrom, et al., 1976 Quinn & Cameron, 1988 Senge, 1990 Thietart & Forgues, 1995 Weick, 1977; 1979

Software Development Authors Cited

Paulk, 1994 Humphrey, 1995; 1996; 1999

Grady, 1997 Humphrey, 1989 McConnell, 2001



Table 4

SPI Studies: Categorization of Studies by Organizational Paradigm

Rational Systems (n=16) Natural Systems (n=4) Open Systems (n=9) Complex Adaptive Systems (n=5)

Aaen & Damsgaard, 1998 Conradi, 1997 Conradi & Fuggetta, 2002* Diaz & Sligo, 1997 Esteves, Pastor, & Casanovas, 2002 Goldenson & Herbsleb, 1995 Humphrey, Snyder, & Willis, 1991 Iverson, Nielsen, & Norberg, 1998 Kautz & Nielsen, 2004* Paulk, 1999 Pourkomeylian, 2000 Rainer & Hall, in press Statz, Oxley, & O'Toole, 1997 Stelzer & Mellis, 1999 Wiegers, 1996b Wixom & Watson, 2001

Abrahamsson & Jokela, 2000 Baddoo, Hall, & Wilson, 2000 Halvorsen & Conradi, 2002* Kautz & Nielsen, 2004*

Berander & Wohlin, 2003 Ceschi, et al., 2005 Conradi & Fuggetta, 2002* Dyba, 2005 Halvorsen & Conradi, 2002* Jalote, n.d. Kautz & Nielsen, 2004* McConnell, 2001 Wiegers, 1999

Lehman, 1997 Lehman, 2000 Little, 2005 Peculis, 2005 Thomsett, 2002

*Multiple paradigms: study appears in more than one column



Table 5

SPI Studies: Frequency of Factors Influencing Success and Failure by Organizational Paradigm (Factors grouped and normalized)

Rational Systems (n=16 ) Natural Systems (n=4) Open Systems (n=9) Complex Adaptive Systems (n=5)

Management Factors

Management buy-in to SPI (2)

Management commitment to SPI (9)

Management expectations realistic (2)

Management focus on business goals (vs. CMM level) (2)

Management leadership (3)

Management financial commitment Management commitment to SPI (4)

Management concern for measurement

Management expectations realistic for time required

Management includes software people

Management short- and long-term SPI goals

Management emphasis on self-organization

Management emphasis on shared pur-pose, values, principles

Management loves change

Organization Factors

Organization business requirement for change

Organization business reengineering

Organization SPI champion (3)

Organization change resistance over-come, unfreezing (2)

Organization knowledge transfer (2)

Organization history of change success (2)

Organization history of project success

Organization mentoring

Organization process orientation

Organization process ownership

Organization project sponsor

Organization quality culture

Organization capacity for additional work

Organization capacity for change

Organization empowerment

Organization integration between func-tions and groups

Organization innovation (change) rea-diness

Organization process ownership

Organization SEPG group

Organization SPI skills

Organization acceptance of frame-works and measurement

Organization change management

Organization change readiness

Organization communication richness

Organization competition from market

Organization core competence

Organization culture change

Organization culture is familial

Organization goal to move up the val-ue chain

Organization export orientation

Organization growth

Organization industry

Organization interaction openness and

Organization communication and feed-back

Organization culture of few but essen-tial boundaries

Organization fast-paced and innovative

Organization flexibility through empo-werment

Organization partnerships

Organization staffed with intelligent people

Organization support system simplicity



Table 5



Organization risk management

Organization SPI skills

Organization training and mentoring

informality

Organization internal pressure for im-provement

Organization knowledge creation

Organization offshore model

Organization pain motivation

Organization participation and en-gagement (2)

Organization problem-solving focus

Organization process and product fo-cus

Organization SEPG group (2)

Organization services nature

Organization stakeholder involvement

Organization subcontracting emphasis

Process Factors

Process assessment effectiveness

Process alignment with TQM

Process automation

Process data collection/analysis (2)

Process defined by practitioners, not outside experts

Process focus on new projects

Process improvement execution

Process metrics

Process suitability to organization

Process best practices

Process assessment with a simple sco-recard

Process alignment with business goals and strategy

Process baselining

Process documentation

Process feedback between initiatives

Process focus on pressing needs

Process improvement execution

Process dynamics mastery

Process evolution

Process feedback understanding

Process modeling (2)

Process parallelism

Process self-stabilization



Table 5



Process quality assurance

Process quantitatively measured

Process scaling to project size

Process stability before automation

Process stabilization after change

Process standardization

Process suitability to organization

Process tailored to organization (2)

Process top-down approach

Process repository of good practices

Process structure

Process synchronization

Process tools adapted to the organiza-tion

Project Factors

Project controls

Project deliverables defined

Project estimation tools

Project planning and tracking

Project requirements management

Project reviews and inspections

Project risk management

Project scope management

Project schedule management

Project standards

Project formality and structure Project agile methods

Project documentation

Project sequencing vs. parallelism

Project evolutionary methods

Project focus on value-added

Project interdependencies minimized

Project mission criticality minimized

Team Factors

Team change agents

Team change management expertise

Team average age

Team business orientation

Team co-location

Team collaboration



Table 5



Team commitment to outcome

Team communication and collabora-tion (2)

Team empowerment

Team leadership

Team project manager allocation (2)

Team project manager effectiveness (3)

Team resources adequate (3)

Team rewards (2)

Team software engineering expertise (4)

Team skills

Team SPI expertise (7)

Team training (3)

Team collaboration

Team composition more engineers than computer scientists

Team existing knowledge

Team expertise (2)

Team motivation

Team multidisciplinary

Team physical work environment

Team project manager involvement in project work

Team practitioner involvement

Team resources sufficient

Team roles clear

Team domain knowledge

Team experience

Team inquiry, learning, experimentation

Team size appropriate

User Factors

User involvement (3)

User kept informed

User involvement (2)



Table 5



Factor Totals

Management: 5

Organization: 15

Process: 17

Project: 10

Team: 14

User: 2

Total: 63

Management: 1

Organization: 8

Process: 2

Project: 1

Team: 0

User: 0

Total: 12

Management: 5

Organization: 24

Process: 11

Project: 3

Team: 13

User: 1

Total: 57

Management: 3

Organization: 7

Process: 6

Project: 4

Team: 6

User: 0

Total: 26

critical success factors in software process improvement: research

Documents