Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
IT Project Complexity, Complication, and Success
Summary of a Research Study Conducted as part of the requirements for the degree
Doctor of Philosophy
Capella University by
David J. Williamson In cooperation with the
Project Management Institute
Information Systems Community of Practice
October, 2011
CxmCn
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Summary
• Problem: IT project failure
• Causes: Many, but project complexity is
underdiagnosed
• Theory: Most PM methods are based on
the rational systems view; however, a
complex adaptive systems view more
effectively describes project behavior
• Methodology: Model distinguishing
between complexity and complication;
quantitative, correlational study using a
web-hosted survey
• Survey: New survey instrument
developed, tested, and administered to the
U.S. membership of the PMI IS CoP
• Results: ITPCx and ITPCn were positively
correlated; ITPCx had a greater negative
correlation with ITPS
• Implications: Identify and mitigate project
attributes related to IT project complexity to
increase the likelihood of IT project
success
January 10, 2012 3 Copyright © 2012, David J. Williamson
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Methodology
Review Literature
Identify Variables, Factors, Elements
Build Model
Build and Test Survey
Collect Data Analyze Data
Describe Results Draw
Conclusions Suggest
Implications
January 10, 2012 4 Copyright © 2012, David J. Williamson
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Complication vs. Complexity
Attribute Complication Complexity
Cause Size, detail, number of parts Interactions between the parts
Analysis Can be decomposed, analyzed,
and described in terms of
components and parts
Cannot be analyzed and
described completely as
separate components
Behavior Consistent and predictable over
time
Adapts to environmental
influences
Response Responds linearly and
predictably to external events
Responds non-linearly and
unpredictably to external events,
evolves and changes over time,
exhibits emergent behaviors,
changes environment
Managing Can be managed with rational
systems approaches
Cannot be managed directly—
can only be accommodated or
mitigated
(Benbya & McKelvey, 2006; Cilliers, 1998; Hass, 2009)
January 10, 2012 5 Copyright © 2012, David J. Williamson
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Model: IT Project Complexity, Complication, and Success
• Identify project characteristics that
contribute to IT project complexity (ITPCx)
and IT project complication (ITPCn)
• Select a definition for IT project success
(ITPS)
• Investigate relationships among:
• ITPCx – ITPCn
• ITPCx – ITPS
• ITPCn – ITPS
• ITPCx – ITPS vs. ITPCn – ITPS
• Provide evidence that complexity and
complication are related but different sets
of project characteristics, with different
relationships to project success
IT Project
Complexity
(ITPCx)
IT Project
Complication
(ITPCn)
IT Project
Success
(ITPS)
Legend:
Primary Relationship
Secondary Relationship
October, 2011 6 © 2011, David J. Williamson
New model—first to distinguish complexity and complication
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Factors and Elements of IT Project Complexity
January 10, 2012 7 Copyright © 2012, David J. Williamson
ITPCx
Project
Complexity
ITPCx1 Objectives
ITPCx2
Opportunity
ITPCx7
Requirements
ITPCx3
Solution
ITPCx6
Schedule
ITPCx4
Team
ITPCx5
Methodology
ITPCx11
Org Change
ITPCx10
Tech Change
ITPCx8
Environment
ITPCx9
IT Complexity
ITPCx13
IT Integration
ITPCx2a Clear
ITPCx2b Familiar
ITPCx3a Familiar
ITPCx3b Available
ITPCx4a Experience
ITPCx4b Track Record
ITPCx5a Formal
ITPCx5b Consistent
ITPCx6a Reasonable
ITPCx6b Flexible
ITPCx7a Clear
ITPCx7b Stable
ITPCx8a Political
ITPCx8b Strategic
ITPCx8c Stakeholders
ITPCx8d Dependency
ITPCx8e Regulatory
ITPCx8f Legal
ITPCx9a Complexity
ITPCx9b Innovation
ITPCx11a Bus Proc
ITPCx11b Org Scope
ITPCx12
Staffing
Thirteen elements with 1 to 6 factors:
• Project objectives: Clarity
• Opportunity: Clarity and familiarity
• Solution: Familiarity and availability
• Team: Experience and track record
• Methodology: Formality and consistency
• Schedule: Reasonableness and flexibility
• Requirements: Clarity and stability
• Project environment: Political, strategic,
stakeholders, dependencies, regulatory, legal
• Information technology: Complexity and
innovation
• Technology: Degree of change
• Organizational change: Business processes
and scope
• Project staffing: Number of organizations
• Integration: Number of interfaces
(Austin, et al., 2002; Baccarini, 1996; Bardyn & Fitzgerald, 1996;
Brockhoff, 2006; Cilliers, 1998; Cooke-Davies, et al., 2007; Fitzgerald &
Bardyn, 2006; Frame, 1994; Hass, 2009; Jaafari, 2003; Singh & Singh,
2002; Whitty & Maylor, 2007, 2009)
Complexity is related to change and unknowns
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Factors and Elements of IT Project Complication
Nine elements with 1 to 3 factors:
• Project leadership: Experience and competence
• Project duration: Length of original schedule
• Project team size: Number of members
• Project cost: Planned cost and flexibility
• Project scope: Flexibility
• Technology content: Percent of total scope
• Organizational support: Executives and users
• Organizational units: Number of units involved
• Contractors: Number, familiarity, and track
record
(Austin, et al., 2002; Baccarini, 1996; Bardyn & Fitzgerald, 1996;
Brockhoff, 2006; Cilliers, 1998; Cooke-Davies, et al., 2007; Fitzgerald &
Bardyn, 2006; Frame, 1994; Hass, 2009; Jaafari, 2003; Singh & Singh,
2002; Whitty & Maylor, 2007, 2009)
January 10, 2012 8 Copyright © 2012, David J. Williamson
ITPCn
Project
Complication
ITPCn5
Scope
ITPCn2
Duration
ITPCn1a Experience
ITPCn1b Competence
ITPCn4
Cost
ITPCn3
Team Size
ITPCn1
Leadership
ITPCn7
Support
ITPCn9
Contractors
ITPCn6
Technology
ITPCn7a Executives
ITPCn7b Users
ITPCn4a Planned
ITPCn4b Flexibility
ITPCn8
Org Units
ITPCn9a Number
ITPCn9b Familiarity
ITPCn9c Track Record
Complication is related to size and familiarity
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Factors and Elements of IT Project Success
Three elements with 2 to 3 factors:
• Project completion: percent completed and
percent implemented
• Project performance vs. initial baseline:
percent of schedule, budget, and scope
• Project performance vs. final baseline:
percent of schedule, budget, and scope
(Baccarini, 1999; Glass, 2006; Standish Group, 1994, 1999, 2009)
January 10, 2012 9 Copyright © 2012, David J. Williamson
ITPS
Project
Success
ITPS1a %Completed
ITPS1b %Implemented
ITPS1
Completion
ITPS2a %Schedule
ITPS2b %BudgetITPS2
Performance (B1)ITPS2c %Scope
ITPS3a %Schedule
ITPS3b %BudgetITPS3
Performance (Bn)ITPS3c %Scope
Used a definition similar to Standish Group’s CHAOS Studies
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Survey
Instrument
• No existing surveys comparing complexity
and complication
• New Internet survey instrument developed
for the study
• Not previously validated; testing required
Field Test
• Qualitative review in a PMI-SWVA seminar
on agile project management
• Quantitative review using a sample with a
feedback page
• Feedback incorporated into design,
instructions, layout, and questions
Pilot Test
• IRB approval prior to pilot testing
• Responses n = 42 exceeded the minimum
of n = 35 to 40 (Johanson & Brooks, 2010)
• Responses were well-distributed; non-
response rate was minimal
January 10, 2012 10 Copyright © 2012, David J. Williamson
The survey was new, but tested and shown to be valid
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Data Collection: Population and Sample
Population
• U.S. IT project managers
• PMI IS CoP: Over 15,000 members
worldwide, 40% to 50% in the U.S.
• Other studies have used this population
• Wallace, Keil, and Rai (2004)
• Xia and Lee (2005)
• Mishra, et al. (2009)
• Response rates ranged from 6% to 15%
Sample
• 6,000 U.S. members of the PMI IS CoP
• 100% probability sample
• Sample power analysis indicated minimum
sample size n = 115 for bivariate normal
correlation ( = .05, 1- = .95, r = .30)
• Actual response rate was 3.9% (n = 235)
January 10, 2012 11 Copyright © 2012, David J. Williamson
The population was reasonably representative The sample was statistically reliable
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Data Collection: Survey Responses
Survey
• Hosted by SurveyMonkey
• Invitations sent by PMI IS CoP
Responses
• Total qualified responses n = 235
exceeded minimum of n = 115
indicated by power analysis
• Response rate indicated data
collection period could have been
shortened
• Follow-up reminders could have
increased total response
January 10, 2012 12 Copyright © 2012, David J. Williamson
Responses were more than adequate, could have been higher
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Data Analysis: Demographics
• Responses: 235 qualified responses; organizations ranging from fewer than 10 to
more than 10,000 employees, annual budgets less than $10 million U.S. to more
than $5 billion U.S.
• Job Titles: Project Manager (46%), Program Manager (17%)
• Project Roles: Project manager (55%), program manager (25%), project team
member (7%).
• PMP Certification: 76%
• Industries: Finance, insurance, and banking (22%), information technology
(20%), healthcare (11%), other (10%) including pharmaceuticals, media,
government
• Project Types: Information technology (39%), software development (33%),
application package implementation (11%)
January 10, 2012 13 Copyright © 2012, David J. Williamson
The sample appeared to be reasonably representative
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Data Analysis: Distributions
• ITPCx and ITPCn
were reasonably
normal
• ITPS was non-normal,
required a normal
transform; NITPS
distribution was
reasonably normal
January 10, 2012 14 Copyright © 2012, David J. Williamson
5.04.03.02.01.0
ITPCx
30
25
20
15
10
5
0
Fre
qu
en
cy
5.04.03.02.01.0
ITPCn
30
20
10
0
Fre
qu
en
cy
5.04.03.02.01.0
ITPS
40
30
20
10
0
Fre
qu
en
cy
2.01.00.0-1.0-2.0-3.0
NORMAL of ITPS using RANKIT
30
25
20
15
10
5
0
Fre
qu
en
cy
The data could be used for correlation analysis
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
• Statistically significant rank order
nonparametric correlations
between all pairs, with the exception
of the ITPCxITPCn and NITPS pair
• Statistically significant Pearson’s
product-moment parametric
correlations between all construct
pairs and construct-transform pairs
Data Analysis: Correlations
Paired constructs Pearson’s Kendall’s taub Spearman’s rho
r r2 p p rs p
ITPCx-ITPCn .530 .281 .000 .338 .000 .483 .000
ITPCx-ITPS -.356 .127 .000 -.256 .000 -.363 .000
ITPCn-ITPS -.247 .061 .000 -.123 .006 -.181 .005
ITPCx-NITPS -.350 .123 .000 -.256 .000 -.363 .000
ITPCn-NITPS -.228 .052 .000 -.123 .000 -.181 .005
ITPCxITPCn-NITPS -.185 .034 .004 -.051 .248 -.078 .236
January 10, 2012 15 Copyright © 2012, David J. Williamson
Relationships existed, and they were statistically significant
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Results: ITPCx and ITPCn
• Positive Pearson’s correlation existed
between ITPCx and ITPCn, r = .530, r2 =
.281, p < .001
• Positive nonparametric rank order
correlations also existed:
• Kendall’s taub = .338, p < .001
• Spearman’s rho rs = .483, p < .001.
• Finding 1: ITPCx was positively
correlated with ITPCn; p = .000 (less
than the significance level .05 for
bivariate normal correlation indicated by
power analysis)
January 10, 2012 16 Copyright © 2012, David J. Williamson
Complexity and complication are related, but different
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Results: ITPCx and ITPS
• Negative Pearson’s correlation existed
between ITPCx and ITPS, r = -.350, r2 =
.123, p < .001
• Negative nonparametric rank order
correlations also existed:
• Kendall’s taub = -.256, p < .001
• Spearman’s rho rs = -.363, p < .001
• Finding 2: ITPCx was negatively
correlated with ITPS; p = .000 (less than
the significance level .05 for bivariate
normal correlation indicated by power
analysis)
January 10, 2012 17 Copyright © 2012, David J. Williamson
Complexity has a negative correlation with success
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Results: ITPCn and ITPS
• Negative Pearson’s correlation existed
between ITPCn and ITPS, r = -.228, r2 =
.052, p < .001
• Negative nonparametric rank order
correlations also existed:
• Kendall’s taub = -.123, p < .01
• Spearman’s rho rs = -.181, p < .01
• Finding 3: ITPCn was negatively
correlated with ITPS; p = .000 (less than
the significance level .05 for bivariate
normal correlation indicated by power
analysis)
January 10, 2012 18 Copyright © 2012, David J. Williamson
Complication has a negative correlation with success
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Results: ITPCx vs. ITPCn
• Pearson’s correlation for ITPCx and
ITPS, r = -.350, r2 = .123, p < .001 was
greater negative than for ITPCn and
ITPS, r = -.228, r2 = .052, p < .001.
• Nonparametric rank order correlations
for ITPCx and ITPS
• Kendall’s taub = -.256, p < .001
• Spearman’s rho rs = -.363, p < .001
were also greater negative than ITPCn
and ITPS
• Kendall’s taub = -.123, p < .01
• Spearman’s rho rs = -.181, p < .01
• Finding 4: ITPCx had a greater
negative correlation with ITPS than did
ITPCn
January 10, 2012 19 Copyright © 2012, David J. Williamson
Complexity affects success more than complication
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Results: Summary
Results
• ITPCx and ITPCn were positively
correlated (r = .530, r2 = .281,
p < .001) to a greater degree than either
variable and ITPS
• ITPCx and ITPS were negatively
correlated (r = -.350, r2 = .123,
p < .001) to a greater degree than were
ITPCn and ITPS (r = -.228,
r2 = .052, p < .001)
Revised model
January 10, 2012 20 Copyright © 2012, David J. Williamson
Complexity affects success more than complication,
but complexity and complication are even more strongly related
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Conclusions and Implications
Conclusions
• Complexity and complication are
related but distinct
• IT projects can have different levels
of complexity and complication
• Stronger relationship between
complexity and complication means
other factors are involved, or the
model needs to be refined
Implications
• There is a difference between
complexity and complication
• Focus on identifying, mitigating, and
accommodating IT project complexity
in order to increase the likelihood of IT
project success
January 10, 2012 Copyright © 2012, David J. Williamson 21
There is a difference between complexity and complication
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Application: Effect on project performance
• The impact of complication on
project difficulty and performance
is linear
• The impact of complexity is non-
linear and unpredictable
January 10, 2012 Copyright © 2012, David J. Williamson 22
Complication
Co
mp
lex
ity
n
x
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Application: Selection of PM methods
• The choice of PM methods
should be driven by the project
characteristics
• Using an inappropriate method
increases the probability of failure
• Agile methods are more effective
for handling complexity than
traditional methods
• Agile methods are also effective
for small projects with low
complexity, if flexibility is
important
January 10, 2012 Copyright © 2012, David J. Williamson 23
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Application: Risk management
• Complication increases risk
• Complexity multiplies risk
• Mitigate complexity and risk with
frequent delivery
January 10, 2012 Copyright © 2012, David J. Williamson 24
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Study published January 6, 2012
January 10, 2012 25 Copyright © 2012, David J. Williamson
http://gradworks.umi.com/34/81/3481367.html
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
Paper submitted to PMI Research and Education Conference 2012
January 10, 2012 26 Copyright © 2012, David J. Williamson
Research Summary: IT Project Complexity, Complication, and Success SOUTHWEST VIRGINIA CHAPTER
References
Austin, S., Newton, A., Steele, J., & Waskett, P. (2002). Modelling and
managing project complexity. International Journal of Project
Management, 20(3), 191-198. doi: 10.1016/S0263-7863(01)00068-0
Baccarini, D. (1996). The concept of project complexity - a review.
International Journal of Project Management, 14(4), 201-204. doi:
10.1016/0263-7863(95)00093-3
Baccarini, D. (1999). The logical framework method for defining project
success. Project Management Journal, 30(4), 25-32.
Bardyn, J., & Fitzgerald, D. (n.d.). The uses of chaos theory in project
management. Retrieved June 7, 2005, from http://www.cyber-
office.org/writings/usesof.htm
Benbya, H., & McKelvey, B. (2006). Toward a complexity theory of
information systems development. Information Technology & People,
19(1), 12-34. doi: 10.1108/095938406106499
Brockhoff, K. (2006). On the novelty dimension in project management.
Project Management Journal, 37(3), 26-36.
Burkatzky, F. H. H. (2007). Development of measurement scales for
project complexity and system integration performance (Doctoral
dissertation). Walden University.
Cavaleri, S., & Reed, F. (2008). Leading dynamically complex projects.
International Journal of Managing Projects in Business, 1(1), 71-87.
doi: 10.1108/17538370810846423
Cilliers, P. (1998). Complexity and postmodernism: Understanding
complex systems. London, England: Routledge.
Cooke-Davies, T., Cicmil, S., Crawford, L., & Richardson, K. (2007). We're
not in Kansas anymore, Toto: Mapping the strange landscape of
complexity theory, and its relationship to project management. Project
Management Journal, 38(2), 50-61.
Dillman, D. A. (2000). Mail and Internet surveys: The tailored design
method (2nd ed.). New York, NY: John Wiley & Sons.
Fitzgerald, D., & Bardyn, J. (2006). Chaos theory and project
management: A perspective on managing the complex project.
Retrieved August 24, 2006, from
http://www.nimblepm.com/writings/chaostheory.html
Frame, J. D. (1994). The new project management: Tools for an age of rapid
change, corporate reengineering, and other business realities (1st ed.).
San Francisco, CA: Jossey-Bass.
Glass, R. L. (2006). The Standish report: Does it really describe a software
crisis? Communications of the ACM, 49(8), 15-17. doi:
10.1145/1145287.1145301
Hass, K. B. (2009). Managing complex projects: A new model. Vienna, VA:
Management Concepts.
Jaafari, A. (2003). Project management in the age of complexity and change.
Project Management Journal, 34(4), 47-57.
Johanson, G. A., & Brooks, G. P. (2010). Initial scale development: Sample
size for pilot studies. Educational and Psychological Measurement, 70(3),
394-400. doi: 10.1177/0013164409355692
Mishra, A., Sinha, K., & Thirumalai, S. (2009). Comparative evaluation of
efficiency across distributed project organizations: A stochastic frontier
analysis. University of Minnesota. Minneapolis, MN.
Singh, H., & Singh, A. (2002). Principles of complexity and Chaos theory in
project execution: A new approach to management. Cost Engineering,
44(12), 23-33.
Standish Group. (1994). The chaos report. West Yarmouth, MA: Author.
Standish Group. (1999). Chaos: A recipe for success. West Yarmouth, MA:
Author.
Standish Group. (2009). Chaos summary 2009. Boston, MA: Author.
Wallace, L., Keil, M., & Rai, A. (2004). Understanding software project risk: A
cluster analysis. Information & Management, 42, 115-125.
Whitty, S. J., & Maylor, H. (2007). And then came complex project
management. Paper presented at the 21st IPMA World Congress on
Project Management.
Whitty, S. J., & Maylor, H. (2009). And then came complex project
management (revised). International Journal of Project Management,
27(3), 304-310. doi: 10.1016/j.ijproman.2008.03.004
Xia, W., & Lee, G. (2005). Complexity of information systems development
projects: Conceptualization and measurement development. Journal of
Management Information Systems, 22(1), 45-83.
January 10, 2012 27 Copyright © 2012, David J. Williamson