frontiers 2015, by 3 pillar, ces, rockbridge
TRANSCRIPT
What Drives Successful Product Development and Innovation in the Software Development Process? The Product Development Success Index (PDSI)
PRESENTED BY: CHARLES COLBY, ROCKBRIDGE
SUNIL MITHAS, ROBERT H. SMITH SCHOOL
TONY ORLANDO, 3 PILLARRIICCARDO DE MARCHI TREVISAN, 3
PILLAR ERIN NORMAN, ROCKBRIDGE
2015 Frontiers in Services Conference
Rockbridge Associates, Inc.
2
Introduction to 3Pillar Global
• 3Pillar Global is a product lifecycle management and development company that helps clients implement products that drive revenue.
• Services include strategy and consulting, customer experience design, platform development, software engineering based on agile practices, and devops.
• Our clients include mid-sized to large companies in the media/entertainment, edtech, financial services, information services, and health and wellness sectors.
3
Background and Environment for this Study
• Service sector companies increasingly seek to develop new software applications to…
• Drive their core business• Add customer value• Create new revenue streams• Create entirely new products that monetize information
captured in their processes
• Efforts are usually driven by a functional area within the company headed by a Chief Innovation Officer or Chief Technology Officer
• Companies have varying degrees of success in this product development process
4
Objectives of this Study
• Determine what drives success in new software product development – the focus is on companies where software development is not their core mission (e.g., a bank, TV network, healthcare provider)
• Create an instrument that can be used to…• Evaluate success• Benchmark success against other companies• Diagnose issues and identify priorities for improvement
This was a collaborative effort between 3Pillar Global and the Center for Excellence in Service at the Robert H. Smith School of Business. Rockbridge Associates was a research partner that spearheaded much of the effort.
5
Methodology
• Exploratory• In-depth meetings with 3Pillar management/experts • 6 in-depth interviews conducted by phone with heads of
corporate functions • Survey of relevant literature• Creation of a Product Development Success framework
• Quantitative• 20-minute online survey of 204 professionals responsible for
new software product development (145 from online panel, 59 from 3Pillar lists)
• Respondents worked in range of service businesses• Had decision input for development function• Mid-sized to large companies (Minimum $1M)
6
Product Development & Innovation Success Framework
Business Outcomes
Macro Factors• New entrants/products• Globalization• Regulation/legal constraints• Speed of technological
development• Economy
Long Range Industry Shifts• Move towards product standardization• Big Data opportunities• Search for new revenue streams• Increasing investment uncertainty
Product Development Success Index
Perceived Development & Innovation Success
Organizational Characteristics• Revenue• Size• Industry• Geographic reach• Strategic vision
New Software Product Development Function Characteristics• R&D budget• Number of employees• Third party services used
Culture
Feedback
Communication
Staffing
Collaboration
Time/Budget Focus
The Product Development Success Index (PDSI)
8
Survey Topics
Topic MeasuresProduct Development Success
53 items, scaled on a 1 to 7 scale covering culture, process and resources
Product Development Process
Degree of structure, development time, planning/review cycle, testing, capture of high-value data, product customization, revenue streams
Macro Factors Competitive environment, industry growth, globalization, geographic coverage, regulation
Product Development Outcomes
Confidence in process, success in achieving dev goals, innovation leader vs. follower, impact on brand, impact on efficiency, barriers
Business Traits Revenue, employment, industry, revenue growth, customer churn
Function Traits Employee size, budget, strategy focus, reliance on outsourcing, decision-making process
9
Index Development
· Goal was to create an instrument that was concise, provided diagnostic detail, and met thresholds for reliability and validity
· Input consisted of the battery of 53 success attributes (49 after dropping 4 items with high item non-response)
· Exploratory factor analysis used to identify underlying dimensions of success, account for redundancy, and eliminate items with low communality
· Resulting index included:· 25 items· 6 dimensions (3 to 7 items per dimension)· Explained 67% of variance· Cronbach Alphas ranged from .74 to .89· Aggregated index had a .69 correlation with perceived overall success of
the product development function
10
7359 87
Highly Successful
(19%)
Somewhat Successful
(34%)
Marginally Successful
(28%)
Rarely Successful
(19%)
The Product Development Success Index (PDSI) is calibrated to a scale of 0 to 100• The PDSI correlates with success in meeting (1) software product
development goals and (2) hard business outcomes, such as revenue, internal efficiency and customer retention.
• Companies are divided into four tiers for purposes of studying the correlates and consequences of success.
11
The overall PDSI score is comprised of six dimensions
· Culture: attract and retain top talent, have creative leadership, encourage interaction, provide career growth, and reward and recognize innovation
· Feedback: degree of input on products collected from customers and end-users
· Communication: interaction between business and technical team members as well as within the development groups
· Staffing: appropriate mix of staff from in-house, outsourced, domestic and foreign workers
· Collaboration: extent to which team members assist each other and can cut across hierarchy to interact
· Time/Budget Focus: succeeding on traditional project metrics
12
The 6 sub-indices vary in importance
Sub-Index Importance to Product Development Success Index
· Importance is based on correlation with overall perceived success (normalized pairwise estimates)
· Dimension scores are weighted accordingly
Percentages do not add up to 100% due to rounding. See notes for more exact figures.
Across the sample, the biggest pain-point is improving culture
13
Performance on Sub-Indices vs. Importance
Top Challenge: Recognizing and
Rewarding Innovation
Correlates and Consequences of Product Development Success
15
What do the successful PD functions look like?
Most Successful Least Successful
Large Divisions (1,000+ Staff) Small (<100)R&D Budget $161M R&D Budget $39M
Larger technology budgets Smaller budgetsInformation Services Education and Hospitality
Revenue > Cost Cutting Cost Cutting > RevenueSell high margin products Not a focus
Select high quality vendors Not a focusBusiness execs & IT
collaboratively drive strategy Business execs drive strategy
Business execs & IT collaboratively drive technical IT drives technical decisions
Seek to lead their industry Tend to follow or track with peersConfident in Innovation Strategy Not Confident
16
Measures of Agile Development
Q1: For each one, please indicate how well the statement describes your organization.Q2a: How frequently does your team review and re-plan the next highest priorities for ongoing software development efforts?Q2c: How frequently during the development process do end users test working software, including new features?Q3a: How often do you discuss and/or implement improvements to your software development process?Q2: How long does it take in your organization to go from an idea to software that is working in a production environment?Q2b: What percentage of your new software, including new features, is covered by automated tests that run every time a feature is added or modified?
Component Frequency/RatingTime from idea to working software Three weeks or lessNew software covered by automated tests 76%+Frequency of review and re-planning priorities
Every three weeks or more often
Frequency of end user testing Every three weeks or more oftenFrequency of process improvement Every three weeks or more oftenTeam Members Exchange Information and Learn From Each Other
Describes company (6 or 7 on a 7-point scale)
Business and Technical Teams Collaborate of Software Development Projects
Describes company (6 or 7 on a 7-point scale)
17
Agile practices correlate strongly with PD success
Number of Agile Practices Used
Total(n=204)
RarelySuccessful
(39)B
MarginallySuccessful
(57)B
Somewhat Successful
(70)C
HighlySuccessful
(38)D
0.02 0.010.080.07 0.110.1814%
8% 7%
17%
24%18%
10%23%
16%
21%26%
13%
25%
34%
26%20%
28%
32%
14%
3%14%
41%
14% 6%
0 Agile Practices
1 Agile Practice
2 Agile Practices
3 Agile Practices
4 Agile Practices
5 Agile Practices
6 Agile Practices
7 Agile Practices
Q1: For each one, please indicate how well the statement describes your organization.Q2a: How frequently does your team review and re-plan the next highest priorities for ongoing software development efforts?Q2c: How frequently during the development process do end users test working software, including new features?Q3a: How often do you discuss and/or implement improvements to your software development process?Q2: How long does it take in your organization to go from an idea to software that is working in a production environment?Q2b: What percentage of your new software, including new features, is covered by automated tests that run every time a feature is added or modified? *Caution, small base size
AB
A
D
D
CDBC
18
Companies need to cross thresholds on their way to success – consensus, financial, knowledge
Q34: Why does your organization not move faster in software product development and technology? (Select all that apply)*Caution, small base size
Constraints on Speed of Software Product Development
Total(n=204)
Rarely Successful
(n=39*)A
Marginally Successful
(n=57)B
Somewhat Successful
(n=70)C
Highly Successful
(n=38*)D
Financial constraints 47% 51% 58%D 41% 34%
Disagreement among decision makers
29% 49%BC 23% 23% 29%
Lack of time 29% 31% 30% 30% 24%
Ineffectiveness of consensus decision making
25% 51%BCD 18% 20% 18%
Fear of risk 23% 41%CD 25% 19% 11%
Lack of vision 19% 41%BCD 12% 16% 11%
Lack of knowledge 19% 38%BCD 14% 19%D 5%
Legal constraints 19% 13% 21% 20% 21%
None of these 15% 5% 14% 16% 24%A
19
More successful companies operate industries with more dynamic competitive activity
Q9: How frequently do the following actions occur in your industry? Seven-point scale from “Never” to “Constantly”*Caution, small base size
Software products with brand new capabili-ties are released
There are new entrants to the industry
Firms leave the industry
33%
33%
18%
37%
23%
7%
56%
27%
21%
68%
55%
32%
Series5Highly Successful(n=36-38*) DSomewhat Successful(n=69) CMarginally Success-ful(n=56-57)BRarely Successful(n=37-38*)A
Industry Changes(% rated 6-7 on a 7-point frequency scale)
ABAB
BC
BB
Perceived Development & Innovation Success
21
PDSI correlates with success on a variety of goals connected to new software product development
Q16: How successful is your organization in achieving the following goals? *Caution, small base size
Index Score versus Software Product Development Goals(% rated 6-7 on a 7-point success scale)
Total(n=196-204)
Rarely Successful(n=38-39*)
A
Marginally Successful(n=52-57)
B
Somewhat Successful(n=69-70)
C
Highly Successful
(n=37*)D
Overall success of new product innovation 37% 8% 18% 43% AB 87% ABC
Capabilities meet customer needs 42% 10% 21% 59% AB 74% AB
Stay competitive with dev 42% 5% 24% A 56% AB 84% ABC
Products competitive in market 41% 10% 23% 50%AB 82%ABC
Keep software products current 37% 3% 25% A 46% AB 76% ABC
Products create new revenue 31% 5% 19% A 37% AB 68% ABC
Quickly roll out new products 31% 8% 14% 33% AB 76% ABC
Based on radically new tech 30% 5% 16% 29% A 79% ABC
22
Product development success also tracks with business outcomes like revenue growth & customer retention
Number of Positive Business Outcomes
Total(n=204)
Rarely Successful(n=39*)
A
Marginally Successful
(n=57)B
Somewhat Successful
(n=70)C
Highly Innovative(n=38*)
D
15%5% 14% 10%
34%20%
8%
19% 29%
16%23%
23%
18%27%
21%19%
31%18%
17% 13%16%
21% 21%11% 11%
8% 13% 11% 6% 5%
0 Positive Outcomes1 Positive Outcome2 Positive Outcomes3 Positive Outcomes4 Positive Outcomes5 Positive Outcomes
Q18: What impact do new software products developed by your organization have on the reputation of your brand?Q19: How, if at all, have software products developed by your organization in the past three years affected the internal efficiency of your organization? Q20: How would you describe the change in your company’s total revenue over the last three years?Q21: How would you describe the change in your company’s total number of customers over the last three years?Q22: Thinking of the last 12 months, which best describes the change (if any) in your company’s customer retention rate?*Caution, small base size
ABCA
Key Takeaways
24
The overall PDSI score is comprised of six dimensions
· Key opportunity is to build a culture of talented people where innovation is rewarded
· Most successful companies tend to be larger, better funded, focused on quality, and strive to be leading edge
· Success is related to a dynamic competitive landscape· Agile development practices truly matter · Success requires overcoming hurdles: consensus -> financial ->
know-how· It is possible to score and diagnose success
For more information, visit:Visit: http://productdevelopmentsuccess.com/
25
Author Contact Information
· Charles Colby, Principal, Chief Methodologist and Founder, Rockbridge Associates, Inc.: [email protected]
· Sunil Mithas, Professor, Robert H. Smith School of Business & Research Director, Center for Excellence in Service: [email protected]
· Tony Orlando, Executive Vice President, 3 Pillar Global: [email protected]
· Riccardo de Marchi Trevisan, Business Development Executive: [email protected]
· Erin Norman, Director, Rockbridge Associates, Inc.: [email protected]
Appendix: Detailed Index Scores