how to avoid continuously delivering faulty software
DESCRIPTION
As organizations continue to compress development and delivery lifecycles, the risk of regressions, integration errors, and other defects rises. But how can development teams integrate defect prevention strategies into their release cycles to ensure that they're not continuously delivering faulty software? In this session, learn the key development testing processes to add to your Continuous Delivery system to reduce the risk of automating the release of software defects.TRANSCRIPT
#
Mark LambertVP Product Management and Support
How to Avoid Continuously Delivering Faulty Software
#
Software Development ChallengesSoftware Development Challenges
APIs drive interconnectivity across the expanded internet
Compliance with regulatory, industry and internal standards
SDLC Speed will be the difference between a first mover vs follower
#
Software Drives InnovationSoftware Drives Innovation
The Cost of Quality associated with software has shifted dramatically
Software has shifted from process enabler to business differentiator
Quality drives brand loyalty
#
The Cost of Software QualityThe Cost of Software Quality
• After a rash of software failures and security breaches left Sony’s gaming services down for weeks, analysts called for the ousting of the Sony CEO.
• Cumulative Loss = $18B
Mar
ket
Val
ue
Event 15d 30d
-22% -33% -30%-19% -11% -12%
#
• Constant Trade-offs that have business impact
SDLC - The Era of Acceleration SDLC - The Era of Acceleration
Time
Quality
Scope
#
• Continuous testing accelerates the SDLC bymanaging quality expectations and actionable tasks
From Automated to Continuous
Expectations
Policy Management
Development
Defect Prevention
Development
Development Testing
Static Analysis
Unit/Component
Peer Review
Automated Tests
Integration Testing
API/Service Tests
Smoke Test
Security Tests
Automated Tests
System Testing
Functional Tests
Scenario Tests
Performance Tests
ContiniousBuild
Remediation Tasks
Go
Release
No Go
???
Service Virtualization – Test Environment Access
#
Static Code Analysis
Pattern-Based Static Analysis
Prevention technique
Analyzes code structure (parse
tree) to apply best practices
Flow-Based Static Analysis
Detection technique
Analyzes code flow to determine
“dangerous paths”
Metric Threshold Analysis
Advisory technique
Finds complex/hard-to-test code prone to
errors
#
• Well understood often under valued• Define the goal of the analysis and the Policy for
compliance• Focus on reduction of business risk not pursuit of
perfection• Start small to promote adoption and monitor for areas of
improvement
Static Code Analysis
#
Peer Code Reviews
• Highly valuable in finding REAL bugs– Algorithms/Design
• Use carefully– Only apply after Static
Code Analysis– Only apply where
there is Business RISK
Image: http://www.jasonawesome.com/2010/06/01/executing-a-php-code-review/
#
• Unit Testing– Developer focuses on the code– Typically not true Unit Test– Code needs to be built to be testable
• Where is the ROI?– Did we design it properly
• How much is enough?– Code Coverage + Peer Review
Unit vs. Functional Testing
#
• Functional Testing– QA focused on the user-story/function
• Where is the ROI?– Does it function correctly– Did we break functionality
• How much is enough?– User-story coverage
• Assoc. code coverage provides additional insight
Unit vs. Functional Testing
#
• Ad-hock/Unstructured Testing of functional areas
• Important part of QA/feedback process
• Requires traceability to user-stories and code
• Should be ‘reinforced’ with automated tests
Explorative Testing
#
• Limitations– Often at the end of the cycle
• Wait until the whole system is ready– Requires specialized skills and specialized tools
• Often not “real tests”– Too late for cost effective remediation
• “Shift Left” Performance and Security– Reuse automated functional tests and tooling– Eliminated the system constraints … Service Virtualization …
Performance/Security Testing
#
• Complexity is a Barrier to Innovation– Accessible– Stable – Controllable
• Constrained Testing
Service Virtualization
3rd Party System
Evolving Component
Mainframe
Scheduled Access
#
• Emulates dependencies for the Test Environment– Reduces the complexity for early stage testing– Increases predictability
• Enables “Test Anytime, Anywhere, Anyway”– Automated Provisioning for different use-cases– Automated Test Data Management/Simulation
• Does not eliminate the need for System/Integration Testing
Service Virtualization
#
Continuous Test Characteristics
• Logically componentized• Correlated with business
requirements • Incremental, Repeatable• Maintainable,
Deterministic• Process is prescriptive
based on results
Continuous Testing
Policy
Traceability
Analysis Risk Assessment
Environment Access
Optimization
#
Development Testing Platform
• Centralize and Automated “Quality Hub”
• Provide Controls and visibility onto variable and ad-hoc usage of quality tools (incl. open source)
• Enables centralized policy to drive consistent results of the SDLC practices
DTP
Source Control
Defects
Require-ments
Code Review
Static Analysis
MetricsFlow Analysis
Unit Testing
Coverage
Functional Testing
Load Testing
#
Workflow drives improvement
• Aggregation of objective SDLC data transformed into actionable information
• Identify, and prevent, potential defects to reduce project risks
• Developer workflow driven from the Developer
Code Analysis
Data Aggregation
Post Analysis Analysis
(PIE)
Reporting and
Prioritization
Download to IDE
Developer Remediation
Source Check-in
DTP
#
• Real-time feedback on compliance and certification with industry, regulatory or standards initiatives during active development.
Visibility for Compliance
#
• Bridge the gap between technical findings and business impact– Aggregation of deep SDLC data into informative dashboards that.
Provide Clarity on Risk
#
Demonstration
#
1. Define Business Expectations in a Policy
2. Automate Key Software Quality Practices1. Code Analysis
2. Peer Review
3. Automated Testing with Traceability
3. Apply Continuously and with a Workflow for remediation
4. Translate to Business Impact and Monitor for improvements
How to Avoid Continuously Delivering Faulty Software