Download - Managed Services Using SLAs and KPIs
CONNECT WITH US:
Testing as a Managed Service Using SLA’s and KPI’s
Sarat Addanki
Head of Practice
CONNECT WITH US:Public | Copyright © 2014 Prolifics
Agenda
2
01
03
04
05
06 02
Corporate overview
SLAs
KPIs
Sample SLAreporting
Demo
Q&A
CONNECT WITH US: 3
5-Years CompoundAnnual Growth Rate
19%
EmployeesWorldwide
1,500
Global PresenceUnited States, United Kingdom, Germany, India
20+Technology
Accelerators
550+Technical
Certifications
Over 10 Technology and Solutions Awards since 2009
including Business Agility, Customer Integration and Digital
Experience
Over 150 global customersare currently Fortune
1000 companies
Best-in-class architects andspecialty experts:
BPM, IIB, DataPower, WSRR, and other IBM products and
tools.
industry-high 96% success rate of deploying robust solutions.
96%
Prolifics at a Glance
Years inBusiness
35+
Offices14
Awards
IBM Technology Expertise
Fortune 1000
Rate of Repeat Engagements
91%
Public | Copyright © 2014 Prolifics
CONNECT WITH US:
Our Services
Public | Copyright © 2014 Prolifics 4
• Application Development, Maintenance and Support
• Outsourced Product Development
• Application Modernization
• Portal Content & Collaboration
• Test Advisory• Functional Testing
• Packaged Apps
• Custom Apps• Cloud
• Test Automation• Middleware and
SOA• Performance Engg• Mobile Testing
BPM & ConnectivityUser ExperienceSecurity
Application Performance ManagementEnterprise Content ManagementBusiness/Data Analytic
CONNECT WITH US:
How Prolifics Can Help You
5
Adapt faster than the competition with continuous process improvement
Transform your applications to increase business agility
Create best-in-class customer and employee experiences. Become more social and collaborative
Gain actionable insight with an intelligent approach to security
BUSINESS PROCESSMANAGEMENT
SOA INTEGRATION USER EXPERIENCE SECURITY
Drive business growth with proactive IT management
Maximize IT investments with end-to-end testing
Optimize business performance with smarter, faster decisions
Increase efficiency within your organization and achieve “straight through processing”
APPLICATIONINFRASTRUCTURE
TEST ADVISORY, APPLICATION TESTING & TEST
AUTOMATION
BUSINESS ANALYTICSOPERATIONAL DECISION
MANAGEMENT (ODM)
Simplify employee and customer access to the documents they need when they need them
Get the most out of your IT to create competitive advantage.
Leverage global expertise to support your IT environment
Optimize your software spend to achieve business value
ENTERPRISE CONTENT MANAGEMENT
ADVISORY SERVICES MANAGED SERVICES SOFTWARE
Public | Copyright © 2014 Prolifics
CONNECT WITH US: 6
Prolifics Customers
Healthcare
6
Financial Services
Utilities
Retail & Distribution Media & Entertainment
Government Technology
Telecommunications Insurance
HealthCare
Public | Copyright © 2014 Prolifics
CONNECT WITH US: 7Companies are moving from Outsourcing Model to Managed Services Model as it allows them to govern the vendor
services with the help of mutually decided upon Service Level Agreements (SLA’s).
Evaluate value
Measure quality
Continues improvement
Determine cost of testing
Create accountability
Measure productivity
Process Compliance
Within Budget
% of Automation
DREIncident Mgmt.
Business Process
Knowledge
Test Design Efficiency
Manage vendor
SLA’s are vital signs. You can tell where a testing organization is headed and also measure it’s maturity level
Why Managed Service and SLAs?
CONNECT WITH US:
Key Performance Indicators (KPI’s)
8
Test case effectiveness
Test progress plan vs actual
Cost of testing as a % development
Skill set matrix
Quality of development
No. of regression test cases updated per release
Quantifiable regression execution
CONNECT WITH US:
SLAs and Measurement Intervals
Defect Removal EffectivenessProcess Compliance Business Process KnowledgeGeneral Incident ManagementAutomated Regression library - ExecutionRegression Library - Updates
123456
789
1011121314151617
Offshore Technical SkillsetEffort per DefectSavings thru DefectsThrough by Project typesResource UtilizationSubmission of SLA ReportReview of RCATest Design EfficiencyDelivery within budgetSizing and Estimation TurnaroundMetrics Accuracy
MonthlyMonthlyMonthlyMonthlyMonthlyMonthlyMonthlyMonthly
QuarterlyQuarterlyQuarterly
MonthlyQuarterlyQuarterlyQuarterlyMonthlyMonthly
S. No
SLA
Non SLA Metrics
Measurement Interval
9
CONNECT WITH US:
DRE Process Compliance Business Process Knowledge
General Incident Mgmt. Automated Regression Library - Execution
Regression Library - Updates
100%
98%
100%
91%
100%
100%
Snapshot of SLA MetricsJune, 2014
10
SLA Target, 100%
SLA Target, 100%
SLA Target, 98%
SLA Target, 90%
SLA Target, 98%
SLA Target, 95%
CONNECT WITH US:
Description Formula
Frequency
Target
Measure of the quality delivered in a given engagement by measuring the number of defects that seeped into production
Performance = Number of non rejected defects found prior to production / (total number of non rejected defects registered during development life cycle from requirements phase up to and including warranty period) * 100
Critical and High - 95%
Medium – 90 %
Low – 85 %
Monthly
Defect Removal
Effectiveness
11
CONNECT WITH US:
Process ComplianceDescripti
onA quarterly audit to measure vendor’s compliance with SV&T key process areas Formula
Significant deficiency score = Sum of % non-compliance of critical process activities/number of audit assessmentsCompliance score = Sum of % compliance of key process activities/number of audit assessmentsMeasurement IntervalMonth
ly
<2% significant deficiencies on critical process activities≥90% compliance on key process activities
Target
12
CONNECT WITH US:
15
Business Process (“BP”) Knowledge
Description
Formula
Measurement Interval
Target
Quarterly assessment to measure service provider’s staff proficiency in McKesson business process knowledge and knowledge management.Performance = Count of number of people rated at different expertise levels (Advanced, Proficient) segmented by McKesson business process
MonthlyFor each BP rated as Critical or High:1. A minimum of 2 resources rated at advanced2. A minimum of 2 rated as proficientFor each BP rated as Medium or Low:1. A minimum of 1 resources rated at advanced level2. A minimum of 1 rated as proficient level
13
CONNECT WITH US:
General Incident
ManagementDescription
A quarterly assessment of incident closure rates
within the prescribed response time
Formula
Measurement Interval
TargetPerformance = Number of
occurrences where incident is closed on or before the
expected closure date/Number of total
occurrences – Segmented by severity
Quarterly
High Severity Issue - Respond within 2 hours (from the time of notification). Resolve within 8 hours.Medium Severity Issue - Respond within 4 hours (from the time of notification). Resolve within 24 hours.Low Severity Issue - Respond within 8 hours (from the time of notification). Resolve within 5 business days unless both parties agree to incorporate the resolution in the next scheduled release (not to exceed 90 days)
14
CONNECT WITH US:
Automated Regression Test
Library
2 week rolling cycle for periodic runs and on demand cycle based on specific requestPerformance = Actual test completion date ≤ expected completion date
Quarterly>=90% Successful execution of Test Automated Regression test Cases<=10% Successful execution of Test automated regression test cases through manual approach in case of Data setup issues.
Description
Formula
Measurement Interval
Target
15
CONNECT WITH US:
Offshore Technical Skillset 1
SAP ABAP
ABAP WebDyn
pro
SAP XI
SAP EP
SAP SD
SAP CRM
SAP FI/CO
SAP HR
SAP MM
SAP PP
SAP PS
SAP SCM
SAP SRM0
5
10
15
20
25
12
0 13
10
2 3
0 1 0 0 1 2
6
2
0 0
23
5
3
1
6
0 0 0 0
11
0 0 0
12
4
2
6
11
21 1
0
Advanced Proficient BasicOffshore SAP Techni-cal Skill set
SAP BO SAP BW Datastage SAP BODS SAP HANA0
1
2
3
4
5
1
4
2
1
2
1
2 2 2
1
2
3
2
1
3
Advanced ProficientOffshore BI/BW Skill set
16
CONNECT WITH US:
Offshore Technical Skillset 2
Acumax FSCM SCORE TANDEM QTP WMI0
1
2
3
4
1
0
1 1
3
0
2
3
1
0
2
1
0
1
2
0 0
3Advanced Proficient Basic
Offshore Legacy & Automation Skill set
VISTEX CHARGEBACKS
VISTEX CONTRACTS VISTEX MDM VISTEX REBATES0
1
2
3
4
1
2
1 1
3
0
2
3
0
2
3
0
Advanced Proficient BasicOffshore Vistex Skill
set
SAP FI
SAP BI/BW, BO
0 1 2
1
2
2
2
Offshore Bench Strength
No. of Resources
17
Peoplesoft Manual Testing ETL Testing0
0.51
1.52
2.5 2
1
0
2 2
1
2 2
0
Offshore HRIS team Skillset Advanced Proficient
CONNECT WITH US:
Effort per Defect & DREFY13YTD, FY14YTD & April ’14, May ‘14
18
70%
75%
80%
85%
90%
95%
100%
0
10
20
30
40
50
60
7099.00%99.30%98.40%98.03%
50
3943
63
DRE Effort per Defect
CONNECT WITH US:
19
Submission of SLA Report
Reporting Period
Target Date
Publishing Date
Published on time
Jan 20142/5/201
4 2/5/2014 Yes
Feb 20143/5/201
4 3/3/2014 Yes
Mar 20144/5/201
4 4/2/2014 Yes
Apr 20145/5/201
4 5/1/2014 Yes
May 20146/5/201
4 6/4/2014 Yes
CONNECT WITH US:
20
Review of RCA’s
This SLA is included to track RCA’s created vs. percentage of RCA’s reviewed and signed off.
Total of 20 RCA’s are created till May 2014.5 RCA’s are added in May 2014, two for SO 3163 (One Medium,
One Low), one medium defect each for SO 2344, SO 4921, SO 3264.
Total RCA's Created Total RCA's Reviewed0
2
4
6
8
10
12
14
16
18
2020 20
No.
of R
CA's
CONNECT WITH US:
Description Formula
Frequency
Target
A quarterly assessment to measure turnaround time for estimation and sizing of testing initiatives
Performance = number of sizing requests closed on or before scheduled closure date/number of sizing requests submitted – segmented by project type (large, medium and small)
Large Projects – 10 Business DaysSmall & Medium projects – 5 Business Days
Quarterly
Estimation Turnaround
Time
21
CONNECT WITH US:
Description Formula
Frequency
Target
A quarterly assessment of accuracy of metrics provided by service provider. McK SV&T will identify ≥30% of metrics to audit. Prolifics will provide data used to calculate metric. Prolifics will provide/ identify source of data for analysis.
Number of measurement items reported accurate/Total number of measurement items reported.
≥95% accuracy of reported metrics
Quarterly
Metrics Accuracy
22
CONNECT WITH US:
Description
Formula
Frequency
Target
Budget = baseline budget + delta cost. Baseline budget includes effort and cost estimated for test lifecycle activities, Delta cost includes Service Level Credit to cost and effort caused due to authorized changes to scope or other project constraints
.
Performance will be measured on 2 dimensions: Significant Deviation Measure =( Number of projects where actual QA cost exceeds budgeted/forecasted QA cost by 25%/Total number of projects)*100 Accuracy Measure =( Number of projects where actual QA cost is within +/- 10% of budgeted/forecasted QA cost/Total number of projects)*100
Significant deviation measure = 0% Accuracy measure = >= 95%
Quarterly
Delivery within Budget
23
CONNECT WITH US:
Delivery within Budget
No specific data point for this, based on the comments from RTM’s.This metric is difficult to accurately track at this point as the total budget for Prolifics and SV&T employees is combinedSVTCs and RTMs do the estimates in most casesIn BI space, (which is completely outsourced to Prolifics) all SO’s have been delivered with in the budget
TARGET ACTUAL (YTD)
95%100
%
24