CONNECT WITH US:

Testing as a Managed Service Using SLA’s and KPI’s

Sarat Addanki

Head of Practice

CONNECT WITH US:Public | Copyright © 2014 Prolifics

Agenda

2

01

03

04

05

06 02

Corporate overview

SLAs

KPIs

Sample SLAreporting

Demo

Q&A

CONNECT WITH US: 3

5-Years CompoundAnnual Growth Rate

19%

EmployeesWorldwide

1,500

Global PresenceUnited States, United Kingdom, Germany, India

20+Technology

Accelerators

550+Technical

Certifications

Over 10 Technology and Solutions Awards since 2009

including Business Agility, Customer Integration and Digital

Experience

Over 150 global customersare currently Fortune

1000 companies

Best-in-class architects andspecialty experts:

BPM, IIB, DataPower, WSRR, and other IBM products and

tools.

industry-high 96% success rate of deploying robust solutions.

96%

Prolifics at a Glance

Years inBusiness

35+

Offices14

Awards

IBM Technology Expertise

Fortune 1000

Rate of Repeat Engagements

91%

Public | Copyright © 2014 Prolifics

CONNECT WITH US:

Our Services

Public | Copyright © 2014 Prolifics 4

• Application Development, Maintenance and Support

• Outsourced Product Development

• Application Modernization

• Portal Content & Collaboration

• Test Advisory• Functional Testing

• Packaged Apps

• Custom Apps• Cloud

• Test Automation• Middleware and

SOA• Performance Engg• Mobile Testing

BPM & ConnectivityUser ExperienceSecurity

Application Performance ManagementEnterprise Content ManagementBusiness/Data Analytic

CONNECT WITH US:

How Prolifics Can Help You

5

Adapt faster than the competition with continuous process improvement

Transform your applications to increase business agility

Create best-in-class customer and employee experiences. Become more social and collaborative

Gain actionable insight with an intelligent approach to security

BUSINESS PROCESSMANAGEMENT

SOA INTEGRATION USER EXPERIENCE SECURITY

Drive business growth with proactive IT management

Maximize IT investments with end-to-end testing

Optimize business performance with smarter, faster decisions

Increase efficiency within your organization and achieve “straight through processing”

APPLICATIONINFRASTRUCTURE

TEST ADVISORY, APPLICATION TESTING & TEST

AUTOMATION

BUSINESS ANALYTICSOPERATIONAL DECISION

MANAGEMENT (ODM)

Simplify employee and customer access to the documents they need when they need them

Get the most out of your IT to create competitive advantage.

Leverage global expertise to support your IT environment

Optimize your software spend to achieve business value

ENTERPRISE CONTENT MANAGEMENT

ADVISORY SERVICES MANAGED SERVICES SOFTWARE

Public | Copyright © 2014 Prolifics

CONNECT WITH US: 6

Prolifics Customers

Healthcare

6

Financial Services

Utilities

Retail & Distribution Media & Entertainment

Government Technology

Telecommunications Insurance

HealthCare

Public | Copyright © 2014 Prolifics

CONNECT WITH US: 7Companies are moving from Outsourcing Model to Managed Services Model as it allows them to govern the vendor

services with the help of mutually decided upon Service Level Agreements (SLA’s).

Evaluate value

Measure quality

Continues improvement

Determine cost of testing

Create accountability

Measure productivity

Process Compliance

Within Budget

% of Automation

DREIncident Mgmt.

Business Process

Knowledge

Test Design Efficiency

Manage vendor

SLA’s are vital signs. You can tell where a testing organization is headed and also measure it’s maturity level

Why Managed Service and SLAs?

CONNECT WITH US:

Key Performance Indicators (KPI’s)

8

Test case effectiveness

Test progress plan vs actual

Cost of testing as a % development

Skill set matrix

Quality of development

No. of regression test cases updated per release

Quantifiable regression execution

CONNECT WITH US:

SLAs and Measurement Intervals

Defect Removal EffectivenessProcess Compliance Business Process KnowledgeGeneral Incident ManagementAutomated Regression library - ExecutionRegression Library - Updates

123456

789

1011121314151617

Offshore Technical SkillsetEffort per DefectSavings thru DefectsThrough by Project typesResource UtilizationSubmission of SLA ReportReview of RCATest Design EfficiencyDelivery within budgetSizing and Estimation TurnaroundMetrics Accuracy

MonthlyMonthlyMonthlyMonthlyMonthlyMonthlyMonthlyMonthly

QuarterlyQuarterlyQuarterly

MonthlyQuarterlyQuarterlyQuarterlyMonthlyMonthly

S. No

SLA

Non SLA Metrics

Measurement Interval

9

CONNECT WITH US:

DRE Process Compliance Business Process Knowledge

General Incident Mgmt. Automated Regression Library - Execution

Regression Library - Updates

100%

98%

100%

91%

100%

100%

Snapshot of SLA MetricsJune, 2014

10

SLA Target, 100%

SLA Target, 100%

SLA Target, 98%

SLA Target, 90%

SLA Target, 98%

SLA Target, 95%

CONNECT WITH US:

Description Formula

Frequency

Target

Measure of the quality delivered in a given engagement by measuring the number of defects that seeped into production

Performance = Number of non rejected defects found prior to production / (total number of non rejected defects registered during development life cycle from requirements phase up to and including warranty period) * 100

Critical and High - 95%

Medium – 90 %

Low – 85 %

Monthly

Defect Removal

Effectiveness

11

CONNECT WITH US:

Process ComplianceDescripti

onA quarterly audit to measure vendor’s compliance with SV&T key process areas Formula

Significant deficiency score = Sum of % non-compliance of critical process activities/number of audit assessmentsCompliance score = Sum of % compliance of key process activities/number of audit assessmentsMeasurement IntervalMonth

ly

<2% significant deficiencies on critical process activities≥90% compliance on key process activities

Target

12

CONNECT WITH US:

15

Business Process (“BP”) Knowledge

Description

Formula

Measurement Interval

Target

Quarterly assessment to measure service provider’s staff proficiency in McKesson business process knowledge and knowledge management.Performance = Count of number of people rated at different expertise levels (Advanced, Proficient) segmented by McKesson business process

MonthlyFor each BP rated as Critical or High:1.      A minimum of 2 resources rated at advanced2.      A minimum of 2 rated as proficientFor each BP rated as Medium or Low:1.      A minimum of 1 resources rated at advanced level2.      A minimum of 1 rated as proficient level

13

CONNECT WITH US:

General Incident

ManagementDescription

A quarterly assessment of incident closure rates

within the prescribed response time

Formula

Measurement Interval

TargetPerformance = Number of

occurrences where incident is closed on or before the

expected closure date/Number of total

occurrences – Segmented by severity

Quarterly

High Severity Issue - Respond within 2 hours (from the time of notification). Resolve within 8 hours.Medium Severity Issue - Respond within 4 hours (from the time of notification). Resolve within 24 hours.Low Severity Issue - Respond within 8 hours (from the time of notification). Resolve within 5 business days unless both parties agree to incorporate the resolution in the next scheduled release (not to exceed 90 days)

14

CONNECT WITH US:

Automated Regression Test

Library

2 week rolling cycle for periodic runs and on demand cycle based on specific requestPerformance = Actual test completion date ≤ expected completion date

Quarterly>=90% Successful execution of Test Automated Regression test Cases<=10% Successful execution of Test automated regression test cases through manual approach in case of Data setup issues.

Description

Formula

Measurement Interval

Target

15

CONNECT WITH US:

Offshore Technical Skillset 1

SAP ABAP

ABAP WebDyn

pro

SAP XI

SAP EP

SAP SD

SAP CRM

SAP FI/CO

SAP HR

SAP MM

SAP PP

SAP PS

SAP SCM

SAP SRM0

5

10

15

20

25

12

0 13

10

2 3

0 1 0 0 1 2

6

2

0 0

23

5

3

1

6

0 0 0 0

11

0 0 0

12

4

2

6

11

21 1

0

Advanced Proficient BasicOffshore SAP Techni-cal Skill set

SAP BO SAP BW Datastage SAP BODS SAP HANA0

1

2

3

4

5

1

4

2

1

2

1

2 2 2

1

2

3

2

1

3

Advanced ProficientOffshore BI/BW Skill set

16

CONNECT WITH US:

Offshore Technical Skillset 2

Acumax FSCM SCORE TANDEM QTP WMI0

1

2

3

4

1

0

1 1

3

0

2

3

1

0

2

1

0

1

2

0 0

3Advanced Proficient Basic

Offshore Legacy & Automation Skill set

VISTEX CHARGEBACKS

VISTEX CONTRACTS VISTEX MDM VISTEX REBATES0

1

2

3

4

1

2

1 1

3

0

2

3

0

2

3

0

Advanced Proficient BasicOffshore Vistex Skill

set

SAP FI

SAP BI/BW, BO

0 1 2

1

2

2

2

Offshore Bench Strength

No. of Resources

17

Peoplesoft Manual Testing ETL Testing0

0.51

1.52

2.5 2

1

0

2 2

1

2 2

0

Offshore HRIS team Skillset Advanced Proficient

CONNECT WITH US:

Effort per Defect & DREFY13YTD, FY14YTD & April ’14, May ‘14

18

70%

75%

80%

85%

90%

95%

100%

0

10

20

30

40

50

60

7099.00%99.30%98.40%98.03%

50

3943

63

DRE Effort per Defect

CONNECT WITH US:

19

Submission of SLA Report

Reporting Period

Target Date

Publishing Date

Published on time

Jan 20142/5/201

4 2/5/2014 Yes

Feb 20143/5/201

4 3/3/2014 Yes

Mar 20144/5/201

4 4/2/2014 Yes

Apr 20145/5/201

4 5/1/2014 Yes

May 20146/5/201

4 6/4/2014 Yes

CONNECT WITH US:

20

Review of RCA’s

This SLA is included to track RCA’s created vs. percentage of RCA’s reviewed and signed off.

Total of 20 RCA’s are created till May 2014.5 RCA’s are added in May 2014, two for SO 3163 (One Medium,

One Low), one medium defect each for SO 2344, SO 4921, SO 3264.

Total RCA's Created Total RCA's Reviewed0

2

4

6

8

10

12

14

16

18

2020 20

No.

of R

CA's

CONNECT WITH US:

Description Formula

Frequency

Target

A quarterly assessment to measure turnaround time for estimation and sizing of testing initiatives

Performance = number of sizing requests closed on or before scheduled closure date/number of sizing requests submitted – segmented by project type (large, medium and small)

Large Projects – 10 Business DaysSmall & Medium projects – 5 Business Days

Quarterly

Estimation Turnaround

Time

21

CONNECT WITH US:

Description Formula

Frequency

Target

A quarterly assessment of accuracy of metrics provided by service provider. McK SV&T will identify ≥30% of metrics to audit. Prolifics will provide data used to calculate metric. Prolifics will provide/ identify source of data for analysis.

Number of measurement items reported accurate/Total number of measurement items reported.

≥95% accuracy of reported metrics

Quarterly

Metrics Accuracy

22

CONNECT WITH US:

Description

Formula

Frequency

Target

Budget = baseline budget + delta cost. Baseline budget includes effort and cost estimated for test lifecycle activities, Delta cost includes Service Level Credit to cost and effort caused due to authorized changes to scope or other project constraints

.

Performance will be measured on 2 dimensions: Significant Deviation Measure =( Number of projects where actual QA cost exceeds budgeted/forecasted QA cost by 25%/Total number of projects)*100 Accuracy Measure =( Number of projects where actual QA cost is within +/- 10% of budgeted/forecasted QA cost/Total number of projects)*100

Significant deviation measure = 0% Accuracy measure = >= 95%

Quarterly

Delivery within Budget

23

CONNECT WITH US:

Delivery within Budget

No specific data point for this, based on the comments from RTM’s.This metric is difficult to accurately track at this point as the total budget for Prolifics and SV&T employees is combinedSVTCs and RTMs do the estimates in most casesIn BI space, (which is completely outsourced to Prolifics) all SO’s have been delivered with in the budget

TARGET ACTUAL (YTD)

95%100

%

24