approaches for auditing software vendors - … · approaches for auditing software vendors data...

Approaches for Auditing

Software Vendors

Data Integrity Validation Europe

30 March 2017

Chris Wubbolt, QACV Consulting, LLC

Objectives

www.QACVConsulting.com 2

• Understanding impact of vendor processes on validation

• Review of Agile SDLC processes

• New approaches to auditing software vendors

• Understanding how SDLC and test tools are used by vendors

• How SaaS vendors impact your company’s validation approaches and data integrity controls.

Impact of Vendor Practices on

Validation


Validation Plan

User Requirements

Functional Specifications

Configuration Specification

Installation Qualification

System Testing

User Acceptance Testing

Traceability Matrix

Validation Summary Report

Standard Operating Procedures

Internal Validation Vendor

SDLC Deliverables

Software

Internal Validation vs. SaaS-based

Saas-based vs. Internal Validation

Validation Plan

User Requirements




System Testing


Traceability Matrix



SaaS Validation Vendor

SDLC Deliverables

Software

Validation Plan

User Requirements


Traceability Matrix






System Testing

Traceability Matrix

SOPs

Release Management

Quality Agreement


Software Vendor Truisms


Software vendors develop and maintain

software.

All software vendors are software developers.

“Quality” software development is essential to

the validation of a system.

21 CFR Part 11.10 (a):

Validation of systems to ensure accuracy, reliability,

consistent intended performance, and the ability to

discern invalid or altered records.

Software Quality Truisms


Quality cannot be tested into a

system.

Quality must be designed into a

system.

Software Development


Software Development Life Cycle (SDLC)

• The set of activities that constitute the processes that are mandatory for the development and maintenance of software.

• The management and support processes that continue throughout the entire life cycle, as well as all aspects of the software life cycle from concept exploration through retirement, are covered.

• Utilization of the processes and their component activities maximizes the benefits to the user when the use of this standard is initiated early in the software life cycle.(1)

(1) IEEE Standard for Developing Software Life Cycle Processes, 1992

SDLC Methodologies


Code and Fix

Waterfall

Prototyping

Incremental Development

Spiral

Rapid Application Development

Agile

(Cowboy Coding)


Require-ments

Design

Testing (unit,

module, system,

etc.)

Bug Fixes

Config-uration

Manage-ment

SQA Testing

Release Manage-

ment

Mainte-nance(Cus-tomer

Support)

Elements of an SDLC


Quality Manual

Document Management

Training Program

Quality Assurance

Supplier Management

CAPAs / Investigations

SDLC Procedures

Customer Support

Vendor Quality System Elements

Requirements

Analysis

Design

Implementation

Verification /

Testing

Operation /

Maintenance

Requirements

Analysis

Design

Implementation

Verification /

Testing

Operation /

Maintenance


Waterfall Methodology


SDLC – Agile Methodology



• Focus on short iterations of development

• Delivery of minimum viable product within short periods of time (2-3 weeks)

• Collaboration between end user and development team

• Continuous end user involvement is critical

An iterative and incremental agile development framework.

A flexible, holistic strategy where a development team works as a unit to reach a common goal.

Enables teams to self-organize by encouraging physical co-location or close online collaboration and daily face-to-face communication among all team members and disciplines in the project.


Agile - Scrum

A key recognition is that during end users can change their minds about the system requirements.

Scrum adopts an approach to deliver quickly and respond to emerging requirements.

Agile - Scrum



Software Vendor Truisms

All software vendors are software developers.

The software development life cycle

methodology is arguably the most important

process for a software vendor.

Requirements

Backlog

User StoriesDesign/Development

Unit Testing

Code Reviews

Design Documents

SQA Testing Release

Management


Why is this important?

Requirements

Backlog

User StoriesDesign/Development

Unit Testing

Code Reviews

Design Documents

SQA Testing Release

Management

1. The vendors SDLC determines the quality of

the software.2. For SaaS vendors, the SDLC documentation

may also be used as validation deliverables.3. The SDLC documentation is likely to be

maintained within vendor SDLC tools.


Use of SDLC and Test Tools

Requirements

Backlog

User StoriesDesign/Development SQA Testing Release

Management

Creation and Management of

Requirements & User Stories

Documentation of Unit

Testing, Code Reviews

& Design Documentation

SQA Test

DocumentationOften used as

“validation” tests.

Configuration / Source Code Management

Management of Bugs and Customer Support Tickets

SDLC/Vendor Tools


Requirements Management

Source Code Management

Configuration Management

Code Review and Unit Testing

Testing – including automated testing

Issue Management

Customer Support

Document Management

SDLC/Vendor Tools - Examples


Test Stuff

Test Track

CoSign

SharePoint

Wiki Pages

Salesforce.com

Team Foundation

Server (TFS)

HP Quality Center

HP Load Runner

Altassian (Jira)

Subversion

SDLC Tools


Team Foundation Server (TFS)

• Requirements Management

• Use Cases

• User Stories

• Design

• Code Review

• Unit Testing

• Traceability

• Testing

• Approvals

• Release Management

SDLC Tools – Questions to ask


What do the tools do?

Do the tools impact software quality?

Do the vendor’s procedures reflect the use of these tools?

Are the tools controlled, qualified, or validated?

How are the records maintained by the tools managed and controlled?

How are records approved?

SDLC Tools – What can go wrong?


Issue Management

• Vendor used a cloud “hosted” version of Jira, which was used for issue management and change control.

• The license was not renewed and all records were lost.

Electronic Approval

• Vendor used a local implementation of CoSign for approval of records.

• When license expired the electronic signatures applied previously could not be validated.




Document Management

• Vendor used SharePoint workflow for approval of quality documents. The SharePoint configuration was setup to delete workflows after 90 days.

• All workflows (and subsequent document approvals) were deleted for all quality documents.

Testing

• Test Stuff testing records could not be located for SQA testing.




Automated Testing

• Automated test tools passed failing results.

• Test tools were not qualified.

Tool Upgrades / Replacements

• Inability to migrate records from legacy tools.

Records

• Unable to present records of SDLC activities, including test results.


• GxP Electronic Recordkeeping Program

• Standard Operating Procedures

• Trained Personnel (including IT)

• Qualified Infrastructure

• Validated Applications

Data Integrity

Data Availability

Data Retention

Computerized Systems

www.QACVConsulting.com 27www.QACVConsulting.com 27


Historical

Software Applications

QMS

LIMS



Historical


QMS

LIMS


Pharma A

GxPElectronic Recordkeeping

Controls

Qualified Infrastructure


Trained Personnel (including IT)

Validated ApplicationsSTILL NEED

Data Center Inc


Historical


Software as a Service


Fail Over Site


QMS

LIMS

Saas Provider

Data Center

31

Software

Vendor

• Quality System

• SDLC Processes

• Customer Support

Typically not directly regulated or inspected by regulatory agencies.

Audited by clients for adherence to standards.

Quality of SLC Documentation, Testing, etc. varies considerably for each

vendor.

Sponsor responsible for installation, validation, and electronic

recordkeeping controls at sponsor location.

Software as a Service Provider• Quality System

• SDLC Processes

• Customer Support

• Validation

• Data Integrity Controls

Hosted Environment is used for a direct GxPfunction (record keeping)

and is more likely to be inspected by regulatory agencies.

Audited by clients for adherence to standards (GxP, Part 11).

Quality of SDLC Documentation, Testing, etc. varies considerably for

each vendor.

SaaSprovider responsible for some aspects of installation, validation,

and electronic recordkeeping controls.


Software Vendor

Hosted

Environment

32

SaaS Vendor Responsibilities

• Validation (with Pharma Company)

• Change Control

• Incident Management

• Maintenance

• Security (Physical and Logical)

• Electronic recordkeeping

• Backup and Restore

• Disaster Recovery


Vendor Audit Observations -

Considerations• Specifications

– Not complete

– Not updated periodically after changes

• Test Records

– No pre-approved Test Plans

– Results not reviewed by second person

– Integrity of test results

– No approved summary reports

• Release Management


Vendor Audit Observations –

Considerations

• Test Record Integrity

– Results and signatures/initials typed into Word document or Excel spreadsheet

– No failures documented

– Test dates and times do not correlate


Vendor Audit Observations –

Considerations• Record Integrity

– Lack of records to demonstrate successful backup

– Failed backups

– Lack of documentation of disaster recovery testing


Summary


• Reviewed impact of vendor processes on validation

• Review of Agile SDLC processes

• Discussed new approaches to auditing software vendors

• Reviewed how SDLC and test tools are used by vendors

• Discussed ow SaaS vendors impact your company’s validation approaches and data integrity controls.

Questions


Chris Wubbolt

QACV Consulting, LLC

Telephone: 610-442-2250

E-mail: [email protected]