Best Practices on Incorporating Quality Assurance Into Your

Software Development Life Cycle

Katya Sadovsky, Applications ArchitectCarmen Roode, Associate Director of Systems Development

Marina Arseniev, Associate Director of Enterprise Architecture

University of California, Irvine

University of California, Irvine

• Located in Southern California• Year Founded:  1965• Enrollment: over 24K students• 1,400 Faculty (Academic Senate)• 8,300 Staff• 6,000 degrees awarded annually• Carnegie Classification:  Doctoral/Research – Extensive• Extramural Funding - 311M in 2005-2006• Undergoing significant enrollment growth

Today’s World

What’s key with today’s systems• Security – data and reputation• Expectations are different – accessibility, availability

– Systems tied to employee mobility – 24 hours a day– No training, intuitive, flexible/changeable

• SDLC needs to support more complex environment – distributed computing environment, different architecture, components, web services

• SDLC has to reflect today’s business environment

What we still see

Examples of avoidable problems

• Systems failing during peak usage

• Insecure systems – break-ins

• e-Voting debacles

Quality Assurance within the SDLC

Common life cycle phases• Budget & Staffing• Requirements/Prototyping• Design• Development • Testing• Roll-out• Maintenance• Decommissioning of the system

Agenda

• Education and Staffing

• How-tos in the Software Development Life Cycle Process

• Templates and Checklists

• Outcome/Summary

• Useful URLs and Q&A

Education and Staffing

• Train staff on SDLC

• Train on minimum competencies– Technologies – i.e. Java, databases– Project management– UML– Certifications

• Ongoing training budget allocation for all staff

Education and Staffing

• Split out project by experience level– Coding of key software done by more

experienced staff (i.e. authentication & authorization)

– Code and design reviews– Training and mentoring

• Have minimum technical competency hiring standards

Business Unit Education

• Educate business units on acquiring secure and quality systems

• Business modeling and functional requirements

• Knowing security issues, laws, regulations and avoiding bad practices:

– Regulated or confidential information can be unintentionally transmitted through email

– Private data can be entered into a text field

Agenda

• Education and Staffing

• How-tos in the Software Development Life Cycle Process

• Templates and Checklists

• Outcome/Summary

• Useful URLs and Q&A

Software Development Life Cycle Without QA

The Software Development Life Cycle

• Traditional waterfall method puts QA at the end of the development process

• Performance testing after QA• What you find

– The system functions correctly but it’s too slow to use

– This means architecture solution is incorrect– Example – object to relational re-architecture

required for performance

The Software Development Life Cycle

How to embed QA into your SDLC

• Documented formal steps

• Templates

• Checklists

• Reviews

• Approvals

• Formal Change Control

Integrating Quality Assurance in Requirements

Integrating QA in Requirements

Acquisition or development• Users to identify needs, document requirements• Prototype what they want• Identify things that can change• Identify business service constraints

• Acceptable levels of downtime• Acceptable levels of data loss• Business cycles

• Identify Security requirements (examples: Compliance requirements – PCI, SB1386, FERPA, HIPAA, Risk assessment – normal or high risk application?)

Integrating QA in Requirements

• Know your business cycles

• Students – know cutoff dates for students

• Financial – know fiscal cycles, month end, year end

• Payroll – hiring waves related to academic year, paycheck generation & view

• Plan on peak usage during load testing

Integrating Quality Assurance in Design

Integrating QA in Design

• Costliest mistakes to fix are introduced in design• To ensure QA, SDLC must include a design review

process• Use your most experienced staff!• Checklist for adherence to architecture and

technology standards• ex: Java, .Net, Portal, new technology? Exception?

• Check for appropriate design solution – use rubric  • If 24/7 is required, what needs to be in the design to assure

Quality?• scalability, availability, accessibility, redundancy?• design for clustering, fault tolerance, load balancing…

Integrating QA in Design – Design for Reuse

• SDLC process must promote identification, design, and reuse of common tested components to achieve QA

• Requirements phase extracts reuse candidates for design• document management, user/group management, workflow…

• Design phase must identify and provide solutions for vulnerable points

• authentication and authorization/access control• database or file stores of sensitive data• logging/auditing

– QA has to test the ability to do forensics – Design security solutions with forensics QA in mind– QA your encryption, auditing, and logging design

Integrating QA in Design – Design for Testability

• Design system and code for load testing QA– Load testing must test redundancy, clustering and peak

concurrency requirements– code is written differently for load testing

• Determine how you will test and write test plans at design time!– Distributed components / Web Services testing

• QA is done at different layers – network, database, messaging

– Design for Disaster Recovery QA

• SDLC Design phase must guarantee Testability

Integrating Quality Assurance During Implementation, Testing, and Remaining

Phases

Integrating QA During Implementation

Implementation – make QA “routine” • Schedule code reviews • Require developers to build unit test harnesses – Junit• Automate nightly code and application security scanning –

Jtest, AppScan, Nessus, database security scanning• Schedule network and configuration scanning - Foundstone• Write and use manual security test procedures• Perform concurrency and stress testing - Jmeter, OpenSTA• Integration testing

– Services and APIs– Are services or distributed components using encryption?– How does an application authenticate to a service?

Integrating QA During Testing

• Functional testing – Do you use formal Test Plans or AdHoc? Tied to

Requirements?– Done by developers and end users?– Do Pilot Users test methodically using Test Plans? – How do you ensure testing coverage is adequate?– SQL Injection and Cross-site Script testing– Browser Compatibility Testing (ex: browser cache)– Regression testing

Integrating QA in Remaining Phases

• Deployment– Helpdesk, Sys Admin, support staff cross-trained?– Policy issues identified?– System and data backups, disaster recovery

• Operations/Maintenance– Repeated “routine” reviews and scanning– Change control

• Decommissioning of Application and Data– Retention/preservation of information and data– Sanitize media, properly dispose hardware and

software

Agenda

• Education and Staffing

• How tos in the Software Development Life Cycle Process

• Templates and Checklists

• Outcome/Summary

• Useful URLs and Q&A

Templates and Checklists

ASP Vendor Security Checklist

• What certification or audits does the University have that the system will be managed per our guidelines and contract agreement?

• How do you manage the system for detection of intrusion.

• How often is the system patched, by whom and when?

• How are we notified if system security is breached? Notification handling?

• How is data purged from the vendor's hardware?

• How are disks, tapes, or computers that might store sensitive data disposed of? Are the media erased before disposal or reuse?

• Where is the hardware location? Is it inside or outside of the United States? Is it subject to our laws?

• Are the personnel who administer and use the hardware located within the United States and subject to our laws?

• Is data encrypted? • If private data is transmitted, either via

Internet, on CD-ROM or file transfer, is it encrypted?

• Is SSL enabled to the application so that traffic over the Internet, including authentication is secure and private?

• Data loss, data backups: what are the guarantees? Are backups stored offsite? If backups have sensitive data, are the backups encrypted? Can we store the backup at UCI? How about disaster recovery planning?

• How is the hardware or database distributed by the vendor among customers? Is one hardware used for all customers? Is a single database used for all customers or does each customer have a private database?

• How are user accounts managed?

Our Change Control Process

• Coordinate and schedule changes in network, database, applications, OS, firewalls and configurations– Avoid downtime due to collisions– Avoid accidental security exposures – We use Oracle Calendar

• All developers, system and network admins meet every Tuesday morning for at least 15 minutes!

• 2 week notice of all planned changes– Test Plan and checklist required– Identification of required security tasks

• High/low risk identified on all changes• Changes recorded in AdCom Service Desk

Agenda

• Education and Staffing

• How-tos in the Software Development Life Cycle Process

• Templates and Checklists

• Outcome/Summary

• Useful URLs and Q&A

Achieved Outcome

Before AfterAdHoc Manual Testing Automated – JTest, JUnit, OpenSTA,

JMeter, AppScan, FoundScan

12 PCs, 6 People (2 PCs per Person), 12 Concurrent Test Users

1 PC, 1 Person, 100s of Concurrent Virtual Test Users

Unscheduled Changes Coordinated, Scheduled Changes

No Checklists 10 Checklists and Templates

No Reviews Code and Security Reviews Before Production Release

AdHoc Training SDLC Training, Certification

No Architectural Controls Technical Reference Architecture

Limited Capacity Planning Capacity Planning via Load Tests

Q&A

Useful Links• AdCom’s SDLC Guidelines: http://

snap.uci.edu/viewXmlFile.jsp?resourceID=1535

• Development for technology professionals: http://snap.uci.edu/viewXmlFile.jsp?resourceID=1433