atos origin - testing - an olympic challenge

28
Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. September 2006 © 2006 Atos Origin Testing an Olympic Challenge Eurostar, The Hague, 12 November 2008 Magnus Alvarsson

Upload: eurostar-software-testing-conference

Post on 15-Jul-2015

63 views

Category:

Software


1 download

TRANSCRIPT

Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. September 2006© 2006 Atos Origin

Testing an Olympic Challenge

Eurostar, The Hague, 12 November 2008

Magnus Alvarsson

2 Major Events

The Olympic Challenge

» There is no second chance» Proven solutions» Reliable solutions» Meet the Users’

expectations

» World Wide Visibility

» Large IT Deployment

» Multiple partners, suppliers and technology

3 Major Events

Agenda

» Introduction» The Olympic Challenge

» Framework» Project Timeline» Organization

» Testing» Strategy» Testing environments» Process

4 Major Events

Objective

» Ensure stable Games systems that:» Conform to functional requirements» Satisfy or exceed agreed service levels» Support the business of the customers

AND:

» Ensure the integration of:» The numerous and varied Games

systems » The partners, people and processes

supporting these systems

Such that ….

» The systems and business processes complement each other to ensure a seamless delivery of service to our customers

5 Major Events

The Olympic Challenge - Scope

» Atos Origin Systems- 9 Core Games Management Systems (CGS)- 9 Info Diffusion Systems (IDS)- The Olympic Family Intranet in 2 or 3 languages- A large number of interfaces between these systems

» On Venue Results- One On Venue Result System per sport (15/36)- One Timing and Scoring System per Sport and/or Venue- One Scoreboard per Venue- Minimum of 1 TV-Graphics solution per sport

» External Interfaces- On Venue Results- 8 World News Press Agencies- 20 Internet Data Feeds clients (Rights Holding Broadcasters, Media, IF’s and

NOC’s)- Weather- Transportation- Biography and Background Provider(s)- Background Check

6 Major Events

Integration of Partners, People, Processes

» Organizing Committee

» Timing and Scoring / On Venue Results Provider

» Hardware providers» Network» Servers» Desk tops» Cameras» Score boards / Video boards» Mobile devices

» Software Providers» Databases» Operating Systems» Office suites» Etc

The Olympic Challenge – Scope (Cont’d)

7 Major Events

» Operations» Software Distribution Solution» Call Management System» Monitoring» etc

» External Stakeholders» International Olympic Committee» International Sporting Federations» Rights-holding Broadcasters and News Agencies» OCOG Functional Areas» etc

The Olympic Challenge – Scope (cont’d)

8 Major Events

The Olympic Challenge - Scope

INTEGRATION

People, Partners & Processes

• Integration of sponsors/partners

• Customers (VANOC, NBC, WNPAs etc)

• Atos TS, Ops, MEP

• Roles & Resp.

• Policies & Procedures

• Communication

• Project / Risk / Issue Management

• Omega – On-Venue Results Systems

• TV Graphics : Olympic Broadcast Services

• Scoreboard

• RCMP, Immigration

• Data providers

Non-Atos SystemsCore Games Management Systems

• Sport Entries &Qualification

• Accreditation

• Workforce

• Medical

• Arrivals &Departures

• Transportation

• Commentator Information System

• INFO 2010

• Internet Data Feed

• Results Data Feed to WNPAs

• Print Distribution System

• Central Repository System

Information Diffusion Systems

Testing / Integration Test Lab

9 Major Events

Project Timeline

2006

Design

Planning Business Process Reviews GAP Process Functional Requirements

Interface Definition Software development Lab build Staff build-up Software deliveriesTest Tools AcquisitionTest Case preparation

Build

Connectivity Tests Functional Tests Sport/Test Events User Accesptance System Tests Multi Sport Tests

Test

Technical Rehearsals Games TOC operational Applications in production Venue Deployment Maximum staff numbers

Operate

Feb 2010Olympic Games

2010

40

100

200

1,200Resources

10 Major Events

The Challenges

» Challenges» Applications from suppliers worldwide» Synchronizing deliveries and compatible code» Management and control of:

- defects- testing environments- the numerous & varied Integration Test Lab activities

11 Major Events

The Answers

» The answers» A purpose-built Test Lab (500 m2; 5000 ft2)» A dedicated Testing team (80+ people including partners) » A comprehensive multi-faceted test program» Strict test procedures and defect tracking» > 200,000+ hours of testing; > 9,000+ test cases: happy v unhappy path» Support from Technical Services and Operations» Communication & Coordination

Testing: the single most important success factor!

12 Major Events

Objective of Integration testing

Ensure that:» the Games systems work together seamlessly» Fulfill the required functionality and service levels

To successfully manage and support the Olympic and ParaolympicGames.

13 Major Events

To achieve the objective…

» Test functionality of different systems

» Availability test

» Verify operating procedures of the systems

» Test systems under extreme situations

» Focus the tests on data flowing end-to-end through the systems.

» Simulate games

14 Major Events

Integration Test Program

» Pre-Delivery Testing of:» Supplier Development» Supplier “Connectivity”» Supplier Technical

» Integration Testing of:» Delivery Acceptance» Stand-alone» Interface» End-to-End» Regression

15 Major Events

Integration Test Program

Integration Testing of:» Delivery Acceptance» Stand-alone» Interface» End-to-End» Regression

16 Major Events

Integration Test Program

» System Testing (3 cycles of > 20 days each per application)

» Load» Stress » Fault Tolerance» etc

» Acceptance Testing» Test Cases supplied

by the IOC and the Sports Federations or the end users

17 Major Events

Integration Test Program

» Test Events» Every Sport» Elite level competition» Real Games conditions» 1 year before

» Multi-sport Tests» Three cycles» Subset of Games schedule each

time» Overall system performance and

functionality» 2 months period

18 Major Events

Integration Test Program

» Technical Rehearsal (TR1 & TR2)

» Two cycles» Subset of Games schedule

each time» Operational readiness of

systems and people» 10 days period

19 Major Events

Technical Rehearsal main objectives

Technical rehearsals primary focus is to:

» Practice and evaluate:» operational policies and procedures

applied during Games» communication processes

» Establish relationships

» Verify roles and responsibilities

» Test staff utilizing systems/applications in a Games time environment

Beijing example

35 Sports

39 Venues

500 Scenarios

2000 Staff

20 Major Events

Test Organization

Integrated team made up with the following groups:

» Integration Test Board - Atos Origin / OCOG / OmegaOversees, Highest Point of Escalation

» Integration Test Manager and Test Management Team - Atos OriginDefiner & Coordinator of testing program & lab

» Test Group Leaders - Atos Origin / OCOGTest plan managers & delivery managers, escalation point for Test Team Leaders

» Test Team Leaders - Atos Origin / OCOGManagers of test activities and defects by Application, Sport, Intel, Unix, Security

» Testers - Atos Origin / OCOGPerform tests, log and track defects

21 Major Events

Test Organization (cont’d)

A set of integration testing plans, policies and procedures ensuring quality and

consistency in the testing process, including:

» Integration Testing Strategy

» Integration Test Plan

» Defect Tracking Process

» Software Delivery Procedure

» Change Management Process

» Test plans for each sport/system

» etc

22 Major Events

Test Lab Characteristics

» Production like environment

» Tools used for testing activities» Test Management tool (Quality Center)

- Test Cases and Scenarios- Defect Tracking

» Test automation tool (QuickTest Professional) - Test scripts

» Load and Performance testing tools (Load Runner / Win Runner)

» Staff» 35 Atos Origin» 30 VANOC» 6 Omega» 10 Others

23 Major Events

» Physical Infrastructure» 300 PCs» 100 laptops» 50 printers» 32 switches» 450 servers

(30 ITL, 420 TER)

Customer Engagement

Test Lab Characteristics

24 Major Events

Test Cell

25 Major Events

Criteria for Environment Strategy

» Meet objectives stated in the Integration Test Strategy

» Integration and operation activities can be smoothly executed.

» Minimize the total number of environments

» Consider the needs for conducting several continuous tests in parallel without delays.

» Use experience gained from previous Olympic Games.

26 Major Events

Environments

» All applications are installed on all environments.

» Three environments must be production like

» Three environments must have the Games-time configuration

» The rest environments will meet the minimum requirements of the Integration team

» Four environments will be available during the Games,

Date

Environment J F M A M J J A S O N D J F M A M J J A S O N D J F MPDCE2ET1T2SDCPARPPPMEP BCN

Paralympic Testing

Para PreProduction

Testing

Testing Test Events System / MSTs TRs Games

Testing

Testing

2008 2009 2010

Systems Test / DR DR

Homologation PreProduction

27 Major Events

Integration Control

Keys to Integration Testing success for big scale projects:

1. Correlation with Project Plan

2. Defect Tracking Process

3. Control of Software Deliveries

4. Change Management Process

5. Clear Testing Strategy

6. Partner management/Stakeholder agreement

7. Support Tools

8. Common policies & procedures

To minimise risk, the overall IT system and applications must be exhaustively tested

Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. September 2006© 2006 Atos Origin

Questions and Answers

Atos Origin Major EventsAv. Diagonal, 210-218

08018 Barcelona (Spain)www.atosorigin.com

Magnus Alvarsson+1 778 328 [email protected]