the company…. the market leader in globalization technology –pioneers in visual translation...

The Company…. The Market Leader in Globalization

Technology– Pioneers in visual translation environments– Solutions for major platforms &

programming languages– 24.9% owned by Corel Corporation– Sales growing 40+% annually– Currently 9,000+ customers worldwide– Offices in Ireland and USA

• Japanese office to open Q3 2003

Challenges

Simultaneous Translation,

Engineering & Testing We live in a Parallel Universe

– Translation• Seldom starts or finishes on time• Difficulty getting source materials from development

teams• Diverse goals of translation/development teams• Goal : Translate as much as possible as quickly as

possible

- Quality Assurance • To monitor quality levels of Engineering• To monitor quality levels of Translation• Goal : To locate, verify and report as many quality

performance issues during the schedule

– Engineering• Struggle to control inputs to drive definite outcomes• Create continuous streaming of localization materials to

Translators and Quality Assurance team• Goal : To locate, debug and fix as many quality performance issues during the schedule

QA Engineering Overview of Test Phases

– Evaluation & Planning Phase – Goal : Defines the scope of the Test Plan– Includes Resource Plan & Scehdule– Product Modularization Report & Baseline Defined

- Exploratory Phase- Goal : Gain familiarity with application- Pretesting of application

- Development Phase- Goal : Create Test Scenarios to support Test Plan

- Execution Phase- Modular testing of application areas- Internationalization & Language Verification

- Pruning Phase- Interlaced throughout your testing cycles- Provides audit for improving process and test scenarios

- Regression & Verification Phase- Retest and verify Engineering activities

- Sign-off Phase- Additional testing requirements for release and mastering

QA Engineering Designing a Test Case

Clearly Defined Goal* What is the function of the test ?

Clear Test Activity* What steps do I follow to create the test scenario ?

Deterministic Outcome* How do I determine success/failure of test ?

Remember!Remember!

Tests can only prove that errors exist, Tests can only prove that errors exist, not that a product is free of errors!not that a product is free of errors!

Tests can only prove that errors exist, Tests can only prove that errors exist, not that a product is free of errors!not that a product is free of errors!

– Common Types of Tests– MenuWalker– Functional- Scenario- Random- Automated- Interoperability- Language Quality- Data Path- Boundary - I18N

QA Engineering Types of Testing

Incremental Testing StrategyIncremental Testing Strategy

For optimal efficiency, your L10N teams should not test baseline functionality! It should focus on

incrementally testing that the product integrity is maintained through the localization process.


– Common Types of Tests– MenuWalker– Functional- Scenario- Random- Automated- System- Acceptance- Language Quality- Data Path- Boundary - I18N- Performance

MenuWalker TestsMenuWalker Tests

ObjectiveObjective

• Exercise all UI of a Exercise all UI of a translated producttranslated product

• Locate all cosmetic Locate all cosmetic issuesissues

ExecutionExecution

• Easily AutomatedEasily Automated

• Manual Scripts easy to Manual Scripts easy to developdevelop

• Various ToolsVarious Tools : : Alchemy CATALYST RTV, Alchemy CATALYST RTV, ToolProof, Visual TestToolProof, Visual Test

Deterministic OutcomesDeterministic Outcomes

• UI devoid of all cosmetic UI devoid of all cosmetic and layout bugsand layout bugs

• Translation Translation Completeness Completeness


Functional TestsFunctional Tests

ObjectiveObjective

• Verify the functional Verify the functional integrity of applicationintegrity of application

ExecutionExecution

• Modular approach is Modular approach is bestbest

• Focus on Unit, Module Focus on Unit, Module and Componentsand Components

• Data Comparative cases Data Comparative cases are easily automatedare easily automated


•Product is functionally Product is functionally consistent with original consistent with original base language applicationbase language application



Scenario TestsScenario Tests

ObjectiveObjective

• Verify product functional Verify product functional integrity in various real-integrity in various real-world test scenariosworld test scenarios

ExecutionExecution

• Previous regression tests Previous regression tests and technical support and technical support feedback can be usefulfeedback can be useful

• Can be complex to Can be complex to automateautomate

• Focus on many System Focus on many System variationsvariations


•Product is functionally Product is functionally consistent with original consistent with original base language applicationbase language application



Random TestsRandom Tests

ObjectiveObjective

• Verify product functional Verify product functional integrity using random integrity using random test data inputs and test data inputs and scenariosscenarios

ExecutionExecution

• Unstructured testing of Unstructured testing of application in real-world application in real-world scenariosscenarios

• Almost impossible to Almost impossible to automateautomate

• Only useful with Only useful with experienced QA Engineersexperienced QA Engineers


• Generally none, but may Generally none, but may prove ‘Product prove ‘Product Robustness’Robustness’




Automated TestsAutomated Tests

ObjectiveObjective

• Verify product functional Verify product functional integrity using automated integrity using automated test casestest cases

ExecutionExecution

• Structured testing of Structured testing of application in controlled application in controlled scenariosscenarios

• Very effective with Very effective with comparative test casescomparative test cases


• Software integrity is Software integrity is maintained in controlled maintained in controlled test casestest cases

• Acceptance of software Acceptance of software at significant project at significant project milestonesmilestones



Automated TestsAutomated Tests

ObjectiveObjective

• Verify product functional Verify product functional integrity using automated integrity using automated test casestest cases

ExecutionExecution

• Structured testing of Structured testing of application in controlled application in controlled scenariosscenarios

• Very effective with Very effective with comparative test casescomparative test cases


• Software integrity is Software integrity is maintained in controlled maintained in controlled test casestest cases

• Acceptance of software Acceptance of software at significant project at significant project milestonesmilestones

Automator, beware !Automator, beware !

Automation is Resource Intensive

•Developing Automated Scripts can be time-consuming

• Need understanding of programming techniques

• If not developed correctly, can be very high on maintainence

No Replacement for real testing

• Can compliment but not replace real ‘testers’

Not everything adapts well to test automation

• Good : Tests that verify an application works in a particular way

• Bad : Tests that find previously undiscovered errors


System TestsSystem Tests

ObjectiveObjective

• Verify product functional Verify product functional integrity under different integrity under different system environmentssystem environments

ExecutionExecution

• Structured test cases Structured test cases using various OSusing various OS

• Lends itself to Lends itself to automationautomation


• Functional completness Functional completness on various OS on various OS environmentsenvironments

• Performance of Performance of application application

• Verification of product Verification of product license integritylicense integrity

•Completeness of Product Completeness of Product International SupportInternational Support



Acceptance TestsAcceptance Tests

ObjectiveObjective

• Verify minimum product Verify minimum product quality to commence QA quality to commence QA cyclecycle

ExecutionExecution

• Very Structured test Very Structured test casescases

• Improve by including Improve by including previously regressed SPRspreviously regressed SPRs

• Significant automation Significant automation canidatecanidate


• Verification of minimum Verification of minimum functional completenessfunctional completeness

• First line of defence for First line of defence for detecting ‘corrupted’ detecting ‘corrupted’ buildsbuilds




Language Quality TestsLanguage Quality Tests

ObjectiveObjective

• Verify translation is Verify translation is consistent with linguistic consistent with linguistic guidelinesguidelines

ExecutionExecution

• Some fundamental tests Some fundamental tests can be easily automatedcan be easily automated

• Sampling generally Sampling generally creates the best results creates the best results due to volumesdue to volumes


• Translation is consistent Translation is consistent with Corporate with Corporate Terminology guidelines Terminology guidelines across component across component boundariesboundaries

• No cosmetic/layout No cosmetic/layout issues exist (e.g. Spelling issues exist (e.g. Spelling errors)errors)



Language Quality TestsLanguage Quality Tests

ObjectiveObjective

• Verify translation is Verify translation is consistent with linguistic consistent with linguistic guidelinesguidelines

ExecutionExecution

• Some fundamental tests Some fundamental tests can be easily automatedcan be easily automated

• Sampling generally Sampling generally creates the best results creates the best results due to volumesdue to volumes


• Translation is consistent Translation is consistent with Corporate with Corporate Terminology guidelines Terminology guidelines across component across component boundariesboundaries

• No cosmetic/layout No cosmetic/layout issues exist (e.g. Spelling issues exist (e.g. Spelling errors)errors)

Examples of Language Quality AutomationExamples of Language Quality Automation

Translation Consistency & Completeness Check

•Relatively easy to automate

•Should always be part of an acceptance test

Spell Checking

• Relatively easy to automate

• Should always be part of an acceptance test

Duplicate Hotkeys/Layout Issues/Placeholders

• Relatively easy to automate & locate


Data Path TestsData Path Tests

ObjectiveObjective

•Verify data integrity of Verify data integrity of application based on application based on knowledge of internal knowledge of internal workings of applicationworkings of application

ExecutionExecution

• Structured data used to Structured data used to test applicationtest application

• Focus on inputs driving Focus on inputs driving deterministic outputsdeterministic outputs

• Easy to automate with Easy to automate with comparative data comparative data outcomesoutcomes


• Data input and Data input and generated outputs are generated outputs are consistent with base consistent with base language applicationlanguage application


NOTE : Sometimes referred to as Grey box testing


Boundary TestsBoundary Tests

ObjectiveObjective

•Verify application Verify application functionality with extreme functionality with extreme data inputsdata inputs

ExecutionExecution

• Structured data used to Structured data used to test applicationtest application

• Relatively easy to Relatively easy to automate with automate with comparative data comparative data outcomesoutcomes


• Data input and Data input and generated outputs are generated outputs are consistent with base consistent with base language applicationlanguage application

• Applicatio robustness – Applicatio robustness – illegal data values do not illegal data values do not crash applicationcrash application



L18N TestsL18N Tests

ObjectiveObjective

• Verify application Verify application supports all Localessupports all Locales

ExecutionExecution

• Structured data that Structured data that focus on locale usedfocus on locale used


• Break down into Break down into categories : Script & categories : Script & RegionalRegional


• Application supports Application supports local script and regional local script and regional conventionsconventions



L18N TestsL18N Tests

ObjectiveObjective

• Verify application Verify application supports all Localessupports all Locales

ExecutionExecution

• Structured data that Structured data that focus on locale usedfocus on locale used


• Break down into Break down into categories : Script & categories : Script & RegionalRegional


• Application supports Application supports local script and regional local script and regional conventionsconventions


Structure of L18N TestsStructure of L18N Tests

Language/Script

• Collation

• Casing

• Character Set (SBC, DBCS, BIDI)

• Inputs – Keyboard, IME

• Output – Fonts Display/Printing

• Character Conversions

Regional

• Number, Currency, Time, Date, List Seperator, Address Format, Telephone Format

LocaleLocaleLocaleLocale

RegionalRegionalRegionalRegional

Language/ScriptLanguage/ScriptLanguage/ScriptLanguage/Script


Performance TestsPerformance Tests

ObjectiveObjective

•Determine how system Determine how system operates under ‘Load’ operates under ‘Load’ conditionsconditions

ExecutionExecution

• Structured environments Structured environments used to test applicationused to test application

• Relatively easy to Relatively easy to automateautomate


• Application operates Application operates successfully under successfully under stressful load conditionsstressful load conditions

• May identify areas of May identify areas of functionality for functionality for optimization by optimization by Development teamDevelopment team



Performance TestsPerformance Tests

ObjectiveObjective

•Determine how system Determine how system operates under ‘Load’ operates under ‘Load’ conditionsconditions

ExecutionExecution

• Structured environments Structured environments used to test applicationused to test application

• Relatively easy to Relatively easy to automateautomate


• Application operates Application operates successfully under successfully under stressful load conditionsstressful load conditions

• May identify areas of May identify areas of functionality for functionality for optimization by optimization by Development teamDevelopment team


Factors to Consider during Performance Factors to Consider during Performance TestingTesting

Network Capacity

• Concurrent users

• Remote Sessions/Connections

Disk, Memory and CPU Usage

• Run system under minimum and maximum boundary conditions

Transaction Loading

• Continuous transaction loading over time

QA Engineering Managing & Tracking

– Tracking your QA Engineering Projects– Coverage Reports

– By Platforms– By Module,

Component, Unit– By Tester– By Build, Release– By Language

– Bug Tracking– 3 Point Tracking– By Category,

Severity– Find vrs Fix Ratio– Pending Tracking

– Language Quality Reports

– Terminology Consistency Report

– Sampling Reports– Error Density

-

50

100

150

200

250

300

350

400

450

1 2 3 4 5 6 7 8

Open

Closed

Pending

-

50

100

150

200

250

300

350

400

450

1 2 3 4 5 6 7 8

Open

Closed

Pending

QA Engineering Tools to Help you Manage

– Alchemy CATALYST 4.0– RTV Component

– Locates and tracks common localization issues

– Validate Expert– Automates detection and

tracking of localization issues

– SDL ToolProof– Identifies UI Issues

– General Automation Tools– Seque ( Tester)

– SilkTest International– SilkPerformer– SilkRadar

– Rational Systems – Wide selection of Developer

Tools– Code Analysis– WEB Services Analysis

– Mercury Interactive– WinRunner– Test Director 7.x

QA Engineering When to start IQA

Base Product QA CycleBase Product QA CycleBase Product QA CycleBase Product QA Cycle

Localization Project CycleLocalization Project CycleLocalization Project CycleLocalization Project Cycle

IQA CycleIQA CycleIQA CycleIQA Cycle

Base Product DevelopmentBase Product DevelopmentBase Product DevelopmentBase Product Development

DisadvantagesDisadvantages

–Localization & IQA not integral to base developmentLocalization & IQA not integral to base development–Base Product completes before any focus on International Issues

–IQA bugs only addressed after release of base product

–No sharing and coupling of test plans between Base QA and IQA teams

–No International Product Development Planning

–IQA Starts after Localization beginsIQA Starts after Localization begins–Translated product required to start IQA effort

–Any IQA issue that needs to be fixed is eventually addressed post-base produuct release

Traditional Product Development CycleTraditional Product Development Cycle

QA Engineering When to start IQA

Localization Project CycleLocalization Project CycleLocalization Project CycleLocalization Project Cycle

IQA CycleIQA CycleIQA CycleIQA Cycle

Base Product DevelopmentBase Product DevelopmentBase Product DevelopmentBase Product Development

AdvantagesAdvantages

–IQA is an integral part of the PD Lifecycle–Releasing localized products is a Corporate Wide Goal

–Base QA team are aware of and implementing IQA in their Base Product QA plans

–Development fix IQA issues as they develop Base product

–Corporate Goal : Universal English Product

–Localization Cycle is parallel to development effort–Localization – Start as late as possible to finish as early as possible

–Simultaneous Release of language products now feasible!

Ideal Product Development CycleIdeal Product Development Cycle

Product QA and IQA CycleProduct QA and IQA Cycle

Thank you….

After Lunch Challenge

Devise a test plan for the following code segment…Devise a test plan for the following code segment…

integer foo(integer x, integer y, integer z ){

integer Result

let Result = x * y / z

return Result}

the company…. the market leader in globalization technology –pioneers in visual translation...

Documents

engineering testing

quality levels of engineering

quality assurance team

quality performance

possible quality assurance

exploratory phase goal

defined goal

test case