webinar: test engineering and test data management
TRANSCRIPT
V1.0 | 2020-05-11
PREEvision v9.0 Service Pack 10
Webinar: Test Engineering and Test Data Management
2
u Basics - Overview
Test Project Preparation (Test Planning)
Test Specification
Test Implementation
Setting up the Test Project (Test Planning)
Preparing the Test Execution (Test Planning)
Performing the Test Execution
Test Analysis and Reporting
Toolchain: PREEvision - vTESTstudio - CANoe
Agenda
5
Basics - Overview
u Test Project Preparation (Test Planning)
Test Specification
Test Implementation
Setting up the Test Project (Test Planning)
Preparing the Test Execution (Test Planning)
Performing the Test Execution
Test Analysis and Reporting
Toolchain: PREEvision - vTESTstudio - CANoe
Agenda
6
1) Preparation of the Test Project
u Recommended Perspective:
u Test Engineering and Test Management
u Model View:
u Create these artifacts below a Product Line:
u Test container
u Test Data> Test Data Package
> Test Specification
u Test Execution Package> Test Project
u Test Environment Package> Test Environment(s)
> e.g. Test vehicles, test benches, HIL, FMU…
u Test Object Package> Device under Test (ECUs, Subsystems, …)
> Subject under Test (SW-Versions)
Preparation (1/2)
Test Project Preparation (Test Planning)
7
2) Prepare test team
u Create Resources
u Administration section
u Optional: assign test team to the Test Project
u Precondition: Test Project exists> 1. Select a Test Project in the Model View.
> 2. in PE, open the Test Team page.
> 3. Drag the required Resources from the Artifact Picker and drop them into the Tester table.
u Advantage: When creating new Test Execution Increments, Test Tasks are automatically created> One Task created for each assigned Resource
Preparation (2/2)
Test Project Preparation (Test Planning)
8
Basics - Overview
Test Project Preparation (Test Planning)
u Test Specification
Test Implementation
Setting up the Test Project (Test Planning)
Preparing the Test Execution (Test Planning)
Performing the Test Execution
Test Analysis and Reporting
Toolchain: PREEvision - vTESTstudio - CANoe
Agenda
9
u Creating Test Specification artifacts:> Test Headings, Test Information
> Test Items, Test Item Steps (init, run, shutdown)
> Test Conditions (Pre/Post-Conditions)
> Test Specification Library (Test Items, Test Conditions)
u Linking to REQ layer (traceability)
u Column Req/CF Links
u Linking between Test Artifacts
u Linking to Test Specification Library> E.g. Link Test Conditions with Test Items
u Linking to test design demands> E.g. Link between Test Information and Test
Item (column Test Links)
u arranging artifacts
u Change order, change hierarchy
u Add comments
Test Specification Editor
Test Specification
10
Test Item:
u Test Items describes the test focus/test idea
u Attributes:
u Pre/Postcondition, Description…
u Later, on Test Execution level:
u Same verdict for precondition, action,
postcondition
u In case of failed:> No differentiation whether precondition or action
failed…
Test Items vs. Test items steps
Test Specification
Test item steps:
u Devide tests into test steps > ☺ verdict for each test step
u 3 roles available: Init/Run/Shutdown
u Example of a Test Item with Steps:> Init: preparation of the test
> initialization of test platform
> variant coding of device under test
> Run: the actual test…
> Only Run steps relevant for testing of REQ
> Shutdown: Cleaning up
> reset of variant coding
u Traceability: only Run Step linked to REQ
u ☺ evaluate verdict depending on test focus
> Especially important for test automation
u E.g. failed at “Init” step> verdict not relevant for test focus / REQ
> “Run” step can be skipped
11
Tests must be linked with REQ
u Traceability, Test coverage analysis
How to set links:
u In TSE, select test artifact to be linked
u e.g. Test Item
u Drag REQ artifact from Artifact Picker
u Drop it onto a test item / Test Item Step
u Into column Req/CF Links
u How to open Test Specification Editor:
u In Model View, double click on Test Specification artifact
u Or open Test Explorer, category Test Specification> Double click on Product Line
Traceability: Linking Tests with Requirements/Customer Features
Test Specification
12
Basics - Overview
Test Project Preparation (Test Planning)
Test Specification
u Test Implementation
Setting up the Test Project (Test Planning)
Preparing the Test Execution (Test Planning)
Performing the Test Execution
Test Analysis and Reporting
Toolchain: PREEvision - vTESTstudio - CANoe
Agenda
13
u Test Implementation:
u Derived from Test Specification
u Technical refinement of Test Specification
u Example: > Test Specification: „engine running“
> Test Implementation: „CAN signal: IGN_On“
u Specific „implementation“ for:
u Different test platforms> Test vehicles
> Test benches (FMU, HIL)
u Variant specific implementation> ECU variants
> Supplier specific
u Implementation for manual or automatic testing:
u Manual Test Sequence
u Automatic Test Script
Meaning of Test Implementation
Test Implementation
14
Automatically create manual Test Sequences
u By manual TC synthesis:
u On Test Specification artifact
u Context menu ➔ Refactoring ➔ Manual Test
Sequence Synthesis
u Test Headings result into Test Sequences
u New creation or update of existing Manual Test Sequences
Manual Test Sequence Synthesis
Test Implementation
Synthesis
Test SpecificationTest Implementation
15
3 Editors available:
u Minimal View
u Rough Overview only – Test Data Packages
u Test Implementation Editor
u Manual Test Sequence Editor
How to open:
u In Test Explorer ➔ category Test Implementation
Test Implementation Layer Editors
Test Implementation
16
u Overview of implementation of Test Specification
u No Test Steps listed
u Used for creation of…
u A) Manual Test Sequences (for manual tests)
u B) Test Scripts (for automatic tests)
u Alternatively…
u A) use Manual Test Sequence Synthesis
u B) import Test Scripts from Test Design Tool (vTESTstudio)
u Assigning Test Environment
Test Implementation Editor
Test Implementation
17
u Used to refine Manual Test Sequences
u Create Manual Test Cases
u Assigning Test Items to Manual Test Cases> Alternative: Manual Test Sequence Synthesis
u Description of Test Steps: Precondition, Action, Expected Result
u Manual Test Sequence Synthesis:
u Test Step details from Test Specification layer copied to Test Implementation layer> Action, …
u Technical implementation defined in Manual Test Sequence Editor
u Note: Manual Test Sequence Synthesis Update> Update overwrites changes in Action, Expected…
> Use Action Comment, Expected Comment fields for implementation description
Manual Test Sequence Editor
Test Implementation
18
Basics - Overview
Test Project Preparation (Test Planning)
Test Specification
Test Implementation
u Setting up the Test Project (Test Planning)
Preparing the Test Execution (Test Planning)
Performing the Test Execution
Test Analysis and Reporting
Toolchain: PREEvision - vTESTstudio - CANoe
Agenda
19
Associate test data with the Test Project.
u How to open Test Project Editor:
u In Model View ➔ select a Test Execution Package
u Open with -> Test Project Editor> All Test Projects of the Execution Package are listed
u Set Scope of Requirements Package and Test Data
u Assign by Drag/Drop into corresponding columns
u Pre-selection for Artifact Picker tables used in later steps…
u Calculation basis for Analysis results
u Define Start/End Date
Test Project
Setting up the Test Project (Test Planning)
20
Basics - Overview
Test Project Preparation (Test Planning)
Test Specification
Test Implementation
Setting up the Test Project (Test Planning)
u Preparing the Test Execution (Test Planning)
Performing the Test Execution
Test Analysis and Reporting
Toolchain: PREEvision - vTESTstudio - CANoe
Agenda
21
Set up Test Execution Increments
u E.g.: for each SW build one Increment
Open the Test Increment Editor:
u In the Test Explorer ➔ click on the ”Test
Execution” category
u Select a Test Project
u Select Tab “Test Increment Editor”
How to add a Test Increment:
u Context menu => New
u Select Subject Under Test
u Defining the test scope:
u Column “Planned Increment Content”: drag/drop from Artifact Picker> Test Specification scope
Creating Test Execution Increments
Preparing the Test Execution (Test Planning)
22
Precondition: Resources assigned as test team
u In Test Explorer ➔ category Test Execution
u Click on an existing Test Execution Increment
u Inside Category list
u Test Tasks are created as children of a Test Execution Increment
u When creating new Test Execution Increments, Test Tasks are automatically created
u One Task created for each assigned Resource
u “Responsible” automatically assigned to the Resource
> See Test Task Editor
Assigning Test Tasks - Test Tasks Editor
Preparing the Test Execution (Test Planning)
23
2 ways of planning the test execution
u Test Task Editor: Column “Test Implementation to Execute”
u Case a) Test manager assigns tests to task
u Case b) “Picking Method”
u Each tester selects Test Sequences on his/her own> Drag/drop from Artifact Picker
> Artifact Picker offers only unassigned tests
Assigning Tests to be Executed
Preparing the Test Execution (Test Planning)
24
Picking Test Cases (not Test Sequences)
u From Artifact Picker
u Drop into “Test Volume” column
Test Sequences: Complete or in Parts:
u In Test Task Editor, “Test Implementation to Execute” column
u click on the […] button of a Manual Test Sequence
u Completely or in parts
u In parts: Select Test Cases for test execution
Test Volume
Preparing the Test Execution (Test Planning)
25
Basics - Overview
Test Project Preparation (Test Planning)
Test Specification
Test Implementation
Setting up the Test Project (Test Planning)
Preparing the Test Execution (Test Planning)
u Performing the Test Execution
Test Analysis and Reporting
Toolchain: PREEvision - vTESTstudio - CANoe
Agenda
26
How to open Test Execution Editor:
u in Test Explorer ➔ Category “Test Execution”
u Tab “Test Task Editor”
u Click on 2nd column ➔ open “Test Execution
Editor”
u Add Test Step Run for manual test
u Icon in 2nd column
u if not already done…
u Verdicts can only be added to Test Step Runs
u Enter test results into Column “Verdicts”
u Verdicts entered on “lowest level”> A) test steps exist ➔ verdicts on test steps level
> B) no Test steps ➔ verdicts on test case run level
> See greyed / white cells
u Double click opens verdicts drop down list
Test Execution / Setting verdicts
Performing the Test Execution
27
u In the editor, verdicts are aggregated from the lowest to the next level
u Verdicts are liable to a hierarchy
u the lowest verdict is aggregated
u verdict Failed lower than Passed
u ➔Failed “wins”
Verdict Logic
Performing the Test Execution
28
Basics - Overview
Test Project Preparation (Test Planning)
Test Specification
Test Implementation
Setting up the Test Project (Test Planning)
Preparing the Test Execution (Test Planning)
Performing the Test Execution
u Test Analysis and Reporting
Toolchain: PREEvision - vTESTstudio - CANoe
Agenda
29
u coverage of REQ by test items / Test Item Steps
How to open:
u A) main menu ➔ Test ➔ Open Test Analyzer |
Specification Analyzer
u Or B) Select a Requirement Package
u context menu ➔ select Open with | Test Specification
Analyzer
Coverage:
u REQ covered ➔ blue bullet
u REQ not covered ➔ white bullet
u Coverage information is aggregated up to all parent artifacts
u Aggregated Covered ➔ blue bar graph
u Aggregated not covered ➔ number on white
background
Test Specification Analyzer
Test Analysis and Reporting
30
u Coverage of Test Specification by Test Sequences / Test Scripts
How to open:
u A) main menu ➔ Test | Open Test Analyzer |
Implementation Analyzer
u Or B) Select e.g. Test Data artifact ➔ context menu ➔ Open with… | Test Implementation
Analyzer
Test Implementation Analyzer
Test Analysis and Reporting
31
u Test execution coverage
u Generate Report
How to open:
u A) main menu ➔ Test | Open Test Analyzer |
Execution Analyzer | flat or structured.
u Or B) Select a Test Project, open the context menu and select Open with | …
u Test Project Execution Analyzer (flat) or Test Project Execution Analyzer (structured)
Test Project Execution Analyzer
Test Analysis and Reporting
32
Perspective: Cockpit Test Engineering / Test Management
Test Analysis and Reporting
Select Test Project
different calculation
basis
33
Basics - Overview
Test Project Preparation (Test Planning)
Test Specification
Test Implementation
Setting up the Test Project (Test Planning)
Preparing the Test Execution (Test Planning)
Performing the Test Execution
Test Analysis and Reporting
u Toolchain: PREEvision - vTESTstudio - CANoe
Agenda
35
u PREEvision:
u Create Test Specification
u Linking to Requirements
u vTESTstudio:
u Test Implementation Design
u PREEvision:
u Planning> Test Project, Increments, Test Task
u CANoe:
u Test Execution
u PREEvision:
u Analysis, Reporting
Workflow Overview
Toolchain: PREEvision - vTESTstudio - CANoe
36
u In Test Explorer, Category Test Specification
u Select Test Specification to export
u Open tab Test Specification Editor
u Click on toolbar button Export to Test Design Tool…> .vti-tso file
u Currently, only vTESTstudio export format supported
PREEvision: Export Test Items to vTESTstudio
Toolchain: PREEvision - vTESTstudio - CANoe
37
u Create Test Cases
u Import Test Items from PV
u .vti-tso file
u Traceability
u Map vTS Test Cases to PV Test Items
u 2 „exports“ for toolchain:
u 1) for PREEvision:
u Export Traceability Matrix
u .vtc-tso file
u 2) for CANoe:
u Compile Test Cases
u .vtuexe file
vTESTstudio: Actions in vTESTstudio
Toolchain: PREEvision - vTESTstudio - CANoe
38
In Test Implementation Editor
u Toolbar button: Import from Test Design Tool
u Traceability Matrix exported from vTESTstudio(.vtc-tso)
u Added artifacts after import:
u Test Scripts > Automatic Test Case
PREEvision: Import into Test Implementation Editor
Toolchain: PREEvision - vTESTstudio - CANoe
u Optional step
u Toolchain can be used without import of Test Implementation
39
u Import compiled test cases from vTESTstudio
u .vtuexe
u Execute…
u Export CANoe Test Report
u .xml file
Actions in CANoe
Toolchain: PREEvision - vTESTstudio - CANoe
40
u Open Test Task editor
u Select Test Task
u Toolbar button: Import from Test Execution Tool…
u Load Test Protocol(s) ➔Select CANoe report
u CANoe report contains traceability information
u Therefore import process offers to import Test Scripts:> Drop-down List
> Select Plan Test Script + import Test Run
PREEvision: Import CANoe Test Report
Toolchain: PREEvision - vTESTstudio - CANoe
41 © 2019. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.0 | 2020-05-11
Author:Thomas TeegenVector Germany
For more information about Vectorand our products please visit
www.vector.com