abstract test case development phil beecher (bcc)

7 September 2010 Abstract Test Cases 1

Abstract Test Case DevelopmentPhil Beecher (BCC)

Edge / Enterprise Conformity


Edge / Enterprise Conformity Abstract Test Cases

• References

• Prerequisites• Define System Under Test• Production Process

• Identify Interfaces• Define Functionality and plan test coverage• Define Tests


Reference Documents

ITU Recommendation X.291 – OSI Conformance Testing Methodology and Framework for Protocol Recommendations for ITU-T Applications – Abstract Test Suite Specification

This recommendation is particularly aimed at developing Abstract Test Suites for testing communications protocols but has relevance for Products (Systems and Devices) which comprise an application and one or more layers of a communications protocol


Prerequisites

Identify System Under Test / Interface(s)

SG System Components

Utility Operational and Enterprise Systems

CIS MDMS

Head End

Managed Integration Layer

OpenADE Server

OpenADR Server

Network

Public

AMI

Residential Customer

Meter

ESI

HAN Devices

DERHAN

C&I Customer Facility

Meter

EMS/Gateway

Loads

DER

Building Control Network

3rd Party Service Provider

OpenADR Server

OpenADE Server

Portal

· Consistent OpenADR,OpenADE,AMI-Ent, SE 2.0 semantics based on IEC CIM

· Different Service Interfaces based on OpenADR,OpenADE,AMI-Ent, SE 2.0


Prerequisites

Define Functionality

Implementation Conformance Statement

• Specific Requirements are defined in an Implementation Conformance Statement (ICS) proforma

• Format is typically a questionnaire

• Each function / feature of the specification(s) should be clearly described with reference to the specific clause(s) in the specification(s)

• All features / functions should be clearly described as mandatory or optional, with dependencies described

• An IUT should have an accompanying ICS: this describes which features / functions of the specification(s) are implemented


Abstract Test Suite Production Process

• Specifications and ICS proforma(s)– Derive conformance requirements which need testing

• Determine test groups– Understand/document test groups required to achieve

conformance requirements

• Develop Test Purposes– Description of tests to provide adequate coverage of

conformance requirements

• Determine Testing Context and Method(s)– Requirements for Upper and Lower Tester and Test

Coordination Procedures (see next slide)


Test Methods Overview

• Testing Context: Single or Multi Party testing?– Single Party Testing Context is used when IUT is required (by

test purpose) to communicate with only one real other system.– Multi Party Testing Context is used when IUT is required to

communicate with multiple other real open systems concurrently.

• Testing Functions LT: Lower TesterLTCF: Lower Tester Control Function UT: Upper TesterTCP: Test Coordination Procedures

7 September 2010 Abstract Test Cases

Testing Functions

• Single Party Testing Context requires:– LT behaves as the peer real open system to the IUT and assigns the

verdict for the test case

– UT behaves as a user of the IUT

– TCP coordinates the LT and the UT

• Multi Party Testing Context requires:– A set of LTs which execute in parallel, each LT behaves as a peer real

open system to the IUT.

– LTCF coordinates the activity of the LTs and UT (if any) and assigns a verdict for the test case

– Optionally, a set of UTs which execute in parallel, each behaving as a user of the IUT

– TCP between each associated LT and UT, among the LTs between the LTs and LTCF among the UTs and between the UTs and LTCF


Test Methods for Single Party Testing

• Local

• Distributed

• Coordinated

• Remote


Single Party Testing – Local Method

• Test events at the LT specified only in terms of X-ASPs and/or P1 to Pn PDUs

• Test events at UT PCO specified in terms of Y-ASPs

• Upper service boundary of the IUT shall be a standardised (hardware) interface which can be used for testing purposes – test suites place no additional requirements on realisation of interface in the SUT

• The specification of the (hardware) upper interface of the IUT defines mapping between relevant ASPs and/or PDUs and their realisation at the interface

• UT is located within the test system

• Requirements for the TCP are specified in the Test Suite but are realised in the test system


Local Test Method

LT – Lower Tester

UT – Upper Tester

IUT – Implementation Under Test

TCP – Test Coordination Procedures

SUT – System Under Test

ASP – Abstract Service Primitives

PCO – Point of Control and Observation

Test System

X-Service-Provider

UT

LT

TCP

SU

TIUT

PCO ASPs

P1 to Pn PDUs

X-ASPs


Single Party Testing – Distributed Method

• The test events at the LT PCO are specified only in terms of X-ASPs and/or P1 to Pn-PDUs

• The test events at the UT PCO are specified in terms of Y-ASPs

• The upper service boundary of the IUT shall be either a human user interface or a standardized programming language interface which can be used for testing purposes; the test suites shall not place any requirements on the realization of the interface in the SUT, additional to those in the standardized programming language interface specification, if applicable

• There shall be a mapping between the relevant ASPs and their realization at the upper interface of the IUT

• The UT is located within the SUT

• The requirements for the TCP shall be specified in the ATSs, although the procedures themselves shall not be

• If the upper interface of the IUT is a human user interface, then the human operator of the SUT fulfils the requirements of the TCP

• If the upper interface is a standardized programming language interface, then the UT is realized in software and the UT and LT together fulfil the requirements of the TCP.


SUT

Distributed Test Method

LT – Lower Tester

UT – Upper Tester





Note: Duplicate UT, TCP for multi-user single party testing.

Test System

X-Service-Provider

LT TCP

IUT

P1 to Pn PDUs

X-ASPs

Y-ASPs

UT


Single Party Testing – Coordinated Method

• The test events at the LT PCO are specified in terms of X-ASPs, and/or (P1 to Pn)-PDUs plus Test Management PDUs (TM-PDUs);

• Y-ASPs are not used in the specification of the ATS; no assumption is made about the existence of an upper service boundary of the IUT;

• the UT is located within the SUT;

• The requirements for the TCP shall be specified in the ATS by means of a standardized Test Management Protocol (TMP) , referenced by the ATS;

• The UT shall be required by the test suite specifier to implement the TMP and achieve the appropriate effects on the IUT;

• Test cases shall be added to the ATS for the purpose of testing that the UT conforms to the requirements of the TMP specification; such test cases do not contribute to the conformance assessment of the IUT.


Coordinated Test Method

LT – Lower Tester

UT – Upper Tester





PDU – Protocol Data Unit

SUTTest System

X-Service-Provider

LT

IUT

P1 to Pn PDUs

X-ASPs

UTTM-PDUs


Single Party Testing – Remote Method

In this test method, provision is made for the case where it is not possible to observe and control the upper service boundary of the IUT. Also in this test method:

• The test events at the LT PCO are specified only in terms of X-ASPs, and/or (P1 to Pn)-PDUs

• Y-ASPs are not used in the specification of the ATS; no assumption is made about the existence of an upper service boundary of the IUT

• Some requirements for TCP may be implied or informally expressed in the ATS but no assumption shall be made regarding their feasibility or realization

• Abstractly the SUT needs to carry out some UT functions to achieve whatever effects of the TCP and whatever control and/or observation of the IUT are implied or informally expressed in the ATS for the given base specification(s); these functions are not specified nor are any assumptions made regarding their feasibility or realization

• The LT should attempt to achieve the implied or informally expressed TCP in accordance with the relevant information in the IXIT(s).


Single Party Testing – Remote Method (2)

Additional notes:

• In order to overcome the lack of specification of behaviour above the IUT, where necessary, the required behaviour of the SUT shall be specified in terms of the X-ASPs or (P1 to Pn)-PDUs which need to be observed by the LT. This form of implicit specification shall be taken to mean “do whatever is necessary within the SUT in order to provoke the required behaviour”.

• It is possible that some of the test cases in the ATS cannot be executed (e.g. transmission of consecutive unacknowledged Data PDUs, etc.).

• With such implicit specification of control of the IUT, in this test method, it is possible to specify control but not observation above the IUT. This is a major difference between this and the other test methods.


Remote Test Method

LT – Lower Tester

UT – Upper Tester





SUTTest System

X-Service-Provider

LT TCP

IUT

P1 to Pn PDUs

X-ASPs

UT


Multi Party Testing Context

• Used when the IUT is required to communicate with multiple other real open systems.

• Only one IUT is tested, but multiple lower testers are used to test it.

• Each Lower tester represents one of the real open systems with which the IUT needs to communicate.

• Each LT communicates with appropriate part of IUT, by observing and controlling ASPs and PDUs

• LTs provide preliminary results but not assign verdict.

• Lower Tester Control Function (LTCF)

– coordinates activity of Lower Testers.

– LTCF assigns verdict to test case

• 0 or more Upper testers may be used.


LT3

General Model for Multi-Party Testing

Lower Tester Control Function

X-Service-Provider

(P) PDUs

X-ASPs

LT2

LT1

TCPs

PCO

PCO

PCO

UT3

IUT

Y-ASPs

UT2

UT1PCO

PCO

PCO


Test Methods Comparison

Add details comparing Test methods….

Meanwhile, see part 11 of x291_04_95