steps towards model-based, inference-driven soa testing
DESCRIPTION
This presentation reports the progress of a project whose goal is to develop a model-driven, contract-based SOA testing environment in which grey-box testing strategies for large services architectures are driven by stochastic inference. In order to achieve this goal, a Bayesian network is built directly by mapping and model transformation from the services architecture design and test models. Because services architectures are complex and large systems, the generated BN and its associated probability tables may also be very large and the inference process may be excessively slow. Nevertheless, the structures of the services architecture design and test models let techniques of compilation improve the inference task, enabling the use of BN inference as a sustainable tool for driving failure discovering and troubleshooting strategies in large scale services architectures.TRANSCRIPT
simplengineering Service Oriented Architects
November 30, 2011
Steps towards model-based, inference-driven SOA Testing Ariele P. Maesano¹ ², Fabio De Rosa ², Libero Maesano ², Perre-Henri Wuillemin¹ {Ariele.Maesano, Pierre-Henri.Wuillemin}@lip6.fr {ariele.maesano, fabio.de-rosa, libero.maesano}@simple-eng.com
¹ Laboratoire d’Informatique de Paris 6 (UPMC), Paris, France ² SIMPLE ENGINEERING, 31-35, rue St. Ambroise, 75011 Paris, France
simplengineering Service Oriented Architects
30/11/2011 - 2
Table of contents
1. SOA Functional Testing
2. Black-box and Grey-box Test
3. SOA Testing Environment
4. Model Driven Test Engineering
5. BN Inference Approach
6. Discussion & Future work
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 3
Motivation In the next 20 years, tens, hundreds, thousands, … applications, systems, devices will be connected and will collaborate without human intermediation - that’s the Brian Arthur’s second economy *
Distributed enterprise architectures involving companies, government and for-no-profit organizations allow the automation of business processes that support daily activities
Service Oriented Architecture (SOA) is a design and implementation style that allows organizations to put in practice dynamic collaborations of loosely coupled systems, in order to achieve flexible, dependable and secure business process automation
SOA is the paradigm able to support the second economy
* W. Brian Arthur, "The second economy." McKinsey Quarterly, October 2011
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 4
SOA conceptual framework The collaboration among systems is carried out through the exchange of services which are regulated by contracts
Service contracts are formal models of the service functions and of the provider and consumer interfaces and external behaviors (including security and quality of service aspects) Service contracts do not include any information about internals of the systems that comply with them - system implementations are black boxes, hidden and private A services architecture is a network of participant systems, playing roles bound by service contracts, that collaborate in order to achieve business goals
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 5
SOA testing conceptual framework Validation question: "Are you building the right services architecture?", is translated in the question: "Are you formalizing the right service contracts and participants’ roles (combinations of contract parts)?"
Verification question: "Are you implementing participant systems right, i.e. in compliance with the validated service contracts and participants’ roles?“
A model-driven approach simplifies the validation and verification task:
once the service contract model has been validated, the remainder of the overall validation task is the verification of the correct application of the model mapping and transformation, until the implementation
The last verification question (about compliance between implementation and contract) “stays open”,
SOA Architects and Integrators cannot employ program static check methods or white-box testing
The only available verification methods are black-box testing of individual SUTs (System Under Test) and grey-box testing of interactions among SUTs of the Services Architecture Under Test (SAUT), where they are observable
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 6
Probabilistic Inference Approach
The ongoing research on SOA testing spans three main areas: the automated generation of test cases from contract models
(positive, negative, quality of service…) the automation of the test execution and evaluation environments inference-based methods and tools in order to help Testers to
efficiently manage test campaigns
There are not yet applications of probabilistic inference to the management of SOA testing campaigns it appears to be adequate for discovery of failures and search and
identification of faulty participants in a services architecture, where the only available information is the match/mismatch between the actual external behavior and the expected one
Bayesian Networks for Services Architecture Testing (BN4SAT) project(*) is centered on the development of a Bayesian network (BN) inference approach and technology to services architecture testing management
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
(*) The project is conducted in cooperation between the Laboratoire d'Informatique de Paris VI (LIP6 - Université Pierre et Marie Curie - Paris VI - France), the Centre National de la Recherche Scientifique (CNRS - France) and Simple Engineering, a European group specialized in the design of services architectures. The project is partially funded by the Association Nationale de la Recherche et de la Technologie (ANRT - France)
simplengineering Service Oriented Architects
30/11/2011 - 7
Black-box Test
Black-box tests are run against individual SUTs whose internals are hidden
Connections (channels) with other participants are not observed
Only the SUT responses to stimuli are observed
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
sd TestSuite_Wiring_003
usbg
:BankGateNode
usam
:AccountMngtNode
CheckBalanceResponder/inquiry(m-usbg-usam-001)
CheckBalanceRequester/inform(m-usam-usbg-002)
WithdrawalResponder/invoke(m-usbg-usam-003)
WithdrawalRequester/report(m-usam-usbg-004)
CheckBalanceResponder/inquiry(m-usbg-usam-005)
CheckBalanceRequester/inform(m-usam-usbg-006)
simplengineering Service Oriented Architects
30/11/2011 - 8
Grey-box Test
Grey-box tests are run against a SAUT (network of SUTs)
SUTs are involved in an end-to-end interaction
Channels among SUTs are observed
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
sd TestSuite_Wiring_003
usatm
:ATM_Node
usbg
:BankGateNode
usam
:AccountMngtNode
FindAccountResponder/inquiry(m-usatm-usbg-001)
FindAccountRequester/inform(m-usbg-usatm-002)
CheckBalanceResponder/inquiry(m-usatm-usbg-003)
CheckBalanceResponder/inquiry(m-usbg-usam-001)
CheckBalanceRequester/inform(m-usam-usbg-002)
CheckBalanceRequester/inform(m-usbg-usatm-004)
CheckCredentialsResponder/invoke(m-usatm-usbg-005)
CheckCredentialsRequester/report(m-usbg-usatm-006)
WithdrawalResponder/invoke(m-usatm-usbg-007)
WithdrawalResponder/invoke(m-usbg-usam-003)
WithdrawalRequester/report(m-usam-usbg-004)WithdrawalRequester/report(m-usbg-usatm-008)
CheckBalanceResponder/inquiry(m-usatm-usbg-009)
CheckBalanceResponder/inquiry(m-usbg-usam-005)
CheckBalanceRequester/inform(m-usam-usbg-006)
CheckBalanceRequester/inform(m-usbg-usatm-010)
simplengineering Service Oriented Architects
30/11/2011 - 9
SOA testing environment
Testing and Test Control Notation (V.3)* environment
Arbiter, Scheduler, Test components as TTCN-3 scripts
Engine as a plug-in (or a “service”)
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
* http://www.ttcn-3.org/
simplengineering Service Oriented Architects
30/11/2011 - 10
TTCN-3
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
deployment BankNet DeploymentTokens_1
usatm4ushwcinterceptor :
ATM4HW_Control
usatm4usbginterceptor
:ATM4BankGate
«SUT»
usatm :ATM_Node
usatm4ushwcinterceptor :
ATM4HW_Control
usatm4usbginterceptor
:ATM4BankGate
usbg4usatminterceptor :
BankGate4ATMusbg4usaminterceptor :
BankGate4AccountMngt
«SUT»
usbg :BankGateNode
usbg4usatminterceptor :
BankGate4ATMusbg4usaminterceptor :
BankGate4AccountMngt
usam4usbginterceptor :
AccountMngt4BankGate«SUT»
usam :AccountMngtNode
usam4usbginterceptor :
AccountMngt4BankGate
usbginterceptor4usam:
BankGate4AccountMngt
usbginterceptor4usatm:
BankGate4ATM
ushwcstimulator:
HW_Control4ATM
usaminterceptor4usbg:
AccountMngt4BankGate
usatminterceptor4usbg:
ATM4BankGate
a4e :
Arbiter4Engine
s4e :
Scheduler4Engine
«executionEnvironment»
:TTWorkbench
usbginterceptor4usam:
BankGate4AccountMngt
usbginterceptor4usatm:
BankGate4ATM
ushwcstimulator:
HW_Control4ATM
usaminterceptor4usbg:
AccountMngt4BankGate
usatminterceptor4usbg:
ATM4BankGate
a4e :
Arbiter4Engine
s4e :
Scheduler4Engine
«TestComponent»
usaminterceptor :
AccountMngtInterceptorNode
«TestComponent»
usatmstimulator :
ATM_StimulatorNode
«TestComponent»
usbginterceptor :
BankGateInterceptorNode
arbiter :Arbiter
scheduler :Scheduler
e4a :
Engine4Arbiter
e4s :
Engine4Scheduler
:Engine
e4a :
Engine4Arbiter
e4s :
Engine4SchedulerAutomated test execution & evaluation environment
simplengineering Service Oriented Architects
30/11/2011 - 11
SOA Testing in WS* platform
SAUT representation = extended UDDI V3 Data Structure + collection of WSDLs
UML Interaction (XMI) + collection of SOAP messages
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 12
BN Inference Approach
Bayesian network inference is used as a failure discovering driver, by getting the next test case to run in
order to reveal a failure. As troubleshooting driver, by getting the next test to run in order to
locate faulty participants
BN4SAT engine integrates failure discovering and troubleshooting, by managing a well-adjusted dynamic test sequence (the strategy) that, with a minimum number of test case runs: reveals a maximum number of failures and locates a maximum number of faulty participants
The BN4SAT graph is built by the BN4SAT loader directly from: the SAUT deployment UDDI files the WSDL files for all the involved service contracts and the test suite files
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 13
BN4SAT Metamodel The BN4SAT graph model has six Boolean stochastic variable
types: Entity - business organization, {notFaulty, faulty} Participant - participant system, {notFaulty, faulty} Port - participant required interface, {notFaulty, faulty} ActionType - type of communicative action (message, rpc, rpc reply), {notFaulty, faulty} Action (evidence): test case action {pass, fail} Transaction: test case {pass, fail}
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 14
Discussion A BN for SOA testing can exhibit a very large size, and can be very dense, therefore with large conditional probability tables (inefficiency in time and space)
We have implemented and assessed different classical inference methods such as value elimination, lazy propagation, Shafer-Shenoy and proved that they are not appropriate to cope in a sustainable way with BN4SAT graphs
Inference by compilation: Bayesian network can be interpreted as a multi-linear function(MLF) and therefore implemented by an arithmetic circuit (AC)
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
SAUT + test suite specification
Bayesian network
Coarse Conjunctive Normal Form (Coarse CNF)
Refined Conjunctive Normal Form (Refined CNF)
deterministic Disjunctive Normal Negational Form
(d-DNNF)
Arithmetic Circuit
SAUT + test suite specification
Refined Conjunctive Normal Form (Refined
CNF)
deterministic Disjunctive Normal Negational Form
(d-DNNF)
Arithmetic Circuit
Darwiche “classical” method
BN4SAT method (Model-driven inference by compilation)
- No intermediary generation of BN - No CNF optimization step
- Refined CNF in linear time (experimental results) - Push the boundaries with size limitation
simplengineering Service Oriented Architects
30/11/2011 - 15
Thanks
Questions ?
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 16
SOA testing environment
Services Architecture Under Test - SAUT is a network of SUTs (System Under Test) connected by channels conveying communicative actions such as messages, remote procedure calls (rpc), rpc replies Channels may be observable (grey-box stance). Conversely, SUTs internals are never observable (black-box stance)
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
deployment BankNet Deployment
usatm4ushwcinterceptor :
ATM4HW_Control
atm4bg :ATM4BankGate«SUT»
usatm :ATM_Node
usatm4ushwcinterceptor :
ATM4HW_Control
atm4bg :ATM4BankGate
am4bg :AccountMngt4BankGate
«SUT»
usam :AccountMngtNode
am4bg :AccountMngt4BankGate
bg4am :BankGate4AccountMngt
bg4atm :BankGate4ATM
«SUT»
usbg :BankGateNode
bg4am :BankGate4AccountMngt
bg4atm :BankGate4ATM
hwc4atm :HW_Control4ATM
«SUT»
ushwc :HW_ControlNode
hwc4atm :HW_Control4ATM
BankNet SAUT
simplengineering Service Oriented Architects
30/11/2011 - 17
SOA Test Running
Each SUT, in a well established SAUT, is initialized with a text context (the SAUT state), that is coherent with the next selected test suite (i.e., a collection of test cases) is going to run A test run takes place when Tester submits one or more stimuli to one or more SUTs Tester compares the SAUT actual response - the exchange of actions between SUT's following the stimuli - with the expected response - the expected action exchange in a specified SAUT state A test matches (mismatches) if the actual response corresponds to (differs from) the expected one Test run has a verdict: pass - the test matches and there is no evidence of failure; fail - there is evidence of a failure; inconclusive - a decision cannot be reached; error - a run-time error has occurred in the test environment
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 18
SOA Testing Environment Test Components: specialized components allow reaping SUTs actions, addressing actions to SUTs, comparing SUTs actions with the expected ones, formulating local verdicts and transmitting them to the Arbiter: Stimulators - send stimuli to one or more SUT, in order to trigger
test runs Interceptors - sit transparently between two connected
participants, intercept actions and perform different tasks Emulators - emulate the behavior of SUTs
Arbiter - the Arbiter collects the Test Components local test verdicts and produces final test verdicts Scheduler - the Scheduler drives the execution of the test runs Engine, which is notified by the Arbiter with test verdicts, manages the Scheduler by handling the test strategy on the basis of probabilistic inference on the acquired test verdicts
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 19
Model-driven Approach
The starting point for testing a SAUT is in the Platform Independent Model (PIM), specifically, the collection of samples A sample is a modeled, schema-compliant occurrence of services interactions, that coordinates either the services deliveries (positive sample), or the services refusals, when the delivery conditions are not satisfied (negative sample) A sample is composed of a: sample interaction, a UML Interaction together with its
MessageTypes instances, and a sample context, a collection of domain objects (instances)
representing the states of the participants' resources in which the interactions take place
The sample collection is the starting point for the design and the implementation of test cases
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 20
Test Engineering - Negative sample
CUSTOMER
customer#
ABCDEFG
CUSTOMER-NAME
customer# name
ABCDEFG Ariele
ABCDEFG Paolo
CUSTOMER-ACCOUNT
customer# account#
ABCDEFG 123456
ACCOUNT
account# currency state balance
123456 EUR active 900.00
Sample context (fact base)
sd TestSuite_Wiring_003
usbg
:BankGateNode
usam
:AccountMngtNode
WithdrawalResponder/invoke(m-usbg-usam-003)
WithdrawalRequester/decline(m-usam-usbg-005)
class Sample oracle
«MessageType»
m-usbg-usam-003 :WithdrawalInv okeMessage
acc# = 123456
am = 1000,00
curr = USDollar
dateTime = 2011-06-15T11:28:38.100+01:00
«MessageType»
m-usam-usbg-005 :WithdrawalDeclineMessage
acc# = 123456
am = 1000,00
curr = USDollar
dateTime = 2011-06-15T11:28:14.109+01:00
code = NotEnoughBalance
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 21
SOA Testing in WS* platform
The SAUT is translated into a set of WSDL files and an extended UDDI V3 Data Structure, in order to capture use links (the observable channels in a specific deployment scenario) - each participant system is represented as UDDI Business Service Sample collection is translated in a test suite (a collection of test cases), by the mapping and mechanical transformation of the PIM into the WS* Interoperability PSM A test case includes: a test case oracle - a XML infoset that represents the interchange of SOAP messages
between SUTs (resulting from the transformation of the sample UML Interaction), together with the collection of the corresponding SOAP envelopes (resulting from the transformation of the Message Type instances exchanged in the interaction)
a test case fact base (test context) - a relational database resulting from the object/relation transformation of the sample context
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 22
SAUT Description
<?xml version="1.0" encoding="UTF-8"?>
<uddi:businessService businessKey="urn:My_US_BankNet" serviceKey="urn:My_US_BankNet:BankGate">
<uddi:name>BankGate</uddi:name>
<uddi:bindingTemplates>
<uddi:bindingTemplate bindingKey="urn:BankGate-AccountMngt:BankGate4AccountMngt:usbg4usam"
serviceKey="urn:My_US_BankNet:BankGate">
<uddi:description>urn:BankGateNode:usbg</uddi:description>
<uddi:accessPoint useType="endpoint">http://usbg.My_US_BankNet.com/BankGate4AccountMngt/usbg4usam</uddi:accessPoint>
<uddi:tModelInstanceDetails>
<uddi:tModelInstanceInfo tModelKey="urn:BankGate-AccountMngt">
<uddi:instanceDetails>
<uddi:overviewDoc>
<uddi:description>required</uddi:description>
<uddi:description>urn:IAccountMngt</uddi:description>
<uddi:description>urn:AccountMngtNode:usam:BankGate-AccountMngt:AccountMngt4BankGate:usam4usbg</uddi:description>
</uddi:overviewDoc>
</uddi:instanceDetails>
</uddi:tModelInstanceInfo>
</uddi:tModelInstanceDetails>
</uddi:bindingTemplate>
...
</uddi:businessService>
deployment BankNet Deployment
am4bg :AccountMngt4BankGate «SUT»
usam :AccountMngtNode
am4bg :AccountMngt4BankGate
bg4am :BankGate4AccountMngtbg4atm :BankGate4ATM
«SUT»
usbg :BankGateNodebg4am :BankGate4AccountMngtbg4atm :BankGate4ATM
use links
UDDI v3 Business Service
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 23
Test Case Oracle - Interaction sd TestSuite_Wiring_002
usbg
:BankGateNode
usam
:AccountMngtNode
WithdrawalResponder/invoke(m-usbg-usam-003)
WithdrawalRequester/report(m-usam-usbg-004)
<?xml version="1.0" encoding="UTF-8"?>
<ss20tc:test-case xmlns:ss20tc="urn:simpleSOAD2.0:test-case" id="urn:InterExchangeNet:Scenario01:Wiring_001">
<ss20tc:sequence>
...
<ss20tc:message id="m-usbg-usam-003" sn="m033" from="urn:BankGateNode:usbg"
to="urn:AccountMngtNode:usam" mode="WithdrawalResponder/invoke"/>
<ss20tc:message id="m-usam-usbg-004" sn="m034" from="urn:AccountMngtNode:usam"
to="urn:BankGateNode:usbg" mode="WithdrawalRequester/report"/>
...
</ss20tc:sequence>
</ss20tc:test-case>
XML Infoset describing interaction among SUT
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing
simplengineering Service Oriented Architects
30/11/2011 - 24
Test Case Oracle - Envelope MessageType Instance
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:ns="urn:ien:bank:services:withdrawal:wsdl:types"
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
<soapenv:Header>
<wsa:MessageID>m-usbg-usam-003</wsa:MessageID>
</soapenv:Header>
<soapenv:Body>
<ns:WithdrawalInvokeMsg xmlns:ns="urn:ien:bank:services:withdrawal:wsdl:types">
<ns:sessionId>
<ns:account>123456</ns:account>
<ns:dateTime>2011-06-15T11:28:38.100+01:00</ns:dateTime>
</ns:sessionId>
<ns:currency>USDollar</ns:currency>
<ns:amount>10000,00</ns:amount>
</ns:WithdrawalInvokeMsg>
</soapenv:Body>
</soapenv:Envelope>
SOAP Message
class Sample oracle
«MessageType»
m-usbg-usam-003 :WithdrawalInvokeMessage
acc# = 123456
am = 1000,00
curr = USDollar
dateTime = 2011-06-15T11:28:38.100+01:00
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing