outsourcing, subcontracting and cots tor stålhane
DESCRIPTION
Responsibility It is important to bear in mind that The company that brings the product to the marketplace carries full responsibility for the product’s quality. It is only possible to seek redress from the company we outsourced to if we can show that they did not fulfill their contractTRANSCRIPT
![Page 1: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/1.jpg)
Outsourcing, subcontracting and COTS
Tor Stålhane
![Page 2: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/2.jpg)
Contents
We will cover the following topics• Testing as a confidence building activity• Testing and outsourcing• Testing COTS components• Sequential testing• Simple Bayesian methods
![Page 3: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/3.jpg)
Responsibility
It is important to bear in mind that• The company that brings the product to the
marketplace carries full responsibility for the product’s quality.
• It is only possible to seek redress from the company we outsourced to if we can show that they did not fulfill their contract
![Page 4: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/4.jpg)
Testing and confidence
The role of testing during:• Development – find and remove defects.• Acceptance – build confidence in the component
When we use testing for COTS or components where the development has been outsourced or developed by a subcontractor, we want to build confidence.
![Page 5: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/5.jpg)
A product trustworthiness pattern
Product is trustworthy
Trustworthinessdefinition
Product related
Process related
People related
Environment definition
System definition
![Page 6: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/6.jpg)
Means to create product trustBased on the product trust pattern, we see that
we build trust based on • The product itself – e.g. a COTS component• The process – how it was developed and
tested• People – the personnel that developed and
tested the component
![Page 7: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/7.jpg)
A process trustworthiness pattern
Activity is trustworthy
Argument byconsideringprocess
Trustworthinessdefinition
Process definition
Team is competent
Method addressproblem
Process istraceable
![Page 8: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/8.jpg)
Means to create process trust
If we apply the pattern on the previous slide we see that trust in the process stems from three sources:
• Who does it – “Team is competent”• How is it done – “Method addresses problem”• We can check that the process is used
correctly – “Process is traceable”
![Page 9: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/9.jpg)
Testing and outsourcing If we outsource development, testing need to
be an integrated part of the development process. Testing is thus a contract question.
If we apply the trustworthiness pattern, we need to include requirements for
• The component - what• The competence of the personnel – who• The process – how
![Page 10: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/10.jpg)
Outsourcing requirements - 1
When drawing up an outsourcing contract we should include:
• Personnel requirements – the right persons for the job. We need to see the CV for each person.
• Development process – including testing. The trust can come from– A certificate – e.g. ISO 9001– Our own process audits
![Page 11: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/11.jpg)
Outsourcing requirements - 2
Last but not least, we need to see and inspect some important artifacts:
• Project plan – when shall they do what?• Test strategy – how will they test our
component requirements?• Test plan – how will the tests be run?• Test log – what were the results of the tests?
![Page 12: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/12.jpg)
Trust in the component
The trust we have in the component will depend on how satisfied we are with the answers to the questions on the previous slide.
We can, however, also build our trust on earlier experience with the company. The more we trust the company based on earlier experiences, the less rigor we will need in the contract.
![Page 13: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/13.jpg)
Testing COTSWe can test COTS by using e.g. black box testing
or domain partition testing.Experience has shown that we will get the
greatest benefit from our effort by focusing on tests for
• Internal robustness• External robustness
![Page 14: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/14.jpg)
Robustness – 1
There are several ways to categorize these two robustness modes. We will use the following definitions:
• Internal robustness – the ability to handle faults in the component or its environment. Here we will need wrappers, fault injection etc.
• External robustness – the ability to handle faulty input. Here we will only need the component “as is”
![Page 15: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/15.jpg)
Robustness – 2
The importance of the two types of robustness will vary over component types.
• Internal robustness - components that are only visible inside the system border
• External robustness – components that are part of the user interface.
![Page 16: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/16.jpg)
Internal robustness testing
Internal robustness is the ability to• Survive all erroneous situations, e.g.
– Memory faults – both code and data– Failing function calls, including calls to OS
functions• Go to a defined, safe state after having given
the error message• Continued after the erroneous situation with a
minimum loss of information.
![Page 17: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/17.jpg)
Why do we need a wrapperBy using a wrapper, we obtain some important
effects:• We control the component’s input, even
though the component is inserted into the real system.
• We can collect and report input and output from the component.
• We can manipulate the exception handling and effect this component only.
![Page 18: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/18.jpg)
What is a wrapper – 1 A wrapper has two essential characteristics • An implementation that defines the functionality
that we wish to access. This may, or may not be an object (one example of a non-object implementation would be a DLL whose functions we need to access).
• The “wrapper” class that provides an object interface to access the implementation and methods to manage the implementation. The client calls a method on the wrapper which access the implementation as needed to fulfill the request.
![Page 19: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/19.jpg)
What is a wrapper – 2 A wrapper provides interface for, and services to, behavior that is defined elsewhere
![Page 20: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/20.jpg)
Fault injection – 1 On order to test robustness, we need to be able
to modify the component’s code – usually through fault injection.
A fault is an abnormal condition or defect which may lead to a failure.
Fault injection involves the deliberate insertion of faults or errors into a computer system in order to determine its response. The goal is not to recreate the conditions that produced the fault
![Page 21: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/21.jpg)
Fault injection – 2 There are two steps to Fault Injection:• Identify the set of faults that can occur
within an application, module, class, method. E.g. if the application does not use the network then there’s no point in injecting network faults
• Exercise those faults to evaluate how the application responds. Does the application detect the fault, is it isolated and does the application recover?
![Page 22: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/22.jpg)
Examplebyte[] readFile() throws IOException { ... final InputStream is = new FileInputStream(…); ... while((offset < bytes.length) && (numRead = is.read(bytes,offset,(bytes.length-offset))) >=0) offset += numRead; ... is.close(); return bytes;}
What could go wrong with this code?
• new FileInputStream() can throw FileNotFoundException• InputStream.read() can throw IOException and
IndexOutOfBoundsException and can return -1 for end of file
• is.close() can throw IOException
![Page 23: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/23.jpg)
Fault injection – 3
• Change the code– Replace the call to InputStream.read()
with some local instrumented method– Create our own instrumented InputStream
subclass possibly using mock objects– Inject the subclass via IoC (requires some
framework such as PicoContainer or Spring)• Comment out the code and replace with throw new IOException()
![Page 24: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/24.jpg)
Fault injection – 4
Fault injection doesn’t have to be all on or all off. Logic can be coded around injected faults, e.g. for InputStream.read():
• Throw IOException after n bytes are read
• Return -1 (EOF) one byte before the actual EOF occurs
• Sporadically mutate the read bytes
![Page 25: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/25.jpg)
External robustness testing – 1
Error handling must be tested to show that• Wrong input gives an error message• The error message is understandable for the
intended users• Continued after the error with a minimum loss
of information.
![Page 26: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/26.jpg)
External robustness testing – 2
External robustness is the ability to• Survive the input of faulty data – no crash• Give an easy-to-understand error message
that helps the user to correct the error in the input
• Go to a defined state• Continue after the erroneous situation with a
minimum loss of information.
![Page 27: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/27.jpg)
Easy-to-understand message – 1
While all the other characteristics of the external robustness are easy too test, the error message requirement can only be tested by involving the users.
We need to know which info the user needs in order to:
• Correct the faulty input• Carry on with his work from the component’s
current state
![Page 28: Outsourcing, subcontracting and COTS Tor Stålhane](https://reader036.vdocuments.net/reader036/viewer/2022062413/5a4d1b5b7f8b9ab0599ab031/html5/thumbnails/28.jpg)
Easy-to-understand message – 2
The simple way to test the error messages is to have a user to
• Start working on a real task • Insert an error in the input at some point
during this taskWe can then observe how the user tries to get
out of the situation and how satisfied he is with the assistance he get from the component.