supervisor: prof. jyri hämäläinen instructor: jari simolin (m.sc), nokia siemens networks espoo,...
TRANSCRIPT
Supervisor: Prof. Jyri Hämäläinen
Instructor: Jari Simolin (M.Sc),
Nokia Siemens Networks
Espoo,15.12.2009
Jyri Ilama
Testing terminology Test automation process & Benefits of TA UMTS architecture IPA2800 platform
◦ Call Management domain Continuous Integration The CI project of Call Management
◦ CI FRT pilot project Lessons learned
◦ What went wrong? Results and analysis
2
(our) Functional testing”Using” the ”system”• Actually, a script uses a sub system of an individual
entity (one, separate network element)• The actual, acceptance / system testing is done later
in a System Verification phase
Regression testing◦ Testing that the software still works after changes
are made in some unit
3
4
How can this be achieved? What’s inside the black boxes?
•Pitfalls to avoid•Good practices
The required effort for selecting a set of test cases and executing them takes a few minutes
More tests can be executed more often Some cases are even impossible to perform
by manual testing Use of resources Repeatability Reusability Simply: Saving resources and time!
5
6
We are interested in RNC’s and MGW’S
Network elements are complex networks of interconnected computer units, communicating via message connections
Network elements have very strict requirements on fault tolerance◦ ”the five nines availability performance”
3G network element platformbased on DX200, running in RNC’sand MGW’s◦ But much more efficient and clearer
of its architecture
7
Responsible of all call related operations◦ Setup, release, and connecting of calls◦ Capacity and performance◦ Remaining calls in case of recovery actions
E.g. Unit failures
8
A practice of software development, where the members of a team integrate their work very frequently, usually at least daily
Whenever an integration is made, it is verified by an automated build to detect possible integration errors as quickly as possible
A complex system that involves lots of people, servers, tools and connections working tightly together
9
10
A computer that runs some CI software and executes integration builds whenever software changes are made
Tools◦ CruiseControl & Hudson
Java-based free software, e.g. Ant, Maven and Rake builds supported
Modularity: plug-in support
11
Started in May 2007: only PRB compilation builds in the beginning, unit tests added later
No decent FRT software available◦ Hipla: an output report of even hundreds of
thousands of lines of plain text, not so nice Summer of 2008: HiBot was designed for
running old (Hipla) test cases in a new way◦ HIT2Python translator, all old Hipla code -> HiBot◦ Telnet functions implemented manually, in a
hurry, by a person who left the company◦ This was the point where this tool was left in the
hands of a summer trainee; me
12
I got HiBot up and running quickly, against all estimates
Started implementing and/or debugging all the functionalities
Lots of occasional problems with the Telnet connections
13
MGW test cases of Call Resource Handling Other pilots by other teams in Call
Management◦ Even more work with correcting all the commands◦ Still random crashes and annoying connection
and prompt detection problems External problems
◦ IPA Reporting Server◦ Other tools, such as CruiseControl and SVN
14
Test automation problems leading to jams and crashes◦ The old, manual test cases had to be automated
properly June 2009: The "Telnet guy" came back
◦ The problems were found and defects corrected◦ Time for me to focus on the essential: to find and
fix defects in IPA2800 platform◦ I implemented a very useful script
15
A very useful script for investigating results
16
Old manual cases are the most difficult ones
Changed outputs of functions: scanning strings◦ If the printout changes a little, does your macro
still work? The principle of modularity
◦ No hard-coded values◦ Reusability of test cases
a Forever-loop is definitely not your friend
17
Consecutive execution◦ A test case should be possible to run in any kind
of situation, no matter if there are e.g. other calls already, hanging, or whatever
◦ Proper teardowns At the end of test suites, so the execution of a new
suite can be started from scratch
Execution order◦ Problematic suites: Which first?
Possible system breakers as last ones
18
New software: Focus on the essential◦ Critical features/functionality
R&D&T should be done instead of R&D◦ And regression testing should be as important as
the testing of new features Scrum mode: we’re too deep inside it. Tasks
that are not backlog items won’t get enough priority, at least easily◦ Needed: RT sprints or a team dedicated to RT!
Close all external problems out of sight Write totally new test cases if possible
19
A stabile, working CI environment for our FRT◦ Real defects are found all the time
Also Backlog Item Testing can be done efficiently The "output2report" script was found very
useful Good lessons about test automation
◦ As well as writing modular, intelligent test cases
A very good quality level of the product release!
20
21