Download - Testing a Media Proxy with … QuickCheck
Testing a Media Proxy with …
QuickCheckThomas Arts
John Hughes
Chalmers/ITU
Joakim JohanssonUlf Wiger
Ericsson
QuickCheck: Properties not Test Cases
• Write general properties of code instead of test cases
• Test in many, many randomly generated cases
prop_reverse() -> ?FORALL(Xs,list(int()), ?FORALL(Ys,list(int()), lists:reverse(Xs++Ys)== lists:reverse(Xs)++lists:reverse(Ys))).
• Simplification is extremely important—separates the signal from the noise!
QuickCheck Testing
3> eqc:quickcheck(test:prop_reverse())............Failed! After 12 tests.[-3,2][-3,1]Shrinking..........(10 times)[0][1]false
A random counter-example: Xs and Ys
NEW: Automatic simplification of failing
cases
Why Did it Fail?
• QuickCheck says: Xs=[0], Ys=[1]
• reverse([0,1])==reverse([0])++reverse([1])?• Xs, Ys the wrong way round
prop_reverse() -> ?FORALL(Xs,list(int()), ?FORALL(Ys,list(int()), lists:reverse(Xs++Ys)== lists:reverse(Xs)++lists:reverse(Ys))).
QuickCheck in a Nutshell
• Features– Properties, not test cases– Controlled random generation– Automated simplification
• Result– Compact, reusable test code– Specification checked against the code– Shortcut from property to bug
Property First Development?
• Code code code… quickcheck
• Fix it… quickcheck
• Fix it… quickcheck… quickcheck quickcheck quickcheck
• Code some more…
A Bug!
A Bug!
Ericsson’s Media Proxy
IP network, servers, etc etc
Media stream
Media Gateway Controller
How hard can it be?
• All we need to do is open and close ”media pinholes” when the controller says…
• Megaco H.248 protocol– ITU standard… 212 pages!– Ericsson Interwork Description… 183 pages!– Control software… 150KLOC
• 20-30K Megaco
Our StrategyQuickCheck
Sends random command sequences
Results make sense?
Completely random commands are all just
rejected—non-sensical!Poor test data
Generating Sensible Messages
MediaDescriptor ::= SEQUENCE{
streams CHOICE{
oneStream StreamParms,multiStream SEQUENCE OF StreamDescriptor
}}
Message Generators
• Example: A media descriptor contains a list of streams– ASN.1 from the standard (simplified):
QuickCheck Generator
mediadescriptor(Streams) when Streams=/=[]-> {mediaDescriptor, #'MediaDescriptor'{ streams = case Streams of [{Id,Mode}] -> oneof([{oneStream,streamParms(Mode)}, {multiStream,[stream(Id,Mode)]}]); _ -> {multiStream, [stream(I,M) || {I,M}<-Streams]} end}}.
stream(I,Mode) -> #'StreamDescriptor'{ streamID = I, streamParms = streamParms(Mode)}.
Message constructionLogic from the IWD
QuickCheck
Records generated by ASN.1 compiler
Conditions in the IWDAdd Request
Desc. Desc. Desc. Properties, Package M Comment
Media M
Stream M Multiple Stream descriptors can be included.
Localcontrol
O LocalControl will be included in all cases except when no media (m-line) is defined in the remote SDP.
mode O The default value of the mode property is “Inactive”, the property is not mandatory if the wanted value is “Inactive”.
StreamParms ::= SEQUENCE{
localControlDescriptor LocalControlDescriptor OPTIONAL,localDescriptor LocalRemoteDescriptor OPTIONAL,remoteDescriptor LocalRemoteDescriptor OPTIONAL,…,statisticsDescriptorStatisticsDescriptor OPTIONAL
}
Remote SDP in here
streamParms(Mode) -> ?LET(RemoteMediaDefined, bool(), if RemoteMediaDefined ->
#'StreamParms'{ localControlDescriptor = localControl(Mode),
localDescriptor = localDescriptor(RemoteMediaDefined),
remoteDescriptor = remoteDescriptor(RemoteMediaDefined)}; true -> …… end).
Two Cases: With and Without Remote Media
Included in this case
Passed on to ensure an m-line is generated
Generating Sensible Commands
Megaco Commands
ContextContext
Megaco Commands
Context
ContextContext
Termination
Add
Megaco Commands
Context
ContextContext
Termination
Termination
Add
Megaco Commands
Context
ContextContext
Termination
TerminationStream
StreamModify
Uses termination ID
Megaco Commands
Context
ContextContext
Termination
TerminationStream
Stream
Subtract
Megaco Commands
Context
ContextContext
Termination
StreamStream
Subtract
Megaco Commands
ContextContext
Sensible Command Sequences?
• Track the state of each test case– What calls in progress, which terminations in each
call…– Simple to model in Erlang
• State machine model– Preconditions say which commands are appropriate– Postconditions check command results– Next state function specifies how commands behave
• QuickCheck library– generates valid sequences, tests, and shrinks them– (developed ”just in time”)
Faults
• Proxy was already well tested
• 6 days of work, mostly writing generators
• 5 faults found:– One in Megaco message encoding/decoding– Four command sequences crashed
Error-provoking Sequences
•
•
•
Add Mod
Add Sub
Add ModAdd
Streams must have two ends
Here today, gone tomorrow…
…with differing numbers of streams
The Best Error!
Add Add Sub Add Sub Add Sub
Due to data corruption here
Shrinking reduced 160 commands to seven!
No one in their right minds would test this!
”Most Likely Bug” Effect
• Once a bug is found, almost every QuickCheck run finds the same one!– (or at least, shrinks to it)
• Add bug preconditions to the model to avoid it and find the next one– Formulate a ”bug hypothesis”– Test the hypothesis!– Document it as a precondition
Real bugs or not?
• Most of these sequences are not sent by Ericsson’s Media Gateway Controller– Controller and Proxy were tested together!
• But the Interwork Description doesn’t say so!
• One of the IWD/the code was buggy!
Using QuickCheck Earlier?
• Testing an older version– Using the same generators and model
• 9 errors found in 6 hours! – (2 TRs for this version)
• Most of the work: writing bug preconditions
QuickCheck is Moreish…
• Follow-up projects under way
• Testing non-Erlang systems (the big one!)
Conclusions
• QuickCheck + Erlang– Simple declarative models of the SUT– Concise, maintainable test code
• Testing: a great application for FP– Performance irrelevant– No need to commit to Erlang in the product
• A winning combination?