![Page 1: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/1.jpg)
Testing Asynchronous APIs With QuickCheck
Thomas Arts Quviq AB
![Page 2: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/2.jpg)
QuickCheck in a Nutshell
Properties Test case Test case Test case Test case Test case
Minimal Test case
2
![Page 3: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/3.jpg)
Property-Based Testing
Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component Testing, System Testing Less work, better testing, more fun
3
![Page 4: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/4.jpg)
From Unit test to Property
Most developers agree that writing unit tests is useful
…. but also quickly gets boring …
An example:
4
API take reset
![Page 5: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/5.jpg)
QuickCheck
API under test
A minimal failing example
5
![Page 6: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/6.jpg)
ok = 1 = 2 = 3 = ok = 1 =
A Unit Test in Erlang
test_dispenser() ->
Expected results
reset(), take(), take(), take(), reset(), take().
6
![Page 7: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/7.jpg)
State machine model
The generator for testing a sequence of commands is a state machine specification The property is that a run of the generated sequence satisfies all postconditions.
7
![Page 8: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/8.jpg)
Modelling the dispenser
reset take take take
1 2 3
ok 1 2 3
8
![Page 9: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/9.jpg)
State Machine for arbitrary sequence
initial_state() -> undefined.
reset_args(_State) -> [].
reset() -> get("http://localhost:4000/reset").
reset_next(_State, _Result, []) -> 1.
take_pre(State) -> State /= undefined.
take_args(_State) -> [].
take() -> get("http://localhost:4000/take").
take_next(State, _Result, []) -> State + 1.
take_post(State, [], Result) -> eq(Result, State + 1).
reset take
Live
DEMO
9
![Page 10: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/10.jpg)
A Parallel Unit Test
• Three possible correct outcomes!
reset
take
take
take
1 2
31 3
22 3
1
ok
1 2
1
10
![Page 11: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/11.jpg)
Another Parallel Test
30 possible correct outcomes!
reset
take
take
take
take
reset
11
![Page 12: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/12.jpg)
Finding race-conditions
Writing unit tests for concurrent events: Headache! Thus, people don’t! QuickCheck does it for you!
12
![Page 13: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/13.jpg)
Arbitrarysequencesprop_dispenser() -> ?FORALL(Cmds, parallel_commands(?MODULE), begin
{H, S, Res} = run_parallel_commands(?MODULE, Cmds),pretty_commands(?MODULE, Cmds, {H, S, Res}, Res == ok)
end).
Live
DEMO
15
![Page 14: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/14.jpg)
Concurrency errors
QuickCheck properties: Property specifies behaviour of any command sequence QuickCheck • runs the sequences with different threads • collect the results • checks whether this can be explained from
sequential behaviour
16
![Page 15: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/15.jpg)
Abstraction
Commonly asked question:
Do we need to re-implement the software as a model?
17
No... not for real systems
![Page 16: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/16.jpg)
A complete system
How to test project-fifo?
18
project-fifo.net
![Page 17: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/17.jpg)
A complete system
Project-fifo • Cloud Orchestration • Manage private and public clouds • Based on SmartOS / Solaris Containers • OSS and Commercial Support
• Self hosted, Distributed, Highly available, Eventually consistent
19
![Page 18: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/18.jpg)
Project-fifo
Architecture
20
Mgmt
.
AAA
Agent Agent Agent
API HTTP / REST API
AAA: OAuth2, RBAC
Business logic, database, tracking
Agents to manage physical components
60,000 lines of code
![Page 19: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/19.jpg)
Lets test
Mgmt.
AAA
Agent Agent Agent
API Create
The
prob
lem
21
![Page 20: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/20.jpg)
Lets test
Mgmt.
AAA
Agent Agent Agent
API Create Login
The
prob
lem
22
![Page 21: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/21.jpg)
Lets test
Mgmt.
AAA
Agent Agent Agent
API Create Login GET fail
The
prob
lem
23
This fails, create is asynchronous
![Page 22: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/22.jpg)
Lets test
Mgmt.
AAA
Agent Agent Agent
API Create Login 3m GET OK
The
prob
lem
24
![Page 23: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/23.jpg)
Lets test
Mgmt.
AAA
Agent Agent Agent
API Create Login 3m GET OK
The
prob
lem
25
• utterly slow testing • hard to test around "just started" • forget race condition testing
![Page 24: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/24.jpg)
Asynchronous API
Create Login stop
26
delete
running
deleted
stopped
start
States of one specific VM
waiting
IDEA: Perform actions on other VMs while waiting !
![Page 25: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/25.jpg)
Asynchronous API
Create Login stop
27
delete
running
deleted
stopped
start
polling
Each operation is waiting before it makes sense to execute and add:
sleep
![Page 26: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/26.jpg)
QuickCheck
Running QuickCheck tests revealed 25 errors... all fixed now J Timing errors, race conditions, type errors, incorrect use of library API, error in documentation, errors in the logic, system limits error, errors in fault handling... and coincidently a hardware error
28
![Page 27: Testing Asynchronous APIs With QuickCheck · Property-Based Testing Instead of writing test cases.... they are automatically generated from properties Useful for Unit Testing, Component](https://reader030.vdocuments.net/reader030/viewer/2022041019/5ece2c97ee11c142a623db40/html5/thumbnails/27.jpg)
A complete system
60,000 lines of code 460 lines of QuickCheck Any reasonable test suite would contain more lines of code... ... and find less errors. 30