release with confidence
TRANSCRIPT
![Page 1: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/1.jpg)
Release with ConfidenceINTEGRATION TEST AUTOMATION AND COVERAGE FOR WEB SERVICE APPLICATIONS
![Page 2: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/2.jpg)
About Me
Currently CTO at a health tech start-up AristaMD Developing in PHP for ~14 years Author of Dialect (advanced PostgreSQL for
Eloquent) https://github.com/darrylkuhn/dialect I like to surf, scuba dive, travel, and read San Diego native The last movie I watched was “What we do in the
Shadows” I occasionally say something at:
https://followingvannevar.wordpress.com/ @darrylkuhn
![Page 3: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/3.jpg)
Ground we’re going to cover
Quick intro to postman (calling web services) Quick intro to Jenkins (build automation) Test automation using postman/Jenkins Generating code coverage reports Some philosophy about test automation
This presentation utilizes Laravel 5 but nothing here is really Laravel specific…
![Page 4: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/4.jpg)
A simple service app
We’re going to demo using a fictitious application called fooblog.com
Exposes a RESTful interface to Authenticate with a simple oAuth layer
Get user data
Manage Blog entries
Source at:https://github.com/darrylkuhn/fooblog
![Page 5: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/5.jpg)
…but before we get started a little survey:
Survey: Who is familiar with the term API?
What about REST or RESTful (who’s going to correct me for using them interchangeably)?
Who’s consumed a web service? Built web services?
Who’s built unit tests? Who’s built integration tests?
Who knows what code coverage is?
Who’s using test automation now?
Who’s ever pushed a change to a production and crossed their fingers?
![Page 6: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/6.jpg)
Postman
API workflow tool (more @ getpostman.com) It’s FREE! Create requests quickly Replay and organize into Collections Switch context quickly with Environments Use JetPacks (a $10 add-on) to test responses with
simple JavaScript Use newman (free) to run tests (built in postman)
on the command line
![Page 7: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/7.jpg)
Postman Interface
Collections give you a simple way to organize your web services, and call them over and over
![Page 8: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/8.jpg)
Postman Interface
Environments provide variables which make it easy to switch from test to production
![Page 9: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/9.jpg)
Postman Interface
Make GET, POST, PUT, DELETE, etc… calls. Easily include JSON, XML, or plain text payloads
![Page 10: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/10.jpg)
Postman Interface
See your responses
![Page 11: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/11.jpg)
Postman Interface
Test your responses with simple JavaScript (using Jetpacks). Set elements in a “tests” array to capture test results.
![Page 12: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/12.jpg)
Let’s make some calls
![Page 13: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/13.jpg)
Jenkins
Build automation tool (more @ http://jenkins-ci.org/) It’s also FREE! Create “Jobs” which are just a series of actions to run in
sequence. Keeps a history of job runs, who ran them, what the result
was. Plugin architecture allows for a rich set of customizations.
Some of the stuff I use: Git Client (build from github source) Junit/CloverPHP (run unit tests and see coverage) Post-Build Script (deploy build artifacts) LDAP Plugin (centralize authentication)
![Page 14: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/14.jpg)
Jenkins Interface
Push button deployment with progress indicator
![Page 15: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/15.jpg)
Jenkins Interface
Dashboard overview of current coverage and test results
![Page 16: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/16.jpg)
Jenkins Interface
Get a history of the jobs you’ve executed. Who, what, when. You get a full change history (if integrated into git) and shell output.
![Page 17: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/17.jpg)
Jenkins Interface
See code coverage (which tests covered which lines of code)
![Page 18: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/18.jpg)
Let’s build a job
![Page 19: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/19.jpg)
Jenkins/Postman Coverage Recipe
1. Create a command to start / stop capturing coverage
2. Add coverage capability to our app3. Create a command to merge newman results
into our PHPUnit results4. Configure Jenkins job to execute the test suite
and capture pass/fail and coverage details
+
![Page 20: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/20.jpg)
But First…
Let’s quickly dive into sebastianbergmann/php-code-coverage
+
![Page 21: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/21.jpg)
php-code-coverage project
Authored by Sebastian Bergmann (PHPUnitanyone?)
Provides several classes that we’ll be using to store and write coverage details including: PHP_CodeCoverage (this is the main class)
PHP_CodeCoverage_Filter (only capture coverage on specific files/directories)
PHP_CodeCoverage_Report_Clover / PHP_CodeCoverage_Report_HTML to output coverage details in different formats
+
![Page 22: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/22.jpg)
Step 1: Start/stop capturing coverage
Web Service calls take place over several PHP life-cycles unlike PHPUnit (which runs in a single master thread)
We need to Identify at the start of the call’s lifecycle that code
execution should be covered
Persist the captured coverage data somewhere until we’re done with all requests
Persistent storage engine: file (you can use anything really – I use redis in the real world)
+
Let’s see some codeapp/Console/Commands/TestCoverage.php
![Page 23: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/23.jpg)
Step 2: Add coverage capability to our app
For Laravel that means adding a small piece of Middleware to HTTP/Kernel.php
1. Check if we should be recording coverage
2. Pull any existing coverage from cache or create a new coverage object
3. Register a shutdown function to save off the coverage details when the process is complete
+
Let’s see some more codeapp/Http/Middleware/Coverage.php
![Page 24: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/24.jpg)
Step 3: Merge results
Load PHPUnit XML Load Postman/Newman JSON Walk the JSON results adding each testsuite &
testcase to the XML result set Write the merged results
+
![Page 25: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/25.jpg)
Step 3: Merge results
Full XSD at: https://windyroad.com.au/dl/Open%20Source/JUnit.xsd
<testsuites><testsuite name="Suite Name" tests="int" assertions="int" failures="int"
errors="int" time="seconds"><testsuite name="Request Name" time="seconds" tests="int"
assertions="int" failures="int"><testcase name="Test Name" time="seconds" />
</testsuite></testsuite>
</testsuites>
General Structure of the output:
"results": [{
"name": "Request Name","totalTime": int (seconds),"tests": {
"Test 1 Name": bool,"Test 2 Name": bool
}}
General Structure of the input:
+
![Page 26: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/26.jpg)
Step 3: Merge results
Load PHPUnit XML Load Postman/Newman JSON Walk the JSON results adding each testsuite &
testcase to the XML result set Write the merged results
+
Code please…app/Console/Commands/MergeTestResults.php
![Page 27: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/27.jpg)
Step 4: Create Jenkins job
Add build action to Turn on coverage collection
Run phpunit
Run newman
Write and merge test and coverage data
+
Let’s take a look under the hood…
![Page 28: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/28.jpg)
Real life Jenkins example Build Script:composer install
./artisan Testing:Coverage collect
vendor/bin/phpunit --log-junit results/phpunit/phpunit.xml -c phpunit.xml
mkdir -p results/newman/
newman -c postman/collection.json -e postman/build.json -o results/newman/build.json --noColor
./artisan Testing:Coverage write
./artisan Testing:MergeResults
Post-Build Script (success):mkdir -p /var/builds/project/ #Make sure the path exists
cp -rpf ../workspace /var/builds/project/release_$BUILD_ID #Save artifacts
chmod -R g+w /var/builds/project/release_$BUILD_ID
chown -R :ops /var/builds/project/release_$BUILD_ID
rm -f /var/www/project #Remove symlink to old build
ln -s /var/builds/project/release_$BUILD_ID /var/www/project #Symlink new build
cd /var/www/project
./artisan migrate --force #Required for production environment
/var/lib/jenkins/jobs/project/sync.sh #Push build out to all servers
![Page 29: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/29.jpg)
Scalability
In our production environment we have: 549 tests across 111 requests On my Sandbox testing takes
~4 min 30 seconds with coverage
~1 min 30 seconds no coverage
Coverage object reaches 3,350,401bytes (3.2MB) Writing coverage output
Coverage XML: ~15 seconds
Coverage HTML: ~10 seconds
![Page 30: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/30.jpg)
Some philosophy about test automation and coverage
What is the purpose of test automation?
I can change code with confidence
![Page 31: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/31.jpg)
Some philosophy about test automation and coverage
Unit Tests v.s. Integration tests
Write unit tests to test your code, unit tests are for developers
Write integration tests to test your application, integration tests are for the business
![Page 32: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/32.jpg)
Some philosophy about test automation and coverage
What does code coverage really get you?
✓ I know where to focus my testing✓ I know when I add lots of new code that isn’t tested
⃠ I know all my code works all the time
![Page 33: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/33.jpg)
Some philosophy about test automation and coverage
How much coverage is “good”?
✓ 100% coverage doesn’t mean 100% perfect code✓ Coverage establishes a baseline to manage to
✓ Worry about the things that matter – go after low hanging fruit
⃠ Don’t chase a number
![Page 34: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/34.jpg)
Q&A
![Page 35: Release with confidence](https://reader031.vdocuments.net/reader031/viewer/2022012405/55a960461a28ab65308b4769/html5/thumbnails/35.jpg)
Thanks!