automate thyself

AUTOMATE THYSELF

Luis Majano @lmajano @ortussolutions

WHO AM I?

• Luis Majano• Computer Engineer• El Salvador • CEO of Ortus Solutions• Creator of all things Box• Sandals & Beaches -> ESRI -> Ortus

Story of Why People don’t testTesting Tools

Executing TestsAutomation

Continuous Integration

NOT ABOUT:BDD

Basics of Testing

Where to start?

Test Harness

What to test?What not to test?

WHY PEOPLE DON’T TEST

COMFORT

WHY PEOPLE DON’T TEST

New MethodologyNew Learned

Behavior

BIGGEST LIES IN SOFTWARE

Don’t worry, we will create the tests and

refactor it later!

Testing will double my estimates on time

HOW DO YOU KNOW YOUR FEATURE IS DONE?

Manual testing is only a momentary satisfaction!

HOW IT WORKS?

Bugs cost money

Cost To Fix Time detected Requirements Design Building Testing Post-Release

Time Introduced

Requirements 1x 3x 5-10x 10x 10-100x

Design --- 1x 10x 15x 25-100x

Building -- -- 1x 10x 10-25x

^ Kaner, Cem; James Bach, Bret Pettichord (2001). Lessons Learned in Software Testing: A Context-Driven Approach. Wiley. p. 4. ISBN 0-471-08112-4.

• Just do it!

• You will get dirty

• It can hurt (a little)

• Learned behavior

NO MORE EXCUSES

It will accelerate your development!

Continuous IntegrationContinuous Delivery

Continuous Improvement

AUTOMATION

AGENDA• Proven event-driven ColdFusion Framework

• Follows OO principles and best practices

• Conventions Based

• Non-intrusive framework you code in ColdFusion

• More than MVC => ColdBox Platform

• Software Toolkit

• Remote Framework

• NOT A MAGIC FRAMEWORK, YOU STILL NEED TO CODE!!

cbVue applicationinstall coldbox-samples/cbVue

TestBox Exposed

API Docs

Core

Test Browser

Test Harness

Test Runner

Samples

install testbox —saveDev

install testbox@be —saveDev

Test Harness

Automated test results!

xUnit/BDD Test Bundles

Harness bootstrap

HTML Runner

ANT Runner

Test Bundle CFCs

• Inherits from testbox.system.BaseSpec

RUN TESTS

• Execute bundles via the URL• http://localhost/tests/bundle.cfc

• SOAP Runner• HTTP/REST Runner• ANT Runner• NodeJS Runner• CommandBox Runner• Custom Runner

HOW IT WORKS?TESTBOX REPORTERS

•ANTJunit : Variant of JUnit for Jenkins•Codexwiki : Mediawiki syntax•Console : Server console•Doc : Semantic HTML•Dot : Awesome Dots•JSON•JUnit•Raw : CFML Struct•Simple : A basic HTML reporter•Text : Back to the 80’s, Text•MinText : Great for CommandBox Runners•XML •Tap : A test anything protocol reporter•Min : Minimalistic Report

Developer Automation

HOW IT WORKS?• Execute bundles via the URL

• http://localhost/tests/bundle.cfc?method=runRemote• SOAP Runner• HTTP/REST Runner• ANT Runner• NodeJS Runner• CommandBox Runner• Custom Runner

Bundle CFC Runner

HTML Runner

CommandBox Runner

Update box.json with runner location

testbox run

{ "runner": "http://127.0.0.1:57538/tests/runner.cfm", "directory": "/tests/specs", "recurse": true}

NodeJS Test Runnernpm install -g testbox-runner

create .testbox-runnerrc

1

2

Run it : testbox-runner2

HOW IT WORKS?• Execute bundles via the URL

• http://localhost/tests/bundle.cfc?method=runRemote• SOAP Runner• HTTP/REST Runner• ANT Runner• NodeJS Runner• CommandBox Runner• Custom Runner

ANT TESTING

HOW IT WORKS?DEVELOPER

AUTOMATION

1.CommandBox Watchers2.Grunt Watchers3.Grunt Watchers + testbox-runner4.Grunt Watchers + CommandBox5.ColdBox Elixir

HOW IT WORKS?

Code->Tests->Commit->Pull->Run->Results->Refactor->Repeat->

CONTINUOUS INTEGRATION

HOW IT WORKS?

Code->Tests->Commit->Pull->Run->Results->Refactor->Repeat->

CI SERVERS

• Luis Majano

• Computer Engineer

Jenkins + CommandBoxwget http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war

box server start warpath=jenkins.war

1

2

Jenkins Features

•FREE - Open Source Java application•Not so easy to configure and use•CI = Jenkins Jobs•Thousands of plugins and extensions• Jobs can be scheduled, parallelized, executed over different machines

• Jenkins Slaves•Powerful but convoluted

Travis CI - Sign Up

https://travis-ci.org/

Sign Up

1

2

Travis Features

•FREE for Open Source Projects•Runs distributed VM’s and Container Support•Triggers Build Script via git repository commits (.travis.yml)•Multiple language support•Many integrations and extensions•Many notification types•No ability to schedule/manual builds•Great for open source projects!

Accounts

Activate For Repositories

1

2

Create .travis.yml3

Commit to repository4

Travis CI - Getting Started

CI Script runs on each commit, pull request, branches

TRAVIS + ANTlanguage: javabranches: only: - developmentsudo: requireddist: trusty

before_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.list

install:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- box install- box server start

script: - ant -f tests/build.xml

TRAVIS + COMMANDBOXlanguage: java

sudo: requireddist: trusty

before_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.list

install:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- cd tests- box install- box server start

script:- box testbox run

TRAVIS + MULTIPLE ENGINESlanguage: java

sudo: required

env: matrix: - ENGINE=lucee@4.5 - ENGINE=lucee@5 - ENGINE=adobe@2016 - ENGINE=adobe@11 - ENGINE=adobe@10

before_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.list

install:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- box install

before_script:- box server start cfengine=$ENGINE port=8500

script:- box testbox run

Gitlab

https://gitlab.com/users/sign_in

Sign Up

1

2

or Self-Hosted3

Gitlab Features•All in one tool: CI, repository, docker registry, issues/milestones, etc.• Same as Travis in concept - CI Runner•Docker based•CI Runners can be decoupled to a docker swarm• Idea of CI pipelines

•Pipelines composed of jobs executed in stages• Jobs can have dependencies, artifacts, services, etc•Schedule Pipelines (Cron)• Jobs can track environments

•Great stats + charting

.GITLAB-CI.YMLimage: ortussolutions/commandbox:alpine

stages: - build - test - deploy

build_app: stage: build script: # Install dependencies - box install production=true

run_tests: stage: test only: - master # when: manual, always, on_success, on_failure script: - box server start - box testbox run

Resources• TestBox : ortussolutions.com/products/testbox

• CommandBox: ortussolutions.com/products/commandbox

• Slack: boxteam.herokuapp.com

• CFML Slack: #box-products

• travis-ci.org

• jenkins.io

• gitlab.com

• Twitter

• @lmajano

• @ortussolutions

Q & A