Automating your testing with Functionize

ALP test creation

Functionize delivers the first autonomous, AI cloud-based testing platform. This guide

will help you get started. We’ll show you how to create tests, how to run tests across all

major browsers including mobile devices (beta) and how to view the results of the tests.

We’ll also explain how Functionize ensures that tests are self-maintaining and explain how

things like 1-click updates work. Finally, we’ll briefly touch on API and load testing with

Functionize.

Right from the start, Functionize makes life easy for you. Our Adaptive Language

Processing (ALP) test creation engine is powered by NLP (Natural Language Processing).

That means you can simply create your test in plain English , upload it to our system and

your test will be created for you. The system understands commands based on keywords

(“SCROLL down the page and VERIFY that the buy now button is shown”) as well as

natural language (“The top menu bar should be the full width of the page”).

A brief guide for first-time users

Learn more at:www.functionize.com

Example ALP Test Creation

After the test has been generated, you have the chance to approve it or tweak it step by step.

Test approvals process

Learn more at:www.functionize.com

Test execution

Test Maintenance

The Functionize platform makes it extremely easy to execute tests. You can run individual

tests or create orchestrations which run suites of tests, and you can test across

multiple browsers and hardware architectures. We partner with Google to provide the

Functionize Test Cloud which allows you to test your site from any geographic location

and with simulated bandwidth restrictions – essential if you expect your users to come

from other countries or to use bandwidth-limited connections.

As the tests run you can see their progress on your dashboard. Once the test has

completed you can quickly see if any problems were found. The system also takes a

screenshot before, during and after each and every step in the test, which you can

scroll through. This will highlight any problems it found to save you having to search for

ages, hence helping with fault diagnosis. The visual comparison is done using template

recognition, an AI-powered “fuzzy comparison” which won’t trigger a failure just because

of a simple CSS style change or rendering artifact.

One of the biggest time sinks with automated testing is test maintenance. Traditional

Selenium-based systems are dumb and 1-dimensional, being limited to a single selector at

a time. The slightest change to a style sheet or a simple change to the page layout breaks

the script. Our tests are completely self-healing.

The ALP-generated script being run

Learn more at:www.functionize.com

Each action is fingerprinted by the system, which uses machine learning to establish what

the action is actually doing. So if you restyle your “Buy Now” button as “Add to Cart” the

system will cope since it can tell the underlying functionality is the same. Likewise, if you

move the button from the bottom left of the page to the bottom right, the system still knows

it’s the same button.

Some test problems are harder to find though. These are typically cases where an action

completes without error, you move on several steps in the test, and then the problem

appears. Analyzing such failures is one of the hardest things in test maintenance. For

instance, imagine your test places an item in the cart, then at a later stage checks if the

correct item is in the cart. Imagine that your page has 2 Add to Cart buttons. One for the

product being viewed and one for a special offer at the right of the page. If you redesign the

page so that the main button changes to Add Now, the system may accidentally select the

special offer instead. Now the test will happily proceed, but the failure will become apparent

when it finds the wrong item in the cart.

Functionize’s Adaptive Event Analysis (AEA™) engine includes a Root Cause Analysis

feature which is able to deal with this situation. By learning from previous tests, RCA can

track-back, find the most likely problems, test the solutions and then present you with the

best solutions ranked by probability.

If you click show suggestion, it will take you to a screenshot from the test (see below)

Learn more at:www.functionize.com

As you see, this page has 2 buttons to add an item which the system has highlighted. Simply by clicking the

correct one, the system will repair the test. Now any tests that rely on this step will also be repaired.

The Functionize CLI

We know that many of you are more at home using the command line than a GUI! Which

is why we have developed the Functionize CLI . We’ve built it on top of Java to allow

portability across OSes. Simply go to your account, download the CLI zip file to your

computer, extract it and you’re good to go. The file will already have your API credentials

embedded so there’s no need for any setup. It also includes inline help so if you pass the

wrong variables it will tell you. Here are a couple of cool things it can do.

Often you want to test a feature before you push it to staging. One of the coolest features

in our CLI is the ability to test against a local server. Spin up a local instance of your site

with the changes you want. Put it on a suitable port and then tell the CLI to use that port

for testing. Simply use the start-tunnel command and the CLI will create a secure tunnel:

Testing a local server

Learn more at:www.functionize.com

functionize start-tunnel 5005

Now when you run a test, Functionize will run the tests against your local server over this

secure tunnel.

One of the most popular features in Functionize is the ability to view all the screenshots

taken during a test. The Functionize CLI supports this using the view-screenshots

command.

functionize view-screenshots 

The Functionize CLI will launch a browser and take you to a custom URL showing the

results of your CLI test. The URL is secure, but it can be shared with other members of your

team if they are logged in. This feature is ideal for creating custom test integrations.

Functionize works seamlessly as part of a continuous integration/continuous deployment

(CI/CD) pipeline. For instance, here at Functionize, we use Spinnaker as part of our own

pipeline. Spinnaker is ideal because it is open-source and works on IaaS clouds (clouds that

run virtual machines like GCP, OpenStack, AWS, VMware, etc.) and container orchestration

platforms (clouds that run containers like Docker Swarm, Kubernetes, etc.).

Spinnaker makes it easy to deploy new builds to staging. It can trigger a full set of

Functionize tests (e.g. smoke tests, regression tests, API tests, etc.), and if the tests pass,

the new code can be automatically deployed using the Promote to Production pipeline. Our

customers have also used other CI tools to achieve the same including Travis CI, Heroku

and Jenkins. If you want more advice, talk to one of our Solutions Architects.

Integrating Functionize as part of your CI pipeline

Viewing test output

Learn more at:www.functionize.com

Functionize also provides tools for testing other aspects of your system, not just the web

app frontend.

Functionize’s load testing solution runs on Google Cloud and is designed to perform far

more realistic load testing than the traditional method of making multiple API calls. This is

because it uses realistic user sessions generated by the Functionize test system from any

location around the world. As a result, the tests accurately predicts how real users will see

the performance change as load increases. There are two modes of testing – testing how

the system performs under a specified load and steadily increasing the load to trigger a

failure. In the former case, the results will show you details, such as how quickly different

page elements loaded, which elements were outliers and the relative performance of each

page. Results are available in real-time and give you a detailed breakdown of the page

performance.

Load testing

Other test tools

The load tests results dashboard gives you detailed statistics

Learn more at:www.functionize.com

You can compare the relative performance of each page to see what is adding delay

You can see how long each step in your test took to complete

Learn more at:www.functionize.com

You can quickly spot outliers

API testing

Functionize’s API testing solution uses an AI-powered recorder to capture the API test

steps. Unlike traditional API testing, our approach tests APIs visually, reducing time and

simplifying the process of creating API tests. Our platform also allows you to take responses

such as API keys and store them using local variables. They can then be accessed later in

the test just by calling the variable. This makes the tests far easier to create, and means an

end to repeated copy-pasting of such details. The system also builds in extensive validations

and uses machine learning to track how the API response should look. This allows it to be

resilient to things like changes in response order.

Learn more at:www.functionize.com

As you add steps you specify the type of call, response type and output format

You can now copy the access token from the response

© Copyright 2018 Functionize, Inc. All Rights Reserved 156 2nd Street | San Francisco, CA 94105 +1-800-826-5051 | www.functionize.com

You can store the key as a parameter called apikey . Now when you run your tests, the API key will be automatically

copied in the previous step and then it can be referred to in subsequent steps by putting apikey .