devops and automated testing - free university of bozen ...russo/ttst/automatedtesting.pdf · •...
TRANSCRIPT
![Page 1: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/1.jpg)
Tools and Techniques for Software Testing - Barbara Russo SwSE - Software and Systems Engineering group
�1
DevOps and Automated testing
![Page 2: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/2.jpg)
�2
• We have seen that we can implement pipelines in Maven
• Exercise run Maven to test your project • Import the example of POM file from ole • Customize and run for your project • Refactor some of the classes changing the name with
postfix IntegrationTest • Re-run test with Maven excluding Unit Tests • Report any issue
Pipelines in shared environments
![Page 3: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/3.jpg)
�3
• What you experienced in the exercise above is a sort of DevOps process
• Manual • Offline
DevOps
![Page 4: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/4.jpg)
�4
• The term DevOps emerged a decade ago as an amalgamation of Development and Operations
• A reaction to a perceived disconnect between developers and operators within the same organization
• Adopted to develop and operate software solutions in the cloud
DevOps
![Page 5: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/5.jpg)
�5
• Virtualized infrastructure enables perform deployment and configuration using pre-configured scripts
• This allows organizations to deploy new versions several times a day
• Operations can be configured to provide feedback from daily operations, which in turn supports development and deployment of new and enhanced features at a rapid pace
Automation in development
![Page 6: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/6.jpg)
�6
• Manual testing is more prone to human error • To reduce QA effort • Reduce waiting time to know build quality • Categorize tests and run against build (smoke/
regression test) • Schedule tests and make them under control
(nightly regression)
Why did we set up an automated system to run tests?
![Page 7: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/7.jpg)
�7
Continuous Development lifecycle
!
![Page 8: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/8.jpg)
�8
• Continuous Integration • Automated Testing and Artifact Creation (e.g.,
build creation) • Continuous Delivery
• Automated deployment to test and staged environments
• Manual deployment to production • Continuous Deployment
• Automated Deployment to production
DevOps
![Page 9: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/9.jpg)
�9
• In Continuous Integration, several builds in a day are executed (nightly builds as well)
• Each build should pass through various sandboxed environments leading up to production and stages (stable, develop, master)
Continuous Development
![Page 10: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/10.jpg)
�10
• In Continuous Delivery, code changes are continuously deployed, although the deployments are triggered manually
• If the entire process of moving code from source repository to production is fully automated, the process is called Continuous Deployment
Continuous Delivery
![Page 11: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/11.jpg)
�11
• A strategy for software releases where in any code commit that passes the automated testing phase is automatically released into the production environment, making changes that are visible to the software's users
Continuous Deployment
![Page 12: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/12.jpg)
�12
• A test environment is where test are performed. No real environment bounds tests
• A staging environment (stage) is a nearly exact replica of a production environment
• Staging environments are made to test codes, builds, and updates to ensure quality under a production-like environment before application deployment
Test and staging environments
![Page 13: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/13.jpg)
�13
![Page 14: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/14.jpg)
�14
• Clean up your project • How many branches do you have? • Did you merge your branch into master before
deploy? • Did you create test environment? (e.g., test
package)
Exercise
![Page 15: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/15.jpg)
�15
• A pipeline is a group of job/tasks that get executed in stages also called batches
• All jobs in stages run sequentially/parallel based on the runners available
Pipelines
![Page 16: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/16.jpg)
�16
• Pipelines’ major components: jobs and stages • Pipelines are configured using a YAML file
called .gitlab-ci.yml within each project • Jobs and stages are defined in the .gitlab-ci.yml
file for each project
Pipelines’ configuration
![Page 17: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/17.jpg)
�17
• “stages” term is used to define stages that can be used by jobs
• Stages allow multiple pipelines • The ordering of elements in stages defines the
ordering of jobs’ execution: • Jobs of the same stage are run in parallel • Jobs of the next stage are run after the jobs from
the previous stage complete successfully
Stages (careful not for environments!)
![Page 18: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/18.jpg)
�18
• The .gitlab-ci.yml file defines the structure and order of the pipelines and determines: • What to execute using GitLab Runner. • What decisions to make when specific
conditions are encountered • For example, when a process succeeds or fails
.gitlab-ci.yml
![Page 19: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/19.jpg)
�19
• Build (default if no stages are defined in yml) • Test (default if no stages are defined in yml)
(assigned to a job when no stage specified for a job) • Deploy (default if no stages are defined in yml) • Review • Dynamic Application Security Testing (DAST) • Staging • Canary
Typical stages
![Page 20: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/20.jpg)
�20
• Jobs are defined with constraints stating under what conditions they should be executed
• Top-level elements with an arbitrary name and must contain at least the script clause
• Not limited in how many can be defined • If all the jobs in a stage
• Succeed, the pipeline moves on to the next stage • Fail, the next stage is not (usually) executed and the pipeline
ends early
Jobs
![Page 21: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/21.jpg)
�21
job1: script: test.sh
job2: script: mvm install
Example
![Page 22: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/22.jpg)
�22
• Job A(1,3), B(2,4) and C(6,7)
0 1 2 3 4 5 6 7 AAAAAAA BBBBBBB CCCC
• Total duration: 4-1 + 7-6 = 4
Job duration
![Page 23: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/23.jpg)
�23
image: maven:latest
variables: MAVEN_CLI_OPTS: "-s ./Applications/maven/conf/settings.xml --batch-mode" cache: paths: - target/
stages: - build - test - doc - deploy - trigger
build: stage: build script: - mvn $MAVEN_CLI_OPTS compile
test: stage: test script: - mvn $MAVEN_CLI_OPTS test artifacts: paths: - target/
Globallydefined
shellcommandsshellcommands
![Page 24: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/24.jpg)
�24
• Jobs are picked up by Runners and executed within the environment of the Runner
• Multiple jobs in the same stage are executed by Runners in parallel, if there are enough concurrent Runners
Runners
![Page 25: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/25.jpg)
�25
• GitLab Runner is used to run jobs and send the results back to GitLab
• Runners run the code defined in .gitlab-ci.yml • After installation, Runners must be registered
• as shared (multiple projects with same requirements) or specific (individual project)
GitLab Runner
![Page 26: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/26.jpg)
�26
![Page 27: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/27.jpg)
�27
>sudo curl --output /usr/local/bin/gitlab-runner https://gitlab-runnerXXX >sudo chmod +x /usr/local/bin/gitlab-runner
install GitLab Runner
![Page 28: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/28.jpg)
�28
• Registering a Runner is the process that binds the Runner with a GitLab instance
• Read more: • https://docs.gitlab.com/runner/register/
index.html • Example: • gitlab-runner register -n --name <your-runner-name> -r
<your-project-token> -u https://gitlab,com --executor shell --tag-list <your-runner-tag> --run-untagged false
Register the runner
![Page 29: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/29.jpg)
�29
• cd ~ • gitlab-runner install • gitlab-runner start
Start the Runner
![Page 30: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/30.jpg)
�30
• When commit is pushed to the repository, GitLab will look for .gitlab-ci.yml from the root directory and trigger a build according to the settings configured
• GitLab Runner uses this file to manage project’s jobs which defines how the project should be built
Executing the pipeline
![Page 31: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/31.jpg)
�31
• Each instance of GitLab CI has an embedded debug tool called Lint, which validates the content of your .gitlab-ci.yml
Lint
![Page 32: DevOps and Automated testing - Free University of Bozen ...russo/TTST/AutomatedTesting.pdf · • The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:](https://reader034.vdocuments.net/reader034/viewer/2022042802/5f42914621ebf92c460a65e9/html5/thumbnails/32.jpg)
�32
• Write the yml file for the simple pipeline example
• build, with a job called compile • test, with two jobs called test1 and test2 • staging, with a job called deploy-to-stage • production, with a job called deploy-to-prod
Exercise