jenkins job builder: our experience
TRANSCRIPT
Jenkins Job BuilderOur experience
Open Source and Linux Lab
OSLL
Our setup
• Product to build and test: Maxscale – database proxy, lots of plugins (routers, filters, protocols, etc, etc, etc)
• Test environment:
– at least 9 VM (1 Maxscale server, 4 Master/Slave servers, 4-servers Galera cluster)
– libvirt/qemu, AWS virtual machines + Docker containers
– need to configure backend for every test run
– need to carefully clean up
• 12 Linux distributions, 3 major MariaDB version, 2 major MySQL version, a lot of Maxscale plugins – any combination should be testable
• dependency on VM provider: it can easily FAIL
What does it mean for Jenkins?
• Lots of parameters for every job
• Cascades of jobs: e.g. prepare backend, install Maxscale, run test, cleanup
• Error processing (e.g. backend VMs preparation failure)
• Matrix jobs: test a lot of combinations of distros, backedn MariaDB version, backend parameters
• very dynamic setup
We wanted to put all Jenkins jobs into GIT repository
Jenkins Job Builder
• A tool from OpenStack developers https://docs.openstack.org/infra/jenkins-job-builder/
• YAML
• Pipelines did not exist when we started
• our jobs: https://github.com/mariadb-corporation/maxscale-jenkins-jobs
What we get from JJB?
• All our jobs (54) are represented as a code (branches, forks, review, pull requests, …)
• Parameters values are in one place for all jobs (easy maintenance of parameters defaults and lists)
• Automatic validation of jobs takes place at every push to jobs repo
• very easy jobs deployment
• at the beginning: app. 20 jobs, we spent 1 week to convert to JJB code (1.5 persons)
- job: name: run_test description: 'This job perform integration testing of maxscale' parameters: - !include: './maxscale_jobs/include/boxes_all_incl.yaml' - !include: './maxscale_jobs/include/products_incl.yaml' - !include: './maxscale_jobs/include/versions_incl.yaml' .... - !include: './maxscale_jobs/include/test_branch.yaml' - !include: './maxscale_jobs/include/slave.yaml' scm: - git: # TODO parametrize this url url: https://github.com/mariadb-corporation/maxscale-system-test.git branches: - $test_branch builders: - !include: './maxscale_jobs/include/build_parser/create_env_vars.yaml' - !include: './maxscale_jobs/include/build_parser/inject_initial_env.yaml' - !include: './maxscale_jobs/include/build_parser/run_test_and_collect.yaml' - !include: './maxscale_jobs/include/build_parser/parse_build_log.yaml' - !include: './maxscale_jobs/include/build_parser/inject_build_results.yaml' - !include: './maxscale_jobs/include/build_parser/create_env_coredumps.yaml' - !include: './maxscale_jobs/include/build_parser/inject_coredumps_var.yaml' - !include: './maxscale_jobs/include/build_parser/write_build_results.yaml' publishers: - !include: './maxscale_jobs/include/build_parser_mail_subject_with_name.yaml' - !include: './maxscale_jobs/include/call_cleanup.yaml' wrappers: - !include: './maxscale_jobs/include/workspace-cleanup.yaml' - !include: './maxscale_jobs/include/timeout.yaml' concurrent: true
choice: name: box choices: !include: './maxscale_jobs/include/boxes_all.yaml' description: 'Virtual machine OS (name of Vagrant box)'
- centos_7_libvirt- centos_6_libvirt- centos_5_libvirt- ubuntu_wily_libvirt- ubuntu_wily_aws…- suse_13_libvirt
boxes_all_incl.yaml
boxes_all_incl.yaml
shell: '$HOME/build-scripts/test/run_test.sh | tee $WORKSPACE/build_log_$BUILD_ID; echo ${PIPESTATUS[0]} > result_$BUILD_ID'
boxes_all_incl.yaml
email-ext: recipients: !include-raw: "./maxscale_jobs/include/mail_recipients.yaml" reply-to: $DEFAULT_REPLYTO content-type: default subject: $DEFAULT_SUBJECT ($name) body: !include-raw: "./maxscale_jobs/include/build_parser_email_body" attach-build-log: false ...
boxes_all_incl.yaml
Problems: !includechoice: name: box choices: !include: './maxscale_jobs/include/boxes_all.yaml' description: 'Virtual machine OS (name of Vagrant box)'
boxes_all.yaml
!include: './maxscale_jobs/include/boxes_libvirt.yaml'!include: './maxscale_jobs/include/boxes_aws.yaml'
boxes_libvirt.yaml
- centos_7_libvirt- ubuntu_trusty_libvirt- debian_jessie_libvirt- suse_13_libvirt
boxes_aws.yaml
- ubuntu_wily_aws- centos_7_aws- sles_12_aws- rhel_7_aws
choice: name: box choices: !include: './maxscale_jobs/include/boxes_libvirt.yaml' description: 'Virtual machine OS (name of Vagrant box)'
choice: name: box choices: !include: './maxscale_jobs/include/boxes_aws.yaml' description: 'Virtual machine OS (name of Vagrant box)'
Problems: !includechoice: name: box choices: !include: './maxscale_jobs/include/boxes_all.yaml' description: 'Virtual machine OS (name of Vagrant box)'
boxes_all.yaml
- centos_7_libvirt- ubuntu_trusty_libvirt- debian_jessie_libvirt- suse_13_libvirt- ubuntu_wily_aws- centos_7_aws- sles_12_aws- rhel_7_aws
Problems: !include
- job: name: build description: 'This job builds Maxscale' parameters: - !include: './maxscale_jobs/include/value.yaml' - !include: './maxscale_jobs/include/source.yaml' - !include: './maxscale_jobs/include/boxes_all_incl.yaml' - !include: './maxscale_jobs/include/target.yaml' - !include: './maxscale_jobs/include/cmake_flags.yaml' - !include: './maxscale_jobs/include/do_not_destroy_vm.yaml' - !include: './maxscale_jobs/include/build_experimental.yaml' - !include: './maxscale_jobs/include/run_upgrade_test.yaml' - !include: './maxscale_jobs/include/old_target.yaml' - !include: './maxscale_jobs/include/cnf_file.yaml' - !include: './maxscale_jobs/include/maxadmin_command.yaml' - !include: './maxscale_jobs/include/slave.yaml' scm: - git: url: !include: './maxscale_jobs/include/maxscale_repo.yaml' branches: - 'develop' builders: - shell: '$HOME/build-scripts/prepare_and_build.sh' - shell: 'export new_target=$target; $HOME/build-scripts/upgrade_test.sh'
build.yaml
- job: name: build_all description: 'This job builds Maxscale' project-type: matrix execution-strategy: sequential: false axes: - axis: type: user-defined name: box values: !include: './maxscale_jobs/include/boxes.yaml' parameters: - !include: './maxscale_jobs/include/value.yaml' - !include: './maxscale_jobs/include/source.yaml' - !include: './maxscale_jobs/include/target.yaml' - !include: './maxscale_jobs/include/cmake_flags.yaml' - !include: './maxscale_jobs/include/do_not_destroy_vm.yaml' - !include: './maxscale_jobs/include/build_experimental.yaml' - !include: './maxscale_jobs/include/run_upgrade_test.yaml' - !include: './maxscale_jobs/include/old_target.yaml' - !include: './maxscale_jobs/include/cnf_file.yaml' - !include: './maxscale_jobs/include/maxadmin_command.yaml' - !include: './maxscale_jobs/include/slave.yaml' scm: - git: url: !include './maxscale_jobs/include/maxscale_repo.yaml' branches: - 'develop' builders: - !include: './maxscale_jobs/include/build_all/call_build.yaml'….
build_all.yaml
Problems
• No parametrized !include
• a bit tricky to put right number of spaces in the beginning of every line (YAML problem)
• can not delete job automatically (rename case)
• Jenkins plugins have to be installed manually
• Views have to be configured manually
Jenkins CLI can solve it!
Our additions
• simple wrapper to load jobs https://github.com/OSLL/jjg.git
– adds timestamps and git commit ID to job description
– easier way to manage Jenkins creds• Job Visualizer (not ready yet, but working)
https://github.com/OSLL/jenkins_job_builder_visualization
Authors:
Evgeny Bovykin [email protected] Zaslavskiy [email protected]
Job Visualizer: Call Graph
● Do you call all the jobs correctly?
● Do you have cyclic dependencies?
Our additions
• simple wrapper to load jobs https://github.com/OSLL/jjg.git
– adds timestamps and git commit ID to job description
– easier way to manage Jenkins creds• Job Visualizer (not ready yet, but working)
https://github.com/OSLL/jenkins_job_builder_visualization
Authors:
Evgeny Bovykin [email protected] Zaslavskiy [email protected]
Job Visualizer: Include Graph
If you change one file, which jobs you can break?
MariaDB Maxscale Continuous Integration
Open Source and Linux Lab
OSLL
in cooperation with
Open Source and Linux Lab
Open Innovations Association FRUCT
and
http://fruct.org/
https://osllblog.wordpress.com/
https://mariadb.com/products/mariadb-maxscale
Mark Zaslavskiy <[email protected]>
Sergey Balandin <[email protected]>