how to utilise
TRANSCRIPT
![Page 1: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/1.jpg)
![Page 2: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/2.jpg)
Alexander Tolstikov Smile Open Source Solutions
How to utilise Open Source tools to create CI/CD & DevOps workflows
without vendor lock-in
![Page 3: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/3.jpg)
Possible workflows:
- Create Kubernetes cluster- Build release artifact (composer install, yarn build, npm)
- Deliver artifact (ansistano, capistrano, etc)- Drush operations
- Automated functional / load / smoke testing - Anything else?
![Page 4: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/4.jpg)
Workflow example:Kubernetes cluster creation on GCP
![Page 5: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/5.jpg)
Evolution of the workflow
![Page 6: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/6.jpg)
Iteration 1:Create k8s cluster manually with UI
ToolsGCP UI
Secrets/Access You need to be authenticated in the browser
Pros • Very fast • No need to maintain dependencies / tools on the local PC
Cons • Need to remember all command params • Need to do the same for all resources (VPC, subnets, DNS, etc) • Human error-prone • Only you know what you did
![Page 7: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/7.jpg)
Congratulations! We just created a snowflake cluster!
![Page 8: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/8.jpg)
Snowflakes are beautiful…
![Page 9: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/9.jpg)
…but so different (and fragile)
![Page 10: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/10.jpg)
“ “Martin Fowler
The true fragility of snowflakes, however, comes when you need to change them. Snowflakes soon become hard to understand and modify. You're not sure what parts of the configuration are important. Their fragility leads to long, stressful bouts of debugging.
![Page 11: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/11.jpg)
Toolsgcloud (from Google Cloud SDK)
Secrets/Access Need to be authenticated on the local PC
Pros More “fixed” compared to manual UI operation
Cons • Need to remember all command params • Need to do the same for all other resources (VPC,
subnets, …) • Human error-prone • No centralised history / logs • Still “Snowflake” infrastructure
Iteration 2: Create k8s cluster manually with “gcloud” command
![Page 12: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/12.jpg)
Tools • terraform Secrets • Need to use GCP key file stored on the local PC Principles / Concepts • Infrastructure as Code Pros • Logic and configuration are separated (different terraform files) • Can be (should be!) checked-in into Git repo Cons • Need to copy/paste files with variables for different environments manually • Configuration drift is still possible with multiple environments
Iteration 3:Create k8s cluster with terraform: Manual terraform commands execution
![Page 13: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/13.jpg)
Iteration 4:PROD & TEST environments & single master config file
![Page 14: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/14.jpg)
Need to introduce some tools!
![Page 15: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/15.jpg)
“ “Unix philosophy
• Do one thing and do it well. • Make programs to work together. • Write programs to handle text streams,
because that is a universal interface.
![Page 16: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/16.jpg)
Tool: uniconf (https://github.com/aroq/uniconf)Config processing (YAML)
Sorry, no documentation…
![Page 17: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/17.jpg)
Input config Output config
OutputInput
![Page 18: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/18.jpg)
Tool: uniconf (https://github.com/aroq/uniconf)Config processing (YAML)
![Page 19: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/19.jpg)
Tool:jq (https://github.com/stedolan/jq)
Config processing (JSON)
Use to extract data from JSON
A lot of filters are available
Check the documentation!
![Page 20: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/20.jpg)
Tool:yq (https://github.com/mikefarah/yq)
Config processing (YAML)
Same as jq tools but for YAML files
Basically is a wrapper over jq tool
![Page 21: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/21.jpg)
uniconf | yq
![Page 22: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/22.jpg)
Output
![Page 23: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/23.jpg)
Tool:gomplate (https://github.com/hairyhenderson/gomplate)
Template rendering. Supports lots of local and remote datasources.
Check the documentation!
![Page 24: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/24.jpg)
Output
Output
Template
![Page 25: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/25.jpg)
Tool:variant (https://github.com/mumoshu/variant)
“One glue to glue them all!”
“Makefile on steroids”
Task definition
Task execution
![Page 26: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/26.jpg)
Variant task execution
![Page 27: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/27.jpg)
Plain command vs Variant task execution
![Page 28: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/28.jpg)
Iteration 4:PROD & TEST environments & single master config file
Now you need to maintain more tools / dependencies:
terraform unicorn
jq / yq gomplate
variant
It can be an issue if we want to execute this workflow somewhere else because we’ll need to sync all software versions, dependencies, etc
![Page 29: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/29.jpg)
Dependency hellPythonRuby
NodeJS…
![Page 30: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/30.jpg)
How to fix the Dependency Hell issue: Put all tools into the container (Docker)
+ =love
![Page 31: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/31.jpg)
Secrets management
![Page 32: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/32.jpg)
Managing secrets
Git Secret (https://github.com/sobolevn/git-secret#git-secret)
Chamber (https://github.com/segmentio/chamber)
GPG (https://gpgtools.org/)
AWS Vault (https://github.com/99designs/aws-vault)
![Page 33: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/33.jpg)
Execute the workflow in the CICD engine: Gitlab CI
![Page 34: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/34.jpg)
Gitlab CI pipeline
![Page 35: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/35.jpg)
Demo time!
![Page 36: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/36.jpg)
Next steps:GitOps? Check Atlantis:
https://github.com/runatlantis/atlantis
![Page 37: How to utilise](https://reader031.vdocuments.net/reader031/viewer/2022020700/61f3a9bed91c781606629df3/html5/thumbnails/37.jpg)
Alexander [email protected] / [email protected]
You can find me on Drupal slack as well!
https://www.smile.eu/