Hi, my name is Torsten Bresser Today I will show you how we scaled our Magento development process with containers, Docker containers in this particular case. Before we start with the actual talk, just a few words about me and my company.
About meAndreas Koch
• Software developer at arvato SCM• Located in Germany• Passionate about
– Deployment automation– Scalable software and processes– PHP, C# and Go
andreaskoch
andykdocs.de
Presenter
Presentation Notes
As you probably can see, this guy here on the slide isn’t me. It is my colleague Andreas Koch. Andy was supposed to talk to you here today, but unfortunately he got sick and is now in a hospital in Germany (send greetings). So I jumped in on very short notice. I am a software architect from Germany. I work for more than 10 years on international e-commerce sites at arvato. Since 4 years we work with Magento (before C#), and I lead our German Magento developer teams. Similar interests as Andy, but no Go programmer
arvato – Bertelsmann
Presenter
Presentation Notes
arvato SCM – Supply Chain Management Part of arvato, a Bertelsmann subsidiary, Europe‘s biggest media company Bertelsmann not well known in the US → Penguin Random House arvato is a big company → put key figures together
arvato SCM Solutions
Shop Development
Digital Imaging
Customer Service
Global Online Marketing
E-Commerce Consulting
Warehousing & Distribution
Transport Management
Financial Services
Business Intelligence
Returns Management
Omnichannel Value added services
Service portfolio Reference customers
Presenter
Presentation Notes
Offers world-wide, full-service e-commerce solutions Various industry segments like Consumer products like fashion Hightech & entertainment Healthcare Telecommunications etc. Various frontend technologies like Microsoft .NET Demandware Native iOS and Android app development Magento → Presentation topic: Simplifying development + hosting of Magento shops with containers
Development,Hosting &Operations
of Magento Shops
Presenter
Presentation Notes
What is needed to develop, host and operate Magento shops?
Hosting Magento is simple. Right?
Presenter
Presentation Notes
It is easy, right? Just A web-server like Apache or Nginx PHP MySQL ... and of course a developer
Hosting Magento is simple. Right? Nope!
Presenter
Presentation Notes
Not that simple unfortunately If goal is to find bugs in dev instead of production → same setup on all environments Additional components: Redis ElasticSearch RabbitMQ etc. + whole team of developers
Challenge #1: Knowledge Transfer«I dunno how to configure Nginx. Andy always does that. But he is currently on vacation in Vegas.» Knowledge Monopolies
Presenter
Presentation Notes
First challenge: Knowledge transfer and monopolies Mention „personal experiences“ Explain situation: The only developer with special know-how is never available One knows Nginx, otherone MySQL Building up and maintaining knowledge requires effort Setup maybe not well documented or automated But at least developers manage the situation somehow
Challenge #2: Parallelization«Don’t touch the integration environment the next days – we are going to test feature xyz!» Parallelization
Presenter
Presentation Notes
Second challenge: Parallelization of development and QA streams Explain situation: QA wants to test features and bugfixes Multiple dev streams vs. one integration environment → Downtimes during deployments → Interferences between features → Bottleneck situation → automation needed
Challenge #3: Operations«How could that have ever worked on the integration environment?!» Operations
Presenter
Presentation Notes
Challenge number 3: Operations Deploying complex infrastructure isn‘t easy Deploy all software components to multiple production servers Same versions across environments ... maybe even highly available Next slide: Patching
Challenge #3: OperationsKeeping environments in sync is hard.
Presenter
Presentation Notes
... and then there are also patches: Every now and then updates of all development Integration environments and servers production → you know from experience → takes a lot of time + error prone process
Challenge #4: Scaling«Just a little something for the current sprint, Andy: We need a new integration and production for the new project. asap!» Scaling
Presenter
Presentation Notes
Last challenge: Scaling Usually a good think, but If no automation, then operating more environments → more manual effort = potential errors Fortunately Docker can help
Introduction to DockerContainer virtualization made easy
What is Docker?Docker automates the deployment of applications inside containers
• Based on Linux technologies like– LXC– cgroups– aufs
• Avoids the overhead of classical virtual machines– Small and portable containers– Ideal for application packaging
Presenter
Presentation Notes
Lightweight virtualization technology for Linux Allows you to package your application components into small Isolated containers – can be linked together + managed portable IS vs. USE Since June 2014
Solving the Challenges1. Knowledge Monopolies2. Parallelization3. Operations4. Scaling
Presenter
Presentation Notes
How can Docker help? First: Knowledge Transfer
Solving Knowledge MonopoliesDocker allows you to
• split your infrastructure components into immutable units of execution by using standard Docker images such as Nginx, PHP and MySQL
• make your software configuration transparent and traceable with Dockerfiles
• combine all your software dependencies into a single docker-compose.yml file
Presenter
Presentation Notes
Docker → each infrastructure component turned into immutable unit of execution Easily connectable Standard images available Docker Composer
Leveraging Docker Standard Images
Docker provides a long list of standard images that are maintained byDocker or the software authors
Operating System Images• Alpine• Ubuntu• Debian• CentOS• …
Software components• PHP• MySQL• Nginx• Redis• …
Presenter
Presentation Notes
Rely on experts for software required for hosting Your developers → focus on shop development Standard images for all components needed for Magento 1 or 2 Available on website „Docker Hub“
Docker Standard Images: PHP
Software components• PHP• MySQL• Nginx• Redis• …
Presenter
Presentation Notes
PHP
Docker Standard Images: MySQL
Software components• PHP• MySQL• Nginx• Redis• …
Presenter
Presentation Notes
MySQL
Docker Standard Images: Nginx
Software components• PHP• MySQL• Nginx• Redis• …
Presenter
Presentation Notes
Nginx
Docker Standard Images: Redis
Software components• PHP• MySQL• Nginx• Redis• …
Presenter
Presentation Notes
Redis ... and many, many more Otherwise Rely on community or Build your own
Infrastructure Definition with Dockerfiles
FROM ubuntu:latest
RUN apt-get update && apt-get install nginx
EXPOSE 80 443
CMD ["nginx", "-g", "daemon off;"]
1. Base image
2. Setup commands
3. Expose ports
4. Entry point
Presenter
Presentation Notes
Sounds complicated, but is very easy Dockerfiles Simple text files Define Base Image Installation commands Ports Command for start It‘s as easy as that, even for „official“ images
Infrastructure Definition with Dockerfiles
Presenter
Presentation Notes
Actual content not really interesting Only one page Tracked in Git Exactly the same for each Linux host Portable Immutable
Running a Default Nginx Container
$ docker run –p 8080:80 nginx:1.9
Presenter
Presentation Notes
Example shown. Command will: Bind port 80 of container to port 8080 of host Download Nginx Docker image Start container → always same result → customize content
Customization via Docker VolumesCustomizing dockers container with volumes:
$ docker run -p 8080:80 -v `pwd`/custom-content:/usr/share/nginx/htmlnginx:1.9
Presenter
Presentation Notes
Use volumes to map files into containers „custom-content“ is mapped into container path „/usr/share/nginx/html“ Let‘s run the command...
Customization via Docker Volumes
Presenter
Presentation Notes
Right-hand side on top of the screen: empty custom-content folder Error message, because of empty folder
Customization via Docker Volumes
Presenter
Presentation Notes
Create index.html File is directly linked into container → Example works Just Nginx is not enough → More software components for Magento needed → Docker Compose
Docker Compose Orchestration utility for Docker Build and run multi-container applications docker-compose.yml Define parameters and relationships between software
… sharing files/a volume with a PHP-FPM container → Demo
Orchestration with Docker Compose
Presenter
Presentation Notes
Docker Compose reads the YAML file and starts all containers → Nginx connected to PHP-FPM both running in different containers
Solving Knowledge Monopolies
Automate the environment setup with Docker Compose
Presenter
Presentation Notes
Docker Compose Replace all manual environment setup steps By a simple config file
Solving Knowledge Monopolies
Automate the environment setup with Docker Compose
docker-compose.yml
docker-compose.yml
Presenter
Presentation Notes
All that is left to do for the developers: Copy project with the docker-compose.yml And run it
Solving the Challenges1. Knowledge Monopolies ✓2. Parallelization3. Operations4. Scaling
Presenter
Presentation Notes
We have our first dockerized PHP server → Knowledge transfer problem solved Implicit knowledge → infrastructure code Next challenge: Parallelization
Enable ParallelizationCombination of tools and processes:
• Docker Compose• Git Combine infrastructure and software• Composer
• git-flow Branch on feature level
• Cloud hosting Create feature instances for testing on-the-fly
Develop and test features in parallel
Presenter
Presentation Notes
Magento 2 setup more complicated, but same approach What do we have to do? Combine infrastructure setup + software Use feature branches Use clouds (either private or public ones) → Develop and test in parallel
Enable ParallelizationGet rid of bottlenecks
Presenter
Presentation Notes
So instead of testing all changes on a single environment...
Enable ParallelizationGet rid of bottlenecks … by creating feature instances
Presenter
Presentation Notes
... Create temporary cloud instances After testing is finished → Ready for deployment on the actual integration environment
Enable ParallelizationCreate feature-instance with the arvato CloudAPI on-the-fly:
Presenter
Presentation Notes
CloudAPI Internal tool Creation of cloud instances Performs additional steps, like copying databases + media folders After development → create a new feature instances After testing → merge into develop branch Open Source?
Solving the Challenges1. Knowledge Monopolies ✓2. Parallelization ✓3. Operations4. Scaling
Presenter
Presentation Notes
Integration environment no longer a bottleneck → Parallelization solved Next: Simplifying operations with Docker
Ease OperationsDocker and Docker Compose will reduce operation efforts:
• Better quality. All environments look the same
Presenter
Presentation Notes
With a dockerized infrastructure Less bugs because environments look similar Bugs can be found earlier in development process
Ease OperationsDocker and Docker Compose will reduce operation efforts:
• Better quality. All environments look the same
• Easier patching. Download the latest Docker images and restart
Presenter
Presentation Notes
Patching also easier Attack surface of host OS reduced, because only Docker Easy updates of Docker images Delete container Download new image Restart container
Ease OperationsDocker and Docker Compose will reduce operation efforts:
• Better quality. All environments look the same
• Easier patching. Download the latest Docker images and restart
• Easier handling. All projects can be controlled by the same commands
Presenter
Presentation Notes
Easier handling Same project structure (even if different components) Same commands for each project
Ease OperationsDocker and Docker Compose will reduce operation efforts:
• Better quality. All environments look the same
• Easier patching. Download the latest Docker images and restart
• Easier handling. All projects can be controlled by the same commands
• Easier handover. Between development and operations team(s)
Presenter
Presentation Notes
Easier handover Software and infrastructure “documented” in code No other deployment documents needed in our departments anymore
Ease Operations: Deployment & Patching1. Copy the new project version to production2. Pull the latest Docker images3. Run the new project version
docker-compose.yml
docker-compose.yml
Presenter
Presentation Notes
Since infrastructure + code combined in single project → Deployments + patches easier as well Steps for a deployment/patching: Copy new project version to production Pull latest Docker images Run with Docker Compose
Reasons for scaling More load Adding more features Winning new businesses
ScalingWhat can you do to scale?
• Add more developers and testers to your team
• Optimize the infrastructure and code
• Add more RAM, CPU and storage to your existing servers
• Add new servers to your production environment
Presenter
Presentation Notes
What can you do to scale? Add more developers + testers → increased throughput Optimize existing infrastructure and code to be more efficient Add capacity to production (vertically or horizontally) to handle more load BUT...
Complexity with Automation
Presenter
Presentation Notes
... this will always increase the complexity Fortunately, only linearly if automation is used OTHERWISE...
Complexity without Automation
Presenter
Presentation Notes
... exponentially Why? More developers + testers → More environments, more deployments More servers → More patching More computational power → ensure software USES the resources New businesses → More environments Admit: Not 100% accurate from a scientific point of view (but looks good)
Scaling with Docker
– More Load: Scale horizontally or vertically
– More Features: We can now parallelize our development
– More Businesses: Copy an existing project and customize it
Still not easy. But easier than without it!
Presenter
Presentation Notes
… and with Docker? Scaling get easier More load → add container (on existing machines, or new ones, e. g. in the cloud) More features → parallelize New businesses → use as templates Still not easy, but easier than without automation Docker not only solution → Just our way, but automation needed
Docker can help to solve typical challenges in Development Hosting Operations by automating infrastructure setups
CombiningInfrastructure & Code
Project structure of a dockerized Magento shop
Presenter
Presentation Notes
Let‘s have a look at how that works with Magento 2
Combining Code + Infrastructure
− app− bin− vendor− pub− composer.json− index.php
− config− mysql− nginx− php
− docker-compose.yml
Code
Infrastructure
Presenter
Presentation Notes
Quite simple structure Magento project (in red) docker-compose.yml (blue) Configs (blue) Start containers from there with Docker Compose That‘s it Next: Different environments
Handling different environments− app− …− config
− mysql− nginx− php-dev− php-prod
− docker-compose.yml− docker-compose.prod.yml
Configs for other environmentsshould also be tracked in thesame project, e.g.
• The production config of yourPHP container should not contain Xdebug
• Usage of a central MySQL cluster on production
Presenter
Presentation Notes
Usually differences between environments Xdebug MySQL cluster Developer Mode → Environment specific docker-compose.ymls
Handling different environments− app− …− config
− mysql− nginx− php-dev− php-prod
− docker-compose.yml− docker-compose.prod.yml
Using a different Docker Composefile:
> docker-compose -f docker-compose.prod.yml up -d
Presenter
Presentation Notes
-f parameter We use 3 docker-compose.ymls Development Testing Without Xdebug No Developer Mode Production Central MySQL cluster + Redis server etc.
Development EnvironmentUsing Docker on Linux, OS X and Windows
Presenter
Presentation Notes
More into practice now! What do you need to use Docker on your local machine?
Development Environment
The dockerized development process works on Linux, OS X & Windows. All you need is:
– Docker Machine (only for Mac and Windows)– Docker & Docker Compose– Git + PHP + Composer
+ An editor or IDE of your choice
Presenter
Presentation Notes
You can use Linux Windows Mac OS X on your host (we will use Mac OS X in the following examples) Install Docker Docker Compose Docker Machine (for Mac OS X and Windows) Git PHP + Composer (for installing + debugging Magento)
Our setup: Ubuntu VMs with PHPStorm Open project folder like normal in editor of choice Start containers with Docker Compose
Path in container/var/www/html
Local path~/magento2
Presenter
Presentation Notes
Path mapping is different for debugging → Explain: Code not executed locally Different paths on host and in container ~/magento2 → /var/www/html
Docker for Windows and MacInstall the Docker Toolbox to use Docker on Mac or Windows
Presenter
Presentation Notes
Time check → Go to slide 70 On Mac + Windows → extra work because Docker = Linux feature → Install Docker Toolbox Docker client Docker Compose Docker Machine
Docker Machine
Presenter
Presentation Notes
Docker Machine works much like Vagrant. Creates and manages virtual machines – that run the Docker daemon for you. Use Docker or Docker Compose on Windows or Mac OS → command will be executed in virtual machine → Example
docker-machine ls
docker-machine create
docker-machine create
Docker Machine
Using Docker on Linux, Mac and Windows
• Linux works best• Mac OS works good
– Use docker-machine-nfs for better performance• Windows is (imho) a bit awkward to use• Always get your file system permissions right!
Presenter
Presentation Notes
Our experience Native Linux machines works best. Docker on Mac OS X works good. Windows is a bit awkward to use – mostly because of file system incompatibilities and missing Unix tools. Improve the performance on Mac OS – use docker-machine-nfs Enabled NFS share Faster than VirtualBox standard way Regardless of OS Always get your file-system permissions right
We will dockerize Magento 2 (on a Mac) Steps we will perform: Create a new Magento project Install our dockerized-magento2 package, which we prepared Run the project
Tool check
Presenter
Presentation Notes
Time check → go to slide 80 Tools used Docker Machine Docker Docker Compose Git git-flow Composer
docker-machine version
docker-machine ls
docker version
docker-compose version
git version
git flow version
composer --version
composer create-project
Presenter
Presentation Notes
We will now create a new project using Composer Should look familiar to Magento 2 developers Magento repository is used Username and password from Magento website (log in and generate them)
composer create-project
Presenter
Presentation Notes
... afterwards we open the project folder...
git flow init
Presenter
Presentation Notes
... and turn the project into a Git repository using git-flow...
git commit
Presenter
Presentation Notes
... next we commit everything so that we can perform our local changes
git flow feature start
Presenter
Presentation Notes
... we create a new feature branch in which we will work...
Install arvatoscm/dockerize-magento2
Presenter
Presentation Notes
... and now the most important step: We add a Composer package to dockerize Magento 2 It is called arvatoscm/dockerize-magento2 Creative name, uh?
Adapt .gitignore
Presenter
Presentation Notes
The .gitignore file coming with Magento 2 ignores some of our new files → In order to track them, .gitignore needs to be changed
Review installed files
Presenter
Presentation Notes
... and then we can see the new files in our project Let‘s have a closer look...
docker-compose.yml
Presenter
Presentation Notes
A docker-compose.yml More complicated than our examples before, because more components But same structure
composer.json
Presenter
Presentation Notes
The composer.json with the new package in it...
config folder
Presenter
Presentation Notes
A config folder, e. g. with files for Nginx MySQL
config folder
Presenter
Presentation Notes
... and a customized PHP.ini because the one from the Docker standard PHP image is not optimized for Magento 2
bin/console script
Presenter
Presentation Notes
... and last but not least: A utility script for controlling the dockerized infrastructure
Make bin/console executable
Presenter
Presentation Notes
We need to make the scripts executable
bin/console
Presenter
Presentation Notes
Let‘s run the console script: Actions Starting and stopping the infrastructure Action to trigger the Magento 2 installation (Alternative: Use Docker commands to perform the same steps) → And that‘s all we need to dockerize Magento 2
Install Magento 2
Presenter
Presentation Notes
Let‘s run the install action...
Install Magento 2: Generate SSL Certificates
Presenter
Presentation Notes
It will generate self-signed SSL certificates... if you haven‘t placed your own certificates in the config/nginx folder
Install Magento 2: Start containers
Presenter
Presentation Notes
... it then starts the Docker containers for which the images are available on the host already...
Install Magento 2: Download Docker Images
Presenter
Presentation Notes
... download missing images...
Install Magento 2: Trigger Magento Installer
Presenter
Presentation Notes
... and start the Magento 2 installer with Given domain name Database credentials from the config files etc. It should look familiar for Magento 2 developers
Install Magento 2: Display Shop URLs
Presenter
Presentation Notes
At the end, the script displays Frontend URL Backend URL Backend credentials
Open Magento Frontend & Backend
Presenter
Presentation Notes
We can now open the frontend ... and the backend → A fully dockerized Magento 2 instance is born And now comes the best: If you find that interesting, you can try it on your own!
Try it yourself
1. Installarvatoscm/dockerize-magento2
2. Have fun with Magento & Docker
Presenter
Presentation Notes
We open sourced this setup You can find the Composer package at Packagist It is called „arvatoSCM/docker-magento2“ Code at github.com/arvatoSCM/dockerize-magento2 Play with it and have fun! We are happy about any comments or pull requests. And then, all that is left to do, is...
Questions
Feedback, questions and pull-requests are welcome.
Thank you.
github.com/arvatoSCM/dockerize-magento2
Presenter
Presentation Notes
Thank you for your attention on how to scale Magento development processes with Docker If you have questions, I am happy to answer them now or later via internet
