gitlab-ci and docker registry - froscon...about introduction gitlab 101 deploying on-premise known...

ABOUT INTRODUCTION GitLab 101 Deploying on-premise Known issues END

GitLab-CI and Docker Registry

Oleg Fiksel

Security Consultant @ CSPI GmbH

[email protected] | [email protected] | Matrix: @oleg:fiksel.info

FrOSCon 2017

mailto:[email protected]



AGENDA

ABOUT

INTRODUCTION

GitLab 101

Deploying on-premise

Known issues

END

Q & A


ABOUT ME

I Security Consultant @ CSPI 1 (former MODCOMP 2)I Main topics

I Architecture

I Development cycle

I Perl Coding

1About CSPi2Wikipedia: MODCOMP

http://www.cspi.com/de/uber-uns/

https://en.wikipedia.org/wiki/MODCOMP


GOALS OF THIS TALK

I This is not a comparision of CI toolsI Provide an overview of dependencies needed to deploy

GitLab-CI Community Edition and Docker Registryon-premise

I Disclamer: The means and methods presented are my ownexpirience


GOALS OF THIS TALK

I This is not a comparision of CI tools

I Provide an overview of dependencies needed to deployGitLab-CI Community Edition and Docker Registryon-premise



GOALS OF THIS TALK

I This is not a comparision of CI toolsI Provide an overview of dependencies needed to deploy

GitLab-CI Community Edition and Docker Registryon-premise



GITLAB 101


WHAT IS GITLAB?

I Web-based Git repository manager and more...I Started as a pet-project in 2011 and now has more then 150

employeesI Introduced Pipelines (CI) in version 8.8 (2016-05-28)I GitLab is used by many organisations such as: IBM, Sony,

NASA, Alibaba, SpaceX and CSPi


WHAT IS GITLAB?

I Web-based Git repository manager and more...

I Started as a pet-project in 2011 and now has more then 150employees

I Introduced Pipelines (CI) in version 8.8 (2016-05-28)I GitLab is used by many organisations such as: IBM, Sony,



WHAT IS GITLAB?


employees

I Introduced Pipelines (CI) in version 8.8 (2016-05-28)I GitLab is used by many organisations such as: IBM, Sony,



WHAT IS GITLAB?


employeesI Introduced Pipelines (CI) in version 8.8 (2016-05-28)

I GitLab is used by many organisations such as: IBM, Sony,NASA, Alibaba, SpaceX and CSPi


WHAT IS GITLAB?


employeesI Introduced Pipelines (CI) in version 8.8 (2016-05-28)I GitLab is used by many organisations such as: IBM, Sony,



WHAT IS DOCKER?

client

docker build

docker pull

docker run

docker host

docker daemon

imagescontainers

registry

...


WORDING

I GitLab Server: git repository hosting serviceI GitLab-CI Runner: user-space daemon that executes

build/testsI Artifacts: build results pushed into an internal GitLab

storageI GitLab Container Registry: integrated docker registry

frontendI Docker Registry: mandatory container registry service


WORDING

I GitLab Server: git repository hosting service

I GitLab-CI Runner: user-space daemon that executesbuild/tests

I Artifacts: build results pushed into an internal GitLabstorage

I GitLab Container Registry: integrated docker registryfrontend

I Docker Registry: mandatory container registry service


WORDING


build/tests

I Artifacts: build results pushed into an internal GitLabstorage




WORDING



storage




WORDING




frontend



WORDING




frontendI Docker Registry: mandatory container registry service


DEPLOYING ON-PREMISE


CHECKLIST

I 2 VMs or Rancher/Kubernetes/Mesos clusterI Reverse proxy/loadabalancer for SSL offload (optional)

supporting HTTP 1.1 to the backend (! Lighttpd)I Direct internet connection (for pulling docker images)I SSL Certificates (own CA or official)


CHECKLIST

I 2 VMs or Rancher/Kubernetes/Mesos cluster

I Reverse proxy/loadabalancer for SSL offload (optional)supporting HTTP 1.1 to the backend (! Lighttpd)

I Direct internet connection (for pulling docker images)I SSL Certificates (own CA or official)


CHECKLIST


supporting HTTP 1.1 to the backend (! Lighttpd)

I Direct internet connection (for pulling docker images)I SSL Certificates (own CA or official)


CHECKLIST


supporting HTTP 1.1 to the backend (! Lighttpd)I Direct internet connection (for pulling docker images)

I SSL Certificates (own CA or official)


CHECKLIST


supporting HTTP 1.1 to the backend (! Lighttpd)I Direct internet connection (for pulling docker images)I SSL Certificates (own CA or official)


PITFALLS

I Internal CA

I Forward proxy

I DNS split horizon (not handled in this talk)


GITLAB-CI RUNNER ARCHITECTURE

GitLab-CI GitLab-CI-RunnerDocker

GitLab-CI-RunnerShell

Container

Container

Container

GitLab-CI-RunnerGitLab-CI-Runner

GitLab-CI-Runner


ON-PREMISE DEPLOYMENT ARCHITECTURE

Docker Container

GitLab

GitLab-CI

Auth

Docker registry(frontend)

Artifacts

GitLab-CI Runner

git clone

run

Test, Build, etcpush (HTTPS)

Docker registry(container)push/pull (HTTPS)

localS3

AzureGCSSwiftstore blob

read/write access

Docker client

pull/push (HTTPS)

auth token (HTTPS)

[separate CA]

auth (HTTPS)

hub.docker.com

Pull (HTTPS)


INTERNAL CA

Every GitLab HTTPS client must trust internal CA including:

I gitlab-ci-runnerI docker container building docker images


INTERNAL CA

Every GitLab HTTPS client must trust internal CA including:I gitlab-ci-runner

I docker container building docker images


INTERNAL CA

Every GitLab HTTPS client must trust internal CA including:I gitlab-ci-runnerI docker container building docker images


INTERNAL CA

I Problem: docker images are pulled from docker hub anddoesn’t trust intern CA.

I Solution: extend all base images with internal CA and usethem for building.


SWITCH DOCKER STORAGE BACKEND

By default, when using docker:dind, Docker uses the vfs storagedriver which copies the filesystem on every run. This is a very

disk-intensive operation which can be avoided if a different driver isused, for example overlay.1

1Source

https://docs.gitlab.com/ce/ci/docker/using_docker_build.html



OS Setup:

I add overlay to /etc/modules (Ubuntu 16.04)I modprobe overlay or reboot the system



OS Setup:I add overlay to /etc/modules (Ubuntu 16.04)

I modprobe overlay or reboot the system



OS Setup:I add overlay to /etc/modules (Ubuntu 16.04)I modprobe overlay or reboot the system



Adjust /etc/docker/daemon.json1 {2 " storage−dr iver " : " overlay "3 }

and restart Docker.

Warning: make sure you have no important local images orcontainers. You will start with an empty Docker storage.


INTERNAL CA - BOOTSTRAP PROCEDURE

I Adjust runner configurationI Build docker first docker images locally and push them to

the registryI Create CI configuration and build images automaticallyI Update images daily using scheduled builds (CI feature)



I Adjust runner configuration

I Build docker first docker images locally and push them tothe registry

I Create CI configuration and build images automaticallyI Update images daily using scheduled builds (CI feature)




the registry

I Create CI configuration and build images automaticallyI Update images daily using scheduled builds (CI feature)




the registryI Create CI configuration and build images automatically

I Update images daily using scheduled builds (CI feature)




the registryI Create CI configuration and build images automaticallyI Update images daily using scheduled builds (CI feature)



Adjust runner configuration:1 # / e t c / g i t l a b−runner/conf ig . toml2 [ [ runners ] ]3 . . .4 executor = " docker "5 [ runners . docker ]6 . . .7 p r i v i l e g e d = true8 volumes = [ "/ cache " , "/ var/run/docker . sock :/ var/run/docker .

sock : rw " ]


INTERNAL CA - DOCKER IMAGE

Dockerfile for Docker image with internal CA:

1 # D o c k e r f i l e2 FROM docker : l a t e s t3

4 COPY my_ca . c r t /tmp/5 RUN c a t /tmp/my_ca . c r t >>/ e t c / s s l / c e r t s /ca−c e r t i f i c a t e s . c r t &&

rm /tmp/my_ca . c r t6

7 ENTRYPOINT [ " docker−entrypoint . sh " ]8 CMD [ " sh " ]



Dockerfile for Docker image with internal CA:1 # D o c k e r f i l e2 FROM docker : l a t e s t3



7 ENTRYPOINT [ " docker−entrypoint . sh " ]8 CMD [ " sh " ]



CI configuration for Docker image with internal CA:

1 # .gitlab-ci.yml2 v a r i a b l e s :3 DOCKER_DRIVER: overlay4 IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME5

6 b e f o r e _ s c r i p t :7 - docker log in −u g i t l a b−c i−token −p $CI_JOB_TOKEN

$CI_REGISTRY8

9 build_docker_image:10 s tage : bui ld11 image: $CI_REGISTRY/ g i t l a b−c i /docker:master12 s e r v i c e s :13 - $CI_REGISTRY/ g i t l a b−c i /dind:master14 tags :15 - dind16 s c r i p t :17 - docker bui ld −t $IMAGE_TAG .18 - docker push $IMAGE_TAG



CI configuration for Docker image with internal CA:1 # .gitlab-ci.yml2 v a r i a b l e s :3 DOCKER_DRIVER: overlay4 IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME5


$CI_REGISTRY8



INTERNAL CA - DOCKER-IN-DOCKER IMAGE

Dockerfile for Docker-in-Docker image with internal CA:

1 # D o c k e r f i l e2 FROM docker : dind3



7 VOLUME /var/ l i b /docker8 EXPOSE 23759

10 ENTRYPOINT [ " dockerd−entrypoint . sh " ]11 CMD [ ]



Dockerfile for Docker-in-Docker image with internal CA:1 # D o c k e r f i l e2 FROM docker : dind3



7 VOLUME /var/ l i b /docker8 EXPOSE 23759

10 ENTRYPOINT [ " dockerd−entrypoint . sh " ]11 CMD [ ]



CI configuration for Docker-in-Docker image with internal CA:



$CI_REGISTRY8




CI configuration for Docker-in-Docker image with internal CA:1 # .gitlab-ci.yml2 v a r i a b l e s :3 DOCKER_DRIVER: overlay4 IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME5


$CI_REGISTRY8



INTERNAL CA - BUILDING IMAGES

Now we can build Docker images with GitLab-CI!



$CI_REGISTRY8



INTERNAL CA - BUILDING IMAGES

Now we can build Docker images with GitLab-CI!1 # .gitlab-ci.yml2 v a r i a b l e s :3 DOCKER_DRIVER: overlay4 IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME5


$CI_REGISTRY8



FORWARD PROXY

I Not every application have proxy supportI Some application configuration is trickyI Configuring proxy every time bloats CI configurationI Set proxy configuration via environmental variables while

integrating your CA in the docker image


FORWARD PROXY

I Not every application have proxy support

I Some application configuration is trickyI Configuring proxy every time bloats CI configurationI Set proxy configuration via environmental variables while



FORWARD PROXY

I Not every application have proxy supportI Some application configuration is tricky

I Configuring proxy every time bloats CI configurationI Set proxy configuration via environmental variables while



FORWARD PROXY

I Not every application have proxy supportI Some application configuration is trickyI Configuring proxy every time bloats CI configuration

I Set proxy configuration via environmental variables whileintegrating your CA in the docker image


FORWARD PROXY

I Not every application have proxy supportI Some application configuration is trickyI Configuring proxy every time bloats CI configurationI Set proxy configuration via environmental variables while



FORWARD PROXY - LOCAL TRANSPARENT PROXY

For applications not supporting proxy−→ local squid in tranparent mode (doesn’t work for HTTPS)

1 # squid c o n f i g u r a t i o n2 a c l docker s r c 1 7 2 . 1 7 . 0 . 0 / 1 63 a c l SSL_ports port 4434 cache_mem 16 MB5 # upstream proxy ip6 cache_peer 1 0 . 0 . 0 . 1 0 parent 8080 0 no−query proxy−only d e f a u l t7 d n s _ v 4 _ f i r s t on8 h t t p _ a c c e s s allow docker9 h t t p _ a c c e s s deny CONNECT ! SSL_ports

10 h t t p _ a c c e s s deny ! Sa fe_por t s11 ht tp_por t 3129 i n t e r c e p t12 memory_pools o f f


FORWARD PROXY - LOCAL TRANSPARENT PROXY

iptables configuration:1 i p t a b l e s −t nat −A PREROUTING −s 1 7 2 . 1 7 . 0 . 0 / 1 6 −p tcp −m tcp −−

dport 80 − j REDIRECT −−to−ports 3129


KNOWN ISSUES


GITLAB-CI WITH SUBMODULES

Submodule init failing due to "SSL certificate problem".f a t a l : unable to a c c e s s ’ h t tps :// github . com/minio/minio−go / ’ :

SSL c e r t i f i c a t e problem : unable to get l o c a l i s s u e rc e r t i f i c a t e

I Issue: 2148I Will be fixed in gitlab-ci-multi-runner v9.4

https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/2148





I Issue: 2148

I Will be fixed in gitlab-ci-multi-runner v9.4






I Issue: 2148I Will be fixed in gitlab-ci-multi-runner v9.4



GIT-LFS

Git Large File Storage (LFS) replaces large files such as audiosamples, videos, datasets, and graphics with text pointers inside Git,

while storing the file contents on a remote server.1

1https://git-lfs.github.com

https://git-lfs.github.com


GIT-LFS

I Problem: GitLab-CI doesn’t download git-LFS objects onCI run (probably fixed by now)

I Workaround: download git-LFS objects “manually” via CIscript


GIT-LFS

1 # .gitlab-ci.yml2 s t a g e s :3 − bui ld4

5 create_package:6 s tage : bui ld7 image: $CI_REGISTRY/ g i t l a b−c i /ubuntu: x e n i a l8 s c r i p t :9 - apt−get update && apt−get i n s t a l l −y wget g i t

10 - wget ht tps ://packagecloud . io/github/g i t−l f s /packages/ubuntu/x e n i a l /g i t−l f s _ 1 . 5 . 2 _amd64 . deb/download −O /tmp/g i t−l f s _ 1. 5 . 2 _amd64 . deb && dpkg −i /tmp/gi t−l f s _ 1 . 5 . 2 _amd64 . deb

11 - g i t l f s i n s t a l l && g i t l f s f e t c h && g i t−l f s checkout12 - t a r c z f appl i c a t ion −‘ c a t a p p l i c a t i o n /vers ion . t x t ‘ . t a r . gz

a p p l i c a t i o n13 a r t i f a c t s :14 e x p i r e _ i n : 2 weeks15 paths:16 - appl i c a t ion−* . t a r . gz17 only:18 - /^ r e l e a s e . * $/


SUMMARY

I GitLab is a great product evolving rapidly

I Deploying GitLab-CI in an enterprise environment can bequite challenging

I Some of use cases and videos are focused on frontenddevelopment using Ruby-On-Rails and deployment to aKubernetes cluster


Q & A


Thanks!

Oleg [email protected] | [email protected] | Matrix: @oleg:fiksel.info




LINKS

I Files from this talk on GithubI Introduction to GitLab pipelinesI Install a root CA in Ubuntu

https://github.com/oleg-fiksel/Setting_up_GitLab-CI_und_Docker_Registry

https://docs.gitlab.com/ee/ci/pipelines.html

https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate

gitlab-ci and docker registry - froscon...about introduction gitlab 101 deploying on-premise known...

Documents