deployment tools and techniques at spotify: virtualenv in debian by chris angove

1

Spotify Tech

Web/iOS/Android/Native clients Service oriented architecture > 50 services in Java and Python Primarily using Cassandra for persistent storage 7,000 Servers in production ~500 in Engineers worldwide

2

Lost in the forest of dependency building

What have we been doing?

3

Deploy via Debian Packages

Use Debian Stable repositories Package/Backport needed libraries as needed All Dependencies defined in Debian package Deployed via Puppet/Aptitude

4

Package  

Mirroring  

Apps  

PyPi  

5

Spending too much time managing dependencies and not enough building cool shit

What’s wrong with that?

6

Using Debian Stable does not included latest and greatest

7

Requires “Backporting” for newer libraries

8

Does not used normal python packaging

9

Limited use on developer systems

(or bad/duplicated information)

10

Full rollback non-existent

11

What you test is NOT what you deploy

12

Dependency lock in due to sharing a common python environment

13

Avoid duplication of effort

There MUST be a better way

14

Increase stability and repeatability

There MUST be a better way

15

Offer easy rollback

There MUST be a better way

16

Scale

There MUST be a better way

17

Deploying Python @ Scale

What about now?

18

Enter Dh-viurtualenv

19

Dh == Debian Helper

•  Registers itself as a Debian Helper •  Bundles the requirements in requirements.txt into virtual environment •  Deploys virtualenv into /usr/share/python/<package-name> •  Still requires native dependencies to be installed via packaging •  Just include the following in the debian/rules:

%:                                dh  $@  -­‐-­‐with  python-­‐virtualenv

20

Virtualenv  

Mirroring  

Apps  

Pip  

21

IT Works!

22

Example: Using Sentry

Define Dependencies in requirements.txt

sentry[postgres]==6.2.0  eventlet==0.13.0  hiredis==0.1.1  django-­‐auth-­‐ldap==1.1.4

23

Example: Using Sentry

Define Debian dependencies in control file

Build-Depends: python (>= 2.6.6-3~), debhelper (>= 8), dh-virtualenv, python-dev, libpq-dev, libldap2-dev, libsasl2-dev, Standards-Version: 3.9.3 X-Python-Version: >= 2.6

24

Example: Using Sentry

Use dh_virtualenv in Debian rules file

%: dh $@ --with python-virtualenv

override_dh_virtualenv:

dh_virtualenv --index-url='http://localhost/simple'

25

Have done this with RPMs as well (@ Amplify)

Within RPM build virtaulenv and mark it as relocatable:

/usr/local/bin/virtualenv --no-site-packages build_venv source build_venv/bin/activate env CFLAGS="$RPM_OPT_FLAGS" easy_install -Z . deactivate /usr/local/bin/virtualenv --relocatable build_venv

26

Benefits

No longer stuck on Debian stable

27

Benefits

No more backporting of new libraries or versions

28

Benefits

Code isolation/use own dependencies

29

Benefits

Single package rollback option

30

Benefits

Development and Deployed systems using same cross-platform setup

31

What’s Next?

32

Not Sure…

33

Dh_virtualenv now available (ironically) in Debian Unstable repositories

34

Still requires managing native dependencies locally

35

But experimenting with containers!

36

Stay Tuned!

37

Whitepaper: http://labs.spotify.com/2013/10/10/packaging-in-your-packaging-dh-virtualenv/ Code: https://github.com/spotify/dh-virtualenv

Thanks to Jyrki Pulliainen and Jim Whitehead @ Spotify Stockholm!

38

Chris Angove, Chapter Lead - Backend Email: cangove@spotify.com Blog: http://labs.spotify.com/ Listen: https://play.spotify.com/

Questions?

January 24, 2014

Check out spotify.com/jobs or @Spotifyjobs for more information.

Want to join the band?