be simple. be boring. be obvious

KRIS KÖHNTOPP · THURSDAY, SEPTEMBER 1, 2016 163 Reads

Code is not your friend.

On Core.Infra day, I was invited to speak. This is my talk. There were many like it, but this

was mine.

In operations, code is not your friend.

This is about what I did the two years I have not been around here.

I first worked for Booking in 2006

as a MySQL consultant doing

databasey things. I joined Booking

as an employee later, in 2008. By

then, I was working in

Amsterdam, but living in Berlin,

so I was flying around quite a bit.

In 2014, that became unworkable

and I signed on with a Berlin

based eCommerce hoster to do

some awesome stuffs.

Edit Note

SysEleven has about 3000 customers, mostly German eCommerce or news sites. The company had less then 50 employees in 2014 and grew to 70 people now. I joined a newly formed team, whose task was to build the new hosting platform for SysEleven.

Given that we were to build the new platform, there has to be an old one. The old platform was based on single machines with all storage local, and maybe a NetApp or two mounted from external. Hosting was done on OpenVZ based containers, and the Underlay (Hypervisor) was generally managed by puppet. The Overlay (contents of the container), too, mostly. But due to limitations in Puppet, architectural decisions of the past and the nature of the workload imposed by customers that was not done consistently and completely. There was a lot of fuzz around the edges, and somethings were completely on manual.

The new platform as based on Openstack, with a software designed storage solution, and a software designed network solution underneath, no local storage on the machine (all bytes always come from the SDS).

Underlay and Overlay were to be completely controlled by Puppet,

and in fact, the actual installation process was to be done by Puppet as well.

We actually succeeded in doing this.

So you have several racks full of HP DL380s, all powered off and cold. The bootstrap talks to the iLOs, turns on the hardware, inventories everything, zaps fresh ROMs and iLOs onto everything, and then paints a base operating system onto the machines that is just enough to bootstrap Puppet. Puppet then continues to install

and personalise the machines and build an Openstack from this, including SDS and SDN cluster builds before the Openstack install.

The total installation time was 30-45 minutes, if everything went well, but usually not everything went well. :-) Also, Puppet as a tool is extremely focused on managing a single host and does not work well with the concept of a cluster. If you want to build such a thing, you end up coding a lot of stuff against the internal logic of Puppet, using Stages to have synchronisation points, or invoking very ugly hacks written as external Python scripts that sync up the cluster before allowing everybody to continue to the next phase.

I learned that there is a place that can genuinely and rightfully be called Puppet Hell.

(Isotopp)Kris Köhntopp

Notes

Notes Drafts

Kris KöhntoppSeptember 14 at 3:19pm ·

HEAT - a declarative language

for virtualized infrastructure

description

HEAT is the declarative language Openstack uses todescribe a deployment of anything. This articlestrives to transport the concepts behind it in orderto give us an idea what is possible.

What we can learn from it

I am showing this as a relatively concrete examplein Openstack, despite the fact that we are not usingOpenstack and hopefully never will.

What I want to show is what a deploymentspecification can look like, declaring resourceusage, network connections, even networkelements provided through a SDN. Similar syntaxexists for storage, databases, and other elements ofthe Openstack uni...

...

See More

2Anastasios Pingios and Hans Tool 1 Share

Comment Share

Kris KöhntoppSeptember 1 ·

Code is not your friend.

On Core.Infra day, I was invited to speak. This ismy talk. There were many like it, but this was mine.

This is about what I did the two years I have notbeen around here.

I first worked for Booking in 2006 as a MySQLconsultant doing databasey things. I joinedBooking as an employee later, in 2008. By then, Iwas working in Amsterdam, but living in Berlin, soI was flying around quite a bit. In 20...

...

See More

19

JP Schoombee, Ugur Basak and 17 others

1 Comment 2 Shares

Comment Share


Kris KöhntoppJuly 7 ·

Add Note

Timeline About Followers Photos More

View Activity Log

Like

Like

Home 11

KrisSearch Booking.com groups and more

We made the decision to not write Puppet modules when we could reuse external work. That turned out to be a not so smart solution. We imported external repositories into our internal Gitlab, and in order to structure things, we set up a bunch of different segregated areas, teams. In the end we had about 200 repos, in six different teams.

That led to a lot of problems. We did not set out to have these problems, it just kind of inevitably happened.

200 Repos is quite a lot, and even with tooling you end up making mistakes. Some people will not be able to see some repos, not because they should not, but

because of mistakes. Some things are done manually, others are done automatically, but rules change or team members move, but the rules are not adjusted and other stuff.

Anyway, you inevitably end up in a situation where you are making a change and you are not aware of all the consumers of the change - either because you can’t see them, or because you did not bother to check out that repo, or because your search did not find them. Not fixing your consumers will break them, and hence the build. That is annoying.

You also can’t make your changes atomically. You may make your change at the source, and merge it, but you still need to merge and push the 13 consumers of your change, and that can take time, or fail. Until you complete the merge in all Repos, nobody in the team can roll out. That sucks.

Finally, if visibility is limited, it may be that two people are producing or importing duplicate solutions for a problem. Worse, it may be that these duplication are actually duplicate external imports, but at different version requirements. That makes it very hard to manage project dependencies and size.

We are doing Puppet, so what is it that we import when we import stuff from Puppetlabs, or github?

Well, mostly generalised configuration management classes for Puppet.

What is that?

Most configuration files are actually Hashes of Hashes with a unique kind of syntactic sugar. MySQL my.cnf files, Bind configuration, Openstack ini-Files, and so on - the are all HoH’s.

expire-mailHow do you handle mailboxes that have potentialto overflow, and where you do not necessarily haveto review each and every message?

[:~]↥ $ expire-mail

From folder INBOX/lists/live-updates before 26-Aug-2016 delete 6 messages: .

...

See More

3Sara Santos, Dominic Montanez and Nelson Ferraz 1 Comment

Comment Share

Let’s have a chat - a taxonomyand some contextThe first chat network I have been using was IRC -Internet Relay Chat. This has been in the late 80iesor early 90ies, so I now have had an interactiveonline presence on the upside of 25 years or so.

Later I got to use ICQ, Jabber, Skype, GoogleHangouts, Facebook Messenger, WhatsApp, Slackand RocketChat among many others. I also gotexposed to a number of collaborative documentprocessing syste...

...

See More

9

Michael Schlesiger, Mike Hamrick and 7 others

2 Comments 1 Share

Comment Share

Kris KöhntoppJune 23 ·

Adventures in ServerDBIt started with a rather innocent question: »Kris,can you find out how many servers of each kind wehad on our datacenters in the past, so that we canplot growth for planning.« So I looked at theServerDB API, laughed cold and hard and lookedinto the ServerDB database.

ServerDB is a Django instance with severaldifferent applications in a single installation. TheMySQL tables are to be found in ...

...

See More

16

Kayla Meduna, Luciano Rocha and 14 others

15 Comments 2 Shares

Comment Share


About the cloudBetween April 2014 and April 2016 I have beenworking for a german hosting company namedSysEleven, building a hosting grade Openstackcluster with them. This is a talk I gave on the OpenSource Congress of Deutsche Bahn AG (GermanRail) in November 2015, summarising a few thingsI learned in those two years.

Some of that also applies to our environment, someof it doesn’t. Particularly, unlike a h...

...

See More

16

German Gutiérrez, Walter Aprile and 14 others

3 Comments 1 Share

Comment Share


Outlook mangles code in mymailsIt seems that the auto-correct keeps coming back,and it's turning my minus,greaterthan's into awindings arrow, but only in some mails, some ofthe time for some reason.

There are multiple places that could be doing this,and they all need individual turning off,unfortunately.

System Preferences → Keyboard

...


Riding a Scooter in theNetherlandsI have a Peugot Elystar (50cc) scooter, which I wasriding in Berlin. This is the type of vehicle which issupposedly limited at 45 km/h and which inGermany you ride with an insurance plate and forwhich you do not pay car tax.

Traffic rules

The rules for riding these things in the Netherlandsare at VVN. That’s the site that has all the trafficrules. The particular page for Scooters is at Brom-e...

Like

Like

Like Like

Chat (43)

Home 11



Notes

Notes Drafts


HEAT - a declarative languagefor virtualized infrastructuredescriptionHEAT is the declarative language Openstack uses todescribe a deployment of anything. This articlestrives to transport the concepts behind it in orderto give us an idea what is possible.




...

See More


Comment Share


Code is not your friend.On Core.Infra day, I was invited to speak. This ismy talk. There were many like it, but this was mine.



...

See More

19


1 Comment 2 Shares

Comment Share



Add Note


View Activity Log

Like

Like

Home 11


So what you get is a YAML-shovel that will generate the HoH that is, say, an Apache vHost-definition from the Hash of Hashes inside Puppet variables. Which in turn are being loaded from the Hash of Hashes that is a Hiera YAML file.

When you want to make a config change, you can’t edit a native Apache config file. You need to read the code, seeing how the Puppet in-memory HoH is being transformed into the actual Apache config. Then you need to look further back and understand how the twelve dozen Hiera files actually overlay each other to form these Puppet structures, and inject your change in the right place.

Then you need to try this out and actually see what config is being generated.

You need a lab. For a config change.

That is because your config and your config generation became so complicated that it is no longer obvious. It is no longer obvious what that code does, and it is no longer obvious what your change does.

In our case the test environment is an entire datacenter, the vanguard data center. That is clearly insane.

How did we end up in this place?

Let me introduce The Configuration Complexity Clock.

You write a piece of actual code, in a proper programming language. Values are pieces of code, constants. They are baked in. It is noon on the config clock.

Things change, values need to change. You need different values in different places, so you load variables from a file, the config file. It is now 3.

Somebody needs more than that - there are rules. “Some rooms are only available at some price if the stay includes a Saturday” is such a rule. Or “The amount of memory we give to the Innodb buffer pool is all machine memory minus 20% or 4GB, whichever is smaller, unless it’s a mz box with all that Blob stuff going on.” is another such rule. It is now 6 on the config clock.

Of course some things can no longer be expressed as rules. You need a fully blown language with data structures, control structures, loops and include files. You get a config management DSL. Something like puppet. It’s 9 on the config clock.

»The team spend most of their time writing code in the new DSL.

See More

2Rory Michaelis and Daniël van Eeden 2 Comments

Comment Share

e...

...

See More

2 Comments 1 Share

Comment Share

Kris KöhntoppMay 27 ·

10 years agoI just found an old file from the very first consultingengagement I had with Bookings.nl. This would bein August 2006 or maybe October 2006.

At that time I had been a Database Consultant forMySQL Germany GmbH and had been asked byHerald van der Breggen to have a look at theconfiguration of this small travel agency inAmsterdam, which resided in two floors atWeteringschans 28 in Amsterdam.

From...

...

See More

16

Jeroen van Wolffelaar, Sean Cutrufello and 14 others

1 Comment 1 Share

Comment Share


No Outlook No MoreSo Microsoft Outlook decided that it wants a Coreof its own on my Mac. Fans spin up and never stopand the battery drains in no time. I made that aticket. Yet what do do in the meantime until this isfixed?

Well, if you go to Preferences, Apple promisesExchange:

Let’s try that, then. If you [+] that, you’ll get a loginscreen that does not look too intimidating:

...

See More

9Artur Burtsev, Jeremy Cornuau and 7 others 6 Comments

Comment Share


Who do you want to be today?Chrome Browser ProfilesexplainedKeeping work and private accounts separated iseasy in Google Chrome, and allows you amongother things to be logged into Facebook @ Workand private Facebook at the same time.

What is this good for?

Chrome Profiles are basically “virtual browsers”,like different installations altogether. They get theirown set of password stores, chrome plugins,cookies and stuff. This means that you can create a...

...

See More

3

Andrey Tokarev, Martin de Koning and John Loemban Tobing

1 Comment 2 Shares

Comment Share


Puppet: Manifest DestinyIn our puppet, I see a lot of code that looks like this(or similar):

if ($i) {

notify { 'one': }

...

See More

4Martin de Koning, Walter Aprile and 2 others 7 Comments

Comment Share

Like

Like

Like

Like

Like

Like

Chat (43)

Home 11





...

See More


Comment Share



...

See More

9


2 Comments 1 Share

Comment Share




...

See More

16



Comment Share




...

See More

16


3 Comments 1 Share

Comment Share





...



Traffic rules


Like

Like

Like Like

Chat (43)

Home 11



Notes

Notes Drafts






...

See More


Comment Share





...

See More

19


1 Comment 2 Shares

Comment Share



Add Note


View Activity Log

Like

Like

Home 11


After some embarrassing episodes, they now go through a complete release cycle before deploying new DSL code.

The DSL text files are version controlled and each release goes through regression testing before being deployed.«

That’s SysEleven, doing puppet. That’s us, doing Puppet - if not now, then it’s going to be us very soon.

That is, because what we are doing is no longer obvious, so we need to check the presence and type of parameters to our Puppet classes.

We need to write tests that check the validity and content of the subconfigs generated.

We then need to do integration testing, in order to prevent catastrophic loss of the entire site from centralised config changes.

At SysEleven, when we were at that point, we stopped. We threw away our complete stack of “Setup Openstack from scratch” and started over.

We chose a different config management system, in our case that was Ansible, but that is not really important.

We did choose something else than Puppet, mainly so that we were unable to directly reuse any component we previously wrote.

We did come up with some rules in order to prevent ourselves from descending into Puppet Hell again. Our goal was to enforce simplicity, and obviousness.

We did switch to a Monorepo, at least as much as possible. There were other, external dependencies with other parts of the company that prevented us from going completely Monorepo, but it kind of worked well enough for us.

We did use Ansible, and while the actual choice of the config management system officially did not matter as long as it is not Puppet, Ansible does have a

Chat (43)

Home 11


See More


Comment Share

e...

...

See More

2 Comments 1 Share

Comment Share




From...

...

See More

16


1 Comment 1 Share

Comment Share





...

See More


Comment Share





...

See More

3


1 Comment 2 Shares

Comment Share



if ($i) {

notify { 'one': }

...

See More


Comment Share

Like

Like

Like

Like

Like

Like

Chat (43)

Home 11





...

See More


Comment Share



...

See More

9


2 Comments 1 Share

Comment Share




...

See More

16



Comment Share




...

See More

16


3 Comments 1 Share

Comment Share





...



Traffic rules


Like

Like

Like Like

Chat (43)

Home 11



Notes

Notes Drafts






...

See More


Comment Share





...

See More

19


1 Comment 2 Shares

Comment Share



Add Note


View Activity Log

Like

Like

Home 11


number of advantages that Puppet does not have. One of them being that it deals much better

with distributed software than Puppet can do, and the other, much more important one being

that it is very limited.

It is impossible to do complicated things in Ansible. At some point, if Ansible is not sufficient,

you need to stop ansibilizing and switch to Python. That’s a good point to raise the issue with

the team and asking around what’s actually the problem and how an Ansible extension

written in Python is going to make this better.

This is good, because it gives you a point where reflection and goal-setting are in order, and a

hook where project management processes can be invoked.

We also looked at our YAML shovels and asked ourselves what they are good for. We found

that they are overgeneralised solutions for our problems. We do not need to be able to write

any conceivable Apache config from our config management code, we only need to be able to

write our Apache config from it.

Or, more generally:

I cannot stress the importance of that enough. It was at the core of all of our problems.

We decided to write config files directly. If there is an Apache config, write that Apache config

file in Apache config file syntax into the Ansible template directory, and install it as literally

as possible. Changes will be obvious in what they do.

Do not use variables, unless you can show that in our config there is actually variation.

We did catch ourselves in review in anticipating problems we didn’t have, introducing

complexity we didn’t need. Good thing we did establish review, because every single team

member failed at being as simple as possible, multiple times.

We did have a few cases were it was in order to write code, remove duplication and generate

stuff in a loop or similar things, but these cases were few and far between.

Our new setup was an order of magnitiude smaller and much, much easier to understand.

We did bring in external people and tried to explain to them what we did, and how things

work, and the new setup was so much easier to work with that this was actually possible.

Feature development is about generalisation, avoiding limits, delaying

decisions and being more flexible.

In�astructure development is about being concrete, making decisions, setting

limits and being obvious.

Chat (43)

Home 11


Chat (43)

Home 11


See More


Comment Share

e...

...

See More

2 Comments 1 Share

Comment Share




From...

...

See More

16


1 Comment 1 Share

Comment Share





...

See More


Comment Share





...

See More

3


1 Comment 2 Shares

Comment Share



if ($i) {

notify { 'one': }

...

See More


Comment Share

Like

Like

Like

Like

Like

Like

Chat (43)

Home 11





...

See More


Comment Share



...

See More

9


2 Comments 1 Share

Comment Share




...

See More

16



Comment Share




...

See More

16


3 Comments 1 Share

Comment Share





...



Traffic rules


Like

Like

Like Like

Chat (43)

Home 11



Notes

Notes Drafts






...

See More


Comment Share





...

See More

19


1 Comment 2 Shares

Comment Share



Add Note


View Activity Log

Like

Like

Home 11


We did not need tests any more,

because changes were obvious.

Code that is not present does not

need to be tested in the first place.

So, looking at Booking: What’s in

it for us?

I do believe, after reading a lot of code in our Puppet and in other places, that Simplicity as a

value in itself would work well for us, too. Code isn’t our friend, either, especially in

Core.Infra and Operations related places. We are here to make decisions, not delay them, and

to be obvious, and as simple as we can get away with. Which is hard enough given the fact

that our size and growth force more and more distributed stuff on us, which by it’s very

nature has a high inherent complexity.

I think the entire container thing is a brilliant idea, and we should be doing it more.

A lot more.

Immutable containers, being created and then loading values from a Zookeeper or Consul,

instead of running Puppet on every host, painting state changes over state changes that may

or may not have been previously applied.

Eventually we will no longer have any need for Puppet at all any more, despite the fact that

we are a larger company that is doing a lot more distributed stuff than now.

Be simple. Be boring. Be obvious.

Because every time you aren’t,

you get to write a postmortem.

Comment Share

19JP Schoombee, Ugur Basak and 17 others

2 shares 1 Comment

Like

Chat (43)

Home 11


Chat (43)

Home 11


Chat (43)

Home 11


See More


Comment Share

e...

...

See More

2 Comments 1 Share

Comment Share




From...

...

See More

16


1 Comment 1 Share

Comment Share





...

See More


Comment Share





...

See More

3


1 Comment 2 Shares

Comment Share



if ($i) {

notify { 'one': }

...

See More


Comment Share

Like

Like

Like

Like

Like

Like

Chat (43)

Home 11





...

See More


Comment Share



...

See More

9


2 Comments 1 Share

Comment Share




...

See More

16



Comment Share




...

See More

16


3 Comments 1 Share

Comment Share





...



Traffic rules


Like

Like

Like Like

Chat (43)

Home 11



Notes

Notes Drafts






...

See More


Comment Share





...

See More

19


1 Comment 2 Shares

Comment Share



Add Note


View Activity Log

Like

Like

Home 11


Pat Lasswell "Be simple. Be boring. Be obvious." I want that on a t-shirt.

Unlike · Reply · 5 · September 1 at 7:45pm

Walter Aprile replied · 7 Replies

Write a comment...