the mozaic

The Mozaic

Mark Safronov (compiled only)

i

DedicationDedicated to all newbs in programming.

ii

Table of Contents

Preface ................................................................................................................... iiiYou’re just another carriage return line feed in the wall ........................................ 1What every programmer absolutely, positively needs to know about encodingsand character sets to work with text ..................................................................... 6Everything Is Broken ............................................................................................ 27Don’t go live with simple security problems - 10 tips to help .............................. 43Extreme Programming, a Reflection .................................................................... 47Surviving Legacy Code With Golden Master and Sampling .................................. 51Hexagonal Architecture ........................................................................................ 57Stop. Write a learning test ................................................................................... 74The Long, Painful History of Time ........................................................................ 78The make it work, make it right, make it fast misconception ............................ 103The Myth of Schema-less ................................................................................... 105NoDB .................................................................................................................. 1097 coding tasks you should probably not write yourself ...................................... 113You Need This One Skill to Succeed in IT .......................................................... 118ORM Is Not a Choice and How to Make It Suck Less .......................................... 120Programming Sucks ........................................................................................... 123Revenge of the Nerds ........................................................................................ 131Screaming Architecture ...................................................................................... 148How to securely hash passwords? ..................................................................... 151Smart Guy Productivity Pitfalls ........................................................................... 162TDD, Straw Men, and Rhetoric ........................................................................... 171Teach Yourself Programming in Ten Years ........................................................ 176Untested code is broken code: test automation in enterprise software delivery .. 184The Three Laws of TDD. .................................................................................... 193

iii

PrefaceThis book is a collection of essays from various authors and times. Those essays areabout different themes of the software engineering practice. When I was startingmy career as a software developer, I was reading a lot of books, blogs and chatdiscussions trying to catch up with all the experience accumulated in this profession.

Some articles were so groundbreaking to me then that I started to collect them overthe years. And now, here they are.

This "fan-made" book is completely non-commercial and has been made in good faiththat it hadn’t offended any of the authors. I didn’t ask any of the authors about thepermission to reproduce their text here. All articles are unchanged, with carefullypreserved layout, where it was possible.

I put this work under Creative Commons Attribution-NonCommercial-ShareAlike 3.0Unported license1. Some of articles probably were licensed under different terms,but I strongly believe CC BY-NC-SA 3.0 is a best possible common ground for anyof such articles.

Hope this collection will help somebody get a better understanding of our craft.

1 https://creativecommons.org/licenses/by-nc-sa/3.0/

https://creativecommons.org/licenses/by-nc-sa/3.0/



1

You’re just another carriage returnline feed in the wallWritten by: Scott Hanselman at 2013-02-201

I love getting pull requests on GitHub. It’s such a lovely gift when someone wantsto contribute their code to my code. However, it seems there are three kinds of pullrequests that I get.

1. Awesome, appreciated and wanted.

2. Not so good, thanks for trying, but perhaps another time.

3. THE WALL OF PINK

I’d like to talk about The Wall of Pink. This is a pull request that is possibly useful,possibly awesome, but I’ll never know because 672 lines (GitHub tells me) changedbecause they used CRs and I used LFs or I used CRLF and they used LF, or I used…well, you get the idea.

There is definitely a problem here. But what’s the problem? Well, it’s kind of likeendianness, except we’re still talking about it in 2013.

1 http://www.hanselman.com/blog/YoureJustAnotherCarriageReturnLineFeedInTheWall.aspx

http://www.hanselman.com/blog/YoureJustAnotherCarriageReturnLineFeedInTheWall.aspx

http://www.hanselman.com/blog/YoureJustAnotherCarriageReturnLineFeedInTheWall.aspx

You’re just another carriage return line feed in the wall

2

"A big-endian machine stores the most significant byte first—at thelowest byte address—while a little-endian machine stores the leastsignificant byte first."

— Wikipedia http://en.wikipedia.org/wiki/Endianness

Did you know for a long time Apple computers were big endian and Intel computerswere little endian? The Java VM is big endian. I wrote shareware code generator16 years ago that generated a byte array on an Intel PC that was later enteredinto a PalmPilot running a Motorola 68328. This was the last time I thought aboutendianness in my career. Folks working on lower-level stuff do think about thissometimes, admittedly, but the majority of folks don’t sweat endianness day to day..

TCP/IP itself is, in fact, big endian. There was a time when we had to really think aboutthe measurable performance hit involved in using TCP/IP on a little-endian processor.But we don’t think about that anymore. It’s there but the abstraction is not very leaky.

It’s years later, but CR/LF issues plague us weekly. That Wall of Pink I mentioned?It looks like this. I had to scroll 672 lines before I saw the +green where the addedlines were added. Who knows what really changed here though? Can’t tell since thisdiff tool thinks every line changed.

Sigh. Whose fault is this?

Perhaps we blame Émile Baudot2 in 1870 and Donald Murray3 in 1899 for addingcontrol characters to instruct a typewriter carriage to return to the home position

2 http://en.wikipedia.org/wiki/%C3%89mile_Baudot3 http://en.wikipedia.org/wiki/Donald_Murray_(inventor)

http://en.wikipedia.org/wiki/%C3%89mile_Baudot

http://en.wikipedia.org/wiki/Donald_Murray_(inventor)

http://en.wikipedia.org/wiki/%C3%89mile_Baudot

http://en.wikipedia.org/wiki/Donald_Murray_(inventor)


3

plus a line feed to advance the paper on the roller. Or we blame Teletype machines.Or the folks at DEC, or perhraps Gary Kidall and CP/M for using DEC as a convention.Then the bastards at IBM who moved to ASCII from EBCDIC and needed a carriagereturn when punch-cards fell out of favor.

The text files we have to day on Windows still have a CR LF (0D 0A) after every line.But Apple uses just uses a line feed (LF) character. There’s no carriage to return, butthere are lines to advance so it’s a logical savings.

Macs and PCs are sharing text more than ever. We live in a world where Git is FTPfor code, we’re up a level, above TCP/IP where Endianness is hidden, but still in textwhere CR LF’s aren’t.

We store our text files in different formats on disk, but later when the files arecommitted to Git, how are they stored? It depends on your settings and the defaultsare never what’s recommended.

You can setup a .gitattributes per repo to do things like this:

*.txt -crlf

Or you can do what GitHub for Windows suggests with text=auto .

# Auto detect text files and perform LF normalization* text=auto

What’s text=auto 4 do?

4 http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html

http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html

http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html


4

This ensures that all files that git considers to be text will havenormalized (LF) line endings in the repository. The core.eolconfiguration variable controls which line endings git will use fornormalized files in your working directory; the default is to use thenative line ending for your platform, or CRLF if core.autocrlf is set.

It uses the native line ending for your platform. But if you spend a few minutesgoogling around you’ll find arguments several ways with no 100% clear answer,although most folks seem to believe GitHub has the right one5.

If this is the right answer, why isn’t it a default? Is it time to make this the default?

This is such a problem that did you know GitHub for Windows has dedicated"normalize your repo’s CRLF" code? They’ll fix them all and make a one-time committo fix the line endings.

I think a more complete solution would also include improvements to the online difftool. If the GitHub repro and server knows something is wrong, that’s a great chancefor the server to suggest a fix, proactively.

SolutionsHere’s some possible solutions as I see it.

• Make Windows switch all text files and decades of convention to use just LF

• Git needs to install with correct platform-specific defaults withoutneeding .gitattributes file

• Have the GitHub web application be more proactive in suggesting solutions andpreventing badness

5 http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git

http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git

http://stackoverflow.com/questions/170961/whats-the-best-crlf-handling-strategy-with-git


5

• Have the GitHub for Windows desktop application proactively notice issues(before I go to settings) and offer to help

• Make the diff tool CR/LF aware and "do the right thing" like desktop diff tools thatcan ignore line ending issues

Until something is done, I’ll always tense up when I see an incoming pull request andhope it’s not a Wall of Pink.

6

What every programmer absolutely,positively needs to know aboutencodings and character sets to workwith textWritten by: David C. Zentgraf at 2013-08-051

If you are dealing with text in a computer, you need to know about encodings. Period.Yes, even if you are just sending emails. Even if you are just receiving emails. Youdon’t need to understand every last detail, but you must at least know what thiswhole "encoding" thing is about. And the good news first: while the topic can getmessy and confusing, the basic idea is really, really simple.

This article is about encodings and character sets. An article by Joel Spolsky entitledThe Absolute Minimum Every Software Developer Absolutely, Positively Must KnowAbout Unicode and Character Sets (No Excuses!)2 is a nice introduction to the topicand I greatly enjoy reading it every once in a while. I hesitate to refer people to it whohave trouble understanding encoding problems though since, while entertaining, itis pretty light on actual technical details. I hope this article can shed some more lighton what exactly an encoding is and just why all your text screws up when you leastneed it. This article is aimed at developers (with a focus on PHP), but any computeruser should be able to benefit from it.

Getting the basics straight

Everybody is aware of this at some level, but somehow this knowledge seems tosuddenly disappear in a discussion about text, so let’s get it out first: A computercannot store "letters", "numbers", "pictures" or anything else. The only thing it canstore and work with are bits. A bit can only have two values: yes or no , trueor false , 1 or 0 or whatever else you want to call these two values. Since acomputer works with electricity, an "actual" bit is a blip of electricity that either isor isn’t there. For humans, this is usually represented using 1 and 0 and I’ll stickwith this convention throughout this article.

1 http://kunststube.net/encoding/2 http://www.joelonsoftware.com/articles/Unicode.html

http://kunststube.net/encoding/

http://www.joelonsoftware.com/articles/Unicode.html


http://kunststube.net/encoding/


What every programmer absolutely, positively needs toknow about encodings and character sets to work with text

7

To use bits to represent anything at all besides bits, we need rules. We need toconvert a sequence of bits into something like letters, numbers and pictures usingan encoding scheme, or encoding for short. Like this:

01100010 01101001 01110100 01110011b i t s

In this encoding, 01100010 stands for the letter "b", 01101001 for the letter "i",01110100 stands for "t" and 01110011 for "s". A certain sequence of bits standsfor a letter and a letter stands for a certain sequence of bits. If you can keep this inyour head for 26 letters or are really fast with looking stuff up in a table, you couldread bits like a book.

The above encoding scheme happens to be ASCII. A string of `1`s and `0`s is brokendown into parts of eight bit each (a byte for short). The ASCII encoding specifies atable translating bytes into human readable letters. Here’s a short excerpt of thattable:

bits character

01000001 A

01000010 B

01000011 C

01000100 D

01000101 E

01000110 F

There are 95 human readable characters specified in the ASCII table, including theletters A through Z both in upper and lower case, the numbers 0 through 9, a handfulof punctuation marks and characters like the dollar symbol, the ampersand and afew others. It also includes 33 values for things like space, line feed, tab, backspaceand so on. These are not printable per se, but still visible in some form and usefulto humans directly. A number of values are only useful to a computer, like codesto signify the start or end of a text. In total there are 128 characters defined in theASCII encoding, which is a nice round number (for people dealing with computers),3Yes, that means ASCII can be stored and transferred using only 7 bits and it often is. No, this is notwithin the scope of this article and for the sake of argument we’ll assume the highest bit is "wasted" inASCII.


8

since it uses all possible combinations of 7 bits ( 0000000 , 0000001 , 0000010through 1111111 ).3

And there you have it, the way to represent human-readable text using only `1`sand `0`s.

01001000 01100101 01101100 01101100 01101111 00100000 01010111 01101111 01110010 01101100 01100100

"Hello World"

Important terms

To encode something in ASCII, follow the table from right to left, substituting lettersfor bits. To decode a string of bits into human readable characters, follow the tablefrom left to right, substituting bits for letters.

encode |enˈkōd|verb [ with obj. ]

convert into a coded form

code |kōd|noun

a system of words, letters, figures, or other symbols substituted for otherwords, letters, etc.

To encode means to use something to represent something else. An encoding is theset of rules with which to convert something from one representation to another.

Other terms which deserve clarification in this context:

character set, charsetThe set of characters that can be encoded. "The ASCII encoding encompasses acharacter set of 128 characters." Essentially synonymous to "encoding".

code pageA "page" of codes that map a character to a number or bit sequence. A.k.a. "thetable". Essentially synonymous to "encoding".


9

stringA string is a bunch of items strung together. A bit string is a bunch of bits,like 01010011 . A character string is a bunch of characters, like this .Synonymous to "sequence".

Binary, octal, decimal, hex

There are many ways to write numbers. 10011111 in binary is 237 in octal is 159 indecimal is 9F in hexadecimal. They all represent the same value, but hexadecimal isshorter and easier to read than binary. I will stick with binary throughout this articleto get the point across better and spare the reader one layer of abstraction. Do notbe alarmed to see character codes referred to in other notations elsewhere, it’s allthe same thing.

Excusez-moi?

Now that we know what we’re talking about, let’s just say it: 95 characters reallyisn’t a lot when it comes to languages. It covers the basics of English, but what aboutwriting a risqué letter in French? A Straßenübergangsänderungsgesetz in German?An invitation to a smörgåsbord in Swedish? Well, you couldn’t. Not in ASCII. There’sno specification on how to represent any of the letters é, ß, ü, ä, ö or å in ASCII, soyou can’t use them.

"But look at it," the Europeans said, "in a common computer with 8 bits to the byte,ASCII is wasting an entire bit which is always set to 0 ! We can use that bit to squeezea whole 'nother 128 values into that table!" And so they did. But even so, thereare more than 128 ways to stroke, slice, slash and dot a vowel. Not all variations ofletters and squiggles used in all European languages can be represented in the sametable with a maximum of 256 values. So what the world ended up with is a wealthof encoding schemes4, standards, de-facto standards and half-standards that allcover a different subset of characters. Somebody needed to write a document aboutSwedish in Czech, found that no encoding covered both languages and invented one.Or so I imagine it went countless times over.

And not to forget about Russian, Hindi, Arabic, Hebrew, Korean and all the otherlanguages currently in active use on this planet. Not to mention the ones not inuse anymore. Once you have solved the problem of how to write mixed languagedocuments in all of these languages, try yourself on Chinese. Or Japanese. Both

4 http://en.wikipedia.org/wiki/Character_encoding#Common_character_encodings

http://en.wikipedia.org/wiki/Character_encoding#Common_character_encodings




10

contain tens of thousands of characters. You have 256 possible values to a byteconsisting of 8 bit. Go!

Multi-byte encodings

To create a table that maps characters to letters for a language that uses more than256 characters, one byte simply isn’t enough. Using two bytes (16 bits), it’s possibleto encode 65,536 distinct values. BIG-55 is such a double-byte encoding. Insteadof breaking a string of bits into blocks of eight, it breaks it into blocks of 16 andhas a big (I mean, BIG) table that specifies which character each combination ofbits maps to. BIG-5 in its basic form covers mostly Traditional Chinese characters.GB180306 is another encoding which essentially does the same thing, but includesboth Traditional and Simplified Chinese characters. And before you ask, yes, thereare encodings which cover only Simplified Chinese. Can’t just have one encodingnow, can we?

Here a small excerpt from the GB18030 table:

bits character

10000001 01000000 丂

10000001 01000001 丄

10000001 01000010 丅

10000001 01000011 丆

10000001 01000100 丏

GB18030 covers quite a range of characters (including a large part of latincharacters), but in the end is yet another specialized encoding format among many.

5 http://en.wikipedia.org/wiki/Big-56 http://en.wikipedia.org/wiki/GB18030

http://en.wikipedia.org/wiki/Big-5

http://en.wikipedia.org/wiki/GB18030

http://en.wikipedia.org/wiki/Big-5

http://en.wikipedia.org/wiki/GB18030


11

Unicode to the confusion

Finally somebody had enough of the mess and set out to forge a ring to bind themall create one encoding standard to unify all encoding standards. This standard isUnicode. It basically defines a ginormous table of 1,114,112 code points that canbe used for all sorts of letters and symbols. That’s plenty to encode all European,Middle-Eastern, Far-Eastern, Southern, Northern, Western, pre-historian and futurecharacters mankind knows about.7 Using Unicode, you can write a documentcontaining virtually any language using any character you can type into a computer.This was either impossible or very very hard to get right before Unicode came along.There’s even an unofficial section for Klingon8 in Unicode. Indeed, Unicode is bigenough to allow for unofficial, private-use areas.

So, how many bits does Unicode use to encode all these characters? None. BecauseUnicode is not an encoding.

Confused? Many people seem to be. Unicode first and foremost defines a table ofcode points for characters. That’s a fancy way of saying "65 stands for A, 66 standsfor B and 9,731 stands for ☃" (seriously, it does). How these code points are actuallyencoded into bits is a different topic. To represent 1,114,112 different values, twobytes aren’t enough. Three bytes are, but three bytes are often awkward to work

7And if it isn’t, it will be extended. It already has been several times.8 http://www.evertype.com/standards/csur/klingon.html

http://www.evertype.com/standards/csur/klingon.html

http://www.evertype.com/standards/csur/klingon.html


12

with, so four bytes would be the comfortable minimum. But, unless you’re actuallyusing Chinese or some of the other characters with big numbers that take a lot ofbits to encode, you’re never going to use a huge chunk of those four bytes. If theletter "A" was always encoded to 00000000 00000000 00000000 01000001 , "B"always to 00000000 00000000 00000000 01000010 and so on, any documentwould bloat to four times the necessary size.

To optimize this, there are several ways to encode Unicode code points into bits.UTF-32 is such an encoding that encodes all Unicode code points using 32 bits. Thatis, four bytes per character. It’s very simple, but often wastes a lot of space. UTF-16and UTF-8 are variable-length encodings. If a character can be represented using asingle byte (because its code point is a very small number), UTF-8 will encode it witha single byte. If it requires two bytes, it will use two bytes and so on. It has elaborateways to use the highest bits in a byte to signal how many bytes a character consistsof. This can save space, but may also waste space if these signal bits need to beused often. UTF-16 is in the middle, using at least two bytes, growing to up to fourbytes as necessary.

character encoding bits

A UTF-8 01000001

A UTF-16 00000000 01000001

A UTF-32 00000000 00000000 0000000001000001

あ UTF-8 11100011 10000001 10000010

あ UTF-16 00110000 01000010

あ UTF-32 00000000 00000000 0011000001000010

And that’s all there is to it. Unicode is a large table mapping characters to numbersand the different UTF encodings specify how these numbers are encoded as bits.Overall, Unicode is yet another encoding scheme. There’s nothing special about it, it’sjust trying to cover everything while still being efficient. And that’s A Good Thing.™


13

Code points

Characters are referred to by their "Unicode code point". Unicode code points arewritten in hexadecimal (to keep the numbers shorter), preceded by a "U+" (that’sjust what they do, it has no other meaning than "this is a Unicode code point"). Thecharacter Ḁ has the Unicode code point U+1E00. In other (decimal) words, it is the7680th character of the Unicode table. It is officially called "LATIN CAPITAL LETTERA WITH RING BELOW".

TL;DR

A summary of all the above: Any character can be encoded in many different bitsequences and any particular bit sequence can represent many different characters,depending on which encoding is used to read or write them. The reason is simplybecause different encodings use different numbers of bits per characters anddifferent values to represent different characters.

bits encoding characters

11000100 01000010 WindowsLatin 1

ÄB

11000100 01000010 Mac Roman ƒB

11000100 01000010 GB18030 腂


14

characters encoding bits

Føö WindowsLatin 1

01000110 11111000 11110110

Føö Mac Roman 01000110 10111111 10011010

Føö UTF-8 01000110 11000011 10111000 1100001110110110

Misconceptions, confusions and problems

Having said all that, we come to the actual problems experienced by many usersand programmers every day, how those problems relate to all of the above and whattheir solution is. The biggest problem of all is:

Why in god’s name are my characters garbled?!

ÉGÉìÉRÅ[ÉfÉBÉìÉOÇÕìÔÇµÇ≠Ç»Ç¢

If you open a document and it looks like this, there’s one and only one reason for it:Your text editor, browser, word processor or whatever else that’s trying to read thedocument is assuming the wrong encoding. That’s all. The document is not broken(well, unless it is, see below), there’s no magic you need to perform, you simply needto select the right encoding to display the document.

The hypothetical document above contains this sequence of bits:

10000011 01000111 10000011 10010011 10000011 01010010 10000001 0101101110000011 01100110 10000011 01000010 10000011 10010011 10000011 0100111110000010 11001101 10010011 11101111 10000010 10110101 10000010 1010110110000010 11001000 10000010 10100010

Now, quick, what encoding is that? If you just shrugged, you’d be correct. Who knows,right‽

Well, let’s try to interpret this as ASCII. Hmm, most of these bytes start 9 with a1 bit. If you remember correctly, ASCII doesn’t use that bit. So it’s not ASCII. What

9Please note that when I’m using the term "starting" together with "byte", I mean it from the human-readable point of view.


15

about UTF-8? Hmm, no, most of these sequences are not valid UTF-8.10 So UTF-8 isout, too. Let’s try "Mac Roman" (yet another encoding scheme for them Europeans).Hey, all those bytes are valid in Mac Roman. 10000011 maps to "É", 01000111to "G" and so on. If you read this bit sequence using the Mac Roman encoding,the result is "ÉGÉìÉRÅ[ÉfÉBÉìÉOÇÕìÔÇµÇ≠Ç»Ç¢". That looks like a valid string, no?Yes? Maybe? Well, how’s the computer to know? Maybe somebody meant to write"ÉGÉìÉRÅ[ÉfÉBÉìÉOÇÕìÔÇµÇ≠Ç»Ç¢". For all I know that could be a DNA sequence.11

Unless you have a better suggestion, let’s declare this to be a DNA sequence, saythis document was encoded in Mac Roman and call it a day.

Of course, that unfortunately is complete nonsense. The correct answer is that thistext is encoded in the Japanese Shift-JIS encoding and was supposed to read "エンコーディングは難しくない". Well, who’d’ve thunk?

The primary cause of garbled text is: Somebody is trying to read a byte sequenceusing the wrong encoding. The computer always needs to be told what encodingsome text is in. Otherwise it can’t know. There are different ways how different kindsof documents can specify what encoding they’re in and these ways should be used.A raw bit sequence is always a mystery box and could mean anything.

10Peruse the UTF-8 specification if you want to follow this with pen and paper.11Hey, I’m a programmer, not a biologist.


16

Most browsers allow the selection of a different encoding in the View menu under themenu option "Text Encoding", which causes the browser to reinterpret the currentpage using the selected encoding. Other programs may offer something like "Reopenusing encoding…" in the File menu, or possibly an "Import…" option which allows theuser to manually select an encoding.

My document doesn’t make sense in any encoding!

If a sequence of bits doesn’t make sense (to a human) in any encoding, the documenthas mostly likely been converted incorrectly at some point. Say we took the abovetext "ÉGÉìÉRÅ[ÉfÉBÉìÉOÇÕìÔÇµÇ≠Ç»Ç¢" because we didn’t know any better andsaved it as UTF-8. The text editor assumed it correctly read a Mac Roman encodedtext and you now want to save this text in a different encoding. All of these charactersare valid Unicode characters after all. That is to say, there’s a code point in Unicodethat can represent "É", one that can represent "G" and so on. So we can happily savethis text as UTF-8:

11000011 10001001 01000111 11000011 10001001 11000011 10101100 1100001110001001 01010010 11000011 10000101 01011011 11000011 10001001 0110011011000011 10001001 01000010 11000011 10001001 11000011 10101100 1100001110001001 01001111 11000011 10000111 11000011 10010101 11000011 1010110011000011 10010100 11000011 10000111 11000010 10110101 11000011 1000011111100010 10001001 10100000 11000011 10000111 11000010 10111011 1100001110000111 11000010 10100010

This is now the UTF-8 bit sequence representing the text"ÉGÉìÉRÅ[ÉfÉBÉìÉOÇÕìÔÇµÇ≠Ç»Ç¢". This bit sequence has absolutely nothing to dowith our original document. Whatever encoding we try to open it in, we won’t everget the text "エンコーディングは難しくない" from it. It is completely lost. It would bepossible to recover the original text from it if we knew that a Shift-JIS document wasmisinterpreted as Mac Roman and then accidentally saved as UTF-8 and reversedthis chain of missteps. But that would be a lucky fluke.

Many times certain bit sequences are invalid in a particular encoding. If we triedto open the original document using ASCII, some bytes would be valid in ASCII andmap to a real character and others wouldn’t. The program you’re opening it withmay decide to silently discard any bytes that aren’t valid in the chosen encoding, orpossibly replace them with ? . There’s also the "Unicode replacement character" �(U+FFFD) which a program may decide to insert for any character it couldn’t decodecorrectly when trying to handle Unicode. If a document is saved with some characters


17

gone or replaced, then those characters are really gone for good with no way toreverse-engineer them.

If a document has been misinterpreted and converted to a different encoding, it’sbroken. Trying to "repair" it may or may not be successful, usually it isn’t. Any manualbit-shifting or other encoding voodoo is mostly that, voodoo. It’s trying to fix thesymptoms after the patient has already died.

So how to handle encodings correctly?

It’s really simple: Know what encoding a certain piece of text, that is, a certain bytesequence, is in, then interpret it with that encoding. That’s all you need to do. If you’rewriting an app that allows the user to input some text, specify what encoding youaccept from the user. For any sort of text field, the programmer can usually decideits encoding. For any sort of file a user may upload or import into a program, thereneeds to be a specification what encoding that file should be in. Alternatively, theuser needs some way to tell the program what encoding the file is in. This informationmay be part of the file format itself, or it may be a selection the user has make (notthat most users would usually know, unless they have read this article).

If you need to convert from one encoding to another, do so cleanly using toolsthat are specialized for that. Converting between encodings is the tedious taskof comparing two code pages and deciding that character 152 in encoding A isthe same as character 4122 in encoding B, then changing the bits accordingly.This particular wheel does not need reinventing and any mainstream programminglanguage includes some way of converting text from one encoding to another withoutneeding to think about code points, pages or bits at all.

Say, your app must accept files uploaded in GB18030, but internally you are handlingall data in UTF-32. A tool like iconv can cleanly convert the uploaded file with aone-liner like iconv('GB18030', 'UTF-32', $string) . That is, it will preservethe characters while changing the underlying bits:

character GB18030 encoding UTF-32 encoding

縧 1011111101101100

00000000 00000000 0111111000100111

That’s all there is to it. The content of the string, that is, the human readablecharacters, didn’t change, but it’s now a valid UTF-32 string. If you keep treatingit as UTF-32, there’s no problem with garbled characters. As discussed at the very


18

beginning though, not all encoding schemes can represent all characters. It’s notpossible to encode the character "縧" in any encoding scheme designed for Europeanlanguages. Something Bad™ would happen if you tried to.

Unicode all the way

Precisely because of that, there’s virtually no excuse in this day and age not to beusing Unicode all the way. Some specialized encodings may be more efficient thanthe Unicode encodings for certain languages. But unless you’re storing terabytesand terabytes of very specialized text (and that’s a lot of text), there’s usually noreason to worry about it. Problems stemming from incompatible encoding schemesare much worse than a wasted gigabyte or two these days. And this will becomeeven truer as storage and bandwidth keeps growing larger and cheaper.

If your system needs to work with other encodings, convert them to Unicode uponinput and convert them back to other encodings on output as necessary. Otherwise,be very aware of what encodings you’re dealing with at which point and convert asnecessary, if that’s possible without losing any information.

Flukes

I have this website talking to a database. My app handles everything as UTF-8 and stores it as such in the database and everything works fine, but when I look at my database admin interface my text is garbled.

— Anonymous code monkey

There are situations where encodings are handled incorrectly but things still work.An often-encountered situation is a database that’s set to latin-1 and an app thatworks with UTF-8 (or any other encoding). Pretty much any combination of 1`s and`0`s is valid in the single-byte `latin-1 encoding scheme. If thedatabase receives text from an application that looks like 11100111 1011100010100111 , it’ll happily store it, thinking the app meant to store the three latincharacters "ç¸§". After all, why not? It then later returns this bit sequence back to theapp, which will happily accept it as the UTF-8 sequence for "縧", which it originallystored. The database admin interface automatically figures out that the database isset to latin-1 though and interprets any text as latin-1, so all values look garbled onlyin the admin interface.


19

That’s a case of fool’s luck where things happen to work when they actually aren’t.Any sort of operation on the text in the database may or may not work as intended,since the database is not interpreting the text correctly. In a worst case scenario, thedatabase inadvertently destroys all text during some random operation two yearsafter the system went into production because it was operating on text assumingthe wrong encoding.12

UTF-8 and ASCII

The ingenious thing about UTF-8 is that it’s binary compatible with ASCII, which isthe de-facto baseline for all encodings. All characters available in the ASCII encodingonly take up a single byte in UTF-8 and they’re the exact same bytes as are used inASCII. In other words, ASCII maps 1:1 unto UTF-8. Any character not in ASCII takes uptwo or more bytes in UTF-8. For most programming languages that expect to parseASCII, this means you can include UTF-8 text directly in your programs:

$string = "漢字";

Saving this as UTF-8 results in this bit sequence:

00100100 01110011 01110100 01110010 01101001 01101110 01100111 0010000000111101 00100000 00100010 11100110 10111100 10100010 11100101 1010110110010111 00100010 00111011

Only bytes 12 through 17 (the ones starting with 1 ) are UTF-8 characters (twocharacters with three bytes each). All the surrounding characters are perfectly goodASCII. A parser would read this as follows:

$string = "11100110 10111100 10100010 11100101 10101101 10010111";

To the parser, anything following a quotation mark is just a byte sequence which itwill take as-is until it encounters another quotation mark. If you simply output thisbyte sequence, you’re outputting UTF-8 text. No need to do anything else. The parserdoes not need to specifically support UTF-8, it just needs to take strings literally.Naive parsers can support Unicode this way without actually supporting Unicode.Many modern languages are explicitly Unicode-aware though.

12And of course there’ll be no recent backup.


20

Encodings and PHP

This last section deals with issues surrounding Unicode and PHP.Some portions of it are applicable to programming languages ingeneral while others are PHP specific. Nothing new will be revealedabout encodings, but concepts described above will be rehashedin the light of practical application.

PHP doesn’t natively support Unicode. Except it actually supports it quite well. Theprevious section shows how UTF-8 characters can be embedded in any programdirectly without problems, since UTF-8 is backwards compatible with ASCII, which isall PHP needs. The statement "PHP doesn’t natively support Unicode" is true thoughand it seems to cause a lot of confusion in the PHP community.

False promises

One specific pet-peeve of mine are the functions utf8_encode13 andutf8_decode14. I often see nonsense along the lines of "To use Unicode in PHP youneed to utf8_encode your text on input and utf8_decode on output". Thesetwo functions seem to promise some sort of automagic conversion of text to UTF-8which is "necessary" since "PHP doesn’t support Unicode". If you’ve been followingthis article at all though, you should know by now that

1. there’s nothing special about UTF-8 and

2. you cannot encode text to UTF-8 after the fact

To clarify that second point: All text is already encoded in some encoding. When youtype it into the source code, it has some encoding. Specifically, whatever you savedit as in your text editor. If you get it from a database, it’s already in some encoding.If you read it from a file, it’s already in some encoding.

Text is either encoded in UTF-8 or it’s not. If it’s not, it’s encoded in ASCII, ISO-8859-1,UTF-16 or some other encoding. If it’s not encoded in UTF-8 but is supposed to contain"UTF-8 characters",15 then you have a case of cognitive dissonance. If it does contain

13 http://php.net/utf8_encode14 http://php.net/utf8_decode15A "Unicode character" is a code point in the Unicode table. "あ" is not a Unicode character, it’s theHiragana letter あ. There is a Unicode code point for it, but that doesn’t make the letter itself a Unicodecharacter. A "UTF-8 character" is an oxymoron, but may be stretched to mean what’s technically calleda "UTF-8 sequence", which is a byte sequence of one, two, three or four bytes representing one Unicode

http://php.net/utf8_encode

http://php.net/utf8_decode

http://php.net/utf8_encode

http://php.net/utf8_decode


21

actual characters encoded in UTF-8, then it’s actually UTF-8 encoded. Text can’tcontain Unicode characters without being encoded in one of the Unicode encodings.

So what in the world does utf8_encode do then?

"Encodes an ISO-8859-1 string to UTF-8"16

Aha! So what the author actually wanted to say is that it converts the encoding of textfrom ISO-8859-1 to UTF-8. That’s all there is to it. utf8_encode must have beennamed by some European without any foresight and is a horrible, horrible misnomer.The same goes for utf8_decode . These functions are useless for any purpose otherthan converting between ISO-8859-1 and UTF-8. If you need to convert a string fromany other encoding to any other encoding, look no further then iconv 17.

utf8_encode is not a magic wand that needs to be swung over any and all textbecause "PHP doesn’t support Unicode". Rather, it seems to cause more encodingproblems than it solves thanks to terrible naming and unknowing developers.

Native-schmative

So what does it mean for a language to natively support or not support Unicode?It basically refers to whether the language assumes that one character equals onebyte or not. For example, PHP allows direct access to the characters of a string usingarray notation:

echo $string[0];

If that $string was in a single-byte encoding, this would give us the first character.But only because "character" coincides with "byte" in a single-byte encoding. PHPsimply gives us the first byte without thinking about "characters". Strings are bytesequences to PHP, nothing more, nothing less. All this "readable character" stuff isa human thing and PHP doesn’t care about it.

01000100 01101111 01101110 00100111 01110100D o n ' t01100011 01100001 01110010 01100101 00100001c a r e !

character. Both terms are often used in the sense of "any letter that ain’t part of my keyboard" though,which means absolutely nothing.16http://www.php.net/manual/en/function.utf8-encode.php17 http://www.php.net/manual/en/function.iconv.php

http://www.php.net/manual/en/function.iconv.php

http://www.php.net/manual/en/function.utf8-encode.php

http://www.php.net/manual/en/function.iconv.php


22

The same goes for many standard functions such as substr , strpos , trim andso on. The non-support arises if there’s a discrepancy between the length of a byteand a character.

11100110 10111100 10100010 11100101 10101101 10010111漢字

Using $string[0] on the above string will, again, give us the first byte, which is11100110 . In other words, a third of the three-byte character "漢". 11100110 is,by itself, an invalid UTF-8 sequence, so the string is now broken. If you felt like it,you could try to interpret that in some other encoding where 11100110 representsa valid character, which will result in some random character. Have fun, but don’tuse it in production.

And that’s actually all there is to it. "PHP doesn’t natively support Unicode" simplymeans that most PHP functions assume one byte = one character, which maylead to it chopping multi-byte characters in half or calculating the length of stringsincorrectly if you’re naively using non-multi-byte-aware functions on multi-bytestrings. It does not mean that you can’t use Unicode in PHP or that every Unicodestring needs to be blessed by utf8_encode or other such nonsense.

Luckily, there’s the Multibyte String extension18, which replicates all importantstring functions in a multi-byte aware fashion. Using mb_substr($string, 0,1, 'UTF-8') on the above string correctly returns 11100110 1011110010100010 , which is the whole "漢" character. Because the mb_ functionsnow have to actually think about what they’re doing, they need to know whatencoding they’re working on. Therefore every mb_ function accepts an $encodingparameter as well. Alternatively, this can be set globally for all mb_ functions usingmb_internal_encoding .

18 http://www.php.net/manual/en/book.mbstring.php

http://www.php.net/manual/en/book.mbstring.php

http://www.php.net/manual/en/book.mbstring.php


23

Using and abusing PHP’s handling of encodings

The whole issue of PHP’s (non-)support for Unicode is that it just doesn’t care.Strings are byte sequences to PHP. What bytes in particular doesn’t matter. PHPdoesn’t do anything with strings except keeping them stored in memory. PHP simplydoesn’t have any concept of either characters or encodings. And unless it triesto manipulate strings, it doesn’t need to either; it just holds onto bytes that mayor may not eventually be interpreted as characters by somebody else. The onlyrequirement19 PHP has of encodings is that PHP source code needs to be saved inan ASCII compatible encoding. The PHP parser is looking for certain characters thattell it what to do. $ ( 00100100 ) signals the start of a variable, = ( 00111101 ) anassignment, " ( 00100010 ) the start and end of a string and so on. Anything elsethat doesn’t have any special significance to the parser is just taken as a literal bytesequence. That includes anything between quotes, as discussed above. This meansthe following:

1. You can’t save PHP source code in an ASCII-incompatible encoding. For example,in UTF-16 a " is encoded as 00000000 00100010 . To PHP, which tries to readeverything as ASCII, that’s a NUL byte followed by a " . PHP will probably get ahiccup if every other character it finds is a NUL byte.

2. You can save PHP source code in any ASCII-compatible encoding. If the first 128code points of an encoding are identical to ASCII, PHP can parse it. All charactersthat are in any way significant to PHP are within the 128 code points defined byASCII. If string literals contain any code points beyond that, PHP doesn’t care.You can save PHP source code in ISO-8859-1, Mac Roman, UTF-8 or any otherASCII-compatible encoding. The string literals in your script will have whateverencoding you saved your source code as.

3. Any external file you process with PHP can be in whatever encoding you like. IfPHP doesn’t need to parse it, there are no requirements to meet to keep the PHPparser happy.

$foo = file_get_contents('bar.txt');

The above will simply read the bits in bar.txt into the variable $foo . PHPdoesn’t try to interpret, convert, encode or otherwise fiddle with the contents.The file can even contain binary data such as an image, PHP doesn’t care.

19 http://www.php.net/manual/en/mbstring.php4.req.php

http://www.php.net/manual/en/mbstring.php4.req.php

http://www.php.net/manual/en/mbstring.php4.req.php


24

4. If internal and external encodings have to match, they have to match. A commoncase is localization, where the source code contains something like echolocalize('Foobar') and an external localization file contains somethingalong the lines of this:

msgid "Foobar"msgstr "フーバー"

Both "Foobar" strings need to have an identical bit representation if you wantto find the correct localization. If the source code was saved in ASCII butthe localization file in UTF-16, the strings wouldn’t match. Either some sort ofencoding conversion would be necessary or the use of an encoding-aware stringmatching function.

The astute reader might ask at this point whether it’s possible to save a, say, UTF-16byte sequence inside a string literal of an ASCII encoded source code file, to whichthe answer would be: absolutely.

echo "UTF-16";

If you can bring your text editor to save the echo " and "; parts in ASCII andonly UTF-16 in UTF-16, this will work just fine. The necessary binary representationfor that looks like this:

01100101 01100011 01101000 01101111 00100000 00100010e c h o "11111110 11111111 00000000 01010101 00000000 01010100(UTF-16 marker) U T00000000 01000110 00000000 00101101 00000000 00110001F - 100000000 00110110 00100010 001110116 " ;

The first line and the last two bytes are ASCII. The rest is UTF-16 with two bytes percharacter. The leading 11111110 11111111 on line 2 is a marker required at thestart of UTF-16 encoded text (required by the UTF-16 standard, PHP doesn’t give adamn). This PHP script will happily output the string "UTF-16" encoded in UTF-16,because it simple outputs the bytes between the two double quotes, which happensto represent the text "UTF-16" encoded in UTF-16. The source code file is neither


25

completely valid ASCII nor UTF-16 though, so working with it in a text editor won’tbe much fun.

Bottom line

PHP supports Unicode, or in fact any encoding, just fine, as long as certainrequirements are met to keep the parser happy and the programmer knows what he’sdoing. You really only need to be careful when manipulating strings, which includesslicing, trimming, counting and other operations that need to happen on a characterlevel rather than a byte level. If you’re not "doing anything" with your strings besidesreading and outputting them, you will hardly have any problems with PHP’s supportof encodings that you wouldn’t have in any other language as well.

Encoding-aware languages

What does it mean for a language to support Unicode then? Javascript for examplesupports Unicode. In fact, any string in Javascript is UTF-16 encoded. In fact, it’s theonly thing Javascript deals with. You cannot have a string in Javascript that is notUTF-16 encoded. Javascript worships Unicode to the extent that there’s no facilityto deal with any other encoding in the core language. Since Javascript is most oftenrun in a browser that’s not a problem, since the browser can handle the mundanelogistics of encoding and decoding input and output.

Other languages are simply encoding-aware. Internally they store strings in aparticular encoding, often UTF-16. In turn they need to be told or try to detect theencoding of everything that has to do with text. They need to know what encodingthe source code is saved in, what encoding a file they’re supposed to read is in,what encoding you want to output text in; and they convert encodings on the fly asneeded with some manifestation of Unicode as the middleman. They’re doing thesame thing you can/should/need to do in PHP semi-automatically behind the scenes.That’s neither better nor worse than PHP, just different. The nice thing about it isthat standard language functions that deal with strings Just Work™, while in PHP oneneeds to spare some attention to whether a string may contain multi-byte charactersor not and choose string manipulation functions accordingly.

The depths of Unicode

Since Unicode deals with many different scripts and many different problems, ithas a lot of depth to it. For example, the Unicode standard contains information


26

for such problems as CJK ideograph unification20. That means, information that twoor more Chinese/Japanese/Korean characters actually represent the same characterin slightly different writing methods. Or rules about converting from lower case toupper case, vice-versa and round-trip, which is not always as straight forward inall scripts as it is in most Western European Latin-derived scripts. Some characterscan also be represented using different code points. The letter "ö" for examplecan be represented using the code point U+00F6 ("LATIN SMALL LETTER O WITHDIAERESIS") or as the two code points U+006F ("LATIN SMALL LETTER O") and U+0308 ("COMBINING DIAERESIS"), that is the letter "o" combined with "¨". In UTF-8that’s either the double-byte sequence 11000011 10110110 or the three-bytesequence 01101111 11001100 10001000 , both representing the same humanreadable character. As such, there are rules governing Normalization within theUnicode standard, i.e. how either of these forms can be converted into the other.This and a lot more is outside the scope of this article, but one should be aware of it.

Final TL;DR

• Text is always a sequence of bits which needs to be translated into humanreadable text using lookup tables. If the wrong lookup table is used, the wrongcharacter is used.

• You’re never actually directly dealing with "characters" or "text", you’re alwaysdealing with bits as seen through several layers of abstractions. Incorrect resultsare a sign of one of the abstraction layers failing.

• If two systems are talking to each other, they always need to specify whatencoding they want to talk to each other in. The simplest example of this is thiswebsite telling your browser that it’s encoded in UTF-8.

• In this day and age, the standard encoding is UTF-8 since it can encode virtuallyany character of interest, is backwards compatible with the de-facto baselineASCII and is relatively space efficient for the majority of use cases nonetheless.

◦ Other encodings still occasionally have their uses, but you should have aconcrete reason for wanting to deal with the headaches associated withcharacter sets that can only encode a subset of Unicode.

• The days of one byte = one character are over and both programmers andprograms need to catch up on this.

Now you should really have no excuse anymore the next time you garble some text.

20 http://en.wikipedia.org/wiki/CJK_Unified_Ideographs

http://en.wikipedia.org/wiki/CJK_Unified_Ideographs

http://en.wikipedia.org/wiki/CJK_Unified_Ideographs

27

Everything Is BrokenWritten by: Quinn Norton at May 20, 20141

Once upon a time, a friend of mine accidentally took over thousands of computers.He had found a vulnerability in a piece of software and started playing with it. In theprocess, he figured out how to get total administration access over a network. Heput it in a script, and ran it to see what would happen, then went to bed for aboutfour hours. Next morning on the way to work he checked on it, and discovered hewas now lord and master of about 50,000 computers. After nearly vomiting in fear hekilled the whole thing and deleted all the files associated with it. In the end he saidhe threw the hard drive into a bonfire. I can’t tell you who he is because he doesn’twant to go to Federal prison, which is what could have happened if he’d told anyonethat could do anything about the bug he’d found. Did that bug get fixed? Probablyeventually, but not by my friend. This story isn’t extraordinary at all. Spend muchtime in the hacker and security scene, you’ll hear stories like this and worse.

It’s hard to explain to regular people how much technology barely works, how muchthe infrastructure of our lives is held together by the IT equivalent of baling wire.

Computers, and computing, are broken.

Build it badly, and they will come.

For a bunch of us, especially those who had followed security and the warrantlesswiretapping cases, the revelations weren’t big surprises. We didn’t know thespecifics, but people who keep an eye on software knew computer technology wassick and broken. We’ve known for years that those who want to take advantage ofthat fact tend to circle like buzzards. The NSA wasn’t, and isn’t, the great predatorof the internet, it’s just the biggest scavenger around. It isn’t doing so well becausethey are all powerful math wizards of doom.

The NSA is doing so well because software is bullshit.

Eight months before Snowden’s first revelation I tweeted this:

It was my exasperated acknowledgement that looking for goodsoftware to count on has been a losing battle. Written by people with

1 https://medium.com/message/everything-is-broken-81e5f33a24e1

https://medium.com/message/everything-is-broken-81e5f33a24e1

https://medium.com/message/everything-is-broken-81e5f33a24e1

Everything Is Broken

28

either no time or no money, most software gets shipped the moment itworks well enough to let someone go home and see their family. Whatwe get is mostly terrible.

— @quinnnorton https://twitter.com/quinnnorton/status/236655133046501376

Software is so bad because it’s so complex, and because it’s trying to talk to otherprograms on the same computer, or over connections to other computers. Even yourcomputer is kind of more than one computer, boxes within boxes, and each one ofthose computers is full of little programs trying to coordinate their actions and talk toeach other. Computers have gotten incredibly complex, while people have remainedthe same gray mud with pretensions of godhood.

Your average piece-of-shit Windows desktop is so complex that no one person onEarth really knows what all of it is doing, or how.

Now imagine billions of little unknowable boxes within boxes constantly tryingto talk and coordinate tasks at around the same time, sharing bits of data andpassing commands around from the smallest little program to something huge, like


29

a browser—that’s the internet. All of that has to happen nearly simultaneously andsmoothly, or you throw a hissy fit because the shopping cart forgot about your movietickets.

We often point out that the phone you mostly play casual games on and keepdropping in the toilet at bars is more powerful than all the computing we used to goto space for decades.

NASA had a huge staff of geniuses to understand and care for their software. Yourphone has you.

Plus a system of automatic updates you keep putting off because you’re in the middleof Candy Crush Saga every time it asks.

Because of all this, security is terrible. Besides being riddled with annoying bugs andimpossible dialogs, programs often have a special kind of hackable flaw called 0daysby the security scene. No one can protect themselves from 0days. It’s their definingfeature—0 is the number of days you’ve had to deal with this form of attack. Thereare meh, not-so-terrible 0days, there are very bad 0days, and there are catastrophic0days that hand the keys to the house to whomever strolls by. I promise that rightnow you are reading this on a device with all three types of 0days. “But, Quinn,” I canhear you say, “If no one knows about them how do you know I have them?” Becauseeven okay software has to work with terrible software. The number of people whosejob it is to make software secure can practically fit in a large bar, and I’ve watchedthem drink. It’s not comforting. It isn’t a matter of if you get owned, only a matterof when.


30

Figure 1. This is a thing that actually happened several yearsago.2 To get rid of a complaining message from another pieceof software, a Debian developer just commented out a line ofcode without realizing that it left their encryption open to easyattack (https://www.xkcd.com/424/)

Look at it this way—every time you get a security update (seems almost daily on myLinux box), whatever is getting updated has been broken, lying there vulnerable, forwho-knows-how-long. Sometimes days, sometimes years. Nobody really advertisesthat part of updates. People say “You should apply this, it’s a critical patch!” andleave off the “…because the developers fucked up so badly your children’s identitiesare probably being sold to the Estonian Mafia by smack addicted script kiddies rightnow.”

2 https://www.schneier.com/blog/archives/2008/05/random_number_b.html

https://www.schneier.com/blog/archives/2008/05/random_number_b.html


https://www.xkcd.com/424/



31

The really bad bugs (and who knows which ones those are when they click the“Restart Later” button?) can get swept up by hackers, governments, and otherhorrors of the net that are scanning for versions of software they know they canexploit. Any computer that shows up in a scan saying “Hey! Me! I’m vulnerable!”can become part of a botnet, along with thousands, or hundreds of thousands ofother computers. Often zombied computers get owned again and become part of yetanother botnet. Some botnets patch computers to throw out the other botnets so theydon’t have to share you with other hackers. How can you tell if this is happening? Youcan’t! Have fun wondering if you’re getting your online life rented out by the hour!

Next time you think your grandma is uncool, give her credit for her time helpingdangerous Russian criminals extort money from offshore casinos with DDoS attacks.


32

Figure 2. A map of things which were hacked for the InternetCensus.

Recently an anonymous hacker wrote a script that took over embedded Linuxdevices.3 These owned computers scanned the whole rest of the internet andcreated a survey that told us more than we’d ever known about the shape of theinternet. The little hacked boxes reported their data back (a full 10 TBs) and quietlydeactivated the hack. It was a sweet and useful example of someone who hackedthe planet to shit. If that malware had actually been malicious, we would have beenso fucked.

This is because all computers are reliably this bad: the ones in hospitals andgovernments and banks, the ones in your phone, the ones that control light switchesand smart meters and air traffic control systems. Industrial computers that maintaininfrastructure and manufacturing are even worse. I don’t know all the details, butthose who do are the most alcoholic and nihilistic people in computer security.Another friend of mine accidentally shut down a factory with a malformed ping atthe beginning of a pen test. For those of you who don’t know, a ping is just about thesmallest request you can send to another computer on the network. It took them aday to turn everything back on.

Computer experts like to pretend they use a whole different, more awesome class ofsoftware that they understand, that is made of shiny mathematical perfection andwhose interfaces happen to have been shat out of the business end of a cholericdonkey. This is a lie. The main form of security this offers is through obscurity—so

3 http://internetcensus2012.bitbucket.org/paper.html

http://internetcensus2012.bitbucket.org/paper.html




33

few people can use this software that there’s no point in building tools to attack it.Unless, like the NSA, you want to take over sysadmins.

A well written encrypted chat, what could gowrong?

Let’s take an example computer experts like to stare down their noses at normalpeople for not using: OTR. OTR, or Off The Record messaging, sneaks a layer ofencryption inside normal plain text instant messaging. It’s like you got on AIM orJabber or whatever and talked in code, except the computer is making the code foryou. OTR is clever and solid, it’s been examined carefully, and we’re fairly sure ithasn’t got any of those nasty 0days.

Except, OTR isn’t a program you use, as such.

There is a standard for OTR software, and a library, but it doesn’t do anything onits own. It gets implemented in software for normal human shlubs to use by othernormal human shlubs. By now, you know this ends in tears.

The main thing that uses OTR is another piece of software that uses a library calledlibpurple4. If you want to see infosec snobs look as distressed as the donkeys thatshit out their interfaces, bring up libpurple. Libpurple was written in a programminglanguage called C.

C is good for two things: being beautiful and creating catastrophic 0days in memorymanagement.

4 https://developer.pidgin.im/wiki/WhatIsLibpurple

https://developer.pidgin.im/wiki/WhatIsLibpurple

https://developer.pidgin.im/wiki/WhatIsLibpurple


34

Figure 3. Heartbleed attack5

Heartbleed6, the bug that affected the world over, leaking password and encryptionkeys and who knows what? Classic gorgeous C.

5 http://xkcd.com/1354/6 http://heartbleed.com/

http://xkcd.com/1354/

http://heartbleed.com/

http://xkcd.com/1354/

http://heartbleed.com/


35

Libpurple was written by people who wanted their open source chat client to talk toevery kind of instant messaging system in the world, and didn’t give a shit aboutsecurity or encryption. Security people who have examined the code have said thereare so many possible ways to exploit libpurple there is probably no point in patchingit. It needs to be thrown out and rewritten from scratch. These aren’t bugs that letsomeone read your encrypted messages, they are bugs that let someone take overyour whole computer, see everything you type or read and probably watch you pickyour nose on your webcam.

This lovely tool, OTR, sits on top of libpurple on most systems that use it. Let memake something clear, because even some geeks don’t get this: it doesn’t matterhow good your encryption is if your attacker can just read your data off the screenwith you, and I promise they can. They may or may not know how to yet, but they can.There are a hundred libpurples on your computer: little pieces of software written ona budget with unrealistic deadlines by people who didn’t know or didn’t care aboutkeeping the rest of your system secure.

Any one of these little bugs will do when it comes to taking over everything else onyour computer. So we update and update, and maybe that throws any intruders out,and maybe it doesn’t. No one knows!

When we tell you to apply updates we are not telling you to mend your ship. We aretelling you to keep bailing before the water gets to your neck.

To step back a bit from this scene of horror and mayhem, let me say that thingsare better than they used to be. We have tools that we didn’t in the 1990s, likesandboxing, that keep the idiotically written programs where they can’t do as muchharm. (Sandboxing keeps a program in an artificially small part of the computer,cutting it off from all the other little programs, or cleaning up anything it tries to dobefore anything else sees it.)

Certain whole classes of terrible bugs have been sent the way of smallpox. Securityis taken more seriously than ever before, and there’s a network of people respondingto malware around the clock. But they can’t really keep up. The ecosystem of theseproblems is so much bigger than it was even ten years ago that it’s hard to feel likewe’re making progress.


36

People, as well, are broken.

“I trust you…” was my least favorite thing to hear from my sources in Anonymous.Inevitably it was followed by some piece of information they shouldn’t have beentelling me. It is the most natural and human thing to share something personal withsomeone you are learning to trust. But in exasperation I kept trying to remind Anonsthey were connecting to a computer, relaying though countless servers, switches,routers, cables, wireless links, and finally to my highly targeted computer, beforethey were connecting to another human being. All of this was happening in the timeit takes one person to draw in a deep, committal breath. It’s obvious to say, but bearsrepeating: humans were not built to think this way.

Everyone fails to use software correctly. Absolutely everyone fucks up. OTR doesn’tencrypt until after the first message, a fact that leading security professionalsand hackers subject to 20-country manhunts consistently forget. Managing all theencryption and decryption keys you need to keep your data safe across multipledevices, sites, and accounts is theoretically possible, in the same way performing


37

an appendectomy on yourself is theoretically possible. This one guy did it once inAntarctica7, why can’t you?

Every malware expert I know has lost track of what some file is, clicked on it tosee, and then realized they’d executed some malware they were supposed to beexamining. I know this because I did it once with a PDF I knew had something bad init. My friends laughed at me, then all quietly confessed they’d done the same thing.If some of the best malware reversers around can’t keep track of their malicious files,what hope do your parents have against that e-card that is allegedly from you?

Executable mail attachments (which includes things like Word, Excel, and PDFs) youget just about everyday could be from anyone—people can write anything they wantin that From: field of emails, and any of those attachments could take over yourcomputer as handily as an 0day. This is probably how your grandmother ended upworking for Russian criminals, and why your competitors anticipate all your productplans. But if you refuse to open attachments you aren’t going to be able to keepan office job in the modern world. There’s your choice: constantly risk clicking ondangerous malware, or live under an overpass, leaving notes on the lawn of yourformer house telling your children you love them and miss them.

Security and privacy experts harangue the public about metadata and networkedsharing, but keeping track of these things is about as natural as doing blood panels onyourself every morning, and about as easy. The risks on a societal level from givingup our privacy are terrible. Yet the consequences of not doing so on an individualbasis are immediately crippling. The whole thing is a shitty battle of attrition between

7 http://www.southpolestation.com/trivia/igy1/appendix.html

http://www.southpolestation.com/trivia/igy1/appendix.html




38

what we all want for ourselves and our families and the ways we need community tosurvive as humans—a Mexican stand off monetized by corporations and monitoredby governments.

I live in this stuff, and I’m no better. Once I had to step through a process to verifymyself to a secretive source. I had to take a series of pictures showing my locationand the date. I uploaded them, and was allowed to proceed with my interview. Itturns out none of my verification had come through, because I’d failed to let theupload complete before nervously shutting down my computer. “Why did you let methrough?” I asked the source. “Because only you would have been that stupid,” mysource told me.

Touché.

But if I can’t do this, as a relatively well trained adult who pays attention to theseissues all the damn time, what chance do people with real jobs and real lives have?

In the end, it’s culture that’s broken.

A few years ago, I went to several well respected people who work in privacy andsecurity software and asked them a question.


39

First, I had to explain something:

Most of the world does not have install privileges on the computer theyare using.

That is, most people using a computer in the world don’t own the computer theyare using. Whether it’s in a cafe, or school, or work, for a huge portion of the world,installing a desktop application isn’t a straightforward option. Every week or two, Iwas being contacted by people desperate for better security and privacy options,and I would try to help them. I’d start, “Download th…” and then we’d stop. The nextthing people would tell me was that they couldn’t install software on their computers.Usually this was because an IT department somewhere was limiting their rights as apart of managing a network. These people needed tools that worked with what theyhad access to, mostly a browser.

So the question I put to hackers, cryptographers, security experts, programmers, andso on was this: What’s the best option for people who can’t download new softwareto their machines? The answer was unanimous: nothing. They have no options. Theyare better off talking in plaintext I was told, “so they don’t have a false sense ofsecurity.” Since they don’t have access to better software, I was told, they shouldn’tdo anything that might upset the people watching them. But, I explained, these arethe activists, organizers, and journalists around the world dealing with governmentsand corporations and criminals that do real harm, the people in real danger. Thenthey should buy themselves computers, I was told.

That was it, that was the answer: be rich enough to buy your own computer, orliterally drop dead. I told people that wasn’t good enough, got vilified in a fewinconsequential Twitter fights, and moved on.

Not long after, I realized where the disconnect was. I went back to the same expertsand explained: in the wild, in really dangerous situations—even when people arebeing hunted by men with guns—when encryption and security fails, no one stopstalking. They just hope they don’t get caught.

The same human impulse that has kept lotteries alive for thousands of years keepspeople fighting the man against the long odds. “Maybe I’ll get away with it, mightas well try!”

As for self-censoring their conversations in the face of hostile infrastructure, non-technical activists are just as good at it as Anons are, or people told to worry aboutmetadata, or social media sharing, or that first message before OTR encryption kicksin. They blow.


40

This conversation was a wake-up call for some security people who hadn’t realizedthat people who become activists and journalists routinely do risky things. Someof them joined my side of the time-wasting inconsequential Twitter fights, realizingthat something, even something imperfect, might be better than nothing. But manyin the security scene are still waiting for a perfect world into which to deploy theirperfect code.

Then there’s the Intelligence Community, who call themselves the IC. We might likeit if they stopped spying on everyone all the time, while they would like us to stopwhining about it.

After spending some time with them, I am pretty sure I understand why they don’tcare about the complaining. The IC are some of the most surveilled humans in history.They know everything they do is gone over with a fine-toothed comb—by their peers,their bosses, their lawyers, other agencies, the president, and sometimes Congress.They live watched, and they don’t complain about it.

In all the calls for increased oversight, the basics of human nature gets neglected.You’re not going to teach the spooks this is wrong by doing it to them more.

There will always be loopholes and as long as loopholes exist or can be constructedor construed, surveillance will be as prevalent as it possibly can be. Humans aremostly egocentric creatures. Spooks, being humans, are never going to know whyliving without privacy is bad as long as they are doing it.

Yet that’s the lesser problem. The cultural catastrophe is what they’re doing to maketheir job of spying on everyone easier. The most disturbing parts of the revelationsare the 0day market, exploit hoarding, and weakening of standards. The question iswho gets to be part of the “we” that are being kept allegedly safe by all this exploiting


41

and listening and decrypting and profiling. When they attacked Natanz with Stuxnetand left all the other nuclear facilities vulnerable, we were quietly put on notice thatthe “we” in question began and ended with the IC itself. That’s the greatest danger.

When the IC or the DOD or the Executive branch are the only true Americans, and therest of us are subordinate Americans, or worse the non-people that aren’t associatedwith America, then we can only become lesser people as time goes on.

As our desires conflict with the IC, we become less and less worthy of rights andconsiderations in the eyes of the IC. When the NSA hoards exploits and interfereswith cryptographic protection for our infrastructure, it means using exploits againstpeople who aren’t part of the NSA just doesn’t count as much. Securing us comesafter securing themselves.

In theory, the reason we’re so nice to soldiers, that we have customs around honoringand thanking them, is that they’re supposed to be sacrificing themselves for thegood of the people. In the case of the NSA, this has been reversed. Our wellbeing issacrificed to make their job of monitoring the world easier. When this is part of theculture of power, it is well on its way to being capable of any abuse.

But the biggest of all the cultural problems still lies with the one group I haven’t takento task yet—the normal people living their lives under all this insanity.

The problem with the normals and tech is the same as the problem with the normalsand politics, or society in general. People believe they are powerless and alone, butthe only thing that keeps people powerless and alone is that same belief. People,working together, are immensely and terrifyingly powerful.

There is certainly a limit to what an organized movement of people who share amutual dream can do, but we haven’t found it yet.

Facebook and Google seem very powerful, but they live about a week from totalruin all the time. They know the cost of leaving social networks individually is high,but en masse, becomes next to nothing. Windows could be replaced with somethingbetter written. The US government would fall to a general revolt in a matter ofdays. It wouldn’t take a total defection or a general revolt to change everything,because corporations and governments would rather bend to demands than die.These entities do everything they can get away with—but we’ve forgotten that we’rethe ones that are letting them get away with things.

Computers don’t serve the needs of both privacy and coordination not becauseit’s somehow mathematically impossible. There are plenty of schemes that could


42

federate or safely encrypt our data, plenty of ways we could regain privacy and makeour computers work better by default. It isn’t happening now because we haven’tdemanded that it should, not because no one is clever enough to make that happen.

So yes, the geeks and the executives and the agents and the military have fuckedthe world. But in the end, it’s the job of the people, working together, to unfuck it.

43

Don’t go live with simple securityproblems - 10 tips to helpWritten by: eviltester at Tuesday, 23 July 20131

I feel anger when I stumble across very, very, very simple security issues. Especiallywhen they compromise my data.

Yes I do. And I hope, as a tester, that you do too.

But I face a problem… As a tester, I can’t say "Did no-one test this!" because I knowthat they might have done, and someone else might have chosen to go live anyway.

But on the off chance that no-one did "test this", I offer you this post.

Security by obscurity

If I visit your site and I can gain access to top level pages that I shouldn’t have, thenI get angry, because that should never happen.

Even if you haven’t told me about the URL, I can try to guess it from your othernaming conventions.

Please make sure you secure your URLs.

Security by obscurity doesn’t work for very long.

Validate Parameters on the server

And if I see parameters in your URLs. I’ll change them.

Yes I will.

I will:

• Because I don’t want the list of items to stay limited at 25, I want to see 2500

◦ No I don’t care about the performance impact on your database - fix that adifferent way

1 http://blog.eviltester.com/2013/07/dont-go-live-with-simple-security.html

http://blog.eviltester.com/2013/07/dont-go-live-with-simple-security.html

http://blog.eviltester.com/2013/07/dont-go-live-with-simple-security.html

Don’t go live with simple security problems - 10 tips to help

44

• Because I want to skip ahead more pages than you have listed on the page. "Iwant to go to page 15 now!"

◦ No I don’t care about the user experience you want me to have, I care aboutthe user experience that I want to have

• Because I can

◦ Yes, because I can

Security by ignorance

When I visit your site, I look at the traffic you issue when I hit a top level URL.

I look at what requests you make to build the page.

Yes I do.

Most browsers have the functionality to view traffic built in now. Any user can viewthe traffic and see the URLs that you access.

And then I use that information…

I take the URLs you’ve used. And I use them.

Sometimes I change them. Simple idea, but sadly all too effective.

• So if you access

◦ http://site/api/123456/report (note: not a real domain)

• I’ll access

◦ http://site/api/123455/report (note: I changed the number)

Yes I will.

And if you haven’t locked down the API to specific users, with specific role levelcontrols. Then I’ll probably see another user’s report. Then I get annoyed, becauseas a user it means that other people can see my reports. And I don’t like that.

No I don’t.

Assume that anything you do automatically someone else will do manually.Just because they can.

http://site/api/123456/report

http://site/api/123455/report


45

Make sure you have low level permissions on your API, don’t assume that no-onewill notice it.

I frequently do this because I want to bypass the horrendous GUIs that web sites putin my face, when I want to achieve a result, rather than experience your broken GUI.So I script it. Or if I can get away by posting a URL with some data, then I’ll do it.

I bet other people do this too.

Testers should do this too, because…

Security by ignorance doesn’t work.

Security through sequential numbering

And if you’re using sequential IDs for reports, or users, or accounts, etc. you activelyencouraged people to hack you.

Yes you did.

No-one has ever recommended - Security through Sequential numbering.

No they haven’t. Never Ever.

Security through sequential numbering doesn’t work.

Tips for Testing

So now, the inevitable 10 tips for testing:

1. Play with the URLs

2. Change URL Parameters

a. to check that permissions surround the public level

b. to check that request validation takes place

3. Try URLs when logged out to make sure permissions apply

4. Guess some URLs

5. Use an HTTP Debug proxy and look at the traffic

6. Investigate the traffic and see what the requests do

7. Issue the traffic requests out of context on a page to understand the 'real' staterules in place


46

8. Change the URL parameters in the traffic URLs

a. to check that permissions surround the AP

b. to check that request validation acts at the API level, not just the GUI level

9. Issue the requests when logged out to check the permissions still apply

10.And if you do test like this, and your organisation keeps ignoring these types ofdefects, check if you reported them effectively, and if you did, then leave becausethat company doesn’t deserve you.

You wouldn’t like me when I’m Angry

I didn’t even describe security testing above. I described functionality testing.

And really basic functionality testing at that, just simple input variations. I haven’tmessed with cookies, I haven’t done anything hard (because cookie editing ain’teasy, 'right kids).

If you don’t include this "really really simple stuff" level of test activity, then pleaselet me know so that I can avoid your site and find a competitor quickly before wedevelop a user/supplier relationship.

I really don’t like getting angry when I act as a user.

Trust me, you wouldn’t want me as an Angry user.

P. S.:

• Yes, this blog post does describe problems found at a specific web site.

• No I will not name that site.

• Yes, I have already told them… and more than once.

• Yes I have started looking at alternatives, sigh.

47

Extreme Programming, a ReflectionWritten by: Robert Martin at 2013-12-101

In my hand I am holding a little white book that, fourteen years ago, changed thesoftware world forever. The title of that book is: Extreme Programming Explained;and the subtitle is: Embrace Change. The author is Kent Beck, and the copyright dateis 1999.

The book is small, less than 200 pages. The print is large and widely spaced. Thewriting style is casual and accessible. The chapters are short. The concepts aresimple.

The implications were an Earthquake whose tremors haven’t even begun to die down.

Chapter 10, which begins on page 53, lays out the 12 practices that caused ourindustry to erupt into controversy; and spawned a revolution that has changedeverything about the way we write software. Those practices are:

• The Planning Game: Nowadays known as SCRUM. The idea that software isproduced in short increments from a prioritized list of work items.

• Small Releases: The notion that deployments should be frequent and incremental.

• Metaphor: Finally crystalized by Eric Evans in his book Domain Driven Design. Thenotion that the structure of the system is based upon a simple mental model ofthe problem domain.

• Simple Design: The notion that it is best to keep the system as simple as possibleat all times regardless of what we fear about the future.

• Testing: The notion that programmers, and customers, write automated teststhat verify that the production code actually does what they think it should.Nowadays we call this Test Driven Development (TDD) and Acceptance TestDriven Development (ATDD).

• Refactoring: The notion that the internal structure of software can, and should,be continuously improved.

• Pair Programming: The notion that members of a team cannot be a team if theywork separately. To be a team they must regularly collaborate, at the keyboard. In

1 http://blog.8thlight.com/uncle-bob/2013/12/10/Thankyou-Kent.html

http://blog.8thlight.com/uncle-bob/2013/12/10/Thankyou-Kent.html

http://blog.8thlight.com/uncle-bob/2013/12/10/Thankyou-Kent.html

Extreme Programming, a Reflection

48

so doing they share knowledge sufficient to cover for each other as team membersshould.

• Collective Ownership: The notion that the code belongs to the team, not to theindividual.

• 40 Hour week: The notion that teams who consistently work overtime are failing.

• On Site Customer: The notion that someone from the business, who is responsiblefor requirements, must be readily and consistently available to the programmingteam.

• Coding Standards: The notion that the team adopts a consistent style in their codeemphasizing cleanliness and communication.

Controversial?

Strange isn’t it? This doesn’t seem all that controversial does it? But fourteen yearsago it was wildly controversial. Indeed, it was so controversial that whole books werepublished describing how this couldn’t possibly work, and how all the proponentswere knuckle-dragging, money-grubbing, nitwits who never wrote a line of code intheir lives and….

Ah, but I shouldn’t let those old feelings overtake me… Because, after all, they’regone — and we’re still here.

Look at those twelve practices. Which ones don’t you do? Most of you, my gentlereaders, likely do most of these practices on a regular basis. While it’s certainly astretch to say that they have become universal, it is by no means a stretch to saythat they are now considered main-stream. What’s more, those teams that don’t doall these practices today, are trying to move towards them. These practices havebecome an ideal, a goal to be achieved as opposed to a heresy to be reviled.

The Churn

The last fourteen years have been strange. The Agile movement, which was spawnedout of the controversy over Extreme Programming, skyrocketed into success, andwas subsequently taken over by the project managers who all but pushed theprogrammers out. We’ve seen the creation, the wild success, and the corresponding(and predictable) impotence, of certifications. We saw the adoption of the planninggame (i.e. SCRUM) without the other eleven practices; and we saw that strategyfail — becoming what Martin Fowler called: Flaccid Scrum. We’ve experienced


49

continuous and vocal process churn as consultants and authors split and competedover Kanban, Lean, and every new project-management prefix-of-the-day. We’veseen the growth of the software craftsmanship movement, and the slow degradationand dilution of the Agile meme.

But in the midst of all that hype and churn, those twelve practices have remained.Some of their names have changed a bit. 40 Hour Week became Sustainable Rate.Testing became TDD. Metaphor became DDD. Small Releases became ContinuousIntegration and Continuous Deployment. But despite these changes the practicesremain very much as they were described fourteen years ago.

We also saw the name Extreme Programming fade almost entirely out of use. Veryfew people use that term nowadays. Some still use the abbreviation XP; but for themost part the name has evaporated. It is very rare for me to hear a team describewhat they do as Extreme Programming, even when they are practicing all twelvepractices as described. The names change. The practices remain. The practices arepersistent.

Amidst the churn, the hype, the controversy, the bluster and blather. Amidst all thechaos of humans jockeying for position over one-another. Amidst all the messinessof human avarice, passion, and pride. Amidst all that politics, the practices persist.

Stable Values

I believe the practices persist because they are based on a firm foundation of stablevalues. Values that Kent Beck described in Chapter 7 on page 29 of his book:

• Communication

• Simplicity

• Feedback

• Courage.

I could try to argue why these are the right values; but I think they speak forthemselves. What software craftsman would reject any one of those values? Whatsoftware craftsman would not strive to ensure that each one of those values wererepresented in their work? These values are values of software craftsmanship.

I could try to argue that the twelve practices embrace and exemplify thesevalues, but their persistence — despite the churn and dissolution of the names andmovements that surrounded them, is evidence enough.


50

Success

Extreme Programming succeeded! It succeeded beyond the wildest dreams of itsproponents. It succeeded because it survived the controversy of its birth and thesubsequent, and inevitable, churn of its advocacy. It succeeded because it outlivedeven its own name!

Extreme Programming has succeeded in the way that Structured Programmingsucceeded. Nobody even thinks about structured programming any more — they justdo it. Nobody even thinks about Extreme Programming any more, we are all justtrying to do it.

That’s success! An idea succeeds when it outlives the movement that spawns it andsimply becomes part of our everyday lives. That’s SUCCESS!

Looking Back

So today, in these last weeks of 2013, take a moment to reflect back on 1999. A timewhen Kent Beck wrote a ground-breaking book. A book that changed everything.Look back and remember: Extreme Programming; and recognize it as the core ofwhat we, today, simply think of as:

Good Software Practice.

51

Surviving Legacy Code With GoldenMaster and SamplingWritten by: J. B. Rainsberger at 2014-09-281

You have inherited some code. Congratulations. Now you need to change it.

There, there.

Michael Feathers once wrote that legacy code is “code without unit tests”. I use aslightly more general definition.

Legacy code is valuable code that we feel afraid to change.

I think that both parts matter. You probably accepted the “afraid to change” partwithout any need for convincing. (If not, then this article probably won’t interest you.)Moreover, if the code doesn’t generate significant value, then I don’t see much risk inchanging it. If the cost of “getting it wrong” doesn’t significantly outweigh the profitwe derive from “getting it right”, then who cares? Probably not I.

I treat valuable code with considerable respect. It provides food for families. I treatdifficult-to-change code also with consider respect, although this comes more fromfear than admiration. If we put these two things together, then, quite simply, onefalse move and I might destroy an order of magnitude more profit than the yearlycost to keep me around.

This brings me to Rule Number Zero of Surviving Legacy Code:

Maximise safety.

We find ourselves in the typical chicken-and-egg problem: we want to write testsin order to refactor more safely, but then we remember that integrated tests are ascam℠2 and decide that we’d rather break things apart a little in order to write less-costly-to-maintain tests. So which do we do first?

1 http://blog.thecodewhisperer.com/2014/09/28/surviving-legacy-code-with-golden-master-and-sampling/2You can find a series of articles on that topic at http://link.jbrains.ca/17NH9w3

http://blog.thecodewhisperer.com/2014/09/28/surviving-legacy-code-with-golden-master-and-sampling/



http://link.jbrains.ca/17NH9w3

Surviving Legacy Code With Golden Master and Sampling

52

In a situation like this, I like to go back to my guiding principles.

Integrated tests are a scam℠ in part because they don’t put enough positive pressureon my designs and thereby don’t give me enough useful design feedback. Right now,I don’t care about this. I already know that the design needs significant work. I alsoknow that I can’t handle the torrent of feedback that microtests would give me aboutthe design.34 If I want to use this principle to guide my behavior, then I need to findanother justification.

Integrated tests remain a scam℠ in part because of the combinatoric explosionin the number of tests I need to achieve a strong level of coverage, which in thiscase correlates to confidence. I might have to write millions of tests to achieve highcoverage. I probably only have time to write hundreds of tests, in which case I haveto gamble about the level of coverage. Perchance, could I not care about coveragein this situation?

Test coverage — however one measures or defines it — links directly to safety inchanging code. I want to use those tests as change detectors. I want the red lightthat flashes the moment I make a mistake. Microtests, especially if I write them first,give me that. They help me find mistakes immediately. They help drive down thecost of making a mistake, an essential technique for managing risk.5 If I can’t writemicrotests cost-effectively, then what can I do?

What if, instead of a red light that flashes the moment I make (almost) any mistake,I had a pink light that flashes when I make a really obvious mistake? I can’t havewhat I want, but I can afford this; will it do? It will help more than doing nothing. Iwill simply buy as much of this confidence as I can afford. To do this, I combine twosimple ideas: Golden Master and sampling.

3When diving into legacy code, I find it more important than ever to keep stuff out of my head. Duringthe two hours it takes to safely refactor some large function, I’m probably going to spot 14 potentially-useful refactorings. I can’t chase every bunny, no matter how cute they are. I need to write those ideasdown, get them out of my head, and get back to the tricky surgery at hand.4 I see little point in spending energy generating a backlog knowing full well that I will never get aroundto doing about 80% of it. Who would volunteer to do that? (Ask your project manager if value-drivenproduct [http://www.jbrains.ca/training/value-driven-product-development] development is right forem.)5 I claim that “the agile approach” to risk management complements the typical approach to riskmanagement of limiting the probability of failure in order to limit exposure. “The agile way”, if you willpermit me to use this shorthand, involves limiting the cost of failure instead. Eventually I will replacethis sentence with a link to an article that goes into this topic in more detail.

http://www.jbrains.ca/training/value-driven-product-development




53

Golden Master

I use Golden Master to help me detect changes in the behavior of a system when Ican’t justify writing the typical kind of assertion that you’ve grown used to seeing intests. I use this trick, for example, when I find it difficult to articulate the expectedresult of a test. Imagine a function whose output consists of an image. It happensquite often that a binary comparison between actual and expected result yields ahyperactive assertion—one which frequently fails even when a human would judgethat the test had passed. I suppose some people know tricks to make it easier toarticulate “looks similar enough” for images, but I don’t know how to do that, and thatleaves me to choose either a hyperactive bit-by-bit comparison or ongoing, manualinspection. Rather than revert to the Guru Checks Output antipattern 6, however, Itake a snapshot of the last-known acceptable output—I call that the golden master— and save it for future use. When I run the test again, I compare the output to thegolden master, and if they match, then the test passes; if they don’t match, thenthe test fails. This doesn’t make the code wrong, but it means that I need to checkthe result and decide whether the code needs fixing or the golden master needsreplacing.

You can use Golden Master wherever you already have some output to check, evenif you find the form of that output particularly challenging. With this technique,you simply diff the output and inspect the situation only when you find differencesbetween the current test run and the golden master. If your system already sendstext to an output stream that you can capture, then you have the tools to use thistechnique.

I warn you, however, not to give in to the temptation to start scraping your outputfor specific information to check. Unless you have no other alternative, you willprobably find it more cost-effective to carefully extract that information from thecode and check it directly using good, old-fashioned assertEquals() . Don’t builda parser for an arbitrary, unplanned, probably context-sensitive grammar. That waylies madness. (Of course, if a context-free grammar happens to describe the format,then go for it. You’ve always wanted to learn lexx and yacc , haven’t you?)

6Marcia, the guru, looks at the output, pauses for a moment, then says, “Yep. That’s it.” If you want tore-run the test, then you need Marcia. That doesn’t seem to scale particularly well.


54

Sampling

I find one glaring problem with the Golden Master technique: if the output describes along-running algorithm, process, or path through the system, then the golden masteritself might describe only one of a thousand, million, or even billion potentially-interesting possible outputs. Welcome back to the combinatoric explosion problemthat makes integrated tests such a scam℠. How do we proceed when we can’tpossibly check the variety of paths through the system that we need to check?

Ideally, we refactor! I know that if I can break my system into many smaller,composable pieces, then I turn products into sums: instead of checking combinationsof paths through multiple parts of the system at once, I can check the handful ofpairwise connexions between parts of the system in relative isolation. I could turnmillions of tests into hundreds. Unfortunately, in our current situation, I don’t feelcomfortable refactoring, so that means that I have to sample the inputs and hopefor the best.

You can find more sophisticated sampling systems out there among blogs written byexperienced testers, but they all amount to sampling: if I can’t try every combinationof inputs, then I try some combinations of some of the inputs and aim for the bestcoverage that I can.

This shouldn’t surprise you. You’ve done this before. You’ve written a functionthat operates on an integer, and you knew enough about the algorithm to identifyboundary cases at, for example, -1, 0, and 1, as well as around 100 and 1000, so youcheck on the order of ten inputs and feel satisfied that the algorithm will work for theremaining few billion inputs. You were sampling.

In the case of legacy code, however, sometimes we can’t sample quite sointentionally. Sometimes even when we limit our scope to characteristic inputs, wehave so many combinations of those inputs that we still can’t afford to write and runall those tests. In some cases, we don’t even know how to identify the characteristicinputs. In other cases, the algorithm itself has a random element, defeating our goalof writing nice, deterministic, repeatable tests. Random sampling to the rescue.

If you can use the random number generator to generate a stream of inputs toyour system, then you can use this generate a collection of output files, and thatcollection can act as your golden master. You only need to control the randomnumber generator by seeding it with the same stream of seeds every time. I usea simple linear generating function like m + p * i where m and p representarbitrarily-chosen numbers and i represents a loop index. Now I simply have to


55

decide how big a sample to take. Generally speaking, a larger sample gives me moreconfidence in the sensitivity of the pink flashing light that signals danger.

I adjust the size of the sample depending on how long it takes to execute a test run,and how much significantly that affects my flow while programming. I also adjustthe size of the sample to match my fear level: the more worried I feel about gettingsomething wrong, the larger sample I take while working, and I accept the cost ofslowing down. I’d usually rather go a little too slow than a little too fast, becauseI know that the cost of making a mistake would likely dominate the savings fromgoing more quickly.

The Techniques in Action

You can see an example of this technique in action by reading this code7. If you’dlike to see how I added this behavior to some legacy code, then start at this commit8

and follow the process step by step.

Although these techniques do not, on their own, guarantee success, when I combineGolden Master and Sampling, I can usually find a way to proceed safely. When Icombine these with microcommitting 9, I can proceed at an even quicker pace. Theyhelp me avoid the Catch-22 problem that arises from needing to refactor dangerouslyunsafely in order to be able to refactor safely and sensibly. Where might you useGolden Master and Sampling to help get your arms (at least a little) around yourlegacy code?

References

• Michael Feathers, Working Effectively with Legacy Code10. Still the classic workon winding your way through legacy code.

• J. B. Rainsberger, “Getting Started with Getting Things Done”11. You don’t havetime to read Getting Things Done? Start here. Four pages. It’ll be fine.

7 https://github.com/jbrains/SurvivingLegacyCode-solutions/tree/08bb50de492cd5edc089dd358693bd00652306bd8 https://github.com/jbrains/SurvivingLegacyCode-solutions/tree/110dc1308c05a7c43a1d991c66f7dae7633e921a9Really frequent committing, like after changing a single line of code. No, really. Eventually I willreplace this sentence with a reference to an article that explores this topic in more detail.10 http://www.amazon.com/gp/product/0131177052/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&tag=jbrains.ca-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=013117705211 http://articles.jbrains.ca/GettingStartedWithGettingThingsDone.pdf

https://github.com/jbrains/SurvivingLegacyCode-solutions/tree/08bb50de492cd5edc089dd358693bd00652306bd

https://github.com/jbrains/SurvivingLegacyCode-solutions/tree/110dc1308c05a7c43a1d991c66f7dae7633e921a

http://www.amazon.com/gp/product/0131177052/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&tag=jbrains.ca-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131177052

http://articles.jbrains.ca/GettingStartedWithGettingThingsDone.pdf







http://articles.jbrains.ca/GettingStartedWithGettingThingsDone.pdf


56

• David Allen, Getting Things Done12. I use it. Not all of it, and not all the time, butI use its core principles quite significantly in managing my work and home lives.No cult, I promise.

• J. B. Rainsberger, “A Proven Path to Effectiveness”13. A “method” that combinesGetting Things Done and Test-Driven Development aimed specifically at thebeleaguered programmer.

• texttest.org14. A library to help you write text-based tests, such as I would use toprovide golden masters. Do not download this tool until you have writtenyour own golden master at least once. That is an order. After that, useTextTest, because it really helps.

12 http://www.amazon.com/gp/product/0142000280/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0142000280&linkCode=as2&tag=jbrains.ca-2013 http://www.amazon.com/gp/product/0142000280/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0142000280&linkCode=as2&tag=jbrains.ca-2014 http://texttest.sourceforge.net/

http://www.amazon.com/gp/product/0142000280/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0142000280&linkCode=as2&tag=jbrains.ca-20


http://texttest.sourceforge.net/





http://texttest.sourceforge.net/

57

Hexagonal ArchitectureWritten by: Alistair. Cockburn at 2005-01-041

Create your application to work without either a UI or a database so you canrun automated regression-tests against the application, work when the databasebecomes unavailable, and link applications together without any user involvement.

The Pattern: Ports and Adapters (‘’ObjectStructural’’)

• Alternative name: ‘’Ports & Adapters’’

• Alternative name: ‘’Hexagonal Architecture’’

Intent

Allow an application to equally be driven by users, programs, automated test or batchscripts, and to be developed and tested in isolation from its eventual run-time devicesand databases.

As events arrive from the outside world at a port, a technology-specific adapterconverts it into a usable procedure call or message and passes it to the application.The application is blissfully ignorant of the nature of the input device. When the

1 http://alistair.cockburn.us/Hexagonal+architecture

http://alistair.cockburn.us/Hexagonal+architecture

http://alistair.cockburn.us/Hexagonal+architecture

Hexagonal Architecture

58

application has something to send out, it sends it out through a port to an adapter,which creates the appropriate signals needed by the receiving technology (human orautomated). The application has a semantically sound interaction with the adapterson all sides of it, without actually knowing the nature of the things on the other sideof the adapters.

Motivation

One of the great bugaboos of software applications over the years has beeninfiltration of business logic into the user interface code. The problem this causesis threefold:

• First, the system can’t neatly be tested with automated test suites because partof the logic needing to be tested is dependent on oft-changing visual details suchas field size and button placement;

• For the exact same reason, it becomes impossible to shift from a human-drivenuse of the system to a batch-run system;

• For still the same reason, it becomes difficult or impossible to allow the programto be driven by another program when that becomes attractive.

The attempted solution, repeated in many organizations, is to create a new layer inthe architecture, with the promise that this time, really and truly, no business logic


59

will be put into the new layer. However, having no mechanism to detect when aviolation of that promise occurs, the organization finds a few years later that the newlayer is cluttered with business logic and the old problem has reappeared.

Imagine now that ‘’every’’ piece of functionality the application offers were availablethrough an API (application programmed interface) or function call. In this situation,the test or QA department can run automated test scripts against the applicationto detect when any new coding breaks a previously working function. The businessexperts can create automated test cases, before the GUI details are finalized, thattells the programmers when they have done their work correctly (and these testsbecome the ones run by the test department). The application can be deployed in‘’headless’’ mode, so only the API is available, and other programs can make useof its functionality — this simplifies the overall design of complex application suitesand also permits business-to-business service applications to use each other withouthuman intervention over the web. Finally, the automated function regression testsdetect any violation of the promise to keep business logic out of the presentationlayer. The organization can detect, and then correct, the logic leak.

An interesting similar problem exists on what is normally considered “the other side”of the application, where the application logic gets tied to an external databaseor other service. When the database server goes down or undergoes significantrework or replacement, the programmers can’t work because their work is tied to thepresence of the database. This causes delay costs and often bad feelings betweenthe people.

It is not obvious that the two problems are related, but there is a symmetry betweenthem that shows up in the nature of the solution.

Nature of the Solution

Both the user-side and the server-side problems actually are caused by the sameerror in design and programming — the entanglement between the business logicand the interaction with external entities. The asymmetry to exploit is not thatbetween ‘’left’’ and ‘’right’’ sides of the application but between ‘’inside’’ and‘’outside’’ of the application. The rule to obey is that code pertaining to the ‘’inside’’part should not leak into the ‘’outside’’ part.

Removing any left-right or up-down asymmetry for a moment, we see that theapplication communicates over ‘’ports’’ to external agencies. The word “port” issupposed to evoke thoughts of ‘’ports’’ in an operating system, where any device that


60

adheres to the protocols of a port can be plugged into it; and ‘’ports’’ on electronicsgadgets, where again, any device that fits the mechanical and electrical protocolscan be plugged in.

The protocol for a port is given by the purpose of the conversation between the twodevices.

The protocol takes the form of an application program interface (API).

For each external device there is an ‘’adapter’’ that converts the API definition tothe signals needed by that device and vice versa. A graphical user interface or GUIis an example of an adapter that maps the movements of a person to the API ofthe port. Other adapters that fit the same port are automated test harnesses suchas FIT or Fitnesse, batch drivers, and any code needed for communication betweenapplications across the enterprise or net.

On another side of the application, the application communicates with an externalentity to get data. The protocol is typically a database protocol. From theapplication’s perspective, if the database is moved from a SQL database to a flat fileor any other kind of database, the conversation across the API should not change.Additional adapters for the same port thus include an SQL adapter, a flat file adapter,and most importantly, an adapter to a “mock” database, one that sits in memoryand doesn’t depend on the presence of the real database at all.

Many applications have only two ports: the user-side dialog and the database-sidedialog. This gives them an asymmetric appearance, which makes it seem naturalto build the application in a one-dimensional, three-, four-, or five-layer stackedarchitecture.

There are two problems with these drawings. First and worst, people tend not to takethe “lines” in the layered drawing seriously. They let the application logic leak acrossthe layer boundaries, causing the problems mentioned above. Secondly, there maybe more than two ports to the application, so that the architecture does not fit intothe one-dimensional layer drawing.

The hexagonal, or ports and adapters, architecture solves these problems by notingthe symmetry in the situation: there is an application on the inside communicatingover some number of ports with things on the outside. The items outside theapplication can be dealt with symmetrically.

The hexagon is intended to visually highlight


61

1. the inside-outside asymmetry and the similar nature of ports, to get away fromthe one-dimensional layered picture and all that evokes, and

2. the presence of a defined number of different ports – two, three, or four (four ismost I have encountered to date).

The hexagon is not a hexagon because the number six is important, but rather toallow the people doing the drawing to have room to insert ports and adapters asthey need, not being constrained by a one-dimensional layered drawing. The term‘’hexagonal architecture’’ comes from this visual effect.

The term “port and adapters” picks up the ‘’purposes’’ of the parts of the drawing.A port identifies a purposeful conversation. There will typically be multiple adaptersfor any one port, for various technologies that may plug into that port. Typically,these might include a phone answering machine, a human voice, a touch-tone phone,a graphical human interface, a test harness, a batch driver, an http interface, adirect program-to-program interface, a mock (in-memory) database, a real database(perhaps different databases for development, test, and real use).

In the Application Notes, the left-right asymmetry will be brought up again. However,the primary purpose of this pattern is to focus on the inside-outside asymmetry,pretending briefly that all external items are identical from the perspective of theapplication. Structure


62

Figure 2 shows an application having two active ports and several adapters for eachport. The two ports are the application-controlling side and the data-retrieval side.This drawing shows that the application can be equally driven by an automated,system-level regression test suite, by a human user, by a remote http application,or by another local application. On the data side, the application can be configuredto run decoupled from external databases using an in-memory oracle, or ‘’mock’’,database replacement; or it can run against the test- or run-time database. Thefunctional specification of the application, perhaps in use cases, is made against theinner hexagon’s interface and not against any one of the external technologies thatmight be used.

Figure 3 shows the same application mapped to a three-layer architectural drawing.To simplify the drawing only two adapters are shown for each port. This drawingis intended to show how multiple adapters fit in the top and bottom layers, andthe sequence in which the various adapters are used during system development.The numbered arrows show the order in which a team might develop and use theapplication:


63

1. With a FIT test harness driving the application and using the mock (in-memory)database substituting for the real database;

2. Adding a GUI to the application, still running off the mock database;

3. In integration testing, with automated test scripts (e.g., from Cruise Control)driving the application against a real database containing test data;

4. In real use, with a person using the application to access a live database.

Sample Code

The simplest application that demonstrates the ports & adapters fortunately comeswith the FIT documentation. It is a simple discount computing application:

discount(amount) = amount * rate(amount);

In our adaptation, the amount will come from the user and the rate will come from adatabase, so there will be two ports. We implement them in stages:

• With tests but with a constant rate instead of a mock database,

• then with the GUI,

• then with a mock database that can be swapped out for a real database.

Thanks to Gyan Sharma at IHC for providing the code for this example.

Stage 1: FIT App constant-as-mock-database

First we create the test cases as an HTML table (see the FIT documentation for this):

Table 1. TestDiscounter

amount discount()

100 5

200 10

Note that the column names will become class and function names in our program.FIT contains ways to get rid of this “programmerese”, but for this article it is easierjust to leave them in.


64

Knowing what the test data will be, we create the user-side adapter, theColumnFixture that comes with FIT as shipped:

import fit.ColumnFixture;public class TestDiscounter extends ColumnFixture{ private Discounter app = new Discounter(); public double amount; public double discount() { return app.discount(amount); }}

That’s actually all there is to the adapter. So far, the tests run from the commandline (see the FIT book for the path you’ll need). We used this one:

set FIT_HOME=/FIT/FitLibraryForFit15Feb2005java -cp %FIT_HOME%/lib/javaFit1.1b.jar;%FIT_HOME%/dist/fitLibraryForFit.jar;src;binfit.FileRunner test/Discounter.html TestDiscount_Output.html

FIT produces an output file with colors showing us what passed (or failed, in case wemade a typo somewhere along the way).

At this point the code is ready to check in, hook into Cruise Control or your automatedbuild machine, and include in the build-and-test suite.

Stage 2: UI App constant-as-mock-database

I’m going to let you create your own UI and have it drive the Discounter application,since the code is a bit long to include here. Some of the key lines in the code arethese:

... Discounter app = new Discounter();public void actionPerformed(ActionEvent event){ ... String amountStr = text1.getText(); double amount = Double.parseDouble(amountStr); discount = app.discount(amount)); text3.setText( "" + discount ); ...


65

At this point the application can be both demoed and regression tested. The user-side adapters are both running.

Stage 3: (FIT or UI) App mock database

To create a replaceable adapter for the database side, we create an ‘’interface’’ to arepository, a ‘’RepositoryFactory’’ that will produce either the mock database or thereal service object, and the in-memory mock for the database.

public interface RateRepository{ double getRate(double amount); }public class RepositoryFactory{ public RepositoryFactory() { super(); } public static RateRepository getMockRateRepository() { return new MockRateRepository(); }}public class MockRateRepository implements RateRepository{ public double getRate(double amount) { if(amount <= 100) return 0.01; if(amount <= 1000) return 0.02; return 0.05; } }

To hook this adapter into the Discounter application, we need to update theapplication itself to accept a repository adapter to use, and the have the (FIT or UI)user-side adapter pass the repository to use (real or mock) into the constructor ofthe application itself. Here is the updated application and a FIT adapter that passesin a mock repository (the FIT adapter code to choose whether to pass in the mock orreal repository’s adapter is longer without adding much new information, so I omitthat version here).

import repository.RepositoryFactory;import repository.RateRepository;public class Discounter{


66

private RateRepository rateRepository; public Discounter(RateRepository r) { super(); rateRepository = r; } public double discount(double amount) { double rate = rateRepository.getRate( amount ); return amount * rate; }}import app.Discounter;import fit.ColumnFixture;public class TestDiscounter extends ColumnFixture{ private Discounter app = new Discounter(RepositoryFactory.getMockRateRepository()); public double amount; public double discount() { return app.discount( amount ); }}

That concludes implementation of the simplest version of the hexagonal architecture.

Application Notes

The Left-Right Asymmetry

The ports and adapters pattern is deliberately written pretending that all portsare fundamentally similar. That pretense is useful at the architectural level.In implementation, ports and adapters show up in two flavors, which I’ll call‘’primary’’ and ‘’secondary’’, for soon-to-be-obvious reasons. They could be alsocalled ‘’driving’’ adapters and ‘’driven’’ adapters.

The alert reader will have noticed that in all the examples given, FIT fixtures are usedon the left-side ports and mocks on the right. In the three-layer architecture, FIT sitsin the top layer and the mock sits in the bottom layer.

This is related to the idea from use cases of “primary actors” and “secondary actors”.A ‘’primary actor’’ is an actor that drives the application (takes it out of quiescent


67

state to perform one of its advertised functions). A ‘’secondary actor’’ is one thatthe application drives, either to get answers from or to merely notify. The distinctionbetween ‘’primary ‘’and’’ secondary ‘’lies in who triggers or is in charge of theconversation.

The natural test adapter to substitute for a ‘’primary’’ actor is FIT, since thatframework is designed to read a script and drive the application. The natural testadapter to substitute for a ‘’secondary’’ actor such as a database is a mock, sincethat is designed to answer queries or record events from the application.

These observations lead us to follow the system’s use case context diagram and drawthe ‘’primary ports ‘’and’’ primary adapters’’ on the left side (or top) of the hexagon,and the ‘’secondary ports’’ and ‘’secondary adapters’’ on the right (or bottom) sideof the hexagon.

The relationship between primary and secondary ports/adapters and their respectiveimplementation in FIT and mocks is useful to keep in mind, but it should be used asa consequence of using the ports and adapters architecture, not to short-circuit it.The ultimate benefit of a ports and adapters implementation is the ability to run theapplication in a fully isolated mode.

Use Cases And The Application Boundary

It is useful to use the hexagonal architecture pattern to reinforce the preferred wayof writing use cases.

A common mistake is to write use cases to contain intimate knowledge of thetechnology sitting outside each port. These use cases have earned a justifiably badname in the industry for being long, hard-to-read, boring, brittle, and expensive tomaintain.

Understanding the ports and adapters architecture, we can see that the use casesshould generally be written at the application boundary (the inner hexagon), tospecify the functions and events supported by the application, regardless of externaltechnology. These use cases are shorter, easier to read, less expensive to maintain,and more stable over time.

How Many Ports?

What exactly a port is and isn’t is largely a matter of taste. At the one extreme,every use case could be given its own port, producing hundreds of ports for many


68

applications. Alternatively, one could imagine merging all primary ports and allsecondary ports so there are only two ports, a left side and a right side.

Neither extreme appears optimal.

The weather system described in the Known Uses has four natural ports: the weatherfeed, the administrator, the notified subscribers, the subscriber database. A coffeemachine controller has four natural ports: the user, the database containing therecipes and prices, the dispensers, and the coin box. A hospital medication systemmight have three: one for the nurse, one for the prescription database, and one forthe computer-controller medication dispensers.

It doesn’t appear that there is any particular damage in choosing the “wrong” numberof ports, so that remains a matter of intuition. My selection tends to favor a smallnumber, two, three or four ports, as described above and in the Known Uses.

Known Uses

Figure 4 shows an application with four ports and several adapters at each port. Thiswas derived from an application that listened for alerts from the national weatherservice about earthquakes, tornadoes, fires and floods, and notified people on theirtelephones or telephone answering machines. At the time we discussed this system,the system’s interfaces were identified and discussed by ‘’technology, linked topurpose’’. There was an interface for trigger-data arriving over a wire feed, one


69

for notification data to be sent to answering machines, an administrative interfaceimplemented in a GUI, and a database interface to get their subscriber data.

The people were struggling because they needed to add an http interface fromthe weather service, an email interface to their subscribers, and they had to find away to bundle and unbundle their growing application suite for different customerpurchasing preferences. They feared they were staring at a maintenance and testingnightmare as they had to implement, test and maintain separate versions for allcombinations and permutations.

Their shift in design was to architect the system’s interfaces ‘’by purpose’’ ratherthan by technology, and to have the technologies be substitutable (on all sides) byadapters. They immediately picked up the ability to include the http feed and theemail notification (the new adapters are shown in the drawing with dashed lines).By making each application executable in headless mode through APIs, they couldadd an app-to-add adapter and unbundle the application suite, connecting the sub-applications on demand. Finally, by making each application executable completelyin isolation, with test and mock adapters in place, they gained the ability to regressiontest their applications with stand-alone automated test scripts.

Mac, Windows, Google, Flickr, Web 2.0

In the early 1990s, MacIntosh applications such as word processor applications wererequired to have API-drivable interfaces, so that applications and user-written scriptscould access all the functions of the applications. Windows desktop applications haveevolved the same ability (I don’t have the historical knowledge to say which camefirst, nor is that relevant to the point).

The current (2005) trend in web applications is to publish an API and let other webapplications access those APIs directly. Thus, it is possible to publish local crime dataover a Google map, or create web applications that include Flickr’s photo archivingand annotating abilities.

All of these examples are about making the ‘’primary ‘’ports’ APIs visible. We see noinformation here about the secondary ports.

Stored Outputs

This example written by Willem Bogaerts on the C2 wiki:

“I encountered something similar, but mainly because my application layer had astrong tendency to become a telephone switchboard that managed things it should


70

not do. My application generated output, showed it to the user and then had somepossibility to store it as well. My main problem was that you did not need to storeit always. So my application generated output, had to buffer it and present it to theuser. Then, when the user decided that he wanted to store the output, the applicationretrieved the buffer and stored it for real.

I did not like this at all. Then I came up with a solution: Have a presentation controlwith storage facilities. Now the application no longer channels the output in differentdirections, but it simply outputs it to the presentation control. It’s the presentationcontrol that buffers the answer and gives the user the possibility to store it.

The traditional layered architecture stresses “UI” and “storage” to be different. ThePorts and Adapters Architecture can reduce output to being simply “output” again. ”

Anonymous example from the C2-wiki

“In one project I worked on, we used the SystemMetaphor of a component stereosystem. Each component has defined interfaces, each of which has a specificpurpose. We can then connect components together in almost unlimited ways usingsimple cables and adapters.”

Distributed, Large-Team Development

This one is still in trial use and so does not properly count as a use of the pattern.However, it is interesting to consider.

Teams in different locations all build to the Hexagonal architecture, using FIT andmocks so the applications or components can be tested in standalone mode. TheCruiseControl build runs every half hour and runs all the applications using the FIT+mock combination. As application subsystem and databases get completed, themocks are replaced with test databases.

Separating Development of UI and Application Logic

This one is still in early trial use and so does not count as a use of the pattern.However, it is interesting to consider.

The UI design is unstable, as they haven’t decided on a driving technology or ametaphor yet. The back-end services architecture hasn’t been decided, and in factwill probably change several times over the next six months. Nonetheless, the projecthas officially started and time is ticking by.


71

The application team creates FIT tests and mocks to isolate their application, andcreates testable, demonstrable functionality to show their users. When the UI andback-end services decisions finally get met, it “should be straightforward” to addthose elements the application. Stay tuned to learn how this works out (or try ityourself and write me to let me know).

Related Patterns

Adapter

The ‘’Design Patterns’’ book contains a description of the generic ‘’Adapter’’ pattern:“Convert the interface of a class into another interace clients expect.” The ports-and-adapters pattern is a particular use of the ‘’Adapter’’ pattern.

Model-View-Controller

The MVC pattern was implemented as early as 1974 in the Smalltalk project. It hasbeen given, over the years, many variations, such as Model-Interactor and Model-View-Presenter. Each of these implements the idea of ports-and-adapters on theprimary ports, not the secondary ports.

Mock Objects and Loopback

“A mock object is a “double agent” used to test the behaviour of other objects. First,a mock object acts as a faux implementation of an interface or class that mimics theexternal behaviour of a true implementation. Second, a mock object observes howother objects interact with its methods and compares actual behaviour with presetexpectations. When a discrepancy occurs, a mock object can interrupt the test andreport the anomaly. If the discrepancy cannot be noted during the test, a verificationmethod called by the tester ensures that all expectations have been met or failuresreported.” — From http://MockObjects.com

Fully implemented according to the mock-object agenda, mock objects are usedthroughout an application, not just at the external interface The primary thrust of themock object movement is conformance to specified protocol at the individual classand object level. I borrow their word “mock” as the best short description of an in-memory substitute for an external secondary actor.

The Loopback pattern is an explicit pattern for creating an internal replacement foran external device.

http://MockObjects.com


72

Pedestals

In “Patterns for Generating a Layered Architecture”, Barry Rubel describes a patternabout creating an axis of symmetry in control software that is very similar to portsand adapters. The ‘’Pedestal’’ pattern calls for implementing an object representingeach hardware device within the system, and linking those objects together in acontrol layer. The ‘’Pedestal’’ pattern can be used to describe either side of thehexagonal architecture, but does not yet stress the similarity across adapters. Also,being written for a mechanical control environment, it is not so easy to see how toapply the pattern to IT applications.

Checks

Ward Cunningham’s pattern language for detecting and handling user input errors,is good for error handling across the inner hexagon boundaries.

Dependency Inversion (Dependency Injection) andSPRING

Bob Martin’s Dependency Inversion Principle (also called Dependency Injection byMartin Fowler) states that “High-level modules should not depend on low-levelmodules. Both should depend on abstractions. Abstractions should not depend ondetails. Details should depend on abstractions.” The ‘’Dependency Injection ‘’patternby Martin Fowler gives some implementations. These show how to create swappablesecondary actor adapters. The code can be typed in directly, as done in the samplecode in the article, or using configuration files and having the SPRING frameworkgenerate the equivalent code.

Acknowledgements

Thanks to Gyan Sharma at Intermountain Health Care for providing the sample codeused here. Thanks to Rebecca Wirfs-Brock for her book ‘’Object Design’’, which whenread together with the ‘’Adapter’’ pattern from the ‘’Design Patterns’’ book, helpedme to understand what the hexagon was about. Thanks also to the people on Ward’swiki, who provided comments about this pattern over the years (e.g., particularlyKevin Rutherford’s http://silkandspinach.net/blog/2004/07/hexagonal_soup.html).

http://silkandspinach.net/blog/2004/07/hexagonal_soup.html


73

References and Related Reading

• FIT, A Framework for Integrating Testing: Cunningham, W., online at http://fit.c2.com, and Mugridge, R. and Cunningham, W., ‘’Fit for Developing Software’’,Prentice-Hall PTR, 2005.

• The ‘’Adapter’’ pattern: in Gamma, E., Helm, R., Johnson, R., Vlissides, J., ‘’DesignPatterns’’, Addison-Wesley, 1995, pp. 139-150.

• The ‘’Pedestal’’ pattern: in Rubel, B., “Patterns for Generating a LayeredArchitecture”, in Coplien, J., Schmidt, D., ‘’PatternLanguages of Program Design’’,Addison-Wesley, 1995, pp. 119-150.

• The ‘’Checks’’ pattern: by Cunningham, W., online at http://c2.com/ppr/checks.html

• The ‘’Dependency Inversion Principle’‘ : Martin, R., in ‘’Agile SoftwareDevelopment Principles Patterns and Practices’’, Prentice Hall, 2003,Chapter 11: “The Dependency-Inversion Principle”, and online at http://www.objectmentor.com/resources/articles/dip.pdf

• The ‘’Dependency Injection’’ pattern: Fowler, M., online at http://www.martinfowler.com/articles/injection.html

• The ‘’Mock Object’’ pattern: Freeman, S. online at http://MockObjects.com

• The ‘’Loopback’’ pattern: Cockburn, A., online at http://c2.com/cgi/wiki?LoopBack

• ‘’Use cases:’’ Cockburn, A., ‘’Writing Effective Use Cases’’, Addison-Wesley,2001, and Cockburn, A., “Structuring Use Cases with Goals”, online at http://alistair.cockburn.us/crystal/articles/sucwg/structuringucswithgoals.htm

http://fit.c2.com

http://fit.c2.com

http://c2.com/ppr/checks.html

http://c2.com/ppr/checks.html

http://www.objectmentor.com/resources/articles/dip.pdf

http://www.objectmentor.com/resources/articles/dip.pdf

http://www.martinfowler.com/articles/injection.html

http://www.martinfowler.com/articles/injection.html

http://MockObjects.com

http://c2.com/cgi/wiki?LoopBack

http://alistair.cockburn.us/crystal/articles/sucwg/structuringucswithgoals.htm

http://alistair.cockburn.us/crystal/articles/sucwg/structuringucswithgoals.htm

74

Stop. Write a learning testWritten by: J. B. Rainsberger at 2011-12-14 (updated 2013-10-28)1

The 30-second version

• Where did that yak come from?2

• When you try to learn a new library at the same time as explore the behavior anddesign of your application, you slow down more than you think.

• When you can’t figure out how to make the new library work for this thing youwant to build, you might spend hours fighting, debugging, swearing.

Stop. Write a Learning Test.

1. Start a new test suite, test class, spec file, whatever you want to call it.

2. Write a test that checks the things you tried to check earlier with debugstatements.

3. Write a test that has nothing to do with your application and its domain.

4. Remove unnecessary details from your test.

When this test passes, then you understand what that part of the library does. If itbehaves strangely, then you have the perfect test to send to the maintainers of thelibrary 3.

The Details

I just did this on a project using the context-free grammar parser treetop4. Of course,I hadn’t used treetop before, so I had to learn it at the same time as design thegrammar for the language I wanted to parse. I reached the point where I couldn’twrite a grammar rule correctly, and spent probably an hour trying to figure out getit to work 5. Fortunately, at that moment, my laptop ran out of power, so I left the

1 http://blog.thecodewhisperer.com/2011/12/14/when-to-write-learning-tests/2 http://www.google.com/search?q=yak+shaving3Remember, we don’t call them bugs anymore: we call them “mistakes”. In this case, we can’t call it a“mistake” yet, because we might simply have a difference of opinion or mindset.4 https://github.com/nathansobo/treetop5Note the wording: I already assuming that I have it right and they have it wrong. Bad programmer.

http://blog.thecodewhisperer.com/2011/12/14/when-to-write-learning-tests/

http://www.google.com/search?q=yak+shaving

https://github.com/nathansobo/treetop

http://blog.thecodewhisperer.com/2011/12/14/when-to-write-learning-tests/

http://www.google.com/search?q=yak+shaving

https://github.com/nathansobo/treetop

Stop. Write a learning test

75

coffee shop 6 and did the usual thing: I explained the problem to my wife so thatI could hear myself doing that. After about 15 minutes away from the problem, Idecided to write some Learning Tests.

Summary of what I did

1. I wrote a Learning Test for a simple case that I thought I already understood well.

2. I wrote a Learning Test similar to the problem I had to deal with, to make sureI understood that well.

3. I wrote a Learning Test for the exact case that behaved unexpectedly.

The whole thing took an hour, and I understood the problem well enough to explain itto my wife. She understood it and agreed that it sounded like a mistake in the library7. I used this Learning Test to open an issue at github. Now I can proceed withoutpulling my own hair out.

Do you want to see the Learning Tests?

The case I already understood well (single_simple_rule_spec.rb)

require 'treetop'

describe "Grammar with a simple rule" do let(:subject) { Treetop.load_from_string(<<GRAMMARgrammar SimpleRule rule word [A-Za-z]+ endendGRAMMAR )}

let (:parser) { subject.new }

it "doesn't match empty string" do parser.parse("").should be_false end

6They have a Second Cup in Romania. Canadians get why I’d find that weird.7Smart woman, my wife.


76

context "matching single letter, the match result" do let(:result) { parser.parse("a") }

it { result.should be_true } it { result.text_value.should == "a" } it { result.to_s.should_not == "a" } it { result.should_not respond_to(:word) } end

context "matching many letters, the match result" do let(:result) { parser.parse("aBcDeF") }

it { result.should be_true } it { result.text_value.should == "aBcDeF" } it { result.to_s.should_not == "aBcDeF" } it { result.should_not respond_to(:word) } endend

The cases I wasn’t sure I understood (single_rule_using_labels_spec.rb)

require 'treetop'

describe "Grammar with a simple rule that uses a label" do context "Labeled subexpression followed by another expression" do let(:subject) { Treetop.load_from_string(<<GRAMMARgrammar SimpleRuleWithLabel rule word letters:[A-Za-z]+ [A-Za-z]* endendGRAMMAR )}

let (:parser) { subject.new }

context "matching many letters, the match result" do let(:result) { parser.parse("aBcDeF") }

it { result.should respond_to(:letters) } it { result.letters.text_value.should == "aBcDeF" } end end

context "Labeled subexpression without another expression" do


77

it "does not represent a valid grammar, even though I think it should" do lambda { Treetop.load_from_string(<<GRAMMARgrammar SimpleRuleWithLabel rule word letters:[A-Za-z]+ endendGRAMMAR )}.should raise_error(RuntimeError, /Expected \#/) end

it "really should let me refer to the expression as #letters" do pending "https://github.com/nathansobo/treetop/issues/21" end endend

78

The Long, Painful History of TimeWritten by: Erik Naggum at 1999-10-111

The programming language Common Lisp offers a few functions to support theconcept of time as humans experience it, including GET-UNIVERSAL-TIME ,ENCODE-UNIVERSAL-TIME , DECODE-UNIVERSAL-TIME , and GET-DECODED-TIME . These functions assume the existence of a timezone and a daylight savingtime regime, such that they can support the usual expression of time in theenvironment in which a small number of real-life applications run. The majority ofapplications, however, need more support to be able to read and write dates andtimes, calculate with time, schedule events at specific clock times daily, and workwith several time zones and daylight saving time regimes. This paper discussessome of the problems inherent in processing time suitable to humans and describesa solution employed by the author in a number of applications, the LOCAL-TIMEconcept.

0 Introduction

The measurement of time has a very long history, dating back to the first records ofhuman civilization. Yet, the archeological evidence suggests that the concept of timeevolved no further than ordinary human needs, and any notion of time remainedconfined to a fairly short time frame, such as a lifetime past and future. Expressionsof measurements of time were brief and imprecise, rife with the numerous andnefarious assumptions humans bring into their communication, consistent with ourtendency to suppress information believed to be redundant.

For instance, everyone knows which century they are in or that some two-digityear refers to. Until computers came along, the assumptions held by people wereeither recoverable from the context or shared by contemporary communicators. Aftercomputers came to store information for us, we still held onto the context as if thecomputers were as able to recover it as we are. Quite obviously, they aren’t, and inabout three months, we will see whether other humans were indeed able to recoverthe context left unstated by other humans when they wrote down their dates with twodigits and assumed it would never be a problem. The infamous Y2K problem is oneof the few opportunities mankind will get to tally the costs of lack of precision in ourcommon forms of communication. The lesson learned will not be that our notations

1 http://naggum.no/lugm-time.html

http://naggum.no/lugm-time.html

http://naggum.no/lugm-time.html

The Long, Painful History of Time

79

of time need to be precise and include their context, unless the general public stopsrefusing to be educated in the face of dire experience. That so much attention hasbeen granted this silly problem is fortunate for those of us who argue against legacynotations of time. However, the inability of most people to deal with issues of suchextraordinary importance when they look "most harmless" means that those who dounderstand them must be inordinately careful in preparing their information suchthat loss of real information can be minimized.

The basic problem with time is that we need to express both time and place wheneverwe want to place some event in time and space, yet we tend to assume spatialcoordinates even more than we assume temporal coordinates, and in the case oftime in ordinary communication, it is simply left out entirely. Despite the existence oftime zones and strange daylight saving time regimes around the world, most peopleare blithely unaware of their own time zone and certainly of how it relates to standardreferences. Most people are equally unaware that by choosing a notation that is closeto the spoken or written expression of dates, they make it meaningless to people whomay not share the culture, but can still read the language. It is unlikely that peoplewill change enough to put these issues to rest, so responsible computer people needto address the issues and resist the otherwise overpowering urge to abbreviate anddrop context.

This paper is almost all about how we got ourselves into trouble by neglecting tothink about time frames longer than a human lifetime, how we got all confused bythe difference between time as an orderly concept in science and a mess in the restof human existence, and how we have missed every opportunity to fix the problems.This paper proposes a fix to the most glaring problems in a programming languagethat should not have been left without a means to express time for so long.

1 Scientific Time

How long does it take the earth to face the Sun at the same angle? This simplequestion has a definite and fairly simple scientific answer, and from this answer, wecan work out a long list of answers about what time is and how we want to deal withastronomical events. The SI units (Système International d’Unités), probably betterknown as "metric units", define the second as the fundamental unit of time, and this,too, has a very good scientific definition. Time progresses continuously and is onlychopped up into units for human convenience. Agreement on a single reference pointwithin a scientific community has always been easy, and it is useful to count basicunits, like days in the (Modified) Julian Day system, or seconds since some arbitraryepoch in computers.


80

Scientific time also lends itself to ease of computation; after all, that is what we dowith it. For instance, we have a world-wide standard for time, called the CoordinatedUniversal Time, or UTC. (The C used to be subscripted, UTC, just like the digits in UT0and UT1 which are universal time concepts with slightly different reference points,but "UTC" has become the preferred form.) Scientific time naturally has origin 0, asusual with scientific measures, even though the rest of human time notations tendto have origin 1, the problems of which will be treated below.

Most computer-related references to time deal with periods of time, which lendthemselves naturally to use scientific time, and therefore, it makes sense to mostprogrammers to treat the period of time from some epoch until some other time tobe the best way to express said other time. This is the path taken by Common Lisp inits UNIVERSAL-TIME concept, with time 0 equal to 1900-01-01 00:00:00 UTC, andthe Unix time concept, with time 0 equal to 1970-01-01 00:00:00 UTC. This approachworks well as long as the rules for converting between relative and absolute time arestable. As it turns out, they are not.

Not all languages and operating systems use this sensible an approach. Some haveused local time as the point of reference, some use decoded local time as thereference, and some use hardware clocks that try to maintain time suitable for directhuman consumption. There is no need to make this issue more complex than italready is, so they will not be granted any importance.

2 Political Time

How long does it take for the clock to show the same value? The answer to thisquestion is only weakly related to the time the planet takes to make a completerotation. Normally, we would say the political rotation takes 24 hours, just like thescientific, but one day out of the year, it takes only 23 hours, and another day outof the year, it takes 25 hours, thanks to the wonders of daylight saving time. Whichdays these are is a decision made by politicians. It used to be made by the military toconserve fuel, but was taken over by labor unions as a means to get more daylightin the workers' spare time, and most countries have gone through an amazing listof strange decision-making in this area during this century. Short of coming to theirsenses and abolishing the whole thing, we might expect that the rules for daylightsaving time will remain the same for some time to come, but there is no guarantee.(We can only be glad there is no daylight loan time, or we would face decades of toomuch daylight, only to be faced with a few years of total darkness to make up for it.)


81

Political time is closely related to territory, power, and collective human irrationality.There is no way you can know from your location alone which time zone applies atsome particular point on the face of the earth: you have to ask the people who livethere what they have decided. This is very different from scientific time, which couldtell you with great ease and precision what the mean sidereal time at some locationshould be. In some locations, this is as much as three hours off from what the localpopulation has decided, or has had decided for them. The Sun is in zenith at noonat very few places on earth, instead being eclipsed or delayed by political decisionswhere the randomness never ends.

Yet, it is this political time that most people want their computers to produce whenthey ask for the date or the time of day, so software will have to comply with therandomness and produce results consistent with political decisions. The amount ofhuman input into this process is very high, but that is the price we have to payfor our willingness to let politicians dictate the time. However, once the humaninput has been provided, it is annoying to find that most programming languagesand supporting systems do not work with more than one timezone at a time, andconsequently do not retain timezone information with time data.

The languages we use tend to shape the ideas we can talk about. So, too, the waywe write dates and times influence our concepts of time, as they were themselvesinfluenced by the way somebody thought about time a long time ago. Calendarsand decisions like which year is the first, when the year starts, and how to deal withastronomical irregularities were made so long ago that the rationale for them has notsurvived in any form, but we can still look at what we have and try to understand.In solving the problem of dealing with time in computers, a solid knowledge of thelegacy we are attending to is required.

3 Notations for Time

The way we write down time coordinates appears to have varied little over theyears in only one respect: we tend to write them differently depending on thesmallest perceived unit of time that needs to be communicated. For instance, itseems sufficiently redundant to include AD or BC in the dates of birth of contemporarypeople that they are always omitted. Should some being with age >2000 years cometo visit us, it is also unlikely that writing its date of birth correctly would be a pressingconcern. However, we tend to include these markers for the sign of the year whenthe possibility of ambiguity reaches a certain level as determined by the reader. Thisprocess is itself fraught with ambiguity and inconsistency, but when computers needto deal with dates this far back, it does not seem worthwhile to calculate them in


82

terms of standard reference points, so we can ignore the problem for now, but mayneed to deal with it if a system of representation is sufficiently useful to be extendedto the ancient past.

Not only do we omit information that is deemed redundant, it is not uncommon forpeople to omit information out of sheer laziness. A particularly flagrant example ofthe omission of information relative to the current time is the output from the Unixls program which lists various information about files. The customary date andtime format in this program is either month-day-hour-minute or month-day-year.The cutoff for tolerable precision is six months ago, which most implementationsapproximate with 180 days. This reduction in precision appears to have beenmotivated by horizontal space requirements, a necessary move after wasting a lot ofspace on irrelevant information, but for some reason, precision in time always sufferswhen people are short of space.

The infamous Y2K problem, for instance, is said to have started when people wantedto save two columns on punched cards, but there is strong evidence of other, muchbetter alternatives at the time, so the decision to lose the century was not predicatedon the need for space, but rather on the culturally acceptable loss of information fromtime coordinates. The details of this mess are sufficiently involved to fill a separatepaper, so the conclusion that time loses precision first when in need or perceivedneed of space should be considered supported by the evidence.

3.1 Natural-Language Notations

People tend to prefer words to numbers, and go out of their way to name things.Such names are frequently symbolic because they are inherently arbitrary, whichimplies that we can learn much from studying what people call numbers. (French hasa number which means "arbitrarily many": 36, used just like English "umpteen", butit is fascinating that a number has meaning like that. Other numbers with particularmeaning include 69, 666, and 4711. The number 606 has been used to refer toarsphenamine, because it was the 606th compound tested by Paul Ehrlich to treatsyphilis.) In the present context, the names of the Roman months have been adoptedby all Western languages, while the names of days of the week have more recentand diverse names, probably because weeks are a fairly recent concept.

Using names for numeric entities complicates processing a natural languagespecification of time tremendously, yet this is what people seem more comfortablewith. In some cultures, months have only names, while in others, they are nearlyalways written as numbers. The way the names of months and the days of the week


83

are abbreviated varies from language to language, as well, so software that wantsto be international needs to maintain a large repository of names and notations tocater to the vanity of human users. However, the names are not the worst we haveto deal with in natural language notations.

Because dates and times are frequently spoken and because the written forms areoften modeled after the spoken, we run into the problem of ordering the elementsof time and the omission of perceived redundancy becomes a much more seriousproblem, because each language and each culture have handled these problems sodifferently. The orders in use for dates are

• year-month-day

• day-month-year

• month-day-year

• day-month

• month-day

• year-month

• month-year

As long as the year is zero or greater than 31 or the day greater than 12, it is usuallypossible to disambiguate these orders, but we are about to experience renewedproblems in 2001, when the year will probably be still be written with two digits bysome people regardless of the experience of mankind as a whole at 2000-01-0100:00:00 . We live in interesting times, indeed.

Time is fortunately specified with a uniform hour-minute-second order, but theassumption of either AM or PM even in cultures where there is no custom for theirspecification provides us with an ambiguity that computers are ill equipped to dealwith. This and other historic randomness will be treated in full below.

Most of the time people refer to is in their immediate vicinity, and any systemintended to capture human-friendly time specifications will need to understandrelative times, such as "yesterday", "this time tomorrow", "two hours ago", "in fifteenminutes". All of these forms vary considerably from culture to culture and fromlanguage to language, making the process of reading these forms as input non-trivial. The common forms of expression for periods of time is also fuzzy in humancommunication, with units that fail to convert to intervals of fixed length, but insteadare even more context-sensitive than simple points in time.


84

3.2 Language-Neutral Notations

Various attempts have been made to overcome the problems of human-to-humanforms of communication between human and machine and in machine-to-machinecommunication. Machine-to-machine communication generally falls into one of threecategories:

1. Naïve binary

2. Formatted or encoded binary

3. Character sequences (text)

Binary formats in general suffer from a huge number of problems that there is littlevalue in discussing here, but it is worth noting that a binary format that is as robustas a textual format is frequently just as verbose as a textual format, so in the interestof robustness and legibility, this discussion will restrict itself to textual formats

Obviously, a language-neutral notation will have to consist of standardized elementsand possibly codes. Fortunately, a standard like this already exists: ISO 8601. Sinceall the work with a good language-neutral notation has already been done, it wouldbe counter-productive in the extreme to reinvent one. However, ISO 8601 is fairlyexpensive from the appropriate sources and also chock full of weird options, likemost compromise standards, so in the interest of solving some problems with its use,only the extended format of this standard will be employed in this paper.

A language-neutral notation will need to satisfy most, if not all, of the needs satisfiedby natural language notations, but some latitude is necessary when dealing withrelative times — after all, the purpose of the language-neutral notation is to removeambiguity and make assumptions more if not completely explicit. ISO 8601 issufficient to cover these needs:

• absolute positions in time

• duration

• period with absolute start and end

• period with absolute start or end and duration

The needs not covered are mostly related to user convenience with respect to thepresent and absolute positions in time in its immediate vicinity. E.g., the omissionof the date when referring to yesterday, tomorrow, the most recent occurrence of


85

a time of day, and the forthcoming occurrence of a time of day. To make this moreconvenient, the notation employed in the LOCAL-TIME concept described belowhas special syntax for these relative times.

The full, extended format of ISO 8601 is as follows:

1999-10-11T11:10:30,5-07:00

The elements are, in order:

1. the year with four digits

2. a hyphen (omitted in the basic format)

3. the month with two digits

4. a hyphen (omitted in the basic format)

5. the day of month with two digits

6. the letter T to separate date and time

7. the hour in the 24-hour system with two digits

8. a colon (omitted in the basic format)

9. the minute with two digits

10.a colon (omitted in the basic format)

11.the second with two digits

12.a comma

13.the fraction of the second with unlimited precision

14.a plus sign or hyphen (minus) to indicate sign of time zone

15.the hours of the time zone with two digits

16.a colon (omitted in the basic format)

17.the minutes of the time zone with two digits

The rules for omission of elements are quite simple. Elements from the time ofday may be omitted from the right and take their immediately preceding delimiterwith them. Elements from the date may be omitted from the left, but leave theimmediately following delimiter behind. When the year is omitted, it is replaced by ahyphen. Elements of the date may also be omitted from the left, provided no other


86

elements follow, in which case they take their immediately preceding delimiter withthem. The letter T is omitted if the whole of the time of day or the whole of the date areomitted. If an element is omitted from the left, it is assumed to be the current value.(In other words, omitting the century is really dangerous, so I have even omitted thepossibility of doing so.) If an element is omitted from the right, it is assumed to coverthe whole range of values and thus be indeterminate.

Every element in the time specification needs to be within the normal bounds. Thereis no special consideration for leap seconds, although some might want to expressthem using this standard.

A duration of time has a separate notation entirely, as follows:

P1Y2M3DT4H5M6S>

P7W

The elements are, in order:

1. the letter P to indicate a duration

2. the number of years

3. the letter Y to indicate years

4. the number of months

5. the letter M to indicate months

6. the number of days

7. the letter D to indicate days

8. the letter T to separate dates from times

9. the number of hours

10.the letter H to indicate hours

11.the number of minutes

12.the letter M to indicate minutes

13.the number of seconds

14.the letter S to indicate seconds


87

or for the second form, usually used alone

1. the letter P to indicate a duration

2. the number of weeks

3. the letter W to indicate weeks

Any element (number) may be omitted from this specification and if so takes itsfollowing delimited with it. Unlike the absolute time format, there is no requirementon the number of digits, and thus no requirement for leading zeros.

A period of time is indicated by two time specifications, at least one of which has to beabsolute, separated by a single solidus (slash), and has the general forms as follows:

start/end

start/duration

duration/end

the end form may have elements of the date omitted from the left with theassumption that the default is the corresponding value of the element from the startform. Omissions in the start form follow the normal rules.

The standard also has specifications for weeks of the year and days of the week, butthese are used so rarely and are aesthetically displeasing so are gracefully elidedfrom the presentation.

When discussing the read/write syntax of the LOCAL-TIME concept below, theabove formats will be employed with very minor modifications and extensions.

4 Geography

It is amusing that when people specify a time, they tend to forget that they lookedat their watches or asked other time-keeping devices at a particular geographiclocation. The value they use for "current time" is colored by this location so much thatthe absence of a location at which we have the current time, renders it completelyuseless — it could be specified in any one of the about 30 (semantically different)timezones employed around the planet. This is particularly amusing with statementsyou find on the web:

This page was updated 7/10/99 2:00 AM.


88

This piece of information is amazingly useless, yet obviously not so to the personwho knows where the machine is located and who wrote it in the first place. Only bymonitoring for changes to this statement does it have any value at all. Specificationsof time often has this purpose, but the belief that they carry information, too, is quiteprevalent. The only thing we know about this time specification is that it was madein the past, which may remove most of the ambiguity, but not quite all — it couldbe 1999-07-10 .

The geographical origin of a time specification is in practice necessary to understandit. Even with the standard notation described above, people will want to knowthe location of the time. Unfortunately, there is no widely adopted standard forgeographical locations. Those equipped with GPS units may use ICBM or gridcoordinates, but this is almost as devoid of meaning as raw IP addresses on theInternet. Above all, geography is even more rife with names and naming rules thatsuffer from translation than any other information that cries for a precise standard.

Time zones therefore double as indicators of geographical location, much to thechagrin of anyone who is not from the same location, because they use names andabbreviations of names with local meaning. Of course. Also, the indication of thedaylight saving time in the timezone is rather amusing in the probably unintentionalcomplexity they introduce. For instance, the Middle or Central European Time can beabbreviated MET or CET, but the "summer time" as it is called here is one of MEST,CEST, MET DST, or CET DST. Add to this that the "S for summer" in the former twochoices is often translated, and then we have the French.

The only good thing about geography is that most names can be translated intogeographical coordinates, and a mapping from coordinates to time zone and daylightsaving time rules is fairly easy to collect, but moderately difficult to maintain. Thiswork has been done, however, and is distributed with most Unix systems thesedays, most notably the free ones, for some value of "free". In order for a completetime representation to work fully with its environment, access to this informationis necessary. The work on the LOCAL-TIME concept includes an interface to thevarious databases available under most Unix systems.

5 Perspective

An important part of the Y2K problem has been that the information about theperspective on the time stored was lost. Trivialities like the fact that people wereborn in the past, bills were paid in the past and fall due in the future, deliveries will bemade in the future, etc, and most of the time, meaningful specifications of time have


89

hard boundaries that they cannot cross. Few people have problems with credit cardsthat expire 02/02 , say. This was very obviously not 1902-02 . The perspective webring to time specifications usually last beyond the particular time specified.

When dealing with a particular time, it is therefore necessary to know, or to betold, whether it refers to the past or the future, and whether the vantage pointis different from the present. If, for instance, a delivery is due 10/15/99 , and itfails to be delivered that day, only a computer would assume that it was now due2099-10-15 . Unfortunately, there is no common practice in this area at all, andmost people are satisfied with a tacit assumption. That is in large part what causedthe Y2K problem to become so enormously expensive to fix. Had the assumed, butnow missing information been available, the kinds of upgrades required would havebeen different, and most likely much less expensive.

There is more to the perspective than just past and future, however. Most computerapplications that are concerned with time are so with only one particular time: thepresent. We all expect a log file to be generated along with the events, and that itwould be disastrous if the computer somehow recorded a different time than the timeat which an event occurred, or came back to us and revised its testimony because itsuddenly remembered it better. Modern society is disproportionately dependent ona common and coordinated concept of the present time, and we have increasingly letcomputers take care of this perspective for us. Telephones and computers, both voiceand electronic radio broadcasts, watches, wall clocks, the trusty old time clocks infactories where the workers depended on its accuracy, they all portray this commonconcept of a coordinated understanding of which time it is. And they all disagreeslightly. A reportedly Swiss saying goes: "A man with one clock knows the time. Aman with two clocks does not."

Among the many unsolved problems facing society is an infrastructure for time-keeping that goes beyond individual, uncoordinated providers, and a time-keepingtechnology that actually works accurately and is so widely available that thedifferences in opinion over what time it is can be resolved authoritatively. Thetechnology is actually here and the infrastructure is almost available to everyone,but it is not used by the multitude of purported sources of the current time. On theInternet, NTP (the Network Time Protocol) keeps fully connected systems in sync, andmost telecommunications and energy providers have amazingly accurate clocks, butmere mortals are still left with alarming inaccuracies. This fact alone has a tendencyto reduce the interest in accurate representation of time, for the obvious reason thatthe more accurate the notation and representation, the less trustworthy the valueexpressed.


90

6 Calculations with Time

The notation for duration and periods bounded by one absolute position in timeand one duration described above have intuitive meaning, but when pressed foractual meaning, suffer somewhat from the distressing effects of political time. Forinstance, a period of one year that starts 1999-03-01 would end on 2000-02-29or 2000-03-01 with equal probability of being correct. More common problemsoccur with the varying lengths of months, but those are also more widely understoodand the heuristics are in place to deal with them.

Less obvious is the problem of adding one day to a particular time of day. This wasthe original problem that spurred the development of the LOCAL-TIME concept andits implementation. In brief, the problem is to determine which two days of the yearthe day is not 24 hours long. One good solution is to assume the day is 24 hours longand see if the new time has a different timezone than the original time. If so, addthe difference between the timezones to the internal time. This, however, is not thetrivial task it sounds like it should be.

The first complication is that none of the usual time functions can report the absolutetime that some timezone identifier will cause a change in the value of timezone asapplicable to the time of day. Resolving this complications means that we do not haveto test for a straddled timezone boundary the hard way with every calculation, butcould just compare with the edge of the current timezone. Most software currentlydoes this the hard way, including the Unix cron scheduler. However, if we acceptthe limitation that we can work with only one timezone at a time, this becomes muchless of a problem, so Unix and C people tend to ignore this problem.

The second complication is that there really is no way around working with an internaltime representation in any calculation — attempts to adjust elements of a decodedtime generally fail, not only because programmers are forgetful, but also becausethe boundary conditions are hard to enumerate.

Most often, however, calculations fall into two mutually exclusive categories:

1. calculations with the time of day possibly including days

2. calculations with the date with no concept of a time of day

When time is represented internally in terms of seconds since an epoch, only theformer is easy — the latter is irrevocably linked with all the timezone problems. Thelatter may in particular be calculated without reference to timezones at all, and


91

indeed should be conducted in UTC . As far as the author knows, there are no tools orpackages available in modern programming languages or environments that providesignificant support for calculations with dates apart from calculation with times ofday — these are usually deferred to the application-level, and appear not to havebeen solved as far as the application programmer is concerned.

7 Historic Randomness

The Roman tradition of using Ante Meridiem and Post Meridiem to refer to the twohalves have survived into English, despite the departure from the custom of changingthe day of the month at noon. The Meridiem therefore has a very different role inmodern usage than in ancient usage. This legacy notation also carries a numbersystem that is fairly unusual. As seen from members of the 24-hour world, the order12,1,2,…11,12,1,2,…,11 as mapped onto 0,1,2…,23 is not only confusing, it is nearlyimpossible to make people believe that 13 hours have elapsed from 11 AM to 12 AM.For instance, several Scandinavian restaurants are open only 1 hour a day to touristsfrom the world of the 12-hour clock, but open 13 hours a day to natives of the worldof the 24-hour clock.

The Roman tradition of starting the year in the month of March has also been lost.Most agrarian societies were far more interested in the onset of spring than in thewinter solstice, even though various deities were naturally celebrated when the sunreturned Most calendars were designed by people who made no particular effort tobe general or accurate outside their own lifetime or needs, but Julius Cæsar decidedto move the Roman calendar back two months, and thus it came to be known as theJulian calendar. This means that month number 7, 8, 9, and 10 suddenly came in asnumber 9, 10, 11, and 12, but kept their names: September, October, November,December. This is of interest mostly to those who remember their Latin but far moreimportant was the decision to retain the leap day in February. In the old calendar,the leap day was added at the end of the year, as makes perfect sense, when themonth was already short, but now it is squeezed into the middle of the first quarter,complicating all sorts of calculations, and affecting how much people work. In the olddays, the leap day was used as an extra day for the various fertility festivities. Youwould just have to be a cæsar to find this unappealing.

The Gregorian calendar improved on the quadrennial leap years in the Juliancalendar by making only every fourth centennial a leap year, but the decision wasunexpectedly wise for a calendar decision. It still is not accurate, so in a few thousandyears, they may have to insert an extra leap day the way we introduce leap secondsnow, but the simplicity of the scheme is quite amazing: a 400-year cycle not only


92

starts 2000-03-01 (as it did 1600-03-01 ), it contains an even number of weeks:20,871. This means that we can make do with a single 400-year calculation for alltime within the Gregorian calendar with respect to days of week, leap days, etc. PopeGregory XIII may well have given a similar paper to this one to another unsuspectingaudience that probably also failed to appreciate the elegance of his solution., and400 more years will pass before it is truly appreciated.

Other than the unexpected elegance of the Gregorian calendar, the world is nowquite fortunate to have reached consensus on its calendars. Other calendars are stillused, but we now have a global reference calendar with complete convertibility. Thisis great news for computers. It is almost as great news as the complete intercurrencyconvertibility that the monetary markets achieved only as late as 1992. Before thattime, you could wind up with a different amount of money depending on whichcurrencies you traded obscure currencies like the ruble through. The same appliedto calendars: not infrequently, you could wind up on different dates according asyou converted between calendar systems, similar to the problem of adding a year toFebruary 29 any year and then subtracting a year.

8 The LOCAL-TIME Concept

The groundwork should now have been laid for the introduction of the severalcounter-intuitive decisions made in the design of the LOCAL-TIME concept and itsimplementation.

8.1 Time Elements as Fixnums

Unix time has the "advantage" that it is representable as a 32-bitmachine integer. It has the equal disadvantage of not working if thetime is not representable as a 32-bit machine integer, and thus can onlyrepresent times in the interval 1901-12-13T20:45:52/2038-01-19T03:14:07 .If we choose an unsigned machine integer, the intervalis 1970-01-01T00:00:00/2106-02-07T06:28:16 . The Common LispUNIVERSAL-TIME concept has the disadvantage that it turned into a bignum onmost 32-bit machines on 1934-01-10T13:37:04 and runs out of 32 bits two yearsearlier than Unix time, on 2036-02-07T06:28:16 . I find these restrictions to beuncomfortable, regardless of whether there are any 32-bit computers left in 2036to share my pain.

Bignum operations are generally far more expensive than fixnum operations, andthey have to be, regardless of how heavily the Common Lisp implementation has


93

optimized them. It therefore became a pronounced need to work with fixnumsin time-intensive applications. The decision fell on splitting between days andseconds, which should require no particular explanation, other than to point out thatcalculation with days regardless of the time of day is now fully supported and veryefficient.

Because we are very close to the beginning of the next 400-year leap-year cycle,thanks to Pope Gregory, day 0 is defined to be 2000-03-01 , which much lessarbitrary than other systems, but not obviously so. Each 400-year cycle contains146,097 days, so an arbitrary decision was made to limit the day to a maximalnegative value of -146,097, or 1600-03-01 . This can be changed at the peril ofaccurately representing days that do not belong to the calendar used at the time. Noattempt has been made to accurately describe dates not belonging to the Gregoriancalendar, as that is an issue resolvable only with reference to the borders betweencountries and sometimes counties at the many different times throughout historythat monarchs, church leaders, or other power figures decided to change to theGregorian calendar. Catering to such needs is also only necessary with dates priorto the conversion of the Russian calendar to Gregorian, a decision made by Lenin aslate as 1918, or any other conversion, such as 1582 in most of Europe, 1752 in theUnited States, and even more embarrassingly late in Norway.

Not mention above is the need for millisecond resolution. Most events on moderncomputers fall within the same second, so it is now necessary to separate them byincreasing the granularity of the clock representation. This part is obviously optionalin most time processing functions.

The LOCAL-TIME concept therefore represents time as three disjoint fixnums:

1. the number of days since (or until, when negative) 2000-03-01

2. the number of seconds since the start of the day in Coordinated UniversalTime

3. the number of milliseconds since the start of the second.

All numbers have origin 0. Only the number of days may be negative.

The choice of epoch needs some more explanation. Conversion to this system onlyrequires subtracting two from the month and making January and February part ofthe previous year.

The moderate size of the fixnums allows us another enormous advantage overcustomary ways to represent time. Since the leap year is now always at the end ofthe year, it has no bearing on the decoding of the year, month, day, and day-of-week


94

of the date. By choosing this odd-looking epoch, the entire problem with computingleap years and days evaporates. This also means that a single, moderately largetable of decoded date elements may be pre-computed for 400 years, providing atremendous speed-up over the division-based calculations used by other systems.

Similarly, a table of the decoded values of the 86400 possible seconds in a day(86401 if we allow leap seconds) yields a tremendous speedup over division-basedcalculations. (Depending on your processor and memory speeds, a factor of 10 to 50may be expected. for a complete decoding)

8.2 Timezone Representation

David Olsen of Digital Equipment Corporation has laid down a tremendous amountof work in collecting the timezones of the world and their daylight saving timeboundaries. Contrary to the Unix System V approach from New Jersey (insertappropriate booing for best effect), which codifies a daylight saving time regime onlyfor the current year, and apply it to all years, David Olsen’s approach is to maintaintables of all the timezone changes. A particular timezone thus has a fairly long tableof periods of applicability of the specific number of seconds of to add to get localtime. Each interval is represented by the start and end times of the specific value,the specific value, a daylight saving time flag, and the customary abbreviation of thetimezone. On most Unix systems, this is available in compiled files in /usr/share/zoneinfo/ under names based on the continent and capital of the region in mostcases, or more general names in other cases. While not perfect, this is probably ascheme good as any — it is fairly easy to figure out which to use. Usually, a table isalso provided with geographic coordinates mapped to the timezone file.

For the timezone information, the LOCAL-TIME concept implements a package,TZ , or TIMEZONE in full, which contains symbols named after the files, whosevalues are lazy-loaded timezone objects. Because the source files for the zoneinfofiles are generally not as available as the portably coded binary information, theinformation are loaded into memory from the compiled files, thus maintainingmaximum compatibility with the other timezone functions on the system.

In the LOCAL-TIME instances, the timezone is represented as a symbol to aid in theability to save literal time objects in compiled Lisp files. The package TZ can easilybe autoloaded in systems that support such facilities, in order to reduce the load-order complexity.

In order to increase efficiency substantially once again, each timezone object holdsthe last few references to timezone periods in it, in order to limit the search time.


95

Empirical studies of long-running systems have showed that more than 98% of thelookups on a given timezone were for time in the same period, with more than 80%of the remaining lookups at the neighboring periods, so caching these values madeample sense.

8.3 Efficiency Considerations in Table Storage

In order to store 146,072 entries for the days of a 400-year cycle with the decodedyear, month, day, and day-of-week and 86401 entries for the seconds of a daywith the decoded hour, minute and second efficiently, various optimizations wereemployed. The naïve approach, to uses lists, consumes approximately 6519K ona 32-bit machine. Due to their overhead, vectors did worse. Since the decodedelements are small, well-behaved unsigned integers, encoding them in bit fieldswithin a fixnum turns out to save a lot of memory:

+----------+----+-----+---+ +-----+------+------+ | yyyy | mm | day |dow| |hour | min | sec | +----------+----+-----+---+ +-----+------+------+ 10 4 5 3 5 6 6

This simple optimization meant 7 times more compact storage of the exact samedata, with significantly improved access times, to boot (depending on processor andmemory speeds as well as considerations for caching strategies, a factor of 1.5 to 3has been measured in production).

Still, 909K of storage to keep tables of precomputed dates and times may seema steep price to pay for the improved performance. Unsurprisingly, more empiricalevidence confirmed that most dates decoded were in the same century. Worst caseover the next few years, we will access two centuries frequently, but it is still a wasteto store four full centuries. A reduction to 100 years per table also meant the numberof years were representable in 7 bits, meaning that an specialized vector of type(UNSIGNED-BYTE 16) could represent them all. The day of week would be lostin this optimization, but a specialized vector of type (UNSIGNED-BYTE 4) of thefull length (146097) could hold them if a single division to get the day of week wastoo expensive. It turns out that the day of week is much less used than the otherdecoded elements, so the specialized vector was dropped and an option includedwith the call to the decoder to skip the day of week.

Similarly, by representing only 12 hours in a specialized vector of type (UNSIGNED-BYTE 16) , the hour would need only 4 bits and the lookup could do the 12-hour


96

shift in code. This reduces the table memory needs to only 156K, and it is still fasterthan access to the full list representation. This compaction yields almost a factor 42improvement over the naïve approach

For completeness, the bit field layout is now simplified as follows.

+-------+----+-----+ +----+------+------+ | 0-100 |1-12| 1-31| |0-11| 0-59 | 0-59 | +-------+----+-----+ +----+------+------+ 7 4 5 4 6 6

Decoding the day now means finding the 400-year cycle for the day of week, thecentury within it for the table lookup, and adding together the values of the centuriesand the year from the table, which may be 100 to represent January and February ofthe following century. All of this can be done with very inexpensive fixnum operationsfor about 2,939,600 years, after which the day will incur a bignum subtraction tobring it into fixnum space for the next 2,939,600> years. (This optimization has notactually been implemented.)

9 Reading and Printing Time

Common Lisp is renowned for the ability to print and read back almost all of itsdata types. The motivation for the LOCAL-TIME concept included the ability tosave human-readable timestamps in files, as well as the ability to store literal timeobjects efficiently in compiled Lisp files. The former has been accomplished throughthe use of the reader macros. Ignoring all other possible uses of the @ character,it was chosen to be the reader macro for the full representation of a LOCAL-TIMEobject. Considering the prevalence of software that works with the UNIVERSAL-TIME concept, especially in light of the lack of alternatives until now, #@ waschosen to be the reader macro for the UNIVERSAL-TIME representation of a timeobject. This latter notation obviously loses the original time zone information andany milliseconds.

9.1 Timestring Syntax

The Lisp reader is instructed to parse a timestring following the reader macrocharacters. Other functions may call PARSE-TIMESTRING directly. Such atimestring follows ISO 8601 closely, but allows for a few enhancements and anadditional option: the ability to choose between comma and period for the fractionalsecond delimiter.


97

Supported formats of the timestring syntax include

1. absolute time with all elements, the default printed format

2. absolute time with some elements omitted, as per ISO 8601

3. absolute time with date omitted, defaulting to the current

4. absolute time with time omitted, defaulting to 00:00:00Z .

5. the most recent occurrence of a time of day, with a leading < .

6. the forthcoming occurrence of a time of day, with a leading > .

7. the time of day specified, but yesterday, with a leading - .

8. the time of day specified, but tomorrow, with a leading + .

9. the current time of day, with a single = .

Work in progress includes adding and subtracting a duration from the specified time,such as the present, explaining the use of the = , which is also needed to representperiods with one anchor at the present. The duration syntax is, however, rife withassumptions that are fairly hard to express concisely and to use without causingunexpected and unwanted results.

The standard syntax from ISO 8601 is fairly rich with options. These are mostlyunsupported due to the ambiguity they introduce. The goal with the timestringsyntax is that positions and periods of time shall be so easy to read and write in aninformation-preserving syntax that there will be no need to cater to the information-losing formats preferred by some only because of their attempt at similarity to theirspoken forms.

9.2 Formatting Timestrings

Considering that the primary problem with time formats is randomness in the orderof the elements, the timestring formatter for LOCAL-TIME objects allows no optionsin that regard, but allows elements to be omitted as per the standard. The loss of12-hour clocks will annoy a few people for a time, but there is nothing quite likeshaking a bad habit for good. Of course, the persistent programmer will write his ownformatter, anyway, so the default should be made most sensible for representingtime in programs and in lisp-oriented input files.

At present, the interface to the timestring formatter is well suited for a call fromFORMAT control strings with the ~// construct, and takes arguments a follows:


98

1. stream — the stream to receive the formatter timestring

2. local-time — the LOCAL-TIME instance

3. universal — if true, ignore the timezone and use UTC>. This is the colonmodifier.

4. timezone — if true, print a timezone specification at the end. This is the atsignmodifier.

5. date-elements — the number of elements of the date to write, counted fromthe right. This is a number from 0 to 4 (the default if omitted or NIL ).

6. time-elements — the number of elements of the time to write, counted fromthe left. This is a number from 0 to 4 (the default if omitted or NIL ).

7. date-separator — the character to print between elements of the date. Ifomitted or NIL , defaults to the hyphen.

8. time-separator — the character to print between elements of the time. Ifomitted or NIL , defaults to the colon. This argument also applies to the timezonewhen it is printed, and when it has a minute component.

9. internal-separator — the character to print between the date and the timeelements. May also be specified as the number 0, to omit it entirely, which is thedefault if either the date or the time elements are entirely omitted, or the letterT otherwise.

9.3 Exported LOCAL-TIME Symbols

• LOCAL-TIME

[Type]

[Constructor] Arguments: (&key universal internal unix (msec0) (zone 0) .

Produce a LOCAL-TIME instance from the provided numeric timerepresentation.

• MAKE-LOCAL-TIME

[Constructor] Arguments: (&key day sec msec zone)

• LOCAL-TIME-DAY

• LOCAL-TIME-SEC

• LOCAL-TIME-MSEC


99

• LOCAL-TIME-ZONE

[Accessors]

• LOCAL-TIME<

• LOCAL-TIME⇐

• LOCAL-TIME>

• LOCAL-TIME>=

• LOCAL-TIME=

• LOCAL-TIME/=

[Functions] Comparison, just like STRING<op> .

• LOCAL-TIME-ADJUST

[Function] Arguments: (source timezone &optional destination)

Returns two values, the values of new day and sec slots, or, if destination isa LOCAL-TIME instance, fills the slots with the new values and returns thedestination.

• LOCAL-TIME-DESIGNATOR

[Function] Convert a designator (real number) into a LOCAL-TIME instance.

• GET-LOCAL-TIME

[Function] Return the current time as a LOCAL-TIME instance.

• ENCODE-LOCAL-TIME

[Function] Arguments: (ms ss mm hh day month year &optionaltimezone)

Return a new LOCAL-TIME instance corresponding to the specified timeelements.

• DECODE-LOCAL-TIME

[Function] Argument: (local-time)

Returns the decoded time as multiple values: ms , ss , mm , hh , day , month ,year , day-of-week , daylight-saving-time-p , timezone , and thecustomary timezone abbreviation.


100

• PARSE-TIMESTRING

[Function] Arguments: (timestring &key start end junk-allowed)

Parse a timestring and return the corresponding LOCAL-TIME .

• FORMAT-TIMESTRING

[Function] Arguments: (stream local-time universal-p timezone-p date-elements time-elements date-separator time-separatorinternal-separator)

Produces on stream the timestring corresponding to the LOCAL-TIME with thegiven options.

• UNIVERSAL-TIME

[Function] Return the UNIVERSAL-TIME corresponding to the LOCAL-TIME .

• INTERNAL-TIME

[Function] Return the internal system time corresponding to the LOCAL-TIME .

• UNIX-TIME

[Function] Return the Unix time corresponding to the LOCAL-TIME .

• TIMEZONE

[Function] Arguments: (local-time &optional timezone)

Return as multiple values the time zone as the number of seconds east of UTC ,a boolean daylight-saving-p, the customary abbreviation of the timezone, thestarting time of this timezone, and the ending time of this timezone

• LOCAL-TIMEZONE

[Function] Arguments: (adjusted-local-time &optional timezone)

Return the local timezone adjustment applicable at the already adjusted-local-time. Used to reverse the effect of TIMEZONE and LOCAL-TIME-ADJUST .

• DEFINE-TIMEZONE

[Macro] Arguments: (zone-name zone-file &key load)


101

Define zone-name (a symbol or a string) as a new timezone, lazy-loaded fromzone-file (a pathname designator relative to the zoneinfo directory on thissystem). If load is true, load immediately.

DEFAULT-TIMEZONE

+ [Variable] Holds the default timezone for all time operations needing a default.

10 Conclusions

1. The absence of a standard notation for time in Common Lisp required all this work.

2. The presence of International Standards for the representation of time made itall a lot easier.

3. Time basically has the most messed-up legacy you can imagine.

4. Pope Gregory XIII made it a little easier on us all.

5. Adoption of this proposal in Common Lisp systems and applications would maketime a lot easier for almost everyone involved, except users who cling to thehabits that caused the Y2K problems.

6. This package is far from complete.

11 Credits and Acknowledgments

This work has been funded by the author and by NHST, publishers of Norway’sfinancial daily, and TDN, their electronic news agency, and has been a workin progress since late 1997. My colleagues and managers have been extremelysupportive in bringing this fundamental work to fruition. In particular, Linn Iré;nHumlekjæ;r and Erik Haugan suffered numerous weird proposals and false startsbut encouraged the conceptual framework and improved on the execution withtheir ideas and by lending me an ear. My management line, consisting of Ole-Martin Halden, Bjørn Hole, and Hasse Farstad, have encouraged the quality of theimplementation and were willing listeners to the many problems and odd ideas thatpreceded the realization that this had to be done.

The great guys at Franz Inc have helped with internal details in Allegro CL and haveof course made a wonderful Common Lisp environment to begin with. Thanks inparticular to Samantha Cichon and Anna McCurdy for taking care of all the detailsand making my stays so carefree, and to Liliana Avila for putting up with my totallack of respect for deadlines.


102

Many thanks to Pernille Nylehn for reading and commenting on drafts, nudging metowards finishing this work, and for taking care of my cat Xyzzy so I could writethis in peace and deliver it at LUGM '99 without worrying about the little furball’sconstant craving for attention, but also without both their warmth and comfort whencomputers simply refuse to behave rationally.

103

The make it work, make it right, makeit fast misconceptionWritten by: Henrique Bastos at 2009-08-181

On the agile way of doing software development, one of developer’s favorite mantrasis: Make it Work, Make it Right, Make it Fast.

However, it’s not uncommon to see people complaining that by following thisdirections, their project never get to the make it fast stage. That happens becauseof a misconception where people treats these three steps as isolated actions of aproject. And they are three stages of one development task.

While you start a development task, you have an early stage where you need toexplore possibilities. You don’t quite know what is the best approach but havesome ideas. If during that stage you go big looking for THE piece of code thatwill fit the problem perfectly, you will end up spending a considerable amountof time on that quest. That’s because you have too many uncertainties on yoursolution and you need to discover through experimentation to reduce thoseuncertainties to a comfortable amount so you can focus and implement your code.This experimentation process is the Make it Work stage where you just code toachieve the desired behavior or effect.

Just after that, you’d probably discovered what kind of libraries you will use, howthe rest of your code base interacts with this new behavior, etc. That’s when you gofor refactoring and Make it Right. Now you dry things out and organize your codeproperly to have a good design and be easily maintainable.

As Software Developers, we know it’s true that restrictions are everywhere. And forthis we know that there are tradeoffs between design and performance. This is whenyou go for the last stage where you Make it Fast by optimizing your code to achievethe needed performance. By doing this at this point, you will see that it will requiremuch less effort. That’s because you can focus on the small 20% of code you cantune to end up with a 80% performance increase.

Ending all those three stages, you will reach the end of your development task andcan integrate this code into your project, delivering the desired value to your ProductOwner.

1 http://henriquebastos.net/the-make-it-work-make-it-right-make-it-fast-misconception/

http://henriquebastos.net/the-make-it-work-make-it-right-make-it-fast-misconception/

http://henriquebastos.net/the-make-it-work-make-it-right-make-it-fast-misconception/

The make it work, make it right, make it fast misconception

104

If you ignore one of those steps while you’re coding that one task, you will probablysuffer the effects of doing big design up front, increasing technical debit, and havinga awful system performance.

That said, a last piece of advice: Be careful not to compensate bad time estimationsdelivering code that only "works". If you do it as a rule, I assure you that you will endup with some non-tested spaghetti project where your cost of change will be so highthat can put your business at risk and make your "agile" useless.

105

The Myth of Schema-lessWritten by: Jon Haddad at Wed 09 July 20141

I have grown increasingly frustrated with the world as people have become moreand more convinced that "schema-less" is actually a feature to be proud of (or evenexists). For over ten years I’ve worked with close to a dozen different databases inproduction and have not once seen "schemaless" truly manifest. What’s extremelyfrustrating is seeing this from vendors, who should really know better. At best, weshould be using the description "provides little to no help in enforcing a schema" or"you’re on your own, good luck."

Before we dig any further into this topic, I think it’s important to understand how wegot here. Let’s go back 5-10 years. The default database for a majority of projectswas typically a RDBMS. MySQL is all the rage and people can’t get enough PHP. ORMslike hibernate made it easier to work with a RDBMS without actually treating it likeone. Everyone is sort of happy, but the tools really aren’t all that great and mostly doa poor job of abstracting the DB away, or require XML files or some other nonsense.In the end, the ORM could only take away some of the annoyance, mostly manuallywritten SQL queries. You still had to deal with the other (perceived) downsides ofthe RDBMS:

• It gets slow to add columns in the most popular implementations as your dataset grows.

• Joins don’t perform well with large datasets.

• People wanted to think in terms of objects, and see having a schema as pointlessoverhead.

• SQL is perceived as hard

• If you’re interested in OLTP workloads, the RDBMS requires sharding and manualmanagement

• Once you shard you lose out on most the fun SQL features

Looking at each of these limitations, the classic "lets just throw everything out andstart from scratch" mentality kicked in. This seems to be an effort to solve theroot problem, but the problem is misdiagnosed. The decision to eliminate schema

1 http://rustyrazorblade.com/2014/07/the-myth-of-schema-less/

http://rustyrazorblade.com/2014/07/the-myth-of-schema-less/

http://rustyrazorblade.com/2014/07/the-myth-of-schema-less/

The Myth of Schema-less

106

from the database because there are some problems with the implementations ofschemas is total nonsense.

Slow alterations is sometimes hailed as a reason to go schema-less. Logically though,there is no reason why changing your schema inherently must result in downtime ortake hours. The only reason why this is the case with the RDBMS world is becauseit generally requires a rewrite of the db files to include the new column. This is animplementation detail, and a choice, not a requirement. Cassandra2, for instance,uses a schema, and can perform CQL table alterations without rewriting the entiretable, as does sqlite3. So this reason is a crock and can be dismissed as an excuseto not support a typed schema.

Joins don’t perform spectacularly across machines due to a number of reasons. ForOLTP workloads the common advice is to denormalize your data. Copying your dataonce incurs a penalty on writes but returns many fold on reads. For OLAP workloads,the industry has been doing a lot of map reduce, but this is most likely a means toan end. Writing map reduce requires too much developer work and will mostly bereplaced by tools that either utilize a M/R framework or something better. The currentbest candidate for this is SparkSQL4, which integrates well with existing data stores,and uses a DAG (directed acyclic graph) instead of map/reduce. I’m not sure howyou could use a tool like Spark and glean any useful information out of a databasewithout any sort of defined structure somewhere. I think this is a clear case whereexplicit schema is highly beneficial, so I’ll add this to my list of "features that don’tneed to be thrown out just because they’re associated w/ an RDBMS."

The third point, "schema as pointless overhead" is nonsense. In any non trivialapplication, there is going to be a defined data model. The purpose of a schema isnot for the developer writing an MVP by himself, it’s for the team that has to dealwith his code for the foreseeable future. Schema is important, it’s the only way thateveryone that didn’t write the original code can sanely understand the data model.Providing a central location (the database!) to look at this schema, and a humanparsable / writable query language to examine it is undeniably useful. The overheadwith any non-trivial application will quickly shift to the developers maintainingthe codebase and away from managing the schema. Tools like Alembic5 make itstraightforward to manage RDBMS schema changes, as well as the sync_tablefunctionality in cqlengine, for Cassandra. The overhead of managing schema at this

2 http://cassandra.apache.org/3 http://www.sqlite.org/lang_altertable.html4 http://databricks.com/blog/2014/03/26/spark-sql-manipulating-structured-data-using-spark-2.html5 http://alembic.readthedocs.org/en/latest/

http://cassandra.apache.org/

http://www.sqlite.org/lang_altertable.html

http://databricks.com/blog/2014/03/26/spark-sql-manipulating-structured-data-using-spark-2.html

http://alembic.readthedocs.org/en/latest/

http://cassandra.apache.org/

http://www.sqlite.org/lang_altertable.html

http://databricks.com/blog/2014/03/26/spark-sql-manipulating-structured-data-using-spark-2.html

http://alembic.readthedocs.org/en/latest/


107

point is significantly less than the mental overhead your dev team must exert. Notmaintaining your schema in a database becomes a form of technical debt.

SQL can be confusing for someone new to the language, but that’s hardly an excuseto throw either it or explicit schema out. SQL was designed to be human readable andwritable, and a solid understand of it wields its user incredibly power and flexibility.Yes, you can write some really nasty queries, but there’s no reason to use a dullknife all the time just because you are afraid someone might cut themselves. This isstrongly supported if you consider the alternatives so far. As a replacement for SQL,query languages based on JSON are total crap. They discourage you to actually lookat your database directly, requiring code to compose queries frequently based onnested data structures. At a glance, it’s easy enough to compose simple queries for ahandful of rows. As the feature set grows however, you end up with something that’scompletely unmanageable and a nightmare to edit. Even if we decided to ditch SQLforever in favor of some other query language, it STILL does not force our hand toeven consider removing explicit schema.

Manual sharding with an RDBMS is a huge pain. Having to expand your productiondatabase cluster via home grown migrations is at best an annoyance and at worsta massive liability. This certainly has contributed to the decline of the RDBMS, butthere’s no reason why this should lead to a wild west-esque data model. Like theother excuses above, it has nothing to do with the case against schema.

So, for the above reasons, there is a stigma against the RDBMS, and with it, a rejectionof explicit database schema. How the two got to be tangled so close together is amystery - in any of the above points is there a solid technical reason to ditch schema?Not one.

If there isn’t a technical reason, there must be an advantage for a developmentteam. Without schema we’re free to… do what exactly? Create a new class perrecord? That sounds tedious after a handful of classes, and impossible with millions.Put heterogeneous objects in the same container? That’s useful, but nothing new.Sqlalchemy6, a Python ORM has had this for a while now, so has cqlengine7.

We don’t seem to gain much in terms of database flexibility. Is our application moreflexible? I don’t think so. Even without our schema explicitly defined in our database,it’s there… somewhere. You simply have to search through hundreds of thousands oflines to find all the little bits of it. It has the potential to be in several places, making

6 http://docs.sqlalchemy.org/en/rel_0_9/orm/inheritance.html7 https://cqlengine.readthedocs.org/en/latest/topics/models.html#table-polymorphism

http://docs.sqlalchemy.org/en/rel_0_9/orm/inheritance.html

https://cqlengine.readthedocs.org/en/latest/topics/models.html#table-polymorphism

http://docs.sqlalchemy.org/en/rel_0_9/orm/inheritance.html

https://cqlengine.readthedocs.org/en/latest/topics/models.html#table-polymorphism


108

it harder to properly identify. The reality of these codebases is that they are errorprone and rarely lack the necessary documentation. This problem is magnified whenthere are multiple codebases talking to the same database. This is not an uncommonpractice for reporting or analytical purposes.

Finally, all this "flexibility" rears its head in the same way that PHP and Javascript’s"neat" weak typing stabs you right in the face. There are some somethings you canbe cavalier about, and some things you should be strict about. Your data model isone you absolutely need to be strict on. If a field should store an int, it should storenothing else. Not a string, not a picture of a horse, but an integer. It’s nice to knowthat I have my database doing type checking for me and I can expect a field to bethe same type across all records.

All this leads us to an undeniable fact: There is always a schema. Wearing "Idon’t do schema" as a badge of honor is a complete joke and encourages a terribledevelopment practice.

109

NoDBWritten by: Robert Martin at 2012-05-151

In the United States, in 1920, the manufacture, sale, and importation of alcoholicbeverages was prohibited by a constitutional amendment. That amendment wasrepealed thirteen years later. During that period of prohibition, the beer industrydied.

In 1933, when prohibition was lifted, a few giant grain companies started brewingbeer. They completely cornered the market. And for nearly 50 years, we in the UnitedState drank this fizzy bodily effluent and called it “beer”. The only way to toleratethe flavor was to drink it very cold.

As a teenager in the ‘60s, I never understood the attraction. Why beer? It was a pale,yellow, distasteful fluid derived from the urine of sick boars, and had no redeemingqualities that I could see.

In 1984, I went to England; and the scales dropped from my eyes. At last I understood.I had tasted beer for the first time; and I found it to be good.

Since those days the beer situation in the United States has improved dramatically.New beer companies are springing up all over the country; and in many cases thebeer they make is actually quite good. We don’t have anything quite so nice as agood english bitter; but we’re getting close.

In the ‘80s a few giant database companies cornered the market. They did this bypromulgating fear, uncertainty, and doubt amongst managers and marketing people.The word “relational” became synonymous with “good”; and any other kind of datastorage mechanism was prohibited.

I was the lead developer in a startup in those days. Our product measured the qualityof T1 communications lines. Our data model was relatively simple, and we kept thedata in flat files. It worked fine.

But our marketing guy kept on telling us that we had to have a relational database.He said that customers would demand it. I found that to be a strange claim since wehadn’t sold even one system at that time, and no customer had ever mentioned our

1 http://blog.8thlight.com/uncle-bob/2012/05/15/NODB.html

http://blog.8thlight.com/uncle-bob/2012/05/15/NODB.html

http://blog.8thlight.com/uncle-bob/2012/05/15/NODB.html

NoDB

110

data storage technology. But the marketing guy was adamant. We just had to havea relational database. Flat files were prohibited.

As the lead developer, responsible for the quality of the software, my view of arelational database was that it would be a big, stogy, slow, expensive pain in the rear.We didn’t have complex queries. We didn’t need massive reporting capabilities. Wecertainly didn’t need a process with a multi-megabyte footprint sitting in memoryand burning cycles. (Remember, this was the ‘80s). So I fought against this idea witheverything I had; because it was the wrong technical solution.

This was not a politically astute move for me. Over a period of several months,a hardware engineer who managed to write a few lines of code, was moved intothe software group. He was gradually given more and more responsibility, and waseventually named my co-manager. He and I would “share” the responsibility forleading the software team.

Uh huh. Sure. Right. A hardware guy with no real software experience was going to“help” me lead the team. And what do you think his first issue was? Why it was toget a relational database into our system!

I left a month later and started my consulting career. It was best career move I haveever made. The company I left no longer exists. I don’t think they ever made a dime.

I watched the relational database market grow during the ‘90s. I watched as allother data storage technologies, like the object databases, and the B-tree databasesdwindled and died; like the beer companies in the 20s. By the end of the ‘90s, onlythe giants were left.

Those giants were marketing up a storm. They were gods. They were rulers. Duringthe dot com bubble, one of them actually had the audacity to buy television ads thatclaimed that their product was “the power that drove the internet”. That remindedme of a beer slogan from the ‘70s “Ya gotta grab for all the gusto in life ya can.”Oh brother.

During this time I watched in horror as team after team put the database at thecenter of their system. They had been convinced by the endless marketing hypethat the data model was the most important aspect of the architecture, and that thedatabase was the heart and soul of the design.

I witnessed the rise of a new job function. The DBA! Mere programmers could not beentrusted with the data — so the marketing hype told us. The data is too precious,too fragile, too easily corrupted by those undisciplined louts. We need special people

NoDB

111

to manage the data. People trained by the database companies. People who wouldsafeguard and promulgate the giant database companies’ marketing message: thatthe database belongs in the center. The center of the system, the enterprise, theworld, the very universe. MUAHAHAHAHAHAHA!

I watched as SQL slipped through every crack and crevice in the system. I ranscreaming from systems in which SQL had leaked into the UI. I railed endlesslyagainst the practice of moving all business rules into stored procedures. I quailedand quaked and ranted and raved as I read through entire mail-merge programswritten in SQL.

I hammered and hammered as I saw tables and rows permeating the source codeof system after system. I hammered out danger. I hammered out a warning. Ihammered out that the schema had become “The Blob”, consuming everything insight. But I knew all my hammering was just slinging pebbles at a behemoth.

And then, in the first decade of the 21st century, the prohibition was lifted, and theNOSQL movement was born. I considered it a kind of miracle, a light shining forthin the wilderness. Finally, someone realized that there might just be some systemsin the world that did not require a big, fat, horky, slow, expensive, bodily effluent,memory hog of a relational database!

I watched in glee as I saw BigTable, Mongo, CouchDB, and all the other cute littledata storage systems begin to spring up; like little micro-breweries in the ‘80s. Thebeer was back! And it was starting to taste good.

But then I noticed something. Some of the systems using these nice, simple,tasty, non-relational databases were being designed around those databases. Thedatabase, wrapped in shiny new frameworks, was still sitting at the center of thedesign! That poisonous old relational marketing hype was still echoing through theminds of the designers. They were still making the fatal mistake.

“Stop!” I yelled. “ Stop! You don’t understand. You don’t understand.” But themomentum was too great. An enormous wave of frameworks rose up and smasheddown on our industry, washing over the land. Those frameworks wrapped up thedatabases and fought to grab and hold the center of our applications. They claimedto master and tame the databases. They even claimed to be able to turn a relationaldatabase into a NoSQL database. And the frameworks cried out with a great voiceheard all over the land: “Depend on me, and I’ll set you free!”

The name of this article is “No DB”. Perhaps after that rant you are getting an inklingof why I named it that.

NoDB

112

The center of your application is not the database. Nor is it one or more of theframeworks you may be using. The center of your application are the use cases ofyour application.

It makes me crazy when I hear a software developer describe his system as a“Tomcat system using Spring and Hibernate using Oracle”. The very wording putsthe frameworks and the database at the center.

What do you think the architecture of that system would look like? Do you think you’dfind the use cases at the center of the design? Or would you find the source codearranged to fit nicely into the pattern of the frameworks? Would you find businessobjects that looked suspiciously like database rows? Would the schema and theframeworks pollute everything?

Here’s what an application should look like. The use cases should be the highestlevel and most visible architectural entities. The use cases are at the center. Always!Databases and frameworks are details! You don’t have to decide upon them up front.You can push them off until later, once you’ve got all the use cases and businessrules figured out, written, and tested.

What is the best time to determine your data model? When you know what the dataentities are, how they are related, and how they are used. When do you know that?When you’ve gotten all the use cases and business rules written and tested. Bythat time you will have identified all the queries, all the relationships, all the dataelements, and you’ll be able to construct a data model that fits nicely into a database.

Does this change if you are using a NoSql database? Of course not! You still focus ongetting the use cases working and tested before you even think about the database;no matter what kind of database it ends up being.

If you get the database involved early, then it will warp your design. It’ll fight to gaincontrol of the center, and once there it will hold onto the center like a scruffy terrier.You have to work hard to keep the database out of the center of your systems. Youhave to continuously say “No” to the temptation to get the database working early.

We are heading into an interesting time. A time when the prohibition against differentdata storage mechanisms has been lifted, and we are free to experiment with manynovel new approaches. But as we play with our CouchDBs and our Mongos andBigTables, remember this: The database is just a detail that you don’t need to figureout right away.

113

7 coding tasks you should probablynot write yourselfWritten by: Andy Lester at 2014-07-081

As programmers, we like to solve problems. We like to get ideas to spring from ourheads, channel through our fingertips, and create magical solutions.

But sometimes we are too quick to jump in and start cranking out code to solvethe problem without considering all the implications of the issues we’re trying tosolve. We don’t consider that someone else might have already solved this problem,with code available for our use that has already been written, tested, and debugged.Sometimes we just need to stop and think before we start typing.

For example, when you encounter these seven coding problems, you’re almostalways better off looking for an existing solution than trying to code one yourself:

1. Parsing HTML or XML

The task whose complexity is most often underestimated–at least based on howmany times it’s asked about on StackOverflow – is parsing HTML or XML. Extractingdata from arbitrary HTML looks deceptively simple, but really should be left tolibraries. Say you’re looking to extract a URL from an <img> tag like

<img src="foo.jpg">

It’s a simple regular expression to match a pattern.

/<img src="(.+?)">/

The string “foo.jpg” will be in capture group #1 and can be assigned to a string. Butwill your code handle tags with other attributes like:

<img id="bar" src="foo.jpg">

And after you change your code to handle that, will it handle alternate quotes like:

1 http://blog.newrelic.com/2014/07/08/7-things-never-code/

http://blog.newrelic.com/2014/07/08/7-things-never-code/

http://blog.newrelic.com/2014/07/08/7-things-never-code/

7 coding tasks you should probably not write yourself

114

<img src= 'foo.jpg'>

or no quotes at all, like:

<img src=foo.jpg>

What about if the tag spans multiple lines and is self-closing, like:

<img id="bar" src="foo.jpg" />

And will your code know to ignore this commented-out tag:



By the time you’ve gone through the cycle of finding yet another valid case yourcode doesn’t handle, modifying the code, retesting it, and trying it again, you couldhave used a proper library and been done with it.

That’s the story with all of these examples: You’ll spend far less time finding anexisting library and learning to use it rather than trying to roll your own code, thendebug it, then extend it to fit all the cases you hadn’t thought of when you started.

2. Parsing CSV and JSON

CSV files are deceptively simple, yet fraught with peril. Files of comma-separatedvalues are trivial to parse, right?

# ID, name, city1, Queen Elizabeth II, London

Sure, until you have double-quoted values to handle embedded commas:

2, J. R. Ewing, "Dallas, Texas"


115

Once you get around those double-quoted values, what happens when you haveembedded double quotes in a string that have to be escaped:

3, "Larry \"Bud\" Melman", "New York, New York"

You can get around those, too, until you have to deal with embedded newlines inthe middle of a record.

JSON has all the same data type hazards of CSV, with the added headache of beingable to store multi-level data structures.

Save yourself the hassle and inaccuracy. Any data that can’t be handled with splittingthe string on a comma should be left to a library.

If it’s bad to read structured data in an unstructured way, it’s even worse to try tomodify it in place. People often say things like, “I want to change all the <img>tags with such-and-such a URL so they have a new attribute.” But even somethingas seemingly simple as “I want to change any fifth field in this CSV with thename Bob to Steve” is dangerous because as noted above, you can’t just countcommas. To be safe you need to read the data–using a comprehensive library–into aninternal structure, modify the data, and then write it back out with the same library.Anything less risks corrupting the data if its structure doesn’t precisely match yourexpectations.

3. Email address validation

There are two ways you can validate an email address. You can have a very simplecheck like, “I need to have some characters before an @ sign, and then somecharacters after,” matching it against this regular expression:

/.+@.+/

It’s not complete and it lets invalid stuff through, but at least you’ve got an @ signin the middle.

Or you can validate it against the rules in RFC 822. Have you read RFC 8222? It coversall sorts of things that you rarely see but are still legal. A simple regular expressionisn’t going to cut it. You’re going to need to use a library that someone else hasalready written.

2 http://www.ietf.org/rfc/rfc0822.txt

http://www.ietf.org/rfc/rfc0822.txt

http://www.ietf.org/rfc/rfc0822.txt


116

If you’re not going to validate against RFC822, then anything else you do is goingto be a matter of using a subset of rules that seem reasonable but might not becorrect. That’s a valid design tradeoff to make many times, but don’t fool yourselfinto thinking that you’ve covered all the cases unless you go back to the RFC, or usea library written by someone who has.

(For far more discussion about validation of email addresses, see this StackOverflowthread3.)

4. Processing URLs

URLs aren’t nearly as odious as email addresses, but they’re still full of annoyinglittle rules you have to remember. What characters need to be encoded? How doyou handle spaces? How about + signs? What characters are valid to go in that partafter the # sign?

Whatever language you’re working in, there’s code to break apart URLs into thecomponents you need, and to reassemble them from parts, properly formatted.

5. Date/time manipulation

Date/time manipulation is the king of problem sets with rules you can’t possibly wrapyour head around all at once. Date/time handling has to account for time zones,daylight saving time, leap years, and even leap seconds. In the United States, wehave only four time zones to think about, and they’re all an hour apart. The rest ofthe world is not so simple.

Whether it’s date arithmetic where you’re figuring out what three days after anotherday is, or you’re validating that an input string is in fact a valid date, use an existinglibrary.

6. Templating systems

It’s almost a rite of passage. A junior programmer has to create lots of boilerplatetext and comes up with a simple little format like:

Dear #user#,Thank you for your interest in #product#...

3 http://stackoverflow.com/questions/201323

http://stackoverflow.com/questions/201323




117

It works for a while, but then she winds up having to add multiple output formats,and numeric formatting, and outputting structured data in tables, and on and on untilshe’s built an ad hoc monster requiring endless care and feeding.

If you’re doing anything more complex than simple string-for-string substitution, stepback and find yourself a good templating library. To make things even simpler, ifyou’re writing in PHP, the language itself is a templating system (though that is oftenforgotten these days).

7. Logging frameworks

Logging tools are another example of projects that start small and grow intobehemoths. One little function for logging to a file soon needs to be able to log tomultiple files, or to send email on completion, or have varying log levels and so on.Whatever language you’re using, there are at least three log packages that havealready been around for years and will save you no end of aggravation.

But isn’t a library overkill?

Before you pooh-pooh the idea of a module, take a hard look at your objections. TheNumber 1 objection is usually, “Why do I need an entire library just to do (validate thisdate/parse this HTML/etc.),” My response is, “What’s the harm in using it?” Chancesare you’re not writing microcontroller code for a toaster where you have to squeezeout every byte of code space.

If you have speed concerns, consider that avoiding a library may be prematureoptimization. Maybe loading up an entire date/time library makes your datevalidation take 10 times as long as your mostly correct homemade solution, butprofile your code first to see if it actually matters.

We programmers are proud of our skills, and we like the process of creating code.That’s fine. Just remember that your job as a programmer is not to write code but tosolve problems, and often the best way to solve a problem is to write as little codeas possible.

118

You Need This One Skill to Succeed inITWritten by: Jes Schultz Borland at July 17, 20131

The ability to program in five languages, including one machine-level? Not it.

Project management skills, up to and including a PMP certification? Not that either.

Excellent oral and written communication skills, as noted on every job descriptionever? That doesn’t hurt, but can be learned.

All of the best IT professionals I have worked with have excellent problem solvingskills.

That’s it.

We face problems in IT on a regular basis. From the help desk technicians who areasked, “Why am I locked out of my computer?” to the SAN administrators who haveto balance the needs of different servers and workloads to the DBA who is asked,“Why is the server so slow?” we are all given problems to solve.

How we go about solving those problems is what sets the great professionals apartfrom the good or the average.

Problem Solving Methodology

In high school, I was introduced to the scientific method. The process is: Does thisremind you of your server room?

Does this remind you of your server room?

1. Formulate a question.

2. Make a hypothesis.

3. Make a prediction.

4. Test the hypothesis. Measurements are emphasized.

5. Analyze the results.

1 http://www.brentozar.com/archive/2013/07/you-need-this-one-skill-to-succeed-in-it/

http://www.brentozar.com/archive/2013/07/you-need-this-one-skill-to-succeed-in-it/

http://www.brentozar.com/archive/2013/07/you-need-this-one-skill-to-succeed-in-it/

You Need This One Skill to Succeed in IT

119

Can this be applied to your problems? Yes, it can.

Formulate a question – usually, the question is asked of you. “Why is the serverslow?” “Why can’t I connect to the database?” “Why is this report execution timingout?”

Make a hypothesis – perhaps a patch was installed on the server or SQL Server thenight before. Maybe a network cable could have been unplugged. Maybe a developerchanged a line of code in the stored procedure. Make a list of what could affect thesystem in question, so you have items to test.

Make a prediction – take a guess at what your results will be. If this is an erroror problem you’ve encountered before, you’ll probably have a good idea of whatto expect. If it’s a new problem, use your past experiences and deductive skills todetermine what changes you make will do to the system.

Test the hypothesis – make a plan, make a change, and check if the problem issolved. Don’t make three changes and wonder which one fixed it – fix one at a time.Know what success looks like. If a query is slow, know what performance was beforethe problem occurred, what performance is when the problem is happening, andwhat acceptable performance looks like. Metrics are important here. You must beable to measure if things improved, stayed the same, or got worse.

Analyze the results – check those metrics. Did you get the results you expected?If so, is the problem resolved? If not, what is the next item on your list to check?Continue to iterate through your list until the problem is solved.

Anyone Can Do This

It doesn’t require a PhD in computer science. It doesn’t require a master’s degreein chemistry. What it does take is a consistent approach to problems every time. Ittakes curiosity and an ability to see patterns.

With practice, this becomes much easier. Practice your problem-solving skills often.They will make you a great IT professional, and set you apart from the rest of thecrowd.

120

ORM Is Not a Choice and How toMake It Suck LessWritten by: Elnur Abdurrakhimov at 26 Apr 20141

There is a lot of ORM hate lately. Some people even call it The Vietnam of ComputerScience2.

ORM vs ORM Tools

First, let’s clarify the difference between ORM and ORM tools.

ORM — object-relational mapping — is a technique used to map object-orientedconstructs like classes and objects to relational database structures like tables androws.

An ORM tool is a particular tool like Hibernate or Doctrine that does that mappingfor you.

ORM Is Not a Choice

If the model in your application is implemented in object-oriented style and you usea relational database to persist data, ORM is not a choice. Whether you like it or not,you have to map the application code to the relational constructs of the database.

The only choice you have in this case is whether to use an existing ORM tool or rollup your own. That’s it.

Of course, there are other options like using NoSQL tools for persistence orabandoning OO style in the model, but that’s another story.

Making ORM Suck Less

While ORM itself is not a choice, there is a choice in using an ORM tool in a way thatmakes it suck less.

1 http://elnur.pro/orm-is-not-a-choice-and-how-to-make-it-suck-less/2 http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx

http://elnur.pro/orm-is-not-a-choice-and-how-to-make-it-suck-less/

http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx


http://elnur.pro/orm-is-not-a-choice-and-how-to-make-it-suck-less/


ORM Is Not a Choice and How to Make It Suck Less

121

Do Not Let an ORM Tool Generate Database Schema

I see this too often. A lot of developers generate their database schemas with anORM tool and that’s a bad idea.

First, ORM tools suck in schema generation. They generate schemas in probably themost blunt way one could imagine. They also name constraints with useless nameslike FK_D95AB405F92F3E70 or IDX_9C0600CAF6BD1646 . If you don’t name aconstraint explicitly, sane DBMSes like PostgreSQL will come up with much betternames on their own.

Second, ORM tools try to support as many DB vendors as possible, so they use only aportable subset of DDL. This means they can’t take advantage of powerful featuresof a particular DBMS you went with. For example, PostgreSQL has a CHECK constraintthat an ORM won’t generate:

CREATE TABLE person( sex varchar NOT NULL, CHECK (sex IN ('male', 'female')));

Take back the control of your database schema. Write database migrations in plainSQL and adapt ORM mapping to it — not the other way around.

Do Not Let an ORM Tool Dictate a Particular Schema

Design your schema using relation databases idioms and then use an ORM tool tomap the schema to the application code. Do not bend your schema to please aparticular ORM tool you’re using.

For example, there are ORM tools that force you to have an autoincrement ID fieldplaying the role of they primary key in each table. From the relational databasedesign point of view, it doesn’t always make sense to add a surrogate key when youalready have a natural one. For example, ISBN is a great natural key for a book; youdon’t have to add a surrogate ID column to the books table.

If your ORM tool forces to bend your schema to its own will, look for a better tool.

ORM Is Not a Choice and How to Make It Suck Less

122

Do Not Let an ORM Tool Dictate a Particular ModelStructure

Some ORM tools force you to extend some base class or do something else to pleasethem. Those are mostly implementations of the Active Record pattern.

There are other ORM tools that implement the Data Mapper pattern and they let youkeep your model structure the way you would if you were not using an ORM tool atall. Hibernate and Doctrine are examples of that.

Do not let an ORM bend your model to its own will. If it does, throw it away and finda better alternative.

Use ORM Tools for Mapping Only

Since marshalling objects to rows and back is mostly boilerplate code and very boringto write, it’s a good idea to use an existing ORM tool to do that for you. Just don’t letit do anything else because that’s out of ORM and ORM tools concern.

Conclusion

ORM is not a choice when you have OO code on one side and RDBMS on another. Butthere is a choice of ORM tools and ways to use them that suck less.

123

Programming Sucks

Every friend I have with a job that involves picking up something heavier than alaptop more than twice a week eventually finds a way to slip something like this intoconversation: "Bro,1 you don’t work hard. I just worked a 4700-hour week digging atunnel under Mordor with a screwdriver."

They have a point. Mordor sucks, and it’s certainly more physically taxing to diga tunnel than poke at a keyboard unless you’re an ant. But, for the sake of theargument, can we agree that stress and insanity are bad things? Awesome. Welcometo programming.

All programming teams are constructed by andof crazy people

Imagine joining an engineering team. You’re excited and full of ideas, probably justout of school and a world of clean, beautiful designs, awe-inspiring in their aestheticunity of purpose, economy, and strength. You start by meeting Mary, project leaderfor a bridge in a major metropolitan area. Mary introduces you to Fred, after you get

1 It always starts with "Bro"

Programming Sucks

124

through the fifteen security checks installed by Dave because Dave had his sweaterstolen off his desk once and Never Again. Fred only works with wood, so you askwhy he’s involved because this bridge is supposed to allow rush-hour traffic full ofcars full of mortal humans to cross a 200-foot drop over rapids. Don’t worry, saysMary, Fred’s going to handle the walkways. What walkways? Well Fred made a goodcase for walkways and they’re going to add to the bridge’s appeal. Of course, they’llhave to be built without railings, because there’s a strict no railings rule enforced byPhil, who’s not an engineer. Nobody’s sure what Phil does, but it’s definitely full ofsynergy and has to do with upper management, whom none of the engineers want todeal with so they just let Phil do what he wants. Sara, meanwhile, has found severalhemorrhaging-edge paving techniques, and worked them all into the bridge design,so you’ll have to build around each one as the bridge progresses, since each onemeans different underlying support and safety concerns. Tom and Harry have beenworking together for years, but have an ongoing feud over whether to use metricor imperial measurements, and it’s become a case of "whoever got to that part ofthe design first." This has been such a headache for the people actually screwingthings together, they’ve given up and just forced, hammered, or welded their waythrough the day with whatever parts were handy. Also, the bridge was designed asa suspension bridge, but nobody actually knew how to build a suspension bridge,so they got halfway through it and then just added extra support columns to keepthe thing standing, but they left the suspension cables because they’re still sort ofholding up parts of the bridge. Nobody knows which parts, but everybody’s prettysure they’re important parts. After the introductions are made, you are invited tocome up with some new ideas, but you don’t have any because you’re a propulsionengineer and don’t know anything about bridges.

Would you drive across this bridge? No. If it somehow got built, everybody involvedwould be executed. Yet some version of this dynamic wrote every single programyou have ever used, banking software, websites, and a ubiquitously used programthat was supposed to protect information on the internet but didn’t.

All code is bad

Every programmer occasionally, when nobody’s home, turns off the lights, poursa glass of scotch, puts on some light German electronica, and opens up a file ontheir computer. It’s a different file for every programmer. Sometimes they wrote it,sometimes they found it and knew they had to save it. They read over the lines, andweep at their beauty, then the tears turn bitter as they remember the rest of the filesand the inevitable collapse of all that is good and true in the world.

Programming Sucks

125

This file is Good Code. It has sensible and consistent names for functions andvariables. It’s concise. It doesn’t do anything obviously stupid. It has never had tolive in the wild, or answer to a sales team. It does exactly one, mundane, specificthing, and it does it well. It was written by a single person, and never touched byanother. It reads like poetry written by someone over thirty.

Every programmer starts out writing some perfect little snowflake like this. Thenthey’re told on Friday they need to have six hundred snowflakes written by Tuesday,so they cheat a bit here and there and maybe copy a few snowflakes and try to stickthem together or they have to ask a coworker to work on one who melts it and then allthe programmers' snowflakes get dumped together in some inscrutable shape andsomebody leans a Picasso on it because nobody wants to see the cat urine soakinginto all your broken snowflakes melting in the light of day. Next week, everybodyshovels more snow on it to keep the Picasso from falling over.

There’s a theory that you can cure this by following standards, except there are more"standards" than there are things computers can actually do, and these standards areall variously improved and maligned by the personal preferences of the people codingthem, so no collection of code has ever made it into the real world without doing afew dozen identical things a few dozen not even remotely similar ways. The first fewweeks of any job are just figuring out how a program works even if you’re familiar withevery single language, framework, and standard that’s involved, because standardsare unicorns.

There will always be darkness

I spent a few years growing up with a closet in my bedroom. The closet had anodd design. It looked normal at first, then you walked in to do closet things, anddiscovered that the wall on your right gave way to an alcove, making for a handylittle shelf. Then you looked up, and the wall at the back of the alcove gave wayagain, into a crawlspace of utter nothingness, where no light could fall and which youimmediately identified as the daytime retreat for every ravenous monster you keptat bay with flashlights and stuffed animals each night.

This is what it is to learn programming. You get to know your useful tools, then youlook around, and there are some handy new tools nearby and those tools show youthe bottomless horror that was always right next to your bed.

For example, say you’re an average web developer. You’re familiar with a dozenprogramming languages, tons of helpful libraries, standards, protocols, what have

Programming Sucks

126

you. You still have to learn more at the rate of about one a week, and remember tocheck the hundreds of things you know to see if they’ve been updated or broken andmake sure they all still work together and that nobody fixed the bug in one of themthat you exploited to do something you thought was really clever one weekend whenyou were drunk. You’re all up to date, so that’s cool, then everything breaks.

"Double you tee eff?" you say, and start hunting for the problem. You discover thatone day, some idiot decided that since another idiot decided that 1/0 should equalinfinity, they could just use that as a shorthand for "Infinity" when simplifying theircode. Then a non-idiot rightly decided that this was idiotic, which is what the originalidiot should have decided, but since he didn’t, the non-idiot decided to be a dick andmake this a failing error in his new compiler. Then he decided he wasn’t going to tellanyone that this was an error, because he’s a dick, and now all your snowflakes areurine and you can’t even find the cat.

You are an expert in all these technologies, and that’s a good thing, because thatexpertise let you spend only six hours figuring out what went wrong, as opposed tolosing your job. You now have one extra little fact to tuck away in the millions of littlefacts you have to memorize because so many of the programs you depend on arewritten by dicks and idiots.

And that’s just in your own chosen field, which represents such a tiny fraction of all thethings there are to know in computer science you might as well never have learnedanything at all. Not a single living person knows how everything in your five-year-oldMacBook actually works. Why do we tell you to turn it off and on again? Because wedon’t have the slightest clue what’s wrong with it, and it’s really easy to induce comain computers and have their built-in team of automatic doctors try to figure it out forus. The only reason coders' computers work better than non-coders' computers iscoders know computers are schizophrenic little children with auto-immune diseasesand we don’t beat them when they’re bad.

A lot of work is done on the internet and theinternet is its own special hellscape

Remember that stuff about crazy people and bad code? The internet is that except it’sliterally a billion times worse. Websites that are glorified shopping carts with maybethree dynamic pages are maintained by teams of people around the clock, becausethe truth is everything is breaking all the time, everywhere, for everyone. Right nowsomeone who works for Facebook is getting tens of thousands of error messages and

Programming Sucks

127

frantically trying to find the problem before the whole charade collapses. There’s ateam at a Google office that hasn’t slept in three days. Somewhere there’s a databaseprogrammer surrounded by empty Mountain Dew bottles whose husband thinks she’sdead. And if these people stop, the world burns. Most people don’t even know whatsysadmins do, but trust me, if they all took a lunch break at the same time theywouldn’t make it to the deli before you ran out of bullets protecting your cannedgoods from roving bands of mutants.

You can’t restart the internet. Trillions of dollars depend on a rickety cobweb ofunofficial agreements and "good enough for now" code with comments like "TODO:FIX THIS IT’S A REALLY DANGEROUS HACK BUT I DON’T KNOW WHAT’S WRONG" thatwere written ten years ago. I haven’t even mentioned the legions of people attackingvarious parts of the internet for espionage and profit or because they’re bored. Everheard of 4chan? 4chan might destroy your life and business because they decidedthey didn’t like you for an afternoon, and we don’t even worry about 4chan becauseanother nuke doesn’t make that much difference in a nuclear winter.

On the internet, it’s okay to say, "You know, this kind of works some of the time ifyou’re using the right technology," and BAM! it’s part of the internet now. Anybodywith a couple of hundred dollars and a computer can snag a little bit of the internetand put up whatever awful chunks of hack code they want and then attach their littlebit to a bunch of big bits and everything gets a little bit worse. Even the good codersdon’t bother to learn the arcane specifications outlined by the organizations peopleset up to implement some unicorns, so everybody spends half their time coping withthe fact that nothing matches anything or makes any sense and might break at anytime and we just try to cover it up and hope no one notices.

Here are the secret rules of the internet: five minutes after you open a web browserfor the first time, a kid in Russia has your social security number. Did you sign up forsomething? A computer at the NSA now automatically tracks your physical locationfor the rest of your life. Sent an email? Your email address just went up on a billboardin Nigeria.

These things aren’t true because we don’t care and don’t try to stop them, they’retrue because everything is broken because there’s no good code and everybody’sjust trying to keep it running. That’s your job if you work with the internet: hopingthe last thing you wrote is good enough to survive for a few hours so you can eatdinner and catch a nap.

Programming Sucks

128

We didn’t start out crazy, we’re being drivencrazy

ERROR: Attempted to parse HTML with regular expression; system returned Cthulhu.

Funny, right? No? How about this exchange:

"Is that called arrayReverse?"

"s/camel/_/"

"Cool thanks."

Wasn’t that guy helpful? With the camel? Doesn’t that seem like an appropriateresponse? No? Good. You can still find Jesus. You have not yet spent so much ofyour life reading code that you begin to talk in it. The human brain isn’t particularlygood at basic logic and now there’s a whole career in doing nothing but really, reallycomplex logic. Vast chains of abstract conditions and requirements have to be pickedthrough to discover things like missing commas. Doing this all day leaves you in astate of mild aphasia as you look at people’s faces while they’re speaking and youdon’t know they’ve finished because there’s no semicolon. You immerse yourself ina world of total meaninglessness where all that matters is a little series of numberswent into a giant labyrinth of symbols and a different series of numbers or a pictureof a kitten came out the other end.

The destructive impact on the brain is demonstrated by the programming languagespeople write. This is a program:

#include <iostream>

int main( int argc, char** argv ) { std::cout << "Hello World!" << std::endl; return 0;}

That program does exactly the same thing as this program:

`r```````````.H.e.l.l.o. .w.o.r.l.di

And this program:

Programming Sucks

129

>+++++++++[<++++++++>-]<.>+++++++[<++++>-]<+.+++++++..+++.[-]>++++++++[<++++>-] <.>+++++++++++[<++++++++>-]<-.--------.+++.------.--------.[-]>++++++++[<++++>- ]<+.[-]++++++++++.

And this one:

Ook. Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.Ook. Ook. Ook. Ook. Ook! Ook? Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook? Ook! Ook! Ook? Ook! Ook? Ook.Ook! Ook. Ook. Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.Ook. Ook. Ook! Ook? Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook?Ook! Ook! Ook? Ook! Ook? Ook. Ook. Ook. Ook! Ook. Ook. Ook. Ook. Ook. Ook. Ook.Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook! Ook. Ook! Ook. Ook. Ook. Ook. Ook.Ook. Ook. Ook! Ook. Ook. Ook? Ook. Ook? Ook. Ook? Ook. Ook. Ook. Ook. Ook. Ook.Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook! Ook? Ook? Ook. Ook. Ook.Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook? Ook! Ook! Ook? Ook! Ook? Ook. Ook! Ook.Ook. Ook? Ook. Ook? Ook. Ook? Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook! Ook? Ook? Ook. Ook. Ook.Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.Ook. Ook? Ook! Ook! Ook? Ook! Ook? Ook. Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook.Ook? Ook. Ook? Ook. Ook? Ook. Ook? Ook. Ook! Ook. Ook. Ook. Ook. Ook. Ook. Ook.Ook! Ook. Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook.Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook! Ook!Ook! Ook. Ook. Ook? Ook. Ook? Ook. Ook. Ook! Ook. Ook! Ook? Ook! Ook! Ook? Ook!Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook. Ook.

Programming Sucks

130

Ook. Ook. Ook. Ook. Ook! Ook.

And once somebody wrote a programming language that let somebody else writethis:

#:: ::-| ::-| .-. :||-:: 0-| .-| ::||-| .:|-. :||open(Q,$0);while("){if(/^#(.*)$/){for(split('-',$1)){$q=0;for(split){s/|/:.:/xg;s/:/../g;$Q=$_?length:$_;$q+=$q?$Q:$Q*20;}print chr($q);}}}print"n";#.: ::||-| .||-| :|||-| ::||-| ||-:: :|||-| .:|"

According to the author, that program is "two lines of code that parse two linesof embedded comments in the code to read the Mayan numbers representing theindividual ASCII characters that make up the magazine title, rendered in 90-degreerotated ASCII art."

That program won a contest, because of course it did. Do you want to live in a worldlike this? No. This is a world of where you can smoke a pack a day and nobody evenquestions it. "Of course he smokes a pack a day, who wouldn’t?" Eventually everyprogrammer wakes up and before they’re fully conscious they see their whole worldand every relationship in it as chunks of code, and they trade stories about it as ifsleepiness triggering acid trips is a normal thing that happens to people. This is aworld where people eschew sex to write a programming language for orangutans.All programmers are forcing their brains to do things brains were never meant to doin a situation they can never make better, ten to fifteen hours a day, five to sevendays a week, and every one of them is slowly going mad.

</rant>

So no, I’m not required to be able to lift objects weighing up to fifty pounds. I tradedthat for the opportunity to trim Satan’s pubic hair while he dines out of my open skullso a few bits of the internet will continue to work for a few more days.

131

Revenge of the NerdsWritten by: Paul Graham at May 20021

We were after the C++ programmers. We managed to drag a lot ofthem about halfway to Lisp.

— Guy Steele co-author of the Java spec

In the software business there is an ongoing struggle between the pointy-headedacademics, and another equally formidable force, the pointy-haired bosses. Everyoneknows who the pointy-haired boss is, right? I think most people in the technologyworld not only recognize this cartoon character, but know the actual person in theircompany that he is modelled upon.

The pointy-haired boss miraculously combines two qualities that are common bythemselves, but rarely seen together: (a) he knows nothing whatsoever abouttechnology, and (b) he has very strong opinions about it.

Suppose, for example, you need to write a piece of software. The pointy-haired bosshas no idea how this software has to work, and can’t tell one programming languagefrom another, and yet he knows what language you should write it in. Exactly. Hethinks you should write it in Java.

Why does he think this? Let’s take a look inside the brain of the pointy-haired boss.What he’s thinking is something like this. Java is a standard. I know it must be,because I read about it in the press all the time. Since it is a standard, I won’tget in trouble for using it. And that also means there will always be lots of Javaprogrammers, so if the programmers working for me now quit, as programmersworking for me mysteriously always do, I can easily replace them.

Well, this doesn’t sound that unreasonable. But it’s all based on one unspokenassumption, and that assumption turns out to be false. The pointy-haired bossbelieves that all programming languages are pretty much equivalent. If that weretrue, he would be right on target. If languages are all equivalent, sure, use whateverlanguage everyone else is using.

But all languages are not equivalent, and I think I can prove this to you without evengetting into the differences between them. If you asked the pointy-haired boss in

1 http://www.paulgraham.com/icad.html

http://www.paulgraham.com/icad.html

http://www.paulgraham.com/icad.html

Revenge of the Nerds

132

1992 what language software should be written in, he would have answered with aslittle hesitation as he does today. Software should be written in C++. But if languagesare all equivalent, why should the pointy-haired boss’s opinion ever change? In fact,why should the developers of Java have even bothered to create a new language?

Presumably, if you create a new language, it’s because you think it’s better in someway than what people already had. And in fact, Gosling makes it clear in the first Javawhite paper that Java was designed to fix some problems with C++. So there youhave it: languages are not all equivalent. If you follow the trail through the pointy-haired boss’s brain to Java and then back through Java’s history to its origins, youend up holding an idea that contradicts the assumption you started with.

So, who’s right? James Gosling, or the pointy-haired boss? Not surprisingly, Gosling isright. Some languages are better, for certain problems, than others. And you know,that raises some interesting questions. Java was designed to be better, for certainproblems, than C++. What problems? When is Java better and when is C++? Arethere situations where other languages are better than either of them?

Once you start considering this question, you have opened a real can of worms. Ifthe pointy-haired boss had to think about the problem in its full complexity, it wouldmake his brain explode. As long as he considers all languages equivalent, all he hasto do is choose the one that seems to have the most momentum, and since thatis more a question of fashion than technology, even he can probably get the rightanswer. But if languages vary, he suddenly has to solve two simultaneous equations,trying to find an optimal balance between two things he knows nothing about: therelative suitability of the twenty or so leading languages for the problem he needs tosolve, and the odds of finding programmers, libraries, etc. for each. If that’s what’son the other side of the door, it is no surprise that the pointy-haired boss doesn’twant to open it.

The disadvantage of believing that all programming languages are equivalent is thatit’s not true. But the advantage is that it makes your life a lot simpler. And I thinkthat’s the main reason the idea is so widespread. It is a comfortable idea.

We know that Java must be pretty good, because it is the cool, new programminglanguage. Or is it? If you look at the world of programming languages from a distance,it looks like Java is the latest thing. (From far enough away, all you can see is thelarge, flashing billboard paid for by Sun.) But if you look at this world up close, youfind that there are degrees of coolness. Within the hacker subculture, there is anotherlanguage called Perl that is considered a lot cooler than Java. Slashdot, for example,


133

is generated by Perl. I don’t think you would find those guys using Java Server Pages.But there is another, newer language, called Python, whose users tend to look downon Perl, and more2 waiting in the wings.

If you look at these languages in order, Java, Perl, Python, you notice an interestingpattern. At least, you notice this pattern if you are a Lisp hacker. Each one isprogressively more like Lisp. Python copies even features that many Lisp hackersconsider to be mistakes. You could translate simple Lisp programs into Python linefor line. It’s 2002, and programming languages have almost caught up with 1958.

Catching Up with Math

What I mean is that Lisp was first discovered by John McCarthy in 1958, and popularprogramming languages are only now catching up with the ideas he developed then.

Now, how could that be true? Isn’t computer technology something that changesvery rapidly? I mean, in 1958, computers were refrigerator-sized behemoths withthe processing power of a wristwatch. How could any technology that old even berelevant, let alone superior to the latest developments?

I’ll tell you how. It’s because Lisp was not really designed to be a programminglanguage, at least not in the sense we mean today. What we mean by a programminglanguage is something we use to tell a computer what to do. McCarthy did eventuallyintend to develop a programming language in this sense, but the Lisp that weactually ended up with was based on something separate that he did as a theoreticalexercise3 — an effort to define a more convenient alternative to the Turing Machine.As McCarthy said later,

Another way to show that Lisp was neater than Turing machines wasto write a universal Lisp function and show that it is briefer and morecomprehensible than the description of a universal Turing machine.This was the Lisp function eval4…, which computes the value of a Lispexpression…. Writing eval required inventing a notation representingLisp functions as Lisp data, and such a notation was devised for thepurposes of the paper with no thought that it would be used to expressLisp programs in practice.

2 http://www.paulgraham.com/accgen.html3 http://www.paulgraham.com/rootsoflisp.html4 http://lib.store.yahoo.net/lib/paulgraham/jmc.lisp

http://www.paulgraham.com/accgen.html

http://www.paulgraham.com/rootsoflisp.html


http://lib.store.yahoo.net/lib/paulgraham/jmc.lisp



http://lib.store.yahoo.net/lib/paulgraham/jmc.lisp


134

What happened next was that, some time in late 1958, Steve Russell, one ofMcCarthy’s grad students, looked at this definition of eval and realized that if hetranslated it into machine language, the result would be a Lisp interpreter.

This was a big surprise at the time. Here is what McCarthy said about it later in aninterview:

Steve Russell said, look, why don’t I program this eval…, and I said tohim, ho, ho, you’re confusing theory with practice, this eval is intendedfor reading, not for computing. But he went ahead and did it. That is,he compiled the eval in my paper into [IBM] 704 machine code, fixingbugs, and then advertised this as a Lisp interpreter, which it certainlywas. So at that point Lisp had essentially the form that it has today….

Suddenly, in a matter of weeks I think, McCarthy found his theoretical exercisetransformed into an actual programming language-- and a more powerful one thanhe had intended.

So the short explanation of why this 1950s language is not obsolete is that it was nottechnology but math, and math doesn’t get stale. The right thing to compare Lisp tois not 1950s hardware, but, say, the Quicksort algorithm, which was discovered in1960 and is still the fastest general-purpose sort.

There is one other language still surviving from the 1950s, Fortran, and it representsthe opposite approach to language design. Lisp was a piece of theory thatunexpectedly got turned into a programming language. Fortran was developedintentionally as a programming language, but what we would now consider a verylow-level one.

Fortran I5, the language that was developed in 1956, was a very different animalfrom present-day Fortran. Fortran I was pretty much assembly language with math.In some ways it was less powerful than more recent assembly languages; there wereno subroutines, for example, only branches. Present-day Fortran is now arguablycloser to Lisp than to Fortran I.

Lisp and Fortran were the trunks of two separate evolutionary trees, one rooted inmath and one rooted in machine architecture. These two trees have been convergingever since. Lisp started out powerful, and over the next twenty years got fast. So-called mainstream languages started out fast, and over the next forty years gradually

5 http://www.paulgraham.com/history.html

http://www.paulgraham.com/history.html

http://www.paulgraham.com/history.html


135

got more powerful, until now the most advanced of them are fairly close to Lisp.Close, but they are still missing a few things….

What Made Lisp Different

When it was first developed, Lisp embodied nine new ideas. Some of these we nowtake for granted, others are only seen in more advanced languages, and two are stillunique to Lisp. The nine ideas are, in order of their adoption by the mainstream,

1. Conditionals. A conditional is an if-then-else construct. We take these for grantednow, but Fortran I didn’t have them. It had only a conditional goto closely basedon the underlying machine instruction.

2. A function type. In Lisp, functions are a data type just like integers or strings.They have a literal representation, can be stored in variables, can be passed asarguments, and so on.

3. Recursion. Lisp was the first programming language to support it.

4. Dynamic typing. In Lisp, all variables are effectively pointers. Values are whathave types, not variables, and assigning or binding variables means copyingpointers, not what they point to.

5. Garbage-collection.

6. Programs composed of expressions. Lisp programs are trees of expressions, eachof which returns a value. This is in contrast to Fortran and most succeedinglanguages, which distinguish between expressions and statements.

It was natural to have this distinction in Fortran I because you could not neststatements. And so while you needed expressions for math to work, there was nopoint in making anything else return a value, because there could not be anythingwaiting for it.

This limitation went away with the arrival of block-structured languages, but bythen it was too late. The distinction between expressions and statements wasentrenched. It spread from Fortran into Algol and then to both their descendants.

7. A symbol type. Symbols are effectively pointers to strings stored in a hash table.So you can test equality by comparing a pointer, instead of comparing eachcharacter.

8. A notation for code using trees of symbols and constants.


136

9. The whole language there all the time. There is no real distinction between read-time, compile-time, and runtime. You can compile or run code while reading, reador run code while compiling, and read or compile code at runtime.

Running code at read-time lets users reprogram Lisp’s syntax; running code atcompile-time is the basis of macros; compiling at runtime is the basis of Lisp’s useas an extension language in programs like Emacs; and reading at runtime enablesprograms to communicate using s-expressions, an idea recently reinvented asXML.

When Lisp first appeared, these ideas were far removed from ordinary programmingpractice, which was dictated largely by the hardware available in the late 1950s.Over time, the default language, embodied in a succession of popular languages, hasgradually evolved toward Lisp. Ideas 1-5 are now widespread. Number 6 is startingto appear in the mainstream. Python has a form of 7, though there doesn’t seem tobe any syntax for it.

As for number 8, this may be the most interesting of the lot. Ideas 8 and 9 onlybecame part of Lisp by accident, because Steve Russell implemented somethingMcCarthy had never intended to be implemented. And yet these ideas turn out tobe responsible for both Lisp’s strange appearance and its most distinctive features.Lisp looks strange not so much because it has a strange syntax as because it hasno syntax; you express programs directly in the parse trees that get built behind thescenes when other languages are parsed, and these trees are made of lists, whichare Lisp data structures.

Expressing the language in its own data structures turns out to be a very powerfulfeature. Ideas 8 and 9 together mean that you can write programs that writeprograms. That may sound like a bizarre idea, but it’s an everyday thing in Lisp. Themost common way to do it is with something called a macro.

The term "macro" does not mean in Lisp what it means in other languages. A Lispmacro can be anything from an abbreviation to a compiler for a new language. If youwant to really understand Lisp, or just expand your programming horizons, I wouldlearn more6 about macros.

Macros (in the Lisp sense) are still, as far as I know, unique to Lisp. This is partlybecause in order to have macros you probably have to make your language look asstrange as Lisp. It may also be because if you do add that final increment of power,

6 http://www.paulgraham.com/onlisp.html

http://www.paulgraham.com/onlisp.html

http://www.paulgraham.com/onlisp.html


137

you can no longer claim to have invented a new language, but only a new dialectof Lisp.

I mention this mostly as a joke, but it is quite true. If you define a language that hascar, cdr, cons, quote, cond, atom, eq, and a notation for functions expressed as lists,then you can build all the rest of Lisp out of it. That is in fact the defining quality ofLisp: it was in order to make this so that McCarthy gave Lisp the shape it has.

Where Languages Matter

So suppose Lisp does represent a kind of limit that mainstream languages areapproaching asymptotically-- does that mean you should actually use it to writesoftware? How much do you lose by using a less powerful language? Isn’t it wiser,sometimes, not to be at the very edge of innovation? And isn’t popularity to someextent its own justification? Isn’t the pointy-haired boss right, for example, to wantto use a language for which he can easily hire programmers?

There are, of course, projects where the choice of programming language doesn’tmatter much. As a rule, the more demanding the application, the more leverage youget from using a powerful language. But plenty of projects are not demanding atall. Most programming probably consists of writing little glue programs, and for littleglue programs you can use any language that you’re already familiar with and thathas good libraries for whatever you need to do. If you just need to feed data fromone Windows app to another, sure, use Visual Basic.

You can write little glue programs in Lisp too (I use it as a desktop calculator), butthe biggest win for languages like Lisp is at the other end of the spectrum, whereyou need to write sophisticated programs to solve hard problems in the face of fiercecompetition. A good example is the airline fare search program7 that ITA Softwarelicenses to Orbitz. These guys entered a market already dominated by two big,entrenched competitors, Travelocity and Expedia, and seem to have just humiliatedthem technologically.

The core of ITA’s application is a 200,000 line Common Lisp program that searchesmany orders of magnitude more possibilities than their competitors, who apparentlyare still using mainframe-era programming techniques. (Though ITA is also in a senseusing a mainframe-era programming language.) I have never seen any of ITA’s code,but according to one of their top hackers they use a lot of macros, and I am notsurprised to hear it.

7 http://www.paulgraham.com/carl.html

http://www.paulgraham.com/carl.html

http://www.paulgraham.com/carl.html


138

Centripetal Forces

I’m not saying there is no cost to using uncommon technologies. The pointy-haired boss is not completely mistaken to worry about this. But because he doesn’tunderstand the risks, he tends to magnify them.

I can think of three problems that could arise from using less common languages.Your programs might not work well with programs written in other languages. Youmight have fewer libraries at your disposal. And you might have trouble hiringprogrammers.

How much of a problem is each of these? The importance of the first varies dependingon whether you have control over the whole system. If you’re writing software thathas to run on a remote user’s machine on top of a buggy, closed operating system(I mention no names), there may be advantages to writing your application in thesame language as the OS. But if you control the whole system and have the sourcecode of all the parts, as ITA presumably does, you can use whatever languages youwant. If any incompatibility arises, you can fix it yourself.

In server-based applications you can get away with using the mostadvanced technologies, and I think this is the main cause of what JonathanErickson calls the "http://www.byte.com/documents/s=1821/byt20011214s0003/[programming language renaissance]." This is why we even hear about newlanguages like Perl and Python. We’re not hearing about these languages becausepeople are using them to write Windows apps, but because people are using themon servers. And as software shifts off the desktop8 and onto servers (a future evenMicrosoft seems resigned to), there will be less and less pressure to use middle-of-the-road technologies.

As for libraries, their importance also depends on the application. For less demandingproblems, the availability of libraries can outweigh the intrinsic power of thelanguage. Where is the breakeven point? Hard to say exactly, but wherever it is,it is short of anything you’d be likely to call an application. If a company considersitself to be in the software business, and they’re writing an application that will beone of their products, then it will probably involve several hackers and take at leastsix months to write. In a project of that size, powerful languages probably start tooutweigh the convenience of pre-existing libraries.

8 http://www.paulgraham.com/road.html

http://www.paulgraham.com/road.html

http://www.paulgraham.com/road.html


139

The third worry of the pointy-haired boss, the difficulty of hiring programmers, I thinkis a red herring. How many hackers do you need to hire, after all? Surely by now weall know that software is best developed by teams of less than ten people. And youshouldn’t have trouble hiring hackers on that scale for any language anyone has everheard of. If you can’t find ten Lisp hackers, then your company is probably based inthe wrong city for developing software.

In fact, choosing a more powerful language probably decreases the size of the teamyou need, because (a) if you use a more powerful language you probably won’t needas many hackers, and (b) hackers who work in more advanced languages are likelyto be smarter.

I’m not saying that you won’t get a lot of pressure to use what are perceived as"standard" technologies. At Viaweb (now Yahoo Store), we raised some eyebrowsamong VCs and potential acquirers by using Lisp. But we also raised eyebrows byusing generic Intel boxes as servers instead of "industrial strength" servers likeSuns, for using a then-obscure open-source Unix variant called FreeBSD instead of areal commercial OS like Windows NT, for ignoring a supposed e-commerce standardcalled SET9 that no one now even remembers, and so on.

You can’t let the suits make technical decisions for you. Did it alarm some potentialacquirers that we used Lisp? Some, slightly, but if we hadn’t used Lisp, we wouldn’thave been able to write the software that made them want to buy us. What seemedlike an anomaly to them was in fact cause and effect.

If you start a startup, don’t design your product to please VCs or potential acquirers.Design your product to please the users. If you win the users, everything else willfollow. And if you don’t, no one will care how comfortingly orthodox your technologychoices were.

The Cost of Being Average

How much do you lose by using a less powerful language? There is actually somedata out there about that.

The most convenient measure of power is probably code size10. The point of high-level languages is to give you bigger abstractions-- bigger bricks, as it were, so you

9 http://news.com.com/2100-1017-225723.html10 http://www.paulgraham.com/power.html

http://news.com.com/2100-1017-225723.html

http://www.paulgraham.com/power.html

http://news.com.com/2100-1017-225723.html



140

don’t need as many to build a wall of a given size. So the more powerful the language,the shorter the program (not simply in characters, of course, but in distinct elements).

How does a more powerful language enable you to write shorter programs? Onetechnique you can use, if the language will let you, is something called bottom-upprogramming11. Instead of simply writing your application in the base language,you build on top of the base language a language for writing programs like yours,then write your program in it. The combined code can be much shorter than if youhad written your whole program in the base language-- indeed, this is how mostcompression algorithms work. A bottom-up program should be easier to modify aswell, because in many cases the language layer won’t have to change at all.

Code size is important, because the time it takes to write a program depends mostlyon its length. If your program would be three times as long in another language,it will take three times as long to write-- and you can’t get around this by hiringmore people, because beyond a certain size new hires are actually a net lose. FredBrooks described this phenomenon in his famous book The Mythical Man-Month, andeverything I’ve seen has tended to confirm what he said.

So how much shorter are your programs if you write them in Lisp? Most of thenumbers I’ve heard for Lisp versus C, for example, have been around 7-10x. But arecent article about ITA in New Architect12 magazine said that "one line of Lisp canreplace 20 lines of C," and since this article was full of quotes from ITA’s president,I assume they got this number from ITA. If so then we can put some faith in it; ITA’ssoftware includes a lot of C and C++ as well as Lisp, so they are speaking fromexperience.

My guess is that these multiples aren’t even constant. I think they increase when youface harder problems and also when you have smarter programmers. A really goodhacker can squeeze more out of better tools.

As one data point on the curve, at any rate, if you were to compete with ITA and choseto write your software in C, they would be able to develop software twenty timesfaster than you. If you spent a year on a new feature, they’d be able to duplicate it inless than three weeks. Whereas if they spent just three months developing somethingnew, it would be five years before you had it too.

And you know what? That’s the best-case scenario. When you talk about code-sizeratios, you’re implicitly assuming that you can actually write the program in the

11 http://www.paulgraham.com/progbot.html12 http://www.newarchitectmag.com/documents/s=2286/new1015626014044/

http://www.paulgraham.com/progbot.html


http://www.newarchitectmag.com/documents/s=2286/new1015626014044/


http://www.newarchitectmag.com/documents/s=2286/new1015626014044/


141

weaker language. But in fact there are limits on what programmers can do. If you’retrying to solve a hard problem with a language that’s too low-level, you reach a pointwhere there is just too much to keep in your head at once.

So when I say it would take ITA’s imaginary competitor five years to duplicatesomething ITA could write in Lisp in three months, I mean five years if nothing goeswrong. In fact, the way things work in most companies, any development project thatwould take five years is likely never to get finished at all.

I admit this is an extreme case. ITA’s hackers seem to be unusually smart, and C isa pretty low-level language. But in a competitive market, even a differential of twoor three to one would be enough to guarantee that you’d always be behind.

A Recipe

This is the kind of possibility that the pointy-haired boss doesn’t even want to thinkabout. And so most of them don’t. Because, you know, when it comes down to it,the pointy-haired boss doesn’t mind if his company gets their ass kicked, so long asno one can prove it’s his fault. The safest plan for him personally is to stick close tothe center of the herd.

Within large organizations, the phrase used to describe this approach is "industrybest practice." Its purpose is to shield the pointy-haired boss from responsibility: ifhe chooses something that is "industry best practice," and the company loses, hecan’t be blamed. He didn’t choose, the industry did.

I believe this term was originally used to describe accounting methods and so on.What it means, roughly, is don’t do anything weird. And in accounting that’s probablya good idea. The terms "cutting-edge" and "accounting" do not sound good together.But when you import this criterion into decisions about technology, you start to getthe wrong answers.

Technology often should be cutting-edge. In programming languages, as Erann Gathas pointed out, what "industry best practice" actually gets you is not the best, butmerely the average. When a decision causes you to develop software at a fractionof the rate of more aggressive competitors, "best practice" is a misnomer.

So here we have two pieces of information that I think are very valuable. In fact, Iknow it from my own experience. Number 1, languages vary in power. Number 2,most managers deliberately ignore this. Between them, these two facts are literally


142

a recipe for making money. ITA is an example of this recipe in action. If you want towin in a software business, just take on the hardest problem you can find, use themost powerful language you can get, and wait for your competitors' pointy-hairedbosses to revert to the mean.

Appendix: Power

As an illustration of what I mean about the relative power of programming languages,consider the following problem. We want to write a function that generatesaccumulators-- a function that takes a number n, and returns a function that takesanother number i and returns n incremented by i.

(That’s incremented by, not plus. An accumulator has to accumulate.)

In Common Lisp this would be

(defun foo (n) (lambda (i) (incf n i)))

and in Perl 5,

sub foo { my ($n) = @_; sub {$n += shift}}

which has more elements than the Lisp version because you have to extractparameters manually in Perl.

In Smalltalk the code is slightly longer than in Lisp

foo: n |s| s := n. ^[:i| s := s+i. ]

because although in general lexical variables work, you can’t do an assignment to aparameter, so you have to create a new variable s.


143

In Javascript the example is, again, slightly longer, because Javascript retains thedistinction between statements and expressions, so you need explicit returnstatements to return values:

function foo(n) { return function (i) { return n += i } }

(To be fair, Perl also retains this distinction, but deals with it in typical Perl fashionby letting you omit `return`s.)

If you try to translate the Lisp/Perl/Smalltalk/Javascript code into Python you run intosome limitations. Because Python doesn’t fully support lexical variables, you haveto create a data structure to hold the value of n. And although Python does have afunction data type, there is no literal representation for one (unless the body is onlya single expression) so you need to create a named function to return. This is whatyou end up with:

def foo(n): s = [n] def bar(i): s[0] += i return s[0] return bar

Python users might legitimately ask why they can’t just write

def foo(n): return lambda i: return n += i

or even

def foo(n): lambda i: n += i

and my guess is that they probably will, one day. (But if they don’t want to wait forPython to evolve the rest of the way into Lisp, they could always just…)

In OO languages, you can, to a limited extent, simulate a closure (a function thatrefers to variables defined in enclosing scopes) by defining a class with one method


144

and a field to replace each variable from an enclosing scope. This makes theprogrammer do the kind of code analysis that would be done by the compiler ina language with full support for lexical scope, and it won’t work if more than onefunction refers to the same variable, but it is enough in simple cases like this.

Python experts seem to agree that this is the preferred way to solve the problem inPython, writing either

def foo(n): class acc: def __init__(self, s): self.s = s def inc(self, i): self.s += i return self.s return acc(n).inc

or

class foo: def __init__(self, n): self.n = n def __call__(self, i): self.n += i return self.n

I include these because I wouldn’t want Python advocates to say I wasmisrepresenting the language, but both seem to me more complex than the firstversion. You’re doing the same thing, setting up a separate place to hold theaccumulator; it’s just a field in an object instead of the head of a list. And the use ofthese special, reserved field names, especially call , seems a bit of a hack.

In the rivalry between Perl and Python, the claim of the Python hackers seems to bethat that Python is a more elegant alternative to Perl, but what this case shows is thatpower is the ultimate elegance: the Perl program is simpler (has fewer elements),even if the syntax is a bit uglier.

How about other languages? In the other languages mentioned in this talk-- Fortran,C, C++, Java, and Visual Basic-- it is not clear whether you can actually solve thisproblem. Ken Anderson says that the following code is about as close as you canget in Java:


145

public interface Inttoint { public int call(int i);}

public static Inttoint foo(final int n) { return new Inttoint() { int s = n; public int call(int i) { s = s + i; return s; } };}

This falls short of the spec because it only works for integers. After many emailexchanges with Java hackers, I would say that writing a properly polymorphic versionthat behaves like the preceding examples is somewhere between damned awkwardand impossible. If anyone wants to write one I’d be very curious to see it, but Ipersonally have timed out.

It’s not literally true that you can’t solve this problem in other languages, of course.The fact that all these languages are Turing-equivalent means that, strictly speaking,you can write any program in any of them. So how would you do it? In the limit case,by writing a Lisp interpreter in the less powerful language.

That sounds like a joke, but it happens so often to varying degrees in largeprogramming projects that there is a name for the phenomenon, Greenspun’s TenthRule:

Any sufficiently complicated C or Fortran program contains an ad hocinformally-specified bug-ridden slow implementation of half of CommonLisp.

If you try to solve a hard problem, the question is not whether you will use a powerfulenough language, but whether you will (a) use a powerful language, (b) write ade facto interpreter for one, or (c) yourself become a human compiler for one. Wesee this already begining to happen in the Python example, where we are in effectsimulating the code that a compiler would generate to implement a lexical variable.

This practice is not only common, but institutionalized. For example, in the OO worldyou hear a good deal about "patterns". I wonder if these patterns are not sometimesevidence of case (c), the human compiler, at work. When I see patterns in myprograms, I consider it a sign of trouble. The shape of a program should reflect only


146

the problem it needs to solve. Any other regularity in the code is a sign, to me at least,that I’m using abstractions that aren’t powerful enough-- often that I’m generatingby hand the expansions of some macro that I need to write.

Notes

• The IBM 704 CPU was about the size of a refrigerator, but a lot heavier. The CPUweighed 3150 pounds, and the 4K of RAM was in a separate box weighing another4000 pounds. The Sub-Zero 690, one of the largest household refrigerators,weighs 656 pounds.

• Steve Russell also wrote the first (digital) computer game, Spacewar, in 1962.

• If you want to trick a pointy-haired boss into letting you write software in Lisp,you could try telling him it’s XML.

• Here is the accumulator generator in other Lisp dialects:

Scheme: (define (foo n) (lambda (i) (set! n (+ n i)) n))Goo: (df foo (n) (op incf n _)))Arc: (def foo (n) [++ n _])

• Erann Gat’s sad tale about "industry best practice" at JPL inspired me to addressthis generally misapplied phrase.

• Peter Norvig found that 16 of the 23 patterns in Design Patterns were "http://www.norvig.com/design-patterns/[invisible or simpler]" in Lisp.

• Thanks to the many people who answered my questions about various languagesand/or read drafts of this, including Ken Anderson, Trevor Blackwell, Erann Gat,Dan Giffin, Sarah Harlin, Jeremy Hylton, Robert Morris, Peter Norvig, Guy Steele,and Anton van Straaten. They bear no blame for any opinions expressed.

• Related:

Many people have responded to this talk, so I have set up an additional page to dealwith the issues they have raised: Re: Revenge of the Nerds13.

It also set off an extensive and often useful discussion on the LL114 mailing list. Seeparticularly the mail by Anton van Straaten on semantic compression.

13 http://www.paulgraham.com/icadmore.html14 http://www.ai.mit.edu/~gregs/ll1-discuss-archive-html/threads.html

http://www.paulgraham.com/icadmore.html

http://www.ai.mit.edu/~gregs/ll1-discuss-archive-html/threads.html

http://www.paulgraham.com/icadmore.html

http://www.ai.mit.edu/~gregs/ll1-discuss-archive-html/threads.html


147

Some of the mail on LL1 led me to try to go deeper into the subject of languagepower in Succinctness is Power15.

A larger set of canonical implementations of the accumulator generatorbenchmark16 are collected together on their own page.

15 http://www.paulgraham.com/power.html16 http://www.paulgraham.com/accgen.html






148

Screaming ArchitectureWritten by: Robert Martin at 2011-09-301

Imagine that you are looking at the blueprints of a building. This document, preparedby an architect, tells you the plans for the building. What do these plans tell you?

If the plans you are looking at are for a single family residence, then you’ll likelysee a front entrance, a foyer leading to a living room and perhaps a dining room.There’ll likely be a kitchen a short distance away, close to the dining room. Perhapsa dinette area next to the kitchen, and probably a family room close to that. As youlooked at those plans, there’d be no question that you were looking at a house. Thearchitecture would scream: house.

Or if you were looking at the architecture of a library, you’d likely see a grandentrance, an area for check-in-out clerks, reading areas, small conference rooms, andgallery after gallery capable of holding bookshelves for all the books in the library.That architecture would scream: library.

So what does the architecture of your application scream? When you look at thetop level directory structure, and the source files in the highest level package;do they scream: Health Care System, or Accounting System, or InventoryManagement System? Or do they scream: Rails, or Spring/Hibernate, or ASP?

The Theme of an Architecture

Go back and read Ivar Jacobson’s seminal work on software architecture: ObjectOriented Software Engineering. Notice the subtitle of the book: A use case drivenapproach. In this book Ivar makes the point that software architectures are structuresthat support the use cases of the system. Just as the plans for a house or a libraryscream about the use cases of those buildings, so should the architecture of asoftware application scream about the use cases of the application.

Architectures are not (or should not) be about frameworks. Architectures should notbe supplied by frameworks. Frameworks are tools to be used, not architectures to beconformed to. If your architecture is based on frameworks, then it cannot be basedon your use cases.

1 http://blog.8thlight.com/uncle-bob/2011/09/30/Screaming-Architecture.html

http://blog.8thlight.com/uncle-bob/2011/09/30/Screaming-Architecture.html

http://blog.8thlight.com/uncle-bob/2011/09/30/Screaming-Architecture.html

Screaming Architecture

149

The Purpose of an Architecture

The reason that good architectures are centered around use-cases is so thatarchitects can safely describe the structures that support those use-cases withoutcommitting to frameworks, tools, and environment. Again, consider the plans for ahouse. The first concern of the architect is to make sure that the house is usable, itis not to ensure that the house is made of bricks. Indeed, the architect takes painsto ensure that the homeowner can decide about bricks, stone, or cedar later, afterthe plans ensure that the use cases are met.

A good software architecture allows decisions about frameworks, databases, web-servers, and other environmental issues and tools, to be deferred and delayed. Agood architecture makes it unnecessary to decide on Rails, or Spring, or Hibernate,or Tomcat or MySql, until much later in the project. A good architecture makes it easyto change your mind about those decisions too. A good architecture emphasizes theuse-cases and decouples them from peripheral concerns.

But what about the Web?

Is the web an architecture? Does the fact that your system is delivered on theweb dictate the architecture of your system? Of course not! The Web is a deliverymechanism, and your application architecture should treat it as such. The fact thatyour application is delivered over the web is a detail and should not dominate yoursystem structure. Indeed, the fact that your application is delivered over the webis something you should defer. Your system architecture should be as ignorant aspossible about how it is to be delivered. You should be able to deliver it as a consoleapp, or a web app, or a thick client app, or even a web service app, without unduecomplication or change to the fundamental architecture.

Frameworks are tools, not ways of life.

Frameworks can be very powerful and very useful. Framework authors often believein their frameworks. The examples they write for how to use their frameworks aretold from the point of view of a true believer. Other authors who write about theframework also tend to be disciples of the true belief. They show you the way to usethe framework. Often it is an all-encompassing, all-pervading, let-the-framework-do-everything position. This is not the position you want to take.

Look at each framework with a jaded eye. View it skeptically. Yes, it might help,but at what cost. How should I use it, and how should I protect myself from it. How

Screaming Architecture

150

can I preserve the use-case emphasis of my architecture? How can I prevent theframework from taking over that architecture.

Testable Architectures.

If you system architecture is all about the use cases, and if you have kept yourframeworks at arms-length. Then you should be able to unit-test all those use caseswithout any of the frameworks in place. You shouldn’t need the web server runningin order to run your tests. You shouldn’t need the database connected in orderto run your tests. Your business objects should be plain old objects that have nodependencies on frameworks or databases or other complications. Your use caseobjects should coordinate your business objects. And all of them together should betestable in-situ, without any of the complications of frameworks.

Conclusion

Your architectures should tell readers about the system, not about the frameworksyou used in your system. If you are building a health-care system, then when newprogrammers look at the source repository, their first impression should be: “Oh,this is a heath-care system”. Those new programmers should be able to learn all theuse cases of the system, and still not know how the system is delivered. They maycome to you and say: “We see some things that look sorta like models, but where arethe views and controllers”, and you should say: “Oh, those are details that needn’tconcern you at the moment, we’ll show them to you later.”

For more on this topic, see Episode VII – Architecture, Use-cases, and High LevelDesign, at cleancoders.com.

151

How to securely hash passwords?Written by: Thomas Pornin at 2nd March 20131

The Theory

We need to hash passwords as a second line of defence. A server which canauthenticate users necessarily contains, somewhere in its entrails, some data whichcan be used to validate a password. A very simple system would just store thepasswords themselves, and validation would be a simple comparison. But if anhostile outsider gains a simple glimpse at the contents of the file or databasetable which contains the passwords, then that attacker learns a lot. Unfortunately,such partial, read-only breaches do occur in practice (a mislaid backup tape, adecommissioned but not wiped out hard disk, as an aftermath of a SQL injectionattack… the possibilities are numerous). See this blog post2 for a detailed discussion.

Since the overall contents of a server which can validate passwords are necessarilysufficient to, indeed, validate passwords, an attacker who got a read-only snapshotof the server is in position to make an offline dictionary attack3: he tries potentialpasswords until a match is found. We cannot avoid that. So we will want to makethat kind of attack as hard as possible. Our tools are the following:

• Cryptographic hash functions4: these are fascinating mathematical objectswhich everybody can compute efficiently, and yet nobody knows how to invertthem. This looks good for our problem: the server could store a hash of apassword; when presented with a putative password, the server just has to hashit to see if it gets the same value; and yet, knowing the hash does not reveal thepassword itself.

• Salts: among the advantages of the attacker over the defender, is parallelism.The attacker usually grabs a whole list of hashed passwords, and is interested inbreaking as many of them as possible. He may try to attack several in parallels.For instance, the attacker may consider one potential password, hash it, and thencompare the value with 100 hashed passwords; this means that the attackershares the cost of hashing over several attacked passwords. A similar optimisation

1 http://security.stackexchange.com/a/318462 http://security.blogoverflow.com/2011/11/why-passwords-should-be-hashed/3 http://en.wikipedia.org/wiki/Dictionary_attack4 http://en.wikipedia.org/wiki/Cryptographic_hash_function

http://security.stackexchange.com/a/31846

http://security.blogoverflow.com/2011/11/why-passwords-should-be-hashed/

http://en.wikipedia.org/wiki/Dictionary_attack

http://en.wikipedia.org/wiki/Cryptographic_hash_function

http://security.stackexchange.com/a/31846

http://security.blogoverflow.com/2011/11/why-passwords-should-be-hashed/

http://en.wikipedia.org/wiki/Dictionary_attack

http://en.wikipedia.org/wiki/Cryptographic_hash_function

How to securely hash passwords?

152

is precomputed tables, including rainbow tables5; this is still parallelism, with aspace-time change of coordinates.

The common characteristic of all attacks which use parallelism is that they workover several passwords which were processed with the exact same hash function.Salting is about using not one hash function, but a lot of distinct hash functions;ideally, each instance of password hashing should use its own hash function. A saltis a way to select a specific hash function among a big family of hash functions.Properly applied salts will completely thwart parallel attacks (including rainbowtables).

• Slowness: computers become faster over time (Gordon Moore, co-founder ofIntel, theorized it in his famous law6). Human brains do not. This means thatattackers can "try" more and more potential passwords as years pass, while userscannot remember more and more complex passwords (or flatly refuse to). Tocounter that trend, we can make hashing inherently slow by defining the hashfunction to use a lot of internal iterations (thousands, possibly millions).

We have a few standard cryptographic hash functions; most famous are MD57 andthe SHA family8. Building a secure hash function out of elementary operations isfar from easy. When cryptographers want to do that, they think hard, then harder,and organize a tournament where the functions fight each other fiercely. Whenhundreds of cryptographers gnawed and scraped and punched at a function forseveral years and found nothing bad to say about it, then they begin to admit thatmaybe that specific function could be considered as more or less secure. This is justwhat happened in the SHA-3 competition9. We have to use this way of designinghash function because we know no better way. Mathematically, we do not know ifsecure hash functions actually exist; we just have "candidates" (that’s the differencebetween "it cannot be broken" and "nobody in the world knows how to break it").

A basic hash function, even if secure as a hash function, is not appropriate forpassword hashing, because:

• it is unsalted, allowing for parallel attacks (rainbow tables for MD5 or SHA-110

can be obtained for free, you do not even need to recompute them yourself);

5 http://en.wikipedia.org/wiki/Rainbow_table6 http://en.wikipedia.org/wiki/Moore%27s_law7 http://en.wikipedia.org/wiki/MD58 http://en.wikipedia.org/wiki/Secure_Hash_Algorithm9 http://en.wikipedia.org/wiki/NIST_hash_function_competition10 http://www.freerainbowtables.com/tables/

http://en.wikipedia.org/wiki/Rainbow_table

http://en.wikipedia.org/wiki/Moore%27s_law

http://en.wikipedia.org/wiki/MD5

http://en.wikipedia.org/wiki/Secure_Hash_Algorithm

http://en.wikipedia.org/wiki/NIST_hash_function_competition

http://www.freerainbowtables.com/tables/

http://en.wikipedia.org/wiki/Rainbow_table

http://en.wikipedia.org/wiki/Moore%27s_law

http://en.wikipedia.org/wiki/MD5

http://en.wikipedia.org/wiki/Secure_Hash_Algorithm

http://en.wikipedia.org/wiki/NIST_hash_function_competition

http://www.freerainbowtables.com/tables/


153

• it is way too fast, and gets faster with technological advances. With a recent GPU(i.e. off-the-shelf consumer product which everybody can buy), hashing rate iscounted in billions of passwords per second11.

So we need something better. It so happens that slapping together a hash functionand a salt, and iterating it, is not easier to do than designing a hash function — atleast, if you want the result to be secure. There again, you have to rely onstandard constructions which have survived the continuous onslaught of vindicativecryptographers.

Good Password Hashing Functions

PBKDF2

PBKDF212 comes from PKCS#513. It is parameterized with an iteration count (aninteger, at least 1, no upper limit), a salt (an arbitrary sequence of bytes, noconstraint on length), a required output length (PBKDF2 can generate an output ofconfigurable length), and an "underlying PRF". In practice, PBKDF2 is always usedwith HMAC14, which is itself a construction which is built over an underlying hashfunction. So when we say "PBKDF2 with SHA-1", we actually mean "PBKDF2 withHMAC with SHA-1".

Advantages of PBKDF2:

• Has been specified for a long time, seems unscathed for now.

• Is already implemented in various framework (e. g. it is provided with .NET15).

• Highly configurable (although some implementations do not let you choose thehash function, e.g. the one in .NET is for SHA-1 only).

• Received NIST blessings16 (modulo the difference between hashing and keyderivation; see later on).

• Configurable output length (again, see later on).

Drawbacks of PBKDF2:

11 http://www.golubev.com/hashgpu.htm12 http://en.wikipedia.org/wiki/PBKDF213 http://tools.ietf.org/html/rfc289814 http://en.wikipedia.org/wiki/Hash-based_message_authentication_code15 http://msdn.microsoft.com/en-us/library/system.security.cryptography.rfc2898derivebytes.aspx16 http://csrc.nist.gov/publications/nistpubs/800-132/nist-sp800-132.pdf

http://www.golubev.com/hashgpu.htm

http://en.wikipedia.org/wiki/PBKDF2

http://tools.ietf.org/html/rfc2898

http://en.wikipedia.org/wiki/Hash-based_message_authentication_code

http://msdn.microsoft.com/en-us/library/system.security.cryptography.rfc2898derivebytes.aspx

http://csrc.nist.gov/publications/nistpubs/800-132/nist-sp800-132.pdf

http://www.golubev.com/hashgpu.htm

http://en.wikipedia.org/wiki/PBKDF2



http://msdn.microsoft.com/en-us/library/system.security.cryptography.rfc2898derivebytes.aspx

http://csrc.nist.gov/publications/nistpubs/800-132/nist-sp800-132.pdf


154

• CPU-intensive only, thus amenable to high optimization with GPU (the defenderis a basic server which does generic things, i.e. a PC, but the attacker can spendhis budget on more specialized hardware, which will give him an edge).

• You still have to manage the parameters yourself (salt generation andstorage, iteration count encoding…). There is a standard encoding for PBKDF2parameters17 but it uses ASN.1 so most people will avoid it if they can (ASN.1can be tricky to handle for the non-expert).

bcrypt

bcrypt18 was designed by reusing and expanding elements of a block cipher calledBlowfish19. The iteration count is a power of two, which is a tad less configurablethan PBKDF2, but sufficiently so nevertheless. This is the core password hashingmechanism in the OpenBSD20 operating system.

Advantages of bcrypt:

• Many available implementations in various languages (see the links at the endof the Wikipedia page).

• More resilient to GPU; this is due to details of its internal design. The bcryptauthors made it so voluntarily: they reused Blowfish because Blowfish was basedon an internal RAM table which is constantly accessed and modified throughoutthe processing. This makes life much harder for whoever wants to speed up bcryptwith a GPU (GPU are not good at making a lot of memory accesses in parallel).See here21 for some discussion.

• Standard output encoding which includes the salt, the iteration count and theoutput as one simple to store character string of printable characters.

Drawbacks of bcrypt:

• Output size is fixed: 192 bits.• While bcrypt is good at thwarting GPU, it can still be thoroughly optimized with

FPGA22: modern FPGA chips have a lot of small embedded RAM blocks which are

17 http://tools.ietf.org/html/rfc2898#appendix-A.218 http://en.wikipedia.org/wiki/Bcrypt19 http://en.wikipedia.org/wiki/Blowfish_%28cipher%2920 http://www.openbsd.org/21 http://security.stackexchange.com/questions/4781/do-any-security-experts-recommend-bcrypt-for-password-storage/6415#641522 http://en.wikipedia.org/wiki/Field-programmable_gate_array

http://tools.ietf.org/html/rfc2898#appendix-A.2


http://en.wikipedia.org/wiki/Bcrypt

http://en.wikipedia.org/wiki/Blowfish_%28cipher%29

http://www.openbsd.org/

http://security.stackexchange.com/questions/4781/do-any-security-experts-recommend-bcrypt-for-password-storage/6415#6415

http://en.wikipedia.org/wiki/Field-programmable_gate_array


http://en.wikipedia.org/wiki/Bcrypt

http://en.wikipedia.org/wiki/Blowfish_%28cipher%29

http://www.openbsd.org/



http://en.wikipedia.org/wiki/Field-programmable_gate_array


155

very convenient for running many bcrypt implementations in parallel within onechip. It has been done23.

• Input password size is limited to 51 characters. In order to handle longerpasswords, one has to combine bcrypt with a hash function24 (you hash thepassword and then use the hash value as the "password" for bcrypt). Combiningcryptographic primitives is known to be dangerous (see above) so such gamescannot be recommended on a general basis.

scrypt

scrypt25 is a much newer construction (designed in 2009) which builds over PBKDF2and a stream cipher called Salsa20/826, but these are just tools around the corestrength of scrypt, which is RAM. scrypt has been designed to inherently use alot of RAM (it generates some pseudo-random bytes, then repeatedly read themin a pseudo-random sequence). "Lots of RAM" is something which is hard to makeparallel. A basic PC is good at RAM access, and will not try to read dozens of unrelatedRAM bytes simultaneously. An attacker with a GPU or a FPGA will want to do that,and will find it difficult.

Advantages of scrypt:

• A PC, i.e. exactly what the defender will use when hashing passwords, is the mostefficient platform (or close enough) for computing scrypt. The attacker no longergets a boost by spending his dollars on GPU or FPGA.

• One more way to tune the function: memory size.

Drawbacks of scrypt:

• Still new (my own rule of thumb is to wait at least 5 years of general exposure, sono scrypt for production until 2014 — but, of course, it is best if other people tryscrypt in production, because this gives extra exposure).

• Not as many available, ready-to-use implementations for various languages.

• Unclear whether the CPU / RAM mix is optimal. For each of the pseudo-randomRAM accesses, scrypt still computes a hash function. A cache miss will be about

23 http://openwall.info/wiki/john/FPGA24 http://security.stackexchange.com/questions/6623/pre-hash-password-before-applying-bcrypt-to-avoid-restricting-password-length25 http://www.tarsnap.com/scrypt.html26 http://en.wikipedia.org/wiki/Salsa20

http://openwall.info/wiki/john/FPGA

http://security.stackexchange.com/questions/6623/pre-hash-password-before-applying-bcrypt-to-avoid-restricting-password-length

http://www.tarsnap.com/scrypt.html

http://en.wikipedia.org/wiki/Salsa20

http://openwall.info/wiki/john/FPGA



http://www.tarsnap.com/scrypt.html

http://en.wikipedia.org/wiki/Salsa20


156

200 clock cycles, one SHA-256 invocation is close to 1000. There may be roomfor improvement here.

• Yet another parameter to configure: memory size.

OpenPGP Iterated And Salted S2K

I cite this one because you will use it if you do password-based file encryption withGnuPG27. That tool follows the OpenPGP28 format which defines its own passwordhashing functions, called "Simple S2K", "Salted S2K" and "http://tools.ietf.org/html/rfc4880#section-3.7.1.3[Iterated and Salted S2K]". Only the third one can bedeemed "good" in the context of this answer. It is defined as the hash of a very longstring (configurable, up to about 65 megabytes) consisting of the repetition of an 8-byte salt and the password.

As far as these things go, OpenPGP’s Iterated And Salted S2K is decent; it canbe considered as similar to PBKDF2, with less configurability. You will very rarelyencounter it outside of OpenPGP, as a stand-alone function.

Unix "crypt"

Recent Unix-like systems (e.g. Linux), for validating user passwords, use iteratedand salted variants of the crypt()29 function based on good hash functions, withthousands of iterations. This is reasonably good. Some systems can also use bcrypt,which is better.

The old crypt() function, based on the DES block cipher30, is not good enough:

• It is slow in software but fast in hardware, and can be made fast in software too butonly when computing several instances in parallel (technique known as SWAR31

or "bitslicing"). Thus, the attacker is at an advantage.• It is still quite fast, with only 25 iterations.• It has a 12-bit salt, which means that salt reuse will occur quite often.• It truncates passwords to 8 characters (characters beyond the eighth are ignored)

and it also drops the upper bit of each character (so you are more or less stuckwith ASCII).

27 http://www.gnupg.org/28 http://tools.ietf.org/html/rfc488029 http://en.wikipedia.org/wiki/Crypt_%28C%2930 http://en.wikipedia.org/wiki/Data_Encryption_Standard31 http://en.wikipedia.org/wiki/SWAR

http://www.gnupg.org/


http://en.wikipedia.org/wiki/Crypt_%28C%29

http://en.wikipedia.org/wiki/Data_Encryption_Standard

http://en.wikipedia.org/wiki/SWAR

http://www.gnupg.org/


http://en.wikipedia.org/wiki/Crypt_%28C%29

http://en.wikipedia.org/wiki/Data_Encryption_Standard

http://en.wikipedia.org/wiki/SWAR


157

But the more recent variants, which are active by default, will be fine.

Bad Password Hashing Functions

About everything else, in particular virtually every homemade method that peoplerelentlessly invent.

For some reason, many developers insist on designing function themselves, andseem to assume that "secure cryptographic design" means "throw together everykind of cryptographic or non-cryptographic operation that can be thought of". Seethis question32 for an example. The underlying principle seems to be that the sheercomplexity of the resulting utterly tangled mess of instruction will befuddle attackers.In practice, though, the developer himself will be more confused by his own creationthan the attacker.

Complexity is bad. Homemade is bad. New is bad. If you remember that,you’ll avoid 99% of problems related to password hashing, or cryptography, or evensecurity in general.

Password hashing in Windows operating systems used to be mindbogglingly awful33

and now is just terrible (unsalted, non-iterated MD4).

Key Derivation

Up to now, we considered the question of hashing passwords. A close problemis about transforming a password into a symmetric key which can be used forencryption; this is called key derivation34 and is the first thing you do when you"encrypt a file with a password".

It is possible to make contrived examples of password hashing functions which aresecure for the purpose of storing a password validation token, but terrible when itcomes to generating symmetric keys; and the converse is equally possible. But theseexamples are very "artificial". For practical functions like the one described above:

• The output of a password hashing function is acceptable as a symmetric key, afterpossible truncation to the required size.

32 http://security.stackexchange.com/questions/25585/is-my-developers-home-brew-password-security-right-or-wrong-and-why33 http://security.stackexchange.com/questions/2881/is-there-any-advantage-to-splitting-a-password/2883#288334 http://en.wikipedia.org/wiki/Key_derivation_function

http://security.stackexchange.com/questions/25585/is-my-developers-home-brew-password-security-right-or-wrong-and-why

http://security.stackexchange.com/questions/2881/is-there-any-advantage-to-splitting-a-password/2883#2883

http://en.wikipedia.org/wiki/Key_derivation_function





http://en.wikipedia.org/wiki/Key_derivation_function


158

• A Key Derivation Function can serve as a password hashing function as long asthe "derived key" is long enough to avoid "generic preimages" (the attacker isjust lucky and finds a password which yields the same output). An output of morethan 100 bits or so will be enough.

Indeed, PBKDF2 and scrypt are KDF, not password hashing function — and NIST"approves" of PBKDF2 as a KDF, not explicitly as a password hasher (but it is possible,with only a very minute amount of hypocrisy, to read NIST’s prose in such a way thatit seems to say that PBKDF2 is good for hashing passwords).

Conversely, bcrypt is really a block cipher35 (the bulk of the password processingis the "key schedule") which is then used in CTR mode36 to produce three blocks(i.e. 192 bits) of pseudo-random output, making it a kind of hash function. bcryptcan be turned into a KDF with a little surgery, by using the block cipher in CTRmode for more blocks. But, as usual, we cannot recommend such homemadetransforms. Fortunately, 192 bits are already more than enough for most purposes(e.g. symmetric encryption with GCM37 or EAX38 only needs a 128-bit key).

Miscellaneous Topics

How many iterations ?

As much as possible ! This salted-and-slow hashing is an arms race between theattacker and the defender. You use many iterations to make the hashing of apassword harder for everybody. To improve security, you should set that numberas high as you can tolerate on your server, given the tasks that your server mustotherwise fulfill. Higher is better.

Collisions and MD5

MD5 is broken: it is computationally easy to find a lot of pairs of distinct inputs whichhash to the same value. These are called collisions.

However, collisions are not an issue for password hashing. Password hashingrequires the hash function to be resistant to preimages, not to collisions. Collisionsare about finding pairs of messages which give the same output without restriction,

35 http://en.wikipedia.org/wiki/Block_cipher36 http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Counter_.28CTR.2937 http://en.wikipedia.org/wiki/Galois/Counter_Mode38 http://en.wikipedia.org/wiki/EAX_mode

http://en.wikipedia.org/wiki/Block_cipher

http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Counter_.28CTR.29

http://en.wikipedia.org/wiki/Galois/Counter_Mode

http://en.wikipedia.org/wiki/EAX_mode

http://en.wikipedia.org/wiki/Block_cipher

http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Counter_.28CTR.29

http://en.wikipedia.org/wiki/Galois/Counter_Mode

http://en.wikipedia.org/wiki/EAX_mode


159

whereas in password hashing the attacker must find a message which yields a givenoutput that the attacker does not get to choose. This is quite different. As far as weknown, MD5 is still (almost) as strong as it has ever been with regards to preimages(there is a theoretical attack39 which is still very far in the ludicrously impossibleto run in practice).

The real problem with MD5 as it is commonly used in password hashing is that itis very fast, and unsalted. However, PBKDF2 used with MD5 would be robust. Youshould still use SHA-1 or SHA-256 with PBKDF2, but for Public Relations. People getnervous when they hear "MD5".

Salt Generation

The main and only point of the salt is to be as unique as possible. Whenever a saltvalue is reused anywhere, this has the potential to help the attacker.

For instance, if you use the user name as salt, then an attacker (or several colludingattackers) could find it worthwhile to build rainbow tables which attack the passwordhashing function when the salt is "admin" (or "root" or "joe") because there will beseveral, possibly many sites around the world which will have a user named "admin".Similarly, when a user changes his password, he usually keeps his name, leadingto salt reuse. Old passwords are valuable targets, because users have the habit ofreusing passwords in several places (that’s known to be a bad idea, and advertisedas such, but they will do it nonetheless because it makes their life easier), and alsobecause people tend to generate their passwords "in sequence": if you learn thatBob’s old password is "SuperSecretPassword37", then Bob’s current password isprobable "SuperSecretPassword38" or "SuperSecretPassword39".

The cheap way to obtain uniqueness is to use randomness. If you generate yoursalt as a sequence of random bytes from the cryptographically secure PRNG40 thatyour operating system offers ( /dev/urandom , CryptGenRandom() …) then youwill get salt values which will be "unique with a sufficiently high probability". 16 bytesare enough so that you will never see a salt collision in your life, which is overkillbut simple enough.

UUID41 are a standard way of generating "unique" values. Note that "version 4" UUIDjust use randomness (122 random bits), like explained above. A lot of programming

39 http://www.iacr.org/archive/eurocrypt2009/54790136/54790136.pdf40 http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator41 http://en.wikipedia.org/wiki/Universally_unique_identifier

http://www.iacr.org/archive/eurocrypt2009/54790136/54790136.pdf

http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator

http://en.wikipedia.org/wiki/Universally_unique_identifier

http://www.iacr.org/archive/eurocrypt2009/54790136/54790136.pdf

http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator

http://en.wikipedia.org/wiki/Universally_unique_identifier


160

frameworks offer simple to use functions to generate UUID on demand, and they canbe used as salts.

Salt Secrecy

Salts are not meant to be secret; otherwise we would call them keys. You do not needto make salts public, but if you have to make them public (e.g. to support client-sidehashing), then don’t worry too much about it. Salts are there for uniqueness. Strictlyspeaking, the salt is nothing more than the selection of a specific hash function withina big family of functions.

"Pepper"

Cryptographers can never let a metaphor alone; they must extend it with furtheranalogies and bad puns. "Peppering" is about using a secret salt, i.e. a key. If youuse a "pepper" in your password hashing function, then you are switching to a quitedifferent kind of cryptographic algorithm; namely, you are computing a MessageAuthentication Code42 over the password. The MAC key is your "pepper".

Peppering makes sense if you can have a secret key which the attacker will not beable to read. Remember that we use password hashing because we consider that anattacker could grab a copy of the server database, or possible of the whole disk ofthe server. A typical scenario would be a server with two disks in RAID 143. One diskfails (electronic board fries — this happens a lot). The sysadmin replaces the disk,the mirror is rebuilt, no data is lost due to the magic of RAID 1. Since the old diskis dysfunctional, the sysadmin cannot easily wipe its contents. He just discards thedisk. The attacker searches through the garbage bags, retrieves the disk, replacesthe board, and lo! He has a complete image of the whole server system, includingdatabase, configuration files, binaries, operating system… the full monty, as theBritish say. For peppering to be really applicable, you need to be in a special setupwhere there is something more than a PC with disks; you need a HSM44. HSM are veryexpensive, both in hardware and in operational procedure. But with a HSM, you canjust use a secret "pepper" and process passwords with a simple HMAC45 (e.g. withSHA-1 or SHA-256). This will be vastly more efficient than bcrypt/PBKDF2/scrypt and

42 http://en.wikipedia.org/wiki/Message_authentication_code43 http://en.wikipedia.org/wiki/RAID#RAID_144 http://en.wikipedia.org/wiki/Hardware_security_module45 http://en.wikipedia.org/wiki/Hash-based_message_authentication_code

http://en.wikipedia.org/wiki/Message_authentication_code


http://en.wikipedia.org/wiki/RAID#RAID_1

http://en.wikipedia.org/wiki/Hardware_security_module



http://en.wikipedia.org/wiki/RAID#RAID_1

http://en.wikipedia.org/wiki/Hardware_security_module



161

their cumbersome iterations. Also, usage of a HSM will look extremely professionalwhen doing a WebTrust audit46.

Client-side hashing

Since hashing is (deliberately) expensive, it could make sense, in a client-serversituation, to harness the CPU of the connecting clients. After all, when 100 clientsconnect to a single server, the clients collectively have a lot more muscle than theserver.

To perform client-side hashing, the communication protocol must be enhanced tosupport sending the salt back to the client. This implies an extra round-trip, whencompared to the simple client-sends-password-to-server protocol. This may or maynot be easy to add to your specific case.

Client-side hashing is difficult in a Web context because the client uses Javascript,which is quite anemic for CPU-intensive tasks.

In the context of SRP47, password hashing necessarily occurs on the client side.

Conclusion

Use bcrypt. PBKDF2 is not bad either. If you use scrypt you will be a "slightly earlyadopter" with the risks that are implied by this expression; but it would be a goodmove for scientific progress ("crash dummy" is a very honourable profession).

46 http://www.webtrust.org/item64428.aspx47 http://en.wikipedia.org/wiki/Secure_Remote_Password_protocol

http://www.webtrust.org/item64428.aspx

http://en.wikipedia.org/wiki/Secure_Remote_Password_protocol

http://www.webtrust.org/item64428.aspx

http://en.wikipedia.org/wiki/Secure_Remote_Password_protocol

162

Smart Guy Productivity PitfallsWritten by: Hook at 13th March 20131

Productivity is one of my pet topics, because it’s always dogged me a bit, especiallyearly in my career. I’d pull long days and nights and then realize I only actuallyworked (as in, typing in code, debugging stuff, and thinking about problems andtheir solutions) maybe 20% of the time. Upon talking to coworkers, this seemedto be normal, a part of the expected friction costs incurred working in an officeenvironment. Meetings, shooting the shit with coworkers, lunch, email, and, er, stuff.

Figure 4. This is what I mean by "stuff" in 1996

Eventually working around high-productivity professionals like John Carmack mademe realize that if you want to excel, then you have to work hard and focus the wholetime.

1 http://bookofhook.blogspot.de/2013/03/smart-guy-productivity-pitfalls.html

http://bookofhook.blogspot.de/2013/03/smart-guy-productivity-pitfalls.html

http://bookofhook.blogspot.de/2013/03/smart-guy-productivity-pitfalls.html

Smart Guy Productivity Pitfalls

163

Figure 5. John Carmack Productivity Measurement Tool ca.1998

I remember Carmack talking about productivity measurement. While working hewould play a CD, and if he was not being productive, he’d pause the CD player. Thismeant any time someone came into his office to ask him a question or he checkedemail he’d pause the CD player. He’d then measure his output for the day by howmany times he played the CD (or something like that — maybe it was how far he gotdown into his CD stack). I distinctly remember him saying "So if I get up to go to thebathroom, I pause the player".

You know what’s pretty hardcore? Thinking that going to the bathroom is essentiallythe same as fucking off.

(Sidenote: my memory is hazy on this since it was long ago — CD PLAYERS yo).

That level of work ethic and focus is rare, particularly with my generation ofprogrammers who were raised with the vile "work smart, not hard" mantra, coupledwith the sense that we were somehow significantly brighter than most of our peers.Combine those two notions and add the Scotty Principle2 and you get….me.

2 http://www.urbandictionary.com/define.php?term=Scotty%20Principle

http://www.urbandictionary.com/define.php?term=Scotty%20Principle

http://www.urbandictionary.com/define.php?term=Scotty%20Principle


164

If my coworkers were grinding through stuff that took them 20 hours, I’d be all "Iwork smart, not hard" with accompanying snarky eye roll, and I’d assume I couldbust through an equivalent work load in four hours and still look like a goddamnsuperhero.

And sometimes I could, so, hey, validation!

And you can skate by like this (God knows I did) for a long time before a couple thingseventually rear their heads and bite you in your entitled face.

Productivity Deficit: Your Attitude WritingChecks Your Work Ethic Can’t Cash

An overinflated sense of your own abilities creates a constant state of productiondeficit, because you assume that you can make it up with a burst of brilliance and/or crunch.

But there is no countering surplus to offset the deficit. The only way surpluses showup is when you finish a (presumably) hard task much faster than you anticipated. Butinstead of banking the surplus (i.e. moving on immediately to your next task), youspend it relaxing and screwing off because, whew, you just earned a small vacationby busting shit out in an hour that you thought would take all day. Hey, what’s onFacebook and Twitter right now? Do I have any new mail? No? I wonder if anyone hastweeted something recently since the last time I checked…what about Reddit? Oh,


165

an inspirational/funny/cool YouTube video, it’s only eight minutes long, let me watchit now! (Eight minutes later) Sweet…what’s up Twitter? Oh man, I have to make asnarky response, but first I have to Google for that XKCD comic that totally makesmy point for me…

And before you know it, the day is done, and you’re still feeling good because youfinished in one hour what should have taken all day, so all good, right?

Trap of the Easy Task

And yeah, it’s often all good, but when operating at a slight deficit things can gopear shaped quickly when you accidentally spring the trap of the easy task. Tasksso trivial that they barely register as work. Update an SDK? Hour, tops. Implementthis feature that I’ve already done on every other platform? Hour, tops. This is allmindless grunt work that anyone could do, it’s just clicking through dialog boxes,maybe changing a few lines of source, rebuilding, whatever.

In other words, an easy task like this is so easy that it’s a constant time costfor everyone irrespective of ability, so there’s no opportunity nor need for crazyoverestimation since what could possibly go wrong?

Well, as with so many things, it’s the unexpected things that screw you the hardest.The things that should be trivial end up taking days because, shit, that SDK installrequired a new compiler install, and the new compiler install failed and I can’tuninstall it, and now I have to fucking WIPE MY WHOLE GODDAMN HARD DRIVETO RECOVER. Or you need to implement something that was trivial on nine otherplatforms, and it should be 30 minutes on platform number ten, except it lacksa base feature that is present on everything else, the compiler is dying with an"Internal compiler failure 2033", the API call you need to use hangs the dev consoleimmediately and, oh, the dev support site you use to get release notes is totally downfor the foreseeable future.

So what should have taken less than an hour took a week.

It’s like having a perfect monetary budget that assumes no crazy "one time" onlybills, except life is full of crazy one time only bills and the only way you can keep thoseunder control is by giving yourself a budgetary capacitor to dampen the fluctuations.

And now you’re defensive about losing a week to something stupid because youbudgeted an hour for it and waited until the last second to do it and now the schedulehas gone to hell, but it’s not your fault, because it could have happened to anyone!


166

But if you had banked your surplus hours before and/or worked at closer to yourtheoretical peak effectiveness then this type of thing would get absorbed in the wash.

And now you live in a state of mild panic because you’re way smarter than all thisbut you’re never actually getting things done on time.

Identity Recalibration Crisis

Which leads to a potential identity recalibration crisis upon landing at a companywith high performers that work hard and smart. And that will happen if you’re good.Now you’re no longer at the top of the curve. In fact, shit, you’re in the middle orbottom of the curve, a situation your brain probably never considered as an option.

Figure 6. This dude was my wake up call with respect to myover inflated sense of skill

I went through this when I went to id software. I had rationalized that John Carmack’ssuccess was just a factor of timing and luck since, hell, he was my age, and I waspretty smart, so what else could it be? Upon my arrival I had a crash course inhumility, because he was way smarter than me and way more productive as well.

This took a while to sink in, because until then I was used to believing I was one ofthe better programmers at a company. And then working with Carmack I realized he


167

was not just a little better, he was orders of magnitude better. And I couldn’t dismissit with a "But I write a lot of code" hand wave, because he also wrote like 80% ofthe Quake code base.

Sometimes it takes a while for this to sink in. Sometimes it doesn’t sink in at all, andyou’re let go because you’re not very productive compared to your new team of highperformers. Sometimes it sinks in, and you panic and get depressed because yourself-image has always been that of being the strongest swimmer in the school, andright now you’re just trying not to drown, much less keep up with everyone else.

But ideally you shrug off the old counter productive mentality and habits and emergeas another one of the high functioning team members, but that can take a lot ofwork, particularly if you have to get over years of giving it twenty percent.

Killing the Underachiever

If my pithy advice were little more than "Be more like John Carmack" then I canimagine a lot of readers throwing up their hands and saying "Well, fuck it, I’m a lostcause, because that’s not going to happen." But what I can do is relate some of thethings that helped me kill off some of my underachieving habits. The point isn’t tobecome a superstar, it’s to become better, since that’s always the first step.

I don’t believe in mechanical solutions ("Turn off the internet", "Listen to music",and stuff like that) because I don’t think they address the core issues, which arepsychological. Instead I found that I had to do the following.

1. Develop self-awareness. It took working with John Carmack and other highproductivity programmers and admitting that they were way more productivethan me before I really understood how much more productive I could be. In otherwords, if you don’t admit it’s something worth improving, then obviously you’renot going to search for a solution, and if that’s the case, you should go here3

or here4.2. Give a shit. Originally I called this "develop a sense of urgency", but really it’s

just about caring about getting your work done. It doesn’t even matter what youspecifically care about! It can be your professional image, your product, yourteam, your customers, whatever. You just have to care about something that willdrive you to getting things done, because if you don’t, apathy will occupy thatvoid.

3 http://www.reddit.com/4 https://www.facebook.com/

http://www.reddit.com/

https://www.facebook.com/

http://www.reddit.com/

https://www.facebook.com/


168

3. Minimize uncertainty. In another blog article, Productivity vs. Uncertainty andApathy5, I talk about how poorly defined goals can lead to poor productivity. Ifit’s unclear what you need to get done today, then there’s a reasonable chanceyou won’t actually do anything today.

4. Commit to getting something done every day. When you show up in themorning have a well defined set of things to finish that day. Stay as late as youhave to in order to finish. By committing to finishing that task other distractionswill naturally fall by the wayside. For example, I have jiu-jitsu training at 7pm.If I screw off too much during the day, I don’t get to train. Also, by committingto a task, you avoid "being busy" instead of "getting work done", they are notthe same thing6.

5. Never say "I’ll finish it up tomorrow" or "I’ll make up for it by comingin early/staying late/working the weekend". This is an easy trap to get into,where you keep incurring time debt until at some point you realize you’re nowthree weeks behind on a task that should have taken two days. This is like rackingup credit card bills assuming you can pay them off later. Which is fine, until "later"arrives and you’ve only accumulated more debt.

6. Do not overpromise to make up for poor productivity. There’s a tendencywhen we’re falling behind to try to overcompensate with future promises. "WhenI’m done, it’ll be AWESOME" or "I know I’m late, but I’m positive I’ll be done byMonday". By doing those things we just build more debt we can’t pay off, and thatwill eventually lead to a catastrophic melt down when the super final absolutelylast deadline date shows up. Just get shit done, don’t talk about how you’re goingto get shit done.

7. Have an objective productivity metric. This is a mechanical thing, but it actsas a reasonable backstop. If you have changelogs you can reference, then it’seasy to sit down on Friday and ask yourself "What have I done this week?" And ifyou know that it’s possible for others to check on you, then it makes you take eachday a lot more seriously. If you judge your value solely on output, not subjectivethings like "being smart", you will be more productive.

8. Accept that "the grind" is part of the job. A friend of mine’s father has agreat quote: "Son, i don’t wake up every day and go to a place called fun. I wakeup and go to a place called work" You can’t get irate or frustrated that the bulkof the day is typing in boring code and dealing with bugs and other people.

5 http://bookofhook.blogspot.com/2012/09/productivity-vs-uncertainty-apathy.html6 http://bookofhook.blogspot.com/2013/01/being-busy-isnt-same-as-getting-work.html

http://bookofhook.blogspot.com/2012/09/productivity-vs-uncertainty-apathy.html


http://bookofhook.blogspot.com/2013/01/being-busy-isnt-same-as-getting-work.html





169

I still screw off during the day. I am not a code grinding automaton. I read Facebook,chat with others, Tweet, shop on Amazon, get coffee, read forums, write blog postslike this one and I’m totally fine with that.

Because I know I’m committed to getting things done as well. So even though I’mwriting this right now at 3pm, if I have to work until 8pm to get my shit done andchecked in, I’ll do that, so that tomorrow is a fresh start, not a day of "Oh goddamit,let’s try again to finish that task I started a week ago."


170

Once I developed that sense of daily urgency my productivity went up considerably.And if I’m really productive in the morning and afternoon and feel like I can go homeat 4pm with a clean conscience, I do that too, so I don’t burn out.

It’s a push/pull between banking surplus and generating a deficit, just make sure youdon’t do just one or the other.

171

TDD, Straw Men, and RhetoricWritten by: Gary Bernhardt at 2014-04-301

In a blog post called Slow database test fallacy2, David Heinemeier Hansson, thecreator of Rails, begins:

The classical definition of a unit test in TDD lore is one that doesn’ttouch the database.

First, you can immediately tell that this piece of writing is going to be heavilyrhetorical. He refers to "TDD lore" as opposed to, say, "TDD practice". By using theword "lore", he positions it as subjective, unreliable, mythological.

Second, that sentence is false. Isolation from the database, or anything else, isgenerally done with mocks, but mocks didn’t even exist when TDD was rediscoveredby Kent Beck in 1994-19953. They were introduced at XP2000 in a paper called Endo-Testing: Unit Testing with Mock Objects4, and it took a long time after that for themto gain popularity. Their role in software development was still being fleshed out in2004 when Mock Roles, Not Objects5 was published.

Classical TDD does not involve mocking or other forms of synthetic isolation bydefinition. We even use the term "classical TDD" to mean "TDD without isolation".

David used the word "classical" not because it’s correct, but because it implies "old".This is the beginning of a series of rhetorical techniques that he uses to incorrectlyassociate isolated unit testing with "oldness". He continues:

Connecting to external services like that would be too slow to get thefeedback cycle you need. That was probably true in 1997 when youwere connecting to a mainframe database or some Oracle abomination.

In 1997, TDD was only known to a small number of people. Mocks did not exist.Certainly no one was isolating unit tests. Why would David invoke that year? It’s adisingenuous rhetorical technique: by implying that a modern movement actuallyoccurred far in the past, he positions it as outdated, thereby discrediting it.

1 https://www.destroyallsoftware.com/blog/2014/tdd-straw-men-and-rhetoric2 http://david.heinemeierhansson.com/2014/slow-database-test-fallacy.html3 http://c2.com/cgi/wiki?TenYearsOfTestDrivenDevelopment4 http://xunitpatterns.com/EndoTesting.html5 http://jmock.org/oopsla2004.pdf

https://www.destroyallsoftware.com/blog/2014/tdd-straw-men-and-rhetoric

http://david.heinemeierhansson.com/2014/slow-database-test-fallacy.html

http://c2.com/cgi/wiki?TenYearsOfTestDrivenDevelopment

http://xunitpatterns.com/EndoTesting.html


http://jmock.org/oopsla2004.pdf

https://www.destroyallsoftware.com/blog/2014/tdd-straw-men-and-rhetoric

http://david.heinemeierhansson.com/2014/slow-database-test-fallacy.html

http://c2.com/cgi/wiki?TenYearsOfTestDrivenDevelopment


http://jmock.org/oopsla2004.pdf

TDD, Straw Men, and Rhetoric

172

The reality is that interest in sub-millisecond tests, and spending a lot of effort to getthem, is fairly new. In 2011, when I published the Fast Tests With and Without Rails6

screencast, almost no one was talking about doing it in Rails. At that point, if mymemory serves me, only Corey Haines and I were talking about it loudly in public.The history that David implies is completely false.

I’ll stop quoting in a moment, but we need one more. He now repeats a claim fromanother post: that TDD leads to worse design. He extends that idea, tying it to testspeed:

Inflicting such damage may well had been worth it back in the old dayswhen a full suite of tests hitting the database would have taken hours torun. It so much certainly is not when we’re talking about a few minutes.

Again, you see reinforcement of the falsehood with "the old days": that was "then",but today is "now"! However, with this passage, you finally get to see what’s reallygoing on. David’s tests run in a few minutes, and he’s fine with that.

I’m not fine with that. A lot of other people are not fine with that.

This is the fundamental issue. It’s possible that I’m the most impatient programmeron earth; I want my feedback to be so fast that I can’t think before it shows up. If Ican think, then I’ll sometimes lose attention, and I don’t want to lose attention.

I aim for test feedback in 300 ms, from the time I press enter to the time I have aresult. That’s not an abstract desire; I frequently achieve it. Here are the tests forDestroy All Software’s catalog class:

$ time rspec spec/lib/catalog_spec.rb......

Finished in 0.00723 seconds8 examples, 0 failures, 2 pending0.24s elapsed

That’s 105 lines of test covering 59 lines of production code running in 240milliseconds end to end, with only 7 of those milliseconds being actual tests. Thetest:code ratio here is slightly higher than DAS' overall ratio, which is about 1.4:1,including both the unit and acceptance test suites. Most of those tests are fully

6 https://www.destroyallsoftware.com/screencasts/catalog/fast-tests-with-and-without-rails

https://www.destroyallsoftware.com/screencasts/catalog/fast-tests-with-and-without-rails

https://www.destroyallsoftware.com/screencasts/catalog/fast-tests-with-and-without-rails


173

isolated, meaning that they only interact with a single class. Some tests use mocksto achieve that; many don’t need to use them.

These tests are fast enough that I can hit enter (my test-running keystroke) andhave a response before I have time to think. It means that the flow of my thoughtsnever breaks. If you’ve watched Destroy All Software screencasts7, you know thatI’ll sometimes run tests ten times per minute. All of my screencasts are recorded liveafter doing many takes to smooth the presentation out, so you’re seeing my actualspeed, not the result of editing. (They also cost money. I apologize for that, but thisis how I buy food and pay rent.)

Walking through a part of my TDD process will make this whole process moreconcrete. I first write a new test and run it by hitting enter. This runs only the currenttest file. It also sets that file as "active", meaning that hitting enter from anywherein the production code will re-run it.

The tests run in a tmux pane to the right of my editor, so they stick around even whileI’m editing code. I expect them to fail, since I just wrote a new hopefully-failing test.A quarter of a second after I hit enter, the tests have finished running. I flick my eyesto the output for less than a second; I know where the exception name will appear inthe traceback. Usually, I’ll see the exception name that I expect. Because the testsrun so fast, I literally do not have a chance to do anything before they finish, so I can’tget distracted. By the time my eyes have moved to the test pane, the output is there.

While my eyes were moving to the test pane to confirm that the correct exceptionwas raised, my fingers were switching to the production code file. I make whateverchange will cause the tests to pass. It’s usually a small change, which is the natureof TDD when done well. I then kick off another test by hitting enter. This runs thesame test file as before because it was implicitly set as the active test. I expect it topass, and it does, but I only know this because I see green in my peripheral vision;there’s no reason to actually focus my eyes on the test output.

I’ve seen my tests fail, made a production code change, and seen the tests pass. Afew seconds have elapsed since I ran the first failing test. You’ve spent an order ofmagnitude more time reading these few paragraphs about it. Unlike David, I’m notexaggerating here for rhetorical purposes: I’m literally talking about a single-digitnumber of seconds between running the failing test and seeing it pass.

7 https://www.destroyallsoftware.com/screencasts

https://www.destroyallsoftware.com/screencasts

https://www.destroyallsoftware.com/screencasts


174

My TDD loop isn’t always that tight, of course, but most of the second-to-secondsteps in building a software system are mundane and don’t require careful thought.If there’s a design question, I’ll stop and think, possibly aided by pen and paper. Ifthe tests fail in an unexpected way, I’ll stop and analyze the failure. For many tests,I don’t have to do either of these.

Staying under the 300 ms mark necessarily requires isolation. I don’t have timeto load Rails, or to load fixtures, or to connect to databases. The test runtime isdominated by RSpec startup, not the application. In fact, many of these tests loadonly a single source file.

That file is going to be a plain old Ruby class: not a Rails model, controller, etc.Isolating those would lead to pain, as anyone who’s tried to do it knows. I keep modelsvery simple and allow them to integrate with the database; I keep controllers verysimple and generally don’t unit test them at all, although I do integration test them.

When I’m at full speed, as I was in that story, I’m typing at 120 words perminute. That’s 10 keystrokes per second, with most of those keystrokes beingVim commands. I don’t leave home row; I don’t touch a mouse; I don’t stop andread; I don’t wait for tests to run. This process allows me to do the trivial parts ofprogramming as quickly as physically possible. Some people don’t believe this, butthere are 90 screencasts in the Destroy All Software catalog that document it.

David misses all of this. He’s never done it. It takes a lot of practice with TDD anda seriously fast editor that you can use with your eyes closed, but he has neither.(Again, no rhetorical exaggeration: when I’m at full speed, I sometimes close my eyesbecause removing visual stimulus aids my concentration.)

Unsurprisingly, there are other parts of David’s post that I disagree with. Mostnotably, application preloaders like Spring are band-aids that introduce tremendouscomplexity. They cause confusing failures that are fundamental to their nature. Notto mention his straw man attacks on the design feedback of TDD, and especiallyisolated TDD. I’ve focused on test speed here because it provides a nice microcosmof our disagreement and of his tactics. He hasn’t actually done the thing that he’sdecrying, and he’s arguing against it by making things up.

He’s been at this for a couple of weeks: if you watch his RailsConf keynote8, you’llhear him say that he has very little theoretical computer science knowledge. Thenhe pooh-poohs the value of that same theoretical knowledge: knowledge which

8 http://www.justin.tv/confreaks/b/522089408

http://www.justin.tv/confreaks/b/522089408

http://www.justin.tv/confreaks/b/522089408


175

he just said that he doesn’t have! It’s normal human behavior to form opinionsbefore understanding something. I certainly wouldn’t want to deny anyone theirhuman fallibility. However, to stand in front of thousands of eager followers that willbelieve anything you say, and to proclaim radical, outlandish positions, using made-up history as justification, on topics that you admit to not understanding… that justseems bad.

Finally, I should say that despite being one of the early proponents of sub-millisecondtests in Rails, I don’t have any kind of capital-B Belief in isolation, in mocking, or inTDD. I use TDD perhaps 75% percent of the time on web apps, and probably lessthan 50% of the time on console tools like Selecta9. I suspect that test doubles,including mocks, are a stop-gap solution: an intermediate evolutionary form. I’vetalked about this in conference talks10 and in screencasts11. Both of them containa lot of code. The screencast is a tour through an actual software system designedusing the scheme in question. Selecta12 is also designed using the same principles(although its design is imperfect in many ways, as designs tend to be).

The straw man that David has been propagating may apply to people who picked upTDD a year ago, and by whom it’s viewed as a silver bullet that should be applied inan extreme way. I was one of those people around 2006, so I know that they exist.There may be other, more experienced people who talk about it in an extreme wayfor rhetorical purposes. (I don’t know; I don’t read intro-to-TDD material, having doneit for almost ten years now.)

If (if!) those things are true, they say nothing about TDD as a technical practice.Despite writing prose so dripping with rhetoric, David doesn’t seem to separate therhetorical presentation of a topic from the facts of its use.

TDD is useful and test isolation is useful, but they both involve making trade-offs.Unfortunately, doing them 100% of the time seems to be the best way to learn whatthose trade-offs are, and that can temporarily lead beginners toward extremism.TDD and isolation both break down in some situations, and learning to detect thosesituations in advance takes a lot of time. This is true of advanced techniques in anydiscipline, programming or otherwise. That is the honest, non-exaggerated, no-lies-involved truth.

9 https://github.com/garybernhardt/selecta10 https://www.destroyallsoftware.com/talks/boundaries11 https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell12 https://github.com/garybernhardt/selecta

https://github.com/garybernhardt/selecta

https://www.destroyallsoftware.com/talks/boundaries

https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell



https://www.destroyallsoftware.com/talks/boundaries

https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell


176

Teach Yourself Programming in TenYearsWritten by: Peter Norvig1

Why is everyone in such a rush?

Walk into any bookstore, and you’ll see how to Teach Yourself Java in 24 Hoursalongside endless variations offering to teach C, SQL, Ruby, Algorithms, and so on ina few days or hours. The Amazon advanced search for [ title: teach, yourself, hours,since: 20002] and found 512 such books. Of the top ten, nine are programmingbooks (the other is about bookkeeping). Similar results come from replacing "teachyourself" with "learn" or "hours" with "days."

The conclusion is that either people are in a big rush to learn about programming, orthat programming is somehow fabulously easier to learn than anything else. Felleisenet al. give a nod to this trend in their book How to Design Programs3, when they say"Bad programming is easy. Idiots can learn it in 21 days, even if they are dummies."The Abtruse Goose comic also had their take4.

Let’s analyze what a title like Teach Yourself C++ in 24 Hours5 could mean:

• Teach Yourself: In 24 hours you won’t have time to write several significantprograms, and learn from your successes and failures with them. You won’t havetime to work with an experienced programmer and understand what it is like tolive in a C++ environment. In short, you won’t have time to learn much. So thebook can only be talking about a superficial familiarity, not a deep understanding.As Alexander Pope said, a little learning is a dangerous thing.

1 http://norvig.com/21-days.html2 http://www.amazon.com/gp/search/ref=sr_adv_b/?search-alias=stripbooks&unfiltered=1&field-keywords=&field-author=&field-title=teach+yourself+hours&field-isbn=&field-publisher=&node=&field-p_n_condition-type=&field-feature_browse-bin=&field-subject=&field-language=&field-dateop=After&field-datemod=&field-dateyear=2000&sort=relevanceexprank&Adv-Srch-Books-Submit.x=16&Adv-Srch-Books-Submit.y=53 http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html4 http://abstrusegoose.com/2495 http://www.amazon.com/Sams-Teach-Yourself-Hours-5th/dp/0672333317/ref=sr_1_6?s=books&ie=UTF8&qid=1412708443&sr=1-6&keywords=learn+c%2B%2B+days

http://norvig.com/21-days.html

http://www.amazon.com/gp/search/ref=sr_adv_b/?search-alias=stripbooks&unfiltered=1&field-keywords=&field-author=&field-title=teach+yourself+hours&field-isbn=&field-publisher=&node=&field-p_n_condition-type=&field-feature_browse-bin=&field-subject=&field-language=&field-dateop=After&field-datemod=&field-dateyear=2000&sort=relevanceexprank&Adv-Srch-Books-Submit.x=16&Adv-Srch-Books-Submit.y=5


http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html

http://abstrusegoose.com/249

http://www.amazon.com/Sams-Teach-Yourself-Hours-5th/dp/0672333317/ref=sr_1_6?s=books&ie=UTF8&qid=1412708443&sr=1-6&keywords=learn+c%2B%2B+days

http://norvig.com/21-days.html






http://www.ccs.neu.edu/home/matthias/HtDP2e/index.html

http://abstrusegoose.com/249



Teach Yourself Programming in Ten Years

177

• C++: In 24 hours you might be able to learn some of the syntax of C++ (if youalready know another language), but you couldn’t learn much about how to usethe language. In short, if you were, say, a Basic programmer, you could learn towrite programs in the style of Basic using C++ syntax, but you couldn’t learnwhat C++ is actually good (and bad) for. So what’s the point? Alan Perlis6 oncesaid: "A language that doesn’t affect the way you think about programming, isnot worth knowing". One possible point is that you have to learn a tiny bit of C++ (or more likely, something like JavaScript or Processing) because you need tointerface with an existing tool to accomplish a specific task. But then you’re notlearning how to program; you’re learning to accomplish that task.

• in 24 Hours: Unfortunately, this is not enough, as the next section shows.


Researchers ( Bloom (1985)7, Bryan & Harter (1899), Hayes (1989)8, Simmon& Chase (1973)) have shown it takes about ten years to develop expertisein any of a wide variety of areas, including chess playing, music composition,telegraph operation, painting, piano playing, swimming, tennis, and research inneuropsychology and topology. The key is deliberative practice: not just doing itagain and again, but challenging yourself with a task that is just beyond your currentability, trying it, analyzing your performance while and after doing it, and correctingany mistakes. Then repeat. And repeat again. There appear to be no real shortcuts:even Mozart, who was a musical prodigy at age 4, took 13 more years before hebegan to produce world-class music. In another genre, the Beatles seemed to burstonto the scene with a string of #1 hits and an appearance on the Ed Sullivan showin 1964. But they had been playing small clubs in Liverpool and Hamburg since1957, and while they had mass appeal early on, their first great critical success,Sgt. Peppers, was released in 1967. Malcolm Gladwell9 has popularized the idea,although he concentrates on 10,000 hours rather than 10 years.

It may be that 10,000 hours, not 10 years, is the magic number. Or it might be someother metric; Henri Cartier-Bresson (1908-2004) said "Your first 10,000 photographsare your worst." True expertise may take a lifetime: Samuel Johnson (1709-1784)said "Excellence in any department can be attained only by the labor of a lifetime;it is not to be purchased at a lesser price." And Chaucer (1340-1400) complained

6 http://www-pu.informatik.uni-tuebingen.de/users/klaeren/epigrams.html7 http://www.amazon.com/exec/obidos/ASIN/034531509X/8 http://www.amazon.com/exec/obidos/ASIN/08058030929 http://www.amazon.com/Outliers-Story-Success-Malcolm-Gladwell/dp/0316017922

http://www-pu.informatik.uni-tuebingen.de/users/klaeren/epigrams.html

http://www.amazon.com/exec/obidos/ASIN/034531509X/

http://www.amazon.com/exec/obidos/ASIN/0805803092

http://www.amazon.com/Outliers-Story-Success-Malcolm-Gladwell/dp/0316017922


http://www.amazon.com/exec/obidos/ASIN/034531509X/


http://www.amazon.com/Outliers-Story-Success-Malcolm-Gladwell/dp/0316017922


178

"the lyf so short, the craft so long to lerne." Hippocrates (c. 400BC) is known forthe excerpt "ars longa, vita brevis", which is part of the longer quotation "Ars longa,vita brevis, occasio praeceps, experimentum periculosum, iudicium difficile", whichin English renders as "Life is short, [the] craft long, opportunity fleeting, experimenttreacherous, judgment difficult." Of course, no single number can be the final answer:it doesn’t seem reasonable to assume that each of programming, chess playing,checkers playing, and music playing could all require exactly the same amount oftime to master, nor that all people will take exactly the same amount of time.

So You Want to be a Programmer

Here’s my recipe for programming success:

• Get interested in programming, and do some because it is fun. Make sure that itkeeps being enough fun so that you will be willing to put in your ten years/10,000hours.

• Program. The best kind of learning is learning by doing10. To put it moretechnically, "the maximal level of performance for individuals in a given domainis not attained automatically as a function of extended experience, but the levelof performance can be increased even by highly experienced individuals as aresult of deliberate efforts to improve." (p. 366)11 and "the most effective learningrequires a well-defined task with an appropriate difficulty level for the particularindividual, informative feedback, and opportunities for repetition and correctionsof errors." (p. 20-21) The book Cognition in Practice: Mind, Mathematics, andCulture in Everyday Life12 is an interesting reference for this viewpoint.

• Talk with other programmers; read other programs. This is more important thanany book or training course.

• If you want, put in four years at a college (or more at a graduate school). Thiswill give you access to some jobs that require credentials, and it will give you adeeper understanding of the field, but if you don’t enjoy school, you can (withsome dedication) get similar experience on your own or on the job. In any case,book learning alone won’t be enough. "Computer science education cannot makeanybody an expert programmer any more than studying brushes and pigmentcan make somebody an expert painter" says Eric Raymond, author of The New

10 http://www.engines4ed.org/hyperbook/nodes/NODE-120-pg.html11 http://www2.umassd.edu/swpi/DesignInCS/expertise.html12 http://www.amazon.com/exec/obidos/ASIN/0521357349

http://www.engines4ed.org/hyperbook/nodes/NODE-120-pg.html

http://www2.umassd.edu/swpi/DesignInCS/expertise.html



http://www.engines4ed.org/hyperbook/nodes/NODE-120-pg.html

http://www2.umassd.edu/swpi/DesignInCS/expertise.html



179

Hacker’s Dictionary. One of the best programmers I ever hired had only a HighSchool degree; he’s produced a lot of great13 software14, has his own newsgroup15, and made enough in stock options to buy his own nightclub16.

• Work on projects with other programmers. Be the best programmer on someprojects; be the worst on some others. When you’re the best, you get to test yourabilities to lead a project, and to inspire others with your vision. When you’re theworst, you learn what the masters do, and you learn what they don’t like to do(because they make you do it for them).

• Work on projects after other programmers. Understand a program written bysomeone else. See what it takes to understand and fix it when the originalprogrammers are not around. Think about how to design your programs to makeit easier for those who will maintain them after you.

• Learn at least a half dozen programming languages. Include one languagethat emphasizes class abstractions (like Java or C++), one that emphasizesfunctional abstraction (like Lisp or ML or Haskell), one that supports syntacticabstraction (like Lisp), one that supports declarative specifications (like Prolog orC++ templates), and one that emphasizes parallelism (like Clojure or Go).

• Remember that there is a "computer" in "computer science". Know how long ittakes your computer to execute an instruction, fetch a word from memory (withand without a cache miss), read consecutive words from disk, and seek to a newlocation on disk. (Answers here.)

• Get involved in a language standardization effort. It could be the ANSI C++committee, or it could be deciding if your local coding style will have 2 or 4space indentation levels. Either way, you learn about what other people like ina language, how deeply they feel so, and perhaps even a little about why theyfeel so.

• Have the good sense to get off the language standardization effort as quicklyas possible.

With all that in mind, its questionable how far you can get just by book learning.Before my first child was born, I read all the How To books, and still felt like a cluelessnovice. 30 Months later, when my second child was due, did I go back to the books for

13 http://www.xemacs.org/14 http://www.mozilla.org/15 http://groups.google.com/groups?q=alt.fan.jwz&meta=site%3Dgroups16 http://en.wikipedia.org/wiki/DNA_Lounge

http://www.xemacs.org/

http://www.mozilla.org/

http://groups.google.com/groups?q=alt.fan.jwz&meta=site%3Dgroups


http://en.wikipedia.org/wiki/DNA_Lounge

http://www.xemacs.org/

http://www.mozilla.org/


http://en.wikipedia.org/wiki/DNA_Lounge


180

a refresher? No. Instead, I relied on my personal experience, which turned out to befar more useful and reassuring to me than the thousands of pages written by experts.

Fred Brooks, in his essay No Silver Bullet17 identified a three-part plan for findinggreat software designers:

1. Systematically identify top designers as early as possible.

2. Assign a career mentor to be responsible for the development of the prospectand carefully keep a career file.

3. Provide opportunities for growing designers to interact and stimulate each other.

This assumes that some people already have the qualities necessary for being agreat designer; the job is to properly coax them along. Alan Perlis18 put it moresuccinctly: "Everyone can be taught to sculpt: Michelangelo would have had to betaught how not to. So it is with the great programmers". Perlis is saying that thegreats have some internal quality that transcends their training. But where does thequality come from? Is it innate? Or do they develop it through diligence? As AugusteGusteau (the fictional chef in Ratatouille) puts it, "anyone can cook, but only thefearless can be great." I think of it more as willingness to devote a large portion ofone’s life to deliberative practice. But maybe fearless is a way to summarize that.Or, as Gusteau’s critic, Anton Ego, says: "Not everyone can become a great artist,but a great artist can come from anywhere."

So go ahead and buy that Java/Ruby/Javascript/PHP book; you’ll probably get someuse out of it. But you won’t change your life, or your real overall expertise as aprogrammer in 24 hours or 21 days. How about working hard to continually improveover 24 months? Well, now you’re starting to get somewhere…

References

Bloom, Benjamin (ed.) Developing Talent in Young People19, Ballantine, 1985.

Brooks, Fred, No Silver Bullets20, IEEE Computer, vol. 20, no. 4, 1987, p. 10-19.

Bryan, W.L. & Harter, N. "Studies on the telegraphic language: The acquisition of ahierarchy of habits. Psychology Review, 1899, 8, 345-375

17 http://en.wikipedia.org/wiki/No_Silver_Bullet18 http://www-pu.informatik.uni-tuebingen.de/users/klaeren/epigrams.html19 http://www.amazon.com/exec/obidos/ASIN/034531509X20 http://citeseer.nj.nec.com/context/7718/0

http://en.wikipedia.org/wiki/No_Silver_Bullet


http://www.amazon.com/exec/obidos/ASIN/034531509X

http://citeseer.nj.nec.com/context/7718/0

http://en.wikipedia.org/wiki/No_Silver_Bullet


http://www.amazon.com/exec/obidos/ASIN/034531509X

http://citeseer.nj.nec.com/context/7718/0


181

Hayes, John R., Complete Problem Solver21 Lawrence Erlbaum, 1989.

Chase, William G. & Simon, Herbert A. "Perception in Chess"22 Cognitive Psychology,1973, 4, 55-81.

Lave, Jean, Cognition in Practice: Mind, Mathematics, and Culture in Everyday Life23,Cambridge University Press, 1988.

Answers

Approximate timing for various operations on a typical PC:

execute typical instruction 1/1,000,000,000 sec = 1 nanosec

fetch from L1 cache memory 0.5 nanosec

branch misprediction 5 nanosec

fetch from L2 cache memory 7 nanosec

Mutex lock/unlock 25 nanosec

fetch from main memory 100 nanosec

send 2K bytes over 1Gbps network 20,000 nanosec

read 1MB sequentially from memory 250,000 nanosec

fetch from new disk location (seek) 8,000,000 nanosec

read 1MB sequentially from disk 20,000,000 nanosec

send packet US to Europe and back 150 milliseconds = 150,000,000nanosec

Appendix: Language Choice

Several people have asked what programming language they should learn first.There is no one answer, but consider these points:

• Use your friends. When asked "what operating system should I use, Windows,Unix, or Mac?", my answer is usually: "use whatever your friends use." Theadvantage you get from learning from your friends will offset any intrinsicdifference between OS, or between programming languages. Also consider your

21 http://www.amazon.com/exec/obidos/ASIN/080580309222 http://books.google.com/books?id=dYPSHAAACAAJ&dq=%22perception+in+chess%22+simon&ei=z4PyR5iIAZnmtQPbyLyuDQ23 http://www.amazon.com/exec/obidos/ASIN/0521357349


http://books.google.com/books?id=dYPSHAAACAAJ&dq=%22perception+in+chess%22+simon&ei=z4PyR5iIAZnmtQPbyLyuDQ







182

future friends: the community of programmers that you will be a part of if youcontinue. Does your chosen language have a large growing community or a smalldying one? Are there books, web sites, and online forums to get answers from?Do you like the people in those forums?

• Keep it simple. Programming languages such as C++ and Java are designed forprofessional development by large teams of experienced programmers who areconcerned about the run-time efficiency of their code. As a result, these languageshave complicated parts designed for these circumstances. You’re concerned withlearning to program. You don’t need that complication. You want a language thatwas designed to be easy to learn and remember by a single new programmer.

• Play. Which way would you rather learn to play the piano: the normal, interactiveway, in which you hear each note as soon as you hit a key, or "batch" mode, inwhich you only hear the notes after you finish a whole song? Clearly, interactivemode makes learning easier for the piano, and also for programming. Insist on alanguage with an interactive mode and use it.

Given these criteria, my recommendations for a first programming language wouldbe Python24 or Scheme25. Another choice is Javascript, not because it is perfectlywell-designed for beginners, but because there are so many online tutorials for it,such as Khan Academy’s tutorial26. But your circumstances may vary, and there areother good choices. If your age is a single-digit, you might prefer Alice27 or Squeak28

or Blockly29 (older learners might also enjoy these). The important thing is that youchoose and get started.

Appendix: Books and Other Resources

Several people have asked what books and web pages they should learn from.I repeat that "book learning alone won’t be enough" but I can recommend thefollowing:

• Scheme: Structure and Interpretation of Computer Programs (Abelson &Sussman)30 is probably the best introduction to computer science, and it does

24 http://python.org/25 http://www.schemers.org/26 https://www.khanacademy.org/computing/cs/programming27 http://alice.org/28 http://www.squeak.org/29 https://blockly-demo.appspot.com/static/apps/index.html30 http://www.amazon.com/gp/product/0262011530

http://python.org/

http://www.schemers.org/

https://www.khanacademy.org/computing/cs/programming

http://alice.org/

http://www.squeak.org/

https://blockly-demo.appspot.com/static/apps/index.html

http://www.amazon.com/gp/product/0262011530


http://python.org/

http://www.schemers.org/

https://www.khanacademy.org/computing/cs/programming

http://alice.org/

http://www.squeak.org/

https://blockly-demo.appspot.com/static/apps/index.html



183

teach programming as a way of understanding the computer science. You can seeonline videos of lectures31 on this book, as well as the complete text online32.The book is challenging and will weed out some people who perhaps could besuccessful with another approach.

• Scheme: How to Design Programs (Felleisen et al.)33 is one of the best books onhow to actually design programs in an elegant and functional way.

• Python: Python Programming: An Intro to CS (Zelle)34 is a good introductionusing Python.

• Python: Several online tutorials35 are available at Python.org36.

• Oz: Concepts, Techniques, and Models of Computer Programming (Van Roy &Haridi)37 is seen by some as the modern-day successor to Abelson & Sussman.It is a tour through the big ideas of programming, covering a wider range thanAbelson & Sussman while being perhaps easier to read and follow. It uses alanguage, Oz, that is not widely known but serves as a basis for learning otherlanguages.

Notes

T\. Capey points out that the Complete Problem Solver38 page on Amazon now hasthe "Teach Yourself Bengali in 21 days" and "Teach Yourself Grammar and Style"books under the "Customers who shopped for this item also shopped for these items"section. I guess that a large portion of the people who look at that book are comingfrom this page. Thanks to Ross Cohen for help with Hippocrates.

31 http://www.swiss.ai.mit.edu/classes/6.001/abelson-sussman-lectures/32 http://mitpress.mit.edu/sicp/full-text/book/book.html33 http://www.amazon.com/gp/product/026206218634 http://www.amazon.com/gp/product/188790299635 http://wiki.python.org/moin/BeginnersGuide36 http://python.org/37 http://www.amazon.com/gp/product/026222069538 http://www.amazon.com/exec/obidos/ASIN/0805803092

http://www.swiss.ai.mit.edu/classes/6.001/abelson-sussman-lectures/

http://mitpress.mit.edu/sicp/full-text/book/book.html



http://wiki.python.org/moin/BeginnersGuide

http://python.org/




http://www.swiss.ai.mit.edu/classes/6.001/abelson-sussman-lectures/

http://mitpress.mit.edu/sicp/full-text/book/book.html



http://wiki.python.org/moin/BeginnersGuide

http://python.org/



184

Untested code is broken code: testautomation in enterprise softwaredeliveryWritten by: Martin Aspeli at October 17th, 20131

Test automation is a key technical practice of modern software development, yet itis sometimes viewed as an unnecessary luxury in enterprise software development.In this article, Martin Aspeli looks at the reasons why.

In 1975, Fred Brooks wrote The Mythical Man Month, arguably the only book you everneed to read about software development. In his essay “No Silver Bullet”, he wrote:“Software entities are more complex for their size than perhaps any other humanconstruct.” Even a moderately complex piece of software could be engineered in analmost infinite number of different ways, and no two developers will produce identicaloutputs for a non-trivial set of requirements. The upshot of all this: we are humanand we make mistakes. Our software contains defects and bugs.

To remove (or at least reduce) the bugs, we rely on testing. Testing takes two mainforms:

1. Validate that the software performs the functions expected by its users. This iswhat we refer to as external quality – have we built the right thing?

2. Checking the correctness of implementation of each component of a piece ofsoftware. For each possible input, confirm the expected output. We call internalquality – have we built it right?

The first type of testing is hugely important and cannot be automated. It should bedone all the time, with as representative a group of users as possible.

It is the second type of testing that should be automated as much as possible, fora number of reasons:

• It is a labour-intensive process that increases in effort exponentially as thecomplexity of the software under test grows.

1 http://www.deloittedigital.com/eu/blog/untested-code-is-broken-code-test-automation-in-enterprise-software-deliver

http://www.deloittedigital.com/eu/blog/untested-code-is-broken-code-test-automation-in-enterprise-software-deliver



Untested code is broken code: testautomation in enterprise software delivery

185

• It is highly repetitive – even boring. Consider a simple function that divides onenumber by another. To test that this performs as expected, we may want totest conditions where the numerator is negative, positive, zero, or a very largenumber; ditto the denominator, leading to 24 possible conditions that could betested. If manual test execution means clicking through a user interface to set upa condition where the code is executed to produce each of those, it could takemany minutes. Compound that to the thousands of operations performed in evenmoderately complex software, and you quickly run out of time.

• If we want to be Agile – to be responsive to genuinely business-driven change; toallow architecture and technical foundations to evolve as we learn more about aproject; to catch problems as early as possible to minimise the cost of remediation– we will want to test not just once at the end of a project, but every week or evenevery day. Lengthy test execution is profoundly limiting to agility.

• Lack of appropriate, timely testing inevitably leads to regressions. Regressionsseriously undermine business confidence in a software team and the ability topredict project completion. We find seemingly fundamental bugs, and we areterrified of what else may be broken.

• A complex, yet mundane process makes it likely that human error will occur intest execution, reducing the value of testing.

Test automation is actually a very simple idea: we take this labour-intensive,repetitive and largely predictable activity and we let a computer do as much of itas possible, letting the humans focus on the value-adding activities with greaterconfidence that the underlying solution is sound.

There are two main types of automated tests: unit tests and acceptance tests.

Unit tests

Automated unit tests assure internal quality. They exercise a single function or unitof a program. A very simple harness calls the unit under test with a range of inputsand expected outputs. Few or no additional components are involved, to make it aseasy as possible to trace a failure to its root cause.

Writing the test and the code it is testing goes hand-in hand, and should be doneby the same person. To borrow an analogy from “Uncle” Bob Martin: automatedunit tests are akin to double-entry bookkeeping. Just like you would never trustan accountant to make zero mistakes in a complex set of accounts without the


186

confidence provided by dual entry, you should never trust a programmer to writebug-free code based only on his or her own reading of that code.

A unit test is a very simple piece of code (“dumb tests are good tests”, as I like tosay) that describes the expected behaviour of a component for a single set of inputs.By describing the required logic in two different ways – one elegant, performant andfeatureful for production use, the other naïve, repetitive and easy to understand fortesting purposes – we radically improve the chances that the code that ends up inproduction is actually correct.

Unit tests also serve to inspire confidence. If we believe that the individual unitsof our code are highly likely to be correct, then we can more easily alter the waywe assemble those units into larger components and systems. We then test thoseintegrations, again using automation as far as possible, but here out of necessityavoiding the full combinatorial complexity of possible inputs and outputs. This isakin to having the confidence that every component that goes into a car has beenindividually quality assured before the car is assembled.

Finally, unit tests can be viewed as executable developer documentation. The testsuite explains, unambiguously, how the programmer intended a function to work. Ifin future we attempt to change the innards of that function, say to fix an escapeddefect or improve performance, we can run the test suite again and have confidencethat we did not accidentally break an edge case. Good programmers write specifictests for intentional conditions that may not be obvious in the future, for this veryreason.

There are three main ways to write unit tests:

1. Test-last: Write some code, then write some tests. This is how most inexperiencedunit testers begin. The problem is that it is easy to lose concentration or succumbto procrastination and not write as many or as useful tests as we would like.

2. Test-during: Write the code and tests at the same time, and use the test suite asthe starting point for debugging and detailed code analysis. This is how I codemost of the time.

3. Test-first: Write tests that describe the thing you intend to build. Verify that theyfail – you haven’t built it yet! Then write the unit under test, running the testsrepeatedly until they pass. This is the most disciplined, rigorous approach tounit testing and is known by its acronym TDD – Test Driven Development. Somepeople find it difficult to get into this habit, but those who do it well write someof the best code you will ever find.


187

Acceptance tests

Automated acceptance tests assure external quality: are we building the right thing?The practice of Acceptance Test Driven Development (ATDD, sometimes calledBehaviour Driven Development, or BDD) is based on the idea of writing detailedrequirements in the form of acceptance tests. The business warrants that when thosetests pass, the associated feature can be deemed complete.

In our projects, we typically write the acceptance test scenarios “just-in-time” as amedium-level requirement (a user story) is about to scheduled for development. Wethen ask the client to sign off these acceptance criteria, to remove any ambiguityabout what we are building. We have found that this not only increases our sharedunderstanding of requirements, it also leads to radically better requirements andsaves us from having to define separate, possibly inconsistent test cases later.

It is not strictly necessary to automate acceptance tests, but it is a very good idea. Wecreated a tool called CoreJet specifically to support our workflow of writing, signingoff, automating and reporting against these acceptance tests. It presents a two-dimensional visualisation of the project that shows the amount of scope implementedand covered by passing automated acceptance tests. The more “green”, the closerwe are to completion.

Automated acceptance tests do not replace unit tests, and typically test key businessscenarios, but not every combination of inputs and outputs. They often involvescripting a web browser or other user interface, and can be significantly slower torun than lightweight unit tests. The two approaches are very much complementary.

Acceptance test automation serves another purpose as well: they make suredevelopers actually read the specification. Using a tool like CoreJet, the exact text ofthe most detailed expression of a requirement is right there in the developer’s lineof sight, woven into the code of a test suite. There is no separate document to readand understand and no context switching. We are more likely to build the right thingthe first time around.

Continuous integration

Test automation underpins another key agile practice: continuous integration. Itworks like this: a server monitors the source code repository for changes. As soonas something is changed, the full suite of automated tests is run. If they all pass:great! If not, a warning sign goes on (quite literally, in our office, where we’ve rigged


188

a red light to the CI server), and the exact change that caused a test to regress ishighlighted. The developer responsible is made to fix it, buy coffee for the team andoptionally wear a funny hat.

Continuous integration is all about fail fast. The cost of fixing a defect incurredminutes ago is miniscule compared to the cost of fixing it six months down theline after thousands of code changes. We all make mistakes, but we can redeemourselves quickly by fixing them immediately, whilst the errant code is still fresh inour minds.

Good testing discipline

Mature development teams adopt a great deal of discipline around their automatedtesting:

• Monitor and report on test coverage (there are tools that can report on thepercentage of non-test code exercised during a test run). Somewhere north of75% is considered good. However, this is merely one indicator. It is possible tohave high coverage with poor tests and vice-a-versa, so be pragmatic.

• Review the coverage and correctness of tests as part of manual code reviews.

• Always run the tests locally before checking in new code to detect regressions.Never “break the build” for others on the team.

• If you do break the build, don’t go home without fixing it.

For the best developers I know, great automated testing is an obsession, and a sourceof genuine pride.

But…

Alas, not everyone is sold. Below are some of the excuses we’ve heard for not doingtesting, in increasing order of validity.

We have a team of manual testers. They do the testingfor us.

As outlined, intelligent quality assurance by humans is hugely important. Manual andautomated testing are complementary. However, the “somebody else’s problem”mentality simply doesn’t fly. Each developer should feel accountable for the codethey write being correct. Writing something half-baked and waiting for someone else


189

to find problems is a recipe for messy code and long delays between incurring adefect and detecting it. This leads to significantly higher overall costs.

We’re on a deadline. We don’t have time to write tests.

Fear of failure can sometimes create perverse incentives. We think it is better tomeet a deadline with something that doesn’t work, than to miss it by a week, buthave something that does.

The answer to this is simple: do less, to a consistently high level of quality. Alooming deadline that looks unachievable is usually a systemic failure to provide theenvironment in which the team has enough capacity and realistic goals. Negotiatescope down and don’t attempt to start features that won’t be complete in time.Having regular releases (i.e. short iterations in a Scrum team) and making sure youwork on high priority features first makes this significantly easier.

We charge by the hour. The client won’t pay for tests.

Good test automation discipline has the potential to significantly reduce overall effortexpended on a software development project.

The constraining factor of a developer’s productivity is certainly not the rate at whichhe or she can type out code. Measures based on lines of code are largely useless,but even quite productive developers may not produce more than, say, 100 lines ofnon-generated code, net, in a day (the best developers tend to remove a lot of codeby replacing it with more elegant constructs). Typing that out flat would easily takeless than half an hour.

The challenge, of course, is producing the right 100 lines of code. Good automatedunit tests tend to lead to cleaner, better-factored code, because well-factored codeis easier to test. The act of writing a suite of tests tend to help the developer exploreand understand how a piece of logic should behave to be correct in a way that simplystaring at the screen does not. Simply put, test automation accelerates the problem-solving thought process that is software development.

Finally, there is “death by a thousand cuts” impact to productivity of manual testing.Writing a simple component may take a few minutes. To check that it works, youstart up an application server and click through the user interface to exercise thecode. This may take another minute or two. You find a problem. Go back and changethe code, restart the server, click through the user interface again. Repeat. The firstcouple of times, the time to write a decent automated unit test may be longer than


190

the time to click through the user interface. Once you have a test, though, creatingvariants of it to test boundary conditions is normally just copy-and-paste with minoredits. The test should run in less than a second. Economies of scale kick in.

Now consider a situation where you want to improve or refactor a fundamental,shared component, with potentially open-ended consequences. A fully manualregression test of an application to have confidence that the change hasn’t brokensomething could take hours. It is simply unsustainable.

I’m a great developer. I don’t need to test my code.

The best developers in the world are acutely aware of the inevitability that theywill introduce defects. That is probably why they are so passionate about testautomation. That said, if you know of a programmer who writes perfect code, tellthem we want to offer them a job.

The tests take too long to run.

As a project grows, it is not uncommon to have thousands of unit tests and hundredsof acceptance tests, if not more. Sometimes these can take a long time to run. Itis prudent to monitor test execution times. If it takes longer than a few minutes torun automated unit tests and longer than ten minutes to run automated acceptancetests, it is worth investing in ways to speed up the tests. Alternatively, it may benecessary to take a risk-based approach and partition the tests so that some tests(i.e. those at greater risk of failing) are run more frequently than others. At the veryleast, however, you should run all the tests once per day.

We don’t know how to write test.

Test automation is a skill. Just as a developer needs to learn the prevailing languagesand frameworks, he or she should learn the test tools used on a project. For peoplewho often work with cutting-edge technology, programmers can be surprisinglyadverse to learning new things. Have none of it, but invest in training and mentoringwhere required.

Our solution doesn’t support test automation.

This is probably the only valid reason in this whole list. Some software, especiallyolder software, can be difficult to automate, for example due to excessivedependencies between modules or poor abstraction. It may take significantinvestment to make custom code testable.


191

However, there are libraries and tools that can reduce this burden. Mockingframeworks can be used to stub out interfaces, even deep into a third partycomponent. Tools like Selenium can be used to automate at the user interface levelfor web applications. It may not be possible to test perfectly, but even a small numberof tests are infinitely better than none.

One might also suggest that ability to support test automation should be a keydeciding factor when choosing technologies to adopt.

What shouldn’t we test?

Test automation is not simply a case of “more is more”. There are some things thatdon’t make sense to test. For example:

• The layout of a user interface. This is much better judged by a human being thana computer.

• Tautologies, e.g. checking that a constant equals its value.

• Excessively repeated tests, e.g. checking that a shared component behaves thesame in every single instance where it is used. Instead, test the component as asingle unit, and then test the other units that rely on it with the assumption thatthe shared component works.

• Externalised configuration that could change between environments.

• Automatically generated code, e.g. “getters” and “setters” in Java.

• Things unrelated to your code. It is sometimes easy to get a little carried away andtest, for example, that a database knows how to execute a simple query, or thata web browser can render HTML. It’s probably safe to assume those things work.

• Conditions that depend on specific data in an external system that cannot becontrolled as part of the test fixture. The problem with this is that the test maynot be easily repeatable across environments.

• Conditions that depend on dates. I call these New Year’s Regressions, as there isusually some poorly written, date-dependent tests that fail once the system clockticks over into another year.

Conclusion

I’m going to end this article with a bold statement: Untested code is broken code.We simply should not accept software solutions that do not have a decent number


192

of appropriate, passing automated tests. It is one of the first things I look for when Ireview a software project, and one of the best indicators of project quality.

193

The Three Laws of TDD.Written by: Robert Martin at Oct 20051

Over the years I have come to describe Test Driven Development in terms of threesimple rules. They are:

1. You are not allowed to write any production code unless it is to make a failingunit test pass.

2. You are not allowed to write any more of a unit test than is sufficient to fail; andcompilation failures are failures.

3. You are not allowed to write any more production code than is sufficient to passthe one failing unit test.

You must begin by writing a unit test for the functionality that you intend to write.But by rule 2, you can’t write very much of that unit test. As soon as the unit testcode fails to compile, or fails an assertion, you must stop and write production code.But by rule 3 you can only write the production code that makes the test compileor pass, and no more.

If you think about this you will realize that you simply cannot write very much codeat all without compiling and executing something. Indeed, this is really the point. Ineverything we do, whether writing tests, writing production code, or refactoring, wekeep the system executing at all times. The time between running tests is on theorder of seconds, or minutes. Even 10 minutes is too long.

Too see this in operation, take a look at The Bowling Game Kata2.

Now most programmers, when they first hear about this technique, think: "This isstupid!" "It’s going to slow me down, it’s a waste of time and effort, It will keep mefrom thinking, it will keep me from designing, it will just break my flow." However,think about what would happen if you walked in a room full of people working thisway. Pick any random person at any random time. A minute ago, all their codeworked.

Let me repeat that: A minute ago all their code worked! And it doesn’t matter who youpick, and it doesn’t matter when you pick. A minute ago all their code worked!

1 http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd2 http://butunclebob.com/ArticleS.UncleBob.TheBowlingGameKata

http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd

http://butunclebob.com/ArticleS.UncleBob.TheBowlingGameKata

http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd

http://butunclebob.com/ArticleS.UncleBob.TheBowlingGameKata

The Three Laws of TDD.

194

If all your code works every minute, how often will you use a debugger? Answer,not very often. It’s easier to simply hit ^Z a bunch of times to get the code backto a working state, and then try to write the last minutes worth again. And if youaren’t debugging very much, how much time will you be saving? How much time doyou spend debugging now? How much time do you spend fixing bugs once you’vedebugged them? What if you could decrease that time by a significant fraction?

But the benefit goes far beyond that. If you work this way, then every hour you areproducing several tests. Every day dozens of tests. Every month hundreds of tests.Over the course of a year you will write thousands of tests. You can keep all thesetests and run them any time you like! When would you run them? All the time! Anytime you made any kind of change at all!

Why don’t we clean up code that we know is messy? We’re afraid we’ll break it.But if we have the tests, we can be reasonably sure that the code is not broken, orthat we’ll detect the breakage immediately. If we have the tests we become fearlessabout making changes. If we see messy code, or an unclean structure, we can cleanit without fear. Because of the tests, the code becomes malleable again. Because ofthe tests, software becomes soft again.

But the benefits go beyond that. If you want to know how to call a certain API, thereis a test that does it. If you want to know how to create a certain object, there isa test that does it. Anything you want to know about the existing system, there isa test that demonstrates it. The tests are like little design documents, little codingexamples, that describe how the system works and how to use it.

Have you ever integrated a third party library into your project? You got a big manualfull of nice documentation. At the end there was a thin appendix of examples. Whichof the two did you read? The examples of course! That’s what the unit tests are!They are the most useful part of the documentation. They are the living examples ofhow to use the code. They are design documents that are hideously detailed, utterlyunambiguous, so formal that they execute, and they cannot get out of sync with theproduction code.

But the benefits go beyond that. If you have ever tried to add unit tests to a systemthat was already working, you probably found that it wasn’t much fun. You likelyfound that you either had to change portions of the design of the system, or cheaton the tests; because the system you were trying to write tests for was not designedto be testable. For example, you’d like to test some function f . However, f callsanother function that deletes a record from the database. In your test, you don’t

The Three Laws of TDD.

195

want the record deleted, but you don’t have any way to stop it. The system wasn’tdesigned to be tested.

When you follow the three rules of TDD, all your code will be testable by definition!And another word for "testable" is "decoupled". In order to test a module in isolation,you must decouple it. So TDD forces you to decouple modules. Indeed, if you followthe three rules, you will find yourself doing much more decoupling than you may beused to. This forces you to create better, less coupled, designs.

Given all these benfits, these stupid little rules of TDD might not actually be so stupid.They might actually be something fundemental, something profound. Indeed, I hadbeen a programmer for nearly thirty years before I was introduced to TDD. I did notthink anyone could teach me a low level programming practice that would make adifference. Thirty years is a lot of experience after all. But when I started to use TDD,I was dumbfounded at the effectiveness of the technique. I was also hooked. I can nolonger concieve of typing in a big long batch of code hoping it works. I can no longertolerate ripping a set of modules apart, hoping to reassemble them and get them allworking by next Friday. Every decision I make while programming is driven by thebasic need to be executing again a minute from now.

196

ColophonAll articles were manually converted from HTML to AsciiDoc3 and the final workwas built into the single electronic publication in PDF format using AsciiDoctor4 andDocBook5 toolchains.

3 http://asciidoctor.org/docs/asciidoc-writers-guide/4 http://asciidoctor.org/5 http://www.docbook.org/

http://asciidoctor.org/docs/asciidoc-writers-guide/

http://asciidoctor.org/

http://www.docbook.org/

http://asciidoctor.org/docs/asciidoc-writers-guide/

http://asciidoctor.org/

http://www.docbook.org/

the mozaic

Documents

broken code

extreme programming

programming sucks

untested code

software developer

surviving legacy code

enterprise software

software engineering