continuous automated testing - cast conference workshop august 2014
DESCRIPTION
CAST 2014 New York: The Art and Science of Testing The Association for Software Testing www.associationforsoftwaretesting.org COURSE DESCRIPTION Automated tools provide test professionals with the capability to make relevant observations even in the fastest-paced environments. Automated testing is also a powerful tool for improving communication between software engineers. This is important because good communication is a prerequisite for growing a great software engineering organization. This workshop will explore the continuous testing of software systems. Special focus will be given to the situation where the engineering team is deploying code to production so frequently that it is not possible to perform deep regression testing before each release. People who participate in this course will learn pragmatic automated testing strategies like: * Data analysis on the command line with find, grep and wc. * Network analysis with Chrome Inspector, Charles and netcat. * Using code churn to predict hotspots where bugs may occur. * Putting stack traces in context with automated SCM blame emails. * Using statsd to instrument a whole application. * Testing in production. * Monitoring-as-testing. Technical level: participants should have some familiarity with the command line and with editing code using a text editor or IDE. Familiarity with Git, SVN or another version control system is helpful but not required. Likewise some knowledge of Web servers is helpful but not required. It is desirable for participants to bring laptops. BIO From 2010 to 2012 Noah was a Test Architect at Etsy. He helped build Etsy's continuous integration system, and has helped countless other engineers develop successful automated testing strategies.These days Noah is an independent consultant in New York. He is passionate about helping engineers understand and use automated tools as they work to scale their applications more effectively.TRANSCRIPT
Noah Sussman and Joanna Burgess! ! CAST NYC, August 11, 2014
Continuous Automated Testing: A Communication System That Scales!
Jared Tarbell, Substrate
Continuous Automated Testing (C.A.T.)
Schedule
✤ 9:00-10:30: CAT!
✤ 10:30-10:50: Break !
✤ 10:50-12:30: CAT!
✤ 12:30-1:30: Lunch!
✤ 1:30-3:00: CAT!
✤ 3:00-3:30: Snack!
✤ 3:30-5:00: CAT
Load it Up!
✤ Jenkins http://jenkins-ci.org/ !
✤ Golly http://golly.sourceforge.net/
“When I use a word it means just what I choose it to mean-neither more nor less.”
What does QA mean to you?
Safety is not about the breakage or lack of quality of single components. Managing quality is about single components, about seeing how they meet particular specifications.
!
—Sidney DekkerGuy Sie on Flickr
In other words…when are we, and when aren’t we speaking the same language? !
More to the point…how much does it matter?
Rolling the Dice
Contested Terminology
✤ Quality Assurance!
✤ Testing!
✤ Manual Testing !
✤ Bugs!
✤ Accountability!
✤ QA Environment !
4REALZ: “When I use a word it means just what I choose it to mean” (ymmv)
✤ Like Humpty Dumpty, I am using contested terminology to support my own viewpoints!
✤ When I use a word, it means what I intend it to mean!
✤ If you are unsure what I mean, raise your hand!!!
ETTO
✤ Efficiency-to-thoroughness-trade-offs (ETTO)!
✤ We do necessarily not pick the best option, we pick the option that best satisfies our immediate needs namely:!
✤ efficiency!
✤ thoroughness
Automated Tests Are…
✤ Part of the software design process
✤ Runnable documentation
✤ NOT useful for debugging releases
✤ NOT useful for catching bugs in code
Tractable Systems
A system is tractable if the principles of the functioning are known and have simple descriptions with few details. Tractable systems do not change while being described.!!
Intractable SystemsIntractable systems are complex systems. A defining feature of intractable systems is that they are unsafe and defy documentation.
Defining Terminology
✤ Abstraction!
✤ Orders of Magnitude Growth !
✤ Cellular Automaton!
✤ Finite Automaton!
✤ Halting Problem !
✤ CAP Theorem !
✤ Acyclic Directed Graphs !
✤ Path reduction
Core Concepts
✤ Ironies of automation !
✤ Hawthorne Effects!
✤ Goodhart’s Law!
✤ Diseconomies of scale
Such dev. Very ops. Wow.
devops: where is the QA?
✤ STOP ASKING ME where QA fits into devops.!
✤ ROFLMAO if QA and testing aren’t ALREADY part of dev for you!!
✤ The idea that QA is distinct from development is a convenient fiction.!
✤ QA and testing have always been software engineering disciplines.
Sufficiently Advanced Monitoring is Indistinguishable from Testing
✤ A Google engineer proposed this idea in 2007.!
✤ Nothing much has changed since then.!
✤ Sufficiently advanced technology is indistinguishable from magic. —Arthur C. Clarke!
✤ Sufficiently advanced technology is indistinguishable from a rigged demo. —Silicon Valley aphorism
Monitoring vs. Testing
✤ Yes you should monitor all the things.!
✤ Yes you should build real time dashboards.!
✤ Yes you should deploy continuously.!
✤ Yes you should fix production bugs on the fly.!
✤ No, none of the above replaces QA and testing.!
✤ Really, no.
QA and Testing are part of the software design process
✤ Monitoring has nothing to do with design.!
✤ Monitoring provides visibility into implementation.!
✤ QA and testing address design flaws.!
✤ That is, QA and testing are design tools.
Each Necessary but Only Jointly Sufficient
✤Monitoring is part of the picture of overall system safety
✤QA and testing are part of the picture of overall system safety
✤Manual and automated testing — part of the overall picture
✤Developers following standards — part of the picture ✤People cooperating — part of the picture ✤The system safety picture is intrinsically incomplete!
Limiting Risk
Nancy Leveson on Limiting Risk:!!✤ Oversight!
!✤ Limit complexity!!✤ Systems thinking
Apply the best software engineering principles… quality assurance, testing… the highest standards. It's not going to be enough. —Nancy Leveson
Guy Sie on Flickr
Law of Leaky Abstractions
“…tools which pretend to abstract out something, like all abstractions, leak, and the only way to deal with the leaks competently is to learn about how the abstractions work and what they are abstracting. So the abstractions save us time working, but they don't save us time learning.” !
—Joel Spolsky
Failure is Scary But Also Inevitable
Everyone—even cats—must deal with failure.
Leaky Abstractions in Software
✤ How does this relate to QA? ✤ in the new view of systems safety, safety is
derived from being able to predict the behavior of the system.
✤ Since abstractions leak, no general abstraction can ever provide granular insight into system behavior.
The Mechanistic Fallacy: A Leaky Abstraction Pushed Too Far✤ Systems Thinking!
✤ Normal Failure!
✤ ETTO (Efficiency to Thoroughness Tradeoff)!
✤ Intractable Systems!
✤ The New View of system safety
Complex Systems are Intractable
—Erik Hollnagel
“You can take a bicycle apart and put it back together…you cannot do that with a frog.” —Liz Keogh
Bicycle vs. Frog
How is a Software System Like a Frog?
✤ It’s a running system.!
✤ It’s changing all the time.!
✤ Conceptually can’t be decomposed into discrete components.!
✤ A frog is intractable.
Let’s compare a bicycle to the universe
✤ How complex is a bicycle?
Bicycles and universes are not on the same scale of complexity
✤ About 30 parts ~ 1/30 = 0.03
✤ How many components does a program have?!
✤ Consider the range of computational “parts” that exist between 1 microsecond and a half hour of computational time [Edsger Dijkstra]!
✤ 0.001 / 1800 = 5.55-7!
✤ Notice how there’s an exponent this time?
Now let’s compare a the universe to a computer program
Computers and universes are on the same scale of complexity
What does n-7 levels of hierarchy look like?
The Pesticide Paradox and the Complexity Barrier
“Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.”!
“Software complexity (and therefore that of bugs) grows to the limits of our ability to manage that complexity.”!
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! —Boris Beizer
The Pesticide Paradox✤ Roaches have decided to terraform your apartment to suit
their needs. Bug terminator summoned; pesticides released. !
✤ Dead roaches. Problem solved or another problem created? !
✤ An unintended consequence: the more efficient the pesticide is the greater the chance that the genes for pesticide resistance will dominate the gene pool going forward. !
✤ This killing “all the bugs” actually results, over the long term, in large generations of bugs that will not be so easy to kill.
The Complexity Barrier
✤ There is an upper limit to how much complexity the human brain can handle. !
✤ Simplifying an existing system is a very difficult problem.!
✤ There are no general solutions for simplification of systems.
!!
!
✤ Simplicity and design are equally important until you have to pick one; then pick simplicity. !
!
✤ Modern business practices assume that there is no functional upper limit on system complexity. !
!
MIT vs. New Jersey! a.k.a “worse is better”
Want to Reduce Cost?
Communicate!
Conway’s Game of Life
1. Survivals. Every counter with two or three neighboring counters survives for the next generation.
2. Deaths. Each counter with four or more neighbors dies (is removed) from overpopulation. Every counter with one neighbor or none dies from isolation.
3. Births. Each empty cell adjacent to exactly three neighbors--no more, no fewer--is a birth cell. A counter is placed on it at the next move.
Delightfully Simple Rules
! ! -Martin Gardner
In real systems the rules aren’t as delightfully simple.
Jonathan Borofsky, Counting 1 to 3227146, 1969/1976!
Jenkins lab (frequently-running job)
1. We will need test data. So we will create two jobs: a frequently running job and a long running job!
2. Set up a Jenkins instance (download the installer from jenkinsci.org)!
3. Go to http://localhost: 8080/!
4. Click “new item”!
5. Type a name for your frequently-running project!
6. Click “build a freestyle software project” and click “OK”!
7. Create a new build step and type the code for your frequently running project !
8. Schedule the job to run once a minute (using the Cron syntax as described in the inline help)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
A Review of My Previously Published Approach to Jenkins API Automation
Jenkins User Conference New York, May 17 2012 #jenkinsconf
The Jenkins JSON API
To read the documentation, go to http://ci.example.com/api
You can append /api/json to the end of nearly any Jenkins URL to get JSON data. http://ci.example.com/api/json for latest builds http://ci.example.com/job/unit-tests/api/json for history of a specific build. http://ci.example.com/computer/api/json for slave information.
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Using depth= to get more granular data
If the API response doesn't contain some data that you expected, try appending ?depth=1 to the URL. If you still don't get what you want, increase the integer value. Usually you'll keep getting more data up until around ?depth=5 Exactly what and how much data you'll get is dependent on the configuration of your Jenkins instance.
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Drawbacks of using depth=
Depending on how deep you go into the API response, you can wind up with a lot of data. Such large responses can be expensive to download. In some cases you can request a response so large that you will wind up DDoSing Jenkins!
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Using tree= to filter the API response
The tree= URL parameter is like a SQL query.
Use depth= to look at the wealth of information available. Then use tree= to select only the information you actually need. This can dramatically reduce the size of your API responses.
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Jenkins User Conference New York, May 17 2012 #jenkinsconf
?tree=busyExecutors
Jenkins User Conference New York, May 17 2012 #jenkinsconf
?tree=computer[displayName]
Suggested Reading✤ Drive: The Surprising Truth About What Motivates Us by Dan Pink with RSAnimate https://www.youtube.com/watch?v=u6XAPnuFjJc!
✤ Ironies of Automation by Lisanne Bainbridge http://www.ise.ncsu.edu/nsf_itr/794B/papers/Bainbridge_1983_Automatica.pdf!
✤ Big Ball of Mud by Brian Foote and Joseph Yoder http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.316.433&rep=rep1&type=pdf!
✤ The Rise of Worse is Better by Richard Gabriel http://www.jwz.org/doc/worse-is-better.html!
✤ Worse is Better by Richard Gabriel http://dreamsongs.com/worseisbetter.html!
✤ To Err is Human: ETTO Principle by Erik Hollnagel http://www.namahn.com/we-share/interviews/erik-hollnagel-err-human-etto-principle!
✤ The Law of Leaky Abstractions by Joel Spolsky http://www.joelonsoftware.com/articles/LeakyAbstractions.html!
✤ Designers and Women in Open Source by Vitorio Miliano http://old.vi.to/designers-and-women-in-open-source.html!
✤ The fantastic combinations of John Conway’s new solitaire game “life” by Martin Gardner http://www.ibiblio.org/lifepatterns/october1970.html!
Books!✤ The Field Guide to Understanding Human Error by Sidney Dekker http://amzn.to/1mdwNdU!
✤ How Google Tests Software by James Whittaker http://amzn.to/XvMtUl!
✤ Software Testing Techniques (2nd edition) by Boris Beizer http://amzn.to/YqtevB!
✤ The Timeless Way of Building by Christopher Alexander http://amzn.to/1tgW3UJ!