token testing slides

Token Testing TalkEric Holscher

http://ericholscher.comDjangocon 2009

0

2.5

5.0

7.5

10.0

Pycon RailsConf DjangoCon

Testing Talks per Conference

How do you know?

» First 4 months of my job was porting and testing Ellington

» Going from Django r1290 to Django 1.0.

» Suite from 0 to 400 tests. (Now has 1715)

30,000 Ft View

»State of testing in Django

»Why you should be testing

»How you start testing

»Useful tools

»Eventual Goals

State of Django Testing

assertTrue('Hello World', community.testing.status)

Django 1.1

Making Testing Possible since 2009

manage.py startapp creates a tests.py

Basic Test

from django.test import TestCase

class SimpleTest(TestCase): def test_basic_addition(self): """ Tests that 1 + 1 always equals 2. """ self.failUnlessEqual(1 + 1, 2)

__test__ = {"doctest": """Another way to test that 1 + 1 is equal to 2.

>>> 1 + 1 == 2True"""}

Fast Tests(Transactions)

0

15

30

45

60

Django 1.0 Django 1.1

Ellington Test Speedup

Minutes (Lower is better)

You now have no excuse.

Why to test

Scary

Less Scary

Not Scary

Peace of Mind

Code must adapt

“It is not the strongest of the species that survives, nor the most intelligent, but the

one most responsive to change.

- Charles Darwin

Won’t somebody please think of the users?!

Tests as Documentation

Tests as Documentation

Test driven development +

Document driven development=

Test Driven Documentation

“Code without tests is broken as designed

- Jacob Kaplan-Moss

Dizzying Array of Testing Options

What kind of test?

» doctest

» unittest

Doctests

» Inline documentation

» <Copy from terminal to test file>

» Easy

Awesome Documentation

def parse_ttag(token, required_tags): """ A function to parse a template tag.

It sets the name of the tag to 'tag_name' in the hash returned.

>>> from test_utils.templatetags.utils import parse_ttag >>> parse_ttag('super_cool_tag for my_object as obj', ['as']) {'tag_name': u'super_cool_tag', u'as': u'obj'} >>> parse_ttag('super_cool_tag for my_object as obj', ['as', 'for']) {'tag_name': u'super_cool_tag', u'as': u'obj', u'for': u'my_object'}

""" bits = token.split(' ') tags = {'tag_name': bits.pop(0)} for index, bit in enumerate(bits): bit = bit.strip() if bit in required_tags: if len(bits) != index-1: tags[bit] = bits[index+1] return tags

Awesome Documentation

>>> from test_utils.templatetags.utils import parse_ttag >>> parse_ttag('super_cool_tag for my_object as obj', ['as']) {'tag_name': u'super_cool_tag', u'as': u'obj'} >>> parse_ttag('super_sweet for object as obj', ['as', 'for']) {'tag_name': u'super_sweet', u'as': u'obj', u'for': u'object'}

Doctest problems

» Can’t use PDB

» Hide real failures

Unit Tests

» Standard (XUnit)

» More robust

» setUp and tearDown

Basic Unit Test

import randomimport unittest

class TestRandom(unittest.TestCase):

def setUp(self): self.seq = range(10)

def testshuffle(self): # shuffled sequence does not lose any elements random.shuffle(self.seq) self.seq.sort() self.assertEqual(self.seq, range(10))

if __name__ == '__main__': unittest.main()

Django TestCase

» Subclasses unittest

» Fixtures

» Assertions

» Mail

» URLs

Test Client

» Test HTTP Requests without server

» Test Views, Templates, and Context

Django’s TestCase

from django.contrib.auth.models import Userfrom django.test import TestCasefrom django.core import mail

class PasswordResetTest(TestCase): fixtures = ['authtestdata.json'] urls = 'django.contrib.auth.urls'

def test_email_not_found(self): "Error is raised if the provided email address isn't currently registered" response = self.client.post('/password_reset/', {'email': 'not_a_real_email@email.com'}) self.assertEquals(len(mail.outbox), 0)

What flavor of test?

» Unit

» Functional

» Browser

Unit test

» Low level tests

» Small, focused, exercising one bit of functionality

» Great for libraries

Regression test

» Written when you find a bug

» Proves bug was fixed

» Django Tickets

Functional

» ‘Black Box Testing’

» Check High Level Functionality

Functional Testing Tools

» Twill

» Django Test Client

Browser tests

» Run tests in a web browser

» Check compatibility of design

» Basically an IE sanity check

» Only real way to test JS, AJAX, CSS

» Slow

Browser Testing Tools

» Windmill

» Selenium

Other kinds of testing

» Spiders

» Fuzz testing

» Load testing

Where do I start?

Use unittest unless you have a reason not to!

Start with a regression test or a functional test

Fixed a bug

Poking at code on the command line

Pony turned Horse

Use the data, Luke

» Use data as a pivot

» Fixtures means you use Unit Tests

» Creation on the command line, Doctests

Creating Fixtures

»./manage.py dumpdata <app>

»./manage.py makefixture Model[x:y]

»Follows relations

»Slicing

»By Hand

Getting in Context

» ./manage.py testshell <fixture>

» (i)pdb

Making Functional tests

» Usually a relatively annoying process

» Testmaker makes it easy.

» ./manage.py testmaker [app]

» Simply browse and your session is recorded.

Testing your views gets you the most coverage.

80% Case

When > Where

Enough to be useful

Tools

Summer of Code

» Test-Only Models

» Skipping tests

» Coverage

» Windmill tests of the admin

Test-Only Models

» Allow for models available only during tests

» Don’t need test_project now!

Test-Only Models

#test_models.py

from django.db import models

class TestModel(models.Model): name = models.CharField() #tests.py

from django.test import TestCase

class TestMyViews(TestCase): test_models = ['test_models']

def testIndexPageView(self): from myapp.models import TestModel TestModel.objects.get(name='daniellindsleyrocksdahouse')

Skipping Tests

» Views Required

» Models Required

» Specific database type

» Conditional on a Function

Skipping Tests

from django.tests.decorators import conditional_skipimport datetime

class TestUnderCondition(TestCase):

def _check_2009(): # Condition returning True if test should be run and False if it # should be skipped. if datetime.datetime.now() > datetime.datetime(2009, 01, 01): return True

@conditional_skip(_check_2009, reason='This test only runs in 2009') def testOnlyIn2009(self): # Test to run if _my_condition evaluates to True

Skipping Tests

====================================================================== SKIPPED: test_email_found (django.contrib.auth.tests.basic.PasswordResetTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dnaquin/Dropbox/Sandbox/django/django/test/decorators.py", line 43, in _skip raise SkippedTest(reason=reason) SkippedTest: Required view for this test not found: django.contrib.auth.views.password_reset

---------------------------------------------------------------------- Ran 408 tests in 339.663s

FAILED (failures=1, skipped=2)

Coverage

» Basically the hotness

» django-admin.py test --coverage

» django-admin.py test --coverage --report

Basic Coverage

Pretty Coverage

Pretty Coverage

Windmill Tests

» Didn’t get finished

» Allow for functional tests of the admin

» Allow for custom Windmill tests of your apps

Custom Test Runners

» Allow you to do anything you want

» Load custom fixtures

» nose or py.test runners

Django Test Extensions

» Gareth Rushgrove

» Extra Assertions

» Coverage and XML Test Runners

» http://github.com/garethr/django-test-extensions

Django Sane Testing

» Ella Folks

» Based on nosetests

» Selenium

» Live server

» http://devel.almad.net/trac/django-sane-testing/

Django Test Utils

» Mine!

» Testmaker

» Crawler

» Random fanciness

» http://github.com/ericholscher/django-test-utils/tree/master

Goals

Some form of TDD

» Write tests as you write code

» Makes your code easy to test

Follow Django’s Model

» Tests with every commit

» Docs with every commit

» Run tests before commiting

Continuous Integration

NEVER LEAVE THE BUILD BROKEN

Love Green

Profiling

» python -m cProfile manage.py test

» Allows you to profile a repeatable case

» Faster tests = Faster production

Things to remember

Testing is not hard, you just have to get started.

If your code doesn’t have tests, it will be hard/impossible to

refactor

Once you have tests, you need to run them!

Teh Pretty

» Thanks to Idan Gazit for design help

Credits

» http://www.flickr.com/photos/tym/192416981/

» http://www.flickr.com/photos/seandreilinger/2459266781/

» http://www.homebabysafety.com/images/baby_crawl_suit.jpg

Questions?

» twitter.com/ericholscher

» http://ericholscher.com

» eric@ericholscher.com