pycon 2013 : scripting to pypi to github and more

Script to PyPi to GithubScript to PyPi to Github

@__mharrison__@__mharrison__http://hairysun.comhttp://hairysun.com

About MeAbout Me

● 12 years Python12 years Python● Worked in HA, Search, Open Source, BI Worked in HA, Search, Open Source, BI

and Storageand Storage● Author of multiple Python BooksAuthor of multiple Python Books

ContinuumsContinuums

More like perl-TMTOWTDIMore like perl-TMTOWTDI

AgendaAgenda

● Project DevelopmentProject Development

– VersioningVersioning

– ConfigurationConfiguration

– LoggingLogging

– File inputFile input

– Shell invocationShell invocation

● Environment Layout * virtualenv * pipEnvironment Layout * virtualenv * pip

● Project layoutProject layout

Agenda (2)Agenda (2)● DocumentationDocumentation

● Automation/MakefileAutomation/Makefile

● PackagingPackaging

– setup.pysetup.py

– PyPiPyPi

● TestingTesting

● GithubGithub

● Travis CITravis CI

● PoachplatePoachplate

BeginBegin

WarningWarning

● Starting from basic Python knowledgeStarting from basic Python knowledge● Hands onHands on

– (short) lecture(short) lecture– (short) code(short) code– repeat until time is gonerepeat until time is gone

ProjectProject

Create pycat. Pythonic implementation of Create pycat. Pythonic implementation of catcat

ScriptingScripting

hello world hello world cat.pycat.py

importimport syssysforfor line line inin sys sys..stdin:stdin: printprint line, line,

hello world - Python 3hello world - Python 3

importimport syssysforfor line line inin sys sys..stdin:stdin: printprint(line, end(line, end==''''))

2 or 3?2 or 3?

2 or 3?2 or 3?

● Better legacy/library support for 2Better legacy/library support for 2● Possible to support bothPossible to support both

hello worldhello world

importimport syssysforfor line line inin sys sys..stdin:stdin: sys sys..stdoutstdout..write(line)write(line)

AssignmentAssignment

create create cat.pycat.py

Single File or Project?Single File or Project?

LayoutLayoutCan depend on distribution mechanism:Can depend on distribution mechanism:● Single fileSingle file● Zip fileZip file

– Python entry pointPython entry point– Splat/runSplat/run

● System packageSystem package● PyPi/Distribute/pip packagePyPi/Distribute/pip package

Single FileSingle File

● chmodchmod and place in and place in $PATH$PATH● add add #!/usr/bin/env python#!/usr/bin/env python

Single File (2)Single File (2)

● No reuseNo reuse

Zip FileZip File

● PYTHONPATH=cat.zip python -m PYTHONPATH=cat.zip python -m __main____main__

● tar -zxvf cat.zip; cd cat; tar -zxvf cat.zip; cd cat; python cat.pypython cat.py

Zip File (2)Zip File (2)

● No reuseNo reuse● No stacktraceNo stacktrace

System PackageSystem Package

● emerge -av qtileemerge -av qtile (rpm|apt|brew) (rpm|apt|brew)

System Package (2)System Package (2)

● Requires rootRequires root● At mercy of packager (maybe worse than At mercy of packager (maybe worse than

PyPi)PyPi)● ReuseReuse● Limited to single versionLimited to single version● python -m modulenamepython -m modulename

Pip PackagePip Package

““Best practice” is combo of:Best practice” is combo of:● DistributeDistribute● VirtualenvVirtualenv● PipPip

Pip Package (2)Pip Package (2)

$ virtualenv catenv$ virtualenv catenv$ source catenv/bin/activate$ source catenv/bin/activate$ pip install pycat$ pip install pycat

Pip Package (3)Pip Package (3)

● Multiple versionsMultiple versions● ReuseReuse● Effort to create Effort to create setup.pysetup.py

Minimal Example LayoutMinimal Example LayoutProject/Project/ README.txt README.txt project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py

Better Example LayoutBetter Example LayoutProject/Project/ .gitignore .gitignore doc/ doc/ Makefile Makefile index.rst index.rst README.txt README.txt Makefile Makefile bin/ bin/ runfoo.py runfoo.py project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py

AssignmentAssignment

create layout for create layout for cat.pycat.py

Semantic VersioningSemantic Versioning

VersioningVersioning

http://semver.orghttp://semver.org

Formal spec for versioning projectsFormal spec for versioning projects

Python VersioningPython Versioning

PEP 386PEP 386

N.N[.n]+[{a|b|c}N[.N]+][.postN][.devN]N.N[.n]+[{a|b|c}N[.N]+][.postN][.devN]

1.0a1 < 1.0a2 < 1.0b2.post3451.0a1 < 1.0a2 < 1.0b2.post345

setup.pysetup.py

fromfrom distutils.coredistutils.core importimport setup setup

setup(namesetup(name=='PyCat''PyCat',,........ version version=='1.0''1.0'))

Where to store version?Where to store version?

In In module.__version__module.__version__ (might cause (might cause importing issues)importing issues)

VersionVersion

If using Sphinx for docs, be sure to update:If using Sphinx for docs, be sure to update:

docs/docs/ conf.py conf.py

argparseargparse

ap ap == argparse argparse..ArgumentParser(versionArgumentParser(version=='1.0''1.0'))......apap..print_version()print_version()

AssignmentAssignment

Add version to projectAdd version to project

ConfigurationConfiguration

ConfigurationConfiguration● PythonPython

– Django (Django (settings.pysettings.py))

– Python (Python (site.pysite.py, , setup.pysetup.py))

● JSON/YAMLJSON/YAML

– Google App EngineGoogle App Engine

● Environment VariablesEnvironment Variables

– Python (Python (PYTHONPATHPYTHONPATH))

● .ini (.ini (ConfigParserConfigParser, , ConfigObjConfigObj))

– matplotlibmatplotlib

– ipythonipython

● Command line options (Command line options (argparseargparse))

● Sqlite blobSqlite blob

● Shelve/pickle blobShelve/pickle blob

Configuration (2)Configuration (2)

Are not mutually exclusiveAre not mutually exclusive

Configuration (3)Configuration (3)(Unix) Hierarchy:(Unix) Hierarchy:

● System rc (run control) (System rc (run control) (/etc/conf.d/etc/conf.d))

● User rc (User rc (~/.config/app/...~/.config/app/...))

● Environment variablesEnvironment variables

● Command line optionsCommand line options

http://www.faqs.org/docs/artu/ch10s02.htmlhttp://www.faqs.org/docs/artu/ch10s02.html

Filesystem Hierarchy Standard: Filesystem Hierarchy Standard: http://www.pathname.com/fhs/http://www.pathname.com/fhs/

Configuration (4)Configuration (4)

● Plain text config is easily approachablePlain text config is easily approachable● Careful with Python config on process Careful with Python config on process

run by rootrun by root

AssignmentAssignment

Add configuration Add configuration -n-n to to show line numbersshow line numbers

LoggingLogging

LoggingLogging

logginglogging module provides feature-rich module provides feature-rich logginglogging

Logging (2)Logging (2)

importimport logginglogging

logginglogging..basicConfig(levelbasicConfig(level==logginglogging..ERROR,ERROR, filename filename=='.log''.log'))......logginglogging..error(error('Error encountered in...''Error encountered in...'))

AssignmentAssignment

Add configuration Add configuration --verbose--verbose to log file to log file

being “catted”being “catted”

Dealing with File InputDealing with File Input

““Files”Files”

Dealing with?Dealing with?● FilenameFilename● file objectfile object● string datastring data

FilenameFilename

Somewhat analogous to an Somewhat analogous to an IterableIterable..● Can open/iterate many timesCan open/iterate many times● Implementation depends on fileImplementation depends on file● Need to manage closing fileNeed to manage closing file

File ObjectFile Object

Somewhat analogous to an Somewhat analogous to an IteratorIterator..● Can iterate once (unless Can iterate once (unless seekseeked)ed)

● Can accept Can accept filefile, , StringIOStringIO, , socketsocket, , generator, etcgenerator, etc

● Memory friendly - scalableMemory friendly - scalable

String DataString Data

No iterator analogy.No iterator analogy.● Memory hog - less scalableMemory hog - less scalable

Stdlib ExamplesStdlib ExamplesModule String Data File Filename

json loads load

pickle loads load

xml.etree.ElementTree

fromstring parse parse

xml.dom.minidom parseString parse parse

ConfigParser cp.readfp cp.read(filenames)

csv reader DictReader

pyyaml 3rd party load, safe_load load, safe_load

Stdlib Take-awaysStdlib Take-aways

● mostly functionsmostly functions● file interface is required, others optionalfile interface is required, others optional● parseparse or or loadload

ExampleExample>>>>>> importimport syssys

>>>>>> defdef parseparse(fin):(fin):...... forfor line line inin upper(fin): upper(fin):...... sys sys..stdoutstdout..write(line)write(line)

>>>>>> defdef upperupper(iterable):(iterable):...... forfor item item inin iterable: iterable:...... yieldyield strstr(item)(item)..upper()upper()

Create file to parseCreate file to parse

>>> >>> withwith openopen(('/tmp/data''/tmp/data', , 'w''w') ) asas fout: fout:... ... fout fout..write(write('line1'line1\n\n''))... ... fout fout..write(write('line2'line2\n\n''))

Filename to Filename to filefile

>>> >>> filename filename == '/tmp/data''/tmp/data'>>> >>> withwith openopen(filename) (filename) asas fin: fin:... ... parse(fin) parse(fin)LINE1LINE1LINE2LINE2

String data to String data to filefile>>> >>> data data == "string"string\n\ndatadata\n\n"">>> >>> importimport StringIOStringIO>>> >>> parse(StringIOparse(StringIO..StringIO(data))StringIO(data))STRINGSTRINGDATADATA

Parse IterableParse Iterable

>>> >>> data data == [ ['foo'foo\n\n'', , 'bar'bar\n\n'']]>>> >>> parse(data)parse(data)FOOFOOBARBAR

More More filefile benefits benefits

● Combine with generators to filter, tweakCombine with generators to filter, tweak● Easier to testEasier to test

AssignmentAssignment

Add a Add a parseparse function function

Invoking Shell Invoking Shell CommandsCommands

Reading outputReading output

>>> >>> importimport subprocesssubprocess>>> >>> p p == subprocess subprocess..Popen(Popen('id -u''id -u', shell, shell==TrueTrue, , stdoutstdout==subprocesssubprocess..PIPE, stderrPIPE, stderr==subprocesssubprocess..PIPE)PIPE)>>> >>> pp..stdoutstdout..read()read()'1000\n''1000\n'>>> >>> pp..returncode returncode # None means not done# None means not done>>> >>> printprint p p..wait()wait()00

Feeding Feeding stdinstdinCan use Can use communicatecommunicate or or p2.stdin.writep2.stdin.write w/ w/ flush/closeflush/close..>>> >>> p2 p2 == subprocess subprocess..Popen(Popen('wc -l''wc -l', shell, shell==TrueTrue, , stdoutstdout==subprocesssubprocess..PIPE, stdinPIPE, stdin==subprocesssubprocess..PIPE, PIPE, stderrstderr==subprocesssubprocess..PIPE)PIPE)>>> >>> out, err out, err == p2 p2..communicate(communicate('foo'foo\n\nbarbar\n\n'') ) #p.stdin.flush()#p.stdin.flush()

>>> >>> outout'2\n''2\n'>>> >>> p2p2..returncodereturncode00

Chaining scriptsChaining scriptsChaining is pretty straightforward make sure to Chaining is pretty straightforward make sure to closeclose stdinstdin. . http://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subprhttp://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subprocess-pipelineocess-pipeline

>>> >>> p3 p3 == subprocess subprocess..Popen(Popen('sort''sort', shell, shell==TrueTrue,,... ... stdout stdout==subprocesssubprocess..PIPE,PIPE,... ... stdin stdin==subprocesssubprocess..PIPE)PIPE)>>> >>> p4 p4 == subprocess subprocess..Popen(Popen('uniq''uniq', shell, shell==TrueTrue,,... ... stdout stdout==subprocesssubprocess..PIPE,PIPE,... ... stdin stdin==p3p3..stdout,stdout,... ... close_fds close_fds==TrueTrue) ) # hangs w/o close_fds# hangs w/o close_fds

>>> >>> p3p3..stdinstdin..write(write('1'1\n\n22\n\n11\n\n''))>>> >>> p3p3..stdinstdin..flush(); p3flush(); p3..stdinstdin..close();close();>>> >>> p4p4..stdoutstdout..read()read()'1\n2\n''1\n2\n'

Chaining scripts and pythonChaining scripts and python

catcat 0-2, add 10 to them (in python) and 0-2, add 10 to them (in python) and wc wc -l-l results. results.>>> >>> importimport osos>>> >>> p5 p5 == subprocess subprocess..Popen(Popen('cat''cat', shell, shell==TrueTrue, , stdoutstdout==subprocesssubprocess..PIPE, stdinPIPE, stdin==subprocesssubprocess..PIPE, close_fdsPIPE, close_fds==TrueTrue))>>> >>> defdef p6p6((inputinput):):... ... ''' add 10 to line in input '''''' add 10 to line in input '''... ... forfor line line inin inputinput::... ... yieldyield ''%d%s%d%s'' %%((intint(line(line..strip())strip())+10+10, os, os..linesep)linesep)

Chaining scripts and python Chaining scripts and python (2)(2)

>>> >>> p7 p7 == subprocess subprocess..Popen(Popen('wc -l''wc -l', shell, shell==TrueTrue, , stdoutstdout==subprocesssubprocess..PIPE, stdinPIPE, stdin==subprocesssubprocess..PIPE, close_fdsPIPE, close_fds==TrueTrue))>>> >>> [p5[p5..stdinstdin..write(write(strstr(x)(x)++osos..linesep) linesep) forfor x x inin xrangexrange((33)])]>>> >>> p5p5..stdinstdin..close()close()>>> >>> [p7[p7..stdinstdin..write(x) write(x) forfor x x inin p6(p5 p6(p5..stdoutstdout..xreadlines())]xreadlines())]>>> >>> p7p7..stdinstdin..close()close()>>> >>> p7p7..stdoutstdout..read()read()'3\n''3\n'

EnvironmentEnvironment

Python EnvironmentPython Environment

● System Python or “Virtual” PythonSystem Python or “Virtual” Python● Installation methodInstallation method

Which PythonWhich Python

System PythonSystem Python● Requires rootRequires root● Only one version of libraryOnly one version of library● System packages may be out of dateSystem packages may be out of date

Which Python (2)Which Python (2)

““Virtual” PythonVirtual” Python● Runs as userRuns as user● Specify versionSpecify version● Sandboxed from systemSandboxed from system● Create multiple sandboxesCreate multiple sandboxes

Which Python (3)Which Python (3)

Install Install virtualenvvirtualenv

virtualenvvirtualenvInstallation:Installation:

● System package System package $ sudo apt-get install $ sudo apt-get install python-virtualenvpython-virtualenv

● With pip With pip $ pip install virtualenv$ pip install virtualenv● $ easy_install virtualenv$ easy_install virtualenv● $ wget $ wget https://raw.github.com/pypa/virtualenv/master/https://raw.github.com/pypa/virtualenv/master/virtualenv.py; python virtualenv.pyvirtualenv.py; python virtualenv.py

virtualenvvirtualenv (2) (2)

Create virtual environment:Create virtual environment:

$ virtualenv env_name$ virtualenv env_name

Directory StructureDirectory Structure

env_name/env_name/ bin/ bin/ activate activate python python pip pip lib/ lib/ python2.7/ python2.7/ site-packages/ site-packages/

virtualenvvirtualenv (3) (3)

Activate virtual environment:Activate virtual environment:

$ source env_name/bin/activate$ source env_name/bin/activate

virtualenvvirtualenv (4) (4)

Windows activate virtual environment:Windows activate virtual environment:

> \path\to\env\Scripts\activate> \path\to\env\Scripts\activate

May require May require PS C:\> PS C:\> Set-ExecutionPolicy AllSignedSet-ExecutionPolicy AllSigned

virtualenvvirtualenv (5) (5)

Comes with Comes with pippip to install packages: to install packages:

$ pip install sqlalchemy$ pip install sqlalchemy

virtualenvvirtualenv (6) (6)

$ which python # not /usr/bin/python$ which python # not /usr/bin/python/home/matt/work/courses/script-pypi-github/env/b/home/matt/work/courses/script-pypi-github/env/bin/pythonin/python

virtualenvvirtualenv (7) (7)

>>> sys.path>>> sys.path['', ..., ['', ..., '/home/matt/work/courses/script-py'/home/matt/work/courses/script-pypi-github/env/lib/python2.7/site-ppi-github/env/lib/python2.7/site-packages']ackages']

virtualenvvirtualenv (8) (8)

Use Use deactivatedeactivate shell function to reset shell function to reset PATHPATH::

$ deactivate$ deactivate

AssignmentAssignment

create virtualenv create virtualenv envenv

pippip

pippipRecursive Acronym - Recursive Acronym - PPip ip IInstalls nstalls PPackagesackages● installinstall● upgradeupgrade● uninstalluninstall● ““pin” versionspin” versions● requirements.txtrequirements.txt

pippip (2) (2)

Install:Install:

$ pip install sqlalchemy$ pip install sqlalchemy

pippip (3) (3)

Upgrade:Upgrade:

$ pip install --upgrade sqlalchemy$ pip install --upgrade sqlalchemy

pippip (4) (4)

Uninstall:Uninstall:

$ pip uninstall sqlalchemy$ pip uninstall sqlalchemy

pippip (5) (5)

““Pin” version:Pin” version:

$ pip install sqlalchemy==0.7$ pip install sqlalchemy==0.7

pippip (6) (6)

Requirements file:Requirements file:

$ pip install -r requirements.txt$ pip install -r requirements.txt

pippip (7) (7)

Requirements file Requirements file $ cat requirements.txt$ cat requirements.txt::

sqlalchemy==0.7sqlalchemy==0.7foobarfoobarbazlib>=1.0bazlib>=1.0

pippip (8) (8)

Create requirement file from env:Create requirement file from env:

$ pip freeze > req.txt$ pip freeze > req.txt

pippip (9) (9)

● pip docs say to install from virtualenvpip docs say to install from virtualenv● virtualenv docs say to install from pipvirtualenv docs say to install from pip

pippip (10) (10)Install from directory:Install from directory:

pip install --download packages -r pip install --download packages -r requirements.txtrequirements.txtpip install --no-index pip install --no-index --find-links=file://full/path/to/p--find-links=file://full/path/to/packages -r requirements.txtackages -r requirements.txt

distributedistribute, , setuptoolssetuptools, , distutilsdistutils

● pippip wraps wraps distributedistribute adds uninstall, adds uninstall, req.txtreq.txt

● distributedistribute fork of fork of setuptoolssetuptools● setuptoolssetuptools unmaintained, adds eggs, unmaintained, adds eggs,

dependencies, dependencies, easy_installeasy_install● distutilsdistutils stdlib packaging library stdlib packaging library

AssignmentAssignment

use pip to install use pip to install package from internetpackage from internet

More LayoutMore Layout

Better Example LayoutBetter Example LayoutProject/Project/ .gitignore .gitignore doc/ doc/ Makefile Makefile index.rst index.rst README.txt README.txt Makefile Makefile bin/ bin/ runfoo.py runfoo.py project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py

.gitignore.gitignore*.py[cod]*.py[cod]

# C extensions# C extensions*.so*.so

# Packages# Packages*.egg*.egg*.egg-info*.egg-infodistdistbuildbuildeggseggspartspartsbinbinvarvarsdistsdist

.gitignore.gitignoredevelop-eggsdevelop-eggs.installed.cfg.installed.cfglibliblib64lib64__pycache____pycache__

# Installer logs# Installer logspip-log.txtpip-log.txt

# Unit test / coverage reports# Unit test / coverage reports.coverage.coverage.tox.toxnosetests.xmlnosetests.xml

# Translations# Translations*.mo*.mo

https://github.com/github/gitignorehttps://github.com/github/gitignore

.gitignore.gitignore (2) (2)

See See blaze-coreblaze-core for example of C and for example of C and Docs. Docs. https://github.com/ContinuumIO/blaze-corhttps://github.com/ContinuumIO/blaze-coree

.gitignore.gitignore (3) (3)From From requestsrequests::

.coverage.coverageMANIFESTMANIFESTcoverage.xmlcoverage.xmlnosetests.xmlnosetests.xmljunit-report.xmljunit-report.xmlpylint.txtpylint.txttoy.pytoy.pyviolations.pyflakes.txtviolations.pyflakes.txtcover/cover/docs/_builddocs/_buildrequests.egg-info/requests.egg-info/*.pyc*.pyc*.swp*.swpenv/env/.workon.workont.pyt.pyt2.pyt2.py

https://github.com/kennethreitz/requestshttps://github.com/kennethreitz/requests

Other tipsOther tips● localsettings.pylocalsettings.py (for django) (for django)

● *~*~ (emacs) (emacs)

● Run Run git statusgit status and add outliers to and add outliers to .gitignore.gitignore

● Make settings global:Make settings global:git config --global core.excludesfile git config --global core.excludesfile Python.gitignorePython.gitignoregit config --global core.excludesfile git config --global core.excludesfile Python.gitignorePython.gitignore

AssignmentAssignment

add (premptive) add (premptive) .gitignore.gitignore

DocumentationDocumentation

DocumentationDocumentation

Two types:Two types:● Developer docs (README, INSTALL, Developer docs (README, INSTALL,

HACKING, etc)HACKING, etc)● End userEnd user

DeveloperDeveloper

READMEREADME - main entry point for project - main entry point for project● Brief introBrief intro● Links/ContactLinks/Contact● LicenseLicense

READMEREADME

For github integration name it For github integration name it README.rstREADME.rst or or README.mdREADME.md

LICENSELICENSE

Include text of license. Templates at Include text of license. Templates at http://opensource.org/licenses/index.htmlhttp://opensource.org/licenses/index.html

LicensingLicensing

Some include dunder meta in project docstring (Some include dunder meta in project docstring (requestsrequests __init__.py__init__.py):):

:copyright: (c) 2013 by Kenneth Reitz.:copyright: (c) 2013 by Kenneth Reitz.:license: Apache 2.0, see LICENSE for more :license: Apache 2.0, see LICENSE for more details.details.

(note IANAL)(note IANAL)

Licensing (2)Licensing (2)Some include dunder meta in project (Some include dunder meta in project (requestsrequests __init__.py__init__.py):):

__title__ = 'requests'__title__ = 'requests'__version__ = '1.1.0'__version__ = '1.1.0'__build__ = 0x010100__build__ = 0x010100__author__ = 'Kenneth Reitz'__author__ = 'Kenneth Reitz'__license__ = 'Apache 2.0'__license__ = 'Apache 2.0'__copyright__ = 'Copyright 2013 Kenneth Reitz'__copyright__ = 'Copyright 2013 Kenneth Reitz'

(note IANAL)(note IANAL)

Other filesOther files

● AUTHORSAUTHORS● HISTORY/CHANGELOGHISTORY/CHANGELOG● TODOTODO

AssignmentAssignment

create simple create simple READMEREADME

End User DocsEnd User Docs

Sphinx is a tool that makes it easy to create Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, intelligent and beautiful documentation, written by Georg Brandl and licensed written by Georg Brandl and licensed under the BSD license.under the BSD license.

http://sphinx-doc.orghttp://sphinx-doc.org

Suggested setupSuggested setup

Project/Project/ doc/ doc/ # sphinx stuff # sphinx stuff Makefile Makefile

Sphinx in 4 LinesSphinx in 4 Lines

$ cd docs$ cd docs$ sphinx-quickstart$ sphinx-quickstart$ $EDITOR index.rst$ $EDITOR index.rst$ make html$ make html

AssignmentAssignment

write docs at a later write docs at a later time :)time :)

MakefileMakefile

MotivationMotivationRunning commands often:Running commands often:

● nosetestsnosetests (plus options) (plus options)

● create sdistcreate sdist

● upload to PyPiupload to PyPi

● create virtualenvcreate virtualenv

● install dependenciesinstall dependencies

● cleanup cruftcleanup cruft

● create TAGScreate TAGS

● profileprofile

● sdistsdist

● PyPi - register and uploadPyPi - register and upload

● creating pngs from svgscreating pngs from svgs

● docsdocs

● Python 3 testingPython 3 testing

● etc...etc...

MakefileMakefile

● Knows about executing (build) Knows about executing (build) commandscommands

● Knows about dependenciesKnows about dependencies

ExampleExample

To test:To test:● Make virtualenvMake virtualenv● install code dependenciesinstall code dependencies● install nose (+coverage)install nose (+coverage)● run testsrun tests

Clean checkoutClean checkout$ make test$ make testvsvs

$ virtualenv env$ virtualenv env$ env/bin/activate$ env/bin/activate$ pip install -r deps.txt$ pip install -r deps.txt$ pip install nose coverage.py$ pip install nose coverage.py$ nosestests$ nosestests

Enough Enough makemake knowledge to be knowledge to be

dangerousdangerous

MakefileMakefile (1) (1)

Syntax of Syntax of MakefileMakefile::

file: dependentfilefile: dependentfile<TAB>Command1<TAB>Command1......<TAB>CommandN<TAB>CommandN

MakefileMakefile (2) (2)

Running (runs Running (runs MakefileMakefile by default): by default):

$ make file$ make file# will build dependentfile if # will build dependentfile if necessarynecessary# then build file# then build file

MakefileMakefile (3) (3)

Example:Example:

foo: foo.c foo.hfoo: foo.c foo.h<TAB>cc -c foo.c<TAB>cc -c foo.c<TAB>cc -o foo foo.o<TAB>cc -o foo foo.o

MakefileMakefile (4) (4)

Running (echoes commands by default Running (echoes commands by default -s-s for silent):for silent):

$ make$ makecc -c foo.ccc -c foo.ccc -o foo foo.occ -o foo foo.o

MakefileMakefile (5) (5)

Subsequent runs do nothing:Subsequent runs do nothing:

$ make$ makemake: `foo' is up to date.make: `foo' is up to date.

MakefileMakefile (6) (6)

Add a Add a cleanclean command: command:

.PHONY: clean.PHONY: clean

clean:clean:<TAB>rm foo.o foo<TAB>rm foo.o foo

MakefileMakefile (7) (7)

Since Since cleanclean isn't a file, need to use isn't a file, need to use .PHONY.PHONY to indicate that to to indicate that to makemake. (If you had a file . (If you had a file named named cleanclean it wouldn't try to build it). it wouldn't try to build it).

MakefileMakefile (8) (8)

(Simply Expanded) Variables (expanded when set):(Simply Expanded) Variables (expanded when set):

BIN := env/binBIN := env/binPY := $(BIN)/pythonPY := $(BIN)/pythonNOSE := $(BIN)/nosetestsNOSE := $(BIN)/nosetests

.PHONY: build.PHONY: buildbuild: envbuild: env<TAB>$(PY) setup.py sdist<TAB>$(PY) setup.py sdist

MakefileMakefile (9) (9)

(Recursively Expanded) Variables (expanded when used):(Recursively Expanded) Variables (expanded when used):

FILE = fooFILE = fooDATA = $(FILE)DATA = $(FILE)

# If DATA expanded would be foo# If DATA expanded would be foo

FILE = barFILE = bar# If DATA expanded would be bar# If DATA expanded would be bar

MakefileMakefile (10) (10)

Shell functions:Shell functions:

.PHONY: pwd.PHONY: pwdpwd:pwd:<TAB>pushd /etc<TAB>pushd /etc

MakefileMakefile (11) (11)Invoking:Invoking:

$ make pwd$ make pwdpushd /etcpushd /etcmake: pushd: Command not foundmake: pushd: Command not foundmake: *** [pwd] Error 127make: *** [pwd] Error 127

((pushdpushd is a bash function) is a bash function)

MakefileMakefile (12) (12)Shell functions:Shell functions:

SHELL := /bin/bashSHELL := /bin/bash

.PHONY: pwd.PHONY: pwdpwd:pwd:<TAB>pushd /etc<TAB>pushd /etc

MakefileMakefile (13) (13)

Multiple commands:Multiple commands:

SHELL := /bin/bashSHELL := /bin/bash

.PHONY: pwd.PHONY: pwdpwd:pwd:<TAB>pushd /etc<TAB>pushd /etc<TAB>pwd<TAB>pwd<TAB>popd<TAB>popd

MakefileMakefile (14) (14)

Multiple commands:Multiple commands:

$ make pwd$ make pwdpushd /etcpushd /etc/etc /tmp/foo/etc /tmp/foopwdpwd/tmp/foo/tmp/foopopdpopd/bin/bash: line 0: popd: directory stack empty/bin/bash: line 0: popd: directory stack empty

MakefileMakefile (15) (15)

Each tab indented command runs in its Each tab indented command runs in its own process. Use own process. Use ;; and put in one line or and put in one line or use use \\ for line continuation for line continuation

MakefileMakefile (16) (16)

Multiple commands (use line continuation Multiple commands (use line continuation \\):):

SHELL := /bin/bashSHELL := /bin/bash

.PHONY: pwd2.PHONY: pwd2pwd2:pwd2:<TAB>pushd /etc; \<TAB>pushd /etc; \<TAB>pwd; \<TAB>pwd; \<TAB>popd<TAB>popd

MakefileMakefile (17) (17)

Shell variables:Shell variables:

.PHONY: path.PHONY: pathpath:path:<TAB>echo $PATH<TAB>echo $PATH

MakefileMakefile (18) (18)

Make thinks they are make variables:Make thinks they are make variables:

$ make path$ make pathecho ATHecho ATHATHATH

MakefileMakefile (19) (19)

$$ needs to be escaped with needs to be escaped with $$::

.PHONY: path.PHONY: pathpath:path:<TAB>echo $$PATH<TAB>echo $$PATH

MakefileMakefile (18) (18)

Now it works:Now it works:

$ make path$ make pathecho $PATHecho $PATH/tmp/maketest/tmp/maketest

Makefiles for Python Makefiles for Python ProjectsProjects

Inspired by Rick Harding's Inspired by Rick Harding's TalkTalk

http://pyvideo.org/video/1354/starting-youhttp://pyvideo.org/video/1354/starting-your-project-right-setup-and-automationr-project-right-setup-and-automationhttps://github.com/mitechie/pyohio_2012https://github.com/mitechie/pyohio_2012

MakefileMakefile for Python for Python

Make virtualenv:Make virtualenv:

env:env:<TAB>virtualenv env<TAB>virtualenv env

MakefileMakefile for Python (2) for Python (2)

Make dependencies:Make dependencies:

.PHONY: deps.PHONY: depsdeps: envdeps: env<TAB>$(PIP) install -r <TAB>$(PIP) install -r requirements.txtrequirements.txt

MakefileMakefile for Python (3) for Python (3)

Testing with Testing with nosenose::

.PHONY: test.PHONY: testtest: nose depstest: nose deps<TAB>$(NOSE)<TAB>$(NOSE)

# nose depends on the nosetests binary# nose depends on the nosetests binarynose: $(NOSE)nose: $(NOSE)$(NOSE): env$(NOSE): env<TAB>$(PIP) install nose<TAB>$(PIP) install nose

Contrary OpinionsContrary Opinions

““Dynamic languages don't need anything Dynamic languages don't need anything like like makemake, unless they have some , unless they have some compile-time interface dependencies compile-time interface dependencies between modules”between modules”

http://stackoverflow.com/questions/758093http://stackoverflow.com/questions/7580939/why-are-there-no-makefiles-for-automati9/why-are-there-no-makefiles-for-automation-in-python-projectson-in-python-projects

Other optionsOther options

● paverpaver● fabricfabric● buildoutbuildout

PackagingPackaging

setup.pysetup.py overloaded overloaded

● create sdist (source distribution)create sdist (source distribution)● upload to pypiupload to pypi● install packageinstall package

setup.pysetup.py wart wart

requirerequire keyword of keyword of distutilsdistutils doesn't doesn't download reqs only download reqs only documentsdocuments them. Use them. Use requirements.txtrequirements.txt in combo with in combo with pippip..

setup.pysetup.py example exampleFrom From requestsrequests::

setup(setup( name='requests', name='requests', version=requests.__version__, version=requests.__version__, description='Python HTTP for Humans.', description='Python HTTP for Humans.', long_description=open('README.rst').read() + long_description=open('README.rst').read() + '\n\n' +'\n\n' + open('HISTORY.rst').read(), open('HISTORY.rst').read(), author='Kenneth Reitz', author='Kenneth Reitz', author_email='me@kennethreitz.com', author_email='me@kennethreitz.com', url='http://python-requests.org', url='http://python-requests.org',

setup.pysetup.py example (2) example (2)

packages=packages,packages=packages,package_data={'': ['LICENSE', 'NOTICE'], package_data={'': ['LICENSE', 'NOTICE'], 'requests': ['*.pem']},'requests': ['*.pem']},package_dir={'requests': 'requests'},package_dir={'requests': 'requests'},include_package_data=True,include_package_data=True,install_requires=requires,install_requires=requires,license=open('LICENSE').read(),license=open('LICENSE').read(),zip_safe=False,zip_safe=False,

setup.pysetup.py example (3) example (3) classifiers=(classifiers=( 'Development Status :: 5 - Production/Stable', 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'Intended Audience :: Developers', 'Natural Language :: English', 'Natural Language :: English', 'License :: OSI Approved :: Apache Software License', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python', 'Programming Language :: Python', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3', # 'Programming Language :: Python :: 3.0', # 'Programming Language :: Python :: 3.0', 'Programming Language :: Python :: 3.1', 'Programming Language :: Python :: 3.1', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.3', ), ),))

setup.pysetup.py modules modules

If project consists of a few modules this If project consists of a few modules this may be easiestmay be easiest

setup.pysetup.py packages packages

Need to explicitly list Need to explicitly list allall packages, not just packages, not just rootroot

setup.pysetup.py scripts scripts

Add executable Python files hereAdd executable Python files here

setup.pysetup.py non-Python files non-Python files

● add files to add files to MANIFEST.inMANIFEST.in (include in (include in package)package)

● add files to add files to package_datapackage_data in in setup.pysetup.py (include in install) Not recursive(include in install) Not recursive

MANIFEST.inMANIFEST.in language language

● include|exclude pat1 pat2 ...include|exclude pat1 pat2 ...● recursive-(include|exclude) dir pat1 pat2 ...recursive-(include|exclude) dir pat1 pat2 ...● global-(include|exclude) dir pat1 pat2 ...global-(include|exclude) dir pat1 pat2 ...● prune dirprune dir● graft dirgraft dir

http://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-chttp://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-cmdmd

setup.pysetup.py classifiers classifiers

Almost 600 different classifiers.Almost 600 different classifiers.

Not used by pip to enforce versions. For UI Not used by pip to enforce versions. For UI onlyonly

Create sdistCreate sdist

$ python setup.py sdist$ python setup.py sdist

PyPiPyPi

Validate Validate setup.pysetup.py::

$ python setup.py check$ python setup.py check

PyPi RegisterPyPi Register

Click on “Register” on right hand boxClick on “Register” on right hand box

https://pypi.python.org/pypi?%3Aaction=rehttps://pypi.python.org/pypi?%3Aaction=register_formgister_form

PyPi UploadPyPi Upload

$ python setup.py sdist register $ python setup.py sdist register uploadupload

PyPi Upload (2)PyPi Upload (2)$ python setup.py sdist register upload$ python setup.py sdist register upload......Creating tar archiveCreating tar archiveremoving 'rst2odp-0.2.4' (and everything under it)removing 'rst2odp-0.2.4' (and everything under it)running registerrunning registerrunning checkrunning checkWe need to know who you are, so please choose either:We need to know who you are, so please choose either: 1. use your existing login, 1. use your existing login, 2. register as a new user, 2. register as a new user, 3. have the server generate a new password for you (and email it to you), 3. have the server generate a new password for you (and email it to you), oror 4. quit 4. quitYour selection [default 1]:Your selection [default 1]:11

PyPi Upload (3)PyPi Upload (3)

Username: mharrisonUsername: mharrisonPassword:Password:Registering rst2odp to http://pypi.python.org/pypiRegistering rst2odp to http://pypi.python.org/pypiServer response (200): OKServer response (200): OKI can store your PyPI login so future submissions will be faster.I can store your PyPI login so future submissions will be faster.(the login will be stored in /home/matt/.pypirc)(the login will be stored in /home/matt/.pypirc)Save your login (y/N)?ySave your login (y/N)?yrunning uploadrunning uploadSubmitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypiSubmitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypiServer response (200): OKServer response (200): OK

PyPi NotePyPi Note

Though PyPi packages are signed there is Though PyPi packages are signed there is no verification step during package no verification step during package installationinstallation

Non PyPi URLNon PyPi URL

$ pip install --no-index -f $ pip install --no-index -f http://dist.plone.org/thirdparty/ http://dist.plone.org/thirdparty/ -U PIL-U PIL

Personal PyPiPersonal PyPi

https://github.com/benliles/djangopypihttps://github.com/benliles/djangopypi

PyPiPyPiMakefileMakefile integration: integration:

# --------- PyPi ----------# --------- PyPi ----------.PHONY: build.PHONY: buildbuild: envbuild: env<TAB>$(PY) setup.py sdist<TAB>$(PY) setup.py sdist

.PHONY: upload.PHONY: uploadupload: envupload: env<TAB>$(PY) setup.py sdist register upload<TAB>$(PY) setup.py sdist register upload

TestingTesting

TestingTesting

Add tests to your projectAdd tests to your project

Testing (2)Testing (2)

● use use doctestdoctest or or unittestunittest

Testing (3)Testing (3)Use Use nosenose to run: to run:

$ env/bin/nosetests$ env/bin/nosetests....--------------------------Ran 2 tests in 0.007sRan 2 tests in 0.007s

OKOK

Testing (4)Testing (4)MakefileMakefile integration: integration:NOSE := env/bin/nosetestsNOSE := env/bin/nosetests# --------- Testing ----------# --------- Testing ----------.PHONY: test.PHONY: testtest: nose depstest: nose deps<TAB>$(NOSE)<TAB>$(NOSE)

# nose depends on the nosetests binary# nose depends on the nosetests binarynose: $(NOSE)nose: $(NOSE)$(NOSE): env$(NOSE): env<TAB>$(PIP) install nose<TAB>$(PIP) install nose

GithubGithub

GithubGithub

Just a satisfied userJust a satisfied user

Why Github?Why Github?

Not bazaar nor mercurialNot bazaar nor mercurial

Why Github? (2)Why Github? (2)

Code is a first class objectCode is a first class object

GithubGithub

Don't check in keys/passwords!Don't check in keys/passwords!

http://ejohn.org/blog/keeping-passwords-ihttp://ejohn.org/blog/keeping-passwords-in-source-control/#postcommentn-source-control/#postcomment

A Branching StrategyA Branching Strategy

http://github.com/nvie/gitflowhttp://github.com/nvie/gitflow

Travis CITravis CI

Travis CITravis CI

CI (continuous integration) for GithubCI (continuous integration) for Github

Travis CI (2)Travis CI (2)

Illustrates power of (web) hooksIllustrates power of (web) hooks

5 Steps5 Steps

● Sign-in with githubSign-in with github● Sync repos on Profile pageSync repos on Profile page● Enable repoEnable repo● Create a Create a .travis.yml.travis.yml file on github file on github● Push a commit on githubPush a commit on github

travis.ymltravis.ymllanguage: pythonlanguage: pythonpython:python: - "3.3" - "3.3" - "2.7" - "2.7" - "2.6" - "2.6" - "pypy" - "pypy"# command to run tests# command to run testsscript: make testscript: make test

More InfoMore Info

http://about.travis-ci.org/http://about.travis-ci.org/

PoachplatePoachplate

poachplatepoachplate

Example of most of what we have talked Example of most of what we have talked about todayabout today

https://github.com/mattharrison/poachplathttps://github.com/mattharrison/poachplatee

That's allThat's all

Questions? Tweet or email meQuestions? Tweet or email mematthewharrison@gmail.commatthewharrison@gmail.com

@__mharrison__@__mharrison__http://hairysun.comhttp://hairysun.com