Principles, Practices, and Habits

Jeremy LeipzigDecember 18, 2010

Gearing up for a data deluge

• Technical• Scientific• Cultural– Version control– Documentation– Testing– Literate programming

Principles, Practices, and HabitsPrinciple Practice Habit

Transparency Open data Provide data behind plots

Transparency Open source All source code in SVN

Scalability Queueing Design for the cluster

Reproducibility Literate programming Sweave scripts, noweb

Integrity Documentation Method sections are provided by analysts

Longevity Softcoding Use configuration files. No magic variables

Sustainability Req spec Deliverables are specified in project site

Maintainability Central genomes Dynamic links where possible

Maintainability Documentation READMEs, Makefiles

Maintainability Project wiki/website We don’t email results to investigators

Results belong in reports, not emails

• Answer a question once• Constant improvement• Associate results with a set of stated

parameters

Literate programming

• Dynamic reports• Reproducible output• Marry code to reports• Marry data & configuration to reports• Reinforced sanity

Sweave

• R +• In R:

Sweave(“myReport.Rnw”,output=”myReport.tex”)

• In shell:echo "Sweave(\"myReport.Rnw\",output=\"myReport.tex\");" | R --no-save --quiet

• In shell:pdflatex myReport.tex

svn co http://versioncontrol.chop.edu/svn/cbmi/chipSeq/trunk/ chipSeqScripts

chromosomeBias.Rnw sample:

echo "configFile<-\"bias.config.R\";Sweave(\”chromosomeBias.Rnw\",output=\"chromosomeBias.tex\");" | R --no-save --quiet

bias.config.R

Versioning

• Subversion keyword substitution

svn add chromosomeBias.Rnwsvn propset svn:keywords 'Revision' chromosomeBias.Rnwsvn ci –m “added chromosomeBias sweave script”

http://svnbook.red-bean.com/en/1.5/svn.advanced.props.special.keywords.html

$Rev$