EBI is an Outstation of the European Molecular Biology Laboratory.

Sumatra: an electronic labook. Git it right

Michele MattioniComputational Neurobiology groupmattioni@ebi.ac.uk

http://creativecommons.org/licenses/by-nc-sa/3.0/

Open questions which I had no answer:

• Where are my results?• Which parameters I used on that simulation?• Which one was that simulation?• Why it worked yesterday and not today?• At exactly which point it worked yesterday?• Where is my mind stuff?

Fro

m V

ale

r ie

Eve

r ett

's p

ho

tos t

rea

mht

tp:/

/ww

w. f

lick r

.co

m/p

ho

tos/

vale

rieb

b/3

00

63

48

55

0/

How the experimental people solve that?

• Write down on a labbook everything they do :• reagents• times• protocols• why!• what worked, what didn't• ...

Ma

rcin

Wic

ha

ry's

ph

ot o

stre

am

htt

p:/

/ww

w. f

lickr

.co

m/p

ho

tos/

mw

i ch

ary

/32

49

20

29

59

/

We need an electronic labbook

• Able to track (automatically) our 'experiments' details:• script launched• parameters used• reason• outcome

• But we can also:• search them• organize them• Rerun them!

Say hello to Sumatra

Slide from Andrew Davison (main developer),http://neuralensemble.org/meetings/talks/CodeJam3_Davison_Sumatra.pdf

http://neuralensemble.org/trac/sumatra

Sumatra: what I need to use it

• Code under version control• Accepted version control:

• Mercurial (hg)

• Subversion (svn)

• Git (git) (I patched it! Power of the opensource..)

• your choice (exstensible architeture)

• Scripts organized as Main class and parameter_file

smt Usage

$ cd myproject$ smt init MyProject

Normal way you launched the script:

$ python main.py default.param

New way using Sumatra:

$ smt run ­­simulator=python ­­main=main.py default.param

Shorter cmdline to launch the job

Configure it (it has to be done once.)$ smt configure ­­simulator=python ­­main=main.py

From$ smt run ­­simulator=python ­­main=main.py default.param

To$ smt run default.param

What if I use java?

$ java main default.param

$ smt run ­­simulator=java ­­main=main default.param

Parameter file? Excuse me?

dt = 0.05 # Dt for the integrationtstop = 10 # [ms]

Basically a name <-> parameter pair

Not obligatory, but it's very handy because:

• Sumatra can store your parameters in the record• If Sumatra understand them then You can change parameter on the fly

$ smt run default.param tstop=20 dt=0.01

Where is my stuff?

smt monitor a directory (default Data).• Any new file/image saved there is automatically added to the simulation record

You save the resultsSumatra remembers where is it

My way: I create a dir for each day and then save all my stuff in dedicated folder• Data/19­04­2010/Sim_1/, Data/19­04­2010/Sim_2/• Data/20­04­2010/Sim_1/, Data20­04­2010/Sim_2/

Access the results: smt$ smt list default_20100407­163228default_20100407­171249

$ smt list ­l Label        : default_20100420­170241Reason       : Testing the tagging from the cmdlineOutcome      : It works!Duration     : 4.38211798668Repository   : GitRepository at /home/mattions/Work/model/MSN/TimeScalesMain_File    : spineIntegration.pyVersion      : 8a61ecdd999e30d25c29620a87d9f4536aa67e9eExecutable   : Python (version: 2.6.4) at /usr/bin/pythonTimestamp    : 2010­04­20 17:02:41.445588Tags         : test, test2

Access the results: smtweb

Access the results: smtweb

Accessing the plot from the link

How to install

ATM: Git support only available from the latest code:

hg clone https://neuralensemble.org/hg/sumatra

git clone http://github.com/mattions/sumatra_on_github

How sumatra saved the day (yesterday)

Scenario:• Made some massive restructuring/refactoring on neuronvisio• Get lost on the way• Nothing worked anymore.

But ...

Searched the last simulation where it worked

Selected the latest working, and neuronvisio ver.

Find that commit on neuronvisio repo

Branched it

Merged restart back into master

kind of apply twice the same patch, but you know it was worth it.

Summary

• Sumatra can help you to:• keep your stuff organized• not lose your work• speed up the recovery time when somethig get wrong• remember what you used ages ago, to make that figure• be more focus on your science.

• You can help sumatra• Using it• Asking for request (ok, send a patch. It's quicker and easier)

Acknowledgement

• Andrew Davison (main sumatra developer)• people got the picture from (link on each picture)• Nicolas Le Novère (to let me work on this)• Computational Neurobiology group @ EBI.

Got questions?

http://neuralensemble.org/trac/sumatra