a task-based approach to gene ontology evaluation
TRANSCRIPT
A Task-Based Approach to Gene Ontology
Evaluation
Erik Clarke, Benjamin Good, and Andrew SuThe Scripps Research Institute
Bio-Ontologies SIG – ISMB – July 2012
Monday, July 16, 12
2006
mitotic cell cyclesecretory pathwayubiquitin cycleRNA processingvesicle-mediated transportregulation of cell cycleintracellular protein transportmRNA metabolic process
interphasenuclear divisionmitotic cell cycleinterphase of mitotic cell cyclecell divisionmitosisorganelle fissionangiogenesis
!
2012Monday, July 16, 12
What happened?
Monday, July 16, 12
% of terms in top* 100 of both years
25
50
75
100
2004 2005 2006 2007 2008 2009 2010 2011 2012
11%16% 18% 20%
32% 35% 38%
59%
100%
perc
enta
ge
year
*top ranked terms by lowest p-value
Monday, July 16, 12
This shows the percentage of terms in the top 100 each year (ranked by p-value) that appear in the top 100 for 2012.This is from a real dataset! Note the significant change occurring after 2010: we are clearly in a state of flux
100000
200000
300000
400000
2004 2005 2006 2007 2008 2009 2010 2011 2012
Total IEAs
Human GO Annotations
Monday, July 16, 12
run by Uniprot since 2004grown by more than 200k since thenthe GO has also been changing significantly during this timethese factors contribute to our researcher’s differing results
With all this work, are things improving?
And how can we tell either way?
The Problem
Monday, July 16, 12
Define improvement: the ability of the GO and annotations to give us relevant, *accurate* results when we use them
• Depth of terms?
• Number of annotations?
• Evidence codes?
• Other “meta-analyses”?
• Ex: GAQ [1]: annotation quality = evidence code x depth in ontology
[1]: Buza et. al. Nuc. acids research, 2008doi: 10.1093/nar/gkm1167
Monday, July 16, 12
The truth is that you could build a totally useless ontology that scores well with these ad-hoc metrics
Instead...
Monday, July 16, 12
Porzel, R. and Malaka, R. A Task-based Approach for Ontology Evaluation, 2004
Ontology Application
performance results
...evaluate performance
Monday, July 16, 12
Enrichment Analysis
Monday, July 16, 12
2004 20122005 2006 2007 2008 2009 2010 2011
Gene Annotations
Gene Ontology +
Enrichment Analysis
p-value scores
Monday, July 16, 12
Use the results from each year’s GO + annotations to evaluate that year’s relative performance
1. Identify a term or area of interest2. Find datasets that should express the term(s)3. Run an enrichment analysis for each version of
the ontology and annotations under test4. Plot the change in p-values over each version
1E-06
1E-05
1E-04
1E-03
1E-02
1E-01
1E+00
enri
chm
ent
p-va
lue
(log
scal
e)
some ontology aspect under test
term of interest
Monday, July 16, 12
Brain tumor dataset: GDS1962
• Samples of different types of brain tumors
• Glioblastomas are known to be highly angiogenic
• Do we see “angiogenesis” as an enriched term with current GO+annotations?
• Using GOAs from 2004-12, do we see improvement in p-values and/or rank?
Monday, July 16, 12
1E-06
1E-05
1E-04
1E-03
1E-02
1E-01
1E+00
2004 2005 2006 2007 2008 2009 2010 2011 2012
enri
chm
ent
p-va
lue
(log
scal
e)
year
Enrichment of angiogenesis in glioblastomas subset
Monday, July 16, 12
- Note the 100,000x difference in p-values- So we know that GOA is getting better at describing this dataset, and we can imagine pulling those terms for many datasets across many fields to get a broader picture
1.E-25!
1.E-21!
1.E-17!
1.E-13!
1.E-09!
1.E-05!
1.E-01!2004! 2005! 2006! 2007! 2008! 2009! 2010! 2011! 2012!
Enric
hmen
t (p-
valu
e)
Year!
1.E-25!
1.E-21!
1.E-17!
1.E-13!
1.E-09!
1.E-05!
1.E-01!2004! 2005! 2006! 2007! 2008! 2009! 2010! 2011! 2012!
Enric
hmen
t (p-
valu
e)
Year!
Top ranked for 2006
Top ranked for 2012
(GDS1962:glioblastomas vs rest)
Monday, July 16, 12
Suggestion of a trend here: The decreasing p-values for the 2012 terms suggest that they are in fact more biologically accurate than those from 2006, or at least that the annotations and/or ontology structure is narrowing in on these particular terms
• We’re doing a mass analysis of > 200 GEO datasets
• Task-based analysis across representative sample of terms
• Analyzing trends of top-ranked terms across time
[historical annotations]
[enrichment analysis]
Monday, July 16, 12
Do we see the same convergence towards 2012 p-values for many other datasets?
A tool to evaluate potential annotations
Monday, July 16, 12
• We can evaluate:
• Natural language processing results
• New methods of electronic inference
• Crowdsourced annotations
Monday, July 16, 12
Example:
Monday, July 16, 12
% of terms in top 100 of both years
25
50
75
100
2004 2005 2006 2007 2008 2009 2010 2011 2012
perc
enta
ge
year
With “helpful” candidate annotations
Baseline
With “bad” candidate annotations
Monday, July 16, 12
We take our candidate annotations and insert them into a set of annotations from years past. Does it improve our coverage? You can imagine other ways of measuring its delta relative to 2012
• First method that evaluates the GO based on effectiveness at a task
• Demonstrated the GO/ human annotations are improving
• Shown sensitivity of EA to gene set composition and ontology structure
• Broad-scale analysis of the GO underway
• Created tool to evaluate candidate annotations using historical EA+GOA results
With many thanks to Ben Good, Andrew Su, and the Su Lab @ The Scripps Research Institute, and to BMC for travel support
Monday, July 16, 12
• Contact:
• @pleiotrope (twitter)
• http://github.com/eclarke/go-historical-analysis
Monday, July 16, 12