Simple Variant Identification under
ReComp controlJacek Cała, Paolo Missier
Newcastle UniversitySchool of Computing Science
Outline• Motivation – many computational problems, especially Big Data and
NGS pipelines, face an output deprecation issue• updates of input data and tools make current results obsolete
• Test case – Simple Variant Identification • pipeline-like structure, “small-data” process• easy to implement and experiment with
• Experiments• 3 different approaches compared with the baseline, blind re-computation• provide insight into what selective re-computation can/cannot achieve
• Conclusions
The heavyweight of NGS pipelines• NGS resequencing pipelines are an important example of the Big Data
analytics problems• Important:• Are at the core of the genomic analysis
• Big Data:• raw sequences for WES analysis are measured in 1-20 GB per patient• for quality purposes patient samples are usually processed in cohorts of 20-40
or close to 1 TB per cohort• time required to process a 24-sample cohort can easily exceed 2 CPUmonths• WES is only a fraction of what the WGS analyses require
Tracing change in the NGS resequencing• Although the skeleton of the pipeline remains fairly static, many aspects
of the NGS are changing continuously• Changes occur at various points and aspects of the pipeline but are
mainly two-fold:• new tools and improved versions of the existing tools used at various steps in
the pipeline• new updated reference and annotation data
• It is challenging to assess the impact of these changes on the output of the pipeline• the cost of rerunning the pipeline for all or even a cohort of patients is very high
ReComp• Aims to find ways to:• detect and measure impact of changes in the input data• allow the computational process to be selectively re-executed• minimise the cost (runtime, monetary) of the re-execution with the maximum
benefit for the user
• One of the first steps – to run a part of the NGS pipeline under the ReComp and evaluate potential benefits
The Simple Variant Identification tool• Can help classify variants into three categories: RED, GREEN, AMBER• pathogenic, benign and unknown• uses OMIM GeneMap to identify genes- and variants-in-scope• uses NCBI ClinVar to classify variants pathogenicity
• The SVI can be attached at the very end of an NGS pipeline• as a simple, short running process can serve as a test scenario for ReComp• SVI –> a mini-pipeline
High-level structure of the SVI process
Phenotype to genes
genes in scope
Variant selection
<< input data >>patient variants
(from a NGS pipeline)
<< input data >>phenotype hypothesis
<< reference data >>OMIM GeneMap
<< reference data >>NCBI ClinVar
<< output data >>classified variants
variants in scope
Variant classification
Simple Variant Identification
Detailed design of the SVI process
• Implemented as an e-Science Central workflow• graphical design approach• provenance tracking
Detailed design of the SVI process
Phenotype to genes
Variant selection
Variant classification
Patientvariants
GeneMap
ClinVar
Classified variants
Phenotypehypothesis
Running SVI under ReComp• A set of experiments designed to get insight into what and how
ReComp can help in process re-execution:1. Blind re-computation2. Partial re-computation3. Partial re-computation using input difference4. Partial re-computation with step-by-step impact analysis
• Experiments run on a set of 16 patients split by 4 different phenotype hypotheses• Tracking real changes in OMIM GeneMap and NCBI ClinVar
Experiments: Input data setPhenotype hypothesis Variant file Variant count File size [MB]Congenital myasthenic syndrome
MUN0785 26508 35.5MUN0789 26726 35.8MUN0978 26921 35.8MUN1000 27246 36.3
Parkinsons disease C0011 23940 38.8C0059 24983 40.4C0158 24376 39.4C0176 24280 39.4
Creutzfeldt-Jakob disease A1340 23410 38.0A1356 24801 40.2A1362 24271 39.2A1370 24051 38.9
Frontotemporal dementia - Amyotrophic lateral sclerosis
B0307 24052 39.0C0053 23980 38.8C0171 24387 39.6D1049 24473 39.5
Experiments: Reference data sets• Different rate of changes:• GeneMap changes daily• ClinVar changes monthly
Database Version Record count
File size [MB]
OMIM GeneMap
2016-03-08 13053 2.2
2016-04-28 15871 2.7
2016-06-01 15897 2.7
2016-06-02 15897 2.7
2016-06-07 15910 2.7NCBI ClinVar 2015-02 281023 96.7
2016-02 285041 96.6
2016-05 290815 96.1
Experiment 1: Establishing the baseline – blind re-computation• Simple re-execution of the SVI process evoked by changes in reference
data (either GeneMap or ClinVar)• Involves the maximum cost related to the execution of the process
• Blind re-computation is the baseline for the ReComp evaluation• we want to be more effective than that
Experiment 1: Results• Running the SVI workflow on one patient sample takes about 17
minutes• executed on a single-core VM• may be optimised –> optimisation out-of-scope at the moment
• Runtime is consistent across different phenotypes• Changes of the GeneMap and ClinVar version have negligible impact
on the execution time, e.g.:
Run time [mm:ss]
GeneMap version 2016-03-08 2016-04-28 2016-06-07
μ ± σ 17:05 ± 22 17:09 ± 15 17:10 ± 17
Experiment 1: Results• 17 min per sample => the SVI implementation has capacity of only 84
samples per CPUcore per day• May be inadequate considering the daily rate of change of GeneMap
• Our goal is to increase this capacity through smart/selective re-computation
Experiment 2: Partial re-computation• The SVI workflow is a mini-pipeline with well defined structure• Changes in the reference data affect different parts of the process
• Plan:• restart the pipeline from different starting points• run only the part affected by the changed data• measure the savings of the partial re-computation when compared with the
baseline, blind re-comp
Experiment 2: Partial re-computation
Change inClinVar
Change inGeneMap
Experiment 2: Results• Running the part of SVI directly involved in
processing updated data can save some runtime
• Savings depend on:• the structure of the process• the point where the changed data are used
• Savings involve the cost of retaining interim data required in partial re-execution• the size of the data depends on the
phenotype hypothesis and type of change• the size is in range of 20–22 MB for GeneMap
changes and 2–334 kB for ClinVar changes
Run time [mm:ss]
Savings Run time [mm:ss]
Savings
GeneMapversion 2016-04-28 2016-06-07
μ ± σ 11:51 ± 16 31% 11:50 ± 20 31%
ClinVarversion 2016-02 2016-05
μ ± σ 9:51 ± 14 43% 9:50 ± 15 42%
Experiment 3: Partial re-computation using input difference• Can we use difference between two versions of the input data to run the process?
• In general, it depends on the type of process and how the process uses the data• SVI can use the difference• Difference is likely to be much smaller than the new version of the data
• Plan:• calculate difference between two versions of reference data –> compute added, removed
and changed record sets• run SVI using the three difference sets• recombine results• measure the savings of the partial re-computation when compared with the baseline, blind
re-comp
Experiment 3: Partial re-comp. using diff.• The size of difference sets is significantly reduced when compared to the new version of the
data
but:• the difference is computed as three separate sets of: added, removed and changed records• it requires three separate runs of SVI and then recombination of results
GeneMap versionsfrom –> to
ToVersion rec. count
Differencerec. count Reduction
16-03-08 –> 16-06-07 15910 1458 91%
16-03-08 –> 16-04-28 15871 1386 91%
16-04-28 –> 16-06-01 15897 78 99.5%
16-06-01 –> 16-06-02 15897 2 99.99%
16-06-02 –> 16-06-07 15910 33 99.8%
ClinVar versionsfrom –> to
ToVersion rec. count
Differencerec. count Reduction
15-02 –> 16-05 290815 38216 87%
15-02 –> 16-02 285042 35550 88%
16-02 –> 16-05 290815 3322 98.9%
Experiment 3: Results• Running the part of SVI directly involved in
processing updated data can save some runtime• Running the part of SVI on each difference set
also saves some runtime• Yet, the total cost of three separate re-executions
may exceed the savings
• Concluding, this approach has a few weak points:• running the process on diff. sets is not always
possible• running the process using diff. sets requires output
recombination• total runtime may sometimes exceed the runtime of
a regular update
Run time [mm:ss]
Added Removed Changed Total
GeneMap change 11:30 ± 5 11:27 ± 11 11:36 ± 8 34:34 ± 16
ClinVar change 2:29 ± 9 0:37 ± 7 0:44 ± 7 3:50 ± 22
Experiment 4: Partial re-computation with step-by-step impact analysis• Insight into the structure of the computational process
+ Ability to calculate difference sets of various types of data=> step-by-step re-execution
• Plan:• compute changes in the intermediate data after each execution step• stop re-computation when no changes have been detected• measure the savings of the partial re-computation when compared with the
baseline, blind re-comp
Experiment 4: Step-by-step re-comp.• Re-computation evoked by daily update in GeneMap: 16-06-01 –> 16-06-02
• likely to have minimal impact on the results
• Only two tasks in the SVI process needed execution• Execution stopped after about 20 seconds of processing
Experiment 4: Results• The biggest savings in the runtime out of the three partial re-
computation scenarios• the step-by-step re-computation was about 30x quicker than the complete re-
execution
• Requires tools to compute difference between various data types• Incurs costs related to storing all intermediate data• may be optimised by storing only intermediate data needed by long running
tasks
Conclusions• Even simple processes like SVI can significantly benefit from selective re-
computation• Insight into the structure of the pipeline opens a variety of options how re-
computation can be pursued• NGS pipelines are very good candidates to optimise
• The key building blocks for successful re-computation:• workflow-based design• tracking data provenance• access to intermediate data• availability of tools to compute data difference sets