curso de genómica - uat (vhir) 2012 - análisis de datos de ngs

NGS Data analysis http://ueb.vhir.org/NGS2012

Introduction to NGS(Now Generation Sequencing)

Data Analysis

Statistics and Bioinformatics Research GroupStatistics department, Universitat de Barelona

Statistics and Bioinformatics UnitVall d’Hebron Institut de Recerca

Alex Sánchez

Outline

• Introduction• Bioinformatics Challenges• NGS data analysis: Some examples and workflows

• Metagenomics, De novo sequencing, Variant detection, RNA-seq

• Software• Galaxy, Genome viewers

• Data formats and quality control

Introduction

Why is NGS revolutionary?

• NGS has brought high speed not only to genome sequencing and personal medicine,

• it has also changed the way we do genome research

Got a question on genome organization?

SEQUENCE IT !!!

Ana Conesa, bioinformatics researcher at Principe Felipe Research Center

NGS means high sequencing capacity

GS FLX 454(ROCHE)

HiSeq 2000(ILLUMINA)

5500xl SOLiD (ABI)

Ion TORRENT

GS Junior

454 GS Junior35MB

NGS Platforms Performance

454 Sequencing

ABI SOLID Sequencing

Solexa sequencing

Applications of Next-Generation Sequencing

Comparison of 2nd NGS

Some numbers

Platform 454/FLX Solexa (Illumina)AB SOLIDRead length ~350-400bp 36, 75, or 106 bp 50bpSingle read Yes Yes YesPaired-end Reads Yes Yes YesLong-insert (several Kbp) mate-paired reads Yes Yes NoNumber of reads por instrument run 5.00K >100 M 400MMax Data output 0.5Gbp 20.5 Gbp 20GbpRun time to 1Gb 6 Days > 1 Day >1 DayEase of use (workflow) Difficult Least difficult DifficultBase Calling Flow Space Nucleotide space Color sapce

DNA ApplicationsWhole genome sequencing and resequencing Yes Yes Yes

de novo sequencing Yes Yes YesTargeted resequencing Yes Yes Yes

Discovery of genetic variants ( SNPs, InDels, CNV, ...) Yes Yes YesChromatin Immunopecipitation (ChIP) Yes Yes YesMethylation Analysis Yes Yes YesMetagenomics Yes No No

RNA Applications Yes Yes YesWhole Transcriptome Yes Yes YesSmall RNA Yes Yes Yes

Expression Tags Yes Yes Yes

Bioinformatics challenges of NGS

I have my sequences/images. Now what?

NGS pushes (bio)informatics needs up

• Need for computer power• VERY large text files (~10 million lines long)

– Can’t do ‘business as usual’ with familiar tools such as Perl/Python.– Impossible memory usage and execution time • Impossible to browse for problems

• Need sequence Quality filtering• Need for large amount of CPU power

• Informatics groups must manage compute clusters• Challenges in parallelizing existing software or redesign of algorithms to work in a

parallel environment

• Need for Bioinformatics power!!!• The challenges turns from data generation into data analysis!• How should bioinformatics be structured

• Bigger centralized bioinformatics services? (or research groups providing service?)• Distributed model: bioinformaticians must be part of the temas. Interoperability?

Data management issues

• Raw data are large. How long should be kept?• Processed data are manageable for most people

– 20 million reads (50bp) ~1Gb

• More of an issue for a facility: HiSeq recommends 32 CPU cores, each with 4GB RAM

• Certain studies much more data intensive than other– Whole genome sequencing

• A 30X coverage genome pair (tumor/normal) ~500 GB• 50 genome pairs ~ 25 TB

So what?

• In NGS we have to process really big amounts of data, which is not trivial in computing terms.

• Big NGS projects require supercomputing infrastructures

• Or put another way: it's not the case that anyone can do everything.– Small facilities must carefully choose their projects to be scaled

with their computing capabilities.

Computational infrastructure for NGS

• There is great variety but a good point to start with:

– Computing cluster• Multiple nodes (servers) with multiple cores• High performance storage (TB, PB level)• Fast networks (10Gb ethernet, infiniband)

– Enough space and conditions for the equipment ("servers room")

– Skilled people (sysadmin, developers)• CNAG, in Barcelona: 36 people, more than 50% of them

informaticians

Alternatives (1): Cloud Computing

• Pros– Flexibility.– You pay what you use.– Don´t need to maintain a data center.

• Cons– Transfer big datasets over internet is

slow.– You pay for consumed bandwidth.

That is a problem with big datasets.– Lower performance, specially in disk

read/write.– Privacy/security concerns.– More expensive for big and long

term projects.

Alternatives (2): Grid Computing

• Pros– Cheaper.– More resources available.

• Cons– Heterogeneous

environment.– Slow connectivity (specially

in Spain).– Much time required to find

good resources in the grid.

In summary?

•“NGS” arrived 2007/8

•No-one predicted NGS in 2001 (ten years ago)

•Therefore we cannot predict what we will come up against

•TGS represents specific challenges

–Large Data Storage

–Technology-aware software

–Enables new assays and new science

•We would have said the same about NGS….

•These are not new problems, but will require new solutions

•There is a lag between technology and software….

Bioinformatics and bioinformaticians

• The term bioinformatician means many things • Some may require a wide range of skills • Others require a depth of specific skills • The best thing we can teach is the ability to learn and

adapt • The spirit of adventure • There is a definite skills shortage • There always has been

Increasing importance of data analysis needs

NGS data analysis

NGS data analysis stages

Quality control and preprocessing of NGS data

Data types

Why QC and preprocessing

• Sequencer output:– Reads + quality

• Natural questions– Is the quality of my sequenced

data OK?– If something is wrong can I fix it?

• Problem: HUGE files... How do they look?

• Files are flat files and big... tens of Gbs (even hard to browse them)

Preprocessing sequences improves results

How is quality measured?

• Sequencing systems use to assign quality scores to each peak• Phred scores provide log(10)-transformed error probability values:

If p is probability that the base call is wrong the Phred score isQ = .10·log10p

– score = 20 corresponds to a 1% error rate– score = 30 corresponds to a 0.1% error rate– score = 40 corresponds to a 0.01% error rate

• The base calling (A, T, G or C) is performed based on Phred scores.

• Ambiguous positions with Phred scores <= 20 are labeled with N.

Data formats

• FastA format (everybody knows about it)– Header line starts with “>” followed by a sequence ID– Sequence (string of nt).

• FastQ format (http://maq.sourceforge.net/fastq.shtml)– First is the sequence (like Fasta but starting with “@”)– Then “+” and sequence ID (optional) and in the following line are

QVs encoded as single byte ASCII codes• Different quality encode variants

• Nearly all downstream analysis take FastQ as input sequence

The fastq format

• A FASTQ file normally uses four lines per sequence. – Line 1 begins with a '@' character and is followed by a sequence

identifier and an optional description (like a FASTA title line). – Line 2 is the raw sequence letters. – Line 3 begins with a '+' character and isoptionally followed by the same

sequence identifier (and any description) again. – Line 4 encodes the quality values for the sequence in Line 2, and must

contain the same number of symbols as letters in the sequence.• Different encodings are in use• Sanger format can encode a Phred quality score from 0 to 93 using ASCII 33 to 126

@Seq description

GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT

!''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65

Some tools to deal with QC

• Use FastQC to see your starting state.

• Use Fastx-toolkit to optimize different datasets and then visualize the result with FastQC to prove your success!

• Hints: – Trimming, clipping and filtering may improve quality– But beware of removing too many sequences…

Go to the tutorial and try the exercises...

Applications

• [1] Metagenomics• [2] De novo sequencing• [3] Amplicon analysis• [4] Variant discovery• [5] Transcriptome analysis• …and more …

[1] Metagenomics &other community-based “omics”

Zoetendal E G et al. Gut 2008;57:1605-1615

[1] A metagenomics workflow

Gene prediction

Binning

AAGACGTGGACA

CATGCGTGCATG

AGTCGTCAGTCATGGG

GTCCGTCACAACTGA

Short reads (40-150 bps)

AAGACGTGGACAGATCTGCTCAGGCTAGCATGAAC

Contigs

GATAGGTGGACCGATATGCATTAGACTTGCAGGGC

1 3000 6000

Proteins, families, functions

1 3000 6000

Functional profiles

1 2000

Sequences into species

Assembly

Homology searching

Functional classificationOntologies

[1] Metagenomic Approaches

SMALL-SCALE: 16S rRNA gene profilingThe basic approach is to identify microbes in a complex community by exploiting universal and conserved targets, such as rRNA genesPetrosini.

LARGE-SCALE: Whole Genome Shotgun (WGS)Whole-genome approaches enable to identify and annotate microbial genes and its functions in the community.

Environmental Shotgun Sequencing (ESS).A primer on metagenomics.

PLoS Comput Biol. 2010 Feb 26;6(2):e1000667.

Challenges and limitations: Chimeric sequences caused by PCR amplification and sequencing errors.

Challenges and limitations: relatively large amounts of starting material requiredpotential contamination of metagenomic samples with host

genetic materialhigh numbers of genes of unknown function.

[1] Comparative Metagenomics

Other software based on phylogeneticdata are UniFrac.

MEGAN can also be used to compare the OTU composition of two or more frequency-normalized samples.MG-RAST provides acomparative functional and sequence-based analysis for uploaded samples

Comparing two or more metagenomes is necessary to understand how genomic differences affect, and are affected by the abiotic environment.

[1] Some Metagenomics projects

"whole-genome shotgun sequencing" 78 million base pairs of unique DNA sequence were analyzed

"whole-genome shotgun sequencing" was applied to microbial populationsA total of 1.045 billion base pairs of nonredundant sequence were analyzed

To date, 242 metagenomic projects are on going and 103 are completed (www.genomesonline.org).

[2] De novo sequencing

[3] Amplicon analysis

Each amplicon (PCR product) is sequenced individually, allowing for the identification of rare variants and the assignment of haplotype information over the full sequence length

Some applications:● Detection of low-frequency (<1%) variants in complex mixtures

→ rare somatic mutations, viral quasispecies... Ultra-deep amplicon sequencing

● Identification of rare alleles associated with hereditary diseases, heterozygote SNP calling... Ultra-broad amplicon sequencing

● Metabolic profiling of environmental habitats, bacterial taxonomy and phlylogeny 16S rRNA amplicon sequencing

[3] Example of raw data generation with GS-FLX

[3] Data Workflow

[3] Final output examples

Bar plots output example (with circular legend for the AA)

NT substitution (error) matrices

AA frequency tables

[4] Variant discovery

Your aligner decides the type/amount of variants you can identify

Naive SNP callingReads counting

Statistic support SNP callingMaximum likelihood, Bayesian

Quality score recalibrationRecalibrate quality score from whole alignment

Local realignment around indelsRealign reads

Known variants (limited species)dbSNP

[4] Example: Exome Variant Analysis

[4] Genotype calling tools

[4] GATK pipeline

[4] Many ongoing sequencing projects

[5] Transcriptome Analysis using NGS

RNA-Seq, or "Whole Transcriptome Shotgun Sequencing" ("WTSS") refers to use of HTS technologies to sequence cDNA in order to get information about a sample's RNA content.

Reads produced by sequencing

Aligned to a reference genome to build transcriptome mappings.

[5] Applications (1) Whole transcriptome analysis

AAAAmRNAFragmentation

cDNA library

sequencing

Detects expression of known and novel mRNAs

Identification of alternative splicing events Detects expressed SNPs or mutations Identifies allele specific expression patterns

[5] Applications (2) Differential expression

1.Reads are mapped to the reference genome or transcriptome

2.Mapped reads are assembled into expression summaries (tables of counts, showing how may reads are in coding region, exon, gene or junction);

3.The data are normalized;

4.Statistical testing of differential expression (DE) is performed, producing a list of genes with P-values and fold changes.

[5] RNA Seq data analysis - Mapping

•Main Issues:–Number of allowed mismatches–Number of multihits–Mates expected distance–Considering exon junctions

End up with a list of # of reads per transcript

These will be our (discrete) response variable

• Two main sources of bias– Influence of length: Counts are proportional to the transcript

length times the mRNA expression level.– Influence of sequencing depth: The higher sequencing depth, the

higher counts.

• How to deal with this– Normalize (correct) gene counts to minimize biases.– Use statistical models that take into account

length and sequencing depth

[5] RNA Seq data analysis -Normalization

[5] RNA Seq - Differential expression methods

• Fisher's exact test or similar approaches.

• Use Generalized Linear Models and model counts using – Poisson distribution.– Negative binomial distribution.

• Transform count data to use existing approaches for microarray data.

• …

[5] Advantages of RNA-seq Unlike hybridization approaches does not require existing genomic

sequence Expected to replace microarrays for transcriptomic studies

Very low background noise Reads can be unabmiguously mapped

Resolution up to 1 bp High-throughput quantitative measurement of transcript abundance

Better than Sanger sequencing of cDNA or EST libraries Cost decreasing all the time

Lower than traditional sequencing Can reveal sequence variations (SNPs) Automated pipelines available

Software for NGS preprocessing and analysis

Which software for NGS (data) analysis?

• Answer is not straightforward.• Many possible classifications

– Biological domains• SNP discovery, Genomics, ChIP-Seq, De-novo assembly, …

– Bioinformatics methods• Mapping, Assembly, Alignment, Seq-QC,…

– Technology• Illumina, 454, ABI SOLID, Helicos, …

– Operating system• Linux, Mac OS X, Windows, …

– License type• GPLv3, GPL, Commercial, Free for academic use,…

– Language• C++, Perl, Java, C, Phyton

– Interface• Web Based, Integrated solutions, command line tools, pipelines,…

http://seqanswers.com/wiki/Software/list

NGS Data analysis http://ueb.ir.vhebron.net/NGS

Which software for NGS (data) analysis?

• Answer is not straightforward.• Many possible classifications

– Biological domains• SNP discovery, Genomics, ChIP-Seq, De-novo assembly, …

– Bioinformatics methods• Mapping, Assembly, Alignment, Seq-QC,…

– Technology• Illumina, 454, ABI SOLID, Helicos, …

– Operating system• Linux, Mac OS X, Windows, …

– License type• GPLv3, GPL, Commercial, Free for academic use,…

– Language• C++, Perl, Java, C, Phyton

– Interface• Web Based, Integrated solutions, command line tools, pipelines,…

http://seqanswers.com/wiki/Software/list

Some popular tools and places

Galaxy Site

http://galaxy.psu.edu/

Obtain data from many data sources including the UCSC Table Browser,BioMart, WormBase,

or your own data.

Prepare data for further analysis by rearrangingor cutting data columns, filtering data and many

other actions.

Analyze data by findingoverlapping regions,

determining statistics, phylogenetic analysis

and much more

contains links to the downloading,

pre-procession and analysis tools

displaysmenus and data inputs

Shows the history of analysis steps, data and resultviewing

RegisterUser

Click Get Data

Get Data from Database

Upload File File Format

Upload or paste file

68 NGS Data analysis http://ueb.vhir.org/NGS2012

FASTQ file manipulation: format conversation,summary statistics,

trimming reads,filtering reads

by quality score…

Input: sanger FASTQOutput: SAM format

Downstream analysis:SAM -> BAM

Copyright OpenHelix. No use or reproduction without express written consent72

List saved histories andshared histories.

Work on a current history, create new, share workflow

Creates a workflow, allowsuser to repeat analysisusing different datasets.

DATA VISUALIZATION

Why is visualization important?

make large amounts of data more interpretableglean patterns from the datasanity check / visual debuggingmore…

History of Genome Visualization

1800s 1900s 2000s

What is a “Genome Browser”

linear representation of a genomeposition-based annotations, each called a track

continuous annotations: e.g. conservationinterval annotations: e.g. gene, read alignmentpoint annotations: e.g. SNPs

user specifies a subsection of genome to look at

Server-side model(e.g. UCSC, Ensembl, Gbrowse)

• central data store• renders images• sends to client

server

client• requests images• displays images

Client-side model(e.g. Savant, IGV)

• stores dataserve

client• local HTS store• renders images• displays images

HTS machine

Rough comparison of Genome Browsers

UCSC Ensembl GBrowse Savant IGV

Model Server Server Server Client Client

Interactive

HTS support

Database of tracks

Plugins

No support Some support Good support

Limitations of most genome browsersdo not support multiple genomes simultaneouslydo not capture 3-dimensional conformationdo not capture spatial or temporal informationdo not integrate well with analyticscannot be customized

The SAVANT GENOME BROWSERhas been createdto overcome these limitations

Integrative Genomics Viewer (IGV)

he Integrative Genomics Viewer (IGV) is a high-performance visualization tool for interactive exploration of large, integrated datasets. It supports a wide variety of data types including sequence alignments, microarrays, and genomic annotations.

Acknowledgements Grupo de investigación en Estadística y Bioinformática del

departamento de Estadística de la Universidad de Barcelona.

All the members at the Unitat d’Estadística i Bioinformàtica del VHIR (Vall d’Hebron Institut de Recerca)

Unitat de Serveis Científico Tècnics (UCTS) del VHIR (Vall d’Hebron Institut de Recerca)

People whose materials have been borrowed or who have contributed with their work Manel Comabella, Rosa Prieto, Paqui Gallego, Javier

Santoyo, Ana Conesa, Thomas Girke and Silvia Cardona.…

Gracias por la atención y la paciencia

curso de genómica - uat (vhir) 2012 - análisis de datos de ngs

Education

ngs applications i (ueb-uat bioinformatics course - session...

codi: revisió: ite-microscopi de vhir-uat-ite-003 ... ·...

brief overview to amplicon variant analysis (ueb-uat...

curso de genómica - uat (vhir) 2012 - tecnologías de...

cvn - antonio rodríguez sinovas - vhir

medicina genómica: comprendiendo la ingeniería genómica

impronta genÓmica 2015

vall d’hebron institut de recerca (vhir) hospital …...1...

ngs applications ii (ueb-uat bioinformatics course - session...

série uat-600 dispositivo de localização de instalações...

secuenciación genómica bacteriana

fonaments de la imatge digital - vhir · 2013. 7. 23. ·...

vall d'hebron research institute (vhir)

genómica proteómica

course vhir-ucts-ueb - session 3 - statistical analysis

tiki for --dummies-- smarties at vhir

curso de genómica - uat (vhir) 2012 - rt-qpcr

storing and accessing information. databases and queries...

genómica microbiana

information management at vhir ueb using tiki-cms