Darlene Goldstein

A Comparison of Microarray Platforms

NUS – IMS Workshop7 January 2004

Talk Outline

• Bioinformatics Core Facility at ISREC

• Purpose of study

• Platform technologies and study design

• Comparisons between platforms

• Conclusions and study completion

BCF: What is it ?• ISREC-based, supported by

the NCCR for molecular oncology, member group of the SIB

• Created by the NCCR

molecular oncology to assist its DAF (which is now absorbed into the DAFL) and its microarray users in their biomedical research

• A group devoted to the bioinformatics and statistical aspects of gene expression research, in particular to the analysis of data generated with microarray technologies

BCF

DAF NCCR biomedical

BCFDAFL &

BIOINF

microarray researchNCCR biomedical

bioinf. research

biostatisticsEPFL

BCF: Main Components

• Technical Support

– advice in experimental design and data analysis

– production, control, development of spotted arrays

– processing of microarray data, quality assessment

• Education

– practical training through classes / workshops

• Collaboration

– statistical data analysis of research projects

• Research & Development

– development / testing tools & methods

Platform Comparison Study• Purpose

– to assess accuracy and reproducibility of different gene expression platforms

– to compare features of different measurement types

– to understand the system (important for normalization and downstream analysis)

• Impact

– practical advice to DAF(L) and to NCCR microarray users

– benefit to wider scientific community, especially if possible to somehow combine results across array types

Platforms and Study Design• Platforms

– Affymetrix GeneChips, high-density short oligo arrays

– Agilent long oligo arrays– in-house spotted cDNA arrays– MPSS (massively parallel signature sequencing, a

digital gene expression technology patented by Lynx); in collaboration with the Ludwig Institute for Cancer Research; originally intended as ‘gold standard’

• Basic Design– 3 replicate measurements for two mRNAs (human

placenta and testis)– dye swap for two-color systems (Agilent, cDNA)– 2 to 3 million tags sequenced for MPSS

Methods• Experimental Method (as recommended by

‘specialists’):

– Affymetrix: Biozentrum Basel– Agilent: Institut Goustav Roussy, Paris– Spotted cDNA arrays: Otto Hagenbuechle's group

(DAF, now DAFL)– MPSS: Lynx (California), Victor Jongeneel's group

(LICR)– qRT-PCR followup (~ 250 genes), Robert Lyle,

Patrick Descombes (UniGE)

• Expression Quantificationas recommended by ‘specialists’ (above),but : RMA for Affymetrix

Spotted cDNA arrays

Human 10k Array 8x4 subarrays

Affymetrix GeneChips

Image of hybridized array

MPSS•Uses microbeads with

~100k identical DNA molecules attached

•Captures and identifies transcript sequences of expressed genes by counting the number of individual mRNA molecules representing each gene

•Individual mRNAs are identified through generated 17-to 20-base signature sequence

•Can use without organism sequence information

•‘MPSS can accurately quantify transcripts as low as 5 transcripts per million (tpm) to above 50,000 tpm’

(information from Lynx web site)

Other comparison studies (I)

• Yuen et al. 2002; Nuc. Acids Res. 30(10):e48– Affy MGU-74A, cDNA; cell lines; qRT-PCR 47 genes– both arrays sensitive (TP) and specific (TN) at identifying

regulated transcripts– found comparable rank-order of gene regulation, but only

modest correlation in fold-change– both array types biased downwards (FC under-estimated

compared to qRT-PCR)

• Evans et al. 2002; Eur. J. Neuroscience 16:409-413– Affy RG-U34A, SAGE to detect brain transcripts; 43 rat

hippocampi; evaluation based on 1000 transcripts– ~55% low, ~90% high abundance transcripts detected

Other comparison studies (II)

• Li et al. 2002; Toxicological Sciences 69:383-390– Affy HuGene FL, HGU-95Av2, IncyteGenomics UniGemV

2.0 (‘long cDNA’); drug-treated cell lines at 8h and 24h; qRT-PCR 9 genes

– cross-hyb contributed to platform discrepancies – found Affy ‘more reliable’ (sensitive)

• Kuo et al. 2002; Bioinformatics 18:405-412– Affy HU6800, cDNA, publicly available data on NCI 60;

2895 genes– found low correlation between measurements (but no

control over lab procedures – different groups had performed the original studies)

Other comparison studies (III)• Barczak et al. 2003; Genome Res. 13:1775-1785

– 2 versions of spotted long oligo (Operon), Affy HGU-95Av2; cell lines; 7344 genes

– this large-scale analysis found strong correlations between relative expression measurements

– similar results for amplified and unamplified targets

• Tan et al. 2003; Nuc. Acids Res. 31:5676-5684– Agilent Human 1, Affy HGU-95Av2, Amersham Codelink

UniSet Human I (30-mers); cell lines in serum-rich medium and 24h after serum removal; 2009 genes

– modest correlations– little overlap in genes called DE– best agreement on DE calls (varying criteria) only 21%

• comparison studies by other groups world-wide are also in progress

Comparison Principle• Cross-platform gene matching done through the

trome database of transcripts (constructed with the Transcriptome Analyzer program tromer)

• Use only those genes we classify as ‘reliably mapped’ between platforms (~2500 genes); we have not (yet) looked at probe(set)s that could not be well-mapped to known transcripts

• ‘Peak technical performance’ : this is a case study, not a systematic study; does not take into account normal user variation, other mRNAs, etc.

• Comparison based on M (log ratio) and A (average log intensity)

• Unfortunately, accuracy cannot be properly assessed, as true M values are not known

cDNA array Performance

MA plots (examples)

range

background

Affy U133A

NCCR h10kd

Agilent

|M| (putative effect) densities

(Difference in M) vs. A: reproducibility

Affy U133A

Agilent h1A

NCCR h10k

y = difference in M

x = average A

D (error) densities

Gene Matching

3365

549

435

2514

5797

5099 5977

AgilentAffy

NCCR

Probe(sets) / genes18325 Agilent h1A 1568824808 Affy U133A 14876 7812 NCCR h10k 6853

Gene matching also with MPSS

2494

--

-

--

Agilent

Affy

NCCR

2494 Tromer clusters4060 Affy probesets2869 Agilent probes2685 NCCR clones

Concordance in M density plots (I)

Agilent Affy NCCR

Concordance in M density plots (II)

Agilent Affy NCCR

Difficulty in comparing to MPSS ratios

selection on MPSS MPSS MPSS AFFY AGIL NCCRTESTIS PLAC M M ave M ave M ave genedescr

Affy upHTR005199_MPSS_1_AGIL_1_AFFY_1_NCCR_10 9 3.32 6.38 3.91 7.12 Fc fragment of IgE, high affinity I, receptor for; alpha polypeptideHTR000581_MPSS_1_AGIL_1_AFFY_1_NCCR_114 5117 8.41 6.45 4.07 5.91 cytochrome P450, family 19, subfamily A, polypeptide 1HTR000581_MPSS_1_AGIL_2_AFFY_1_NCCR_114 5117 8.41 6.45 1.34 5.91 cytochrome P450, family 19, subfamily A, polypeptide 1HTR004581_MPSS_1_AGIL_1_AFFY_1_NCCR_11 3635 10.83 6.89 4.12 4.84 pregnancy-associated plasma protein AHTR006015_MPSS_1_AGIL_1_AFFY_1_NCCR_19 9632 9.91 6.99 4.20 6.60 glycoprotein hormones, alpha polypeptideHTR000790_MPSS_1_AGIL_1_AFFY_2_NCCR_10 280 8.13 7.94 2.20 4.38 hydroxy-delta-5-steroid dehydrogenase, 3 beta- and steroid delta-isomerase 1HTR000790_MPSS_1_AGIL_1_AFFY_2_NCCR_20 280 8.13 7.94 2.20 5.69 hydroxy-delta-5-steroid dehydrogenase, 3 beta- and steroid delta-isomerase 1

Agil upHTR010250_MPSS_1_AGIL_1_AFFY_1_NCCR_14 211 5.41 5.87 4.56 6.15 Homo sapiens adrenomedullin (ADM), mRNA.HTR004842_MPSS_1_AGIL_1_AFFY_1_NCCR_16 1328 7.57 5.91 4.98 5.36 glypican 3HTR004414_MPSS_1_AGIL_1_AFFY_1_NCCR_10 6 2.81 1.58 5.15 4.56 estrogen-related receptor gammaHTR004414_MPSS_1_AGIL_1_AFFY_2_NCCR_10 6 2.81 4.02 5.15 4.56 estrogen-related receptor gammaHTR002717_MPSS_1_AGIL_1_AFFY_1_NCCR_10 828 9.70 6.06 6.17 7.04 insulin-like growth factor binding protein 1HTR002717_MPSS_1_AGIL_1_AFFY_1_NCCR_10 828 9.70 6.06 6.17 7.04 insulin-like growth factor binding protein 1

NCCR upHTR010250_MPSS_1_AGIL_1_AFFY_1_NCCR_14 211 5.41 5.87 4.56 6.15 Homo sapiens adrenomedullin (ADM), mRNA.HTR003344_MPSS_1_AGIL_1_AFFY_1_NCCR_10 726 9.51 0.02 1.75 6.23 placental growth factor, vascular endothelial growth factor-related proteinHTR003344_MPSS_1_AGIL_1_AFFY_2_NCCR_10 726 9.51 5.01 1.75 6.23 placental growth factor, vascular endothelial growth factor-related proteinHTR006015_MPSS_1_AGIL_1_AFFY_1_NCCR_19 9632 9.91 6.99 4.20 6.60 glycoprotein hormones, alpha polypeptideHTR002717_MPSS_1_AGIL_1_AFFY_1_NCCR_10 828 9.70 6.06 6.17 7.04 insulin-like growth factor binding protein 1HTR005199_MPSS_1_AGIL_1_AFFY_1_NCCR_10 9 3.32 6.38 3.91 7.12 Fc fragment of IgE, high affinity I, receptor for; alpha polypeptide

MPSS difficulties, another illustration

Correlations

MPSS AGILENT 1 AGILENT 2 AFFY NCCR 1 NCCR 2MPSS 1.00 0.43 -0.45 0.44 0.47 -0.47 MPSS

AGILENT 1 0.43 1.00 -0.97 0.65 0.72 -0.73 AGILENT 1AGILENT 2 -0.45 -0.97 1.00 -0.66 -0.73 0.73 AGILENT 2

AFFY 0.44 0.65 -0.66 1.00 0.72 -0.73 AFFYNCCR 1 0.47 0.72 -0.73 0.72 1.00 -0.98 NCCR 1NCCR 2 -0.47 -0.73 0.73 -0.73 -0.98 1.00 NCCR 2

MPSS AGILENT 1 AGILENT 2 AFFY NCCR 1 NCCR 2

first quartile (25% least frequent RNAs)

fourth quartile (25% most frequent RNAs)

MPSS AGILENT 1 AGILENT 2 AFFY NCCR 1 NCCR 2MPSS 1.00 0.72 -0.73 0.73 0.76 -0.76 MPSS

AGILENT 1 0.72 1.00 -0.98 0.77 0.79 -0.79 AGILENT 1AGILENT 2 -0.73 -0.98 1.00 -0.77 -0.79 0.80 AGILENT 2

AFFY 0.73 0.77 -0.77 1.00 0.81 -0.81 AFFYNCCR 1 0.76 0.79 -0.79 0.81 1.00 -0.98 NCCR 1NCCR 2 -0.76 -0.79 0.80 -0.81 -0.98 1.00 NCCR 2

MPSS AGILENT 1 AGILENT 2 AFFY NCCR 1 NCCR 2

Agreement: top up 200 (placenta)

96

2634

44

4840 30Agilent

Affy

NCCR

M range Affy: 1.66 - 7.94Agil: 1.48 - 6.17

NCCR: 1.83 - 7.12

Agreement: top down 200 (testis)

87

2634

53

4638 41Agilent

Affy

NCCR

M range Affy: -8.27 - -1.65Agil: -6.07 - -1.47

NCCR: -6.18 - -1.79

Comparison with MPSS, 99% CI (up)

Comparison with MPSS, 99% CI (Down)

MPSS CI Overlap

Overlap with the 99% CI for MPSS

Affy 31.23%Agilent 28.62%NCCR 30.52%

Overlap with the 99.9% CI for MPSS

Affy 41.53%Agilent 38.90%NCCR 40.52%

Overlap with MPSS

88

112

74MPSS

NCCR

38

missing or classified as unreliably mapped (tag to gene not unique)

(similar numbers also for Affy and Agilent);56 of the 88 are in common to all 4

Conclusions (I)•The three microarray platforms compared

performed very similarly in terms of which genes are detected as differentially expressed, distributions of M values, variability between replicate measurements ...

... so similarly that it seems hard to find real differences

•Most disagreement for low-expressed genes

•RMA M values (Affy) are better variance-stabilized, but reproducibility is good for all platforms except for weak signals in Agilent (likely due to bg treatment)

•RMA M values are more strongly compressed towards zero at low intensity; reduces false positive calls but might make DE at low intensity undetectable (but is it detectable at all?)

Conclusions (II)Microarrays vs MPSS

M values, quantitative comparison:

the disagreement is large ...

... so large that it is hard to reconcile the values, making it impossible to use MPSS as the ‘gold standard’

M values, qualitative comparison:

there is a good degree of agreement - approximately the same to all three

microarray platforms

Conclusions (III)• MPSS predicts many more low-abundance genes to

be (strongly) differentially expressed• The hybridization methods lose signal of low-

abundance genes (due to the background fluorescence estimation?)

• microarrays miss detection of most of the differential expression of low abundance transcripts, but it is also possible that MPSS is biased for many genes or less precise than this approach suggests

approach with confidence intervals for MPSS

(currently approximate CI that takes into consideration the sampling error on the counts, we have no replicated measurements for MPSS)

Completion of Study

Choose genes for qRT-PCR for which the platforms and MPSS disagree and (attempt to) address the questions:

• which platform is more accurate?• how does accuracy depend on the signal intensity?• do the microarrays miss DE frequently....?• ....and especially at weak signal intensity ?• which platform best detects low abundance RNAs?• does MPSS agree with QT-PCR?

• Suggestions are welcome !!

Acknowledgements

• Ludwig Institute for Cancer ResearchVictor Jongeneel, Christian Iseli, Brian

Stephenson

• DAF/DAFLOtto Hagenbuechle, Josiane Wyniger

• UniGERobert Lyle, Patrick Descombes

• BCF

Mauro Delorenzi, Eugenia Migliavacca

• and everyone I inadvertently left out!