1. Lack Of Reproducibility Inp yBiomedical Research How A Common Quality QyStandard For Non- Non-regulated BiomedicalgResearch Could ChangeThe Trendlo Palm, MD, PhD, MBA, CMQ/OESenior Member of the American Society for Quality(ASQ)Food, Drugs & Cosmetics Division (FD&C) Marcus EvansSenior Vice President Clinical Operations & BiometricspEvolution Summit,Forest Research InstituteApril 30, 2012

2. Moores Law:Transistor Count Doubling Every Two YearsSource: Wikipedia 3. Erooms Law: Number Of New Drug ApprovalsPer Billion US Dollars Halved Every Nine YearsSource: Nature Reviews Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March2012, page 191 - 200 4. War On Cancer Going On For 40 Years WarCancer About$200billionspentoncancerresearchintheUSsince1971*US i 1971* Morethan1.5millionpaperspublished* Butdespitesomeprogressinpreventionandtreatment Almost1.5millioncancercasesand560 000cancerdeathsintheUnitedStatesin2009** CancernowthesecondleadingcauseofdeathintheUS** Nearly1in2menandmorethan1in3womenwill ybediagnosedwithcancerduringtheirlifetime** AvastinincreasessurvivalinmCRCpatientsby4.7months***atanannualtreatmentcostof$58.000****Source: * Fortune Magazine March 22, 2004, ** JAMA, March 17, 2010 - Vol 303, *** Avastin label, **** Reuters, Thu Jun 30, 2011 5. Some Causes Of DecliningProductivity Of Biomedical Research BiomedicalR&Dhasbecomemore complex* Regulatory hurdles have gone up* Regulatoryhurdleshavegoneup Industrializationofdrugdiscovery (focusonsingletargets)hasput (focus on single targets) has put R&Donanoveralllesseffective path path* Pharmaceuticalcompanieshave becometoobigtoinnovate bt bi t it*Source: Nature Reviews Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March2012, page 191 - 200 6. Could there be another, major, more fundamental root causeof the declining productivity of biomedical research? 7. Lack Of Reproducibility InBiomedical ResearchBi di l R h NewToolsandtechnologies,massiveamountsof data,longtermstudies,interdisciplinary d ldi i di i li approaches,andthecomplexityofquestionsbeing askedarecomplicatingreplicationefforts Anempiricalassessmentof18publishedpapersof A ii l f 18 bli h d f microarraystudiesshowedthatindependent analystscouldperfectlyreproducetheresultsof onlytwoofthestudies only two of the studies Thisisoneofmedicinesdirtyy secrets:Mostresults,including thosethatappearintopflight peerreviewedjournals,cantbe reproduced 8. Lack Of Reproducibility: Industry ExperienceAmgen* Publishedliteraturedescribedthatinhibitionoftheserine/threoninekinase33(STK33) destroyedcancercells AmgenlaunchedmassiveresearcheffortbutcouldnotreplicatetheresultspBayer* BayerreportedinSeptember2011thatithadhaltednearlytwo thirdsofitsearlydrugtargetprojectsbecausein housetwothirds of its early drug target projects because inhouseexperimentsfailedtomatchclaimsmadeintheliterature.Pfizer*** Pfizer announced January 2012 that it had to write off $750PfizerannouncedJanuary2012thatithadtowriteoff$750millionafterresultsofastudywithDimebonforAlzheimer,publishedoriginallyinthejournalLancet*,couldnotbereproducedreproducedSource: *WSJ, December 2, 2011; ** Fierce Biotech, January 17, 2012 9. Lack Of Reproducibility:Case St d 1 fCStudy from A d Academiai In2002paperpublishedinLancetbyauthorsfrom theFDAandNCIclaimedthatamassspectrometry methodcouldprovidehighlysensitiveandspecific diagnostictestsforovariancancerg NCIannouncedaClinicalProteomicsinitiativeand companieswereformedtotakeassaysbasedonthis methodtotheclinic method to the clinic IndependentanalysisbyMDAndersonresearcher demonstratedthattheresultsweredueto experimentalartifacts(runningofallofthecontrols beforeallofthecancers)Source: K i h A . BS Keith Baggerly and Kevin R . Coombesl dK i C bHandbook of Statistics in Clinical Oncology, Third EditionAntje Crowley and John HoeringChapman and Hall/CRC 2012 Pages 605618 10. Lack Of Reproducibility:Case St d 2 fCStudy from A d Academiai In2006apaperpublishedinNatureMedicinebyDukeUniversityprofessorAnilPotticlaimedthatmicroarraybasedsignaturesofdrugsensitivityderivedfromcelllinescouldpredictpatientresponsetospecificchemotherapeutics Discover magazinedesignatedthispaperoneofthetop100breakthroughsof2006g Largeclinicaltrialswerestartedusingthismethodology In2009,independentanalysisdemonstratedthatthedatawerewrongduetomislabelingandindexingerrorsd t d ti l b li di d iSource: Keith A . Baggerly and Kevin R . Coombes Baggerly KA, Coombes KR.Handbook of Statistics in Clinical Oncology, Third Deriving chemosensiti it Deri ing chemosensitivity from cellEditionlines: Forensic bioinformatics andAntje Crowley and John Hoering reproducible research in high-Chapman and Hall/CRC 2012 Pages 605618throughput biology. Ann Appl Stat 2009; 3(4):13091334. 11. Non-reproducible Research Is Noise And Not Knowledge Whatiftherecentrevelationsindicatethatmostofpublishedbiomedicalresearchisnoise? (Thisisoneofmedicinesdirtysecrets:Most results,includingthosethatappearintopflight peerreviewedjournals,can tbereproduced peerreviewed journals cant be reproducedWSJ, December 2, 2011 ) Whatisthenoise/knowledgeratioofthe22million /grecordsinMedline?30%?,50%?,.?? Howcanbiomedicalresearchworldwidebeproductivewhiletryingtoreproducenoise? d tihil t i td i ? Whatiftheamountofnoiseinmoderng g ybiomedicalresearchisbeginningtosuffocateanyproductiveresearch? 12. What are the reasons for thelack fl k of reproducibility ind ibilit imodern biomedical research? 13. Modern Science Is Based OnCollective I t lliC ll ti Intelligence Collectiveintelligencerequires: asharedbodyofknowledge,methods,andy g techniques,asharedpraxis* asharedandagreeduponqualitystandard howtoplan,conduct,andreportscientific work*Source: Reinventing Discovery The new era of networked science by Michael Nielsen, 2012 14. Modern Biomedical ResearchDoes not Have A Well DefinedQuality StandardGenesweremislabeledduetoanoffbyoneindexing G i l b l dd tff bi d i errorWeconcludedthatthemethoddidntactuallyworkat all;itonlyappearedtoworkduetopoorbookkeeping all; it only appeared to work due to poor bookkeepingAdisconnectbetweenthenumbersandthesample namesrenderedthepredictionsinvalidpoordocumentationallowederrorstogounnoticedpg untilafterthingshadproceededtoclinicaltrialsthemostcommonmistakespeoplemakearesimple onesIftheanalysesareopaque,thenthesimpleerrorsmay gounnoticed,andsimplemistakesarestillimportantSource: Keith A . Baggerly and Kevin R . Coombes, Handbook of Statistics in ClinicalOncology, Third Edition, Antje Crowley and John HoeringChapman and Hall/CRC 2012 Pages 605618 15. In 2000, The WHO Identified The Development Of A Common Quality Standard For Biomedical Research As AQ y Pressing Global NeedTheworldspopulationisfacingserious healthchallenges..thereisincreased demandfornewdrugsandnewprinciplesfor treatment..itisessentialthatbasicscientific (biomedical)researchasawhole,.be (bd l) hh lb conductedinaproperfashionusingprocesses thatminimizewasteofresourcesandreduce theneedforcostlyconfirmationand the need for costly confirmation and repetitionofworkalreadyperformedItishopedthatwideapplicationofthe I i h d hid li i f h qualitypracticesproposedinthishandbook willleadtocosteffective,accelerated discoveryresearchandwillultimatelybenefit discovery research and will ultimately benefit humanhealthSource: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006 16. The WHO Rationale For A Quality St d d I Biomedical Q lit Standard In Bidi l Research Tominimizewasteofresourcesandreducetheneedforcostlyconfirmationandrepetitionofworkalready d titi fk l dperformed To generate reliable data to ensure aTogeneratereliabledatatoensureasolidbasisfordecidingwhethertoinvestinfurtherdevelopmentofastrategyorin further development of a strategy orproduct QualitymeansbetterscienceQy 17. The Flow Of Research Activities From Planning To PublishingSource: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006 18. In 2009, a committee of the FD&CDivision of the American Society forQuality (ASQ) was charged withdeveloping a Quality Standard forBiomedical Research and DrugDevelopment in the USBestqualitypracticesforbiomedicalresearchand drugdevelopment 1. ASQTechnicalReport 2. ASQStandard 3. ISOStandard 3 ISO St d d 19. Best Quality Practices For Biomedical ResearchAnd Drug Development Development- Technical Report Content - Managementsystem g y Documentstorageg Organization TechnicalRequirements ProjectManagement TestEquipment QualityManagement Testmethods/MethodSystem Validation Documentation SamplingandChainofCustody Sampling and Chain of Custody Documentcontrol/ MaterialsppDocumentapprovaland LegalandEthical Legal and Ethicalissue: Considerations Documentchanges VendorSelectionand Qualification 20. The New Quality Standard for BiomedicalResearch and Drug Development Will Be Based esea c a d ug e e op e t e asedOn Current State-Of-The-Art Quality Practices State-Of-The- World Health Organization Handbook: Quality PracticesWorldHealthOrganizationHandbook:QualityPracticesforBiomedicalResearch ISO17025 BARQAGuidelinesforQualityinNonRegulatedScientificResearch ICHQ2ValidationofAnalyticalProcedures:TextandMethodology 21CFRPart58 ICHQ9QualityRiskManagement ISO 900ISO900x 21. CoCommittee Members ttee e be s GeorgeBernstein,MAI g, JuliMotika,Regeneron, gConsulting loPalm,ForestResearch RickCalabrese,Sartorius InstituteStedimSt di MichelePruett,Innovative KeithConerly,Sodexho ConsultantsGXP LiChung Huang Eli LillyLi ChungHuang,EliLilly Sandra R Storli Abbott Labs SandraR.Storli,AbbottLabs AliceKrumenaker, JohnSurack,ClemsonCorePharma,LLCUniversity RichardLombardi,Forest A.MarkTrotter,TrotterBiotechResearchInstitute Solutions JJuneMorita,ConsultantM it Clt t 22. Summary Declining productivity of Drug Development DecliningproductivityofDrugDevelopment PoorreproducibilityofBiomedicalResearch Lack of a common language, a common quality Lackofacommonlanguage,acommonquality standardinBiomedicalResearch Impactingmedicalprogressandhumanhealth WHOdevelopedHandbookonQualityPractices inBasicBiomedicalResearch ASQFD&CDivisiondevelopinganewQuality & d l l StandardforBiomedicalResearch&Drug DevelopmentintheUS Development in the US